there was a third terminal on the back of the switch, if i have the 2 ends of the brown wire going to the wrong terminals would that do it? im going to now use a relay to see if i can make it work, the red power cord is coming from the existing 4x4 harness. does it need to be cut from the harness or how does that work out?
Your switch with three terminals is likely a SPDT (Single Pole Double Throw which is on one way and on the other way).
You only need a SPST (Single Pole Single Throw which is on one way and off the other)
Your SPDT will work just fine if you connect one of the brown wires to the middle and the other to one of the ends. The terminal at the other end of the switch would not have anything hooked up to it. This turns your on - on switch to a on - off switch which is all we need.
There is no need for the whole relay business unless you want an indicator light to turn on next to your switch.
Some switches have an indicator light build in. My switch has a tiny red dot indicating I am in 4X4 mode. The switch is oriented so when I push the switch up the red light comes on and I am in the factory 4x4 mode. (Auto 4x4hi 4x4lo as per my factory switch)
I run 99 percent of the time with switch down in 2x4 mode. The 2x4 mode is great for backing my trailer up a steep driveway to its home.
I suggest you take your switch with nothing hooked up to it and check out what is happening with an ohm meter.
If you do not have a ohm meter handy you can try the following.
I trust you extended your brown wire ends up to behind the glove box / dash area. Just try to twist them back together and cover them with back tape. If this does not get your 4x4 back working you may have done damage by grounding the brown wire. Maybe a fuse got blown.
Try to diagnose the switch and the brown wire / 4x4 system as per above before you try the relay method. Then once it is working you can do the relay hookup if you still want to.