Yeah I would definitely suspect the steering link as the culprit.
And I have to correct myself about the steering link because now I see you're running a linked suspension with a panhard bar. So, earlier I said you want a horizontal steering link -- but this is only 100% true for a leaf-sprung suspension and partially true for a linked suspension + panhard.
With your suspension, a linked suspension + panhard, what you want is a steering link that is the same length as the panhard and the same angle (so that they are parallel). This way, when the panhard pushes the axle sideways (which it does because the panhard rotates on an arc as the axle moves up/down), the steering link undergoes the same exact arc and angle change, which will keep your steering wheel about straight.
If you're going to redo the steering, also redo the panhard because what you want, since you're running a linked suspension + panhard, is a panhard that is (in parallel to what I said earlier) as horizontal as possible. What this does is minimize the amount of lateral (side-side) movement the axle does when the suspension moves up/down. But, with a horizontal panhard, you would also need a horizontal steering link -- which would require you to go hi-steer. And by the way, a panhard that is pushed up as far vertically as possible (but still allowing for movement and not hitting things like the oil pan) has better off-roading characteristics -- such as a better roll-axis which decreases the likelyhood of your vehicle rolling over on its side.