That is a posibilty, but IMHO its worth the work.
The push rods and rocker arm assemblies wearing down is very common on the 4.0 motor. More so than the lifters.. When we tore down our motor at 355k miles more than 1/2 of the cups in the rocker arms were worn way down and more than 6 of the push rods had the tips worn down.
We reused 10 of the lifters from our motor and took 2 from my parts motor to replace the two that had the plastic guides on the top of the lifter broken.
Soo, in short, I used all used (I soaked them in atf for a day or so while working on the motor) lifters and new rocker arm and pushrods and have 0 tap now.
Also, changing the rockers and push rods is very little work.. Pull the valve covers, pull 3 bolts per rock arm assembly, pull off rocker arm assembly and push rods... Install new stuff and call it a day..
I can have them done in 2 about hours.
~Mark