Well, I had a fair bit of crud in my transmission pan, as well there was silver "sparkles" in the pan. That was quite disturbing too. My trans wouldn't shift into OD and the TQ wouldn't lock. I installed the VB upgrades & the two kits as per Glacier's thread (again, many thanks to Glacier) and I installed a remote filter system. Works like a top now. Ah, that is, after I replaced the governor.
There's no doubt that the poor beast got hot if your brakes were dragging that bad. I still think I'd do the VB rebuild, flush the cooler and transmission, remote filter and then see how it works. If you are ready to do a rebuild, these are things that you will most likely do anyway.
If you do the VB rebuild and are going to install the governor spring improvement that comes with one of the kits, I highly recommend that you replace the governor body (stem as it's sometimes called) as it will cause you trouble eventually anyway.
It's not a transmission, it's an adventure!!