Pictures would help.
Not using the parking brake is a bad thing, as the cables can rust to the sheath and become completely seized, which then causes them to be a limiting factor in how much the rear brake shoes can move. The only fix is new cables, or disconnecting the cables and not having a parking brake, which isn't a good idea.
I would suggest checking out the brake cables, and later, possibly engaging the parking brake with the transmission in neutral and, in a safe area, see how well it holds. If you can do your own brake work, or at least take off the wheels and drums, that would be another good idea, to inspect the condition of the springs and mechanisms inside the drum brakes, along with the parking brake cable and associated connections. Operating the parking brake cable with the drums off can show you whether or not the cable, even if it's moving, is still so badly rusted that it's either not allowing full engagement of the parking brake, and/or is interfering with operation of the drum brakes.
While the rear brakes really only contribute stopping force to keep the back end of the vehicle from going to one side or the other, they do serve a function and the front brake pads will wear faster with rear brakes that aren't operating properly. I'd also say a parking brake that doesn't fully engage is a safety concern, as it can be the last resort in the case of a hydraulic failure when the vacuum reserve runs out, or a way to control a back end that is coming around during a tricky stop.
This isn't to say for sure that your vehicle has these issues, but it is something to check for if the parking brake is not normally used, especially in climates that see ice and snow. It is a good idea to get into the habit of engaging the parking brake anytime the vehicle is parked on a slope, as the pin for the auto transmission isn't designed to hold a vehicle like the brake, even though people use it that way all the time. This use of the parking brake, even just once in awhile, keeps the cables and hardware in working order and prevents the problems detailed above.