There should be something called a "weather shield" behind the door trim panel. It's basically a thick piece of plastic that's glued to the door. It purpose is to keep cold air and water from getting into the interior. If it's been removed (or if it's no longer glued to the door) cold air will blow through the trim panel.