Might want to check your brake fluid. I had the same problem, expecially when X engine compartment was warm/hot. (Typically after driving a while during summer months.) After changing the brake fluid my pedal is smooth again.
It seems that after the brake fluid ages it no longer is a good lubricant for the master cylinder.