This issue has come back after a year... but that's not much a surprise considering the quick fix. It seem certain that the brake fluid temperature has big effect on this since if the car is cool the brake pedal feels smooth. If the car has been driven a while and the engine compartment (and brake fluid) is hot, the issue comes up.
I'm pretty sure the issue is with the master cylinder and the hot brake fluid not acting as good lubricant. Maybe if I let a shop replace the brake fluid with new better quality one it should help? Any suggestions? Or is it waste of time/money to even try this?
I think there was some service bulletin for this. My X is already out of warranty, so I doubt they will fix it for free. (Maybe have to wait if they ever recall this?)