If Google hints at something, you should pay attention.
Plug in a site and the tool will analyze its performance then give you feedback and detailed suggestions how to speed it up. Thanks Google!
It makes sense that if a site is slow to load, the user gets annoyed and bounces away; so if a site repeatedly has a user high bounce rate, then the site will be punished with lower organic rankings.
Remember, Google just wants to make us happy, and it wants to make me (and my short attention span) happy by not making us wait.
Give it test drive, how quickly does your site load?