Before the thermals

I was talking to a balloonist a couple of years back and he was saying that they don’t fly early to avoid turbulence but rather to avoid the thermals, which can really mess up their flight trajectory of they get caught in lift when they want to go down and sink when they want to go up.