Sometimes...

... the subject, of a candid shot, looks suspiciously at the camera.

I find the best thing to do is carry on fiddling with the camera, until they have lost interest.

I also very rarely use my camera viewfinder for this type of shot. I use the tilted rear touch screen at waist level to fire the shutter.