The Court House - horrible photographic experience

I went into town to use the cash machine today, and on the way back, I was setting up the camera on a tripod to photograph this pub just opposite the local Police Station. While I was doing this, I was approached rather aggressively by a much younger and much taller man, who asked me what it was and what I was doing. I got as far as saying it was a camera ... and the response was before I got any further that 'I can see that you f*ing stupid b*. I tried to avoid replying to any of what follows, which included lots of accusations of being a pervert, being involved with 'Jim will fix it', and apparently train spotters are perverts too. All this right outside the Police Station, so either they don't have CCTV on the approach to the station, or they don't monitor it. It was VERY unsettling, but I stood my ground and captured the images, but it does leave me thinking that I will never want to be in a position to suffer this in the future.