On Monday, I visited the Imperial War Museum in London and decided to take a few photos of people (most of them unaware of my presence!) as I like to view folks in their natural state - no poses or exaggerated facial expressions. Those that did notice I had the lens pointed directly at them couldn't have cared less. This was London after all. Anything goes!
If somebody did approach me in a 'rude' manner, I would probably agree with them but tell them to lighten up. Yes it might be rude but I'm not really harming anyone so c'est la vie!
As for your situation, I immediately put myself in his shoes and thought, Jesus, I think I'd be pissed off too if I'd been making wooden statues all day long.
I suppose you could say you've been fortunate if this has been the only occurrence in 30 years.