Looks like I'm not alone!
I'm generally in the same position. Right now I'm using a small point and shoot and my camera goes with me just about everywhere, however, I still feel a little out of place when I pull it out for an "artistic" shot.
Most people expect to see cameras at kid's birthday parties or playgrounds, etc. but when you crouch down to take a macro shot of a lone blade of grass because it strikes you as interesting, people look at you kinda strange. At least that's how it feels to me.
I think the problem is that photography is one of the few "personal" artforms that you generally do in public. Most people's art is done in privacy but not so with photography. To get the really good shots you have to go out into the world.
All I can say is to keep working at it. Start by going out at odd times when there is not as many people around and work up to being more at ease in a crowd. The results can be quite satisfying.
Regards, Iguanasan