I would stop at any time I felt the desire to, as long as I can see that I'm not endangering others or myself (in that order). Comes to mind that a traffic cone might not be a bad idea- stop, run back a bit, place the cone, go have fun taking shots. I've had the same experience- wanted to stop, nowhere to pull over, got a little ticked, got over it, didn't get the shot.
I like the idea of having a notebook to record the various things as reminders. I spent a bit of time locally on foot, looking for vantage points, and it would have been handy to refer to some notes on that had I made any. It would be good to be able to browse through the notebook and see various photo ops listed in some detail. When the time suits, you can just pick one and go do it, sort of a little project. On the subject of projects, since I do a lot of little jobs in my home shop, it would be good for me to have that list, which I'd add to whenever I came up with an idea. Whenever the time might be right, I could pick one and do it. If I'm browsing through it, I can add notes to existing entries, and maybe when the time comes to do one of them, there'd be enough information there that the job would get done easier, quicker, and or better. Or in NASA lingo, better-faster-cheaper.
No reason why photo-op projects can't be handled this way. I think I'm going to start this notebook myself. I'll have to carry it with me pretty much always, so maybe I can make it a little case with led reading light, a pair of seeing glasses (getting old can sure suck sometimes), the notebook and pen or pencil, and the pocket camera. Hmm.