Rather than make a big thing about removing things from his mouth I' d monitor and use distraction, ideally you want to get in there before he picks up things. Keep high value treats, fave toys (squeakers) handy. You don't want to make him think you want the things he's exploring, but need to find ways to refocus his interest on you or something else you are happy for him to have in his mouth.
I would also be teaching him a really solid leave command- this takes time but is what you need if you are worried about him approaching or picking up potentially poisonous things. There are certain plants and shrubs that are poisonous so I would make sure you know what these are- obviously if you see him pick up something poisonous then you need to stop him eating it. Here's a link
http://www.dog-first-aid-101.com/toxic-garden-plants.htmlCertain terriers, for instance, can become committed and avid bee catchers. Some are good at it and others end up getting stung in the mouth, which is not great. So these are the sort of things to watch out for and try to nip in the bud, if you can.
Obviously he is going to explore so it's a case of you not giving him free rein in every environment, only the ones where you are confident there is nothing he can ingest that is dangerous.
Did he actually try to swallow the pebble or did he just pick it up?
He will be teething and this may play a big part in his behaviour, he may also be naturally very curious. I would try to play exciting games with him when you are out, to try to see you as his main source of entertainment rather than him entertaining himself.