I recently rescued a 2 yr old entire male dog GSD. Its really embarrassing but every now and again if he rolls on his back or if I am bathing or brushing him I notice he has two very hard lumps either side of his pen*s, these are totally seperate to his balls and he never seems that desperate for a wee etc! Please help I am terrified that I might loose him!
Jackie and Ozzy

Perfectly normal & nothing to worry about  this is the the part of the male that enables him to "tie"with a female during mating

They are perfectly normal...don't worry.All male dogs have them...they are to assist with the tie during mating.
Archer

I think that this is very normal and means that he is pleased to see you :D

Daisy

Don't worry! This is perfectly normal!

What you are noticing is the 'bulb' of his pen*s that, if he mated a bitch, would swell when he had penetrated her and her vag*nal muscles would tighten behind to hold him firmly for the 'tie'. It just means he is happy to see you!
:)