
If his sperm was viable he'd surely have managed to get at least one of all these bitches in whelp? It would be really hard luck otherwise. I'd get his sperm checked again, perhaps by a different vet? and see what it shows - could be low even if some are there. Otherwise I'd take it that it just isn't meant to be and either keep him as a pet, or sell him as a pet.
Castrate him? No. If he's not fertile, why bother!!!