Worrying is never for nothing, it's what we breeders do best, we watch like hawks, keep an eye, if he is still chubby in the hand, not dehydrated and feeding well then the size is just the size. :-) Some pups are just smaller, some big bruisers, they do vary, just like us. :-)
As long as he is healthy and not weak, he is just smaller, he may catch up once food is introduced or he may not, sometimes the smaller pups can grow up as adults to be bigger than some of their siblings, sometimes they don't. It is nothing to worry about if he is healthy. There are usually varying degrees of size in litters - quite normal. :-)

On the other hand not putting weight on at all for more than a day is not normal after the first few days they should gain steadily even if not as fast as bigger siblings.
If the bitch is no longer constantly in with pups I would ensure this pup gets to feed in between the rest of the litter, say every hour pop him on the milk bar. As he is smaller he cannot take on as much milk as the others in one go, and will also tire more easily, so needs more opportunities to feed. If by the next day eh hasn't gained then I would be looking at why.