He's been eating and pooping fine for the last 4 days, but I haven't seen any signs of a bag coming out.  I've heard rumors that it can be dangerous.. what do you think? 
He's also acting fine, active and being his usual self. 
Should I worry?

I would just give your vet a call, ask for advice. I'm suprised its not come out yet, could you have missed it?

Hi,
We had a greyhound that ate the bone I was holding(whole!)and the bag it was in! A few days later, the bag arrived, but no bone I'd give it a couple of more days yet IMO, as the vet will not want to cut him open and go looking if the dog hasn't shown any symptoms. If he stops eating, drinking and is sick and running a temp, then take him.