My GR helps himself to fallen chestnuts in the woods and I haven't noticed any adverse affects so far. He has been doing this every autumn for three years.
Yes, conkers are horse chestnuts and are poisonous (to us, certainly). Chestnuts (aka sweet chestnuts or Spanish chestnuts), which are what we roast and eat, and use for chestnut stuffing, are fine.