If you've got the smooth side it'd seem that the boundary fence isn't your responsibility, and is the property of the other people.Not necassirly I am afraid :( :( :(
They could be party fences where you both own it. It could be yours or it could be the builders as they put it up.
The way round the fence is has no bearing anymore to who owns it. It was a gentlemans handshake agreement that is being used less and less as the years have gone by. There is no law to say you have to have the 'bad' side.
The only way you can find out if the fence is yours or not is look on your deeds. They should be outlined on there.
Sorry still not much help.
Lea :)