People who do this really enfuriate me. It is dangerous and when someone else is feeding your cat it does become hard to get your cat in at night/when it's dark for their own protection.
I had this problem many years ago with a neighbour of mine who used to be a friend. We got a brother and Sister cat from WGAS when our cat was in a RTA & although he survived he lost control of his bowell and bladder and had to be PTS. We got the cats because myself and the children were devastated at losing the family pet, and I'd also recently split from my husband so the children were overly affectionate with the new cats focusing lots of love on them.
The only pet the neighbour had was a dog, she also had 2 children (one at the same school as my eldest daughter) and the boy in the neighbouring family would taunt my daughter (7) that her cat loved them more and had left our family just like her Dad (a lovely boy!). Understandably my daughter would get upset at the taunts, the male of the cat was her cat so this added insult to injury, the female of the pair was very small and vocal, a bit timid and had taken to me and used to avoid the children. I explained to my neighbour that the situation was upsetting my daughter already sensitive from the matrimonial split, & that she'd grown specifically fond of the male cat and it used to go up initially to her room and help her to get to sleep early evening.
Many times the cat would be missing, and because of the RTA on our older cat we would all be worried that he'd been knocked by a car and this was why we couldn't get hold of him, so many a sleepless night was had.
I repeatedly asked the neighbour not to feed the cat, and she assured me that she had been but she wasn't any longer. One day our male cat had been missing for a couple of days and both myself the children were distraught so I approached the neighbour who's been feeding him. The way the houses were set out was with 2 doors at the front of the house, one opening on to the hall and one opening on to the kitchen. She must have been expecting someone else as when I knocked she called to come through the kitchen, as I did so there was the cat bed, the place mat and food bowls with fresh fish (and milk which I never feed to my cats) in them, as well as an array of cat toys.
I was so angry that I walked straight out again and went to my house aware that I might just blow up at her, and I did not want any trouble within the neighbourhood, so I explained to her later when I had calmed down that the cat was on a special diet and it was costing me money in vet bills and also that because the cat was spending so much time at hers, that I was missing veterinary check ups. I figured if she liked the cat that much she wouldn't want this to happen as a result of her intervention. The diet and vet thing was a total fabrication but I was getting desperate and just wanted our cat back!
As it happened this woman also had a husband that worked away, and didn't like cats (this was the reason they didn't have their own cat), so when he would come home they would move all the cat paraphenalia and shoo my cat away. As he'd been away on a long work contract the cat had got used to staying with them and forgotten to come home almost all the time, the child of their family was still saying hurtful things to my daughter about how he had her cat and how lovely and fluffy it was at night. I had rung WGAS who had come up with the dietry idea - that had failed, but I was keeping them informed of the situation, as technicaly the cat remains their property throughout it's life (or it did in those days I am not sure if it's changed now).
We were fed up and went to the Blue Cross in Kimpton and got another kitten, the female cat took to it and my daughter was slightly consoled. Then the neighbours hubby got a new contract and was home a lot more, so the days when they would not allow the cat access increased in frequency and lenght. This meant the cat we'd originally had as ours started to want to come back to ours at these times, but he would attack the other cats in the household in the process. We tried to work through it, but in the end I rang WGAS and they said that they wanted the cat back as it was unsatisfactory. They even did me a letter to show the woman concerned.
We did this and it was horrible to have to do. On top of this we had the whole cliquey neighbourhood turn on us and everyone was stopping us in the street telling us how we broke their hearts by being so selfish!!!!!

I got so fed up of it all that I was glad when we moved away from there to a larger house, I would never have believed how much trouble and ill feeling one cat could cause! My daghter was bullied at school over it, and I was given the cold shoulder at all the Mums and Tots meetings locally - pathetic.
I really hope you manage to sort it out amicably - keep us posted. The tags with the food warning on them sound a good idea :) I think my tactic on that failed as the friend guessed I was making it up!