I suppose, ultimately, it comes down to your operation method (DOO vs Guard) and TOC policy.
We have guards and our policy is we can accept not to calls/additional stops via the official paper slip, via instruction from control or via the guard verbally.
Personally, with a special stop, I'll be reasonably comfortable taking that on the verbal authority of the guard. The way our safety management system is set up means that it's not an imcident per se, for me to stop additionally. It would simply cause delay and if it wasn't an authorised special stop, I might get a slightly awkward to explain "please explain" (a request for delay info). The operational incident would fall to the guard, for releasing the train doors at an unauthorised stop. Of course, if a special stop order comes to me from the guard, I'll check who's told them and make sure they've checked with control.
When it comes to "not to calls", I'm much more careful, as failing to call at a station /is/ a drivers operational incident. Basically the instruction must be officially written or issued directly to me from control. If anyone else (including the signaller) tries to issue me one, I'll always confirm, again, with control. This is also likely TOC specific, but there have been a number of occasions where the signaller has v1 of the plan, whilst control are working to v2 (or v3, or usually something like v17 - they do quite like to change their minds)...
EDIT (just to keep up with the last few posts):
I can't imagine many scenarios where a driver would refuse a special stop order unless it wasn't issued properly. Certainly, we would get a phone call from a manager if we did so. I can't think of many colleagues who would refuse on the grounds that it would take them over their day - each additional stop only causes circa 2 mins of delay - which is usually made up in pathing time or engineering time built into the timetable, or simply through scheduled 2/3 minute dwells at larger stations that become 1 minute dwells. It's hardly going to ruin your after work plans to stop once or twice additionally.
What would ruin your day is to have an operational incident investigation against yourself because you stopped/didn't stop when the instruction wasn't officially issued.