It does (mildly) annoy when people in the window seat ask to get out 10 minutes before the station, when I am getting off at the same place.
I can see your point, but...
If I'm in the Quiet Coach of a VTEC Mk4 set, and have my bike with me, it would be prudent to get out of my seat 10 minutes before the station. Likewise if I have a tight connection to make (especially if the train is running late) and I need to get to the optimum door to exit, and it's vital I'm among the first out to get a chance of making it
On the other hand, I've known people move to the vestibules to get off a lightly loaded train for York... in
Knottingley!

Though that amuses me, rather than annoys me, and telling people they're 15-20 minutes away from York normally results in them sitting back down.
I don't like being 'boxed in' the window seat because it may (mildly) annoy people if you have to get past them.
To go back to the original question, some people are
elective mute/
selective mute, and some people are simply shy in these sort of situations. I agree some are rude, but you can't always be sure.