Anybody who is travelling on a pass which does not have their destination on I used to ask where they were going just to ensure they were on the right train, particularly if it was late at night and they may struggle to get to their intended destination if they were on the wrong train, doesn't matter if they were blind, had one leg, drunk, male, female, old or young, I'd still ask. And I'd probably repeat it back just out of force of habit, to make sure I'd heard it right.
If your worried about others hearing you, just say your getting off at a station further down the line, then they wont be expecting you to get off when you do.