We're not comparing like for like here. I don't class somebody who has a "couple of train books on the shelf" as a proper obsessive trainspotter. They just sound like someone who has a passing interest in railways. In which case this probably wouldn't preclude having lots of sex with a beautiful woman.
Well anything obsessive is a turn off, you are quite correct. But that cause be endlessly down the pub watching football, spending hours doing up cars and ignoring the kids, and many other 'male' pursuits that women, in general, do not have appreciation for. I don't mean that in a derogatory sense, it is a fact some hobbies and interests are more male orientated than others, although I say the more ladies the better!
I would say talking bus, plane, or truck numbers would equally be viewed as slightly odd. But is it any more odd than stamp collection, collecting football shirts, and so on? There is part of the (especially) male species that is genetically disposed towards hunting and gathering, maybe it's related so therefore is more normal than we think?
Then there is a point about obsession. When does a passion turn into an obsession? Love can be unhealthy if obsessive, drinking and better certainly so if done to excess. If were all to do everything in moderation, how many of us are guilty of going over the top in at least one thing we enjoy? Quite a few.
But as I said, these things are subject to fashion and fad, and if you turn back the clock 40 years trainspotting was viewed as very normal, in what was then a very varied railway. I suspect, today's modern traction, which varies little on some lines, doesn't help.
I certainly have no interest in train numbers, never have and never will, nor do I go for 'mileage' or anything like that. While it does give the rest of us a bad name, I will stop short of criticising them, if that's what they want to do, so be it. To my knowledge not all that many enthusiasts do it, people resolve their interests in different ways. Likewise, most car enthusiasts don't take car number plates. And of course, many train enthusiasts are car enthusiasts! Most people are interested in the designs and how things work.