Charlwood House Open Rail Data

[HST125Scorton]

Over the last couple of week while using this handy website to obtain timetables I had noticed it getting sluggish. With Bad Gateway 504 errors becoming more common. On average a page from Crewe lets say on 10-10-2025 used to take seconds to load, took 10mins to load.

It now seems to site owner has decided to close it 'Temp' due to abuse.

login 7410p

 

[alistairlees]

That’s really unfortunate. It’s an excellent resource.

 

[HST125Scorton]

Website has come back online but taking a while to get any sort of data to load up for locations.

 

[trainmania100]

This site has years of data on it, must be billions of train movements on it so it's going to get slower the days go on

 

[Peter749]

The thing I notice is it seems to give a full 24 hours when you pick a location - Unlike RTT and OTT where you can specify the time limit.
If you could specify a time limit it would give a quicker response

Just tried a list for MAN and it was taking ages - changed it to Oban and it give a result in about 20 seconds

 

[Horizon22]

This site has years of data on it, must be billions of train movements on it so it's going to get slower the days go on

Surely some of it can be archived now? Much like apps like RTT only go so far into the past.

 

[trainmania100]

Surely some of it can be archived now? Much like apps like RTT only go so far into the past.

I guess the further it goes back the less likely the data is going to be of use to most People other than off chance .

To show that the archiving actually works, I think railcam has the same software (installed by the charlwoodhouse owner) and utilises the backup function

Could of been a crawler that took the site down rather than the archiving being full up. All a bot has to do is traverse a train schedule page and follow each of the location links (which leads to a full location list for that respective location) , so if a bot crawls all of those links successively it puts high load on the resources .

Wouldn't surprise me if it's the same crawlers that took down wnxx forum the other month, they're all just as bad

 

[pdeaves]

This site has years of data on it, must be billions of train movements on it so it's going to get slower the days go on

I think it's only searchable back three years (assuming it behaves, of course; I can't extract any data at all currently!).

 

[nw-sparks]

I'm sorry about the problems with my site. Unfortunately the server is underpowered, and is struggling under the load. The "abuse", if it even is that, is a steady flow of queries about historic departure screens and train times. These are coming from multiple IP addresses, and each address only makes a few queries a week, which is nowhere near enough to trigger my blocking system.

So either someone is going to lot of effort to scrape the entire contents of my database using a large botnet, or something else is going on that I don't understand.

I am planning to move to a new more powerful server soon, in the mean time there is little I can do.

Phil

P.S. The historic information is regularly trimmed to the last three years, as that is all I can fit on the disc. If the new server performs well I might increase that to four years.

 

[DaveHarries]

I'm sorry about the problems with my site. Unfortunately the server is underpowered, and is struggling under the load. The "abuse", if it even is that, is a steady flow of queries about historic departure screens and train times. These are coming from multiple IP addresses, and each address only makes a few queries a week, which is nowhere near enough to trigger my blocking system.

So either someone is going to lot of effort to scrape the entire contents of my database using a large botnet, or something else is going on that I don't understand.

I am planning to move to a new more powerful server soon, in the mean time there is little I can do.

Phil

P.S. The historic information is regularly trimmed to the last three years, as that is all I can fit on the disc. If the new server performs well I might increase that to four years.

Hi Phil,

No worries at all. I have been using it quite a bit for a SimSig TT which I am compiling and have been working on that on a near daily basis and I have been working on that since July so I hope I am not in any way to blame. I assure you that I am not using a botnet though: they are a curse and should be internationally illegal. In the meantime thanks for all you do in keeping the site going: it is much appreciated.

Dave

 

[josh-j]

a steady flow of queries about historic departure screens and train times. These are coming from multiple IP addresses

I've heard a lot of sites these days are experiencing large increases in random traffic like that, originating from AI crawlers (i.e. trying to read the whole internet for model training). Deliberately using different IPs, not respecting robots.txt, and not identifying themselves as bots making blocking them not so simple, although there are ways of doing it.

 

[nw-sparks]

Sadly, there is an easy solution to this problem, and that is to charge for the website. That would be the nuclear option and is not the way I want to go.

I wonder if I could go half way and require users to register, and log in to use the site. Then I could count their usage and lock out anyone who went over the top.

Just thinking out loud!

 

[HST125Scorton]

Sadly, there is an easy solution to this problem, and that is to charge for the website. That would be the nuclear option and is not the way I want to go.

I wonder if I could go half way and require users to register, and log in to use the site. Then I could count their usage and lock out anyone who went over the top.

Just thinking out loud!

I'm above with Dave, as use your site for SimSig to create timetables. I'm the one who emailed you the other day (Aaron). Now yesterday I was using the site to go back to 28-02-2025 for the freight services. At just after 9am no issues 20sec loaded. At around 12pm I got the 504 Bad Gateway this lasted till just after 1pm. Around 1.35pm I did another go back to 01-03-25 same location freight only. Page took around 1-2min to load everything was ok until around 3.30pm when yep the 504 came back. This seem to last till after 5pm.

I think registering for an account would be a start to try and stop these Bots/AI Crawlers. Those of us like myself who use it everyday to create timetables past and present for SimSig or others could have it listed on there account. Paywall would be the final answer to possibly end the issues? I would be very happy to pay a yearly subscription to keep the site going and having access to the site all the time.

 

[800Travel]

Sadly, there is an easy solution to this problem, and that is to charge for the website. That would be the nuclear option and is not the way I want to go.

I wonder if I could go half way and require users to register, and log in to use the site. Then I could count their usage and lock out anyone who went over the top.

Just thinking out loud!

Registration would be a good first step I'd say. I wonder if real time trains is also going down that route, as I noticed a profile icon on the site the other day. Probably unfortunately as others have said it's something to do with AI training tools crawling the site.

 

[nw-sparks]

The more I think about this the better it seems. As there is no payment involved I don't have to worry too much about data security, I probably won't even collect an email address, just have username and password. It will require a cookie on your machine.

Any account that I deem abusive can simply be deleted from the database.

I wonder if AI can get round this by registering a million accounts?

 

[Watershed]

The more I think about this the better it seems. As there is no payment involved I don't have to worry too much about data security, I probably won't even collect an email address, just have username and password. It will require a cookie on your machine.

Any account that I deem abusive can simply be deleted from the database.

I wonder if AI can get round this by registering a million accounts?

If you require email verification and add a Captcha then you will stop a lot of the bot traffic, even if not all.

I think a small number of searches (e.g. 10 per 24 hour period) should be possible without registration, so that people can see if your site has the kind of content they're after.

You could also look into putting your site behind Cloudflare. Back when I looked into it, they offered a lot of their protection services for free.

 

[Tom]

You could also look into putting your site behind Cloudflare. Back when I looked into it, they offered a lot of their protection services for free.

I think the key thing is here is "until they don't" - they don't have a great track record in recent months if they decide they don't like you or think you have money...

 

[josh-j]

It is also possible to limit crawlers using something like Anubis, which is designed to protect small sites from this kind of thing. A normal visitor to the site sees a message about checking they're not a bot which lets them through after a couple of seconds, no need for registration.

 

[nw-sparks]

It is also possible to limit crawlers using something like Anubis, which is designed to protect small sites from this kind of thing. A normal visitor to the site sees a message about checking they're not a bot which lets them through after a couple of seconds, no need for registration.

That looks like a possibility, thanks, although the fact that their website is currently down doesn't inspire confidence! I'll investigate further.

 

[splashoutradio]

Without going into it too much detail, RailwayData went through a period of being hammered by scraping, bad bots and the like, so I made a script that (largely) deals with it. It is custom made for RD, but if it's potentially of any use, I'll happily share it with you @nw-sparks .

 

[josh-j]

That looks like a possibility, thanks, although the fact that their website is currently down doesn't inspire confidence! I'll investigate further.

I'm not sure why but it works if you go in via https://techaro.lol/solutions/anubis

 

[DaveHarries]

I think registering for an account would be a start to try and stop these Bots/AI Crawlers. Those of us like myself who use it everyday to create timetables past and present for SimSig or others could have it listed on there account.

Yes, probably a good place to start and the use of a CAPTCHA (?) could serve to dispose of Bots / AI Crawlers: I would certainly hope so as I am no fan of AI.

Dave

 

[crosscity]

I'm sorry about the problems with my site. Unfortunately the server is underpowered, and is struggling under the load. The "abuse", if it even is that, is a steady flow of queries about historic departure screens and train times. These are coming from multiple IP addresses, and each address only makes a few queries a week, which is nowhere near enough to trigger my blocking system.

So either someone is going to lot of effort to scrape the entire contents of my database using a large botnet, or something else is going on that I don't understand.

I am planning to move to a new more powerful server soon, in the mean time there is little I can do.

Phil

P.S. The historic information is regularly trimmed to the last three years, as that is all I can fit on the disc. If the new server performs well I might increase that to four years.

Your site is really useful. I don't use it to look at today's trains. I find the site so useful in finding when trains that run only occasionally have actually run. A superb resource.

 

[nw-sparks]

Many thanks all for the nice comments and good advice. I've decided to try a very simple registration solution, and it's just gone live.

Let me know how you get on.

Phil

== Doublepost prevention - post automatically merged: 16 Oct 2025 ==

So far, so good. Load is down to negligible levels, and 20 accounts registered.

Let's see just how I this AI actually is...

 

[trainmania100]

Many thanks all for the nice comments and good advice. I've decided to try a very simple registration solution, and it's just gone live.

Let me know how you get on.

Phil

Works fine for me.
Performance seems to have improved as well.

 

[HST125Scorton]

Many thanks all for the nice comments and good advice. I've decided to try a very simple registration solution, and it's just gone live.

Let me know how you get on.

Just registered to the site, same name handle as here. I have noticed the site performance becoming a lot quicker and pages loading quicker.

 

[WessexEclectic]

So far, so good. Load is down to negligible levels, and 20 accounts registered.

Let's see just how I this AI actually is...

Is the site accepting new registrations at this time? Cant remember my original details...

 

[trainmania100]

Is the site accepting new registrations at this time? Cant remember my original details...

I think you just put any details in and press the relevant option; login or register
Then use those details again to login

 

[nw-sparks]

I think you just put any details in and press the relevant option; login or register
Then use those details again to login

Any username and password to register. If you've forgotten, just register again.

This nonsense was a desperate attempt to stop the continuous background of random queries from AI systems, and much to my surprise it has worked so far.

There are 679 users registered at the moment.

Phil

 

[WessexEclectic]

I think you just put any details in and press the relevant option; login or register
Then use those details again to login

Any username and password to register. If you've forgotten, just register again.

This nonsense was a desperate attempt to stop the continuous background of random queries from AI systems, and much to my surprise it has worked so far.

There are 679 users registered at the moment.

Phil

Success, Thanks both!