• Our new ticketing site is now live! Using either this or the original site (both powered by TrainSplit) helps support the running of the forum with every ticket purchase! Find out more and ask any questions/give us feedback in this thread!

Passenger train consists as Open Data

Freightmaster

Verified Rep
Joined
7 Jul 2009
Messages
4,410
Those don't show allocations, unless I am missing something.

Correct - it was in reply to people pointing out passenger trains which are listed on RTT
but don't show allocations because the operators do not use the GEMINI system, such as
Tyne and Wear Metro, Sheffield supertram, etc.


There was a time when they did show, although at a different Wimbledon station. I seem to remember that they were either 1Ixx or 2Ixx depending on whether they were Edgware Road or 'City' services.

They still show on realtime maps, but the trains don't use different headcodes:

1758367829931.png

(extract from my Woking to Wimbledon map)


MARK
 
Sponsor Post - registered members do not see these adverts; click here to register, or click here to log in
R

RailUK Forums

Peter Mugridge

Veteran Member
Joined
8 Apr 2010
Messages
17,790
Location
Epsom
They still show on realtime maps, but the trains don't use different headcodes:
That's a known issue with the data on the branch; it doesn't even show the trains correctly on the platform indicators at the stations between West Brompton and Wimbledon.
 

takno

Verified Rep - Traksy
Joined
9 Jul 2016
Messages
6,574
Hopefully there are plans to introduce this into websites such as Trasky , RTT and Railrecord quickly.
I'm hoping to get something out for Traksy soon, although I expect it to initially be an extension of the way we currently do C2C tracking - barebones unit numbers, and on the new site only. Will take a couple of weeks to think through whether I want to do a more comprehensive version and how to make it available to free users.

The release comes entirely without documentation in the form of gigantic XML files fed through a new connection method, so it's not going to be trivial, and it's unlikely to be released for a day or two. You may see me hacking away on Monday and Tuesday as I take very stupid way round from London to Edinburgh.
 

trainmania100

Established Member
Joined
8 Nov 2015
Messages
3,031
Location
Newhaven
You may see me hacking away on Monday and Tuesday as I take very stupid way round from London to Edinburgh.
The good thing about c2c is the trains don't divide on route. I've only just started properly looking at this, but now I'm having trouble finding a service that divides so I can handle the deallocation properly on a schedule route...

I've found today's Horsham / bognor splits but I've only just started logging so I've missed the input...

Agree documentation would be good but I guess that takes the fun out of it. Eg not knowing the difference between TrainDest and AllocationDest lol
 

takno

Verified Rep - Traksy
Joined
9 Jul 2016
Messages
6,574
The good thing about c2c is the trains don't divide on route. I've only just started properly looking at this, but now I'm having trouble finding a service that divides so I can handle the deallocation properly on a schedule route...

I've found today's Horsham / bognor splits but I've only just started logging so I've missed the input...

Agree documentation would be good but I guess that takes the fun out of it. Eg not knowing the difference between TrainDest and AllocationDest lol
To be honest I think it's reasonably clear, or not incredibly difficult to figure out, and waiting for somebody to clear the documentation as well would only have delayed the process even further. It's mostly just quite a lot to parse through and get set up, with having to use an external Kafka connection. The Kafka client library I'm using is already a bit dicey for internal connections, so I'll probably need to look at something which just copies the queue over to my local Kafka server.
 

Poggs

Verified Rep - OpenTrainTimes
Joined
28 Aug 2008
Messages
294
Location
London
To be honest I think it's reasonably clear, or not incredibly difficult to figure out, and waiting for somebody to clear the documentation as well would only have delayed the process even further. It's mostly just quite a lot to parse through and get set up, with having to use an external Kafka connection. The Kafka client library I'm using is already a bit dicey for internal connections, so I'll probably need to look at something which just copies the queue over to my local Kafka server.
There is an ICD (Interface Control Document) on its way, and some XSDs (FOI request pending) for the XML data will be available soon too. I will also write a less formal explanation of how to use the data for the Open Rail Data wiki in due course.
 

takno

Verified Rep - Traksy
Joined
9 Jul 2016
Messages
6,574
There is an ICD (Interface Control Document) on its way, and some XSDs (FOI request pending) for the XML data will be available soon too. I will also write a less formal explanation of how to use the data for the Open Rail Data wiki in due course.
Wonderful, thanks for all the hard work on this.

I think looking at the messages that I've got a fairly good grip of what most of the fields mean. It would be good to know what some of the possible values are for operators and liveries (I guess the operators might be the same values from Trust, but just as a 4 digit number?). Would also be handy to know whether the train id (which is embedded in a longer code for the examples I've looked at) can be depended on, or whether I should be figuring it out from operator/station/headcode.
 

Dren Ahmeti

Member
Joined
17 Oct 2017
Messages
602
Location
Reading
Hopefully it will happen at some point given most TOPS data is already available "on demand". Eg through groups.io where various groups publish loco pool lists daily...
Yes, but TOPS access is tracked, and a few similar “holdouts” on the freight side of thing view it as infringing on “commercial sensitivity” - it took a lot of work to get freight headcodes unscrambled - I remember the days of FRGT and random headcodes being displayed in data!
 

trainmania100

Established Member
Joined
8 Nov 2015
Messages
3,031
Location
Newhaven
It would be good to know what some of the possible values are for operator
I was looking this morning for the operator codes I wasn't sure if it had something to do with the 2 digits at the end of the train ID eg (not a real example) 2K21W1234523 as I can't see anything else it could be.
 

Tom

Member
Joined
19 Jan 2008
Messages
868
Location
35,000ft
The last two digits at the end of the Train ID in LINX messaging from memory is the hour of train departure from origin. RTT uses the Core identifier and plucks the UID out of it. From memory in some cases a TRUST ID can also appear in that field (but that might have been changed in the past few years!)

From experience the metadata that goes along with the messages, e.g. livery/operator/seats, can be taken with somewhat of a pinch of salt.
 

littledude

Member
Joined
21 Nov 2011
Messages
137
Location
People’s Republic of South Yorkshire
The last two digits at the end of the Train ID in LINX messaging from memory is the hour of train departure from origin. RTT uses the Core identifier and plucks the UID out of it. From memory in some cases a TRUST ID can also appear in that field (but that might have been changed in the past few years!)

From experience the metadata that goes along with the messages, e.g. livery/operator/seats, can be taken with somewhat of a pinch of salt.
The last 2 digits of the full headcode are the day of departure. Everything departing today will end with 21
 

trainmania100

Established Member
Joined
8 Nov 2015
Messages
3,031
Location
Newhaven
The last 2 digits of the full headcode are the day of departure. Everything departing today will end with 21
Happy to be corrected but the TRUST ID last 2 digits is the day of departure, this linx specific element I think is different.

Guess a cif lookup will be needed to get the operator then.
 
Last edited:

Oxfordblues

Member
Joined
22 Dec 2013
Messages
1,114
Yes, but TOPS access is tracked, and a few similar “holdouts” on the freight side of thing view it as infringing on “commercial sensitivity” - it took a lot of work to get freight headcodes unscrambled - I remember the days of FRGT and random headcodes being displayed in data!
I'd love to be able to access TOPS if only to find out about wagon locations and train-lists, but I can see that they might be deemed commercially-sensitive, so I don't hold out any hope.
 

Tom

Member
Joined
19 Jan 2008
Messages
868
Location
35,000ft
The last 2 digits of the full headcode are the day of departure. Everything departing today will end with 21
That is correct for TRUST but LINX (where this data flow originates from) is not the same.

For instance, a message I've randomly plucked from Friday (19th) has a Core train ID of 1W57G5608010 which was scheduled to leave Cardiff at 1053.

Guess a cif lookup will be needed to get the operator then.
You can get the sector code from the ResponsibleRU field. I'm not sure I would trust the Operator/Livery fields personally on the actual allocations based on past experience - RTT uses a custom mapping.
 

trainmania100

Established Member
Joined
8 Nov 2015
Messages
3,031
Location
Newhaven
You can get the sector code from the ResponsibleRU field. I'm not sure I would trust the Operator/Livery fields personally on the actual allocations based on past experience - RTT uses a custom mapping.
I see it now- thanks.
See what you mean, TL, SN and GN all use sector 88 as part of GTR.

== Doublepost prevention - post automatically merged: ==

Not familiar with ScotRail workings, do any friends here know if 170s are regularly paired with 43 sets?
 
Last edited:

Poggs

Verified Rep - OpenTrainTimes
Joined
28 Aug 2008
Messages
294
Location
London
I'd love to be able to access TOPS if only to find out about wagon locations and train-lists, but I can see that they might be deemed commercially-sensitive, so I don't hold out any hope.
If you have some money, you can get access to the rail mainframe for yourself - we're taking a few thousand pounds per year for two CICS sessions. No guarantees you'll be able to make the TOPS enquiries you want, but it is possible.

== Doublepost prevention - post automatically merged: ==

Wonderful, thanks for all the hard work on this.
Thank you - it's been a priority of mine to get this data open so everyone can use it - not just me!
I think looking at the messages that I've got a fairly good grip of what most of the fields mean. It would be good to know what some of the possible values are for operators and liveries (I guess the operators might be the same values from Trust, but just as a 4 digit number?). Would also be handy to know whether the train id (which is embedded in a longer code for the examples I've looked at) can be depended on, or whether I should be figuring it out from operator/station/headcode.
Operator codes are 99xx, with the 'xx' being the numeric code as used in TRUST messages. Liveries... that's another story, I'm now trying to assemble all the associated reference data to help people make best use of it.

You shouldn't need to work anything out - allocations have start/end times and TIPLOCs which you can match up to a schedule. Likely you won't even need to reference any of the train identification data, but if I were you, I'd work with the UID embedded in the TransportOperationalIdentifiers tag.
 
Last edited:

takno

Verified Rep - Traksy
Joined
9 Jul 2016
Messages
6,574
You shouldn't need to work anything out - allocations have start/end times and TIPLOCs which you can match up to a schedule. Likely you won't even need to reference any of the train identification data, but if I were you, I'd work with the UID embedded in the TransportOperationalIdentifiers tag.
I'm hoping to keep building this as a separate model which doesn't need access to the CIF schedules, but l'll see where we get - it all ends up running in the same process anyway, so it's not the end of the world if it has to do a schedule lookup
 

nesw

Member
Joined
5 Jan 2013
Messages
279
Location
London
I wonder whether there’s scope or sufficient detail to use the allocation data to produce resource diagrams?
 

Class 466

Established Member
Joined
5 Mar 2010
Messages
2,129
Location
Atherton, UK
I wonder whether there’s scope or sufficient detail to use the allocation data to produce resource diagrams?
it would be pretty cool if there was a fleet list site that listed when a train was last allocated and what to. Alas, I have no such skill.
 

trainmania100

Established Member
Joined
8 Nov 2015
Messages
3,031
Location
Newhaven
I wonder whether there’s scope or sufficient detail to use the allocation data to produce resource diagrams?
It's possible, but obviously it's not going to ever be 100% accurate, I've seen 377 diagrams get updated late evening for services they worked in the morning.
 
Last edited:

Poggs

Verified Rep - OpenTrainTimes
Joined
28 Aug 2008
Messages
294
Location
London
I wonder whether there’s scope or sufficient detail to use the allocation data to produce resource diagrams?
Yes, there is enough detail to do that - but only where resource allocations are made. You don't know if a resource diagram is an LTP or STP diagram, therefore you can only guess (but quite reasonably most of the time) until you've had what you think are all allocations in for a particular day.
 

JAWZ

Member
Joined
29 May 2023
Messages
55
Location
Bath
This is excellent news - just started playing with the API myself, hopefully should have it integrated into TrainMapper soon!
 

Bletchleyite

Veteran Member
Joined
20 Oct 2014
Messages
112,941
Location
"Marston Vale mafia"
it would be pretty cool if there was a fleet list site that listed when a train was last allocated and what to. Alas, I have no such skill.

You can enter a unit number into the search on Realtime Trains and if it's currently doing something it will bring that journey up.

One thing that would be REALLY handy would be if this data included which way round a given unit is, particularly on TOCs where units can end up either way like TPE and XC. That would require input by guards, I guess, though, so I doubt it's coming soon.
 

MrJeeves

Established Member
Associate Staff
Senior Fares Advisor
Joined
28 Aug 2015
Messages
4,611
Location
Burgess Hill
One thing that would be REALLY handy would be if this data included which way round a given unit is, particularly on TOCs where units can end up either way like TPE and XC. That would require input by guards, I guess, though, so I doubt it's coming soon.
The data includes this, but its accuracy is unreliable for most TOCs.
 

Bletchleyite

Veteran Member
Joined
20 Oct 2014
Messages
112,941
Location
"Marston Vale mafia"
The data includes this, but its accuracy is unreliable for most TOCs.

That's quite interesting as the formation diagrams on CIS displays usually do show the orientation correctly for most TOCs so the data must be there somewhere. Those diagrams for something like WMT obviously don't show first class (even though it'd be useful for those of us who like the declassified bit to know where it is so we can get there first, same for the bike spaces which if accurate would result in fewer bikes shoved blocking the way in vestibules) but for the likes of XC it often is correct. However there do seem to be multiple sources of this data - one I noted recently was SWR platform screens saying in the text "this train is formed of 6 coaches" but the little icons showing three (the 6 was correct).
 

JAWZ

Member
Joined
29 May 2023
Messages
55
Location
Bath
Yes, there is enough detail to do that - but only where resource allocations are made. You don't know if a resource diagram is an LTP or STP diagram, therefore you can only guess (but quite reasonably most of the time) until you've had what you think are all allocations in for a particular day.
Don’t the “associations” records in the Schedule feed essentially describe diagrams? I played with them a bit recently but not to the extent of trying to build up a UI for them. Looked fairly comprehensive though from just playing with SQL.

== Doublepost prevention - post automatically merged: ==

Where is the API access link? I'd like to start using it too
It’s on the RDM site, you need to sign up for an account, the link isn’t public as such. Easy (and free!) to subscribe though.
 

nesw

Member
Joined
5 Jan 2013
Messages
279
Location
London
Greater Anglia upload formations and capacity for long term and short term each week on the marketplace. I’m guessing that it would be possible to match the data using the ID. C2C also publish their formation data each day and a number of other operatots have uploaded long term plan formation data. I’m told that the open Gemini data includes diagram numbers, NT 301 as an example, that should simplify creating diagrams.

 
Last edited:

stadler

Established Member
Joined
5 Jun 2023
Messages
3,304
Location
Horsley
That's a known issue with the data on the branch; it doesn't even show the trains correctly on the platform indicators at the stations between West Brompton and Wimbledon.
I think that is a major London Underground wide issue. Nothing really to do with this section in particular. Lots of stations between Earls Court and Ealing Broadway also fail to show trains properly on the station boards. Even some stations in Central London can be very out of sync with the reality of when trains are due to arrive. London Underground are just very poor at providing correct next train data at stations.
 

Top