ATOC make full raw fares datafeed available

[Paul Kelly]

This hasn't been mentioned already has it?

Each year in the UK, train fares can be subject to revision in January, May and September, with the main revision taking place in January each year and the May and September revisions being available to make seasonal alterations to fares. Train companies are also able to create promotional fares and make corrections to fares between the three revisions under strict rules overseen by government.

On a trial basis, it is intended that the fares data download available on this website will be released three times a year in January, May and September. These three releases are known as ‘full releases’ of fares data and are being made available here free of charge under the terms of a Creative Commons licence.

If you wish to receive updates to fares, which are made between the three ‘full releases’ and are released on a daily basis, then you can find details of how to access them and the associated conditions and costs by visiting www.atoc.org/about-atoc/rail-settlement-plan/data-feeds

It should be noted that fares priced at £0.05, £0.10, and those ending in .99 pence should not be displayed or advertised by the user, because they are only being included in the fares data download for testing purposes or as vehicles to issue manual rail inclusive package fares which are not held in the fares system and not sold by train companies.

Users should also be aware that Advance fares are mainly quota controlled and therefore whilst these fares appear in the fares data, they are only available for sale subject to availability.

I have had a look it and it is the full raw feed as provided to booking engines. Only the electronic routeing guide feed is now conspicuous by its absence!

Well done ATOC!

 

[Poggs]

This hasn't been mentioned already has it?

I have had a look it and it is the full raw feed as provided to booking engines. Only the electronic routeing guide feed is now conspicuous by its absence!

Well done ATOC!

I just stumbled across it by pure chance.

Now to download and, er...

 

[bakerstreet]

I'm looking at this on a smartphone but when I get to a desktop machine is this in a form that anyone can interrogate the data or is this requiring expert computer knowledge?

 

[soil]

I'm looking at this on a smartphone but when I get to a desktop machine is this in a form that anyone can interrogate the data or is this requiring expert computer knowledge?

Realistically you would want to put this into SQL database. The datafiles are hundreds of megabytes of plain text, one fare pair to a line and it's not really practical to access it as it is.

It's all described here:

http://www.atoc.org/clientfiles/File/RSPDocuments/sp0035.pdf

Unless you have an application to build, I would imagine you would be better using one of the existing interfaces to these data.

 

[maniacmartin]

The Avantix Traveller data is much more compact, as its done with clusters etc, but the downside is you can't access it directly with a standard SQL query.

 

[Paul Kelly]

This data is all structured in clusters too! It's just so massive because it's exported in plain text. It would be tens of gigabytes if it included the fares point-to-point instead of as clusters!

 

[bakerstreet]

Unless you have an application to build, I would imagine you would be better using one of the existing interfaces to these data.

I think that's what I'll do! Out of interest I don't suppose there's a way of combining this data with an old version of Avantix? ie using the program but replacing the old fares data with the new?

 

[maxiboy]

I think that's what I'll do! Out of interest I don't suppose there's a way of combining this data with an old version of Avantix? ie using the program but replacing the old fares data with the new?

Probably not possible (at least not without some major reverse-engineering effort) as the file format seems to be quite different. Although the Avantix Traveller databases are mostly plain text (and so are these files that can be downloaded from ATOC) I haven't entirely figured out their structure.

 

[Paul Kelly]

librailfare (advertising disclaimer: written by me) can read the Avantix Traveller database format. I think Avantix uses Orbix but it can be reverse-engineered without any specific knowledge of Orbix. Once you have got the Orbix wrapper stuff out of the way the data structure is very very similar.

 

[wintonian]

I see this as the important bit:

On a trial basis, it is intended that the fares data download available on this website will be released three times a year in January, May and September. These three releases are known as ‘full releases’ of fares data and are being made available here free of charge under the terms of a Creative Commons licence.

But it doesn't say which license, so can it be used commercially? What about creating derivative works? If you do does the license remain the same?

A very positive development but some important questions about licensing are left unanswered.

 

[Muzer]

It doesn't specify in plain-text; however, by reading and comparing the legal-ese, it looks to me to be CC-BY 2.0. Not NC, not even SA. So, use it however you like, even GPL it if you must

 

[wintonian]

CC-BY 2.0 are you sure? I would be very surprised if they would allow commercial use of the data.

Full details of the licence terms are available in the member area or during registration.

Although this statement makes me wonder if it is in fact compatible with Creative ?Common licensing? After all (and certainly with share alike) there is requirement to register or agree to any additional conditions. Someone in theory could download all the data and make it available for others to download from their own server.


So when is someone going to take this data and change it in order to demonstrate a sensible and realistic fares structure?

 

[DaveNewcastle]

. . . . . . . So, use it however you like, even GPL it if you must

A Licence must be given; it cannot be assumed.

Alternatively, all rights may be waived, but again, this must be stated as a grant and cannot be assumed.

 

[maxiboy]

librailfare (advertising disclaimer: written by me) can read the Avantix Traveller database format. I think Avantix uses Orbix but it can be reverse-engineered without any specific knowledge of Orbix. Once you have got the Orbix wrapper stuff out of the way the data structure is very very similar.

Thank you for the link, I'll have a look at it

 

[soil]

So when is someone going to take this data and change it in order to demonstrate a sensible and realistic fares structure?

What do you mean?

 

[radamfi]

So how do I actually download this data?

 

[greatkingrat]

From http://data.atoc.org/

You will need to register if you haven't already done so.

 

[Paul Kelly]

Someone in theory could download all the data and make it available for others to download from their own server.

That is fine; the person who downloads it then "inherits" the same license from ATOC. The person making it available is not allowed to sub-license under it a different license though; that is specifically prohibited in the terms of the license.

Commercial use is not specifically prohibited, but on the other hand the license does appear to reserve the right to charge royalties for use of the data, which seems like it could potentially be a bit of a time bomb.

 

[wintonian]

That is fine; the person who downloads it then "inherits" the same license from ATOC. The person making it available is not allowed to sub-license under it a different license though; that is specifically prohibited in the terms of the license.

Commercial use is not specifically prohibited, but on the other hand the license does appear to reserve the right to charge royalties for use of the data, which seems like it could potentially be a bit of a time bomb.

How can you be sure of this when we don't even be sure exactly what the license is?

 

[greatkingrat]

As has already been posted, we do know what the license for the data is (CC-BY 2.0). The license on the ATOC site is the same as the one at

http://creativecommons.org/licenses/by/2.0/uk/legalcode

 

[DaveNewcastle]

. . . . the license does appear to reserve the right to charge royalties for use of the data, which seems like it could potentially be a bit of a time bomb.

The Licence "grants to You a worldwide, royalty-free, non-exclusive Licence for use and for the duration of Copyright in the work" so there is no prospect of fees being incurred within any reasonable time.

The right to charge fees arises from any of the uses and rights which are NOT covered by the Licence:

2.4 All rights not expressly granted by the Licensor are hereby reserved, including but not limited to, the exclusive right to collect, whether individually or via a licensing body, such as a collecting society, royalties for any use of the Work.

 

[wintonian]

As has already been posted, we do know what the license for the data is (CC-BY 2.0). The license on the ATOC site is the same as the one at

http://creativecommons.org/licenses/by/2.0/uk/legalcode

I'm yet to see anything by the hand of ATOC that says explicitly it is CC-BY 2.0, the only thing I have seen so far is that it is licensed under a Creative Commons licence.

If you could link to where ATOC say what CC licence they are using that would be great.

 

[greatkingrat]

I think you have to register on the ATOC website before you can see the actual license.

 

[Paul Kelly]

How can you be sure of this when we don't even be sure exactly what the license is?

My quotes were from the actual license text from data.atoc.org! See here: http://data.atoc.org/licence-agreement

The Licence "grants to You a worldwide, royalty-free, non-exclusive Licence for use and for the duration of Copyright in the work" so there is no prospect of fees being incurred within any reasonable time.

Ah, I see - that is much more reassuring. Maybe the issue is that the licensee is not allowed to charge royalties to third parties for accessing the data; that ATOC retain the exclusive right to do this? E.g. a copy of the same data will be being provided as daily updates to booking engine providers, and they pay a lot of money for the privilege of using it.

 

[wintonian]

Well why don't they just come out and say "(CC-BY 2.0)" rather than someone attempting to justify their salary by typing out paragraphs of unnecessary text just because they can?

 

[Muzer]

A Licence must be given; it cannot be assumed.

Alternatively, all rights may be waived, but again, this must be stated as a grant and cannot be assumed.

I meant because it's not SA, you can re-licence it however you want assuming you comply with the other terms.

 

[DaveNewcastle]

I meant because it's not SA, you can re-licence it however you want assuming you comply with the other terms.

No.
Only the rights holder has that authority under the Act. The Licence even states:

2.1 You must not:

sublicense the Work.

 

[maxiboy]

No.
Only the rights holder has that authority under the Act. The Licence even states:

What about the first sale doctrine?
http://en.wikipedia.org/wiki/First-sale_doctrine#Application_to_digital_copies

Meaning it would be possible to transfer one's licence, but only once (with the effect of losing one's own licence, of course).

 

[Muzer]

No.
Only the rights holder has that authority under the Act. The Licence even states:

In which case, what's the point of having ShareAlike as an option?

 

[DaveNewcastle]

. . . . . what's the point of having ShareAlike as an option?

I'm sure there are other differences, but on first sight, the distinction between ATOC's prohibition on sub-licensing and ShareAlike's permission of an endless cascade of sub-licences might be some assurance of currency to the end user.
(After multiple sub-licences, data will become progressively older, which for fares data may be irresponsible to authorise, whereas for an entertainment or artistic product, that may be a neutral or positive quality.)