Transport directions from ChatGPT

[Gaelan]

Here's a fun exercise in demonstrating the limitations of language models like ChatGPT: ask it for directions on any vaguely non-trivial journey by public transport. It'll name several of the relevant services, but more often than not make at least one fundamental mistake that makes the whole itinerary dangerously useless. A few examples:

Several attempts to ask for directions from St Andrews to Edinburgh Airport suggest traveling to Edinburgh by:

• Bus 99, which runs St Andrews - Leuchars - Dundee
• Bus X54, which runs Edinburgh - Dundee, not via St Andrews
• Bus X24, which runs St Andrews - Glasgow

It never mentions a bus that actually gets you from St Andrews to Edinburgh, nor does it mention the tram-train interchange station Edinburgh Gateway which is (in my experience) by far the most convenient way to do that journey.

Asking for directions from St Andrews to Amsterdam is mostly successful, but includes this sage advice:

Once you arrive at London Kings Cross Station, take the London Underground to London St Pancras International Station. The Piccadilly line or Circle line can be used for this part of the journey. This will take around 15-20 minutes.

Asking for directions between my American hometown and Seattle (~25 miles) seems quite insistent that the bus from my hometown terminates at Tacoma's main transport hub (it doesn't, it terminates at a random Park & Ride on the edge of the city), and once tried to put me on the light rail from Tacoma to Seattle, which would be great if it wasn't for the fact that said line isn't planned to open for a decade at least.

 

[Mcr Warrior]

Here's a fun exercise in demonstrating the limitations of language models like ChatGPT: ask it for directions on any vaguely non-trivial journey by public transport.

Certainly not authoritative. Garbage in, garbage out?

Anyone else have any other (transport related) examples of ChatGPT's limitations?

 

[eldomtom2]

Certainly not authoritative. Garbage in, garbage out?

Not so much that as the fact that it's designed to predict the most likely phrase, not know facts. Small differences in phrasing a question can cause it to give very different answers.

 

[Gaelan]

Not so much that as the fact that it's designed to predict the most likely phrase, not know facts.

And that's the fundamental problem - it's designed to produce the most plausible-sounding answer it can; it has no notion of truth. This makes it fundamentally unusable for anything where factual accuracy is key, which is a problem, because people keep using it for that!

 

[DelW]

And that's the fundamental problem - it's designed to produce the most plausible-sounding answer it can; it has no notion of truth. This makes it fundamentally unusable for anything where factual accuracy is key, which is a problem, because people keep using it for that!

Sounds rather like your ex President! (And to a marginally lesser degree, our ex-but-one Prime Minister).

It does concern me that search engines are being steered towards using these things. When that happens, it will be difficult to rely on anything they produce, and alternative facts / fake news will be everywhere.

 

[Gaelan]

Sounds rather like your ex President! (And to a marginally lesser degree, our ex-but-one Prime Minister).

Perhaps, though I dare say ChatGPT does a better job - but I digress.

 

[Bletchleyite]

And that's the fundamental problem - it's designed to produce the most plausible-sounding answer it can; it has no notion of truth. This makes it fundamentally unusable for anything where factual accuracy is key, which is a problem, because people keep using it for that!

Not only that but it isn't trained on current data, it's trained on data a few years old, if I recall.

 

[Gaelan]

Not only that but it isn't trained on current data, it's trained on data a few years old, if I recall.

Yes, certainly this too - although the routes it’s proposing don’t make any more sense on a 2021 or 2019 timetable, as far as I’m aware.

It’d be interesting to try this experiment with Google/Bing’s search-enabled chatbots.

 

[danm14]

Anyone else have any other (transport related) examples of ChatGPT's limitations?

It told me I could get from Glasgow to Dublin without using an aeroplane or ferry, by taking a (direct) train to Holyhead and then another train to Dublin. Although it doesn't seem to do that any more.

 

[Bletchleyite]

Yes, certainly this too - although the routes it’s proposing don’t make any more sense on a 2021 or 2019 timetable, as far as I’m aware.

Fair point. I guess the issue with an AI is that it tries to infer things the same way a human would. Think "Race Across the World" - without Internet access, you might just infer stuff logically, then find out the service that logically should exist doesn't.

 

[pokemonsuper9]

Tried this one here (removed my home station out of it, wanted to see if it could guess my planned route).
1. Accurate based on a previous timetable
2. takes closer to 2 hours exactly than 2:15, but can vary to that
3. Much more reasonable to get Northern to TCR than this but ok
4. West Ruislip is the wrong way
5. c2c doesn't go to Southend Victoria

Other points:
3:30 basically implies teleporting between trains and instant departures.
Doesn't even go via Stratford International.

Playing around with ChatGPT is always fun.

 

[Gaelan]

Tried the St Andrews - Edinburgh Airport test with Bard (google’s thing, I believe able to access up-to-date information from Google’s search index).

Sure, here are the directions by public transport from St Andrews to Edinburgh Airport:

1. Take the X59 bus from St Andrews Bus Station to Leuchars railway station. The journey takes about 20 minutes and costs £4.50 for a single adult fare.
2. At Leuchars railway station, take the train to Edinburgh Waverley station. The journey takes about 30 minutes and costs £10.50 for a single adult fare.
3. At Edinburgh Waverley station, take the tram to Edinburgh Airport. The journey takes about 35 minutes and costs £5.50 for a single adult fare.

The total journey time is about 1 hour and 45 minutes and the total cost is £20 for a single adult fare.

Close, but wrong: the X59 goes directly to Edinburgh not via Leuchars. Changing to the tram at Waverley isn’t the standard route, but possibly justifiable if the next train from Leuchars isn’t a stopper. The fares are in the right ballpark (although the train fare is low unless you’re using a railcard or an LNER advance), the journey times are completely wrong.
Another go produced much the same itinerary, but with a price range of “£21-£110”! This time it also mentioned the airport shuttle (a privately-run minibus), claiming a price range of £15-£20 - nope, website says they start at £30, and much more outside peak dates!

 

[pokemonsuper9]

This one's quite funny, it knows they are on the same line, however still instructs you to go via Piccadilly.
I tested this wanting to see if it'd instruct to change at Dinting.
I like correcting it when it makes mistakes, however I don't think it actually reports back and learns from that.

 

[TUC]

Surely in terms of this aspect of ChatGPT it's no different to what Google Maps and similar apps offer? Google Maps can be quite good at finding the quickest way of getting from A to B on public transport. It just has no concept of permitted routes or which route offers the cheapest fare.

 

[Bletchleyite]

Surely in terms of this aspect of ChatGPT it's no different to what Google Maps and similar apps offer? Google Maps can be quite good at finding the quickest way of getting from A to B on public transport. It just has no concept of permitted routes or which route offers the cheapest fare.

Not really. The problem with ChatGPT (in the context of something absolute) is that in the absence of information it infers, and it can infer wrong. Google Maps won't infer - if there isn't a bus in the data between A and B it won't assume there is one because there logically should be.

ChatGPT will do something like assume that just because Merseyrail has 4tph on weekdays on all lines, that Manchester's local network will too.

A future challenge for the likes of ChatGPT is how to interface a natural language conversation style user interface with being able to understand that certain things need to be absolute.

 

[Neo9320]

Okay, here’s something we can all do (and not just in rail terms) but actually feed chatGPT the correct answers!

A system that is based on data only as recent as 2021 isn’t going to be up to date, and certainly not with the constantly changing uk timetable.

Tell it when it is wrong, that’s the whole point of the project to create an up to date AI system that can help as many people as possible.

As someone mentioned up thread ‘garbage in garbage out’ if people keep on messing with this system and not improving it, it won’t get anywhere!

 

[fwsmVor]

It apparently thinks there's a once-a-day direct bus from Dingwall to Exeter via Glasgow.

 

[Intercity110]

Anyone else have any other (transport related) examples of ChatGPT's limitations?

I asked it about wales and west’s loco hauled services, and it said they were hauled by class 50s, I don’t think they did though.

 

[GatwickDepress]

Anyone else have any other (transport related) examples of ChatGPT's limitations?

What part did H from Steps play in saving the Settle and Carlisle Railway from closure?

H from Steps, whose real name is Ian Watkins, did not play a direct role in saving the Settle and Carlisle Railway from closure. However, he was involved in a campaign to save the railway in the early 1990s, which was led by the Settle and Carlisle Railway Trust. He contributed to the 1994 charity single 'Settle-Carlisle' which saw record sales and helped ensure the future of the railway line.

 

[Purple Train]

In an alternative ChatGPT world...

London Bridge has five platforms
Milton Keynes Central replaced a "Milton Keynes" station opened in 1838
Milton Keynes Central also features a curved roof of steel and glass
Brookmans Park station has an art deco entrance and is situated in a bucolic village named Brookmans Park
Moston has direct services to Halifax
Trains from Paddington to Reading stop at Guildford (haven't noticed that on my journeys into London...)
There is a 100 bus from Redhill to Guildford via Earlswood (Warwickshire)
The National Express coach service from Nottingham to Victoria considerately stops at Bexleyheath Bus Garage
Cannon Street has six platforms, is served by Thameslink, and has direct trains to Sussex
Charing Cross is served by Southern and Southeastern, and has direct trains to Brighton
Moorgate is served by Thameslink trains to Cambridge
All termini in London are situated in "the financial district"
Pudding Mill Lane provides a super-interchange with the DLR, Central Line, and the Elizabeth line, including a 15-space car park
Southgate tube station has lifts and is not served by Uxbridge trains
Lincoln Central has six platforms and a variety of shops
Mornington Crescent features a "unique and distinctive facade" that is "often photographed by visitors", and is named after the elder brother of the Duke of Wellington

I'm not sure which I prefer: ChatGPT's world, or this one...

 

[Mcr Warrior]

Anyone else have any other (transport related) examples of ChatGPT's limitations?

What part did H from Steps play in saving the Settle and Carlisle Railway from closure?
H from Steps, whose real name is Ian Watkins, did not play a direct role in saving the Settle and Carlisle Railway from closure. However, he was involved in a campaign to save the railway in the early 1990s, which was led by the Settle and Carlisle Railway Trust. He contributed to the 1994 charity single 'Settle-Carlisle' which saw record sales and helped ensure the future of the railway line.

Take it that the above is mostly/entirely a load of tosh?

Believe 'H', born 1976, was working at Butlin's Minehead in the early 1990's, rather than single-handedly rescuing the Settle & Carlisle line (and hadn't the line already effectively been rescued/reprieved in 1989?)

 

[greatkingrat]

Looks like it will make up something for any random celebrity. At least the dates are more accurate this time.

The Chuckle Brothers, Barry and Paul Elliott, were British entertainers and television personalities known for their slapstick comedy. They were not directly involved in the campaign to save the Settle-Carlisle railway, but they did lend their support to the cause.

The Chuckle Brothers were among the celebrities who publicly supported the campaign to save the railway. In 1989, they performed a special show on board a train that travelled the length of the Settle-Carlisle line, raising awareness of the campaign and helping to draw attention to the beauty of the route.

 

[GatwickDepress]

Take it that the above is mostly/entirely a load of tosh?

Believe 'H', born 1976, was working at Butlin's Minehead in the early 1990's, rather than single-handedly rescuing the Settle & Carlisle line (and hadn't the line already effectively been rescued/reprieved in 1989?)

Oh yes, I believe it's what the boffins call a 'hallucination'.

If you ask it something absolutely outlandish and disprovable like 'why did Sir Isambard Kingdom Brunel build a viaduct between Belfast and Liverpool?' then ChatGPT will rebuke you for trying to spread misinformation and that it never happened, but nudget it something semi-plausible or write a leading question, it makes it all up.

Looks like it will make up something for any random celebrity. At least the dates are more accurate this time.

To be quite frank, I would have paid good money to watch a live Chuckle Brothers show on a train.

 

[Mcr Warrior]

What's with all these C list celebrities and their supposedly supporting the Settle-Carlisle railway?

 

[A Challenge]

Another example is the Hither Green Viaduct, which was from a Diamond Geezer post on the River Quaggy on the 1st April.

diamond geezer

diamondgeezer.blogspot.com

Today's post has been written by ChatGPT and illustrated by DALL·E.
It's entirely AI-generated.

A walk along the River Quaggy

.........


A photograph of the Hither Green railway viaduct over the River Quaggy in southeast London.

One of the most striking sights along the river is the old stone viaduct that spans the river at Hither Green. Built in the 19th century, the viaduct is an impressive feat of engineering, with arches that soar high above the river below. It's a beautiful reminder of the rich history of the area, and I take a moment to appreciate the craftsmanship and design of the structure.

"Tell me more about the Hither Green viaduct."

The Hither Green Viaduct is a stunning structure that spans the River Quaggy in the Hither Green area of Lewisham, southeast London. It was built in 1897 as part of the London, Chatham and Dover Railway Company's expansion plans.

A photograph of the Hither Green railway viaduct over the River Quaggy in southeast London.

The viaduct is an impressive example of Victorian engineering, with a total of 31 arches spanning the river below. Each arch is made up of 10 to 12 brick piers, with the largest central arch spanning over 40 meters. The viaduct stands around 22 meters tall and is approximately 220 meters in length. The viaduct was constructed to carry trains on the South Eastern Main Line between Charing Cross and Kent. The line was electrified in the 1920s, and the viaduct remained in use until 1961, when the railway line was rerouted.

After the railway line was decommissioned, the viaduct was abandoned and became dilapidated over time. However, in recent years, the viaduct has been restored and is now an important part of the local community. It was listed as a Grade II* listed building in 1980, which means that it is recognized as a building of exceptional historic and architectural interest.

The restoration of the Hither Green Viaduct was carried out as part of the Green Chain Walk project, which aims to create a network of walking routes that link green spaces across southeast London. The viaduct now forms part of the Green Chain Walk and is a popular spot for walkers and cyclists.



The viaduct is a beautiful and impressive structure, and its restoration has brought it back to its former glory. The arches of the viaduct provide a unique vantage point from which to view the River Quaggy below, and the sound of the flowing water can be heard echoing through the arches. The viaduct is an important part of the area's history, and its restoration is a testament to the importance of preserving our architectural heritage for future generations to enjoy.

 

[Purple Train]

What's with all these C list celebrities and their supposedly supporting the Settle-Carlisle railway?

Would the attention of Pink Floyd convince you?

Pink Floyd, the legendary English rock band, played a significant role in raising awareness and funds to preserve the Settle-Carlisle railway line in the 1980s. The band held two benefit concerts at London's Roundhouse in 1986, which raised over £250,000 for the preservation efforts. Pink Floyd's lead guitarist David Gilmour was particularly involved in the campaign and even produced a short film titled "The Settling of the Carlisle" to raise awareness of the line's beauty and significance. The band's efforts helped to save the line from closure, and it remains an important and scenic railway route to this day.

Though, to the computer's credit, it did reject the notion that the 45th President of the United States might have been involved in said efforts... though, on a tangent, he did end up doing the draw for the League Cup in 1990 (I think) for Saint and Greavsie, through a slightly strange set of circumstances.
That's firmly off-topic though, so sticking to the main theme...

Taplow is now served by South Western Railway services to Windsor & Eton Central in ChatGPT's world! I wish...

 

[GatwickDepress]

Taplow is now served by South Western Railway services to Windsor & Eton Central in ChatGPT's world! I wish...

Hastings apparently has direct Javelin services to London St Pancras now, although the Marshlink has been reduced to stop at Rye and Winchelsea only, and Southeastern run a shuttle to Bexhill via Battle!

 

[Intercity110]

Northampton has East midlands railway, both West Midlands and London northwester railway services, Avanti west coast and crosscountry services. I wish this was the case desperately. The Northampton and law port railway also extends through the whole line. I want to live in chatgpt’s world so badly.

 

[GatwickDepress]

Darlo would love this one:

• Other destinations: Fenny Stratford is also served by some peak-time services that operate to and from London Euston via Milton Keynes Central, and some special event services to places such as Wembley Stadium.

 

[Purple Train]

Darlo would love this one:

• Other destinations: Fenny Stratford is also served by some peak-time services that operate to and from London Euston via Milton Keynes Central, and some special event services to places such as Wembley Stadium.

It suggests the same is true of Bow Brickhill as well... additionally the Jubilee Line from Baker Street to Waterloo terminates at Stanmore!
ChatGPT doesn't have the greatest internal compass...

== Doublepost prevention - post automatically merged: 10 May 2023 ==

This one's quite funny, it knows they are on the same line, however still instructs you to go via Piccadilly.
I tested this wanting to see if it'd instruct to change at Dinting.
I like correcting it when it makes mistakes, however I don't think it actually reports back and learns from that.

Or, even better, go direct?

Anyway, ChatGPT's world now encompasses:
1) Regular, direct trains from the Hertford Loop to King's Cross
2) Regular services from St Pancras to Edinburgh quicker than via King's Cross

I wonder who would operate the St P-Edinburgh services? One for Speculative Discussion?