2012 West Coast Franchise Failure: That 'Spreadsheet Error'.

[sbt]

Having seen a number of references to the problems with the 2012 West Coast Franchise award being due to a 'Spreadsheet Error' I thought, as someone involved in government modelling, I would clarify things a bit, explaining a bit about what happened in relation to 'the spreadsheet' and what has happened as a result.

This is, by the way, my own view and not necessarily that of my employers, the government etc.

First off, a link to the key report, the Laidlaw Inquiry report.

https://www.gov.uk/government/publications/report-of-the-laidlaw-inquiry

NOTE: Amended to correct my belief that the Ready Reckoner was 'the spreadsheet'.

If you read it you find that find that 'the problem' was not an 'error' in a Spreadsheet as most people conceive of it. The spreadsheet in question was used to produce a 'Ready Reckoner' supplied to the bidders. This was a tabular guide calibrated from detailed runs of the 'GSP Resilience Model' (GDPRM). Except that the runs of the GDPRM had been carried out incorrectly, so the numbers in the Ready Reckoner were wrong. The runs could have been done correctly but what the report describes as 'technical flaws and inconstancies' made it more likely that the mistakes that were made would be.

Of the many recommendations from Laidlaw only one refers to modelling.

8.14.3 formalised Quality Assurance procedures are established in respect of modelling, encompassing best practice, audit and other testing procedures at appropriate stages of procurements; and​

The rest of the recommendations deal with a whole host of other problems - there wasn't one 'problem', there were many and the problem with the Ready Reckoner and the way the GDPM was used was a symptom of the other, more fundamental, problems.

The result of this, and other issues, was the MacPherson 'Review of quality assurance of government models'. The title of the report is a bit misleading in that it also looked at the 'QA' regarding how an otherwise high quality model was used. For example you might have a perfect model of your mortgage repayment plan but its not that useful if you put the wrong interest rate in or use it to predict your pools winnings.

https://www.gov.uk/government/uploa...ovt_analytical_models_final_report_040313.pdf

Its hard to summarise the results of this, but I think the key one is 'no need to panic' - there was no major cross-governmental issue with the QA of models. However it did recommend some standards etc. to be followed across government and create a 'one stop shop' for 'best practise'.

I build and use models for the MOD, working in the Policy and Capability Department of the Defence Science and Technology Laboratory. MacPherson has led to very few changes in the way we do things as, basically, we were following 'best practice' already. This doesn't mean we are complacent however. For example I spent, as a personal initiative, part of Friday afternoon writing code to make it easier to maintain the QA of some of our models.

I hope people found that interesting and useful and that it cleared up a few misconceptions.

 

[brianthegiant]

Interesting post, so the mistakes were made before the meta model was supplied to the bidders, so it would have been very difficult at that stage to weight the process in favour of a particular bidder (as was frequently suggested on here - that DfT did because Virgin had 'got on the wrong side of them'). And since the modeller wouldn't be the same person as whoever deals directly with the bidders who might have some motivation to fudge the results, it would have required collusion between them which makes it even less likely.

 

[rebmcr]

Sounds similar to what went wrong with the Hubble Space Telescope's primary mirror — shaped correctly and precisely, but to incorrect shape data.

 

[WatcherZero]

Ah the See no Evil, Hear no Evil report when Laidlaw said the Government hadn't provided the evidence he had asked for and since he couldn't see it he had to conclude it didn't exist. I don't think anyone ever suggested the model was flawed, but that it was late and inappropriately used. The failings were all due to human elements of organisation, leadership and individual decision making.

 

[sbt]

I don't think anyone ever suggested the model was flawed, but that it was late and inappropriately used. The failings were all due to human elements of organisation, leadership and individual decision making.

First off, I would like to alert people to a small change in my first post - I now understand the nature of the 'Ready Reckoner' better.

The reason I started this thread was I have seen lots of people on this forum and in the press blaming a 'spreadsheet error'. I regard as indicating that lots of people do indeed believe and suggest that 'the model' (GDPRM) was flawed.

The GDPRM wasn't late - it was in existence prior to the franchising process. What was 'late', in the sense that there was insufficient time to develop it, was a tool that did its job that was suitable for distribution to the bidders.

Its important not to confuse the 'Initial Findings' Laidlaw report, where he states that he has not yet received various documents, and the final one where he appears to have received these as he doesn't comment on the issue. For completeness a link to the initial report is below.

http://www.railwaysarchive.co.uk/documents/DfT-LaidlawInterimReport2012.pdf

There were also another report into the failures, the NAO one, which comes to the same conclusions.

http://www.nao.org.uk/report/lessons-from-cancelling-the-intercity-west-coast-franchise-competition/

I'm not going to be drawn into the emails issue other than to make two points:

a) A check was subsequently made, after the Laidlaw report, of the departmental emails that Laidlaw was criticised for not having investigated for the sentiments people were concerned about. This was at the request of the Transport Committee in Parliament. I can find no evidence online that anything untoward was found.

b) The circumstances described in the reports were in themselves sufficient to produce the events that are known to have occurred without any deliberate bias in favour of or against one or other of the bidders.​

 

[Baggypants]

So... it pivoted on a function which did not excel?

 

[WatcherZero]

2.16.2 In my Initial Findings Report I explained (at paragraph 2.4.5) that I considered that there had been insufficient time to conduct an email capture and review in advance of preparing that report and so had not instructed that to be done. On 31 October 2012 the Inquiry team submitted to the DfT a request for email searches to be done by the DfT by reference to specified custodians, time periods and search terms. In the event, the DfT was unable to transfer the data to the Inquiry team in sufficient time for an email review to be carried out because of Government data security and related issues. No email capture and review has therefore been carried out for the specific purpose of this report.

2.16.3 The DfT instructed the Inquiry team that in interviews of DfT officials the Inquiry team should not ask questions which go to the personal accountability, culpability or mindset of DfT officials. The DfT has explained this limitation on the basis that such questions might prejudice the HR investigations. Given that the Inquiry is not focussed on individual culpability (see paragraph 2.9 above), I consider that this limitation has not, in practice, prevented the Inquiry team from obtaining through interviews the evidence that it needed to ensure that the Inquiry was carried out in accordance with the Terms of Reference.

Sounds like he still didn't receive them to me. See no evil, hear no evil.