A-level Results 2020

[JamesT]

If we did simply go by teacher assessment (for one year only), would it be such a bad thing?

Those who would have done better in the exam than the algorithm thought will get what they deserve. Others will get more than they deserve - it may open a door through which they will stride purposefully onto a path leading to the sunlit uplands (other metaphors are available) - a potential realised. Or it may be a step too far and they stumble but then, hopefully, find a better path for them.

In one of my A-Level subjects, the entire cohort did worse than the mock exams had predicted - those expected to pass well, scraped through; those expected to pass so-so, failed; no one got better than grade E. I was saved by the fact that, aged 11, I had chosen a scarcity subject where the universities couldn't be picky. It was a very lucky break.

Sometimes people get lucky breaks. Mostly it's the children of the rich and powerful; it would be nice if sometimes it's other people.

As I've mentioned in a previous post, many universities over offer on places as they know a certain proportion won't get the grades. If everyone got the predicted grades, they suddenly have more students than they have places. There's a numbers cap in place, so even if they could find the resources to run the course for everyone, they're probably not allowed to.
Conversely those universities that don't over-offer and rely on making up their numbers through clearing are facing a shortfall which could be quite serious on their finances.

If you were going to decide to use predicted grades after abandoning exams, that decision needed to have been much earlier in the year so the universities could adjust the criteria they use to make the offers on. (Oxbridge make their offers in January, so that ship had long sailed).

 

[si404]

Although this was announced, there doesn't seem to be any way for a student to actually request that their mock exam result be used instead of the grade awarded by the algorithm. The appeals process published on Saturday, and then withdrawn, did not do this.

OfQual have asserted their independence from the Ministry and rejected his (stupid) proposal that Mock grades should count.

It's good that Williamson's nonsense isn't happening, but OfQual doubling down on their nonsense algorithm isn't good (and will thankfully make it easier to hold them accountable for their actions as they aren't hiding behind the politicians).

So maybe 'bold' but not especially helpful.

Indeed, but the charge of inaction was made against Williamson. That's untrue. The action was a bad one (and I said it was), but it was action.

If we're going to pillory the guy, and I'm not saying we shouldn't, we should at least get the charges accurate!

Ultimately, the buck stops with the Secretary of State.

Sure, and I can only imagine Williamson is still in his job because they haven't come up with a solution for Thursday (let alone a sacrificial lamb to replace him and be given a very short time to try and save a sinking ship) and it's just easier if he goes down for that too.

However, sacrificing the SoS shouldn't save those lower down the order who screwed up. Likewise sacrificing the UK minister shouldn't save devolved ministers who have complete authority in their countries and did very similar things to Williamson.

This was an institutional failing, and to blame it entirely all on the ministerial flailings to try (and totally fail) to fix this is to not understand what went wrong.

You say it’s all down to Ofqual

No I don't. I'm saying it's not all down to the Tories, as some were suggesting it was.

If we did simply go by teacher assessment (for one year only), would it be such a bad thing?

No it wouldn't have been - this cohort was already being judged by a different standard due to the cancellation of exams.

However, now the horse has bolted and decisions have been made - the genie cannot go back in the box. Universities have filled their places, etc, etc.

 

[Typhoon]

It cannot just be a case of teacher over prediction.

An analysis by the Sixth Form Colleges Association (SFCA) looked at 65,000 exam entries in 41 subjects from its members and found that grades were 20% lower than historic performances for similar students in those colleges.
The SFCA said it had not found a single one where the results were above the three-year average.
Ofqual states that its objective for A-level results this year was to ensure "national results are broadly similar to previous years".
The research showed that Ofqual "not only failed to produce broadly similar results, but has in fact produced worse results in every single subject", the SFCA said.
Bill Watkin, chief executive of the SFCA, said Ofqual should "immediately recalibrate and rerun the model to provide all students with an accurate grade".

From https://www.bbc.co.uk/news/education-53804323 (my emphasis)

65,000 entries - no small sample. I cannot think of any reason why the allocated results should be lower in every subject. It cannot be down to GCSE results as those that take A levels at sixth form colleges are likely to have taken most of their GCSEs at schools prior to moving to sixth form college. Whatever is in the algorithm, it does not appear to be background-neutral.

This is not the time for blame-allocating but, for the sake of the students concerned, it needs to be sorted out pretty quickly.

 

[SuspectUsual]

The PM himself is still missing instead of being out there and leading in this crisis

What else would you expect? The one public pronouncement he’s made that I’ve heard has been a standard load of Boris bullsh*t with no grasp of detail and vague platitudes like “these are a robust set of results”.

 

[oldman]

No it wouldn't have been - this cohort was already being judged by a different standard due to the cancellation of exams.

However, now the horse has bolted and decisions have been made - the genie cannot go back in the box. Universities have filled their places, etc, etc.

Well, if there is a u-turn, the unis (actually the over-subscribed ones) and the regulators will have to find a way around it.

Daily Telegraph site (timed 13:59)

Gavin Williamson will announce a full U-turn on exams this afternoon, confirming that all A-level and GCSE grades will be based on teacher assessment, the Telegraph understands.

 

[Bayum]

Announcement at 16:00, should be interesting. U-turn on not using CAGs, or resignation? Perhaps both?

 

[Bayum]

Another thing that surprises me is the complete lack of thought from Ofqual to contact centres they thought were inflating their grades. If their argument was ‘Some centres predicted nothing but A/A*’ - investigate. Don’t just sit back and punish others. This whole fiasco is so disheartening to students, their teachers and their families. University offers have already been made, and students have had offers rescinded based on these results. Some courses will already be full. The cynic in me questions whether this was about restricting access to universities this year...

 

[SuspectUsual]

Another thing that surprises me is the complete lack of thought from Ofqual to contact centres they thought were inflating their grades. If their argument was ‘Some centres predicted nothing but A/A*’ - investigate. Don’t just sit back and punish others. This whole fiasco is so disheartening to students, their teachers and their families. University offers have already been made, and students have had offers rescinded based on these results. Some courses will already be full. The cynic in me questions whether this was about restricting access to universities this year...

Indeed. The same goes for the Department for Education, of course. Surely the first questions to be asked when decisions like this are made are “how does it impact results?”, “does it work fairly?”, “will the public accept it?” etc etc.

I honestly don’t know how if it’s arrogance, ineptitude, or both

 

[Bayum]

Indeed. The same goes for the Department for Education, of course. Surely the first questions to be asked when decisions like this are made are “how does it impact results?”, “does it work fairly?”, “will the public accept it?” etc etc.

I honestly don’t know how if it’s arrogance, ineptitude, or both

I think it’s a combination of factors, unfortunately. I think they’ve just used whatever grade curve they have had in previous years for that school and places students on it in the order their teacher’s ranked them. Wholly unfair when you consider that a bright cohort might easily not be any lower than a B. They’ve focussed too much on trying to keep it ‘stable’ and ‘average’, but they’ve not worked hard enough on disadvantaged areas, groups of children who suffer etc.

 

[Senex]

The more we hear, the more puzzling the whole business seems to be. It appears that OfQual worked entirely in its own on the algorithm and what it was designed to do, didn't discuss it with any of the "interested parties" like schools, universities, and teaching unions, and didn't take advice even from bodies like the Royal Statistical Society. And it seems that the neither the civil servants of the DoE nor their political masters over a number of months were curious enough to try to find out what was being developed. I find it very surprising indeed that in a matter that was bound to be very politically sensitive government ministers do not seem to have taken a direct interest.
That some people were beginning to suspect that things would not go smoothly comes out in two Guardian articles, https://www.theguardian.com/educati...l-result-predictions-to-be-downgraded-england, and https://www.theguardian.com/educati...he-father-who-foresaw-a-level-algorithm-flaws. The second of these is especially interesting, as it appears that a member of the public with the necessary specialist skills did try to warn politicians and others of what was likely to happen (with a surprising degree of accuracy). And then, of course, there were the Scottish results, which should surely have warned London not only of what was coming but also of the furore it was inevitably likely to create. Even at that very late stage, surely politicians on the ball — and it is with the Secretary of State himself that the buck stops — would have at the very least tried to prepare for what was coming, or better, despite the problems for the universities and the disappointment for the candidate, stopped the issue of the results until there had been a thorough review.
How could all this have happened? How far is the British establishment's love of secrecy in public life to blame? Surely there was a case, the moment it seemed likely that no examinations could be sat this summer, for all the interested parties to be brought together to discuss the situation and work out what was to be done. But instead a group of officials got to work behind closed doors, produced their algorithm, ran it, and brought chaos. And they were allowed by their complacent and compliant political masters to do so. And when the storm broke, no-one was in the least degree prepared.

 

[eMeS]

... But instead a group of officials got to work behind closed doors, produced their algorithm, ran it, and brought chaos. And they were allowed by their complacent and compliant political masters to do so. And when the storm broke, no-one was in the least degree prepared.

Could this be as a result of so many of their political masters choosing PPE at university, rather than statistics? Or am I, a retired engineer, simply having silly ideas?

 

[najaB]

How could all this have happened? How far is the British establishment's love of secrecy in public life to blame? Surely there was a case, the moment it seemed likely that no examinations could be sat this summer, for all the interested parties to be brought together to discuss the situation and work out what was to be done. But instead a group of officials got to work behind closed doors, produced their algorithm, ran it, and brought chaos. And they were allowed by their complacent and compliant political masters to do so. And when the storm broke, no-one was in the least degree prepared.

It just shows how much the current government holds the average Briton in contempt.

 

[Bayum]

The decision that has been made is the correct decision. I recognise that in some circumstances, there will be students who have been given grades that they shouldn’t, however this is where Ofqual ‘should’ have been focusing their work on; identifying the inconsistencies and working with centres to assure both parties that those grades are appropriate for those students. Instead, Ofqual have spent their time putting together s**t on a plate.

 

[SuspectUsual]

Does anyone know what proportion of results in a “normal” year differ from the teachers’ predictions?

 

[JamesT]

Given previous performance, in most cases if you give students the teacher's predicted grade then it will be the wrong one. Various reports have them predicting the correct grade only 16% of the time, the vast majority of the time they over-estimate rather than under-estimate.
Before the results came out, there were people campaigning against using the predicted grades, e.g. https://www.itv.com/news/2020-06-17...students-gcse-alevel-coronavirus-uk-education

Relying on students' predicted grades during the coronavirus crisis could be unfairly "predicting futures" of some pupils, a report has warned.

The Equality Act Review concluded the teacher assessment system - in place of cancelled exams - could negatively impact black, Asian and minority ethnic (BAME) pupils and those from more disadvantaged backgrounds.

There do seem to be some particularly odd individual results, but without actually doing exams it's hard to say how far off the algorithm is from what might have been.
All four nations decided originally that this was the correct path to go down, so it's not just a few rogue civil servants.

 

[JamesT]

Does anyone know what proportion of results in a “normal” year differ from the teachers’ predictions?

A report in 2016 produced for the UCU says teacher predictions were correct 16% of the time, 9% underestimate, and 75% overestimate - https://www.ucu.org.uk/media/8409/P...ec-16/pdf/Predicted_grades_report_Dec2016.pdf

 

[najaB]

A report in 2016 produced for the UCU says teacher predictions were correct 16% of the time, 9% underestimate, and 75% overestimate...

It also says that the overestimation occurs most commonly at the bottom of the actual results and underestimation occurs most commonly at the top. So this is likely the least worst option to go with, the students who will be most disadvantaged are the ones who are most likely to be able to recover from it - either through challenges or through hard work next year.

 

[JamesT]

It also says that the overestimation occurs most commonly at the bottom of the actual results and underestimation occurs most commonly at the top. So this is likely the least worst option to go with, the students who will be most disadvantaged are the ones who are most likely to be able to recover from it - either through challenges or through hard work next year.

To some extent that's expected. If you've predicted AAA and it's wrong, there's an awful lot more grades below than above. (And vice-versa for the other end).
It also says that for high-achieving students, disadvantaged ones are more likely to be under-predicted compared to their advantaged compatriots. Which leads to them applying to less prestigious universities as they don't think they have the grades for the better ones.
The drive of the report is that predicted grades shouldn't be used as they're so inaccurate for university admissions and 'real' results should be instead. Unfortunately that makes a complete mess of the academic calendar (how do you fit admissions between results in August and term starting in September?) and really doesn't help this year as there are no exams to provide the results.

 

[DelayRepay]

I will repeat a point that's been made a few times, but is worth repeating.

Teachers can predict grades based on potential, based on the student's work and assessments during the year. What they cannot do is predict which of their students, for whatever reason, will not reach their potential during the exam.

Example:
Teacher has a class of 20 and predicts 5 students will get A*. In fact one of them has a bad exam and 'only' gets A. It is reasonably likely that this will happen, and probably happens every year meaning that the aggregated historical results are lower than teacher's predictions. Teacher cannot predict which of their 5 A* students is going to be the one that has a bad exam though.

Algorithm addresses this by predicting that whichever of the students was ranked 5th by teacher is the one who has the bad exam, and their grade is changed to an A.

Teacher's class are all well performing this year, and teacher hasn't predicted anyone to get less than a C. But last year there was a student who had an almighty row with his girlfriend on the morning of the exam, and didn't turn up, so he got a U. Algorithm decides that someone has to get a U this year to fit the curve, so whichever student has been ranked bottom goes from a C to a U, failing the exam that never took place. Meanwhile, this year's student who had some kind of drama on the day of the exam that never happened gets a C. So algorithm has given the right grades, but to the wrong students. Teacher could not possibly have factored in the drama to their prediction because if they were able to do that, they would have also predicted the Euromillion results and would not be working as a teacher!

In summary it is unfair on teachers to say they over-predicted. They did, but that's because all they can predict on is potential. They can't predict which of their students are going to have bad exams.

When I did my A-levels, I was predicted AAAA. A* did not exist then. I got AAAC. Wasn't a bad prediction - the C was further maths, my uni offer was based on 3 As, so I focused less on further maths and more on the other three subjects to make sure I got my 3 As, rather than risk ending up with AABB.

 

[SuspectUsual]

That’s very interesting. So really, the thing that’s really unfair isn’t so much the teachers predicting the grades, but also them being asked to tank the pupils. Obviously in a “normal” year, if ten pupils get an A, it doesn’t matter if one is only a mark short of an A* and another is only a mark above a B, but this year it really matters.

You’d imagine that data would exist to prove or far more likely disprove a correlation between ranking in mock exams (rather than grades) and eventual exam grades

== Doublepost prevention - post automatically merged: 17 Aug 2020 ==

When I did my A-levels, I was predicted AAAA. A* did not exist then. I got AAAC. Wasn't a bad prediction - the C was further maths, my uni offer was based on 3 As, so I focused less on further maths and more on the other three subjects to make sure I got my 3 As, rather than risk ending up with AABB.

Again, very interesting. I was predicted ABB but actually got ACC, because the university course offer I had required only BD or CC. I did well enough, rather than as well as I could. No algorithm can allow for that

 

[158756]

This U turn might have taken the heat off Williamson, but it's going to create massive problems for universities.

I read they're legally obliged to take any student who met the conditions of an offer, and everyone they've accepted through clearing. So the more prestigious universities, which offer far more places than they actually have, will be massively oversubscribed, without the teaching space or accommodation for all their students. Universities relying on insurance choices and clearing on the other hand have just lost a substantial part of their income - some are already struggling, this decision could send quite a few to the wall.

 

[cactustwirly]

I don't understand the issue with the algorithm.
From what I've read, it's exactly the same one as they use every single year to calculate the grade boundaries for each subject.

Using teachers grades is stupid, because they're under pressure to overinflate grades because the of school league tables. And teachers marking of mocks is actually quite generous compared to a real examiner.

Personally I think the teachers should have predicted a mark, for each paper the students would have sat. Then the normal algorithm used to set the grade boundaries.

 

[JamesT]

That’s very interesting. So really, the thing that’s really unfair isn’t so much the teachers predicting the grades, but also them being asked to tank the pupils. Obviously in a “normal” year, if ten pupils get an A, it doesn’t matter if one is only a mark short of an A* and another is only a mark above a B, but this year it really matters.

You’d imagine that data would exist to prove or far more likely disprove a correlation between ranking in mock exams (rather than grades) and eventual exam grades

I can’t immediately find a reference, but I had read that teachers were considered to be much more accurate when it came to ranking their students. They might not get it quite right whether Jonnie gets an A or a B, but they’re much better at saying he’ll do better than Bob.

 

[DelayRepay]

I don't understand the issue with the algorithm.
From what I've read, it's exactly the same one as they use every single year to calculate the grade boundaries for each subject.

Not really. It is true that every year they use an algorithm of sorts to set the grade boundaries, but that's after everyone has taken the exams and the numerical results are available. So this year I might get 86%, which the algorithm decides is a B. But everyone else getting 86% will also get a B. People who only get 82% might end up with a C.

This year, there is no exam, so the algorithm has taken teacher rankings (not estimated grades), and allocated grades in rank order based on historical performance. It is different because the normal process applies consistently to all candidates, but this process applies differently depending on the historical performance of your school.

== Doublepost prevention - post automatically merged: 17 Aug 2020 ==

I can’t immediately find a reference, but I had read that teachers were considered to be much more accurate when it came to ranking their students. They might not get it quite right whether Jonnie gets an A or a B, but they’re much better at saying he’ll do better than Bob.

I can't find a reference either, but I think it's a generally accepted principle that humans are better estimating anything relatively, rather than absolutely.

"How tall was he?"
"About the same height as you"

 

[Bayum]

I don't understand the issue with the algorithm.
From what I've read, it's exactly the same one as they use every single year to calculate the grade boundaries for each subject.

Using teachers grades is stupid, because they're under pressure to overinflate grades because the of school league tables. And teachers marking of mocks is actually quite generous compared to a real examiner.

Personally I think the teachers should have predicted a mark, for each paper the students would have sat. Then the normal algorithm used to set the grade boundaries.

I mean, I don’t know where you have all this experience from marking mock papers or plucking grades out of thin air comes from, but I can tell you quite honestly what you’ve suggested is a load of garbage. These results have been moderated with other colleagues within the various disciplines. Most schools and centres will have set these predicted grades with a great deal of discussion between various teachers of subjects including moderation in their own disciplines.
Mock tests are a very difficult ballgame. Most schools will set mocks far harder than the papers or will mark more harshly so come the paper, students aren’t making silly mistakes because they’ve learnt from the feedback they have got. The other problem with mock exams is that centres will set mock exams differently based on modular work, questions their cohorts find particularly challenging and past papers which cover the whole curriculum. Much of the curriculum will not have been taught at the point of students leaving to lockdown, so will not have achieved full marks on these tests because of that. Some exam boards will have a larger number of past papers whereas others might not, depending how soon they changed after the A-Levels were increased in difficulty. Many students won’t prepare for mock exams, nor will they necessarily do as well as they might on the final test later.

 

[cactustwirly]

I mean, I don’t know where you have all this experience from marking mock papers or plucking grades out of thin air comes from, but I can tell you quite honestly what you’ve suggested is a load of garbage. These results have been moderated with other colleagues within the various disciplines. Most schools and centres will have set these predicted grades with a great deal of discussion between various teachers of subjects including moderation in their own disciplines.
Mock tests are a very difficult ballgame. Most schools will set mocks far harder than the papers or will mark more harshly so come the paper, students aren’t making silly mistakes because they’ve learnt from the feedback they have got. The other problem with mock exams is that centres will set mock exams differently based on modular work, questions their cohorts find particularly challenging and past papers which cover the whole curriculum. Much of the curriculum will not have been taught at the point of students leaving to lockdown, so will not have achieved full marks on these tests because of that. Some exam boards will have a larger number of past papers whereas others might not, depending how soon they changed after the A-Levels were increased in difficulty. Many students won’t prepare for mock exams, nor will they necessarily do as well as they might on the final test later.

That's nowhere near as robust as the moderation on actual exam papers. Teacher's marking will always be biased, however much you try and mitigate this. They are also biased because of league tables, and the pressure to maintain the schools position.
Obviously the exams are marked anonymously by complete strangers giving candidates a fairer mark that actually reflects their ability.
University exams are marked internally, but then they are moderated by external examiner's to avoid bias.
From personal experience going through the system, teacher's marks are definitely less harsh than examiners, not hugely but enough to make a difference.

The whole point of the moderation is to make sure that the results were fair and matched the performance of the past 5 years. The fact that lots of grades were moderated suggests that the teacher's grades weren't accurate and were inflated.

 

[Bayum]

That's nowhere near as robust as the moderation on actual exam papers. Teacher's marking will always be biased, however much you try and mitigate this. They are also biased because of league tables, and the pressure to maintain the schools position.
Obviously the exams are marked anonymously by complete strangers giving candidates a fairer mark that actually reflects their ability.
University exams are marked internally, but then they are moderated by external examiner's to avoid bias.
From personal experience going through the system, teacher's marks are definitely less harsh than examiners, not hugely but enough to make a difference.

The whole point of the moderation is to make sure that the results were fair and matched the performance of the past 5 years. The fact that lots of grades were moderated suggests that the teacher's grades weren't accurate and were inflated.

But there were no exams? ‍♂️
That isn’t the problem. The problem is that there were so many outliers that weren’t being taken into account. The average used didn’t take out anything regarding deprivation, low SES, BAME or anything, so all we’ve been left with is a classist, elitist set of grades reserved for those paying for education or the ability to live in a more expensive area. Ofqual have had ample amount of opportunity to do random sampling in order to ensure that teacher’s have been moderating effectively and correctly, but chose not to, and instead just applied the ranking order instead of taking grades into account.

 

[eoff]

The more we hear, the more puzzling the whole business seems to be. It appears that OfQual worked entirely in its own on the algorithm and what it was designed to do, didn't discuss it with any of the "interested parties" like schools, universities, and teaching unions, and didn't take advice even from bodies like the Royal Statistical Society. And it seems that the neither the civil servants of the DoE nor their political masters over a number of months were curious enough to try to find out what was being developed. I find it very surprising indeed that in a matter that was bound to be very politically sensitive government ministers do not seem to have taken a direct interest.
That some people were beginning to suspect that things would not go smoothly comes out in two Guardian articles, https://www.theguardian.com/educati...l-result-predictions-to-be-downgraded-england, and https://www.theguardian.com/educati...he-father-who-foresaw-a-level-algorithm-flaws. The second of these is especially interesting, as it appears that a member of the public with the necessary specialist skills did try to warn politicians and others of what was likely to happen (with a surprising degree of accuracy). And then, of course, there were the Scottish results, which should surely have warned London not only of what was coming but also of the furore it was inevitably likely to create. Even at that very late stage, surely politicians on the ball — and it is with the Secretary of State himself that the buck stops — would have at the very least tried to prepare for what was coming, or better, despite the problems for the universities and the disappointment for the candidate, stopped the issue of the results until there had been a thorough review.
How could all this have happened? How far is the British establishment's love of secrecy in public life to blame? Surely there was a case, the moment it seemed likely that no examinations could be sat this summer, for all the interested parties to be brought together to discuss the situation and work out what was to be done. But instead a group of officials got to work behind closed doors, produced their algorithm, ran it, and brought chaos. And they were allowed by their complacent and compliant political masters to do so. And when the storm broke, no-one was in the least degree prepared.

It seems though that Ofqual are saying there were various interested parties engaged in this (perhaps as a GDPR appeal defence)...

I found this story today...
https://www.computerweekly.com/news...es-GDPR-concerns-over-A-Level-results-scandal

Quote from the ICO...

“Ofqual has stated that automated decision-making does not take place when the standardisation model is applied, and that teachers and exam board officers are involved in decisions on calculated grades. Anyone with any concerns about how their data has been handled should raise those concerns with the exam boards first, then report to us if they are not satisfied."

 

[LAX54]

So the handling of this, only topped by the handing of the 'virus' !

 

[SuspectUsual]

So the handling of this, only topped by the handing of the 'virus' !

And Brexit, obviously. Oven ready apparently, just seems that we don’t actually possess an oven