A-level Results 2020

[JamesT]

There was an explanation of the algorithm used to calculate the results by Timandra Harkness (Fellow of the Royal Statistical Society) on The World at One today (14/08) (interview starts at 15' 17").

Apparently the past records of that school for each subject over several years is taking into account to calculate the number of each grade to award; this is then adjusted to take into account that year's GCSE performance compared to that of previous years. The adjusted number of A*, A, B, etc are then allocated by the rank order for that subject that the teachers have placed them in. No use is made of the estimated grade.

There would appear to several weaknesses in this. How does this system allocate grades for a subject never taught before at that institution? What about subjects where the general GCSE scores would not necessarily indicate aptitude (like creative arts)? What happens in subjects which attract low numbers of students?

My understanding that for subjects where the cohort is insufficiently large or there isn't past performance to feed in, the moderation is not applied. Therefore the predicted grades are used as-is.

This is likely one of the reasons for the apparent bias towards private schools, as they tend to smaller class sizes, they may be more likely to have classes too small for the moderation to apply.

 

[Typhoon]

My understanding that for subjects where the cohort is insufficiently large or there isn't past performance to feed in, the moderation is not applied. Therefore the predicted grades are used as-is.

This is likely one of the reasons for the apparent bias towards private schools, as they tend to smaller class sizes, they may be more likely to have classes too small for the moderation to apply.

Thank you. I heard this later, so a good year to be studying Ancient Greek or Classical Civilisation!

 

[Senex]

Thank you. I heard this later, so a good year to be studying Ancient Greek or Classical Civilisation!

There seem to have been several detailed examinations of the algorithm in various newspapers and on the BBC, although an official version doesn't seem to have been issued. There seem to be all sorts of possible criticisms, but to me the most unfair aspect of all seems to be this one of small school/course v. large school/course. If I've followed right (and I'm no statistician), if an outstanding pupil took let's say an A-level in Sanskrit that was being offered for the very first time in their small rural sixth form and the teacher predicted A*, that is the grade that would be awarded, with no intervention. But if the same outstanding pupil were taking, say, English Literature in a large bog-standard urban comp with a general mediocre record and if the teacher saw the outstanding talent and predicted A*, that would count for nothing, because if there had been no A* marks at that school in the preceding three years, then no-one could be awarded an A* in this year'a algorithm process. How can that possibly be a fair scheme of assessment?
I'm also a little puzzled by the way in which the universities, having received the marks a few days early as they always do, did not realise there were massive problems and at least consider not rushing to confirm places, so that they could remain in a position to respond to appeals results or any general changes. As it is, a number of places seem to have filled their quotas already, so that any candidates who get their marks changed to such as would meed their admissions requirements are going to be told to come back next year. How can it possibly be right to delay those who have met requirements, even if that has only been established on appeal, whilst letting in those who perhaps did not meet requirements in the first place? Particularly when so many questions have already so quickly been raised about the soundness of the whole system. And what happens if the whole scandal is repeated in a few days' time with the GCSE results?

 

[Typhoon]

If an outstanding pupil took let's say an A-level in Sanskrit that was being offered for the very first time in their small rural sixth form and the teacher predicted A*, that is the grade that would be awarded, with no intervention. But if the same outstanding pupil were taking, say, English Literature in a large bog-standard urban comp with a general mediocre record and if the teacher saw the outstanding talent and predicted A*, that would count for nothing, because if there had been no A* marks at that school in the preceding three years, then no-one could be awarded an A* in this year's algorithm process. How can that possibly be a fair scheme of assessment?

I quoted a distraught female student on Any Questions who went to a college which specialised in vocational course but had an 'A' level programme. There are several reasons why she might have made the move but schools tend to hang on to the more able students. She had been accepted by the Royal Veterinary College, subject to results, so she must have looked good on paper (and, possibly, at interview) but all her results in all subjects were down, one by 3 grades, probably because the college had not had outstanding students before. The chief regulator and an executive director of Ofqual appeared before the House of Commons Education Select Committee on 10th June, very reassuring. Expected grades in by the end of that week. That is two month's ago, how come they did not see a problem soon after? Did they flag it up with DfE?
The chief regulator of Ofqual stated that it would be up to the exam boards to deal with appeals, they are going to be swamped because not only with justified appeals but with those who've got the right result but think they might get bunged up a grade. I hope no key school staff are holidaying in France!

I'm also a little puzzled by the way in which the universities, having received the marks a few days early as they always do, did not realise there were massive problems and at least consider not rushing to confirm places, so that they could remain in a position to respond to appeals results or any general changes.

Good point, individual admissions tutors may not have noticed too much of a problem with cohorts of 50 or so but Faculties, certainly at Russell Group universities, should have. And universities will want to fill up numbers as soon as possible if there are fewer students to go round.

And what happens if the whole scandal is repeated in a few days' time with the GCSE results?

Inevitable, I would have thought. Plus there will be no 'back up' of being able to look back at that cohort's GCSE results. I hope that KS2 Sats are not being considered!

Education Committee transcription https://committees.parliament.uk/oralevidence/493/pdf/ (Q 479 onwards)

 

[JamesT]

There seem to have been several detailed examinations of the algorithm in various newspapers and on the BBC, although an official version doesn't seem to have been issued. There seem to be all sorts of possible criticisms, but to me the most unfair aspect of all seems to be this one of small school/course v. large school/course. If I've followed right (and I'm no statistician), if an outstanding pupil took let's say an A-level in Sanskrit that was being offered for the very first time in their small rural sixth form and the teacher predicted A*, that is the grade that would be awarded, with no intervention. But if the same outstanding pupil were taking, say, English Literature in a large bog-standard urban comp with a general mediocre record and if the teacher saw the outstanding talent and predicted A*, that would count for nothing, because if there had been no A* marks at that school in the preceding three years, then no-one could be awarded an A* in this year'a algorithm process. How can that possibly be a fair scheme of assessment?

My understanding was there was also supposed to be some sort of influence from GCSE results as a cohort getting top marks two years ago were likely to get good results in the same subjects this time. But I've not seen much mention of that actually working.

I'm also a little puzzled by the way in which the universities, having received the marks a few days early as they always do, did not realise there were massive problems and at least consider not rushing to confirm places, so that they could remain in a position to respond to appeals results or any general changes. As it is, a number of places seem to have filled their quotas already, so that any candidates who get their marks changed to such as would meed their admissions requirements are going to be told to come back next year. How can it possibly be right to delay those who have met requirements, even if that has only been established on appeal, whilst letting in those who perhaps did not meet requirements in the first place? Particularly when so many questions have already so quickly been raised about the soundness of the whole system. And what happens if the whole scandal is repeated in a few days' time with the GCSE results?

The main problem at universities is they're geared up for the normal distribution of results. Many will typically over offer on the number of places they have, based on the knowledge that in a normal year a percentage of applicants will miss their grades or decide to go elsewhere etc. The objective of the algorithm was to moderate the results back down to a more normal distribution, i.e. within a few percent of previous years. So when the universities received the results they would look relatively normal, not something to raise the alarm.
I believe that conditional offers are binding, the university has no choice but to give a place to someone who has achieved the grades their offer was based on. (I don't know whether this is also the case if the grade is achieved after an appeal). So they couldn't wait and see with this.
So they're now rather stuck. They can't tell people who got grades based on the moderation to go away, (on what basis could you?) and they don't necessarily have space this year to take everyone if they succeed on appeal or reverted to the predicted grades for all.

 

[Bayum]

There was an interview with an Oxford admissions tutor this morning. She spoke about the way that the results were given using the statistical models.

Looking at the results, we discovered something far more disturbing. Disadvantaged students were being further disadvantaged by the way in which grades were given. The algorithm literally copied the inequalities already in the education system. We made offers of 83% our places to state schools which was a big thing for us. Nearly a quarter of those students saw their grades downgraded, whereas it was only a tenth of private school offer holders who were downgraded. Ofqual did something to the teacher predictions that made grade distribution across the whole country look right, but they can't possibly tell us that they've given the right grades to the right people.

 

[chorleyjeff]

It’ll certainly be interesting to see how it goes. I was talking with my Dad about it the other day (he is a secondary teacher). The part that surprised me most was the SQA were changing results without looking at a pupils work from throughout the year. I think that’s what’s caused the controversy up here - results seemed to have been changed to fit statistics, and not based on any work.

Good luck. I hope you get what you’re looking for.

In my day results were moderated to make sure that there were not many As. And numerous people were very disappointed with actual results. It was, after all, just a filter to restrict university access. As for the rest two grade Es was enough to get access to professional qualification courses.

 

[Belperpete]

In my day results were moderated to make sure that there were not many As. And numerous people were very disappointed with actual results. It was, after all, just a filter to restrict university access. As for the rest two grade Es was enough to get access to professional qualification courses.

The results have always been moderated. As anyone who has tried writing exams will tell you, the only way to get two exams equally difficult is to make them identical. While a lot of effort goes into making the exams equivalent, inevitably there will be minor differences that can only be overcome by moderating the results. However, this is done equally across the whole intake of a particular exam, so that the grade boundaries are set at particular percentages to get roughly the same mark distributions as previous years. What has been done this year is totally different. Small groups of students are having their grades adjusted in totally different ways, depending on their particular school's past results.

It does seem as though the algorithm used is way too simplistic, such that small variations in previous years' performances can have a dramatic effect on individual students this year. A case was given on the radio of a subject where the students at a particular school normally achieved As or Bs, but because one student in the last 3 years got an E, then the lowest achieving pupil this year was marked down from a B to an E. If true, that is ridiculous. In my opinion, anyone who has been downgraded by more than one grade should have their result reviewed as a matter of course.

 

[Typhoon]

It does seem as though the algorithm used is way too simplistic, such that small variations in previous years' performances can have a dramatic effect on individual students this year. A case was given on the radio of a subject where the students at a particular school normally achieved As or Bs, but because one student in the last 3 years got an E, then the lowest achieving pupil this year was marked down from a B to an E. If true, that is ridiculous. In my opinion, anyone who has been downgraded by more than one grade should have their result reviewed as a matter of course.

Too right. One in 3 years is obviously an outlier. Presumably this description accounts for the (named) student

from Telford, told BBC Radio 4's Today programme that her results had been downgraded from As and A* grades to a B, E and U.
"I know I didn't sit the exam but I didn't think I'd be treated as if I didn't turn up for the exam," she said.

(https://www.bbc.co.uk/news/uk-53791736 - about halfway down the page). Now it could be that the school overestimated the grades but not by that much; presumably she was last on the ranking list so gets the 'U' because someone failed last year.

I still think this could have been detected weeks ago and something done - it must be incredibly stressful for the students. their parents and teachers. The more I hear, the worse it appears.

 

[thejuggler]

There are two ways this will probably go.

Universities will be desperate to get students signed up, they will ignore actual grades and accept on the basis of offers and mock exams.

There will be a legal challenge via Judicial Review of the system, especially the ranking which sounds a shambles.

Just read that lawyers are already looking into the legality under GDPR of using school results data from the last three years for the algorithm.

 

[Bayum]

It does seem as though the algorithm used is way too simplistic, such that small variations in previous years' performances can have a dramatic effect on individual students this year. A case was given on the radio of a subject where the students at a particular school normally achieved As or Bs, but because one student in the last 3 years got an E, then the lowest achieving pupil this year was marked down from a B to an E. If true, that is ridiculous. In my opinion, anyone who has been downgraded by more than one grade should have their result reviewed as a matter of course.

The algorithm used is based on statistical modelling of a school’s previous results and the ranking system. What we have found, unfortunately, is that the ranking system still takes into account deprivation, the discordance for BME students, richer areas as well as disadvantaged pupils.

 

[Bayum]

Interestingly, Ofqual released information this morning stating exactly what a ‘mock exam’ entailed and the various bits and pieces which it ruled as ‘worthy’ to be acceptable. This was the basis on which centres would be able to appeal the results of students on their mock exams. Again, mock exams are widely variable and based on the centre. Some will do end of year exams on a regular basis which will focus on the breadth of the syllabus, whereas others will do end of module and end of year exams, whilst others will do something completely different.

Tonight, Ofqual have reneged this advice with no explanation for why.

One of the things that really annoys me recently is this assumption that teachers have graded their students too high. Under the Labour government, there were cries every single year of the exams being ‘dumbed down’ and it being a race to the bottom. What we have seen each year since the new ‘challenging’ syllabi have been in place are consistent rises in the results year after year, yet Ofqual, the Government and the Press don’t take this into account. Teachers are working their asses off, students are working their asses off and we are seeing the fruits of their labour - yet we are still willing to focus on a statistical curve of exam results from previous years.

 

[ScottGallagher]

I feel sorry for this school year. They will have this cloud over their grades for the rest of their lives.

The idea though that some teachers won't have committed skulduggery in all this to bump grades isn't credible. I took my GCSEs in 1998 when your coursework between years 10 and 11 counted at least 25 percent of the grade in most subjects. Coursework was "marked" in order to redraft.
In my school I noticed students who were say predicted A and B and sometimes C grades were basically cut adrift. The idea was to zero in on D students, not to help them per say but to get them over the line to a C to make the school or borough stats look better. I don't know about now but back then, schools were measured on A*-C grades as one bracket overall.

 

[si404]

The idea though that some teachers won't have committed skulduggery in all this to bump grades isn't credible.

Indeed - predictions are a different measure to typical results. They don't take into account a lack of revision, or getting into the exam room and tanking - and so are some times higher. They are estimates of potential and, while some are underestimates, they are rarely overestimates: just that, come exams, people might not make their potential.

They will have this cloud over their grades for the rest of their lives.

Indeed - it's not their fault at all that they weren't allowed to sit exams, and the mucking around of the last couple of weeks would never have removed this cloud as algorithms, mocks, etc instead of predicted grades is still a different yardstick people are being measured against than other years.

 

[Senex]

The more we hear about OfQual's algorithm and the impact it has had on large numbers of able young people it has had, the less defensible something designed to ensure consistency across the whole but with no care for the performmance of individuals (unless they happened to be from groups of less than the magic mnumber of five) seems to be. We cannot afford such an attitude towards the brightest of our young people. But who is going to intitiate the U-turn / climbdown that is now despersately needed? We live in an age when no Westminster politicians ever seem to accept that they or their departments might be responsible for a mistake.

 

[si404]

We live in an age when no Westminster politicians ever seem to accept that they or their departments might be responsible for a mistake.

The fiasco is people trying to fix mistakes they admit they made, but doing so badly...

There was an initial decision to cancel exams and use predicted grades.

Once all this was collated, using predicted grades was flagged as a mistake by OfQual, and - backed by government(s) and Teachers Unions - made an algorithm to fix the 'grade inflation'.

Then the results in Scotland came out, and the Holyrood politician managed to keep supporting the algorithm for a couple of days before admitting a mistake.

The relevant politicians in Cardiff and Westminster - who lucked out by the results day being a week later and would have done exactly the same - got to admit their mistakes ahead of time.

All three (I've not heard anything about Northern Ireland, but probably them too) then tried to fix these mistakes on the fly...

 

[Senex]

The Scottish government did admit they had stuffed up (a stronger and coarser expression would be more appropriate). But I haven't heard Gavin Williamson say, "We got this badly wrong" yet.

 

[DelayRepay]

This year's students have my sympathy.

It seems that the algorithm has, at a macro level, awarded the correct grades given that the overall distribution is similar to previous years. But it doesn't seem to have awarded the correct grades to individual students.

In reality, not all students with predicted A*s would have actually achieved A*s. Some would have had a bad exam for whatever reason and ended up with a lower grade. The algorithm has worked on the basis that whichever A* student was at the bottom of the ranking would have been the student who didn't get the A*. In reality it could have been any of them. But there is no way of knowing, across the whole population of students, which ones would not have achieved their projected grades. Obviously we don't hear so much from students who got an A but think the exam would realistically have resulted in a B.

Overall it just seems unfair, especially for those who have lost university places or been 'awarded' a U. I am waiting for a government U-turn, but I also expected them to sack Cummings so who knows what will happen. I see there is a legal challenge being prepared.

The only positive I can see is that this will turn a generation against the Conservative party.

 

[eoff]

I don't think (many) teachers have intentionally inflated grades. What's probably more likely to have happened is that if you have a group of borderline students who could pass the exam on a good day, realistically in an exam perhaps only half of them will pass, but teachers will be inclined to give all of them the benefit of the doubt. There's no doubt the grades estimated by teachers were far higher than anything likely to have been achieved had the exams gone ahead.

This was my concern and is exactly the sort of effect an exam board would not like to admit to.

 

[eoff]

A few questions/observations, and I apologise if they were already answered.

Would the algorithm used here count as an automated decision process without human involvement under GDPR, If so then people have a right to appeal and ask for a decision by a human. From the ICO website..

"Individuals have the right not to be subject to a decision when:

• it is based on automated processing; and
• it produces an adverse legal effect or a significantly affects the individual.

You must ensure that individuals are able to:

• obtain human intervention;
• express their point of view; and
• obtain an explanation of the decision and challenge it."

(https://ico.org.uk/for-organisation...t-to-be-subject-to-automated-decision-making/)

Secondly, given my understanding so far it seems that a pupil moving school could get different marks than they would have got if they had not moved (say from a private school to a historically low performing school resulting in a downgrade of 2 grades).

Third, I presume that there are only some valid grounds of appeal and these would not include "I don't agree that my marks should be graded low based on the results of people in my school who took exams in previous years".

 

[Domh245]

The only positive I can see is that this will turn a generation against the Conservative party.

*further turn a generation against them. The Tories seem hell bent on destroying their support with anyone under the age of 50, and have been doing so for a good few years now!

 

[DelayRepay]

*further turn a generation against them. The Tories seem hell bent on destroying their support with anyone under the age of 50, and have been doing so for a good few years now!

But in this case, it's not just the students themselves. It's the parents and grandparents too.

 

[si404]

The only positive I can see is that this will turn a generation against the Conservative party.

Why? The issues of downgrading poor students for being at rubbish schools were the same, if not slightly worse, in Labour-run Wales and SNP-run Scotland.

It's a political problem - not a party political problem.

 

[Senex]

Why? The issues of downgrading poor students for being at rubbish schools were the same, if not slightly worse, in Labour-run Wales and SNP-run Scotland.

It's a political problem - not a party political problem.

It wasn't just poor pupils at rubbish schools. It was also very good pupils at decent state schools. As for the parties, yes, they all got things wrong, but at least the SNP acknowledged the problem quickly and dealt with it. Maybe their response wasn't perfect, but they acted boldly when action was desperately needed.

 

[si404]

It wasn't just poor pupils at rubbish schools. It was also very good pupils at decent state schools.

Indeed, but the 'target the poor' (I meant financially poor, not academically poor) is an anti-Tory dig I've seen by people trying to pin an algorithm that was devised by the Independent body OfQual, and had the backing of Teachers Unions and all the main political parties, on "the baddies".

I note you say 'state schools' - private school pupils have had the same problem, albeit on a lesser scale (due to smaller classes being exempt for statistical reasons).

at least the SNP acknowledged the problem quickly and dealt with it.

As did Welsh Labour and UK Conservatives - but they did so before results day because they had the luxury of an extra week.

Maybe their response wasn't perfect, but they acted boldly when action was desperately needed.

You don't think stuff like Williamson's announcement that "your mocks can count if they are higher" isn't bold action? Perhaps Sir Humphrey would call it courageous, and I would definitely call it foolhardy, but it certainly was bold action when action was needed!

There's an institutional failing here, with quangos and unions and politicians of many colours all making/complicit to mistakes - and if anyone is dumb enough to think the blame falls on just one political party, then they lack the thinking to deserve good grades.

 

[DelayRepay]

You don't think stuff like Williamson's announcement that "your mocks can count if they are higher" isn't bold action? Perhaps Sir Humphrey would call it courageous, and I would definitely call it foolhardy, but it certainly was bold action when action was needed!

Although this was announced, there doesn't seem to be any way for a student to actually request that their mock exam result be used instead of the grade awarded by the algorithm. The appeals process published on Saturday, and then withdrawn, did not do this. And it's still an unfair solution - what about students who didn't do a mock, or didn't put much effort in because mocks don't normally mean anything? What about students who go to schools that deliberately under mark mocks to give the students a shock and make them work harder? What about the students who put in huge effort after the mock to improve their performance?

So maybe 'bold' but not especially helpful.

There's an institutional failing here, with quangos and unions and politicians of many colours all making/complicit to mistakes - and if anyone is dumb enough to think the blame falls on just one political party, then they lack the thinking to deserve good grades.

Ultimately, the buck stops with the Secretary of State. I have not seen evidence that the unions agreed the specifics of the model, and the outcomes, but even if they did the government has a track record of ignoring the unions when it suits them. I'm not sure how other parties can be blamed for this situation in England, when we have a Conservative government.

I am not of the view that the SoS instructed Ofqual to devise a system that deliberately favoured certain types of school and student, but for whatever reason that is what we have ended up with. They have had weeks to fix this but have not done so. The appeals process is complete confusion. Students were told they could use their mock grades, but there is no way to do this in practice. Appeals processes published then withdrawn a few hours later. And all the while talented students are missing out on university places. Even if their grades are re-assessed/improved, what are they supposed to do if in the mean time their university place has been given to someone else?

 

[Bayum]

Indeed, but the 'target the poor' (I meant financially poor, not academically poor) is an anti-Tory dig I've seen by people trying to pin an algorithm that was devised by the Independent body OfQual, and had the backing of Teachers Unions and all the main political parties, on "the baddies".

I note you say 'state schools' - private school pupils have had the same problem, albeit on a lesser scale (due to smaller classes being exempt for statistical reasons).
As did Welsh Labour and UK Conservatives - but they did so before results day because they had the luxury of an extra week.
You don't think stuff like Williamson's announcement that "your mocks can count if they are higher" isn't bold action? Perhaps Sir Humphrey would call it courageous, and I would definitely call it foolhardy, but it certainly was bold action when action was needed!

Private schools have had it on a far less scale than state institutions. Pupils have been marked up/down based on postcode lotteries. You say it’s all down to Ofqual - Williamson chose to go with it and is staunchly sticking by it. His ‘triple lock’ guarantee is a load of sh**e. Even Ofqual have recognised and understood the vast limitations of the use of mock exams.

 

[PHILIPE]

The PM himself is still missing instead of being out there and leading in this crisis

 

[wireforever]

'Isn't the PM on a 2 week holiday 'crisis what crisis'A tory MP has called for a statement by 5pm today to resolve the matter.Perhaps the education secretary will say Ofqual is to be scapped as it is all their fault?

 

[oldman]

If we did simply go by teacher assessment (for one year only), would it be such a bad thing?

Those who would have done better in the exam than the algorithm thought will get what they deserve. Others will get more than they deserve - it may open a door through which they will stride purposefully onto a path leading to the sunlit uplands (other metaphors are available) - a potential realised. Or it may be a step too far and they stumble but then, hopefully, find a better path for them.

In one of my A-Level subjects, the entire cohort did worse than the mock exams had predicted - those expected to pass well, scraped through; those expected to pass so-so, failed; no one got better than grade E. I was saved by the fact that, aged 11, I had chosen a scarcity subject where the universities couldn't be picky. It was a very lucky break.

Sometimes people get lucky breaks. Mostly it's the children of the rich and powerful; it would be nice if sometimes it's other people.