Should we stop using exams for assessment permanently?

[Bletchleyite]

There's more to ICT than Information Technology - IT doesn't include the Communications side of things.

I disagree. Pretty much the entire industry is about some form of communication these days. It's a term invented by schools that is simply not used in the real world. Schools could do with packing this in and using real world terminology.

 

[najaB]

It's a term invented by schools that is simply not used in the real world.

I suggest then that you let people like the EU, HM Government, ESTI, The UN, and many others know because they seem to be making the mistake of using it in the real world.

 

[si404]

Schools could do with packing this in and using real world terminology.

When I did ICT GCSE, it would have been better termed "BS".

It was almost about jumping through the coursework mark schemes hoops rather than actual ability with office programs' functions.

Which meant I had:
- about 60 hours of teaching on what I needed to do in the coursework, worth 3/4s of the GCSE
- 3 hours on how to do some of the more obscure functions we had to use in the coursework.
- 3 hours (including the 3 hour mock, which they gave us 2 hours to do) on the exam.

Total waste of time. And the ICT department were trying to dissuade people who weren't natural BSers from doing A level as it was more of the same and not the CompSci class people wanted - though every year some got through the net and ended up dropping out, often with a bad grade at AS level (typically inversely proportional to their ability with computers).

 

[Belperpete]

You didn't know what the question asked would involve, so were expected to remember loads of quotes in the chance that one of them would be appropriate.

When I took Latin, there were two syllabuses (syllabii). The one that most schools took used set books, and any translation that you would be required to do in the exam would only come from one of these set books. Pupils who took this syllabus tended to be taught to remember vast chunks of Latin text. Our school did the other syllabus, where you could be presented with any Latin text to translate. I soon worked out that the Romans wrote mainly about two topics: sex and war. We weren't going to be given anything about sex to translate, so I memorised all the words most likely to be used in a text about war (dagger, spear, kill, maim, etc.), and so achieved a reasonable pass. Having said that, the only Latin word that I can still decline all these years later is amo, amas, amat..... (I love, you love, .....). None of it came in any use during my career, but then that wasn't the point of O Levels.

 

[Belperpete]

Can't stand [multiple choice], easy to mark, unreliable test of ability (the monkey gets 25%). When I started teaching, we had multiple choice with 5 answers, 4 marks for a right answer, -1 for a wrong one!

The fact that you can get 25% without knowing anything just means that the base starts at 25%, not 0%.

Having both set and marked multiple choice papers, while they may be easier to mark, they are most definitely harder to set. Trying to find one unambiguously right answer, and 3 unambiguously but not too obviously wrong answers is very difficult.

Exam technique is very important in any exam, and I am certain that I only passed some of my O Level exams through it. In the case of multiple choice papers, I used to advise my students:
1) if you pick an answer, you stand a 25% chance of getting it right
2) if you don't put anything, you stand 0% chance of getting it right, so always put an answer even if you don't have a clue.
3) you can usually rule at least one or possibly two of the answers out. Even if all you can do is rule two of the answers out, that still leaves you with a 50% chance.

Multiple choice papers are good at testing memory. They are not very good at testing reasoning, which is why you tend not to find them in degree papers.

 

[Belperpete]

I've been surprised how much harm the amount of stress the exams caused and the effect they've had, the course itself was incredibly useful and finishes with practical tests where you have to put all you've learnt to use but the exams were focused much more on being able to memorise certain facts instead which is no use in the real world.

I fully agree about the stress. I had a couple of friends who were physically sick while actually sitting exams through stress. Unfortunately, while there are problems with exams, I think they are the least-worst alternative. With course work there is too much opportunity for cheating, copying others work, parental assistance, and subjective marking by teachers.

Almost none of the information that I ever learnt at school or at university, aside from basic maths and english, has ever been of any use to me at work. I suspect that the same applies for most people. Most jobs are inherently very specialised, and so inherently require specific on-the-job training to do them. All that exams do is to prove that you are capable of the relevant level of learning, and have a general aptitude and enthusiasm for certain broad subject areas.

When I went through the sequence of O levels, A levels and then degree, I found that the O Level papers were almost entirely designed to test rote-learning of basic information. The more O level papers you passed, the better your ability to rote-learn facts. A levels were aimed more at how you applied information. For example, in A level maths we were told that if we were running out of time, to not bother working out the answer - most marks were given for method (such as picking the right formula to solve a particular problem), and very little for doing the basic number-crunching (that in real life you would probably get a calculator or computer to do). A levels proved that you were capable of more than rote learning, you could apply what you had learnt. When I was revising for my A levels, I looked back at my old O Level papers and was horrified at how little I could answer. If you don't keep refreshing memories, you lose them, and A level exams were not about regurgitating lots of facts like O level exams were.

By the time I got to the final year of my degree course, it was all about my understanding of the subject. Again, virtually none of it would be relevant to my real-life job, but then it is quite common for people who obtain a degree to go onto a job that has very little relevance to the subject they studied. Having a degree proves that you can study on your own initiative and gain a detailed understanding of a subject.

I was sponsored through university, and when my boss demanded to know why I had chosen final year subjects that seemed to have little relevance to my future job, my reply was that none of them seemed to have much relevance, so I picked the ones I was most likely to do well in. He agreed.

at university we had a module where we worked in groups of 10-12 for something like ten weeks with each group given a set of requirements we had to formally specify then implement and unlike most of the university courses I did was very similar to a real work environment.

How did they assess which of you had actually done the work, and who had ridden on the others' coat-tails?

 

[Bletchleyite]

How did they assess which of you had actually done the work, and who had ridden on the others' coat-tails?

Direct supervision by the group's tutor, in my experience.

 

[Meerkat]

It’s easy for bluffers to get less than the monkey score in a multiple choice as they usually put in an answer that is wrong but has semi relevant words to catch out those who weren’t paying attention in class and/or didn’t revise.
The finance industry exams i did at work were utter sods for that
Which brings us to the issue that kids need to do exams at school so they are ready for doing exams at work

 

[najaB]

They are not very good at testing reasoning, which is why you tend not to find them in degree papers.

I don't know if it's still the case, but medical course quite often use multiple choice papers with a large number of questions - say 120 - to be done in one hour, where there is one very wrong answer and two neutral answers. So, for example:

"What drug should you prescribe for a patient presenting at A&E with a heart rate of...":

• A. Drug that has no effect
• B. Drug that will kill them
• C. Drug that has no effect
• D. Correct drug

The idea being to put the candidate in a high-pressure situation and see if they are likely to crack. I agree that outside of medicine or similar professions they are of little use.

(As an aside, it's fairly rubbish that Xenforo doesn't seem to believe in ordered lists using letters!)

 

[Jozhua]

When I took Latin, there were two syllabuses (syllabii). The one that most schools took used set books, and any translation that you would be required to do in the exam would only come from one of these set books. Pupils who took this syllabus tended to be taught to remember vast chunks of Latin text. Our school did the other syllabus, where you could be presented with any Latin text to translate. I soon worked out that the Romans wrote mainly about two topics: sex and war. We weren't going to be given anything about sex to translate, so I memorised all the words most likely to be used in a text about war (dagger, spear, kill, maim, etc.), and so achieved a reasonable pass. Having said that, the only Latin word that I can still decline all these years later is amo, amas, amat..... (I love, you love, .....). None of it came in any use during my career, but then that wasn't the point of O Levels.

Of course they didn't have anything about sex!

Horrific violence is okay as long as there's no nooky words

 

[edwin_m]

When I took Latin, there were two syllabuses (syllabii). ... Having said that, the only Latin word that I can still decline all these years later is amo, amas, amat..... (I love, you love, .....).

You've kind of proved your point there - the Latin plural of syllabus is syllabi. Sillabii would be the plural of syllabius.

 

[Bletchleyite]

I don't know if it's still the case, but medical course quite often use multiple choice papers with a large number of questions - say 120 - to be done in one hour, where there is one very wrong answer and two neutral answers. So, for example:
The idea being to put the candidate in a high-pressure situation and see if they are likely to crack. I agree that outside of medicine or similar professions they are of little use.

(As an aside, it's fairly rubbish that Xenforo doesn't seem to believe in ordered lists using letters!)

The other thing you can do with them is score creatively, e.g. +1 for the correct answer, -1 for the dangerous one and 0 for the other two.

 

[Adam Williams]

Tentative yes, or at least a "we should consider reducing our reliance on large final exams which make up a big proportion of grades".

Few reasons for me:

1. From a Computer Science perspective: The number of graduates who finish their course in a position where they're useless at doing practical software engineering work is far too high. Given that the proportion of marks for practical tasks (go write a parser and interpreter, go implement some common data structures in Java etc) hovered between 20% and 30% for me and it was often quite difficult to fail these courseworks, I am really not surprised to see grads who struggle to actually write software and work collaboratively. I suspect that if less of an emphasis was placed on doing dumb things like writing code on a piece of paper with a pen, we'd see an improvement here.
   1. Obviously if you're going to go for more courseworks, you need them to be relevant to work that's actually done in industry. This will be more difficult for university departments with poorly maintained industry connections.
   2. Yes, plagiarism is a risk. I think it can be mitigated by something like Moss (Turnitin for humanities/essay-based assignments) and by discussing submitted solutions with students afterwards.
2. We've discovered very fast that we're screwed for determining grades now that modular A Levels and other qualifications have been scrapped, because there's very little data to go off before the final exams are taken. Making everything linear was a dumb idea in the first place (thanks Michael Gove), made it much more difficult for students to remedy failure and has greatly increased the amount of stress that we put on students in our education system. A return to something modular, even if still somewhat exam based, would be a step in the right direction.

 

[Typhoon]

We weren't going to be given anything about sex to translate, so I memorised all the words most likely to be used in a text about war (dagger, spear, kill, maim, etc.), and so achieved a reasonable pass. Having said that, the only Latin word that I can still decline all these years later is amo, amas, amat..... (I love, you love, .....). None of it came in any use during my career, but then that wasn't the point of O Levels.

Our school must have done the other syllabus. Caesars Gallic Wars wasn't too bad but the poetry was awful, made Vogon poetry seem good. I now find some of it was filthy, literally. (We didn't get that, just poems to canaries or gerbils or chairs or something).

Having said that, the only Latin word that I can still decline all these years later is amo, amas, amat..... (I love, you love, .....). None of it came in any use during my career, but then that wasn't the point of O Levels.

I'm sorry sir, but I am going to have to disappoint you. You conjugate verbs and decline nouns. (So you decline Mensa, Mensa, Mensam, Mensae, …). And I failed Latin but have retained a load of useless junk in my brain about passive and subjunctive. It didn't even help with English. A list in the Exam of the more unlikely words that might occur would have helped but you were expected to have an encyclopedic knowledge of all sorts of obscure words, it didn't even help that the verses didn't rhyme. Pointless!

 

[najaB]

The other thing you can do with them is score creatively, e.g. +1 for the correct answer, -1 for the dangerous one and 0 for the other two.

They used to do something like that. I think it was also -½ for questions that were blank.

 

[Typhoon]

They used to do something like that. I think it was also -½ for questions that were blank.

One syllabus for GCSE Maths had a limited number of multiple choice questions within the exam. The intention was to be mark neutral. I think there were 5 questions and 5 options for each so you would expect to get 1 right and 4 wrong if you guessed. There were 4 marks for a correct answer, -1 for a wrong answer, 0 for no answer (to discourage guessing). So the student who guessed would expect 0 marks. It wasn't popular!

 

[DynamicSpirit]

Tentative yes, or at least a "we should consider reducing our reliance on large final exams which make up a big proportion of grades".

Few reasons for me:

1. From a Computer Science perspective: The number of graduates who finish their course in a position where they're useless at doing practical software engineering work is far too high. Given that the proportion of marks for practical tasks (go write a parser and interpreter, go implement some common data structures in Java etc) hovered between 20% and 30% for me and it was often quite difficult to fail these courseworks, I am really not surprised to see grads who struggle to actually write software and work collaboratively. I suspect that if less of an emphasis was placed on doing dumb things like writing code on a piece of paper with a pen, we'd see an improvement here.
   1. Obviously if you're going to go for more courseworks, you need them to be relevant to work that's actually done in industry. This will be more difficult for university departments with poorly maintained industry connections.
   2. Yes, plagiarism is a risk. I think it can be mitigated by something like Moss (Turnitin for humanities/essay-based assignments) and by discussing submitted solutions with students afterwards

I kinda agree with some of that - particularly, asking people to do coding with pen and paper seems pretty daft. However I would observe that you seem to be viewing computer science entirely from a perspective of, does it train you to be a good coder. And that's not really education, that's more like, training. I get that, from the perspective of a student who's paying £10K in fees and wants to recoup that in salaries, that's an important consideration, but I wonder whether that's ideal from the point of view of wider society. Yes, I'd expect someone with a computer science degree to have some experience of programming, to know what a compiler is - and indeed to have some familiarity with a variety of different languages from different domains - from conventional programming languages like Java or C# to more domain-specific things CSS, HTML and SQL. But from the perspective of general education and wider society, I'd have liked to think that a computer science graduate would have a good understanding of the role of computers in society and how computers have shaped society over the last few years. They'd be aware of ethical issues about data collection and so on. They'd know something about the principles of good UI design, and be aware of accessibility issues. They'd hopefully know about hardware and how computers work - and on a practical side, would be comfortable doing things like taking a PC to bits to replace a CPU (OK, that one is heading towards training too). They'd be aware of AI and what kinds of things are and aren't likely to be achievable with AI, and be able to think at least a little about the impact of it on the economy. They'd understand issues about standardisation - why standards exist, and, if pushed, be able to read and roughly understand an ISO document or an RFC. And they'd be familiar with security and the principles behind how to prevent both people and businesses becoming victims of security hacks and scams. In short they'd have a well rounded understanding of both much of the technology behind computers and how computers interact with society.

Off the top of my head, I'm not really familiar with University computer science syllabuses, but I'd have thought that some (obviously not all) of those things would be appropriate to test by traditional exam.

Or am I being too old fashioned?

 

[Jozhua]

I kinda agree with some of that - particularly, asking people to do coding with pen and paper seems pretty daft. However I would observe that you seem to be viewing computer science entirely from a perspective of, does it train you to be a good coder. And that's not really education, that's more like, training. I get that, from the perspective of a student who's paying £10K in fees and wants to recoup that in salaries, that's an important consideration, but I wonder whether that's ideal from the point of view of wider society. Yes, I'd expect someone with a computer science degree to have some experience of programming, to know what a compiler is - and indeed to have some familiarity with a variety of different languages from different domains - from conventional programming languages like Java or C# to more domain-specific things CSS, HTML and SQL. But from the perspective of general education and wider society, I'd have liked to think that a computer science graduate would have a good understanding of the role of computers in society and how computers have shaped society over the last few years. They'd be aware of ethical issues about data collection and so on. They'd know something about the principles of good UI design, and be aware of accessibility issues. They'd hopefully know about hardware and how computers work - and on a practical side, would be comfortable doing things like taking a PC to bits to replace a CPU (OK, that one is heading towards training too). They'd be aware of AI and what kinds of things are and aren't likely to be achievable with AI, and be able to think at least a little about the impact of it on the economy. They'd understand issues about standardisation - why standards exist, and, if pushed, be able to read and roughly understand an ISO document or an RFC. And they'd be familiar with security and the principles behind how to prevent both people and businesses becoming victims of security hacks and scams. In short they'd have a well rounded understanding of both much of the technology behind computers and how computers interact with society.

Off the top of my head, I'm not really familiar with University computer science syllabuses, but I'd have thought that some (obviously not all) of those things would be appropriate to test by traditional exam.

Or am I being too old fashioned?

Yeah, that makes sense, I mean some of that background learning is definitely important!

In regards to AI, I always find it hilarious when some relatively advanced chatbot or robot comes on the news and they ask if it'll suddenly become sentient

 

[Adam Williams]

I kinda agree with some of that - particularly, asking people to do coding with pen and paper seems pretty daft. However I would observe that you seem to be viewing computer science entirely from a perspective of, does it train you to be a good coder. And that's not really education, that's more like, training.

I'm definitely guilty as viewing it as an investment to enable work rather than a balanced education that you do for the sake of broadening your understanding. As you point out, the fees factor into it. I had pretty high expectations for what I'd get for spending £30+k.

I'd have liked to think that a computer science graduate would have a good understanding of the role of computers in society and how computers have shaped society over the last few years. They'd be aware of ethical issues about data collection and so on.

There was some of this in one of the modules I did in my degree - e.g. we discussed "disruptive technologies", the gig economy, usability etc. Not as much as I'd have liked and it was never really delivered in an engaging way which then lead to complaints about the content not being particularly interesting.

They'd be aware of AI and what kinds of things are and aren't likely to be achievable with AI, and be able to think at least a little about the impact of it on the economy.

I think this was one area that was too theoretical, which a lot of Russel Group universities sometimes fall into the trap of doing. I remember doing what felt like a dozen different search algorithms, Bayesian probability and a bit of game theory. There was very little/nothing in the way of looking at AI in general, discussing its impact on society/the economy or really considering the "big picture".

They'd hopefully know about hardware and how computers work - and on a practical side, would be comfortable doing things like taking a PC to bits to replace a CPU (OK, that one is heading towards training too).

Yes to the former from a theoretical perspective, no to the latter (but we did do some basic hardware/electronics - I think we made an adder in the lab).

They'd understand issues about standardisation - why standards exist, and, if pushed, be able to read and roughly understand an ISO document or an RFC.

I think this in particular was neglected which was a bit of a shame. There was much more focus on being able to read academic papers for the sake of being able to do research than e.g. actually showing us IEE 754 when discussing floating point, or RFC 793 when introducing networking concepts.

Off the top of my head, I'm not really familiar with University computer science syllabuses, but I'd have thought that some (obviously not all) of those things would be appropriate to test by traditional exam.

Or am I being too old fashioned?

I guess they could be tested by exam, but a lot of these I think could be approached with an essay assignment (which can only help technical writing skills later on) too. At the end of the day you want to ensure grads can think critically too, and put across their ideas in writing. To me, this seems more representative of what we might ask a graduate to do in the workplace, or what they might do if they stay in academia and end up doing their own research.

 

[najaB]

They'd hopefully know about hardware and how computers work - and on a practical side, would be comfortable doing things like taking a PC to bits to replace a CPU (OK, that one is heading towards training too).

That's most definitely not computer science.

 

[Adam Williams]

That's most definitely not computer science.

I'd argue the former is, falling under the umbrella of computer architecture (and there's scope to explore e.g. the Von Neumann & Harvard models of computation, memory systems, modern CPU design etc).

The latter, practical stuff.. yeah, not really pure computer science. But still useful, I'd say. It gives some context to all the theoretical work.

 

[yorkie]

They could start by stopping calling it that. It's an invention of the education sector, similar to "CDT", "Food Tech" and the likes. We are the IT industry. They are the manufacturing or engineering industries, or the food production or catering industries. Start calling the subjects what they actually are in the real world and stop running petty little isolated fiefdoms.

It's not called that any more. Last I heard, there was no such thing as GCSE ICT any more. Or, if there is, hardly anyone does it and most schools don't offer it. The choice at GCSE level is generally between Computer Science and Creative iMedia.

If you'd like to know what the iMedia course involves, see https://www.ocr.org.uk/Images/115888-specification.pdf; it is mostly coursework. I've heard some schools choose not to offer it any more though because the burden of marking coursework on teacher workload is high.

On the other hand, grades for GCSE Computer Science are mostly (or entirely) based on exams, but the course is generally only deemed suitable for students who have a good understanding of Maths.

 

[Bletchleyite]

FWIW when I did it (1995-1997) it was called Information Systems, and was based around the principles of business analysis, data storage and system design (though with a bit of coding thrown in). It would actually have been quite good as a way of starting someone off at 16 into an IT based Apprenticeship had those really been on offer back then.

I did an A-level in it at night school (as my school didn't offer it and I otherwise wanted to stay there). That was called "Computing" but was pretty much a techie/coding based thing like most university CS courses.

The department itself was known as the "IT department", though it sort of sat under "Technology" alongside the more practical subjects like what we then called CDT, though I have no idea what it's called now - i.e. woodwork, metalwork, working with plastics, technical drawing and electronics all in one. I think "Home Economics" also technically sat there but in practice there was a big male-female split and very little socialising between the two sets of staff.

 

[najaB]

The latter, practical stuff.. yeah, not really pure computer science. But still useful, I'd say. It gives some context to all the theoretical work.

Oops, I had meant to bold "would be comfortable doing things like taking a PC to bits to replace a CPU". You are correct that computer architectures are definitely computer science.

 

[Bletchleyite]

Oops, I had meant to bold "would be comfortable doing things like taking a PC to bits to replace a CPU". You are correct that computer architectures are definitely computer science.

They are, though slotting together hardware doesn't generally feature in CS courses. It's dead easy anyway.