[Random Musings] How I write mock exams - the sequel

 This is the second post of the series. In part 1 we discussed the prep work required to produce what I deem to be a reasonable exam paper at the level that the students are expected to achieve. Today we’ll focus on writing the actual questions and assembling them to form a complete paper.

Hey there!

Back with a second post on this. It’s been over a week since I first sat down and write, and since I’m not in the mood to do schoolwork it’s a good idea to just grab a coffee and share my thoughts.

Throughout this rambling, ER refers to examiners’ reports, NOT emergency rooms.

Even though I have read numerous papers about the theory behind assessment design and awarding process, I’m still a newbie when it comes to producing valid assessment tools. As outlined in part 1, I’m writing the papers on my own from my experience interacting with IB students as well as looking at what other exam boards are doing, and another person – ideally a subject specialist (shoutout to RShields#5160 for proofreading them and providing excellent comments on the content as well as the visual of the paper). My papers definitely aren’t perfect because of the lack of manpower, but I’ll still strive to produce better papers in the future. Keep an eye on this blog every now and then – I’ll try to blog about the papers I’ve read by including a summary of the paper and the takeaways that I could implement in my own writing process.

Step 5:  Write the actual questions

From the table of topics you’ve planned in step 4, write one question at a time, paying particular attention to the level of demand. I’m aware that exam writers cannot always accurately predict the difficulty of the question, but we’re just trying to replicate a decent-to-high quality paper to give students the experience of taking an exam. The problems with validity and reliability depends on how you want to use the data afterwards and the purpose of the assessment – personally, I’d rather use unit tests to clearly pinpoint the common misconceptions that students hold rather than RAG-ing the spreadsheet [1].

 

Key things that might be useful:

•     The first few questions should be relatively accessible to everyone – you don’t want your students to cry at question 2.
•      Starting 1/3 to halfway through the section, the difficulty starts to pick up and peaks at the last question. You might want to chuck in a slightly easier question around 2/3 of the section.
•      Every section B question has several parts – make sure that some parts are easier and some are harder to create a discriminating paper. If you want to refresh your memory, reread part 1 (add link) and read the second paragraph under Step 1.

 

It might be useful to look at the AQA’s graph on the level of difficulty (especially the Higher Tier) for section B questions – link, page 5. You want to have a steady rise of difficulty through the section, but all candidates should be able to attempt a part of the question. By “raising difficulty” I didn’t mean every successive part has to be harder than the previous – follow the development of the question – but there has to be some harder parts.

 

Advice

·       When you’re just started, you might only be able to write standard, routine, straightforward questions. The expertise will eventually come when you’ve written more exams. Looking at my first mock in 2020, I’ve had close to zero experience in writing mocks – the stuff I wrote myself is mostly standard and nothing particularly thought-provoking.

·       Authentic questions run the risk of not assessing the topics we want to assess, as students might be confused with the context or the wording.^[2]

 

Example

My AAHL Mocks (Set B 2021), question 11, reproduced below.

 

This question assesses polynomial functions and complex roots.

Since it is a section B question, it can combine several concepts under the same roof – in this case, transformations and further equations.

Here’s a peak of my thoughts when I was writing this question. It definitely isn’t perfect because it has been a while since I wrote this paper, but the key points are there.

·       This question is on functions, so we can test sketching a polynomial function. Of course we could combine transformations and using the function to solve another equation using a dummy variable – candidates should be well-versed with this.

·       Part (a) provides an entry for essentially ALL candidates. From all ER I’ve read, candidates have massive success with a similar type of question, either show that an expression is a factor or find the constant in f(x) if a factor is given.

·       The topic of conjugate roots is slightly more challenging but most candidates taking maths at this level should be comfortable with this. Slightly more algebra going on, from forming the quadratic that gives the two complex roots to finding the remaining factor, though this is still a routine question.

·       Part (c) is definitely routine – this is the (n+1)^th time that candidates see this type of question. The only key points we’re looking for are the intersections with the axes clearly labelled and shape and/or long-term behaviour of f(x) – easy routine 3 marks. Lack of care in the sketches will them 2 marks at the very least, so this shouldn’t be a huge issue.

·       Part (d) is slightly more challenging – aiming at grade 5s and above (grade B, in A-level currency) – as candidates have to unpack the transformations. The concept of finding the coordinates of a point after transformations is not well done compared to translating an entire curve, though I’ll need to reread some ERs to confirm this. This question could be tricky if candidates didn’t notice that the absolute value sign doesn’t affect the y-coordinate of the point, because it’s already on the -axis.

·    The point chosen on part (d) is intentional, as we’re given that (x-3) is a factor of f(x). If we decided to give another point that’s not already there, we’re giving students another way to answer the question and risking not assessing the concept of conjugate roots.

·       Part (e) is also somewhat predictable. Considering that this is the second of the four questions in section B, this is acceptable to me. Some rearrangements and manipulation of indices are necessary to show that the given equation has a similar form to the given function. Answers without sufficient working are not accepted. When the equation is formed, candidates write down their “roots”, which they have to reject the second part to get the correct answer. This is intentional, as I would also like to assess exponential equations.

·       If this is a question to be released as part of an official exam, I’m sure there will be a lot more rounds to refine this question, including changing the mark allocation. I’m specifically stingy when it comes to marks awarding – this is not an official paper, the grades aren’t gonna be awarded and there’s no external interpretation of the grades – so I could squeeze in more questions on the paper and assess slightly more content.

·       One thing I would change about this question is the wording of (d) – I should’ve clarified by writing “find the coordinates of A’” instead of simply “find A’”. It’s a minor change but could make a difference to some students – one pointed this out to me a couple weeks back.

 

Step 6: Write the mark scheme (this includes doing the paper yourself)

I don’t think I have to explain the importance of doing your own paper.

 

There’s a very high chance that you made an error in the question, say, a u-sub that doesn’t work, a question that’s way more demanding than expected compared to the rest of the paper and the paper as a whole is too easy or too challenging. You might also realise that you allocated too many marks or too few marks to one part/one question on the paper and it has a domino effect on the remaining questions.

 

Allocating marks

When you’re doing a paper, ideally you should time yourself then type up the MS, but otherwise, you could write the MS as you work through the questions. From your experience doing and reading mark schemes, you should have a good idea of how the marks are awarded. Make sure that other papers have a roughly similar “pace” – the number of minutes per mark – to the paper you’re writing. For example, a mark on CAIE A-level Maths requires a bit more than normal A-level (~1.4 minutes per mark vs ~1.2 minutes per mark).

 

Quick refresher on different kinds of marks

M     Method marks

        Evidence of a correct method/procedure being applied.

A     Accuracy marks

        Aka correct answers

()     Implied marks, usually implied method marks.

Stronger candidates tend to ignore/combine the steps (say, writing the formula with the numbers plugged in instead of doing them separately).

 AG Answer given – the entire solution must be checked if the candidate skipped any steps/illegal algebra to get the printed answer.

 

Usually a significant step is worth 2 marks (M1A1) and a simpler step is worth 1 (M1 or A1). However, when students mess up the last step, they tend to lose 2 marks (M1A1), and this could be prevented by lowering the number of marks for that question (aka ‘tariff’).

 

For example, part 11(b) from the photo was awarded as below.

·       A standalone mark for writing down the second root (in A-level, this would be a B1 mark)

·       M1 for finding the factor – A1 could have been included, but this only further penalises the candidate if they made an error here.

·       M1 for using long division or synthetic division or any valid method to factorise.

·       A1 is only earned if they achieved the printed line.

·       The last A1 is for writing down the roots – there’s no need to assign a method mark here because it’s straightforward, and the A1 is for finding the correct roots.

 

Step 7: Defend the papers

It’s crucial to have a fresh pair of eyes to check your papers before sharing em to the public. Since my background is not in mathematics (I’m currently doing chemical engineering), I really need a subject specialist to make sure I know what I’m doing and they can call me out if I mess up.

 

Step 8: All the presentation stuffs!

They are definitely more important than you think.

 

We have all had some instructors whose exams turned us off so much – either the formatting, lack of contrast between the question number and the body, improper maths notation, or just sloppiness in general.

 

Students can only perform their best when they’re comfortable, and we should make an attempt to make it a pleasing experience for them. Here are some tips that I used myself (check out the latest set of mocks [link] to see it in action).

·       Cover page – as close to the actual exam as possible, including headers/footers. However, there must be an element of “fakeness” – some students thought that the actual exams were leaked when they see the paperhead for the first mock I wrote.

·       Consistent font and consistent layout as in the official papers.

·       The answer lines are grey instead of black to enhance contrast – it’s definitely not eye-pleasing when you see several black lines together.

·       Line spacing between question parts and different lines of the same part – I use double spacing to distinguish parts and 1.5 spacing for text.

·       If a question takes two pages, you want to have the first page on the left and the other page on the right so students won’t have to flip back and forth and potentially miscopying their own answers.

·       For a section B question, to save space, I tried to put two questions on the same space, allowing at least 4-5 empty lines between them.

·       A blank page for the purpose of pagination must be clearly marked with sentences such as “Please do not write anything on this page.”

·       The bottom right of an odd page (except the cover) should be printed with an instruction for students to turn over. In middle school, several of my classmates didn’t know the existence of the remaining two questions on a physics test and lost 50% of the entire evaluation.

·       Colour blindness: please don’t use red and green together, or different shades of the same colour. This is just a principle of good teaching, and it obviously extends to exam writing.

 

And I think that’s all I would like to say for today!

It’s been a long post – approx. 2200 words when I wrap this up and added the references. I’ll post an update of this article if it’s of interest to others! C:

Andrew’s out!

 

References

[1] Article by Adam Boxer.

[2] Daisy Christodoulou, Making Good Progress?

[3] Interview with Christian, Mr Barton’s Maths Podcast