In a previous blog post, I listed five errors pretty much guaranteed to make teachers scream. In the first part of this blog, I explained how students could overcome four of them, but never explained the last one. In this blog, everything should become clear. Again, feel completely free to share with your students – in fact I would positively encourage you to.

If a student has called every type of research by its proper name instead of calling everything a “case study”, his or her examiner will have started to feel relaxed. If this student labelled all their graphs or charts correctly, the examiner will have known they were dealing with someone who cares about the quality of evidence as much as they do. If this same student avoided using “proves” when talking about evidence, the examiner will have started to get a little excited, thinking that they might have a future psychologist on their hands and will have fully let down their guard. If the student then avoided criticising the ecological validity of every study whether it’s relevant or not, the examiner may start to get a little excited. Now it’s time to move in for the kill.

5. The killer phrase!

In the first blog in this series, I explained why you should never say a result is “very significant” because significance is an all-or-nothing concept. Here I explain what you should say instead to show that you have a complete understanding of how statistical tests are used.

First learn this phrase:

• A 1% level of significance means that the probability of a Type I error is 1/100.

This is easy to remember because it’s full of ‘ones’ (1%, Type I, 1/100) and so you can focus on the other terms.

Then learn this phrase:

• A result that is significant at this level is unlikely to have happened by chance and can be generalised with confidence.

Put them together and you have two sentences that can be adapted to answer any question on statistical testing because they show how all the concepts you need to learn about are linked.

——

A level of significance is the amount of chance that a researcher allows before they accept their hypothesis. If the level is too high, they might accept results that are just due to chance. If the level is too low, they will reject results that are valid because they are being too strict.

A Type I error is when the results of a study persuade you to support your research hypothesis, but there was actually something wrong with the sample or method and you should not have. In other words, it is when something random suggests your results are significant, but they are not. So, the likelihood of a Type I error depends on the amount of chance that you allow before you accept your results. Again, too high a level of significance and the likelihood of a Type I error increases. Of course, you can reduce the likelihood of a Type I error by choosing a low level of significance, but then you increase the likelihood that you will reject results that should be significant – this is called a Type II error.

So with the first sentence, you have managed to link together significance, Type I error and probability in one phrase. If a question asks you about any one of those concepts, use the killer phrase to imply that you understand how all the concepts are linked.

The final sentence is then used to link your answer to the specific question you have been asked as well as showing that you know the whole point of statistical testing is generalising from samples to populations.

——

All you need to do is adapt the phrases to the particular question.

As the convention is to use a 5% level of significance and you are likely to be explaining whether a particular set of results are significant, you will usually write:

• A 5% level of significance means that the probability of a Type I error is 5/100 or 1/20. A result that is significant at this level is unlikely to have happened by chance and can be generalised

You could also use this phrase to elaborate on any question that asks you to explain what a Type I error is or simply to explain how levels of significance are used in statistical testing.

If a question asks about a 10% level of significance, you might write:

• A 10% level of significance means that the probability of a Type I error is 10/100 or 1/10. A result that is significant at this level is too likely to have happened by chance and should not be generalised.

Finally, if a question shows that a result is too likely to have happened by chance, you might write:

• A 5% level of significance means that the probability of a Type I error is 5/100 or 1/20. A result that is not significant at this level is too likely to have happened by chance and should not be generalised.

Unfortunately, there is no simple way to use these phrases when talking about Type II errors, but in the unlikely event that you are asked about them, you could write something like this:

• A 5% level of significance means that the probability of a Type I error is 5/100 or 1/20. Any result that is not significant at this level implies that the Null Hypothesis should be accepted, but there is always the possibility that random effects have prevented a significant result so that a Type II error has occurred.

Outside of the exam, you could use the phrase to test your own knowledge.  Read it carefully, to check that you appreciate how every concept mentioned is linked to every other concept.

——

Now imagine you are an examiner reading one of those phrases. By the time they get to your paper, they may have read dozens of vague, unimpressive, confusing or simply incorrect explanations of how to use statistical tests, and then they get to your answer.

Obviously, when you manage to link “level of significance” to “Type I error” in the same phrase, they will start to get excited. When you add “5/100” to link them both to the concept of probability, their heart will start beating faster. As they begin to read the next sentence their fingers will be quivering, so that when you link significance, Type I errors the likelihood of results happening by chance together, they won’t know how to control themselves.

The final blow lands when you add “can be generalised with confidence” to show that you understand that the whole point of statistical tests is to decide when it is appropriate to generalise from a sample to a population. The examiner will explode in a volcano of red hot statistical lava, and the board will be forced to grant you a top grade rather than risk any more lives.

Dr George Smith is Head of Psychology at Millfield School in Somerset.  After his degree, he combined his love of technology and language by conducting research into computer-mediated communication. He moved into teaching when he discovered that he enjoyed the daily excitement of dealing with the challenging questions asked by A Level students more than sitting in a lab by himself writing software.