(Dis)trusting statistics: a one-page guide

Theory of Knowledge banner

A numbers expert declares he’ll sum up everything he knows about analyzing statistics on the back of a postcard. Could any TOK teacher NOT instantly spring to the alert? He’s inspired me to attempt my own lean summary: a single page mini-guide on (dis)trusting statistics, useful in our own educational context of Theory of Knowledge.

The postcard version

The numbers expert is Tim Harford of the BBC podcast “More or Less”, which regularly offers listeners commentary on statistics in the world around us. His 10-minute episode “Debunking guide – on a postcard” is worth playing in a TOK class. (It also gives a sample of a podcast to which you may well wish to subscribe!)  Harford gives the following five tips, illustrating each with examples:

  1. Observe your feelings.
  2. Understand the claim.
  3. Get the back story.
  4. Put things in perspective.
  5. Be curious.

If I were using this podcast episode in class, I’d write these points prominently on a board as they are brought up, and use the list at the end to review with students what he has said.

A one-page version for TOK

I’m prompted, though, to adapt this approach of a terse summary to mesh better with the broader critical thinking skills of Theory of Knowledge. The advantage of a lean summary is that it provides a framework for ideas, into which up-to-date stories and statistics of the day can readily be fitted for illustration.

If you follow this blog or use my OUP Theory of Knowledge book , you’ll already be familiar with my critical framework of The Three S’s: Source, Statement, Self. You can download it here:  SSS-GUIDE-TO-EVALUATING-KNOWLEDGE-CLAIMS. For further comments on internet evaluation to supplement it, see my post from March 27, 2017 TOK and ‘fake news’: 3 tips, 2 downloads, and 3 resources.”

So here goes for my own try at statistics – outlined below and available here for download as a single page class handout: DOMBROWSKI STATS MINI-GUIDE

And here is my challenge to you: Can you and your students come up with – and share! – a one-page framework that’s even better?

(Dis) trusting Statistics: A Mini-Guide

  1. What is the source of these statistics?

What person or organization is providing this information? What are its qualifications and area of expertise? What are likely to be its biases, if any? How readily can you check the source – and possibly its sources in turn – for their qualifications, reputation, and consistency with other statistics offered by respected organizations and journals?

2. What is actually being claimed?

Are words and terms clear, and used in a way consistent with definitions in the relevant field? Is the knowledge claim a factual one about the present or past, or is it a hypothetical prediction about the future? Does it report on a single study or a meta-study? If it reports a survey, how large and representative is the sample population? (Become familiar with the following: different kinds of averages; difference between correlation and causation; the terms “statistically significant”, “p-value”, “p-hacking”, “background noise”.)

3.How are the statistics framed in context?

Are the statistics used as supporting evidence for a knowledge claim or argument? (And how valid is that argument?) Are the numbers being used – instead or as well – to impress in a more emotional way? Do accompanying images or language clarify the significance of the statistics – and/or possibly heighten an emotional impact? Does it seem that other important statistical information has been omitted?

4. What is your own emotional response to the statistics?

Do you notice in your own reaction to the statistics any inclination to accept or reject the statistics even before you’ve examined them as above? Do you detect in yourself any signs of confirmation bias – the inclination to believe whatever harmonizes with what you already think, or what you wish were true, regardless of the quality of the information?



Eileen Dombrowski, “TOK and ‘fake news’: 3 tips, 2 downloads, and 3 resources,” Oxford Education Blog. March 27, 2017. https://educationblog.oup.com/theory-of-knowledge/tok-and-fake-news-3-tips-2-downloads-and-3-resources

Eileen Dombrowski, Lena Rotenberg, and Mimi Bick. Theory of Knowledge. Oxford University Press, 2013. https://global.oup.com/education/product/9780199129737/?region=international

Tim Harford, “Debunking guide – on a postcard”, More or Less, BBC World Service. Feb 18. 2018. https://www.bbc.co.uk/programmes/w3csvq3v

Cartoons copyright Theo Dombrowski, used here with permission, and permission granted to teachers using the materials above in their own classrooms.