Sherlock Holmes and the telltale clue of the Unesco graph

by John Watson

At the UNESCO Institute for Statistics, Sherlock Holmes examined the line chart showing education levels, region by region.(1)

“Aha!” Holmes suddenly exclaimed. “Watson, this isn’t a statistics institute. It’s a propaganda department!”

I’d heard similar grumblings from a friend, who objected to UNESCO’s taking money from the tech industry, then pushing for more screens in schools.(2) But global education wasn’t Holmes’s usual field. How had he reached the same conclusion, so quickly?

This chart appeared in a report from the UNESCO Institute for Statistics (UIS) which concluded that worldwide, education was in a perilous state, and thus UIS needed more funding to collect more data about it. Look at the chart showing how many children, in each of seven global regions, fail to achieve Minimum Proficiency Levels (MPLs) in reading. What telltale clue did Holmes see?

“Actually, I’m confused,” I admitted. “I’d say they’ve included too many variables. Some dots cover up others. I don’t know what’s going on.”

“You’ve indeed pointed out one problem,” Holmes agreed, and I felt a quick glow of pride. “An amateur did this. A bar graph would show more, but even that wouldn’t clarify much. And this shows only the number, not the percentage, who do not achieve the MPLs. Oceania ranks very low. Is that because it has a very small population… or does it have great schools? This tells us nothing. A chart should be used because it makes the data clearer, not just because the software readily spits it out.”

Holmes puffed on his pipe, in blatant disregard of the No Smoking sign posted in the UNESCO office. “But that’s the least of it. The line — that’s the big story. Something appears to be dropping. Precipitously! But that’s not what these numbers show. The line drops down because seven regions were arbitrarily arranged to make it go down. Unesco wanted to sound an alarm. There is good reason for alarm, but it’s unrelated to the falling line. The line could go in whatever direction you want.” He sketched out a few graphs, to make his point:

  • A graph from the the Unesco institute for statistics suggests that something is declining. But that just reflects the order in which they arranged 7 regions.
  • Same data, in a different sequence, and it looks like things are improving.
  • Same data, in a different sequence, gives a different impression.

“A line chart shows a relationship between one axis and the other. For example, it might show population growth over time, or how price influences demand. For this data, a line chart is entirely wrong. The line reveals nothing about the data. In fact, it suggests a relationship — between reading levels and region — that simply does not exist. What it does reveal is that whoever created and published this graph has no basic grasp of statistics.

“We all make mistakes, and this error would be understandable from a high school or college student. That’s why they’re students. When the UNESCO Institute for Statistics publishes this, it’s like a baseball player running around the bases in the wrong direction. This is not who you want as your pinch hitter.” I smiled my appreciation. Holmes knows I’m trying to develop a wider readership, and he tosses me an occasional American analogy.

“In this report, the UNESCO Institute for Statistics wanted to portray a dire situation. The falling line looks dire. But there are 5040 different ways to arrange these seven regions. They did not, just by coincidence, arrange themselves to show a steadily falling line.”

I followed his logic, but I was incredulous. “Are you suggesting that someone at UNESCO created this chart with the deliberate intent of misleading global policy-makers?”

“We can’t rule that out,” Holmes replied, “but it seems unlikely. This is clearly the work of a propaganda department, not a statistics office. There’s more evidence of that, for example, its fondness for publishing statistics that show 3- and 4-digit accuracy when even a single digit is overstating the case. I was writing a monograph about that, until the Crown Jewels affair interrupted me.

“But is it a savvy propaganda department? No, it is bumblers. Propagandists are rarely the cream of the intellectual crop, and savvy propagandists wouldn’t leave such damning evidence that they are clueless about statistics. Recall what I’ve often told you: ‘Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth.’ Here’s what I deduce:

“Those who understand statistics, and are dismayed to find themselves in such a place, have either left UNESCO in disgust, or accepted that silence is the price of keeping a comfortable, well-paying job. Whoever made this graph understood that their job was to portray a crisis, so they arranged the regions in the necessary sequence, and then added a line which unintentionally announced that they had no idea what they were doing. Somebody higher up, but equally clueless about statistics, saw a chart that conveyed the message they wanted, and approved it.

“Can you come up with any other plausible explanation for how the Unesco Institute for Statistics could have published this graph, unless it serves primarily as a propaganda department?”

Data from the Unesco Institute for Statistics shapes United Nations policy about global health and education. It’s scary to think of it as a propaganda department. But I had confess that I had no answer to his question.

Sources:

The chart appears in More Than One-Half of Children and Adolescents Are Not Learning Worldwide, UIS Fact Sheet No. 46, Unesco and Unesco Institute for Statistics, September 2017. (UIS/FS/2017/ED/46) The above link leads to a Unesco news release explaining that the “not learning” figure is intended as a “Global Composite Indicator for Education.” The report itself can be downloaded directly from: More than half of children not learning, UNESCO-UIS 46

UNESCO’s links to the tech industry run deep. In 2014 the agency announced: “UNESCO study shows effectiveness of mobile phones in promoting reading and literacy in developing countries.” The study was funded by Nokia, the telecom giant, and it did not in fact show what UNESCO claimed. Sherlock has just about wrapped up the Crown Jewels affair, and we will write more about this soon. [Editor’s note: The story is now written, and linked below.]

Related stories

Leave a Reply