Sherlock Holmes and the telltale clue of the UNESCO graph

by John Watson

At the UNESCO Institute for Statistics, Sherlock Holmes examined the line chart showing education levels, region by region.(1)

“Aha!” Holmes suddenly exclaimed. “Watson, this isn’t a statistics institute. It’s a propaganda department!”

I’d heard similar grumblings from a friend, who objected to UNESCO’s taking money from the tech industry, then pushing for more screens in schools.(2) But global education wasn’t Holmes’s usual field. How had he reached the same conclusion, so quickly?

This chart appeared in a report from the UNESCO Institute for Statistics (UIS) which concluded that worldwide, education was in a perilous state, and thus UIS needed more funding to collect more data about it. Look at the chart showing how many children, in each of seven global regions, fail to achieve Minimum Proficiency Levels (MPLs) in reading. What telltale clue did Holmes see?

“Actually, I’m confused,” I admitted. “I’d say they’ve included too many variables. Some dots cover up others. I don’t know what’s going on.”

“You’ve indeed pointed out one problem,” Holmes agreed, and I felt a quick glow of pride. “An amateur did this. A bar graph would show more, but even that wouldn’t clarify much. And this shows only the number, not the percentage, who do not achieve the MPLs. Oceania ranks very low. Is that because it has a very small population… or does it have great schools? This tells us nothing of value. A chart should be used because it makes the data clearer, not just because the software readily spits it out.”

Holmes puffed on his pipe, in blatant disregard of the No Smoking sign posted in the UNESCO office. “But that’s the least of it. The line — that’s the big story. Something appears to be dropping. Precipitously! But that’s not what these numbers show. The line drops down because seven regions were arbitrarily arranged to make it go down. UNESCO wanted to sound an alarm. There is good reason for alarm, but it’s unrelated to the falling line. The line could go in whatever direction you want.” He sketched out a few graphs, to make his point:

  • A graph from the the Unesco institute for statistics suggests that something is declining. But that just reflects the order in which they arranged 7 regions.
    Click the arrow on the right, or dots below, to see an alternative way to present the same data.
  • Same data, in a different sequence, and it looks like things are improving.
    Arrange the regions in the opposite order, and it looks like something’s going up.
  • Same data, in a different sequence, gives a different impression.
    Or arrange the regions this way, to suggest that disaster was narrowly averted. You’re lucky we’re here!!!

“A line chart shows a relationship between one axis and the other. For example, it might show population growth over time, or how price influences demand. For this UNESCO data, a line chart is entirely wrong. The only reason to draw the line is if you’re designing a skateboard ramp.”

I grimaced. My publisher usually edits out these anachronisms, let’s see if this one gets through. Holmes continued: “The line reveals nothing about the data. In fact, it suggests a relationship — and a trend — that simply does not exist. What it does reveal is that whoever created and published this graph has no basic grasp of statistics.

“We all make mistakes, and this error would be understandable from a high school student. That’s why they’re students. When the UNESCO Institute for Statistics publishes this, it’s like a baseball player running around the bases in the wrong direction. This is not who you want as your pinch hitter.” I smiled my appreciation. Holmes knows I’m trying to develop a wider readership, and he tosses me an occasional American analogy.

“In this report, the UNESCO Institute for Statistics wanted to portray a dire situation. The falling line looks dire. But there are 5040 different ways to arrange these seven regions. They did not, just by coincidence, arrange themselves to show a steadily falling line.”

I followed his logic, but I was incredulous. “Are you suggesting that someone at UNESCO created this chart with the deliberate intent of misleading global policy-makers?”

“We can’t rule that out,” Holmes replied, “but it seems unlikely. This is clearly the work of a propaganda department, not a statistics office. There’s more evidence of that, for example, its fondness for publishing statistics that show 3- and 4-digit accuracy when even a single digit is overstating the case. I was writing a monograph about that, until the Crown Jewels affair interrupted me.

“But is it a savvy propaganda department? No, it is bumblers. Propagandists are rarely the cream of the intellectual crop, and savvy propagandists wouldn’t leave such damning evidence that they are clueless about statistics. Recall what I’ve often told you: ‘Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth.’ Here’s what I deduce:

“Those who understand statistics, and are dismayed to find themselves in such a place, have either left UNESCO in disgust, or accepted that silence is the price of keeping a comfortable, well-paying job. Whoever made this graph understood that their job was to portray a crisis, so they arranged the regions in the necessary sequence, and then added a line which unintentionally announced that they had no idea what they were doing. Somebody higher up, but equally clueless about statistics, saw a chart that conveyed the message they wanted, and approved it.

“Can you come up with any other plausible explanation for how the UNESCO Institute for Statistics could have published this graph, unless it serves primarily as a propaganda department?”

Data from the UNESCO Institute for Statistics shapes United Nations policy about global health and education. It’s scary to think of it as a propaganda department. But I had confess that I had no answer to his question.

Sources:

The chart appears in More Than One-Half of Children and Adolescents Are Not Learning Worldwide, UIS Fact Sheet No. 46, UNESCO and UNESCO Institute for Statistics, September 2017. (UIS/FS/2017/ED/46) The above link leads to a UNESCO news release explaining that the “not learning” figure is intended as a “Global Composite Indicator for Education.” The report itself can be downloaded directly from: More than half of children not learning, UNESCO-UIS 46

UNESCO’s links to the tech industry run deep. In one case, the agency announced: “UNESCO study shows effectiveness of mobile phones in promoting reading and literacy in developing countries.” The study was funded by Nokia, the telecom giant, and it did not in fact show what UNESCO claimed. Sherlock finally wrapped up the Crown Jewels affair, and a report on the Nokia affair is linked below.

Related stories

Left: Cellphones and literacy. UNESCO took money from big tech to publish a deceitful report that benefited the company that gave it the money. If an African president had done that, what would we call it?
Right: Francophonie. By forcing children in its former African colonies to study in French, France thought it was spreading the glory of the French language. But students ended up learning neither French, nor much of anything else.
Left: Schools in the global South are getting worse. The aid industry doesn’t ask why, and for good reason. It created much of the problem.
Right: Cooking the numbers. 59% of 10-year-olds in Senegal can’t read, says a UN-World Bank report. Actually, the number is only about 7%. After denying the problem, now the global agencies exaggerate it. There’s a reason.
Left: Experts vs. Chimps: Do “development experts” have any more wisdom than dart-throwing chimpanzees? Well, it’s a tight contest. It turns out, the experts who get in the news have a sorry track record, though they try to hide it.
Right: Garbage in… UNESCO takes garbage data, runs it through a fancy formula, and claims to show a picture of education around the globe. This merely disguises the fact that UNESCO really has no idea where things stand.

Other stories of interest

Left: Chocolate hands. You can buy chocolate hands at shops in Antwerp. So what? Well, one of the colonial era’s great atrocities involves Belgians chopping off the hands of Africans.
Right: Boys thrown under the schoolbus. U.N. agencies and NGOs focus almost exclusively on girls’ education. But girls are faring better than boys, whose needs are ignored and who are falling behind.
Left: Charity is not development. The aid industry promises development but does charity. They are opposites. That’s why aid projects so often achieve the opposite of what they promise.
Right: Aid for the richest and whitest: UK Aid has funneled hundreds of millions of dollars through AgDevCo, a UK “impact investor,” to help white European investors run agribusinesses in Africa.
Left: What would make a better future? There are ways that wealthier countries can genuinely help others, if they want to. Give the aid money directly to the poor, for example. Here are ideas.
Right: The cholera epidemic: The epidemic that killed 10,000 Haitians revealed more than just sloppy sanitation practices by the U.N. It revealed institutional rot and an imperial attitude of impunity.