“Remarkably—as has been shown—if you can say something straightforward in a complicated manner with complex theorems, even if there is no large gain in rigor from these complicated equations, people take the idea very seriously.”
—Nassim Nicholas Taleb, in Antifragile

I read Taleb’s observation just a day after wincing at this equation from the UNESCO Institute for Statistics:(1)

I was lost after “AgI-1.” How about you?

In the past several years, UNESCO and UNICEF have sounded the alarm about an “education crisis” in developing countries. Worldwide, says UNESCO, 56% of primary-school-age children are not achieving minimum proficiency levels (MPLs) in reading. In sub-Saharan Africa, the figure is 85%.(2)

I got curious about how the UNESCO Institute for Statistics comes up with such precise numbers, and that eventually led me to the formula above.

On a closer look, the formula is a high-falutin’ way of stating what would be obvious to anyone who knows a little math. Let’s say you know three things:

• 1000 children are in grade 4.
• In the past, 90% of 4th-graders finished primary school (grade 5) on schedule.
• In the past, 80% of 5th-grade graduates met the MPL.

Then you assume that past performance will remain unchanged.

Now you can easily calculate how many of these children will achieve “minimum proficiency.” It’s this:

A x B x C

That is, 1000 x 90% x 80%, or 720 children. Do this for each grade, and add them up. UNESCO/UIS refer to their “new methodology,” and their gussied-up formula – using an 11-character variable instead of just writing “A” — makes this look like a job for a PhD because… well, Mr. Taleb, above, already explained why.

For its conclusion that 56% of primary-school-age children are not achieving MPLs in reading, UNESCO needs a couple more variables: The percentage of children who are not in school now, and how many of them are expected to enroll and reach the MPLs.(2)

There’s no new insight here, and the result will be accurate only to the extent that the underlying data and assumptions are correct. There’s a lot of those:

• All the data, including out-of-school rates, must be reasonably accurate.
• The tests that measure MPLs must be designed and administered well.
• Completion rates and MPL achievement rates must both be recorded accurately.
• Previous achievement rates must remain fairly steady year-to-year.
• The formula assumes that all out-of-school children can be classified as “not learning.”
• It assumes that the items tested – usually reading and math – represent “learning.” If you achieve MPL on those subjects, you have learned. Otherwise, you did not.
• It assumes that teachers or administrators don’t change the answer sheets to make their own performance look better.

These assumptions range from plausible, to dubious, to laughable, to flat-out wrong. For example, if you can’t read but can fix a motorbike, your job prospects in many countries will be far better than if you can pass a basic reading test but have no common sense. As for out-of-school rates: Among many other problems, UNESCO data routinely considers children in private or religious schools, such as madrasas, to be “out of school” — and it classifies them as therefore “not learning.”(3)

Yet from this, the UNESCO Institute for Statistics produces a string of tidy numbers that fill up page after page, report after report. For example:(4)

Not only are these numbers based on the flimsiest of foundations, but we don’t even know what they represent. UNESCO itself notes that “there is currently no global consensus on how to define minimum proficiency levels in reading and mathematics.” A child who achieves “proficiency” in one country might, in another, be judged to fall short.

Why does it matter?

These numbers, like so many other U.N. statistics, are too weak to be of any actual use for planning and evaluation. Rather, they are used to claim success, or to announce a crisis – whichever will justify a call for more funding. In 2015 the U.N. used bad numbers to claim great success for its development goals, which included education.(5) (At the time, it was seeking approval to set development goals for another 15 years.) Now there is a “crisis in education” — which also calls for more funding.

By publishing countless reports filled with bad data, the U.N. also subtly channels attention. Now we’re thinking about how to improve in-school learning. But really, we need to question whether school is even the right place for all — or most — or even any — children.

Furthermore, it diverts from the issue of who should decide how a society educates its children. UNICEF has stated that “Education needs and priorities differ hugely across different countries and regions.”(6) Why not encourage countries to try various approaches? We’d learn a lot from that. But the U.N.’s one-plan-fits-all mentality leaves no room for original thinking.

Finally, UNESCO shows considerable bad faith when it publishes so much data while hiding the poor quality of it. These reports suggest that the U.N. and UNESCO have a solid understanding of the situation, and thus will know what to do. The reality is just the opposite: Their disastrously bad policy — pushing all children into school, regardless of whether they learn anything — got us into this. And they have no idea what to do next. But that won’t stop them from doing what they do best: appealing for more money.

Notes and Sources

1. Counting the Number of Children Not Learning: Methodology for a Global Composite Indicator for Education, UNESCO Institute for Statistics, 2017, page 10. There are many reasons to question whether this institute should be in charge of U.N. statistics. The cover states that this is “Information Paper No. 47” and some page headers say the same; then they switch to saying “Information Paper No. 32.” One note states that “This assumption does not imply that those who dropped out of primary school did not learn anything.” UNESCO-UIS then proceeds to count these children in the group that “did not learn.” After calculating that 85% of children in sub-Saharan Africa do not achieve minimum proficiency levels, UNESCO-UIS insists as a matter of extreme urgency that all children not in these failing schools should be made to attend.

2. Here, from the same document, is how UNESCO calculates how many children are not in school but are expected to enroll and reach the MPL:

(Yes, the explanations should be labeled a-b-c-d, not a-b-c-a. As I’ve noted before, sloppiness is rampant at the UNESCO Institute for Statistics.)

3. These out-of-school definitions are often quite fuzzy, or aren’t stated at all, but it’s clear that many children who do attend a private or religious school — which may be better than the public school — are counted as “out of school.” In Africa’s largest country, for example, the Country Level Evaluation: Nigeria, 2020 (Global Partnership for Education) states: “Among those counted as being out of school, a significant proportion attend un-registered Islamiyya and Quranic Schools, which in some cases outnumber registered schools.” Also see Estimation of the numbers and rates of out-of-school children and adolescents using administrative and household survey data, by Kristi Fair, 2016. (UNESCO Institute for Statistics, Information Paper No. 35, October 2016.) This report looks at the two common sources of enrollment numbers, and the potential pitfalls of each. The bottom line is: Children in private and religious schools are often counted as “out of school,” but we don’t know if that happens 40% or 80% of the time, and we don’t even know how many children attend such schools. Yet UNESCO plugs this number, along with other equally dubious values, into its formula to produce sweeping statements that, for example, 85% of children in sub-Saharan Africa are not learning.

Furthermore, in 2021-2022 UNESCO itself reported that 3 in 10 of all young adults who did not attend school, can read. In short, the fancy equation is manipulating garbage data.

4. The “not achieving MPL” statistics and the big data chart are from More Than One-Half of Children and Adolescents Are Not Learning Worldwide, UNESCO, and UNESCO Institute for Statistics, 2017.

5. U.N. extreme-hunger data changed dramatically, just in time for its 2015 report; see Are there more hungry people than before… or fewer?

6. Every Child Learns: UNICEF Strategy 2019-2030, UNICEF, 2019, page 36.

Related stories

Other stories of interest