This post refers to an article by Canadian statistician Pierre-Jérôme Bergeron in McGill Journal of Education / Revue des sciences de l’éducation de McGill

John Hattie’s work, and particularly his seminal book Visible Learning, ranks among the best cited publications in education research. To many, the results of his many meta-analyses of more than 50,000 studies are a trusted guide in their teaching. It is the basis of many education reforms at the school or higher levels. To non-statisticians the work is quite impressive and very convincing, but recently is has come under fire from mathematical and statistical experts.

Some of the attacks have been quite personal and unfair, which does not help most of us who are non-experts to determine where his work can be trusted or not. A well-written paper by Shaun Killian (Australian Society for Evidence-based Teaching, 2015) may help us to judge for ourselves what will be useful in the daily classroom practice. Killian shares six concerns of Visible Learning:

Visible Learning Focuses On Academic Results.

Hattie Relies On Meta-Analyses.

He Says SES [social economic status] & Class Size Don’t Matter.

Effect Size Is Not A Valid Statistical Measure.

Half Hattie’s Statistics Are Wrong.

Hattie Based His Findings On Shonky Research.

It is important to note that this paper has been written with a focus on evidence-based education, which our blog collective OnderzoekOnderwijs.net is not hugely enthousiastic about – we favour evidence-informed education. My personal critique of Hattie’s work relates to Killian’s points 1 and 3.  My main concern, as with many other studies of effective education, is that the question “Effective for what? For what goal?” is hardly ever asked. And even before that question comes the question “What is good education?” These go beyond questions about the effect size of certain interventions.

While Killian’s paper provides plenty of food for thought, even if Killian himself concludes that Hattie’s work should not be dismissed. More serious critics of his work include Dylan Wiliam, cited in Old Hat(tie)? Some things you ought to know about effect sizes.

A recent paper by Canadian statistician Pierre-Jérôme Bergeron raises more serious doubt about the fundamentals of the metastudies underlying Visible Learning.

pseudoscience

Reading Bergeron’s paper I wondered if he has some personal axe to grind. He is definitely not being very nice, identifying major errors and calling Visible Learning ‘fiction’ and ‘pseudoscience’. Yet he makes some points worth considering:

Hattie talks about success in learning, but within his meta-analyses, how do we measure success? An effect on grades is not the same as an effect on graduation rates. An effect on the perception of learning or on self-esteem is not necessarily linked to “academic success” and so on. A study with a short timespan will not measure the same thing as a study spanning a year or longer. And, of course, we cannot automatically extend observations based on elementary school students to secondary school or university students. The same applies to the way we group different factors under a category without defining inclusion and exclusion criteria. For example, the gender effect reported by Hattie is, in fact, a mean of differences between boys and girls in the set of studies selected, regardless of the duration, the level, or the populations studied.

He continues:

Basically, Hattie computes averages that do not make any sense. A classic example of this type of average is: if my head is in the oven and my feet are in the freezer, on average, I’m comfortably warm. Another humoristic example is: the average person has one testicle and one ovary and thus is a hermaphrodite. We wouldn’t say that the person making this kind of statement holds the Holy Grail of biology research, yet this is exactly what Hattie does when he aggregates every gender difference under the same effect. This is also true for his other aggregations, whether they be “major” contribution sources (the student, the home, the school, the teacher, the programme, or the teaching method) or “individual” influences, such as the “disease” effect which combines together disparate health problems, including cancer, diabetes, sickle-cell anemia, and digestive problems. It goes without saying that certain of these individual influences are much less frequent than others.

Most of the problems Bergeron identifies are rather technical and will probably not be appreciated by most teachers, including myself. Yet we can consult at any time those among our colleagues who are skilled statisticians to help us decide whether and how to use Visible Learning to develop new teaching strategies.

sources

Pierre-Jérôme Bergeron (2017). How to engage in pseudoscience with real data: a criticism of John Hattie’s arguments in Visible Learning from the perspective of a statistician. McGill Journal of Education / Revue des sciences de l’éducation de McGill, vol. 52, no. 1.

John Hattie (2013). Visible Learning: A Synthesis of 800 Meta-Analyses Relating to Achievement. Routledge.

Shaun Killian (2015). An Objective Critique of Hatties Visible Learning Research. Australian Society for Evidence Based Teaching.