jump to navigation

universities boycott, “macleans” gets defensive August 29, 2006

Posted by Brad Richert in academia, media.

August 14, 2006

Dear Mr. Keller:
We regret to advise you that our universities will not be participating in the 2006 Maclean’s questionnaire.
We share Maclean’s goal of providing good information for students and their families who are researching post-secondary education. We also compliment you on your editorial coverage of the post-secondary sector. Many of the articles in Maclean’s have contributed to the national discussion about post-secondary education, and have helped to frame students’ choices. Our concern relates specifically to Maclean’s attempts to generate a global ranking of Canadian universities.

In various ways and for some years, many institutional spokespersons have expressed considerable reservations about the methodology used in the Maclean’s university survey and the validity of some of the measures used. Thus far, these serious concerns have gone largely unaddressed, and there is still no evidence that Maclean’s intends to respond to them.

We welcome public assessment of our work, and all our institutions devote significant resources to that end. We already publish a great deal of data about our own institutions on-line and intend to publish more in future, ideally in the form of standardized datasets that will facilitate valid temporal and institutional comparisons. However, it is truly hard for us to justify the investment of public funds required to generate customized data for your survey when those data are compiled in ways that we regard as over-simplified and arbitrary.

Our concerns about Maclean’s misuse of data in its rankings issue can be briefly recapitulated here.
To begin with, the Maclean’s rankings aggregate data from a range of variables related to the student body, class sizes, faculty, finances, library and reputation. It is inappropriate to aggregate information across a range of programs at a large and multi-dimensional research university into a single ranking number. Consider how such an approach might pervert one’s understanding of a general hospital that is ranked #1 in obstetrics and #10 in cancer care. Averaging these rankings would result in this hospital being ranked “#5 overall”. For the patient seeking care in one of these areas, such a measure would be useless at best and misleading at worst. This is, effectively, the method that Maclean’s applies to Canadian universities by its calculation of “league tables” based on the arbitrary assignment of weights to variables which, by themselves, are of questionable validity. The variables selected by Maclean’s also fail to capture the breadth of experiences students say are important in their university education such as, for example, extra-curricular activities or the opportunity for rich and diverse interactions with peers and faculty outside the classroom.

We are also concerned by Maclean’s recent attempt to draw comparisons of student experience across incomparable surveys of student engagement, and Maclean’s reliance on survey data with low response rates and all the associated response biases that arise from skewed profiles of respondents. The responsible compilation and comparison of data is a core tenet of academic research. Several universities already show student survey data, in context, on their own web sites and question Maclean’s decision to pull different kinds of data out of context and compare “apples and oranges”. Maclean’s treatment of these survey data, in our view, fails to give appropriate notice to these methodological limitations.

It is not just the Maclean’s student survey that has suffered from low response rates. Equally troubling is the fact that a clear majority of individuals who receive the Maclean’s reputational survey do not respond. This is a particular concern as the results of the reputational survey not only affect rankings in a significant way, but are given prominence separately by your magazine.

This is only a partial accounting of the methodological flaws in the Maclean’s rankings. In short, the ranking methodology used by Maclean’s is oversimplified and arbitrary. We do find it ironic that universities are being asked to subsidize and legitimize this flawed methodology, when many faculty, staff and students at our institutions are dedicated in their research to ensuring that data are collected rigorously and analyzed meticulously.
We remain open to the possibility of collaborating with Maclean’s at some future date, particularly if we can agree on means to ensure that the data will be valid and the analyses truly informative. Meanwhile, we will continue to publish data on our websites to facilitate informed student and family choice.
Yours truly,
Tom Traves, Dalhousie University
Peter George, McMaster University
Michael Stevenson, Simon Fraser University
Indira Samarasekera, University of Alberta
Stephen Toope, University of British Columbia
Harvey Weingarten, University of Calgary
William Cade, University of Lethbridge
Emőke Szathmáry, University of Manitoba
Luc Vinet, Université de Montréal
Gilles Patry, University of Ottawa
David Naylor, University of Toronto

Macleans Magazine, Volume 119 Number 35, p.2

Nobody likes being graded, particularly those used to giving tests, not sitting them. Thist may explain why 11 Canadian universities sent Maclean’s a letter last week, saying that they would “not be participating in” an informational questionnaire that we send each year to the universities, as we gather data for our annual University Ranking issue.

Sour grapes? (University of Alberta)

Absolutely not. The University of Alberta has consistently been ranked among the top universities in our medical/doctoral category. Our aim is solely to make the information about post-secondary education more useful by eliminating the distortions of ranking. It should be noted that the University of Toronto has ranked first in our category since the rankings began in 1990, and they joined our group because they also want changes in the way the data are compiled and presented. What we are hoping to achieve is to have the magazine adjust its presentation of information to reflect provincial and other differences that affect such measures as entering grades, retention and graduation rates. We believe the rankings, as they now stand, do not provide students, parents and counsellors with accurate information which they can refer to when making important post-secondary decisions.

Sources and more info:



1. beepbeepitame - August 29, 2006

Welcome to the blogroll 🙂

2. hiutopor - September 16, 2007

Hi all!

Very interesting information! Thanks!


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: