Measuring what matters: A convening

by | Sunday, May 17, 2020

All of us involved in social design (and I include education in among those as well) ask ourselves, or are asked this question:

How do we measure the impact of the work we do?

This begs the question, why measure in the first place? Lord Kelvin, one of the greatest physicists of the 19th century, provided the canonical answer to this question back in 1883, saying:

…When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind.— William Thompson Kelvin (Popular Lectures, Vol. I, “Electrical Units of Measurement,” p. 73, 1883).

In other words, according to Lord Kelvin “to measure is to know” (which as it happens is another of his oft-quoted remarks).

This emphasis on measurement can be seen in all aspects of our lives. We measure our children’s heights, and our GDP; we measure learning gains in students, and recidivism rates of inmates released from prison; we measure the number of steps we walk and the calories we eat.

That said, we also know that there are things, important things, that we have not yet managed to measure, or do not know how to. These are often intangible, ineffable—and yet, important.

How do we measure if students are seeing beauty in mathematics? Or the economic value of public art? Or the capacity of music to move us?

These issues are of concern for all of us engaged in social design (i.e. the application of design methods to tackle complex human problems). In a recent blog post, prompted by the COVID19 crisis and global school closures, I asked the question about the value of school (in two parts, One & Two)– and argued that measures that focus on one dimension (that of learning, via standardized tests) miss all the other values that schools bring to our lives.

The issue, of course, is that if we can’t measure the impact of our work how can we know if what we are doing is working or not? The issue is not whether we should measure but rather if we are measuring what we ought to be measuring. Often, we are limited to measuring what we can easily measure rather than what we ought to measure. Alternatively:

Do we measure what we value or do we merely value what we can measure?

These are not new questions, but they are important ones, and worth revisiting.

Recently we organized a virtual convening of scholars and leaders from across ASU on this very topic. Titled: The Importance of the Ineffable: Measuring What Matters, the convening was organized by the Office of Scholarship and Innovation, the Principled Innovation Team and InnovationSpace. Participants came from a range of units within and outside of ASU, bringing their disciplinary expertise as well as their deep expertise into the conversation. (Complete list of colleges and units that participated given at the end.)

This event had initially been planned as being a face-to-face convening, till COVID19 got in the way. What this shift to an online convening meant that we had to get creative to ensure that the different approaches and perspectives could get properly and thoughtfully addressed and some synthesis reached.

To us that meant that we could not (and should not) just drop people into a zoom room and hope for good things would happen.

Jennifer Stein and Henry Borges put in some significant effort to design a genuinely powerful experience – with additional support from Clarin Collins, Ben Scragg, and Lok-Sze Wong. We moved seamlessly from whole group to smaller teams and back to whole group, toggling between Zoom and Google Slides and Google Draw, sharing ideas, collapsing them in to broad categories of questions and then digging deeper into them.

Screenshot from virtual card-sorting exercise. Names have been deleted.

A range of topics were generated and were virtually categorized and then people self-selected to discuss each of these overarching topics further.

Overall it was an exciting and intellectually challenging event, providing us with lots to chew on and we continue to explore these ideas in actual projects in the future.

Note: Participants included people from the following units or organizations: The School for the Future of Innovation in Society, School of Earth and Space Exploration, College of Public Service and Community Solutions, School of Computing, Informatics, and Decision Systems Engineering, Herberger Institute for Design and the Arts, School of Mathematical and Statistical Sciences, Mary Lou Fulton Teachers College, and the Kern Family Foundation.

End note:

Finally, for two slightly different, and idiosyncratic yet relevant, takes on the whole issue of measurement see these two blog posts from a few years ago:  

A few randomly selected blog posts…

Peer review in the science classroom

Peer review in the science classroom

Fig. 1: Header image. Credits: Illustration by Punya Mishra. License CC-BY-NC. The scientific method is a myth. In more ways than one. Typically in school you are taught that the scientific method consists of making observations, developing hypotheses, testing them by...

How to complete a half-marathon

I completed the Capital City River Run Half Marathon today. This race has become an annual event for me, this being my 7th outing so far (the first three years being a 10 mile run before it shifted into a half-maraton). [See links here, here and here]. I completed the...

New ambigram: Nihal

My friend, Hartosh (I had written previously about his mathematical novel here) and his wife Pam, recently had a baby boy. This ambigram is of his name: Nihal Enjoy.

A Silver Lining side conversation with S. Giridhar:

A Silver Lining side conversation with S. Giridhar:

S. Giridhar (Giri), Chief Operating Officer of Azim Premji University (APU) and I had a chance to chat for a Silver Lining for Learning side conversation. Giri is a good friend and we connect at multiple levels. We both went to the same undergraduate institution (BITS...

Tactical creativity in sports

Tactical creativity in sports

Daniel Memmert is Professor and Executive Head of the Institute of Exercise Training and Sport Informatics at the German Sport University Cologne. A lifelong sports player and enthusiast, Memmert’s research is at the intersection of human movement science, sport...

Research to practice : 3 articles

Matt Koehler and I are co-editors for an ongoing series of articles "From Research to Practice" for Education Matters, an educational magazine published by Educational Technology & Management Academy (ETMA). ETMA is non-profit organization based in New Delhi...

Engineering education, past & future

Neeraj Buch and I were invited to talk to a group of engineering educators from India. This was a meeting organized by the College of Engineering and the Indo-US Collaboration for Engineering Education. The topic I spoke about was was Improving Engineering Education:...

New Orleans, here we come…

Matt and I will be at New Orleans next week for the AACTE Annual Conference. The last time I went to New Orleans must have been in 2000 or 2001... so I am looking forward to going there. There are three specific things we will be involved with. Meeting of the AACTE's...

@ Purdue

Sending this note from Purdue University where I am visiting the School of Engineering Education. Had a great dinner last night with Karl Smith (whom I am catching up with after a couple of years) and Johannes Strobel. Karl picked me up this morning and drove me over...


Submit a Comment

Your email address will not be published. Required fields are marked *