Measuring what matters: A convening

by | Sunday, May 17, 2020

All of us involved in social design (and I include education in among those as well) ask ourselves, or are asked this question:

How do we measure the impact of the work we do?

This begs the question, why measure in the first place? Lord Kelvin, one of the greatest physicists of the 19th century, provided the canonical answer to this question back in 1883, saying:

…When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind.— William Thompson Kelvin (Popular Lectures, Vol. I, “Electrical Units of Measurement,” p. 73, 1883).

In other words, according to Lord Kelvin “to measure is to know” (which as it happens is another of his oft-quoted remarks).

This emphasis on measurement can be seen in all aspects of our lives. We measure our children’s heights, and our GDP; we measure learning gains in students, and recidivism rates of inmates released from prison; we measure the number of steps we walk and the calories we eat.

That said, we also know that there are things, important things, that we have not yet managed to measure, or do not know how to. These are often intangible, ineffable—and yet, important.

How do we measure if students are seeing beauty in mathematics? Or the economic value of public art? Or the capacity of music to move us?

These issues are of concern for all of us engaged in social design (i.e. the application of design methods to tackle complex human problems). In a recent blog post, prompted by the COVID19 crisis and global school closures, I asked the question about the value of school (in two parts, One & Two)– and argued that measures that focus on one dimension (that of learning, via standardized tests) miss all the other values that schools bring to our lives.

The issue, of course, is that if we can’t measure the impact of our work how can we know if what we are doing is working or not? The issue is not whether we should measure but rather if we are measuring what we ought to be measuring. Often, we are limited to measuring what we can easily measure rather than what we ought to measure. Alternatively:

Do we measure what we value or do we merely value what we can measure?

These are not new questions, but they are important ones, and worth revisiting.

Recently we organized a virtual convening of scholars and leaders from across ASU on this very topic. Titled: The Importance of the Ineffable: Measuring What Matters, the convening was organized by the Office of Scholarship and Innovation, the Principled Innovation Team and InnovationSpace. Participants came from a range of units within and outside of ASU, bringing their disciplinary expertise as well as their deep expertise into the conversation. (Complete list of colleges and units that participated given at the end.)

This event had initially been planned as being a face-to-face convening, till COVID19 got in the way. What this shift to an online convening meant that we had to get creative to ensure that the different approaches and perspectives could get properly and thoughtfully addressed and some synthesis reached.

To us that meant that we could not (and should not) just drop people into a zoom room and hope for good things would happen.

Jennifer Stein and Henry Borges put in some significant effort to design a genuinely powerful experience – with additional support from Clarin Collins, Ben Scragg, and Lok-Sze Wong. We moved seamlessly from whole group to smaller teams and back to whole group, toggling between Zoom and Google Slides and Google Draw, sharing ideas, collapsing them in to broad categories of questions and then digging deeper into them.

Screenshot from virtual card-sorting exercise. Names have been deleted.

A range of topics were generated and were virtually categorized and then people self-selected to discuss each of these overarching topics further.

Overall it was an exciting and intellectually challenging event, providing us with lots to chew on and we continue to explore these ideas in actual projects in the future.

Note: Participants included people from the following units or organizations: The School for the Future of Innovation in Society, School of Earth and Space Exploration, College of Public Service and Community Solutions, School of Computing, Informatics, and Decision Systems Engineering, Herberger Institute for Design and the Arts, School of Mathematical and Statistical Sciences, Mary Lou Fulton Teachers College, and the Kern Family Foundation.

End note:

Finally, for two slightly different, and idiosyncratic yet relevant, takes on the whole issue of measurement see these two blog posts from a few years ago:  

A few randomly selected blog posts…

TPACK newsletter #33, June 2017

TPACK newsletter #33, June 2017

TPACK triplet design by Punya Mishra The latest version of the TPACK newsletter (#33) can be found here (pdf). All previous issues are archived here. A shout-out to Judi Harris for all the work that goes into this.

Best of SkyMall

I love browsing through the SkyMall catalog when I am flying. I never cease to be amazed by human ingenuity - the range of things we have built, irrespective of how useful (or useless they may be). Anyway, someone has now listed the 10 best (or worst, depending on...

Speaking of leadership

Matt and I were invited to Sydney, Australia a year ago as a part of the Teaching Teachers for the Future (TTF) project. You can see a report in the New Educator: TPACK takes hold in Australia. As a part of this visit we were interviewed to speak a bit about...

Organizational & Team Creativity

Organizational & Team Creativity

Illustration for Group-Creativity by Punya Mishra The next article in our series Rethinking Technology & Creativity in the 21st Century just got published by the journal TechTrends. This article features an interview with Dr. Roni...

Links of interest

During Dr. Jalaluddin's keynote I took some time to search online for some reports, prompted by what he had been saying. (Yes I was listening not just browsing). The first is an European study: ICT in Schools: Trends, Innovations and Issues in 2006-2007. You can...

Creativity & Courage

Creativity & Courage

Here is the next article in our series Rethinking Technology & Creativity in the 21st Century for the journal TechTrends. This article features an interview with Dr. Yong Zhao, Foundation Distinguished Professor in the School of Education at...

Creativity in Las Vegas

I was recently invited to present a keynote address at the 21st Century Instructional Technology Conference (titled Elements of Technology) at the Clark County School District in Las Vegas, Nevada. Clark County is the 5th largest school district in the country with...

Design: Fixing clocks | Negotiating Systems

Design: Fixing clocks | Negotiating Systems

I just came across a quote from Alan Kay while browsing the web. Alan Kay is a programmer, educator, jazz musician and one of the key inventors of computing as we know it today. He received the A. M. Turning award (informally known as the Nobel Prize of Computing) and...

MAET Graduate Celebration, Plymouth

Last Friday we celebrated the latest graduates from the MAET off campus program. These were students, who for the most part, have completed the MAET program over three summers in Plymouth, England. We here at MAET headquarters are extremely proud of their...


Submit a Comment

Your email address will not be published. Required fields are marked *