Making it in academia! Hmmm…?

by | Monday, January 10, 2022

The question of impact of one’s work is something that all researchers and scholars care about, particularly in applied fields like education. The question, however, is how is impact to be measured? Over the past few weeks I have had a few instances where my work has been recognized for its impact—on public discourse, in citations from other scholars and in teaching. In at least two of these cases, I am proud of the recognition—and yet, even in those two, I have to admit to some mixed feelings. I question what that “metric” on which my work is “scored” really means, who is included in the scoring and who is not. So it is with these somewhat ambivalent feelings that I present three examples that, just happened, to pop up in the past few weeks.

The first, and maybe, the most meaningful to me personally, was that I made it to the 2022 RHSU Edu-Scholar Public Influence Rankings, at number 77 no less. For those who don’t know what this means, the Edu-Scholar Public Influence Rankings list the 200 university-based scholars who had the biggest influence on educational practice and policy last year. This is the first year that I made the list – and though I was pleased, I was also somewhat puzzled, particularly by the fact that I had landed at number 77! In my very first year? Should I not have started somewhere at the bottom, in the 190’s and worked my way up?

As I looked over the process that was used to generate this list, I wasn’t sure anything significant had changed in the past year in how these rankings were generated – so why did I make this year and not before? The only thing I could think of was that my name had just not been included in the list of names that were researched. This, of course, makes me wonder about all the other people who should be on this list and are not.

This is not to say I am not happy to see my name on this list – but it also gives me a sense of humility as I recognize the inherent randomness in the process. (To be fair, Rick Hess, who curates this list, provides a summary of the process and recognizes possible weaknesses in this, and actually, any approach that seeks to create such a list). As he says

This whole endeavor is an imprecise, highly imperfect exercise. Of course, the same is true of college rankings, NFL quarterback ratings, or international scorecards of human rights. Yet, for all their imperfections, such efforts convey real information and help spark useful discussion.

If the first piece of news was about public influence, the second item focuses more on the academic side of things, on numbers that academics really care about i.e. citations. To give some context, the standard way of measuring the impact of a piece of research is by counting the number of times the piece (usually a journal article) is cited by others. The idea here is that more important research will be cited more often than research that is less important. Citation analysis is typically used by governments, funding agencies, and university tenure and promotion committees to evaluate the productivity and quality of a piece of research – and, via that measure, to judge a researcher’s work. So, one can imagine this is something academics care for even though we know that 90% of papers published in academic journals are never cited.

As you can imagine citation analysis is complicated (for a range of reasons that I will not get into here). Recently, a few researchers attempted to create a standardized citation metrics author database of 100,000 researchers. They made this database (and its updates) public, so essentially anyone can go to the links at the end of this post and download the Excel spreadsheets and see/play with the data. This, being as ego-driven as the other guy, is exactly what I did.

Long story short, of the approximately 70,000 scholars that had education as their field of interest, I was ranked 466 (across my career) and 196 (for the year 2020). This sounds quite amazing really… but a few key caveats are in order. First, the point I had made above, that 90% of published papers are never cited, so that 70,000 number is not necessarily as meaningful as it sounds!

Second, and maybe more important, is the bias inherent in which journals are included in this tabulation. A little bit of digging reveals that the authors extracted data from the Scopus bibliographic database, which contains abstracts and citations for academic journal articles from over 21,000 titles from over 5,000 publishers, primarily focusing (and this the important part) on the scientific, technical, medical, and social sciences. This sounds impressive, but is deceptive, since the journals in this database are not necessarily representative of impactful work in my field, education. This often includes incredibly important and critical work done by scholars working in areas of social justice, philosophy, educational history, teacher education and more. So once again, I am happy to see these numbers, but I also take them with a healthy dose of salt.

The third and final indication that I had arrived was when I learned this morning that students could now purchase course assignment “reflection” papers about my work from an online paper-mill. And there are more than one of these paper-mills out there, letting students download (for a fee of course) papers about the application of the TPACK framework. I am not linking to those websites – since I don’t want to support this line of work that, in some fundamental way, undermines the important work that I, and many of my faculty colleagues, are engaged in—namely to get students to think and to express their thoughts and perspectives through writing. As the screenshots below show, these sites make some impressive claims and promises.

I guess I should feel good about the fact that the TPACK framework is established enough that it has become part of the routine questions that are asked in teacher education courses. But, to me it is more an indication of the fact that TPACK is now just one more framework, one more set of acronyms, that students are supposed to memorize. It is not something they need to engage with intellectually, conceptually and practically…. and that, to be honest, makes me sad. And the words of T. S. Eliot, in the Love Song of J. Alfred Prufock, come to mind,

 “That is not it at all,
               That is not what I meant, at all.”

________________________

References

Ioannidis JPA, Baas J, Klavans R, Boyack KW (2019) A standardized citation metrics author database annotated for scientific field. PLoS Biol 17(8): e3000384. https://doi.org/10.1371/journal.pbio.3000384. (You can also access the updated database as of August 2021).

Topics related to this post: Worth Reading

A few randomly selected blog posts…

EDUsummIT 2017: Summary Report

EDUsummIT 2017: Summary Report

EDUsummIT 2017 is the fifth International Summit on Information Technology (IT) in Education (EDUsummIT 2017) recently held in Borovets, Bulgaria, on September 18-20, 2017. EDUsummIT 2017 was co-hosted by the University of Library Studies & Information...

The civilizing effects of technology

Martin Amis was recently interviewed in Guernica (Amis Unfiltered, Santiago Wills interviews Martin Amis). The interview covered a wide range of topics, literature, Obama, and a fascinating digression on the relationship between food and national character!  What...

Happy New Year (and a new illusory video)

Happy New Year (and a new illusory video)

Since 2008 we have been creating short videos to welcome in the New Year. These videos, created on a shoe-string budget, are usually typographical in nature with some kind of an optical illusion or aha! moment built in. Check out our latest creation to welcome 2019...

New optical illusion: An oscillating visual paradox!

New optical illusion: An oscillating visual paradox!

A design for the word "illusions" inspired by a design by Scott Kim.  I have been obsessed with optical illusions for for a long time. This interest has played out in many ways: from the hundreds of ambigrams I have created to the new year’s videos we create as a...

Cosmetic changes

I have made some cosmetic changes to the way the blog looks. The sidebars are now light blue, to differentiate them from the middle (content heavy) column. Once I did this I realized that I did not need that boxy border around the middle column, and pouf, it was gone....

Teaching to learning styles, what hogwash

There is an article in today's Chronicle titled Matching Teaching Style to Learning Style May Not Help Students. I have been somewhat skeptical of the learning styles literature for a while, not the least for hearing the phrase being bandied about without much...

The pleasures of being a teacher

Yesterday, as I was watching the second presidential debate, and following various bloggers who were live-blogging the event, I took a moment to check my email. I found that I had received a note from a former student. This individual had been in my summer cohort last...

iSad

Steve Jobs (1955 - 2011) To all of us who value creativity and design the passing of Steve Jobs is the passing of an era. I know my world changed when I first saw the Macintosh. I was a freshly minted electrical engineer who was trying to get out of engineering...

Chiayi, Chung Cheng & on to Kuosheng

I had been looking forward to the high speed rail journey though I had some concerns about navigating through the train station since most of the signs were in Chinese and Waiway (the graduate student who had come to pick me up from the airport) could not come with me...

2 Comments

  1. Punya Mishra

    Thank you Robin. Your comment was deeply meaningful to me. Maybe more so than the three mentioned in this post (well the first two for sure). Truly. ~ punya

    Reply
  2. Robin Angotti

    Another measure is less obvious. It is the unseen impact you have made on individuals like me. The time you took to have conversations with me at conferences, over lunch, helping me think about how my own work was an example of the TPACK framework. I don’t think I would be where I am today had it not been for your impact on me and your willingness to share some of your time with me. Thank you.

    Reply

Leave a Reply to Robin Angotti Cancel reply

Your email address will not be published. Required fields are marked *