Measuring Research Benefits
With “Are Moonshots Giant Leaps of Faith?” (Issues, Spring 2017), Walter D. Valdivia has joined the distinguished ranks of science and technology policy analysts who have written eloquent explanations of why ex post evaluation of research and development (R&D) investments is so difficult, if not impossible, at any but the highest levels of aggregation. He poses an interesting question: whether abnormally large increases in government-funded R&D program budgets, which he calls, somewhat infelicitously, “moonshots,” yield proportionately large benefits. He then details many of the reasons we are not generally able to analyze the benefits of more routine R&D budgets, never mind those that receive large injections of new money in a short time.
Though one might quibble with one or two of his claims, the overall thrust of his article is right on point. Quite naturally, citizens, politicians, and all manner of experts would like to be able to quantify the benefits that result from our huge public (and private) investments in R&D. There are good reasons for asking this question about the aggregate R&D budget as well as about various parts of it, right down to the level of the individual research project and the individual researcher.
Unfortunately, as Valdivia nicely demonstrates, we can’t provide a straightforward and fully satisfying answer to the benefits question at any level. At best, we can examine various surrogates, indicators, partial measures, and indirect hints to try to get some empirical purchase on the answer. In keeping with Valdivia’s final claim, at the end of the day there is still no substitute for informed expert judgment, with all its biases and aided by the available inadequate measures, to tell us both what got from past R&D investments and what we might get from future ones.
Christopher T. Hill
Walter Valdivia provides a good summary of the literature on the effects of science on society at three levels: (technological) innovation, knowledge, and research organization. His views have been well known for decades. He cites the difficulty of measuring the links between research and economic growth, the limitations of publication and citation counts, and the limited administrative capacity for making enlightened choices in promising fields.
Valdivia’s recommendations, however, do not cover the full scope of his criticisms. His discussion is essentially concerned with technological impacts, but it does not address the full array of impacts, particularly those less quantifiable, such as cultural impacts. Neither does he discuss the negative impacts of the application of science. He does suggest that the “full array of means by which knowledge production meets people’s needs” should be considered, but that is all. Valdivia calls for investments in administrative capacity, in general-purpose technologies, for specific goals, and he calls for agencies to pool their political capital for greater effect.
I think it is time to articulate the issue of science and society in totally new terms. A new paradigm and, above all, a new discourse are needed. First, we must admit that social scientists have never managed to produce the evidence necessary to demonstrate a link between science and society (although we all believe intuitively that there is such a link). Second, we (scientists and their representatives) still defend science publicly based on a decades-old discourse. Yet we have never convinced policy makers with a discourse on social and economic impacts, because “science and technology funding is more likely to be increased in response to threats of being overtaken by others (Sputnik, Japan, Germany, now China) than it is to respond to the promise of general welfare or eventual social goods,” as Caroline Wagner said on the National Science Foundation’s Science of Science Policy Listserv.
I have no ready answers as to what this new discourse should be, although training of students certainly should be a central part of it, and knowledge as a concept should be less abstract than it is now. One thing I am sure of is that the scholarly analyses and the public discourses of scientists have to make a tabula rasa of everything we have long assumed. Everyone proclaims the linear model, in which all innovation begins with basic scientific research, is dead, but in fact it is still alive and kicking. The issue is not whether the model (and its many variants under different names, such as the chain-linked model) is right or wrong, but that it is not the appropriate “marketing” tool to sell science to the public. Today, innovation has taken the place of research as a cultural value responsible for growth and welfare, and research has very few hearings in the discourse of progress. For better or worse, scientists have to take this into account.