Money Well Spent?

By Irwin Feller

Measuring the Economic Value of Research is an important book, edited by a distinguished group of researchers who focus on the science of science policy, an emerging interdisciplinary approach to evaluating the scientific enterprise. The book offers a case study of government support of academic research on food safety, mainly by the US Department of Agriculture, the National Institutes of Health, and the National Science Foundation. The case study showcases the analytical and empirical prowess of a new data platform called UMETRICS (Universities: Measuring the Impacts of Research on Innovation, Competitiveness, and Science). The UMETRICS platform is a conceptually and technically innovative approach for obtaining, refining, and integrating heterogeneous data sets that can be used to better inform decisions relating to the level, apportionment, human capital needs, and physical facility requirements of public investments in research and development (R&D) and the impacts or returns that these investments have on policy objectives.

The food safety case study is a collaborative undertaking, with individual chapters authored by interdisciplinary teams of experts on the subject matter as well as on statistics, information science, and other areas as needed. As such, it presents a multidimensional array of findings relevant to decisions about the funding and assessment of food safety research, the future supply and competencies of the field’s labor force, and prospective career and employment opportunities for students enrolling in graduate studies in related disciplines. Additionally, interspersed throughout the book’s opening and concluding descriptive chapters are salient observations that connect its methodology and findings to a range of contemporary science policy trends of relevance both to food safety R&D and a swath of federal research investment decisions. These include the increasingly targeted character of federal support of food safety research; trends away from smaller, less costly, single-investigator-driven work to larger, more expensive multidisciplinary projects; and overall declines in real levels of research support.

In this review, I focus on the book’s claim about generalizability, namely that UMETRICS is a “novel template that the science of science policy community can use to assess the impact and value of research that extends to other scientific fields.” In good part, the book justifies this claim.

The initial inputs to the UMETRICS platform are the employee files of individuals involved with federal and nonfederal research projects, provided under strict confidentiality protocols by the human resource systems of cooperating universities. (There are currently 50 cooperating universities in total, of which 19 provided the data employed in the food safety study.) The platform consists of a data architecture that links demographic and expenditure data from these files with earnings, placement, bibliometric, patent, and other forms of data from the US Census Bureau and other groups. These include the Longitudinal Employer-Household Dynamics Data program, which provides data on employment and earnings, and the Census Business Register, which provides information about the characteristics of the businesses where individuals get jobs. Other public sources provide details about dissertations, publications, and patents.

This approach makes several notable contributions to the evaluation process. By starting with research project data, rather than, say, data from surveys of principal investigators alone, the UMETRICS platform captures the full set of individuals—faculty, students, postdoctoral researchers, technical staff, and others supported by federal research grants. This expanded coverage allows for a much wider assessment (by number of observations and functional roles) and flexibly disaggregated analysis of the downstream impacts of academic research grants. It also generates a sufficiently sizeable database to address several related science policy issues, such as the size, composition, and productivity of variously sized research teams and the demographic characteristics—age, gender, race, and national origin—of the food safety research labor force.

Noteworthy among the platform’s innovative features is the construction of comparison groups of graduate students and postdoctoral students engaged in food safety research. The platform distinguishes among three groups: those engaged in food safety; those engaged in intense food safety programs, but not food safety per se; and the far larger number of all other students captured in the UMETRICS data set. Given current requirements or expectations for some modicum of evaluation design in presentations of evidence of program accountability or effectiveness, this construction of comparison groups strengthens the case presented about the impacts of food safety research.

Although readily describable in terms of emerging “big data” approaches to social science research and program evaluation, the strength of the book rests in its “small data,” theory-driven, and exacting construction of its constituent datasets. Consider the necessary first step of data taxonomy upon which other datasets are based, namely how the scope of food safety, a “diverse field, encompassing many different disciplines,” should be defined. Noting that current scientific taxonomies do not provide clear labeling for many new or interdisciplinary fields, including food safety research, the study draws upon advances in text analysis, such as searchstring, wiki-labeling, and topic modeling, to identify an initial set of research grants. Subsequent chapters describe similar adjustments in isolating, filtering, and then aggregating federal agency academic R&D expenditures, doctoral program enrollments, career placements, and other information to construct a consistent series from among diverse and heterogeneous datasets.

Indeed, Measuring the Economic Value of Research is so strong as an expert technical monograph on the construction of new and improved datasets that the book justifies its claim as a template for studying the diverse and extensive range of impacts of academic research in other domains. Only a small percentage of the UMETRICS database is employed in the study of food safety. Programmatically, it could readily be applied to other fields, such as health or environmental research, where academic personnel represent a large percentage of the R&D performers and account for substantial shares of related research expenditures. Adjustments, though, would be necessary to bracket the aggregate economic significance of agency support of academic research in domains such as national security, space exploration, and technological innovation, where the university role in total national research activity is relatively smaller.

The shift in the book’s latter chapters, from filling in the components of the platform’s data architecture to presenting the economic impacts of research projects, however, points to the current limitations of UMETRICS as a template for addressing the full suite of questions typically embedded in the formulation and assessment of science policies. One of these, to cite John Marburger III, the respected physicist and university president who served as the science adviser and director of the US Office of Science and Technology Policy under President George W. Bush in his initial articulation of the need for a science of science policy, is whether trends in the absolute or relative levels for funding for specific agencies and missions are “disturbing”—far too high or too low.

The book’s characterization of the ways in which scientific ideas are transmitted to and constitute value to the broader economy encompasses publications and patents, but most importantly includes the employment of people trained in food safety research. This emphasis on human capital reflects a core proposition of UMETRICS, namely the “importance of people—students, principal investigators, postdoctoral researchers, and research staff—who conduct research, create new knowledge, and transmit that knowledge into the broader economy.”

In particular, the chapters on workforce dynamics relating to employment, earnings, occupations, and early careers highlight the nuanced, disaggregated, and policy-relevant information made possible by UMETRICS. These data provide much-needed reinforcement to the historic proposition advanced by research-oriented universities that their major contribution to societal well-being—economic and beyond—is through the joint production of research and graduate education, more than patents or other metrics of technology transfer or firm formation.

As novel and flexible a contribution as the UMETRICS platform may be in addressing academic research performance, its importance and probable use as a template for assessing the impact and value of scientific research is likely to be highly contingent on the types of science policy questions being asked and the preferences, for whatever reasons, of decision-makers among different modes of assessment, methodology, and evidence.

For example, the findings from Measuring the Economic Value of Research do not demonstrate the contribution of federal investments in academic food safety research to food safety per se, as measured, say, by changes in the number and severity of unsafe food incidences or the health and economic effects caused by these incidences. Nor are connections made between the production of new knowledge and the adoption of new safety standards, voluntarily or via regulation, in the production, processing, and distribution of food, or the enhanced ability of regulatory agencies to present scientific evidence to defend their food safety regulations.

Brief note is made of the high rates of private and social rates of return to research reported in econometrically based production function studies, but the study itself does not venture into this area. Thus, at the technical level, it is not clear how one would integrate into or compare findings from this or other UMETRIC-based studies with the earlier and still significant line of research on the economics of agricultural research. This research is exemplified by Robert Evenson, Paul Waggoner, and Vernon Ruttan’s classic 1979 survey published in Science, “Economic Benefits from Research: An Example from Agriculture,” and by the economists Keith Fuglie and Paul Heisey’s more recent 2007 report for the Agriculture Department’s Economic Research Service, Economic Returns to Public Agricultural Research. Nor is it evident from the book how the UMETRICS approach can be employed to address one of the fundamental questions of science policy: what is the optimal level of research support?

Also not treated in the book are the means and channels through which research findings affect the behaviors of individual consumers, producers, governmental entities, and others that in the end shape final outcomes. Here the recent studies of the adoption of agricultural innovations by France’s Institut National de la Recherche Agronomique and its ASIRPA project (which in English translates to Socio-Economic Analysis of the Impacts of Public Agricultural Research), a program that seeks to disentangle the roles of networks of actors in the innovation process, provide a useful complementary approach.

The book is dedicated to John Marburger III, who was an early advocate of using empirical tools to evaluate science and innovation policy. Measuring the Economic Value of Research is thus a fitting tribute to his vision and leadership, representing the research rigor and empirical orientation contained in his call for a new science of science policy.