Data-Driven Science Policy
A critical challenge for science policy decision makers is determining how to spend limited resources most productively. To do so, one must have a basic understanding of the inner workings of the science, technology, and innovation (STI) system, knowledge of where the most productive research is being done, and an awareness of how progress proceeds across numerous individuals and institutions. Advances in computational power, combined with the unprecedented volume and variety of data on science and technology developments, create ideal conditions for the development and application of data mining and modeling approaches that reveal the dynamics of research progress and can augment human judgment in allocating resources. STI studies use large-scale publication, patent, funding, news, social media, and other data to rigorously study the structure and evolution of the science and technology landscape; they use advanced visualizations to communicate the results of these studies; and they can empirically validate the results of various policy and funding strategies.
Today, science advice is provided by senior researchers in universities, industry, and government. Most experts make rather limited use of the high-quality and high-coverage datasets or the advanced data mining and modelling tools that are now available. The time is ripe to augment the human intellect with data and tools.
Industry has long embraced big data and advanced data mining, modelling, and visualization algorithms. Computational models are widely used by Amazon and Netflix to anticipate consumer behavior, by financial companies to detect credit card fraud, and by insurance companies to set rates. Many companies use models internally to support strategic decision making and to guide investment decisions.
Several scientific disciplines have established billion-dollar international data infrastructure and distributed computing systems in close collaboration with government and industry partners. Examples include meteorology for weather forecasts and storm warnings, epidemiology to predict the next pandemic or to identify the best intervention strategies, and climate research to develop future scenarios or to set carbon prices.
Although no comparable infrastructure exists yet for the study and management of the STI system, some experts have begun to interlink data silos and use computational models to improve STI decision making. The goal is to use simulations of possible futures to help increase our understanding of the dynamics of the STI system and to model the likely results of possible policy interventions. Models are being applied to explore questions such as: Which career paths are more likely to lead to high-impact work? Which institutions will likely be most productive in the future? Which funding strategy has the highest return on investment? Other models examine the influence on research of larger social factors such as changing demographics, alternative economic growth trajectories, and power relationships among nations. Decision makers who embrace data and models have better means to identify the factors that were most important in explaining previous events and they can use this insight to help project which levers might be most effective in influencing future developments. A few case studies can illustrate the variety of ways in which models have been employed.
Small is beautiful
Contemporary science is a collaborative effort within an intricate network of people, institutions, concepts, and technology. Many projects are of such complexity or scope that they require the joint efforts of many individuals with diverse expertise, sometimes reaching team sizes of a few hundred. The evidence indicates that large interdisciplinary teams are likely to produce high-impact work. But it’s not simply a matter of bigger being better.
Staša Milojević at Indiana University developed a model of how teams emerge and grow and found that the key to the success of large teams was the existence of relatively small, core teams or even single investigators responsible for key pieces of the research. Surprisingly, the model shows that relatively small teams dominate knowledge production in most fields, so that cumulatively, they still contribute more new knowledge than large teams. These findings are of key importance to policy because they show that increased funding emphasis on large teams may undermine the very process by which large, successful teams emerge.
The wisdom of crowds
Johan Bollen and colleagues at Indiana University argue that scholars “invest an extraordinary amount of time, energy, and effort into the writing and reviewing of research proposals” with the result that funding agencies are consuming resources that could be more productively used to finance research. In a 2014 paper, they use National Science Foundation (NSF) and Taulbee Survey data to provide a simple calculation of return on investment for scholars in computer science. They find that a representative proposal would require the work of four professors for four weeks, which would cost roughly $35,000; given the current success rate of 21% in the discipline, it takes an average of five proposals to win a grant, for a total labor cost of $175,000. The average NSF grant is about $165,000 per year, and when university indirect cost rates are deducted, it leaves about $110,000 for research. In other words, average success results in a net loss for faculty. This calculation does not include the time that faculty spend reviewing grant proposals—in 2015 alone, NSF commissioned 231,000 reviews to evaluate 49,600 proposals.
Bollen et al. then go on to propose a FundRank model to (partially) replace the current process of government research funding allocation with expert-based crowdsourced funding. In the proposed system, every researcher with acceptable credentials would receive a certain dollar amount each year, let’s say $100,000. Each researcher would then designate a certain fraction, say 50%, to colleagues who are most deserving. That is, scholars collectively assess each other’s merit and fund-rank other scholars, with highly ranking scholars receiving the most funding.
Instead of spending weeks writing and reviewing proposals, scholars would be incentivized to spend time communicating the value and impact of their past, current, and planned work so that others can judge their contributions and ambitions. Using a fully digital system, conflicts of interest can be easily identified and networks of mutual favors can be detected automatically.
To test how this system might affect the distribution of research funds, Bollen et al. used the PageRank algorithm pioneered by Sergey Brin and Larry Page to implement a computational model that uses citation data as a proxy for how scholars might allocate funding to other scholars. In the model, the “importance” (i.e., reputation, value, impact) of scholar S depends not only on the number of citations received but also the “importance” of the scholars that cited S—the more citations by important scholars, the more important S must be. The FundRank model was validated using citation data from 37 million papers over 20 years as a proxy for how each scientist might distribute funds in the proposed system. For each scientist, his or her actual NSF and National Institutes of Health funding over a decade was compared with the amount of funding predicted by FundRank; results show that the two are closely correlated, at a fraction of the cost required by the current system.
A virtual test drive
Policy decision makers need to understand and trust modelling results, or they will not use them in practice. Visualizations of the modelling process and modelling results have proven invaluable for providing a strong intuitive feel for model predictions and insights. William Rouse’s team at the Stevens Institute of Technology has been working closely with the National Academies of Engineering and Medicine to implement “policy flight simulators” that let decision makers fly the future before they write the check.
The technique works by gathering a group of five to 15 policymakers in a room equipped with many large screens. The group discusses options that they can then run through the simulation model. Graphic presentations of the results are projected on the screens. In one session, representatives of New York City’s 66 hospital corporations met to explore how the provisions of the Affordable Care Act could affect merger and acquisition activities. Using the simulator, they were surprised to find that major players’ strategies, relative to their primary competitors, very strongly affect the “pecking order” resulting over the coming decade. For example, a hospital that for a long time had been the first or second largest discovered that failure to consider the strategies of competitors could result in it dropping out of the top five over the coming 10 years.
Another policy flight simulator session focused on the adoption of automobile power-train technologies, comparing internal combustion, hybrid, electric, and hydrogen systems. As expected, modelling results show that electric vehicle purchases are likely to increase if the federal government provides subsidies and states invest in battery-charging infrastructure. Surprising, however, is the indirect effect of Corporate Average Fuel Economy (CAFE) standards. With fuel costs very low, Americans are buying more pickup trucks and large SUVs, which can have profit margins approaching $10,000 per vehicle. To meet CAFE standards for their fleets, automakers have had to lower the cost of smaller, fuel-efficient cars—sometimes selling at a loss of up to $2,000—to stimulate sales. As economy cars become cheaper, they take away sales from hybrid and electric vehicles, undercutting government incentives.
Inventing the future
The development and implementation of easy-to-use and actionable models for STI decision making pose diverse challenges and great opportunities. An interdisciplinary approach and a close collaboration among academia, government, and industry are needed to identify “grand challenges” and to develop data infrastructures and systems-science modelling approaches that truly address these challenges. Computational models will need to be vetted by experts and earn the trust of the scientific policy making community before many start using them in practice. The key to building trust is transparency and the engagement of all stakeholders in the design and application of STI models.
To bring relevant stakeholders together, a “Conference on Modeling Science, Technology & Innovation” will be held at the National Academy of Sciences in Washington, DC, on May 17-18, 2016. The conference is funded by the NSF’s Science of Science and Innovation Policy program. A draft agenda is available at http://modsti.cns.iu.edu.
The conference will feature presentations by leading experts from academia, government agencies, foundations, and industry that develop or apply computational models to increase understanding of the structure and dynamics of STI systems. Case studies will be presented by the National Institutes of Health, the National Oceanic and Atmospheric Administration, the Centers for Disease Control and Prevention, and other agencies that use models to recommend reviewers, estimate the success of clinical trials, or predict the impact of different funding types. More than 20 academic researchers from economics, social science, scientometrics and bibliometrics, information science, and science policy that develop mathematical, statistical, and computational models of different types will present their results. All of these scholars worked closely with decision makers to ensure their models address key insight, communication, or decision making needs. Industry will showcase how large-scale datasets and advanced computational models are used in commerce, education, or finance.
The conference aims to synthesize the current state-of-the-art in the area of modeling STI and to provide substantive input for future research and development endeavors. It welcomes broad participation in the consensus-building process among producers (researchers, industry, and government) and users (science policy makers and other decision makers) of models when identifying “grand challenges” in fundamental research, applied research, cyberinfrastructure, education, and outreach.
Katy Börner ([email protected]) is the Victor H. Yngve Distinguished Professor of Information Science in the Department of Information and Library Science, School of Informatics and Computing, Indiana University, and a visiting professor at the Royal Netherlands Academy of Arts and Sciences in The Netherlands and the University of Duisburg-Essen, Germany.