It’s about More than Money

Policymakers often talk about a “U.S. innovation system,” within which different sectors are funded to perform specific roles and which connect, tightly or loosely, to one another in well-known ways. Policymaking thus tends to focus on the level of funding and its distribution among sectors. But the framework used to study federal policy no longer fits the reality. As expressed in two recent National Academies reports, Measuring Research and Development Expenditures in the U.S. Economy (2004) and Understanding Business Dynamics (2007), there is increased recognition that the categories are outdated. Measuring R&D, for example, notes that the models of innovation that underlie the data are “increasingly unrepresentative of the whole of the R&D enterprise,” omitting factors such as “the growth of the service sector, the growing … role of small firms in R&D, the shift in funding from manufacturing R&D to health-related R&D, changes in geographic location, and the globalization of R&D.” Further, some scholars are describing new modes of “open innovation” occurring outside corporate R&D departments, and indeed often outside firms entirely. These trends transcend traditional vocabulary and concepts. We cannot assess the economic value of federal programs without an up-to-date understanding of how the economy functions and how innovation happens.

The rigorous but narrowly focused studies of the effect of the Bayh-Dole Act on university patenting and licensing, for example, do not examine whether changes in the practices of U.S. universities are leading multinational firms to shift their research support to universities and institutes in other countries. If this happens, it could also reduce the number of foreign nationals who enroll in U.S. universities and often stay as faculty, corporate researchers, or entrepreneurs to make significant contributions to the U.S. innovation system.

National and state science and innovation policies and programs are founded on a grab bag of theories, big and small. Big theories include the contributions that S&T are asserted to make to national defense, economic competitiveness, and human understanding of the physical world. Smaller theories address geographic, institutional, and demographic equity; specific perceived scientific or technological opportunities; and various short-lived policy fads. Although some of these rationales rest on firm empirical evidence, such as the suboptimal levels of private-sector investment in basic research and the contributions of scientific and technological innovation to national and regional economic growth, others hinge on symbolic value or wishful thinking. Some are simply wrongheaded, requiring agencies and performers to torque their activities to meet wasteful, vague, or counterproductive performance expectations and outcomes.

The fundamental problem with these unreliable analytic methods is that they can lead to flawed decisions about R&D policies and programs. For example, the Office of Management and Budget (OMB) relies heavily on the notion of market failure and the supposedly quantifiable contribution that federal R&D can make to economic growth as the measure of the effectiveness of programs, even as contemporary economic analysis finds it impossible to quantify with any precision the effect that a given unit of knowledge will have on economic performance. Conversely, other evaluations are the polar opposite of precision because they fail to include explicit statements of program objectives, agreement on what constitutes success, use of comparison or control groups in evaluations, or consideration of the value of foregone alternatives. These observations point to the need to take seriously Will Rogers’s adage that “It isn’t what we know that gives us trouble, it’s what we know that ain’t so.” We need to develop a more rigorous R&D evaluation system built on a foundation of current reliable data and new analytic methodologies.

Calls for research

To experienced participants in U.S. R&D policy debates, Marburger’s 2002 call for a new science of science policy that provided for “a systematic way of ordering the opportunities so finite resources can be invested to best effect” sounds a familiar refrain that dates back at least to the 1960s, when Alvin Weinberg advanced his intrinsic and extrinsic criteria for scientific choice. In effect, many new evaluative and predictive techniques, such as bibliometrics, patent analyses, foresight, and roadmapping, have been designed to provide evidence of past and current performance to better inform prospective actions. Recent calls for even more new models suggest that whatever progress may have been made along these lines over the past 40 years, it is still deemed inadequate, at least at the level of program and budget detail needed for policymaking. For example, the recent National Academies report A Strategy for Assessing Science (2007) concludes that “No theory exists that can reliably predict which research activities are most likely to lead to scientific advances or to societal benefit.”

The absence of a reliable model for predicting the value of R&D investments, at least for the near term, means that answers to the S&T policy questions cited above will likely continue to be made on the basis of a combination of expert judgment and competitive peer-review processes. This observation is not, however, an uncritical reaffirmation of the views expressed in various recent reports that expert review is the most effective means of evaluating federally funded research programs or that peer review is an international gold standard for evaluating science and engineering proposals. A recent stream of empirical research and participant observation has found that peer-review decisions can be skewed by a number of systemic factors, including panelist attitudes toward scientific risk and their cognitive maps concerning the structure of knowledge, small-group dynamics, the number and ordering of criteria, voting rules, instructions provided by funding agencies, and the intervention of program managers. In particular, the limitations of peer-review mechanisms, at least as conventionally implemented, are increasingly evident in at least three important areas: capacity to forecast trends in fundamental research not only within but across fields of science, receptivity to discontinuous or transformative research, and acceptance of interdisciplinary research.

Numerous modifications to peer-review procedures, including changes in the way reviewers are chosen and in the voting rules, have been proposed. Some have even suggested that a little randomness be included. These variations deserve to be tested because we know that the current system is not perfect.

If there is one area of S&T policy where there should be a solid research base, it is human resources. But as one examines the rationales for policy prescriptions in recent congressional testimony and high-profile reports, it becomes clear that there is little empirical evidence for the claims of shortages or surpluses in scientific and technical personnel. Recent legislation, however well-intentioned, reflects a time-honored tradition of making policy on the basis of simplistic analysis and simple solutions.

The enormously influential National Academies report Rising Above the Gathering Storm, which calls for recruiting 10,000 new science and mathematics teachers annually, essentially ignores the impressive data base of human resource surveys with roots as far back as the 1950s, as well as subsequent analyses by sociologists and economists who provide important insights into S&T career patterns. Research on the S&T labor force has failed to address the challenge of how the federal government can most effectively pursue national S&T objectives in the context of historic divisions across levels of government for K-12 education as well as distributed responsibilities among agencies.

The unfavorable international comparisons of student math and science achievement that underpin much contemporary U.S. discourse and policy also only scratch the surface of a complicated phenomenon. Many industrialized countries with high-scoring students are facing a science and engineering recruitment challenge that looks much like that in the United States. A reasonable hypothesis is that major structural shifts in the global economy are connected to the recruitment issues; demand for technical talent is increasing in other parts of the world and dampening immigration rates. The S&T policy research communities have barely begun to look at the connections among these trends or to generate policy options under these conditions.

The statistic that most sharply separates U.S. public-sector R&D spending from that of other industrialized economies is the prominence of defense-related spending in the United States. Because of the size of the U.S. defense R&D enterprise, researchers need to devote more attention to its effect on nondefense technological innovation; the competitiveness of U.S. industries; regional growth patterns; and the recruitment, training, and distribution of S&T personnel.

For example, although documented examples exist of major industries, including computers and aerospace, that have started or been sustained by military R&D, efforts of major defense contractors to diversify into the civilian sectors suggest that military requirements often do not stimulate the development of products that can survive in these markets. Relatedly, military installations are spread around the country, but military laboratories and contractors are more spatially concentrated. Does an R&D-intensive military activity produce different effects in local economies than does a more standard installation?

The role of politics

Science policy, to the extent that it addresses specific national objectives and the production of benefits for different constituencies, is manifestly a political process, with the usual battles among the branches of government, various constituencies, and warring ideologies. All this is obvious.

What is less obvious—or at least not systematically understood—is how these actors and factors interact over time to produce specific outcomes. Retrospectively, one can perhaps account for the constellation of influences that led to the doubling of the National Institutes of Health (NIH) budget. But can this help us understand what is likely to become of efforts to “balance” this effort by increasing the budget for the physical sciences and engineering while the NIH budget is held constant? The rich economics and political science literature about the social dynamics shaping the formation, force, and staying power of coalitions of interest suggests that the future will not be explained by anything as simple as a desire for balance. More important, given the increasingly widespread view that sharp discontinuities in the level of federal R&D funding disrupt scientific careers and result in unwise investments in physical infrastructure, what changes in the structures, processes, or criteria by which the executive and legislative branches form budgets might produce a more predictable funding pattern, and thus a more stable and productive scientific enterprise?

Similarly, with a view toward understanding and perhaps predicting the outcomes of the dynamics of political processes on S&T policy, how are coalitions of interest formed to promote or oppose specific initiatives? For example, what coalitions, using what arguments and with what influence, might be expected to form around recently advanced proposals to increase the Small Business Innovation Research set-aside above 2.5%?

Any large-scale federal S&T program can be expected to generate some positive results, be it peer-reviewed publications or improved performance of a technology. Conversely, the inherent uncertainty surrounding R&D, especially basic research undertakings, guarantees that there will be a number of projects that can be described as unsuccessful. Without clearly defined expectations and initial agreements on what S&T programs are designed to achieve, current reviews of federal S&T undertakings, whether by OMB or a congressional committee, are unavoidably subjective.

Scientific research has always been an international activity, and the rapidly evolving geopolitical environment is of critical importance. As recently as a decade ago, U.S. policymakers could talk confidently about broad S&T leadership. A National Academies committee recommended that the United States seek to be a leader in some fields of science and a fast second in others. Yet as the evidence of growing strength elsewhere mounts, the United States clearly needs to plan for a future in which it is among the world leaders, rather than the dominant force, in most areas of science and engineering. Given the high cost of major scientific undertakings, increased patterns of international scientific cooperation, the polycentric location of R&D laboratories by multinational firms, the emergence of new and potentially significant contributors to international science, and rapid diffusion and transfer of scientific and technological knowledge across national borders, what strategic principles should guide U.S. S&T policy in a world increasingly characterized by complex sets of interactions—partly cooperative, partly competitive, partly independent—with other countries?

These questions also spill over to innovation policy and concerns about U.S. economic competitiveness. For example, if scientific leadership shifts to other countries, are U.S. industries equipped to be fast followers? Is their absorptive capacity equal to their innovative capacity? Will U.S. universities be as capable of providing a window on the rest of the world as they have been of providing a window for the rest of the world? Under what conditions will U.S.-based economic activities thrive in the new global economy?

Research on these topics has to move fast to keep up with the changing realities. But existing national industry surveys do not do much to track international patterns, and national data sets on the workforce include scant information on the increasing flows of scientists and engineers among countries and regions. With a few exceptions, the research on international S&T collaboration is not charting this process. Part of that literature focuses on international collaboration in Big Science, a game played mostly by the rich countries. A large part focuses on the movement of technology through foreign aid programs but neglects the greater volume of technology transfer through multinational firms. There is thus considerable scope for new research on the S&T elements of relationships in a developing world economy. Because the dominant theoretical model relies on the idea of science as a self-organizing system, there is plenty of scope for testing the effect of policy interventions in that system. Mapping technology transfer through its public and private routes will be an important part of the agenda, and studying the conditions under which technology transfer builds lasting capacity and continuing partnerships will also be useful.

Not surprisingly, many items on the above agenda are familiar ones. Their inclusion reflects the perennial challenges that researchers and policymakers confront as they try to reduce the uncertainties and complexities surrounding processes of scientific discovery and technological innovation. Given these challenges, we should have modest expectations about the new science of science and innovation policy. Perhaps even more so, modesty may be needed on the part of those advancing claims about new and improved theories, models, or algorithms. As we have demonstrated, serious gaps exist in the knowledge base on which new theories must be built. The success of the current initiative for a science of science policy will depend less on having more pieces of data than on how well the pieces are put together. Increased dialogue between the policy and research communities is a necessary precondition for completing the puzzle successfully.