The Return of Science and Technology Assessment for Congress
The COVID-19 pandemic has thrown into stark relief the nation’s urgent need for timely, reliable scientific information. Though information on the pandemic is in ample supply, less readily available is reliable analysis of what this flood of information means.
The United States Government Accountability Office (GAO) focuses on just that careful evaluation. Our newest internal unit—the Science, Technology Assessment, and Analytics (STAA) team—provides Congress with timely, independent, and relevant analysis to both respond to and anticipate shifts that will, like COVID-19, profoundly impact daily life.
Congress has invested in the team because of a growing, bipartisan recognition that science, technology, and innovation challenges require a new approach. GAO’s century-long mandate to ensure the accountability of the federal government positions us to maintain strong oversight of taxpayer spending on science and technology. And by growing our expertise and focus on this topic, we can help Congress navigate the increasingly complex technologies it must govern, including artificial intelligence, quantum computing, and rapid vaccine development.
In the year and a half since our team was established, we have led 40 reports and contributed to over 250. Our work includes technology assessments, program evaluations, two-page explainers called “Science & Tech Spotlights,” and participation in hundreds of external events.
By growing our expertise and focus on this topic, we can help Congress navigate the increasingly complex technologies it must govern, including artificial intelligence, quantum computing, and rapid vaccine development.
The STAA team is also increasing the speed at which the GAO responds to emerging issues. For example, the team responded swiftly to the COVID-19 outbreak, releasing an overview of coronaviruses on March 3, 2020—six weeks after the virus was detected in the United States—and followed up with several related Spotlights. We reported on the many uncertainties in the science behind social distancing, as well as on the potential for accelerating vaccine development. In the latter Spotlight, we reported that 110 vaccines were in development around the world as of May 15, with at least three receiving federal funding, and that expedited vaccines might raise unintended safety and distribution risks.
In a more in-depth report, the team reviewed COVID-19 forecasting models, which can help predict trends such as infection or mortality rates. We found that because these models rely on data collected by different jurisdictions and reported under different standards, it is difficult to compare data across places and over time. In our overview of herd immunity, we similarly reported that the data are insufficient to show how long COVID-19 immunity might last or whether it’s enough to prevent reinfection.
The team has also contributed to larger GAO efforts around COVID-19. In June 2020, the GAO issued a major report on federal response and recovery efforts, which found inconsistencies in viral testing numbers, critical supply shortages, and confusion about the Paycheck Protection Program. The team provided the technical expertise that helped Congress better understand what key actions the federal government had taken to address the pandemic and, importantly, what lessons could be learned from the response.
Even before COVID-19 cases surged in the United States, the GAO was examining urgent issues in health and technology. In a December 2019 technology assessment, we reported on how machine learning could reduce the time and costs required to bring new drugs to market by finding new insights in large biomedical or health-related data sets.
Our position in the nation’s leading oversight agency affords us a look at the entire federal enterprise and an opportunity to lead research in crucial science and technology areas. One such example was our May 2020 assessment of forensic technology. We examined how federal law enforcement agencies are using algorithms to partially automate the assessment of evidence, and found that these tools can potentially speed up investigations and reduce human bias and error.
In our overview of herd immunity, we reported that the data are insufficient to show how long COVID-19 immunity might last or whether it’s enough to prevent reinfection.
In testifying to Congress in July 2019, we took a closer look at the chemical industry, which supports nearly 26% of the US gross domestic product, the broadest measure of goods and services produced by the nation’s economy. Our work on chemical sustainability found that stakeholders in government, industry, and academia vary in how they define and assess the sustainability of chemical processes and products, and that these differences hinder the development of more sustainable chemistry technologies. In response, we recommended that stakeholders create an industry consortium that would work with key federal partners to help make sustainable chemistry a priority and to develop a national initiative or strategy aimed at that goal. Crucially, our report and corresponding testimony spurred Congress to enact the Sustainable Chemistry Research and Development Act of 2019, which directed the Office of Science and Technology Policy to convene an interagency entity responsible for coordinating federal programs and activities in support of sustainable chemistry.
Taken together, GAO’s work provides in-depth, critical analysis of emerging technologies and how they might shape society.
Expertise, transparency, trust
Though many observers are voicing concern about a decline in the federal government’s deference to scientific expertise, the STAA team, consistent with GAO’s tradition, is using its position as a nonpartisan resource for Congress to provide trusted recommendations rooted in rigorous, transparent methodology. Fundamental to this is the team’s wide-ranging science and technology expertise, including external experts across academia, think tanks, and industry. At present, the team comprises 94 staff members, at least 61 with an advanced degree in science, technology, engineering, or mathematics. It also has operations research analysts and project controls engineers as well as members who hold advanced degrees in public policy, rounding out the team to expertly advise at the nexus of technology and policy.
This expertise carries over to each of our technology assessments, reviews that begin with and consistently reflect a keen attention to detail. In our Technology Assessment Design Handbook, published in December 2019, we outlined the four stages of this process that are critical to developing rigorous work: initiation, design, message development, and external review.
In the initiation phase of our work on machine learning in drug development, for instance, we began by discussing the scope and focus of the project with congressional requesters, their central questions being: how are artificial intelligence technologies used in drug development and what are the potential policy implications?
GAO’s work provides in-depth, critical analysis of emerging technologies and how they might shape society.
With that key question in mind, we moved into the design phase, where we performed initial research on the use of AI in drug development, consulted with GAO subject matter experts, and began identifying initial policy options. By the message development phase, we were collecting and analyzing evidence, assessing results, and drafting findings. Among our conclusions were that more high-quality data, increased data sharing, and uniform standards would go a long way in addressing the research gaps, regulatory uncertainties, and other challenges hindering the use of machine learning in drug development.
Our work is rooted, too, in a rigorous review process. Indeed, our reports are thoroughly cross-checked and referenced to ensure that the data presented and the recommendations proposed are reliable. In practice, this means that a person independent of the research team checks each statement of fact in a report or other product. He or she confirms that all facts, figures, and dates are independently traced back to the supporting evidence, that the findings are adequately supported, and that an independent methodologist has approved any technical data used.
Beyond these internal reviews, we also involve external experts—often with the assistance of the National Academies of Sciences, Engineering, and Medicine—over the course of our work. Once we have selected a group of experts that represents the needed cross-sectoral expertise, we convene a meeting of those specialists to highlight and discuss the latest research in the field. We then contact those experts over the course of our work to gain additional input as needed. Once we have developed a draft report, the specialists who participated in our initial meeting review our draft and other studies for technical and scientific accuracy to ensure the assessments are of the highest quality. Many of our reports, too, are selected at random for additional internal reviews, as was the case for our reviews of artificial intelligence oversight and nuclear microreactors. In each case, a team of internal specialists reevaluated our work to ensure that it met GAO’s quality standards and answered substantively the questions it set out to address.
In addition to involving external experts in any one report, we have established a body of cross-sectoral, interdisciplinary science and technology policy experts to advise our team on emergent and emerging issues facing Congress and the nation. Beginning in the fall of 2020, this nonvoting advisory council will meet annually to discuss emerging trends and to ensure that we are providing relevant, fact-based, nonpartisan foresight, insight, and oversight on key issues and related policy implications.
Innovating to assess innovation
As the GAO works to keep Congress informed on the latest developments in science and technology, we are also applying the latest technological advances to improve our own research and analysis. One of the leaders in this effort is the Innovation Lab, an independent arm of STAA established to experiment on the use of advanced analytics and emerging technologies. For example, the lab is now examining congressional remote voting technology, both how an online system might function and, perhaps more importantly, what opportunities and challenges these tools pose to the security and veracity of the congressional voting process. And the lab is leading an effort to track real-time COVID-19 data to provide Congress with responsive, interactive visualizations on the pandemic’s spread to rapidly understand new developments. In each case, close study of the data and of emerging technology shapes the lab’s findings and recommendations.
Our science and technology expertise, taken together with our wealth of research into the pandemic to date, uniquely positions us to help Congress address its fundamental questions.
In the months ahead, we plan to explore key aspects of the COVID-19 pandemic; namely, how technology is being used to track the spread of the disease and what vaccine development efforts promise for the future. Critical to this effort will be our review of Operation Warp Speed, the public-private partnership focused on developing vaccines, therapeutics, and diagnostics to counter COVID-19, chiefly to gauge how effectively it is using new technology to facilitate and accelerate the development of one or more vaccines. Our science and technology expertise, taken together with our wealth of research into the pandemic to date, uniquely positions us to help Congress address its fundamental questions: How is science and technology being used to develop a COVID-19 vaccine? What opportunities and challenges do these new tools present? What role will new technology play in evaluating new vaccines and, critically, in assuring that they are distributed fairly?
The science and technology questions surrounding the COVID-19 pandemic, like many of the topics we explore, are vast. It is only by studying these areas, by rigorously reviewing our work, and by anticipating new developments, that we can understand these shifts, respond to unforeseen events, and chart a new path.
This hinges, critically, on a sound understanding of the field and of the innovations afoot. We believe that we have the experience and the expertise to not only understand the shifts in the science and technology landscape but also to anticipate the developments to come.
If the COVID-19 pandemic has brought anything to light, it is that science and technology have never been more critical in understanding and responding to the developments around us, shifts that shape our lives and that determine our future.
What the STAA team’s mandate was a year and a half ago is even clearer now. Science and technology developments and impacts are in no short supply, and the effective steering of those innovations will take sustained attention from policy-makers and the oversight community. With critical help from the team, the GAO is better positioned than ever to provide that oversight both to understand where the nation now stands and, more importantly, where it is going.