Governance of the Inconceivable
How do scientists and policymakers work together to design governance for technologies that come with evolving and unknown risks? In the Winter 1985 Issues, seven experts reflected on the possibility of a large nuclear conflict triggering a “nuclear winter.” These experts agreed that the consequences would be horrifying: even beyond radiation effects, for example, burning cities could put enough smoke in the atmosphere to block sunlight, lowering ground temperatures and threatening people, crops, and other living things. In the same issue, former astronaut and then senator John Glenn wrote about the prospects for several nuclear nonproliferation agreements he was involved in negotiating. This broad discussion of nuclear weapons governance in Issues—involving legislators Glenn and then senator Al Gore as well as scientists, Department of Defense officials, and weapons designers—reflected the discourse of the time. In the culture at large, fears of nuclear annihilation became ubiquitous, and today you can easily find danceable playlists containing “38 Essential ’80s Songs About Nuclear Anxiety.”
But with the end of the Cold War, the breakup of the Soviet Union, and the rapid growth of a globalized economy and culture, these conversations receded from public consciousness. Issues has not run an article on nuclear weapons since 2010, when an essay argued that exaggerated fear of nuclear weapons had led to poor policy decisions. “Albert Einstein memorably proclaimed that nuclear weapons ‘have changed everything except our way of thinking,’” wrote political scientist John Mueller. “But the weapons actually seem to have changed little except our way of thinking, as well as our ways of declaiming, gesticulating, deploying military forces, and spending lots of money.”
All these old conversations suddenly became relevant again as our editorial team worked on this issue. On February 27, when Vladimir Putin ordered Russia’s nuclear weapons put on “high alert” after invading Ukraine, United Nations Secretary-General Antonio Guterres declared that “the mere idea of a nuclear conflict is simply unconceivable.” But, in the space of a day, what had long seemed inconceivable was suddenly being very actively conceived.
In the space of a day, what had long seemed inconceivable was suddenly being very actively conceived.
As remarkable as it was that Americans were suddenly talking about nuclear conflict again, it was even more extraordinary that society had managed to generally ignore the idea for 30 years or so—even as most of the weapons in question lurked in their silos. Those years of blissful forgetting were not the result of changes in the technology itself, which continued to become more sophisticated, but in the governance system—all the official and unofficial connections that seemed to prevent catastrophe—that large numbers of Americans came to trust. The experience of this last month is a reminder that the role of governance is not only to prevent accidents and horrible calamities, but also to create the trust and transparency that allows society to adopt new technologies without being frozen by fear of their unknown outcomes.
The Spring 2022 Issues looks at governance for a broad slate of today’s new and yet-to-be-imagined technologies, including emerging health and medical technologies, bioengineering and biosecurity, remedies for the debris that is accumulating in space, and lethal autonomous weapons. Our contributors explore ways that policymakers, institutions, researchers, industry, and communities can work together to effectively govern emerging or unpredictable technologies, creating a more peaceful world where more people can thrive.
The role of governance is not only to prevent accidents and horrible calamities, but also to create the trust and transparency that allows society to adopt new technologies without being frozen by fear of their unknown outcomes.
Governance is essential to harness still-nascent technologies for social good. Consider brain-computer interfaces: in 2019, a German man who had been completely paralyzed for several years used a brain implant to communicate with his family, ask for beer, and tell his son that he loved him. But the history of discriminatory use of neurosurgical treatments, including lobotomies, suggests that an innovation promising such transformation at the individual level could also harm society’s most vulnerable. Debra Mathews, Rachel Fabi, and Anaeze Offodile present the work of a National Academy of Medicine (NAM) committee that set out to develop a methodological framework to enable policymakers to examine emerging medical technologies to “anticipate and address the social, legal, ethical, and governance issues they raise.” The framework they’ve developed weaves case studies, historical analysis, and scenario building into a uniquely panoptic vision of how the technology may evolve and how its social impacts could be felt. And by using the framework to get a bird’s-eye view across multiple technologies, NAM’s Committee on Emerging Science, Technology, and Innovation (CESTI) has delineated how governance institutions across sectors could implement principles reflecting the broader social goals of justice, fairness, autonomy, and individual and social good. The CESTI project shows the power of applying a rigorous ethical imagination to a sphere of innovation that has often been left to markets, the military, and regulators.
A similar set of ethical opportunities and challenges animates “Building A Bottom-Up Bioeconomy” by Philip Shapira and his coauthors. They envision using bioengineering to create “distributed governance systems that empower communities to tailor new approaches to their particular situations, creating a break with the industrial patterns and practices of the past.” In this telling, previous visions of the potential of engineering biology to create sustainable alternatives to fossil fuels and plastics fell short: by imagining massive, centralized economies of scale, they replicated the waste and injustices of the fossil fuel industry. To address those shortcomings, the authors suggest a new model with thousands of local refineries processing local feedstocks under local governance. However, any bioeconomy requires “deep social engagement,” they write, to prevent “repeating the mistakes of the past and reinforcing currently inadequate economic and ecological systems.”
Sam Weiss Evans argues for a congruent understanding of society’s role in biosecurity in his essay, “When All Research is Dual Use.” Current biosecurity paradigms that rely on researchers’ intentions and “guards, gates, and guns” are not sufficient for today’s complex biosecurity landscape. “Having special governance only for known threats makes little sense when new security concerns are likely to emerge from the unknown—whether from unintended consequences, natural evolution, or malicious use. What is needed instead is a curiosity-based system that attunes researchers, funders, and policymakers to attend to security throughout the research lifecycle.” Only by embracing the social context of biology can regulators begin to assure future biosecurity with democratic principles.
The CESTI project shows the power of applying a rigorous ethical imagination to a sphere of innovation that has often been left to markets, the military, and regulators.
While we were editing this issue, the bioinnovations Weiss writes about became ever more interwoven with society itself. On February 27, the very same day that nuclear war became “conceivable” again, a citizen biolab in New York started selling genetically engineered loops of DNA called plasmids on Etsy, the online marketplace better known for scented candles and decorative pillows. The plasmids, sold in small tubes for $20, could be inserted into E. coli cells to make them express different candy-colored pigments in a petri dish. The marketing of plasmids as a handicraft shows how rapidly do-it-yourself biology has spread, and it also suggests the silly and glorious ways that biotechnology, once out of the lab and into pop culture, may evolve. Although Etsy took the shop down a week later—thus turning the site into a de facto biotechnology governance institution—it’s not hard to imagine that someday our grandchildren will admire the primitive exuberance of “38 Essential DNA Sequences About BioAnxiety From the ’20s.”
Along with imagination and creativity, wider community involvement will be essential to the discussion of how biotechnology evolves. In California, a summer camp called BioJam brings teenagers, scientists, and artists together to learn the principles and practice of biodesign to create art and then share what they’ve learned with their communities. “Framing science as ancestral, playful, and creative can … inspire deep innovation, and empower more people to see themselves as shapers of the questions science asks and explores,” write Callie Rodgers Chappell, Rolando Perez, and Corinne Okada Takara in their essay “Bioengineering Everywhere, For Everyone.” And while BioJam is fun, it is also advancing a deeper agenda: as a former camper explains, creating durable rules and guidelines for gene editing requires a diversity of people and knowledge.
Beyond institutional and community involvement, durable governance also requires political leadership. Ronald Reagan and Margaret Thatcher acted surprisingly quickly on the threat of ozone depletion, write Stephen Garber and Lisa Ruth Rand in “A Montreal Protocol For Space Junk?” Perhaps it was education (Thatcher was a chemist) or self-interest (Reagan had skin cancer), but the ratification of the Montreal Protocol to phase out ozone-depleting chemicals demonstrates how effective global governance can be—leading the authors to suggest that something similar could be used for the growing problem of space debris. “The technical and legal challenges of dealing with space debris are daunting,” they write, “as is the complexity inherent in managing a global common resource.” Few people understand how much of daily life—ATM transactions and air travel, for example—requires satellites that could be harmed by orbital debris. “Whether or how the same ordinary citizens who feared the looming personal health risks of a thinned ozone layer could learn to take as seriously the myriad threats posed by failing satellite infrastructure remains a complicated question.”
The very same day that nuclear war became “conceivable” again, a citizen biolab in New York started selling genetically engineered loops of DNA called plasmids on Etsy, the online marketplace better known for scented candles and decorative pillows.
And this points to how narratives can influence governance. In his essay “Banning Lethal Autonomous Weapons: An Education,” Stuart Russell tells the story of how he first set off for Geneva in 2015, intending to give testimony that would move forward a global ban on lethal autonomous weapons systems. But as he encountered obstacles to the diplomatic process, he reconsidered media portrayals casting these evolving weapons as Terminator-style robots, which “misleads the public into thinking that autonomous weapons are science fiction.” In response, he and a group of artists and funders took up Einstein’s challenge to “change our ways of thinking” by producing Slaughterbots, a short film about autonomous weapons, which has now been seen by millions.
Bringing a feeling of security to a time of great techno-political anxiety is not an easy task. As the Winter 1985 Issues showed, governance for fearsome technological developments requires experts with the fortitude to imagine the worst alongside a vision of the best. And in society at large, all that conceiving—not to mention cathartic pop songs and chaotic declaiming and gesticulating—helps to create spaces where choices can be made. It’s clear that society now needs many more such broad frameworks to create new spaces for consideration to ameliorate our new anxieties as well as the old ones.
Institutions are essential to governance, and the stories in this issue show how government, science, philanthropy, civil society, and communities may forge new relationships around visions of a shared future. This issue’s interview with psychologist Daniel Kahneman, who won the Nobel Prize in Economics in 2002, explores how institutions, including the justice system, sometimes make bad decisions. Ironically, using artificial intelligence could potentially make some decisions fairer and more transparent, but as he told Issues editor Sara Frueh, humans prefer the natural over the artificial and are “strongly biased against algorithms.” And although algorithms are currently seen in well-earned dystopic terms, it’s possible that, with proper governance and new narratives, the public may someday see their use in certain situations as fair.
This issue’s interview with psychologist Daniel Kahneman, who won the Nobel Prize in Economics in 2002, explores how institutions, including the justice system, sometimes make bad decisions.
The many other articles in this issue delve into open science as well as Operation Warp Speed, explore the potential for philanthropy to champion engaged research, and explain the role of narrative in ensuring that science and technology policies take the needs of workers into account. Plus a fabulous and hard-to-characterize story about the electrification of Mexico City in the early twentieth century and a dissection of the ways that local policymakers desire—but do not get—expert advice.
And for a more literal dissection, a poem by Marianne Boruch and artwork by Lisa Nilsson—inspired by the time both artists spent in the cadaver labs where young medical doctors learn about the human body. Their rigorous application of observation, social awareness, and imaginative empathy create striking portraits of how knowledge itself is attained.