Histories of an Innovation Icon

By Richard Van Atta

Sharon Weinberger’s new book, The Imagineers of War, provides a history of the Defense Advanced Research Projects Agency (DARPA), a technology research and development (R&D) agency at the Department of Defense. Her book is remarkably similar in structure and coverage to one published 18 months earlier by Annie Jacobsen, titled The Pentagon’s Brain. Both books dwell on DARPA’s early history, particularly the agency’s founding after the Soviet Union’s launch of Sputnik and the agency’s decadelong involvement in Vietnam. And both omit or skim some critical periods in DARPA’s past that are necessary for understanding the organization’s role in innovation today.

DARPA was founded in 1958. (At that time it was called ARPA—“Defense” wasn’t added until 1972; for consistency, I will refer to the agency as DARPA.) The late 1950s were a time of enormous concern about the possibility of thermonuclear Armageddon between the United States and the Soviet Union, as well as about the threat the Soviet’s conventional military posed to Europe. President Dwight Eisenhower feared that the United States was falling behind the Soviets. These concerns were exacerbated when the Soviets launched a small satellite—Sputnik—that, though itself harmless, signaled We can hit you. The US military technology development apparatus looked unresponsive. Eisenhower was exasperated with inter-service rivalries that thwarted effective R&D, particularly into new realms such as outer space, which no military service could legitimately call its own. He thus was open to an alternative approach and agreed with his advisers to create DARPA as a new agency reporting directly to the secretary of defense.

The origins of DARPA are better depicted by Weinberger. It was founded with three presidential initiatives: to get the United States into space, to detect Soviet nuclear tests, and to develop missile defenses. Eisenhower had made it clear that space was to be the realm of a civilian agency, what became the National Aeronautics and Space Administration (NASA). Herbert York, who was charged with overseeing the Defense Department’s technology strategy as the first Director of Defense Research and Engineering, saw DARPA’s job as initiating space programs until NASA was established, whereas DARPA’s first director, Roy Johnson, sought to maintain a space role for DARPA.

When the space program transitioned to NASA, DARPA’s main remaining projects were the Vela programs, focused on nuclear test detection, and the Defender missile defense program. Weinberger describes the Vela Uniform program on seismic detection of Soviet nuclear tests, noting that this revolutionized the field of seismology. Importantly, Vela provided the technical means to support the limited nuclear test ban and facilitated the US and later Soviet moratoriums on underground nuclear testing. The Vela Hotel satellite program, which was first launched in 1963 and proved successful almost immediately in detecting above-ground nuclear explosions, is passingly mentioned by Weinberger. Despite Vela and Defender representing the preponderance of DARPA’s programs in the 1960s, Jacobsen gives them scant mention.

Both books describe the earliest missile defense program, Operation Argus, in some detail. Based on the theory of Nicholas Christofilos, who worked for York at the Lawrence Radiation Laboratory, the notion was to create a radiation belt in the extreme upper regions of the Earth’s atmosphere that would disable incoming missiles. The idea of exploding numerous nuclear weapons on the fringes of space to test the “Christofilos effect” seems outlandish today, but the fact that this project was pursued illuminates the national security environment of those times and DARPA’s willingness to undertake risky projects.

Weinberger notes that the Defender missile defense program, though consuming half of DARPA’s budget in the 1960s, was seen as a “god-awful mess” that involved “loony” ideas. The agency’s problem was coming up with approaches that were significantly better than current ones, but not completely unrealistic. Indeed, out of this came DARPA programs such as Arpat, a complicated missile interception scheme that the head of the program himself labeled only “kind of nutty.” These programs, after expending tens of millions of dollars, provided little technological payoff—despite generally being considered good science.

Both authors spend the bulk of their books on Project Agile, a highly classified counterinsurgency program deployed in Vietnam and, eventually, other countries in Southeast Asia. The program began in 1961 under William Godel, deputy director of DARPA, whose role in the early years is little understood and not well-documented in the agency’s archives. (However, much of the history of Godel and Agile was presented in detail in The Advanced Research Projects Agency, 1958-1974, by Richard J. Barber Associates, published in 1975.)

Each of these books presents compelling information about Godel’s activities, indicating a program run amok and with little oversight. Out of Project Agile came such debacles as the “strategic hamlets” program of rural pacification; fallacious assessments of the Vietnamese people under the guise of social science; and the egregious use of chemical defoliants, particularly the infamous Agent Orange. In retrospect, much of Agile was naïve, poorly managed, and rife with amateurism and even corruption. As Weinberger puts it, Godel “was running the Agile office as his own covert operations shop.”

As a fascinating and excruciating example of errant public policy, these deep dives into the Godel episode are astounding. They portray an overzealous, misguided operator who hijacked a technological agency to perpetrate an outlandish and failed program of social engineering at a massive scale. The episode illuminates the fact that there existed at least two DARPAs with little in common: the “strategic” DARPA, pursuing missile defense and nuclear test detection; and the “operational” DARPA, in which programs such as Agile attempted to bring technology into a combat zone. The latter was hardly scientific and raised, as Weinberger states, a battle over competing visions over the agency’s future. Neither book draws out the relevant lessons, however.

Given DARPA’s fundamental impact on what was to become computer science, both authors give information technologies short shrift—perhaps mindful that this history has been well told in other books, most notably Mitchell Waldrop’s The Dream Machine (2001). Moreover, both Weinberger and Jacobsen focus narrowly on the ARPANET, the precursor to today’s Internet, giving scant attention to the broader, increasingly coherent program begun as the Information Processing Techniques Office under J. C. R. Licklider. His concept of “man-computer symbiosis” is not captured in either book. Waldrop and others have elucidated how DARPA fostered a multipronged development of the technologies underlying the transformation of information processing from clunky, inaccessible machines to the ubiquitous network of interactive and personal computing capabilities. These two histories offer little on this transformation—perhaps the most significant of DARPA’s impacts—which continues today in DARPA’s pursuit of artificial intelligence, robotics, and cognitive computing.

Weinberger documents how DARPA was a troubled agency in the late 1960s and early 1970s, a victim of the Vietnam malaise and resource cutbacks that affected all of the Defense Department, as well as the fact that the Defender and Vela programs had essentially run their course. In 1965, Deputy Secretary of Defense Cyrus Vance advocated abolishing the agency. Crucial to understanding how DARPA evolved organizationally and programmatically from this are the actions of John S. Foster, who became the Defense Department’s Director of Defense Research and Engineering in 1965 and remained for eight years. Weinberger gives this crucial period some mention, but fast-forwards to the post-1970 resolution in which DARPA jettisoned the Agile program and moved Defender to the Army. There is no mention of this history by Jacobsen.

Foster was unhappy with how Agile was being conducted and dismayed with DARPA’s “toleration of an ‘academic’ atmosphere, undue pretensions to independence … [and] management deficiencies,” as the 1975 Barber report noted. Foster forced major changes and became deeply involved in DARPA programs and budgets. A new DARPA director appointed in 1967, Eberhart Rechtin, saw himself as Foster’s man, assigned to clean up and provide institutional discipline while reinvigorating DARPA by emphasizing the transition of successful projects to the military. He saw DARPA as a research agency—not development—that took risks that the military services would not, and he sought new directions with new programs. Reshaping DARPA continued under Rechtin’s successor, Steven Lukasik. Whereas Rechtin was relatively supportive of Agile but disenchanted with its focus and results, Lukasik saw it as “an embarrassment” and closed it down.

Weinberger capably describes DARPA’s post-Vietnam transition, prompted by the refocusing of the White House and Office of the Secretary of Defense on the Soviet threat. This entailed searching for new alternatives to the use of tactical nuclear weapons to defend Europe against Soviet attack. DARPA played a lead role in supporting studies on how to respond to such an attack and developed the underlying capabilities required to achieve these new alternatives. Weinberger succinctly presents the “systems of systems” concept for countering the Soviet Union’s numerical superiority with advanced technology, what Under Secretary of Defense William J. Perry and Secretary of Defense Harold Brown subsequently would call the “Offset Strategy.”

To implement this concept, during the mid-1970s into the 1980s DARPA launched transformative programs on stealth technologies, standoff precision strike capabilities (the ability to accurately destroy targets from a distance), and tactical surveillance via unmanned aerial vehicles. Weinberger provides a crisp chapter on the agency’s Have Blue stealth program, including the role that Malcolm Currie, Foster’s successor as the Director of Defense Research and Engineering, played in persuading the Air Force to commit to a prototype program. Missing from her account, though, is the transition from this “proof of principle” prototype to the F-117A, which was accomplished in just four years by Lockheed and the Air Force, with extraordinary top-level management oversight by Under Secretary Perry.

Weinberger gives much briefer mention to DARPA’s efforts to develop unmanned aerial vehicles—a mere two pages—and there is no mention of the Assault Breaker program to develop standoff precision strike capabilities. These are major omissions, as these were large programs with significant consequences for future defense technology capabilities. Jacobsen covers these programs even less, briefly describing stealth and then jumping to the F-117A stealth aircraft and JSTARS surveillance plane used in the first Gulf War more than a decade later. There is a brief mention of Assault Breaker, but no discussion of the program itself, and a few words mentioning the tactical unmanned aerial vehicles, but nothing on how they were developed by and transitioned from DARPA. Given that these programs are often touted as evidence of DARPA’s impact in transforming tactical warfare, this lack of treatment is baffling.

In both books, it is as if much of the 1990s did not exist. In the decade following the end of the Cold War, DARPA struggled to redefine itself and its programs. Moreover, the United States was in a budget crisis due, in part, to the vast defense spending of the 1980s. At the time, the White House, the Department of Defense, and DARPA were creating highly innovative and controversial programs aimed at bringing defense technology to bear on national economic competitiveness. This era of “dual-use” programs was a major redirection of DARPA under then-Secretary of Defense Perry—and was highly contentious due in part to questions about government investment in R&D that some contended could more effectively be performed by private industry. The midterm elections in 1994 produced a Republican majority in Congress that set out to end dual-use programs, creating a crisis of direction for DARPA during the remaining six years of the Clinton administration. None of this is covered in either book.

Another major gap in both books is the lack of discussion of the Future Combat Systems program, which DARPA conducted in partnership with the Army starting in the late 1990s. Under this program, tanks were to be replaced with a networked system of distributed robots and sensors—but the program was a massively expensive debacle. It was cancelled in 2009 by Secretary of Defense William Gates, after over $18 billion had been expended with nothing to show for it. Weinberger acknowledges this in a single sentence. There is no mention of the program by Jacobsen.

For most of the 2000s, the DARPA director was Anthony Tether, formerly director of the agency’s Strategic Technology Office from 1982 to 1986. The terror attacks of September 11 occurred within months of Tether becoming director, enmeshing DARPA in the “War on Terror.” The Total Information Awareness program, run by former Admiral John Poindexter (a controversial selection due to his role in the Iran Contra affair), is covered in both books. These tell the story of Tether and Poindexter making serious misjudgments that resulted in public outcry over the program on the grounds of its potential for invading privacy. The controversy led to Poindexter’s resignation and the program’s termination. But as both books document, the technologies that DARPA developed for deep data-mining were transferred to the intelligence agencies—particularly the National Security Agency.

Weinberger then segues to DARPA’s Grand Challenge, an incentive prize competition for demonstrating autonomous self-driving automobile technology. In the first Challenge, no vehicles came even close to completing the off-road course. Based on new technology and experience from the first race, five vehicles finished the second Challenge, pushing autonomous vehicles closer to reality. These and subsequent Challenges were successful in creating interest and incentivizing teams of researchers, often university-based, to demonstrate integrated capabilities. Importantly, the Challenges built on DARPA’s technology-focused programs in robotics, sensing, autonomy, communications, and energy storage as underlying enabling technologies.

Neither book provides much coverage of DARPA’s support for the wars in Iraq and Afghanistan under Tether. Weinberger does discuss the Nexus 7 data-analysis effort started by Tether’s successor, Regina Dugan. This program embedded social science “insurgency experts” and information technology scientists working on crowd-sourcing and social networking into Afghanistan—the first time since Vietnam that DARPA ventured into an operational area. Nexus 7 sought to use “reality mining” for “computational counterinsurgency.” After a period of months with more than 100 personnel in and out of country, the results of Nexus 7 in Afghanistan are debatable: although arguably a technical success, its operational impacts were limited. DARPA comes out of this recent experience facing questions similar to those raised by its earlier engagement in Vietnam: Should the agency develop social science and information technology for “population-centric” counterinsurgency? Should it focus on such immediate wartime efforts at all, or should its role remain long-term research of the kind that in the past has produced game-changing technologies? These types of questions are barely raised in either book.

Jacobsen ends The Pentagon’s Brain with roughly a hundred pages that largely veer off from DARPA’s history. She extensively discusses the improvised explosive devices that slaughtered US troops in Iraq and Afghanistan, but DARPA’s role in dealing with these was marginal. From there, she forays into technology and security issues only loosely connected to DARPA, including an aside on terror attacks and several pages on the Human Terrain System (a failed Army social science-based counterinsurgency program that had no DARPA involvement). Her final 50 pages are essentially speculation as to what the agency may currently be working on and what technologies it might advance in the future in areas such as robotics, autonomy, artificial intelligence, human-machine systems, and biological regeneration. She posits that these technologies may be leading to a new adversary—the autonomous “killer robot”—and leaves it at that.

Weinberger finishes The Imagineers of War by asking whether DARPA has devolved into a narrowly focused technology development agency, as opposed to one that takes on “truly high risk” projects. She ends with a vignette about the agency’s neuroscience programs aimed to modulate the human brain with neural implants to treat illness or injury. She concludes that its “neuroscience work could transform the world by revolutionizing medicine, and it could lead to weapons that change the way we fight future wars. Whether that world will be a better place is unclear.”

In sum, neither book is a comprehensive history of DARPA. Rather, they illuminate selective aspects of the agency’s past while leaving out crucial aspects of its history. Both lack a clear organizing principle, though Weinberger’s treatment is more thorough and carefully documented, based on numerous interviews, including most of the DARPA directors and a large number of office directors and program managers.

DARPA has dramatically affected many areas of defense capabilities. It has also produced much broader, revolutionary economic and societal advances in information technologies, microelectronics, materials, and other areas. Although Agile and its offshoots—which take up disproportionately large portions of both books—were ill-founded failures, they represented but one element of DARPA, and one that became decreasingly relevant over time to how the agency was defined and what it did. Largely missing from both books is what DARPA did over its 60-year existence to garner its reputation as an innovation icon. Missing is any discussion of its seminal technology work in microelectronics, cognitive computing, and synthetic biology. A more coherent and complete treatment that elucidates how the agency changed over time in various historical and organizational contexts is crucial to understanding today’s DARPA and its ongoing mission to demonstrate what the future could be.