Improving Scientific Advice to Government

Congress and federal policymakers draw on independent expert panels for scientific and technical advice in addressing some of society’s most controversial and economically significant issues. It is imperative that these panels operate in the most productive manner possible. Yet recent “reforms” intended to increase the panels’ transparency and reduce the potential for bias and conflict of interest may be weakening the advisory process.

Concern centers particularly on two of the most important sources of advice: the National Research Council (NRC), which is part of the National Academy of Sciences (NAS), and the Science Advisory Board (SAB), which serves the Environmental Protection Agency (EPA). Both the NAS and SAB create panels to develop consensus opinions on some of the most controversial scientific topics of our time, testing science’s predictive tools and directly affecting the outcome of federal policies involving billions of dollars in private and public outlays. Interest groups acknowledge the crucial influence of the panels by focusing considerable resources on how NAS and SAB panels are populated and managed.

Beyond these similarities, considerable differences separate the NAS from the SAB. Since the NRC’s creation during the Wilson administration, the NAS has convened several thousand panels and last year alone published 280 consensus reports on almost every conceivable topic of policy-related science. The SAB dates only to 1978 and–a crucial difference–advises a single agency on topics of the EPA’s choosing via an executive committee appointed by the EPA administrator. In many respects, the SAB is simply a more evolved and “institutionalized” federal advisory panel. It is fair to say that by comparison to the NAS, the SAB is much more beholden to its creating sponsor, vulnerable to influence by agency staff scientists, and far less robust in providing for accountability for the quality of its final reports.

In recent years, both the NRC and the SAB processes for rendering advice have undergone change in response to criticism, largely from activist nongovernmental organizations, regarding their supposed lack of transparency and the potential for bias and conflict of interest of panel members. NRC reforms were codified in 1997 in amendments to the Federal Advisory Committee Act, which were stimulated by a federal circuit court decision that animal rights activists won against the academy. The SAB adopted elaborate guidelines in the wake of a 2001 General Accounting Office report that was highly critical of SAB’s conflict-of-interest practices.

My perceptions of the undesirable side effects of these changes are impressionistic. No survey research has been done; no data have been collected. Yet I do have 30 years of experience serving on NAS committees, boards, commissions, and panels, and I follow the SAB in my law practice. Further, I have shared my ideas with a number of key participants in the NAS and SAB processes. They acknowledged the tendencies I describe, gave confidential examples from their experience, and urged me to share my views for discussion.

At the heart of the matter, the processes employed by the NAS and the SAB in forming advisory panels and the processes by which the panels write their reports and the conveners review those reports, have become much more formal and proceduralized. These processes are now more open to stakeholder input and influence, whereas the panels themselves have been constituted to be more neutral on the issues and more balanced in interests represented. As a result, panel meetings increasingly resemble public hearings before neutral science judges.

These developments may attract praise in a democratic society that highly values procedural due process and fairness. But, on balance, their unintended consequences may well end up diluting the quality of NAS and SAB scientific advice. Many scientists now look on with increasing concern, and some of them have become reluctant either to serve on panels or to recommend others to serve.

One concern centers on privacy, which, in turn, is part of a wider issue regarding injury to reputation and career from unwarranted disclosures and advocacy efforts. For example, new SAB rules require panel candidates to disclose all of their financial, professional, and personal ties–and to do so as well for their spouses and dependent children. Candidates are required to complete a lengthy questionnaire that includes listing assets, affiliations, research projects, consultancies, liabilities, and compensated expert testimony. The rules also call for a period of open public commentary after “biosketches” of potential panel members are released. Critics now argue that all this does not go nearly far enough.

Many scientists worry that efforts to make panel deliberations as transparent as possible mean that essentially all meetings will be conducted in a public fishbowl with the attendant risks of stakeholder “grandstanding” and opportunities for interest-group pressure on individual members who speak their minds openly. The NAS mandate, for example, now requires a 20-day period for public comment on committee slates, advance public notice of committee meetings and agendas, a public access file for all materials provided by anyone to committees, a publicly available summary of the closed sessions, and after-the-fact public disclosure of the names of peer reviewers.

There is concern, too, about the increased pressure on conveners to scrub panels vigorously for conflict and bias. This sometimes leads to selection of panel members for mere neutrality or disinterest–or even selection of less qualified candidates in order to balance panel membership by ideology, gender, age, and geographic distribution. The NAS and the SAB now appear to believe that if they solicit nominees widely from stakeholders and professional groups, obtain detailed information about panel candidates, vet this information publicly with stakeholders, limit panel service to conflict-free survivors, “balance” final membership to counteract bias, and maximize the openness of deliberations, then they will obtain the best scientific advice that consensus panels can provide.

But these measures can discourage the best scientists from serving because of the paperwork involved, the public disclosure of personal information, the attention that any knowledgeable scientist almost inevitably will attract from one interest group or another, and the stigma attached to anyone passed over by conveners reluctant to risk a fight over a nominee. Panel members are generally volunteers. As much as they want to provide public service and garner the prestige of a membership on a blue-ribbon panel, the best and the brightest scientists have plenty to do at their workbenches, without running the gauntlet of panel service.

There also is a real risk that valuable technical expertise will be lost in the quest for neutrality. The problem often comes down to how conveners view the twin threats of conflict of interest and bias. The “black-letter law” of conflict and bias, much discussed in a large body of literature, comes down to this: Direct or apparent financial conflict disqualifies and may be illegal unless an explicit waiver for otherwise unobtainable expertise is secured. Bias does not ordinarily disqualify, but once disclosed should be balanced against the countervailing biases of other nominees. But this “settled” practice clashes with the reality that financial, professional, and ideological concerns are not clearly separable. Each candidate reflects a complex mix of experiences, judgments, and prejudices. What is more, the greater his or her expertise, the more likely that the candidate will appear to have at least some financial conflicts and biases, however mild, based on employment, personal wealth, prior publications, public statements, personal insights, and research agendas. No candidate is capable of a pure passion for dispassionate public service.

The NAS points out that stakeholders (and even potential nominees who are judged too conflicted or biased to serve) can present their views at one of the several open committee meetings now required by internal mandates or law. SAB panels also have made greater use of public open-mike sessions. But this loss of panel expertise, coupled with hearings to compensate, is not a commendable alternative to the more informal consensus panels, whatever their faults, that existed previously.

This is not to say screening for conflicts and biases is inappropriate. Acute and direct financial conflicts, and intensely advocated policy positions, create a strong potential for mischief. However well-intentioned, a scientist so burdened would be hard-pressed, and probably unlikely, to leave these behind upon entering the panel’s meeting room. Nor can even the most carefully managed waiver overcome this problem. No one can reasonably expect a scientist to function effectively under conditions that require a direct, clear choice between sacrificing scientific judgment and either defying an employer’s wishes or abandoning an intensely advocated public position.

Steps to improvement

As the bottom line, the NRC and the SAB already do a good job of assembling panels and seeking to create an environment that fosters collegiality, the highest professional standards, and reliable assessments of the science at issue. Still, certain corrections and additional measures do seem warranted to protect the integrity of consensus science advice to government.

Against the backdrop of recent reforms and the principles of democratic legitimization that drive them–due process, transparency, public participation, and impartiality–the recommendations made here may seem mildly reactionary. At the very least, they may echo a bygone era (if it ever existed) when society was less contentious and more trusting, when professional norms and peer pressure were stronger, and when scientists seemed more capable of impartial public service. But the key questions here are of degree rather than absolutes; tendencies rather than extremes.

• Conveners of consensus scientific panels should continue to vet candidates publicly, both for conflicts of interest and for passionate biases that would have to be balanced by selecting an equally passionate member of the opposite persuasion. Although this may increase the risk of a “hung” panel, the appearance of impartial, accurate scientific consensus is vitally necessary to federal policymakers and the public. The conveners might try to be less beholden to stakeholders, though, if a candidate has needed expertise. Also, conveners should limit the opportunity for public comment to the slate actually proposed to serve. Panel formation should not be a competition; no one’s reputation should have to suffer a public airing and rejection after volunteering to serve.
• The conveners’ primary concern should be populating panels with the best scientific and technical expertise available. Balancing interest groups has no place in panel formation; balancing disciplines does. The important thing is to ascertain in advance if a difference in scientific opinion exists that requires different viewpoints to be represented. Remember: These panels offer expert advice; they do not make policy.

In addition, a variety of nonscientists and near-scientists–for example, attorneys, ethicists, and political scientists–serve on NAS and other panels. Again, their expertise is crucial, but their policy views peripheral. Their presence is perhaps necessary in view of the expanding range of advice Congress and the agencies seek. But panels of scientists should listen very critically to the odd lawyer or philosopher in their midst, since balancing all types of relevant expertise is impractical. In the end, panels should pronounce on science as the scientists see it, not as advised by others. Further, conveners should resist taking on policy-laden topics best left to other venues. Science cannot assume the responsibility to objectively and “scientifically” resolve tangled policy issues in which science may play only a supporting role. Consensus science panels are but one part of the federal policy selection process. Economic, political, and equity considerations belong in final federal policies but not in science panel reports.

• Conveners should seek expertise despite the potential for conflict or bias. In casting a wide net for expertise, conveners will find that some of the best candidates have significant potential conflicts and biases. The most egregious of these should, of course, disqualify. But merely coming from industry or a nonprofit background should not disqualify a candidate. Indeed, industry and nonprofit groups are prime sources for nominees. In every instance, conveners should value expertise over the use to which it may have been put. This means that conveners must thoroughly satisfy themselves in advance that their nominees’ potential conflicts and biases can be managed by self-disclosure and questioning by other panelists.
• Some of the most controversial NRC and SAB panels involve the science that underpins health, environment, and safety regulation. Some public advocates have argued that because Congress has enacted precautionary legislation, and federal agencies have codified the precautionary principle in their risk-reduction rules and guidance, consensus panels of scientists must interpret and apply science to conform to these legal mandates. Yet it is not clear why science should be guided by these legal norms. Panels are not “science juries” rigidly bound by legislative instructions. They should be free to give their best scientific opinion on the technical merits of the bridging inferences and assumptions–that is, the “defaults” that enable science to support risk-averse regulatory standards despite significant scientific uncertainty.

The practices of science demand close attention to facts and data; empirical observation; rigorous logic; an explanatory chain of scientific causation, reproducibility, and predictability; and peer review. This is what scientists “do.” When asked to provide advice to government, this background fundamentally determines, and limits, what scientists may offer. Scientists want to be helpful, but they should not too willingly become handmaidens to regulators or to legislators who are all too eager to pass the policy buck by labeling an issue “scientific” when it really is not. Supreme Court Chief Justice John Marshall long ago wrote, “It is emphatically the province of the judicial department to say what the law is.” Risk analysts need to reassess the primacy of science in risk assessment. To paraphrase Marshall, it is the province of science to say what risk assessment is. EPA risk assessment and management admittedly require an accommodation between science and regulatory policymaking. But this accommodation should not be allowed to obscure the central role of science in determining what is scientifically plausible in risk assessment. Objective, neutral science is the necessary baseline from which precautionary risk management should proceed. Without this essential contribution from science, federal risk managers will be at sea as to what the best scientific estimate of risk is.

• Conveners should discourage, and possibly prohibit, outside one-on-one exchanges between panel members and groups with a stake in the outcome of the panel’s deliberations. As but one example of such meddling, an academic scientist serving on an SAB panel reviewing the EPA’s 2000 draft dioxin risk reassessment has reported that lobbying by environmental organizations, industry, and EPA staff scientists alike was intense and contributed to a sense that the deliberative panel meetings were mere window dressing.
• Prospective panelists should make an explicit and perhaps written commitment to listen with an open mind to the discussion and participate fairly and impartially, supplying facts and opinion without regard to employment or prior positions taken on the issues. In the event of violation, no legal obligation would attach; rather, the sanction would be embarrassment in the scientific community.
• The best antidote to undue influence caused by financial conflict or personal bias is full understanding by copanelists. A confidential discussion among copanelists regarding bias and conflict (preceded by “ground-truthing” by the conveners) serves to alert copanelists to background and perspective that may shape a panelist’s contribution to the consensus effort. The NAS has long used such discussions and proven their worth.

Disclosure rather than disqualification is realistic and honest: The best scientists with the most to contribute will have a rich background of professional and public experience, with at least some type of financial stake and a record of expert opinion in the scientific and public literature. In this way, panelists are implicitly burdened to critically evaluate their copanelists’ contributions. This is more realistic than falsely assuring them that the conveners alone are responsible because they have conducted an inquiry and found the panel to be conflict-free and bias-balanced. Disclosure rather than disqualification makes very clear that face-to-face “peer review” and peer pressure are expected to produce consensus on a science-based report. Disclosure works best in concert with the personal commitment to neutrality and an emphasis on populating committees with strong expertise that will correct weak (that is, biased or conflicted) argumentation in panel discussions.

• The practice of conducting open public meetings and presentations should, of course, continue. The problem is, there are too many of them, and they deflect attention away from the obligation of the conveners and panelists to obtain the best expertise for the panel itself, to deliberate, and to write a consensus report. Holding more confidential discussions, and fewer public sessions, permits a panel with limited time and budget to work more with each other and “bond” to lay a foundation for consensus. Confidential discussions shield panelists from stakeholder oversight, enable scientists to shed public positions, minimize opportunities for stakeholder grandstanding, and–most important of all–give peer pressure more leeway to function effectively. Coupled with limits on outside contacts, confidentiality can play a far more constructive role than any proponent of the current reforms has been willing to acknowledge.
• In the end, the panel’s report is what counts. It should be carefully drafted by the panel itself (a condition of membership) or, on occasion, by staff under tight instructions and panel supervision. This practice already is the hallmark of the finest NAS reports. As with a judicial opinion, a panel report should rest on an underlying rationale that speaks for itself and can withstand the most withering critical reviews. Dissenting or supplementary views detract and may represent a last opportunity for stakeholder grandstanding. In consensus reports, anonymity lends strength, as the NAS has long realized. The NAS places a high premium on dissent-free final reports, and its multilayer approval process labors to reconcile views within a report’s consensus framework.
• Panel reports should be independently reviewed and independently refereed. The NRC Report Review Committee (RRC) supervises the peer review of draft panel reports. More important, the RRC decides when the panel has adequately responded to its concerns–that is, the RRC decides when a panel report is of sufficient quality to be released publicly. The SAB has no such independently refereed peer review, although its executive committee plays some of this role. The value and seriousness of independently refereed peer review have been confirmed again and again at the NRC. Reports have been rewritten; major conclusions have been changed. The very existence of the RRC process exerts a discipline on NRC panels that appears to be lacking at the SAB.

These recommendations cannot cure all the ills of the current “reforms.” Nor should they be implemented inflexibly or in elaborate or formal guidelines, lest they fall into the same procedurally heavy trap into which current policies, particularly the SAB’s, have fallen. But if the NAS and the SAB are to avoid the public hearing and adversarial due process models more appropriate to the making of rules and policies, then the recommendations should receive consideration.