Corin R. Stone is a Scholar-in-Residence and Adjunct Professor at the Washington College of Law. Stone is on leave from the Office of the Director of National Intelligence (ODNI) where, until August 2020, she served as the Deputy Director of National Intelligence for Strategy & Engagement, leading Intelligence Community (IC) initiatives on artificial intelligence, among other key responsibilities. From 2014-2017, Ms. Stone served as the Executive Director of the National Security Agency (NSA). In 2005, Ms. Stone helped stand-up the ODNI and served in a number of key leadership roles, including the first Principal Deputy General Counsel, the first IC Information Sharing & Safeguarding Executive, and the Assistant DNI for Policy & Strategy, where she led the development of the 2014 National Intelligence Strategy and oversaw the formulation and implementation of IC‐wide policy and strategy on the full range of intelligence issues.
A Roadmap for AI in the Intelligence Community
(Editor’s Note: This article was first published by our friends at Just Security and is the fourth in a series that is diving into the foundational barriers to the broad integration of AI in the IC – culture, budget, acquisition, risk, and oversight. This article considers a new IC approach to risk management.)
OPINION — I have written previously that the Intelligence Community (IC) must rapidly advance its artificial intelligence (AI) capabilities to keep pace with our nation’s adversaries and continue to provide policymakers with accurate, timely, and exquisite insights. The good news is that there is strong bipartisan support for doing so. The not-so-good news is that the IC is not well-postured to move quickly and take the risks required to continue to outpace China and other strategic competitors over the next decade.
In addition to the practical budget and acquisition hurdles facing the IC, there is a strong cultural resistance to taking risks when not absolutely necessary. This is understandable given the life-and-death nature of intelligence work and the U.S. government’s imperative to wisely execute national security funds and activities. However, some risks related to innovative and cutting-edge technologies like AI are in fact necessary, and the risk of inaction – the costs of not pursuing AI capabilities – is greater than the risk of action.
The Need for a Risk Framework
For each incredible new invention, there are hundreds of brilliant ideas that have failed. To entrepreneurs and innovators, “failure” is not a bad word. Rather, failed ideas are often critical steps in the learning process that ultimately lead to a successful product; without those prior failed attempts, that final product might never be created. As former President of India A.P.J. Abdul Kalam once said, “FAIL” should really stand for “First Attempt In Learning.”
The U.S. government, however, is not Silicon Valley; it does not consider failure a useful part of any process, especially when it comes to national security activities and taxpayer dollars. Indeed, no one in the U.S. government wants to incur additional costs or delay or lose taxpayer dollars. But there is rarely a distinction made within the government between big failures, which may have a lasting, devastating, and even life-threatening impact, and small failures, which may be mere stumbling blocks with acceptable levels of impact that result in helpful course corrections.
The Cipher Brief hosts private briefings with the world’s most experienced national and global security experts. Become a member today.
As a subcommittee report of the House Permanent Select Committee on Intelligence (HPSCI) notes “[p]rogram failures are often met with harsh penalties and very public rebukes from Congress which often fails to appreciate that not all failures are the same. Especially with cutting-edge research in technologies … early failures are a near certainty …. In fact, failing fast and adapting quickly is a critical part of innovation.” There is a vital difference between an innovative project that fails and a failure to innovate. The former teaches us something we did not know before, while the latter is a national security risk.
Faced with congressional hearings, inspector general reports, performance evaluation downgrades, negative reputational effects, and even personal liability, IC officers are understandably risk-averse and prefer not to introduce any new risk. That is, of course, neither realistic nor the standard the IC meets today. The IC is constantly managing a multitude of operational risks – that its officers, sources, or methods will be exposed, that it will miss (or misinterpret) indications of an attack, or that it will otherwise fail to produce the intelligence policymakers need at the right time and place. Yet in the face of such serious risks, the IC proactively and aggressively pursues its mission. It recognizes that it must find effective ways to understand, mitigate, and make decisions around risk, and therefore it takes action to make sure potential ramifications are clear, appropriate, and accepted before any failure occurs. In short, the IC has long known that its operations cannot be paralyzed by a zero-risk tolerance that is neither desirable nor attainable. This recognition must also be applied to the ways in which the IC acquires, develops, and uses new technology.
This is particularly important in the context of AI. While AI has made amazing progress in recent years, the underlying technology, the algorithms and their application, are still evolving and the resulting capabilities, by design, will continue to learn and adapt. AI holds enormous promise to transform a variety of IC missions and tasks, but how and when these changes may occur is difficult to forecast and AI’s constant innovation will introduce uncertainty and mistakes. There will be unexpected breakthroughs, as well as failures in areas that initially seemed promising.
The IC must rethink its willingness to take risks in a field where change and failure is embraced as part of the key to future success. The IC must experiment and iterate its progress over time and shift from a culture that punishes even reasonable risk to one that embraces, mitigates, and owns it. This can only be done with a systematic, repeatable, and consistent approach to making risk-conscious decisions.
Today there is no cross-IC mechanism for thinking about risk, let alone for taking it. When considering new activities or approaches, each IC element manages risk through its own lens and mechanisms, if at all. Several individual IC elements have created internal risk assessment frameworks to help officers understand the risks of both action and inaction, and to navigate the decisions they are empowered to make depending upon the circumstances. These frameworks increase confidence that if an activity goes wrong, supervisors all the way up the chain will provide backing as long as the risk was reasonable, well-considered and understood, and the right leaders approved it. And while risk assessments are often not precise instruments of measurement – they reflect the quality of the data, the varied expertise of those conducting the assessments, and the subjective interpretation of the results – regularized and systematic risk assessments are nevertheless a key part of effective risk management and facilitate decision-making at all levels.
Go beyond the headlines with expert perspectives on today’s news with The Cipher Brief’s Daily Open-Source Podcast. Listen here or wherever you listen to podcasts.
Creating these individual frameworks is commendable and leading-edge for government agencies, but more must be done holistically across the IC. Irregular and inconsistent risk assessments among IC elements will not provide the comfort and certainty needed to drive an IC-wide cultural shift to taking risk. At the same time, the unique nature of the IC, comprised of 18 different elements, each with similar and overlapping, but not identical, missions, roles, authorities, threats and vulnerabilities, does not lend itself to a one-size-fits-all approach.
For this reason, the IC needs a flexible but common strategic framework for considering risk that can apply across the community, with each element having the ability to tailor that framework to its own mission space. Such an approach is not unlike how the community is managed in many areas today – with overarching IC-wide policy that is locally interpreted and implemented to fit the specific needs of each IC element. When it comes to risk, creating an umbrella IC-wide framework will significantly improve the workforce’s ability to understand acceptable risks and tradeoffs, produce comprehensible and comparable risk determinations across the IC, and provide policymakers the ability to anticipate and mitigate failure and unintended escalation.
Critical Elements of a Risk Framework
A common IC AI risk framework should inform and help prioritize decisions from acquisition or development, to deployment, to performance in a consistent way across the IC. To start, the IC should create common AI risk management principles, like its existing principles of transparency and AI ethics, that include clear and consistent definitions, thresholds, and standards. These principles should drive a repeatable risk assessment process that each IC element can tailor to its individual needs, and should promote policy, governance, and technological approaches that are aligned to risk management.
The successful implementation of this risk framework requires a multi-disciplinary approach involving leaders from across the organization, experts from all relevant functional areas, and managers who can ensure vigilance in implementation. A whole-of-activity methodology that includes technologists, collectors, analysts, innovators, security officers, acquisition officers, lawyers and more, is critical to ensuring a full 360-degree understanding of the opportunities, issues, risks, and potential consequences associated with a particular action, and to enabling the best-informed decision.
Given the many players involved, each IC element must strengthen internal processes to manage the potential disconnects that can lead to unintended risks and to create a culture that instills in every officer a responsibility to proactively consider risk at each stage of the activity. Internal governance should include an interdisciplinary Risk Management Council (RMC) made up of senior leaders from across the organization. The RMC should establish clear and consistent thresholds for when a risk assessment is required, recommended, or not needed given that resource constraints likely will not allow all of the broad and diverse AI activities within organizations to be assessed. These thresholds should be consistent with the IC risk management principles so that as IC elements work together on projects across the community, officers have similar understandings and expectations.
The risk framework itself should provide a common taxonomy and process to:
- Understand and identify potential failures, including the source, timeline, and range of effects.
- Analyze failures and risks by identifying internal vulnerabilities or predisposing conditions that could increase the likelihood of adverse impact.
- Evaluate the likelihood of failure, taking into consideration risks and vulnerabilities.
- Assess the severity of the potential impact, to include potential harm to organizational operations, assets, individuals, other organizations, or the nation.
- Consider whether the ultimate risk may be sufficiently mitigated or whether it should be transferred, avoided, or accepted.
AI-related risks may include, among other things, technology failure, biased data, adversarial attacks, supply chain compromises, human error, cost overruns, legal compliance challenges, or oversight issues.
An initial risk level is determined by considering the likelihood of a failure against the severity of the potential impact. For example, is there is a low, moderate, or high likelihood of supply chain compromise? Would such a compromise affect only one discrete system or are there system-wide implications? These calculations will result in an initial risk level. Then potential mitigation measures, such as additional policies, training, or security measures, are applied to lower the initial risk level to an adjusted risk level. For example, physically or logically segmenting an organization’s systems so that a compromise only touches one system would significantly decrease the risk level associated with that particular technology. The higher the likelihood of supply chain compromise, the lower the severity of its impact must be to offset the risk, and vice versa. Organizations should apply the Swiss Cheese Model of more than one preventative or mitigative action for a more effective layered defense. Organizations then must consider the adjusted risk level in relation to their tolerance for risk; how much risk (and potential consequence) is acceptable in pursuit of value? This requires defining the IC’s risk tolerance levels, within which IC elements may again define their own levels based upon their unique missions.
Understanding and considering the risk of action is an important step forward for the IC, but it is not the last step. Sometimes overlooked in risk assessment practices is the consideration of the risk of inaction. To fully evaluate potential options, decision-makers must consider whether the overall risk of doing something is outweighed by the risks of not doing it. If the IC does not pursue particular AI capabilities, what is the opportunity cost of that inaction? Any final determination about whether to take action must consider whether declining to act would cause greater risk of significant harm. While the answer will not always be yes, in the case of AI and emerging technology, it is a very realistic possibility.
And, finally, a risk framework only works if people know about it. Broad communication – about the existence of the framework, how to apply it, and expectations for doing so – is vital. We cannot hold people accountable for appropriately managing risk if we do not clearly and consistently communicate and help people use the structure and mechanisms for doing so.
Buy-in To Enhance Confidence
An IC-wide AI risk framework will help IC officers understand risks and determine when and how to take advantage of innovative emerging technologies like AI, increasing comfort with uncertainty and risk-taking in the pursuit of new capabilities. Such a risk framework will have even greater impact if it is accepted – explicitly or implicitly – by the IC’s congressional overseers. The final article in this series will delve more deeply into needed changes to further improve the crucial relationship between the IC and its congressional overseers. It will also provide a link to a full report that provides more detail on each aspect of the series, including a draft IC AI Risk Framework.
Although Congress is not formally bound by such a framework, given the significant accountability measures that often flow from these overseers, a meeting of the minds between the IC and its congressional overseers is critical. Indeed, these overseers should have awareness of and an informal ability to provide feedback into the framework as it is being developed. This level of transparency and partnership would lead to at least two important benefits: first, increased confidence in the framework by all; and second, better insight into IC decision-making for IC overseers.
Ultimately, such a mutual understanding would encourage exactly what the IC needs to truly take advantage of next-generation technology like AI: a culture of experimentation, innovation, and creativity that sees reasonable risk and failure as necessary steps to game-changing outcomes.
Read also AI and the IC: The Tangled Web of Budget and Acquisition
Read also Artificial Intelligence in the IC: Culture is Critical
Read also AI and the IC: The Challenges Ahead
Read more expert-driven national security insights, perspective and analysis in The Cipher Brief
