The democratization of intelligence is changing how espionage is done – and by whom. How can states best achieve their aims in an age of silo-spotting, open source sleuths and other “intelligence auxiliaries”?
EXPERT PERSPECTIVE — Late in the summer of 2021, researchers affiliated with the Federation of American Scientists (FAS) discovered evidence that China was upgrading its ballistic missile silo count in what they described as “the most significant expansion of the Chinese nuclear arsenal ever.” To reach this conclusion, the analysts relied on open source intelligence (OSINT) techniques, including commercial satellite imagery and unclassified knowledge about Chinese missile technology, that has become commonplace in recent years.
FAS’ research did not go unnoticed by the US government. In response to the findings, Admiral Charles Richard of USSTRATCOM used a press conference to rhetorically ask the researchers, “If you enjoy looking at commercial satellite imagery or stuff in China, can I suggest you keep looking?"
Admiral Richard’s deadpan aside, his comment raises interesting questions about the future of public-private partnerships in intelligence. What happens when sophisticated intelligence capabilities exist outside of government? What intelligence functions should only be performed by states – and which should be done by non-state actors? And most pressingly, what tools can we give leaders like Admiral Richard so that they can stop “suggesting” and start working effectively with outside groups to achieve common policy objectives?
Since Admiral Richard’s comment, we have seen yet more examples of how the “democratization of intelligence” is creating new opportunities for private sector and NGO groups to provide insight on important geopolitical questions. Today, as Russia marshals its forces to widen its existing invasion of Ukraine, OSINT practitioners are continually scanning for new troop movements, giving policymakers ways to discuss publicly what once would have been secret intelligence. More broadly, across government and the think tank world, there has been widespread recognition of the increasing sophistication of open source intelligence capabilities. Most recently, we have also seen intelligence leaders, including most prominently MI6 Chief Richard Moore, comment on the exciting possibilities of public-private partnerships founded in part on open source techniques.
But so far, we have seen few new ways of working designed to actually capture this potential. Typically, the conversation about open source centers on how government can do more – rather than on how the public sector can forge innovative partnerships. As a result, national security professionals who wish to work with a burgeoning non-state intelligence sector are often left with unclear guidance. There has been plenty of work on integrating OSINT tools into state arsenals; but less toward creating the new policies, procedures, and ways of working needed to actually leverage non-state OSINT capability.
This is a missed opportunity. The IC should articulate new ways of working that effectively leverage the activities of non-state “intelligence auxiliaries” to help achieve national priorities. Such auxiliaries, whether they are directly tasked, paid, or integrated into the IC’s structure or not, have proven that they can make vital contributions to important tasks in mission awareness and information operations.
Leveraging intelligence auxiliaries is not without risk, and should be engaged in with care and according to a transparent and understandable framework. But intelligence auxiliaries are not going away – the state’s option is not whether to engage with them, but instead how.
Intelligence Auxiliaries in History
Intelligence Auxiliaries are not new – non-state intelligence activity has been a part of the world’s “second oldest profession” since the beginning.
Historically, the idea that non-state actors should work on the frontlines of intelligence would hardly have seemed out of place. Private individuals in ancient Rome were prolific practitioners of intelligence. European religious orders were famous collectors of intelligence – a long-standing tradition that continues today. Journalists have long been adjuncts, wittingly and unwittingly, to intelligence work - Napoleon was a famous consumer of British newspapers for information on troop movements, relying on them more than secret intelligence.
Moreover, states often collaborated with non-state intelligence gathering networks to supplement their own awareness. In the ancient Aztec empire, intelligence networks organized by merchants were often leveraged by the state. In the age of European expansion, commercial actors regularly engaged in espionage for private as well as state gain – with European commercial actors famously stealing the techniques for processing tea leaves and for making porcelain from China. The East India Company, itself a commercial actor, routinely made use of pre-existing espionage networks to inform its policy before the handover to the Crown in 1858.
For most of human history, states didn’t necessarily hold better espionage technology than did non-states – but they were still able to out-perform and effectively leverage non-state intelligence actors because, by virtue of being states, they possessed certain abilities that non-states could not hope to equal. These “enduring state characteristics” helped ensure that, even when non-states had superior information, states were still able to remain the most important players in the espionage market.
- States had control over the information environment. The state’s historical ability to publish, censor, or encourage the production of information shaped the intelligence-gathering environment. Historical censorship of the press during times of conflict, such as during Britain’s Boer War, shows how states have used control over the information environment to achieve objectives related to their own or foreign actors’ espionage. Today, China’s ability to control information leaving its borders is a continuation of the exercise of this power.
- States had agenda-setting ability. In times of peace or conflict, the state’s ability to guide non-state practitioners toward common outcomes was a major lever through which it exercised indirect control over non-state auxiliaries. Historically, mission-definition was an important way for states to guide the activities of private groups such as privateers, explorers, missionaries, and private trading corporations.
- States had fiat power. States could simply declare activities or types of information legal or illegal, and could engage in activities that they made illegal for others without fear of punishment. This historical ability to “live above the law”, whether through law courts or secret police, gave states powerful levers to compel the production of information.
- States had superior historical memory. One of the single most important advantages of historical intelligence bureaucracies was their ability to develop and sustain large archives. Archeological evidence of intelligence archives has attested to the importance that this state advantage had even in ancient times. Though less relevant today with the development of large-scale information storage in the private sector, state advantages in collecting and classifying information still persist today.
- States had market-making ability. States could incentivize or de-incentivize the production of information through their market-making power. Not only were states the largest players in information markets – they have had the ability to set the rules by which other players engage in market activity. This gave states the ability to offer bounties or create attractive commercial partnerships for information.
- Direct Contracting Ability. Finally, states could and did directly commission private groups to conduct intelligence work on their behalf. While not unique to states, this ability nonetheless is a major way that states have been able to leverage capabilities possessed by non-state actors throughout history.
The Rise and Fall of Information Overmatch
During the Cold War, states could rely on qualitatively better information than non-states. As this advantage recedes, “intelligence auxiliaries” are coming back on the scene.
The Cold War saw the technological gap between states and non-state “intelligence auxiliaries” widen. During this period of rapid advances in technology, states added a new intelligence advantage vis a vis non-states – the simple ability to reliably access and exploit information that far outclassed in quantity and type that available to non-states. As this advantage erodes in the modern day, the difference between the classified and non-classified worlds is diminishing – and with it a way of working based on information overmatch.
Access The Us is Engaging in a Strategy to Share Intelligence on Russia more broadly. Is it Worth the Risk? This is content reserved for Subscriber+ Members
Upgrade your access to Subscriber+ today
Today’s premier intelligence bureaucracies were largely formed during the “long” 20th century. This was an era when states increasingly came to enjoy privileged access to intelligence based on capabilities that non-states could not match, at least not reliably. During the long 20th century, as war and intelligence-gathering became more mechanized and technological, militaries and intelligence agencies became increasingly invested in industrial development and production as a way of securing the necessary materials to win wars. With the private sector unlikely to support the cost of expensive investments in military and intelligence technology on its own, “the role of the state [became] vital because it was the state that provided the critical financial resources required to take embryonic technologies and develop them at a speed unlikely to be matched by the civilian market.”
This sponsorship gave rise to sophisticated intelligence technologies such as computers, satellites, maritime domain awareness tools, unmanned aerial vehicles, and more. With sponsorship came control, and for most of the long 20th century states were the only entities with access to sophisticated espionage technology. Groups outside of government were largely restricted to low-grade versions of the same technology, or to classic “first generation” OSINT sources such as media and grey literature collection. While some non-state groups had significant impact on events throughout the Cold War, top-tier intelligence activities were largely restricted to states.
This state monopoly on access to top-tier information began to break down at the beginning of the 21st century. As the price of computing continued to fall, the state’s role as the principal investor in military and intelligence technology became less important. As a result, private companies increasingly took the lead in creating, and funding, technologies that transformed the practice of intelligence, such as the internet, social media, and artificial intelligence.
In place of the government monopoly on espionage technology, today there is a boisterous bazaar of information and data vendors. These companies sell a wide variety of social media intelligence tools, earth observation capabilities, large-scale information storage and processing, mobile phone location data, global HUMINT platforms, and sophisticated telemetry intelligence capabilities. This private market has changed the game. Today, with enough money and focus, a small group of dedicated individuals can leverage private sector capability to rival a well-funded intelligence agency. For example, the following capabilities are all within easy, legal reach for any government or non-governmental organization (so long as the budget is right):
- An on-call satellite imagery service from providers such as Planet, Maxar, IceEye, or others. Today, non-government researchers can access reams of satellite imagery on any area in the world, often at time increments of less than 24 hours for refresh. This means that together with sophisticated, openly available image recognition algorithms, a small team can scour the desert for Chinese siloes – or monitor North Korea’s nuclear program, and study deforestation trends globally.
- A flexible HUMINT capability that can source insights from anywhere on the globe. While journalism could always have been considered “OSINT-enabled HUMINT,” today a range of social media intelligence tools or distributed online survey platforms that allow a user to query “sources” all over the globe. Finally, natural language processing platforms allow anyone to conduct a sophisticated, global information-gathering operation completely out of the box.
- A mature SIGINT capability, from private firms whose satellites allow for the measurement of RF and microwave signals from emitters on earth.
- Finally, all of this is supported by a diverse, evolving, and multi-participant marketplace for both data and AI/ML capabilities, many of which are hungry for non-governmental researchers to show, in unclassified spaces, the power of their tools.
This only scratches the surface of the tools available. New low and no-cost OSINT platforms are consistently replicating capabilities once held only by top-tier spy agencies – for anyone to use. From NGOs to private companies to non-state terrorist groups and hacker collectives, the increasing sophistication and scope of OSINT capabilities has meant that states no longer have reliable information overmatch vis a vis non-states.
How to Work with Intelligence Auxiliaries
During the Cold War, it simply wasn’t possible for, say, the Bacardi corporation to charter U2 overflights of Cuba to provide snapshots of ongoing missile deployments, or for a group of disaffected Soviet emigres to conduct large-scale online surveys on food availability in supermarkets. Today, similar groups routinely use capabilities such as the tools highlighted above to produce impactful reporting on global issues of crime, corruption, and conflict around the world.
While profit is part of the equation, for many of these organizations, a dedication to mission is a key motivation. This new generation of intelligence auxiliaries combines the mission-motivation of a non-profit with the nimble structure of a startup and the technological reach of an intelligence agency. These groups rapidly collect, analyze, and disseminate research products to audiences both inside and outside of government. Because they often work almost entirely with publicly available information (PAI), such groups frequently have more flexibility than government bodies in whom they hire, how they work, what tools they use, and how they leverage their research, giving them the ability to move faster than any state actor.
States can leverage this energy by recognizing the existence of intelligence auxiliaries aligned with their goals, and looking at how they can enable these groups. Loosely speaking, state responses to the capability of intelligence auxiliaries can be arrayed on a spectrum control, borrowed from principles of agency law. On the “loosest” end of the spectrum, states can engage in Admiral Richard’s “opportunistic enlistment” of an intelligence auxiliary. He (presumably) was not aware of FAS’ research before it was published, learned about it in the news, and was pleased that it happened to align well with his mission priorities. He chose to amplify the research through independent, uncoordinated strategic communications. The defining feature of this model is a total absence of control and coordination.
On the “tightest” side of the spectrum, states can exercise a “direct control” style tasking of auxiliaries, as one might do with a traditional defense contractor. Under this model, a state actor directly contracts with an outside organization to conduct an activity on the state’s behalf, subject to conditions which the state imposes. Different degrees of control over working methods, personnel, timelines, and requirements may be imposed by the state under this model, but its defining feature is a high degree of control over methods and tools of work.
Between these two ends of the spectrum, however, are a variety of ways of interacting with intelligence auxiliaries. These ways of working aren’t based on information overmatch, but instead on long-term, enduring capabilities that states have had throughout history. A hypothetical spectrum of options based on these “enduring state functions” might look something like this:
- Opportunistic Enlistment of Intelligence Auxiliaries. Relying on the fact that intelligence auxiliaries pursue their activities independently from the State, states can simply observe their production and choose to amplify it when convenient, with little to no coordination between the two. This looks much like the case of Admiral Richard, or of other policymakers who seek to leverage emerging news stories.
- Using the Information Environment as Intelligence Terrain. States have the ability to define what information is open and what isn’t – and can do so with an eye to granting intelligence auxiliaries more access to mission-important information. For instance, the US has recently enacted new beneficial ownership laws, changing what information companies must provide when incorporating themselves. Information in public registers is a gold mine for intelligence auxiliaries working on anti-corruption issues; if states act to change laws with the idea that information disclosed will be used by intelligence auxiliaries for citizen OSINT policing, then they’re shaping the environment for good in a way that frees up state resources.
- Using Mission-Definition Power to Signal What is Important. Intelligence auxiliaries depend for funding and credibility on their ability to achieve important missions. There are many ways that policymakers can help guide intelligence auxiliaries toward important goals without revealing classified information or exercising direct control. This might take the form of an expanded campaign of academic outreach or of embedding select personnel within non-governmental intelligence auxiliary groups. In one example, the government could expand ongoing efforts to bring commercial expertise into mixed unclassified and classified spaces. Alternatively, states might create common information spaces virtually: instant messaging spaces could be created for trusted intelligence auxiliaries to join and share ongoing research and leads. Such groups could also serve as impromptu coordinating spaces for quick reaction OSINT monitors, as they are currently doing for Ukraine contingencies.
- Using Fiat Power to Empower Auxiliaries. States could simply declare activities or types of information legal or illegal depending on the identity of the actor. A simple example of this in action is how Bank Secrecy Act (BSA) Section 314(b) gives financial institutions the ability to share sensitive information with one another “in order to identify and report activities that may involve terrorist activity or money laundering.” An expanded version of this law could expand 314(b)’s safe harbor to chosen intelligence auxiliaries, giving banks a way to collaborate with trusted outside experts to better combat money laundering.
- Using Historical Memory to Feed Private Efforts. States have significant troves of information, both classified and unclassified, that they often find difficult to truly leverage. If more of these resources are given to non-states, the burden on states to create value from data can be more widely shared. This can (but doesn’t have to) mean selective declassification. Government departments such as Commerce and CBP have significant amounts of unclassified trade and corporate data that could help identify human traffickers, proliferators, and other bad actors.
- Using Market-Making Ability to Create Information Marketplaces. States have significant ability to create markets for information, both as participants and as rule-setters. They can use this power to incentivize groups to work together toward state goals. States can both signal priorities and make market connections among non-state actors by hosting short term surges or “hackathons” designed to temporarily gather expertise and tooling to answer an important question. Governments could incentivize private sector tech providers to make “in kind” donations of capability for set periods of time to boost intelligence auxiliaries, with the results of a short-term surge going to support policymaker awareness or strategic communications campaigns. Hackathons often offer more direct control than do similar “open data” initiatives, making them more attractive for sensitive missions.
- Using Commissioning Power to Task Directly. Finally, states can pursue familiar, tried and true models of direct tasking. This model can be effective in certain circumstances, but often is slow and laborious to implement.
The above are not new ways of working – in nearly every case, there are examples of government leveraging similar models to achieve important goals. But thinking of intelligence auxiliaries and the ways of working with them together as part of a single toolkit can help clarify the operational and legal issues at play. Rather than try to be overly prescriptive with how states can engage with non-state intelligence auxiliaries, policymakers should focus on creating a playbook of workable collaboration frameworks that can be relied on by intelligence professionals at different levels in government to engage with outside organizations. Without such models, commanders are likely to improvise – which may lead to good outcomes in some cases and bad ones in others.
These are far from the only models possible – and which model is appropriate will depend on the specifics of a given situation. Just as there is no one-size fits all approach to a given intelligence question, so there will not be an appropriate universal model for collaboration between state actors and non-state intelligence auxiliaries.
Read also Open Source Intelligence and Uncovering Secrets Hidden in Plain Sight in The Cipher Brief
Not whether, but how
Ultimately, the method of interacting with an intelligence auxiliary will be situation-dependent. Should it be covert or overt? Paid or unpaid? Public or private? Short-term or long-term? Policymakers will have to negotiate these questions with reference to specific facts.
But they should not have to negotiate them without functioning models. Admiral Richard is far from the only policymaker who has identified an outside capability that he has no tools to use. We must give intelligence officials, policymakers, and commanders flexible, clear, and transparent ways of working with intelligence auxiliaries. A failure to define the rules of engagement will not deter interaction, but instead leave it less clear and more likely to lead to uncontrollable outcomes. Without clear rules, a government actor could collaborate with an intelligence auxiliary in ways that endanger civil liberties or even lives.
But doing nothing is also a strategy. Policymakers do not get to negotiate the existence of intelligence auxiliaries. Intelligence technology is likely to continue to develop outside the walls of government, and private groups will continue to leverage this technology either in pursuit of a self-defined charitable mission, or for private gain. For non-authoritarian countries with strong civil societies and robust data governance regimes, the rise of intelligence auxiliaries is likely to be a significant force-multiplier vis a vis authoritarian rivals. The question for states is not whether they engage with outside intelligence capability, but how.
Read additional expert perspectives on open source intelligence in The Cipher Brief
Read more expert-driven national security insights, perspective and analysis in The Cipher Brief
The Cipher Brief may receive a small commission for products purchased via links
