From Secrets to Sensors: Why Open Source Data Must Drive Modern Intelligence

THE BLUF / COLUMN — The Department of Defense is on a tear to revamp technology for warfighters. Secretary Hegseth signed an AI strategy on 9 January. Prior to that he called for an Enterprise Command and Control Program Office that would provide a real time battlespace picture for military commanders, something that has been talked about for years but never completed. The urgency is clear. The intelligence community must play a large part in this Department of Defense technology transformation if it wants to remain relevant to the warfighter and decision makers. Key for the intelligence community will be to fundamentally rethink its relationship with open-source data. To be most useful for the decision makers and operators, the intelligence community must rebuild itself on the backbone of open-source data and commercial technology.

Not only is data important but because of the volume and complexity of that data, so is the technology that can sift through the data. Public domain data must be the first place to gather trends and threat warnings that feed the many watch centers across the department of defense and intelligence community which should be the basis of the enterprise command and control picture. Much of the technology already exists for sifting through the data; fusing it into one picture will be a feat. The intelligence community, however, must make cultural changes first to accept the importance of the technology and public data, and secondly to acquire the technology and data on a timely basis.

Need for Change

The intelligence community has struggled with adopting the reality that to remain relevant, it must embrace publicly and commercially available data into its threat and warning process and use the powerful technologies that the commercial world is developing to sift through that data. There has been much work in this area across the intelligence community and some of it has been groundbreaking but the work has not been comprehensive, integrated, or fast. There are boutique enterprises that have developed their own high-tech way forward but when it comes to scaling such technology across the intelligence community or within the Defense Intelligence Enterprise, the hurdles are huge. Each intelligence agency claims its own security issues and erects fences against cross intelligence technology modernization. Even organizations like the Defense Innovation Unit or IN-Q-TEL focus on discreet requirements, not the wholesale cultural change needed to bring in the latest commercial technology that can support warning and security. Neither the Director of National Intelligence nor the Office of the Secretary of Defense for Intelligence and Security have been able to lead the intelligence community to making this cultural and technology change a key priority. As the largest part of the intelligence community, the Defense Intelligence Enterprise needs to lead the way in developing a high tech, open source, data rich environment for its customer base. Military commanders need a real-time, complete, and high-quality battlespace picture to quickly make informed decisions, take direct actions, and assess the high volume of potential targets and threats.

The capabilities that are available for plugging into this warning and targeting picture are endless. For example:

Warning Against Biological Events: Technical sensors and human sensors worldwide record biological events and provide data on disease upticks and hazardous environments. Environmental & Biosensors are global and are critical tools for detecting, quantifying, and interpreting dynamic changes in environmental or biological systems. There are multiple crowd sourced disease outbreak platforms. Biosensors, local reporting, and crowdsourcing technologies monitor everything from air and water quality to metabolic and cellular activity in real time. Technology can combine these complex platforms and sensors to read the data and provide intelligence analysts a real time global map of trends ad anomalies, that once fused with other intelligence could lead to early detection of a pandemic, an industrial or environmental bio event, or development of a WMD. Such a real time platform, based on open source data would also support military and clandestine operations by providing information to the operator about the environment they will encounter. Such maps are currently available either through a subscription or to the public but not one that fuses all this data.

Humans as Sensors: Combing through social media to provide on the ground sentiment analysis can help intelligence analysts provide warning of threats or instability or to provide information on how populations are responding to real time events. This commercial tool, used heavily by marketing firms, can augment State Department and other reporting on in country sentiments and provide insights into socio cultural priorities and concerns.

BLOCK Chain Intelligence: Blockchain intelligence can enable proactive disruption and analysis of adversaries by providing data on how they are using virtual assets for revenue generation. This process allows provides a high-level overview of a country's cryptocurrency usage, which could provide valuable insights into the nation’s overall economic health and strategy. It also provides detailed information on an individual or entity’s financial activities that can illuminate patterns and processes. The importance of blockchain analysis was underscored in 2024 when the US Senate Committee on Armed Services recommended that the Department of Defense (DOD) explore blockchain technology to potentially revolutionize national security. In the committee’s 2025 National Defense Authorization Act (NDAA), they instructed the Secretary of Defense to investigate and test potential blockchain applications for DoD.

As the Department of Defense has made clear, commercial solutions for trend analysis, finding anomalies, and providing data for decision makers are abundant. The intelligence community already has some of these abilities. What is missing is the fusion of this data into one picture and the ability to use the tools at scale.

Starting with Cultural Change

In order to provide the intelligence needed for an increasingly complex world, we need to fundamentally change the practice of intelligence to accept the importance of publicly available and commercially available data. Additionally, the intelligence cycle needs to change. The intelligence cycle needs to start with open source information and data. This means that the intelligence community also needs to incorporate the technology and the algorithms that will allow today’s intelligence professionals to see trends, anomalies, and threats in that data. This data then should be fused with intelligence collected by the traditional intelligence disciplines to provide decisionmakers and operators with all the information they need to perform their jobs.

Further, to demonstrate the importance of harnessing the publicly and commercially available data, the intelligence community must develop an open source center that brings together the myriad open source/PAI/CAI and data sifting technologies that industry currently is building. The high-tech center would be the first step for intelligence analysts of all types in building threat and warning analysis and targeting. This center would be a place where the US and its allies could share data and techniques to build a global warning picture. It would be the intelligence community’s beginning of the Enterprise Command and Control picture or the Common Intelligence Picture.

The Cipher Brief is committed to publishing a range of perspectives on national security issues submitted by deeply experienced national security professionals. Opinions expressed are those of the author and do not represent the views or opinions of The Cipher Brief.

Have a perspective to share based on your experience in the national security field? Send it to Editor@thecipherbrief.com for publication consideration.