Legal Limbo Leaves Killer Robots Off-Leash

The UN’s Convention on Conventional Weapons (CCW) Group of Governmental Experts (GGE) met last week to discuss lethal autonomous weapons systems. But while most member states called for a legally-binding process to ensure that some form of meaningful human control be maintained over these prospective weapons systems, there is a sense of distrust among states that could fuel an artificial intelligence and robotics arms race.

• A total of 86 countries participated in the meetings, including the UN Institute for Disarmament Research (UNIDIR), the International Committee of the Red Cross (ICRC) and the Campaign to Stop Killer Robots, a coalition of over 60 civil society groups across 26 countries coordinated by Human Rights Watch. The goal of the Campaign to Stop Killer Robots is broadly to prohibit the development, production and use of lethal autonomous weapons systems in order to retain meaningful human control over life-and-death decisions in battle, policing and other situations.
• The Pentagon, in 2012 directive, has laid out the U.S. policy that such weapons systems should allow “appropriate levels of human judgment” over the use of lethal force. But what constitutes “appropriate” remains unclear and the national policy could change depending on the evolving capabilities of other states, particularly China and Russia.
• One of the major contentions in last week’s GGE discussions was the definition of what a fully autonomous weapons system actually is. The Netherlands proposed this definition: “a weapon that, without human intervention, selects and engages targets matching certain predefined criteria, following a human decision to deploy the weapon on the understanding that an attack, once launched, cannot be stopped by human intervention.” No formal agreements on a definition emerged from the GGE discussions.
• States will determine steps forward at the annual CCW meeting on Friday, November 24, but a few prominent states, including the U.S. and Russia, believe it is too soon to begin negotiating new arms control measures for weapons that do not yet exist.

Notably, autonomous systems are not the same as automatic systems, though automated systems are the natural precursor to futuristic lethal autonomous weapons. While the distinction appears conceptually clear, agreeing on where that line between autonomous and automatic weapons systems resides is a main focus of discussions at the United Nations.

• Systems such as the Phalanx Close-In Weapon System, Israel’s Iron Dome missile defense system, or South Korea’s Samsung SGR-A1 border sentries all represent automatic weapons systems, which are primarily static defensive capabilities preprogrammed to target inanimate objects such as incoming munitions when reaction times would often overwhelm human operators.
• Autonomous systems, while not yet developed, would be mobile weapons platforms equipped with onboard sensors and computing systems capable of decision-making through algorithms to seek, identify, track and attack a variety of targets by adapting to their surroundings after being activated. The United Kingdom’s Taranis, France’s nEUROn, and the American X-47B are all examples of a push toward autonomy in unmanned aerial combat vehicles.

The prospective development of autonomous weapons is largely driven by states that believe they need these weapons capabilities for self-preservation out of fear that adversarial states also will develop them – a spiraling cycle that breeds a global arms race. “Artificial intelligence is the future, not only for Russia, but for all humankind,” Russian President Vladimir Putin said at the beginning of September when speaking to students. “It comes with colossal opportunities, but also threats that are difficult to predict. Whoever becomes the leader in this sphere will become the ruler of the world.”

• Weapons autonomy removes requirements of a communication link to remotely controlled weapons systems such as unmanned aerial systems (UAS), which can prompt command delays and is vulnerable to electromagnetic disruption, such as spoofing or jamming, as well as capture that could reveal the drone’s location and sensor feeds. Operating in contested airspace, as has become apparent over eastern Ukraine and is a central component of China’s anti-access area denial (A2/AD) strategy in the South China Sea, will require some level of autonomy should unmanned aircraft wish to remain a viable operational tool.
• Autonomous weapons close the time gap between action and reaction by enabling decision-making at speeds likely incomprehensible to humans – giving states with autonomous weapons capabilities a clear tactical advantage on the battlefield.
• Unlike humans, autonomous systems conceptually know no fear, stress, fatigue, nor are prone to emotional overreaction or self-preservation instincts. Such qualities, in the right hands, might render warfare more humane by preventing militaries from committing atrocities of war. Greater restraint coupled with enhanced discrimination between civilians and lawful combatants could result in the use of force more aligned with international humanitarian law.

While there are qualities that could drive militaries to develop lethal autonomous weapons, some of those same qualities give credence to groups seeking to apply preventive arms control measures. Many of the operational advantages militaries could gain from the development of autonomous weapons also present operational risks.

• The high-tempo and adaptive decision making of autonomous systems could lead to unpredictable outcomes similar to flash crashes in the financial trading sector. The possibility of high-speed friendly fire by autonomous weapons could mean militaries will retain a human mechanism as a fail-safe.
• Autonomous systems could also be destabilizing by engaging in quick, disarming surprise attacks, where a swarm of small, cheap, 3-D printed drones could fly under the radar and target strategic command-and-control systems or even nuclear capabilities – disturbing a fragile strategic balance of power and perhaps escalating a diplomatic crisis to all out war, without human intervention.
• Given the proliferation of drone technologies, either through national development or commercial and military export, it is likely autonomous weapons will also eventually find their ways into the hands of nefarious actors. Much like how autonomous weapons could enable more precise targeting to differentiate friend from foe, in the hands of oppressive states or terrorist groups, these systems could wreak havoc on an unsuspecting civilian population.

• Should autonomous weapons engage in unlawful acts, the question of accountability becomes a challenge – are humans liable for the harm caused by a weapon acting independently? While decisions to use lethal force are already decentralized in kinetic drones strikes, accountability would be even more diluted for autonomous weapons – and developers and weapons manufacturers are often granted immunity. This “accountability gap” could weaken deterrence of unlawful acts and leave victims without any avenue of recourse.

Despite ongoing discussions at the multilateral level within the CCW, it remains uncertain whether states will take meaningful action to prevent a global arms race toward autonomous weapons – a prospect that could be a defining characteristic of global security for years to come.

Levi Maxey is a cyber and technology analyst at The Cipher Brief. Follow him Twitter @lemax13.