Our Propensity for Addiction
To a certain extent, we’ve been misled to believe that growth in our IT budgets as well as the sheer number of systems we operate and support, translates into a positive measure of merit, when in reality, simply accumulating more without shedding legacy, actually results in less of everything that really matters to an organization, like security, agility, performance and cost.
In today’s digital world, the adage “if it isn’t broke, don’t fix it” is not relevant. Achieving the organizational agility to continuously upgrade and/or remove-and-replace legacy hardware and applications before they become cyber security liabilities is more than a support process, it is an essential 21st century tradecraft. We must overhaul traditional approaches to budgeting, acquisition and O&M of IT systems, driven by an informed mission risk calculus and an accurate life cycle cost assessment, associated with maintaining legacy.
Sean Roche, Former CIA Senior Executive
A new leadership imperative that makes improving cyber security a true priority for every mission element while refusing to succumb to the learned behavioral impediments that perpetuate legacy systems: functional fixedness, outdated onsite IT support models, irrational nostalgia and fear of change is essential.
Legacy systems are a digital opioid
How many of us are still using a cell phone or a laptop we purchased in 2001? Why then, do we accept and even defend maintaining legacy systems in the workplace? Legacy systems, most simply defined, are systems that could be replaced with a new system that offers improved performance, security and capability. This has often been incorrectly confused with the word ‘update’, which for purposes of this article, can be defined as merely a revision that does not change the overall capability.
In the digital world, considerations for obsolescence must begin during the release of any new hardware or software system. Even for behemoths like Microsoft and Apple, the mean time between issuing new product operating systems has decreased by 33% and 50% respectively and keeps accelerating. Yet, government agencies continue to seek special arrangements to extend support well beyond the market lifetime. At a minimum, legacy must include operating systems, applications and hardware that is no longer supported within the original universal robust support period declared by the manufacturer.
Beyond specific capabilities, foundational legacy also needs to be addressed as new operating models and architectures that recognize the current security and performance environment (example: transition from mainframe to client-server to cloud) are introduced into the marketplace.
Today, we are far too passive about identifying and replacing legacy systems. Every organization needs to identify and track current legacy IT systems and take aggressive steps to actively annihilate them while having transition plans for new systems and applications as they are launched to prevent accumulation of legacy in the future.
Getting and Staying Hooked
A key weakness is our faithful adherence to processes that are the enablers that keep us on vulnerable legacy systems, delaying the moment we have our digital “rock bottom” and then being forced to admit our failures. We are perpetuating a cycle of legacy addiction by asking our IT support contractor to find the digital workforce that is content to service yesterday’s technology. This degrades the strength and vitality of these companies that are a large and increasingly vital part of our workforce.
Sean Roche, Former CIA Senior Executive
On the sidelines of a large public sector trade conference, a senior IT services provider noted the challenge to attract and retain critical digital talent for the national security mission. He said “if I am somehow able to lure them to come work for us and then, by miracle, get them successfully through the clearance process in less than 14 months, what is my chance of retaining high demand, hot shot new cyber, IT, data science and engineering graduates long term when I tell them they will be assigned to work in a windowless room on a 5-year contract to maintain Lotus Notes?”
Perpetuation of legacy IT hardware and software is perhaps the single greatest impediment faced by nation states, corporations and individuals seeking to reach a higher level of cyber hygiene and mission performance. Surveying the major cyber security data breaches and intrusions that have occurred over the past five years, exploitation of legacy systems is often the direct method or a key contributing factor. When seeking to gain access to a restricted network there are certain “aim points” that intrusion teams favor –legacy operating systems and applications are at the top of the list.
For example, consider Microsoft’s Windows XP, a digital antique of an operating system that provides hackers with a cornucopia of vulnerabilities. When it debuted, XP was a significant improvement in terms of user experience and integrated capability over the deservedly maligned Windows ME (millennium edition). Unfortunately, it was released just at the time when awareness of hacking and malware at scale was just starting to evolve. Built with user experience, and easy interoperability in mind, but not security as the priority, it was quickly revealed to have a series of increasingly severe cyber security vulnerabilities. For the last 10 years, in the cyber security community the question of “how do you make an XP machine secure?” is met with the universal response of “uninstall”.
Microsoft introduced XP in 2001 and updated it with newer versions until 2008 when they ceased general licensing and terminated retail sales, exiting mainstream support in 2009 and then ceasing “extended period support” (double secret probation) six years ago. Unsupported in the best of circumstances, why would any organization that values protection of their data, continue to operate Windows XP on even one terminal of their system today?
Across the USG and across the globe, the number of machines running XP today defies logic, but not explanation. Most often, the core network has been extended over time with outlying machines that are not considered during upgrades to new operating systems. The momentum created by perpetually extending the organizational licensing agreements, lack of even basic efforts to maintain up to date network configuration and most importantly, failure to consider cyber security decisions as part of the core mission workflow, are to blame. The cyber hygiene of a vast network of machines running the very latest Windows OS is significantly degraded by even a few connected and enabled nodes running a legacy OS like Windows XP.
Mainlining: Your Bank – Your Doctor
The USG is not alone in the struggle to retire vulnerable legacy. According to a 2015 report published by the ATM Industry Association, despite being no longer supported, more than 94% of the world’s ATM machines were still running highly vulnerable Windows XP. Even though ATM machines have a 7 to 10-year lifecycle, the industry sought and received support for the embedded version of XP until January 2016. Following a number of embarrassing incidents, US banks and manufacturers at the time made the decision to migrate to the very latest version of Windows and more importantly to move to Linux-based platforms to provide the agility required for near continuous patching and upgrades in the future. The medical industry continues to run XP on many critical devices that monitor vital signs, detect signs of illness and determine dosage of medicines. This is largely driven by lack of awareness within the medical community as well as hardware devices that are designed to be compatible with only one operating system.
Our Partners and Allies
In an increasingly connected world, you are only as secure as your most vulnerable vendor or mission partner. In April 2014, trade journals reported that the Dutch and British governments rushed to create last minute desperation deals with Microsoft to continue to keep XP “on life support” for them under Microsoft’s Custom Support program at a cost of $9.1 million for British public sector customers. The Dutch government quickly followed suit creating its own multi-million Euro deal with Microsoft for custom XP support for their public sector machines still running XP.
An Expensive and Dangerous Habit
Those deals are a fraction of what the USG has been paying to support the hundreds of thousands of systems that were still running XP and other end of life software. Despite years of advance warning that support for the operating system was ending, there are still far too many large systems running Windows XP across the USG on sensitive networks and embedded mission systems.
Sean Roche, Former CIA Senior Executive
Shedding legacy is the fastest way to generate the investment needed to re-capitalize IT infrastructure and systems on a recurring basis. Beyond vulnerability, the true cumulative financial burden associated with retaining and maintaining these outdated systems is often obscured, elusive and purposefully hidden. While there may be some federated IT budget under the purview of the CIO the true IT spend within organizations is often fragmented among disparate elements that procure and operate mission specific networks, develop applications and apply cyber security as well as the growing discipline of data curation. Calculating the true IT spend most often reveals a much larger financial burden than is captured in the budget associated with the CIO.
Additionally, the unnecessarily complex, risk adverse and often soul crushing USG procurement process motivates even the most stalwart public sector employee to support the seemingly endless rubber stamp renewal of legacy licensing agreements and exercise out year options on IT support contracts without the slightest bit of evaluation of the cost of the now obsolete product, much less the cyber vulnerability. Thus, the decision to license an operating system or worse yet, a software package that creates proprietary structured databases, will be renewed far beyond the normal useful life. The result is a lifecycle expenditure that greatly exceeds the utility of the legacy system and effectively blocks investment from upgraded capabilities that are inherently more secure in the current and highly dynamic cyber environment.
The highest risk - cost situation can develop from the decision to procure a truly custom software solution that is developed based on USG derived requirements and supplied by a team of contractors. Far beyond licensing, this approach perpetuates the myth of “sunk cost” decision making, often creating a 20-25 year captive on-site contracting arrangements, creates embedded, retrenched legacy while locking out other solutions and provides a cyber security vulnerability footprint that is assumed away to be adequate but is not driven by an accurately informed risk calculus.
Denial…Getting Our Fix
The unjustified hyperbole of excuses we accept and tolerate to retain legacy systems are increasingly unsupportable and illogical. These are most often offered by senior officials with NO background in cyber security, IT or any type of technology. They are often manipulated into making false, unsupportable arguments by the embedded mission support contractors who have a vested interest in perpetuating the infinite “Y-2K feeding trough”. In fact, it was the overblown fear mongering of the 1999/2000 digital turnover (perpetuated by USG and others) that revealed that most senior officials would fight to maintain legacy for all the wrong reasons driven by the innate turf war sense that they are rewarded for in government.
Let’s consider some of the unsupportable justifications for maintaining and perpetuating legacy systems that are known to be highly vulnerable, barely used and expensive to maintain.
Upgrades are Too Expensive: Most components within large government organizations struggle to identify what they are actually spending on IT and cyber. But they will offer that it is too expensive to constantly tear down and replace legacy systems. The reality is very different. Even a most basic review of the budgetary implications, do not support this non-sequitur. Given the cost to maintain a cleared workforce tasked with the burden of maintaining and upgrading secure terminals, the cost of maintaining fixed place hardware and software configurations to keep them secure more than 4 years is greater than replacing them outright after 48 months. More convincingly, for those organizations that issue a company configured laptop for less sensitive, offsite teleworking via the Internet, the cost of maintaining the cyber security on those devices dictates that they should be outright replaced every 14 months. Increasingly, devices and software applications are racing to the bottom of the commodity pricing model. Yet, the outdated traditional legacy support model attempts to keep them viable for 3+ years. This is not only costly, but it adds a constantly increasing number of attack vectors that cannot be adequately patched, despite significant costs.
Legacy = Under the Radar = Safe
Another rather ludicrous argument to justify our continued addiction is that legacy systems are inherently less vulnerable because they do not draw the attention of nefarious intrusion artists. The amount of core re-use for new releases of Windows operating systems means that newly discovered flaws that become CVEs are effective on current and older operating systems. Examining the actual data and trends behind cyber intrusions provides the stark reality that the vast majority of current year incursions and attacks rely on well-known CVEs that have been revealed and patched for years. In 2018, the two most frequently used CVEs had been declared with patches issued more than 9 years earlier. It is not just the number of vulnerabilities, but most importantly, the type. Despite the end of support for Windows XP, Microsoft has been compelled to release three “emergency” security updates for the operating system to patch major security vulnerabilities:
- A patch released in May 2014 to address recently discovered vulnerabilities in Internet Explorer 6 through 11 on all versions of Windows.
- A patch released in May 2017 to address a vulnerability that was being leveraged by the WannaCry ransomware attack.
- A patch released in May 2019 to address code execution vulnerability similar to the WannaCry vulnerability
One of the most lethal vulnerabilities used to target Windows XP was revealed in August 2019. A remote code execution vulnerability (aka: “Bluekeep”) exists in Remote Desktop Services formerly known as Terminal Services when an unauthenticated attacker connects to the target system and sends specially crafted requests, aka 'Remote Desktop Services Remote Code Execution Vulnerability'. In terms of risk, it doesn’t get any more critical than this, earning a 10 out of 10 for serious for digital lethality. One final note to those still in denial: XP was the initial poster child for legacy but not unique. Windows-7 extended support ended on Jan 13th 2020 and within a week, a new critical level CVE for Windows-7 was announced that targeted mobile users who were guests in luxury hotels throughout Asia, but there was no patch available nor was one expected anytime soon as Microsoft had ended support.
Do you have executives traveling with legacy operating systems? In the same way, it is increasingly difficult to maintain a vintage classic car on the road for daily use, keeping legacy systems secure is a losing battle.
Legacy User Tantrums
One of the most fascinating aspects of attack vectors for cyber incursions are those created on a large network by only a handful of recalcitrant users who refuse to accept the sunsetting of a legacy system because they cannot adapt to the changes in user interface of the new version. Consider the case of a sophisticated, high-tech sub-contractor that was using a software design tool to develop sensitive component systems as a supplier to a major government program. After critical vulnerabilities were discovered in a legacy version of the software design tool, the sub-contractor swiftly replaced it with an entirely new version only recently released.
The newer version comprehensively addressed all known flaws or vulnerabilities, not surprisingly the new version also had new features to enhance capability. While most of the staff accepted the change, a few announced that they disliked several aspects of the newer user interface and the display layout and therefore they REFUSED to use it. They complained loudly and often until their operations manager demanded the IT team re-install the older version on their machines. Not surprisingly, running the older version reinstalled the key vulnerability that was the eventual intrusion path that resulted in their designs being hacked and exfiltrated. Many who have managed USG IT have stories of maintaining legacy systems at great cost while waiting for the last long serving curmudgeon user to retire.
The model for leadership in retiring legacy government systems was former NASA Administrator Michael Griffin, who made the decision to retire the Space Shuttle after 30 years, for all the right reasons. Former members of the astronaut corps, contractors and members of Congress launched extensive, maligning attacks against this inevitable decision. When those failed, there were an array of increasingly desperate delay tactics aimed at extending the program beyond the declared retirement date. Less than 10 years later, the dire consequences and morbid predictions of what would happen when the STS was no longer flying have been replaced by the emergence of a real revolution in commercial space, one that has delivered the first low cost, truly reusable launch system pioneered by SpaceX.
Treating IT, Security and Cyber as a Secondary “Support” Functions
The fact that the OCD functional fixedness behavior of recalcitrant users is tolerated has its roots in the way IT, security and cyber were considered ancillary and subordinated to the workflow of organizations created before the dawn of the desktop computer era (pre 1986). Despite elevating the CIO, CISO and CDO positions within large government agencies, these disciplines were still viewed as “children of a lesser God”. Additionally, they were (and still are) often forced to refer to all others in the organization as “customers” and hence their mission vital functions are referred to as “customer support”. That’s the bureaucratic equivalent of wearing a “kick me” sign on your back.
When individual PC style workstations were introduced at scale into the workspace for Federal agencies during the mid-late 80’s it quickly became clear that the typical procurement program government budgeting cycle did not support the kind of agile and constant investment that IT would require. Initial forays into the use of the IBM PC-AT model quickly revealed the need for an operations and maintenance budget as well as a growing army of specialized support personnel who would later evolve into the IT support services workforce. As information systems became a more integral and essential part of the workflow, the required investment started to compete with major baseline programs. IT was still viewed as an administrative “support” function.
Patching and extended support is the new digital methadone
Patching and extending support is offered as an effective substitute for replacement but quickly becomes a type of digital placebo to placate anxious leadership that something is being done to address the issue. The true numbers are often elusive. Patching and extended support require larger numbers of hands-on IT support personnel at increasing cost with lower effectiveness. It prolongs the digital addition and expensive habit of legacy systems.
For a number of technical and configuration reasons, relying on patching to keep legacy software systems secure is insufficient. The challenges with a reactive patching strategy are multifold. First, individuals just don’t do it consistently and large organizations are typically slow to implement, sometimes years after the patch is issued. Next, the nature of the digital “stack” is being transformed as capabilities are continuously integrated in software defined configurations, this will only accelerate as 5G is rolled out. The idea that patching any single component will be sufficient is going to be replaced by the concept that the software defined network must be continuously “repaved”.
Patching must occur with a 100% success rate for the entire network. Anything less than 100% results in a gaping vulnerability gap. Finally, as more organizations go to 24/7 operations with higher reliance on their IT infrastructure, operational users are reluctant to allow any intentional or unintentional downtime that might be occur as a result.
To be clear, patching is still an important component of cyber hygiene, but the current dosage and heavy reliance on it as a single point of treatment is not a cure. Likewise, despite the excessive cost to those who buy it, periods of “extended support” provide an overall lower level of cyber hygiene and become the equivalent of a long-term care policy instead of preventive care. Thus, vulnerable and expensive legacy systems are enabled into a permanent state of “digital hospice” with no one willing to mercifully euthanize them.
In part two of this special series on the risks posed by legacy systems from Cipher Brief Expert and former CIA senior executive Sean Roche, we’ll talk about the 8 initial steps to better cyber hygiene.
Read more expert-driven national security insights, opinions and analysis only in The Cipher Brief
