Why did some US institutions not migrate their very old software systems to use somewhat newer ones?

Question

This article shows us that some US institutions rely on very old systems (written decades ago, which is very long in IT time perspective):

New Jersey Gov. Phil Murphy says that the state is looking for volunteers with skills that can be used to help in the COVID-19 coronavirus outbreak, and one of those skills is knowing your way around a 61-year-old programming language used on big, old, mainframe computers.

COBOL is an old computer programming language that was first developed in the 1950′s in conjunction with the Department of Defense. Today, most programmers prefer and use more modern languages, but there are pockets where old software written in COBOL remains in use, particularly financial applications and in large enterprises or government agencies.

I understand that state institutions are typically much slower than private companies when it comes to IT investments, but having such old systems seems strange. Relying on systems written so long ago can cause lots of issues which translates to wasted time and money:

• initial developments done way before modern software development frameworks were around
• it is much harder to find specialists who, not only know the language, but also are able to adapt to such an old software development style
• inability to act in a timely fashion when changes or maintenance must be performed (e.g. Coronavirus outbreak)

It makes sense for banks to be reluctant to rewrite their systems due to the large costs and risks, since they must also make profit. But a public institution is not profit based and, in theory, it should be easier for them to invest in modernizing their software systems.

Question: Why did some US institutions not migrate their very old software systems to use somewhat newer ones?

Note: I am not thinking about big leaps, but why not at least to work with languages and frameworks that are at most 20 years old, not 60 years old.

---

While the main arguments are related to software development risks, I am interested in the political aspect of the "why", since we are talking about public institutions making the decision to partially / totally rewrite some systems. Such aspects might be related to the size of projects or quality of management in the public sector.

Answer 1

It's not just US institutions, there are many countries, institutions - including banks - and companies running on ancient software and hardware.

There are multiple reasons for that:

First, from the business standpoint, replacing software costs money. Why spend money to replace something that already meets requirements? If you have software A that does what you want, already exists and is "free", since you already own it, and software B, which doesn't exist yet, costs years of development time, probably requires new hardware and has the associated costs for all of that, which would your boss or your (often very much non-technical) boss 's boss approve? What would a government employee with a tight budget approve? What would a politician calling for less government spending approve?

Secondly, the exact workings of the system are often lost in time. No one actually knows what the system does exactly, but it interfaces with dozens of other systems through APIs that are equally unknown, obscure or deprecated. Creating a specification for such a system that a contractor/contracted company can work with is difficult, if not impossible. There is a huge risk that the resulting system will not replicate the behavior of the old system and stuff will stop working. It might do things like round .5 up instead of rounding to even like the old system, which could cause all the bills or transactions to suddenly be off by 1 cent compared to the previous system.
In government, a broken system could mean that the unemployed might not get their benefits, the tax payers not their yearly tax payback or any number of things that will lead to an elected politician not being reelected. Politicians like being reelected, so even if they understand the technical need, which is rare anyways, they might avoid the risks for political reasons. Often there were attempts at replacing the system, which then failed and then those failures get pointed at whenever someone proposes replacing the system.

Thirdly, the old systems usually have a very bad user interface. A new system will have a shiny, new interface, sometimes because creating the old interface wouldn't even be feasible in a new application. This requires training people. People that have worked in the company or institution for 40-50 years and are very set in their ways. Those people often have a lot of blocking power, which can completely halt development on the new system, because their requirements, i.e. "don't change anything," aren't met. They are usually also very hard to fire and replace, so the institution has to hire new people without firing them, which requires a bigger budget, which comes back to the first point.

In summary, as technical people we might understand that those systems will someday inevitably go "poof" and everything will break down catastrophically. Other people either don't know that it can happen, or they just hope that it will happen after their time, because in the end no one wants to pay the money, take the risk and take the blame for a project that will probably fail.

Answer 2

Because it's hard to do, prone to failure and very costly. And because the newer alternatives aren't always a big improvement.

Very costly in the sense that failure costs in the multiple 100M$ are frequent and could easily become an election issue (which makes it a political issue).

Take a look, for example, at Canada's revamped payroll system. $1B+ and counting, IBM was the lead on that. Or look at Capita's activities in the UK and its litany of project failures. Or, 70M$ for a failed asset management system in Australia, for a population base of 250 000 people .

TLDR:

Even with the best of intentions, we in the IT industry do not yet have a proven, repeatable, low-risk solution to migrating old government IT systems. Nor is it all that clear what the next good-for-60-years technology solution is. Until that is the case, there will be a strong case for keeping seemingly obsolete, but working, systems running as long as possible.

The details:

Bad project management in public sector

It is very difficult to reduce risks when doing this kind of work.

One option is to do a strict migration from the old system to the new system, with it doing the exact same thing as before. This is very rarely done, usually all sorts of new bells and whistles are attached which makes it very hard to judge if the new system does indeed provide the same results as the old. Often, under the guise of the rewrite, the existing business processes and organization are revamped, on the basis of the promises made by the promoters of the new system.

Governments IT is often driven by consensus and stakeholders. Give too many people a say in what a complex system should do and what should be fairly straightforward degenerates into endless meetings and shifting requirements. Worse, rather than picking a representative sample of senior front-line users of the existing system, the stakeholders are usually their managers. Often, retraining of, and engagement with, front-line users takes a backseat to more glamorous techy and PM stuff.

Many government departments also consider themselves special and do not like off-the-shelf existing solutions. That ranges from not buying a commercial product but writing it from scratch to acquiring an off-the-shelf product but then customizing it all out of recognition.

Even when intentionally trying to keep things simple there are inherent complexities to government IT. A payroll system will have to keep track of multiple union agreements and seniority scales, for example. Not all payroll vendors are able to do this without custom development.

Unclear and shifting requirements, and to a lesser extent, incompetence by the outsourcing contractors is what kills most government IT projects.

Big IT projects require a Mr./Ms. No. The successful projects I have participated in all usually circled back to the project director having the authority and willingness to tell the different stakeholders to buzz off, but only when necessary. Their driving principle is to keep things simple, a single-minded focus on reducing risk and to get a solid working basic system out as quickly as possible. That person needs to be backed at the highest level. It's rare to see that recognized, and doubly so in a public sector setting.

Limited choice of, not always good, contractors

Public sector IT relies heavily on outsourcing to big, high-profile, companies. IBM, Capita, Accenture, etc... (in the distant past, when governments were a major driver in early IT a lot of this work was done in-house).

The government consulting companies are often more skilled in navigating the complexities of government procurement and contracts than they are at actually producing working products. However, when a project fails, the immediate government reaction is to be more careful the next time and select a trustworthy, credible, vendor.

Small companies, even assuming they somehow knew better than the big consulting giants, don't have that credibility, or the commercial skills to win a bid, and the next big project tends to be awarded to yet again the IBMs and Capitas of this world, even while these companies outsource more and more to lower cost countries like India.

The next line of defense is to write longer and longer specifications documents, with every contingency covered. That 1.2B$ Canadian payroll project had a 6000 page spec at some point. Specs cost, but are not working code at the end of the day. No one is going to read 6000 pages of tech specs, making it very hard for any one person to fully understand the project.

A primary point of effort for a consultancy (their core competency to cite a comment) then becomes managing the spec and limiting their contractual risks, rather than getting things done. At the opposite end, the much vaunted spec-less methods like Agile are woefully inadequate in these contexts.

Technology

COBOL is 60 yrs old, about as unhip as you can get and is suffering from a shrinking pool of talent. On the other hand, it is very stable, you can take 30 yr old code, run it in a modern environment and expect it to work and it also quite clear to read. It's there and it works.

The primary alternative would be Java. 25 yrs old, no shortage of talent. But... there is much less version-to-version stability, the codebases are rarely very clear and there is a strong culture of overcomplexity in its practitioners. Java is also quite allergic to non-Java stuff, its perfect world is one in which everything is also written in Java. Unfortunately, that is not the way the real world works.

Java's shortcomings are being recognized and it's start to have the whiff of "legacy" as well. In the last 5-10 years not many would have picked Java as a technology stack for complex new websites for example, unless they are already invested in that technology.

However, there are no other real contenders for massive-scale, government-level, IT development languages.

Such a language would need to be:

• predictably available 30-50 years from now
• secure by design, plays well with others, not overly verbose
• suitable for line-of-business apps (i.e. processing health claims, rather than doing high-performance 3D graphics)
• clear and with low initial complexity for junior programmers (John Carmack ain't gonna be working at NJ's DMV)
• appropriate to projects in which tens or hundreds of developers work together on a tightly integrated system

COBOL comes close to this ideal - its main weakness are scarcity of skilled labor and verbosity. Java less so IMHO and I can't think of another widely adopted language that combines these attributes. Simple modern compiled languages like Swift, Go (C#?) might be adapted/adopted for this, but that's not their current aim and they lack vendor neutrality.

Why does this matter to politics?

Because software matters. Banks have sometimes been described as IT companies specialized in customer service and money management. A modern government relies on IT in order to deliver all sorts of services and to collect taxes. Large scale IT failures have a budgetary impact, crowding out spending in other areas and can directly affect how citizens interact with their government. And they become talking points during election cycles.

On a related subject, the inability of local government IT, for schools, hospitals or cities, to manage and secure their systems have made them prey to ransomware attacks, which can very directly affect thousands of citizens.

In order to function well, governments are going to have to improve considerably in this field.

Answer 3

I would like to address the underlying misconception that was expressed in the note:

I am not thinking about big leaps, but at least to work with languages and frameworks that are at most 20 years old, not 60 years old.

Yes, COBOL is a pretty old language and new systems are likely not made with it anymore. However, FORTRAN is even older (63 years) and still used in science and engineering.

20 years, on the other hand, is young for a programming language. I can think of very few languages that are "at most 20 years old" and already known and used. Rust, Go, Kotlin are about ten years old. Swift is 5. C# is 20.

Most of the other languages are more mature. Web servers mostly use PHP (26 years old), Java (25), Python (30), also C#. Web frontend uses JavaScript (25). HTML is not a programming language, but it's about 30. Operating systems are written in C (48) and C++ (35). And the assembly language obviously precedes even FORTRAN, it's close to 70 and still used.

Of course, these languages have evolved since initial appearance, but so (slightly) has COBOL with the latest additions in 2014. My aim is to drive home that age does not make language bad and most software is built on languages older than 20 years.

Answer 4

I think that the premise of the question is not correct: there are often technical reasons to keep using legacy systems.

To contribute to ideas that have already been mentioned, I want to mention two facts:

• COBOL it not a stagnant language from 60 years ago. Yes, it is old and lacks many features of modern languages. But its standard was last updated in 2014.

• Rewriting big systems is no minor decision. Several cases have been mentioned above, and I wanted to add that of Netscape: in 1997, when they were the clear leaders in the browser wars, they decided to rewrite their code from scratch. Three years later, when their code was finally more or less ready, their market lead was gone. This article from 20 years ago, by Joel Spolsky, mentions this and other examples.

Answer 5

People are quite insensitive to gradual degradation of something. This is not limited to IT infrastructure. Roads, bridges, dams, power grids, internet RFCs, international agreements, etc, ... are in general not redesigned and rebuilt until they fail spectacularily.

Prior to failing, they are seen as "not broken, won't fix".

Almost no one gets praised, re-elected or promoted for saving a disaster that didn't happen.

Answer 6

Information Systems used by the state and in general IT in the public sector is a very tricky subject and always have been. From the strategic R&D perspective, you need to formulate requirements on what do you want to achieve by having given IT infrastructure. On a wide level, as you can find in the public sector, it's a very complicated thing to do.

First point, you have an existing IT system and infrastructure. If you want to change some part of the legacy system, it may be extremely hard to do, because of it's complexity, unknow documentation and requirements, people working and managing this area of IT system not being there anymore, or even when you have people who made those changes, they won't anymore remember exactly what was the requirement and need for their implementation in the first place, because it was a long time ago.

Another thing you can do is to change the entire IT system, which has its own problems:

The second point, you have political cycles, which means some administration is going to formulate the need for IT innovation. Now, when we are talking about a whole area like defence, social security, taxation, healthcare... Those things take a long time and money to be implemented. A lot of times, however, it spans administrations or people in charge of those innovation project. After new administration or strategic unit emerges, for example as a result of an election or even a simple change in management of the given institution, the new unit doesn't want to continue in given innovation project, they cut funding and stay on the same level or formulate different requirements. This is usually a more dominant problem of public sector than a private one.

Thirdly, you can very often find old methodologies used for developing existing systems. They are very inflexible, usually not very effective, often going back to beginning of the 2000s when the level of IT project management was usually quite frankly catastrophic. So when the strategic unit wants to hire a new company which uses modern methodologies of software development, they don't want to go into changing these old legacy systems because in general, it won't work and won't be worth their time and effort, so they will want to go from scratch, which usually takes time and a lot of money, where we are getting back to the problem of political cycles and strategic endeavour discussed in the previous point.

Answer 7

COBOL is a pain to write. But it comes from a time when storage and processing were slow and expensive. Brutally simple, ruthlessly optimized. Not a bad choice for systems where speed and transparency matter.

Banks suffer few of the government contract issues mentioned yet they mostly still use COBOL for things like ATM transaction processing.

FORTRAN still has its niche too. For complex math problems, everything else is either 1) even harder to use ( C ) or 2) slower (everything else).

Answer 8

The political aspect of the "why" : I guess there is nothing to gain, politically.

At first, we assume we have a system, which has been set up to do its job, and works without major errors. Time passes, and the system continues to work, yet is increasingly becoming outdated.

Now, x years after the system has been set up; enter a new official keen to update outdated systems. This can go either way: the migration to the new system works, or it doesn't. If the migration does not work, massive negative PR ensues. If the migration works, the new system works just as intended as the old system did. In the best case, there are no side-effects; in the worst case, the official gets reprimanded for wasting money on updating a system which still did its job.

Politicians usually don't get praise, when nothing is happening. Social security didn't break down yesterday, good job!

Answer 9

The only reason this is a political question at all is because the services are important and such a change is expensive.

Unfortunately the political process is pretty much exactly wrong process for deciding how and when to do such a migration. There isn’t a constituency for it, there’s no immediate benefit. Also, because of the above the projects that do get undertaken have a tendency to fail, which raises yet another barrier to doing so.

Answer 10

It's not just government institutions, any private company that is old will have the same issues, e.g. Ticketmaster: https://www.cio.com/article/3448036/ticketmaster-tackles-tech-debt-with-streaming-data-platform.html

We had tech debt that was older than most of the companies I’d worked at,” he says.

Ticketmaster had achieved early data-science successes through custom data integrations with its various IT systems — and there were plenty of those. After 40 years of acquisitions and internal software development, the company had around 300 IT systems, each on its own island of data.

I happened to meet a guy at a tech Meetup, who worked at TicketMaster and he said they don't even dare breathe on the old code that's at the core of their ticket allocation algorithm, because it's so complicated that no-one is sure they understand all the intricacies and edge cases and they just emulate the hardware it was designed to run on.

One of my favorite parts of Vernor Vinge's awesome sci-fi novel "A Deepness in the Sky", is that the most in-demand, yet demanding profession is "Programmer Archaeologist", i.e. someone who can dig through 5 thousand years worth of software dependencies and have a smidgen of a clue of what it's actually doing:

There were programs here that had been written five thousand years ago, before Humankind ever left Earth. The wonder of it—the horror of it, Sura said—was that unlike the useless wrecks of Canberra’s past, these programs still worked! And via a million million circuitous threads of inheritance, many of the oldest programs still ran in the bowels of the Qeng Ho system. Take the Traders’ method of timekeeping. The frame corrections were incredibly complex—and down at the very bottom of it was a little program that ran a counter. Second by second, the Qeng Ho counted from the instant that a human had first set foot on Old Earth’s moon. But if you looked at it still more closely. . .the starting instant was actually some hundred million seconds later, the 0-second of one of Humankind’s first computer operating systems.(i.e. they still run on Unix time, presumably the Year 2038 problem has been solved)

Answer 11

Budgets, if you want the most simple answer. While less efficient on a longer-term timescale, maintaining old code is cheaper on a year to year basis, and legislative budgets, authorizations, funding and appropriations operate on a yearly basis. When projects span more than a single government fiscal year, if money appropriated for one budget year isn't spent by the end of that year (what project, government or private, doesn't run behind schedule), those funds are often lost unless specially "encumbered" or authorized to carry over into the next fiscal year. With every two-year election cycle, there runs the danger that the powers that control the purse strings might change to someone who does not view that kind of spending as necessary, or a priority.

It's very difficult to get politicians who are concerned with their short-term popularity to approve something that will show a fiscal benefit by the six, seventh or eighth year after implementation, with huge up-front price tags, inevitable cost-overruns (which will be paid, because once you sink X dollars/Euros in, you're committed to seeing it through) and glitches when these massive, complex systems are brought online.

There is no competition or economy of scale to bring to bear because these systems are mostly custom-built to perform very singular functions. As long as they can keep something functioning with bale wire and spit, they will do so until they absolutely can't.