Does it matter if software is unreliable and insecure?

I put a lot of effort into making software, and websites in particular, reliable, secure, and maintainable. I am beginning to question whether there is any real benefit to doing so. There is little correlation between the quality of software, and whether users are willing to use or how much money it makes.

The immediate trigger for writing this is recent events, that I might explain in a blog post as well as including here, but it reflects years of observing poor quality, often horribly insecure, code running for years. People in software frequently bemoan this (looking at you, Hacker News, as well bloggers and social media posters). I have been complaining about the prevalance of code quality problems for years myself.

It aint broke

The problem is that software seems to run for years without “essential” updates: with insecure code and annoying bugs, and everyone seems to be perfectly happy with it. Even serious security issues just do not ever seem to happen in real life. My experience is primarily with small websites and apps, but these are custom code so I do not usually see things that are just Wordpress brochure sites and the like. I will start with these, and move on to big business later.

However, Wordpress is a great place to begin, because it is widely used. I know lots of people who use it, and many over the years, and pay little if any attention to security. They do not update Wordpress, and they do not update the OS in their VPS. They assume that AWS does all the updating required for their “instance”, or they think that the managed VPS includes application updates as well as OS updates. How many of these have been compromised. One, ever, compromised though an oout of date wordpress install, and one other with an OS compromise (and, of course, that was the person running a hardened kernel) and I have been doing this a long time.

Its far from being just Wordpress. Over two decades ago my very first customer asked me to look at his (entirely custom PHP code) site, and I was immediately appalled. The site had multiple forms, and no escaping at all before the forms were sent to the database. SQL injection attacks were trivial. The site had been running for years without an incident. I inisted the site needed an upgrade, of course. Last year I found a site, that has again been running for many years, calling Python’s exec on user input in two places.

Right now, not long after the highly publicised PHP exploit do you think the average small business with a site on a cloud platform or a VPS will have updated? Something that was probably not that secure because it was set up a “web designer” who used a Wordpress plugin for every bit of functionality and a drag and drop theme. Those with shared hosts will get updates. Those with managed servers/VPSes will get the OS part of the update, but, usually, those services are not got to the the application code is updated or using the right PHP version. Everyone else will have an unsupported version of everything: OS, database, language libraries, application code…

When I get a new client (usually Python + Django on some form of Linux) it is usually safe to assume no one has done security upgrades. Its also usually safe to assume that something will not have been upgraded for years, and very often nothing has been. The best I see is an OS configured for automatic updates. In just the last few weeks I upgraded a system from Django 3.2, and there are definitely much older systems out there.

So with all this, where are the waves of compromised sites? Anyone who admins a web server sees the constant probles for weaknesses (e.g. requesting things such as /wp-admin or /.env ) but there never seem to be consequences. Maybe a lot of this is coming from bots and is meant for follow up if they find a site worth attacking.

The same goes for quality. Lots of sites have slightly broken functionality or lots of usability rough edges and lots of happy customers. Of course people are more interested in the product than the website selling it, but this seems to apply even when the software is the product. Some of it is network effects (e.g. Facebook, Microsoft) so people will complain about it but keep using it.

It happens to big business

That takes me neatly on to bigger businesses. A lot of people will no doubt be thinking these are just small businesses with limited technical skills. Its not. Microsoft was responsible for a breach of US military email. Tesco Bank did not store passwords securely (for those unfamiliar with the issue, you do not actually store the password itself if you are remotely competent). Between them Google and an Australian pension fund managed to delete all the latter’s data. Cloudstrike managed to crash 8.5 million different systems.

The last is particularly interesting for two reasons. The first is that it required mistakes (in my opinion incompetence or carelessness) on both sides for it to happen. Insufficient testing on the part of Cloudstrike, and also on the part of their customers. Yes, a few organisations can smugly tell you that they deployed it to their staging environment, found the bug, and held of deploying it to production and so avoided taking down their systems. However, everyone else (countless organisations running literally millions of systems) simply did not bother with the basics. Again, it would be excusable in a small business that does not have staging environments, but this affected a huge range of businesses from airlines to hospitals.

The other is that it seems to have no consequences. Cloudstrike did not lose sales or customers as a result. The same seems to be true for big tech too. People just keep using them because they are the “safe” option. This is not because they are the safe option in operational or technical terms (i.e. the lowest risk of security compromises for failures to operate or bugs) . They are the safe options in career terms. You will not get fired for making the same mistake as everyone else using AWS us-east-1 when it fails yet again, nor for using Github (which currently seems to have outages weekly). In the 1970s it was said “no on ever got fired for buying IBM”. the attitude is the same, but the names have changed.

What it means for me

Going back to my own concerns. Am I wasting my time worrying about quality? Maintainability pays because saves time later on, but I am not sure the people paying for the maintenance know or care. As for bugs, people just work around them. When a big website or their Windows PC is difficult to work with what do people do? They complain, but they do not take their business elsewhere. They accept that when systems do not work them need to work around them.

As for security and reliability, out entire software stack is complex and fragile. No one wants to put money into making even the fundamentals secure and reliable. Features and performance win every time, and you can see this in the design choices made in every layer (and, oh, there are so many layers now) from hardware to apps. You can tell people a system is fragile or insecure or not user friendly or hard to work with, and they will happily go ahead and fix the entirely predictable problems when they occur.

I care, and I want to make the effort, but is it worth it? If no one else does? Yo be honest, most of what I do is a small component driving a complex system of hardware and software. Many millions of lines of kernel code, even more user space code, hardware containing billions of transistors. It is the layer I can control, and for anything connected to a network the place where failure is most likely. It seems out of sync with what everyone else does, or wants to do. It is a Canute ¹ holding back the tide.