A few weeks ago an unbelievable story made the rounds in tech and even got some mainstream attention: the Australian superannuation fund UniSuper, which has over $100bn AUD under management, almost would have gone down. They were only saved because they had backups of their data on cloud providers not called Google. Google speaks of an “unprecedented sequence of events” that led to the deletion of the entire account of UniSuper, i.e. all their data. I have my own hypotheses regarding this alleged misconfiguration.

Years ago I worked for Pajeet Soft but got out quickly. There, the most unbelievable event happened, easily about as unbelievable as the aforementioned accident at Google. This was during the scamdemic. We were happily working from home, putting effort into our Nintendo Switch games while writing a bit of code on the side. At one point, I got a message from a colleague on my work phone. It could be summarized as ,”The fuck just happened?” He was forcibly logged out and no longer able to log into his computer. Minutes later, his work phone got shut down, too. Minutes later, I got kicked out of the system, and in private chats messages, exchanged via our private phones, we were wondering whether Pajeet Soft had just gotten shut down for good, US-style. This was not true, however. Within a few hours, some people were able to log in again, and within a few more days, all user accounts were restored. The cleanup took two to three weeks, and even afterwards there was the occasional message on internal chat channels when people realized that they no longer had access to some part of the system they rarely worked with but now needed to make some changes to.

Pajeet Soft also suffered from a “terrible configuration error”, and if that error had become public, they could have gotten fined massively as they were active in a regulated industry. The root cause was that Rajesh Kumar, a high-ranking “DevOps” engineer, made some changes he did not really understand. By literally ticking the wrong checkbox he disabled every user account that did not have administrative powers. There was a “four-eye principle” in place so before this change got live, meaning that it had to be approved by a colleague first. Pajeet Kumar was happy to do that. He readily approved this change, and then all hell broke loose.

Within a little less than about 30 minutes, the all-star team at Pajeet Soft realized that something was not quite right and they worked on finding a solution. Meanwhile, they kept employees updated via emails (which nobody could access) and posts on an internal Slack channel (yes, exactly). I have never seen something so idiotic unfold. Worse was that this incident was never acknowledged. They just pretended it did not even happen and even the internal post-mortems were manipulated to downplay the impact or make up bogus reasons. The internal records were falsified, i.e. there was a mandate to record step-by-step what you did to resolve an issue. These records were basically fiction. Eventually, this was simply an unavoidable error nobody could have prevented, let alone predicted, but thanks to the heroic efforts of the all-star Pajeets, it was fixed as quickly as possible, and business impact was negligible.

My completely outlandish hypothesis is that Google has geniuses of a similar caliber on their payroll. These people use printed notes or text files with step-by-step instructions to do their jobs. You have to see it to believe it. They understand hardly anything they do. I would bet money that this explains the “misconfiguration” at Google that almost would have killed a $100bn+ company, and would have done so had they blindly trusted Google.

People seem to not understand that the “cloud”, in the context of computing, is basically just a computer that is owned and maintained by someone else, and you pay for that. It is similar to renting an apartment versus owning one. If you rely on the “cloud”, you may be in for a big surprise because you have no idea about who is maintaining these systems. A lot of critical tasks could be outsourced to countries not know for their high quality of work, for instance. From Edward Snowden we furthermore know that the US-based cloud providers are complicit in industrial espionage, and also do not protect your private data. Your data never really gets deleted. Facebook has a flag for deletions, which means that such data is gone from your profile but still on their server. They just no longer show it to you. Recently, there was also a “bug” affecting Apple devices. Suddenly supposedly deleted pictures kept reappearing. This is not even a new phenomenon because similar “bugs” were reported years ago already. I am quite certain that the “bug” is that images are only flagged as being deleted while the data is still on iCloud, the name of their cloud-storage solution, and kept there. Apple tells us that iCloud is not to blame but you the PR cleanup does not seem to address all issues people have reported. It’s just a “glitch“, goy!