NOTES

Google Accidentally Deletes $135 Billion Pension Fund

Alternate Title: Why Backups Are A Necessity

By Paul DiMaggioAugust 12, 20242 min read

Summary

UniSuper is an Australian “superannuation fund manager” that manages around $135 billion Australian dollars for ~620,000 Australians. UniSuper’s online platform was migrated to Google’s Cloud infrastructure in 2023 by Google engineers using an internal Google infrastructure deployment tool. Google’s engineers neglected to modify the default behavior of this type of deployment, so UniSuper’s infrastructure was configured to automatically delete itself in about 12 months. It did so on May 2, 2024. Luckily, UniSuper had backups on that could be restored.

Breakdown

  • Google engineers used Google Cloud's VMware engine to migrate UniSuper’s platform to the Google Cloud
  • Google’s engineers accidentally left default behavior of 1 year expiration in place. This led to the deletion of UniSuper’s entire infrastructure.
  • Unisuper manages around 135 billion Australian dollars for over 600,000 Australians.
  • Employers in Australia must contribute 11.5% to a superannuation account.
  • Unisuper had backups another cloud provider.
  • The incident caused significant downtime and customer complaints.

Recommendations for Businesses

  • Backups are essential for disaster recovery and operational resilience.
  • The 3-2-1 backup rule: three copies, two types of media, one offsite
  • Make sure you actually test that your backups can be restored
  • Document all processes and configurations meticulously.
  • Regularly review and update default settings in tools.
  • Maintain continuity by retaining critical staff or knowledge transfer processes.
  • Use insurance as a safety net for unexpected disasters.
  • Continuously improve operational resilience through regular audits.