California Based startup –GitLab was in utter catastrophe after experiencing data loss as a result of what it has suddenly discovered are ineffectual backups. A series of human errors caused as the service to go down overnight. GitLab lost 6 hours of database, data including issues, merge requests, users, comments, snippets, etc.
Gitlab is similar to gitbub and Bitbucket, where companies store their code repositories. GitLab is supported by Khosla ventures and Y combinator.
On Tuesday evening, the startup issued a sobering series ocomplaints on the internet. The system admin was working late at night from Netherlands, had accidentally deleted a directory on the wrong server during a frustrating database replication process. He wiped a folder containing 300GB of live production data that was due to be replicated. Just 4.5GB remained by the time he realized and canceled the rm-rf command. The last potentially viable backup was taken six hours beforehand.
GitLab is an open-source code collaboration platform that enables developers to create and manage code bases collaboratively. The platform enables developers to create, review, and deploy code bases. It is built to run on clients’ infrastructures. GitLab was launched in 2014 and is operated from San Francisco, California. The total equity funding of the startup is $ 26.62 Million.
While in the process of restoring that older version of the database, the site went completely down for at least six hours which worsened and led to intermittent failures. To ge the service back online it took another several hours.
However, the good news is,, the data which accidentally got erased didn’t actually contain anyone’s code, just the data like comments and bug reports got erased. The customers who installed GitLab’s software on their own servers weren’t affected too and the most importantly the paying customers were also not affected at all. The crisis minimized the financial impact.
When this entire crisis was taking place, GitLab team was active on social media and explaining the situation to their customers and apologizing for the inconvenience.
As for the systems administrator who made the mistake, the VP of Gitlab is hesitant to place blame, since it was really the whole team’s fault as none of their other backup systems were working. At the time of writing this post, Gitlab is back to work, But not sure how much data is lost.
Read Startup news.