Chip downtime. What actually happened?

At around 11pm on Friday evening we ran a maintenance script, the primary goal of which is to prevent downtime. Ironically, it caused our database to malfunction. We followed procedure and switched it off. However, it had entrenched several bugs in the stack which in turn caused numerous disruptions to Chip throughout the night until approximately 8:00 am Saturday morning when the issues were finally resolved.

To make matters worse, we had an unexpected spike of user activity on Friday evening and the server struggled to cope with the maintenance issue and the increase in volume. Sod's law as they say - the perfect storm of problems.

Has the issue now been resolved?

I would like to reassure you that the issue has now been resolved and we won't be running that maintenance script again! I can also confirm that all personal data and customer savings were completely secure at all times.

Our promise to you

It is never acceptable for our savers to be without access to their savings  — period. We have learnt a great deal from this experience and we are actively working on implementing a number of processes to constantly improve our technical infrastructure and eliminate incidents like this.

Thank you for your trust and continued support.

Simon

face.png