On August 15th, 2022, at 3:20 AM PST, Files.com Customer Support received a customer support email indicating issues with a failed sync, followed 30 minutes later by a second customer report with failed sync issues. The Customer Support team started researching both issues and found a common thread between the two tickets. On August 15th, 2022, at 6:32 AM PST, Customer Support initiated an Incident based upon the impacts to outbound FTP and SFTP on customer syncs in the Frankfurt and London data centers.
The Incident Management Team immediately began an in-depth investigation and quickly identified that a security certificate on one of our main management systems was not updated, which blocked the systems which provided outbound FTP and SFTP functionality to customer syncs from working properly. The security certificates were quickly updated in the Frankfurt and London data centers which restored functionality on August 15th, 2022, at 7:07 AM PST. The customers who submitted Help Scout support tickets were contacted, and they verified that their sync issues were resolved.
The IMT continued to update all the security certificates in all data centers in every region and was completed on August 15th, 2022, at 7:15 AM PST. The root cause of this incident was an automated alert letting internal staff know the security certificates were within a week of expiration that did not function.
During the incident postmortem, two new monitoring alerts were proposed. The first alert would identify the soon to expire certificates, allowing notification to internal staff at least a week in advance. This replaced the non-functioning alert identified as the root cause. The second new alert will notify internal staff should the errors from an expired certificate be seen. Both new alerts are now in place and operational.
We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.