Outbound SFTP and FTP Issues in Frankfurt and London Data Centers
Incident Report for Files.com
Postmortem

On August 15th, 2022, at 3:20 AM PST, Files.com Customer Support received a customer support email indicating issues with a failed sync, followed 30 minutes later by a second customer report with failed sync issues. The Customer Support team started researching both issues and found a common thread between the two tickets. On August 15th, 2022, at 6:32 AM PST, Customer Support initiated an Incident based upon the impacts to outbound FTP and SFTP on customer syncs in the Frankfurt and London data centers.

The Incident Management Team immediately began an in-depth investigation and quickly identified that a security certificate on one of our main management systems was not updated, which blocked the systems which provided outbound FTP and SFTP functionality to customer syncs from working properly. The security certificates were quickly updated in the Frankfurt and London data centers which restored functionality on August 15th, 2022, at 7:07 AM PST. The customers who submitted Help Scout support tickets were contacted, and they verified that their sync issues were resolved.

The IMT continued to update all the security certificates in all data centers in every region and was completed on August 15th, 2022, at 7:15 AM PST. The root cause of this incident was an automated alert letting internal staff know the security certificates were within a week of expiration that did not function.

During the incident postmortem, two new monitoring alerts were proposed. The first alert would identify the soon to expire certificates, allowing notification to internal staff at least a week in advance. This replaced the non-functioning alert identified as the root cause. The second new alert will notify internal staff should the errors from an expired certificate be seen. Both new alerts are now in place and operational.

We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Posted Aug 16, 2022 - 08:13 PDT

Resolved
All services have been restored and are operating normally. As of 7:15 AM, a fix has been implemented and we will continue to monitor the results. We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause. If you need additional support, please do not hesitate to contact our Customer Success team by email or phone. Thanks for your support while we resolved this issue.
Posted Aug 15, 2022 - 07:23 PDT
Investigating
We are actively investigating an issue with Outbound SFTP and FTP in the Frankfurt and London data centers. We will provide additional details as they become available. Customers with urgent questions are encouraged to contact our Customer Success team by email. Thank you for your patience.
Posted Aug 15, 2022 - 07:01 PDT
This incident affected: FTP/FTPS and SFTP.