Connection Issues to Files Servers in Australia

Incident Report for Files.com

Postmortem

On October 20th, 2022, starting at 4:15 PM PST, Files.com started receiving reports of issues connecting to all services in the Australia storage region. The Customer Support team was able to recreate the reported issues and correlate multiple tickets as the same issue, and on October 20th, 2022, at 4:34 PM PST declared an Incident due to the scope of the impact on multiple customers. The scope of customer impact was all services in the Australia storage region.

The Incident Management Team (IMT) immediately began investigation and determined that DNS entries for the Australia region were not populating as intended, sending all interactions in the Australia storage region to a non- functioning legacy server. On October 20th, 2022, at 5:16 PM PST, the IMT deployed a change forcing the DNS entries to register properly and point to the correct functioning proxy servers, and were able to verify that the Australia storage region was back online. Customer Support then reached out to the customers who had submitted support tickets and ensure their testing validated the fix, confirming the RTO time listed above.

During the Incident Postmortem meeting, the root cause was determined to be a combination of factors that caused the DNS issue. These factors included a deployment to the production proxy servers, as well as a legacy proxy server that had been recently removed, but the DNS entry for that legacy system impacted the deploy in an unexpected way. Several improvements were identified in both processes and system changes, including several new complex monitors, which have been submitted to the engineering teams for completion.

We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Posted Nov 07, 2022 - 14:47 PST

Resolved

All services have been restored and are operating normally. As of 5:16 PT, a fix has been implemented and we will continue to monitor the results. We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause. If you need additional support, please do not hesitate to contact our Customer Success team by email or phone. Thanks for your support while we resolved this issue.
Posted Oct 20, 2022 - 17:21 PDT

Investigating

We are actively investigating connection issues for users connecting to servers in Australia. We will provide additional details as they become available. Customers with urgent questions are encouraged to contact our Customer Support team by email. Thank you for your patience.
Posted Oct 20, 2022 - 17:00 PDT
This incident affected: Core Services / API, Web Interface, FTP/FTPS, and SFTP.