On October 20th, 2022, starting at 4:15 PM PST, Files.com started receiving reports of issues connecting to all services in the Australia storage region. The Customer Support team was able to recreate the reported issues and correlate multiple tickets as the same issue, and on October 20th, 2022, at 4:34 PM PST declared an Incident due to the scope of the impact on multiple customers. The scope of customer impact was all services in the Australia storage region.
The Incident Management Team (IMT) immediately began investigation and determined that DNS entries for the Australia region were not populating as intended, sending all interactions in the Australia storage region to a non- functioning legacy server. On October 20th, 2022, at 5:16 PM PST, the IMT deployed a change forcing the DNS entries to register properly and point to the correct functioning proxy servers, and were able to verify that the Australia storage region was back online. Customer Support then reached out to the customers who had submitted support tickets and ensure their testing validated the fix, confirming the RTO time listed above.
During the Incident Postmortem meeting, the root cause was determined to be a combination of factors that caused the DNS issue. These factors included a deployment to the production proxy servers, as well as a legacy proxy server that had been recently removed, but the DNS entry for that legacy system impacted the deploy in an unexpected way. Several improvements were identified in both processes and system changes, including several new complex monitors, which have been submitted to the engineering teams for completion.
We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.