Intermittent Connectivity Issues in Frankfurt Data Center

Incident Report for Files.com

Postmortem

On August 25th, 2022, at 8:13 PM PST, Files.com received automated alerting identifying issues with a specific server subsystem in the Frankfurt, Germany region. The alerts indicated issues with all connectivity. The Infrastructure team began immediate investigation and based upon the affected services, called an incident on August 25th, 2022, at 8:24 PM PST.

The Incident Management Team (IMT) immediately began an in-depth investigation and quickly identified two servers that were offline. Those two servers were restarted and all connectivity was restored on August 25th, 2022, at 8:39 PM PST. The IMT continued to investigate the specific cause of the servers to go offline, however nothing was shown in the logs or monitoring metrics. The IMT decided to add additional logging on the two servers, with the theory that what caused the systems to trip offline occurred before the process could be written to the operational logs.

The automated alerting triggered on August 26th, 2022, at 1:04 AM PST showing the two servers offline again. The paged Infrastructure engineer immediately restarted the systems and had connectivity restored on August 26th, 2022, at 1:09 AM PST. FIles.com received the first of three customer initiated tickets on this problem on August 26th, 2022, at 1:03 AM PST. The additional logging did not pick up the process responsible for tripping the servers offline at this time.

The automated alerting triggered on August 26th, 2022, at 5:01 AM PST showing the two servers offline again. Both servers were immediately restarted and connectivity restored on August 26th, 2022, at 5:08 AM PST. The IMT could not find anything in the logging that indicated why the servers had tripped offline. A decision was made to add two additional servers in the Frankfurt, Germany region to see if that would allow connectivity to stay operational should the two servers trip offline again. The two additional servers were online at operational on August 26th, 2022, at 5:46 AM PST.

The automated alerting triggered on August 26th, 2022, at 9:02 AM PST showing two servers offline again. Connectivity was not interrupted at this time due to the additional servers added previously. The IMT was able to identify the process which was severely spiking the servers memory and causing them to trip offline. That process identification led to the discovery of a customer sync job to OneDrive that would trip the servers offline. The sync job was disabled on August 26th, 2022, at 9:22 AM, which stopped the offline behavior and allowed the declaration of RTO. All three customers who had submitted tickets were contacted, and they verified full functionality.

An enhancement was identified to the particular servers which would prevent them from tripping offline should this particular sync job behavior occur in the future. The enhancement was tested and deployed to production on August 26th, 2022, at 1:11 PM PST.

During the incident postmortem, discussion took place about the timeliness of Files.com Status Page updates to keep customers better informed. A more formalized process is being developed to ensure timeliness of the Status Page updates. A backup logging process has been identified and is currently in development that would have been able to catch the process that was tripping the servers the first time. Once tested, that new process will be deployed.

We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Posted Sep 12, 2022 - 08:50 PDT

Resolved

All services have been restored and are operating normally. As of 09:22 PST, a fix has been implemented and we will continue to monitor the results. We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause. If you need additional support, please do not hesitate to contact our Customer Support team by email or phone. Thanks for your support while we resolved this issue.
Posted Aug 26, 2022 - 09:41 PDT

Update

We've added capacity and enhanced monitoring around the impacted systems. We will continue to actively monitor the situation to confirm resolution. If you need additional assistance, please do not hesitate to contact our Customer Support team by email. Thank you for your patience.
Posted Aug 26, 2022 - 07:00 PDT

Investigating

We have been actively investigating an intermittent connectivity issue in our Frankfurt data center. We will provide additional details as they become available. Customers with urgent questions are encouraged to contact our Customer Support team by email. Thank you for your patience.
Posted Aug 26, 2022 - 05:48 PDT
This incident affected: Core Services / API, Web Interface, FTP/FTPS, SFTP, WebDAV, Background Processing, including Notifications and Webhooks, and Remote Server Integrations.