Connection Errors / Login Failures
Incident Report for Files.com
Postmortem

5-16-19 Incident Report

The Event

On May 16th, 2019 at ​01:55 AM PST​, the Files.com system started running very slowly affecting all operations including API calls.

The proactive monitoring system for Files.com alerted our staff who immediately responded to the alert. During the response the problem was identified to be a missing temporary directory that was removed during critical vendor security patch updates on several servers. This behavior was not seen during testing.

The missing temporary directory was identified and corrected at ​03:24 AM PST​. Once corrected, Files.com was fully operational.

At no time was customer data affected. The missing temporary directory was on server infrastructure, not in the data storage system.

Lessons Learned

Files.com has implemented a script upon reboot that checks for all the required temporary folders and recreates them if needed. Testing of critical patch updates will continue as before, even though testing did not reveal this issue prior to rolling out the critical vendor security patch.

Posted May 30, 2019 - 04:14 PDT

Resolved
Connectivity issue identified. System restored to operational and being monitored. We are conducting a thorough investigation and will have more details once that process has been completed. We are continually improving our system, and we apologize for the inconvenience.
Posted May 16, 2019 - 04:06 PDT
Monitoring
Connectivity issue identified. System restored to operational and being monitored. We are conducting a thorough investigation and will have more details once that process has been completed. We are continually improving our system, and we apologize for the inconvenience.
Posted May 16, 2019 - 04:06 PDT
Identified
We are actively investigating increased error count on the core Files.com / BrickFTP application, API, and FTP, SFTP, and WebDAV interfaces. We will post more as we are able to share more. First notification received at 2:09 AM PST.
Posted May 16, 2019 - 02:06 PDT
This incident affected: Core Services / API, Web Interface, FTP/FTPS, SFTP, WebDAV, and Background Jobs, including Sync and Webhooks.