Slower Performance Speeds

Incident Report for Files.com

Postmortem

On June 6th, 2022 at 1:01 PM PST, Files.com received several high priority alerts by several of our monitoring systems of extremely high API usage, resulting in slower response times across the entire Files.com platform.

Our Engineering team began investigation upon receiving these alerts through PagerDuty and determined the scope of the event and called an incident on June 6th, 2022 at 1:17 PM PST.

Our Incident Management Team immediately started looking at logs and identified the issue. A customer had an automated script that was using SFTP incorrectly, resulting in uploading multiple files in parallel (not the root cause) and then was incorrectly pulling a new LIST command right after the upload on a directory with 10’s of thousands of files. These list commands were running in parallel at the same time as the uploads, which resulted in the API servers being overloaded.

We disabled the customer’s user account that was running the bad script which returned the system back to normal functionality (RTO) on June 6th, 2022 at 3:15 PM PST.

Customer Support worked with the specific customer to adjust their SFTP script to account for the SFTP limitations with multiple uploads and LIST commands in parallel. The fix was tested with the customer and verified to be working correctly, which allowed us to mark WRT on June 7th, 2022 at 1:22 PM PST.

During our internal Postmortem meeting we identified multiple areas for process, testing, monitoring and platform improvements, which have been identified and added to our development process, and the process improvements have been implemented in our code deployment procedures.

We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Posted Aug 08, 2022 - 08:58 PDT

Resolved

All services have been restored and are operating normally. As of 3:30 p.m. PDT, a fix has been implemented and we will continue to monitor the results.

We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause.

If you need additional support, please do not hesitate to contact our Customer Success team by email or phone. Thanks for your support while we resolved this issue.
Posted Jun 06, 2022 - 15:34 PDT

Investigating

We are actively investigating performance speed issues. All services are available, but may be running slower than normal. We will provide additional details as they become available. Customers with urgent questions are encouraged to contact our Customer Success team by email. Thank you for your patience.
Posted Jun 06, 2022 - 13:00 PDT
This incident affected: Core Services / API, Web Interface, FTP/FTPS, SFTP, WebDAV, Background Processing, including Notifications and Webhooks, and Remote Server Integrations.