On June 6th, 2022 at 1:01 PM PST, Files.com received several high priority alerts by several of our monitoring systems of extremely high API usage, resulting in slower response times across the entire Files.com platform.
Our Engineering team began investigation upon receiving these alerts through PagerDuty and determined the scope of the event and called an incident on June 6th, 2022 at 1:17 PM PST.
Our Incident Management Team immediately started looking at logs and identified the issue. A customer had an automated script that was using SFTP incorrectly, resulting in uploading multiple files in parallel (not the root cause) and then was incorrectly pulling a new LIST command right after the upload on a directory with 10’s of thousands of files. These list commands were running in parallel at the same time as the uploads, which resulted in the API servers being overloaded.
We disabled the customer’s user account that was running the bad script which returned the system back to normal functionality (RTO) on June 6th, 2022 at 3:15 PM PST.
Customer Support worked with the specific customer to adjust their SFTP script to account for the SFTP limitations with multiple uploads and LIST commands in parallel. The fix was tested with the customer and verified to be working correctly, which allowed us to mark WRT on June 7th, 2022 at 1:22 PM PST.
During our internal Postmortem meeting we identified multiple areas for process, testing, monitoring and platform improvements, which have been identified and added to our development process, and the process improvements have been implemented in our code deployment procedures.
We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.