FTP, SFTP, and WebDAV Only: Elevated Error Rates

Incident Report for Files.com

Postmortem

At 8:31 AM PST on September 5th, Files.com made a routine code deployment which introduced a bug preventing SFTP, FTP, and WebDav operations from completing. Files.com detected this issue at 8:36 AM and reverted a change restoring the SFTP, FTP, and WebDAV services at 8:40 AM PST.

Additionally a small number of customers experienced a continuation of authentication failures for sessions that were incorrectly cached as failures. Files.com received an escalated report of this problem and resolved it at 9:36 AM PST by clearing the login caches for SFTP, FTP, and WebDAV connections.

The elevated error rates during this period were caused by an update to our internal service authentication to add authentication to a new service. While this update was needed to provide new connectivity for services, it introduced a regression for SFTP, FTP, and WebDav’s internal authentication methods. This problem was promptly detected by the Files.com monitoring and alerting services and we immediately began remediation.

The root cause of this incident was Files.com’s insufficient testing of a change that was deployed. We have updated the testing that failed to identify this issue to improve our future delivery.

We promise a system that works perfectly, all of the time, and today we failed to deliver that to you. Our entire engineering team is working hard to prevent issues like this one from occurring in the future.

If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Posted Sep 10, 2024 - 15:45 PDT

Resolved

We have resolved an incident causing elevated error rates on the FTP, SFTP, and WebDAV services on Files.com in all regions. This incident did not impact other network services such as our API, AS2, or any others. This incident occurred between the times of 8:31am and 8:40am Pacific Time.

We are compiling a final Root Cause Analysis for this incident, which we will post here when it is complete.
Posted Sep 05, 2024 - 08:30 PDT