Zip Archive Creation Failures [US-East]
Incident Report for Files.com
Postmortem

On April 13th, 2023, at 11:29 AM PST, Files.com received automated alerts of zip archive creation errors in the US East region, which resulted in an incident being declared.  The Incident Management Team (IMT) convened and immediately began investigation. 

Files.com released an initial Status Page posting on April 13th, 2023, at 11:48 AM PST,  stating: 

Zip Archive Creation Failures [US-East]:  We are currently investigating issues with zip archive creation on US-East Servers. Only accounts with custom domains or which have enabled security opt-out settings will be impacted by this issue. On affected accounts, users who are trying to download multiple files at a time through the web interface will not be able to complete the download.” 

The zip archive creation errors were resolved on April 13th, 2023, at 11:49 AM PST, returning the platform to full functionality. 

Files.com released a resolution Status Page posting on April 13th, 2023 at  11:52 AM PST, stating:  

“All services have been restored and are operating normally.  As of 11:49 a.m. all accounts should again be able to download multiple files at a time through the web interface. Zip archive creation for these files is occurring as normal.”  

This change began when a security patch was deployed to our HTTP server configuration on our production proxy servers.  Due to a mistake in the patch configuration, source IP addresses did not get properly tagged for certain customers who are using both custom domains and our “insecure ciphers” feature that enables older TLS versions for compatibility.    

Due to the error in IP tagging, ZIP downloads did not function for customers due to a perceived IP mismatch at the ZIP service. 

The root cause of this issue was inadequate code review and testing of the security patch prior to production deployment.  This issue occurred only in a rare combination of customer environments, but it is still our responsibility to test every possible combination. 

Extensive review and testing was conducted by Files.com staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future.   

We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Posted May 15, 2023 - 10:33 PDT

Resolved
All services have been restored and are operating normally.

As of 11:49 a.m. all accounts should again be able to download multiple files at a time through the web interface. Zip archive creation for these files is occurring as normal.

We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause. If you need additional support, please do not hesitate to contact our Customer Support team by email or phone. Thanks for your support while we resolved this issue.
Posted Apr 13, 2023 - 11:52 PDT
Investigating
We are currently investigating issues with zip archive creation on US-East Servers. Only accounts with custom domains or which have enabled security opt-out settings will be impacted by this issue. On affected accounts, users who are trying to download multiple files at a time through the web interface will not be able to complete the download.

We will provide additional details as they become available. Customers with urgent questions are encouraged to contact our Customer Support team by email. Thank you for your patience.
Posted Apr 13, 2023 - 11:48 PDT
This incident affected: Web Interface.