On April 12th, 2023, at 10:37 AM PST, Files.com received automated alerting of elevated error rates, which resulted in an incident being declared. The Incident Management Team (IMT) convened and immediately began investigation.
Files.com released an initial Status Page posting on April 12th, 2023, at 10:59 AM PST, stating Performance Degradation and/or High Error Rates: We are investigating a performance degradation and/or elevated error rates.
The elevated error rates were resolved on April 12th, 2023, at 11:31 AM PST, returning the platform to full functionality.
Files.com released a resolution Status Page posting on April 12th, 2023, at 11:41 AM PST, stating:
“All services have been restored and are operating normally. We have resolved a performance degradation on Files.com which affected certain API calls as well as uploads to Remote Server Mounts. This incident occurred between the times of 10:29 PST to 11:33 PST.”
This incident was began when an internal code change was deployed to the part of our infrastructure that initiates outbound connections for sync and mounts. This code change resulted in unexpected data being sent from those services to our API services which caused 500 Server Errors on the API, preventing requests from being fulfilled.
The unexpected data was simply additional data added to the payload, not any incorrect or corrupt data.
The root cause of this incident was Files.com’s failure to design its API to operate correctly even in the event of unexpected additional data arriving from another internal service. The relevant internal code libraries have been updated to successfully handle this sort of situation in the future without any customer impact. Because Files.com uses one primary library for inter-service communication, this update will be applied across the entire Files.com infrastructure.
Extensive review and testing was conducted by Files.com staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future.
We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.