On April 20th, 2023, at 11:57 AM PST, Files.com received customer reports of 2FA login errors which resulted in an incident being declared. The Incident Management Team (IMT) convened and immediately began investigation.
Files.com released an initial Status Page posting on April 20th, 2023, at 11:57 AM PST, stating:
“Reports of Elevated Errors - 2FA and SSO Logins: We are investigating reports of errors on the Files.com service. This is affecting user logins on the web portal that use two-factor authentication or single-sign on.”
The 2FA login errors were resolved on April, 20th, 2023, at 11:58 AM PST, returning the platform to full functionality.
Files.com released a resolution Status Page posting on April 20th, 2023, at 12:08 PM PST stating:
“All services have been restored and are operating normally. We have resolved an issue that was causing web logins to fail when using single sign-on or two-factor authentication. This incident occurred between the times of 11:28 to 11:58 a.m.”
This incident began when our frontend team deployed a code change designed to fix a bug in the login process in the Files.com web interface. Unfortunately, the code change actually made things worse and prevented 2FA or SSO logins from working at all. This change was rolled back as soon as the incident was identified.
The root cause of this incident was Files.com’s failure to properly identify a change to the login page as a high-risk change and implement a sufficient level of automated and manual testing prior to deployment. Files.com will be conducting remedial training on change risk identification at its upcoming Engineering meeting.
Extensive review and testing was conducted by Files.com staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future.
We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.