Reports of Elevated Errors - 2FA and SSO Logins

Incident Report for Files.com

Postmortem

On April 20th, 2023, at 11:57 AM PST, Files.com received customer reports of 2FA login errors which resulted in an incident being declared.  The Incident Management Team (IMT) convened and immediately began investigation.  

Files.com released an initial Status Page posting on April 20th, 2023, at 11:57 AM PST, stating:   

“Reports of Elevated Errors - 2FA and SSO Logins:  We are investigating reports of errors on the Files.com service.  This is affecting user logins on the web portal that use two-factor authentication or single-sign on.”  

The 2FA login errors were resolved on April, 20th, 2023, at 11:58 AM PST, returning the platform to full functionality. 

Files.com released a resolution Status Page posting on April 20th, 2023, at 12:08 PM PST stating:  

“All services have been restored and are operating normally. We have resolved an issue that was causing web logins to fail when using single sign-on or two-factor authentication.  This incident occurred between the times of 11:28 to 11:58 a.m.”  

This incident began when our frontend team deployed a code change designed to fix a bug in the login process in the Files.com web interface. Unfortunately, the code change actually made things worse and prevented 2FA or SSO logins from working at all.  This change was rolled back as soon as the incident was identified.  

The root cause of this incident was Files.com’s failure to properly identify a change to the login page as a high-risk change and implement a sufficient level of automated and manual testing prior to deployment.  Files.com will be conducting remedial training on change risk identification at its upcoming Engineering meeting.  

Extensive review and testing was conducted by Files.com staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future. 

We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Posted May 15, 2023 - 12:47 PDT

Resolved

All services have been restored and are operating normally.

We have resolved an issue that was causing web logins to fail when using single sign-on or two-factor authentication. This incident occurred between the times of 11:28 to 11:58 a.m.

We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause. If you need additional support, please do not hesitate to contact our Customer Support team by email or phone. Thanks for your support while we resolved this issue.
Posted Apr 20, 2023 - 12:08 PDT

Investigating

We are investigating reports of errors on the Files.com service. This is affecting user logins on the web portal that use two-factor authentication or single-sign on.

We will provide additional details as they become available. Customers with urgent questions are encouraged to contact our Customer Support team by email. Thank you for your patience.
Posted Apr 20, 2023 - 11:57 PDT
This incident affected: Web Interface.