All Regions: Major Network Outage
Incident Report for Files.com
Postmortem

On May 5th, 2023 at 11:39 AM, Files.com received internal monitoring alerts of issues related to DNS, which resulted in an incident being declared.  The Incident Management Team (IMT) convened and immediately began investigation. 

The DNS resolution issue was resolved on May 5th, 2023, at 3:20 PM, returning the platform to full functionality. 

Files.com released an initial investigation to its Status Page posting on May 5th, 2023, at 12:00 PM PST stating: 

“All regions: We are investigating a major network outage on Files.com affecting Files.com services in all regions. This outage is affecting our gateway networking in regions other than USA, and our Core Services are running correctly. 

At this time, we are also investigating elevated error rates in our primary USA region. 

We will provide additional details as they become available. Customers with urgent questions are encouraged to contact our Customer Support team by email. Thank you for your patience.” 

Files.com released an updated investigation to its Status Page posting on May 5th, 2023, at 12:58 PM PST stating: 

“We are continuing to investigate this issue.  We will post an update as soon as the issue has been identified and a fix is being implemented.  If you need additional assistance, please do not hesitate to contact our Customer Support team by email. Thank you for your continued patience.” 

Files.com released a second updated investigation to its Status Page posting on May 5th, 2023, at 14:03 PM PST stating: 

“We are continuing to investigate the DNS issue causing Files.com sites to be inaccessible for some users.  We will post an update as soon as the issue has been identified and a fix is being implemented. 

If you need additional assistance, please do not hesitate to contact our Customer Support team by email. Thank you for your continued patience.” 

Files.com released a resolution Status Page posting on May 5th, 2023, at 3:30 PM PST stating: 

“All services have been restored and are operating normally. We have identified and resolved the root cause underlying these DNS issues.  

We will follow up with an Incident Report within one business day including the root cause and steps taken to address the root cause. If you need additional support, please do not hesitate to contact our Customer Support team by email or phone.”  

This incident started when a prominent FinTech company sent a fraudulent and erroneous report accusing Files.com of cybercrime to GoDaddy, the .COM domain name  registrar where the Files.com domain name is registered.  

This FinTech company is a Files.com customer, and the report was sent in error. 

Upon receipt of the accusation, GoDaddy chose to immediately suspend the Files.com domain without providing Files.com any advance notice or warning. 

Once the domain suspension was identified, we worked with GoDaddy and the source of the erroneous cybercrime report to escalate and respond to the erroneous report, which ultimately resulted in GoDaddy removing the suspension from the Files.com domain, returning the services back to functionality. 

Unfortunately, escalating this situation within GoDaddy took a disappointingly long time and we ended up with a 3+ hour downtime.  We are thoroughly disappointed with the way they handled the situation. 

You may be questioning why we use GoDaddy as our domain registrar, and we want to speak to that briefly.  They're actually a customer of ours and neighbor of ours in Scottsdale, AZ, and we have a generally good relationship with the company.  

With that said, we were already in the process of moving all of our domain registrations to CSC Domains, an enterprise-focused and security-focused domain registrar whose business is structured to prevent exactly these sorts of mishaps.  

Although we have our GoDaddy account secured as strongly as possible, including with two-factor authentication, CSC Domains offers a much stronger level of enterprise security and protections, and domain registrar risks are something we identified in a previous meeting of our risk committee. 

Unfortunately, we missed the mark on timing, because we ended up having an incident with GoDaddy prior to completing that migration.  

As of Monday, May 8th, 2023, we are actively underway with migrating the Files.com domain (and all other domains that we own) to CSC.  The transfer process, which is controlled by GoDaddy, can take up to a week to complete.  

Ultimately, the root cause of this incident was Files.com’s use of GoDaddy as its domain registrar and our failure to complete the project to switch to CSC Domains in a more timely manner. 

We recognize the impact that this incident has had on our customers. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Posted May 08, 2023 - 20:11 PDT

Resolved
All services have been restored and are operating normally. We have identified and resolved the root cause underlying these DNS issues.

We will follow up with an Incident Report within one business day including the root cause and steps taken to address the root cause. If you need additional support, please do not hesitate to contact our Customer Support team by email or phone.

Thanks for your support while we resolved this issue.
Posted May 05, 2023 - 15:30 PDT
Update
We are continuing to investigate the DNS issue causing Files.com sites to be inaccessible for some users. We will post an update as soon as the issue has been identified and a fix is being implemented.

If you need additional assistance, please do not hesitate to contact our Customer Support team by email. Thank you for your continued patience.
Posted May 05, 2023 - 14:03 PDT
Update
We are continuing to investigate this issue. We will post an update as soon as the issue has been identified and a fix is being implemented. If you need additional assistance, please do not hesitate to contact our Customer Support team by email. Thank you for your continued patience.
Posted May 05, 2023 - 12:58 PDT
Investigating
All regions: We are investigating a major network outage on Files.com affecting Files.com services in all regions. This outage is affecting our gateway networking in regions other than USA, and our Core Services are running correctly.

At this time, we are also investigating elevated error rates in our primary USA region.

We will provide additional details as they become available. Customers with urgent questions are encouraged to contact our Customer Support team by email. Thank you for your patience.
Posted May 05, 2023 - 12:00 PDT
This incident affected: Core Services / API, Web Interface, FTP/FTPS, SFTP, and WebDAV.