On March 25th, 2024, at 7:43 AM PST, Files.com correlated multiple customer tickets indicating ‘elevated error rates related to outbound emails sent by Files.com’, which resulted in an incident being declared.  The Incident Management Team (IMT) convened and immediately began investigation. 

The ‘elevated error rates related to outbound emails sent by Files.com’ issue was resolved on March 25th, 2024, at 8:48 AM PST, returning the platform to full functionality.

Files.com released an initial investigation posting to the Status Page on March 25th, 2024, at 8:01 AM PST stating: 

‘We are investigating elevated error rates related to outbound emails sent by Files.com. This impacts any emails, including password reset requests, email 2-FA codes, and activity notifications.’

Files.com released a resolution posting to the Status Page on March 25th, 2024, at 9:16 AM PST stating:

‘We have resolved the issue causing elevated error rates on outbound emails sent from Files.com. This incident occurred between the times of 3:32 AM Pacific Time on March 23rd, 2024 and 8:48 AM Pacific Time on March 25th, 2024.

We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause.

Unfortunately, emails that failed to send during this incident cannot be recovered and resent. We apologize for the impact that this may have had on your operations.

 If you need additional support, please do not hesitate to contact our Customer Support team by email or phone.’

‌

Files.com released a more accurate resolution posting to the Status Page on March 25th, 2024, at 9:25 AM PST stating:

‘Emails from Files.com failing to send

We have resolved the issue causing a total failure of outbound emails sent from Files.com.  Emails from Files.com failed to deliver between the times of 3:32 AM Pacific Time on March 23rd, 2024 and 8:48 AM Pacific Time on March 25th, 2024.

 We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause.

 Unfortunately, the emails that failed to send during this incident cannot be recovered and resent.  We apologize for the impact that this may have had on your operations.

 If you need additional support, please do not hesitate to contact our Customer Support team by email or phone.’

In this incident, outbound E-Mail sending from the Files.com application did not work for a period of time. Files.com sends outbound E-Mail using Amazon Web Services.

This incident began when Amazon sent two messages to Files.com about our E-Mail delivery at times outside of business hours.  Those messages went to an inbox that is only monitored during regular business hours. Because Files.com did not reply to the message from Amazon, Amazon disabled our access to their E-Mail service.  The original message from Amazon was submitted to Files.com on Saturday, March 23rd, 2024, at 3:32 AM PST, and then Amazon disabled our access on Saturday, March 23rd, at 2:24 PM PST.

As soon as Files.com became aware of the issue, Files.com switched E-Mail delivery to a separate vendor (Postmark) that we also have a relationship with.  This returned the platform to full functionality on March 25th, 2024, at 8:48 AM PST.  We switched back to Amazon on March 27th, 2024, at 6:37 AM PST.

The root cause of this incident is that Files.com did not have an automatic means of escalating the emails from Amazon to an after-hours operational pager address.  Files.com has since implemented this automated escalation tool.  Additionally, Files.com has added support for more quickly switching to our backup E-Mail vendor (Postmark) and we will maintain this capability for any future events.

Finally, Files.com is also adding additional E-Mail logging which would have made it easier for our monitoring systems to detect the actual bounced E-Mails themselves.  To the extent possible, we intend to extend access to these logs to you, our customers, so that you have direct visibility into outbound email deliverability as well in the future.

We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.