Delays To Batch and Scheduled Operations

Incident Report for Files.com

Postmortem

On April 11th, 2023, at 2:05 PM PST, Files.com received automated alerting of delays to batch and scheduled operations, which resulted in an incident being declared.  The Incident Management Team (IMT) convened and immediately began investigation. 

Files.com released an initial Status Page posting on April 11th, 2023, at 2:15 PM PST, stating: 

“Delays To Batch and Scheduled Operations: We are investigating reports of delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations.  This situation should not affect real-time operations such as FTP, SFTP, AS2, and other operations where Files.com acts as a server.” 

The delays to batch and scheduled operations were resolved on April 11th, 2023, at 2:18 PM PST, returning the platform to full functionality. 

Files.com released a resolution Status Page posting on April 11th, 2023, at 2:41 PM PST,  stating:  

“All services have been restored and are operating normally.   We have resolved a situation causing delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations.  Operations were delayed beginning at 2:00 p.m. PDT and ending at 2:08 p.m. PDT as well as 2:11 p.m. PDT and 2:15 PDT.  All operations did successfully complete despite delays.”  

Files.com uses a pool of worker servers that perform a variety of batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations.  At 2:00, 4 of those servers crashed due to to running out of RAM.  Upon investigation, it was determined that many jobs were running which relate to cleaning up sites that no longer represent an active customer (cancelled sites) and that those jobs all had disproportionately high CPU and RAM usage.  The root cause is inconclusive at this time, but likely due to a memory leak in this job type.  To prevent this issue from recurring while the job is further investigated, additional configuration was added to limit this job type's parallelism.  With the restricted parallelism, the job should not be able to crash multiple worker machines at once.  The total customer impact of this incident was very minimal because worker servers did properly run throughout the duration of the incident, and any impact was likely limited to small delays in processing. 

Extensive review and testing was conducted by Files.com staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future.   

We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Posted May 15, 2023 - 10:15 PDT

Resolved

All services have been restored and are operating normally.

We have resolved a situation causing delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations.

Operations were delayed beginning at 2:00 p.m. PDT and ending at 2:08 p.m. PDT as well as 2:11 p.m. PDT and 2:15 PDT. All operations did successfully complete despite delays.

We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause. If you need additional support, please do not hesitate to contact our Customer Support team by email or phone. Thanks for your support while we resolved this issue.
Posted Apr 11, 2023 - 14:40 PDT

Investigating

We are investigating reports of delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations.

This situation should not affect real-time operations such as FTP, SFTP, AS2, and other operations where Files.com acts as a server.

We will provide additional details as they become available. Customers with urgent questions are encouraged to contact our Customer Support team by email. Thank you for your patience.
Posted Apr 11, 2023 - 14:14 PDT
This incident affected: Background Processing, including Notifications and Webhooks.