On April 11th, 2023, at 2:05 PM PST, Files.com received automated alerting of delays to batch and scheduled operations, which resulted in an incident being declared. The Incident Management Team (IMT) convened and immediately began investigation.
Files.com released an initial Status Page posting on April 11th, 2023, at 2:15 PM PST, stating:
“Delays To Batch and Scheduled Operations: We are investigating reports of delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations. This situation should not affect real-time operations such as FTP, SFTP, AS2, and other operations where Files.com acts as a server.”
The delays to batch and scheduled operations were resolved on April 11th, 2023, at 2:18 PM PST, returning the platform to full functionality.
Files.com released a resolution Status Page posting on April 11th, 2023, at 2:41 PM PST, stating:
“All services have been restored and are operating normally. We have resolved a situation causing delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations. Operations were delayed beginning at 2:00 p.m. PDT and ending at 2:08 p.m. PDT as well as 2:11 p.m. PDT and 2:15 PDT. All operations did successfully complete despite delays.”
Files.com uses a pool of worker servers that perform a variety of batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations. At 2:00, 4 of those servers crashed due to to running out of RAM. Upon investigation, it was determined that many jobs were running which relate to cleaning up sites that no longer represent an active customer (cancelled sites) and that those jobs all had disproportionately high CPU and RAM usage. The root cause is inconclusive at this time, but likely due to a memory leak in this job type. To prevent this issue from recurring while the job is further investigated, additional configuration was added to limit this job type's parallelism. With the restricted parallelism, the job should not be able to crash multiple worker machines at once. The total customer impact of this incident was very minimal because worker servers did properly run throughout the duration of the incident, and any impact was likely limited to small delays in processing.
Extensive review and testing was conducted by Files.com staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future.
We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.