We have identified a coding error that caused Automations and manual bulk file move/copy/delete jobs to fail to fully complete in certain situations. This coding error was caused by a race condition in the code that manages our orchestration layer for parallelized file operations. The race condition was in our production environment from 2:22 PM PST on April 13 through 2:18 PM PST on April 16, when we deployed a fix. This issue was more pronounced in larger jobs. The logging output from the jobs is correct.
We understand how important it is for your automated jobs to run on time and complete as expected, and we are absolutely committed to ensuring reliability on our platform. We sincerely apologize for any impact that this error had on your operations.
We have the capability to re-submit the affected Automation Runs and File Migrations for processing again. We elected not to do this by default because of the potential for unexpected outcomes, but we are happy to manually re-submit your jobs if you would like us to.
Please get in touch with our support team, and we can perform this task for you, as well as answer any other questions you may have.