We recently addressed issues affecting slow-arriving Email Notifications. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.
Between 2:32 pm and 4:05 pm PST on November 3, 2023, some users may have experienced difficulties while working in Box. During this time, file-related notifications were temporarily delayed. The issue occurred due to an expired authentication token used in one of our environments. We were able to resolve the issue by rotating the token, at which point entire backlog of unsent notifications was sent. In addition, we are working on putting additional safeguards in place to alert us beforehand in such cases earlier to prevent similar issues from occurring in the future.
Analysis
This issue was caused by an expired token used to distribute software within Box infrastructure. As a result, there was temporary inability for the affected mechanism to distribute the current version of software. This has led to a number of jobs triggering continuously to pause, since these could not obtain the current version of software due to this issue. One of these jobs that was impacted was batching job for notifications, and this caused the temporary impact to end users. Due to a limitation in alerting mechanism in this area, the alert did not fire as expected as a warning beforehand.
We were able to resolve this issue by rotating the token, which restored the impacted functionality. Once the problem was mitigated, the notification job started firing again and quickly sent out the historical backlog of unsent notifications.
Corrective Actions
The following corrective actions have been completed or are planned:
We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.
Sincerely,
The Box Team