Issue with Webhooks not going through for some users
Incident Report for Box
Postmortem

Notice and disclaimer: Box is providing this preliminary information subject to further review and analysis. To the best of our knowledge, this is the current state and we will update as more information is confirmed.

 

Between 9:16pm PDT on June 2, 2021 and 12:30pm PDT on June 3, 2021, some users may have experienced difficulties while working in Box. During this time, webhook event delivery was delayed. The issue occurred as a result of an unexpectedly configured user being deleted, causing a high number of unprocessable webhook requests. We were able to resolve the issue by temporarily lowering the retry number for failed webhook events and deleting the problematic webhooks. All delayed webhook events were delivered once the issue was resolved. In addition, we are working on improving our webhook cleanup process to prevent similar issues from occurring in the future. 

We are conducting a full engineering postmortem and our overview is subject to change with further analysis and findings. We will publish the results as soon as we have concluded our investigation.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team

Posted Jun 21, 2021 - 10:01 PDT

Resolved
After further monitoring, this incident is now considered resolved. Box services have been restored to full functionality. Please contact Box Support at https://support.box.com/ if you continue to experience any issues.
Posted Jun 03, 2021 - 14:49 PDT
Monitoring
We have resolved the issue impacting v2 Webhooks. We are continuing to monitor to ensure no further impact is observed
Posted Jun 03, 2021 - 12:51 PDT
Investigating
Box is investigating reports of an issue with v2 Webhooks. Users of this service may not receive Webhooks events. Affected services are subject to change as the investigation continues.
Posted Jun 03, 2021 - 11:33 PDT
This incident affected: Box Platform / API (Webhooks).