We recently addressed issues affecting timeliness of the Events API. We would like to take the opportunity to further explain these issues and the steps we have taken to keep it from happening in the future. 

Between March 29, 2021, and April 12, 2021, some users may have experienced difficulties while working in Box. During this time, the real-time Events API sporadically experienced unusual latency. This issue occurred as a result of multiple hardware performance issues in the data-backing store and an issue in the service that populated the data-backing store with the events. We were able to resolve the issue by removing certain nodes from the cluster, addressing the issue in the service that populated the events into the data-backing store, and adding capacity to handle multiple node failures within the cluster.

Analysis 

On the evening of March 29, 2021, both data-backing store clusters experienced multiple simultaneous hardware failures at the same time. This caused a performance impact on both data-backing stores that service the Events API.

Corrective Actions

The following corrective actions have been completed or are planned:

• Additional capacity has been added to both of the data-backing store clusters to enable them to handle multiple simultaneous hardware failures without performance impact to the Events API.

• Correction of the issue that caused the service that populates the cluster to run low on memory and slow down.

• Addition of new monitoring and alerts in our production environment to more quickly identify this issue and remove failed nodes from the cluster.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team