[Major] Issues with Uploads, Downloads and Box Notes
Incident Report for Box

We recently addressed issues affecting Box Notes. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 6:15pm PDT on June 10th and 12:56am PDT on June 11th, some users may have experienced difficulties while working in Box. During this time, Box uploads, downloads and Notes experienced some failures / errors. This issue occurred due to a code change that caused our storage databases to replicate invalid events that impacted the availability of the system. We were able to resolve the issue by restarting our storage database clusters to remove the invalid events from replication and resume normal operation. In addition, we are working towards a fix for the configuration change to prevent similar issues from occurring in the future. 


The issue occurred as a result of a recent code change in an ongoing effort to improve the underlying technology that powers Box storage services. Our engineering team prioritized executing these changes on a database cluster that was not in customer path but encountered an unexpected failure that caused invalid events to replicate and overrun our customer-facing databases capacity. The team immediately rolled back the code change but invalid events continued to saturate our storage database clusters. In response, our team took further remediation steps to remove these invalid events and failover our storage service applications to other Box regions. No impact to data occurred as part of this issue or the remediation work to restore service.

Corrective Actions

The following corrective actions have been completed or are planned:

  • Document a more expedient recovery procedure for invalid replication events in our storage database clusters.

  • Modify our Box Storage improvement plan to reduce the risk of introducing invalid replication events.

  • Increase the speed of our Storage Service failover process.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 



The Box Team

Posted Jul 01, 2021 - 04:59 PDT

After further monitoring, this incident is now considered resolved. Box services have been restored to full functionality. Please contact Box Support at https://support.box.com/ if you continue to experience any issues.
Posted Jun 11, 2021 - 05:19 PDT
We have resolved the issue impacting Uploads, downloads and Box Notes. We are continuing to monitor to ensure no further impact is observed
Posted Jun 11, 2021 - 01:36 PDT
We are continuing to monitor for any further issues.
Posted Jun 11, 2021 - 00:28 PDT
Our team has taken action to remediate impact and is seeing recovery across metrics. We are continuing to monitor for any additional impact.
Posted Jun 10, 2021 - 23:56 PDT
Box continues remediation efforts to restore full functionality to Upload, Download and Box Notes.

Users of this service may continue to observe some failures while remediation is in process.
Posted Jun 10, 2021 - 23:06 PDT
We are continuing to investigate this issue.
Posted Jun 10, 2021 - 22:57 PDT
We are continuing to investigate this issue.
Posted Jun 10, 2021 - 22:56 PDT
We are continuing to investigate this issue.
Posted Jun 10, 2021 - 22:31 PDT
We are investigating an ongoing issue affecting uploads, downloads and Notes. We will provide more information as soon as it is available.
Posted Jun 10, 2021 - 22:30 PDT
This incident affected: Box Notes (Web Application) and Box Web Application (Uploads/Downloads).