[Minor] Item Counts Not Updating
Incident Report for Box
Postmortem

We recently addressed issues affecting file and folder counts in Box. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 4:00 PM PST on February 18, 2024 and 9:10 PM PST on February 19, 2024, updates to file and folder counts displayed in Box were delayed for some customers. The issue occurred as a result of an intentional workload migration to mitigate a partial outage in one region of one of our cloud providers. We were able to resolve the issue by adding resources to the target region to reduce the cross-region latency impact. These delays only affected the visibility of file and folder counts; the counts themselves were maintained.

Analysis

On February 16th, a resource migration from one of our regions to another was performed as a mitigation effort for an unrelated issue. An increase in traffic on February 18th combined with the region shift resulted in a backlog in the file and folder counts workflow.

We addressed the issue by adding more resources to the new region. We are currently in the process of further consolidating the resources for this workflow to avoid significant delays and backlogs. Furthermore, to minimize the likelihood of similar situations occurring again, we are implementing checks and alerts to prevent future scenarios in which resource shortages could lead to delays in updating file and folder counts.

Corrective Actions

The following corrective actions have been completed or are planned:

  • We will implement checks and alerts to trigger if the workflow processing slows down and starts to back up.
  • We will add further documentation to our run book as references to be used in the event of similar region outages, ensuring that our service remains resilient to such incidents.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team

Posted Feb 29, 2024 - 12:34 PST

Resolved
After further monitoring, this incident is now considered resolved. Box services have been restored to full functionality. Please contact Box Support at https://support.box.com/ if you continue to experience any issues.
Posted Feb 19, 2024 - 18:24 PST
Update
Our team is continuing to monitor the progress of File and Folder counts. While customers should continue to see gradual improvement, it will take some time for all counts to be up to date. We will provide another update once the issue is considered fully resolved or if the status of the issue changes.
Posted Feb 19, 2024 - 11:09 PST
Monitoring
Our team has taken steps to remediate this issue and is seeing improvement in file and folder counts. Customers should begin to see gradual improvement with counts updating across Box. We are continuing to monitor for any additional impact and to ensure the issue will be fully resolved by the steps we have taken.
Posted Feb 19, 2024 - 10:44 PST
Investigating
Our team is investigating an issue with File and Folder counts not updating. This issue only affects the aesthetic count fields. It does not affect the ability to use or access any Box features or services. We will provide additional information as it becomes available.
Posted Feb 19, 2024 - 09:59 PST