*****Updated Oct 18, 2021 @ 12:44 PM PT*****

We recently addressed issues affecting user reports. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

 From August 31, 2021 at 5:30 am PDT to September 11, 2021 at 12:47 PDT, some users may have experienced difficulties while working in Box. During this time, customers experienced stale data in certain Batch Admin Reports. The issue occurred as a result of latent scale issues. We were able to resolve the issue by tuning configurations in our ingestion pipeline. In addition, we are working on a more scalable ingestion pipeline to prevent similar issues from occurring in the future. 

Analysis 

The pipeline that ingests data to be used for reports relies on temporary buffer tables before compacting the data into final tables. The buffer tables have an index to keep track of all of its files. These indexes require maintenance to avoid placing an undue strain on the system. In this case, the size of the indexes and resulting strain created a backlog that impacted data freshness of Batch Admin Reports.

We are currently working on an overhaul of how we ingest and maintain the data for reports. In this system, the index will constantly be maintained and not put pressure on the system.

Corrective Actions

The following corrective actions have been completed or are planned:

• Truncate the buffer tables indexes

• Add additional metrics to root cause the issue faster

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

 Sincerely,

The Box Team

************************************************************************************************************************

Notice and disclaimer: Box is providing this preliminary information subject to further review and analysis. To the best of our knowledge, this is the current state and we will update as more information is confirmed.

From August 31, 2021 at 5:30am PDT to September 11, 2021 at 12:47 PDT, some users may have experienced difficulties while working in Box. During this time, customers experienced stale data in their reports. The issue occurred as a result of latent scale issues. We were able to resolve the issue by tuning configurations in our ingestion pipeline. In addition, we are working on a more scalable ingestion pipeline to prevent similar issues from occurring in the future.

We are conducting a full engineering postmortem and our overview is subject to change with further analysis and findings. We will publish the results as soon as we have concluded our investigation.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.

Sincerely,

The Box Team