[Major] Issues with Office Online
Incident Report for Box
Postmortem

We recently addressed issues affecting Box’s Office Online integration. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

On Tuesday, September 27th 2022 between 3:00 PM PDT and 5:09 PM PDT, some users may have experienced difficulties while working in Box. During this time, users experienced degraded performance opening and editing Office files in Box. The issue occurred as a result of a configuration change made as part of our ongoing effort to improve security. We were able to resolve the issue by provisioning more resources to the impacted Box systems. In addition, we are implementing code changes to increase the resiliency of the impacted systems to prevent similar issues from occurring in the future. 

Analysis

The issue occurred as a result of a configuration change intended to improve the security of inter-service communication at Box. This change resulted in an unexpected increase in CPU load on one of our backend services, which prevented it from readily processing Office Online requests. This issue was compounded when impacted requests were re-attempted, which ultimate resulted in performance issues across the Office Online integration. The team provisioned additional resources to the impacted services, which mitigated the increased load and allowed Box’s functionality to return to normal.

Corrective Actions

The following corrective actions have been completed or are planned:

  • Improving our accounting of the resource impact of similar security changes to ensure our services can handle them.
  • Improving the resiliency of the services at issue to minimize the impact of an increase in retried requests to prevent CPU issues from compounding.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team

Posted Oct 05, 2022 - 11:08 PDT

Resolved
From approximately 03:00 PM to 05:10 PM US Pacific time, we had an issue impacting Office Online. Our systems automatically detected and corrected the underlying issue. There is no current impact and no further updates will be provided here. If you are currently seeing any issues, please let us know at https://support.box.com.
Posted Sep 27, 2022 - 06:00 PDT