[Minor] Users in Asia-Pacific region may be unable to reach Box.
Incident Report for Box
Postmortem

UPDATED as of 2/4/2022 6:50 AM

We recently addressed issues affecting Box Web, Uploads/Downloads, Notes, and Mobile Applications. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 4:00 PM PST and 6:53 PM PST on January 12, 2022, some users may have experienced difficulties while working in Box. During this time, Box users in the APAC region may have experienced poor performance and increased latency while using Box. The issue occurred as a result of a misconfiguration that impacted the addition of a new Internet transit provider in the APAC region. We were able to resolve the issue by disabling BGP traffic to the new provider. In addition, we have modified our BGP policy and are working with our vendor to address a misconfiguration on their side to prevent similar issues from occurring in the future. 

Analysis 

As part of our ongoing effort to improve the customer experience for users in the APAC region, Box added a new Internet transit provider. However, as a result of a misconfiguration, this new provider began taking traffic from some customer networks that should not have been routed over the circuit. As a result of this unexpected traffic, the new circuit became oversaturated, resulting in the performance issues and increased latency. We were able to resolve the issue by disabling BGP traffic to the new provider and are gradually increasing traffic in a slow, controlled manner to prevent performance issues from reoccurring.

Corrective Actions

The following corrective actions have been completed or are planned:

  • We have worked with the vendor to identify the underlying issue and conduct a formal post-mortem on the vendor's side.

  • We are gradually increasing traffic in a slow, controlled manner to this new provider. 

  • We are addressing a monitoring gap that prevented us from detecting the issue earlier.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team

Notice and disclaimer: Box is providing this preliminary information subject to further review and analysis. To the best of our knowledge, this is the current state and we will update as more information is confirmed.

On January 12, 2022 between 4:00 PM PST and 7:00 PM PST, some users may have experienced difficulties while working in Box. During this time, users of Box in Japan experienced increased latency and poor performance. The issue occurred as a result of a recent link addition introduced to improve performance for users in other APAC regions. We were able to resolve the issue by rolling back the change. In addition, we are evaluating options to gradually increase service to this new provider to prevent similar issues from occurring in the future.

We are conducting a full engineering postmortem and our overview is subject to change with further analysis and findings. We will publish the results as soon as we have concluded our investigation.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.

Sincerely,

The Box Team

Posted Jan 18, 2022 - 16:52 PST

Resolved
We have taken steps to resolve this issue and are considering the issue to be fully resolved. If you are seeing any new issues, please let us know at https://support.box.com.
Posted Jan 12, 2022 - 19:14 PST
Monitoring
The Engineering team has been able to identify the issue and have implemented a fix. We currently as seeing a drastic decrease in errors and are monitoring the situation. If you are experiencing any issues, please don't hesitate to submit a ticket at support.box.com.
Posted Jan 12, 2022 - 19:03 PST
Investigating
We are currently investigating an issue where some users in the Asia-Pacific region are unable to access Box via Box.com, Box Drive, and Box Notes. Box engineers are actively reviewing the issue and will post updates at the next change in status.
Posted Jan 12, 2022 - 18:56 PST
This incident affected: Box Notes (Web Application, Desktop Client), Box Web Application (Login/SSO, Uploads/Downloads, Admin Console & Functionality), Mobile Applications (Uploads/Downloads), and Box Website.