UPDATED as of 2/4/2022 6:50 AM
We recently addressed issues affecting Box Web, Uploads/Downloads, Notes, and Mobile Applications. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.
Between 4:00 PM PST and 6:53 PM PST on January 12, 2022, some users may have experienced difficulties while working in Box. During this time, Box users in the APAC region may have experienced poor performance and increased latency while using Box. The issue occurred as a result of a misconfiguration that impacted the addition of a new Internet transit provider in the APAC region. We were able to resolve the issue by disabling BGP traffic to the new provider. In addition, we have modified our BGP policy and are working with our vendor to address a misconfiguration on their side to prevent similar issues from occurring in the future.
Analysis
As part of our ongoing effort to improve the customer experience for users in the APAC region, Box added a new Internet transit provider. However, as a result of a misconfiguration, this new provider began taking traffic from some customer networks that should not have been routed over the circuit. As a result of this unexpected traffic, the new circuit became oversaturated, resulting in the performance issues and increased latency. We were able to resolve the issue by disabling BGP traffic to the new provider and are gradually increasing traffic in a slow, controlled manner to prevent performance issues from reoccurring.
Corrective Actions
The following corrective actions have been completed or are planned:
We have worked with the vendor to identify the underlying issue and conduct a formal post-mortem on the vendor's side.
We are gradually increasing traffic in a slow, controlled manner to this new provider.
We are addressing a monitoring gap that prevented us from detecting the issue earlier.
We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.
Sincerely,
The Box Team
Notice and disclaimer: Box is providing this preliminary information subject to further review and analysis. To the best of our knowledge, this is the current state and we will update as more information is confirmed.
On January 12, 2022 between 4:00 PM PST and 7:00 PM PST, some users may have experienced difficulties while working in Box. During this time, users of Box in Japan experienced increased latency and poor performance. The issue occurred as a result of a recent link addition introduced to improve performance for users in other APAC regions. We were able to resolve the issue by rolling back the change. In addition, we are evaluating options to gradually increase service to this new provider to prevent similar issues from occurring in the future.
We are conducting a full engineering postmortem and our overview is subject to change with further analysis and findings. We will publish the results as soon as we have concluded our investigation.
We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.
Sincerely,
The Box Team