[Outage] Issue with Box Services
Incident Report for Box
Postmortem

NOTICE AND DISCLAIMER: CUSTOMER UNDERSTANDS AND AGREES THAT THIS INFORMATION IS SUBJECT TO CHANGE. TO THE BEST OF OUR KNOWLEDGE, THIS IS THE CURRENT STATE AND WILL BE UPDATED AS INFORMATION IS CONFIRMED.

LAST UPDATED: August 17, 2020

Summary Impact

Between 4:02 pm and 4:15 pm PDT on July 8, 2020, some users may have experienced difficulties while working in Box. During this time, users may have been unable to connect to Box services. The issue occurred as a result of routing updates associated with a scheduled network maintenance to increase data center capacity.  We were able to detect the issue and resolve it by rolling back the configuration changes.  In addition, we are working to improve our network resiliency by increasing our testing and validation processes and increasing our service isolation to prevent similar issues from occurring in the future.

Analysis

Upon investigation, analysis identified that the tooling used to generate BGP (Border Gateway Protocol) routing configurations for new data center expansion space unintentionally generated a configuration that overlapped with existing data center space.   The subsequent deployment of the new configuration overwrote the BGP routes of the existing space, isolating traffic for critical services in the existing space. While the tooling that generated the configuration has been in use for several years, it lacked a validation test to look for conflicts with existing configurations. It was further identified that application configurations were not properly isolating services to limit the area of impact.

Defect Remediation

The following remediation actions have been completed or are planned:

  • Update tooling to validate and test candidate routing changes do not conflict with existing configurations
  • Increase application service isolation to limit area of impact 

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team

Posted Jul 09, 2020 - 11:05 PDT

Resolved
No additional was observed and this incident is now considered resolved. Please contact Box Support at https://support.box.com if you experience any additional issues.
Posted Jul 08, 2020 - 18:18 PDT
Monitoring
Our teams validated that all affected services are recovered at this time. We are continuing to monitor for any additional impact.
Posted Jul 08, 2020 - 18:07 PDT
Update
Our team continues to take steps to fully remediate impact. Our monitoring indicates continued recovery and we will provide additional updates as they become available.
Posted Jul 08, 2020 - 17:33 PDT
Identified
Our team identified the cause of this issue and our monitoring indicates continued recovery. We will continue to provide additional updates as they become available.
Posted Jul 08, 2020 - 17:16 PDT
Update
We're continuing to see recovery and will continue to provide additional updates as they become available.
Posted Jul 08, 2020 - 17:00 PDT
Update
Our monitoring indicates initial recovery is underway, but customers may be experiencing slowness or 429 errors. We will continue to provide additional updates as they become available.
Posted Jul 08, 2020 - 16:43 PDT
Investigating
We are currently investigating an issue with Box services. We will provide more information as it becomes available.
Posted Jul 08, 2020 - 16:13 PDT
This incident affected: Box Web Application (Login/SSO, Uploads/Downloads, Collaboration, Search, Preview, Sharing (Shared Links), Email Notifications, Admin Console & Functionality, Governance (Retention, Legal Holds), Workflows and Automations, Comments and Tasks, Accessible Site (a.box.com)), Box Notes (Web Application, Desktop Client), Partners / Integrations (Microsoft Online, Apple iWork, Box for Salesforce, Google Apps, NetSuite), Mobile Applications (Login/SSO, Preview, Search, Uploads/Downloads, Mobile-Optimized Website (m.box.com)), Desktop Applications (Login/SSO, Box Sync, Box Drive, Box Edit / Tools), Box Platform / API (Content API, Content Preview, Authentication (OAuth 2.0 / JWT), Search, Uploads/Downloads, Webhooks), and Box Relay, FTP, Box Website.