[Outage] Issue with Box Services
NOTICE AND DISCLAIMER: CUSTOMER UNDERSTANDS AND AGREES THAT THIS INFORMATION IS SUBJECT TO CHANGE. TO THE BEST OF OUR KNOWLEDGE, THIS IS THE CURRENT STATE AND WILL BE UPDATED AS INFORMATION IS CONFIRMED.
LAST UPDATED: August 17, 2020
Between 4:02 pm and 4:15 pm PDT on July 8, 2020, some users may have experienced difficulties while working in Box. During this time, users may have been unable to connect to Box services. The issue occurred as a result of routing updates associated with a scheduled network maintenance to increase data center capacity. We were able to detect the issue and resolve it by rolling back the configuration changes. In addition, we are working to improve our network resiliency by increasing our testing and validation processes and increasing our service isolation to prevent similar issues from occurring in the future.
Upon investigation, analysis identified that the tooling used to generate BGP (Border Gateway Protocol) routing configurations for new data center expansion space unintentionally generated a configuration that overlapped with existing data center space. The subsequent deployment of the new configuration overwrote the BGP routes of the existing space, isolating traffic for critical services in the existing space. While the tooling that generated the configuration has been in use for several years, it lacked a validation test to look for conflicts with existing configurations. It was further identified that application configurations were not properly isolating services to limit the area of impact.
The following remediation actions have been completed or are planned:
- Update tooling to validate and test candidate routing changes do not conflict with existing configurations
- Increase application service isolation to limit area of impact
We are continuously working to improve Box and want to make sure we are delivering the best product and user experience. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.
The Box Team
No additional was observed and this incident is now considered resolved. Please contact Box Support at https://support.box.com
if you experience any additional issues.
Our teams validated that all affected services are recovered at this time. We are continuing to monitor for any additional impact.
Our team continues to take steps to fully remediate impact. Our monitoring indicates continued recovery and we will provide additional updates as they become available.
Our team identified the cause of this issue and our monitoring indicates continued recovery. We will continue to provide additional updates as they become available.
We're continuing to see recovery and will continue to provide additional updates as they become available.
Our monitoring indicates initial recovery is underway, but customers may be experiencing slowness or 429 errors. We will continue to provide additional updates as they become available.
We are currently investigating an issue with Box services. We will provide more information as it becomes available.
This incident affected: Box Notes (Web Application, Desktop Client), Box Platform / API (Content API, Content Preview, Authentication (OAuth 2.0 / JWT), Search, Uploads/Downloads, Webhooks), Desktop Applications (Login/SSO, Box Sync, Box Drive, Box Edit / Tools), Mobile Applications (Login/SSO, Preview, Search, Uploads/Downloads, Mobile-Optimized Website (m.box.com)), Partners / Integrations (Microsoft Online, Apple iWork, Box for Salesforce, Google Apps, NetSuite), Box Web Application (Login/SSO, Uploads/Downloads, Collaboration, Search, Preview, Sharing (Shared Links), Email Notifications, Admin Console & Functionality, Governance (Retention, Legal Holds), Workflows and Automations, Comments and Tasks, Accessible Site (a.box.com)), and Box Relay, FTP, Box Website.