We recently addressed issues affecting Uploads, Downloads and PublicAPI. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.
Between 03:37 PM and 03:47 PM PST on January 22, 2024, some users may have experienced difficulties while working in Box. During this time, an internal service was restarted, which resulted in some users experiencing degraded performance and/or intermittent errors on Box Uploads, Downloads and Public API. The issue occurred as a result of a previous infrastructure change and was automatically resolved once the impacted service successfully restarted and stabilized. We are enhancing our deployment automation and defining additional processes to minimize the risk of similar issues occurring in the future.
Analysis
On January 22, 2024, a change was rolled out on an availability zone in one of our primary regions in which an internal database interaction service was running. There was a pre-existing discrepancy between the code and infrastructure originating from a previous operation performed in response to an unrelated incident. This discrepancy resulted in the impacted service restarting, which caused some users to experience degraded performance.
When the change went live, the availability of the service dropped. That caused traffic from downstream services that directed to the service fleet in this zone to fail. As a result the top level metrics were impacted for 10 minutes. The impacted service came back up shortly after, which remediated the issue.
Corrective Actions
The following corrective actions have been completed or are planned:
We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.
Sincerely,
The Box Team