***UPDATED: Oct 18, 2021 @ 12:41 PM PT
We recently addressed issues affecting uploads via Box Embed Widget and a subset of accelerated uploads. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.
Between 1:00PM PDT on September 23, 2021 and 10:23PM PDT on September 27, 2021, some users may have experienced difficulties while working in Box. During this time, customers uploading via Box Embed Widget would see an error notification and multiple uploads of the same file. For customers uploading via accelerated nodes with the IP Allowlist feature enabled, users experienced errors in the upload. The issue occurred as a result of a recent code change in our upload service intended to clean up some deprecated pathways. We were able to resolve the issue by reverting the change and restoring to a previous known version. In addition, we are adding several new tests and alerts as part of the deployment process to prevent similar issues from occurring in the future.
Analysis
As part of regular updates, an upload service deployment was done in production. This change incorrectly overwrote a header that is used by downstream services to determine the original context of the upload. As such, in the Embed Widget case, the downstream service failed to recognize that CORS headers need to be set in the response. This caused errors only on the client side, which were not visible to the service. Corrective actions are being taken to ensure both the client and server side is aware of this change going forward in a timely manner.
In the IP Allowlisting and accelerated nodes case, the upload was not recognized to be from the original source through accelerated node and thus failed authorization check.
We were able to resolve the issue by reverting the change and restoring to a previous known version. After the revert of the deployment, the issue was mitigated.
Corrective Actions
The following corrective actions have been completed or are planned:
Adding additional testing and alerting during the development for html5 uploads
Adding additional testing and alerting during the development for IP Allowlist and accelerator combination
Adding a sanity environment before full deployment of upload-proxy service
We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.
Sincerely,
The Box Team
***********************************************************************************************************************
Notice and disclaimer: Box is providing this preliminary information subject to further review and analysis. To the best of our knowledge, this is the current state and we will update as more information is confirmed.
Between 1:00PM PDT on September 23, 2021 and 10:23PM PDT on September 27, 2021, some users may have experienced difficulties while working in Box. During this time, some customers experienced issues uploading files to Box via the Box Embed Widget and for a subset of accelerated uploads. The issue occurred as a result of a recent code change in our upload service to clean up some deprecated pathways. We were able to resolve the issue by reverting the change and restoring to a previous known version. In addition, we are adding several new tests and alerts as part of the deployment process to prevent similar issues from occurring in the future.
We are conducting a full engineering postmortem and our overview is subject to change with further analysis and findings. We will publish the results as soon as we have concluded our investigation.
We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.
Sincerely,
The Box Team