[Major] Box Notes inbox unavailable
Incident Report for Box
Postmortem

**UPDATED AS OF May 13, 2021 15:35PM PST

We recently addressed issues affecting Box Notes. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 12:15am and 1:22am PDT on April 17th, 2021, some users may have experienced difficulties while working in Box. During this time, Box users may have experienced a degradation to services such as Box Notes. The issue occurred as a result of incorrect DC flip for SKS service (Box Notes service backend) for network maintenance . The issue was resolved after reverting the DC flip for SKS service. In addition, we are working to improve our testing processes to prevent similar issues from occurring in the future.

Analysis

Box's SKS Events API  retrieves data from an internal database cluster that is updated as events occur within the Box application.  This SKS events API powers the left sidebar in Box Notes. As part of this issue, we identified an issue with our configuration update process in which multiple configurations can point to same internal cluster and that prevents the startup of SKS service. After we identified the problem configuration, we standardized our configuration deploy process so that this issue does not reoccur.

Corrective Actions

The following corrective actions have been completed or are planned:

  • Document and validate the process to switch SKS service from one cluster to another and make one cluster primary vs another.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.

Sincerely,

The Box Team

************************************************************************************

We recently addressed issues affecting Box Notes. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 12:15 am and 1:22 am PDT on April 17th, 2021, some users may have experienced difficulties while working in Box. During this time, Box users may have experienced a degradation to services such as Box Notes. The issue occurred as a result of incorrect DC flip for SKS service (Box Notes service backend) for network maintenance. The issue was resolved after reverting the DC flip for SKS service. In addition, we are working to improve our testing processes to prevent similar issues from occurring in the future

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.

Sincerely,

The Box Team

Posted May 13, 2021 - 12:04 PDT

Resolved
We have completed our monitoring period and are considering the issue fully resolved. If you are seeing any new issues, please let us know at https://support.box.com.
Posted Apr 17, 2021 - 01:36 PDT
Monitoring
As of 1:20 AM PST, we no longer are seeing failures when attempting to load Box Notes Inbox. We will continue to monitor recovery.
Posted Apr 17, 2021 - 01:26 PDT
Identified
Starting from 12:15AM PST, Users would be experiencing an increase of failed loadings for the Box Notes inbox. We have identified the issue and are working to remediate. More information to come.
Posted Apr 17, 2021 - 01:06 PDT
This incident affected: Box Notes (Web Application, Desktop Client).