March 24, 2023 Regarding the event that occurred on March 22, 2023, in which the space could not be accessed

Thank you for using oVice.

We deeply apologize for the inconvenience caused to users regarding the inability to access the space that occurred on March 22, 2023.

We will report on the results of the investigation and measures to prevent recurrence of the event.

At around 15:00 on March 22, 2023, the ovice UI was updated and the update was successfully completed, but the server load increased more than expected during the processing process (processing for active users). An event has occurred.
As a result, the CPU utilization rate of the server reached 100%, the response was slow, and memory failure occurred on some servers. It became a situation where the response could not be accepted, and it became impossible to access the service.

■Cause The
main cause is an increase in the number of accesses due to reacquisition of user information when updating the DB, and inadequate predictions and countermeasures against the accompanying increase in system load.

In order to avoid this event, the memory was increased and the problem was resolved.

Recurrence prevention
After the failure on February 7, 2023, we have not implemented system updates on the backend side during the daytime in Japan time until we can guarantee safety by design. However, due to the design decision that the possibility of causing a system failure due to the front-end update is extremely low, and that no failure has occurred due to the front-end update so far, we will continue the update during the daytime in Japan time. I was doing it.

Regarding the future, we will not update during the daytime in Japan time until it is in a state where safety can be guaranteed by design, regardless of whether it is the front end or the back end. In addition, we will continue to consider the design and mechanism of safer deployments for updates outside of Japan time. 

Was this article helpful?
1 out of 1 found this helpful

Articles in this section

See more