We have performed the root cause analysis of the incident as part of the long-term resolution of incident management.
Impact:
Learners were facing intermittent issues loading the learning site dashboard widgets and the series pages.
Root Cause Analysis:
We have identified that this issue was caused due to a deadlock-like situation in the MySQL database server.
This resulted in increased I/O operations on the MySQL server that shot up the resource consumption to 100% within a few seconds.
Actions Taken:
As an immediate action, we restarted the MySQL database node to free up the I/O resources. Further, we took additional steps to over-provision the I/O operations capacity and upgrade to a more optimized database volume type.
As part of the preventive actions, we have optimized the queries that could possibly lead to such deadlock-like situations.
Posted Jun 01, 2021 - 06:46 PDT
Resolved
This incident has been resolved.
Posted May 27, 2021 - 10:32 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 27, 2021 - 10:24 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted May 27, 2021 - 10:15 PDT
Update
We are continuing to investigate this issue. Learners are facing intermittent issues loading the learning site dashboard, including the widgets and series pages. Learners can open the modules from the invitation/reminder emails and through the direct links. We will keep you posted on the progress.
Posted May 27, 2021 - 09:54 PDT
Investigating
We are currently investigating this issue.
Posted May 27, 2021 - 09:44 PDT
This incident affected: Knowledge (Course / Quick-Update / Assessment, Instructor-Led Training, Spaced Reinforcement), Practice and Execution (Mission, Coaching Sessions), Mindtickle Platform, and Interface (Learning Site).