We have performed the root cause analysis of the incident as part of the long-term resolution of incident management.

Impact

• MindTickle users could not access the platform on June 30, 2021, between 1:53 PM and 2:50 PM PDT.

Root Cause Analysis

• After receiving the alerts from our monitoring systems, we reviewed the impacted database (Elasticsearch) performance and health metrics.
• An ad-hoc read query was executed at 1:48 PM PDT for an internal analysis which turned out to be an expensive query and exhausted memory resources (JVM) of the database. By the time we figured out the impact, the database cluster could not serve new requests.
• A human error resulted in the expensive query being triggered.

Actions Taken

• We immediately terminated the query that caused this heavy database resource consumption.
• We restarted the impacted database nodes to free up the resource consumption, and at 2:50 PM PDT, the platform was up with minimum required resources. By 3:09 PM PDT, the platform was up and running in its normal capacity.
• As a next step, we are augmenting a mechanism to identify such expensive queries and process them through our existing gating system.