We have performed the root cause analysis of the incident as part of the long-term resolution of incident management.

Impact

• Learners were facing issues loading coaching and mission modules in the learning site.

Root Cause Analysis

• During our planned downtime, we migrated the coaching applications database from a self-managed MySQL instance to the AWS-managed RDS service.
• After the data migration, the database instances could not start due to a bug in the specific version of MySQL used by AWS RDS.

Actions Taken

• Following AWS Support Team’s recommendation to resolve the bug, we upgraded the MySQL used by the AWS RDS to a newer version in which the bug was resolved.
• After the MySQL version upgrade, the database instances were up and coaching applications started loading normally.
• As part of the preventive actions, we have added necessary checks in our migration process to ensure we identify such bugs during our dry run migration activity.