Admins and Reps were unable to view modules on the Series page

Incident Report for MindTickle, Inc.

Postmortem

On May 29, 2024,Admins & reps were unable to see modules on Mindtickle platform

Incident Start: May 29, 2024, 01:00 AM PT

Incident Resolved: May 29, 2024, 02:50 AM PT

We sincerely apologize for any inconvenience caused. We are committed to learning from this incident and improving our processes and systems.

Below is the incident's timeline, the root cause, and action items.

Incident timeline:

May 29, 2024, 01:00 AM PT: Backward incompatible code deployed to production
May 29, 2024, 01:35 AM PT: Team alerted via alerts
May 29, 2024, 02:30 AM PT: Root cause identified
May 29, 2024, 02:40 AM PT: Rollback Triggered
May 29, 2024, 02:50 AM PT: System restored to normal operation

Root Cause:

The incident was caused by an upgrade of the protobuf-java library version from 3.5.1 to 3.24.0 in critical services of the Mindtickle Platform through the platform-protos dependency. This version upgrade was not backward compatible and had breaking changes that affected the deserialisation of the protos leading to disruption of service.

‌

Learning and Next Steps:

Improve version control: It is a standard practice to use the latest version of libraries we depend on, going forward we will move to using specific predetermined versions. This will help us avoid any issues that may arise from conflicting changes.

Posted Jun 05, 2024 - 08:39 PDT

Resolved

On 29 May 2024, between 01:05 PT - 02:46 PT, admins, and reps were unable to view the modules (Courses, Quick-Update, Assessments, Checklist, ILT, Reinforcements, Missions & Coaching) on the Series page. However, any users already accessing the modules before 01:05 PT, would not have faced any concerns.

The issue was resolved completely at 02:46 PT. We will share a detailed RCA of the incident by Tuesday, 04 June 2024.

Posted May 29, 2024 - 04:12 PDT

This incident affected: Knowledge (Course / Quick-Update / Assessment, Instructor-Led Training, Spaced Reinforcement), Practice and Execution (Mission, Coaching Sessions), and Interface (Admin Site, Learning Site, Mobile App).