Users Unable to Access Series & Modules on the Mindtickle Platform

Incident Summary On June 5, 2025, a temporary issue was observed where users were unable to view the Series Listing Page across the platform.

The root cause was traced to a misconfiguration in the infrastructure setup of a new caching service (Amazon DAX) introduced to improve system performance. The configuration error blocked communication between application services and the DAX cluster, leading to service disruptions. The issue was promptly identified and resolved by rolling back the change.

Impact Area The following functionality was impacted during the incident:

• Series Listing Page (site-wide impact)
  

Incident Timeline

• June 5, 2025, 1:37 AM PT: Users began experiencing issues with the Series Listing Page.
• June 5, 2025, 1:56 AM PT: The engineering team detected the issue and initiated an investigation.June 5, 2025, 2:08 AM PT: Rollback completed and services were restored to a stable state.
  

Root Cause Analysis The issue was caused by a misconfiguration in the security settings of the newly introduced Amazon DAX cluster in the production environment. The security group associated with the cluster did not allow required inbound traffic from the application services. This caused service pods to fail and led to errors on the Series Listing Page.

Next Steps and Preventive Actions

• Improved Health Checks: Applications will include readiness and liveness probes for external dependencies like DAX to detect failures earlier.
• Enhanced Monitoring and Alerts: DAX metrics will be integrated into the monitoring framework to ensure real-time visibility and proactive alerts.

We apologize for the inconvenience caused by this incident.