We have performed the root cause analysis of the incident as part of the long-term resolution of incident management.

Impact:

• On May 24, 2021, between 4:30 PM to 5:52 PM PDT, a set of users received intermittent errors on the admin and learning site pages.

Root Cause Analysis:

• Upon receiving reports of these errors, we identified that a set of users on the admin and learning site were intermittently receiving 502 bad gateway and 503 service unavailable errors. These errors are caused when the server, while acting as a gateway or proxy, either gets no response or receives an invalid response from the upstream server.
• In this case, one of the upstream servers providing user management service was not reachable from the application gateway server. The IP addresses used to reach the user management service are cached by the application gateway and are updated once the IP addresses of the user management service are rotated.
• Upon further investigation, we found that the IP addresses of the user management service were rotated; however, some of the IP addresses were not updated in the DNS cache of the application gateway server.

Actions Taken:

• As an immediate action, we restarted the gateway server processes to flush the DNS entries cached by the application libraries.
• Further, we have added proactive alerting around such DNS issues to ensure we are immediately notified of such events.
• Additionally, we are exploring long-term approaches to automatically flush the DNS cache if we receive 502/503 errors continuously.