Small subset of users unable to log into application.
Incident Report for

5-10-19 Incident Report

The Event

On May 10th, 2019 at 08:46 AM PST, a small subset of users were unable to initially log into the application. Those customers who were using a custom domain name for their site, or those customers who were already attached to the system were not affected.

The proactive monitoring system for alerted our staff who immediately responded to the alert. During the response the problem was identified to be an internet DNS system that was unable to process changes in a timely manner.

This caused the DNS system to not respond to a client request when they were initially trying to log into the system.

The DNS system in question finished all changes and was responding again to all internet lookups at 08:58 AM PST. At no time was the actual system unavailable, just the DNS lookup to reach the system at initial login was failing.

Lessons Learned now has full control over all Internet DNS records for the system. This will allow a much faster response to rate limits imposed by the internet DNS systems if this issue occurs again in the future. also will take the rate limits of the Internet DNS system into account while reviewing any proposed changes to the system in the future with the goal of working around those rate limits.

Posted May 30, 2019 - 04:19 PDT

A small subset of users were unable to initially log into the application.
Posted May 10, 2019 - 08:46 PDT