While this incident is resolved and we are fully operational, we wanted to share some additional details.
At approximately 2:21pm EDT, we detected a spike in traffic to element451.io (our internal admin console) of roughly 14x normal levels. This overwhelmed the ECS tasks powering the console, causing them to fail. While AWS typically auto-recovers this type of failure within under 60 seconds, it took approximately 4 minutes this time for ECS to replace the failed tasks and restore access.
Existing user sessions were not affected, and access to our other public-facing services (Events, Pages, Appointment Booking, and Apps) remained fully available throughout.
We're continuing to investigate the root cause of the traffic spike. In the meantime, we've increased our baseline task count and memory allocation for both the app and proxy layers to add headroom and reduce the chance of a recurrence.
Resolved
While this incident is resolved and we are fully operational, we wanted to share some additional details.
At approximately 2:21pm EDT, we detected a spike in traffic to element451.io (our internal admin console) of roughly 14x normal levels. This overwhelmed the ECS tasks powering the console, causing them to fail. While AWS typically auto-recovers this type of failure within under 60 seconds, it took approximately 4 minutes this time for ECS to replace the failed tasks and restore access.
Existing user sessions were not affected, and access to our other public-facing services (Events, Pages, Appointment Booking, and Apps) remained fully available throughout.
We're continuing to investigate the root cause of the traffic spike. In the meantime, we've increased our baseline task count and memory allocation for both the app and proxy layers to add headroom and reduce the chance of a recurrence.
Resolved
This incident has been resolved.
The issue affecting internal user/admin access to the Element451 console has been fully addressed, and all systems are operating normally.
If you continue to experience any problems, please contact our support team.
Thank you for your patience while we worked to resolve this issue.
Monitoring
A fix has been implemented for the issue affecting access to the Element451 dashboard. Which has now stabilized.
An issue at AWS caused a 4 minute outage for new sessions to load admin/internal user access to the Element451 console. AWS was able to heal the issue quickly. Existing sessions were unaffected as was access to other public facing sites like Events, Pages, Appt Booking and Apps.
We will provide a final update once we are confident the issue is fully resolved.