Resolved -
The incident has been resolved, and all previously-available assemblers are once again fully available.
Jun 23, 16:04 EDT
Monitoring -
We have implemented a fix and believe the incident to be resolved. We are monitoring the health of the infrastructure as individual assemblers come back online, and will resolve this if there are no outstanding issues. Please let us know if you are experiencing any problems. We will provide another update by 4:30pm EDT.
Jun 23, 15:50 EDT
Update -
We have identified a further cause and are working on a remediation. Individual assemblers are coming back online. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 12:30 EDT.
Jun 23, 11:32 EDT
Update -
We have identified a further cause and are working on a remediation. Individual assemblers are coming back online. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 11:30 EDT.
Jun 23, 10:31 EDT
Update -
We have identified a further cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 10:30 EDT.
Jun 23, 09:30 EDT
Update -
We have identified a further cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 9:30 EDT.
Jun 23, 08:42 EDT
Update -
We have identified a further cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 8:30 EDT.
Jun 23, 08:08 EDT
Update -
We are continuing to work on a fix for this issue.
Jun 23, 08:07 EDT
Update -
We have identified the root cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 8:00 EDT.
Jun 23, 07:47 EDT
Update -
We have identified the root cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 7:30 EDT.
Jun 23, 07:00 EDT
Identified -
We have identified the root cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 7:00 EDT.
Jun 23, 06:30 EDT
Investigating -
At approximately 4am EDT, we began experiencing a problem with our assembler VPN system that is preventing alerts from being ingested through assemblers. Alert ingestion is delayed for some devices behind an assembler. We will provide an update by 6:30am EDT.
Jun 23, 05:51 EDT