Resolved -
The incident has been resolved, and all delayed notifications have been delivered. No customer data was lost, and SOC operations were not impacted.
Feb 2, 17:33 EST
Monitoring -
We are observing that delayed messages are starting to be delivered. We will monitor the backlog of delayed notifications and the health of the system. We will provide an update by 22:30 UTC.
Feb 2, 17:12 EST
Update -
Outbound notifications from Workbench to User Email, Email Ticketing, Slack, MS Teams, PagerDuty, and OpsGenie continue to be delayed. We will provide an update by 22:30 UTC.
Feb 2, 17:00 EST
Update -
Outbound notifications from Workbench to User Email, Email Ticketing, Slack, MS Teams, PagerDuty, and OpsGenie continue to be delayed. We will provide an update by 22:00 UTC.
Feb 2, 16:41 EST
Identified -
We have identified the root cause as an issue with a 3rd party vendor. They are actively working on resolving the issue. We have confirmed that outbound notifications from Workbench have not been delivered to to User Email, Email Ticketing, Slack, MS Teams, PagerDuty, and OpsGenie since 19:30 UTC. These notifications are queued and will be re-sent when the issue is resolved. Alert ingestion is not affected. We will provide another update by 21:30 UTC.
Feb 2, 16:15 EST
Investigating -
Expel is investigating an issue with Workbench notifications that may be preventing users from receiving outbound notifications from Workbench to User Email, Email Ticketing, Slack, MS Teams, PagerDuty, and OpsGenie integrations since 19:30 UTC. We will provide an update by 21:30 UTC.
Feb 2, 16:00 EST