We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We investigated the remaining unhealthy devices, which are not related to the customer secrets vault upgrade. This incident is resolved.
Posted Apr 20, 2026 - 16:54 EDT
Update
We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We are investigating the remaining unhealthy devices, which do not seem related to credentials. We will provide another update by 4:30pm EDT.
Posted Apr 20, 2026 - 16:01 EDT
Monitoring
We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We are investigating the remaining unhealthy devices, which do not seem related to the credentials. We will provide another update by 4:00pm EDT.
Posted Apr 20, 2026 - 15:34 EDT
Update
We have restored the credentials for the subset of devices. We are monitoring closely to ensure proper ingestion of delayed alerts. We will provide another update by 3:30pm EDT.
Posted Apr 20, 2026 - 15:02 EDT
Identified
We have restored the credentials for the affected subset of devices. We are monitoring closely to ensure proper ingestion of delayed alerts. We will provide another update by 3:00pm EDT.
Posted Apr 20, 2026 - 14:34 EDT
Update
We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 2:30pm EDT.
Posted Apr 20, 2026 - 14:01 EDT
Update
We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 2:00pm EDT.
Posted Apr 20, 2026 - 13:35 EDT
Update
We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 1:30pm EDT.
Posted Apr 20, 2026 - 13:00 EDT
Investigating
At 10:59AM EDT, we noticed that a set of devices have been unhealthy since last week due to invalid credentials. We believe the cause to be a change to our device credentials storage subsystem. We are working to identify the specific set of devices and restore the credentials. Once restored, all delayed alerts will be ingested. We will provide an updated by 1pm EDT.