Microsoft has confirmed the underlying infrastructure issue in West US 2 has been mitigated, with the vast majority of downstream services restored. Log Analytics remains in active mitigation upstream, which continues to limit some of our internal observability, but all customer-facing services have returned to normal operation.

We are marking this incident resolved.

Microsoft has confirmed that datacenter utility power in West US 2 has been fully restored. Cooling has normalized, and the majority of upstream network and storage infrastructure has recovered. Two storage components are still completing final recovery and data integrity validation, with Microsoft estimating full restoration within the next couple of hours.

On our side, service metrics remain within baseline range and we have not observed any further degradation since this morning. We are continuing to monitor closely as the remaining upstream components finish validation, and will confirm full mitigation once Azure declares the event resolved.

We are not out of the woods yet. Azure has provided a more detailed update.

Cause:
Severe thunderstorm hit the West US 2 datacenter at ~04:27 UTC. Complete utility power loss, generator failover partially failed (synchronization + thermal protection issues), cooling systems also failed. Cascading failure exceeded N+1 redundancy. This has been ongoing for 13+ hours.

Services still impacted:
Azure SQL - DB
Redis - Cache
Azure Functions - Webhooks, edge compute
Storage
Application Insights/Log Analytics
VMs/Container Apps - Compute

We are monitoring metrics available for each of the above services. Log Analytics is the only service no longer live for us. The rest of the services are within range of baseline metrics.

Although we do not have an "all clear" from Azure, we have seen most service metrics return to baseline. The logging service is still down (Log Analytics) so we aren't able to see traces and errors. We do not believe this outage caused any loss of data in transit or at rest.

Timeline:
Though Azure reported their power event at 4:30 UTC, we started to see first impacts to our service at around 8am UTC. We then saw intermittent impact to emails, connectivity and backend performance until 17:30 UTC. Since then, we have seen metrics return to normal.

We will provide a more detailed outage report once the event is clearly behind us.

We are monitoring a power outage related incident currently underway at the Azure USW region. We have noticed elevated error rates in email delivery and device authentication. We will post updates as they are made available.

Further details about the incident and it's recovery are also available here. https://azure.status.microsoft/en-us/status