Sora ID - Widespread API issues – Incident details

Widespread API issues

Resolved
Major outage
Started almost 3 years agoLasted about 2 hours

Affected

API

Major outage from 7:16 AM to 8:47 AM

Updates
  • Resolved
    UTC
    Resolved

    Service has been restored. The database was restored using a backup from about 12:01am ET on 30 Sep. This means that about 3 hours of data was lost in the immediate recovery (12am - 3am ET) and we will work to restore this data as we regain access to the original database, which is still non-operational.

  • Monitoring
    UTC
    Monitoring

    We have restored service using a backup database. We are monitoring the fix and when verified will communicate recovery point and any further action.

  • Identified
    UTC
    Identified

    We have identified an issue with our database and we're working to get back up and running as quickly as possible. In parallel we are pursuing two paths:

    1. Restoring service using our existing database. We are having trouble getting our existing managed database on GCP to become healthy but are optimistic about this path
    2. A point-in-time recovery of the original database from just before this incident is being performed
      Whichever of these completes first will resolve this outage.
  • Investigating
    UTC
    Investigating

    We are investigating widespread issues with our API and will update soon once a cause is identified.