Sora ID - Widespread API issues – Incident details

Widespread API issues

Resolved
Major outage
Started 12 months agoLasted about 2 hours

Affected

API

Major outage from 7:16 AM to 8:47 AM

Updates
  • Resolved
    Resolved

    Service has been restored. The database was restored using a backup from about 12:01am ET on 30 Sep. This means that about 3 hours of data was lost in the immediate recovery (12am - 3am ET) and we will work to restore this data as we regain access to the original database, which is still non-operational.

  • Monitoring
    Monitoring

    We have restored service using a backup database. We are monitoring the fix and when verified will communicate recovery point and any further action.

  • Identified
    Identified

    We have identified an issue with our database and we're working to get back up and running as quickly as possible. In parallel we are pursuing two paths:

    1. Restoring service using our existing database. We are having trouble getting our existing managed database on GCP to become healthy but are optimistic about this path
    2. A point-in-time recovery of the original database from just before this incident is being performed
      Whichever of these completes first will resolve this outage.
  • Investigating
    Investigating

    We are investigating widespread issues with our API and will update soon once a cause is identified.