Service Incident Postmortem

Summary
On May 19, 2026, between approximately 16:11 and 16:46 UTC, Yield API 2.0 and the Legacy API experienced elevated latency and an increased rate of request failures (HTTP 5xx errors) following a routine production deployment. Service was fully restored within roughly 35 minutes. We have not identified any failed customer write actions during the window.

Impact
- Window: ~16:11–16:46 UTC (approximately 35 minutes)
- Affected services: Yield API 2.0, Legacy API
- Symptoms: increased response times, intermittent request timeouts, and elevated 5xx errors
- Synthetic monitoring probes timed out during the window; no customer write/POST operations were confirmed to have failed.

Root cause
The deployment included a routine database schema migration. That migration required a brief exclusive lock on a heavily-used table. Because the table was simultaneously serving a high volume of concurrent activity, the migration could not acquire the lock and instead created lock contention. This backed up queries and produced the timeouts and elevated error rates seen across both APIs.

Resolution
Our team opened incident response within minutes of the first alert. We canceled the in-progress release, rolled the deployment back across all regions, and cleared the contending database activity. Once contention was relieved, the migration completed near-instantly and all services recovered, with monitoring alerts auto-resolving by ~16:46 UTC. The release was subsequently re-applied successfully with the migration handled separately.

Preventative actions
Enforce a strict timeout on deployment-time database migrations so a blocked migration fails fast rather than holding locks.
Move schema migrations into a dedicated, isolated release step instead of running them inline with application deployments.
Fix automated rollback so a canceled release reverts without manual intervention.
Expand monitoring coverage (database metrics and logs, deployment events) for faster detection and diagnosis.
Strengthen incident-response and customer-notification procedures.

We apologize for any disruption. If you have questions about how this incident may have affected your integration, please reach out to your Yield.xyz contact.

Error rates on our API have returned to normal. The incident was automatically resolved. Duration: 7 minutes.

We are investigating elevated error rates on our API. Some requests may experience failures or degraded performance. Our team is actively monitoring the situation. Started at 16:28 UTC.