Everything is running normally.
Between about 7:45 and 10:07 AM PT, some outbound message requests to api.agentphone.ai were slower than normal during traffic bursts. Messages were still accepted and delivered; inbound messages, calls and the dashboard were unaffected. A fix was deployed and latency returned to normal.
Send latency has stayed at normal levels (median around 0.2 seconds) for two hours since the fix. Marking resolved.
12:00 PM PTSend latency back to normal since about 10:07 AM PT after a second fix. Requests that slowed down this morning were still accepted and delivered.
10:15 AM PTSome outbound message requests taking longer than normal during traffic bursts, in some cases 20 to 60 seconds. Messages are being accepted and delivered; nothing is lost.
9:50 AM PTA degraded host at our hosting provider caused some API requests to time out or return 502 between 12:43 and 12:55 PM PT. The API process stayed up; moving it to a healthy host resolved it. Dashboard and docs were unaffected.
API moved to a healthy host; all requests succeeding since. Calls that failed in the window can be retried.
12:55 PM PTDegraded host at our hosting provider; the API process stayed up but some requests routed through it failed.
12:50 PM PTSome requests to api.agentphone.ai timing out or returning 502.
12:43 PM PTOur hosting provider moved our primary database to new hardware. The copy ran while the database stayed online; the final sync and restart interrupted database-backed requests for about 20 seconds.
Migration finished. Interruption at the final sync lasted about 20 seconds; requests that failed in that window could be retried, nothing already accepted was affected.
10:17 PM PTBackground copy of the database volume to the new host began.
10:01 PM PTMaintenance window announced for 10:00 PM PT.
11:00 AM PTA hardware fault at one of our hosting providers degraded storage and private networking in their US West region, where our database runs. API requests were slow or timed out for roughly 40 minutes. Message delivery itself was unaffected.
Our hosting provider isolated the faulty hardware. Error rates and response times returned to normal.
6:40 PM PTTraced to a hosting provider outage affecting storage and private networking in US West.
6:20 PM PTElevated API error rates and timeouts.
6:00 PM PT