Sonic S3 gateway errors
Affected
Partial outage from 7:44 AM to 8:46 AM, Degraded performance from 8:46 AM to 9:49 AM, Partial outage from 9:49 AM to 11:53 AM, Operational from 11:53 AM to 8:55 PM
- Resolved · UTC
We consider this incident to be resolved. Since the last update there have been zero upload errors logged, and Sonic has been fully operational. We've also taken note of the fact that Sonic was not returning an error when uploads failed and resulted in 0 byte files. While failure due to the same circumstances as in this incident should no longer be possible in the future, we will be issuing another patch to address proper handling of such type of events.
- Monitoring · UTC
We've deployed a permanent fix and we observe a drop in upload errors to the S3 gateway. We expect error counts to reach zero within 30 minutes. Customers are advised to check the integrity of the files uploaded during this partial outage as some may actually be 0 bytes in size. We continue to monitor the situation and are prepared to resume work on this incident should the applied fixes are not enough to remedy the issue. No data has been lost during this incident.
- Update · UTC
Work on a permanent solution continues. At present some uploads towards Sonic might continue to fail. Updates will follow.
- Identified · UTC
We've identified the cause of the errors and have restored the operational state of the gateway. The team is now starting to work on a permanent solution to the issue.
- Investigating · UTC
We are aware of gateway timeouts on Sonic object storage. We are currently investigating this incident.

