pushr - Sonic S3 gateway errors – Incident details

We consider this incident to be resolved. Since the last update there have been zero upload errors logged, and Sonic has been fully operational. We've also taken note of the fact that Sonic was not returning an error when uploads failed and resulted in 0 byte files. While failure due to the same circumstances as in this incident should no longer be possible in the future, we will be issuing another patch to address proper handling of such type of events.

We've deployed a permanent fix and we observe a drop in upload errors to the S3 gateway. We expect error counts to reach zero within 30 minutes. Customers are advised to check the integrity of the files uploaded during this partial outage as some may actually be 0 bytes in size. We continue to monitor the situation and are prepared to resume work on this incident should the applied fixes are not enough to remedy the issue. No data has been lost during this incident.

Work on a permanent solution continues. At present some uploads towards Sonic might continue to fail. Updates will follow.

We've identified the cause of the errors and have restored the operational state of the gateway. The team is now starting to work on a permanent solution to the issue.