pushr - Notice history

This incident has been resolved.

We've managed to resolve the temporary HTTP5xx errors on newly uploaded content. Step 2 of the patch will now be applied, and it should provide a permanent fix for the drives unavailability issue, but will not provide a permanent fix for the HTTP5xx issue. We will keep this incident open, but Sonic's state is now back to fully operational.

We are continuing to work on a fix for this incident. A 2-step patch is being applied. In step 1 we've reconnected the offline drives and content previously unavailable is now online. In step 2 we are addressing the root cause of the issue which so far is believed to be related to temporary loss in connectivity between some servers in the cluster. This connectivity loss is also believed to be the cause of another issue that we've received reports for today - some newly uploaded files may be returning HTTP5XX errors upon download. We are still attempting to confirm the link between the two issues and will be holding back step 2 until we have better understanding of the second issue.

We’ve identified the issue. A patch is being prepared that should fix the issue. We’ll be working on this non-stop until fully resolved. Next update will come in a few hours.

Closing this incident as resolved, and we continue to work on permanent fix.

We've traced the issue down to a malfunction in our origin shield implementation. We've temporarily disabled the origin shields on pull zones to allow proper edge synchronisation. A permanent fix is coming.