pushr - Notice history

The team is preparing the master nodes in the cluster for a hardware reset, in our continuous attempts to resolve the issue which causes random temporary unavailability on file uploads and the S3 API. During the last maintenance window on March 22, a complete replacement of the hardware of a single master node was done, which ruled out faulty hardware component(s) as the root cause. Based on all collected information available from testing, available system logs and known specific bugs related to the AMD platform (on which Sonic is built), we will perform this reset to load the systems' kernel with the "iommu=pt" flag. This will allow us to pass through AMD's technology which enables virtualisation of I/O resources (AMD-Vi). Should this attempt fail at resolving the issue, a decision has been made to initiate a switch of all master nodes to a different type of servers powered by Intel.

The expected unavailability during this maintenance window is 15 minutes. It could be extended in the event that we hit a boot issue with the kernel flag enabled and need to revert the configuration.

We continue to observe instability in one of the master nodes in the Sonic cluster which leads to connectivity loss. After two unsuccessful attempts at fixing the issue, we've decided to replace the entire server in order to rule out the majority of possible hardware errors, and to possibly avoid the need for further maintenance windows and downtime.
At 1700 CET today the master node will be shut down, and it's disk drives will be moved to a new server. During this event, uploads will be disabled and the S3 API will be temporarily shut down. Availability of some of the stored objects will be impacted too. We do not expect any data loss. The procedure is expected to take up to 90 minutes until services are restored.

We would like to notify all customers that due to internal structure changes in our team, we expect to be slower than usual to respond to support tickets and live chat requests in the following days. We expect the get back to normal operation by early next week. We thank you for your understanding and your patience during this period.

Our team will be applying an update to the master nodes in the storage cluster to fix a runtime error that was noticed earlier today. Since this error affects the accessibility of the files in the cluster, we will be pushing this update on short notice. Expected unavailability of uncached content is 30 minutes.