PUSHR's global system status is being updated automatically when our monitoring systems detect an issue with any of our services. If you are aware of an ongoing issue that is not listed here, please report it to us by clicking on the link above, or by opening a ticket from your account's dashboard.
Sonic Object Storage - Hardware replacement
Completed
Scheduled for March 22, 2023 at 3:00 PM – 4:50 PM
Affects
Sonic Object Storage
Under maintenance from 3:00 PM to 4:50 PM
Updates
Completed
March 22, 2023 at 4:50 PM
Completed
March 22, 2023 at 4:50 PM
We've migrated all disk drives successfully. All services have been re-enabled. This maintenance is now over and we are monitoring for errors.
Update
March 22, 2023 at 4:03 PM
In progress
March 22, 2023 at 4:03 PM
Data center technicians have now begun the process of moving the disk drives to a new server.
Update
March 22, 2023 at 3:04 PM
In progress
March 22, 2023 at 3:04 PM
We are now starting to drain existing upload connections in preparation to take a metadata snapshot. New upload connections (FTP) will be blocked, while existing ones will be allowed to finish. S3 API will be shutdown shortly.
In progress
March 22, 2023 at 3:00 PM
In progress
March 22, 2023 at 3:00 PM
Maintenance is now in progress
Planned
March 22, 2023 at 3:00 PM
Planned
March 22, 2023 at 3:00 PM
We continue to observe instability in one of the master nodes in the Sonic cluster which leads to connectivity loss. After two unsuccessful attempts at fixing the issue, we've decided to replace the entire server in order to rule out the majority of possible hardware errors, and to possibly avoid the need for further maintenance windows and downtime.
At 1700 CET today the master node will be shut down, and it's disk drives will be moved to a new server. During this event, uploads will be disabled and the S3 API will be temporarily shut down. Availability of some of the stored objects will be impacted too. We do not expect any data loss. The procedure is expected to take up to 90 minutes until services are restored.
