pushr - Sonic Object Storage - Hardware replacement – Maintenance details

All systems operational

PUSHR's global system status is being updated automatically when our monitoring systems detect an issue with any of our services. If you are aware of an ongoing issue that is not listed here, please report it to us by clicking on the link above, or by opening a ticket from your account's dashboard.

Sonic Object Storage - Hardware replacement

Completed
Scheduled for March 22, 2023 at 3:00 PM – 4:50 PMUTC

Affects

Sonic Object Storage

Under maintenance from 3:00 PM to 4:50 PM

Updates
  • Completed
    March 22, 2023 at 4:50 PMUTC
    Completed
    March 22, 2023 at 4:50 PMUTC

    We've migrated all disk drives successfully. All services have been re-enabled. This maintenance is now over and we are monitoring for errors.

  • Update
    March 22, 2023 at 4:03 PMUTC
    Update
    March 22, 2023 at 4:03 PMUTC

    Data center technicians have now begun the process of moving the disk drives to a new server.

  • Update
    March 22, 2023 at 3:04 PMUTC
    Update
    March 22, 2023 at 3:04 PMUTC

    We are now starting to drain existing upload connections in preparation to take a metadata snapshot. New upload connections (FTP) will be blocked, while existing ones will be allowed to finish. S3 API will be shutdown shortly.

  • In progress
    March 22, 2023 at 3:00 PMUTC
    In progress
    March 22, 2023 at 3:00 PMUTC

    Maintenance is now in progress

  • Planned
    March 22, 2023 at 3:00 PMUTC
    Planned
    March 22, 2023 at 3:00 PMUTC

    We continue to observe instability in one of the master nodes in the Sonic cluster which leads to connectivity loss. After two unsuccessful attempts at fixing the issue, we've decided to replace the entire server in order to rule out the majority of possible hardware errors, and to possibly avoid the need for further maintenance windows and downtime.
    At 1700 CET today the master node will be shut down, and it's disk drives will be moved to a new server. During this event, uploads will be disabled and the S3 API will be temporarily shut down. Availability of some of the stored objects will be impacted too. We do not expect any data loss. The procedure is expected to take up to 90 minutes until services are restored.