pushr - DNS connectivity issues – Incident details

PUSHR's global system status is being updated automatically when our monitoring systems detect an issue with any of our services. If you are aware of an ongoing issue that is not listed here, please report it to us by clicking on the link above, or by opening a ticket from your account's dashboard.

DNS connectivity issues

Resolved11 hours 14 minutes

Affected

Anycast DNS

Operational from 12:23 AM to 11:37 AM

  • ResolvedUTC

    This incident has been resolved and the issue has been confirmed to be a software error in upstream providers equipment. We've re-enabled all affected DNS PoPs and latency is back to normal.

  • MonitoringUTC

    We now have global connectivity restored but will not rush into putting the affected name servers online again. We continue to monitor the situation and to wait on information from upstreams on the root cause of this incident. The team will gather at start of business hours to discuss this unusual event and the way forward to avoid such disruptions in the future. This incident will be updated once again when all name server nodes are re-introduced to the network.

  • IdentifiedUTC

    We've identified the scope of the issue and it seems to affect EU and parts of the middle east. We do not yet know the actual cause, but it appears that our anycast IP prefix is being advertised from our provider that serves all EU locations, but the name server IPs themselves are actually not reachable which leads to a black hole. We are now shutting down all instances in EU and expect that this will route DNS traffic to US and SA. This should provide a temporary fix until we are able to understand this situation better.

  • InvestigatingUTC

    We are currently investigating this incident. PUSHR's DNS seems to be unreachable from multiple locations globally. Reason not yet known. Some services may be unavailable.