pushr - DNS connectivity issues – Incident details

This incident has been resolved and the issue has been confirmed to be a software error in upstream providers equipment. We've re-enabled all affected DNS PoPs and latency is back to normal.

We now have global connectivity restored but will not rush into putting the affected name servers online again. We continue to monitor the situation and to wait on information from upstreams on the root cause of this incident. The team will gather at start of business hours to discuss this unusual event and the way forward to avoid such disruptions in the future. This incident will be updated once again when all name server nodes are re-introduced to the network.

We've identified the scope of the issue and it seems to affect EU and parts of the middle east. We do not yet know the actual cause, but it appears that our anycast IP prefix is being advertised from our provider that serves all EU locations, but the name server IPs themselves are actually not reachable which leads to a black hole. We are now shutting down all instances in EU and expect that this will route DNS traffic to US and SA. This should provide a temporary fix until we are able to understand this situation better.