** Update: we published a service that allows you to check whether your organization is vulnerable here
Today at Black Hat, Wiz CTO Ami Luttwak and I are presenting on a new class of vulnerabilities we discovered that exposes valuable dynamic DNS data from millions of endpoints worldwide. DNS (Domain Name Service) is one of the foundations of the Internet, an immensely complex and decentralized system that, at its core, translates readable domain names (like nytimes.com) to numerical IP addresses.
There’s a proud tradition of DNS research at Black Hat, most famously in 2008 when the late great Dan Kaminsky prevented Internet Armageddon by exposing some of its fundamental flaws. Generally speaking, DNS has become a lot safer since then. Still, DNS vulnerabilities are usually critical because they put billions of devices around the world at risk.
Today the rise of managed DNS providers (like Amazon Route53, Google Cloud DNS, and Akamai, to name just a few) and the ubiquity of remote work are stretching and tearing new holes in the fabric of this decades-old protocol designed for a world where employees and servers were all “on-prem.”
Our research in a nutshell
We found a simple loophole that allowed us to intercept a portion of worldwide dynamic DNS traffic going through managed DNS providers like Amazon and Google. Essentially, we “wiretapped” the internal network traffic of 15,000 organizations (including Fortune 500 companies and government agencies) and millions of devices. It was a bottomless well of valuable intel - computer names, employee names and locations, and details about organizations’ web domains including entry points that are exposed to the internet.
We have no way of knowing whether the loophole has already been exploited: Anyone could have collected data undetected for over a decade.
We do know this is still an active threat vector – while two major DNS providers (Amazon and Google) have fixed the issue, others may still be vulnerable. As a result, millions of devices are potentially vulnerable.
How managed DNS providers work — and how we exploited it
For our purposes, there are two major players overseeing DNS: Domain registrars and DNS hosting providers. Your DNS host is the service that is authoritative for hosting your DNS records. A domain registrar is where you purchase domain names. There are DNS hosting providers that offer domain registration and vice versa, but the two services should not be confused.
DNS hosting providers have an easy-to-use, self-service platform that allow customers to make updates to their domain name and what name servers it points to. Customers can add any domain name they want (for instance, amazon.com or ami.is.the.greatest.com) because it’s not supposed to have any impact on web traffic (they’re not the authoritative domain registrar). The basic assumption is that there is full isolation between you and other customers. Route53 doesn’t verify that I own amazon.com because nothing that I register on my DNS is supposed to have any impact on other customers.
What we found was that registering certain "special" domains, specifically the name of the name server itself, has unexpected consequences on all other customers using the name server. It breaks the isolation between tenants. We successfully registered one type of special domain, but we suspect there are many others.
What did we do exactly?
AWS Route 53 has around 2,000 DNS servers that are used and shared among all their customers. We registered a new domain on the Route53 platform with the same name as their official DNS server. (Technically, we created a new “hosted zone” inside AWS name server ns-1611.awsdns-09.co.uk and named it “ns-852.awsdns-42.net”.)
Whenever a domain is added to Route53, four different DNS servers are selected to manage the domain. We made sure that any nameserver we register on the platform falls under the management of the same server. In fact, we repeated this process on about 2,000 name servers on AWS alone ( ns-1611.awsdns-09.co.uk is just one example).
Now we partially control the hosted zone, so we can point it to our IP address. Whenever a DNS client queries this name server about itself (which thousands of devices do automatically to update their IP address within their managed network – more on that in a minute), that traffic goes directly to our IP address.
What traffic did we receive?
When we first registered this new domain with the same name as the Route53 name server, we immediately started receiving a flood of traffic from more than a million unique endpoints around the world. After analyzing it, we learned it was dynamic DNS traffic from Windows machines that were querying the hijacked name server about itself. Dynamic DNS keeps DNS records automatically up to date when an IP address changes. It’s traditionally been used in large networks that host internal services, and use their own internal servers. In short, the traffic we received contained sensitive information that was never supposed to leave an organizations internal network.
The dynamic DNS traffic we “wiretapped” came from over 15,000 organizations, including Fortune 500 companies, 45 U.S. government agencies, and 85 international government agencies. The data included a wealth of valuable intel like internal and external IP addresses, computer names, employee names and office locations.
Why did we receive this traffic?
The short answer is that Microsoft machines use a unique algorithm to find and update the master DNS server on IP address changes. Eventually the algorithm will query the hijacked nameserver for its own address. The result? Since we’ve directed that server to our malicious IP address, we start receiving all of the query traffic.
To better understand this, imagine a Wiz employee decides to work from home - like most of us lately - and connects to their home WiFi. Their work laptop gets an internal IP address from their home router, and will try to find the company’s local master server to update it with this new address.
Eventually, the endpoint will try to update the master server, which is an AWS shared server that manages thousands of customers. AWS name servers do not support dynamic DNS updates, so the update request will fail.
So far the Microsoft algorithm works exactly as expected, and at this point it should stop and give up on updating the master server. But that’s not what happens – and here’s where the problem arises. Instead of giving up, Microsoft tries to find the master DNS server in another way. The next step will be to check if Wiz’s name servers have records for the master server.
AWS’s name server responds with the IP address we’ve provided, in this case 1.3.3.7. This is where the Windows endpoint will send the dynamic update...inadvertently leaking it’s internal IP address, computer name, and other info to our malicious DNS server.
How could this data be used?
Here’s where things get scary. The traffic that leaked to us from internal network traffic provides malicious actors all the intel they would ever need to launch a successful attack. More than that, it gives anyone a bird’s eye view on what's happening inside companies and governments. We liken this to having nation-state level spying capability – and getting it was as easy as registering a domain.
For example, below is one of the world’s largest services companies. We mapped the locations of their employees and offices based on the traffic we received from 40,000 endpoints.
Our snooping didn’t stop there: We were able to identify companies that appear to be operating in violation of OFAC (Office of Foreign Assets Control) sanctions. A major commodity trading company had employees working in the Ivory Coast and Myanmar, and a subsidiary of a large credit union with a branch in Iran.
Who’s responsible for solving the problem?
The DNS vulnerability we discovered falls into murky “international waters” — it’s unclear who should be responsible for fixing it. Microsoft? Managed DNS services? Individual organizations?
Microsoft could provide a global solution by updating its dynamic DNS algorithm. However, when we reported our discovery to Microsoft, they told us that they did not consider it a vulnerability but rather a known misconfiguration that occurs when an organization works with external DNS resolvers.
Managed DNS providers can also make changes to help their customers operate securely. For instance, they could start verifying ownership of domains before allowing customers to register them.
But for the time being, every organization should take steps to prevent their data from leaking. Ultimately, customers are responsible for configuring their DNS resolvers properly so dynamic DNS updates do not leave the internal network.
