Techies - please note that I am trying to explain this to non-techies...
This FYI about one of the things that makes the internet work would be enlightening to some here who are new to the notion of "metadata", where it goes, and to whom it is available. Specifically, it might be a good idea to understand something about the Domain Name System (DNS). I'm going to simplify a little bit, so before someone wants to jump on me about DNS caching, DNS TTL records, private and alternate DNS servers and other complications, it is important to nail down the basics, which obviously most people do not understand.
------
A "domain name" is sort of like an address for identifying the location of a resource on the internet - be it a web site, mail server, file server, and so on. "Democraticunderground.com" is a domain name.
Now, you can reach "Democraticunderground.com" by typing that address into your browser, but what happens after you hit return is that a series of events takes place to eventually cause your computer to retrieve information from a server located at a numeric address - specifically
216.158.28.195 - which is the address needed by your computer, and a bunch of machines in between, to communicate with Democraticunderground.com.
Go ahead and type
216.158.28.195 into your address bar - it will connect you to Democraticunderground.com just as if you had typed the name "Democraticunderground.com" instead of the IP address of the server addressable by 216.158.28.195. If you are afraid to type a strange number into your browser, I'm sure other readers can confirm that:
http://216.158.28.195 - indeed goes straight to DU.
Domain names are handy because it is a lot easier to remember "Democraticunderground.com" instead of
216.158.28.195. They are also handy for a lot of other reasons, such as if DU were to move to another server, then the name "Democraticunderground.com" can remain the same, while records indicating "where do I find democraticunderground.com" can simply be updated with a new numerical address.
Now, backing up for a second, you might appreciate that in order for this to work, there needs to be a system for answering that question "where do I find democraticunderground.com" in order for your computer to be able to look up the domain name, find the IP address, and then connect to the resource. That function is provided by the Domain Name System (DNS).
Your computer does not have, somewhere inside of it, a big "phone book" for looking up IP addresses associated with domain names. If it did, it would defeat the purpose of being able to have a system which can handle address changes of resources, locating new resources, and so on. In other words, if you had a big local directory of these things, it would be constantly obsolete.
Instead, the way that DNS lookups happen is a multi-step process. At its most basic level, When you type in "www.democraticunderground.com" the Domain Name System does the following things:
1. Your computer first asks, "where do I look up names in .com?" As you know, there are many "top-level domains" (TLDs) such as .com, .net, .org and so on, including all of the country-code TLDs like .uk, .de, and so on. Just who your computer asks, I'll get to in a minute. But your computer gets the answer "you look up .com names at a DNS server located at IP address X" where "X" is the address of a server run by Verisign, the .com registry. If it had been a .org name, X would be a server run by Afilias, which handles the .org registry. If it had been a .biz name, X would be a server run by Neustar, the .biz registry. And so on.
2. Your computer takes that address X - the location of the DNS server for that TLD - and then goes to that address and asks "where do I find democraticunderground in .com?". The DNS server for the TLD, run by Verisign, provides the answer to that question, which is the address of a server that was specified by Elad to his domain registrar, and is the IP address of the DNS server for "democraticunderground" in the ".com" TLD. Let's call it address Y.
3. Using address Y, your computer asks that server, "what address do I use for communicating in hypertext for democraticunderground?" and is then given the final answer, from the DNS server for democraticunderground, of 216.158.28.195.
Now, looking at those steps, your machine asks a series of questions to different machines along the way of simply finding the IP address of democraticunderground.com. It first asks the "root server" where is .com. It then asks the .com DNS server where is democraticunderground. And then it finally asks the democraticunderground DNS server where is "w w w".
All of these machines are operated by different people, and each of them is able to log (a)
the IP address of your machine (because of course, your machine wouldn't get an answer otherwise) and (b)
the address you are looking up. In other words, there are several parties who, in the course of making the internet work, have access to EVERY address to which you have sought to connect using DNS. That's every time you connect to a website, send an email, retrieve a file from an ftp server, and so on.
Now, let's back up and take a look at step 1 in that process - the part where your computer says to the "root server", "I am looking up www.democraticunderground.com, can you tell me where to find .com?"
Do you know who RUNS that server?
Before getting to the answer to that question - i.e. just who is it that can effectively know everything you look at or communicate with on the internet - I want to describe in a bit more detail what the root server system does. The root server system is exceedingly simple. All it does is to keep a list of all the top-level domains, .com, .net, .org, and all of the other TLDs - some 150 or so of them (including the country codes). A complete list is here
https://en.wikipedia.org/wiki/List_of_Internet_top-level_domains
The entire data set of "top level domain - DNS server for that domain" is simply a small file, less than 200 kB, which contains that information just like a telephone book.
As it turns out the "root server" is not one machine, but is a distributed set of 13 machines in different locations around the world. They, and their mirrors, all copy the master file kept by the A root server, and DNS queries are distributed among the root servers depending on where you are, DNS traffic volume, and other conditions. However, operating any ONE of the root servers, gives you a huge slice of "who is communicating with what".
Here is the list of the 13 root servers, designated as A through M:
http://en.wikipedia.org/wiki/Root_name_server
You may note that three of them are directly operated by the US government:
E 192.203.230.10 ns.nasa.gov NASA
G 192.112.36.4 ns.nic.ddn.mil Defense Information Systems Agency
H 128.63.2.53 aos.arl.army.mil U.S. Army Research Lab Aberdeen Proving Ground, Maryland, U.S.
The Aberdeen Proving Ground, as you may know, is just a short hop up US Route 40 from Fort Meade.
So, one thing that anyone who uses the internet should understand is that by using the Domain Name System to visit websites, send email, retrieve files, and so on, you are CONSTANTLY sending metadata to three machines operated by the US government, and two of which are operated by the US Department of Defense. That goes for everyone on the planet who uses the internet.
It is also worth knowing that companies like Verisign, which runs the .com servers, have very, very cozy relationships with other organizations nearby their Herndon, Virginia base of operations.
It is an inherent feature of the internet that three of the root DNS servers are, and have long been, operated by the US government. It is a legacy of the history of the internet that it was an ARPA project which, had it not been for the intervention of Senator Al Gore, would have not been made available to the public. At the bottom of the "Al Gore invented the internet" story is the truth that it was his legislation which transitioned the development of policy around the root server system, and its continued operation, from the DoD to the Department of Commerce. The DoC, in turn, essentially contracts out the root server policy-making function (e.g. who gets to operate which TLDs) to a California non-profit corporation called the Internet Corporation for Assigned Names and Numbers (about which, if I went into any detail, this post would never end).
I've had the opportunity to visit a root server installation, and observe in real time the flood of information that pours in during the course of serving DNS queries from the root level - "Someone's computer at address X is looking up 'thing.example.com'" and so on. Gazillions of those queries, every day.
You do not have any kind of "user agreement" with the operators of the root server system, because the system assumes that if you are using it - and of course you are using it every day - then you accept the inherent features of how it operates. In other words, the internet assumes you know what you are doing.
Now, yes, techie friends, there is a lot of caching going on, not all queries go all the way to the root because your ISP is not looking up ".com" all day long, etc. and so on. However, unless you have assigned your DNS settings elsewhere (and most people don't) then your ISP certainly has the full data. But for the type of "traffic analysis" relevant to what intelligence agencies do with the data, there is enough traffic data at the root level which can indicate statistically significant traffic variations among, say, a set of IP addresses in Yemen which are looking up IP addresses in Milwaukee, and so on. Again, the DNS system does not have access to the contents of the packets going between those IP addresses, but it is an inherent feature of plain vanilla DNS that the identity of a huge swath of "what IP addresses are looking up what URLs" is directly transmitted to any of three US government computers -
BECAUSE THAT'S THE WAY THE INTERNET WORKS.
