What they mean when the government says “We do not have ‘direct’ access to your info”

Summary: Even the best journalists and national security experts have difficulty with technical stories like the recent NSA revelations. Today Marcus Ranum (bio) cuts through the government’s lies, explaining the truth behind the NSA’s tapping vital telephone and email communication systems.

These are the small ones.
These are the small ones; America’s nerves


When politicians and spokespeople choose their words with exquisite care, then it’s time to examine them with extra care. Let’s talk a little bit about the realities of how one might monitor a data center, shall we?


“We have no direct access to their systems.”

Of course you don’t. By “direct access” you mean that you can log in and collect data directly from the system, or have database administrators’ credentials and can issue queries, or whatever. You wouldn’t want that, anyway, because the queries and the activities might then become public knowledge — those are traceable, you know.

When someone logs into a system, gains administrative rights, and looks at someone’s email in-box that leaves traces in the system logs, and that’s completely unacceptable because what you’re querying for is classified and suddenly those system logs contain extremely sensitive data, indeed.

Here’s how you do it

Those big outfits decrypt all their traffic at the edges of the network using a load-balancer/redirector that’s capable of offloading the CPU-intensive activity of decryption from the backend servers. Inside the provider’s core network, the traffic carried within their switches is all in the clear.

You show up with a national security letter and maybe a warrant and tell the provider that you’ve got a system that does classified stuff and they’re going to plug it into their network and have the core switches span some of the traffic between, say, the mail servers and everything else, and the user authentication servers and everything else, and send a copy of that traffic to the mystery box (or boxes, depending on the load you need to consume) and that’s it.

There’s no need even to give the box an IP address, which is a feature also, because that makes the box impossible for anyone to see other than in the configuration of the core switch or if they get into the special locked room in the data center and count the number of boxes in the rack there.

The box is a sniffer. Remember the old FBI CARNIVORE system that was “outed” back in 2000? That’s how CARNIVORE worked, pre 9-11. The newer systems may look like Insight.

Sniffing traffic is fairly straightforward

Government cyberwarriors: our enemy
Government cyberwarriors: our enemies


You collect raw packets, reassemble them into virtual streams, collect statistics about the stream, extract whatever data you’re interested from the stream, and do whatever analysis you want to on that data. This is how load balancers (like the SSL accelerator I referred to earlier) and intrusion detection systems work.

This is ideal for a classified program since the actual analysis method used: what data is collected, whether it’s message headers or full contents, etc, can remain completely internal to the collecting device. That way there’s no need to pester the security people at the provider if you want to update your collection rules: you just give yourself a classified order to now start collecting something new, on your box, in their data center.

Remember how google famously claimed that their wireless sniffers weren’t collecting sensitive data, and they later “remembered” that oops, dear me, the system was “misconfigured” to collect too much? That’s how you do it. Except if you can layer the whole thing under layers of classification, it’s even harder for anyone to learn what’s going on. Besides, not having an IP and not actually having to touch the providers’ systems keeps you out of the potential problem-space of being a cause of failure; facebook or google are going to be mighty touchy about down-time caused by your collection system and you can take your sniffer offline whenever you want to, without impacting any of the surrounding systems.

In other words, having “direct access” would be a huge disadvantage for you, because you now have a greater potential for information about your collection program leaking out. No, they do not have “direct access” to google, facebook, twitter, etc. They have something better: field-programmable completely invisible classified and unregulated access.

“We have never heard of a program called PRISM”

So says the corporate spokesperson. Well, that’s true. Because you’re a marketing spokesperson and you’re completely unaware of the boxes in the special room in the data center. But even if you were, you wouldn’t call it “PRISM” because that was (until now) the classified name of the program. You’d call it “the special room in the data center” or “the spook closet” or “shut up we don’t talk about that.”

If you work for one of the agencies that consumes the data, you probably don’t even call it PRISM. To you it’s just “the data you work on” and how it was collected and managed is protected source and methods outside of your need to know.

“We don’t listen to all the phone calls …”

… just key words “all” and “listen.”

If you collect the call data while it’s moving on a network, it’s just encoded voice signals. If you run those through a speech-to-text engine and then search it as text you’re not “listening” to anything because no ears are involved and no sound is produced. Of course you run that text through a scoring engine and pattern-match for interesting keywords, and perhaps — just perhaps — a human analyst eventually plays the data to make sounds that they listen to with their ears.

If you’d like to see this kind of sophistry in action, I suggest you watch Mike McConnell’s comments in the Intelligence-squared debate with Bruce Schneier and Mark Rothenberg. The debate took place in 2010, and I think that now, you’re a bit more qualified to read between the lines of what McConnell is saying.

First off, you’ll realize that he’s not lying out of ignorance — he’s very very carefully spinning the truth. There’s one moment in the debate when McConnell rebuts Schneier by saying “If I wanted to tap your home phone, I’d have to get a warrant…”   The key word is “home“. Presumably McConnell threw that in there deliberately to deceive the audience because he knew that mobile phones were being tapped without warrants. If you do decide to watch that video, I suggest you stop before you get to 7:51 into it, when McConnell displays disgusting cynicism:

I would summarize by saying: we have laws, and the key is getting the laws correct. If the law is written correctly and there’s the appropriate oversight committee – if you violate the law, you will be held accountable.

In a nation as free and as wonderful as ours is, leading the world in human rights and privacy and civil liberties it’s getting the debate framed right — to mitigate the risk to protect the nation, consistent with our values and our laws.

By “consistent with our values” I suppose McConnell meant “us, in the intelligence community.”

“There’s too much data to handle”

This is another trope I’ve often heard. Oddly, it’s often uttered by people who use google: a service that collects practically every page of data on the entire web, indexes it, and searches for complex combinations of keywords in microseconds.

How is this done? By massively parallellizing processing: big data is managed, massaged, and refined as it moves up the data pyramid, with additional hints and analysis propagated up to each layer. The underlying data is retained so it can be referred to when necessary, and as long as there are at least 2 copies somewhere, nothing is ever lost.

When you start using hierarchical storage systems, you can store (for all intents and purposes) infinite amounts of data, as long as you can keep buying media for your robot jukeboxes. The point, again, is not that someone looks at all the messages. They are stored, scored, analyzed, clustered, and if something appears interesting (based on origin, destination, keywords, clustering, vocabulary used, etc) then maybe someone has to look at it.

Does anyone here actually think data mining is a new idea? SAS and IBM have been doing it since the 1970s and post 9-11 there are a raft of companies building new data mining and correlation tools for the security industrial complex.

The identity and location of all communicants“‘

Judith Emmel, an NSA spokeswoman, told the Guardian in a response to the latest disclosures: “NSA has consistently reported – including to Congress – that we do not have the ability to determine with certainty the identity or location of all communicants within a given communication. That remains the case.” (Source: Glenn Greenwald)

I’m pretty sure Emmel was speaking the literal truth, but only because she carefully placed the word “all” in the second to last sentence, and “certainty”.

What about “most of“? What about “fairly confident”? This is more sophistic world-slicing: deny that you have a certain capability while strategically neglecting to mention that you’re trying to have that capability. I’m sure Tiger Woods could honestly say “I lack the ability to always knock the ball into the cup” but that sure as hell doesn’t mean he’s not a really good golfer.

Google and Facebook respond

David Drummond, Chief Legal Officer of Google, sent a letter to to the Attorney General and the Federal Bureau of Investigation. It denies the allegations about Google’s data-sharing with the government — in very carefully worded terms.

Mark Zuckerberg, CEO of Facebook, denied the allegations in a posting at Facebook

For More Information


The revelations:


Consequences: the US government has just trashed the overseas reputation of our tech & telecom industries

Posts about these revelations, and what they show about America:

We must conquer the future, or it will conquer us





24 thoughts on “What they mean when the government says “We do not have ‘direct’ access to your info”

  1. It’s simply disgusting. Do we have a Constitution, or don’t we? Do we have a Rule of Law, or not? Are all people created equal, or are there only prisoners and wardens? One thing is certain, the lunatics ARE running the asylum now.


    1. All good questions. To which I have a simple answer.

      These are not things given by God, or Nature’s God. They are things we make so by our will, acting together, and pass on to our children so they can do the same. Should we find the burden of doing so too great, than others will govern.

      Because the nation will be governed. We will be governed, by ourselves or by others.

      None of this implies or requires lunatics, nor makes America an asylum.


  2. A small comment on the last section, about the impact all of this will have on tech companies over seas:

    Most folks in the U.S. are not aware, but the Chinese government has launched a sustained media campaign against Apple since April. There were four or five weeks in a row where a week did not go by without a prime-time special report on Apple’s corruption, an announcement from officials that Apple had broken law x or law z, or a special editorial in the People’s Daily slamming Apple for distributing pornography or something else. It is all very reminiscent of the attacks that landed on Google before Google was kicked out of the country.

    With Google it seemed fairly obvious why – just a few months before Google disclosed that China had hacked its servers in order to access e-mails accounts of dissidents. With Apple things are a lot harder to read. There really isn’t a reason for this new found hostility, save a realization on China’s part that Apple is too entwined with the U.S. government or just too independent to further China’s national interest.

    A few weeks ago that might have seemed kind of silly. Not so now! These PRISM revelations justify that reasoning, don’t they? It makes one wonder what the Chinese know that we do not.

    How my Chinese friends have crooned. Just chatted with one yesterday – she was quite blunt. “Everybody gets upset when China spies on people, but we are not the only ones who do it. America is just as bad. We have always known that. Now you do too.”


  3. I guess the only source of comfort is that the speech-to-text algorithms probably work about as well as this speech-to-text transcription of president Obama’s inaugural address:

    England reveals that the whom and and in the thousand 1006 hundred and was last illusion Hall William Law will little wilderness blinking wilderness in the room is being 00z7J no longer a linked to the little were illusion he Moorhouse will growa long school illusion of the 00z7J in the North t of functional is those motion world is the gross endorsement for all the rebel will is the whom the present will only for what a shows the I and in a genre while the was England in usual with a 0200 an are so is 100 and close problems longer what all the more of the Sir’s 31 and no row was will not the war against the little for the o to lose their than the moreover one little of it is the at at 0 0 drinking had England and the little is all the below the loan England with the of the under whose appeal those of 2 it was the 2/room day care England 102 00z7J a 001 hundred and mainly acknowledgingcan and get a wall revealed in the visible illusion reserves in the those 00z7J will lose the than in the in the winning the will be e room through the those who the those warrant it will not a 0 loss in a 001 loss the block the Englan children are less you are listening the room in the day it will is a 0 loan the roof of a long 0 am and the 2007 00 in the was will those and conclusions were more possible that they will those 0clusion of all have a 2000 and are from a 0 am and resources are the in a and about the and the so 0100 for a 0 00z7J the 0 00z7J 0


  4. A little more fine-parsing of the descriptions of these systems makes me inclined to project another piece of their architecture: I’ve heard it described as the system collecting metadata and then an analyst can “task” it for more detailed collection. So here’s a hypothetical that’s in line with Drake’s disclosures of the “thin thread” system: the sniffer/sensor collects everything it can, then generates metadata from it. So, for example, an email message going past might be collected and the From:/To:/Subject:/Message-ID/IP source/IP destination/PGP key fingerprint (if any) are recorded. That information is compressed and sent “upstream” to additional analysis engines. The message itself is saved on the sensor in a round-robin database – depending on the amount of storage in the sensor, it might last weeks or months. If an analyst on the back-end decides to examine data in more detail, he “tasks” the system to retrieve the actual message, which triggers a query to the sensor to submit the entire message into the data stream.

    Drake described ‘thin thread’ as protecting citizens’ data by not collecting everything – in this case we define “collecting” as “keeping and bringing to someone’s attention.” The data is simply kept around long enough for the rest of the system to decide whether or not it matters.

    I am willing to bet a stack of donuts that’s how it works. And I’ll go a step further and point out that architecture lends itself to abuse (in spite of what Drake says) because there’s no way to tell a “tasking” operation from an “archiving” operation. Someone with, say, a great big data center in Utah, might extend that idea to the point of retrieving nearly everything and storing it then classifying the complete retrievals. The left hand has no need to know that the right hand is collecting everything – the system is designed to obscure that fact even from itself.


  5. One thing that no one seems to have picked up is that these programs are a massive threat to US (and associated countries such as Britain, Australia, Canada and New Zealand) national security themselves. When you have such far reaching systems and so many people having access to it (estimates vary between 500,000 to a million) then they offer very easy access by other parties to use these systems against themselves.

    For example, if I wanted to spy on the US I wouldn’t waste time setting up spy network or bugging people, I’d just simply corrupt (or blackmail, whatever) one of that million and use the US’s own systems to spy on the people, organisations, etc that I am interested in.

    I will guarantee out of that 500,000 to a milllion crowd there are many that have already been bought off (etc) by Russia, China, Israel, India, et al. Thereby turning the whole system against itself. It is also wide open to be abused by criminal elements. For example, there must be some of those people corrupted (etc) by various drug lords. There will almost certainly be some of those people, with all that access, selling information on people and organisations on the black markets right now.

    If the system is not already corrupt it will soon be. Politicians will use it to get access to information against other ones, corporations using to gain advantage over others, drug lords finding out about actions against them, blackmailers finding out things to make money from, et al. As well, what about when it is (and it will be) hacked, then all that information will be available everywhere….

    Human nature, taking a simple example, you are a low level person, making say 70,000 a year, then you find you can track (say) a politician who is a into kinky sex or cheating or their wife, well easy way to make some money isn’t it? Or if someone comes up to you and offers you (say) half a million for all the phone records of a particular person what do you think will happen? And organised crime, if it hasn’t already, will latch onto to this and use it.

    This is a system designed for total corruption.


    1. Unfortunately your forecasts about America have proven correct so far. And these warnings certainly appear correct.

      But I fear these are the almost-trivial aspects, in one sense. These things were kept secret, I suspect, because of fear. What might happen if the public learns about them? Now the public has learned about them. What if there is no reaction? I have seen nothing substantial so far. Our ruling elites will be emboldened, quite rightly, by our apathy and disinterest.

      If so, then we will be moving to the 2nd part of the downward “S” curve. The steep part.


Leave a comment & share your thoughts...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s