UPDATE Seriously, just go watch this video with the whistleblower who is the source all the below.

I've been tweeting up a storm and got a couple questions along the lines of "what do you think of PRISM?" so I thought I'd sit down and exercise my ability to explain in long-form what I think is going on, and what it means.

The News

Late last week, Glenn Greenwald broke a story at the Guardian about how the NSA — the National Security Agency; the camera-shy and more data/computation driven cousin to the CIA — was collecting phone records for millions of Americans:

National Security Agency is currently collecting the telephone records of millions of US customers of Verizon, one of America's largest telecoms providers, under a top secret court order issued in April.

The order, a copy of which has been obtained by the Guardian, requires Verizon on an "ongoing, daily basis" to give the NSA information on all telephone calls in its systems, both within the US and between the US and other countries.

Then, the day after the Guardian and the Washington Post published stories based on leaked documents about a program called PRISM which allows the US and UK intelligence services to mine data from popular internet destinations:

The National Security Agency and the FBI are tapping directly into the central servers of nine leading U.S. Internet companies, extracting audio and video chats, photographs, e-mails, documents, and connection logs that enable analysts to track foreign targets, according to a top-secret document obtained by The Washington Post.

...

Equally unusual is the way the NSA extracts what it wants, according to the document: “Collection directly from the servers of these U.S. Service Providers: Microsoft, Yahoo, Google, Facebook, PalTalk, AOL, Skype, YouTube, Apple.”

In the ensuing 48 hours, the internet companies and their CEOs issued remarkably similarly-worded denials, which insiders basically think are bollocks. Simultaneously, the Obama administration and the national security apparatus defended the programs and prepared to launch criminal investigations against the whistleblowers.

What It Means

It's been clear for quite a while that the NSA has been engaged in large-scale electronic intelligence-gathering. They didn't build a giant datacenter in Utah for nothing. We learned in 2007 that the NSA was working with AT&T (and presumably other tier-one internet backbone carriers) to access live traffic in their facilities. The actual extent and purpose of those operations remain unknown as there was quick action in congress to provide blanket/retroactive immunity for the telecoms, and no criminal investigation was ever launched. However, the original leaker says the NSA monitors all web traffic.

Given this background, it's not super-surprising that they're also hoovering up phone records, credit card transactions, internet search histories, IMs, emails, and facebook messages. All of this and more is alleged. This is what a 20,000-strong organization with a history of being on the cutting edge of computing and signals intelligence would do in this modern world, especially if it were effectively un-checked in the post-9/11 "anything goes" legal environment.

The phone records are likely being used in a massive pattern matching system to identify networks of individuals who are under suspicion. David Simon, creator of The Wire, draws the same conclusion, but is sadly rather blase about it. He's got the right perspective on how call metadata is useful from his background covering police investigations — the use-case here isn't far off from The Wire's first season — but I don't think he quite understands how this whole "Big Data" thing works.

Simon asks "how many computer runs do you think the NSA can do?" as if that made the whole thing kind of harmless, as if the scale of the data-set is so big that it's hard to do anything except super-focused high-value terrorist investigations. That's where he's wrong. Google does billions of searches a day and monitors the whole internet to do that, and it's not rocket science. It's a set of industry standard best-practices. There's no technical reason the NSA can't be running constant automated pattern analyses, in addition to letting any agent get the call tree for any person in seconds.

That's a far cry from running down some phone records to bust a Baltimore drug gang. The scale and speed with at the action takes place gives it a different character entirely.

Not to mention, you don't need a title three wiretap to snoop on communication anymore. First of all because you can get legal paper from a secret FISA court that has no oversight or public accountability, and secondly because there's no need to tap any wires: the listening devices are already in place and the tape has been running. Given the evidence that there's massive amounts of passive traffic filtering, data-mining, and possibly live archival of ongoing communication, it quickly gets creepy. For instance, there's no real technical barrier stopping the NSA from piping every skype call made into a data warehouse, and then looking them up if they become curious, even if that's decades after the fact.

As for PRISM, it sounds like that's just the data portal they've built into all their various collection mechanisms. Most likely the actual collection varies from source to source: some of it might be pulled from the backbone internet snooping, some of it might be from info handed over by Google and Microsoft. Some of it might be commercially acquired data from existing consumer databases. Based on what we've seen, PRISM is the viewport into aggregated data, not the source itself. We'll probably learn more about these sources in the days and weeks to come.

Why It Matters

None of this is super surprising, but that doesn't make it any less dangerous. These recent revelations are part of a long trend towards increasing surveillance and privacy intrusion, and as citizens we need to start taking a stand against this, and soon. The news has prompted people from other countries to speak out and try to help Americans understand how dark this can become. It's something we should not be willing to accept. It can get ugly fast.

Even if one trusts the Obama Administration or the NSA as they're currently constituted, the systems they're putting into place — legal and technical — are fundamentally antithetical to a free and open society. These aren't isolated incidents: beginning with the PATRIOT act — a sweeping piece of legislation passed with no review, literally in the dead of night, under the thread of imminent national security — continuing through the FISA courts, the retroactive immunity for telcos, the agressive prosecution of whistleblowers, and so on. Secret programs like this are insidious: people in charge get addicted to the power and control, and those who are swept up in it are compromised and forced into silence.

In Dulles UAL lounge listening to 4 US intel officials saying loudly leaker & reporter on #NSA stuff should be disappearedrecorded a bit

— Steve Clemons (@SCClemons) June 8, 2013

This is a direction here, and if we don't change the direction we're headed we're liable to end up where we're going.

President Obama says he "welcomes the debate", even as he prepares to investigate and jail the people who brought forward the information that started it. Needless to say, he's lost is "Constitutional Scholar" merit badge. But imagine if these trends continue, and imagine a future where there's a war, a major resource shortage, or an internal upheaval against the current status quo. Do you really think that all these powers of surveillance would be used exclusively for Truth, Justice and the American Way?

Personally I don't have much to fear from all of this. I've made a point of having nothing to hide for years. But it's not about me, or you, or who has something to hide. It's about the relationship we have as citizens to our government, and how that dynamic works in an era where information is the common currency of power. If we get this wrong, our kids and grandkids will be the ones paying the price.

What We Can Do

The first thing to do is make a point that this matters. Challenge anyone who shrugs it off. Challenge anyone who asserts it's a good thing. We're not living in a totalitarian state yet, and the tide of public opinion still carries sway.

Secondly, start paying attention. There are entities out there who fight for our rights on this. Sign this EFF petition and start supporting their actions generally.

Thirdly, give the companies who cooperate hell. It's true they don't legally have a lot of wiggle room. Those carefully-worded CEO statements were carefully worded because they'd be vulnerable to a criminal whistleblower probe, but the people need allies in this process. If we can pressure internet companies to resist this kind of thing as a matter of policy, that'll help.

Fourth, familiarize yourself with encryption software and how to use it. Not because you have anything to hide, but simply because you value your privacy. Encourage other people to do this too: if a critical mass of people get onto encrypted channels, that will seriously hamper the powers that be.

Fifth, we've got to change the law. There will increasingly be more and more technical mechanisms to perform this kind of surveillance, and until we get serious privacy laws in place, it's nothing but a running battle and arms race. We do not need to sacrifice privacy for security. That's a false choice. We need clear and unambiguous legislation that prevents this kind of Orwellian operation, whether it emanates from a government agency or a private corporation.