Do you really think the NSA has the manpower to sift through billions of keystrokes and metadata

[apples and oranges]

Last edited Wed Jul 31, 2013, 02:06 PM - Edit history (2)

from everyone, including you and me? What is your opinion of the efficiency of federal employees? What is the population of your city?

Perhaps our subjective answers to such questions might explain the disconnect we're seeing between DU members on the issue of NSA "spying."

Update: OFFS, I didn't realize I'd have to spell everything out. OF COURSE the computers are collecting the data, but who the fuck is going to take the time to look at everything? Unless we're expecting the computers to make the decisions on who to arrest and robots are sent to our doors.

Update 2: I never said that what the NSA is doing is alright with me. Just offering my opinion on why some people aren't as angry or worried.

[geckosfeet]

1. No. But they do have unlimited funding and technology. We lose.

[FreakinDJ]

13. They'll Outsourse the sifting to India

[bemildred]

2. I saw numbers as high as 850,000 "analysts" back in June.

I'm still waiting for real data on that subject.

[1-Old-Man]

14. The total number of contracted employees of the NSA is estimated to be 40,000 or so

and they are the largest employer of mathematicians in the world. They also have the most sophisticated computer and analytical systems in the world. Add those things together and you will quickly understand how absurd this discussion is.

[think]

25. I am guessing this number doesn't include private contractors?

[bemildred]

45. Or our "partners", The UK was neck deep in it too. nt

[backscatter712]

71. There's the Five Eyes.

When domestic spying is so unconstitutional and impolitic, get your allies to do it for you, and fortuitously collect the data through intel-sharing!

[onehandle]

3. No.

From the moment I connected to the Internet decades ago, I assumed that my every action could be monitored.

I also assume that it's not. Especially now that in the massive cacophony of data that's out there.

[leftstreet]

4. wtf - the 'efficiency of federal employees?'

First, the NSA would use software and programming, not many employees

Second...wtf?

[frylock]

28. yeah, nice right-wing talking point there..

but what else is new.

[Tikki]

5. NO

Tikki

[L0oniX]

6. I hear that there are NSA shills participating in blog postings that are being recorded by the NSA.

[Harmony Blue]

7. Private contractors provide the manpower

which is why it is so dangerous.

[KittyWampus]

10. and expensive

[Scuba]

8. Not manpower, technology...

... and your smear of federal employees is pathetic.

[apples and oranges]

38. Smear? I asked a hypothetical question!

Why are some interpreting that as a smear?

[Savannahmann]

9. Opinion of efficiency of Federal Employees?

Well, we know that Federal Employees do things they're not supposed to do. For example, it is against the rules to view porn in Federal Offices, but the inspectors who were supposed to be checking out Deepwater Horizon were back at the office checking out porn and ignoring their duties.

http://www.azcentral.com/news/articles/2010/05/25/20100525oil-spill-report-cites-drugs-gifts-porn-politico.html

So they are very efficient at doing things they are NOT SUPPOSED TO BE DOING.

Next question?

[1-Old-Man]

11. Jesus you are naive.

Manpower is not required for sifting through data and there are computers these days. Guess who has the most and most powerful computers on earth along with all the storage money can buy.

[apples and oranges]

40. You do realize that at some point a human has to make decisions on anything the computer

compiles?

[NoOneMan]

55. We all realize that. That is part of the NSA leak

What is your point? The better the algorithms, the fewer false flags it throws. The more intelligent the system is, the fewer humans you really need for analysis.

What exactly are you trying to get at here?

[FreakinDJ]

12. Nice - A Constitution SHREDDING Apologist

[Xithras]

15. Why would anyone presume that humans would be doing the sifting?

There's no need to sift through any of it right now. The system collects and stores it, the system analyzes it looking for patterns, and if you ever become of interest to anyone, the system can retrieve everything you've ever done and make it available to whomever wants to review it. When it comes to sifting and sorting, a single computer can do the work of hundreds of thousands of human analysts.

And the NSA has a lot of computers.

[cbdo2007]

16. Hmmm, you aren't a data analyst are you?

It's easy to sift through billions of keystrokes and metadata with computers and queries.

If you need peanut butter, do you have trouble finding a jar of peanut butter to buy with the millions of stores and other products for sale out there?? No, first you narrow down what type of building to look for - grocery store. Then you narrow down what type of food it is - canned type good. Then you go to that isle and look for the peanut butter and jelly section. Then you look on the shelf with the peanut butter.

[1awake]

17. No, but then they don't need to.

It's all recorded, cross referenced and cataloged for later use.

[hootinholler]

18. They don't need manpower to "sift through it"

Just like google doesn't need manpower to sift through the intertubes.

[reformist2]

44. +1. It's scary how many people there are carrying water for the NSA here.

[DetlefK]

19. They aren't splitting peas by hand there. We have data-mining programs for that.

Do you think the guys at the LHC are sifting through their terabytes of data (generated daily!) by looking on a monitor and hitting the "scroll down"-button?

We already have incredibly powerful programs based on the "neuronal network"-architecture that can sift through vast amounts of data and deliver better results than human intuition.
And you don't have to program search-parameters: You show the neuronal network "this is the pattern we are looking for" and the network learns by itself to identify that pattern.

For example:
A professor told me of a test-run they had done with a neuronal-network-program: They had uploaded all match-results of the soccer-league of Germany into it, from five decades prior until that day. And then they had the program participate in a sort of a "fantasy-league": All the humans in the research-group and the program were predicting how the season would end.
How good was the program? It came in second.

[kamikaze762]

20. Erm...

Are you asking whether or not a program in which billions of dollars are spent each year to mine the data can interpret the data? It would be a pretty useless waste of resources to create a program sophisticated enough to probe every communication without developing the algorithms needed to flag relevant messages... They can, they have, and they will.

[Brickbat]

21. It's like you're not familiar with computers or software.

[Capt. Obvious]

22. Don't be silly

They outsource that job.

[PowerToThePeople]

23. Yes. Humans won't do it though.

[Tierra_y_Libertad]

24. If it's so useless, why are they collecting it?

[AnotherMcIntosh]

32. Good question. n/t

[LondonReign2]

26. Derp!

Last edited Wed Jul 31, 2013, 12:34 PM - Edit history (1)

Are we meant to take you seriously?

Because, yeah. what the NSA employees are doing, by hand, is checking every keystroke.

#FAIL

[frylock]

27. man, if only they had a computer or two to run a few queries

[Incitatus]

29. Maybe not, but now they have the data stored to search for blackmail material on anyone.

While I am not personally concerned, there are politicians, business executives and any number of people that could be coerced into doing things against our best interests by someone else.

[djean111]

30. You or I can buy a program named CallTrunk, which has Argosearch built in. $5 - $50 a month.

The pricing goes up as storage needs go up, not related to functionality.
CallTrunk records your calls. Argosearch "sifts" through those calls - the actual spoken words - for keywords.
Of course, all the "metadata" is saved and searchable, too.
If this is quick and easy on a home computer, imagine how quick and easy this is on those NSA computers.

[ohheckyeah]

31. They don't need the manpower to sift through

billions of keystrokes and metadata - that's what they have computer programs for.

[MineralMan]

33. NSA has the computer power to do that, of course.

The question is: Are they doing that with everyone, or just for targeted people who meet their mission requirements?

The NSA is not interested in the doings of US citizens in general. In fact, they are prohibited from being interested in that. Only when US citizens metadata intersects with targets of interest outside of the US is NSA supposed to even examine that data. That's how they are supposed to behave.

Do they behave that way? We do not know. In fact, nobody really knows, except the NSA.

However, it is the FBI's mission to investigate people and actions within the United States. And they do that, because it's their job. The FBI is the agency people who are concerned about their domestic communications should be looking at, not the NSA.

[Downtown Hound]

34. Who fucking cares?

the fact that they can do if they wanted to IS NOT OKAY WITH ME. DO YOU GET IT?

[1-Old-Man]

35. That is exactly the key point - everything else is simply subterfuge.

I not only have a right to privacy but I have rights that inhibit my Government from spying on me - and that is the Key point. Everything else is just clutter, subterfuge, shit meant to shift our attention away from the key point, that you so accurately point out.

[The Second Stone]

36. They missed the Boston Marathon bombers

and the public used cell phone photos to identify them. They NSA is not about protecting US citizens from terrorist attacks, it is about having all the conversations of everyone so that that citizens are cowed and compliant and blackmailed if necessary, just like J. Edgar Hoover did with Washington elites when he was FBI Director.

[nadinbrzezinski]

43. ^^^^____^^^^ This

[NoOneMan]

72. Tread softly when watched. Don't make waves

They already have your dick pictures. Cower and follow the crowd, then no one has to see your dirty little texts

Isn't that the message we are all getting? Shut up or maybe, just maybe, your dick pictures, porn viewing, drug using, communist reading, etc, might make it out. It doesn't even have to happen to scare people into compliance. That threat is real and its there. We can all be destroyed by this data.

So we have a society where your dirty secrets aren't secret, especially if you are in public service. Where whistleblowers are sent to jail, so sources stop talking. Fear leads to control

[ladjf]

37. It won't be long before that won't be able to afford the massive storage costs. nt

[apples and oranges]

46. That too. It's infeasible

[djean111]

53. No, storage costs get cheaper and cheaper, and compression algorithms get better and better.

And they have all of our money to play with.
In addition, they can just commandeer anything they want, in the name of "national security".
Thinking the storage will get too expensive is like thinking Snowden had to worry about his hotel and room service bill.
Once something is politicized, the costs do not matter.

[Arctic Dave]

39. You mean these "federal employees"?

[Zorra]

41. The 1% has all the money in the world, and all the time in the world to spend it

on protecting and furthering their interests.

[Quantess]

76. That's really what it boils down to. (no text)

[nadinbrzezinski]

42. Nice smear on federal employees

That said, bother reading the Guardian articles. It will be self explanatory when you are done...first layer of analysis, if not more, are done by sophisticated technology, using very specific dictionaries, triggering the five eyes.

I used in this post a good number of those words, in the dictionary you know. This post will trigger the system

[leeroysphitz]

47. Once Skynet comes online the need for human involvement

in data analysis and National Security matters in general will be over.

[NCTraveler]

48. No.

But I believe they can preemptively collect data on me for the possibility of future use. That is bullshit.

You line of argument is along the lines of "if you aren't doing anything wrong, you have nothing to worry about".

[mick063]

49. It is stored.

Ponder the difference of stored verse not stored.

Consider the combination of stored data with tecnological advancement over time.

[Safetykitten]

50. Yes.

[randome]

51. They have help in the intern department.

http://www.google.com/imgres?newwindow=1&hl=en&biw=1217&bih=744&tbm=isch&tbnid=GFOH-SlyyuB9OM:&imgrefurl=http://harrypotter.wikia.com/wiki/Alastor_Moody&docid=132ja_znSvfDBM&imgurl=

[Atman]

52. Please read "Digital Fortress" by Dan Brown.

Yes, that Dan Brown, the Da Vinci Code guy. It was his first novel, way back in 1998, all about the NSA supercomputers. Think how far they've come by now?

I could have done without the schmaltzy love story sub-plot, but that's to be expected in these kind of books. What makes it a page-turner are the discussion about massive NSA computers and how incredibly it parallels so much of what is going on today. Again, it was published in 1998.

http://www.amazon.com/Digital-Fortress-Thriller-Dan-Brown/dp/0312944926/ref=sr_1_1?s=books&ie=UTF8&qid=1375288953&sr=1-1&keywords=digital+fortress

[NoOneMan]

54. Does Google have the ability to sift through the entire internet and perform searches?

Yes. And the NSA has algorithms for analysis as well.

but who the fuck is going to take the time to look at everything

Computers.

Unless we're expecting the computers to make the decisions on who to arrest and robots are sent to our doors

Unless you haven't been paying attention, these algorithms identify likely targets by building profiles and matching patterns. When a high confidence threshold is passed, that profile is forward to a human to investigate (and there is a chain of command until it can be opened and investigated by human eyes).

Frankly, I just don't understand if you've been living in a hole these last few weeks

[backscatter712]

58. Exactly. The NSA's computer systems are Google for Tyrants. n/t

[Atman]

66. I hope the NSA likes my porn surfing habits.

Oh, wait...not "habits." Just proclivities.

Hi, Agent Mike!

[NoOneMan]

69. 77% of Americans view internet porn once a month

And while it might be funny to think the NSA is recording that trivial data, its not always funny to look forward to a full life knowing that someone, somewhere, has hard proof of your vices on record in case they ever need to twist your arms.

Hell, if not you, it sucks to know we live in a society where 77% of people--no matter how powerful--may easily be blackmailed.

[Go Vols]

56. Maybe

The heavily fortified $2 billion center should be up and running in September 2013. Flowing through its servers and routers and stored in near-bottomless databases will be all forms of communication, including the complete contents of private emails, cell phone calls, and Google searches, as well as all sorts of personal data trails—parking receipts, travel itineraries, bookstore purchases, and other digital “pocket litter.”

“You can watch everybody all the time with data- mining,” Binney says. Everything a person does becomes charted on a graph, “financial transactions or travel or anything,” he says. Thus, as data like bookstore receipts, bank statements, and commuter toll records flow in, the NSA is able to paint a more and more detailed picture of someone’s life.

Sitting in a restaurant not far from NSA headquarters, the place where he spent nearly 40 years of his life, Binney held his thumb and forefinger close together. “We are, like, that far from a turnkey totalitarian state,” he says.

At the DOE’s unclassified center at Oak Ridge, work progressed at a furious pace, although it was a one-way street when it came to cooperation with the closemouthed people in Building 5300. Nevertheless, the unclassified team had its Cray XT4 supercomputer upgraded to a warehouse-sized XT5. Named Jaguar for its speed, it clocked in at 1.75 petaflops, officially becoming the world’s fastest computer in 2009

http://www.wired.com/threatlevel/2012/03/ff_nsadatacenter/all/

[backscatter712]

57. They've got Google for Tyrants - why do you think they've got that new datacenter in Utah?

All you have to do is surf to google.com, and you can see for yourself what's available to you at the click of a mouse and the typing of a few keys - Google's got this art down!

But Google sticks mostly to indexing public web pages and Internet data that people have chosen, or by default, made available to it.

The NSA does the same damn thing - indexing and searching. What do they do with all those supercomputers? The same thing Google does.

Except they're indexing your private phone records, your emails, your metadata, your private information.

I'll say it again - the NSA is Google for Tyrants.

They don't have people sifting through that data, they've got the search engines doing that for them. Hundreds of billions of dollars of computer infrastructure.

[NoOneMan]

60. Google also tracks and records user behavior

Google isn't just about indexing public pages at all. Their real value is their ability to track people and build profiles about who they are and what they are interested in. The more they know about the individuals browsing pages, the better they are at serving ads and making money off people. In fact, it also goes that sometimes its more profitable for them to focus on people rather than search results, in order to steer people from organic clicks.

Its actually a bit creepy knowing that google is watching nearly every move people on the internet make (with analytics and doubleclick tracking scripts). It can figure out your gender, age, interests, favorite sites, favorite products, people similar to you, thing you might be interested in, etc. The NSA can figure all this out as well.

[backscatter712]

62. Touche! A valid point!

Of course, Google is a civilian company acting on information that we provide to them, either explicitly, or by default when we fail to prevent ourselves from giving them our information.

Think of what the NSA has access to and puts in their search engines. Think about what information they can use when constructing profiles on people.

[NoOneMan]

65. From what I know about what Google can do, it makes me very uncomfortable with the NSA

Google uses "metadata" to build a virtual reflection of who you are. Its a unique fingerprint that has proven value (in the billions) in its ability to crack your personality in order to sell things to you. These unique profiles of who we are, floating in their databases, is an electronic expression of our habits, desires, wishes, interests, etc.

It isn't simply "metadata", any more than a synapse firing in our brain is just an electronic signal. The forest that contains the trees is being encapsulated and they are capturing the essence of who we are. This is more revealing than a urine sample, a fingerprint, DNA, and other forms of collection we deem to invade our privacy. They are watching our electronic shadows as we tread across this new virtual world of the internet.

Google is of course just trying to sell us stuff (maybe). But what is the NSA doing with our unique metadata reflections--our electronic shadows? What could they do? What will they do? Even if today they have the best intentions, our shadow will exist tomorrow when those intentions change. Why on earth would we want something to contain a unique reflection of who we are forever, when we don't know what they really are doing with it and we can never know?

Its difficult not to be hyperbolic about this. The very concerning issue is that if the least of our fears is realized, this system is a doorway to absolute tyranny because anyone can be silenced (by targeting and eliminating or blackmailing us into silence). They already know who the problems will be, in case there comes a time when we NEED problem individuals to fix the broken system.

[Atman]

68. Ghostery.

That is all.

It's almost fun watching which scripts and cookies are being thwarted. Just don't sign up for Ghostery's "options." Set it to block EVERYTHING, then wait to see what doesn't work and unblock it individually. Seems to work pretty well.

[usGovOwesUs3Trillion]

59. Software Algorithms do

[Safetykitten]

61. Amazing is it not? People posting that there could not possibly be a way that ALL that info could

not be processed, so...why worry?

[usGovOwesUs3Trillion]

63. Yeah, back to their original 'argument'

It's booggles my mind, so it must simply not be possible"

[Puzzledtraveller]

64. lol, manpower

[NoOneMan]

67. NSA spy manpower in action

They put him in the basement and told him to investigate all records starting with A through D. We expect to see him in a few thousand years.

[Lint Head]

70. Which proves it is a waste of tax payers money and extreme overreach of spying powers.

The NSA should be prosecuted for espionage against it's own citizens.

[Hosnon]

73. Why on Earth would they use manpower?

Computers collect the data and computers sift through the data for pre-determined "hits". Those "hits" then likely get reviewed by a person.

As others have pointed out, sifting through that data is no harder for the NSA than it is for Google

[Egalitarian Thug]

74. Do you really think that it's necessary for humans to do it?

[Vashta Nerada]

75. The problem is, the could single any of us out.

If they suspected that you or I were a terrorist, they could pull up everything we've ever done-online and in our phone records-and sift through it all to make sure we weren't terrorists. That means they could listen to private phone calls to our loved ones to what we post on facebook.

They won't look through everything, but they'll look through your stuff if they suspected anything.

[Atman]

80. So what is your point?

First red flag for the NSA...your name.

Now, if you're just surfing some cooking sites and chatting with your friends about plans for the weekend, it is highly unlikely that you will otherwise cause a blip on the radar screen. Sure, you get a couple points for your name, but most likely the algorithms won't pick you out of the "crowd" which needs to be analyzed further.

So then, what if you start texting and e-mailing people in the countries that we've deemed "unfriendly" toward the US? You get a few more points, another red flag. What if these texts and e-mails also coincided with phone calls in which you just happened to discuss certain nefarious activities against the United States? BINGO...then the big red flag goes off. You've hit several of the key indicators, and the computer flags you and your name shows up on Agent Mike's To Do list tomorrow morning.

Just to be clear, I'm not accepting it or supporting it. Just trying to clarify what is happening. The OP was correct to a limited degree...there simply aren't enough people to listen to or directly monitor everything we do or say every day. It's impossible. But there is computer processing power, and algorithms which search for patterns.

And yes, if they suspect you're a terrorist, you might want to question why they suspect this. If they look through "your stuff" and find nothing, you'll never know they looked through your stuff. They'll just close the file. Maybe put a red flag on it for future scrutiny, depending upon what you said to your uncle.

AGAIN, I am not in any way condoning this or apologizing for what the NSA is doing. I am just trying to put it into some context. The OP was way off the mark and woefully ignorant of the process. But then again, so are many of the other posters. No one is sitting at a desk watching your e-mails and recently visited URL's pass by on his screen. It simply isn't the way it works. It's still insidious, but it just doesn't work that way. You won't pop up on Agent Mike's screen until you order some ..... ... online, (I'm not stupid enough to type the actual words into a DU forum), or call an emergency meeting of ________________ (insert group name here).

[alc]

77. yes

There are (at least) 2 ways it is done

1) they have a lot of heuristics (rules) that help them identify likely communications of interest. 'hits' from the first level of rules may go through more heuristics or go to a human. There will be false hits and misses. To minimize misses, they increase the number of false hits. These need more resources and lead nowhere. That's one of the problems with this type of system - lots of 'wasted' resources which could be spent on targeted investigations. And very inconvenient for targets of the false hits - all 600,000 people on the no-fly list are not actually terrorists, but did match a heuristic somewhere. Would you like to be added because a suspected terrorist was in your city and your phone happened to be near his a few times in one week (maybe you both like starbucks before work)? And that "suspected terrorist" may not even belong on the list either but took some business trips that "looked fishy" to the heuristics.

A company I worked for in the past actually worked with the NSA on these rules and visualization systems (many companies and universities and government agencies work together on this type of thing - big data analysis). My company purely for marketing (e.g. who should get a coupon and what value/type) and optimization (e.g. analyze product returns) while the NSA for obvious reasons. Our 'wasted' resources was coupons that were not redeemed or didn't result in long term consumers. The cost was pretty significant but still better than blasting an entire zip code with coupons. Other companies send out personalized coupon books that don't appear personalized. The cost is very high and most aren't used, but the up-sell and cross-sell results are pretty amazing. And it's done without any humans involved and significantly less computer resources than the NSA has.

We are talking about 50+ million consumers and 100s of millions of items of marketing data each run. Walmart crunches billions of sales records a day looking for all sorts of patterns (customers, inventory, distribution issues, sales effectiveness, pricing, etc). Billions of records is nothing for this type of analysis whether you're trying to identify a small number of individuals or find bigger patterns (e.g. companies have found pretty cool patterns around running out of products on shelves and how to avoid it and sometimes without even searching for those but looking at visualizations of the data).

2) They identify someone or a group (e.g. known terrorists or senators who will be voting on an NSA oversight bill). Then they have the computers pull out all metadata involving those people. If there isn't enough data, they widen it (i.e. include families or staff). Then they have computers cross-reference that metadata with other metadata. If there are only a few dozen (or 100) individuals, this may be human-guided. For example

"show me every time Mr X's cell phone was within 100 yards of a suspected felon".
No hits there so try "200 yards", or "prostitute".
No hits there so look for daily patterns and days that didn't follow the pattern.
Look for every other phone that was within 100 yards of this phone more than 5 times over the last month (excluding work and home). Anyone look suspicious? Spend some time looking at their metadata.
Then look for times the phone was at work/home while the car wasn't (from all of the license plate cameras).
Then look for other phones with very similar location patterns since this person may have an official and off-the-shelf untraceable phone.

You get the idea. If you're focusing on a small number of individuals but have a HUGE amount of data about everyone you can find a lot. In the case of terrorists this is good. But, since there are other potential uses they should have to start with the terrorists and expand from there by getting warrants to collect more data rather than having all of the data available and being able to navigate through it.

[Rex]

78. Manpower? What century are you from?

Man is obsolete for that task, keep up please.

[LanternWaste]

79. No. However I do imagine that there are computer algorithms which may easily search through and sort

No. However I do imagine that there are computer algorithms which may easily search through and sort key words and phrases, and then flag those communiques, bringing them to the direct attention of the relevant personnel, who may then read through a much smaller and much more practical accumulation of data.

[upaloopa]

81. Then why are they collecting the data? Just bored I guess.

[cthulu2016]

82. Really? You cannot be Serious...

You, along with everyone else in th world, would have said 3-4 years ago that there was no way you could push a button on your phone and it would tell you the name of the song playing in the car next to you at a red light based on your phone 'listening' to a snippet of the song and comparing it to all songs.

I've done it.

For about 30 years running, the most efficient way to be wrong has been to talk about what computers will not be able to do in five years.

You put the googly eye cartoon after your last sentence because the only way to argue your point is the hope of fooling somebody into thinking the alternative view is crazy.

But it is not crazy.

Unless we're expecting the computers to make the decisions on who to arrest and robots are sent to our doors.

No, smarta**... we are expecting the computers to study all those calls and key-strokes and then spit out names of persons of interest for humans to potentially arrest.

And storing data subjects it to future technologies. (Like the way we do DNA testing of evidence collected before DNA testing existed.)

Any data being saved today can be read by computers five years from now, and arguing that computer's five years from now won't be able to read it all is a fool's errand.

[AgingAmerican]

83. Yes

They have sorting algorithms that do all the heavy lifting.

[Bradical79]

84. Snowden

Wasn't he a private contractor rather than an NSA employee? Our intelligence and military apparatus has many corporate partnerships.