The National Security Agency developed a pilot program
in the late 1990s that would have enabled it to gather and analyze
massive amounts of communications data without running afoul of privacy
laws. But after the Sept. 11 attacks, it shelved the project -- not
because it failed to work -- but because of bureaucratic infighting and
a sudden White House expansion of the agency's surveillance powers,
according to several intelligence officials.
The agency opted instead to adopt only one component of the program,
which produced a far less capable and rigorous program. It remains the
backbone of the NSA's warrantless surveillance efforts, tracking
domestic and overseas communications from a vast databank of
information, and monitoring selected calls.
Four intelligence officials knowledgeable about the program agreed to
discuss it with The Sun only if granted anonymity because of the
sensitivity of the subject.
The program the NSA rejected, called ThinThread, was developed to handle
greater volumes of information, partly in expectation of threats
surrounding the millennium celebrations. Sources say it bundled together
four cutting-edge surveillance tools. ThinThread would have:
* Used more sophisticated methods of sorting through massive phone and
e-mail data to identify suspect communications.
* Identified U.S. phone numbers and other communications data and
encrypted them to ensure caller privacy.
* Employed an automated auditing system to monitor how analysts handled
the information, in order to prevent misuse and improve efficiency.
* Analyzed the data to identify relationships between callers and
chronicle their contacts. Only when evidence of a potential threat had
been developed would analysts be able to request decryption of the records.
An agency spokesman declined to discuss NSA operations.
"Given the nature of the work we do, it would be irresponsible to
discuss actual or alleged operational issues as it would give those
wishing to do harm to the U.S. insight and potentially place Americans
in danger," said NSA spokesman Don Weber in a statement to The Sun
"However, it is important to note that NSA takes its legal
responsibilities very seriously and operates within the law."
In what intelligence experts describe as rigorous testing of ThinThread
in 1998, the project succeeded at each task with high marks. For
example, its ability to sort through massive amounts of data to find
threat-related communications far surpassed the existing system, sources
said. It also was able to rapidly separate and encrypt U.S.-related
communications to ensure privacy.
But the NSA, then headed by Air Force Gen. Michael V. Hayden, opted
against both of those tools, as well as the feature that monitored
potential abuse of the records. Only the data analysis facet of the
program survived and became the basis for the warrantless surveillance
The decision, which one official attributed to "turf protection and
empire building," has undermined the agency's ability to zero in on
potential threats, sources say. In the wake of revelations about the
agency's wide gathering of U.S. phone records, they add, ThinThread
could have provided a simple solution to privacy concerns.
A number of independent studies, including a classified 2004 report from
the Pentagon's inspector-general, in addition to the successful pilot
tests, found that the program provided "superior processing, filtering
and protection of U.S. citizens, and discovery of important and
previously unknown targets," said an intelligence official familiar with
the program who described the reports to The Sun. The Pentagon
report concluded that ThinThread's ability to sort through data in 2001
was far superior to that of another NSA system in place in 2004, and
that the program should be launched and enhanced.
Hayden, the president's nominee to lead the CIA, is to appear Thursday
before the Senate Select Committee on Intelligence and is expected to
face tough questioning about the warrantless surveillance program, the
collection of domestic phone records and other NSA programs.
While the furor over warrantless surveillance, particularly collection
of domestic phone records, has raised questions about the legality of
the program, there has been little or no discussion about how it might
be altered to eliminate such concerns.
ThinThread was designed to address two key challenges: The NSA had more
information than it could digest, and, increasingly, its targets were in
contact with people in the United States whose calls the agency was
prohibited from monitoring.
With the explosion of digital communications, especially phone calls
over the Internet and the use of devices such as BlackBerries, the NSA
was struggling to sort key nuggets of information from the huge volume
of data it took in.
By 1999, as some NSA officials grew increasingly concerned about
millennium-related security, ThinThread seemed in position to become an
important tool with which the NSA could prevent terrorist attacks. But
it was never launched. Neither was it put into effect after the attacks
in 2001. Despite its success in tests, ThinThread's information-sorting
system was viewed by some in the agency as a competitor to Trailblazer,
a $1.2 billion program that was being developed with similar goals. The
NSA was committed to Trailblazer, which later ran into trouble and has
been essentially abandoned.
Both programs aimed to better sort through the sea of data to find key
tips to the next terrorist attack, but Trailblazer had more political
support internally because it was initiated by Hayden when he first
arrived at the NSA, sources said.
NSA managers did not want to adopt the data-sifting component of
ThinThread out of fear that the Trailblazer program would be
outperformed and "humiliated," an intelligence official said.
Without ThinThread's data-sifting assets, the warrantless surveillance
program was left with a sub-par tool for sniffing out information, and
that has diminished the quality of its analysis, according to
Sources say the NSA's existing system for data-sorting has produced a
database clogged with corrupted and useless information.
The mass collection of relatively unsorted data, combined with system
flaws that sources say erroneously flag people as suspect, has produced
numerous false leads, draining analyst resources, according to two
intelligence officials. FBI agents have complained in published reports
in The New York Times that NSA leads have resulted in numerous dead
The privacy protections offered by ThinThread were also abandoned in the
post-Sept. 11 push by the president for a faster response to terrorism.
Once President Bush gave the go-ahead for the NSA to secretly gather and
analyze domestic phone records -- an authorization that carried
no stipulations about identity protection -- agency officials regarded
the encryption as an unnecessary step and rejected it, according to two
intelligence officials knowledgeable about ThinThread and the
warrantless surveillance programs.
"They basically just disabled the [privacy] safeguards," said one
Another, a former top intelligence official, said that without a privacy
requirement, "there was no reason to go back to something that was
perhaps more difficult to implement."
However two officials familiar with the program said the encryption
feature would have been simple to implement. One said the time required
would have involved minutes, not hours.
Encryption would have required analysts to be more disciplined in their
investigations, however, by forcing them to gather what a court would
consider sufficient information to indicate possible terrorist activity
before decryption could be authorized.
While it is unclear why the agency dropped the component that monitored
for abuse of records, one intelligence official noted that the feature
was not popular with analysts. It not only tracked the use of the
database, but hunted for the most effective analysis techniques, and
some analysts thought it would be used to judge their performance.
Within the NSA, the primary advocate for the ThinThread program was
Richard Taylor, who headed the agency's operations division. Taylor who
has retired from the NSA, did not return calls seeking comment.
Officials say that after the successful tests of ThinThread in 1998,
Taylor argued that the NSA should implement the full program. He later
told the 9/11 Commission that ThinThread could have identified the
hijackers had it been in place before the attacks, according to an
intelligence expert close to the commission.
But at the time, NSA lawyers viewed the program as too aggressive. At
that point, the NSA's authority was limited strictly to overseas
communications, with the FBI responsible for analyzing domestic calls.
The lawyers feared that expanding NSA data collection to include
communications in the United States could violate civil liberties, even
with the encryption function.
Taylor had an intense meeting with Hayden and NSA lawyers. "It was a
very emotional debate," recalled a former intelligence official.
"Eventually it was rejected by [NSA] lawyers."
After the 2001 attacks, the NSA lawyers who had blocked the program
reversed their position and approved the use of the program without the
enhanced technology to sift out terrorist communications and without the
The NSA's new legal analysis was based on the commander in chief's
powers during war, said former officials familiar with the program. The
Bush administration's defense has rested largely on that argument since
the warrantless surveillance program became public in December.
The strength of ThinThread's approach is that by encrypting information
on Americans, it is legal regardless of whether the country is at war,
according to one intelligence official.
Officials familiar with Thin Thread say some within NSA were stunned by
the legal flip-flop. ThinThread "was designed very carefully from a
legal point of view, so that even in non-wartime, you could have done it
legitimately," the official said.
In a speech in January, Hayden said the warrantless surveillance program
was not only limited to al-Qaida communications, but carefully
implemented with an eye toward preserving the Constitution and rights of
"As the director, I was the one responsible to ensure that this program
was limited in its scope and disciplined in its application," he said.
Copyright © 2006, The Baltimore Sun