DARPA's "Memex" program is developing new search tools for the vast majority of the Web that isn't indexed by Google and other commercial search engines.
Their aim, they say, is to track down human traffickers. DARPA gave Scientific America a brief demo:
Dark Web sites have, of course, attracted DARPA’s attention because they are good candidates for human trafficking activity. As a result, White and his team are developing a “Dark Web crawler” that explores the Tor-accessible, peer-to-peer areas of the deep Web and has thus far done much to enlighten the researchers as to the extent of dark Web activity. Whereas it was once thought to consist of 1,000 or so pages, White says there could be anywhere between 30,000 and 40,000 dark Web pages. “Just finding these pages and seeing what’s on them is a new aspect of search technology,” he says.
DARPA chose law enforcement efforts to disrupt human trafficking as a concrete cause around which it could quickly develop and deploy its new approach to searching the Internet. White is confident that Memex technology can likewise be applied to any type of investigative effort, including counterterrorism, missing persons, disease response and disaster relief.
"Human Traffickers Caught on Hidden Internet"
"Memex aims to create a new paradigm for domain specific search" (DARPA)
