OnionCity a search engine founded by Virgil Griffith and the late Aaron Swartz
Anonymising networks benefit journalists, whistleblowers and even law enforcement
SURPRISE … Google can’t find everything.
Or rather, there are parts of the Internet not indexed by Google and other standard search engines, known as the ‘Deep Web.’
A subsection of this un-indexed web that garners the lion’s share of media attention is the ‘Dark Web,’ home to various anonymising networks such as Tor and the resources that they provide access to – via ‘onion addresses,’ a pseudo-top-level domain host suffix designating an anonymous, hidden service reachable via the Tor network.
Tor, short for The Onion Router, is free software for enabling anonymous communication, and can also also provide anonymity to websites and other servers.
Servers configured to receive inbound connections only through Tor are called hidden services. Rather than revealing a server's IP (Internet Protocol) address (and thus its network location), a hidden service is accessed through its .onion address.
The Tor network understands these addresses and can route data to and from hidden services, even to those hosted behind firewalls or network address translators (NATs), while preserving the anonymity of both parties. Tor is necessary to access hidden services.
Bring up the Dark Web and for many, images of seedy marketplaces pop up, where for the right price, the motivated can procure anything from guns and drugs to hackers-for-hire.
The most high profile was online black market Silk Road, best known as a platform for selling illegal drugs. It was shut down after its administrator Ross Ulbricht, who went by the alias Dread Pirate Roberts, was arrested in 2013.
Most recently, it was reported that Evolution Marketplace, a deep-web marketplace, disappeared with over US$12 million worth of the cryptocurrency Bitcoin, in what was described as an “exit scam” by its administrators.
Right now, it’s quite a hassle for those who want to explore the Dark Web with no onion address or Tor browser installed.
But that’s about to change with a handful of services cropping up and aiming to make exploring the un-indexed corner of the Internet that much easier.
Welcome to OnionCity
Virgil Griffith (pic above) is the man behind OnionCity, a search engine that aims to enable access to Tor's onionsites.
In an interview with Digital News Asia (DNA) about the project, Griffith says that OnionCity is an implementation of the original goals of Tor2web, a project to let Internet users access anonymous servers.
It was founded by Griffith along with Aaron Swartz in 2008, but was put on hold for a few years while Swartz was busy with JSTOR and Griffith was finishing his doctorate in Computation and Neural Systems at the California Institute of Technology.
“Aaron [Swartz] and I had some struggles deciding what to block. Unfortunately we can't say too much about the outcome of those discussions.
“But in short we decided that the United States is one of the better legal jurisdictions for our servers to reside and we comply with all American blocking laws and court orders,” says Griffith.
[Editor’s Note: Swartz, best known for his work in the development of the web feed format RSS, the Markdown publishing format and social news site Reddit, passed away on Jan 11, 2013]
OnionCity was officially ‘launched’ on Feb 11 this year, when Griffith announced the project via a Tor-talk mailing list.
How it works is relatively simple, he explains: The back-end is a Tor2web proxy which passes content between the regular Internet and Tor's ‘Onion Web,’ which is where these onion sites live.
“The front-end is a lesser Google product called Custom Search Engine which lets you build your own specialised search engine on top of Google’s index.
“Right now both of these work great, but as OnionCity grows we'll probably have to change the tech,” he adds.
Why unpeel the onion?
The thought of making it easier for users to search the Dark Web may bring up knee-jerk reactions about lowering the barriers for those inclined to engage in illicit activities.
In his Vox article outlining the Dark Web, Timothy B. Lee argued that it is important to remember that the technologies used to facilitate Dark Web activities aren't inherently good or bad.
“The same technologies used by drug dealers and child pornographers to hide their identity can also be used by whistleblowers and dissidents in repressive regimes,” he noted.
But the role advocates of projects such as Tor envision is one that protects privacy and anonymous publishing in an environment where tracking and identification in the digital realm to offline lives has become the norm.
Griffith points out that anonymous publishing has substantial historical precedent for mitigating corruption and ensuring a free society, and continuing this capability into the digital age is a good idea.
“With OnionCity I wish to see anonymous Internet publishing become more popular and less arcane,” he adds.
Dr Jeremy Malcolm, senior global policy analyst at the Electronic Frontier Foundation (EFF), notes that Internet is often seen as fostering anonymity, but this is “only true on a superficial level.”
“If you have serious concerns about remaining anonymous you need to use a service like Tor, and to do so correctly, with the Tor Browser Bundle to solve the problem of anonymity for information that you access,” he says.
But Malcolm points out that there is also a second problem: Anonymity for information that you provide, for which you need Tor hidden services.
“We prefer this phrase than talking about the ‘Dark Web,’ which is laden with negative connotations. But just like on the regular Web, there is not much use in setting up a resource if nobody knows where to find it.
“So search engines like OnionCity that open up these Tor hidden services to regular users are a good idea to facilitate broader access to information that people provide anonymously,” adds Malcolm (pic).
OnionCity isn’t the only search engine out there that enables easier access to Tor’s hidden services. There is also Ahmia, whose mission statement includes “creating an environment to share meaningful statistics, insights and news about the Tor network itself.”
But perhaps the most famous effort in this category is Memex, being developed by 17 different contractor teams which are working with the US military’s Defence Advanced Research Projects Agency (Darpa).
The stated goal of Memex is to build a better map of more Internet content, and to also seek to determine just how much of Tor traffic is related to hidden services sites.
Enabling more access to anonymising networks doesn’t just benefit journalists who need a secure place for sources to provide anonymous tips, or whistleblower sites such as WikiLeaks, there are also benefits for law enforcement and those that watch them.
“The advantage of criminals using hidden services is that at least it provides transparency about the problem. Often law enforcement agencies will spout made-up figures about how much crime is conducted online, which others have no way of verifying,” says Malcolm.
“But with hidden services, it is possible to get a better idea of what previously happened under wraps. This is the first step towards catching and prosecuting those criminals using conventional investigation methods,” he adds.
Malcolm also points out that its potential role in authoritarian countries.
“Even relatively tame criticism of the government can get you into trouble, so it is important for hidden services to be easily accessible so that dissidents can get their message out,” he argues.
In addition, researchers on edgy topics like crime and terrorism can find some of the most raw information about their subjects using hidden services. The more control that authorities exercise over the regular Web, the more content will be pushed onto hidden services.
“This provides a useful safety valve for those affected by state or corporate censorship, who are being pushed off mainstream networks as the Internet increasingly becomes safe and ‘vanilla’,” says Malcolm.
In November 2014, Facebook launched its own hidden service, with a goal to better serve people who already access its services via Tor but are sometimes blocked by its automatic security controls.
For example, from within Iran and China, countries whose governments block Facebook access, hundreds of thousands of people access the site this way, according to the organisation behind Tor.
A blog post about Facebook’s hidden service points out that the key point here is that anonymity “isn't just about hiding from your destination.”
“There’s no reason to let your ISP (Internet service provider) know when or whether you’re visiting Facebook. There’s no reason for Facebook’s upstream ISP, or some agency that [conducts surveillance on] the Internet, to learn when and whether you use Facebook.
“And if you do choose to tell Facebook something about you, there’s still no reason to let them automatically discover what city you’re in today while you do it,” it added.
Griffith says that if people believe that citizens should have a voice online without the US National Security Agency (NSA) knowing who said it, onion-sites are the “best protection currently available.”
“In fact I’m a little surprised that Asian governments aren’t already funding Tor and/ or OnionCity to better protect themselves and businesses from foreign intelligence.
“These services also protect socially unpopular groups such as the discussion boards for Gay Ugandans or people with compromising ailments and psychological disorders seeking community and treatment,” he adds.
Enter at your own risk
In his NakedSecurity blog post about OnionCity, independent web consultant Mark Stockley notes that caution must still be exercised.
“OnionCity users would be foolish to use it for anything illegal though because users get no protection whatsoever.
“This is a Tor2Web proxy so the Tor part where the .onion sites reside is as secure as Tor and the Web part where you and I reside is as insecure as the Web. It isn't even available over HTTPS yet. What we users get is convenience, nothing more,” he wrote.
The EFF’s Malcolm notes that sites accessed using OnionCity do nothing for the anonymity of the person accessing the site, only those who run the site.
“The administrators of OnionCity can see everything that you access over their service, even including any passwords or other confidential details that you type.
“OnionCity should not be used if you wish to conceal your identity while you are browsing hidden sites,” he adds.
Those who want to browse the Dark Web without leaving a trail, you still need to be on the Tor network using a Tor browser.
“Because they are the same place – don’t do anything on OnionCity that you wouldn’t do on the regular Internet,” says Griffith.
In addition, .onion domains are only available when owner is also online. Griffith says the search engine caches anything that the originating onion-site does not explicitly ask not to be cached.
“When an onion-site disappears, its content gradually disappears from OnionCity over the next few days,” he adds.
Since going live, OnionCity traffic currently hovers around around 40 hits per second, with about 20,000 unique visitors per day, reports Griffith.
“Our index size increases every day and we’re currently at 1.6 million pages. We estimate there’s currently about six million pages in .onion that have yet to be indexed.
“Last week the index was at 1.5 million pages, and it's nice to see every measure increase a little bit each week,” he adds.
Asked what he has planned for OnionCity, Griffith says that he would like to expand it to include search for other ‘dark webs’ such as I2P and GNUnet.
“On the front-end, the default search engine interface leaves a lot to be desired. I would love to add a few team members and funding!
“OnionCity isn't quite self-supporting yet, but there is slowly increasing ad-revenue. Maybe after this article we’ll get an investor?” he quips.
Right now, as the sole operator behind the project, his goal has been to “ensure growth is continuing and response times remain quick.”
“Just keeping up with the users and growth has been enough of a job. When that settles down, I’ll publish a longer report on typical search terms, and other insights,” he adds.
The biggest challenge for Griffith and the project has been dealing with legal threats from lawyers who “don't know that it’s even possible to publish anonymously on the Internet.”
“They just see ‘onion.city’ in the URL bar and they, perhaps understandably, assume this means we have full control over the content. Instead of thinking of us like a telecommunications provider, they see us like wordpress.com.
“We spend a lot of effort gently educating lawyers about US laws for telecommunications companies, and as we grow the situation is slowly improving,” Griffth reports.
As consultant Stockley notes: “OnionCity isn’t doing anything wrong and it’s not ‘outing’ anyone on the Dark Web; it’s just providing a means for regular web users to search things they would otherwise have to work a little harder to find.”
In the meantime, Griffth is busy getting settled down in Singapore, having moved to the Lion City from Silicon Valley, where he leads the Toroken Project, which aims to add a cryptocurrency incentive to running a Tor relay.
“Toroken is on hold until more funding appears, and I moved to Singapore in search of entrepreneurial opportunities in both security and financial technology.
“Relative to Silicon Valley, I felt there were bigger problems to solve in South-East Asia and fewer technical people solving them. The region is a rich yet underserved market, and I look forward to bringing Silicon-Valley know-how to the local startup environment,” he adds.
The Silk Road arrests, and why users should be worried
How the PRISM surveillance scandal affects Asia
McAfee report looks at ‘dark web’ industry, data breaches
Average of 900 online resources active on Tor daily: Kaspersky
For more technology news and the latest updates, follow us on Twitter, LinkedIn or Like us on Facebook.