Forwarding Tor Hidden Services to Another Server Across the Internet

---

Tuesday 26th February 2019

I recently re-deployed my entire infrastructure onto two new servers using Ansible, and as part of this I wanted to remove all stored secrets from my public-facing web servers.

Let's Encrypt certificates were no problem as they are generated on the server and can be easily replaced if needed, and I removed the need for an SSH private key for Git by just using the public repo over HTTPS.

The only secrets that posed a challenge were my Tor Hidden Service private keys, both for Onion v3 and the historic Onion v2. The impact of one of these keys breaching would be very high, since the associated hostnames are already widely known and indexed. Because of this, it would absolutely not be appropriate to store them in my Ansible playbooks Git repository, nor would it be ideal to store them locally on my Ansible control machine.

One option would be to manually upload them whenever I deployed a new server, however this goes against the complete automation that I am achieving with Ansible. Instead, I decided to not run Tor on my public web server fleet at all, and instead host the Hidden Services elsewhere, with traffic forwarded to the web server fleet securely over the internet with an Apache reverse HTTP proxy.

Skip to Section:

Forwarding Tor Hidden Services to Another Server Across the Internet
┣━━ Hidden Service Traffic
┣━━ Why can't you natively forward Hidden Service traffic over the internet?
┣━━ Forwarding Hidden Service Traffic with an Apache Reverse Proxy
┣━━ Troubleshooting
┗━━ Conclusion

Hidden Service Traffic

When configuring a Tor Hidden Service, the HiddenServicePort configuration is used to redirect requests to your Hidden Service on one port to another service running on a different port. The most common configuration is to redirect requests to port 80 to a local web server also running on port 80:

HiddenServiceDir /var/lib/tor/onion_v3/
HiddenServiceVersion 3
HiddenServicePort 80 127.0.0.1:80

Alternatively, if you wanted to host SSH behind a Hidden Service, you could use:

HiddenServicePort 22 127.0.0.1:22

The important point to note is that Hidden Services are not protocol-aware - they just redirect raw packets. This means that you can freely make Tor redirect the packets wherever you want to, but you are responsible for making sure that it does this securely.

The Onion Service Protocol provides confidentiality, anonymity and integrity between Tor clients (users) and Hidden Services, but once the traffic is forwarded by the Hidden Service it is in its raw format.

For example, if HTTP traffic on port 80 is forwarded, then it gets forwarded as-is (plaintext HTTP). As drastic as this may sound, it's not normally a problem as most Hidden Services forward traffic to localhost (127.0.0.1), so the unencrypted traffic isn't traversing any insecure networks. As long as the server machine is configured correctly and isn't directly accessible by adversaries, there generally isn't a security problem.

However, if you want to forward your Hidden Service traffic to another server across the internet, you will need to provide a layer of security yourself.

Why can't you natively forward Hidden Service traffic over the internet?

You can if you want... but unless you have added your own extra layer of security (e.g. TLS), it will be completely unencrypted.

I looked into this further by setting up a Hidden Service on a test machine, configuring it to forward traffic to one of the public JamieWeb servers (157.230.83.95, which is nyc01.jamieweb.net), and monitoring the network interface with Wireshark.

I used the following Hidden Service configuration in /etc/tor/torrc:

HiddenServiceDir /var/lib/tor/onion_v3_test/
HiddenServiceVersion 3
HiddenServicePort 80 157.230.83.95:80

I restarted the Tor service with sudo service tor restart, and a new Hidden Service had been successfully created. I started a Wireshark capture, and put the Onion hostname into Tor Browser:

My server blocked the request as the test Hidden Service I had created is not an authorised hostname, however you can see that the request did successfully reach my server (which to clarify, is a completely different machine to where the Hidden Service is running).

In the Wireshark packet capture, you can see that this request was sent completely unencrypted between the Hidden Service and the remote server:

Tracing the TCP stream shows that the request and response was unencrypted, which is the expected and intended behaviour:

Modifying HiddenServicePort to forward the traffic to port 443 does not resolve this problem though. As I discussed earlier, Hidden Services are not protocol-aware, so the packets will just be forwarded in their raw format to port 443. This is no good, as web servers listening on port 443 are generally expecting a TLS connection first, before HTTP traffic is sent:

As you can see in the screenshot above, Wireshark actually notices that plain HTTP traffic was sent to the usual HTTPS port.

Tor does not intelligently create the TLS connection it needs, as ultimately it's not supposed to - this is way beyond the scope of what Hidden Services are designed to do.

Forwarding Hidden Service Traffic with an Apache Reverse Proxy

In order to securely forward my Tor Hidden Service traffic to a remote server across the public internet, I set up an Apache reverse proxy to forward requests over HTTPS.

This works by having the Hidden Service forward packets to a local web server running on 127.0.0.1, which will then proxy the requests to the remote server natively using HTTPS.

In order to set this up for your own Hidden Service, you will need to create a new Virtual Host. On Debian-based systems, you can create a new file in the /etc/apache2/sites-available directory named something applicable like tor-forward.conf, and add the following configuration to the file:

<VirtualHost 127.0.0.1:80>
    SSLProxyEngine On
    ProxyRequests Off
    ProxyPass "/" "https://your-website-here.example/"
    ProxyPassReverse "/" "https://your-website-here.example/"
</VirtualHost>

This configuration will forward requests to 127.0.0.1:80 on to the remote server specified over HTTPS.

• SSLProxyEngine On allows mod_proxy to use HTTPS, and requires mod_ssl to be enabled.
• ProxyRequests Off prevents your Apache server being used as a forward proxy server, which prevents unauthorised people connecting and using your machine as a front for their nefarious activity.
• ProxyPass passes requests to the first argument ("/" in this case, which is essentially every request) through to the the second argument (which in this case, will the public-facing address of your remote web server).
• ProxyPassReverse allows Apache to rewrite the Location, Content-Location and URI headers in HTTP redirects, to ensure that they continue working properly and do not accidentally redirect out of the reverse proxy setup.

If you don't want to bind this VirtualHost to port 80, you can use a different one if you want, but you'll need to update the HiddenServicePort configuration accordingly. A ServerName is also not needed, as this Virtual Host will only accept connections from localhost.

Also make sure that you always use trailing slashes for the arguments in the ProxyPass and ProxyPassReverse directives, otherwise requests will be improperly proxied and could allow for your server to be used as an open proxy (since without a trailing slash, requests to /index.html will be forwarded to https://your-website-here.exampleindex.html [note the missing slash], which can be exploited to reach unauthorized destinations).

Before enabling the new VirtualHost, you'll need to enable the proxy, proxy_http and ssl Apache modules.

On Debian-based systems, you can do this with sudo a2enmod module_name. Once this is done, you can also enable the new Virtual Host with sudo a2ensite tor-forward.conf (or whatever you named the Virtual Host file).

Then, test your Apache config with apachectl configtest, and restart the Apache server with sudo service apache2 restart.

Now when you make a request to the web server and hit the Virtual Host that you created, the response should be from the remote server specified in your configuration.

If everything works as expected, you can update your Tor Hidden Service configuration in /etc/tor/torrc to set HiddenServicePort to 80 127.0.0.1:80 (or whichever IP/port you used), and then restart Tor with sudo service tor restart.

Connecting to the Hidden Service will now result in Apache establishing a TLS connection with the remote server and proxying the request through:

Tracing the TCP stream shows the TLS handshake taking place:

And the website is displayed as expected in Tor Browser:

As an additional configuration, if you wish to prevent people from connecting directly to your origin server, you could use a firewall rule to restrict connections to port 443 to only allow your Hidden Service, or you could use mod_authz_host's Require directive to whitelist the IP address required.

Troubleshooting

I've documented some common errors that you may encounter with this setup:

AH01144: No protocol handler was valid for the URL /. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.:

Ensure that the proxy_http module is enabled.

Invalid command 'SSLProxyEngine', perhaps misspelled or defined by a module not included in the server configuration:

Ensure that the ssl module is enabled.

AH01144: No protocol handler was valid for the URL / (scheme 'https'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.:

Ensure that the SSLProxyEngine configuration is enabled for the relevant Virtual Host, and that the ssl module is enabled for the server.

AH00961: HTTPS: failed to enable ssl support:

Ensure that the SSLProxyEngine configuration is enabled for the relevant Virtual Host.

AH00898: DNS lookup failure:

Ensure that you used trailing slashes in the ProxyPass and ProxyPassReverse directives.

Conclusion

This setup isn't ideal for some use cases, but for me it has allowed me to vastly improve the resilience and disaster recovery time of my infrastructure, without posing an undue risk to my Hidden Service private keys.

In future I may even bring the reverse proxy completely on-site and host it on a Raspberry Pi or something similar, as that would allow for further cost savings and ease of setup.

There are also other ways that you could securely forward a Hidden Service across the internet, for example using an SSH tunnel, however a reverse HTTP proxy seems like the most suitable way, as it is the most resilient. Either end can go down and it will start working again automatically when it comes back online, but with an SSH tunnel you'd have to resort to a potentially unreliable monitoring scripts to re-establish the connection if/when it goes offline.