Skip to content

Onionspray Survival Guide

Introduction

This is a practical guide for managing, diagnosing and fixing Onion Services hosted using Onionspray.

It does not cover installation or login into Onionspray or Onionprobe instances, which are already handled by the installation and usage documents.

Managing Onionspray instances

Checking Onionspray instance status

ubuntu@instance:~$ cd onionspray
ubuntu@instance:~/onionspray$ ./onionspray ps -a
:::: onionspray processes ::::
ubuntu      1347  0.3  1.1  48364 44588 ?        S    18:22   0:03 tor -f /home/ubuntu/onionspray/projects/sites.d/tor.conf
ubuntu      1350  0.0  0.0  88412  1748 ?        Ss   18:22   0:00 nginx: master process nginx -c /home/ubuntu/onionspray/projects/sites.d/nginx.conf
ubuntu      1351  0.0  0.1  89632  4188 ?        S    18:22   0:00 nginx: worker process
ubuntu      1352  0.0  0.1  89632  4188 ?        S    18:22   0:00 nginx: worker process
ubuntu      1353  0.0  0.0  88028  3528 ?        S    18:22   0:00 nginx: cache manager process
ubuntu      1580  0.0  0.0   2616  1736 pts/1    S+   18:39   0:00 /bin/sh ./onionspray ps -a
ubuntu      1587  0.0  0.0   6468   724 pts/1    S+   18:39   0:00 grep -E \b(onionspray|onionbalance|nginx)\b
ubuntu@instance:~/onionspray$

Restarting Onionspray instances

Use this when, for various reasons, you decide to restart all Onionspray instances for a given set of services, like when they become unresponsive.

  • Open a terminal session for each Onionspray instance that is hosting the service that seems offline. At each instance:

  • Go to the Onionspray folder and restart it:

    cd ~/onionspray
./onionspray bounce -a

  • After this is done for all relevant instances, give some time to the Onion Service get back online (1 to 5 min to be sure).

  • Test service reachability at Tor Browser using a fresh circuit.

  • Close all terminal sessions.

  • If service is still unreachable, please do further diagnostics or contact other system administrators.

Basic upgrade procedure

As usual:

sudo apt update && sudo apt upgrade && sudo apt autoremove

Then reboot (see next session).

Rebooting

Overall reboot procedure when you have multiple instances

  • Reboot all primary instances, such as those tagged with first.
  • Then reboot all secondary instances, such as those tagged with second.
  • Make sure that the primary and secondary instances are not rebooted at the same time, but with some minutes between each reboot, to increase the probability that the Onion Services descriptor for each site are published in different HSDirs.

Reboot procedure for each instance

Recommendation: always stop Onionspray before rebooting the machine:

cd /home/ubuntu/onionspray
./onionspray shutdown -a
sudo reboot

This avoids the following issue right after restarting an Onionspray instance virtual machine:

./onionspray bounce -a
:::: bounce sites ::::
nginx: [emerg] bind() to unix:/home/ubuntu/onionspray/projects/sites.d/xxxxxxxxxxxxxxxxxxxx-v3.d/port-80.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/home/ubuntu/onionspray/projects/sites.d/xxxxxxxxxxxxxxxxxxxx-v3.d/port-443.sock failed (98: Address already in use)
nginx: [emerg] still could not bind()

But if you happen to come to this issue, it can be solved with this:

find /home/ubuntu/onionspray/projects/sites.d -name '*.sock' -exec rm {} \;
rm /home/ubuntu/onionspray/projects/sites.d/nginx.pid
cd /home/ubuntu/onionspray
./onionspray bounce -a

Backing up the Onionspray instances

Backing up and Onionspray instance can be done by simply copying it's entire folder like this:

rsync -av --delete instance-address:~/onionspray/ onionspray/

Backing up the Onionspray instances logs

Check the Onionspray Log Parser.

Getting metrics from the Onionspray instance logs

Check the Onionspray Log Parser.

What to do if the instances are gone? Starting from a backup copy.

  • Onionspray, and Onion Services in general, are easily portable: you can move service around if you have their keys.

  • So if your Onionspray instances are unavailable and you want to put the services back online, just configure Onionspray as usual, making sure to deploy:

  • The Onionspray configuration file, like sites.conf.

  • The Onion Service keys at secrets.
  • The HTTPS certificates, in a folder like projects/sites.d/ssl.d.

These paths are relative to the Onionspray installation folder.

  • Please note that it won't hurt your instances if you start a backup/emergency Onionspray instance: the Onion Services descriptors are updated automatically, so when your instances get back online you can just stop your emergential instance.

  • You can even start the instances from a copy of your backup, as long as you initialize it in a machine compatible with the specs. By the time this guide was written, Onionspray requires Ubuntu 20.04.5 LTS to run properly.

What to do if the certificate expired?

Check this guide.