How did the moose begin

My standard "everyday" solution when it comes to connecting computers into a single network is Wireguard.
Wireguard is good, supports p2p, and generally has no downsides.

The downsides come from having part of my home infrastructure located in territory controlled by a country that has blocked Wireguard by signatures.
This is, of course, utterly disgusting, and what's even more disgusting is that these blocks have long since stopped following any kind of legislation.
The result is an incomprehensible black box that can do anything, behave however it wants, and nobody knows how this shaitan-machine even works anymore.

So it's time for penetration.

Why not obfuscation?

Actually, there are several projects that allow obfuscating Wireguard traffic and punching through firewalls.
udp2raw, wstunnel and others handle this excellently.
And Amnezia VPN has made their own fork of Wireguard, specifically for breaking through government censorship.

But the main problem with obfuscation is the reduction of effective packet MTU. Because we wrap one packet in another packet, and this overhead takes up space.
And that's not good.

What I want from a VPN

• p2p mesh network
  Wireguard is good, of course, but routing all traffic through one server has consequences.
  The consequences usually include launching a Mars rover to switch the VPN to another server in case of IP blocking or just because the server started feeling unwell.
  And routing traffic halfway around the planet just to get access to a machine that's within arm's reach — that's just wrong.

• Open source and selfhosted
  In matters like this, relying on a third-party provider is either dangerous or useless.
  Tailscale, for example, is famous for its geographical blocks, so relying on it is pointless.
  And since Tailscale doesn't do this on a whim (I hope), there's no guarantee that other services won't do the same.

• Ideologically correct VPN
  This point exists here specifically for Headscale and ZeroTier.
  Creating a crippled open-source product to advertise a commercial one is a vicious practice and I personally don't approve this.

• Not Wireguard
  For obvious reasons. Signature-based blocking.

• Packaged in nixpkgs
  This one's even more obvious. I'm not going to package a VPN into nix myself.

Test subjects

EasyTier

GitHub, Official site

This is probably the simplest way to create a p2p network. So simple that there isn't even a module in nixpkgs to run it.

For security, there's only a password in --network-secret, which is used for traffic encryption.

To work, it immediately opens TCP, UDP, WG, WS, WSS and whatever Lucifer's IT department cooked up. If one gets blocked, it'll break through via another.

Essentially all nodes in the network are identical and you can specify multiple peers for initial connection establishment.
You can use either public ones, which can be viewed here, or specify one of your own nodes.
It doesn't require any additional configuration.

By the way, it has clients for Android, Windows and Mac OS, so it's a good time to dig out those old games you didn't finish in childhood and organize LAN party with friends who aren't very tech-savvy.

The main disadvantage is that you can't bind IP addresses to specific machines.

And yes, this is a project from China, which might not appeal to some for ideological reasons, but personally I hope it was created by enthusiasts specifically for breaking through the Great Firewall of China.

{
  networking.firewall = {
    allowedTCPPorts = [ 11010 11011 11012 ];
    allowedUDPPorts = [ 11010 11011 11012 ];
  };

  environment.systemPackages = [ pkgs.easytier ];
  systemd.services."easytier" = {
    enable = true;
    script = "easytier-core -d --network-name sumeragi --network-secret changeme -p tcp://public.easytier.cn:11010 --dev-name et0 --multi-thread";
    serviceConfig = {
      Restart = "always";
      RestartMaxDelaySec = "1m";
      RestartSec = "100ms";
      RestartSteps = 9;
      User = "root";
    };
    wantedBy = [ "multi-user.target" ];
    after = [ "network.target" ];
    path = with pkgs; [
      easytier
      iproute2
      bash
    ];
  };
}

Nebula

GitHub, Official site

This is a more pompous commercial solution from the creators of Slack.

It has elliptic curve encryption, suggests using its own PKI and looks generally reliable.
Though the prospect of manually distributing certificates to machines doesn't thrill me.

For its operation it requires "lighthouses" that will connect all other nodes.
Inside, everything works on Noise Protocol.
On the outside it exposes only a single UDP port.

Among the nice features there's a firewall and zoning, to build slightly more complex networks than "everyone with everyone."

And also Nebula's interface is absolutely shit.
Instead of a normal CLI, you need to configure an internal sshd and connect via SSH to localhost.
Maybe it's more secure, but it's utterly disgusting.

let
  isLighthouse = if (config.networking.hostName == "lighthouse") then true else false;
in
{
  services.nebula.networks.sumeragi = {
    enable = true;
    ca = "/etc/nebula/ca.crt";
    cert = "/etc/nebula/node.crt";
    key = "/etc/nebula/node.key";

    isLighthouse = isLighthouse;
    lighthouses = if (isLighthouse) then [] else [ "10.1.0.1" ];

    listen = {
      host = "0.0.0.0";
      port = 4242;
    };

    staticHostMap = {
      "10.1.0.1" = [ "266.266.266.266:4242" ];
    };

    settings = if (isLighthouse) then {
      sshd = {
        enabled = true;
        listen = "127.0.0.1:2222";
        host_key = "/etc/nebula/id_ed25519";
        authorized_users = [
          {
            user = "nommy";
            keys = [
              "ssh-ed25519 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
            ];
          }
        ];
      };
    } else {
    };

    firewall = {
      outbound = [
        { port = "any"; proto = "any"; host = "any"; }
      ];
      inbound = [
        { port = "any"; proto = "any"; host = "any"; }
      ];
    };
  };

  networking.firewall.allowedUDPPorts = [ 4242 ];
}

Tinc

GitHub, Official site

When I found this, my first thought was "The fuck is this?"
The project is over 10 years old and is still in an unstable state.
The current version is 1.1pre18, released way back in 2021.
The last commit to the 1.1 branch was over a year ago.
It's packaged in Nix as Lucy knows what.
How is this even a thing?

But actually, Tinc can surprise you quite a bit.

Under the hood it uses its own protocol over UDP, elliptic curves and a ton of black magic (which, by the way, is properly documented) that makes it all work.

Of course, it still needs a node for initial connection bootstrapping, but there's no special setup required — any node can do it, and afterwards it's all direct node-to-node communication.

It has a relatively normal CLI, can show a graph of the entire network, has other tasty features, but really lacks some kind of TUI, or at least ASCII art for rendering that graph.

let
  hostName = config.networking.hostName;
in
{
  networking.firewall.allowedTCPPorts = [ 655 ];
  networking.firewall.allowedUDPPorts = [ 655 ];

  services.tinc = {
    networks = {
      sumeragi = {
        name = hostName;
        ed25519PrivateKeyFile = "/etc/tinc/sumeragi/ed25519_key.priv";
        interfaceType = "tun";
        debugLevel = 3;

        hostSettings = {
          lighthouse = {
            settings.Ed25519PublicKey = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
            subnets = [
              { address = "10.2.0.1/32"; }
            ];
            addresses = [
              { address = "266.266.266.266"; port = 655; }
            ];
          };
          laptop = {
            settings.Ed25519PublicKey = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
            subnets = [
              { address = "10.2.0.2/32"; }
            ];
          };
          rpi = {
            settings.Ed25519PublicKey = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
            subnets = [
              { address = "10.2.0.3/32"; }
            ];
          };
        };
      };
    };
  };

  environment.etc = {
    "tinc/sumeragi/tinc-up".source = pkgs.writeScript "tinc-up-sumeragi" ''
        #!${pkgs.stdenv.shell}
        ${pkgs.nettools}/bin/ifconfig $INTERFACE ${(builtins.elemAt config.services.tinc.networks.sumeragi.hostSettings."${hostName}".subnets 0).address} netmask 255.255.255.0
        /run/current-system/sw/bin/ip r add 10.2.0.0/24 dev tinc.sumeragi
    '';
    "tinc/sumeragi/tinc-down".source = pkgs.writeScript "tinc-down-sumeragi" ''
        #!${pkgs.stdenv.shell}
        ${pkgs.nettools}/bin/ifconfig $INTERFACE down
        /run/current-system/sw/bin/ip r del 10.2.0.0/24 dev tinc.sumeragi
    '';
  };
}

Methodology of measurment

This is actually a huge topic and you could write a whole book about it, but the most important thing is — IPerf lies.
Different versions of IPerf show different numbers, use different measurement methodologies by default, have many tuning options that affect results, and sometimes their readings differ significantly from reality.

So along with two versions of IPerf, it's worth adding some real-world network usage cases.

Internet speeds in both directions are roughly the same for all nodes, so I'll take numbers from the first direction that comes up, since the difference will be within the margin of error.

Infrastructure

For realistic measurements I'll use three machines:

• Home laptop (Laptop) in Spain
• Intermediate server with public IP (Lighthouse) in Finland
• Raspberry Pi (RPi) behind the Russian firewall

The mesh network coordinators are hosted on Lighthouse, while speed is measured between Laptop and RPi.

Ping

ping -c 300 10.1.0.3

We send ICMP packets, wait for the response to arrive, measure the time it took to get the response.
Here we can check latency, jittering and the number of lost packets.

Latency is the average ping response time.
Jittering is how much the response time "wanders" relative to the average. Measured in ms.
The number of lost packets is self-explanatory.

For more or less stable results, 300 packets should be enough.

/dev/zero through SSH

ssh 10.1.0.3 'dd if=/dev/zero bs=128M count=3 2>/dev/null' | dd of=/dev/null status=progress

We read three times 128 MB of zeros through SSH, then look at the reading speed.
Generally not a bad way to determine data transfer speed inside an SSH tunnel.

The main reason for using this test is that through some solutions SSH works so hellishly slow that more than a second can pass between pressing a key and the character appearing on screen, which is completely unacceptable.
And sometimes it doesn't work at all.

Wget

wget 10.1.0.3:5201/testfile

As a test file — the same 384 MB of zeros from /dev/null.

As a server I use simple-http-server, setting the number of threads equal to the number of CPU cores (8).
Of course, with compression disabled, otherwise megabytes of zeros risk turning into kilobytes of headers.

iperf2 and iperf3

Yes, they show orange prices in Africa. Hell knows how to tune this.
So we just run them with standard configuration and then normalize the results from megabits to megabytes.

Reference values

Measuring exact values for speed, ping and all this stuff that we could use as a baseline is somewhat impossible, since both machines are behind NAT.
But since the infrastructure includes a Lighthouse with a public IP, we can run a few tests and fantasize about some results.

Now we can start fantasizing.

Speed between nodes is limited by the slowest link, so we use the minimum values as our reference.
Latencies can simply be added together. But what to do with jittering isn't entirely clear.
Supposedly you can't add such values,
I don't want to recalculate every packet manually, so I'll just take the maximum value.

And it's time for the final results.

Results

All speeds are normalized in bytes. To convert to bits, multiply by 8.

Egyptian power of those iperfs...

Tinc, as I already said, is very capable of surprising.

EasyTier can be forgiven for such overheads, it's ad-hoc after all and generally "be thankful there's any connection at all."

But Nebula frankly disappointed me. Here I really want to crack a joke about the Slack client on Electron, but... I expected better, seriously.

So if you want to get something like this — Tinc is the best choice performance-wise.
I'll keep all of them at once for myself.
I don't like launching Mars rovers unnecessarily.

That's all, folks.

And it all started when mom asked me to fix the robot vacuum...