5 - February 2025: Skarabox Intake: P1

5 - February 2025: Skarabox Intake: P1

February 10, 2025
Nix, NixOS
NixOS, SkaraBoxOS

Excerpt #

Ibizaman started a project of their opinionated Nix Configuration. This configuration that Ibizaman put together, contains the following excerpt:

SkaraboxOS is an opinionated and simplified headless NixOS installer.

It provides a flake template which combines:

  • Creating a bootable ISO, installable on an USB key.
  • nixos-anywhere to install NixOS headlessly.
  • disko to format the drives using native ZFS encryption with remote unlocking through ssh.
  • sops-nix to handle secrets.
  • deploy-rs to deploy updates.

This repository does not invent any of those wonderful tools. It merely provides an opinionated way to make them all fit together for a seamless experience.

Source: https://github.com/ibizaman/skarabox#skaraboxos

Some of these components I had already experienced myself; Installed NixOS already on a few machines, sops-nix and deploy-rs using and configured. Others I had heard about, I was very interested in. I had heard about these features from recent podcasts.

I reached out to Ibizaman, letting him know I would like to use SkaraboxOS and SHB for my WGU Capstone project. I loved the idea of how you could build a system and infrastructure, from a project. Truly an awesome example of Infrastructure as Code.

With every stakeholder, with a project, there is another perspective. today, I began tearing apart my configuration, making SkaraboxOS and SHB the core to my configuration. I had multiple flakes, based on architecture type. My Nix configuration already became very crazy, I needed to get my Nix configuration rewritten. Along the core implementation of SkaraboxOS and SHB, the idea for the machines to be organized by the location of airports by code (IATA and ICAO)…. I mean the location; this is because it’s all about the “location, location, location”.

Jokes aside, may I present my perspective of the process to implement SkaraboxOS, a current mix with SkaraboxOS and my experience! All utilizing the copy-ssh-key mechanism with nixos-anywhere.

This is 90% Theory #

Upon testing, I didn’t test the SOPS correctly, I commented out a line that was important regarding a password for an account with SOPS. I was able to confirm that I was able to decrypt the system with the root key I set.

Off to Part Two (https://tech.fish-maule.com/posts/nc-6-skarabox-intake-p1-feb_25) hopefully for a config with SOPS working after using nixos-anywhere to install.

Differences #

There are some differences between Ibizaman’s repository and my perspective, all broken down in the results:

  • Step 1
    • Using dd instead of usbimager.
  • Step 3:
    • Use of a script with no Internet to generate secrets; leveraging the SSH key, generated on system boot-up.
  • Step 8:
    • How secrets are stored and accessed.

Result #

1 Build ISO and Write to USB Drive #

  1. Build ISO: [<USER>@laptop:~]$ nix build github:ibizaman/skarabox#beacon.
  2. Write ISO to USB Drive: [<USER>@laptop:~]$ dd if=nixos.iso of=/dev/sdX bs=4M status=progress conv=fdatasync.1

!!! Step two will erase the USB Drive !!!

2 Boot Device from USB Drive #

This will be a weird reference but…. https://tails.net/install/linux - Step 5/7 details guidelines one person can use to boot a USB Stick from.

These are not hackers you are looking for…

Here is the account information:

  • Username: nixos
  • Password: skarabox123

3 Generate Secrets #

!!! Do not connect the Internet !!!

With the System booted up, we need to generate the secrets for [<USER>@server:~] :

#!/run/current-system/sw/bin/bash

echo "Create Directory skarabox/secrets and Changing directory to skarabox/secrets..."
mkdir -p skarabox/secrets && cd skarabox/secrets

echo "Generating HostID..."
uuidgen | head -c 8 > hostid

sudo -i

echo "Changing directory to /etc/ssh..."
cd /etc/ssh

echo "Generating Age Key..."
ssh-to-age -- -private-key -i ./ssh_host_ed25519_key -o ./age_host_key

echo "Coping SSH Private and Public Key to Secrets directory..."
cp ssh_host_ed25519_key ssh_host_ed25519_key.pub /home/nixos/skarabox/secrets

echo "Generating Age Public Key..."
ssh-to-age -- -i ./ssh_host_ed25519_key.pub -o ./age_host_key.pub

echo "Coping Age Public Key to Secrets directory..."
cp age_host_key.pub /home/nixos/skarabox/secrets

echo "Changing Permissions of the age_host_key..."
chmod 600 age_host_key

echo "Create Directory secrets and Changing directory to secrets..."
mkdir ~/secrets && cd ~/secrets

echo "Generating Rook Passphrase..."
openssl -- rand -hex 64 > root_passphrase

echo "Generating Data Passphrase..."
openssl -- rand -hex 64 > data_passphrase

4 Connect to Internet #

Now we connect to the internet. If help is needed, the NixOS Installation Guide has been my go to resource: https://nixos.org/manual/nixos/stable/#sec-installation-manual-networking.

Make sure you note down the IP Address, we will need that later!

5 Download SkaraboxOS Templates #

  1. Download SkaraboxOS template Flake to use for system configuration: [<USER>@server:~]$ nix flake init --template github:ibizaman/skarabox.
  2. Move secrets directory to Flake: [<USER>@server:~]$ mv secrets skarabox.

6 Transfer Files to Laptop #

We will use SFTP to transfer the files from the Server to our Laptop: [<USER>@laptop:~]$ sftp nixos@<IP ADDRESS FROM STEP 4>:/home/nixos/skarabox ..

!!! LEAVE THE SERVER ON !!!

7 Identify and Edit Configuration Requirements #

  1. Edit the skarabox options found within the [<USER>@laptop:~/skarabox/flake.nix] file as shown below:
 in
    {
      nixosModules.skarabox = {
        imports = [
          ...
          ({ config, ... }: {
            skarabox.hostname = "skarabox";
            skarabox.username = "skarabox";
            skarabox.disks.rootDisk = "/dev/nvme0n1";  # Update with result of running `fdisk -l` on the USB stick.
            # 10% of size SSD.
            skarabox.disks.rootReservation = "100G";
            skarabox.disks.dataDisk1 = "/dev/sda";  # Update with result of running `fdisk -l` on the USB stick.
            skarabox.disks.dataDisk2 = "/dev/sdb";  # Update with result of running `fdisk -l` on the USB stick.
            # Disable if only an SSD for root is present.
            skarabox.disks.enableDataPool = true;
            # 5% of size Hard Drives.
            skarabox.disks.dataReservation = "500G";
            ...
          })
          ./configuration.nix
        ];
      };

			...
      };
  1. Edit Deployment options found in the [<USER>@laptop:~/skarabox/flake.nix] file as shown below:
 # Used with deploy-rs for updates.
      deploy.nodes.skarabox = {
        hostname = <IP ADDRESS FROM STEP 4>;
        sshUser = "<USERNAME>";
        sshOpts = [ "-o" "IdentitiesOnly=yes" "-i" "ssh_host_ed25519_key" ];
        profiles = {
          system = {
            user = "root";
            path = deployPkgs.deploy-rs.lib.activate.nixos self.nixosConfigurations.skarabox;
          };
        };
      };
      # From https://github.com/serokell/deploy-rs?tab=readme-ov-file#overall-usage
      checks = builtins.mapAttrs (system: deployLib: deployLib.deployChecks self.deploy) deploy-rs.lib;
    };
}
... Bottom of File

8 Identify and Edit Secrets #

  1. Edit the skarabox + sopsoptions found within the [<USER>@laptop:~/skarabox/flake.nix] file as shown below:
 in
    {
      nixosModules.skarabox = {
        imports = [
          ...
            skarabox.sshAuthorizedKeyFile = config.sops.secrets.PUBLIC_KEYS.path;
            skarabox.hostId = config.sops.secrets.HOST_ID.path;
						 sops.defaultSopsFile = ./secrets.yaml;
            sops.age = {
              sshKeyPaths = [ config.sops.secrets.ROOT_SSH_KEY.path ];
            };
          })
          ./configuration.nix
        ];
      };

			...
      };

Going forward, I would recommend use another device with to edit the files through SFTP and SSH.

  1. Add Age Public Keys to [<USER>@laptop:~/skarabox/.sops.nix] file:
    1. Get Public Key from Laptop:
      1. Navigate to [<USER>@laptop:~/skarabox/secrets].
      2. Run [<USER>@laptop:~/skarabox/secrets]$ nix run nixpkgs#ssh-to-age -- -i /etc/ssh/ssh_host_ed25519_key.pub -o ./age_laptop_host_key.pub.
    2. Edit [<USER>@laptop:~/skarabox/.sops.nix] file:
keys:
  - &laptop <CONTENTS OF skarabox/secrets/age_laptop_host_key.pub FILE>
  - &server <CONTENTS OF skarabox/secrets/age_host_key.pub FILE>
creation_rules:
  - path_regex: secrets\.yaml$
    key_groups:
    - age:
      - *laptop
      - *server

  1. Execute [<USER>@laptop:~/skarabox]$ sops secrets.yaml:

    !!! In order for this these secrets to be stored and accessed in the future, make sure the age public keys from step 8.2.2 are in the [<USER>@laptop:~/skarabox/.sops.nix] file !!!

    Execute [<USER>@laptop:~/skarabox]$ cat .sops.yaml BEFORE EXECUTING THE COMMAND ABOVE to make sure the Age Keys can be found!

    ROOT_SSH_KEY: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
    -----END OPENSSH PRIVATE KEY-----    
PUBLIC_KEYS: |
    #SERVER
    - ssh-ed25519 .. root@<MACHINE>
    #LAPTOP
    - ssh-ed25519 ... root@<MACHINE>    
HOST_ID: ...

    Note: Formatting is important with YAML, here is an example of a [<USER>@laptop:~/skarabox/secrets.yaml] from my working system!

  2. Delete Secrets:

    We will no longer need the secrets that we stored within the [<USER>@laptop:~/skarabox/secrets.yaml] file; execute this command on the Laptop to delete the secrets so they are not stored as plaintext in our configuration files: [<USER>@laptop:~/skarabox]$ rm -r secrets.

9 Stage Requirements #

We now have our configuration requirements on the Laptop. The next step is to build the system. We have two different approaches when it comes to building a system:

  1. Basic Option: Move Configuration Requirements back to the Server.
  2. Advanced Option: Commit Configuration Requirements to a Repository.

9.1 Move Configuration back to Server #

We will use sftp to move the files back to the server:

  1. Remove the files from the Server: [<USER>@server:~]$ rm -r skarabox.
  2. Copy the files from the Laptop: [<USER>@laptop:~]$ sftp -r skarabox nixos@<IP ADDRESS FROM STEP 4>:~ .

9.2 Commit Configuration to a Repository #

The scope of this guide is SkaraboxOS intake and will leave it up to other resources to provide details with committing configuration to a repository. I have added footnotes for a resource that is a great start when it comes to using a git repository.2

10 Perform Implementation of Configuration #

Now for the fruits of our labor! As mentioned in step 9, we have two different approaches when it comes to staging the requirements. The same approaches reflect in this step:

  1. Basic Option: Implement Configuration from Server.
  2. Advanced Option: Implement Configuration from Laptop/Server.

For these steps, we have the data_passphrase and root_passphrase in the [<USER>@server:~/secrets] location, store the content of these files in a safe place!

10.1 Implement Configuration from Server #

With the configuration files on the server, we can implement the configuration following these steps:

  1. Move secrets to [<USER>@server:~/skarabox]: [<USER>@server:~]$ mv secrets/* skarabox .

  2. Change directory to [<USER>@server:~/skarabox]: [<USER>@server:~]$ cd skarabox .

  3. Execute the nixos-anywhere installation:

    [<USER>@server:~/skarabox]$ nix run github:nix-community/nixos-anywhere -- \
  --copy-host-keys \
  --flake '.#skarabox' \
  --ssh-option "IdentitiesOnly=yes" \
  --disk-encryption-keys /tmp/root_passphrase root_passphrase \
  --disk-encryption-keys /tmp/data_passphrase data_passphrase \
  nixos@<IP ADDRESS FROM STEP 4>

    Note: You will be prompted to enter a password; same password as in Step 2: skarabox123.

10.2 Implement Configuration from Laptop/Server. #

With the configuration files in a Git Repository, we can implement the configuration following these steps:

  1. Clone Repository that contains Flake: [<USER>@server:~]$ git clone https://<USER>@<DOMAIN>/<REPO OWNER>/<REPO>.git.

  2. Move secrets to [<USER>@server:~/skarabox]: [<USER>@server:~]$ mv secrets/* skarabox .

  3. Change directory to [<USER>@server:~/secrets]: [<USER>@server:~]$ cd secrets .

  4. Execute the nixos-anywhere installation:

    [<USER>@server:~/skarabox]$ nix run github:nix-community/nixos-anywhere -- \
  --copy-host-keys \
  --flake '#skarabox' \
  --ssh-option "IdentitiesOnly=yes" \
  --disk-encryption-keys /tmp/root_passphrase root_passphrase \
  --disk-encryption-keys /tmp/data_passphrase data_passphrase \
  nixos@<IP ADDRESS FROM STEP 4>

    Note: You will be prompted to enter a password; same password as in Step 2: skarabox123.

11 Testing Implementation #

Once you have executed the commands, the server will reboot. Congratulations! SkaraboxOS has been installed! To access this server, there will be two steps we will follow to access the console:

  1. Unlock Disk Encryption
  2. Unlock Root Encryption

Sources #

Session 1: Past Experience #

Current Date: 2024 - February 9, 2025

Experience #

I played with Ibizaman’s SkaraboxOS during this timeframe, trying to piece it together. My goal was get it going, no matter how much work is involved. I did feel it took longer through this session than anticipated. I got all of the details I needed to make a system that worked with Skarabox. Through the process of trial and error, I put all of my notes in the result.

Research #

Capstone Blog Post: https://tech.fish-maule.com/posts/4-skarabox-shb-capstone

  1. https://nixos.wiki/wiki/NixOS_Installation_Guide#From_Linux ↩︎

  2. https://git-scm.com/book/en/v2/Git-Basics-Getting-a-Git-Repository ↩︎