6 - March 2025: Skarabox Intake: P2 | Fish-Maule Family Tech Site

6 - March 2025: Skarabox Intake: P2

March 29, 2025

Excerpt #

As I have outlined in my previous blog post: https://tech.fish-maule.com/posts/skarabox-capstone/5-skarabox-intake-p1-february_25; I approached the issue to perform a system configuration based on option 1 from this issues: https://github.com/ibizaman/skarabox/issues/3#issuecomment-2632587337. This approach was practically based on theory. About 90% of the concepts I’ve tested myself but I did not test this amazing work I did to the fullest.

With what I know, help from others, and what is provided; my goal for this post and to solve the issue:

Have a successful test with the result of an efficient and easy-to-use workflow.
Overhaul my configurations based on what I mentioned in my last post:
- The idea for the machines to be organized by the location of airports, by a code (IATA and ICAO).

Differences #

There are some differences between Ibizaman’s repository and my perspective, all broken down in the results:

Step 1
- Using dd instead of usbimager.

Result #

1: Download SkaraboxOS Template #

Create skarabox directory in the home directory: [<USER>@laptop:?]$ mkdir ~/skarabox.
Change to skarabox directory: [<USER>@laptop:?]$ cd ~/skarabox.
Download SkaraboxOS template Flake to use for system configuration: [<USER>@laptop:~/skarabox]$ nix flake init --template github:ibizaman/skarabox.

2: Build ISO and Write to USB Drive #

Build ISO: [<USER>@laptop:~/skarabox]$ nix build github:ibizaman/skarabox#beacon.
Write ISO to USB Drive: [<USER>@laptop:~]$ dd if=result/iso/beacon.iso of=/dev/<REPLACE WITH DRIVE LETTER> bs=4M status=progress conv=fdatasync.
- Note: Use [<USER>@laptop:~]$ fdisk -l To find which drive to write ISO. ¹

!!! Step two will erase the USB Drive !!!

3: Boot Device from USB Drive #

This will be a weird reference but…. https://tails.net/install/linux - Step 5/7 details guidelines one person can use to boot a USB Stick from.

These are not hackers you are looking for…

As mentioned in the terminal when first booting up the server ([<USER>@server:?]$ skarabox-help), note the following details for future steps:

IP Address: [<USER>@server:?]$ ip -brief a.
Disk layout: [<USER>@server:?]$ sudo fdisk -l.

If help is needed to connect to the internet, the NixOS Installation Guide has been my go to resource: https://nixos.org/manual/nixos/stable/#sec-installation-manual-networking.

4: Generate Secrets #

Generate secrets using script: [<USER>@laptop:~/skarabox]$ ./generate_secrets.sh.

Notes:

You may need to change the shell that is being called for the script.
You may need to make script executable: chmod +x generate_secrets.sh

Below is a copy of the contents of the script:

#!/run/current-system/sw/bin/bash

mkdir -p extra-files/etc/ssh

mkdir secrets && cd secrets

echo "Generating Rook Passphrase..."
nix --extra-experimental-features "flakes nix-command" run nixpkgs#openssl -- rand -hex 64 > root_passphrase

echo "Generating Data Passphrase..."
nix --extra-experimental-features "flakes nix-command" run nixpkgs#openssl -- rand -hex 64 > data_passphrase

echo "Generating HostID..."
uuidgen | head -c 8 > hostid

echo "Generating Data+User SSH Private and Public Keys..."
ssh-keygen -t ed25519 -N "" -f ssh_skarabox && chmod 600 ssh_skarabox

echo "Generating Machine SSH Private and Public Keys..."
ssh-keygen -t ed25519 -N "" -f  ssh_host_ed25519_key && chmod 600  ssh_host_ed25519_key

echo "Generating Age Key..."
nix --extra-experimental-features "flakes nix-command" run nixpkgs#ssh-to-age -- -private-key -i ./ssh_host_ed25519_key -o ./age_host_key

echo "Generating Public Age Key..."
nix --extra-experimental-features "flakes nix-command" run nixpkgs#ssh-to-age -- -i ./age_host_key -o ./age_host_key.pub

cp ssh_host_ed25519_key ssh_host_ed25519_key.pub age_host_key ../extra-files/etc/ssh

5: Identify and Edit Configuration Requirements #

Edit the skarabox options found within the [<USER>@laptop:~/skarabox/flake.nix] file as shown below:

in
    {
      nixosModules.skarabox = {
        imports = [
          ...
          ({ config, ... }: {
            # Server System Configuration
            # ---
            skarabox.hostname = "skarabox";
            skarabox.hostId = builtins.readFile ./secrets/hostid;

            # User Configuration
            # ---
            skarabox.username = "<USERNAME>";
            skarabox.initialHashedPassword = "$y$j9T$7EZvmryvlpTHSRG7dC5IU1$lBc/nePnkvqZ//jNpx/UpFKze/p6P7AIhJubK/Ghj68"; # Update with result of running 'mkpasswd -m sha-512' on any device.
            skarabox.sshAuthorizedKeyFile = ./secrets/ssh_skarabox.pub;


            # Disk Configuration
            # ---
            skarabox.disks.rootDisk = "/dev/nvme0n1";  # Update with result of running `fdisk -l` on the USB stick.
            # 10% of size SSD.
            skarabox.disks.rootReservation = "100G";
            skarabox.disks.dataDisk1 = "/dev/sda";  # Update with result of running `fdisk -l` on the USB stick.
            skarabox.disks.dataDisk2 = "/dev/sdb";  # Update with result of running `fdisk -l` on the USB stick.
            # Disable if only an SSD for root is present.
            skarabox.disks.enableDataPool = true;
            # 5% of size Hard Drives.
            skarabox.disks.dataReservation = "500G";
            ...
          })
          ./configuration.nix
        ];
      };

      ...
      };

Edit Deployment options found in the [<USER>@laptop:~/skarabox/flake.nix] file as shown below:

# Used with deploy-rs for updates.
      deploy.nodes.skarabox = {
        hostname = "<IP ADDRESS FROM STEP 2>";
        sshUser = "<USERNAME>";
        sshOpts = [ "-o" "IdentitiesOnly=yes" "-i" "ssh_skarabox" ];
        profiles = {
          system = {
            user = "root";
            path = deployPkgs.deploy-rs.lib.activate.nixos self.nixosConfigurations.skarabox;
          };
        };
      };
      # From https://github.com/serokell/deploy-rs?tab=readme-ov-file#overall-usage
      checks = builtins.mapAttrs (system: deployLib: deployLib.deployChecks self.deploy) deploy-rs.lib;
    };
}
... Bottom of File

6: Identify and Edit Secrets #

Add Age Public Keys to [<USER>@laptop:~/skarabox/.sops.nix] file:
1. Get Public Key from Laptop:
  1. Navigate to [<USER>@laptop:~/skarabox/secrets].
  2. Run [<USER>@laptop:~/skarabox/secrets]$ nix --extra-experimental-features "flakes nix-command" run nixpkgs#ssh-to-age -- -i /etc/ssh/ssh_host_ed25519_key.pub -o ./age_laptop_host_key.pub.
2. Edit [<USER>@laptop:~/skarabox/.sops.nix] file:
```
keys:
  - &laptop <CONTENTS OF skarabox/secrets/age_laptop_host_key.pub FILE>
  - &server <CONTENTS OF skarabox/secrets/age_host_key.pub FILE>
creation_rules:
  - path_regex: secrets\.yaml$
    key_groups:
    - age:
      - *laptop
      - *server
```
Execute [<USER>@laptop:~/skarabox]$ sops secrets.yaml:
!!! In order for this these secrets to be stored and accessed in the future, make sure the age public keys from step 4 are in the [<USER>@laptop:~/skarabox/.sops.nix] file !!!
Execute [<USER>@laptop:~/skarabox]$ cat .sops.yaml BEFORE EXECUTING THE COMMAND ABOVE to make sure the Age Keys can be found!
```
SSH_KEY: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
    -----END OPENSSH PRIVATE KEY-----    
PUBLIC_KEYS: |
    #SERVER
    - ssh-ed25519 .. root@<MACHINE>
    #LAPTOP
    - ssh-ed25519 ... root@<MACHINE>    
```
*Note: Formatting is important with YAML, here is an example of * [<USER>@laptop:~/skarabox/secrets.yaml] ; one of my working system!

7 Perform Implementation of Configuration #

Now for the fruits of our labor!

With our configuration files all set, we can implement the configuration following these steps:

Change directory to [<USER>@laptop:~/skarabox]: [<USER>@laptop:~]$ cd skarabox .

Execute the nixos-anywhere installation:

[<USER>@laptop:~/skarabox]$ nix run github:nix-community/nixos-anywhere -- \
  --flake '.#skarabox' \
  --ssh-option "IdentitiesOnly=yes" \
  --extra-files "extra-files"
  --disk-encryption-keys /tmp/root_passphrase secrets/root_passphrase \
  --disk-encryption-keys /tmp/data_passphrase secrets/data_passphrase \
  nixos@<IP ADDRESS FROM STEP 2>

Note: You will be prompted to enter a password; same password as in Step 2: skarabox123.

8 Testing Implementation #

Once you have executed the commands, the server will reboot. Congratulations! SkaraboxOS has been installed! To access this server, there will be two steps we will follow to access the console:

Unlock Disk Encryption
1. Once the system boots up and finds an IP Address, we can SSH to enter the root passphrase: [<USER>@laptop:~/skarabox/secrets]$ ssh -p 2222 root@<IP ADDRESS OF SERVER> -o IdentitiesOnly=yes -i ssh_skarabox.
  - You will be prompted to verify the key fingerprint if you don’t have the keys added to your known.
```
The authenticity of host '[<IP ADDRESS OF SERVER>]:2222 ([<IP ADDRESS OF SERVER>]:2222)' can't be established.
ED25519 key fingerprint is SHA256:<REDACTED>.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
```
    - Note: It is recommended to verify the authenticity, the <REDACTED> section should match ssh_skarabox.pub.
2. Enter root passphrase:
```
Enter passphrase for 'root':
```
  Once you enter the correct passphrase, copied and pasted in terminal (CTRL+SHIFT+V) from root_passphrase file, the machine will restart. Normal behavior would show Connection to <IP ADDRESS OF SERVER> closed.
SSH into Server
Once the server’s disks has been decrypted, we can no SSH into our User account: [<USER>@laptop:~/skarabox/secrets]$ ssh <USERNAME>@<IP ADDRESS OF SERVER> -o IdentitiesOnly=yes -i ssh_skarabox.

Sources #

Session 1: Past Experience #

Current Date: 2024 - February 9, 2025

Experience #

I played with Ibizaman’s SkaraboxOS during this timeframe, trying to piece it together. My goal was get it going, no matter how much work is involved. I did feel it took longer through this session than anticipated. I got all of the details I needed to make a system that worked with Skarabox. Through the process of trial and error, I got some notes.

Research #

Capstone Blog Post: https://tech.fish-maule.com/posts/4-skarabox-shb-capstone

Session 2: A Renewed Experience #

Current Date: March 15th, 2025 - March 20th, 2025

Experience #

I soon learned that the approach I’m taking here is not the expected approach based on the notes. The issue here is how I decided to approach the configuration, filling in the gaps with what makes sense. With what I theorized, this is not the case.

Research #

Session 3: Deploy! #

Current Date: March 20th, 2025 - March 29th, 2025

Experience #

I was able to successfully deploy to some of my own machines. While I didn’t provide the updates to the process, I still could piece together the notes from memory and apply the experience I encountered from Part 1.

Research #

https://nixos.wiki/wiki/NixOS_Installation_Guide#From_Linux ↩︎