posts: add nix update script post

src/posts/2024-12-28-nix-update-scripts-made-easy.md

@@ -0,0 +1,336 @@
+---
+author: "Youwen Wu"
+authorTwitter: "@youwen"
+desc: "keep your flakes up to date"
+image: "https://wallpapercave.com/wp/wp12329537.png"
+keywords: "nix, update, zen browser"
+lang: "en"
+title: "Nix automatic hash updates made easy"
+---
+
+Nix users often create flakes to package software out of tree, like this [Zen
+Browser flake](https://github.com/youwen5/zen-browser-flake) I've been
+maintaining. Keeping them up to date is a hassle though, since you have to
+update the Subresource Integrity (SRI) hashes that Nix uses to ensure
+reproducibility.
+
+Here's a neat method I've been using to cleanly handle automatic hash updates.
+I use [Nushell](https://www.nushell.sh/) to easily work with data, prefetch
+some hashes, and put it all in a JSON file that can be read by Nix at build
+time.
+
+First, let's create a file called `update.nu`. At the top, place this shebang:
+
+```nu
+#!/usr/bin/env -S nix shell nixpkgs#nushell --command nu
+```
+
+This will execute the script in a Nushell environment, which is fetched by Nix.
+
+## Get the up to date URLs
+
+We need to obtain the latest version of whatever software we want to update.
+In this case, I'll use GitHub releases as my source of truth.
+
+You can use the GitHub API to fetch metadata about all the releases of a repository.
+
+```
+https://api.github.com/repos/($repo)/releases
+```
+
+Roughly speaking, the raw JSON returned by the GitHub releases API looks something like:
+
+```
+[
+   {tag_name: "foo", prerelease: false, ...},
+   {tag_name: "bar", prerelease: true, ...},
+   {tag_name: "foobar", prerelease: false, ...},
+]
+
+```
+
+Note that the ordering of the objects in the array is chronological.
+
+> Even if you aren't using GitHub releases, as long as there is a reliable way to
+> programmatically fetch the latest download URLs of whatever software you're
+> packaging, you can adapt this approach for your specific case.
+
+We use Nushell's `http get` to make a network request. Nushell will
+automatically detect and parse the JSON reponse into a Nushell table.
+
+In my case, Zen Browser frequently publishes prerelease "twilight" builds which
+we don't want to update to. So, we ignore any releases tagged "twilight" or
+marked "prerelease" by filtering them out with the `where` selector.
+
+Finally, we retrieve the tag name of the item at the first index, which would
+be the latest release (since the JSON array was chronologically sorted).
+
+
+```nu
+#!/usr/bin/env -S nix shell nixpkgs#nushell --command nu
+
+# get the latest tag of the latest release that isn't a prerelease
+def get_latest_release [repo: string] {
+  try {
+	http get $"https://api.github.com/repos/($repo)/releases"
+	  | where prerelease == false
+	  | where tag_name != "twilight"
+	  | get tag_name
+	  | get 0
+  } catch { |err| $"Failed to fetch latest release, aborting: ($err.msg)" }
+}
+```
+
+## Prefetching SRI hashes
+
+Now that we have the latest tags, we can easily obtain the latest download URLs, which are of the form:
+
+```
+https://github.com/zen-browser/desktop/releases/download/$tag/zen.linux-x86_64.tar.bz2
+https://github.com/zen-browser/desktop/releases/download/$tag/zen.aarch64-x86_64.tar.bz2
+```
+
+However, we still need the corresponding SRI hashes to pass to Nix.
+
+```nix
+src = fetchurl {
+   url = "https://github.com/zen-browser/desktop/releases/download/1.0.2-b.5/zen.linux-x86_64.tar.bz2";
+   hash = "sha256-00000000000000000000000000000000000000000000";
+};
+```
+
+The easiest way to obtain these new hashes is to update the URL and then set
+the hash property to an empty string (`""`). Nix will spit out an hash mismatch
+error with the correct hash. However, this is inconvenient for automated
+command line scripting.
+
+The Nix documentation mentions
+[nix-prefetch-url](https://nix.dev/manual/nix/2.18/command-ref/nix-prefetch-url)
+as a way to obtain these hashes, but as usual, it doesn't work quite right and
+has also been replaced by a more powerful but underdocumented experimental
+feature instead.
+
+The [nix store
+prefetch-file](https://nix.dev/manual/nix/2.18/command-ref/new-cli/nix3-store-prefetch-file)
+command does what `nix-prefetch-url` is supposed to do, but handles the caveats
+that lead to the wrong hash being produced automatically.
+
+Let's write a Nushell function that outputs the SRI hash of the given URL. We
+tell `prefetch-file` to output structured JSON that we can parse.
+
+Since Nushell _is_ a shell, we can directly invoke shell commands like usual,
+and then process their output with pipes.
+
+```nu
+def get_nix_hash [url: string] {
+  nix store prefetch-file --hash-type sha256 --json $url | from json | get hash
+}
+```
+
+Cool! Now `get_nix_hash` can give us SRI hashes that look like this:
+
+```
+sha256-K3zTCLdvg/VYQNsfeohw65Ghk8FAjhOl8hXU6REO4/s=
+```
+
+## Putting it all together
+
+Now that we're able to fetch the latest release, obtain the download URLs, and
+compute their SRI hashes, we have all the information we need to make an
+automated update. However, these URLs are typically hardcoded in our Nix
+expressions. The question remains as to how to update these values.
+
+A common way I've seen updates performed is using something like `sed` to
+modify the Nix expressions in place. However, there's actually a more
+maintainable and easy to understand approach.
+
+Let's have our Nushell script generate the URLs and hashes and place them in a
+JSON file! Then, we'll be able to read the JSON file from Nix and obtain the
+URL and hash.
+
+```nu
+def generate_sources [] {
+  let tag = get_latest_release "zen-browser/desktop"
+  let prev_sources = open ./sources.json
+
+  if $tag == $prev_sources.version {
+	# everything up to date
+	return $tag
+  }
+
+  # generate the download URLs with the new tag
+  let x86_64_url = $"https://github.com/zen-browser/desktop/releases/download/($tag)/zen.linux-x86_64.tar.bz2"
+  let aarch64_url = $"https://github.com/zen-browser/desktop/releases/download/($tag)/zen.linux-aarch64.tar.bz2"
+
+  # create a Nushell record that maps cleanly to JSON
+  let sources = {
+    # add a version field as well for convenience
+	version: $tag
+
+	x86_64-linux: {
+	  url:  $x86_64_url
+	  hash: (get_nix_hash $x86_64_url)
+	}
+	aarch64-linux: {
+	  url: $aarch64_url
+	  hash: (get_nix_hash $aarch64_url)
+	}
+  }
+
+  echo $sources | save --force "sources.json"
+
+  return $tag
+}
+```
+
+Running this script with
+
+```bash
+chmod +x ./update.nu
+./update.nu
+```
+
+gives us the file `sources.json`:
+
+```json
+{
+  "version": "1.0.2-b.5",
+  "x86_64-linux": {
+    "url": "https://github.com/zen-browser/desktop/releases/download/1.0.2-b.5/zen.linux-x86_64.tar.bz2",
+    "hash": "sha256-K3zTCLdvg/VYQNsfeohw65Ghk8FAjhOl8hXU6REO4/s="
+  },
+  "aarch64-linux": {
+    "url": "https://github.com/zen-browser/desktop/releases/download/1.0.2-b.5/zen.linux-aarch64.tar.bz2",
+    "hash": "sha256-NwIYylGal2QoWhWKtMhMkAAJQ6iNHfQOBZaxTXgvxAk="
+  }
+}
+```
+
+Now, let's read this from Nix. My file organization looks like the following:
+
+```
+./
+| flake.nix
+| zen-browser-unwrapped.nix
+| ...other files...
+```
+
+`zen-browser-unwrapped.nix` contains the derivation for Zen Browser. Let's add
+`version`, `url`, and `hash` to its inputs:
+
+```nix
+{
+  stdenv,
+  fetchurl,
+  # add these below
+  version,
+  url,
+  hash,
+  ...
+}:
+stdenv.mkDerivation {
+   # inherit version from inputs
+  inherit version;
+  pname = "zen-browser-unwrapped";
+
+  src = fetchurl {
+    # inherit the URL and hash we obtain from the inputs
+    inherit url hash;
+  };
+}
+```
+
+Then in `flake.nix`, let's provide the derivation with the data from `sources.json`:
+
+```nix
+let
+   supportedSystems = [
+     "x86_64-linux"
+     "aarch64-linux"
+   ];
+   forAllSystems = nixpkgs.lib.genAttrs supportedSystems;
+in
+{
+   # rest of file omitted for simplicity
+   packages = forAllSystems (
+     system:
+     let
+       pkgs = import nixpkgs { inherit system; };
+       # parse sources.json into a Nix attrset
+       sources = builtins.fromJSON (builtins.readFile ./sources.json);
+     in
+     rec {
+       zen-browser-unwrapped = pkgs.callPackage ./zen-browser-unwrapped.nix {
+         inherit (sources.${system}) hash url;
+         inherit (sources) version;
+
+         # if the above is difficult to understand, it is equivalent to the following:
+         hash = sources.${system}.hash;
+         url = sources.${system}.url;
+         version = sources.version;
+       };
+}
+```
+
+Now, running `nix build .#zen-browser-unwrapped` will be able to use the hashes
+and URLs from `sources.json` to build the package!
+
+## Automating it in CI
+
+We now have a script that can automatically fetch releases and generate hashes
+and URLs, as well as a way for Nix to use the outputted JSON to build
+derivations. All that's left is to fully automate it using CI!
+
+We are going to use GitHub actions for this, as it's free and easy and you're
+probably already hosting on GitHub.
+
+Ensure you've set up actions for your repo and given it sufficient permissions.
+
+We're gonna run it on a cron timer that checks for updates at 8 PM PST every day.
+
+We use DeterminateSystems' actions to help set up Nix. Then, we simply run our
+update script. Since we made the script return the tag it fetched, we can store
+it in a variable and then use it in our commit message.
+
+```
+name: Update to latest version, and update flake inputs
+
+on:
+  schedule:
+    - cron: "0 4 * * *"
+  workflow_dispatch:
+
+jobs:
+  update:
+    name: Update flake inputs and browser
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v4
+
+      - name: Check flake inputs
+        uses: DeterminateSystems/flake-checker-action@v4
+
+      - name: Install Nix
+        uses: DeterminateSystems/nix-installer-action@main
+
+      - name: Set up magic Nix cache
+        uses: DeterminateSystems/magic-nix-cache-action@main
+
+      - name: Check for update and perform update
+        run: |
+          git config --global user.name "github-actions[bot]"
+          git config --global user.email "github-actions[bot]@users.noreply.github.com"
+
+          chmod +x ./update.nu
+          export ZEN_LATEST_VER="$(./update.nu)"
+
+          git add -A
+          git commit -m "github-actions: update to $ZEN_LATEST_VER" || echo "Latest version is $ZEN_LATEST_VER, no updates found"
+
+          nix flake update --commit-lock-file
+
+          git push
+```
+
+Now, our repository will automatically check for and perform updates every day!