Marin Atanasov Nikolov

Exporting org-roam notes to Hugo and Quartz

Sun, 22 Feb 2026 00:00:00 +0000

I use Org Mode for note taking and tracking purposes.

Recently I have also made the switch to Org Roam, that is great extension to Org Mode, which leverages the Zettlekasten method for organizing and linking your notes.

I have also found this approach quite useful and intuitive once you start using it. Being able to establish relationships between your notes and then inspect them by looking at the backlinks is quite useful.

There are plenty of resources about org-roam available, so if you are just starting out I’d suggest checking out the Org Roam Manual, and the Build a Second Brain in Emacs with Org Roam guide for a good overview.

In this post we will see how we can export our existing org-roam notes to Markdown, which can then be served by a static-site generator like Hugo or Quartz.

My Emacs configuration for org-mode looks like this.

(use-package ox-pandoc
  :ensure t
  :after ox)

(use-package ob-go
  :ensure t
  :after ox)

(use-package ox-hugo
  :ensure t
  :after ox)

(defun dnaeon/set-creation-date-heading-property ()
  "Sets the CREATED property on each org-mode heading"
  (save-excursion
    (org-back-to-heading)
    (org-set-property "CREATED" (format-time-string "%Y-%m-%d %T"))))

(use-package org
  :defer t
  :init
  ;; (setq org-src-preserve-indentation t)
  (require 'ob-shell)
  (require 'ob-lisp)
  (require 'ob-python)
  (require 'ob-go)
  (org-babel-do-load-languages
   'org-babel-load-languages
   '((shell . t)
     (lisp . t)
     (python . t)))
  :config
  (add-hook 'org-insert-heading-hook #'dnaeon/set-creation-date-heading-property))

It is a pretty simple and standard setup. I like to have a timestamp associated with each org-mode heading, and the dnaeon/set-creation-date-heading-property takes care of that.

When creating a new heading using org-insert-heading-respect-content or C-<enter> we automatically have the CREATED property like in the example below.

* TODO Just another entry
:PROPERTIES:
:CREATED:  2026-02-23 13:14:20
:END:

Something useful.

My org-roam configuration looks like this.

(defun dnaeon/org-set-created-property ()
  "Sets the CREATED property when a new org-roam node is created"
  (org-set-property "CREATED" (format-time-string "%Y-%m-%d %T")))

(defun dnaeon/tag-new-org-roam-node-as-draft ()
  "Tags org-roam nodes as draft"
  (org-roam-tag-add '("draft")))

(use-package org-roam
  :defer t
  :bind
  (("C-c n f" . org-roam-node-find)
   ("C-c n i" . org-roam-node-insert)
   ("C-c n t" . org-roam-buffer-toggle)
   ("C-c n g" . org-roam-graph)
   ("C-c n c" . org-roam-capture)
   ;; Dailies
   ("C-c n d" . org-roam-dailies-capture-today))
  :config
  ;; Display mtime for nodes
  (cl-defmethod org-roam-node-mtime ((node org-roam-node))
    (format-time-string "%Y-%m-%d %T" (org-roam-node-file-mtime node)))

  ;; A method to display the :CREATED: property from each node when browsing the
  ;; org-roam nodes. In order to display the :CREATED: property add the
  ;; following to the ORG-ROAM-NODE-DISPLAY-TEMPLATE var.
  ;;
  ;; (propertize "${created-property}" 'face 'org-date)
  (cl-defmethod org-roam-node-created-property ((node org-roam-node))
    (cdr (assoc-string "CREATED" (org-roam-node-properties node))))

  ;; Add CREATED property to each org-roam node
  (add-hook 'org-roam-capture-new-node-hook #'dnaeon/org-set-created-property)

  ;; Add a `draft' tag to each newly created node, until we remove it.
  (add-hook 'org-roam-capture-new-node-hook #'dnaeon/tag-new-org-roam-node-as-draft)

  ;; Richer context information when browsing nodes
  (setq org-roam-node-display-template (concat
                                        "${title:80} "
                                        (propertize "${mtime}" 'face 'org-date)
                                        "  "
                                        (propertize "${tags:25}" 'face 'org-tag)))

  ;; Configure path to database and notes
  (setq
   org-roam-directory (file-truename "~/Projects/docs")
   org-roam-db-location (file-truename "~/.emacs.d/org-roam.db"))

  (setq-default
   org-roam-capture-templates
   '(("d" "default" plain "%?" :target (file+head "notes/%<%Y>/%<%m>/${slug}.org" "#+title: ${title}\n") :unnarrowed t)
     ("e" "encrypted" plain "%?" :target (file+head "notes/%<%Y>/%<%m>/${slug}.org.gpg" "#+title: ${title}\n#+filetags: :gpg:encrypted:\n") :unnarrowed t)))

  ;; Configure what sections to display in the org-roam buffer.
  ;;
  ;; https://www.orgroam.com/manual.html#Configuring-what-is-displayed-in-the-buffer-1
  (setq org-roam-mode-sections
        '((org-roam-backlinks-section :unique t)
          org-roam-reflinks-section
          ;; #'org-roam-unlinked-references-section
          ))

  ;; Enable automatic database sync or invoke manually via `M-x org-roam-db-sync'
  (org-roam-db-autosync-mode))

My org-roam-capture-templates contain two templates – one for regular notes, and another one for encrypted GPG notes.

All notes reside in the /path/to/docs/notes/YYYY/MM/<note>.org path. I prefer organizing my notes in a timestamped directory structure, but that is a personal preference. The nice thing about org-roam (or capture templates in org in general) is that you can tweak it to your personal needs.

Upon creating a new note I use a hook to add the CREATED property, similar to the other hook for regular org files and headings. Another hook takes care of adding a draft tag to newly created notes, which is manually removed once I’m done with the note. This helps me keep track of things I need to go back and work on.

I also use a couple of functions for Customizing Node Completions via the org-roam-node-mtime and org-roam-node-created-property functions.

This is what it looks when browsing the list of notes via consult and marginalia.

A nice addition to org-roam is the graphical frontend org-roam-ui, with which you can browse and read your notes.

At the time of writing this post this is what the graph of my notes looks like.

Other Emacs packages that I’ve found useful when working with org-roam (or org-mode in general) include org-transclusion, org-ql, consult-org-roam and org-roam-ql.

Whenever I need to export my notes and share them with others I might create a single note composed of multiple, standalone notes using org-transclusion, and then export it using ox-pandoc or another exporter, depending on what the audience of the document would be.

We can also export our org-roam notes in Markdown format, which can then be served by a static-site generator like Hugo. The ox-hugo exporter does exactly this, and I also use it in order to self-host my notes on an internal system.

Other options include Org Mode Publishing.

My workflow for publishing notes usually involves two repos – one repo contains my org-roam notes, which is specifically being used for storing and tracking my notes only. And then I also use a second repo, which contains static-site generator specific configurations and files.

I prefer keeping them separate, because I don’t want to pollute my main org-roam notes repo with anything else, which would simply consume these notes. This also allows me to experiment with different static-site generators without causing too much noise in the main notes repo.

So, how do I get the notes from the org-roam notes repo to the static-site generator repo?

Well, I use an Emacs Lisp script, which iterates through each org-roam node and exports it via ox-hugo. However, there were a few issues I have faced.

The first one is related to the date-metadata, which is associated with each entry. ox-hugo expects that your org contain various File-level properties, in order to produce a valid front matter for Hugo.

Since all of my org-roam notes already contain the CREATED property, then the ideal solution would be for ox-hugo to use that when generating the Hugo front matter.

Unfortunately ox-hugo does not support mapping properties to front-matter variables. But with Advising Emacs Lisp Functions we can implement a function that affects the org-mode export environment, which ox-hugo uses in order to generate the Markdown document.

The following advice function pushes the :date property to the org export environment. This property represents the date property from the Hugo front-matter. If needed, we can also push the :hugo-publishdate and :hugo-lastmod properties, which correspond to publishDate and lastmod properties respectively.

;; All my org-roam notes contain the `CREATED' property, which specifies the
;; creation date of the note. This wrapper adds the `:date' property to the
;; export environment, which `ox-hugo' will use when exporting to Markdown.
(defun org-export-get-environment/add-date (orig-fun &rest args)
  (let ((env (apply orig-fun args))
        (created (org-entry-get nil "CREATED")))
    (plist-put env :date created)))

(advice-add 'org-export-get-environment :around #'org-export-get-environment/add-date)

Another issue I have faced when using ox-hugo is related to how links between notes are generated. Since my notes reside in the notes/YYYY/MM/ directory structure, when exporting them ox-hugo would generate invalid links, in the form of ../MM/some-note.md.

This is documented in this issue about broken links when exporting. Since the final Markdown documents, which will be served by Hugo (or Hugo-compatible service), we can implement the following hook, which strips away the directory part of the generated link.

;; My org-roam capture templates store the notes in `notes/YYYY/MM' directory
;; structure.
;;
;; When exporting to Markdown via `ox-hugo' the resulting links are invalid,
;; because they would point to `../MM/filename.md'.
;;
;; Since all files are being exported to the `$HUGO_BASE_DIR/content/notes'
;; directory we are simply stripping any leading directories from the filenames
;; here, in order to have valid links.
(defun org-export-resolve-id-link/strip-directory (val)
  (car (last (file-name-split val))))

(advice-add 'org-export-resolve-id-link :filter-return #'org-export-resolve-id-link/strip-directory)

Finally, we can assemble all the pieces together and automate the process of exporting all of our org-roam notes using the script below.

#!/usr/bin/env emacs --script

;;
;; A utility script to export all my org-roam notes via `ox-hugo'
;;
;; The script expects the `HUGO_BASE_DIR', `ORG_ROAM_DIR' and `ORG_ROAM_DB' env
;; vars to be set.
;;
;; `HUGO_BASE_DIR' specifies the directory where org-roam nodes will be
;; exported.
;;
;; Upon successful completion the script produces the following directories.
;;
;; $HUGO_BASE_DIR/content/notes - contains the generated Markdown files
;; $HUGO_BASE_DIR/static/ox-hugo - contains static files, e.g. images
;;
;; In order to serve the notes generated by the script simply sync the Markdown
;; files and static files from the directories above to your Hugo installation.
;;

(dolist (env-var '("HUGO_BASE_DIR" "ORG_ROAM_DIR" "ORG_ROAM_DB"))
  (unless (getenv env-var)
    (error (format "%s env var is not set" env-var))))

(require 'package)
(package-initialize)

(require 'org)
(require 'cl-lib)
(require 'ox-hugo)

(use-package org-roam
  :ensure t
  :config
  ;; Configure path to the org-roam database and notes
  (setq
   org-roam-directory (file-truename (getenv "ORG_ROAM_DIR"))
   org-roam-db-location (file-truename (getenv "ORG_ROAM_DB"))))

;; My org-roam capture templates store the notes in `notes/YYYY/MM' directory
;; structure.
;;
;; When exporting to Markdown via `ox-hugo' the resulting links are invalid,
;; because they would point to `../MM/filename.md'.
;;
;; Since all files are being exported to the `$HUGO_BASE_DIR/content/notes'
;; directory we are simply stripping any leading directories from the filenames
;; here, in order to have valid links.
(defun org-export-resolve-id-link/strip-directory (val)
  (car (last (file-name-split val))))

(advice-add 'org-export-resolve-id-link :filter-return #'org-export-resolve-id-link/strip-directory)

;; All my org-roam notes contain the `CREATED' property, which specifies the
;; creation date of the note.
;;
;; This wrapper adds the `:date' property to the export environment, which
;; `ox-hugo' will use when exporting to Markdown.
;;
;; Additional keys that may be added here include `:hugo-publishdate',
;; `:hugo-lastmod', etc.
(defun org-export-get-environment/add-date (orig-fun &rest args)
  (let ((env (apply orig-fun args))
        (created (org-entry-get nil "CREATED")))
    (plist-put env :date created)))

(advice-add 'org-export-get-environment :around #'org-export-get-environment/add-date)

;; Iterate through the org-roam nodes and export each of them
(let* ((hugo-base-dir (file-truename (getenv "HUGO_BASE_DIR")))
       (hugo-static-dir (file-name-concat hugo-base-dir "static"))
       (org-roam-nodes (org-roam-node-list))
       (unique-roam-nodes (cl-remove-duplicates
                           org-roam-nodes
                           :key (lambda (node) (org-roam-node-id node)))))

  ;; ox-hugo expects that the $HUGO_BASE_DIR/static directory exists in advance.
  (unless (file-accessible-directory-p hugo-static-dir)
    (make-directory hugo-static-dir t))

  ;; Export each org-roam node
  (dolist (node unique-roam-nodes)
    (let ((node-title (org-roam-node-title node))
          (node-file (org-roam-node-file node)))
      (with-current-buffer (find-file-noselect node-file)
        (setq-local org-hugo-base-dir hugo-base-dir
                    org-hugo-front-matter-format "yaml"
                    org-hugo-section "notes"
                    org-agenda-files nil
                    org-export-with-broken-links t)
        (org-hugo-export-wim-to-md))))
  (message (format "Exported %d org-roam node(s)" (length unique-roam-nodes))))

You can also find the full script here.

Download the script and make it executable.

wget https://gist.githubusercontent.com/dnaeon/87427d319ae0b0a14bf7bf2bc0c49a77/raw/5034f599a2beb234c6bd28e67dba4f47b17f126d/export-roam-notes-to-hugo.el
chmod +x export-roam-notes-to-hugo.el

The script expects the following env vars to be provided.

HUGO_BASE_DIR - base directory of your Hugo site
ORG_ROAM_DIR - directory, which contains your org-roam notes
ORG_ROAM_DB - path to the org-roam database

In order to run the script either export the env vars, or pass them using env, e.g.

env HUGO_BASE_DIR=/path/to/hugo \
    ORG_ROAM_DIR=/path/to/org-notes \
    ORG_ROAM_DB=/path/to/org-roam.db \
    export-roam-notes-to-hugo.el

Upon successful completion it will generate the following directories.

$HUGO_BASE_DIR/content/notes - contains the generated Markdown files
$HUGO_BASE_DIR/static/ox-hugo - contains static files, e.g. images

At this point you can start up your Hugo instance and browse your org-roam notes.

Another alternative to Hugo, which I actually like is Quartz. Quartz comes with plugins like Graph View, backlinks, and many others.

In case you are building a site to host your org-roam notes with Quartz you should check out the Publish org-roam notes to personal wiki using ox-hugo and Quartz post, which provides details on how to get started with Quartz.

Using the Emacs Lisp script from this post you can bulk-export your org-roam notes and serve them with Quartz. One thing to keep in mind is that when syncing your generated Markdown files and static files you need to copy them here.

$HUGO_BASE_DIR/content/notes - needs to be synced at $QUARTZ_BASE_DIR/content/notes
$HUGO_BASE_DIR/static/ox-hugo - needs to be synced at $QUARTZ_BASE_DIR/content/ox-hugo

After that, simply start your Quartz instance using the command below.

npx quartz build --serve

Here is an example of Quartz serving my org-roam notes.

And here we can see the graph view of Quartz.

You can also deploy your site to any of the supported hosting providers.

Fixing the clicking fan noise of Synology DS224+

Tue, 19 Aug 2025 00:00:00 +0000

Couple of days ago I got my brand new Synology DS224+ disk station, which was meant to replace my current (and old) Synology DS213j system.

My old DS213j system has served me well for the past ten years, and I have been pretty happy with it. Synology also did an awesome job by providing security updates for all these years to old devices such as the DS213j I have. However, it was time for an upgrade.

After some researching I’ve decided to grab the DS224+, which is a great improvement over my current DS213j. It runs with with 4 cores Intel Celeron J4125 CPU and comes with 2GB of memory, and you have the option to upgrade the memory to 6GB in total. And it also has dual-port NICs. Sweet.

For comparison my DS213j runs with a single core Marvell Kirkwood 1.2GHz and has 512MB of memory. And it has a single NIC.

Like I’ve mentioned – my DS213j has served me well for the past years, and I still keep it around. One of the things that I really like about the DS213j is how quiet it actually is.

So, after getting the DS224+ and installing latest DSM 7.2.2 one thing immediately caught my attention – the noise coming from the fan. And that noise was simply annoying and clicking. I’ve recorded a video of the fan noise, coming out of my brand new DS224+, which you can also listen to.

The first I did was to make sure that the fan is configured in Quiet Mode and to my surprise – that was already the case.

Unfortunately, it turned out that I’m not the only one having such issues with the DS224+. A number of people have already reported similar issues, which you can find in the links below.

Interestingly, the only way you can make that annoying noise disappear is if you simply switch from Quiet Mode to Full-speed mode in the Fan Speed Mode settings. However, that’s not what I really want to do.

The DS213j and DS224+ while being different in terms of CPUs, architecture, amount of memory, etc., are identical in one thing – and that is the fan they run with.

The Synology Spare Parts page confirms that as well.

Back in the day I had to replace the fan of my old DS213j and I’ve used a Noctua NF-A9 FLX / 3 pins, which to this day is still running. So, I thought about doing the same with my DS224+ as well – replace the stock fan with a NF-A9 FLX 3-pin fan, however some people have reported issues when using this fan in particular with DSM 7.2.x (check the links above for more details).

Another Noctua fan, for which people have reported good results in replacing their stock fan is the Noctua NF-B9-redux-1600, 92 x 92 x 25 mm / 3 pins, however I didn’t have a spare one I could use immediately, and I had to order one.

And while waiting for the NF-B9-redux-1600 to arrive I had to keep listening to that annoying, clicking sound coming from the DS224+ stock fan. Needless to say, I had time to tinker with the new DS224+ and see what can be done to fix this.

After some researching and testing various things I eventually found out about /usr/syno/etc.defaults/scemd.xml, which is the configuration for the fan curve. This file is parsed by the scemd service and is used to configure various disk and CPU temperature thresholds, which when reached will configure the fan to run either at low and high speeds, so that the system can cool down.

Comparing this file on my DS224+ with the same file on my old DS213j showed some differences, which I expected anyways. However, having the exact same fan on both systems, I would at least expect that the fans behave the same (unless one of them is actually broken).

You can find the original scemd.xml files for DS213j and DS224+, which I’ve extracted from my systems here.

Some time later I’ve also found a post from a folk at Reddit, who has provided details about how he/she fixed the same annoying noise by adjusting the fan curve profile in /usr/syno/etc.defaults/scemd.xml. You can find the post here.

I’ve tried the instructions he left and it actually made a difference on my DS224+, however there was still a minor low frequency noise coming out of that fan, which I was not happy with.

Then I’ve decided to simply adapt my existing DS213j config for DS224+. Turned out it was pretty simple, and I didn’t have to deal with any of the conversions from frequency (Hz) to RPM values.

A snippet of changes I did to my /usr/syno/etc.defaults/scemd.xml file are provided below.

        <fan_config period="20" threshold="6" type="DUAL_MODE_HIGH" hibernation_speed="UNKNOWN">
		<disk_temperature fan_speed="ULTRA_LOW"  action="NONE">0</disk_temperature>
		<disk_temperature fan_speed="VERY_LOW"   action="NONE">33</disk_temperature>
		<disk_temperature fan_speed="LOW"        action="NONE">41</disk_temperature>
		<disk_temperature fan_speed="VERY_HIGH"  action="NONE">47</disk_temperature>
		<disk_temperature fan_speed="ULTRA_HIGH" action="NONE">53</disk_temperature>
		<disk_temperature fan_speed="ULTRA_HIGH" action="SHUTDOWN">61</disk_temperature>

		<cpu_temperature fan_speed="ULTRA_LOW"   action="NONE">0</cpu_temperature>
		<cpu_temperature fan_speed="LOW"         action="NONE">55</cpu_temperature>
		<cpu_temperature fan_speed="HIGH"        action="NONE">65</cpu_temperature>
		<cpu_temperature fan_speed="ULTRA_HIGH"  action="SHUTDOWN">95</cpu_temperature>
	</fan_config>
	<fan_config period="20" threshold="6" type="DUAL_MODE_LOW" hibernation_speed="UNKNOWN">
		<disk_temperature fan_speed="ULTRA_LOW"  action="NONE">0</disk_temperature>
		<disk_temperature fan_speed="VERY_LOW"   action="NONE">38</disk_temperature>
		<disk_temperature fan_speed="LOW"        action="NONE">46</disk_temperature>
		<disk_temperature fan_speed="HIGH"       action="NONE">52</disk_temperature>
		<disk_temperature fan_speed="VERY_HIGH"  action="NONE">55</disk_temperature>
		<disk_temperature fan_speed="ULTRA_HIGH" action="NONE">58</disk_temperature>
		<disk_temperature fan_speed="ULTRA_HIGH" action="SHUTDOWN">61</disk_temperature>

		<cpu_temperature fan_speed="ULTRA_LOW"   action="NONE">0</cpu_temperature>
		<cpu_temperature fan_speed="LOW"         action="NONE">55</cpu_temperature>
		<cpu_temperature fan_speed="HIGH"        action="NONE">65</cpu_temperature>
		<cpu_temperature fan_speed="ULTRA_HIGH"  action="SHUTDOWN">95</cpu_temperature>
	</fan_config>

You can also find the full updated scemd.xml file here.

Feel free to adjust the threshold temperatures for disk and CPU to other values, if needed.

After you make changes to the /usr/syno/etc.defaults/scemd.xml file make sure that you restart the scemd service.

sudo systemctl restart scemd

And with these changes made to scemd.xml the clicking noise coming from the DS224+ fan went away for me, so now I don’t even have to replace it.

Remote SDR setup with SDRplay & Raspberry Pi

Sun, 13 Jul 2025 00:00:00 +0000

One of the things I like doing in my spare time is messing around with Software-defined radio (SDR).

Few years ago I started out with a simple setup – got myself an RTL-SDR dongle, hooked it up to the computer and connected an antenna. My choice of operating system is GNU/Linux and (btw) I run Arch Linux as my daily driver. Next thing was to configure the RTL-SDR driver, install gqrx and start it up. Pure joy.

Ever since I’ve started with SDR I’ve been using it to listen to random ham operator conversations, track satellites, listen to distant short-wave stations, receive and decode radio faxes, listening to the International Space Station (ISS), etc.

It’s a fun thing to do with lots of possibilities! My wife thinks differently, though. She thinks that “I’m just playing with some toys”. And in a way she might be right, but it’s still a fun thing to mess with.

This is a radio fax transmitted by the Deutscher Wetterdienst (DWD), which is more than 1000 kilometers away from where I live and captured the broadcast.

At some later point I got a Ham it Up Plus Upconverter, which you can use in combination with your RTL-SDR dongle for short-wave listening.

Then an Airspy R2 with a Spyverter R2. Compared to RTL-SDR, the Airspy R2 is the better choice, at least to me. I was getting better reception with the Airspy, than I was getting with the RTL-SDR dongle.

I should also mention that I live in Sofia, which is the capital of Bulgaria. I also live in an apartment, which means that I don’t have much spare space to practice my hobby. I can’t just install an antenna pole on the balcony here or spread out some long antenna wires across the buildings. So I have to plan things a bit.

I’m using one GRA-D220 144/440/900/1200MHz Mini Discone Antenna, which is a multi-band antenna and is good enough for listening on 2m and 70cm bands.

The other antenna I use is the MLA-30+ Loop Antenna,0.5-30MHz Active Receiving Antenna, which is used for short-wave listening.

One of the challenges I’ve always had with my SDR setup is that each time I wanted to start it up and just scan through the frequencies I had to take out the SDR receivers, get the long coax cable from the balcony and take it inside the room with me, connect it to the SDR, and then plug into the computer. This takes time, then I have cables crossing the room, and later I’ll have to undo everything I did once I’m done with it.

I managed to save myself some time by mounting the SDRs on a wooden plate, which I then mounted next to my computer, but still the problem with dragging the antenna cables with me inside the room was bugging me. And drilling holes through the wall was not an option.

This setup worked well for some time, but it was about time to take things to the next level.

Using rtl_tcp allows you to stream your SDR over the network, so if you have an SDR running on a remote location, you could connect to it via the network from a remote client. That means I could just leave the SDRs in a waterproof box on the balcony and then connect to them over the network. And that also means no more cables running across the room. I could also access my SDRs from elsewhere by simply connecting to my home network via WireGuard. Easy as that.

Okay, so what do I need to get this up and running?

I’ve got the SDRs already, and the antennas, I would also need a computer as well, which I can simply put inside the box with the rest of the SDRs. The obvious choice here is to use a Raspberry Pi, first because of it’s dimensions, and second because it’s powerful enough to serve this purpose.

However, putting all these things in a single box ended up as quite a challenge and eventually I started looking for a new compact SDR, with dual antenna ports, which I could put inside the box along with the Raspberry Pi.

Eventually, I ended with the RSPduo from SDRplay. The RSPduo can operate between 1kHz and 2GHz, which meant that I don’t actually need an upconverter anymore, and that also means less space is required to fit all these components in a single waterproof box.

With all these details out of the way it was time to get some work done.

Here’s a list of the components I’ve used.

Raspberry Pi 5 - 16GB (8GB is more than enough, but I’ve got the 16GB anyways)
SanDisk SD Card
Aluminium Passive Cooling Case
Raspberry Pi 27W USB-C Power Supply
RSPduo
USB-A to USB-B cable (RSPduo doesn’t come with USB cables)

After putting all these components in a box it looks like this. This is the Raspberry Pi in it’s aluminium case.

All components packed nicely in the box. I’ve used some velcro strips to get the components mounted in the box, which I think ended up good enough.

And this is how the box looks like when closed.

Having all the components in place it was time to install the software. The Raspberry Pi runs with DragonOS, which comes with many SDR-related tools pre-installed. In order to install it download the DragonOS_Pi64 image and install it on your SD card using the Raspberry Pi Imager.

Once installed boot your Raspberry Pi and login to it. The default username and password in DragonOS are:

Username: ubuntu
Password: dragon

Now it’s time to configure the Pi. You can use the raspi-config tool to configure various system settings such as hostname, password, WiFi, etc.

Make sure that your system is up-to-date as well.

sudo apt-get update
sudo apt-get upgrade

In case you need any other packages, now is the time to install them as well, e.g. install command-line editors (mg), sudo(8) access, tmux, lm-sensors for monitoring the sensors of your Pi, etc.

Once you are done with updating and configuring your Pi we need to reboot it.

sudo reboot

NOTE: In case you are connecting to your Pi using VNC and experiencing strange issues such as blank screen you can follow the steps below to get VNC up and running. If you don’t use VNC at all, or don’t experience such issue, you can safely skip the next few steps, which discuss how to fix the VNC connection to your Pi.

Check the status of the vncserver-x11-serviced service.

ubuntu@sdr01:~$ sudo journalctl -u vncserver-x11-serviced.service
May 21 06:45:45 ubuntu systemd[1]: Started vncserver-x11-serviced.service - VNC Server in Service Mode daemon.
May 21 06:45:46 ubuntu vncserver-x11[2568]: ServerManager: Server started
May 21 06:45:48 ubuntu vncserver-x11[2568]: ConsoleDisplay: Cannot find a running X server on vt1
May 21 06:46:08 ubuntu vncserver-x11[2568]: ConsoleDisplay: Found running X server (pid=3471, binary=/usr/lib/xorg/Xorg)
May 29 11:25:12 ubuntu vncserver-x11[2568]: Connections: connected: 192.168.88.70::63330 (TCP)
May 29 11:25:12 ubuntu vncserver-x11[2568]: Connections: disconnected: 192.168.88.70::63330 (TCP) ([ConnFailed] No configured security type is supported by 3.3 VNC Viewer)
May 29 11:25:32 ubuntu vncserver-x11[2568]: Connections: connected: 192.168.88.70::63335 (TCP)
May 29 11:25:32 ubuntu vncserver-x11[2568]: Connections: disconnected: 192.168.88.70::63335 (TCP) ([ConnFailed] No configured security type is supported by 3.3 VNC Viewer)
May 29 11:26:51 ubuntu vncserver-x11[2568]: Connections: connected: 192.168.88.70::63352 (TCP)
May 29 11:26:51 ubuntu vncserver-x11[2568]: Connections: disconnected: 192.168.88.70::63352 (TCP) ([ConnFailed] No configured security type is supported by 3.3 VNC Viewer)

I was having issues with VNC on my Pi, and from the logs above you can see that as soon as the session is established it is followed immediately by closing the session. Here is how I’ve fixed it on my Pi.

First, stop and disable the VNC services.

sudo systemctl stop vncserver-virtuald.service
sudo systemctl stop vncserver-x11-serviced.service
sudo systemctl disable vncserver-virtuald.service
sudo systemctl disable vncserver-x11-serviced.service

Then remove the realvnc-vnc-server server package.

sudo apt-get remove realvnc-vnc-server

Now we will install tigervnc server instead, which works just fine.

sudo apt-get install tigervnc-standalone-server tigervnc-xorg-extension tigervnc-tools

For tigervnc we need to configure user mappings in /etc/tigervnc/vncserver.users file.

# TigerVNC User assignment
#
# This file assigns users to specific VNC display numbers.
# The syntax is <display>=<username>. E.g.:
#
# :2=andrew
# :3=lisa
:1=ubuntu

In the config above we have mapped user ubuntu to the :1 display on port 5901.

Create a VNC password for your ubuntu user.

vncpasswd

Type and verify your password. The password will be hashed and stored in ~/.vnc/passwd.

Password:
Verify:
Would you like to enter a view-only password (y/n)? n
A view-only password is not used

Enable and start an instance of the tigervncserver service.

sudo systemctl start tigervncserver@:1.service
sudo systemctl enable tigervncserver@:1.service

You should be able to verify that the VNC server is up and running and it’s already listening for new connections.

ubuntu@sdr01:~$ netstat -tlpan | grep 5901
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:5901          0.0.0.0:*               LISTEN      248895/Xtigervnc
tcp6       0      0 ::1:5901                :::*                    LISTEN      248895/Xtigervnc

Note that it listens on localhost interface only, which is fine with me, since I don’t want it exposed anyways. In order to access the VNC server I’m doing it over an SSH tunnel.

ssh -L 5901:localhost:5901 sdr01.internal

Now we can start up a VNC client and connect to localhost:5901 using the password we’ve set before with vncpasswd.

For additional information about TigerVNC, please refer to the TigerVNC Arch Linux Wiki page.

With the VNC issues out of the way, we can proceed and install the SDRplay API and SDRconnect on our Pi.

You can also watch this nice intro about SDRconnect in this video here.

At the time of writing this document the version of SDRconnect I’m installing is release 83273bcd8, which was downloaded from the SDRplay website.

ubuntu@sdr01:~$ wget https://www.sdrplay.com/software/SDRconnect_linux-arm64_83273bcd8.run
ubuntu@sdr01:~$ ./SDRconnect_linux-arm64_83273bcd8.run

Once SDRconnect is successfully installed you should see output similar to the one below.

Press y and RETURN to accept the license agreement and continue with
the installation, or press n and RETURN to exit the installer [y/n] y

Testing for arm64 compliance.......Done

Checking for package tools...apt found

Checking for libusb-1.0...found
Checking for libasound2...found
Checking for libuuid1...found
Checking for libudev1...found

udev directory found, adding rules for RSPs...Done

SDRconnect will be installed into /opt/sdrconnect
To change the installation path, enter the new location at the prompt.
Installation location [/opt/sdrconnect]:

SDRconnect will be installed to: /opt/sdrconnect
To start the installation press y and RETURN or n and RETURN to exit [y/n] y
Installing to /opt/sdrconnect.............Done

Create a system menu entry to launch SDRconnect client? [y/n] y
Installing menu entry......Done

Checking SDRconnect Server Service status...not installed

Do you want this installation to operate the SDRconnect server as a continuous
background service? [y/n] y
SDRconnect Server Service options file installed.
 --> /opt/sdrconnect/server_options.txt

SDRconnect Server Service file created.
Installing service... (may take a while, please wait...)
SDRconnect Server Service started.

To change the server service command line options, edit the
/opt/sdrconnect/server_options.txt file and start the service again.

To stop and start the service, use...
      sudo systemctl stop sdrconnect
      sudo systemctl start sdrconnect

To disable the service (e.g. to stop the service starting after a reboot) and
then enable it again, use...
      sudo systemctl disable sdrconnect.service
      sudo systemctl enable sdrconnect.service

Make a note of this information!
Important Note: When the server service is running, the RSP that it is using
cannot be used by another application on this host. You will need to stop the
server service to make that RSP available to other applications.

Press any key to continue

To add the new installation folder to your search path, please add the following
line to the bottom of your ~/.bashrc file and restart your terminal window...

export PATH=/opt/sdrconnect:$PATH

Warning:
System files have been changed on your system, so it is recommended that you
reboot your system for correct operation.

SDRconnect installation finished.

Add /opt/sdrconnect to your PATH as mentioned in the output above.

echo 'export PATH=/opt/sdrconnect:$PATH' >> ~/.bashrc

Time to install SDRplay RSP API.

ubuntu@sdr01:~$ wget https://www.sdrplay.com/software/SDRplay_RSP_API-Linux-3.15.2.run
ubuntu@sdr01:~$ ./SDRplay_RSP_API-Linux-3.15.2.run

This is what the output from the install script looks like.

Verifying archive integrity... All good.
Uncompressing SDRplay RSP API Install Package V3.15 (installer v2)
SDRplay API 3.15 Installation
=============================

This installation requires to either be run as root, or it will request root
access through the sudo program. This is required to install the daemon into
the system files.

Press RETURN to view the license agreement

...

Press y and RETURN to accept the license agreement and continue with
the installation, or press n and RETURN to exit the installer [y/n] y

A copy of the license agreement can be found here: /home/ubuntu/sdrplay_license.txt


Architecture reported as being 64 bit
System reports 64 bit files found
System is also setup to produce 64 bit files
Architecture reports machine as being arm64 compliant


Checking for root permissions. You may be prompted for your password...

The rest of the installation will continue with root permission...


This API requires the following system dependencies...
libusb and lidudev
If the installer cannot detect the presence of these, you will
have the option to continue and you will need to install them

Checking for package tools...
apt found

Checking for packages...
libusb-1.0 found
libudev1 found
Udev directory found, adding rules...rules added
Adding SDRplay devices to the local hardware database...Done
USB name database not found, continuing...

Installing API files, the default locations are...
API service : /opt/sdrplay_api
API header files : /usr/local/include
API shared library : /usr/local/lib
Daemon start scripts : /etc/systemd/system
Daemon start system : SystemD

To continue the installation with these defaults press y and RETURN
or press n and RETURN to change them [y/n] y
Cleaning old API files...Done.
Installing /usr/local/lib/libsdrplay_api.so.3.15...Done
Installing header files in /usr/local/include...Done
Installing API Service in /opt/sdrplay_api...Done
Installing Service scripts and starting daemon...Created symlink /etc/systemd/system/multi-user.target.wants/sdrplay.service → /etc/systemd/system/sdrplay.service.
Done

The API Service has been installed and started. After the installation has
finished, please reboot this device.

To start and stop the API service, use the following commands...

sudo systemctl start sdrplay
sudo systemctl stop sdrplay

If supported on your system, lsusb will now show the RSP name

SDRplay API 3.15 Installation Finished

Time to reboot the system as mentioned in the output above.

sudo reboot

Verify that you can see your SDRPlay device after rebooting and connecting your SDR.

ubuntu@sdr01:~$ lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 1df7:3020 SDRplay RSPduo
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

In order to connect remotely via SDRConnect the sdrconnect service should already be running and listening on port 50000.

Verify that the sdrconnect.service is up and running.

ubuntu@sdr01:~$ sudo systemctl status sdrconnect
● sdrconnect.service - SDRconnect Server
     Loaded: loaded (/etc/systemd/system/sdrconnect.service; enabled; preset: enabled)
     Active: active (running) since Sun 2025-07-13 13:19:50 EEST; 26min ago
    Process: 2052 ExecStartPre=/bin/sleep 30 (code=exited, status=0/SUCCESS)
   Main PID: 6096 (startServer.sh)
      Tasks: 24 (limit: 18709)
     Memory: 38.9M (peak: 39.3M)
        CPU: 9.191s
     CGroup: /system.slice/sdrconnect.service
             ├─6096 /bin/sh /opt/sdrconnect/startServer.sh
             └─6098 /opt/sdrconnect/SDRconnect --server --port=50000

Jul 13 13:19:52 sdr01.internal startServer.sh[6098]: RF Notch Disabled
Jul 13 13:19:52 sdr01.internal startServer.sh[6098]: DAB Notch Disabled
Jul 13 13:19:52 sdr01.internal startServer.sh[6098]: Hardware Control: 1st Client can control the hardware
Jul 13 13:19:52 sdr01.internal startServer.sh[6098]: Maximum number of clients: 8
Jul 13 13:19:52 sdr01.internal startServer.sh[6098]: PPM Correction: 0
Jul 13 13:19:52 sdr01.internal startServer.sh[6098]: Full IQ: Allowed
Jul 13 13:19:52 sdr01.internal startServer.sh[6098]: Audio mode: Allowed
Jul 13 13:19:52 sdr01.internal startServer.sh[6098]: Virtual Receivers (VRXs): Allowed
Jul 13 13:19:52 sdr01.internal startServer.sh[6098]: Server started
Jul 13 13:19:52 sdr01.internal startServer.sh[6098]: Press CTRL-C to stop the server

And confirm that it’s listening on port 50000 port.

ubuntu@sdr01:~$ sudo netstat -tlpan | grep 50000
tcp        0      0 0.0.0.0:50000           0.0.0.0:*               LISTEN      6098/SDRconnect

NOTE: In case you want to use gqrx with your RSPduo device and upon starting up gqrx you don’t see the RSPduo in the device list, then make sure to stop the sdrconnect service as the device is already in use by it.

So, to use gqrx with your RSPduo first stop the sdrconnect service.

sudo systemctl stop sdrconnect

Start up gqrx now and you should see the RSPduo in the device list. If you don’t intend to use gqrx and want to stick to SDRconnect you should leave the sdrconnect service up and running.

In order to connect to your Pi using SDRconnect you should first add a new remote device. Go ahead and add your Pi to the SDRconnect remote devices list, test the connection and save.

Once you’ve saved your device you need to Refresh Device List as well.

And now you should see your device and be able to connect to it.

Here’s a screenshot of SDRconnect receiving from some local WFM broadcast stations.

Unfortunately at the time of writing this document there was no activity on the 2m and 70cm bands, so I could not take some screenshots of them as well.

Well, that’s all for now.

LZ1DFB, 73!

A new member of the homelab -- HP Pro Mini 400 G9

Mon, 07 Jul 2025 00:00:00 +0000

My homelab setup hasn’t changed much for the last few years.

I have been happily running a couple of Intel NUC systems over the years. Initially they have been running ESXi, then they got migrated to Arch Linux and KVM.

The workloads running on these systems vary, depending on what I’m actually working on or testing at a given point in time, but the ones that keep running on them at all times are Kubernetes, AdGuard, SyncThing, Plex, Gitea and some Gitea Action Runners.

For running Kubernetes I’m using Talos Linux nodes, and I really like the way Talos handles the Kubernetes lifecycle management using minimal and immutable nodes.

I should mention that the Intel NUC systems are quite old. One of them is running an Intel i3 5th generation, and the other one is Intel NUC i5 8th generation. Over the years the only maintenance I had to do on these systems is to make sure to replace their fans, but besides that they have been rock solid.

While these systems do serve their purpose just fine for my humble homelab requirements, I thought it was about time I upgrade it a bit, so that I can future-proof it for the years to come.

In case you haven’t heard of this yet, Intel has discontinued development of the NUC systems. Future development of NUC is now handled by ASUS, so before I bite the bullet and order a new NUC, I’ve decided to do a bit of research about these new ASUS NUCs.

Unfortunately, when reading reviews for the ASUS NUCs you will find lots of mixed feelings – some people claim they are good, others mention that they run pretty hot and thermal throttle. One of the most concerning things to me is the fact that I’ve been reading a lot of reviews about the ASUS NUC 14 (and 15) Pro from users complaining about the loud fan noise. And that is not something I’d like to have.

With the old Intel NUCs it was easy – the CPUs back in the day were not that powerful and with low enough TDP, making them a no brainer when it comes to picking up an Intel NUC for your homelab.

Eventually, I gave up on the idea of having an ASUS NUC as my new homelab node, and after some time spent on researching I decided to go with HP Pro Mini 400 G9.

I went with the following specs.

Intel Core i5-13500T
64GB memory
2TB Samsung 990 Evo Plus NVME

Additional specs about this machine can be found in the HP Pro Series 400 G9 specs page.

Compared to my old Intel NUCs this system is more powerful and capable of replacing my two NUC nodes. The case is all metal, which is better than any plastic case, and helps with thermal dissipation.

Besides the NVME drive you can also install a 2.5 inch SATA SSD as well. One thing that was not clear to me when ordering this machine is the fact that when you purchase this system with an NVME disk it will arrive without the 2.5 inch SATA Drive Bay kit.

So, if you are planning on installing a 2.5 inch SATA disk you will need to order the HP Desktop Mini 2.5” SATA Drive Bay kit v2 (13L70AA) separately. I got my brand new 2.5 inch SATA drive bay from eBay.

I’ve also got two SOLIDIGM D3-S4520 SATA SSD disks. These are enterprise SATA SSD disks with Power Loss Protection (PLP) and should serve me well for the coming years.

As the system only has a single drive bay one of these disks goes into the drive bay, and the other one will be connected to the machine via a USB Type C enclosure. Then both disks will be part of a ZFS mirror pool. The NVME disk will be added to a separate ZFS pool and will be used for workloads which have faster I/O needs.

Another nice thing about this system is how easy it is to access the mainboard, throw some RAM in it, replace an SSD or fan, which makes maintenance an easy and hassle-free job. The installation of the drive bay was easy and nicely fits into the chassis.

One thing I really like about the HP Pro Mini 400 G9 is how quite it actually is, even under load.

I’ve installed Proxmox VE on this system with OpenZFS and have already migrated my workloads from the Intel NUC systems to it.

$ sudo zpool status
  pool: rpool
 state: ONLINE
  scan: resilvered 38.3G in 00:01:42 with 0 errors on Mon Jul 14 21:43:27 2025
config:

        NAME                                                     STATE     READ WRITE CKSUM
        rpool                                                    ONLINE       0     0     0
          mirror-0                                               ONLINE       0     0     0
            ata-SOLIDIGM_SSDSC2KB019TZ_PHYI513202JB1P9DGN-part2  ONLINE       0     0     0
            ata-SOLIDIGM_SSDSC2KB019TZ_PHYI513202F91P9DGN-part2  ONLINE       0     0     0

errors: No known data errors

And here’s the system installed in the homelab rack.

In the picture above you will see one of the Intel NUCs, the new HP Pro Mini 400 G9 next to it, a MikroTik RouterBOARD RB3011 UIAS-RM, JetKVM currently connected to the Intel NUC and my old Synology DS213j storage server, which needs an upgrade as well.

Using JWT Authentication Method with the Go SDK for Vault

Thu, 03 Jul 2025 00:00:00 +0000

In the Setting up a Kubernetes cluster with Vault and OIDC trust post we’ve seen how to configure identity federation between a Kubernetes cluster and Vault.

Once you have the JWT Authentication Method configured properly in Vault, we can get access to Vault by first authenticating against the auth method’s /login endpoint. This is how you would do it when using the Vault CLI.

vault write auth/jwt/login role=my-role-name jwt=@/path/to/my/token

On successful authentication Vault will give us back a token, which we can use in subsequent calls to Vault, e.g. for retrieving secrets, adding new secrets, etc.

And what about the SDK? How do authenticate against our JWT/OIDC Auth Method when using the Go SDK?

The Vault SDK for Go can be found in the github.com/hashicorp/vault/api package. There’s also a quick start example for Go in the github.com/hashicorp/vault-examples repo. And in the same repo there’s also an example of using JWT Auth Method from a Kubernetes cluster.

But what about a more generic JWT/OIDC Auth Method support in the Go SDK? You know, one that doesn’t actually rely on Kubernetes Service Account Tokens.

Well, it turns out that a more generic JWT Auth Method does not exist in the SDK yet.

Technically speaking the auth.KubernetesAuth implementation of the vault.AuthMethod interface does that already, and you could use it outside of Kubernetes by tweaking the various options such as WithServiceAccountToken, WithServiceAccountTokenPath, etc., but at the same time this implementation contains too much Kubernetes details such as service accounts, default token path in a Kubernetes pod, etc.

Besides, the name of the Auth Method implementation might confuse people when reading your code making them think that Kubernetes is a requirement, while actually it is not.

So, what can we do? Well, we can implement our own vault.AuthMethod based on the existing auth.KubernetesAuth implementation. And this is what we are going to do in this post.

Will start off by creating a new Go module for our JWT Auth Method implementation.

mkdir vault-jwt-auth-method
cd vault-jwt-auth-method
go mod init vault-jwt-auth-method
go get -v github.com/hashicorp/vault/api

Our JWT Auth Method implementation will reside in the auth/jwt package, similar to the way how the existing Auth Method implementations reside in the upstream Vault SDK repo.

mkdir -p pkg/auth/jwt

Open up pkg/auth/jwt/jwt.go in your favorite $EDITOR. And this is what the JWT Auth Method implementation looks like.

package jwt

import (
	"context"
	"errors"
	"fmt"
	"os"
	"strings"

	vault "github.com/hashicorp/vault/api"
)

// DefaultMountPath specifies the default mount path for the JWT
// Authentication Method.
const DefaultMountPath = "jwt"

// ErrNoToken is an error, which is returned when [JWTAuth] is configured
// with an empty token.
var ErrNoToken = errors.New("no token specified")

// ErrInvalidMountPath is an error, which is returned when configuring [JWTAuth]
// to use an invalid mount path for a Vault Authentication Method.
var ErrInvalidMountPath = errors.New("invalid auth method mount path specified")

// JWTAuth implements support for the [JWT Authentication Method].
//
// [JWT Authentication Method]: https://developer.hashicorp.com/vault/docs/auth/jwt
type JWTAuth struct {
	// roleName specifies the name of the role to use.
	roleName string

	// mountPath specifies the mount path for the JWT Authentication Method.
	mountPath string

	// token specifies the JWT token which will be used for authenticating
	// against the Vault Authentication Method endpoint.
	token string
}

var _ vault.AuthMethod = &JWTAuth{}

// Option is a function which configures [JWTAuth].
type Option func(a *JWTAuth) error

// New creates a new [JWTAuth] and configures it with the given options.
//
// The default mount path for the JWT Authentication Method is
// [DefaultMountPath]. In order to configure a different mount path for the
// Authentication Method you can use the [WithMountPath] option.
//
// The JWT token which will be used for authentication against the Vault
// Authentication Method login endpoint may be specified either as a string,
// from path, or via an environment variable. In order to configure the token
// for authentication use the [WithToken], [WithTokenFromPath] or
// [WithTokenFromEnv] options.
func New(roleName string, opts ...Option) (*JWTAuth, error) {
	jwtAuth := &JWTAuth{
		roleName:  roleName,
		mountPath: DefaultMountPath,
	}

	for _, opt := range opts {
		if err := opt(jwtAuth); err != nil {
			return nil, err
		}
	}

	if jwtAuth.token == "" {
		return nil, ErrNoToken
	}

	if jwtAuth.mountPath == "" {
		return nil, ErrInvalidMountPath
	}

	return jwtAuth, nil
}

// Login implements the [vault.AuthMethod] interface.
func (a *JWTAuth) Login(ctx context.Context, client *vault.Client) (*vault.Secret, error) {
	path := fmt.Sprintf("auth/%s/login", a.mountPath)
	data := map[string]any{
		"jwt":  a.token,
		"role": a.roleName,
	}

	return client.Logical().WriteWithContext(ctx, path, data)
}

// WithToken is an [Option], which configures [JWTAuth] to use the given token
// when authenticating against the Vault JWT Authentication Method.
func WithToken(token string) Option {
	opt := func(a *JWTAuth) error {
		a.token = token

		return nil
	}

	return opt
}

// WithTokenFromPath is an [Option], which configures [JWTAuth] to read the
// token from the given path.
func WithTokenFromPath(path string) Option {
	opt := func(a *JWTAuth) error {
		token, err := os.ReadFile(path)
		if err != nil {
			return err
		}
		a.token = string(token)

		return nil
	}

	return opt
}

// WithTokenFromEnv is an [Option], which configures [JWTAuth] to read the token
// from the given environment variable.
func WithTokenFromEnv(env string) Option {
	opt := func(a *JWTAuth) error {
		value := os.Getenv(env)
		a.token = value

		return nil
	}

	return opt
}

// WithMountPath is an [Option], which configures [JWTAuth] to use the given
// mount path for the Vault Authentication Method.
func WithMountPath(mountPath string) Option {
	opt := func(a *JWTAuth) error {
		// Remove any trailing slashes from the given mount path
		a.mountPath = strings.TrimRight(mountPath, "/")

		return nil
	}

	return opt
}

And here is a simple example of how we can use our JWT Auth Method in Go.

package main

import (
	"context"
	"errors"
	"fmt"
	"os"

	vault "github.com/hashicorp/vault/api"

	jwtauth "vault-jwt-auth-method/pkg/auth/jwt"
)

func printErr(err error) {
	fmt.Println(err)
	os.Exit(1)
}

func main() {
	ctx := context.Background()

	// Create a JWT Authentication Method using the token from the following
	// role and token stored in the path
	authMethod, err := jwtauth.New(
		"my-role-name",
		jwtauth.WithTokenFromPath("/path/to/my/token"),
	)
	if err != nil {
		printErr(err)
	}

	// Create Vault client and authenticate using our JWT Authentication
	// Method implementation
	config := vault.DefaultConfig()
	client, err := vault.NewClient(config)
	if err != nil {
		printErr(err)
	}

	// Login using our Auth Method
	authSecret, err := client.Auth().Login(ctx, authMethod)
	if err != nil {
		printErr(err)
	}
	if authSecret == nil {
		printErr(errors.New("no auth info returned after login"))
	}

	// Read a sample KV v2 secret
	kvv2 := client.KVv2("kvv2")
	secret, err := kvv2.Get(ctx, "my/test/secret")
	if err != nil {
		printErr(err)
	}

	fmt.Println(secret.Data)
}

Additional improvement to the example code above would be to renew the auth secret using a vault.LifetimeWatcher, so that our API client is kept authenticated, especially if we are working on a service, which is supposed to be long-running.

Btw, in case you have missed this one, just want to remind the readers that Hashicorp Vault got forked a couple of years ago as a response to Hashicorp relicensing their OSS projects.

The new OSS friendly fork is now known as OpenBao. The code from this post has been adapted and submitted to OpenBao as a feature.

Setting up a Kubernetes cluster with Vault and OIDC trust

Tue, 01 Jul 2025 00:00:00 +0000

In this post we will see how to create and configure a development Kubernetes cluster with Vault running in it and setup OpenID Connect trust between Vault and Kubernetes workloads.

Create the Kubernetes cluster

First, create a new Kubernetes cluster using minikube.

env KUBECONFIG=$(pwd)/kubeconfig minikube start --profile vault-k8s-dev

If you are using direnv, you can also create a local .envrc file to point to our cluster and kubeconfig when you enter the directory where your development cluster kubeconfig is stored.

export KUBECONFIG=$(pwd)/kubeconfig
minikube profile vault-k8s-dev

Enable and reload the config.

direnv allow
direnv reload

Once the cluster is up and running, verify that it runs with enabled Service account issuer discovery.

kubectl get --raw '/.well-known/openid-configuration' | jq

The command above will print output similar to the one below. The /.well-known/openid-configuration endpoint provides the OpenID Connect Provider Metadata. It includes information about the issuer, where to find the public keys for verifying JWT tokens, the supported signing algorithms, etc.

{
  "issuer": "https://kubernetes.default.svc.cluster.local",
  "jwks_uri": "https://10.0.2.15:8443/openid/v1/jwks",
  "response_types_supported": [
    "id_token"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ]
}

Alternatively, simply check the options, which are used to start the kube-apiserver static pod, e.g.

kubectl --namespace kube-system describe pod kube-apiserver-vault-k8s-dev | grep service-account

Above command should print the following output.

--service-account-issuer=https://kubernetes.default.svc.cluster.local
--service-account-key-file=/var/lib/minikube/certs/sa.pub
--service-account-signing-key-file=/var/lib/minikube/certs/sa.key

We will also enable anonymous access to the API endpoints related to OpenID Connect, since these API endpoints don’t serve any sensitive data and should be public.

In order to do that we will bind the system:service-account-issuer-discovery cluster role to the system:anonymous user. This is what the cluster role represents.

> kubectl describe clusterrole system:service-account-issuer-discovery
Name:         system:service-account-issuer-discovery
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
  Resources  Non-Resource URLs                     Resource Names  Verbs
  ---------  -----------------                     --------------  -----
             [/.well-known/openid-configuration/]  []              [get]
             [/.well-known/openid-configuration]   []              [get]
             [/openid/v1/jwks/]                    []              [get]
             [/openid/v1/jwks]                     []              [get]

Create the cluster role binding.

kubectl create clusterrolebinding system:anonymous-service-account-issuer-discovery \
        --clusterrole system:service-account-issuer-discovery \
        --user=system:anonymous

Start a test pod in the cluster and verify that you can access the OpenID Discovery endpoint, e.g.

curl -X GET https://kubernetes.default.svc.cluster.local/.well-known/openid-configuration

Deploy Vault in the Kubernetes Cluster

Next, we will create the Kubernetes resources, which describe our Vault configuration, service and statefulset. There are many different ways to get Vault installed in your Kubernetes, but in order to keep things simple and straight forward we will use these simple resources.

This is the ConfigMap providing the Vault config.

apiVersion: v1
data:
  config.json: |
    {
      "storage": {
        "file": {
          "path": "/vault/file"
        }
      },
      "listener": [
        {
          "tcp": {
            "address": "0.0.0.0:8200",
            "tls_disable": true
          }
        }
      ],
      "default_lease_ttl": "168h",
      "max_lease_ttl": "720h",
      "ui": true
    }
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: vault-config

The Service resource.

---
apiVersion: v1
kind: Service
metadata:
  name: vault
  labels:
    app: vault
spec:
  ports:
  - port: 8200
  selector:
    app: vault
  type: ClusterIP

And the StatefulSet resource.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vault
spec:
  serviceName: "vault"
  replicas: 1
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      containers:
      - name: vault
        image: hashicorp/vault:1.20
        args: ['server']
        securityContext:
          capabilities:
            add: ["IPC_LOCK"]
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /ui/sys/health
            port: 8200
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /ui/sys/health
            port: 8200
        ports:
        - containerPort: 8200
        volumeMounts:
        - name: vault-data
          mountPath: /vault/file
        - name: vault-config
          mountPath: /vault/config

      volumes:
      - name: vault-config
        configMap:
          name: vault-config
  volumeClaimTemplates:
  - metadata:
      name: vault-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

Apply these resources and confirm that Vault is up and running.

> kubectl get pods,services
  NAME          READY   STATUS    RESTARTS   AGE
  pod/vault-0   1/1     Running   0          24m

  NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
  service/kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP    64m
  service/vault        ClusterIP   10.108.56.80   <none>        8200/TCP   48m

Initialize and unseal Vault

Once Vault is up and running we need to initialize it. In order to make things simpler when interfacing with Vault we will port-forward it’s port to our local system.

export VAULT_ADDR=http://localhost:8200/
kubectl port-forward service/vault 8200:8200

And now, we can initialize it.

vault operator init

Above command will print the unseal keys, e.g.

Unseal Key 1: 2K/...
Unseal Key 2: xEKFbr...
Unseal Key 3: VFiwfbI43...
Unseal Key 4: 8Je/INW6lQOI...
Unseal Key 5: /1AFwxmvgw7UnFa...

Initial Root Token: hvs...

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Now, we can unseal the Vault. Repeat the command below at least 3 times in order to unseal it.

vault operator unseal

We will also mount the KV v2 secrets engine, which we’ll later use.

vault secrets enable --version=2 --path kvv2 kv

List the enabled secret engines and we should see our kvv2 mount.

> vault secrets list
Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_f86afe98    per-token private secret storage
identity/     identity     identity_fa281c9c     identity store
kvv2/         kv           kv_7e911817           n/a
sys/          system       system_f6068170       system endpoints used for control, policy and debugging

Enable JWT/OIDC authentication method in Vault

Now, we can continue with enabling the jwt authentication method.

vault auth enable jwt

List the enabled authentication methods.

> vault auth list
Path      Type     Accessor               Description                Version
----      ----     --------               -----------                -------
jwt/      jwt      auth_jwt_cbaae747      n/a                        n/a
token/    token    auth_token_468286d4    token based credentials    n/a

Configure the authentication method for jwt by using the /auth/<auth-method>/config endpoint.

You can check the additional parameters, which can be set in the JWT/OIDC auth method (API) documentation.

vault write auth/jwt/config \
      oidc_discovery_ca_pem="<Kubernetes-API-Server-CA>" \
      oidc_client_id="" \
      oidc_client_secret="" \
      default_role="k8s-dev"

The Kubernetes API Server CA bundle can also be obtained from the running vault-0 pod and the location where the service account token is mounted, which is /var/run/secrets/kubernetes.io/serviceaccount/ca.crt. Since we are running our development Kubernetes cluster in minikube the CA bundle can also be found in ~/.minikube/ca.crt on your local system.

If your Kubernetes cluster uses a certificate signed an official Certificate Authority such as DigiCert, Let’s Encrypt, etc. then you don’t need to specify the oidc_discovery_ca_pem parameter, but since this is a development cluster it comes with self-signed certificate and it’s own CA, which is why we need to specify it here.

The previous command can also be executed against the running pod container like this.

kubectl exec vault-0 -- vault write auth/jwt/config \
        oidc_discovery_ca_pem=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
        oidc_client_id="" \
        oidc_client_secret="" \
        default_role="k8s-dev"

Verify that the authentication method has been correctly configured.

> vault read auth/jwt/config
Key                                     Value
---                                     -----
bound_issuer                            n/a
default_role                            k8s-dev
jwks_ca_pem                             n/a
jwks_pairs                              []
jwks_url                                n/a
jwt_supported_algs                      []
jwt_validation_pubkeys                  []
namespace_in_state                      true
oidc_client_id                          n/a
oidc_discovery_ca_pem                   -----BEGIN CERTIFICATE-----...
oidc_discovery_url                      https://kubernetes.default.svc.cluster.local
oidc_response_mode                      n/a
oidc_response_types                     []
provider_config                         map[]
unsupported_critical_cert_extensions    []

Next thing we need to do is to create the k8s-dev role, which we’ve specified in the previous step. This role will be associated with the k8s-dev-writer policy, allowing Kubernetes workloads read/write access to Vault for a given path.

vault write auth/jwt/role/k8s-dev \
      bound_subject="system:serviceaccount:default:vault-writer" \
      bound_audiences="vault-dev" \
      user_claim="sub" \
      policies=k8s-dev-writer \
      role_type=jwt \
      ttl=1h

The bound_subject parameter from the command above specifies a Kubernetes service account named vault-writer from the default namespace. Kubernetes workloads, which will access Vault should be running with this service account, and also the service account token should be issued with the vault-dev audience.

Finally, we need to create the k8s-dev-writer policy, which will give our Kubernetes workloads read/write access to Vault. This is what our simple policy looks like.

path "kvv2/metadata/k8s-dev/*" {
    capabilities = ["list"]
}

path "kvv2/data/k8s-dev/*" {
    capabilities = ["create", "update", "patch", "read", "delete"]
}

We can now apply this policy.

vault policy write k8s-dev-writer /path/to/k8s-dev-writer-policy.hcl

Time to test things out. Let’s create the vault-writer Kubernetes service account.

kubectl create sa vault-writer

We can now create a test token for our vault-writer service account and test accessing Vault using this token.

kubectl create token vault-writer --audience=vault-dev

Save the token somewhere, because we will need it a bit later. This is what the payload of our JWT token looks like when decoded.

{
  "aud": [
    "vault-dev"
  ],
  "exp": 1751300275,
  "iat": 1751296675,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "jti": "066083b5-1219-4c91-8338-728365776f22",
  "kubernetes.io": {
    "namespace": "default",
    "serviceaccount": {
      "name": "vault-writer",
      "uid": "bc5cffb5-65e6-4921-835c-a0a8cdaa69db"
    }
  },
  "nbf": 1751296675,
  "sub": "system:serviceaccount:default:vault-writer"
}

In order to access Vault using our Kubernetes service account token, first we need to get a Vault token. This step is similar to what OAuth 2.0 Token Exchange does. We use our Kubernetes service account token in order to get a Vault token. And this is how we do it.

vault write auth/jwt/login role=k8s-dev jwt="<k8s-sa-token>"

On successful authentication we should see a similar output.

Key                  Value
---                  -----
token                hvs...
token_accessor       ...
token_duration       1h
token_renewable      true
token_policies       ["default" "k8s-dev-writer"]
identity_policies    []
policies             ["default" "k8s-dev-writer"]
token_meta_role      k8s-dev

And now we can use the token from the output above in order to access Vault and write a sample secret.

export VAULT_TOKEN="<vault-token-goes-here>"
vault kv put kvv2/k8s-dev/foo bar=baz

Read back the secret we’ve just created.

> vault kv get kvv2/k8s-dev/foo
==== Secret Path ====
kvv2/data/k8s-dev/foo

======= Metadata =======
Key                Value
---                -----
created_time       2025-06-30T15:42:46.718550078Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

=== Data ===
Key    Value
---    -----
bar    baz

Since our policy allows read/write access to the kvv2/data/k8s-dev/* path only we should not be able to read or write anything else, so let’s try that out.

> vault kv put kvv2/some-secret bar=baz
Error writing data to kvv2/data/some-secret: Error making API request.

URL: PUT http://localhost:8200/v1/kvv2/data/some-secret
Code: 403. Errors:

* 1 error occurred:
        * permission denied

Looks good. Finally, we can create a test pod which uses our vault-writer service account and access Vault from it. The following example Kubernetes manifest can be used to start a test pod using the vault-writer service account.

This example uses Service Account Token projection in order to inject a token into the running pod, using an audience, which Vault expects our clients to have as part of their JWT claims.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: vault-client
  name: vault-client
spec:
  containers:
  - command:
    - sleep
    - infinity
    env:
      - name: VAULT_ADDR
        value: http://vault:8200/
    image: hashicorp/vault:1.20
    name: vault-client
    volumeMounts:
      - name: token-vol
        mountPath: /service-account
        readOnly: true
  volumes:
    - name: token-vol
      projected:
        sources:
          - serviceAccountToken:
              audience: vault-dev
              expirationSeconds: 3600
              path: token
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  serviceAccountName: vault-writer

Create the test pod and log into it.

kubectl apply -f /path/to/vault-client.yaml
kubectl exec -it vault-client -- /bin/sh

Once you get a shell in the running pod, simply exchange the Kubernetes service account token for Vault token, similar to the way it was done in the previous examples.

vault write auth/jwt/login role=k8s-dev jwt=@/service-account/token

References

Make sure to check the following documents for additional information on the topic.

Making Emacs and consult-imenu better with tree-sitter

Tue, 17 Jun 2025 00:00:00 +0000

Not long ago I have switched from lsp-mode to eglot and tree-sitter.

Approximately at the same time I’ve also moved from helm to vertico and the complementary packages for it such as marginalia, consult and orderless.

So far I’ve been pretty happy with the transition. eglot and vertico feel snappy and more responsive than lsp-mode and helm. And marginalia is great as well, because it provides nice annotations next to your completion candidates.

Also, eglot is part of Emacs for quite some time already, and vertico (and it’s complementary packages) build on top of the standard Emacs APIs such as completing-read, which is another good reason to stick with them.

consult provides search and navigation commands, and also comes with support for (amongst other things) imenu via M-x consult-imenu.

consult and marginalia pair nicely together. Here’s an example of M-x consult-imenu displaying the symbols for an Elisp buffer, which look great. The extra annotations next to each completion candidate are provided by marginalia itself.

You get a nice menu with entries grouped by Types, Functions, Variables, etc. You can also narrow down the search for entries of specific kind, e.g. when you want to browse the functions only, simply type f SPC in the consult-imenu prompt and it will give you the function definitions only.

At $WORK I’m primary working on Go code these days, and one thing I’ve noticed when using M-x consult-imenu is that the results are less impressive when compared to the Elisp code.

As you can see the results are nowhere near to what M-x consult-imenu (enriched by marginalia) displays for Elisp code.

In this post we will see how to make M-x consult-imenu look better when working with Go code (or any other language with tree-sitter support).

After some time spent on reading documentation and manuals I was not able to fix it myself, so I made a post at r/emacs asking whether someone else was having the same issues as I do. In the meantime I kept digging.

So, what turned out to be the issue for me, is that eglot provides it’s own imenu-create-index-function called eglot-imenu, which actually wraps treesit-simple-imenu, and treesit-simple-imenu is tree-sitter’s approach to building the imenu index.

Here’s what the value of imenu-create-index-function looks like, when set by eglot.

Use C-h v imenu-create-index-function (or M-x describe-variable imenu-create-index-function).

imenu-create-index-function is a variable defined in ‘imenu.el’.

Its value is
#f(advice eglot-imenu :before-until treesit-simple-imenu)
Local in buffer main.go; global value is
imenu-default-create-index-function

The function to use for creating an index alist of the current buffer.

It should be a function that takes no arguments and returns
an index alist of the current buffer.  The function is
called within a ‘save-excursion’.

See ‘imenu--index-alist’ for the format of the buffer index alist.

  Automatically becomes buffer-local when set.
  This variable may be risky if used as a file-local variable.

My current tree-sitter configuration looks like this. Make sure to check How to Get Started with Tree-Sitter if you are just starting out with tree-sitter.

After configuring tree-sitter you should also install the language grammars using M-x treesit-install-language-grammar.

;; tree-sitter settings

(use-package treesit
  :config
  ;; Language grammars and sources
  (setq treesit-language-source-alist
        '((bash "https://github.com/tree-sitter/tree-sitter-bash")
          (c "https://github.com/tree-sitter/tree-sitter-c")
          (go "https://github.com/tree-sitter/tree-sitter-go")
          (go-mod "https://github.com/camdencheek/tree-sitter-go-mod")
          (json "https://github.com/tree-sitter/tree-sitter-json")
          (python "https://github.com/tree-sitter/tree-sitter-python")))

  ;; Remap major modes to treesitter modes
  (setq major-mode-remap-alist
        '((go-mode . go-ts-mode)
          (python-mode . python-ts-mode)
          (js-json-mode . json-ts-mode)
          (json-mode . json-ts-mode)
          (sh-mode . bash-ts-mode))))

In order to make eglot stop messing with imenu we need to tell it so. My current (simplified) eglot config looks like this.

Note that we are configuring eglot-stay-out-of in the :config section, so that eglot doesn’t interfere with imenu index generation.

;; eglot settings

(defun eglot-format-buffer-before-save ()
  (add-hook 'before-save-hook #'eglot-format-buffer -10 t))

(defun eglot-organize-imports-before-save ()
  (add-hook 'before-save-hook (lambda ()
                                (call-interactively #'eglot-code-action-organize-imports))
            nil t))

(use-package eglot
  :ensure t
  :init
  (require 'company)
  (require 'yasnippet)
  (require 'go-mode)
  :config
  ;; We want to use treesit-simple-imenu for imenu index creation
  (add-to-list 'eglot-stay-out-of 'imenu)

  ;; Shutdown LSP server when last project buffer is closed as well
  (setq eglot-autoshutdown t)

  ;; Enable eglot for the following modes
  (dolist (mode '(go-mode-hook go-ts-mode-hook))
    (add-hook mode 'eglot-ensure))

  ;; Buffer-local hooks which install before-save hooks
  (add-hook 'go-ts-mode-hook #'eglot-format-buffer-before-save)
  (add-hook 'go-ts-mode-hook #'eglot-organize-imports-before-save))

;; eglot-booster requires that `emacs-lsp-booster' is installed [1]
;; [1]: https://github.com/blahgeek/emacs-lsp-booster
(use-package eglot-booster
  :ensure t
  :after (eglot)
  :config
  (eglot-booster-mode 1))

And my (simplified) settings for go-ts-mode look like this.

;; Go specific settings

;; Help out the builtin project find the Go module path
(require 'project)

(defun project-find-go-module (dir)
  (when-let ((root (locate-dominating-file dir "go.mod")))
    (cons 'go-module root)))

(cl-defmethod project-root ((project (head go-module)))
  (cdr project))

(add-hook 'project-find-functions #'project-find-go-module)

(use-package go-ts-mode
  :ensure t
  :config
  (add-hook 'go-ts-mode-hook #'company-mode))

Instead of using eglot-imenu we will configure a hook for our go-ts-mode which sets imenu-index-create-function to treesit-simple-imenu.

(add-hook 'go-ts-mode (lambda () (setq-local imenu-index-create-function #'treesit-simple-imenu)))

Later we will move this hook to the use-package definition for go-ts-mode.

Next, we need to tell treesit-simple-imenu how to generate the imenu index. treesit-simple-imenu relies on treesit-simple-imenu-settings variable when generating the imenu index.

treesit-simple-imenu-settings is a variable defined in ‘treesit.el’.

Its value is nil

Settings that configure ‘treesit-simple-imenu’.

It should be a list of (CATEGORY REGEXP PRED NAME-FN).

CATEGORY is the name of a category, like "Function", "Class",
etc.  REGEXP should be a regexp matching the type of nodes that
belong to CATEGORY.  PRED should be either nil or a function
that takes a node an the argument.  It should return non-nil if
the node is a valid node for CATEGORY, or nil if not.

CATEGORY could also be nil.  In that case the entries matched by
REGEXP and PRED are not grouped under CATEGORY.

NAME-FN should be either nil or a function that takes a defun
node and returns the name of that defun node.  If NAME-FN is nil,
‘treesit-defun-name’ is used.

‘treesit-major-mode-setup’ automatically sets up Imenu if this
variable is non-nil.

  Probably introduced at or before Emacs version 30.1.

By default the go-ts-mode.el defines treesit-simple-imenu-settings like this.

    ;; Imenu.
    (setq-local treesit-simple-imenu-settings
                `(("Function" "\\`function_declaration\\'" nil nil)
                  ("Method" "\\`method_declaration\\'" nil nil)
                  ("Struct" "\\`type_declaration\\'" go-ts-mode--struct-node-p nil)
                  ("Interface" "\\`type_declaration\\'" go-ts-mode--interface-node-p nil)
                  ("Type" "\\`type_declaration\\'" go-ts-mode--other-type-node-p nil)
                  ("Alias" "\\`type_declaration\\'" go-ts-mode--alias-node-p nil)))

The docstring of treesit-simple-imenu-settings shown above shows the structure of the list. The second item in the each list as mentioned there is a regexp which matches the type of the tree-sitter node.

When making any changes to treesit-simple-imenu-settings it is best that you review the AST tree as you go by using M-x treesit-explore-mode.

This is what the AST looks like for our sample Go program.

So, after spending some time configuring treesit-simple-imenu-settings with the help of M-x treesit-explore-mode here’s what my current config looks like.

;; Treesitter helpers to extract node names for various symbols such as consts,
;; vars, imports, etc.

(defun my/go-ts-var-is-global-p (node)
  "Predicate which tests whether a `var_spec' node belongs to the global scope"
  ;; The parent node for `var_spec' which is `var_declaration' belongs to the
  ;; global scope, if the `var_declaration' node belongs to a parent node of
  ;; type `source_file'.
  (let* ((var-declaration (treesit-parent-until
                           node
                           (lambda (item)
                             (string-equal (treesit-node-type item) "var_declaration"))
                           t))
         (var-declaration-parent (treesit-node-parent var-declaration)))
    (string-equal (treesit-node-type var-declaration-parent) "source_file")))

(defun my/go-ts-const-is-global-p (node)
  "Predicate which tests whether a `const_spec' node belongs to the global scope"
  ;; The parent node for `const_spec' which is `const_declaration' belongs to the
  ;; global scope, if the `const_declaration' node belongs to a parent node of
  ;; type `source_file'.
  (let* ((const-declaration (treesit-parent-until
                             node
                             (lambda (item)
                               (string-equal (treesit-node-type item) "const_declaration"))
                             t))
         (const-declaration-parent (treesit-node-parent const-declaration)))
    (string-equal (treesit-node-type const-declaration-parent) "source_file")))

(defun my/go-ts-const-spec-node-name (node)
  "Returns the name of a `const_spec' node"
  (let ((const-name (treesit-node-child-by-field-name node "name")))
    (treesit-node-text const-name)))

(defun my/go-ts-var-spec-node-name (node)
  "Returns the name of a `var_spec' node"
  (let ((var-name (treesit-node-child-by-field-name node "name")))
    (treesit-node-text var-name)))

(defun my/go-ts-import-node-name (node)
  "Returns the name of a `import_spec' node"
  (let* ((import-path (treesit-node-child-by-field-name node "path"))
         (import-alias (treesit-node-child-by-field-name node "name"))
         (package-name (treesit-search-subtree import-path "interpreted_string_literal_content")))
    (cond
     (import-alias
      (format "%s (%s)" (treesit-node-text package-name) (treesit-node-text import-alias)))
      (t (treesit-node-text package-name)))))

(defun my/treesit-simple-imenu-settings ()
  (setq-local imenu-max-item-length 200)
  (setq-local imenu-index-create-function #'treesit-simple-imenu)
  (setq-local treesit-simple-imenu-settings
              '(("Imports" "\\`import_spec\\'" nil my/go-ts-import-node-name)
                ("Functions" "\\`function_declaration\\'" nil nil)
                ("Constants" "\\`const_spec\\'" my/go-ts-const-is-global-p my/go-ts-const-spec-node-name)
                ("Variables" "\\`var_spec\\'" my/go-ts-var-is-global-p my/go-ts-var-spec-node-name)
                ("Interfaces" "\\`type_declaration\\'" go-ts-mode--interface-node-p nil)
                ("Types" "\\`type_declaration\\'" go-ts-mode--other-type-node-p nil)
                ("Aliases" "\\`type_declaration\\'" go-ts-mode--alias-node-p nil)
                ("Structs" "\\`type_declaration\\'" go-ts-mode--struct-node-p nil)
                ("Methods" "\\`method_declaration\\'" nil nil)))

  (setq-local consult-imenu-config
              '((go-ts-mode :toplevel "Imports"
                            :types ((?I "Imports"    font-lock-constant-face)
                                    (?f "Functions"  font-lock-function-name-face)
                                    (?c "Constants"  font-lock-constant-face)
                                    (?v "Variables"  font-lock-variable-name-face)
                                    (?i "Interfaces" font-lock-type-face)
                                    (?t "Types"      font-lock-type-face)
                                    (?a "Aliases"    font-lock-type-face)
                                    (?s "Structs"    font-lock-type-face)
                                    (?m "Methods"    font-lock-function-name-face))))))

What is left now is to add a hook for go-ts-mode, which configures everything for us.

My updated go-ts-mode config looks like this.

(use-package go-ts-mode
  :ensure t
  :config
  (add-hook 'go-ts-mode-hook #'my/treesit-simple-imenu-settings)
  (add-hook 'go-ts-mode-hook #'company-mode))

Here’s how it looks like with the sample Go program shown before.

Things look much better now. We have various categories such as Imports, Functions, Variables, etc. And the index generation happens much faster now, because it relies on tree-sitter only.

makefile-graph with Apache Echarts support

Sat, 24 May 2025 00:00:00 +0000

In a previous post I’ve talked about makefile-graph - a tool which lets you visualize your Makefile targets so you can better understand how targets relate to others.

The makefile-graph tool initially supported two output formats - dot and tsort. The tsort format prints the topological order after performing a topological sort of the graph, and the dot format outputs a representation of the graph in the Dot language.

Starting with version v0.1.4 of makefile-graph we can now render a graph of the Makefile targets using Apache Echarts, which provides a better interactive experience when inspecting your Makefile targets.

In order to install the latest version of makefile-graph execute the following command.

go install github.com/dnaeon/makefile-graph/cmd/makefile-graph@latest

In order to render a graph using Apache Echarts we need to specify the --format echarts option, e.g.

makefile-graph \
    --makefile examples/Makefile \
    --direction LR \
    --format echarts

The command above will generate an HTML document, which you can save and open.

If you want to use a different theme (e.g. dark, vintage, etc.) you can specify the Echarts theme using the --theme option. The font size and style can be controlled via the --font-size and --font-style options respectively. For example this command will render the graph using the dark theme of Echarts.

makefile-graph \
    --makefile examples/Makefile \
    --direction LR \
    --theme dark \
    --format echarts

For additional information and examples, please refer to the dnaeon/makefile-graph repo.

Tracking Kubernetes resource origin with kustomize

Fri, 15 Nov 2024 00:00:00 +0000

kustomize is a configuration management tool, which allows you to manage Kubernetes resources in a template-free way.

With kustomize you can define a base set of resources describing the desired state of an application, and then customize the configuration for a specific environment by leveraging overlays.

You can also use kustomize to generate secrets and configmaps.

The template-free approach of kustomize is something which I like, because it keeps the whole configuration more explicit, and does not introduce complexity via some magic template expansion. Another nice thing about kustomize is that it is already integrated into kubectl.

In order to get familiar with kustomize and the concepts it uses, please make sure to check the Introduction to Kustomize.

In this post we are going to see how we can track the origin of Kubernetes resources produced by kustomize, and then visualize them.

Let’s create a sample kustomization based on the kustomize helloWorld example.

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: hello-world

resources:
  - https://github.com/kubernetes-sigs/kustomize//examples/helloWorld/?timeout=120

Now, let’s build it.

kustomize build

The output from the command above is this.

apiVersion: v1
data:
  altGreeting: Good Morning!
  enableRisky: "false"
kind: ConfigMap
metadata:
  labels:
    app: hello
  name: the-map
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: hello
  name: the-service
spec:
  ports:
  - port: 8666
    protocol: TCP
    targetPort: 8080
  selector:
    app: hello
    deployment: hello
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hello
  name: the-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello
      deployment: hello
  template:
    metadata:
      labels:
        app: hello
        deployment: hello
    spec:
      containers:
      - command:
        - /hello
        - --port=8080
        - --enableRiskyFeature=$(ENABLE_RISKY)
        env:
        - name: ALT_GREETING
          valueFrom:
            configMapKeyRef:
              key: altGreeting
              name: the-map
        - name: ENABLE_RISKY
          valueFrom:
            configMapKeyRef:
              key: enableRisky
              name: the-map
        image: monopole/hello:1
        name: the-container
        ports:
        - containerPort: 8080

This fairly simple example shows how to create a new kustomization.yaml, which is based on a remote repo. The result of our kustomization is just three resources - a Deployment, a Service and a ConfigMap.

Where things may become complicated is when your kustomization.yaml builds on top of multiple bases, each of which provides a lot of resources.

In these situations understanding where a given resource comes from becomes a challenge. Fortunately, in kustomize we can enable originAnnotations, which helps in this regard.

When we enable originAnnotations in our kustomization.yaml kustomize will add annotations to each resource it produces, which describes where the resources originate from.

Let’s try it out.

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: hello-world

buildMetadata:
  - originAnnotations

resources:
  - https://github.com/kubernetes-sigs/kustomize//examples/helloWorld/?timeout=120

This time when we kustomize build we get this output instead. Notice the newly added config.kubernetes.io/origin annotation by kustomize.

apiVersion: v1
data:
  altGreeting: Good Morning!
  enableRisky: "false"
kind: ConfigMap
metadata:
  annotations:
    config.kubernetes.io/origin: |
      path: examples/helloWorld/configMap.yaml
      repo: https://github.com/kubernetes-sigs/kustomize
  labels:
    app: hello
  name: the-map
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    config.kubernetes.io/origin: |
      path: examples/helloWorld/service.yaml
      repo: https://github.com/kubernetes-sigs/kustomize
  labels:
    app: hello
  name: the-service
spec:
  ports:
  - port: 8666
    protocol: TCP
    targetPort: 8080
  selector:
    app: hello
    deployment: hello
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    config.kubernetes.io/origin: |
      path: examples/helloWorld/deployment.yaml
      repo: https://github.com/kubernetes-sigs/kustomize
  labels:
    app: hello
  name: the-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello
      deployment: hello
  template:
    metadata:
      labels:
        app: hello
        deployment: hello
    spec:
      containers:
      - command:
        - /hello
        - --port=8080
        - --enableRiskyFeature=$(ENABLE_RISKY)
        env:
        - name: ALT_GREETING
          valueFrom:
            configMapKeyRef:
              key: altGreeting
              name: the-map
        - name: ENABLE_RISKY
          valueFrom:
            configMapKeyRef:
              key: enableRisky
              name: the-map
        image: monopole/hello:1
        name: the-container
        ports:
        - containerPort: 8080

Now, this is much better as we can clearly see where a given resource came from. However, when having lots of resources it’s still a challenge to visually parse these annotations, but we can do better than that.

Some time ago I’ve worked on creating a tool which parses the origin annotations produced by kustomize and renders a graph of the resources and their origins.

You can find the code at the dnaeon/kustomize-dot repo.

dnaeon/kustomize-dot can operate in two modes - as a standalone CLI application, or as a KRM Function plugin.

In order to generate a graph of the Kubernetes resources and their origin when building a kustomization target we need to enable the originAnnotations build option in our kustomization.yaml file, just like we did for the helloWorld example.

The tool is flexible and supports filtering of resources, highlighting of resources or whole namespaces, setting graph layout direction, etc. This is useful when we want to get a more focused view of the resulting graph.

This here is the representation of all the resources for kube-prometheus operator.

The resulting graph is big, and could be confusing as well. In order to focus on specific resources (or namespaces) we can ask kustomize-dot to filter out the things we do (or don’t) need.

This command here for example will keep only resources from the default and kube-system namespaces.

kustomize-dot generate -f kube-prometheus.yaml \
    --keep-namespace default \
    --keep-namespace kube-system

The kube-prometheus.yaml file can be found in pkg/fixtures/kube-prometheus.yaml.

This is what the result looks like.

We can also apply some highlighting, if needed.

kustomize-dot generate -f kube-prometheus.yaml \
    --keep-namespace default \
    --keep-namespace kube-system \
    --highlight-namespace default=pink \
    --highlight-namespace kube-system=yellow

And here’s another example, which drops all ConfigMap resources, and keeps everything else.

kustomize-dot generate -f kube-prometheus.yaml \
    --keep-namespace monitoring \
    --drop-kind ConfigMap \
    --highlight-kind service=yellow \
    --highlight-kind servicemonitor=orange \
    --highlight-kind serviceaccount=lightgray \
    --highlight-kind deployment=magenta \
    --highlight-kind prometheusrule=lightgreen \
    --highlight-kind networkpolicy=cyan

The result looks like this.

For additional information and examples, please refer to the dnaeon/kustomize-dot repo.

Generating graphs from your Makefiles

Sun, 28 Apr 2024 00:00:00 +0000

Anyone who has done any kind of programming at some point comes across GNU Make and Makefile.

So what is make(1) and what are these Makefiles?

Instead of trying to explain this I’ll simply quote here the official GNU Make documentation.

GNU Make is a tool which controls the generation of executables and other non-source files of a program from the program’s source files.

Make gets its knowledge of how to build your program from a file called the makefile, which lists each of the non-source files and how to compute it from other files. When you write a program, you should write a makefile for it, so that it is possible to use Make to build and install the program.

If you haven’t heard or used make(1) before, I’d recommend that you go and check the GNU Make documentation for introduction and a tutorial on creating your first Makefile.

In the rest of this post I assume that you have already used make(1) and know what it does.

Okay, with that out of the way it’s time to focus on the topic of this post. Working with Makefiles can sometimes be challenging, especially if your project is large enough to contain lots of targets and lots of dependencies between each of them.

Navigating through these targets and their dependencies may not be an easy task. For example including other Makefiles while it helps keeping things logically separated introduces additional complexity when a person needs to understand the additional targets and additional dependencies from the included Makefile.

Okay, so what can we do about that? How to navigate through large Makefiles and get a better understanding of all the targets and their dependencies?

Internally, GNU Make (and other make implementations) are building a dependency graph, where the targets represent the vertices and the edges which connect them are represented by the target prerequisite.

Knowing this we can actually build a graph from our Makefiles and visualize them, which will help us navigate through the jungle of targets and their dependencies.

Okay, first things first. How do we get the graph representation which make(1) is using? This one is easy with GNU Make.

All you have to do in order to dump make’s internal database is call make(1) with these flags.

make --print-data-base \
     --no-builtin-rules \
     --no-builtin-variables \
     --dry-run \
     --always-make \
     --question

Calling this make(1) command inside a directory with a Makefile will dump the internal database of GNU Make.

Here’s a sample output from one of my Makefiles.

# GNU Make 4.4.1
# Built for x86_64-pc-linux-gnu
# Copyright (C) 1988-2023 Free Software Foundation, Inc.
# License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
# This is free software: you are free to change and redistribute it.
# There is NO WARRANTY, to the extent permitted by law.

# Make data base, printed on Sun Apr 28 16:38:33 2024

# Variables

# environment
TMUX_PANE = %0
# environment
GO111MODULE = on
# environment
QT_WAYLAND_RECONNECT = 1
# variable set hash-table stats:
# Load=109/1024=11%, Rehash=0, Collisions=6/141=4%

# Pattern-specific Variable Values

# No pattern-specific variable values.

# Directories

# . (device 65041, inode 3154786): 17 files, no impossibilities.

# 17 files, no impossibilities in 1 directories.

# Implicit Rules

# No implicit rules.

# Files

test_cover:
#  Phony target (prerequisite of .PHONY).
#  Implicit rule search has not been done.
#  File does not exist.
#  File has not been updated.
#  recipe to execute (from 'Makefile', line 8):
	go test -v -race -coverprofile=coverage.txt -covermode=atomic ./...

# Not a target:
Makefile:
#  Implicit rule search has been done.
#  File is secondary (prerequisite of .SECONDARY).
#  Last modified 2022-08-19 12:15:43.700764449
#  File has been updated.
#  Successfully updated.

# Not a target:
.DEFAULT:
#  Implicit rule search has not been done.
#  Modification time never checked.
#  File has not been updated.

test:
#  Phony target (prerequisite of .PHONY).
#  Implicit rule search has not been done.
#  File does not exist.
#  File has not been updated.
#  recipe to execute (from 'Makefile', line 5):
	go test -v -race ./...

get:
#  Phony target (prerequisite of .PHONY).
#  Implicit rule search has not been done.
#  Implicit/static pattern stem: ''
#  File does not exist.
#  File has been updated.
#  Needs to be updated (-q is set).
# automatic
# @ := get
# automatic
# * :=
# automatic
# < :=
# automatic
# + :=
# automatic
# % :=
# automatic
# ^ :=
# automatic
# ? :=
# automatic
# | :=
# variable set hash-table stats:
# Load=8/32=25%, Rehash=0, Collisions=1/11=9%
#  recipe to execute (from 'Makefile', line 2):
	go get -v -t -d ./...

# Not a target:
.SUFFIXES:
#  Implicit rule search has not been done.
#  Modification time never checked.
#  File has not been updated.

.PHONY: get test test_cover
#  Implicit rule search has not been done.
#  Modification time never checked.
#  File has not been updated.

# files hash-table stats:
# Load=7/1024=1%, Rehash=0, Collisions=0/23=0%
# VPATH Search Paths

# No 'vpath' search paths.

# No general ('VPATH' variable) search path.

# strcache buffers: 1 (0) / strings = 26 / storage = 226 B / avg = 8 B
# current buf: size = 8162 B / used = 226 B / count = 26 / avg = 8 B

# strcache performance: lookups = 36 / hit rate = 27%
# hash-table stats:
# Load=26/8192=0%, Rehash=0, Collisions=0/36=0%
# Finished Make data base on Sun Apr 28 16:38:33 2024

If you look closely you will see that this output contains multiple sections:

A header describing the version of the make tool
A section containing the environment variables
Pattern-specific Variable Values
No pattern-specific variable values
etc.

Within the Files section we can also see our targets and this is exactly what we need and what we are going to be parsing.

The good news is that I’ve already implemented the parser and you can install it as a CLI tool, or use it as a Go package in case you want to implement different graph representations.

You can find the code in the dnaeon/makefile-graph and here’s how you can install it.

go install github.com/dnaeon/makefile-graph/cmd/makefile-graph@latest

If you want to use the parser and do whatever you want to do with the parsed graph you can install the parser package.

go get -v github.com/dnaeon/makefile-graph/pkg/parser

Here are some screenshots of makefile-graph in action. The commands below are using the following Makefile, from which we will be generating the graph representation in dot format.

edit : main.o kbd.o command.o display.o \
       insert.o search.o files.o utils.o
	cc -o edit main.o kbd.o command.o display.o \
		   insert.o search.o files.o utils.o

main.o : main.c defs.h
	cc -c main.c
kbd.o : kbd.c defs.h command.h
	cc -c kbd.c
command.o : command.c defs.h command.h
	cc -c command.c
display.o : display.c defs.h buffer.h
	cc -c display.c
insert.o : insert.c defs.h buffer.h
	cc -c insert.c
search.o : search.c defs.h buffer.h
	cc -c search.c
files.o : files.c defs.h buffer.h command.h
	cc -c files.c
utils.o : utils.c defs.h
	cc -c utils.c
clean :
	rm edit main.o kbd.o command.o display.o \
	   insert.o search.o files.o utils.o

Here’s how to render the graph for all targets from our sample Makefile.

makefile-graph --makefile examples/Makefile --direction TB | dot -Tsvg -o graph.svg

And this is what it looks like.

In large enough Makefiles the generated graph may not be easy to understand as well, and for that reason makefile-graph supports highlighting specific targets and their dependencies.

For example this command will highlight the files.o target and it’s dependencies.

makefile-graph \
    --makefile examples/Makefile \
    --direction TB \
    --target files.o \
    --highlight \
    --highlight-color lightgreen

The results look like this.

And if that is not good enough, we can always focus on a specific target and it’s dependencies only.

makefile-graph \
    --makefile examples/Makefile \
    --direction TB \
    --target files.o \
    --related-only

The results look like this.

For additional information and examples, please refer to the dnaeon/makefile-graph repo.