Writing a Guix service from scratch, as a beginner

The documentation defines a service as "something that extends the functionality of the operating system".

This definition is quite vague, but that's because a service can have a wide variety of functionalities. The most obvious type of a service is a daemon running a process in the background (our case), but it's not the only type of service.

As we will see, a service can be a process which is run once, such as creating accounts or copying files to the store. It can be a recurring process, such as a cron job.

In a way, it is a primitive to configure our system in a declarative manner.

One interesting aspect of services, is that they are designed from the ground up to be extensible and composable. A service can extends one or more services, and can itself be extended by one or more services. We will dive a deeper on that subject further below; I find this architecture fantastic.

You can read more about services in the documentation. If you still don't quite understand the concept of services,, it's completely normal. It didn't click for me until I started writing this Caddy service.

So if you don't fully grasp what a service is yet, follow along, it will make more sense as we progress.

Being curious and eager to really assimilate how services extensions work, I tried find out if we can guess the shape of the accounts by looking at the account-service-type service, defined in shadow.scm:547.

This is how the account-service-type is defined. We should be familiar with the <service-type> record structure.

(define account-service-type
  (service-type (name 'account)
    ;; Concatenate <user-account>, <user-group>, and skeleton
    ;; lists.
    (compose concatenate)
    (extend append)
    (extensions
      (list (service-extension activation-service-type
              account-activation)
        (service-extension shepherd-root-service-type
          account-shepherd-service)
        ;; Have 'user-processes' depend on 'user-homes' so that
        ;; daemons start after their home directory has been
        ;; created.
        (service-extension user-processes-service-type
          (const '(user-homes)))
        (service-extension etc-service-type
          etc-files)))
    (default-value '())
    (description
      "Ensure the specified user accounts and groups exist, as well
as each account home directory.")))

The first thing we can see, is that the default-value is an empty list, which means that the account-service-type itself is expecting a list as a value. But a list of what?

In this case, the comments give us the answer: a list of <user-account> and <user-group>. If it was not for the comments, you could look at the account-activation.

Before moving on, let's talk about compose and extend fields. This is how they are defined in the documentation

> compose

This is the procedure to compose the list of extensions to services of this type.

> extend

This procedure defines how the value of the service is extended with the composition of the extensions.

This is a bit vague and left me a bit scratching my head, but what it means is that:

1. Compose defines how to the service composes multiple services extending this service-type.

   For example,if we have three services listing accounts-service-type as an extension, how do we handle each value? Do we only take the first? The last?

   Obviously, if three services are defining users by extending the account-service-type, we want to create all those users. Therefore, we want to concatenate the values provided by the extending services.

   We can imagine other services behaving differently. For example, let's imagine a service whose role is to manage a port. Let's call it open-port-22-service-type. Let's also imagine that two services extend this service: one wants to open the port, the other close it.

   What shall we do? Open it, or close it? We have conflicting values.

   Maybe we want to err on the caution side and define open-port-22-service-type to prioritise 'close' values if it receives contradicting values.

   That's what the compose field is for. It defines how to compose multiple extending service values.

2. Extend defines how the composition from the compose field interact with the service's value itself.

It is, in a way, quite similar to compose

Let's imagine a service-type which displays a welcome message when the computer boots, called welcome-message-service-type. If a service extends it with a different message, we might want to give priority to the extending service value.

In this case, we would use extend to instruct the welcome-message-service-type to replace its own value with the composed values from the extending services.

Note that compose and extend accept procedures which takes the service itself as an argument. It gives the user a lot of freedom and power when defining the rules.

Let's finish with the user-account-service-type as a final example.

In the case of users, we want to:

1. compose: concatenate the values to create all the users and groups received.
2. extend: append to the service value. so we create all the users and groups received AND the ones specified by the user-account-service-type service itself (remember that we can instantiate the service with a value).

Zooming back to the architecture, we can see that services are all built upon each other. A service, when installed, will modify the services listed in its extensions field. Those will in turn modify the service-type of their own extensions.

This "chain reaction" will continue until all services are collected in the system-service-type, which is the root of all services. Under the hood, we end up with a collection of all our services in one, large, single structure. I assume that this is conglomerate of services is then handled by Guix when configuring the system.

Ok, we now know the account-service-type:

1. Accept a list of <user-account> and <user-group> records
2. Concatenate any values received as per its compose rule.
3. Append the (concatenated) received values to its own value, as per the extend rule.

So, by either following in my 'deep dive', reading the documentation or looking up examples in the code base, we found out that our group and user are defined as a <user-account> and <user-group> respectively.

Both records can be found in accounts.scm at line 71 and 90 respectively.

We define our accounts like so:

(define %caddy-accounts
  (list (user-account
          (name "caddy")
          (group "caddy")
          (comment "Run the caddy daemon")
          (system? #t)) ;;<-- the caddy user is a system user, not a real user
    (user-group
      (name "caddy")
      (system? #t))))

Now we can add them the account-service-type extension.

The extensions procedure accepts a list of <service-extension>. A service-extension is a record that accept two parameters.

The first is the name of the target service-type . This is account-service-type in our case. The second parameter to a service-extension is a procedure that takes the service itself (the current caddy service) and returns a list of object to extend the target service (see documentation).

The second parameter seems complicated, but it's simply a procedure that returns the "value" for the extension. In our case, a list of user and group account. You can imagine that using this second argument, we could create the user dynamically based on the value of our caddy-service-type if we wanted make our service configurable. That's how configurable services are structured.

Let's define our service-extension for the account-service-type and add it to the extension field.

(define caddy-service-type
  (service-type
    (name 'caddy)
    (extensions (list (service-extension account-service-type
                        (const %caddy-accounts))))
    (description "Caddy is an extensible server platform that uses TLS by default.")))

We used const here, which is the syntactic sugar for (lambda(_)(%caddy-accounts)).

Alright, time to finally test it out!

Run sudo guix system reconfigure config.scm to apply our changes.

Let's check the presence of the caddy user by running sudo id caddy.

> sudo id caddy
# uid=982(caddy) gid=976(caddy) groups=976(caddy)

Nice! Our caddy user and group were created!

What we have learned so far:

• Services are 'things' that modify the system.
• Services can be extended and can extend other services.
• Services define how they can be extended, via the compose and extend fields.
• Services define the service they extend, via the extensions field.
• All services are eventually "collected" in the system-service-type, the root service.
• We have learned to how to extend the account-service-type to create a user and a group for our service.
• We have also learned to dig around the source code to learn more about services.

Now let's see how we can write the config, the Caddy file, into /etc/caddy.

I won't go to deep into the concept of G-exps; there is a lot to cover and it merits its own article. We will keep a high level overview.

To understand what G-expression are, you need to know that Guix runs in two environments.

When building packages, the guix daemon creates something akin to containers. They aren't quite containers, but the concept is close enough for our understanding.

To ensure reproducibility, each package is built in its own "container", which only includes the required inputs. For this to happen, part of the code in package definition needs to be staged for later execution, inside that "container". G-expressions is how we staging code to be executed at build time.

The two environments of Guix are often referred as:

1. The "host" environment, in which we define packages and configurations (what we are doing here).
2. The "build" environment, in which the Guix daemon is performing the actual build actions inside the "container".

G-expressions are expressed with the #~ macro, such as #~( ;; this is a g-expression ;; ). Whenever you see the macro #~, it means that the code inside the bracket will be executed later, inside the build "container".

Inside a G-exp, we can evaluate some expressions using the #$ macro, or ungexp.

For example, let's have a look at this G-expression:

#~(begin
    (mkdir #$output)
    (mkdir "/another/folder/path"))

In this instance, when the code is evaluated #$output will be replaced (or ungexp) and the expression will be:

#~(begin
    (mkdir "/the/output/dir")
    (mkdir "/another/folder/path"))

In concept, it's similar to the quasi-quote and comma in Scheme.

As the G-expression is evaluated in a different environment, it will not have automatically access to the modules imported in the current environment. In order to give the G-expression access to modules, we need to import them in the "build" environment in which the G-expression will be evaluated.

To do so, we use the with-imported-modules syntax like so:

(with-imported-modules '((guix build utils))
  #~(begin
      (use-modules (guix build utils))
      (mkdir-p "/the/output/dir") ;;mkdir-p is a utility found in guix build utils.
      (mkdir-p "/another/folder/path")))

In the example above, mkdir-p is a procedure from in guix/build/utils.scm. The with-imported-modules provides the modules to the G-expression environment. We can then use the standard use-modules syntax as we would in a normal Scheme environment.

There is a lot more to G-expressions, but it merits its own article so I won't go any further.

This is all we need to know at this stage:

1. G-expression is code staged to be executed in the "build" phase of Guix.
2. G-expression are written with the #~ macro.
3. We can ungexp part of the G-expression with the #$ macro.
4. We can import modules into the G-expression with the with-imported-modules syntax.

If you want to dig deeper, some excellent resources are:

• G-Expressions in the documentation
• FOSDEM Talk about G-exps (video)
• Code Staging in GNU Guix (paper, but very accessible)

With this newly acquired knowledge, let's think about what we are trying to achieve here.

We want to:

1. Create the /etc/caddy directory.
2. Copy the configuration file at /etc/caddy/caddyfile.
3. Ensure that the configuration file is read-only for the caddy user.

We need to express those three instructions in a G-expression that we will pass as an argument to the service-extension for the activation-service-type.

This is how we can do so:

(define %caddyfile (plain-file "CaddyFile" ":8080 { respond \"Hello world!\"}"))
;; Generate a plain caddyfile configuration.
;; Refer to https://caddyserver.com/docs/caddyfile-tutorial for more information

(define (%caddy-activation _)
  (with-imported-modules '((guix build utils)) ;; we need some utilities, such as `mkdir-p
    #~(begin
        (use-modules (guix build utils))
        (mkdir-p "/etc/caddy") ;; 1. Create the directory
        (copy-file #$%caddyfile "/etc/caddy/caddyfile") ;; 2. Copy the configuration file at the right location (notice the ungexp!).
        (chown "/etc/caddy/caddyfile"
          (passwd:uid (getpwnam "root")) ;; 3. Change the ownership to root:caddy.
          (group:gid (getgrnam "caddy")))
        (chmod "/etc/caddy/caddyfile" 750)))) ;; 4. Set permissions.

You notice that the %caddy-activation procedure accepts an argument. As discussed earlier, it can receive the current service as an argument, allowing you to dynamically modify the output. For example, you could imagine giving the caddy-service-type the ability to configure the caddy username.

Ok, let's try it out:

(define caddy-service-type
  (service-type
    (name 'caddy)
    (description "Run Caddy, the simple proxy server.")
    (extensions
      (list (service-extension account-service-type (const %caddy-accounts))
        (service-extension activation-service-type %caddy-activation) ;; <- Adding the extension here.
        (default-value nil)))

Now run sudo guix system reconfigure config.scm.

Inspect the content of /etc/caddy/caddyfile:

> sudo cat /etc/caddy/caddyfile
# :8080 { respond "Hello world!"}

Great, we now have our user and our configuration file. We can test out our service by starting caddy manually.

sudo caddy -c /etc/caddy/caddyfile" # you need sudo here, we will resolve this in part 4.

In a second terminal, run curl localhost:8080. You should obtain the response "Hello world!".

We now have successfully created a service to run caddy!

This concludes the configuration of our service. Our service:

1. creates the daemon user and group.
2. generate the caddy configuration file
3. copy it to the right location.

The last part is about making it run as a daemon managed by Shepherd in the background, so we don't need to start it manually.

On most linux distributions, including Guix, any port numbers below 1024 are special: standard users are not allowed to bind process to them.

Since our caddy user is not a privileged user, it won't be able to run the Caddy server on port 80 (http) and 443 (https). We need to allow the Caddy process to be run on those ports.

The way to go about this in Guix is to use privileged programs.

To make Caddy a privilived program, we append it to the %default-privileged-programs and assign it the necessary extra capabilities.

(privileged-programs
  (append (list (privileged-program
                  (program (file-append caddy "/bin/caddy"))
                  (capabilities "cap_net_bind_service=ep")))
    %default-privileged-programs))

If you're extra curious, you can take a peek at the list of privileged-programs in system.scm:1267. It contains a lot of well known utility requiring extra privileges, such as sudo, su, passwd, ping or mount.

Here, we are adding the capability CAP_{NETBINDSERVICE} as described in man 7 capabilities as Bind a socket to Internet domain privileged ports (port numbers less than 1024).

The =ep means that capability is *e*ffective and *p*ermitted, essentially giving it the capability there and then.

An important thing to note: when we make a program privileged, it is relocated outside of the store. The store cannot contain privileged for security concerns, you can read more about it here. The privileged programs are located in /run/privileged.

Our 'privilged'Caddy program is located at /run/privileged/bin/caddy.

The next step is to configure Shepherd. To do so, we need to extend the shepherd-root-service-type with a shepherd-service record.

In this case, I won't go into too much detail. The Shepherd service documentation is very helpful and detailed, I highly recommend you to read it. In particular the page about the page about service constructor and destructors explains the role of make-forkexec-constructor and make-kill-desstructor, both doing a lot of the heavy lifting here.

Instead, I will comment the intentions on the extension itself:

(define (%caddy-shepherd-service _)                                 ;; We ignore the config argument as we will not use it.
  (list (shepherd-service
          (provision '(caddy))                                      ;; this is purely an arbitrary naming handle.
          (documentation "Run the caddy daemon")                    ;; self explanatory
          (requirement '(networking user-processes))                ;; We instruct shepherd to start the caddy daemon after the networking and user modules.
          (start #~(make-forkexec-constructor                       ;; the command to start the process: note the use of g-exps! Refer to the doc for an exhaustive list of options
                     (list "/run/privileged/bin/caddy"               ;; Note the location of our privileged version of caddy. The "run" commands and its arguments are Caddy related; refer to the caddy documentation
                       "run" "--config" "/etc/caddy/caddyfile"
                       "--adapter" "caddyfile")
                     #:user "caddy"                                  ;; The user and group in charge of the daemon
                     #:group "caddy"
                     #:log-file "/var/log/caddy.log"
                     #:environment-variables                         ;; Required for the logs
                     '("HOME=/var/lib/caddy")))
          (stop #~(make-kill-destructor)))))                        ;; The destructor, called when the process is killed. Refer to the documentation

A few point worth mentioning:

• 'user-processes is a requirement for most shepherd-service as mentioned in the documentation. We need the file systems for the user to be mounted before starting Caddy.
• 'networking is also set as a requirement, we want the network stack up before we start caddy. I found this while looking at how the exim service was written. As far as I can tell, these aren't documented. You will need to look directly in the source code.
• I don't quite recall why I set up the environment variable, but I believe it was a requirement for the logs. The service would otherwise it would complain. It is mentionned in the Caddy documentation when using systemctl.

If you wish to change the Caddyfile, you can modify our %caddyfile and reconfigure the system. Alternatively, for quick debugging purpose, you can modify the file directly at /etc/caddy/caddyfile and restart the caddy server with sudo herd restart caddy.