spacemacs/doc/CI_PLUMBING.org

Continuous Integration

• Description
• Overview
  • TLDR
  • Current stack:
    • CircleCI
    • GitHub Actions
    • Docker
    • Clojure
• Workflows (groups of CI jobs)
  • Pull request jobs
    • Emacs Lisp Tests
    • Documentation validation
    • PR validation
  • Branch updates (runs on merge)
    • Emacs Lisp Tests
    • Project files updates
      • General concepts
      • Built-in updates
      • Documentation updates
      • Web site updates
  • Scheduled jobs:
    • General concepts
    • Built-in files updates
• Potential improvements (PR ideas)
• Side notes
  • We used to have TravisCI (3 CI providers at the same time)
  • CircleCI setup config and cron jobs

Description

This file explains how our continuous integration operates and what problems it solves. It is works in progress.

Overview

TLDR

Spacemacs is big - the active maintainers team is small. The more we can automate - the better. We use  CircleCI, GitHub Actions and Docker to test PRs, update/fixi documentation and exporting it to https://develop.spacemacs.org/.

Most of the code is just a bunch of bash/ELisp scripts and yml files, but some of the documentation tools are written in Clojure. Check out CircleCI and GitHub Actions directories, the code is pretty much self-explanatory but will be examined in depth below.

Current stack:

Wait, what? Why Clojure, why 2 CI providers? I knew you would ask this question, dear reader, so here's my rationale:

CircleCI

It has a cool set of features and a generous quota for open source projects. But most importantly, unlike GitHub Actions, there is a straight forward way to cache build dependencies between runs and using it in tandem with GH Actions provides us with even more concurrency. It means that PR authors have to wait less time for feed back. This is crucial since we have a lot of test and platforms to cover. Also, CircleCI can run jobs on user provided Docker images that it caches, so we do not hit the DockerHub pull quota. On the downside, the CircleCI configuration file can be pretty involved, has unexpected limitations that can leave you puzzled for quite a while.

GitHub Actions

Oh man, that's good. It is clear that GH team had the benefit of hindsight when developed their CI platform. And it runs rely fast (at least for now). Maybe one day we'll fully switch to Actions. The biggest concern here is the vendor lock-in since all the good stuff is highly specific. While CircleCI allows you to run a job locally for free. And run whole CI on your own hardware with "strings attached".

Docker

Having a stable pre-build environment for jobs reduces headaches and improves set up time. Duh! Also DockerHub used to be a cool place to store and build huge images for free, but now it has all sorts of quotas + RAM is pretty limited for memory hungry JVM builds (((foreshadowing))).

Clojure

Besides the obvious fact that Rich Hickey's talks are the best. Before we started with automation, Spacemacs already had a huge set of documentation files that couldn't be fixed by a bunch of regular expressions wrapped into bash/ELisp code. The options were to either fix all README.org files by hand and keep fixing them forever, since contributors often forget to format stuff properly and nagging them constantly both wastes PR reviewer time and makes the contributor less likely to stick. Or go all-in and create a system that can extract data out of documentation files and rebuild them from scratch. Clojure designed to push data around and it has specs that can be used to validate files, generate test data and constructors for org-mode elements. The code is compiled to native-image so pretty much all of the JVM drawbacks are mitigated, for the particular use case anyway.

Workflows (groups of CI jobs)

Pull request jobs

Emacs Lisp Tests

Code tests are handled by GitHub Actions. See the workflow file. The stages are:

1. Emacs installation with purcell/setup-emacs - for UNIX and jcs090218/setup-emacs-windows for Windows. The step is configured by the job matrix. With two keys os and emacs_version. CI runs test for every possible combination. The stage ends up seriously bloated with repetition since the actions sometimes fail (especially for MacOS) so I added 3 sets of retires for the both actions. Currently GitHub doesn't provide a better way to implement this.
2. Checkout - clones the repo.
3. Installation of a local ELPA mirror with packages used be the tests. The archive is build daily in JAremko/testelpa-develop repository and configured by .spacemacs files used in test. The mirror is set as a top priority package repository via Spacemacs lock file this way we actually install the packages (it is important to test that the system works) and if some packages are missing (for example, the mirror can be outdated) then they will be installed from a remote repository.
4. Run the tests! CI run core, base and layer tests sequentially because heaving 20+ CI results in PRs makes people ignore them. And this way they start faster since we cut on setup time. But the tests have to always clean after themselves to avoid affecting the next ones.

TODO Documentation validation

TODO PR validation

Branch updates (runs on merge)

Emacs Lisp Tests

See Emacs Lisp Tests it is the same.

TODO Project files updates

TODO General concepts

TODO Built-in updates

TODO Documentation updates

TODO Web site updates

Scheduled jobs:

TODO General concepts

TODO Built-in files updates

Potential improvements (PR ideas)

• CircleCI config generation stage can test if a PR changes any .org file and schedule documentation testing job only if it does.
• PR validation job can be moved to CircleCI config generation stage. If it isn't valid all CircleCI jobs can be skipped.
• Web site repo becomes too heavy and PR diffs are meaningless. Removing update dates that are embedded into each exported HTML files would reduce the patch size drastically.
• Figure out how to retry installation of Emacs for EmacsLisp tests in more concise manner.
• Emacs Lisp tests step that runs the test isn't DRY.
• Emacs Install retries can use some delay between the attempts since it is likely that a failed upstream repo will fail again if you don't give it any time to recover/change state. But it shouldn't add delay to runs without failures since they vastly outnumber failed ones.
• See if we actually properly clean all they side effects between running EmacsLisp tests.

Side notes

We used to have TravisCI (3 CI providers at the same time)

We ran long running jobs there but ended up dropping the CI since TravisCI doesn't allow collaborators to read/set environment variables anymore, the could be in some kind of trouble or maybe not. Anyway, when TravisCI stopped running jobs on their old domain (as a part of the migration from https://travis-ci.org/ to https://www.travis-ci.com/) I decided to use it as an opportunity to have fewer kinds of configs. Still, it's good environment for building heavy (both in build time and RAM) Docker images.

CircleCI setup config and cron jobs

• Currently configs with setup step don't run cron jobs.
• We have setup config because environment variables aren't accessible at the top level of config files. But we need IS_BRANCH_UDATE environment variable to figure out if CI runs on PR or branch update. So config generation step bakes it into the config that CircleCI will use.