automate - the better. We use [[https://circleci.com/][CircleCI]], [[https://github.com/features/actions][GitHub Actions]] and [[https://www.docker.com/][Docker]] to test PRs,
update/fixi documentation and exporting it to [[https://develop.spacemacs.org/]].
Most of the code is just a bunch of bash/ELisp scripts and yml files, but
some of the documentation tools are written in Clojure.
Check out [[https://github.com/syl20bnr/spacemacs/tree/develop/.circleci][CircleCI]] and [[https://github.com/syl20bnr/spacemacs/tree/develop/.github/workflows][GitHub Actions]] directories, the code is pretty much
self-explanatory but will be examined in depth below.
Code tests are handled by GitHub Actions. See the [[https://github.com/syl20bnr/spacemacs/blob/develop/.github/workflows/elisp_test.yml][workflow]] file.
The stages are:
1. Emacs installation with [[https://github.com/purcell/setup-emacs][purcell/setup-emacs]] - for UNIX and
[[https://github.com/jcs090218/setup-emacs-windows][jcs090218/setup-emacs-windows]] for Windows. The step is configured
by the job matrix. With two keys =os= and =emacs_version=. CI runs test
for every possible combination. The stage ends up seriously bloated with
repetition since the actions sometimes fail (especially for MacOS)
so I added 3 sets of retires for the both actions. Currently GitHub
[[https://github.community/t/how-to-retry-a-failed-step-in-github-actions-workflow/125880][doesn't provide a better way to implement this]].
2. Checkout - clones the repo.
3. Installation of a local ELPA mirror with packages used be the tests.
The archive is build daily in [[https://github.com/JAremko/testelpa-develop][JAremko/testelpa-develop]] repository and
configured by .spacemacs files used in test. The mirror is set as a top
priority package repository via [[https://github.com/syl20bnr/spacemacs/blob/develop/.github/workflows/scripts/dot_lock.el][Spacemacs lock file]] this way we actually
install the packages (it is important to test that the system works) and
if some packages are missing (for example, the mirror can be outdated)
then they will be installed from a remote repository.
4. Run the tests! CI run core, base and layer tests sequentially because
heaving 20+ CI results in PRs makes people ignore them. And this way
they start faster since we cut on setup time. But the tests have to
=always= clean after themselves to avoid affecting the next ones.
** We used to have TravisCI (3 CI providers at the same time)
We ran long running jobs there but ended up dropping the CI since TravisCI
doesn't allow collaborators to read/set environment variables anymore,
[[https://pbs.twimg.com/media/Eoq3OnWW4AIy7ih?format=jpg&name=large][the could be in some kind of trouble]] or [[https://blog.travis-ci.com/oss-announcement][maybe not]]. Anyway, when TravisCI
stopped running jobs on their old domain (as a part of the migration from
- Currently configs with setup step [[https://discuss.circleci.com/t/setup-workflow-and-scheduled-workflow-in-the-same-configuration/39932/6][don't run cron jobs]].
- We have setup config because environment variables aren't accessible at the