spacemacs/doc/LAYERS.org

#+TITLE: Configuration layers development

* Table of Contents                     :TOC_5_gh:noexport:
- [[#introduction][Introduction]]
- [[#nomenclature][Nomenclature]]
- [[#the-emacs-loading-process][The Emacs loading process]]
  - [[#emacs-lisp-files][Emacs Lisp files]]
    - [[#loading-a-file][Loading a file]]
  - [[#features][Features]]
    - [[#the-load-path][The load path]]
  - [[#auto-loading][Auto-loading]]
  - [[#eval-after-load][Eval after load]]
  - [[#use-package][Use-package]]
- [[#anatomy-of-a-layer][Anatomy of a layer]]
  - [[#layersel][layers.el]]
  - [[#packagesel][packages.el]]
  - [[#funcsel][funcs.el]]
  - [[#configel][config.el]]
  - [[#keybindingsel][keybindings.el]]
- [[#the-spacemacs-loading-process][The Spacemacs loading process]]
- [[#case-study-auto-completion][Case study: auto-completion]]
- [[#layer-tips-and-tricks][Layer tips and tricks]]
  - [[#cross-dependencies][Cross-dependencies]]
  - [[#shadowing][Shadowing]]
  - [[#use-package-init-and-config][Use-package init and config]]
  - [[#use-package-hooks][Use-package hooks]]
  - [[#best-practices][Best practices]]
    - [[#package-ownership][Package ownership]]
    - [[#localize-your-configuration][Localize your configuration]]
    - [[#load-ordering][Load ordering]]
    - [[#no-require][No require]]
    - [[#auto-load-everything][Auto-load everything]]

* Introduction
This document is intended as a tutorial for users who are interested in writing
their first configuration layer, whether for private use or for contributing
upstream. It should help clear up some confusion regarding how layers work and
how Spacemacs (and Emacs) loads packages. For an overview of configuration
layers with descriptions see [[https://develop.spacemacs.org/layers/LAYERS.html][Spacemacs layers list]].

* Nomenclature
Layers and packages. What gives?
- A set of Emacs Lisp files that, taken together, provide some
  feature. Packages may be available on a package repository, such as ELPA or
  MELPA or on a third-party service provider (such as github) or even
  locally on the disk.
- A collected unit of configuration that can be enabled (or disabled)
  in Spacemacs. A layer typically brings together one or more packages, as
  well as the glue configuration code required to make them play well with
  each other and Spacemacs in general.

Before writing a layer, it is helpful to consider what you are trying to
achieve. Is there a package that provides the functionality you are after, and
you want to integrate it in Spacemacs? If yes, you should write a layer. Are you
trying to implement a new feature that would be useful for the Emacs community
at large? In that case, consider whether it wouldn't be more appropriate to
write a package first, and then a layer that uses your package.

* The Emacs loading process
To understand how to best implement a layer, we have to investigate how Emacs
loads code.

** Emacs Lisp files
Emacs Lisp files contain code that can be evaluated. When evaluated, the
functions, macros and modes defined in that file become available to the current
Emacs session. Henceforth, this will be termed as /loading/ a file.

One major problem is to ensure that all the correct files are loaded, and in the
proper order. Another issue is to ensure that not too many files are loaded
immediately. This causes startup to take too long. Instead, we want to make sure
that files are loaded only as needed, and not all at once.

How is this done in Emacs, and how is it done in Spacemacs?

*** Loading a file
The simplest way to load a file is to call =load-file=.

#+BEGIN_SRC emacs-lisp
  (load-file "~/elisp/foo.el")
#+END_SRC

This is as primitive as it comes. The path must be exact, and it does not have
to be in the Emacs load path (we'll get to that later). It will not look for a
byte-compiled =.elc= file. It will simply load exactly what you tell it to.

** Features
A better way to load what you need is to use /features/. A feature is a symbol
that typically has the same name as the file it resides in. Let us say you have
the following contents in a file called =my-feature.el=.

#+BEGIN_SRC emacs-lisp
  ;; Your code goes here ...

  (provide 'my-feature)
#+END_SRC

To have Emacs load this file, call =require=, as such:

#+BEGIN_SRC emacs-lisp
  (require 'my-feature)
#+END_SRC

This checks whether the feature =my-feature= has already been loaded. If not, it
looks for a file called =my-feature.el=, =my-feature.elc= or some such. If it
finds such a file, it will load it. When the call to =provide= is evaluated, the
feature is added to the list of loaded features, so that subsequent calls to
=require= will do nothing.

This will cause an error if no such file can be found.

The file =my-feature.el= may very well contain other calls to =require=, and in
fact this is quite a common way to ensure that dependencies are loaded before
your code runs.

Package authors should use this technique to make sure that dependencies are
loaded before their code runs.

*** The load path
When loaded using =require=, Emacs looks for files in its /load path/. This is
nothing more than a list of paths where elisp files can be found, and you can
inspect it through ~SPC h d v load-path~ in Spacemacs. To add to the load path,
simply add to this list, e.g.

#+BEGIN_SRC emacs-lisp
  (add-to-list 'load-path "/some/path/")
#+END_SRC

** Auto-loading
Calling =require= is nothing more than a glorified way of calling =load-file=.
It solves the problem of ensuring that files are loaded in the correct order,
and to some degree it solved the problem of where to find the files on disk but
a long list of calls to =require= at startup would still cause Emacs to take
forever to load.

Emacs uses auto-loading to solve this problem. When a function is registered as
auto-loading, an "empty" definition is provided. When that function is called,
the file that provides the function is immediately loaded (along with all its
required features). Finally, the "empty" function is substituted with the real
one and called normally. The end user will see only a slight delay when first
calling the function, while subsequent calls to that function (or any other
function loaded as part of the same procedure) will be as quick as normal.

To register a function as auto-loadable, we call =autoload=:

#+BEGIN_SRC emacs-lisp
  (autoload 'some-function "some-file")
#+END_SRC

This instructs Emacs that whenever =some-function= is called, load
=some-file.el= first, and then proceed.

After evaluating the above code, you can try to inspect =some-function= by doing
~SPC h d f some-function~. It will say it's an auto-loaded function, and that
nothing else is known about it until it is loaded. The call to =autoload= can
optionally include more information, such as a doc-string, whether the function
can be called interactively, and so on. This provides more information to the
end-user without her having to actually load the file first.

Open your =elpa= directory, go to =helm= and look at the file
=helm-autoloads.el=. This provides all the auto-loads for all the files in Helm.
However, this file is not written by hand. Instead, it is automatically
generated from "magic" comments in the source code of Helm. They look like this:

#+BEGIN_SRC emacs-lisp
  ;;;###autoload
  (defun my-function ()
    ;; Source code...
    )
#+END_SRC

The magic comment =;;;###autoload= instructs Emacs that the following definition
should be auto-loaded. This automatically generates an appropriate call to
=autoload=.

Things that can be auto-loaded generally involve anything "definable", such as
functions, macros, major or minor modes, groups, classes, and so on.

Magic comments also work on other things, such as variable definitions
(=defvar=), but in that case, the definition is just copied verbatim into the
auto-loading file. For example, this code will load Helm on startup, long before
your file is actually evaluated, probably not what was intended:

#+BEGIN_SRC emacs-lisp
  ;;;###autoload
  (require 'helm)
#+END_SRC

It is the responsibility of the package authors to ensure that their package can
be appropriately auto-loaded, and most packages do this quite well.

Spacemacs makes thorough use of auto-loading. Almost everything in Spacemacs is
loaded when needed instead of right away.

** Eval after load
Often, we will want to configure packages after loading them. We may want to set
some variables or call some functions. This is trivial with =require=, because
it loads immediately, but it can be tricky with autoloading, because the
configuration code must also be deferred.

Emacs offers =with-eval-after-load= for this purpose. It can be used like this:

#+BEGIN_SRC emacs-lisp
  (with-eval-after-load 'helm
       ;; Code
       )
#+END_SRC

This arranges for the relevant code to be executed after Helm is loaded (using
either =require= or an autoload), or if Helm is already loaded, the code is
executed immediately.

Since =with-eval-after-load= is a macro and not a function, its argument does
not have to be quoted.

** Use-package
For /end users/ who are trying to put together an efficient Emacs configuration,
there is a very useful /package/ called =use-package= that provides a macro
which is /also/ called =use-package= which does a very good job of streamlining
the whole process of loading packages.

The aspiring layer author is recommended to have a look at the =use-package=
[[https://github.com/jwiegley/use-package][documentation]]. Some examples follow.

#+BEGIN_SRC emacs-lisp
  (use-package helm)
#+END_SRC

This simply loads Helm. It is essentially equivalent to =(require 'helm)=.

#+BEGIN_SRC emacs-lisp
  (use-package helm
    :defer t)
#+END_SRC

This defers the loading of Helm using the auto-load facility and the auto-load
commands provided by the Helm source code. It is, in fact, a no-op.

#+BEGIN_SRC emacs-lisp
  (use-package helm
    :defer t
    :init
    ;; Code to execute before Helm is loaded
    :config
    ;; Code to execute after Helm is loaded
    )
#+END_SRC

This form includes code to execute before and after Helm is loaded. The =:init=
section can be executed immediately, but since Helm is deferred, the =:config=
section is not executed until after loading, if ever. It is essentially
equivalent to simply running the =:init= block, and then adding the =:config=
block in an =with-eval-after-load=.

#+BEGIN_SRC emacs-lisp
  (use-package helm
    :commands (helm-find-files helm-M-x))
#+END_SRC

This creates auto-load references for additional commands, if you find that the
package author has been slacking.

#+BEGIN_SRC emacs-lisp
  (use-package ruby-mode
    :mode "\\.rb\\'")
#+END_SRC

For packages that provide major modes, you can associate file extensions to that
mode by using the =:mode= keyword. This adds an entry to =auto-mode-alist= and
an auto-load for =ruby-mode=. Typically this is not required, as =ruby-mode=
should already be auto-loadable, and the package should associate Ruby files
with itself already.

Use-package supports heaps of useful keywords. Look at the [[https://github.com/jwiegley/use-package][documentation]] for
more.

* Anatomy of a layer
A layer is simply a folder somewhere in Spacemacs's layer search path that
usually contains these files (listed in loading order).
- declare additional layers
- the packages list and configuration
- all functions used in the layer should be declared here
- layer specific configuration
- general key bindings

Additionally, for each local package (see the next section), there should be a
folder =<layer>/local/<package>/= containing the source code for that package.
Before initializing that package, Spacemacs will add this folder to the load
path for you.

** layers.el
This file is the first file to be loaded and this is the place where additional
layers can be declared.

For instance, if layer A depends on some functionality of layer B, then in the
file =layers.el= of layer A, we can add:

#+BEGIN_SRC emacs-lisp
  (configuration-layer/declare-layer 'B)
#+END_SRC

The effect is that B is considered a used layer and will be loaded as if it
was added to =dotspacemacs-configuration-layers= variables.

** packages.el
It contains this list of packages of the layer and the actual configuration for
the packages included in the layer.

This file is loaded after =layers.el=.

It must define a variable called =<layer>-packages=, which should be a list of
all the packages that this layer needs. Some valid package specifications are
as follows:

#+BEGIN_SRC emacs-lisp
  (defconst mylayer-packages
    '(
      ;; Get the package from MELPA, ELPA, etc.
      some-package
      (some-package :location elpa)

      ;; A local package
      (some-package :location local)

      ;; A local package to be built with Quelpa
      (some-package :location (recipe :fetcher local))

      ;; A package recipe
      (some-package :location (recipe
                               :fetcher github
                               :repo "some/repo"))

      ;; An excluded package
      (some-package :excluded t)
      ))
#+END_SRC

The =:location= attribute specifies where the package may be found. Spacemacs
currently supports packages on ELPA compliant repositories, local packages,
remote packages hosted on Git repositories (including specific helpers for
GitHub, GitLab, and Bitbucket) and MELPA recipes (through the Quelpa package).
Local packages should reside at =<layer>/local/<package>/=.

For information about recipes see the [[https://github.com/milkypostman/melpa#user-content-recipe-format][MELPA documentation]].

As you may have noticed from examples above, there are two ways to declare a
local package: using either =:location local= or a Quelpa recipe with the
Spacemacs-specific pseudo-fetcher =local=. The former is for the simplest
packages that declare no external dependencies, since it just adds the package
directory to the =load-path=. The latter is for packages that do have external
dependencies declared and thus have to be built with Quelpa.

Packages may be /excluded/ by setting the =:excluded= property to true. This
will prevent the package from being installed even if it is used by another
layer.

For each included package, you may define one or more of the following
functions, which are called in order by Spacemacs to initialize the package.
1. =<layer>/pre-init-<package>=
2. =<layer>/init-<package>=
3. =<layer>/post-init-<package>=

It is the responsibility of these functions to load and configure the package in
question. Spacemacs will do nothing other than download the package and place it
in the load path for you.
*Note:* A package will not be installed unless at least one layer defines an
=init= function for it. That is to say, in a certain sense, the =init= function
does mandatory setup while the =pre-init= and =post-init= functions do optional
setup. This can be used for managing cross-layer dependencies, which we will
discuss later.

** funcs.el
It contains all the defined functions used in the layer.

This file is loaded after =packages.el= and before =config.el=.

It is good practice to guard the definition of functions to make sure a package
is actually used. For instance:

#+BEGIN_SRC emacs-lisp
  (when (configuration-layer/package-used-p 'my-package)
    (defun spacemacs/my-package-enable () ...)
    (defun spacemacs/my-package-disable () ...))
#+END_SRC

By guarding these functions we avoid defining them when the package =my-package=
is not used.

** config.el
This file configures the layer by declaring layer variables' default values and
setting up some other variables related to the layer.

This file is loaded after =funcs.el=.

** keybindings.el
It contains general key bindings.

This is the last file loaded.

The word /general/ here means /independent of any package/. Since the end user
can exclude an arbitrary set of packages, you cannot be sure that, just because
your layer includes a package, that package will necessarily be loaded. For this
reason, code in these files must be generally safe, regardless of which packages
are installed.

More on this in the next section.

* The Spacemacs loading process
The Spacemacs loading process can be summarized as follows:
1. Spacemacs goes through all the enabled layers and evaluates their files.
   First =layers.el= is loaded to declare layer dependencies. Then =packages.el=
   and =funcs.el= are loaded, but nothing happens from them since these files
   only define functions and variables, then the changes introduced by =config.el= are applied.
2. Spacemacs checks which packages should be downloaded and installed. To be
   installed, a package must be
   - included by a layer that the user has enabled,
   - not be excluded by any other layer that the user has enabled,
   - not be excluded by the user herself, and
   - there must be at least one =<layer>/init-<package>= function defined for
     it.

   Alternatively, if a package is part of the end user's =dotspacemacs-additional-packages=, it will also be installed.
3. All packages which should be installed are installed in alphabetical order, =package.el= built-in Emacs library is in charge of implicit dependencies.
   Installed packages not following the rules of 2. are removed as well as
   their dependencies if possible. (This last behavior is optional but default.)
4. The =pre-init=, =init= and =post-init= functions for each installed package
   are executed in turn.

It is step four that interests us. It is very important that a package is not
installed if no =init= function is defined for it.

We say that a layer *owns* a package if it defines an =init= function for it. A
layer does *not* own a package if it only defines =pre-init= or =post-init=
functions.

Only one layer may own a package. Since layers are processed in order of
specification in the user's dotfile, it is possible for layers to "seize"
ownership of a package that was owned by a previously enabled layer.

* Case study: auto-completion
Spacemacs provides a layer called =auto-completion= which provides
auto-completion features in many modes. It does this using the package
=company=. This layer owns the =company= package, so it defines a function
called =auto-completion/init-company=.

When a user enables the =auto-completion= layer, Spacemacs locates it and finds
=company= in the list of packages. Provided that =company= is not excluded,
either by the user or another layer, Spacemacs then locates and runs the =init=
function for =company=. This function includes a call to =use-package= that sets
up the basic configuration.

However, auto-completion is a two-horse game. By its very nature, it is specific
to the major mode in question. It is pointless to expect the =auto-completion=
layer to include configuration for each conceivable major mode, and equally
futile to expect each programming language layer (python, ruby, etc.) to fully
configure =company= on their own.

This is solved using the =post-init= functions. The Python layer, for example,
includes the =company= package and defines a function called
=python/post-init-company=. This function is called after
=auto-completion/init-company=, but it is not called if
- the =auto-completion= layer is not enabled, in which case no =init= function
  for =company= will be found, or
- the =company= package is excluded either by the user or another layer

As such, =python/post-init-company= is the /only/ safe place to put
configuration related to =company= in Python mode.

If the Python layer had defined an =init= function for =company=, that package
would have been installed even if the =auto-completion= layer had been disabled,
which is not what we want.

* Layer tips and tricks
** Cross-dependencies
Spacemacs provides a couple of additional useful functions you can use to check
whether other layers or packages are included.
- check if a layer is enabled
- check if a package is or will be installed

These are useful in some cases, but usually you can get the desired result just
by using =post-init= functions.

For layers that require another layers to be enabled, use the functions
=configuration-layer/declare-layer= and =configuration-layer/declare-layers= to
ensure that layers are enabled even if the user has not enabled them explicitly.
Calls to these functions must go in the =layers.el= file.

** Shadowing
Shadowing is the operation of replacing a used layer by another one. For
instance if a used layer A can shadow a used layer B and the layer A is listed
after the layer B in the dotfile then the layer A replaces the layer B and it is
like only the layer A is being used.

Examples of this mechanism are helm/ivy layers or neotree/treemacs layers.

A layer can shadow other layers by calling in its =layers.el= file the function
=configuration-layer/declare-shadow-relation=. This function declares a
=can-shadow= relation between all the layers.
=can-shadow= is a commutative relation, if layer A can shadow layer B then layer
B can shadow layer A.

The =shadow= operator is a binary operator accepting two layer names, it is not
commutative and the order of the operands is determined by the order of the
layers in the dotfile (like the ownership stealing mechanism).

If =:can-shadow= property is set explicitly to =nil= in the dotfile then the
layer won't shadow any layer.

For instance to install both ivy and helm layer:

#+BEGIN_SRC emacs-lisp
  (setq dotspacemacs-configuration-layers
   '(
     ivy
     (helm :can-shadow nil)
     )
#+END_SRC

note that due to the commutative relation =can-shadow= the above example can
also be written like this (in this case, =:can-shadow= should be read
=:can-be-shawdowed=):

#+BEGIN_SRC emacs-lisp
  (setq dotspacemacs-configuration-layers
  '(
    (ivy :can-shadow nil)
    helm
    )
#+END_SRC

We will prefer the first form as it is more intuitive.

** Use-package init and config
In the vast majority of cases, a package =init= function should do nothing but
call to =use-package=. Again, in the vast majority of cases, all the
configuration you need to do should be doable within the =:init= or =:config=
blocks of such a call.

What goes where? Since =:init= is executed before load and =:config= after,
these rules of thumb apply.

In =:config= should be
- Anything that requires the package to be already loaded.
- Anything that takes a long time to run, which would ruin startup performance.

The =:init= block should contain setup for the entry points to the package. This
includes key bindings, if the package should be loaded manually by the user, or
hooks, if the package should be loaded upon some event. It is not unusual to
have both!

** Use-package hooks
Spacemacs includes a macro for adding more code to the =:init= or =:config=
blocks of a call to =use-package=, after the fact. This is useful for =pre-init=
or =post-init= functions to "inject" code into the =use-package= call of the
=init= function.

#+BEGIN_SRC emacs-lisp
  (spacemacs|use-package-add-hook helm
    :pre-init
    ;; Code
    :post-init
    ;; Code
    :pre-config
    ;; Code
    :post-config
    ;; Code
    )
#+END_SRC

Since a call to =use-package= may evaluate the =:init= block immediately, any
function that wants to inject code into this block must run =before= the call to
=use-package=. Further, since this call to =use-package= typically takes place
in the =init-<package>= function, calls to =spacemacs|use-package-add-hook=
*always* happen in the =pre-init-<package>= functions, and not in
=post-init-<package>=.

** Best practices
If you break any of these rules, you should know what you are doing and have a
good reason for doing it.

*** Package ownership
Each package should be owned by one layer only. The layer that owns the
package should define its =init= function. Other layers should rely on
=pre-init= or =post-init= functions.

*** Localize your configuration
*Each function can only assume the existence of one package.* With some
exceptions, the =pre-init=, =init= and =post-init= functions can /only/
configure exactly the package they are defined for. Since the user can exclude
an arbitrary set of packages, there is no /a priori/ safe way to assume that
another package is included. Use =configuration-layer/package-usedp= if you
must.

This can be very challenging, so please take this as a guideline and not
something that is absolute. It is quite possible for the user to break her
Spacemacs installation by excluding the wrong packages, and it is not our
intention to prevent this at all costs.

*** Load ordering
In Spacemacs, layers are loaded in order of inclusion in the dotfile, and
packages are loaded in alphabetical order. In the rare cases where you make use
of this property, you should make sure to document it well. Many will assume
that layers can be included in arbitrary order (which is true in most cases),
and that packages can be renamed without problems (which is also in most cases).

Preferably, write your layer so that it is independent of load ordering. The
=pre= - and =post-init= functions are helpful, together with
=configuration-layer/package-usedp=.

*** No require
Do not use require. If you find yourself using =require=, you are almost
certainly doing something wrong. Packages in Spacemacs should be loaded through
auto-loading, and not explicitly by you. Calls to =require= in package init
functions will cause a package to be loaded upon startup. Code in an =:init=
block of =use-package= should not cause anything to be loaded, either. If you
need a =require= in a =:config= block, that is a sign that some other package is
missing appropriate auto-loads.

*** Auto-load everything
Defer everything. You should have a very good reason not to defer the loading
of a package.