In Elixir, macros are used to define things that would be keywords in other languages: defmodule, def, defp, defmacro, and defmacrop are all macros defined by the standard library in Kernel. In ExUnit, assert is able to both run the code passed to it to see if the test is passing, but also print that code when it fails, so that you don’t need a custom DSL to show what was being tested. In GenServer, use GenServer defines default implementations for all the required callbacks.
If you want a head-trip, look at the implementation of defmacro, which is defined using defmacro:
defmacro defmacro(call, expr \\ nil) do
define(:defmacro, call, expr, __CALLER__)
end
Don’t worry, like all languages defined in themselves, defmacro is defined using a “bootstrap” library that’s written in the underlying language, in Elixir’s case :elixir_bootstrap defines minimal versions of @, defmodule, def, defp, defmacro, defmacrop in Erlang: just enough for Kernel to be parsed once and then it defines the full version. This way, you don’t need the last version of Elixir to build the next version, just Erlang.
import Kernel, except: [@: 1, defmodule: 2, def: 1, def: 2, defp: 2,
defmacro: 1, defmacro: 2, defmacrop: 2]
import :elixir_bootstrap
Macros allow us to generate code dynamically at compile time. One of the reasons they were added to Elixir was to reduce the amount of boiler plate needed to be written for behaviours, such as :gen_server. In Erlang, this boiler plate was manually added to each file using Emacs templates.
Before the introduction of -optional_callbacks attribute in Erlang 20, there was no way to add new callbacks without having everyone update their code to add their own copy of the default implementation.
GenServer has 6 callbacks you need to implement. Every GenServer you use would need to have the correct signature and return values for all those callbacks.
So, to implement the bare minimum, we can get away with one-liners in most cases, but we need to remember the shape of each of the returns even if we don’t care about code_change/3 for hot-code upgrades. Additionally, the one-liners with raise won’t type check with dialyzer: it will warn about non-local return, which is just dialyzer’s way of saying you’re raising an exception or throwing. The real code in GenServer is doing more to make dialyzer happy and to give you more helpful error messages that are easier to debug.
def init(args), do: {:ok, args}
def handle_call(msg, _from, state), do: raise "Not implemented"
def handle_info(msg, state) do
:error_logger.error_msg(
'~p ~p received unexpected message in handle_info/2: ~p~n',
[__MODULE__, self(), msg]
)
{:noreply, state}
end
def handle_cast(msg, state), do: raise "Not implemented"
def terminate(_reason, _state), do: :ok
def code_change(_old, state, _extra), do: {:ok, state}
But, if you read the docs for GenServer and know that you don’t need to implement all the callbacks, you can put use GenServer in your callback module and all those default implementation will be defined for you. So, you go from having to hap-hazardly copy default implementations to each callback module to a single line.
Just like defmodule and the various def* for call definitions, use is not a keyword in Elixir, it is a macro in Kernel, so think of use as a convention, not a keyword.
use is not magic. It’s very short piece of code that is only complex to give some convenience:
- It automatically does
require, as__using__is a macro and macros can’t be used without an explicitrequirefirst - It uses
Enum.map, so you can pass multiple aliases (use Namespace.{Child1, Child2}) - It raises an
ArgumentErrorif you called it wrong.
defmacro use(module, opts \\ []) do
calls = Enum.map(expand_aliases(module, __CALLER__), fn
expanded when is_atom(expanded) ->
quote do
require unquote(expanded)
unquote(expanded).__using__(unquote(opts))
end
_otherwise ->
raise ArgumentError, "invalid arguments for use, expected a compile time atom or alias, got: #{Macro.to_string(module)}"
end)
quote(do: (unquote_splicing calls))
end
If use just calls the __using__ macro, what is the __using__ macro supposed to do? The only requirement is that it behaves like any other macro: it returns quoted code. The rest is up to the conventions and best practices in the docs for Kernel.use.
Example
Let’s look at an example of using __using__ and the misteps you can make along the way and how to fix them.
While working at Miskatonic University, William Dyer started a compendium of various species the university had encountered. The university’s not mad enough to try to bring them to Earth, so we use a Client library to establish communication with grad students working in the field.
defmodule Miskatonic.OldOnes do
def get(id) do
with {:ok, client_pid} <- client_start_link() do
Miskatonic.Client.show(client_pid, id)
end
end
defp client_start_link do
Miskatonic.Clients.Portal.start_link(entrance: "witch-house")
end
end
Miskatonic.OldOnes@william-dyer
While researching the Old Ones, Miskatonic grad students found some of their records referring to greater species that the Old Ones were studying. Because naming is hard, Miskatonic has started to call them Great Old Ones.
defmodule Miskatonic.GreatOldOnes do
def get(id) do
with {:ok, client_pid} <- client_start_link() do
Miskatonic.Client.show(client_pid, id)
end
end
defp client_start_link do
Miskatonic.Clients.Boat.start_link(
latitude: -47.15,
longitude: -126.72
)
end
end
Miskatonic.GreatOldOnes@gustaf-johansen
So, we have two modules, that both have a get function for getting the research on a resource, but how we can communicate with the grad students in the fields differ. We want to make communicating with new and exciting things that want to drive us mad easier because we keep losing grad students, so we need to refactor our two modules and extract the common pieces. Here’s the general shape. There’s a get/1 function that takes an id and then internally there’s client_start_link/0 function that hides the different ways we communicate with the realms of the different species.
defmodule Miskatonic.Species do
def get(id) do
with {:ok, client_pid} <- client_start_link() do
Miskatonic.Client.show(client_pid, id)
end
end
defp client_start_link do
??
end
end
Using use
Using the use convention, we can move get/1 definition into a quote block in the __using__ macro for a new, general Miskatonic.Species module. We can move get/1 into it, but we can’t move client_start_link in it.
defmodule Miskatonic.Species do
defmacro __using__([]) do
quote do
def get(id) do
with {:ok, client_pid} <- client_start_link() do
Miskatonic.Client.show(client_pid, id)
end
end
end
end
end
Now we can use Miskatonic.Species allow us to get rid of the duplicate get/1 code in each module, but we still need the client_start_link since it differs in each.
defmodule Miskatonic.OldOnes do
use Miskatonic.Species
defp client_start_link do
Miskatonic.Clients.Portal.start_link(entrance: "witch-house")
end
end
defmodule Miskatonic.GreatOldOnes do
use Miskatonic.Species
defp client_start_link do
Miskatonic.Clients.Boat.start_link(latitude: -47.15,
longitude: -126.72)
end
end
Miskatonic.GreatOneOne@bob-howard
Bob Howard gets pulled off the project and sent to The Laundry, so a new grad student, Carly Rae Jepsen needs contact with the Yithians, who Old Ones fought.
Seeing how useful use Miskatonic.Species was in the other modules, the Carly Rae Jepsen tries the same, but she get a cryptic error message that client_start_link/0 is undefined.
defmodule Miskatonic.Yithians do
use Miskatonic.Species
end
Miskatonic.Yithians@carly-rae-jepsen-compilation-error
== Compilation error in file lib/miskatonic/yithians.ex ==
** (CompileError) lib/miskatonic/yithians.ex:2: undefined function client_start_link/0
(stdlib) lists.erl:1338: :lists.foreach/2
(stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
mix compile
Carly Rae tracks down that Miskatonic.Species depends on client_start_link/0 being defined, but Miskatonic.Species isn’t currently making the best use of the compiler to tell developers that. Using @callback, to declare that client_start_link/0 is required by @behaviour Miskatonic.Species that Carly Rae adds to the quote block.
defmodule Miskatonic.Species do
@callback client_start_link() ::
{:ok, pid} | {:error, reason :: term}
defmacro __using__([]) do
quote do
@behaviour Miskatonic.Species
def get(id) do
with {:ok, client_pid} <- client_start_link() do
Miskatonic.Client.show(client_pid, id)
end
end
end
end
end
Miskatonic.Species@carly-rae-jepsen-client-start-link-callback
So, great, Carly Rae gets a compiler warning now, that’s more specific about why Carly Rae needs client_start_link in Miskatonic.Yithians, but it looks like @callback implementations need to be public, so change all the defp client_start_link to def client_start_link
warning: undefined behaviour function client_start_link/0 (for behaviour Miskatonic.Species)
lib/miskatonic/great_old_ones.ex:1
warning: undefined behaviour function client_start_link/0 (for behaviour Miskatonic.Species)
lib/miskatonic/yithians.ex:1
warning: undefined behaviour function client_start_link/0 (for behaviour Miskatonic.Species)
lib/miskatonic/old_ones.ex:1
mix compile
With the switch to public client_start_link/0, we can learn about the Old Ones, Great Old Ones, and Yithians, but the code could be better. Although we’re not writing the def get in every file, it’s being stored in each, which we can see if we ask for the debug info. For one function, this isn’t a big deal, but if we add more and more functions, this is unnecessary bloat, we know it’s exactly the same code. Code loading still takes time with the BEAM even if it’s faster than languages that need to be interpreted from source first.
iex> {:ok, {module, [debug_info: {_version, backend, data}]}} = :beam_lib.chunks('_build/dev/lib/miskatonic/ebin/Elixir.Miskatonic.Yithians.beam',[:debug_info])
iex> {:ok, debug_info} = backend.debug_info(:elixir_v1, module, data, [])
iex> {:ok, %{definitions: definitions}} = backend.debug_info(:elixir_v1, module, data, [])
iex> List.keyfind(definitions, {:get, 1}, 0)
{:get, 1}, :def, [line: 2, generated: true],
[{[line: 2, generated: true],
[{:id, [counter: -576460752303423100, line: 2], Miskatonic.Species}], [],
{:with, [line: 2],
[{:<-, [line: 2],
[{:ok,
{:client_pid, [counter: -576460752303423100, line: 2],
Miskatonic.Species}}, {:client_start_link, [line: 2], []}]},
[do: {{:., [line: 2], [Miskatonic.Client, :show]}, [line: 2],
[{:client_pid, [counter: -576460752303423100, line: 2],
Miskatonic.Species},
{:id, [counter: -576460752303423100, line: 2],
Miskatonic.Species}]}]]}}]}
The general approach you want to take when making functions in your __using__ quote block to be as short as possible. To do this, I recommend immediately calling a normal function in the outer module that takes __MODULE__ as an argument.
The reason I recommended always passing in the __MODULE__ is illustrated well here, module is needed, so that client_start_link/0 can be called in get/2 because it’s outside the quote block and won’t be in the module that calls use Miskatonic.Species anymore.
defmodule Miskatonic.Species do
@callback client_start_link() ::
{:ok, pid} | {:error, reason :: term}
defmacro __using__([]) do
quote do
@behaviour Miskatonic.Species
def get(id), do: Miskatonic.Species.get(__MODULE__, id)
end
end
def get(module, id) do
with {:ok, client_pid} <- module.client_start_link() do
Miskatonic.Client.show(client_pid, id)
end
end
end
Carly Rae Jepsen is doing such a good job on the code that the university doesn’t want to risk her going mad in the field, so Miskatonic University has decided to fund another graduate position on the team. Nathaniel Wingate Peaslee joins the team and discovers that the Yithian psychic link isn’t limited to just swamping location, but can be used to swap in time. This means to study more of Yithians, the Miskatonic.Yithians module should try mind transferring to a Yithian in a different time, if getting info on a Yithian fails.
defmodule Miskatonic.Yithians do
use Miskatonic.Species
def client_start_link(keywords \\ [yithian: "Librarian"]) do
Miskatonic.Clients.Psychic.start_link(keywords)
end
def get(id) do
case Miskatonic.Species.get(__MODULE__, id) do
{:error, :not_found} ->
with {:ok, pid} <- client_start_link(yithian: "Coleopterous") do
Miskatonic.Client.show(pid, id)
end
found ->
found
end
end
end
Miskatonic.Yithians@clause-cannot-match
Ah, but Nathaniel seems unable to override get/1 that the use Miskatonic.Species is inserting. Line 2 is the line where use Miskatonic.Species is called while line 8 is where Nathaniel wrote the def get.
warning: this clause cannot match because a previous clause at line 2 always matches
lib/miskatonic/yithians.ex:8
mix compile
We can use defoverridable to any function defined above in a quote block as overridden if the outer scope defines the same name and arity, instead of the outer scope appending clauses to the same name and arity. Although mixing clauses from quote blocks and the outer scope is allowed, it’s mostly going to cause confusing bugs, so I recommend always marking any functions defined in a quote block.
defoverridable in quote |
||
|---|---|---|
| Function Clauses Defined In | No | Yes |
quote |
Function uses quote clauses |
Function uses quote clauses |
quote and defmodule |
Function uses both quote and defmodule clauses |
Function uses only defmodule clauses |
So, Nathaniel marks get/1 as overridable, and the override works without warnings.
defmodule Miskatonic.Species do
@callback client_start_link() ::
{:ok, pid} | {:error, reason :: term}
defmacro __using__([]) do
quote do
@behaviour Miskatonic.Species
def get(id), do: Miskatonic.Species.get(__MODULE__, id)
defoverridable get: 1
end
end
def get(module, id) do
with {:ok, client_pid} <- module.client_start_link() do
Miskatonic.Client.show(client_pid, id)
end
end
end
Miskatonic.Species@defoverridable
But, he’s able to do more, when you override a defoverridable function, you can call the overridden function with super. This allows users of your __using__ macro to not have to look at the implementation of the function they are overriding, which means their code is more likely to continue working if you change implementation details.
defmodule Miskatonic.Yithians do
use Miskatonic.Species
def client_start_link(keywords \\ [yithian: "Librarian"]) do
Miskatonic.Clients.Psychic.start_link(keywords)
end
def get(id) do
case super(id) do
{:error, :not_found} ->
with {:ok, pid} <- client_start_link(yithian: "Coleopterous") do
Miskatonic.Client.show(pid, id)
end
found ->
found
end
end
end
Miskatonic.Yithians@defoverridable
Miskatonic University’s library is doing really well, but it still has some slight bugs: every module has a get/1 and it’s overridable, but it’s not a callback. It may seem weird to mark get/1 as a callback, since only client code calls get/1, but if we want to make test mocks, to test code that depends on Miskatonic.Species we really need a get/1 callback. By making get/1 a callback, we can also use the compact form of defoverridable, that takes the name of the behaviour whose callbacks are overridable, instead of listing each function’s name/arity.
defmodule Miskatonic.Species do
@callback client_start_link() ::
{:ok, pid} | {:error, reason :: term}
@callback get(id :: String.t) :: term
defmacro __using__([]) do
quote do
@behaviour Miskatonic.Species
def get(id), do: Miskatonic.Species.get(__MODULE__, id)
defoverridable Miskatonic.Species
end
end
def get(module, id) do
with {:ok, client_pid} <- module.client_start_link() do
Miskatonic.Client.show(client_pid, id)
end
end
end
Miskatonic.Species@defoverridable-behaviour
One final check that Elixir 1.5 gives us is @impl. @impl is like @Override in Java, but better.
- Mark which functions are implementations of callbacks
- Document which behaviour a function is for, which makes finding docs and source easier for readers
- Force all other callbacks for the same behaviour to use
@implto maintain consistent documentation.
In Miskatonic.Species, there is only one behaviour, but if it was a stack of behaviours, such as building on top of GenServer, then marking which callbacks are for GenServer and which are for other behaviours can be very helpful.
defmodule Miskatonic.Species do
@callback client_start_link() ::
{:ok, pid} | {:error, reason :: term}
@callback get(id :: String.t) :: term
defmacro __using__([]) do
quote do
@behaviour Miskatonic.Species
@impl Miskatonic.Species
def get(id), do: Miskatonic.Species.get(__MODULE__, id)
defoverridable Miskatonic.Species
end
end
def get(module, id) do
with {:ok, client_pid} <- module.client_start_link() do
Miskatonic.Client.show(client_pid, id)
end
end
end
TL;DR
Let’s review Miskatonic University’s finding and thank the graduate students for turning mad, so we don’t have to.
- We can use
use, which calls__using__, which callsquoteto inject default implementations - All
defs in thequoteblock should be declared as@callbacks in the outer module wheredefmacro __using__is. - Put
@behaviourwith the outer module as the behaviour name at the top of quote block - The default functions should be one-liners that call functions with the same name in the outer module with
__MODULE__as a prepended argument. - Mark all default functions with
@impl, as it will force other callbacks for the behaviour to also use@impland double check you got the name and arity right between the@callbacksand implementation in the quote block. - Use that passed in
__MODULE__whenever you need to call another callback from the outer module functions, so that overrides for any callback will always be called. Don’t call other outer module functions directly! - Use
defoverridablewith the outer module so that you don’t have confusing errors with clauses mixing from the quote block and theuseusing module.
Note: Updated 2018-10-17 to clarify defoverridable table.
