iex -S mix phx.server: What? How?

iex -S mix phx.server: it starts up my Phoenix dev server under Elixir’s interactive shell program, IEx, so I can call functions from my app and whatnot.

One day, it started to bug me unreasonably much that I didn’t really understand how this command causes that to happen. Perhaps culpable: a touch of delirium, downstream of the flu.

There were a lot of reasons for my mystification (i.e. a lot of basic things I hadn’t thought much about before), so below you’ll find, if you want to, ~2700 words about some things that happen and how. I don’t expect even one whole person to read the entirety. It’s not that kind of blog post.

There may be inaccuracies.

Interacting scripts

Before getting fancy, I want to know what my basic everyday Elixir commands do.

The elixir, iex, and mix commands are all scripts.

elixir, the script

The elixir shell script constructs a string of arguments based on the arguments you supply, plus some information about your Elixir installation, and tacks that onto an erl command.

The elixir script ends with

if [ -n "$ELIXIR_CLI_DRY_RUN" ]; then
  echo "$@"
else
  exec "$@"
fi

It’s assembled the erl invocation, with all its arguments, into "$@", and all that’s left is to execute that command.

But look! By setting the ELIXIR_CLI_DRY_RUN env var, you can view the whole assembly instead of running it! I didn’t know that; I’m going to use it in a minute.

mix, the script

The mix script is exactly this:

#!/usr/bin/env elixir
Mix.CLI.main()

This is an Elixir script! The shebang makes mix a (much) shorter way to write (in my case, with runtime installations managed by asdf):

elixir /Users/chris/.asdf/installs/elixir/1.17.3-otp-27/bin/mix

So mix invokes elixir and we end up, again, with an erl command.

This felt circular to me, because the elixir script doesn’t do anything with the contents of the mix script. But that’s just the way it works; rest assured that something eventually reads the file. We’ll see what, later.

iex, the script

The iex shell script ends with

exec "$SCRIPT_PATH"/elixir --no-halt --erl "-user elixir" +iex "$@"

Which is to say: it calls the elixir command with a specific set of arguments, followed by any arguments it was called with ("$@").

Another erl command.

elixir --help has entries for --no-halt (“[do] not halt the Erlang VM after execution”) and --erl (“[s]witches to be passed down to Erlang”). That +iex option isn’t in the help, but the elixir script does notice it, and omit -s elixir start_cli from its erl arguments.

But we don’t have to work out the final set of arguments by reading the scripts. We can just print them out by setting the ELIXIR_CLI_DRY_RUN environment variable.

Pause to appreciate coolness

My Erlang/OTP installation doesn’t know anything about my Elixir installation.

The arguments in these human-readable erl commands will have to tell it, or point it at, everything it needs to know in order to do whatever Elixir-land thing I wanted. This is not trivial or banal. It is very cool.

How to read erl arguments

So: elixir runs erl with a collection of arguments. And iex and mix both go through elixir. That is, each of these commands generates an erl command.

The erl docs do a pretty good job of explaining the different kinds of argument that erl accepts, and what it does with each kind.

It can be a bit hairy to categorise them in practice. The following paraphrases info that’s in the docs, with some emphasis on “subtleties” that I missed in my first reading, and that would have saved me some casting about in source code.

• Emulator flags are for VM settings. Emulator flags start with a plus sign, except when they start with a hyphen like all the other flags. We won’t run into any emulator flags in this exercise.

• Many, but not all, other defined flags are marked as init flags: flags that are interpreted and used by Erlang’s init and not stored for later use.

• -- and -extra are init flags that indicate that what follows is one or more plain_arguments: values that the init stores in a list that you can get later with init:get_plain_arguments/0.

• You, the user, can create user flags to suit your needs. The init stores user flags as key-value pairs, where the flag becomes an atom key for a value that’s a list of the items that follow (up until the next flag). To get this list, use init:get_arguments/0. If an argument starts with a hyphen, and it’s not an init flag, an emulator flag or a plain argument, it’s a user flag.

• Some code that comes with the Erlang distribution looks for arguments stored with specific user flags. Some of these user flags are documented alongside the init flags.

• Case in point: there’s an important special-case user flag documented as -Application Par Val; when/if the OTP application named Application is started, Application‘s Par configuration option is set to Val. We see this shape of user flag in the elixir command. This one tripped me up; I could see the apparent intent but missed it in the docs.

elixir, mix, iex: the erl angle

Armed with the above clues to how erl statements work, let’s look at minimal examples of each of our Elixir commands.

elixir

If I don’t put something after elixir, it just prints its help at me, so I’ll tell it to print hi instead.

ELIXIR_CLI_DRY_RUN=1 elixir -e 'IO.puts("hi")'

Here’s what that spits out:

erl -noshell \
-elixir_root /Users/chris/.asdf/installs/elixir/1.17.3-otp-27/bin/../lib \
-pa /Users/chris/.asdf/installs/elixir/1.17.3-otp-27/bin/../lib/elixir/ebin \
-elixir ansi_enabled true \
-s elixir start_cli \
-extra -e IO.puts("hi")

That’s an eyeful, but it’s mostly because paths are long and messy. Flag by flag:

-noshell

Skips starting the Erlang shell. We want this init flag if we’re working in Elixir and not Erlang. Either we don’t want a shell, or we want IEx.

-elixir_root /Users/chris/.asdf/installs/elixir/1.17.3-otp-27/bin/../lib

-elixir_root is a user flag. We’re telling the init to store the path to our installed Elixir lib dir under the key :elixir_root.

Spoiler: The start/2 function of the elixir module uses this stored value to set paths to ebin directories, where .beam files are.

-pa /Users/chris/.asdf/installs/elixir/1.17.3-otp-27/bin/../lib/elixir/ebin

-pa is an init flag. Adds the path to the Elixir .beam files to Erlang’s code path.

-elixir ansi_enabled true

This is that special -<application> <parameter> <value> user-flag format. Sets :ansi_enabled to true in the :elixir OTP application’s config when it starts.

-s elixir start_cli

-s is another init flag. Run function start_cli/0 of the elixir module.

This function starts the elixir OTP application (and starts the logger application) and finally passes our list of stored plain arguments to Elixir.Kernel.CLI.main/1, “the API invoked by [the] Elixir boot process” with this line:

  'Elixir.Kernel.CLI':main(init:get_plain_arguments()).

(Good thing we told Erlang where to find Elixir with the -pa flag.)

-extra -e IO.puts("hi")

Each thing after -extra in an erl expression is a plain argument, so -e and 'IO.puts("hi")' are stored by the init as such.

My 'IO.puts("hi")' came out without its single quotes. I can roll with that; I had to type it in a way that would be parsed unambiguously in the shell, and now it’s in the shape it has to be in for the next thing to parse.

On that topic, we can quickly check how the init stores plain arguments:

erl -extra -e 'IO.puts("hi")' 

Erlang/OTP 27 [erts-15.2.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

Eshell V15.2.1 (press Ctrl+G to abort, type help(). for help)
1> init:get_plain_arguments().
["-e","IO.puts(\"hi\")"]

(Plain arguments keep their order in the list, which is important when it comes to parsing and using them.)

Recapping what’s explicit in the erl expression: typing elixir -e 'IO.puts("hi")' into my system shell starts an Erlang VM (BEAM) instance in which the Erlang init process:

• doesn’t start an Erlang shell
• prepares the Erlang runtime environment, including:
  • setting the path to Elixir .beam files
  • storing some arguments, which in this case are for setting up the elixir OTP application
  • storing some plain_arguments that correspond to the arguments I passed to the elixir command
• invokes the start_cli/0 function of the elixir module

That’s the end of what we can read on the face of the erl expression.

Then, elixir:start_cli/0 configures and starts the elixir and logger OTP applications, yoinks the list of plain arguments from the Erlang init, and passes that to our first Elixir function, Elixir.Kernel.CLI.main/1.

So I want to know what Elixir does with these arguments.

So that’s what “the API invoked by [the] Elixir boot process” meant. The options you pass to the elixir CLI command, telling it what you want “Elixir” to do, go to the Kernel.CLI Elixir module, which makes it happen.

My request to evaluate the Elixir syntax IO.puts("hi") is executed. The VM prints hi to stdout. I see it in my terminal. Elixir has done what I asked and halts the system.

mix

ELIXIR_CLI_DRY_RUN=1 mix   

erl -noshell \
-elixir_root /Users/chris/.asdf/installs/elixir/1.17.3-otp-27/bin/../lib \
-pa /Users/chris/.asdf/installs/elixir/1.17.3-otp-27/bin/../lib/elixir/ebin \
-elixir ansi_enabled true \
-s elixir start_cli \
-extra /Users/chris/.asdf/installs/elixir/1.17.3-otp-27/bin/mix

This is identical to the elixir -e 'IO.puts("hi")' example, but with different plain arguments after the -extra flag.

Elixir.Kernel.CLI will recognise the path to the mix script as the path to a file, read it, and call the function inside: Mix.CLI.main().

iex

ELIXIR_CLI_DRY_RUN=1 iex     

erl -noshell \
-elixir_root /Users/chris/.asdf/installs/elixir/1.17.3-otp-27/bin/../lib \
-pa /Users/chris/.asdf/installs/elixir/1.17.3-otp-27/bin/../lib/elixir/ebin \
-elixir ansi_enabled true \
-user elixir \
-extra --no-halt +iex

In the not-IEx commands, -s elixir start_cli started Elixir, but the -s flag doesn’t show up here.

-user elixir

ELUSIVE. But important! -user is a user flag, and isn’t in the docs, but it’s a special one; the Erlang user_sup module looks in the init arguments for the key user.

The user is the process that deals with the Erlang VM’s I/O; to interact with IEx, we need input and output to go through it.

The best resource I’ve found for this Erlang user concept: REPL? A bit more (and less) than that by Fred Hebert. (“If you want to change where the IO takes place, change the user process, and everything gets redirected.”)

It looks like passing -user elixir means user_sup:start_user/3 calls apply(elixir, start, []).

In turn, elixir:start/0 passes user_drv:start/2 iex:shell/0 for its initial_shell argument and iex:shell/0, finally, calls elixir:start_cli/0, which feeds our plain arguments to Kernel.CLI to be acted on.

-extra --no-halt +iex

The plain arguments are --no-halt and +iex.

We can find their significance in the Elixir Kernel.CLI source:

• --no-halt results in a no_halt: true entry in the config map, which results in the decision not to emit System.halt(status) after executing commands.
• +iex sets mode: :iex in config; this seems to be mainly used to decide how to deal with other flags like -v or --version, and --dbg.

Differences from the elixir command:

• start the iex application after elixir
• send all VM I/O through iex
• don’t halt the system once Kernel.CLI has processed the plain arguments and taken any actions they ask for

As we know, the end result is that I’m dropped into an IEx shell once all that’s done, and I can ask IEx to run moar Elixir.

What’s running in the VM?

Bare minimum, that is.

Here are the started applications in my bare IEX shell.

iex -e "IO.inspect(Application.started_applications)"

Erlang/OTP 27 [erts-15.2.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

[
  {:logger, ~c"logger", ~c"1.17.3"},
  {:iex, ~c"iex", ~c"1.17.3"},
  {:elixir, ~c"elixir", ~c"1.17.3"},
  {:compiler, ~c"ERTS  CXC 138 10", ~c"8.5.4"},
  {:stdlib, ~c"ERTS  CXC 138 10", ~c"6.2"},
  {:kernel, ~c"ERTS  CXC 138 10", ~c"10.2.1"}
]
Interactive Elixir (1.17.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> 

I haven’t run into the place that starts the compiler application, but I can wave my hands and say it makes sense that we’d need it; I might ask the VM to run something that hasn’t been compiled yet.

I can compare the applications started in a VM that I spun up to run an Elixir function to list the applications started in it:

elixir -e "IO.inspect(Application.started_applications)"

[
  {:logger, ~c"logger", ~c"1.17.3"},
  {:elixir, ~c"elixir", ~c"1.17.3"},
  {:compiler, ~c"ERTS  CXC 138 10", ~c"8.5.4"},
  {:stdlib, ~c"ERTS  CXC 138 10", ~c"6.2"},
  {:kernel, ~c"ERTS  CXC 138 10", ~c"10.2"}
]

For fun, here’s one just for erl. Just kernel and stdlib in this VM:

erl -eval 'io:format("~p~n", [application:which_applications()]).'

Erlang/OTP 27 [erts-15.2.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

[{stdlib,"ERTS  CXC 138 10","6.2"},{kernel,"ERTS  CXC 138 10","10.2.1"}]
Eshell V15.2.1 (press Ctrl+G to abort, type help(). for help)
1> 

Putting the pieces together

When we start the BEAM with an iex command, all the VM’s I/O goes through IEx, and IEx implements the interactivity, including taking care of interpretation of Elixir expressions, or compilation of code, as necessary.

All that’s left is to add -S mix phx.server:

ELIXIR_CLI_DRY_RUN=1 iex -S mix phx.server  

erl -noshell \
-elixir_root /Users/chris/.asdf/installs/elixir/1.17.3-otp-27/bin/../lib \
-pa /Users/chris/.asdf/installs/elixir/1.17.3-otp-27/bin/../lib/elixir/ebin \
-elixir ansi_enabled true \
-user elixir \
-extra --no-halt +iex -S mix phx.server

No surprises. Just like iex, but with an additional -S mix phx.server after -extra.

Check on the plain arguments:

iex(1)> :init.get_plain_arguments()
[~c"--no-halt", ~c"+iex", ~c"-S", ~c"mix", ~c"phx.server"]

How does Mix get its arguments?

We already talked about how the ["-S", "mix"] part of the arguments list means Elixir.Kernel.CLI will execute the mix script.

And we kinda know that Mix has to see the "phx.server" argument, since that’s the Mix Task we’re trying to run. Let’s just put a tidy bow on how that happens:

As soon as Kernel.CLI.parse_argv/2 matches "-S", it stashes the next argument as a :script into its list of commands to run, and quits looking for things to parse. Everything that’s left in the original arguments list stays in the system command-line arguments list (System.argv/0). In this case, "phx.server" is what’s left.

On the Mix side, the contents of the mix script calls Mix.CLI.main/1, which defaults to getting its argument list from System.argv/0. Voilà: Mix.CLI.main(["phx.server"]).

An analogous thing happens, also via Kernel.CLI, if we run mix phx.server, or elixir -S mix phx.server.

The first thing that Mix.CLI.main/1 does is start the Mix application with Mix.start/0—but I could go on like this forever. Let’s just agree that Mix is now going to run the phx.server task.

Why is it shaped like this?

I now truly believe that iex -S mix phx.server starts my Phoenix dev server on a fresh Erlang VM and drops me into an Elixir shell in that VM.

I’ve had the chance to admire the effort and ingenuity that goes into letting users run programs—no—start up VMs and run programs on them—with simple one-liners.

I still go back and forth between “this is an obvious command that’s really explicit about what it does” and “something about this feels magical and weird.” That -S just sits kinda funny.

If I were in the habit of using elixir -S mix phx.server to run the phx.server Mix task (which works): look, symmetry! iex -S mix phx.server is the same thing but through IEx!

But I don’t do that, because mix phx.server works, thanks to mix being an Elixir script you can also invoke from a system shell. It all makes sense! Not so symmetric, though.

In the end, it’s a balance between elegance and power. I see a lot more elegance in it than I did before.

Trivia: Some history of iex -S mix

Early on, it seems there was some tension between the conciseness of mix iex (possible when mix is a plain shell script) and the flexibility of iex -S mix (possible when mix is an Elixir script); bin/mix went back and forth between shell script and Elixir script in 2012-2013:

• 2012/07/16: “Add mix file executables” (a shell script)
• 2012/07/31: “Make mix an Elixir script” (an Elixir script)
• 2012/11/30: “Fix mix iex” (a shell script again)
• 2012/12/05: J. Valim: “I believe we need to remove support for mix iex as it needs to be iex -S mix.” (v0.7.2 breaks mix #692; some discussion here.)
• 2013/01/20: “Provide iex -S mix instead of mix iex” (an Elixir script again)