Let it Rain - Ditch that Umbrella App!

08 Feb 2020

A little over a year ago, I started working on an Elixir/Phoenix application for my startup. I was brand new to Elixir and Phoenix and learning as I went. Developing an application on your own means making a lot of decisions, and one decision I made back then was to generate the application as an Elixir/Phoenix “umbrella project.” This approach allows you to have a group of applications in one repo and run them together from the top level.

I had misgivings almost immediately and ultimately decided to convert my project to a conventional Elixir/Phoenix layout. I was surprised that I couldn’t find any help online as to how to go about this. So today, I’m writing up my experiences so that maybe I can help someone else who decides to ditch their umbrella.

Trigger warning: this post is going to have far too many occurrences of the words “app,” “application,” “folder,” “directory,” etc. It’s hard to talk about moving things around in a file system in a way that isn’t mind-numbing. Ha!

Looks Like Rain… Bring an Umbrella!

As is explained in the official docs, an umbrella project is “used to build applications that run together in a single repository.” Coming from working on a high-traffic Rails application for over a decade, this resonated with me. I envisioned a cluster of apps similar to the system I’d worked with all those years. This included a primary Rails monolith that served the large web application (both a server-rendered website and extensive APIs), but also several smaller applications like background job workers, deferred HTTP request workers, etc.

Nevermind that these weren’t all written in the same language. I’d found my new love in Elixir, and I would happily write all my little side apps in Elixir and keep them warm and cozy under my umbrella.

Or Maybe Better Off Getting Wet?

Almost immediately, I ran into frustrations. It seemed like each application constantly had to recompile, including dependencies. And speaking of dependencies, I ran into a lot of confusing issues getting the right dependencies listed in the right mix.exs configuration file. The root of the project has a mix.exs that loads each app’s mix.exs. I can’t remember exactly what my problems were, but I know it was confusing.

When I finally started converting my project back to a non-umbrella layout, I sought advice in the Elixir Slack channel and someone mentioned that the current official advice on umbrella configuration is to only have a mix.exs in the root of the project. That definitely makes the applications interdependent though! I thought the whole point of an umbrella project was that you could deploy each app to separate nodes if you needed to? (Note: maybe you still can? Elixir is multi-node-capable.)

Oh, and then there was the “Could not start application …” error I kept getting. That turned out to be an issue in Elixir itself and updating to Elixir 1.8.1 fixed it. I’m pretty sure, however, that that only happened with umbrella projects (a non-umbrella app would by default only start one Elixir application running).

Finally, VSCode has a nice feature where you can alt-click on the file path in a stacktrace in the built-in terminal to open that file at that line. But this didn’t work with an umbrella project because the domain and web apps run as individual applications from their respective directories. So running the tests actually executes from within each app’s directory, and because you are running VSCode in the root of the project, the file paths in the stacktrace don’t match. You can change into and out of each app’s directory all the time, but that’s annoying.

This was probably the most frustrating thing of all. After getting full-time Elixir work last summer, I didn’t get back to working on this personal project for a while. But in the meantime, I did get very entrenched in my tools and workflow at my day-job: VSCode with the ElixirLS plugin. I developed habits quickly, like that alt-clicking on a line in a stacktrace to go straight to the problem. Sure, I can just open the file and navigate to the line, but it takes a couple of seconds longer and when you do it constantly, all day long, it really drags on you.

Really, the only benefit of using an Umbrella layout that I noticed was that running tests from the top-level groups the test output per-app. That’s a pretty small thing though. Perhaps I will eventually wish I had the domain and web apps as separate applications, but it’s not worth the hassle to worry about that now. Classic premature optimization!

Converting to a Standard Phoenix Layout

If you decide to convert your umbrella project to a standard layout, you should definitely do it in a git branch so that you can keep track of what you’ve done so far and diff with master when necessary. It’s best to make small commits too, so you can review more easily and undo something if you need to. The problem is that as soon as you start moving things around, your build is going to break, so you don’t have the usual refactoring reassurance of passing tests.

By default, umbrella projects have an apps directory at the root level, with each application in its own sub-folder:

Regular Phoenix projects just have the domain (e.g.: my_app) and web (e.g.: my_app_web) folders inside a root-level lib directory. Those are a convenience/namespace separation, not separate applications:

I’ll get to some important config changes that this means in a bit, but that’s not what I did first. I just started moving directories around. There’s probably a more disciplined way of doing this, but I’ll explain what I did and you can adjust as you see fit.

The first step I took for the “de-umbrellification” was to move each folder in the apps directory into a new lib folder at the top level. Note, however, that the actual guts of each of these applications is the folder within the lib folder of each. So we move apps/your_app/lib/your_app and apps/your_app_web/lib/your_app_web into a new, top-level lib directory. See the screenshot to better understand what I mean:

Because each of these apps is intended to be standalone, it also has its own test folder. The conventional layout for a standard Phoenix app, however, has a test directory at the root of the project with child folders for the domain and web tests. Our per-app test folders are still hanging out in the old apps directory, so let’s move apps/your_app/test/your_app and apps/your_app_web/test/your_app_web into a new top-level test directory.

This leaves behind a support directory in each test directory; the one in the domain app has data_case.ex and the web app has conn_case.ex and channel_case.ex. Those can all move into one support directory in your new test folder.

Most likely, both of your apps’ test directories have a test_helper.exs file and they are probably identical. It is just some bare Elixir code to start the ExUnit app and the db sandbox. You can move one into your new test directory and delete the other.

Both applications also have a priv directory. For the domain app, this has the migrations, while the web app is likely to have gettext and static directories. All this stuff should also be combined into one top-level priv directory.

The last thing to move is the assets directory for the web app. Move it out of apps/your_app_web/ up to the top level. You will also have to update the include path in app.css and the Javascript dependencies paths in package.json to reflect that change.

For that matter, there may be several paths that need to be fixed. I had a test mock that loaded JSON files to simulate HTTP responses (this was before I knew about things like ExVCR) and had to fix those because the support files moved to a different place than the tests. But we’re just getting started editing and/or combining files.

Combining Configurations

The configuration arrangement of an umbrella application is arguably its boon and its bane. The top-level mix.exs file is generally pretty sparse. Its magic is in the apps_path option in the project function. This tells mix that we’re working with an umbrella project and where to look for the individual applications. (I’m not sure why anyone would change the name of that directory, but there you go.)

The dependencies listed here would be only those used by all your child applications, which may be none. Each app has its own mix.exs file with its own list of dependencies and with extra configurations to tell Elixir where stuff is relative to this child application (e.g.: build_path, deps_path, etc).

Standard Phoenix projects run only one application, named for the domain, and the web “app” Endpoint is a child process of the app. So there is just one set of config files. To continue our conversion to a standard Elixir project then, we need to merge our mix.exs files into one top-level file. Copy all the dependencies into one list in the dep function and replace the apps_path option with app and the name of your main application, e.g.: your_app.

You can probably just toss the umbrella’s top-level mix.exs and replace it with the mix.exs from your domain app, although you’ll want to remove all the configs like “something_path”. See the examples below. This all should be done with some caution to make sure you don’t miss anything. Do be sure to change the module name of whichever mix.exs you keep (i.e. drop Umbrella from it).

Here is the start of the top-level umbrella mix.exs:

defmodule Bravauto.Umbrella.MixProject do
  use Mix.Project

  def project do
    [
      apps_path: "apps",
      start_permanent: Mix.env() == :prod,
      deps: deps()
    ]
  end

  # ... rest of functions

While here is the start of the umbrella domain app’s mix.exs:

defmodule Bravauto.MixProject do
  use Mix.Project

  def project do
    [
      app: :bravauto,
      version: "0.1.0",
      build_path: "../../_build",
      config_path: "../../config/config.exs",
      deps_path: "../../deps",
      lockfile: "../../mix.lock",
      elixir: "~> 1.5",
      elixirc_paths: elixirc_paths(Mix.env()),
      start_permanent: Mix.env() == :prod,
      aliases: aliases(),
      deps: deps()
    ]
  end

  # ... rest of functions

And here is the start of the umbrella web app’s mix.exs:

defmodule BravautoWeb.MixProject do
  use Mix.Project

  def project do
    [
      app: :bravauto_web,
      version: "0.1.0",
      build_path: "../../_build",
      config_path: "../../config/config.exs",
      deps_path: "../../deps",
      lockfile: "../../mix.lock",
      elixir: "~> 1.5",
      elixirc_paths: elixirc_paths(Mix.env()),
      compilers: [:phoenix, :gettext] ++ Mix.compilers(),
      start_permanent: Mix.env() == :prod,
      aliases: aliases(),
      deps: deps()
    ]
  end

  # ... rest of functions

But you wind up with just one mix.exs at the top-level (and without the extra path configs):

defmodule Bravauto.MixProject do
  use Mix.Project

  def project do
    [
      app: :bravauto,
      version: "0.1.0",
      elixir: "~> 1.5",
      elixirc_paths: elixirc_paths(Mix.env()),
      compilers: [:phoenix, :gettext] ++ Mix.compilers(),
      start_permanent: Mix.env() == :prod,
      aliases: aliases(),
      deps: deps()
    ]
  end

  # ... rest of functions, including `deps` with all dependencies

Another delicate chore to perform is to combine your domain and web-app config directories into one set of config.exs, dev.exs, test.exs, and prod.exs files. This can be done by copying and pasting the files together and putting them in a top-level config directory. Just be sure nothing is duplicated. Check each file carefully. A freshly generated Phoenix project is a good reference here, and really throughout this process.

Since we are converting this from a multi-app project to a single app, you can delete the web Application module (e.g.: YourAppWeb.Application in application.ex at the top level of the web app directory.) However, if you just delete this file, your project will crash on startup. In the traditional project structure, your endpoint is started as a child process of the main application. So be sure to add YourAppWeb.Endpoint to the children list in the start function of your domain application.ex file:

children = [
    # Start the Ecto repository
    Bravauto.Repo,
    # Start the endpoint when the application starts
    BravautoWeb.Endpoint
]

There are also places that set config values for your web app as if it was still its own Elixir application. What I mean by this is that there will be lines— mostly in files like config.exs, dev.exs, etc.— that use the config macro to set a value. In the umbrella world, they are trying to set these on the web application but now need to set the value on the lone application for this project. Basically search for config :your_app_web and replace it with config :your_app, like this:

config :bravauto_web, BravautoWeb.Endpoint,
  url: [host: "localhost"],
  secret_key_base: "...",

# changes to:

config :bravauto, BravautoWeb.Endpoint,
  url: [host: "localhost"],
  secret_key_base: "...",

In the Home Stretch

I think once you do all this stuff, you should be home-free. You may have issues specific to your app (I wound up having to deal with upgrading my very small use of LiveView from the pre-release version I was on to the latest version), but you should be able to figure those out by digging into compilation error messages or test failures.

The thing that probably stumped me the most during this process was when I didn’t put my web Endpoint module in the application’s list of children. The error was that it couldn’t find a config value in an ETS table! I finally figured out that the ETS table didn’t exist because the endpoint process wasn’t being started.

Hopefully, this will help someone out there that decides to “de-umbrellify” their Phoenix application. It wasn’t really that hard, but it wasn’t trivial either.

Ironically, my first exposure to Elixir was taking Dave Thomas’ Elixir for Programmers video course, and in that, he comes out against umbrella apps. His attitude was that you can and should use Elixir’s multi-process nature to make separate applications for separate needs, and you can then use these together simply by wiring them up as dependencies of each other. The course does a good job of explaining this. But despite Dave’s advice, I figured I was for sure going to have a herd of apps working together, so why not use the officially-endorsed approach?

Well, I feel strongly now that that was definitely premature optimization (which we know is the root of all evil, right?). It will be ironic if someday I convert this project back to an umbrella application, but I don’t see that happening anytime soon, if ever. And actually, having converted it away from an umbrella structure will certainly make it easier for me to convert it back. If nothing else, it was a good lesson in how Elixir applications work.