Hunting Unicorns

10 May 2019

Yesterday was the six-month anniversary of my layoff, so I thought it would be a good time to post another update. I hate to start every post talking about how long it’s been since I last posted, but I did intend to post much more frequently when I started this blog. I get so focused on my mission that it’s easy to put off everything else. Also, the work has gotten more difficult. Read on to hear all the gory details.

Go Go Gigalixir… Oh No!

As I explained in my last post, I was excited about being nearly “done” with the core web version of this app. After that post, I still had a bit more to do with the LiveView code that replaced the Ajax poller. The VIN API job needed to send the result back to the view. I was able to add my first Elixir message-passing code!

  def start_vehicle_lookup(vin) do
    worker = Task.async(Bravauto.VehicleFromVin, :create_vehicle_from_vin, [vin])
    result = Task.await(worker)
    send(self(), result)
  catch
    :exit, _ ->
      send(self(), {:error, :timeout})
  end

Ironically, although Elixir is known for its processes and message-passing, it is not something you actually often need in a simple Phoenix app. Elixir, Phoenix, Plug, and Ecto start up their processes and you don’t need to worry about it for basic operation. I’ve read that it is a common mistake to reach for a GenServer for something that doesn’t need it.

After that was working correctly, I spent some time fixing bugs and improving the existing web UI. That included things like better validations, adding user and admin links in various places, improving a few forms, etc. Being new to Elixir, I also wanted to make sure that I was writing idiomatic Elixir and following best practices. I installed Credo ran it on my Elixir code and followed all of its suggestions.

Then I needed to deploy the app somewhere so that my business partner could play with it. This took longer than expected, partially because there are a several steps and partially because I had to switch hosting midstream. Ever since I chose Elixir, I have been planning to deploy this app to Gigalixir. They are very Elixir-friendly and do not do daily restarts for free-tier apps. They also have thorough documentation.

Following those docs, I installed Distillery and configured it. This went fairly smoothly. Ironically, this may be the one and only time I have to do so since Elixir 1.9 will include a mix release command. This has some differences from how Distillery works however, so we’ll see how quickly Gigalixir and Heroku come out with support for it.

I also added basic auth to my app, just in case someone stumbled on it. In hindsight, this was silly because the app name generated by Gigalixir (and later Heroku) uses random words and is highly unlikely to be guessed. I also had to add seed data for an initial admin user and ensure the seed data was loaded (vehicle makes are also loaded via seeds).

I got the app successfully deployed to Gigalixir but then tried to run migrations. Boom! I had decided early on to use Postgresql citext type for case-insensitive strings instead of having to constantly lowercase my queries. However, it is a Postgresql extension, so you have to install it and that requires superuser credentials. Turns out that the Postgresql instances in the Gigalixir free tier don’t allow superuser access! Drat!

NB: Free-tier Postgresql on Gigalixir does not offer superuser access; no Citext type!

So, I did some research and found that this is not the case for Heroku’s free tier! Hooray! There were a few config changes to make, but I soon had it up and running. This was only a temporary deploy, so I don’t mind missing out on Gigalixir’s benefits.

My business partner played with the app for a while and we met to discuss where we stood, what important features were still needed for an MVP, and where to go next. We agreed that building an initial mobile app to talk to this backend should come first. I explained the difference between a Progressive Web App and a “real” native mobile app, as well as React Native vs pure native. Pure native just isn’t an option for us without hiring someone. But my partner strongly preferred to go for as “real” a native experience as possible. React Native it is!

Entering a Whole New World (of Pain)

Initially, I allowed myself some learning time. I watched an entire twelve-episode, free, online course on React Native by Harvard University. It’s a bit out of date already, but here is the first episode if you are interested. This filled my head full of information but didn’t grease the rails as much as I had hoped once I started working on my own React Native code.

I have to admit I also suffered some decision paralysis. There are SO many decisions to make! I had already chosen React Native vs a Progressive Web App or using Flutter. I had also, since going through some React Hooks tutorials graciously provided by my brother, adamantly intended to use hooks and functional components as much as possible, preferably avoiding classes altogether. This makes seeking help online a bit frustrating since most resources show class-component examples, but I want to stick with a functional approach as much as possible. And still, there were more decisions to make and more research to do.

For state management, should I go with Redux or Mobx State Trees or do I even need such a library? For app navigation, do I use React Navigation, React Native Flux Router, or React Native Navigation? For communicating with the backend, do I use GraphQL or a traditional JSON REST API? (Spoiler: at least initially, I am using Phoenix Channels, aka WebSockets.) And for API authentication, should I use JWT (Javascript Web Tokens) or is that overkill? There are even decisions about how to write tests. React Native comes with Facebook’s Jest testing framework, but a lot of people recommend using Enzyme as well. Then, there is also the React Native Testing Library, which replaces Enzyme, promising better React Native support. Whew, so many decisions!

On top of all this, the generally preferred way to get started with React Native is using Expo, a suite of tools built on top of React Native to make the development process smoother and provide support for lots of cool features. I used Expo while working on the Harvard online course and played with it some outside of that. It definitely makes things much easier. React Native development is something of a byzantine nightmare. Or maybe a better analogy is a Rube Goldberg Machine.

You have to install Apple XCode (you have a Mac, right?) and Android Studio to get the device emulators. Then, running the app on one platform or the other builds the native code for that platform (your computer fans will get a workout!) and loads that onto the emulator. Meanwhile, a new terminal window opens to run the Metro Bundler process, which handles providing the JavaScript bundle to the mobile app. Then, if you are doing backend communication, you also need to have your development server running. And if you want to debug (you will debug), React Native can open a Chrome window to use its debugger. Finally, there is also the standalone react-devtools app, which is an Electron GUI that shows the actual component tree of your running application. That’s a lot of moving parts! Expo hid a lot of this away and made the development cycle smoother.

So why am I not currently using Expo? Well, the hooks API only made it into React with version 16.8, which came out in February, and then made it into React Native with version 0.59 a month later. Expo is lagging behind and still only supports React Native 0.58. Boo! They are promising RN 0.59 in the SDK 33 release, which is getting close. I have been loading this page every few days to check on the progress. As I write this, it is 88% complete. In the meantime, I have been getting my bearings with an app created with the standard react-native init <AppName> command line tool. This is probably good for me, as I’m getting to know the pieces of React Native better than I would with Expo smoothing the way.

And let’s just say that things have not been smooth. First, I banged out an initial stab at the VehicleEvaluation component. This shows a compact view of vehicle and auction info with the maximum bid. I was aiming to match the wireframe my partner made up for me. React Native styling is more complicated than it seems at first blush. It’s similar to CSS, but with different selectors: camelCase instead of snake_case, since it is JavaScript. I’ve never been a style expert anyway, so that was slow going, and I only got it half-presentable.

Using some JSON sample data, I then built an EvaluationList component. This shows a list of VehicleEvaluation components and will likely be one of the main screens of the app. From here the user will be able to tap an individual vehicle evaluation component and go to a detail view of that evaluation. There will also be a link to a screen to start a new evaluation. But for now, I just wanted to render a list of real evaluation data from my app. So my next step was to hook up frontend-to-backend communication. Oh goodie, more decisions!

Plan 9 Channel 7

Having recently played with Phoenix LiveView, I was interested in the real-time, stateful capabilities of WebSockets. I looked into using Phoenix channels for mobile app communication. The potential downside is in cases of poor network connectivity. However, it is already on our roadmap to require the mobile app to maintain its own state and sync with the backend when a connection is present. Also on the roadmap is to have the app show real-time updates of auctions in progress. So figuring out the channel approach seemed worth doing.

I stumbled on this post which talks about building a useChannel React hook and SocketProvider React component to integrate your app with Phoenix channels. The author had also published a use-phoenix-channel NPM package so that I could use this in my app. Neat!

On the Phoenix side of things, I set up the UserSocket and built a VehicleEvaluationChannel module:

defmodule BravautoWeb.VehicleEvaluationChannel do
  use BravautoWeb, :channel
  alias Bravauto.Accounts
  alias Bravauto.Evaluations
  alias BravautoWeb.VehicleEvaluationView

  # TODO: user token instead of user_id as room topic
  def join("vehicle_evaluations:" <> user_id, params, socket) do
    send(self(), {:after_join, params})
    {:ok, assign(socket, :user_id, String.to_integer(user_id))}
  end

  def handle_info({:after_join, _message}, socket) do
    user = Accounts.get_user!(socket.assigns.user_id)

    evals =
      Evaluations.list_user_vehicle_evaluations(user)
      |> Phoenix.View.render_many(VehicleEvaluationView, "evaluation.json")

    push(socket, "after_join", %{evaluations: evals})
    {:noreply, socket}
  end
end

I still need to plan out exactly what channels I will actually have, but for now, I just wanted to get an existing user’s VehicleEvaluation records.

I then added use-phoenix-channel to the React Native app and wrapped my simple App code in a SocketProvider component:

const App = (props) => {
  return (
    <SocketProvider wsUrl='ws://192.168.0.3:4000/socket' options={{}}>
      <View style={styles.container}>
        <Text style={styles.welcome}>Welcome to Bravauto!</Text>
        <EvaluationList />
      </View>
    </SocketProvider>
  )
}

Then, in my EvaluationList component, I wired up the useChannel hook:

const EvaluationList = props => {
  const initialState = { evaluations: [] }
  const [state, _broadcast] = useChannel('vehicle_evaluations:9', reducer, initialState)
  return (
    <Evaluations evaluations={state.evaluations} />
  )
}

(The ‘9’ in the channel topic is the user ID. No, it won’t ultimately work that way, but good enough for now.) I had to pass in a reducer function to handle the event passed back from the channel:

const reducer = (state, { event, payload }) => {
  switch (event) {
    case 'phx_reply':
      return state
    case 'after_join':
      return { ...state, evaluations: payload.evaluations }
    default:
      return state
  }
}

It took a bit of finagling and head-scratching, but I got it working. Eureka!

Gettin’ Testy About Tests

My test-driven spidey sense was tingling, however. I had a basic Phoenix channel on the backend and a few React Native components on the front end, without any tests! Bad developer! No biscuit!

I knocked out a basic test for the VehicleEvaluationChannel in no time, then moved on to the JavaScript tests. Boom, ReferenceError: window is not defined. The use-phoenix-channel package pulls in the Phoenix.js package (which is what handles the JavaScript end of the WebSocket communication) and something in Phoenix.js was attempting to access the DOM window object.

I foolishly spent some time trying to figure out if there was some way to make Phoenix.js work in a windowless environment (it’s a pain debugging with minimized node_modules; I wound up using the unminimized libraries directly). Ultimately, I read up on how to wire up mocks in React Native tests and created a mock of use-phoenix-channels.js that stubbed the functions I needed.

As a little aside here, I find it odd that the React Native convention is to put test files in a __tests__ directory. The underscores make it feel “hidden” or “not part of the regular code” to me. Tests are essential! On the other hand, the underscores do make the directory float to the top of my editor’s tree view, so one could argue it’s making them more visible.

I also have a philosophical objection to React “snapshot” tests. This is a popular part of the Jest testing framework. Your test renders the component and writes out a serialization of it. Then any future test runs will fail if the rendered output doesn’t match the snapshot. So, as you make changes to your component, the test will fail. You look at the snapshot’s diff and make sure the changes are what you expect, before telling Jest to update the snapshot and committing.

The first problem here is that this is a classic point of failure for a harried developer: “Ugh, I have to get this done so I can get out of here… Yep looks good, update, commit, push…“ Now, if you are in an organization that has embraced workflow best practices, you’ll be doing code reviews and the reviewer should catch an incorrect snapshot update. But in my experience, it is common that the reviewer won’t know the code as well as the author, so I would argue that the review is less likely to catch a mistake.

And beyond the practical reasons, the snapshot workflow requires changing the code and then updating the snapshot after the test fails. This is completely backward in true test-driven-development practice! You should write/update the test first.

But really, like so many things with the JavaScript world, testing in JavaScript has always been clunky, at least in my experience: heavy reliance on mocks and all too easy to let the output drive the test. It has forever been a struggle to make a solid, realistic test suite that exercised DOM-manipulating code without a bunch of parallel markup that had to be kept up-to-date with tests. At least Jest’s React tests render the actual component.

I did briefly play with React Native Testing Library so that I could write assertions about the pieces of my components. But I ditched it, at least for now, and stuck with the snapshots. It is the common practice by far and I have plenty to learn and figure out as it is. As long as I have something exercising my components and verifying their output, I’m happy. I just need to stay alert as I update those snapshots!

Rabbit Hole to China!

Once I had a few simple tests in place, I started looking at next steps for the app: user auth, er wait, no, navigation. But then I discovered something sad. I had settled into always running the app in the iOS emulator. I happened to switch to the Android emulator, and what the heck? The evaluation data is not showing up! Let’s see, no errors. Nothing in the logs. Wait, the backend isn’t even being hit! Argh.

I started debugging and found a phx_error event that I wasn’t catching (next story in Tracker: Add socket error reporting and logging!). After more digging, I figured out that Android just wouldn’t make the WebSocket call. I made the critical mistake of doing some Googling rather than just playing with that dang socket URL. I read that Android 9 banned insecure network access, or so it sounded. I found posts and bug reports talking about having to make your backend work over TLS and wire up a network_security_config.xml in the Android Manifest and upload the self-signed cert to the emulator, etc. I sank into my own little tar pit for a bit.

Luckily a helpful fellow on the Elixir Forum pointed out that Android is finicky about WebSocket URLs. It would not work with localhost or 127.0.0.1. It had to be the actual IP of your machine. Once I changed the URL to ws://192.168.0.3:4000/socket it worked! The whole “must be a secure socket” idea was a red-herring. I will need to have a secure socket for the production app, so learning how to set that up was useful. But that will be a production cert hitting a public endpoint. For now, I just wanted to keep moving with development!

NB: Android is STUPID about localhost URLs!

NB²: If you’re having a hard time finding the solution to a problem with Google, you may be looking at the wrong problem!

Hello Stress, My Old Friend

Well, this is by far my longest post, so I need to wrap it up. I did want to mention, however, that there have been other distractions while switching gears to the mobile app. The primary of these is survival! For the last five months, I have benefitted from a state program to help me financially while I get this thing going. Not enough to cover everything, but enough to slow the leak. But that ends soon and I’m officially raiding my retirement.

I have been working on establishing myself as a consultant for hire, or at least finding part time technical work to replace the funds I’ll no longer be receiving. This isn’t easy. I’ve learned that approaching companies who are advertising full-time positions about helping them part-time is pointless. It’s easy to fall into worry and start eyeing those “apply” buttons on interesting-sounding job listings. But as I mentioned in my first post, the book I read a few months ago inspired me to look beyond the standard full-time software jobs. Also, I’m concerned that I just won’t be able to find time for Bravauto if I take a full-time job.

Anyway, the next few months will tell the tale. We will see if I’m actually up for this. I should have been working much harder on developing work opportunities all along, but it’s hard to steal the time away from my passion. Who knows, maybe trying to find an investor for this thing is the right move. But I worry it would taint things and maybe even ruin it; at least before we stop crawling and start walking. Alas, I may just be hunting unicorns.