Managing your Dart Projects like a Jefe

Even though it’s a little premature, as there are still several things to do before backlog.io comes out of beta (such as actually hosting it on the domain backlog.io), I thought I’d kick off a series of blogs on various aspects of how the app is built.

The first blog is about jefe, a tool I’ve just now built to tame my project dependencies.




I recently decided that it was time to get more sophisticated with the way I manage the Dart projects behind backlog.io.

My project set up is split into two halves:

1. The core bit that has no dependencies on appengine. This is where 99% of the code lives
2. The appengine bit

For most of backlog.io’s brief history, these two parts were contained in two separate (somewhat randomly named) git repos. Each had several Dart projects. The structure was as follows:

• gissue – the non appengine part
  
  • gissue_common
  • gissue_client
  • gissue_server
  • gissue_integration_tests
• gitbacklog – the appengine part
  
  • gitbacklog_client
  • gitbacklog_server
  • gitbacklog_tool

Unsurprisingly, there are dependencies between these projects and these were all set up manually as path dependencies.

In turn the Dockerfile that was needed to deploy to appengine had to be carefully crafted to match.

So what’s wrong with that?

This is already somewhat complex to manage, but it gets worse when I need to change more things.

For example, I maintain several shelf packages and add features / fix bugs on these as needed for backlog.io.

So additionally my workspace might also include a structure like:

• shelfish
  
  • mojito
  • shelf_route
  • shelf_bind
  • shelf_rest
  • shelf_auth
  • …. etc

and as you’d expect, there are more path dependencies between these as well as the backlog projects, whilst developing.

Of course, my code also depends on several third party pub packages and at times I need to fork to add features or fix bugs on these.



All in all, this leads to lots of fiddling around with the dependencies in the pubspec.yaml files. As it’s a manual process it is also prone to getting it wrong, forgetting to remove path dependencies when you are done etc.

Automation to the Rescue

Clearly, the answer to alleviating the pain associated with managing these sets of related projects is to automate it.

So how do we go about that?

Pubspecs already capture the relationships between different Dart packages. However, how do we differentiate between packages which we are working on from those we simply import as is?

We need to make this distinction so that we can automate aspects that relate to only those packages that we are maintaining.

Since this is kinda augmenting the information in pubspec.yaml it seemed sensible to follow a similar style to capture these relationships.

And so jefe.yaml was born.

Breaking up is hard to do

So it was time to take the plunge and break up my projects into separate git repos and link them back together with my new yaml format.

First up gissue. I created a file called jefe.yaml which looks like:

name: gissue

projects:
  gissue_common: git@bitbucket.org:andersmholmgren/gissue_common.git
  gissue_client: git@bitbucket.org:andersmholmgren/gissue_client.git
  gissue_server: git@bitbucket.org:andersmholmgren/gissue_server.git
  gissue_integration_tests: git@bitbucket.org:andersmholmgren/gissue_integration_tests.git

The yaml is pretty straight forward. We are simply defining the set of projects that we want to manage as a group.

We can now do the same for the gitbacklog repo. The only additional concern is to capture that we also want to pull in the gissue group. We do that with a new key called groups




name: gitbacklog

groups:
  gissue: git@bitbucket.org:andersmholmgren/gissue.git

projects:
  gitbacklog_client: git@bitbucket.org:andersmholmgren/gitbacklog_client.git
  gitbacklog_server: git@bitbucket.org:andersmholmgren/gitbacklog_server.git
  gitbacklog_tool: git@bitbucket.org:andersmholmgren/gitbacklog_tool.git

Firing it up

So we’ve defined which projects we want to work on and manage as a unit. Now what can we do with it?

First we need some tools

pub global activate jefe

Let’s see what el jefe can do

> jefe -h

Description:

  Manages a set of related Dart projects

Usage:

  jefe [options] <command> [<args>]

Options:

  -h, --help    Print this usage information.

Commands:

  install             Installs a group of projects                             
  init                Installs or updates a group of projects                  
  start               Sets up for the start of development on a new feature    
  finish              Completes feature and returns to development branch      
  release             Create a release of all the projects                     
  exec                Runs the given command in all projects                   
  set-dependencies    Set dependencies between projects                        
  completion          Tab completion for this command.                         

  See 'jefe help [command]' for more information about a command.

Installing sounds like a good place to start.

> jefe help install

Description:

  Installs a group of projects

Usage:

  jefe install [options] <git-uri>

    <git-uri>    The git Uri containing the project.yaml.    

Options:

  -d, --install-directory    The directory to install into
                             (defaults to ".")

  -h, --help                 Print this usage information.

OK so let’s install the gitbacklog source. Oh it helps to be me to do this step as you won’t have permission ;-)

> jefe install git@bitbucket.org:andersmholmgren/gitbacklog.git

Once it completes we now have the following:

• gitbacklog_root
  
  • gissue_root
    
    • gissue
    • gissue_client
    • gissue_common
    • gissue_server
    • gissue_integration_tests
  • gitbacklog
  • gitbacklog_client
  • gitbacklog_server
  • gitbacklog_tool

So essentially the same as before except for the addition of the _root container directories and the gissue and gitbacklog directories that contain the yaml files for the group definitions.

Moar Interesting

OK so we now have a way to checkout a collection of Dart projects with one command. Kinda useful but not super exciting.

Lets move on to setting things up for the development of a new feature.

jefe start shiny_new_feature

So what just happened?

Firstly, lets take a look at each project’s git workspace

git branch

  develop
* feature/shiny_new_feature
  master

OK so managing the git branching is at least as easy as when we had two repos (actually easier as we can branch with one command). But not huge.

What else? Lets take a look at the pubspec.yaml files. How about gitbacklog_client



dependencies: 
  browser: '^0.10.0+2'
  polymer: '^0.16.0+7'
  gissue_client: 
    path: /mydir/gitbacklog_root/gissue_root/gissue_client
  gissue_common: 
    path: /mydir/gitbacklog_root/gissue_root/gissue_common

Correctly set up path dependencies! - noice.

And as a bit of icing on the cake, pub get was also run on all the projects for us (awthanks).

Now I gotta do some actual work

Sadly, jefe can’t write the code for the shiny new feature for me so I gotta do that myself.

OK so rather than have you wait around while I do that I’m just gonna go ahead and pretend I’ve written some code for the new feature and am ready to test it with the local appengine.

For that I have some grinder commands that integrate with jefe to automatically generate the Dockerfile and then fire up appengine. These live in the gitbacklog_tool project

grind run

The grind task for generating the Dockerfile looks like

Future genDockerfile(GrinderContext context) async {
  final currentFeatureNameOpt = await (await jefeExecutor())
      .executeOnGraph(jefe.feature.currentFeatureName());

  // use pub serve if on a feature branch
  final usePubServe = currentFeatureNameOpt is Some;

  final genDocker = jefe.docker.generateDockerfile(
      'gitbacklog_server', 'gitbacklog_client',
      dartVersion: '1.9.3',
      environment: {'USE_PUB_SERVE_IN_DEV': usePubServe},
      exposePorts: [8080, 8181, 5858],
      entryPointOptions: ['--enable-vm-service:8181/0.0.0.0']);

  await (await jefeExecutor()).executeOnGraph(genDocker);
}

I provide the command the name of my client and server projects and a bunch of other info that will go into the Dockerfile.

As I’m developing a feature, I want to run with pub serve and don’t particularly want to wait while docker creates all the images for my client and related projects, since I’m not going to use that anyway.

Well jefe knows this, so simply omits the client projects and its dependencies. The Dockerfile looks like

FROM google/dart:1.9.3
RUN apt-get update
RUN apt-get install -y ssh
ADD ["id_rsa", "/root/.ssh/id_rsa"]
RUN ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts
RUN ssh-keyscan github.com >> /root/.ssh/known_hosts
ADD ["gissue_root/gissue_common", "/Users/blah/gitbacklog_root/gissue_root/gissue_common/"]
ADD ["gissue_root/gissue_server", "/Users/blah/gitbacklog_root/gissue_root/gissue_server/"]
ADD ["gitbacklog_server/pubspec.yaml", "/Users/blah/gitbacklog_root/gitbacklog_server/pubspec.yaml"]
ADD ["gitbacklog_server/pubspec.lock", "/Users/blah/gitbacklog_root/gitbacklog_server/pubspec.lock"]
WORKDIR /Users/blah/gitbacklog_root/gitbacklog_server
RUN pub get
ADD ["gitbacklog_server", "/Users/blah/gitbacklog_root/gitbacklog_server/"]
WORKDIR /Users/blah/gitbacklog_root/gitbacklog_server
RUN pub get --offline
ENV USE_PUB_SERVE_IN_DEV true
EXPOSE 8080 8181 5858
CMD []
WORKDIR /Users/blah/gitbacklog_root/gitbacklog_server
ENTRYPOINT ["/usr/bin/dart", "--enable-vm-service:8181/0.0.0.0", "/Users/blah/gitbacklog_root/gitbacklog_server/bin/server.dart"]

Feature Complete

My shiny new feature is now complete so time to close it up.

jefe finish shiny_new_feature

If we now go back and look at the git branches and pubspec.yaml files we will notice that we are back on the develop branch and the path dependencies have been replaced by git dependencies referencing the current hash of those projects.

dependencies: 
  browser: '^0.10.0+2'
  polymer: '^0.16.0+7'
  gissue_client: 
    git: 
      ref: 942c4aa3b1fdaaadb8b95d66f02da38e01ac8ead
      url: git@bitbucket.org:andersmholmgren/gissue_client.git
  gissue_common: 
    git: 
      ref: d31aaca406c501a2a8e1bc147e3323044404d616
      url: git@bitbucket.org:andersmholmgren/gissue_common.git

This time when I run the server I want to include the client project in the Dockerfile and build it using pub build. This ensures that there are no difference with how it will run in production.

This time when we run

grind run


jefe sees that we no longer have any path dependencies so includes the client and optimises the paths. It also omits all the dependent projects as they will just add weight and slow down the build and deploy. The Dockerfile now looks like

FROM google/dart:1.9.3
RUN apt-get update
RUN apt-get install -y ssh
ADD ["id_rsa", "/root/.ssh/id_rsa"]
RUN ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts
RUN ssh-keyscan github.com >> /root/.ssh/known_hosts
ADD ["gitbacklog_server/pubspec.yaml", "/app/gitbacklog_root/gitbacklog_server/pubspec.yaml"]
ADD ["gitbacklog_server/pubspec.lock", "/app/gitbacklog_root/gitbacklog_server/pubspec.lock"]
WORKDIR /app/gitbacklog_root/gitbacklog_server
RUN pub get
ADD ["gitbacklog_server", "/app/gitbacklog_root/gitbacklog_server/"]
WORKDIR /app/gitbacklog_root/gitbacklog_server
RUN pub get --offline
ADD ["gitbacklog_client/pubspec.yaml", "/app/gitbacklog_root/gitbacklog_client/pubspec.yaml"]
ADD ["gitbacklog_client/pubspec.lock", "/app/gitbacklog_root/gitbacklog_client/pubspec.lock"]
WORKDIR /app/gitbacklog_root/gitbacklog_client
RUN pub get
ADD ["gitbacklog_client", "/app/gitbacklog_root/gitbacklog_client/"]
WORKDIR /app/gitbacklog_root/gitbacklog_client
RUN pub get --offline
RUN pub build
ENV USE_PUB_SERVE_IN_DEV false
EXPOSE 8080 8181 5858
CMD []
WORKDIR /app/gitbacklog_root/gitbacklog_server
ENTRYPOINT ["/usr/bin/dart", "--enable-vm-service:8181/0.0.0.0", "/app/gitbacklog_root/gitbacklog_server/bin/server.dart"]

Deploy All The Things

So we’ve tested that our feature is working as planned and want to get it to users.

First off we cut a release

jefe release

This bumps the versions of all the projects, merges to master and tags the commits.
Just to make sure all is as we expect, we can now run exactly the code that we will in production

grind runProd

This uses a slightly different command to generate the production Dockerfile. The set up is pretty much the same as for the previous though so I’ll omit here. The resulting Dockerfile is now

FROM google/dart:1.9.3
RUN apt-get update
RUN apt-get install -y ssh
ADD ["id_rsa", "/root/.ssh/id_rsa"]
RUN ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts
RUN ssh-keyscan github.com >> /root/.ssh/known_hosts
RUN git clone -q -b 0.12.0 git@bitbucket.org:andersmholmgren/gitbacklog_server.git /app/gitbacklog_server
WORKDIR /app/gitbacklog_server
RUN pub get
RUN git clone -q -b 0.15.5 git@bitbucket.org:andersmholmgren/gitbacklog_client.git /app/gitbacklog_client
WORKDIR /app/gitbacklog_client
RUN pub get
RUN pub build
ENV USE_PUB_SERVE_IN_DEV false
EXPOSE 8080 8181 5858
CMD []
ENTRYPOINT ["/usr/bin/dart", "--enable-vm-service:8181/0.0.0.0", "/app/gitbacklog_server/bin/server.dart"]

Very similar to the previous Dockerfile except it is now cloning the client and server repos to make sure that we are releasing off a tag committed to our remote repository.

Another grinder task does the actual deploy

grind deploy

Happy Days

This is making my life much easier when managing my codebase and I hope it will be helpful for others.

If so then you may want to consider contributing to jefe.

Da Future

One of the things I want to tackle soon is adding support for hosted packages. In that case a jefe release would also publish to pub and change the dependencies to hosted dependencies rather than git.
May be able to automate some tasks with the changelog too (although that could be done for git projects too).
Let me know what you think.