jbuilder (dune) Beta 17

The 17th beta of jbuilder represents a few months of development. While that’s a bit longer than our usual release cycle for these betas, we do have a larger release than usual. So I’ve decided to write up a little post in addition to just posting the usual change log. I’ll talk about some important new features, and some less important ones as well. A couple of things that I’ll omit are bug fixes and experimental features as this blog post is already long.

Let me start off by reminding everyone that jbuilder will be renamed to dune. The renaming will be done before the release of 1.0. Although the details of the transition are still under discussion, we will try to make the transition as painless as possible for users. A tool that will automatically migrate your projects to dune will be available. This is the last planned beta before the 1.0 release.

Small Usability Improvements

Although dune is a fully featured build system, it’s still relatively young and has a few unpolished corners remaining. We tried to address some of these in the release to avoid the dreaded death by a thousand cuts. Here’s a selection of some of the improvements we’ve made.

Path Related Improvements $ jbuilder {exec,utop}

We improved $ jbuilder exec to completely replace the need for symlinks for executing built binaries (like in ocamlbuild).

Previously, exec would be limited to executing installable (public) binaries. Now dune is able to execute local binaries as well, by passing a relative path to the binary. E.g. if you’d like to execute a binary defined with:

(executable
 ((name foo)))

You can do it with:

$ jbuilder exec ./foo.exe

If the binary is in a sub directory like src. Then you can either give the full relative path from the build root, or a path relative to your CWD.

$ cd src && jbuilder exec ./foo.exe
# works like in the original dir
$ jbuilder exec ./src/foo.exe

Furthermore, exec will now make sure the binary is rebuilt before executing it. Those of you who are used to doing something like:

$ jbuilder build $X && jbuilder exec $X

can rejoice at the key savings. Those who don’t like this new behavior, can disable it with the --no-build flag. I’ll also take this opportunity to remind everyone that exec supports passing arguments to the executed binary by forwarding all the arguments after -- to it. This is a standard feature which is supported everywhere in dune where it makes sense.

Finally, exec can also execute arbitrary binaries in the install environment by providing it an absolute path:

$ jbuilder exec /bin/ls

This example isn’t very useful, as you’re unlikely to have a public ls binary, but you get the idea.

We’ve also done a little fix to $ jbuilder utop related to CWD handling. Previously, launching a top level when your CWD wasn’t the build root wouldn’t quite work.

$ cd src && jbuilder utop .

Now this works the same as $ jbuilder utop src.

Running Tests

A common complaint by users has been that dune is sometimes a little too smart for its own good when running tests. By default, $ jbuilder runtest would rerun only the tests whose dependencies have been modified. This is desirable most of the time, but can still be controlled with the new --force option. Switching on this flag will make sure all the test binaries will rerun.

The --force is actually more general than just for re-running tests. Recall that $ jbuilder runtest just builds the @runtest alias. So we can actually force the building of any alias using the --force flag.

Smarter Variables

Dune allows users to refer to various variables when defining custom rules (to those who are familiar here’s the list). A good chunk of them correspond to a subset of $ ocamlc -config.

These variables end up being quite useful for writing correct rules. For example, here’s an example of using the ${CC} variable to have a custom rule for compiling C code (conceivably useful for compiling and running a bit of C code as part of your build):

(rule
 ((targets (foo.exe))
  (deps    (foo.c))
  (action  (run ${CC} -o ${@} ${<} -lfoolib))))

The example above will only work for dune >= 1.0.0+beta17. Previous versions would treat all of these variables as raw strings, which meant that if ${CC} would contain flags, then dune would look for an invalid binary and fail. Now dune is smarter about these variables and understand which variables are just strings which are lists.

To get back the old behavior, where the variable is treated as a string, is just as easily done by quoting it with "${CC}".

Diffing & Promotion

Diffing & Promotion is dune’s neat little answer to working with build artifacts that we’d like to check in and version control in our project. It’s a small feature with wide implications, and introduces a style of work that’s perhaps a bit unfamiliar to many developers. I will present it by using a couple of case studies.

Avoiding OCaml Syntax in jbuild Files

Advanced users might be aware that dune allows you to write OCaml code that will generate a jbuild file on the fly, rather than use the jbuild file syntax. This feature is not recommended, and should only be used as a last resort. The new promotion feature in conjunction with the static include (which I’ll describe shortly) gives a way to avoid its use in situations where we’re just using it to generate boilerplate.

As an example, consider this jbuild file in the Camomile library. We’d like to replace the tuareg syntax with a normal jbuild file and a mechanism to regenerate it when necessary.

First, let me introduce the new (include ..) stanza:

(include <path>)

This stanza will include the source of <path> as part of our source for this jbuild file. <path> is restricted to be a static path that must be present in our source tree.

In conjunction with promotion, this can be used to replace the tuareg syntax:

(include jbuild.inc)

(rule (with-stdout-to jbuild.inc.gen (run ./gen-jbuild.exe)))

Where gen_build.exe will be the binary that will generate our jbuild.inc.gen (repurposed from the jbuild syntax)

We tie this together by defining an alias to do the promotion using the new diff action:

(alias
 ((name   jbuild)
  (action (diff jbuild.inc.gen jbuild.inc))))

We can now update our generated jbuild files with:

$ jbuilder build @jbuild --auto-promote

This will make sure the generated files are up to date before copying them to the source.

Correction Files

Another model for updating generated sources is by having the generators write so called corrected (or correction) files. Such a corrected file is the most up to date version of the generated source. It can be diffed against the current version, and eventually accepted by the user.

I’ll motivate the use case with toplevel_expect_test and demonstrate how dune supports such a workflow with the diff? stanza.

Here’s a primitive toplevel_expect_test test (foo.ml):

let x = 1 + 'e'
[%%expect{|
Line _, characters 12-15:
Error: This expression has type char but an expression was expected of type
        int
|}]

The basic idea is that the ocaml-topexpect binary will take this source and evaluate every statement in the toplevel. It will make sure that the output from every evaluated statement matches the contents of the subsequent attribute. When the output indeed matches, the test is considered to be passed, otherwise the test fails.

The great thing about expect tests is that they’re far more useful in the failure case. On failure, ocaml-expect will generate a corrected foo.ml where every %%expect attribute has been updated to make the tests pass and a helpful diff between the two foo.ml’s. If we’re happy with the new results of our tests, we can accept them and overwrite our own foo.ml with the corrected one.

How do we make this work with dune? Here’s a simple stanza to run the expect tests above:

(alias
 ((name topexpect)
  (deps (foo.ml))
  (action
   (progn
    (bash "ocaml-expect ${<} || true")
    (diff? foo.ml foo.ml.corrected)))))

A couple of things to note here:

• we need a silly hack to ignore the exit code from ocaml-expect. This is because ocaml-expect will return a non zero error code indicating failure in addition to a corrected file. Dune only expects the corrected file.

• We’ve used a diff? rather than a diff action. This action is tailored for commands that produce a new version of the source only when a difference exists, and treat the failure to generate a new source as success - exactly what we expect with corrected files.

To run the tests, we have our @topexpect alias:

$ jbuilder build @topexpect

If there are differences and we are satisfied by them, we can promote the corrected files:

$ jbuilder promote

toplevel_expect_test based tests might not seem very useful, but the workflow is quite general and will also work for ppx_expect and linting. I’ll talk more about those another time, when those features are a bit more ready.

Cross Compilation

The last notable feature we’ve added is the ability to cross compile dune projects. This is an important feature to those who wish to build binaries for Android, iOS, Windows, etc. on their development or CI boxes. This is quite a large feature, so we’ll save the details for another blog post. For now I’ll just link the official documentation for this feature:

http://jbuilder.readthedocs.io/en/latest/advanced-topics.html#cross-compilation

Credits

I’d like to thank http://ocamllabs.io/, the rest of the dune team, all the external contributors, and everyone who reported bugs or made suggestions.

Being an external contributor can be quite tough. So as a special thanks, I’d like to list all external contributors that have helped out in this release, and invite anyone else to come help with the next version of dune.

Here are github usernames all the external contributors that participated in this release:

Chris00, MiloDavis, seliopou, aantron, xclerc, hhugo.

Extension Points - 3 Years Later   Configurator Reborn

Comments

comments powered by Disqus