.. post:: 2018-01-23 :tags: OCaml, jbuilder, dune jbuilder (dune) Beta 17 ======================= The 17th beta of jbuilder represents a few months of development. While that's a bit longer than our usual release cycle for these betas, we do have a larger release than usual. So I've decided to write up a little post in addition to just posting the usual change log. I'll talk about some important new features, and some less important ones as well. A couple of things that I'll omit are bug fixes and experimental features as this blog post is already long. Let me start off by reminding everyone that jbuilder will be renamed to dune. The renaming will be done before the release of 1.0. Although the details of the transition are still under discussion, we will try to make the transition as painless as possible for users. A tool that will automatically migrate your projects to dune will be available. This is the last planned beta before the 1.0 release. Small Usability Improvements ---------------------------- Although dune is a fully featured build system, it's still relatively young and has a few unpolished corners remaining. We tried to address some of these in the release to avoid the dreaded death by a thousand cuts. Here's a selection of some of the improvements we've made. Path Related Improvements ``$ jbuilder {exec,utop}`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We improved ``$ jbuilder exec`` to completely replace the need for symlinks for executing built binaries (like in ocamlbuild_). Previously, ``exec`` would be limited to executing installable (public) binaries. Now dune is able to execute local binaries as well, by passing a relative path to the binary. E.g. if you'd like to execute a binary defined with: .. code-block:: lisp (executable ((name foo))) You can do it with: :: $ jbuilder exec ./foo.exe If the binary is in a sub directory like ``src``. Then you can either give the full relative path from the build root, or a path relative to your CWD. :: $ cd src && jbuilder exec ./foo.exe # works like in the original dir $ jbuilder exec ./src/foo.exe Furthermore, ``exec`` will now make sure the binary is rebuilt before executing it. Those of you who are used to doing something like: :: $ jbuilder build $X && jbuilder exec $X can rejoice at the key savings. Those who don't like this new behavior, can disable it with the ``--no-build`` flag. I'll also take this opportunity to remind everyone that ``exec`` supports passing arguments to the executed binary by forwarding all the arguments after ``--`` to it. This is a standard feature which is supported everywhere in dune where it makes sense. Finally, ``exec`` can also execute arbitrary binaries in the `install` environment by providing it an absolute path: :: $ jbuilder exec /bin/ls This example isn't very useful, as you're unlikely to have a public ``ls`` binary, but you get the idea. We've also done a little fix to ``$ jbuilder utop`` related to CWD handling. Previously, launching a top level when your CWD wasn't the build root wouldn't quite work. :: $ cd src && jbuilder utop . Now this works the same as ``$ jbuilder utop src``. Running Tests ~~~~~~~~~~~~~ A common complaint by users has been that dune is sometimes a little too smart for its own good when running tests. By default, ``$ jbuilder runtest`` would rerun only the tests whose dependencies have been modified. This is desirable most of the time, but can still be controlled with the new ``--force`` option. Switching on this flag will make sure all the test binaries will rerun. The ``--force`` is actually more general than just for re-running tests. Recall that ``$ jbuilder runtest`` just builds the ``@runtest`` alias. So we can actually force the building of any alias using the ``--force`` flag. Smarter Variables ~~~~~~~~~~~~~~~~~ Dune allows users to refer to various variables when defining custom rules (to those who are familiar here's the list_). A good chunk of them correspond to a subset of ``$ ocamlc -config``. These variables end up being quite useful for writing correct rules. For example, here's an example of using the ``${CC}`` variable to have a custom rule for compiling C code (conceivably useful for compiling and running a bit of C code as part of your build): .. code-block:: lisp (rule ((targets (foo.exe)) (deps (foo.c)) (action (run ${CC} -o ${@} ${<} -lfoolib)))) The example above will only work for ``dune >= 1.0.0+beta17``. Previous versions would treat all of these variables as raw strings, which meant that if ``${CC}`` would contain flags, then dune would look for an invalid binary and fail. Now dune is smarter about these variables and understand which variables are just strings which are lists. To get back the old behavior, where the variable is treated as a string, is just as easily done by quoting it with ``"${CC}"``. Diffing & Promotion ------------------- Diffing & Promotion is dune's neat little answer to working with build artifacts that we'd like to check in and version control in our project. It's a small feature with wide implications, and introduces a style of work that's perhaps a bit unfamiliar to many developers. I will present it by using a couple of case studies. Avoiding OCaml Syntax in ``jbuild`` Files ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Advanced users might be aware that dune allows you to write OCaml code that will generate a ``jbuild`` file on the fly, rather than use the jbuild file syntax. This feature is not recommended, and should only be used as a last resort. The new promotion feature in conjunction with the static ``include`` (which I'll describe shortly) gives a way to avoid its use in situations where we're just using it to generate boilerplate. As an example, consider this jbuild_ file in the Camomile library. We'd like to replace the tuareg syntax with a normal jbuild file and a mechanism to regenerate it when necessary. First, let me introduce the new ``(include ..)`` stanza: .. code-block:: lisp (include ) This stanza will include the source of ```` as part of our source for this jbuild file. ```` is restricted to be a static path that must be present in our source tree. In conjunction with promotion, this can be used to replace the tuareg syntax: .. code-block:: lisp (include jbuild.inc) (rule (with-stdout-to jbuild.inc.gen (run ./gen-jbuild.exe))) Where ``gen_build.exe`` will be the binary that will generate our ``jbuild.inc.gen`` (repurposed from the jbuild syntax) We tie this together by defining an alias to do the promotion using the new ``diff`` action: .. code-block:: lisp (alias ((name jbuild) (action (diff jbuild.inc.gen jbuild.inc)))) We can now update our generated ``jbuild`` files with: :: $ jbuilder build @jbuild --auto-promote This will make sure the generated files are up to date before copying them to the source. Correction Files ~~~~~~~~~~~~~~~~ Another model for updating generated sources is by having the generators write so called *corrected* (or *correction*) files. Such a *corrected* file is the most up to date version of the generated source. It can be diffed against the current version, and eventually accepted by the user. I'll motivate the use case with toplevel_expect_test_ and demonstrate how dune supports such a workflow with the ``diff?`` stanza. Here's a primitive toplevel_expect_test_ test (``foo.ml``): .. code-block:: ocaml let x = 1 + 'e' [%%expect{| Line _, characters 12-15: Error: This expression has type char but an expression was expected of type int |}] The basic idea is that the ``ocaml-topexpect`` binary will take this source and evaluate every statement in the toplevel. It will make sure that the output from every evaluated statement matches the contents of the subsequent attribute. When the output indeed matches, the test is considered to be passed, otherwise the test fails. The great thing about expect tests is that they're far more useful in the failure case. On failure, ``ocaml-expect`` will generate a *corrected* ``foo.ml`` where every ``%%expect`` attribute has been updated to make the tests pass and a helpful diff between the two foo.ml's. If we're happy with the new results of our tests, we can accept them and overwrite our own foo.ml with the corrected one. How do we make this work with dune? Here's a simple stanza to run the expect tests above: .. code-block:: lisp (alias ((name topexpect) (deps (foo.ml)) (action (progn (bash "ocaml-expect ${<} || true") (diff? foo.ml foo.ml.corrected))))) A couple of things to note here: * we need a silly hack to ignore the exit code from ``ocaml-expect``. This is because ocaml-expect will return a non zero error code indicating failure in addition to a corrected file. Dune only expects the corrected file. * We've used a ``diff?`` rather than a ``diff`` action. This action is tailored for commands that produce a new version of the source only when a difference exists, and treat the failure to generate a new source as success - exactly what we expect with corrected files. To run the tests, we have our ``@topexpect`` alias: :: $ jbuilder build @topexpect If there are differences and we are satisfied by them, we can promote the corrected files: :: $ jbuilder promote toplevel_expect_test_ based tests might not seem very useful, but the workflow is quite general and will also work for ppx_expect_ and linting. I'll talk more about those another time, when those features are a bit more ready. Cross Compilation ----------------- The last notable feature we've added is the ability to cross compile dune projects. This is an important feature to those who wish to build binaries for Android, iOS, Windows, etc. on their development or CI boxes. This is quite a large feature, so we'll save the details for another blog post. For now I'll just link the official documentation for this feature: http://jbuilder.readthedocs.io/en/latest/advanced-topics.html#cross-compilation Credits ======= I'd like to thank http://ocamllabs.io/, the rest of the dune team, all the external contributors, and everyone who reported bugs or made suggestions. Being an external contributor can be quite tough. So as a special thanks, I'd like to list all external contributors that have helped out in this release, and invite anyone else to come help with the next version of dune. Here are github usernames all the external contributors that participated in this release: Chris00_, MiloDavis_, seliopou_, aantron_, xclerc_, hhugo_. .. _list: http://jbuilder.readthedocs.io/en/latest/jbuild.html#variables-expansion .. _jbuild: https://github.com/rgrinberg/Camomile/blob/rel-0.8.6/Camomile/locales/jbuild .. _ppx_expect: https://github.com/janestreet/ppx_expect .. _toplevel_expect_test: https://github.com/janestreet/toplevel_expect_test .. _ocamlbuild: https://github.com/ocaml/ocamlbuild .. _Chris00: https://github.com/chris00 .. _MiloDavis: https://github.com/MiloDavis .. _seliopou: https://github.com/seliopou .. _aantron: https://github.com/aantron .. _xclerc: https://github.com/xclerc .. _hhugo: https://github.com/hhugo