.. post:: 2017-04-22
:tags: OCaml, OPAM
:author: Rudi Grinberg
Optional Dependencies Considered Harmful
========================================
This will be a short PSA to opam package maintainers to avoid spurious optional
dependencies. At this point, I think this is all relatively common knowledge.
But open source maintainers are as a rule busy people, and without much
encouragement, they end up dragging their feet. Therefore I hope this post can
be a useful reminder of the negative effects of optional dependencies and what
can be done to avoid them.
Recognizing Illegitimate Optional Dependencies
----------------------------------------------
First I'd like to point that not all optional dependencies are created equal.
Some optional dependencies are in fact harmless and completely legitimate. For
example, `iocaml `__ has an optional
dependency on `ocp-index `__ to provide
code completion when ocp-index is available. The distinction to note here is the
fact that this is truly `optional` functionality of the package and not just an
additional set of interfaces.
On the other hand illegitimate optional dependencies are usually sub-libraries
or sub-packages masquerading as optional dependencies. There's no need to point
fingers at specific packages here - the examples are numerous, as a rule,
anything that optionally depends on ``lwt``, ``async``, ``ppx_tools``. Or
anything that provides an optional findlib subpackage. I'll use cohttp for my
examples below, as it's a package I help maintain and experienced all the
typical problems of optional dependencies.
Why are Illegitimate Optional Dependencies Harmful?
---------------------------------------------------
I'll summarize all the disadvantages of depopts for sub libraries by appealing
to three principles:
* User friendliness
* A preference towards conservative and anti-fragile implementation
* Maximizing the correctness and precision of the metadata provided to opam
And if the problems I'm about to describe don't sound so bad on their own, it
should be remembered that the worse situations are created when more than one of
these principles is violated simultaneously.
Depopts are User Hostile
~~~~~~~~~~~~~~~~~~~~~~~~
This position took a while for me to develop. As depopts themselves sound like a
relatively simple and harmless feature, and I have enough experience to even
claim that I'm familiar and comfortable with their effects. But I've had to deal
with many new users over the years. And just in the context of cohttp, I
encountered dozens of variations on the following questions from new users:
* Where do I find cohttp for Async/Lwt?
* I've installed cohttp, why can't I use it for Async/Lwt?
* How do I enable SSL for cohttp?
All of these questions require fiddling with depopts, and all of them end up
surprisingly difficult for new users to answer on their own. While I'm not
recommending that we optimize everything towards new users. The current
convention clearly ends up consuming a lot of a maintainer's time. The fact that
these optional dependencies are usually absent in other package managers is
perhaps telling of how unfamiliar this method is. In any case, whether it's the
fault of the user or not, the principle of least surprised seems to be heavily
violated here.
Finally, depopts aren't as discoverable as proper opam packages. A user looking
for new functionality will intuitively reach for ``$ opam search`` more often
than not. This gives us a hint that more opam packages as a solution. More on
that later, but for now let's just note that increasing the number of opam
packages "for free" is a positive thing by itself.
Depopts Cause Unnecessary Reinstalls and Build Failures
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A package that is an optional dependency of another package tends to be an
optional dependency for other packages. Consider Lwt for example. But just be
cause I'd like to use cohttp along with lwt, doesn't mean I want to compile a
dozen of other packages that also have Lwt as a depopt. At best, this wastes a
lot of pointless cycles. At worst it introduces new build failures that leave
your switch in a busted state. I know that this shouldn't happen due to build
constraints. But reality begs to differ.
Depopts Result in Over Determined Constraints
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Consider the following ``depopts`` snippet:
::
depopts: ["async" "lwt" "js_of_ocaml"]
conflicts: [
"async" {< "113.24.00"}
"lwt" {< "3.0.0"}
"js_of_ocaml" {< "3.0.0"}
]
This would the hypothetical depopts/conflicts specification for the next version
of cohttp if we wanted to use jsoo's `ocaml-migrate-parsetree
`__ based ppx, and only stay
compatible with the new version of Lwt. Because we wanted to use the new non
blocking version of ``Lwt_unix.bind`` for example.
These constraints are clearly orthogonal and yet they have an unexpected side
effect on cohttp-js [1]_ users. While those users can easily tolerate an older
Lwt in their application, cohttp will now force them to these upgrades at once.
Now imagine if js_of_ocaml 3.0 conflicted with Lwt 3.0. Though awkward
situations like this might be rare, they have occurred to me before and I'm sure
will occur to the unsuspecting user.
What to do?
-----------
Don't use depopts to specify sub libraries. Instead, use opam's ability to
define multiple opam packages out of a single repository. Yes, I know that given
the build system situation we're currently in, this is usually easier said than
done. Nevertheless, it can be accomplished pretty easily if you're using one of
the following build systems:
* `jbuilder `__
* `topkg `__
* `oasis `__
And here's an example of a project accomplishing this using each of these build
systems:
* `js_of_ocaml `__ - Since 3.0 only.
Which is unreleased as of writing this post
* `topkg `__
* `tyxml `__ - doesn't separate the
build/install steps properly but is the easiest path to take if you're using
oasis
As for myself, I will transition the packages I maintain to jbuilder. Consider
this a sneak preview of what to expect from the next version of cohttp.
What not to do?
---------------
Do not under any circumstances take this as a call to split your packages into
multiple git repositories. The majority of sub libraries don't need their own
bug trackers and git histories. You will only confuse your users and sometimes
even fail to synchronize your releases properly and create a real nightmare for
your users. But more on that another day.
.. [1] cohttp-js is not yet a real package. But a cohttp-js-lwt package is
planned for the next release.