Optional Dependencies Considered Harmful¶
This will be a short PSA to opam package maintainers to avoid spurious optional dependencies. At this point, I think this is all relatively common knowledge. But open source maintainers are as a rule busy people, and without much encouragement, they end up dragging their feet. Therefore I hope this post can be a useful reminder of the negative effects of optional dependencies and what can be done to avoid them.
Recognizing Illegitimate Optional Dependencies¶
First I’d like to point that not all optional dependencies are created equal. Some optional dependencies are in fact harmless and completely legitimate. For example, iocaml has an optional dependency on ocp-index to provide code completion when ocp-index is available. The distinction to note here is the fact that this is truly optional functionality of the package and not just an additional set of interfaces.
On the other hand illegitimate optional dependencies are usually sub-libraries
or sub-packages masquerading as optional dependencies. There’s no need to point
fingers at specific packages here - the examples are numerous, as a rule,
anything that optionally depends on lwt, async, ppx_tools. Or
anything that provides an optional findlib subpackage. I’ll use cohttp for my
examples below, as it’s a package I help maintain and experienced all the
typical problems of optional dependencies.
Why are Illegitimate Optional Dependencies Harmful?¶
I’ll summarize all the disadvantages of depopts for sub libraries by appealing to three principles:
User friendliness
A preference towards conservative and anti-fragile implementation
Maximizing the correctness and precision of the metadata provided to opam
And if the problems I’m about to describe don’t sound so bad on their own, it should be remembered that the worse situations are created when more than one of these principles is violated simultaneously.
Depopts are User Hostile¶
This position took a while for me to develop. As depopts themselves sound like a relatively simple and harmless feature, and I have enough experience to even claim that I’m familiar and comfortable with their effects. But I’ve had to deal with many new users over the years. And just in the context of cohttp, I encountered dozens of variations on the following questions from new users:
Where do I find cohttp for Async/Lwt?
I’ve installed cohttp, why can’t I use it for Async/Lwt?
How do I enable SSL for cohttp?
All of these questions require fiddling with depopts, and all of them end up surprisingly difficult for new users to answer on their own. While I’m not recommending that we optimize everything towards new users. The current convention clearly ends up consuming a lot of a maintainer’s time. The fact that these optional dependencies are usually absent in other package managers is perhaps telling of how unfamiliar this method is. In any case, whether it’s the fault of the user or not, the principle of least surprised seems to be heavily violated here.
Finally, depopts aren’t as discoverable as proper opam packages. A user looking
for new functionality will intuitively reach for $ opam search more often
than not. This gives us a hint that more opam packages as a solution. More on
that later, but for now let’s just note that increasing the number of opam
packages “for free” is a positive thing by itself.
Depopts Cause Unnecessary Reinstalls and Build Failures¶
A package that is an optional dependency of another package tends to be an optional dependency for other packages. Consider Lwt for example. But just be cause I’d like to use cohttp along with lwt, doesn’t mean I want to compile a dozen of other packages that also have Lwt as a depopt. At best, this wastes a lot of pointless cycles. At worst it introduces new build failures that leave your switch in a busted state. I know that this shouldn’t happen due to build constraints. But reality begs to differ.
Depopts Result in Over Determined Constraints¶
Consider the following depopts snippet:
depopts: ["async" "lwt" "js_of_ocaml"]
conflicts: [
"async" {< "113.24.00"}
"lwt" {< "3.0.0"}
"js_of_ocaml" {< "3.0.0"}
]
This would the hypothetical depopts/conflicts specification for the next version
of cohttp if we wanted to use jsoo’s ocaml-migrate-parsetree based ppx, and only stay
compatible with the new version of Lwt. Because we wanted to use the new non
blocking version of Lwt_unix.bind for example.
These constraints are clearly orthogonal and yet they have an unexpected side effect on cohttp-js 1 users. While those users can easily tolerate an older Lwt in their application, cohttp will now force them to these upgrades at once.
Now imagine if js_of_ocaml 3.0 conflicted with Lwt 3.0. Though awkward situations like this might be rare, they have occurred to me before and I’m sure will occur to the unsuspecting user.
What to do?¶
Don’t use depopts to specify sub libraries. Instead, use opam’s ability to define multiple opam packages out of a single repository. Yes, I know that given the build system situation we’re currently in, this is usually easier said than done. Nevertheless, it can be accomplished pretty easily if you’re using one of the following build systems:
And here’s an example of a project accomplishing this using each of these build systems:
js_of_ocaml - Since 3.0 only. Which is unreleased as of writing this post
tyxml - doesn’t separate the build/install steps properly but is the easiest path to take if you’re using oasis
As for myself, I will transition the packages I maintain to jbuilder. Consider this a sneak preview of what to expect from the next version of cohttp.
What not to do?¶
Do not under any circumstances take this as a call to split your packages into multiple git repositories. The majority of sub libraries don’t need their own bug trackers and git histories. You will only confuse your users and sometimes even fail to synchronize your releases properly and create a real nightmare for your users. But more on that another day.
- 1
cohttp-js is not yet a real package. But a cohttp-js-lwt package is planned for the next release.