.. post:: 2014-02-05
   :tags: OCaml
   :author: Rudi Grinberg

Benchmarking OCaml Json Libraries
=================================

According to `opam <http://opam.ocaml.org/pkg/index-popularity.html>`__
OCaml has 2 popular libraries for parsing json:

-  `Yojson <https://github.com/mjambon/yojson>`__
-  `Jsonm <https://github.com/dbuenzli/jsonm>`__,
   `Ezjsonm <https://github.com/samoht/ezjsonm>`__

At the time of writing, Yojson is actually the 6th most popular OCaml
library overall!

All things being equal we'd like to use the fastest json library.
Unfortunately for us, things aren't equal. In fact, ``Jsonm`` strives to
be extremely minimalist and low level and the API that it provides
non-blocking encoding and decoding.

Therefore, we will instead be benchmarking ``Yojson`` against
``Ezjsonm``. ``Ezjsonm`` is a wrapper library designed provide a user
friendly API on top of ``Jsonm``. With this wrapper,
serialization/deserialization code looks almost identical with both
libraries. In my benchmarks I test the speed in which I read/write the
following datatype:

.. code-block:: ocaml

    module Event_type = struct
      type t =
        | Login
        | Purchase
        | Logout
        | Cancel
    end
    type event = {
      username: string;
      date: int;
      event_type: Event_type.t;
      payload: string;
    }

This is a simple data type, however it's representative of at least some
of the applications json is used for.

Results
-------

In my benchmarks I've opted for 2 simple tests:

-  How fast can a list of a 1000 ``event`` types be transformed to a big
   json list written to an array

-  How fast can a json string of a 1000 ``event``\ s be transformed into
   ``event list``

The tests are done using the excellent ``Core_bench`` library and here
is the "pretty" output for the first test:

::

     ─────────────────────────┬──────────┬────────────┬──────────┬──────────┬────────────┐
    │ Name                    │ Time/Run │    mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │
    ├─────────────────────────┼──────────┼────────────┼──────────┼──────────┼────────────┤
    │ ezjsonm write           │   2.67ms │ 1_668.16kw │ 124.34kw │  41.75kw │    100.00% │
    │ yojson write            │   1.24ms │    61.27kw │  86.60kw │   4.27kw │     46.35% │
    └─────────────────────────┴──────────┴────────────┴──────────┴──────────┴────────────┘

For the second test:

::

    ┌──────────────┬──────────┬────────────┬──────────┬──────────┬────────────┐
    │ Name         │ Time/Run │    mWd/Run │ mjWd/Run │ Prom/Run │ Percentage │
    ├──────────────┼──────────┼────────────┼──────────┼──────────┼────────────┤
    │ ezjsonm read │   5.47ms │ 1_857.55kw │  67.00kw │  67.00kw │    100.00% │
    │ yojson read  │   2.04ms │   155.36kw │  39.14kw │  22.10kw │     37.30% │
    └──────────────┴──────────┴────────────┴──────────┴──────────┴────────────┘

The verdict is: ``Yojson`` is at least twice as fast as ``Ezjsonm`` in
both benchmarks.

Of course, recall that benchmarks often lie (on behalf of their author)
and it's entirely possible that this difference can be attributed to my
code. Therefore, I welcome you to go over my code
`here <https://github.com/rgrinberg/ocaml-json-benchmark>`__ and draw
your own conclusions.

Even in the case where I did not make a mistake in benchmarking, don't
let me turn you off from using ``Jsonm``, which allows you to write json
parsers at a very low level. I'm certain that a specialized
reader/writer in ``Jsonm`` will easily beat ``Yojson``.

Future work
-----------

This set of tests is not enough to draw any conclusions just yet. Here's
what I'd like to do in the future:

-  OCaml has more implementations of the json serialization format. They
   are uncommonly used, however it would still be nice to include them:
   ``yajl``, ``tiny_json``.

-  Test serializers/deserializer that are automatically generated from
   type definitions: ``atdgen``, ``cow``, ``deriving-yojson``

-  See how well a hand rolled ``Jsonm`` reader/writer fairs.

-  Add more tests for different use cases, data types, write to
   channels, not just strings.