I’m very excited to announce the immediate availability of the version 6 of the Mercure Internet Draft as well as of the version 0.10 of the reference implementation!
Mercure is a real-time protocol built on top of Server-sent Events and leveraging HTTP/2+. It allows to push messages to JavaScript webapps, mobile apps or IoT devices with ease. Client-side, it relies on the native EventSource class and doesn’t require any JS dependency. Server-side, it allows to use a Hub to handle persistent connections and dispatch messages sent, for instance, from your REST or GraphQL APIs. An open source Hub designed for performance and high loads is developed in parallel to the specification. Get started with Mercure.
Since the initial release of Mercure, we gather feedback from the ever-growing community of users and contributors. This allowed us to identify the pain points, some limitations and new use cases. It took months of work to get there, but these new versions of the protocol and of the FOSS hub (written in Go) fix all the limitations and design issues we identified so far. They are by far the best and the fastest versions ever released! Let’s discover these changes and new features.
Protocol Changes
Authorization Mechanism
One of the key features of Mercure is its authorization mechanism. It allows to send private updates, which will be received only by subscribers having the appropriate credentials.
Until the last version of the protocol, this mechanism was using the concept of targets, and was hard to understand for newcomers.
In the 6th version of the protocol, the authorization mechanism has been dramatically simplified, while staying as powerful as before.
The authorization mechanism has been totally revamped and the concept of targets is gone. Now, to send a private update, the publisher must only mark it as private when sending the POST request to the hub using the new private parameter.
On the other hand, to receive a private update, a subscriber must present a JWT to the hub containing a list of topic selectors in the mercure.subscribe claim of the token.
A private update will be received by a subscriber only if:
- the topic of the update is contained in the list of topic selectors
- the topic matches an URI template present in the list
- the topic selectors list contains the special value
*, which allows to receive all updates.
More advanced use cases such as cherry-picking which updates will be received by a subscriber are also supported thanks to existing features such as alternate topics.
Also, the new concept of target selectors can also be used to subscribe to (public or private) updates, making the whole protocol more consistent.
Learn more about topic selectors and the authorization mechanism.
Presence API
One of the most requested feature was the ability to retrieve the list of currently connected subscribers. This is now possible thanks to a new web API exposed by the hub! It’s also possible for a subscriber to receive Mercure updates when other subscribers connect or disconnect.
Hubs can now expose endpoints returning the list of connected users as JSON-LD documents:
/.well-known/subscriptionsreturns the list of all active subscriptions/.well-known/subscriptions/{topic}returns the list of active subscriptions for a specific topic/.well-known/subscriptions/{topic}/{subscriber}returns details for a single subscription
Also, after having fetched the initial list of connected users, clients can subscribe (using the standard subscription mechanism) to connection status updates by using these URIs as topics.
To access to this presence API, subscribers must be authenticated and have topic selectors matched by these URIs.
Finally, arbitrary data (such as the subscriber’s username or IP address) can be attached to a subscription and retrieved by other subscribers using the new mercure.payload JWT claim.
The debug UI shipped with the reference hub now supports this new feature:
Learn more about active Mercure subscriptions.
Event Sourcing
The native features of Mercure, including its built-in history and its state reconciliation mechanism make it a convenient solution to implement Event Sourcing.
The latest version of the protocol introduces changes making dedicated to this kind of usages.
First, it is now possible to set the value of the Last-Event-ID HTTP header (or of the query parameter with the same name) to earliest to retrieve the all past events. This allows to use the Mercure hub similarly to Apache Kafka, but directly from the browser (or any other kind of HTTP client).
Additionally, when a Last-Event-ID header or parameter is set, the hub will now always return a Last-Event-ID header in the HTTP response. If it matches the requested one, it means that all requested events have been returned, otherwise, it means that the requested event may have been deleted by the hub (if it stores a limited number of events for instance), or that this event ID doesn’t exist.
Learn more about Event Sourcing and state reconciliation capabilities of Mercure.
IRIs, Strings, and Fragment Identifiers
The protocol has been made more consistent by allowing to use IRIs (recommended) or plain strings (discouraged but sometimes practical) for every identifiers: topics, event IDs, subscriber IDs…
Event IDs starting by the # character (URI fragment) are now reserved for generation by the hub. This allows a hub to use offsets generated by systems such as Apache Kafka, Apache Pulsar or Redis Streams as update identifiers while keeping these identifiers globally unique (because they are relative to the Hub URI).
Changes in the Reference Implementation
All the features described previously have already been implemented in the reference implementation (the open source hub written in Go). But that’s not all! The hub has also been dramatically improved during the past few months.
New Engine
Until version 0.10, the reference hub was using an engine working like this: when a new update was posted by a publisher, a new light-weight process (goroutine) was started and the update was pushed to every active subscriber in a loop. If a Last-Event-ID header (or parameter) was passed by a subscriber to request past events, these past events were retrieved from the persistence layer and dispatched from another goroutine.
However, this approach had several drawbacks:
- If a subscriber was slow to receive an update, it was slowing down the main loop, and so increasing the latency also for other subscribers.
- If a
Last-Event-IDwas passed, new events could be received before events coming from the history (for example, when a new event was sent from the main goroutine while another goroutine was still retrieving past events from the persistence layer). The order in which events were received wasn’t 100% guaranteed.
Mercure 0.10 contains a brand new engine that fixes these issues! This new engine leverages thoroughly the iconic features of the Go programming language: channels and goroutines. Now, two buffers are assigned to each subscribers: the first one stores events coming from the history while the second one stores “live” events. When a new event is published, the main goroutine pushes this event in the “live” buffers without waiting for events to be dispatched. Then a dedicated goroutine per subscriber pops events from the buffers associated with it. So every subscriber can receive the events as its own pace, without impacting the rest of the system.
This buffer system also fixes the order issue: goroutines dedicated to specific subscribers wait for all updates coming from the history to be dispatched before starting to dispatch updates stored in the “live” buffer.
Last but not least, this new system allows to disconnect subscribers that are too slow to receive the updates. A new config option, dispatch_timeout, has been introduced to configure the maximum time that a subscriber can take to receive an update before being disconnected to free resources.
If you’re interested in the internals, take a look at this piece of Go code!
A big thanks to Dani Kaplan for his precious help testing and debugging this new engine under high loads!
Prometheus and Health Checks Support
Prometheus is a very popular monitoring system and time series database for Cloud Native applications. Jérémy Decool made a series of great contributions that add support for Prometheus to the Mercure hub. Metrics include (among other things) the number of currently connected subscribers, the number of dispatched updates, the number of served HTTP request segregated by status code, the memory usage, the current number of processes and the number of open file descriptors!
In addition, a new URL (/healthz) is now available to check if the hub is up and running without polluting the HTTP logs with useless entries. This particularly useful for Kubernetes’ liveness and readiness probes.
Version Flag
Jérémy Decool also contributed a new --version flag allowing to know what exact version of the Hub is being used. It looks obvious and priceless, but it was a missing feature. Thanks Jérémy for adding it!
Examples and Demos
We updated the documentation as well as all existing examples (in various programming languages) to be in sync with the latest version of the spec. I also added examples and demos illustrating new features, including a chat written in JavaScript and Python:
This demonstration chat is now available online, try it!
Community
Mercure is becoming more and more popular! It recently reached 2k stars on GitHub and integrations with existing ecosystems are now plenty. For this new version, we worked with the community to keep libraries and other Mercure-related projects in sync with the latest changes in the spec. Most of them are already compatible including (but not limited to):
- the Helm chart (to install Mercure on a Kubernetes cluster)
- the Symfony integration
- the Laravel integration
- the API Platform integration
- the Dart library
For some others (for instance, the Java library and the Python library), Pull Requests are open. Finally, some tools still need to be updated including the awesome alternative Hub written in Node created by Nicolas Coutin and the Yii framework integration. Any help is very welcome!
To keep up with Mercure news, you can now follow our brand new Twitter account.
Finally, I’m very happy to announce that Les-Tilleuls.coop now provides official training for Mercure, in English and in French!
Improving Libraries Used by Mercure
One of the most powerful feature of Mercure is the ability to use URI templates to subscribe to topics matching patterns. Under the hood, the reference implementation of Mercure use the uritemplate Go library.
The performance of the Mercure hub highly depends of the performance of this library. So I worked with Kohei Yoshida (the author of this lib) to improve its performance, add benchmarks and fix some bugs! This also dramatically improved the performance of the hub (and of other projects using this library). Thank you Kohei for your quick reviews and releases!
The development of the new version of the hub also allowed me to find an issue in the Go standard library, and to open a Pull Request to fix it.
Hosted and High Availability Versions
Managed and On Premise High Availability (cluster) versions of the Mercure hub are available for a while now. They are already compatible with Mercure 0.10 and are now out of beta!
I want to thank very much our first customers, who trusted us, helped us to battle-test our hosting platform and gave us incomparable feedback allowing us to improve both the open protocol, the Managed and On Premise versions and the Open Source hub! They also allow us to invest more in the protocol’s development and in the Open Source version! So thank you again.
I also want to thank very much my GitHub sponsors, who allow me to work always more on my free software projects, including Mercure but also Vulcain, API Platform and the Symfony components I maintain.
Try and Star Mercure!
If you aren’t using Mercure yet, you can install a hub in a few seconds, or use the online demo, give it a try! If you like the project, show your support by starring it on GitHub!
1 thought on “Say Hello to Mercure 0.10!”