Recursive Message Broker

This is a sketch of a recursive messaging protocol, broker, and client libraries, inspired by AMQP 0-91, PubSubHubBub, STOMP and reversehttp. It's quite different to AMQP 1.0 (but it may be instructive to compare the two approaches).

Currently, the project includes

• a server (written in OCaml), with adapters for
  • Hop's own protocol,
  • a subset of AMQP 0-9-1, and
  • XHR streaming of messages to and from the broker.
• a web-based console for the server
• an OSX GUI for the server
• messaging for the web
• client libraries for various languages (Java, Racket, Javascript)

A sketch?

Honestly, not meant to be production software... yet.

Background

Messaging à la AMQP 0-91 can be broken down into a few core pieces:

• transmission and receipt of messages (publishes, deliveries, and gets)

• subscription management (subsuming enrollment, bindings, consumers and relays)

• directory (naming of resources in the broker)

• object management (creation and destruction of remote resources)

AMQP itself, being a first mover in its space, isn't as orthogonal as it could be. It can be greatly simplified without losing anything of value. This experiment is intended to demonstrate one possible way of paring each of the core pieces of AMQP-style messaging back to their essences.

More detail

TBD.

• what recursive means in this context

• doing things this way gives you shovels (relays) for free

• and effortless interop with legacy messaging networks (including UDP, SMTP, IMAP, HTTP etc)

• and effortless federation

• and a big step closer to a sensible semantics for transactions

• relays (including active client connections!) are just nodes in the network, addressable like any other - so (post! somerelay (post! someremotenode ...)) and so on is the way to cause things to happen remotely.

Compiling the server

The server is written in OCaml. To build and run the server, you will need:

• OCaml itself, version 3.12 or newer
• OCaml Findlib; I have used 1.2.8 and 1.3.1, but older versions may well work
• libev installed somewhere that Findlib can find it
• python to generate parts of the protocol codecs

Make sure you have ocamlopt, ocamlbuild, ocamlfind etc. on your path. Then, in the server subdirectory, run make. It should first compile Lwt, which is included as a third-party library, and then should proceed to compiling the server itself.

If ocamlfind can't find libev, try setting (and exporting) the environment variables C_INCLUDE_PATH and LIBRARY_PATH to point to the include and lib directories containing libev's files.

To run the server, simply run ./server/hop_server.native, or just make run from within the server directory.

Working with the management and monitoring webpages

If you want to edit and/or recompile the server's built-in webpages, you will need to have installed

• xsltproc to make the webpages from the templates
• recess to compile the LESS into CSS

Compiling the Java client library

You will need a recent JDK, and Ant v1.6 or newer. Change to the java subdirectory, and run ant. You will end up with a file build/lib/hop.jar, which contains the client library and some test programs.

Run it

Open three terminals. Run the server in one of them. You should see output like the following:

hop ALPHA, Copyright (C) 2012 Tony Garnock-Jones.
This program comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to redistribute it under certain conditions.
See the GNU General Public License (version 3 or later) for details.
info: ("Node bound" "factory" "factory")
info: ("Registered node class" "queue")
info: ("Registered node class" "fanout")
info: ("Registered node class" "direct")
info: ("Node bound" "meta" "direct")
info: ("Node create ok" "direct" ("meta") "" "" "meta")
info: ("Node bound" "amq.direct" "direct")
info: ("Node create ok" "direct" ("amq.direct") "" "" "amq.direct")
info: ("Node bound" "amq.fanout" "fanout")
info: ("Node create ok" "fanout" ("amq.fanout") "" "" "amq.fanout")
info: ("Accepting connections" "AMQP" "5672")
info: ("Accepting connections" "HTTP" "5678")
info: ("Accepting connections" "Hop" "5671")
info: ("Waiting for milestone" "AMQP ready")
info: ("Achieved milestone" "AMQP ready")
info: ("Waiting for milestone" "HTTP ready")
info: ("Achieved milestone" "HTTP ready")
info: ("Waiting for milestone" "Hop ready")
info: ("Achieved milestone" "Hop ready")
info: ("Achieved milestone" "Server initialized")

In the second terminal, run the consuming half of the Java test program pair:

java -cp hop.jar hop.Test1 localhost

In the third, run the producing half:

java -cp hop.jar hop.Test3 localhost

Wire protocol

Obviously the wire protocol itself here is the simplest thing that could possibly work, and you'd never use anything like this inefficient in a real system. That said, this is what's there right now:

Message transfer

(post <routing-key> <message> <subscription-token>) - Instructs the receiving node to route (or process) the given message according to the given routing-key. Different kinds of nodes will do different things here, and in particular, will interpret the routing key differently. Queues, for example, will ignore the routing key and will deliver the message to only one of their active subscribers, whereas exchanges will generally match the routing key against their active subscriptions and will deliver the message on to all matches.

Subscription management

(subscribe <routing-key-filter> <target-node> <target-routing-key> <reply-node> <reply-routing-key>) - Instructs the receiving node to create a new subscription. The new subscription will only route messages matching the routing-key-filter, which is interpreted on a per-node-type basis as above for routing-key. Matching messages will be sent to target-node using post!, with a routing key of target-routing-key. The reply-node parameter, if nonempty, instructs the receiving node to send confirmation of subscription (along with a token that can be used with unsubscribe below) to the given address and routing key. If reply-node is empty, no confirmation of subscription is sent.

(unsubscribe <token>) - Instructs the receiving node to delete a previously established subscription. The token comes from the subscribe-ok message sent to reply-node after a successful subscribe operation.

Object management

(create <class-name> <argument> <reply-node> <reply-routing-key>) - Instructs the receiving object factory node to construct a new instance of the given class-name, with the given argument supplied to the constructor. The reply-node and reply-routing-key are used to send confirmation of completion to some waiting node.

Copyright and licensing

Hop is Copyright 2010, 2011, 2012 Tony Garnock-Jones tonygarnockjones@gmail.com.

Hop is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Hop is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with Hop. If not, see http://www.gnu.org/licenses/.