Beginnings of a writeup of the idea

doc/hop.md

@@ -0,0 +1,196 @@
+# The Hop Pattern
+
+Hop is a combination of
+
+ - a syntax for encoding data for transport
+ - a subscription and messaging protocol
+ - a design for recursive networks
+
+In this document, I will sketch the third aspect of the system: the
+design for recursive networking.
+
+## Core idea: Abstract Networks
+
+Every virtual machine, TCP connection, overlay network, and other
+object in the system can be viewed abstractly as a communications
+network. This applies just as well to individual TCP connections and
+individual Java objects as it does to AMQP-style exchanges, brokers,
+and queues.
+
+This might not seem like an advantage, but in fact looking at things
+this way permits simple, regular composition of networks (and virtual
+machines, etc.).
+
+### Abstract Networks
+
+The idea of an abstract network that we will be using is this:
+
+A network is a *namespace* within which exist *nodes* which send
+*messages* that are addressed to other *names* in the network. Names
+are mapped back to nodes via a *directory* of some kind. Joining such
+a network is called *enrollment*: it is the process of, first,
+authenticating a node to the network, and second, *binding* of zero or
+more names in the namespace to the new node. These networks also have
+a *routing semantic*: when a message is sent to some name in the
+network, the network will have a characteristic way of using its
+directory to decide which, if any, nodes in the network should receive
+the message.
+
+### A TCP connection, viewed as an abstract network
+
+A single TCP connection can be seen as a network.
+
+ - the set of possible names is exactly `{client, server}`.
+ - joining the network and binding to names happens automatically as
+   part of the creation of the connection.
+ - messages sent by one party are implicitly delivered to the other
+   party.
+
+### A Java virtual machine instance, viewed as an abstract network
+
+A Java virtual machine and all its objects can be seen as a network.
+
+ - the set of possible names is the set of valid (live) pointers to
+   objects.
+ - joining the network and binding to names happens automatically each
+   time a new object is created.
+ - object names are (almost) capabilities.
+ - method calls, returns, and exceptions are the messages sent in the
+   system: calls are sent to object names, returns to continuation
+   names (transient and implicitly specified), and exceptions to
+   handler names (again implicitly specified).
+ - there is a built-in (and unavoidable) notion of *conversation
+   pattern* included in the semantic of this kind of network: each
+   method call message results in either a return message or an
+   exception message.
+
+### The world-wide UDP/IP network, viewed as an abstract network
+
+The entire collection of UDP/IP endpoints can be seen as an abstract
+network.
+
+ - the names are the `{ip, port}` pairs.
+ - joining the network and binding to a name happens in part via the
+   operation of DHCP (giving an IP address) and in part via, for
+   example, BSD sockets' `bind(2)` system call.
+ - messages sent in the system are just UDP packets.
+
+### An AMQP broker, viewed as an abstract network
+
+AMQP 0-9-1 brokers can be seen as abstract networks.
+
+ - the nodes are the exchanges, queues, consumers and active channels
+   in the broker.
+ - some nodes, namely queues and exchanges, join the network via the
+   operation of a broker-internal *factory* node.
+ - other nodes, namely channels and consumers, join the network via
+   connections from the outside world.
+ - name binding for nodes happens at the time they're created.
+
+### An AMQP "direct" exchange, viewed as an abstract network
+
+Exchanges within AMQP 0-9-1 brokers can be seen as abstract
+networks. For example, consider "direct"-style exchanges, though
+similar observations can be made of the other kinds of exchange
+defined by the AMQP specification:
+
+ - the names are the routing-keys to which queues have been bound.
+ - joining the network and binding to names is done via queue binding:
+   queues are the only nodes in AMQP that can join an exchange's
+   network.
+ - messages delivered to an exchange are routed using the exchange's
+   internal routing table (its directory) to zero or more bound
+   queues.
+
+## Bridging between abstract networks
+
+So far, I've mentioned two kinds of network: those where there is no
+explicit notion of a bridge to the outside world, such as individual
+TCP connections, Java virtual machine instances, and the worldwide
+UDP/IP network, and those where some explicit idea of connecting to
+other systems is present, such as AMQP brokers and exchanges.
+
+Considering UDP/IP for a moment, it's clear that the "layer 2"
+protocols underpinning UDP/IP act in some way as connections to other
+fragments of the worldwide network. Similarly, each TCP/IP connection
+has a kind of "border router" at each end embodied in the BSD sockets
+API that routes packets to and from the kernel's TCP stack. Finally,
+Java VM instances have a plethora of options for communicating with
+the outside world and permitting communications from the outside world
+to cause methods to be invoked on Java objects—that is, from the
+abstract point of view, to cause messages to be sent within the VM's
+internal network.
+
+((TODO: distinguish carefully between interior routers and border
+routers: the former glue together subnets of UDP/IP into a single
+worldwide namespace, for example, where the latter permit access to
+other namespaces, or even kinds of namespace, from a given
+network. The former implement a distributed directory and routing
+system, the latter embed entire networks as single nodes in a host
+network.))
+
+It seems clear that every kind of complete computing system has three aspects:
+
+ - a computation facility (which is how it does its job, whatever that may be)
+ - an internal communications facility (its network aspect)
+ - an external communications facility (routing of data to other systems/networks)
+
+The TCP connections that AMQP brokers accept, the bindings between
+AMQP exchanges and queues, the "consumer" relationships between AMQP
+queues and channels,
+
+### Abstract relays
+
+Abstract relays *embed* a remote network as a single node within a
+local network. Messages sent via local names to relay instances are
+transported uninterpreted across some underlying transport to the
+other network, where a corresponding relay instance interprets the
+messages being sent. A symmetric relay setup may also exist, which
+gives a *mutual embedding* of the two networks.
+
+## Core protocol
+
+`post`
+
+`subscribe`
+
+`unsubscribe`
+
+## Refinements
+
+Acknowledgement
+
+Lifetime-coupling
+
+## Frequently useful node classes
+
+Factory
+
+Relay (seen in many various guises)
+
+Exchange
+
+Queue/Buffer
+
+Consumer
+
+Broker (usually explicit only as part of a mutual embedding of a relay
+into a network)
+
+## Elements
+
+An implementation of the Hop pattern will include
+
+Codec
+
+relay
+
+classes
+
+nodes
+
+namespace and a dispatcher
+
+factory
+
+a notion of subscription, same as the notion of binding