marketplace-2014/scribblings/highlevel.scrbl

#lang scribble/manual
@require[racket/include]
@include{prelude.inc}

@title[#:tag "high-level-interface"]{High-level interface}

@declare-exporting[#:use-sources (marketplace/sugar)]

This high-level interface between a VM and a process is analogous to
the @emph{C library interface} of a Unix-like operating system. The
@secref{low-level-interface} corresponds to the @emph{system call
interface} of a Unix-like operating system.

@section[#:tag "hashlang-variations"]{Using @tt{#lang marketplace} and friends}

@;{
@defmodulelang*[(marketplace
		 marketplace/flow-control)]
}

@defmodulelang[marketplace]

Programs written for Marketplace differ from normal Racket modules
only in their selection of language. A Racket module written with
@tt{#lang marketplace}, such as the echo server in
@secref["echo-server-example"], specifies a sequence of definitions
and startup @tech{actions} for an application. Typically, initial
actions spawn application processes and nested VMs, which in turn
subscribe to sources of events from the outside world.

At present, there's just @tt{#lang marketplace}. In future, there will
be languages providing greater support for flow control,
responsibility transfer, and other networking concepts.

@;{
@itemlist[

  @item{@racket[marketplace] is for ordinary Racket programs, and uses
  the @secref{tcp-bare} TCP driver;}

  @item{@racket[marketplace/flow-control] is like
  @racket[marketplace], but uses the flow-controlled @secref{tcp}
  driver;}

]
}

@section{Using Marketplace as a library}

@defmodule*[(marketplace/sugar)
	    #:use-sources (marketplace/sugar)]

Instead of using Racket's @tt{#lang} feature, ordinary Racket programs
can use Marketplace features by requiring Marketplace modules
directly.

Such programs need to use @racket[ground-vm] to
start the ground-level VM explicitly. They also need to explicitly
start any drivers they need; for example, the file
@filepath{examples/echo-plain.rkt} uses @racket[ground-vm] along with
@racket[tcp] and an initial @racket[endpoint] action:

@racketblock[
(ground-vm tcp
           (subscriber (tcp-channel ? (tcp-listener 5999) ?)
             (match-conversation (tcp-channel from to _)
               (on-presence (spawn (echoer from to))))))
]

@defform[(ground-vm maybe-boot-pid-binding maybe-initial-state initial-action ...)
	 #:grammar
	 [(maybe-boot-pid-binding (code:line)
	 			  (code:line #:boot-pid id))
	  (maybe-initial-state (code:line)
	  		       (code:line #:initial-state expr))
	  (initial-action expr)]]{

Starts the ground VM. If @racket[#:boot-pid] is specified, the given
identifier is bound within the form to the PID of the @emph{primordial
process} that performs the initial actions. If
@racket[#:initial-state] is specified, it is used as the initial state
for the primordial process; if it is not supplied, the primordial
process is given @racket[(void)] as its initial state.

}

@section[#:tag "constructing-transitions"]{Constructing transitions}

@declare-exporting[#:use-sources (marketplace
				  marketplace/sugar)]

@deftogether[(
@defform[(transition new-state action-tree ...)]
@defform[(transition/no-state action-tree ...)]
)]{

Each of these forms produces a @racket[transition] structure.

Each @racket[action-tree] must be an @tech{action tree}.
It's fine to include @emph{no} action trees, in which case the
transition merely updates the state of the process without taking any
actions.

In the case of @racket[transition/no-state], the value @racket[(void)]
is used for the process state. @racket[transition/no-state] is useful
for processes that are stateless other than the implicit state of
their endpoints.

}

@defstruct*[transition ([state State] [actions action-tree?]) #:transparent]{

A transition structure. The @racket[transition-state] field is the new
private state the process will have after the transition is applied,
and the @racket[transition-actions] are the actions that will be
performed by the VM in order to apply the transition.

}

@defproc[(action-tree? [value any/c]) boolean?]{

Predicate that recognises an @deftech{action tree}. An action tree is
either

@itemlist[
	  @item{@racket['()];}
	  @item{@racket[#f];}
	  @item{@racket[(void)];}
	  @item{a pair of action trees; or}
	  @item{an @tech{action}.}
]

When performing actions, a VM will traverse an action-tree in
left-to-right order.

@racket['()], @racket[(void)], and @racket[#f] may be present in
action-trees: when the VM reaches such a value, it ignores it and
continues with the next leaf in the tree.

For example, all of the following are valid action trees which will
send messages @racket[1], @racket[2] and @racket[3] in that order:

@racketblock[(list (send-message 1)
		   (send-message 2)
		   (send-message 3))]

@racketblock[(list (list (send-message 1))
		   (cons (send-message 2) (cons '() (send-message 3))))]

@racketblock[(cons (cons (send-message 1)
			 (send-message 2))
		   (list #f #f (send-message 3)))]

Because @racket[#f] and @racket[(void)] are valid, ignored, members of
an action-tree, @racket[and] and @racket[when] can be used to
selectively include actions in an action-tree:

@racketblock[(list (first-action)
		   (when (condition?)
		     (optional-action))
		   (final-action))]

@racketblock[(list (first-action)
		   (and (condition?)
			(optional-action))
		   (final-action))]

Finally, these inert placeholders can be used to represent "no action
at all" in a transition:

@racketblock[(transition new-state) (code:comment "No action-trees at all")
	     (transition new-state '())
	     (transition new-state (void))
	     (transition new-state #f)]

}

@defproc[(sequence-actions [initial-transition transition?]
			   [item (or/c action-tree? (any/c -> transition?))]
			   ...) transition?]{

Returns a transition formed from the @racket[initial-transition]
extended with new actions, possibly updating its carried state. Each
of the supplied @racket[item]s is examined: if it is an
@racket[ActionTree], it is appended to the pending transition's
actions; if it is a procedure, it is called with the @emph{state} of
the pending transition, and is expected to return an updated
transition.

For example,

@racketblock[(sequence-actions (transition 'x
				 (send-message (list 'message 0)))
			       (send-message (list 'message 1))
			       (send-message (list 'message 2))
			       (lambda (old-state)
				 (transition (cons 'y old-state)
				   (send-message (list 'message 3))))
			       (send-message (list 'message 4)))]

produces the equivalent of

@racketblock[(transition (cons 'y 'x)
	       (send-message (list 'message 0))
	       (send-message (list 'message 1))
	       (send-message (list 'message 2))
	       (send-message (list 'message 3))
	       (send-message (list 'message 4)))]

}

@section[#:tag "endpoint-dsl"]{Creating endpoints}

The primitive action that creates new endpoints is
@racket[add-endpoint], but because endpoints are the most flexible and
complex point of interaction between a process and its VM, a
collection of macros helps streamline endpoint setup.

@deftogether[(
@defform[(publisher topic handler ...)]
@defform[(subscriber topic handler ...)]
@defform[(observe-subscribers topic handler ...)]
@defform[(observe-publishers topic handler ...)]
@defform[(observe-subscribers/everything topic handler ...)]
@defform[(observe-publishers/everything topic handler ...)]
@defform[(build-endpoint pre-eid role handler ...)]
)]{

The many variations on the core
@racket[build-endpoint] form exist to give
good control over @racket[InterestType] in the endpoint under
construction;
see @secref{participating-vs-observing}.

Almost everything is optional in an endpoint definition. The only
mandatory part is the topic.

For example, a minimal endpoint subscribing to all messages would be:

@racketblock[(subscriber ?)]

A minimal publishing endpoint would be:

@racketblock[(publisher ?)]

While topic patterns are ordinary Racket data with embedded @racket[?]
wildcards (see @secref{messages-and-topics}), all the other patterns
in an endpoint definition are @racket[match]-patterns. In particular
note that @racket[?] is a wildcard in a topic pattern, while
@racket[_] is a wildcard in a @racket[match]-pattern.

@subsection{Receiving messages}

@defform[(on-message [pattern expr ...] ...)]{

Supply an @racket[on-message] handler clause to an endpoint definition
to handle incoming message events (as distinct from presence- or
absence-events).

The following endpoint @emph{subscribes} to all messages, but only
@emph{handles} some of them:

@racketblock[(subscriber ?
	       (on-message
		['ping (send-message 'pong)]
		['hello (list (send-message 'goodbye)
			      (quit))]))]

}

}

@subsection{Action-only vs. State updates}

@defform[(match-state pattern handler ...)]{

If a group of handlers is wrapped in @racket[match-state], then all
the wrapped handlers are expected to return
@seclink["constructing-transitions"]{transition structures}.

If not, however, the handler expressions are expected to return plain
@tech{action tree}s.

This way, simple handlers that do not need to examine the process
state, and simply act in response to whichever event triggered them,
can be written without the clutter of threading the process state
value through the code.

For example, a simple endpoint could be written either as

@racketblock[(subscriber 'ping
	       (on-message ['ping (send-message 'pong)]))]

or, explicitly accessing the endpoint's process's state,

@racketblock[(subscriber 'ping
	       (match-state old-state
		 (on-message ['ping (transition old-state
				      (send-message 'pong))])))]

}

@subsection{Handling presence and absence events}

@deftogether[(
@defform[(on-presence expr ...)]
@defform[(on-absence expr ...)]
)]{

Other endpoints (in this or other processes) may have matching topics
and complementary orientations to the current endpoint. When such
endpoints come and go, presence and absence events are generated in
the current endpoint.

By default, no actions are taken on such events, but
@racket[on-presence] and @racket[on-absence] handlers override this
behaviour.

For example, say process A establishes the following endpoint:

@racketblock[(subscriber 'ping
	       (on-presence (send-message 'pinger-arrived))
	       (on-absence  (send-message 'pinger-departed))
	       (on-message ['ping (send-message 'pong)]))]

Some time later, process B takes the following endpoint-establishing
action:

@racketblock[(let-fresh (ping-endpoint-name pong-waiter-name)
	       (name-endpoint ping-endpoint-name
		 (publisher 'ping
		   (on-presence
		    (list (name-endpoint pong-waiter-name
			    (subscriber 'pong
			      (on-message
			       ['pong (list (delete-endpoint ping-endpoint-name)
					    (delete-endpoint pong-waiter-name))])))
			  (send-message 'ping))))))]

The sequence of events will be:

@itemlist[

  @item{Process A's @racket[on-presence] handler will run, and the
  @racket['pinger-arrived] message will be sent. At the same
  time,@note{In the current implementation, one happens before the
  other, but it is nondeterministic which is run first.} process B's
  @racket[on-presence] handler runs, installing a second endpoint
  and sending the @racket['ping] message.}

  @item{Process A's endpoint receives the @racket['ping] message, and
  sends the @racket['pong] message.}

  @item{Process B's second endpoint receives the @racket['pong]
  message, and deletes both of process B's endpoints.}

  @item{The @racket[on-absence] handler in process A runs, sending
  the @racket['pinger-departed] message.}

#:style 'ordered]

One possible trace of messages in the VM containing processes A and B is

@racketblock['pinger-arrived
	     'ping
	     'pong
	     'pinger-departed]

By sending the @racket['ping] message @emph{only} once the
@racket[on-presence] handler has fired, process B ensures that
someone is listening for pings.

This way, if process B starts before process A, then B will
automatically wait until A is ready to receive ping requests before
issuing any.

}

@subsection{Exit reasons}

@defform[(match-reason pattern handler ...)]{

If a handler is wrapped in a @racket[match-reason] form, then the exit
reason supplied to the @racket[delete-endpoint] or @racket[quit]
action that led to the @racket[absence-event] is available to the
endpoint's @racket[on-absence] handler expression.

}

@subsection[#:tag "updating-endpoints"]{Updating endpoints}

If, when an endpoint is created, an existing endpoint with an
@racket[equal?] name is already present, then if the existing and
to-be-added endpoints have exactly equal roles (meaning equal
orientations, interest-types, and topic patterns), the @emph{handlers}
for the endpoint are @emph{updated} without emitting presence or
absence notifications.

This dubious feature can be used to avoid "glitching" of presence
signals. A future release of this library will include better
automatic support for avoiding such transients.

@subsection{Who am I talking to?}

@deftogether[(
@defform[(match-orientation pattern handler ...)]
@defform[(match-conversation pattern handler ...)]
@defform[(match-interest-type pattern handler ...)]
)]{

Wrapping a handler in @racket[match-orientation],
@racket[match-conversation], and/or @racket[match-interest-type] gives
a handler access to the contents of the @racket[role] structure
carried in the triggering @racket[EndpointEvent].

The carried role describes the @emph{intersection of interests}
between the current endpoint and the peer endpoint, and so can proxy
for the identity of the other party. It is in a sense a description of
the scope of the current conversation.

It is most common to simply use @racket[match-conversation] to extract
the @racket[role-topic] alone, since it is seldom necessary to examine
@racket[role-orientation] (since it's guaranteed to be complementary
to the orientation of the current endpoint) or
@racket[role-interest-type].

See @secref{Examples} for examples of the use of
@racket[match-conversation] and friends.

}

@subsection[#:tag "participating-vs-observing"]{Participating in a conversation vs. observing conversations}

The core @racket[build-endpoint] form takes an expression evaluating
to a @racket[role], rather than a simple topic. This gives full
control over the new endpoint's @racket[Orientation] and
@racket[InterestType].

The other forms exist for convenience, since usually the orientation
and interest-type is known statically, and only the topic varies
dynamically:

@itemlist[

  @item{@racket[publisher] and @racket[subscriber] are for ordinary
@emph{participation} in conversations;}

  @item{@racket[observe-subscribers] and @racket[observe-publishers]
are for @emph{observing} conversations without participating in them; and}

  @item{@racket[observe-subscribers/everything] and
@racket[observe-publishers/everything] are like the ordinary
@tt{observe-...} variants, but use interest-type @racket['everything]
instead of @racket['observer].}

]

The @racket[publisher], @racket[observe-subscribers] and
@racket[observe-subscribers/everything] forms create
@emph{publisher}-oriented endpoints, and @racket[subscriber],
@racket[observe-publishers] and @racket[observe-publishers/everything]
create @emph{subscriber}-oriented endpoints. The rationale for this is
that as a participant, the code should declare the role being played;
but as an observer, the code should declare the roles being observed.

@subsection[#:tag "naming-endpoints"]{Naming endpoints}

Endpoint names can be used to @seclink["updating-endpoints"]{update}
or @seclink["deleting-endpoints"]{delete} endpoints.

@defproc[(name-endpoint [id any/c] [add-endpoint-action AddEndpoint]) AddEndpoint]{

Returns a copy of the passed-in @racket[add-endpoint] action
structure, with the @racket[id] field set to the passed-in identifying
value.

}

@defform[(let-fresh (identifier ...) expr ...)]{

Binds the @racket[identifier]s to freshly-gensymmed symbols so that
they are available to the @racket[exprs]. @racket[let-fresh] is useful
for inventing a guaranteed-unused name for a temporary endpoint:

@racketblock[(let-fresh (my-name)
	       (name-endpoint my-name
		 (subscriber ?
		   (on-message [_ (list (delete-endpoint my-name)
					...)]))))]

}

@section[#:tag "deleting-endpoints"]{Deleting endpoints}

@defproc[(delete-endpoint [id Any] [reason Any #f]) Action]{

Use this action to delete a previously-added endpoint by name. The
@racket[id] given must be @racket[equal?] to the corresponding
@racket[add-endpoint-pre-eid]; when @racket[endpoint] was used to
construct the endpoint to be deleted, the relevant name is that bound
by @racket[#:let-name] or supplied to @racket[#:name]. See
@secref{naming-endpoints}.

If @racket[reason] is supplied, it is included in the corresponding
action, and made available in any resulting @racket[absence-event]s.

}

@section{Sending messages and feedback}

@defproc[(send-message [body Any] [orientation Orientation 'publisher]) Action]{

Constructs a message-sending action with the given orientation.
Usually the correct orientation to use is @racket['publisher]; it
means that the sender of the message is acting in the "publisher"
role. Use @racket['subscriber] instead when acting in the "subscriber"
role, i.e. sending feedback.

}

@defproc[(send-feedback [body Any]) Action]{

Equivalent to @racket[(send-message body 'subscriber)].

}

@section{Creating processes}

@deftogether[(
@defform[(spawn maybe-pid-binding boot-expr)]
@defform[(spawn/continue maybe-pid-binding
			 #:parent parent-state-pattern k-expr
			 #:child boot-expr)
	 #:grammar
	 [(maybe-pid-binding (code:line)
			     (code:line #:pid identifier))
	  (k-expr expr)
	  (boot-expr expr)]]
)]{

Action describing a new process to create. The @racket[boot-expr]
should be an expression yielding a @racket[transition] that contains
the child process's initial state and initial actions.

If @racket[#:pid] is supplied, the associated identifier is bound to
the child process's PID in both @racket[boot-expr] and the parent's
@racket[k-expr].

The @racket[spawn/continue] variation includes a @racket[k-expr],
which will run in the parent process after the child process has been
created. Note that @racket[k-expr] must return a @racket[transition],
since @racket[parent-state-pattern] is always supplied for these
variations.

}

@defproc[(name-process [id Any] [spawn-action Spawn]) Spawn]{

Returns a copy of the passed-in @racket[spawn] action structure, with
the @racket[debug-name] field set to the passed-in identifying value.
The debug name of a process is used in VM debug output. See also
@secref{logging}.

}

@section{Exiting and killing processes}

@defproc[(quit [who (Option PID) #f] [reason Any #f]) Action]{

Action causing the termination of a process. If @racket[who] is
omitted or @racket[#f], terminates the acting process; otherwise,
terminates the peer process having @racket[who] as its PID.

If @racket[reason] is supplied, it is included in the corresponding
action, and made available in any resulting @racket[absence-event]s.

Terminating the current process is as simple as:

@racketblock[(quit)]

When a process raises an exception that it does not catch, its
containing VM catches the exception and turns it into an implicit quit
action. In that case, the @racket[reason] will be the raised exception
itself.

}

@section{Cooperative scheduling}

@defform[(yield state-pattern k-expr)]{

Lets other processes in the system run for a step, returning to
evaluate @racket[k-expr] only after doing a complete round of the
scheduler.

The state of the yielding process will be matched against
@racket[state-pattern] when the process is resumed, and
@racket[k-expr] must evaluate to a @racket[transition].

}

@section{Creating nested VMs}

@defform[(spawn-vm maybe-vm-pid-binding maybe-boot-pid-binding
		   maybe-initial-state
		   maybe-debug-name
		   boot-action-expr ...)
	 #:grammar
	 [(maybe-vm-pid-binding (code:line)
				(code:line #:vm-pid identifier))
	  (maybe-boot-pid-binding (code:line)
				  (code:line #:boot-pid identifier))
	  (maybe-initial-state (code:line)
			       (code:line #:initial-state expr))
	  (maybe-debug-name (code:line)
			    (code:line #:debug-name expr))
	  (boot-action-expr expr)]]{

Results in a @racket[spawn] action that starts a nested VM. The
primordial process in the new VM executes the boot-actions with the
given initial state. (If no initial state is supplied, @racket[(void)]
is used.)

If @racket[#:vm-pid] is present, the corresponding identifier is bound
in the boot-action expressions to the container-relative PID of the
new VM itself. If @racket[#:boot-pid] is present, however, the
corresponding identifier is bound to the new-VM-relative PID of the
primordial process in the new VM.

}

@section{Relaying across layers}

@defproc[(at-meta-level [preaction (PreAction State)] ...) (Action StateType)]{

Each VM gives its processes access to two distinct IPC facilities: the
@emph{internal} one, provided for the VM's processes to talk amongst
themselves, and the @emph{external} one, the network that the VM
itself is a process within.

Marketplace's actions can apply to either of those two networks. By
default, actions apply to the VM of the acting process directly, but
using @racket[at-meta-level] to wrap an action @emph{level-shifts} the
action to make it apply at the level of the acting process's VM's
container instead.

For example, wrapping an @racket[endpoint] in @racket[at-meta-level]
adds a subscription to the VM's container's network. Instead of
listening to sibling processes of the acting process, the new endpoint
will listen to sibling processes of the acting process's VM. In this
example, the primordial process in the nested VM creates an
endpoint in the VM's own network, the ground VM:

@racketblock[
(spawn-vm
 (at-meta-level
  (subscriber (tcp-channel ? (tcp-listener 5999) ?) ...)))
]

In this example, a new process is spawned as a sibling of the
nested VM rather than as a sibling of its primordial process:

@racketblock[
(spawn-vm
 (at-meta-level
  (spawn (transition/no-state (send-message 'hello-world)))))
]

Compare to this example, which spawns a sibling of the
nested VM's primordial process:

@racketblock[
(spawn-vm
 (spawn (transition/no-state (send-message 'hello-world))))
]

}