1 Concepts
Marketplace integrates ideas from both distributed systems and +
1 Concepts
Marketplace integrates ideas from both distributed systems and virtualized operating system designs to obtain an architecture of nested virtual machines (VMs). Each nested layer is equipped with its own publish/subscribe network that also propagates @@ -15,7 +15,7 @@ carries a VM-specific protocol that is often different from the protocol spoken by its containing VM.
The outermost VM is called the ground VM. The protocol spoken by processes running within the ground VM is a simple protocol relating Racket’s synchronizable events to Marketplace network -messages. See Writing New Drivers and Drivers for +messages. See Drivers for information on using Racket events from Marketplace programs.
1.1 What is a process, what are event handlers?
A Marketplace process is a collection of event
handlers, plus a piece of private process state. Every
processThe exception to this rule is the Ground VM, which plays
diff --git a/Drivers.html b/Drivers.html
index 9e05d53..10c67a1 100644
--- a/Drivers.html
+++ b/Drivers.html
@@ -1,5 +1,5 @@
- procedure procedure struct TODO TODO TODO TODO TODO TODO TODO Choose a tcp-handle, and then create endpoints as follows: The TCP driver will automatically create an outbound connection in
+response to the presence of the endpoints. When the endpoints are
+deleted (or the process exits), the TCP driver will notice the absence
+and will close the underlying TCP socket. For a complete example, see TCP chat client. Choose a port number, and then create an observer endpoint as
+follows: The use of #:observer here indicates that this endpoint isn’t
+actually interested in exchanging any TCP data; instead, it is
+monitoring demand for such exchanges. The TCP driver uses a rare
+#:everything endpoint to monitor the presence of
+#:observers, and creates listening TCP server sockets in
+response. When a connection comes in, the TCP driver spawns a manager
+process which offers regular #:participant endpoints for
+communicating on the newly-arrived socket. To illustrate the code for handling a newly-arrived connection, TCP-related messages will be of the form (tcp-channel remote-address local-address subpacket) where the subpacket is either eof or a
+bytes?. Send data with (send-message (tcp-channel local-address remote-address subpacket)) where, as for receiving data, the subpacket is either
+eof or a bytes?. For examples of the use of the timer driver, see uses of
+set-timer and timer-expired in
+the
+Marketplace-based DNS resolver. For examples of the use of the UDP driver, see uses of
+udp-packet etc. in
+the
+Marketplace-based DNS resolver. Here is a complete Marketplace program: "examples/echo-paper.rkt" The top-level endpoint action subscribes to TCP connections
+ Here is a complete Marketplace program: "examples/echo-paper.rkt" The top-level endpoint action subscribes to TCP connections
arriving on port 5999, and spawns a fresh process in response to
each (#:on-presence). The topic of
conversation (#:conversation) associated with the newly-present
@@ -16,4 +16,4 @@ the #:on-absence handler in echo
action, terminating the connection’s process. The heart of our system
is the interface between a process and its containing VM. Our
implementation instantiates this interface as a collection of Typed
-Racket programs. "examples/chat-paper.rkt" "examples/chat-client.rkt" "examples/chat-paper.rkt" "examples/chat-client.rkt" procedure procedure value matrix-root-logger : logger? value matrix-root-logger : logger? This high-level interface between a VM and a process is analogous to
+ This high-level interface between a VM and a process is analogous to
the C library interface of a Unix-like operating system. The
Low-level interface corresponds to the system call
-interface of a Unix-like operating system. Programs written for Marketplace differ from normal Racket modules
+interface of a Unix-like operating system. Programs written for Marketplace differ from normal Racket modules
only in their selection of language. A Racket module written with
#lang marketplace, such as the echo server in
TCP echo server, specifies a sequence of definitions
and startup actions for an application. Typically, initial
actions spawn application processes and nested VMs, which in turn
-subscribe to sources of events from the outside world. There are a handful of variant languages to choose from: marketplace is for untyped programs, and uses
-the tcp-bare TCP driver; marketplace/flow-control is like
-marketplace, but uses the flow-controlled tcp
-driver; marketplace/typed is like marketplace, but
-for typed programs; marketplace/typed/flow-control is like
-marketplace/flow-control, but for typed programs. Instead of using Racket’s #lang feature, ordinary Racket programs
+subscribe to sources of events from the outside world. At present, there’s just #lang marketplace. In future, there will
+be a variation for Typed Racket, and languages providing greater
+support for flow control, responsibility transfer, and other
+networking concepts. For now, Typed Racket programs must be written as
+#lang typed/racket programs using (require marketplace)
+and ground-vm: explicitly. Instead of using Racket’s #lang feature, ordinary Racket programs
can use Marketplace features by requiring Marketplace modules
directly. Such programs need to use ground-vm/ground-vm: to
start the ground-level VM explicitly. They also need to explicitly
@@ -147,6 +147,25 @@ action. In that case, the reason will be the raised
itself. syntax (yield maybe-state-pattern k-expr) syntax (yield: typed-state-pattern k-expr) If pattern is supplied, k-expr should evaluate to a
-Transition; otherwise it should produce an ActionTree. ***** nested-vm, nested-vm:
-TODO ***** at-meta-level, at-meta-level:
-TODO syntax syntax If #:vm-pid is present, the corresponding identifier is bound
+in the boot-action expressions to the container-relative PID of the
+new VM itself. If #:boot-pid is present, however, the
+corresponding identifier is bound to the new-VM-relative PID of the
+primordial process in the new VM. syntax (at-meta-level: : StateType preaction ...) procedure (at-meta-level preaction ...) → (Action StateType) Marketplace’s actions can apply to either of those two networks. By
+default, actions apply to the VM of the acting process directly, but
+using at-meta-level (or at-meta-level: in typed
+code) to wrap an action level-shifts the action to make it
+apply at the level of the acting process’s VM’s container instead. For example, wrapping an endpoint in at-meta-level
+adds a subscription to the VM’s container’s network. Instead of
+listening to sibling processes of the acting process, the new endpoint
+will listen to sibling processes of the acting process’s VM. In this
+example, the primordial process in the nested-vm creates an
+endpoint in the VM’s own network, the ground VM: In this example, a new process is spawned as a sibling of the
+nested-vm rather than as a sibling of its primordial process: Compare to this example, which spawns a sibling of the
+nested-vm’s primordial process: Every program is a network.
+ Every program is a network.
This is the insight behind the π-calculus. Encoding a program as a
π-calculus term shows it as a network of communicating processes. It
is also one of the original inspirations for Smalltalk, where every
@@ -34,4 +34,4 @@ location of mutable state, failure detection and recovery, access
control, I/O and user interface, debugging and profiling. Marketplace addresses these concerns with a small set of primitives
chosen to make network programming in-the-small as flexible, scalable,
manageable and securable as network programming in-the-large— At its heart, the interface between each process and its
+ At its heart, the interface between each process and its
containing VM is based on handler functions exchanging
event and action structures with the VM. Both events and
actions are simple Racket structures. This low-level interface between a VM and a process is analogous to
@@ -15,7 +15,34 @@ which may have embedded wildcards. struct type TODO: Role type Orientation : (U 'publisher 'subscriber) type InterestType : (U 'participant 'observer 'everything) type EndpointEvent : (U PresenceEvent AbsenceEvent MessageEvent) type type type A role describes the conversational role of a peer as seen by
+some process. For example, a subscriber to topic 'foo with
+interest-type 'participant might receive a presence
+notification carrying the role (role 'publisher 'foo 'participant) Notice that the orientation of the role is the opposite of the
+orientation of the endpoint. type Orientation : (U 'publisher 'subscriber) type InterestType : (U 'participant 'observer 'everything) Using 'observer indicates that the endpoint is intended to
+monitor other ongoing (participant) conversations instead.
+Observer endpoints receive presence and absence notifications about
+participant endpoints, but participant endpoints only receive
+notifications about other participant endpoints, and not about
+observer endpoints. The 'observer interest-type is intended to make it easier to
+monitor resource demand and supply. The monitoring endpoints/processes
+can react to changing demand by creating or destroying resources to
+match. Finally, the 'everything interest-type receives notifications
+about presence and absence of all the types of endpoint,
+'participant, 'observer, and 'everything.
+Endpoints with interest-type 'everything are rare: they are
+relevant for managing demand for observers, as well as in some
+cases of cross-layer presence/absence propagation. Most programs (and
+even most drivers) will not need to use the 'everything
+interest-type. type EndpointEvent : (U PresenceEvent AbsenceEvent MessageEvent) type type type struct4 Drivers
4.1 event-relay
self-id : Symbol 4 Drivers
4.1 event-relay
self-id : Symbol (struct tcp-listener (port) #:prefab) port : (integer-in 0 65535) 4.2.3 Opening an outbound connection
4.2.4 Accepting inbound connections
4.2.5 Receiving data
4.2.6 Sending data
4.3 tcp
4.4 timer (typed and untyped)
4.5 udp (typed and untyped)
4.2.3 Opening an outbound connection
(let ((local (tcp-handle 'some-unique-value)) (remote (tcp-address "the.remote.host.example.com" 5999))) (transition/no-state (endpoint #:publisher (tcp-channel local remote ?)) (endpoint #:subscriber (tcp-channel remote local ?) [(tcp-channel _ _ (? eof-object?)) ; Handle a received end-of-file object (transition ...)] [(tcp-channel _ _ (? bytes? data)) ; Handle received data (transition ...)]))) 4.2.4 Accepting inbound connections
(endpoint #:subscriber (tcp-channel ? (tcp-listener 5999) ?) #:observer #:conversation (tcp-channel them us _) #:on-presence (spawn #:child (chat-session them us))) (define (chat-session them us) (transition/no-state (endpoint #:subscriber (tcp-channel them us ?) #:on-absence (quit) [(tcp-channel _ _ (? bytes? data)) ; Handle incoming data (transition ...)]))) 4.2.5 Receiving data
4.2.6 Sending data
4.3 timer (typed and untyped)
4.4 udp (typed and untyped)
7.1 TCP echo server 7.2 TCP chat server 7.3 TCP chat client 7 Examples
7.1 TCP echo server
#lang marketplace (endpoint #:subscriber (tcp-channel ? (tcp-listener 5999) ?) #:conversation (tcp-channel from to _) #:on-presence (spawn #:child (echoer from to))) (define (echoer from to) (transition stateless (endpoint #:subscriber (tcp-channel from to ?) #:on-absence (quit) [(tcp-channel _ _ data) (send-message (tcp-channel to from data))]))) 6.1 TCP echo server 6.2 TCP chat server 6.3 TCP chat client 6 Examples
6.1 TCP echo server
#lang marketplace (endpoint #:subscriber (tcp-channel ? (tcp-listener 5999) ?) #:conversation (tcp-channel from to _) #:on-presence (spawn #:child (echoer from to))) (define (echoer from to) (transition stateless (endpoint #:subscriber (tcp-channel from to ?) #:on-absence (quit) [(tcp-channel _ _ data) (send-message (tcp-channel to from data))]))) 7.2 TCP chat server
#lang marketplace (nested-vm (at-meta-level (endpoint #:subscriber (tcp-channel ? (tcp-listener 5999) ?) #:observer #:conversation (tcp-channel them us _) #:on-presence (spawn #:child (chat-session them us))))) (define (chat-session them us) (define user (gensym 'user)) (transition stateless (listen-to-user user them us) (speak-to-user user them us))) (define (listen-to-user user them us) (list (endpoint #:publisher `(,user says ,?)) (at-meta-level (endpoint #:subscriber (tcp-channel them us ?) #:on-absence (quit) [(tcp-channel _ _ (? bytes? text)) (send-message `(,user says ,text))])))) (define (speak-to-user user them us) (define (say fmt . args) (at-meta-level (send-message (tcp-channel us them (apply format fmt args))))) (define (announce who did-what) (unless (equal? who user) (say "~s ~s.~n" who did-what))) (list (say "You are ~s.~n" user) (at-meta-level (endpoint #:publisher (tcp-channel us them ?))) (endpoint #:subscriber `(,? says ,?) #:conversation `(,who says ,_) #:on-presence (announce who 'arrived) #:on-absence (announce who 'departed) [`(,who says ,what) (say "~a: ~a" who what)]))) 7.3 TCP chat client
#lang marketplace (require racket/port) (spawn #:debug-name 'console-output-driver #:child (transition/no-state (endpoint #:subscriber (list 'console-output ?) [(list 'console-output item) (begin (printf "~a" item) (void))]))) (spawn #:debug-name 'console-input-driver #:child (transition/no-state (endpoint #:publisher (list 'console-input ?) #:name 'input-relay #:on-absence (list (send-message (list 'console-output "Connection terminated.\n")) (quit))) (endpoint #:subscriber (cons (read-line-evt (current-input-port) 'any) ?) [(cons _ (? eof-object?)) (list (send-message (list 'console-output "Terminating on local EOF.\n")) (delete-endpoint 'input-relay))] [(cons _ (? string? line)) (send-message (list 'console-input line))]))) (spawn #:debug-name 'outbound-connection #:child (let ((local (tcp-handle 'outbound)) (remote (tcp-address "localhost" 5999))) (transition/no-state (endpoint #:subscriber (list 'console-input ?) #:on-absence (quit) [(list 'console-input line) (list (send-message (list 'console-output (format "> ~a \n" line))) (send-message (tcp-channel local remote (string-append line "\n"))))]) (endpoint #:publisher (tcp-channel local remote ?)) (endpoint #:subscriber (tcp-channel remote local ?) #:on-absence (quit) [(tcp-channel _ _ (? eof-object?)) (quit)] [(tcp-channel _ _ data) (list (send-message (list 'console-output (format "< ~a" data))) (void))])))) 6.2 TCP chat server
#lang marketplace (nested-vm (at-meta-level (endpoint #:subscriber (tcp-channel ? (tcp-listener 5999) ?) #:observer #:conversation (tcp-channel them us _) #:on-presence (spawn #:child (chat-session them us))))) (define (chat-session them us) (define user (gensym 'user)) (transition stateless (listen-to-user user them us) (speak-to-user user them us))) (define (listen-to-user user them us) (list (endpoint #:publisher `(,user says ,?)) (at-meta-level (endpoint #:subscriber (tcp-channel them us ?) #:on-absence (quit) [(tcp-channel _ _ (? bytes? text)) (send-message `(,user says ,text))])))) (define (speak-to-user user them us) (define (say fmt . args) (at-meta-level (send-message (tcp-channel us them (apply format fmt args))))) (define (announce who did-what) (unless (equal? who user) (say "~s ~s.~n" who did-what))) (list (say "You are ~s.~n" user) (at-meta-level (endpoint #:publisher (tcp-channel us them ?))) (endpoint #:subscriber `(,? says ,?) #:conversation `(,who says ,_) #:on-presence (announce who 'arrived) #:on-absence (announce who 'departed) [`(,who says ,what) (say "~a: ~a" who what)]))) 6.3 TCP chat client
#lang marketplace (require racket/port) (spawn #:debug-name 'console-output-driver #:child (transition/no-state (endpoint #:subscriber (list 'console-output ?) [(list 'console-output item) (begin (printf "~a" item) (void))]))) (spawn #:debug-name 'console-input-driver #:child (transition/no-state (endpoint #:publisher (list 'console-input ?) #:name 'input-relay #:on-absence (list (send-message (list 'console-output "Connection terminated.\n")) (quit))) (endpoint #:subscriber (cons (read-line-evt (current-input-port) 'any) ?) [(cons _ (? eof-object?)) (list (send-message (list 'console-output "Terminating on local EOF.\n")) (delete-endpoint 'input-relay))] [(cons _ (? string? line)) (send-message (list 'console-input line))]))) (spawn #:debug-name 'outbound-connection #:child (let ((local (tcp-handle 'outbound)) (remote (tcp-address "localhost" 5999))) (transition/no-state (endpoint #:subscriber (list 'console-input ?) #:on-absence (quit) [(list 'console-input line) (list (send-message (list 'console-output (format "> ~a \n" line))) (send-message (tcp-channel local remote (string-append line "\n"))))]) (endpoint #:publisher (tcp-channel local remote ?)) (endpoint #:subscriber (tcp-channel remote local ?) #:on-absence (quit) [(tcp-channel _ _ (? eof-object?)) (quit)] [(tcp-channel _ _ data) (list (send-message (list 'console-output (format "< ~a" data))) (void))])))) 6 Management and Monitoring
6.1 generic-spy
label : Any 5 Management and Monitoring
5.1 generic-spy
label : Any 6.2 logging (MATRIX_LOG)
5.2 logging (MATRIX_LOG)
6.3 debugger (experimental)
5.3 debugger (experimental)
2 High-level interface
2 High-level interface
2.1 Using #lang marketplace and friends
#lang marketplace #lang marketplace/flow-control #lang marketplace/typed #lang marketplace/typed/flow-control 2.1 Using #lang marketplace and friends
#lang marketplace 2.2 Using Marketplace as a library
(require marketplace/sugar-untyped) (require marketplace/sugar-typed) 2.2 Using Marketplace as a library
(require marketplace/sugar-untyped) (require marketplace/sugar-typed) 2.9 Cooperative scheduling
maybe-state-pattern = | #:state pattern typed-state-pattern = : State | pattern : State k-expr = expr 2.10 Creating nested VMs
2.11 Relaying across layers
2.10 Creating nested VMs
(nested-vm maybe-vm-pid-binding maybe-boot-pid-binding maybe-initial-state maybe-debug-name boot-action-expr ...) (nested-vm: : ParentStateType maybe-vm-pid-binding maybe-boot-pid-binding maybe-typed-initial-state maybe-debug-name boot-action-expr ...) maybe-vm-pid-binding = | #:vm-pid identifier maybe-boot-pid-binding = | #:boot-pid identifier maybe-initial-state = | #:initial-state expr maybe-typed-initial-state = | #:initial-state expr : StateType maybe-debug-name = | #:debug-name expr boot-action-expr = expr 2.11 Relaying across layers
preaction : (PreAction State) (nested-vm (at-meta-level (endpoint #:subscriber (tcp-channel ? (tcp-listener 5999) ?) ...))) (nested-vm (at-meta-level (spawn #:child (transition/no-state (send-message 'hello-world))))) (nested-vm (spawn #:child (transition/no-state (send-message 'hello-world)))) Marketplace: Network-Aware Programming
Marketplace: Network-Aware Programming
3 Low-level interface
(require marketplace) 3 Low-level interface
(require marketplace) (struct role (orientation topic interest-type) #:prefab) orientation : Orientation topic : Topic interest-type : InterestType 3.3 Endpoint Events
3.3 Endpoint Events
(struct presence-event (role) #:prefab) role : Role