Background and overview.

marketplace/scribblings/background.scrbl

@@ -0,0 +1,102 @@
+#lang scribble/manual
+@require[racket/include]
+@include{prelude.inc}
+
+@title{Background}
+
+@section{Network programming in the small}
+
+Even the simplest of programs can be seen as a network. This program
+sends a message containing @racket[3] and @racket[4] to @racket[+],
+and awaits a reply message, which contains @racket[7]:
+
+@interaction[(+ 3 4)]
+
+Subroutines are network services. Here, when the message @racket[3] is
+sent to @racket[add-four-to], it delegates to @racket[+]:
+
+@interaction[(define (add-four-to n) (+ n 4))
+	     (add-four-to 3)]
+
+@note{In π-calculus encodings of programs, replication and state are
+made explicit.} Immutable programs (of which stateless programs are a
+special case) are easily able to be @emph{replicated}; programs using
+mutable state, however, must be fixed in place. In fact, strictly
+speaking, it is the @emph{state} that must be fixed in place, not the
+program @emph{manipulating} the state. In the following program, the
+mutable @racket[total] variable must not be replicated:
+
+@interaction[(define accumulate
+	       (let ((total 0))
+		 (lambda (n)
+		   (set! total (+ total n))
+		   total)))
+	     (accumulate 3)
+	     (accumulate 4)
+	     (accumulate 0)]
+
+It would be a mistake to simply replace each occurrence of
+@racket[accumulate] with its definition, since three separate
+@racket[total] locations would be created. However, there is no
+problem replicating the @racket[lambda] term, so long as
+@racket[total] always refers to the same location:
+
+@interaction[(define total 0)
+	     ((lambda (n) (set! total (+ total n)) total) 3)
+	     ((lambda (n) (set! total (+ total n)) total) 4)
+	     ((lambda (n) (set! total (+ total n)) total) 0)]
+
+Programs raise exceptions to signal partial failure:
+
+@interaction[(define accumulate
+	       (let ((total 0))
+		 (lambda (n)
+		   (when (negative? n)
+		     (error 'accumulate "n must be non-negative!"))
+		   (set! total (+ total n))
+		   total)))
+	     (accumulate 3)
+	     (accumulate -2)]
+
+Programs can handle exceptions to @emph{contain} partial failure:
+
+@interaction[(with-handlers ([exn:fail? (lambda (e) 'ok)])
+	       (error 'oh-dear))]
+
+Partial failures interact with shared, mutable state in unpredictable
+ways, especially in larger programs that combine stateful subprograms.
+
+Programmers often ignore issues of security and trust between
+subprograms within a larger program. Most programming languages
+entirely lack any means of securely isolating subprograms from each
+other, leading to predictable failures. Even memory-safe languages
+such as Racket, Java and .NET only offer weak techniques for securely
+composing mutually suspicious subprograms. Techniques such as avoiding
+@emph{ambient authority} and using @emph{object capabilities} to
+compose subprograms
+
+TODO ^
+
+@section{Network programming in the large}
+
+Programs which engage in I/O are very obviously part of a network:
+
+@racketblock[(define username (read-line))
+	     (printf "Hello, ~a!\n" username)]
+
+But look closely! There is a difference here between the kind of communication 
+
+This program is more obviously composed of a number of moving pieces:
+
+@interaction[(let ((b (box 3)))
+               (set-box! b (+ (unbox b) 4))
+               (unbox b))]
+
+@interaction[(define (add-three-to b)
+               (set-box! b (+ (unbox b) 3)))
+             (define (add-four-to b)
+               (set-box! b (+ (unbox b) 4)))
+             (define my-box (box 0))
+             (add-three-to my-box)
+             (add-four-to my-box)
+	     (unbox my-box)]

marketplace/scribblings/concepts.scrbl

@@ -4,25 +4,13 @@
 
 @title{Concepts}
 
-In this paper, we present a novel "marketplace" approach to
-functional systems programming, generalizing
- Felleisen et al.'s "Worlds and Universes" approach to
-functional I/O
-[ICFP 2009]. We integrate ideas from both distributed systems
+We integrate ideas from both distributed systems
 and virtualized operating system designs to obtain a novel
 architecture of nested virtual machines. Each nested layer is equipped with
 its own publish/subscribe network that also propagates information
-about the (dis)appearance of services. Our paper presents
-several case studies, including a recursive DNS resolver that
-has served our lab's DNS traffic for the past nine months.
+about the (dis)appearance of services.
 
-Our goal is to reconcile this tension, starting from the "Worlds
-and Universes" approach to functional I/O and
-generalizing to
-functional systems programming. We augment
-its inherent support for concurrency with publish/subscribe (pub/sub)
-messaging, a notion of presence, and
-nestable virtual machines (VMs). The result suggests a @emph{
+The result suggests a @emph{
 marketplace} metaphor, where communicating programs exist in a
 noisy, crowded, even chaotic context, rather than in a quiet place
 systematically going through their inboxes.

marketplace/scribblings/marketplace.scrbl

@@ -2,16 +2,71 @@
 @require[racket/include]
 @include{prelude.inc}
 
-@title[#:tag "marketplace"]{Marketplace: A Functional Operating System}
+@title[#:tag "marketplace"]{Marketplace: Network-Aware Programming}
+@;{Marketplace: A Functional Operating System}
+@;{Marketplace: A Functional Network Operating System}
+@;{Marketplace: A Functional Distributed Operating System}
 
 @author[(author+email "Tony Garnock-Jones" "tonyg@ccs.neu.edu")]
 
-@defmodulelang[marketplace]
+@bold{Every program is a network.}
+This is the insight behind the π-calculus. Encoding a program as a
+π-calculus term shows it as a network of communicating processes. It
+is also one of the original inspirations for Smalltalk, where every
+object, every value, was imagined to be a separate computer in a vast
+network, and where objects communicated by message-passing.
 
-This manual TODO (overview & motivation)
+@bold{Every program is part of a network.}
+A program that computes a result but cannot communicate it is useless
+indeed. Every complete program both @emph{computes} and
+@emph{communicates}. Furthermore, it does so with some finite set of
+@emph{resources}, which it must carefully manage.
+
+@bold{Our programming languages do not recognise that every program is
+a network.} They blur the distinction between stateful and stateless
+portions of a program, making it difficult for programmers to reason
+about concurrency, contention, and distribution. They often treat
+partial failure as an afterthought, despite its importance in
+reasoning about program behaviour, particularly in connection with the
+effect of exceptions on stateful programs. They seldom consider issues
+of trust and security.
+
+@bold{Our programming languages do not recognise that every program is part of a network.}
+They treat communication with the outside world in an ad-hoc manner.
+They frequently treat network communication separately from
+@tt{1950s-style terminal input and output}. They force the programmer
+to divine failures in other parts of the network by arcane means such
+as timeouts and examining the entrails of dead communication channels.
+They offer no support for allocating or releasing local resources in
+response to changes in other parts of the network. They seldom
+consider issues of trust and security.
+
+@bold{Marketplace is a network-aware programming language.} As a
+corollary, because every program not only computes but also
+communicates and manages its resources, Marketplace is also a
+distributed operating system.
+
+By recognising that programs communicate both
+@emph{internally} (between subprograms) and @emph{externally} (between
+peers), we recognise an inherently recursive layered architecture. We
+see at every level the @emph{same} concerns of resource management,
+location of mutable state, failure detection and recovery, access
+control, I/O and user interface, debugging and profiling.
+
+Marketplace addresses these concerns with a small set of primitives
+chosen to make network programming in-the-small as flexible, scalable,
+manageable and securable as network programming in-the-large---and
+vice versa.
+
+@;{
+Networks must be manageable. Networks must be monitorable. Networks
+must tolerate partial failure. Networks must scale. Networks must
+communicate with other networks, via yet other networks.
+}
 
 @local-table-of-contents[]
 
+@include-section["background.scrbl"]
 @include-section["concepts.scrbl"]
 @include-section["highlevel.scrbl"]
 @include-section["lowlevel.scrbl"]

marketplace/scribblings/prelude.inc

@@ -1,5 +1,6 @@
 ;; -*- scheme -*-
 (require scribble/racket
+	 scribble/eval
 	 scriblib/footnote
 	 (for-syntax racket)
 	 (for-label typed/racket/base))