diff --git a/marketplace/scribblings/background.scrbl b/marketplace/scribblings/background.scrbl new file mode 100644 index 0000000..03a3523 --- /dev/null +++ b/marketplace/scribblings/background.scrbl @@ -0,0 +1,102 @@ +#lang scribble/manual +@require[racket/include] +@include{prelude.inc} + +@title{Background} + +@section{Network programming in the small} + +Even the simplest of programs can be seen as a network. This program +sends a message containing @racket[3] and @racket[4] to @racket[+], +and awaits a reply message, which contains @racket[7]: + +@interaction[(+ 3 4)] + +Subroutines are network services. Here, when the message @racket[3] is +sent to @racket[add-four-to], it delegates to @racket[+]: + +@interaction[(define (add-four-to n) (+ n 4)) + (add-four-to 3)] + +@note{In π-calculus encodings of programs, replication and state are +made explicit.} Immutable programs (of which stateless programs are a +special case) are easily able to be @emph{replicated}; programs using +mutable state, however, must be fixed in place. In fact, strictly +speaking, it is the @emph{state} that must be fixed in place, not the +program @emph{manipulating} the state. In the following program, the +mutable @racket[total] variable must not be replicated: + +@interaction[(define accumulate + (let ((total 0)) + (lambda (n) + (set! total (+ total n)) + total))) + (accumulate 3) + (accumulate 4) + (accumulate 0)] + +It would be a mistake to simply replace each occurrence of +@racket[accumulate] with its definition, since three separate +@racket[total] locations would be created. However, there is no +problem replicating the @racket[lambda] term, so long as +@racket[total] always refers to the same location: + +@interaction[(define total 0) + ((lambda (n) (set! total (+ total n)) total) 3) + ((lambda (n) (set! total (+ total n)) total) 4) + ((lambda (n) (set! total (+ total n)) total) 0)] + +Programs raise exceptions to signal partial failure: + +@interaction[(define accumulate + (let ((total 0)) + (lambda (n) + (when (negative? n) + (error 'accumulate "n must be non-negative!")) + (set! total (+ total n)) + total))) + (accumulate 3) + (accumulate -2)] + +Programs can handle exceptions to @emph{contain} partial failure: + +@interaction[(with-handlers ([exn:fail? (lambda (e) 'ok)]) + (error 'oh-dear))] + +Partial failures interact with shared, mutable state in unpredictable +ways, especially in larger programs that combine stateful subprograms. + +Programmers often ignore issues of security and trust between +subprograms within a larger program. Most programming languages +entirely lack any means of securely isolating subprograms from each +other, leading to predictable failures. Even memory-safe languages +such as Racket, Java and .NET only offer weak techniques for securely +composing mutually suspicious subprograms. Techniques such as avoiding +@emph{ambient authority} and using @emph{object capabilities} to +compose subprograms + +TODO ^ + +@section{Network programming in the large} + +Programs which engage in I/O are very obviously part of a network: + +@racketblock[(define username (read-line)) + (printf "Hello, ~a!\n" username)] + +But look closely! There is a difference here between the kind of communication + +This program is more obviously composed of a number of moving pieces: + +@interaction[(let ((b (box 3))) + (set-box! b (+ (unbox b) 4)) + (unbox b))] + +@interaction[(define (add-three-to b) + (set-box! b (+ (unbox b) 3))) + (define (add-four-to b) + (set-box! b (+ (unbox b) 4))) + (define my-box (box 0)) + (add-three-to my-box) + (add-four-to my-box) + (unbox my-box)] diff --git a/marketplace/scribblings/concepts.scrbl b/marketplace/scribblings/concepts.scrbl index 4a73f23..08dd6ac 100644 --- a/marketplace/scribblings/concepts.scrbl +++ b/marketplace/scribblings/concepts.scrbl @@ -4,25 +4,13 @@ @title{Concepts} -In this paper, we present a novel "marketplace" approach to -functional systems programming, generalizing - Felleisen et al.'s "Worlds and Universes" approach to -functional I/O -[ICFP 2009]. We integrate ideas from both distributed systems +We integrate ideas from both distributed systems and virtualized operating system designs to obtain a novel architecture of nested virtual machines. Each nested layer is equipped with its own publish/subscribe network that also propagates information -about the (dis)appearance of services. Our paper presents -several case studies, including a recursive DNS resolver that -has served our lab's DNS traffic for the past nine months. +about the (dis)appearance of services. -Our goal is to reconcile this tension, starting from the "Worlds -and Universes" approach to functional I/O and -generalizing to -functional systems programming. We augment -its inherent support for concurrency with publish/subscribe (pub/sub) -messaging, a notion of presence, and -nestable virtual machines (VMs). The result suggests a @emph{ +The result suggests a @emph{ marketplace} metaphor, where communicating programs exist in a noisy, crowded, even chaotic context, rather than in a quiet place systematically going through their inboxes. diff --git a/marketplace/scribblings/marketplace.scrbl b/marketplace/scribblings/marketplace.scrbl index feceaa3..8171b4f 100644 --- a/marketplace/scribblings/marketplace.scrbl +++ b/marketplace/scribblings/marketplace.scrbl @@ -2,16 +2,71 @@ @require[racket/include] @include{prelude.inc} -@title[#:tag "marketplace"]{Marketplace: A Functional Operating System} +@title[#:tag "marketplace"]{Marketplace: Network-Aware Programming} +@;{Marketplace: A Functional Operating System} +@;{Marketplace: A Functional Network Operating System} +@;{Marketplace: A Functional Distributed Operating System} @author[(author+email "Tony Garnock-Jones" "tonyg@ccs.neu.edu")] -@defmodulelang[marketplace] +@bold{Every program is a network.} +This is the insight behind the π-calculus. Encoding a program as a +π-calculus term shows it as a network of communicating processes. It +is also one of the original inspirations for Smalltalk, where every +object, every value, was imagined to be a separate computer in a vast +network, and where objects communicated by message-passing. -This manual TODO (overview & motivation) +@bold{Every program is part of a network.} +A program that computes a result but cannot communicate it is useless +indeed. Every complete program both @emph{computes} and +@emph{communicates}. Furthermore, it does so with some finite set of +@emph{resources}, which it must carefully manage. + +@bold{Our programming languages do not recognise that every program is +a network.} They blur the distinction between stateful and stateless +portions of a program, making it difficult for programmers to reason +about concurrency, contention, and distribution. They often treat +partial failure as an afterthought, despite its importance in +reasoning about program behaviour, particularly in connection with the +effect of exceptions on stateful programs. They seldom consider issues +of trust and security. + +@bold{Our programming languages do not recognise that every program is part of a network.} +They treat communication with the outside world in an ad-hoc manner. +They frequently treat network communication separately from +@tt{1950s-style terminal input and output}. They force the programmer +to divine failures in other parts of the network by arcane means such +as timeouts and examining the entrails of dead communication channels. +They offer no support for allocating or releasing local resources in +response to changes in other parts of the network. They seldom +consider issues of trust and security. + +@bold{Marketplace is a network-aware programming language.} As a +corollary, because every program not only computes but also +communicates and manages its resources, Marketplace is also a +distributed operating system. + +By recognising that programs communicate both +@emph{internally} (between subprograms) and @emph{externally} (between +peers), we recognise an inherently recursive layered architecture. We +see at every level the @emph{same} concerns of resource management, +location of mutable state, failure detection and recovery, access +control, I/O and user interface, debugging and profiling. + +Marketplace addresses these concerns with a small set of primitives +chosen to make network programming in-the-small as flexible, scalable, +manageable and securable as network programming in-the-large---and +vice versa. + +@;{ +Networks must be manageable. Networks must be monitorable. Networks +must tolerate partial failure. Networks must scale. Networks must +communicate with other networks, via yet other networks. +} @local-table-of-contents[] +@include-section["background.scrbl"] @include-section["concepts.scrbl"] @include-section["highlevel.scrbl"] @include-section["lowlevel.scrbl"] diff --git a/marketplace/scribblings/prelude.inc b/marketplace/scribblings/prelude.inc index fedaf4d..d43f082 100644 --- a/marketplace/scribblings/prelude.inc +++ b/marketplace/scribblings/prelude.inc @@ -1,5 +1,6 @@ ;; -*- scheme -*- (require scribble/racket + scribble/eval scriblib/footnote (for-syntax racket) (for-label typed/racket/base))