From 07922cec641b2854dcd8f455d4f923ae1ecd129c Mon Sep 17 00:00:00 2001 From: Tony Garnock-Jones Date: Wed, 11 Jul 2012 09:35:20 -0400 Subject: [PATCH] Talk outline --- presentation/talk-outline.org | 316 ++++++++++++++++++++++++++++++++++ 1 file changed, 316 insertions(+) create mode 100644 presentation/talk-outline.org diff --git a/presentation/talk-outline.org b/presentation/talk-outline.org new file mode 100644 index 0000000..a0f49f9 --- /dev/null +++ b/presentation/talk-outline.org @@ -0,0 +1,316 @@ +#lang racket/base +(require "../os2.rkt") +(require "../os2-tcp.rkt") + +#| + +Outline ideas: + - the core of the idea is pubsub and presence. conversational + context - not raw messaging and actors. + - (frequently first-order-)functional programming of networked systems. + - similar to a generalisation of big-bang + - similar advantages obtain + - show code. Stuff that looks nice and illustrates the features of os2. + - transitions + - then the actions (spawn, roles, presence, messaging) + - then ground-vm + - then nested-vm + - then at-meta-level + +Prepare for whiteboard + showing code. Light on the slides, if any. + +Recent work in Network Application Architecture. + +--------------------------------------------------------------------------- + +Desperately needs a name - "os-big-bang", "os2" + +Intro: + +Lots of systems, both formal models and practical implementations, +work with messaging between actors or actor-like entities. Notably +Erlang and the π-calculus. + +In these systems, actors/processes have conversations with one +another, but the programming languages themselves don't talk about +conversations or patterns of interaction larger than a single exchange +of a message from a single sender to a single receiver. + + Whiteboard + First point: receivers = 1 i.e. point-to-point/unicast + Second point: granularity = single message + +By contrast, lots of the communicating systems we want to model have +more interesting conversational patterns. + + HTTP: request/response; streaming; authentication; sessions + TCP: bidirectional; ongoing stream; connected + Bittorrent: multiparty; cooperative; unordered + Mailing lists: pub/sub; subscription management + +So generally we can say that communication is instead + + Whiteboard + receivers = many i.e. multicast/broadcast/M-of-N + granularity = session/conversation/connection + +The research I've been doing has been examining the idea of a +conversational context: the envelope within which a group of +communicating actors/processes work together to achieve some shared +task. + +There's a rich area of work dealing with describing the patterns of +message exchange themselves, for example session types and behavioural +contracts, but I'm looking at it from a different angle: + + - how do actors interested in communicating find each other? + Whiteboard + Discovery + Synchronisation + - how does a conversation start? + Resource allocation + - how are resources managed during a conversation? + Acknowledgement + Nacks + Flow control + - how do participants know when a conversation is over? + Error/crash/exit signalling + Resource release + - how do conversations fit in to the underlying network system? + Routing + - what are the roles involved in various kinds of conversation? + Sender/Receiver + Scribe? Archivist? Statistician? Auditor? + +To experiment with this I've come up with a model operating system, +and used it to implement a DNS caching proxy and an SSH server. + +The model operating system is based on multicast/pubsub, and also +includes a construct for working with conversational context. I've +been calling it "presence", since it's a generalisation of an idea +from the XMPP/Jabber world, but you could equally well call it +"interest" or "subscription". + +It's pure functional and event-based - very reminiscent of big-bang, +and so has similar characteristics, including processes usually having +first-order state - and recursive, meaning that instances of the +kernel can run as user processes within another kernel instance. + +Each process within an instance of the system has private state, and a +collection of active endpoints. Again the terminology here is a bit +loose, and while I usually think of endpoints as subscriptions or +event sources, they also make sense as TCP-like half-connections or +more loosely as representing participation in a particular (set of) +conversation(s). + + Whiteboard + + Process = ∃State . State × [Endpoint] + where + Endpoint = Topic × Role × InterestType + × PresenceHandler + × AbsenceHandler + × MessageHandler + + PresenceHandler = Topic → State → Transition + AbsenceHandler = Topic × Reason → State → Transition + MessageHandler = Message → State → Transition + + Transition = State × [Action] + + Topic = notional set of Messages + Role = 'Subscriber + 'Publisher + ... + InterestType = 'Participant + 'Observer + 'SuperObserver + +The kernel runs a process in response to some external event, and it +expects back a new process state and a list of actions that the +process wishes the kernel to perform on its behalf. + + Process = ∃State . State × [Endpoint] + where (...) + Action = 'Yield (State → Transition) + + 'AtMetaLevel Preaction + + Preaction + Preaction = 'AddRole Endpoint + + 'DeleteRole Endpoint + + 'SendMessage Message + + 'Spawn (∃State . State × [Action]) + + VMState = Process × [Action] + + runVM :: VMState → VMState × [Action] + + nestedVM :: (∃State . State × [Action]) → (VMState × [Action]) + nestedVM bootProcess = ... + + groundVM :: (∃State . State × [Action]) → 0 + groundVM bootProcess = ... + +--------------------------------------------------------------------------- +|# + + + + + + + + + + + +(define (main) + (ground-vm + (transition 'none + (spawn tcp-driver) + (spawn listener)))) + + + + + + + + + + + + + + +(define listener + (let ((local-addr (tcp-listener 5999))) + (transition 'no-state + (role 'inbound-handler + (topic-subscriber (tcp-channel (wild) local-addr (wild)) + #:monitor? #t) + #:state state + #:topic t + #:on-presence + (match t + [(topic 'publisher (tcp-channel remote-addr (== local-addr) _) #f) + (transition state + (spawn (connection-handler local-addr remote-addr)))]))))) + + + + + + + + + + + + + + + +(define (connection-handler local-addr remote-addr) + (transition 'no-state + (send-feedback (tcp-channel remote-addr local-addr (tcp-mode 'lines))) + (send-feedback (tcp-channel remote-addr local-addr (tcp-credit 1))) + (role 'echoer + (topic-subscriber (tcp-channel remote-addr local-addr (wild))) + #:state state + #:on-absence (transition state + (kill)) + [(tcp-channel _ _ (? bytes? line)) + (define reply (bytes-append #"You said: " line #"\n")) + (transition state + (send-feedback (tcp-channel remote-addr local-addr (tcp-credit 1))) + (send-message (tcp-channel local-addr remote-addr reply)))] + [(tcp-channel _ _ (? eof-object?)) + (transition state + (kill))]))) + + + + + + +#| +--------------------------------------------------------------------------- + +Messages include their topics: topics are after all just sets of messages + - This means ground events are (cons/c evt/c any/c) + - This is the set of all possible responses to sync'ing on that event! + - I've scraped by with this so far, but because sync'ing has + side-effects, I'll eventually need better names for event requests + and responsibility transfer. + +Contracts + - for process state (implemented) + - for messages across the bus at each level + - between processes / within conversations + +Functional pub/sub + +Motivate nested VM model + - Helper subprograms that have internal communication actions should + not have those actions manifest in their interface: what would be + stateful/side-effecting subroutine calls in Scheme should retain + that RPC-like request/response flavour in our system. + - Either the states of a subprogram and its callers should remain + separate, or some powerful help should be provided by the system to + make sure they are combined in a clean, manageable, + arbitrarily-composable fashion. + - The system should make it possible to invoke a subprogram without + needing to be aware of whether and how that subprogram in turn + chooses to invoke sub-subprograms. + +Motivate roles vs Erlang-style single mailbox + - The system should make it possible for programs that would + otherwise block on some external operation to remain responsive to + their normal inputs while waiting for the external operation to + complete. + +Can represent the "natural" structure of SSH in terms of nested VMs + Whiteboard + Diagram of 14 Oct 2011 from my research journal + +Ground-level events and a ground-level VM bottom out the recursion + +Examples + HTTP Load Balancer + Bank Teller + +Weaknesses + - The Wart + - Glitching + - There’s maybe a kind of continuousness (in the calculus sense) + at play here: well-behaved presence apps don’t glitch in their + presence. Sets of interests smoothly evolve without transient + drops of interest potentially leading to confusion on the part + of the peer. + - Handoff of responsibility + - the manager can take responsibility, spawn, and then drop when + it sees the child is up or crashed; or + - the manager can spawn without taking, and blip on crash; or + - the manager can spawn and then treat like an established service + instance + - Querying the routing table + - e.g. for sending SSH channel opens: is the channel type + supported? querying the routing table would let us find out + without explicitly sending a message + +Not just roles and sets, but *participation levels* + - participant + - observer/monitor + - "super"-observer/monitor + +This got me thinking: the presence/messaging system I’ve been building +looks like a hybrid between synchronous and asynchronous messaging. Is +it fair to say you can use the presence mechanism to synchronise, and +then the messaging mechanism to communicate? + +Rendezvous via presence: this makes temporal dependencies between +processes during system startup resolve themselves automatically. No +more races at boot time! + +Exceptions and errors are propagated via loss of presence. + - Erlang's process links and monitors can be seen as a special case + of this mechanism. + - Erlang/OTP-like supervisors fit naturally into this picture + +|#