Flow control documentation
This commit is contained in:
parent
e3d1a0a43c
commit
4af561537b
|
@ -1,4 +1,76 @@
|
|||
# Flow control
|
||||
|
||||
- [Account]
|
||||
- [LoanedItem]
|
||||
- struct [`Account`]
|
||||
- struct [`LoanedItem`]
|
||||
|
||||
In order to handle high-speed scenarios where actors can become
|
||||
overloaded by incoming events, this crate takes a
|
||||
([possibly novel](https://syndicate-lang.org/journal/2021/09/02/internal-flow-control))
|
||||
approach to *internal flow control* that is a variant of "credit-based
|
||||
flow control" (as widely used in telephony systems).
|
||||
|
||||
The idea is to associate each individually-identifiable activity in an
|
||||
actor system with an [*account*][Account] that records how much
|
||||
outstanding work has to be done, system-wide, to fully complete
|
||||
processing of that activity.
|
||||
|
||||
Each Actor scheduling new activities in response to some external
|
||||
source (e.g while reading from a network socket in a
|
||||
[linked task][Activation::linked_task]) calls
|
||||
[`Account::ensure_clear_funds`] on its associated [`Account`]. This
|
||||
will suspend the actor until enough of the account's "debt" has been
|
||||
"cleared". (In the case of reading from a socket, this causes the TCP
|
||||
socket's read buffers to fill up and the TCP window to close, which
|
||||
throttles upstream senders.)
|
||||
|
||||
Every time any actor sends an event to any other actor, a
|
||||
[`LoanedItem`] is constructed which "borrows" enough credit from some
|
||||
nominated [`Account`] to cover the event. Crucially, when an actor is
|
||||
*responding* to an event by *sending* more events, the account chosen
|
||||
is *the one that the triggering event was charged to*. This lets the
|
||||
server automatically account for fan-out of events.[^corollary]
|
||||
Finally, once a `LoanedItem` is completely processed (i.e. when it is
|
||||
[dropped][LoanedItem::drop]), its cost is "repaid" to its associated
|
||||
account.
|
||||
|
||||
## Does it work?
|
||||
|
||||
Anecdotally, this approach appears to work well. Experimenting using
|
||||
`syndicate-server` with producers sending as quickly as they can,
|
||||
producers are throttled by the server, and the server seems stable
|
||||
even though its consumers are not able to keep up with the unthrottled
|
||||
send rate of each producer.
|
||||
|
||||
## Example
|
||||
|
||||
Imagine an actor *A* receiving publications from a TCP/IP socket. If
|
||||
it ever "owes" more than, say, 5 units of cost on its account, it
|
||||
stops reading from its socket until its debt decreases. Each message
|
||||
it forwards on to another actor costs it 1 unit. Say a given incoming
|
||||
message *M* is routed to a dataspace actor *D* (thereby charging *A*'s
|
||||
account 1 unit), where it results in nine outbound events *M′* to peer
|
||||
actors *O1*···*O9*.
|
||||
|
||||
Then, when *D* receives *M*, 1 unit is repaid to *A*'s account. When
|
||||
*D* sends *M′* on to each of *O1*···*O9*, 1 unit is charged to *A*'s
|
||||
account, resulting in a total of 9 units charged. At this point in
|
||||
time, *A*'s account has had net +1−1+9 = 9 units withdrawn from it as
|
||||
a result of *M*'s processing.
|
||||
|
||||
Imagine now that all of *O1*···*O9* are busy with other work. Then,
|
||||
next time around *A*'s main loop, *A* notices that its outstanding
|
||||
debt is higher than its configured threshold, and stops reading from
|
||||
its socket. As each of *O1*···*O9* eventually gets around to
|
||||
processing its copy of *M′*, it repays the associated 1 unit to *A*'s
|
||||
account.[^may-result-in-further-costs] Eventually, *A*'s account drops
|
||||
below the threshold, *A* is woken up, and it resumes reading from its
|
||||
socket.
|
||||
|
||||
[^corollary]: A corollary to this is that, for each event internal to
|
||||
the system, you can potentially identify the "ultimate cause" of
|
||||
the event: namely, the actor owning the associated account.
|
||||
|
||||
[^may-result-in-further-costs]: Of course, if *O1*, say, sends *more*
|
||||
events internally as a result of receiving *M′*, more units will
|
||||
be charged to *A*'s account!
|
||||
|
||||
|
|
Loading…
Reference in New Issue