ycl coro relay

ycl coro relay is a small but complete system that provides order/receive facilities among asynchronous coroutines. Features include timeouts, cancellation, error propagation, and scatter/gather capabilities. It has been largely superceded by ycl coro call.

Commands edit

accept ?spec? ?payload?: yield until called, and return a list whose first item is a command prefix that can be called to deliver a result to the requestor. Additional items are arguments passed by the requestor. If spec is provided, it is a list of variable names, and returned values are assigned in order to those names. If the last name in spec is args, a list of remaining values is assigned to it. payload is a value to yield.
call ?delay? args: Like order, but the evaluated command does not have to call accept or use deliver, or even be a coroutine, although it can be. It's just a normal command that returns a result. Returns an order id that can passed to cancel. delay is the number of milliseconds to delay before sending evaluating the command. args is the command to evaluate.
cancel ?coroutine coroutine? id: Cancel an order that hasn't been received. If the order has not been started, it won't be. Otherwise, any results it delivers are discarded. If id is all, all orders made by the current coroutine are canceled. If coroutine is provided, it specifies a coroutine other than the default, and only makes sense when id is all.
iter names coroutine script: Iteratively place orders to $coroutine and evaluate $script for each received order. $names is a list containing the name of a variable in which to place the received order and, optionally, the name of a variable containing a list of items to send as arguments of the next order. Within $script break and continue can be used as might be supposed.
last: Returns the order id of the last delivered task.
order delay ?arg ...`?: Place a new order and return an order id that can be passed to cancel. Number of milliseconds to wait before placing the order. If $delay contains more than one item, the second item is the maximum number of milliseconds alloted for delivery, order times out, after which the order is not be initiated if it hasn't been started, and if it has, any subsequent deliveries are discarded. If there is a third item, the second item is the maximum number of milliseconds alllotted for the order to be started, and the third item is the maximum number of milliseonds alloted for delivery. If the completion timeout occurs before the order is complete, any subsequent delivery is discarded.
receive ?orders orders: Yield until a previously-placed order comes in, and then return the value and the options dictionary of that order. If orders is provided, it is a list that limits which orders are to be received. If there was an error in the fulfilment of an order, receive returns that error.
switch ?option value ...?: Yield until an order was received, and respond to it according to the first word of the order. routes is a dictionary of words and corresponding scripts. These are merged into the default routes.
wrap iter wrapper ?name? name: Returns the name of a new coroutine named name that acts as a proxy for iter, internally placing orders to iter, passing each delivery and the options dictionary to wrapper (a command prefix), and delivering that result to the caller. Useful to create filters, monitors, and multiplexers. If name is not provided, a name is automatically selected.

Examples edit

interps: Communicate with external processes via ycl chan interp.
http: Grab the contents of URLs.
ycl coro relay test: The test suite for the package.
ycl comm http: An http client that makes use of ycl coro relay.

Presentations edit

Coroutine is the New Main, EuroTcl 2016

Example edit

#! /bin/env tclsh

package require {ycl coro relay}

namespace import [yclprefix]::coro::relay

proc adder {} {
    while 1 {
        relay accept {deliver args}
        {*}$deliver [::tcl::mathop::+ {*}$args]
    }
}

proc multiplier {} {
    while 1 {
        relay accept {deliver args}
        {*}$deliver [::tcl::mathop::* {*}$args]
    }
}

after 0 [list coroutine main apply [list {argv0 argv} {
    coroutine a1 adder
    coroutine m1 multiplier
    set count 10
    set tasks {}
    for {set num1 0; set num2 100} {$num1 < $count} {incr num1; incr num2} {
        dict set tasks add [relay order 0 a1 $num1 $num2] {}
        dict set tasks mult [relay order 0 m1 $num1 $num2] {}
    }
    while {[llength [dict keys [dict get $tasks add]]]
        || [llength [dict keys [dict get $tasks mult]]]} {
        set received [relay receive]
        foreach type {add mult} {
            if {[dict exists $tasks $type [relay last]]} {
                dict unset tasks $type [relay last]
                lappend ${type}_results $received
            }
        }
    }
    rename a1 {}
    rename m1 {}
    puts [list {add results} $add_results]
    puts [list {mult results} $mult_results]
    exit 0

} [namespace current]] $argv0 $argv]


vwait forever

result:

{add results} {100 102 104 106 108 110 112 114 116 118}
{mult results} {0 101 204 309 416 525 636 749 864 981}

APN: How compatible is this system with the coroutine package in tcllib ? In particular, can one of these cooperating coroutines call commands from that package such as coroutine::util gets ? Or would there be a conflict between "competing" yields?

PYK: coroutine::util::gets is "blocking", and ycl coro relay receive is also "blocking". Both commands yield, and then expect to receive a response the next time their context is entered. Both commands can be used in the same coroutine context, but after calling ycl coro relay order or ycl coro relay call, the script should consider itself a callback script for the answer, and should only yield for that purpose. The same is true for coroutine::util::gets. It dominates the coroutine context, looping around yield until it gets what it wants. If something else happens to be calling the coroutine command for the same context in the meantime, that's probably a bug.

APN I don't see it as a bug. I see it as an incompatibility between the above framework and coroutine packages like the one in tcllib. Consider a "task" written in terms of your framework above which receives and responds to orders. This task makes use of a brand new coroutine based http client library that happens to make use of the tcllib coro gets to read from the socket. When the gets yields, some other task happens to send an order to the coroutine which now "wakes up" expecting the return value of gets but gets the accept arguments instead. Does chaos not ensue? I would not think it would be feasible to require knowing which library or package calls that a "task" makes might yield in this fashion. One possible solution, which I currently use in my fiber package, is to use an explicit queue which includes a flag that is set when the "task" makes a receive call. A message (order in your case) is not dispatched to the task unless this flag is set so the case I describe above is taken care of. Even then, I'm unsure it covers all cases of such conflict.

PYK 2016-04-26: That wouldn't really happen with this system unless the programmer "crossed the streams." since there's a one-to-one correspondance between a coro relay order and a coro relay receive, all a programmer needs to do is remain aware of what code ends up between those two statements. As long as a call to coroutine::util::gets doesn't get in the middle, it will work as expected, and both can be used in the same coroutine context. But ycl coro relay switch, or something similar can be used for the situations where the coroutine expects to be entered by various third parties. It's up to the programmer to devise some mechanism for identifying the caller, but ycl coro relay switch strongly hints at how one might go about doing that. One of the design goals of ycl coro relay was to not have any explicit queues, and so far, I'm proud to have succeeded on that point. I'm hoping nothing comes up that requires their introduction.

PYK 2016-07-03: Update: Although ycl coro relay is still primarily driven by the Tcl event loop, it does now make some use of queues, partly to address the issues APN and I are discussing above. This mitigates the potential for scripts to "cross the streams", but the issue still exists. relay accept is the point where things can get confused. More precisely, a coroutine that accepts multiple orders can't call anything else that might yield, except for receive which understands the situation and can work with it. If a coroutine does need to yield in some other way, one approach is for the coroutine to only accept once. This turns out to fairly workable since, as noted in Representing Control in the Presence of One-Shot Continuations

, Carl Bruggeman, Oscar Waddell, R. Kent Dybvig, 1996, most continuations are called only once. Another pattern I've had success with is to have one coroutine iteratively receive orders via accept and spin up a new a new coroutine to do the remainder of the processing, which may include various calls to yield, and then make the delivery. See the interps demo for an example.

PYK 2071-06-03: APN's analysis above was spot-on. ycl coro relay now takes care to only dispatch an order when the fulfiller is accepting orders. A record of order fulfillers is kept, and accept and receive both set an "accepting" flag in the record for the current coroutine before they yield. When an order comes up for a coroutine that isn't accepting, that order is shuffled back onto the queue. This continues until the fulfiller is accepting again.

coroutine