Updated 2013-04-29 19:01:12 by pooryorick ▲

Tcl's "exec" command is ... I quote Cameron Laird:
 > One thing that is grossly underappreciated is how well
 > [exec] and [open] (and [bgexec]!) work across Unix and
 > Win*.  Unless you've lived through it, you can't imagine
 > how much pain people working with other languages endure
 > to
 >   exec $myprocess << $some_input
 > or
 >   open |$myprocess r+
 > In fact, the reality continues (in 2001!) to be that
 > several leading languages just give up on proper semantics
 > for Win*, because they believe the problems are insur-
 > mountable.

But still, exec has one bad shortcoming, that is nearly impossible to come by:

  • because the meta-arguments for redirection are mixed with normal arguments (for the external command), it is impossible(!) to pass a normal argument which looks like a redirection. This goes as far as that it is impossible to pass a string starting with a `<' or '>' character.
  • because exec wants the command line spread out argument by argument, it is quite a pain (involving eval and all lots of extra [list]-protection of the other arguments) to include a Tcl-list as part of the argument list.

Example:
 % exec  /bin/echo "<hallo>"
 couldn't read file "hallo>": no such file or directory
 %

A more practical example (someone posted it on c.l.t)
 % exec gvim --remote-send <Esc>

Actually, there is a unix-only workaround:
 % exec sh -c {gvim --remote-send '<Esc>'}

but this involves one extra shell to be invoked, and puts you into the hell of quoting for tcl and then for sh ...

29-jul-03 Note: Due to recent announcements, some of the stuff below soon becomes obsolete.

  • A patch was announced, to compatibly solve the problem inside "exec", using a new option "-quote".
  • Some activity about argument-expansion has come up. (TIP 144)

Extended APIs as proposed below can then be implemented as wrapper-procedures to the new enhanced "exec".

24-feb-05 SLB: well, the limitations of exec remain an issue for some of us. I've made a stab at tackling this and have submitted it as a TIP [1]

To work towards a real solution of this problem, I've proposed an alternative API to the awesome work that made exec possible.

!! I do NOT propose removal nor (incompatible) modification of "exec" !!

Now, there are generally a few possible ways to do it:

  • Use the same bourne-shell-like syntax, but add a way to distinguish the redirection-arguments from normal arguments.
  • Invite a new syntax for redirection that is more tcl'ish, by consisting more of readable words than character sequences.

These items are discussed in the next few sections below.

To me it looks pretty impossible to do it with compatible changes to exec, which means that for compatibility-reasons we won't get around making this a new command.

My original suggestion for the new command name was 'pipe', but in a mail someone expressed a dislike for this name and suggested 'run' / 'bgrun' instead. I could live with these, too :-)

Preserving bourne shell syntax:
 <file, >file, >>file , >&2, 2>&1, |, ...

The simplest way would be to define one marker-argument, e.g. a single or double dash "-", which would remove any meta-syntax meaning of the next argument:
 % pipe /bin/echo - "<hello>" - - >file

would then write the string "<hello> -" to the file named "file". (the doublequotes are actually redundant here, but improve readability)

Although it looks very close to current exec-syntax, it is not compatible; especially not design-bug compatible :-)
   Pro: easy to implement, and the result is safe.
   Con: the syntax is even obscurer than before.

More Tcl'ish syntax:

Like above section, but all the somewhat cryptic symbols are replaced by dash-option (some of them with arguments):
 % pipe /bin/echo "<hello>" - - -outtofile file

The "<Hello>"-strings needs no more protection, the single dash (the second one!) is protected by the preceding one, and -outtofile specifies where the program's stdout should go to.
   Pro: still easy to implement (& safe); clearer meaning, better readability
   Con: Tendence to over-verbosity; a whole large bunch of options to learn.

It should also be possible to mix them, allowing both "<file" and "-filetoin file" ...

Think about a bunch of new features that could be added, when we're at it.

  • embed command and arguments in sublists:
 % pipe {mail -s} - [list $Subject] - $Recipients

This would save us a lot of dangerous eval's. All dash-protected arguments and all other non-redirection-arguments are concat'ed to form the external command line. Redirection-options and their arguments will not be treated as lists, which makes a big difference to the dangerous eval-constructions so far.

  • add new redirections:

for backgrounded jobs one could add redirection to tcl-channels, by which the functionality of '[open "|..."]' would be subsumed and extended, as stdout and stderr could be separately processed in the Tcl-scripts.

for synchronous jobs, one could redirect stdout and stderr to variables, which is what was most often done to the result of "exec", anyway. If stdout is not redirected, it would be returned as in "exec".

  • tee'ing:

why not duplicate the output of a command to both a variable or tcl-channel AND to the input of the next command in the pipeline ?

  • what else ?

Comment: even more verbose variant

Setok: I fully agree that there are problems with exec, and the proposed system solves those problems. However, I also proposed an alternative syntax:
  pipe {
    inputFromFile filename.txt
    command {wc -l}
  } {
    command {mail -s "wc -l" setok@fishpool.com}
  }

This avoids the readability problems you can get with heavy use of dashed options. It's not quite as straightforward as the original proposal but can make complex combinations much cleaner syntactically. The only real issue is that internally substitution would have to be performed on the blocks so you can do the following:
  pipe {
    command [list wc $wcOption]
  }

This can complicate matters a bit for the implementation, but it might be worth it for the cleaner syntax. Naturally I'm willing to hear about any other alternatives...

Andreas Leitgeb: Now that I read this suggestion again, I think it would indeed be nicest to have. Maybe this could be implemented as a tcl-prc wrapper around one of the first two solutions.

it would work like this: for each block, pseudo-commands "inputFromFile", "command", etc. are defined, then the block eval'ed, and the action of each of these pseudo-commands would be to build up a "newexec"-argument-list. the separate blocks are then joined with a "-pipe" or "|" argument.

Thank goodness exec will still exist - I can't imagine ever using this interface, but I have no objection to it existing for others. I just avoid the nasty characters myself...

AvL: does this refer to the whole alternativeexec discussion, or just the later (more verbose) suggestions ? Why do you think it is _posssible_ to avoid nasty chars (short of avoiding exec altogether) ?

2011-02-27 (ali)

I have a suggestion (I call it exec2) that I believe is like the "pipe" suggestion above, but with much more concise syntax:

In the foreground case, new redirection operators
>%, >*%, 2>%, 2>*%, etc.

to redirect output to a variable. "*" means "-keepnewline". Analogous operators for appending and redirecting both stdout and stderr are also included in this suggestion. In the background case, new redirection operators
<%, >%, 2>%, >&%

assign a channel connected to the pipeline to a variable. As for the quoting issue, I suggest simply introducing a quote character, say ', that can always be put in front of an argument to make it literal, e.g.
exec2 command '$arg1 '$arg2

would guarantee that $arg1 is treated literally even if it is a redirection operator. I chose single quote because it isn't a Tcl quote character (and thus doesn't needed to be escaped).

I wrote a UNIX-only implementation in Tcl on top of exec and open.

http://bazaar.launchpad.net/~li-anye-0/+junk/tcl-stuff/view/head:/exec2.tcl

Because it is built on top of exec, I couldn't resolve the quoting issue (without using sh, losing the ability to redirect to Tcl channels). It's also not perfect in many other ways but should be good enough to demonstrate the concept and maybe good enough for real world use for some people.

You're welcome to add remarks and further suggestions. (Please above this line. Thanks)

One final statement (sorry for the Caps): THIS PROPOSAL IS DOOMED IF NO ONE VOLUNTEERS TO IMPLEMENT IT!

Category Suggestions