expr shorthand for Tcl9

This page collects various suggestions from the Tcl 9.0 WishList about shorthands that one could use instead of the expr command. The original numbered suggestions were

34. GPS: I would like the math commands + - * / in the standard Tcl 9.0.

and

41. Larry Smith - some special syntax to invoke expr

the discussions of which follow below (slightly reordered). Two subsequent suggestions are

Lars H - extended $ substitution as expr shorthand

and

Martin Lemburg - special quotation to allow the bytecompiler to recognize expression

NEM 2009-03-20: I've created a patch for Jacl that implements (...) as shorthand for [expr {...}] at [1]. It also includes some new commands as alternatives to if that use the syntax rather than building-in expr, so you can write things like:

proc fac n {
    when ($n <= 1) { return 1 }
    otherwise { return ($n * [fac ($n-1)]) }
}

It's all very experimental, and doesn't pass the Tcl test-suite: doesn't conflict with arrays (except the empty named array), but does conflict with some uses like [regexp (a|b|c...) ....]. Meant as a feasibility test and fun experiment rather than for serious use.

TCV 2009-03-20 Does [set xyzzy(1+2) ohai] set an array element or xyzzy3?

NEM Try it. It sets an array element. As for braces and quotes, the expr syntax is only valid at the start of a word.

Steve Bennett 1 Nov 2010 I experimented with adding expr shorthand to Jim with $(...). I know that some will complain that this conflicts with the empty array syntax in stooop, but Jim is less burdened with strict backwards compatibility. This is a very natural syntax, similar to bash $((...)) but less likely to conflict than plain (...). I think that it is a big win for readability and I am strongly inclined to keep it.

. set x $(3 + 4)
7
. incr y $(7 * [incr x])
56
. set z $("bb" in {aa bb cc})
1

Talk

GPS: I would like the math commands + - * / in the standard Tcl 9.0. Expr is awkward, and having to use braces to workaround the byte-code compiler sucks. The issue with this as I see it is whether to make them operate on longs/ints or doubles. How can we solve this?

Lars H: You can of course have them right here and now. Just define

proc - {term1 term2} {expr {$term1 - $term2}}
proc + {args} {
   set res [lindex $args 0]
   foreach arg [lrange $args 1 end] {set res [expr {$res + $arg}]}
   set res
}
proc * {args} {
   set res [lindex $args 0]
   foreach arg [lrange $args 1 end] {set res [expr {$res * $arg}]}
   set res
}
proc / {numer denom} {expr {$numer / $denom}}

and you'll be able to write things like

+ 1 2 [- 3 4] [* 5 6] [/ 7 8]

for 1+2+(3-4)+(5*6)+(7/8).

The idea that one should decide whether mathematical operations operates on "longs/ints" or "doubles" shows clear signs of too much C programming. In Tcl, everything is a string.

LV In addition, with Tcl 8.5 one has the tcl::mathop namespace, from which one can import the operators as commands as well. So this wishlist item seems, at first glance, to be fulfilled now, rather than in Tcl 9.0.

ZB 23.09.2008. Personally I couldn't understand, if someone finds the solutions proposed above as "not enough". The first one looks just obvious, and usage of namespace will be especially convenient in case of math-oriented procedures. Besides: although "expr" syntax isn't especially pretty, but it's not that ugly anyway; one can live with that. Not sure, is really that "search for new delimiters" (and so on) worthy any further efforts.

peterc 2998-09-24: I wouldn't think there'd be many (any?) scripts which name their procs solely as a number. Surely a nice simple way might be to test the proc name to see if it's numeric, then if so, send it to an expr-like math proc. Then you could have more eye-friendly commands like:

set a [12 + 5]

JJS 2009-05-14: I once wrote an application with some procs named as numbers. It received messages with a header consisting of a command code, a length, and data. Once it had a complete message, it invoked the command code as a proc, passing the data payload and some other info. To define a message handler you'd write something like

proc 12345 {data len chan} {...}

where 12345 was the command code. I was a relative newbie back then, but it worked.

Martin Lemburg 17.02.2003:

In Tcl, everything is a string.? No, definitely not!

For calculations it is definitely important to handle number values only with commands that don't change the internal tcl type to string, so the exact representation is not lost!

If you don't care, if you don't have to care for the internal representation, than you can act like [i]n Tcl, everything is a string.!

We have many matrices to be multiplied and so we have to care about the internal type and not decrease the calculation's precision!

RFox

The concept of an <em>exact representation</em> is nonesense. The best you get on any computing device is an approximate fixed precision representation. In Tcl, you can set the precision to be what you want it to be, but it's still the precision of a stringized - fixed point representation that is just as exact or inexact as the binary fixed precision representation.

Lars H: That everything is a string is one of the basic principles of Tcl. It is quite true that the internal representation of real numbers currently contains more information than the string representation -- this is the subject of the A real problem page -- but IMO this is a bug in Tcl. (It can alternatively be seen as a malfeature of the tcl_precision variable.) The correct way (used by e.g. Knuth for TeX) to handle this problem is rather to make sure that the string representation of a number contains enough digits to uniquely specify all bits in the internal representation. This will mean that you generally "cannot trust the last digit" and will see numbers written with more digits than "what is really there", but since this is always "inside the program" that should not be a problem; anything shown to a user should be explicitly formatted.

Finally, if you are interested in maximal precision then you should probably read what Robert Heller writes on the subject of guard bits on the A real problem page.

Larry Smith - some special syntax to invoke expr - () would be best. This would make code much easier to read and more compact.It would be fairly easy to turn the feature on and off for backward compatibility.

This is not in the Tcl Way. If I (FW) had it my way, the correct thing to do for such a radical addition in that area would be to add math procedures to eliminate the need to use expr, for those who prefer a more purely Polish syntax.

LV In other words, FW, you would prefer to say something like

        set a [add 2 3 [divide [multiply 4 5 ] 7]]

YUCK!

FW: Well, I'm not sure what the command names would be, perhaps they would just be named after the math operators, as in GPS's suggestion in wish #34.

LV I still respond YUCK - I find reverse polish notation difficult to read and more difficult to code. I prefer the notation currently used by expr.

escargo 4 Jan 2006 - Just for the record, I must point out that this is Parsing Polish notation not rpn.

Lars H: Another possibility is to extend $ substitution. I have seen references that it was originally introduced as a shorthand for [set ...], and in view of this, it would be natural to consider making a form of it which is a shorthand for [expr {...}]. How about

set res $={2*$a + 3*rand()}

as a shorthand for

set res [expr {2*$a + 3*rand()}]

i.e., when $ is followed by = (a case not explicitly mentioned in the Tcl syntax rules, but in which currently no substitution will take place) and something in braces then these braces contain an expr-expression rather than the name of a variable. What one should perhaps think hard about here is the possibility to get rid of the need to put $'s before variable names in these $=-expressions. Simplicity and compatibility with expr suggest that they should be required, but forgetting the $ in expressions is probably the most common typo I make. This could be a sign that they are unnatural.

The tcl parser is a simple one - trying to replace [expr ...] with $= seems to me to be asking to break the parser for minimal benefit.

As for a new command to which one need only supply variable names, and not $, that too violates the tcl parser's rules - to get a reference to a variable, use the name only, but to use the value of a variable, precede the name with a $. It doesn't seem to me that the resulting command is going to be all that much beneficial, and on the contrary, it seems like it is going to make tcl slower, more complicated, and more difficult to teach and to use.

In bash one can write things like

$(($i+1))

for what would be

[expr {$i+1}]

in Tcl. The former currently refers to an element in the empty name array in Tcl, but if wish #39 (the empty string is no longer the name of a variable) was to be implemented, then the bash syntax (with double or single parentheses) could be used as an expr shorthand in Tcl. (In fact one could take the implementation of this syntax in Tcl 8 as an exercise in trace hacking. Set up a read trace on the empty name variable, and pass every index through expr to compute the array element value. Should work, but there would be a ridiculous amount of shimmering.) However, I think $={...} would work better syntactically than $((...)) or $(...). The best case for the latter two is the analogy with bash.

Larry Smith $(...) is not bad at all. "$(" is not currently defined so it is backward-compatible, and the () notation is natural for arithmetic expressions. It also plays well with the idea of $[] to expand the results of a command into a list. It even suggests that ${} is a cleaner and more logical way to indicate the "expand w/o eval" that we currently use "" for. This has always felt wrong because "" is otherwise meaningless in Tcl, and is frequently ignored or interpreted oddly (as in "10" == 10 in expr).

ulis Sorry, "$(" *IS* currently defined (empty named array):

set (1+2) 4
puts $(1+2)
->4

Setok -- Somehow that looks perlish. I prefer using the command: +, -, *, / etc. Result in lines like set a [+ $b $c $d]. Indeed, I use this form all the time.

Lars H: Whether it looks perlish is completely irrelevant (even though it is a classical killer argument in Tcl discussions; $ substitution got that critique too, in the beginning). What matters is whether is would be perlish, and I don't think it is.

As for having the math operators as explicit procedures, I think it would be a good idea to add a package for this to tcllib ASAP.

Sarnold In my little experience of Perl, I was surprised I could quite not force the interpreter to handle my numbers as integers. (they were automatically casted to float)

IMHO it is quite better to have both the casting operators, and the math operators as commands. I hope those changes won't make things harder to understand. IMHO it is expected from a developer to have skills with floating-point numbers and integers. Many people don't think so, but it is awfully too often required to know the limits of our machine-related number representations.

LV anyone who wants to write such a package, and submit it to tcllib, could probably get it in pretty quickly. And releases occur frequently enough that it would end up in the library distribution in a small number of months.

There is no compatibility problem, and if people like them then they could get byte-coded by Tcl 8.5. And it would still be possible to provide another shorthand like $= in Tcl 9, if some such suggestion is approved of.

Setok Well my point was not that being like perl is always necessary bad, but I find $={$a + $b} a bit strange to read, although now that I'm not so tired as when first reading this page it's not as impossible as I reckoned. As you mentioned opers-as-procs is a good compromise and I never liked expr having its own functions anyway (a [rand] procedure would be much more natural than rand()).

As long as expr continues to work, other syntactical sugar notations are easy enough to add on. I would not like to see such things become a part of the tcl.tar.gz distribution. However, if they were part of tcllib, that would be fine.

In ksh one could write things like

$(($i+1))

let $i = $i + 1

for what would be

[expr {$i+1}]

incr $i

in Tcl.

Lars H: I hope anonymous at 134.243.40.138 will refrain from making non-editorial changes to signed material in the future.

Martin Lemburg 17.02.2003:

if there is a possibility to mark expression with a special quotation to allow the bytecompiler to recognize expression, than it would easier a lot the writing of calculations.

What's about:

set pi '4 * atan(1)';

And ... even if it's not tcl-ish to do, but if we break tcl style by using "()" for functions, why not applying the nice patch for expr to allow every proc and command to be invoked in the same way and to access variables without the "$"-sign?

Why not ...

proc pi {} [expr {4*atan(1)}];
proc perimeter {r} {expr {2 * r * pi()}};

proc pi {} '4*atan(1)';
proc perimeter {r} {'2 * r * pi()'};

Lars H: The "special quotation" is probably too radical to meet general approval -- cf. the hassle with {*} (Tcl9 wish #3) -- but you're on to something concerning access to variables in expressions without using $. That would be very nice!

AJD Don't personally see any improvement in the above forms over the simple use of "$pi" or "[pi]". However, the use of argumentless functions within expr is something I've hacked around with... in particular, the case above of [expr { pi() }] to denote a constant. The core doesn't currently have any such beasts but they can be defined using the API function Tcl_CreateMathFunc. The idea being the core could define PI and other 'well known constants', saving users having to do it (possibly in a variety of places). There's even the opportunity for a teensy weensy performance gains (e.g., inline "2*pi()"). Maybe the ability to define "expr constants" could be exported to the script level... Overall though, probably doesn't buy enough beer to get taken to the bar.

Maurice.Diamantini (03 mars 2003): I'd like to see the functionality from [vexpr] or the vkit (is it dead ?) to be integrate into the expr command.

so that one could write

set a [list 1.0 4.0 3.0]
set b [list 5.5 6.6 9.8]
set c [expr {$a*$b}]

Another amelioration I'd like in tcl9 is to extend incr, and add a few other command like mult to work with other type than int (double or list ) (il c ist the liste above:

incr c 1    ;# add 1 to every element of the list c
incr c 3.1  ;# add a double
incr c $b   ;# add another list

RJM Picking up the suggestion above to use simple () parentheses: Why not doing it without preceding $? Just as "[" marks the start of a recursive script, "(" could mark the start of a math expression.

(simple example)

set y ($x*$a + 3.5)

Note that the shortcut should only apply to expressions that can be evaluated in an expr {...} command notation. In the case the expression cannot be evaluated with {} quoting, the conventional expr command should still be applied: This is the case where STRINGS must be evaluated as a math expression. In other interpreter languages it is also usual to provide a command for such cases.

Lars H: It would certainly be a language that made sense. The big problem with making ( behave like [ (but for expressions rather than scripts) is however that it makes another character special, and thus breaks every old script in which that character wasn't protected. In particular the syntax for array elements would get into serious trouble. I suspect the change would be too great for any Tcl version increment. Math operations as explicit procedures are probably the way to go.

Larry Smith: Actually, the more I look at arrays the more I think making them part of the language was a bad idea. An array should be accessed the same way other data structures are, with normal tcl syntax:

array set foo $bar "a string"

and

set x [ array set foo $bar ]

set x [ array get foo $bar ]

It really has no call for special syntax. It would also allow people to overload the array procs to add capabilities we don't have, but frequently want:

array set foo $x $y $element

would permit "real" numerically-addressed arrays to use the same keywords (this example keys off of the number of args, but it could also look at the values of the passed args to decide if this is a numerically-addressed array or a hash table.

RJM: I revised my contribution above. "[" may also appear without space as to execute a script. However it does not mean that "(" as an expression shortcut should also evaluate an expression when it appears without a leading space. I suppose expression results mostly happen to appear as command arguments. When it happens to appear inside a string, the shortcut shall not apply. In such cases, the expr command should still be used. When a variable with trailing space exists, there will be a problem. But I hardly believe that anyone would define variables with trailing spaces. Hence, backward compatibility shouldn't be a problem. Moreover: To my opinion, backward compatibility is not a holy cow. It is consequent backward compatibility that made MS-windows a monster. Consider also that scripts can easily be adapted to conform a newer Tcl version, using a script.

Another idea (yep, this is a brainstorm page): When "(" is not acceptable, a new command "=" should. For evaluation purposes, enter the following proc:

proc = {args} {expr $args}

According to my previously stated condition, this command should only work with true expressions (i.e. be equivalent to expr {text}). This way, any [= 5*1.2 + $a] inside a script text would execute the fast way round. Oops! (note few weeks later) This intention violates with Tcl rules - Tcl would do substitutions before passing the argument to the "="-command. So the fast way round is never possible with a new command. Conclusion: only a new substitution type will do (see above: $={...}, '...', $(...) and (...) ). Given the complaints with "(" I vote for $(...). Uhh, this is empty array access (see above)

Recalling backward compatibility issues, I'd recommend a little bit more courage in changing things in Tcl. expr is historically given, because everything was a string in Tcl. When no shortcut will be introduced, I believe Tcl will early or late fade out in computing history, simply because the choice in scripting languages is really large. Anybody who is selecting a scripting language will tend to not select Tcl when he/she encounters "set a [expr {....}]". This is Tcl Marketing-relevant!

RS: You can save a level in call stack by using this instead:

interp alias {} = {} expr

(RJM: still for evaluation purpose: executes slower and does not include {} in expr).

DKF: The problem with special syntax for expressions (or at least special syntax that isn't standard command invokation or the = alias) is that this is a major departure from the existing rules; you'd be making some kinds of syntax special, and one of Tcl's key strengths is that there are no special global syntaxes. The expr syntax is always just a local syntax.

A few notes here...

Unbraced expr was special in the alphas of Tcl 8.0, but this was deeply unpopular with many very influential Tclers.
It is possible to get commands that are as efficient as the expr bytecode, or rather it would be if it wasn't for the fact that we're not very proud of the actual compiler API and so haven't exposed it publically. :^)

slebetman 15 Mar 2006: By unbraced exprs do you mean [expr $foo + $bar]? Why was special treatment of unbraced expr unpopular? Was the C code ugly? To me braced exprs are unpopular to the extent that I'm often willing to sacrifice speed rather than break my old habbit. If special treatment of unbraced exprs were already coded can't we just put it back in so that we don't need the braces to bytecompile exprs. That will also make things like let faster since currently the expr that it depends on is not bytecompiled.

RJM It can be seen, that no practicable shortcut for expr {...} seems to exist. Therefore, another approach should be discussed. Why not let appropriate commands expect an expression in one or more of its arguments, like the if command does. For example coords arguments for canvas operations. They must always be numeric, so rewriting commands involving coords to expect expressions would allow us to embed calculations within such commands in a more compact fashion, which improves readability. I also think this would not affect backwards compatibility problems, since numbers are already "expressions". Further, existing code, where [expr {....}] is already embedded, also yields no problems. I would see this idea closely related to what is written in What some command arguments expect. If the idea is interesting, an inventarisation of commands with this respect should be undertaken. Further, some new commands could be defined, such as xset/nset for expression/numeric set etc.

KPV See tip #176 [2]. Personally, I like what that tip proposes--it would clean up most of the cases where adding expr gets really awkward. But I've not heard much momentum behind it.

Lars H: Another TIP which should be mentioned on this page is #174 [3]. It is pretty much what is listed as Wish #34 above.

HGO: And yet another TIP which should be mentioned on this page is #232 [4]. It may help in many of the above stated cases.

AMG: What about $[...] being shorthand for [expr {...}]?

I saw mention of $[...] above, but if I read correctly it was in regards to a feature already provided by {*}[...], so this character sequence been freed up. Does anyone currently use this sequence? At the moment it means: eval the script "..." and prepend the result with $.

We'd have to change the parser to quote between $[ and ], and it would have to handle nesting correctly. Important inconsistency: we do not do this for $(...)!

07nov05 jcw - This is not necessarily an inconsistency. [...] gets parsed as usual, the $ in front of it takes the inside (say $x) and replaces it with [list expr [list $x]] - IOW, "$" before a [ gets interpreted as "expr" after the [ (more or less, the arg also needs to be treated as surrounded by {}'s).

AMG, 9 Dec 2005: That's not quite right... expr [list $x] and expr {$x} aren't equivalent. Surely you mean the latter.

The inconsistency I refer to is $[...] having implicit {}'s but not $(...), and you haven't addressed this. Maybe I just don't understand what you're trying to say. Care to elaborate?

Question: Should $[...] be recognized by the [expr] parser as well as the Tcl parser? I imagine $[...] inside an [expr] expression would have precisely the same effect as (...).

Larry Smith How about this idea, then:

interp shorthand <srcproc> startchar endchar

This would allow creation of application-specific shorthand expressions. For ordinary tcl, one might declare:

interp shorthand expr ( )

Thereafter the interpreter scanner would spot (...) and turn it into [ expr ... ].

This also allows people with full unicode editing to use things like 〚...〛, ⌊...⌋ or even <...> as shorthand for various options, which could include things like constant constructors, object calls, regexp sugar, whatever.

Lars H: The problem with this is that is changes Tcl syntax in rather unpredictable ways. list-quoting is one issue -- if list is to be usable as a generic command constructor (which is very established in Tcl) then defining "shorthands" like this requires changing the canonical representations of lists to ensure all shorthands to be quoted! That quickly becomes very messy.

Larry Smith Actually the above was rather tongue-in-cheek (and I was amazed by the variety after a very cursory look through a unicode font), but truthfully, this doesn't change tcl's syntax except to provide semantic meanings for new symbols in a context that right now only permits {...} and [...]. The symbols listed would mean nothing if they were inside pairs of the existing delimiters.

Lars H: (Sigh.) Of course they wouldn't be subjected to substitution within {} (however, they would have to be subjected to substitution inside brackets, as that is just another command -- think your syntax suggestions through before you post them!), but that's no help to you. Tcl does not have any context that "only permits {...} and [...]". "Barewords", as they are sometimes called, are legal everywhere and the string representations for lists uses them not only for very simple elements without whitespace, but in fact relies on them for the trickiest cases.

Larry Smith I do think my syntax suggestions through before posting them. Just because you don't like them is no excuse for such a remark.

As far as tcl syntax and parsing go, the scanner is the one place in the interpreter where "", [] or {} have meaning. Providing new delimiters simply converts an extremely unlikely string containing such delimiters into something tcl recognizes and executes.

 set x 〚foo〛

...under the current system sets x to 〚foo〛, true enough - but that is an extremely unlikely sort of string. Giving it a semantic - that is, extending the scanner to recognize ĺ - only breaks a very unlikely case. Frankly, if you are coding that way, you deserve to have your code broken.

Lars H: Well, a rereading with your reply in mind indicates that what I took for a lack of thought may simply have been an unclear re-presentation of the exact same ideas as that you started out with, but as it stands it really sounds as though you introduce the extra principle that "expr substitution" should not be triggered between brackets, which would render it of limited utility. Also note that brackets and braces don't have anything in common syntactically; there is no "one place" in which both are handled and (unlike brackets) braces are only special at the beginning of a word.

Regarding your argumentation:

I personally don't write things like "set x 〚foo〛" right now,
but people who do don't deserve to have their code broken.
Indeed, the argument that this "is an extremely unlikely sort of string" is the syntax analogue of the classical argument that the government should be allowed to violate human rights: "We probably won't do it to anyone you know (and in the unlikely event that anything bad ever happens to anyone because of this then -- as is evident from us even invoking this extraordinary law -- these people obviously must have been evildoers anyway)."
None of this has even begun to address the problem of list-quoting that I raised. If string representations are not changed to quote new shorthands, then you open up for attackers getting malicious code evaluated. If string representations are changed, then you have to make them interpreter-relative, but Tcl_Objs don't belong to a particular interpreter!

Sure, if you implement a Tcl/2 then you can make its syntax mutable to your heart's content, but doing so is far from free from side-effects, and you do not seem to appreciate that.

Lars H: My present (2006) opinion is that shorthands in general are unTclish. What should be done instead is to create more capable little languages, in which syntax rules could be more algebraic. Rather than

set r <hypot($x,$y)+1>
set x1 <$x/$r>
set y1 <$y/$r>

where there is constant switching between Tcl and expr, one should do also the variable assignments in some expr-like language, e.g.

calculate {
   r = hypot(x,y)+1;
   x1 = x/r;
   y1 = y/r;
}

Larry Smith I like the idea, but the real problem here is the messy syntax that arises from the need to calculate expressions being provided to other commands. List extraction, for example, where the first index is given (j) and the end is computed: (j+5). The innocuous-looking (j+5) balloons into [ expr { $j + 5 } ] (not forgetting the need to curly brace arguments to expr).

Lars H: TIP#176 has already adressed that problem in the case of indices. For other small calculations TIP#174 provides a nice solution. That only leaves the high end of expr usage to deal with!

Larry Smith #176 works but it does put the onus on the command to provide such functionality. #174 is not, IMHO, a nice solution. It is at best a slightly less ugly one.

In all honesty this is a very restricted domain, and becomes even more so if commands can accept simple expressions like $j+1 or end-1, but it's a wart, and it adds complexity to commands to support this. General access to simple computations is really needed, but I'm begining to think expr is over-specified. Expr also supports floating point, trancendental functions and lots of other stuff not really needed to solve the problem we really are trying to deal with here.

<sarcasm> Now that we've broken down and added {*} maybe we want to extend the idea to include {compute} - that is, expand and expr in context - where {compute}j+1 is read as [ expr { $j+1 } ]. </sarcasm>

In any case, expr is really starting to rub me the wrong way. As a means of computing indices or other such trivia, it is way over-specified. As a means of doing advanced math, it's way under-specified. As a means of doing other types of computation problems, such as matrices, vectors, or what-have-you, it fails utterly, yet it remains the repository of the logic needed to build upon and so insinuates itself into every solution, and again our code grows cluttered with [expr {...}].

I'm thinking more and more fondly of Tcl/2. And even more after this last exchange.

AM (17 march 2006) Actually a procedure like [calculate] has already been written - A little math language revisited

Gosh, it would be nice if the mechanism of expr was expanded to vectors and matrices and what not ... I have contemplated and started several experiments in that direction. All the ingredients are there. Now it is a matter of concocting the right receipe and bake the penultimate script. Main problem: time.

Sarnold (7 sept 2006) I am currently experimenting a syntax

{math}{...}

to be handled by a modified Jim interpreter. What I expect is RPN generated at parse-time from expr-like syntax. This means operators and math functions should be available as commands at Tcl-level.

Example:

{math}{$i + 2 > 0}  -> [mathfunc.gt [+ $i 2]]      (gt = greater than, second arg defaults to 0)
{math}{$j && $i>0}  -> [and {$j} {mathfunc.gt $i}]

(and is a command, but to allow lazy evaluation, it needs braces around its args)

But I face a huge problem : I still need the expr parser because, if I wanted to bash it, I had to write {math} before each expression, including if, while, for, etc. Though, there is an advantage to this approach: you CAN redefine math operators to handle vectors, matrices, bignums, everything you need... (because +, -, *, / are commands in Jim) But don't forget it is *experimental* and does not concern Tcl but a little bit.

CMcC Mon 11th Sept 2006 sees no need for incorporating infix notation in tcl's global syntax - it's a major global change for not much local virtue and significant global cost (IMHO), I refer you to little language for a growing collection of the places in tcl where specialised parsing is contained within a linguistic {}-scope.

However, I would like to see expr syntax expanded to permit #-comments, because it would be nice to be able to comment complex if conditionals.

NEM: I actually quite like the idea of () as syntax for expr, but it would have to be done in the same way as {} and "" -- i.e., ( is only special at the start of a word. This would rule out any clash with array notation, and also with parens in ""-quoted strings (which are rife). The resulting syntax seems quite elegant for the common cases:

lindex $xs ($ptr/2+1)
set res ($expr + [mycommand ($subexpr)])

It doesn't really work for while or for, where the expression needs to be evaluated repeatedly (I'm assuming eager evaluation of expressions, i.e., (...) is equiv to [expr {...}]), but I can live with those as they are already concise except for the nested sub-expression case, which this proposal also nicely handles:

for {set i 0} {$i < [lindex $foo ($x+1)]} {incr i} { ... }

You could clean up if by making it accept a plain boolean value rather than an expression. That would result in syntax like the following:

if ($a < 12) { ... }
if [mycmd ...] { ... }

Which both seem nicer to my eye than the current braced varieties.

Justification for adding a new global syntax? Well, firstly, I think expr is probably important enough to warrant it. Use of mathematical expressions is extremely common. If I was looking for parts of the language that justified syntax, expr would probably come after $-syntax for variables, but before either array syntax or {*}. I also think it makes the language more newbie-friendly, as explaining expr and about always bracing expressions is perhaps less than inspiring. Finally, most of the alternatives (if you believe there is a problem to be addressed at all) seem to involve adding features to the expr little language, until it is no longer a little language, but actually quite a large one. With comments, assignment etc, the emphasis seems to be on including more and more of your source code inside expr -- the logical conclusion would be that eventually all source code is in expr syntax and Tcl becomes the little embedded language! I'd prefer to keep expr minimal and just give it some sugar.

RS: Note however that the (...) syntax isn't free for the taker any more - it rather refers to the array named "":

% set (x+1) 5
5
% puts $(x+1)
5
% parray ""
(x+1) = 5

At least Stooop (for which this feature was introduced) would break if (...) received a different meaning.

NEM: Note that $(...) could continue to work. Only [set (...) ...] would break. You could use [array set] to work around that (perhaps aliases to [self]?), but it would definitely require a change to Stooop. I think it would be worth it -- empty array names are cute, but little used.

CMcC (Wed 13th Sept 2006) Discussion on tkchat suggests there's a problem with some aspects of expr being made into global syntax: specifically, &&, || and ?: operators explicitly don't evaluate their second argument in some cases. This is a major departure from the tcl evaluation/substitution model.

RS: On the other hand, you sometimes need such short-circuiting - consider these not atypical cases:

if {$d != 0 && ($n/$d) > 1} ...
if {$d == 0 || ($n/$d) > 1} ...
expr {$d==0? 0: $n/$d}

NEM: I don't understand. As I said, a global syntax for expr would be precisely equivalent to [expr { ... }]. Short-circuit operators would continue to work exactly as they do now in braced expressions. This is an argument against both maths operations as commands, and against changing Tcl_GetIntFromObj et al to recognise some cut-down expr syntax.

Robert Joy 2006-09-13 6:32am PST: While I respect that one might want to drive expr into a more common paradigm. I personally don't see a need to move in that direction. In some ways I prefer the use of expr as it it becomes abundantly clear that one is defining a mathmatical expression and not just creating a string that looks like an expression. For the sacrifice of a few extra characters to change to a new paradigm it's just not worth all the effort. Just my 2 cents for what it's worth and I certainly don't mean to upset anyone but I think those who originally created expr had the right idea.

WHD: For some time now, I've been using a helper command called "let". The two following statements are equivalent:

set a [expr {$a + $b*$b + $c*$c}]
 
let a {$a + $b*$b + $c*$c}

This doesn't help when you're using an expression in-line, as an argument to another command....but it makes complex computations much easier to read. - RS: Something like this?

proc let {_var expr} {
  upvar 1 $_var var
  set var [uplevel 1 [list expr $expr]]
}

CGM 2008-06-11: Another suggestion - use { .... }= as shorthand for [expr { .... }] . This is both compact and backward-compatible to the same degree as {*}, since it's currently an error "extra characters after close-brace". Using {} braces preserves the quoting semantics to avoid double-substitution of variables. There's an ambiguity over {*}= but the expand meaning should precedence here as [expr {*}] is an error anyway.

JMN speaking of {*} .. it seems to me that it would now be natural to use another such prefix for expr.. e.g {#}{1 + 2}

Looking above I see Sarnold has already suggested something like this with {math}{...}

JMN 2008-06-14

While at first glance this mightn't seem much better than [expr {...}], I think that in the vast majority of cases the braces surrounding the actual expression could be left off simply by formatting it without whitespace. E.g.,

{#}1+2

If a tcl command with whitespace is included in the expression, you still wouldn't need braces.

 {#}[llength $somelist]/2

tcltkd : 2008-06-12: A classic technique:

proc unknown {args} {uplevel 1 eval expr $args}

Is it dangerous? If is,some global interpreter flag should be employed.

NEM There's too many evals in there, so it is dangerous. A more correct version would be:

proc unknown args { uplevel 1 [list expr $args] }

But note that it has problems with some expressions, such as && and ||:

% [set a 1] || [set a 2]
1
% set a
2

Versus using expr directly:

% expr {[set a 1] || [set a 2]}
1
% set a
1

NEM A possible approach is to write a pre-processing version of proc that implements syntax for expr:

proc func {name params body} {
    regsub -all {([^\\])\(} $body "\\1\[expr {" body
    regsub -all {([^\\])\)} $body "\\1}\]" body
    uplevel 1 [list proc $name $params $body]
}
func foo {x y} { return (1+[lindex $x ($y*4)]) }

It is a bit too eager on substitutions, requiring backslash-escaping to include a literal parenthesis anywhere, and not respecting braces. Still, it does allow a glimpse at a remarkable reimagining of Tcl:

func if {cond then {_else_ ""} args} {
    set args ([llength $args]==1 ? [lindex $args 0] : $args)
    uplevel 1 ($cond ? $then : $args)
}
func foo a {
    if ($a < 0) {
        puts Negative
    } else if ($a > 10) {
        puts Large
    } else {
        puts Small
    }
}

FM I like the concept of little language. But we could need to obtain the value of each calculation, so we should return a list. If we write :

set L [math {
   x=0;
   y=1;
   z=$y+1
}]]

we should get :

lindex $L 0 --> 0 (x value)
lindex $L 1 --> 1 (y value)
lindex $L 2 --> 2 (z value)

Then, we should be able to write in tk for instance :

wm minsize . {*}[math {[winfo screenwidth]/2];[winfo screenheight]/2}]
wm geom . [join [math {[winfo screenwidth]/2];[winfo screenheight]/2}] x]

instead of :

wm minsize . [expr {[winfo screenwidth]/2]}] [expr {[winfo screenheight]/2}]
wm geom . [expr {[winfo screenwidth]/2}]x[expr {[winfo screenheight]/2}]

or somethings like

[canvas .c] create line {*}[set L [list]; for {set i -50} {i<=50} {incr i} { lappend L {*}[math {x=0.1*$i; y=$x**2}] }; set L]

Lars H: When only one result is wanted, it would be quite a nuisance to have to lindex it out of a list of results. The approach I took in infix was to introduce , as a list-construction operation — this way one can get the effect you ask for by doing

set L [infix {} {
   x=0;
   y=1;
   z=y+1;
   x,y,z
}]]

set L [infix {} {
   x=0,
   y=1,
   z=y+1
}]]

FM: I found it more clear to return a list. But maybe it should be another command which "list expressions"

proc lexpr {expr} {
    namespace path {::tcl::mathfunc ::tcl::mathop}
    set L [list]
    foreach ::EXPR [split [string trim $expr] "\n\;"] {
        if { ! [expr {$::EXPR eq "" || $::EXPR eq {\{} || $::EXPR eq {\}} }] } {
            lappend L  [uplevel {
                namespace path {::tcl::mathfunc ::tcl::mathop}
                expr [subst {$::EXPR}]
            }]
        }
    }
    set L
}

Test :

set Central [dict create {*}[lexpr {
    "Rayon"; [set r 3]
    "Perimeter"; 2*[set pi [acos -1]]*$r
    "Area"; [set A [* $pi [** $r 2]]]
    "Solid Angle"; 4*$A
    "Volume"; 4*$A*$r/3
}]]

I know that you've made a lot of works on these proc infix. But I think that implementing a little langage by this way make us loosing tcl inside. Will we need to develop a new parser ?

Lars H: My point was only that with a list construction operation, you can return a list as easily as a single value. How the little language supporting that operation is implemented is a separate issue.

Let's think about Tcl. On one hand Tcl is verbose, but on the other hand programs are tiny. Why ? It's seems to me thats it's because commands could easily communicate together. I come to LOP, we could have a whole set of little langage but the problem is to be able to make them communicate together. In tcl we could write things like this :

expr {[
       set x 1
       if {$x == 1} {
           set y [expr {[info frame]*cos($x)}]
       }
       set z [expr {$y+$x+[llength [info proc ::*]]}]
       list [info level] $x $y $z [winfo children .]
      ]}

This is why Tcl is wonderfull. In fact, with the introspection facility of tcl, I think it would be possible for all this command to "know" they are in the expr context. Let push further this idea. I could implement this only this with lexpr because "info frame" tells only which proc is the caller (but not a command like expr !). Before let's do an little proc as "set" :

proc = args {
    set dico [dict create {*}[info frame -1]];
    if {[dict exist $dico proc] && [dict get $dico proc] eq "::lexpr"} {
        uplevel "set [lindex $args 0] \[expr {[lrange $args 1 end]}\]"
    } else {
        uplevel "set [lindex $args 0] {[lrange $args 1 end]}"
    }
}

This proc is looking upframe, test if it is call by lexpr (the better would be by expr) and adapt himself. let's do some trivial test :

lexpr {
    [= x 1]
    [= y 2]
    [= z $x + $y]
}
-> 1 2 3

= z $x + $y
-> 1 + 2

The proc "=" adapt himself to the caller. Imagine this kind of things could be implement with set and other command like list, so that they detect an expr context and apply expr to their argument. We could write things like this :

set Central [dict create {*}[expr {[
                                    set pi (acos(1))
                                    set r 3
                                    set P (2*$pi*$r)
                                    set A ($pi*$r**2)
                                    list Rayon $r Perimeter $P Area $A {Solid Angle} (4*$A) Volume (4*$A*$r/3)
                                   ]}]]

Another way to achieve this is to use the proc know in a modified version of expr

namespace eval ::lexpr {
    proc know {cond body} {
        proc ::lexpr::unknown {args} [string map [list @c@ $cond @b@ $body] {
            if {![catch {expr {@c@}} res] && $res} {
                return [eval {@b@}]
            }
        }][info body unknown]
    }

    know {[expr $args] || 1} {expr $args}

    proc let args {
        eval eval [list expr $args]
    }
    namespace unknown ::lexpr::unknown
    namespace ensemble create -map {let ::lexpr::let}
}

Test :

lexpr let {[
     set pi [acos(-1)]
     set r 3
     list Rayon $r Perimeter [2*$pi*$r] Area [set A [$pi*$r**2]] \
          {Solid Angle} [4*$A] Volume [4*$pi*$r**3/3]
]}

In this way, propagating the context, Tcl could approach a contextual grammar (here in 15 lines of code) and all Tcl commands remain inside !

Maybe a flag could be send between TclParseExpr and TclParseCommand to keep the knowledge of the context ? So, called from TclParseExpr, TclParsecommand would first try to parse a script inside it as an expression and in case of error, parse it as usuel. This could be a new concept in tcl, the ability for the parser to "know" in which context it is, and so it could be implemented for other commands. In those cases (only for expressions in a first step), {[ ... ]} mean propagate the context.

Lars H: All that is really an orthogonal idea (not tied to expr, although it can be applied to it), but anyhow… As I see it, the main problem with this kind of "adapt behaviour to context" solutions is that they don't scale at all well as the complexity (number of possible contexts) of the system grows — there are simply so many things that have to know about each other to make it all work, that it effectively becomes unextendable.

It is often possible to achieve much of the intended benefits by using a namespace as "context", i.e., having different behaviours of a command really be different commands. Besides already being supported in the language, this also scales well, because it is not based on having the computer guess (is this a command or expression or ...?) what you meant by what you wrote! It doesn't support every syntax one might imagine, but it very often supports a syntax that is as compact and expressive as the one that might be found in other languages.

FM So you mean that the better way should be to give the hability to the programmer to implement context sensitive procedure rather than put it in the core, because the extrem complexity of the approch ? You certainly right. Additional tests to add would be a lost of time. In fact, tcl'ers don't have so many way to explore this complexity, and so to see how it is extrem.[ ] substitution is not context sensitive, substitutions have no "color", since it occure before the invokation of the command prefix of a line. To experiment such a behavior, I would like to write something like :

expr 4+[Tclproc $a+3]*5

... and be able to know I'm in the expr context, so I can apply expr to the arguments of TclProc. But, as far as I know, neither info frame, nor info level can give me the information about the first word of the command line beeing evaluated, but only the first word of the nested script currently evaluated. It should possible since the parser has seen the command prefix of the line before recursively evaluate the nested script. Maybe NRE could offer such an opportunity. Suppose we have a command : info lineprefixe

rename set ::_set
proc set {var val} {
    if {[info lineprefix] eq "expr"} {
        uplevel ::_set $var [expr $val]
    } else {
        uplevel ::_set $var $val
    }
}

expr {[set B 4+[set A 3+3]*5]}

set B [expr {4+[set A [expr {3+3}]*5}]

rename list ::_list

proc list args {
    if {[info lineprefix] eq "expr"} {
        foreach e $args {
            lappend L [expr {$args}]
        }
        return $L
    } else {
        return [::_list $args]
    }
}

expr {[set L [list 1+1 2+2 3+3 4+4 5+5]]}
set L [list [expr {1+1}] [expr {2+2}] [expr {3+3}] [expr {4+4}]]

This is limited to the cases where expr is the first word of the line, and equaly when all arguments of the nested command are to be evaluated as arithmetics expressions. There is also a (big)problem since all those procs above (set, list) become slower. Other ideas should be found

if I write at the beginning of a line

y=sin($x)/$x

I get an error. Of course y=... is not a command ! I could write that y is a proc

proc y {= args} {
    uplevel set y [expr {$args}]
}

But I'm obliged to write

y = sin($x)/$x

But if I write

y= sin($x)/$x
y =sin($x)/$x

it's return an error. So, I'm obliged to handle all this case (splitting, ...) with many procs and to define this procs for each variables ! I could use the unknow handler, but as it happens after everything has fail, it is too slow for arithmetic, and of course, won't work if y is a command. A solution I see, is a new interp command which would glob the command prefix and apply the good transformation to it.

interp match path <glob pattern> path commandTransformation

# detect a command prefix glob pattern and return the command to be executed
interp match {} *=* {} apply {{exp args} {lassign [split $exp =] var e; return {set var [expr $e $args]}}}

X=2*3+cosh(3)-5

A=[B=[winfo width .] > [C=[winfo screenwidth]] ? $C : $B]

The limitation is that all the command is affected. What if we want only a few arguments to be expr(essed)? Another idea, it is well know that :

{something}$List

always throw an error (except for {*}, the expansion prefix). An error in the parser means that we can extend the parser semantic in a backwarded compatibility way. The place has be done in the parser to handle such case. So it should be extend. A wonderfull way should be to let the programmer extend it by himself !

interp elementprefix path <elementprefix> path command args

interp elementprefix {} = {} apply {{args} {foreach e $args {lappend L [expr $e]}; return $L}}

lassign {=}[list 1+1 2+2 3+3 4+4] A B C D
lassign [list {=}1+1 2+2 {=}3+3 4+4] A B C D

wm geometry . [join {=}[list [winfo width]/2 [winfo heigth]/2] x]

interp elementprefix {} mid {} apply {{args} {expr double([join $args +])/[llength $args]}}
set M {mid}$L

Justifications for a global syntax

Code correctness (protection against double substitution): unbraced use of expr is a source of bugs and security holes; for example expr 1 + $a may execute code in $a (try set a {[puts "gotcha"]}). Unbraced use is nontheless common (due to laziness and lack of understanding of the risks). A more terse (read: lazier) syntax that doesn't permit double-substitution will lead to more correct code. To avoid double-substitution we have to introduce new syntax. Even an expr variant that accepts a single argument (rather than concatenating a list of args) could be called as expr 1+$a rather than expr {1+$a}, leading to double-substitution.
Terseness and prettiness: a terse syntax for expr is frequently requested on this wiki, on tcl-core and on c.l.t; this is clearly a popular desire. NEM (on this page) also cites mathematical expressions being common enough to warrant a global syntax, and that such a syntax will make the language more newbie-friendly. A terse syntax will combat the growth of expr into a less-little language.

Concerns

Backwards compatibility: many of the global syntax proposals would reserve syntax that has existing meaning (in Tcl 8.5 & 8.6), potentially breaking existing code. Other constructs use previously-illegal syntax, but are less intuitive. Opinions vary on whether or not to break compatibility, and how widespread the impact of a break would be for a particular syntax.
Intuitive/ugly syntax: there is much debate around how intuitive or ugly each syntax proposal is. For newbie-friendliness the goal should be an intuitive syntax.
Global syntax is unTclish: DKF and Lars H have expressed this view on this page. See Terseness above for a counter-argument.
Global syntax should not restrict expr syntax: Lars H noted on tcl-core that the parsing rules of the global syntax - which would become part of the Dodekalogue, should not impose any limitations on the language of expr, as such limitations could prevent the evolution of the expr little language. I note that the effective limit is that the global syntax should be as expressive as the contents of a braced expression (matching what can be achieved using [expr {...}]). Ideas for extending expr that may influence the language syntax include: omit $'s from varnames in expr, support comments, support list/array/vector expresisons.
Nesting in expr: should the global syntax be recognised by the expr parser as well? Currently one can nest expr by writing [expr { 1 + [expr { 2 + 3 }]] - should a global syntax need to nest in the same way?

Non-solutions

Salt and Sugar "Cheap sugar" proposes interp alias {} = {} expr (credited to Jeff Hobbs). CMcC makes a similar suggestion on tcl-core. RJM and I point out that this merely addresses terseness and doesn't fix double-substitution. The slightly safer proc = {exp} { expr $exp } makes double-substitution less likely but doesn't prevent it (e.g. = 1+$a).
tcl::mathop For short expressions, e.g. a simple addition of two numbers, using the tcl::mathop commands can be every bit as terse as any expr shorthand suggested. Thus instead of [expr {$i+1}], write [+ $i 1].

Syntax proposals

I have collected syntax proposals from this page, other pages on the wiki, and tcl-core discussions:

Proposal	Source	Interpretation (*)	Works within string (**)	Backwards compatible	Parser impact (***)	Comments
$={...}	Lars H (this page)	Substitution	Yes	Yes (maybe)	Minimal
$=(...)	Mistyping of $={...}	Substitution	Yes	Yes (maybe)	Major	Twylite 2012-12-14: Uncertainty over rules for identifying the end of the word; in particular the existing rules for array variables don't allow nested parenthesis so this proposal may break compatibility.
{=}{...} {=}"..." {=}[...]	Twylite on tcl-core, others?	Word	No	Yes	Minimal	Twylite 2012-12-14: The definition must not permit a bareword after the {=}, ensuring that double-substitution must be intentional.
{...}=	CGM (this page)	Word	No	Yes	Minor
{#}(...}	JMN (this page)	Word	No	Yes	Minimal	Twylite 2012-12-14: Discussion on {expand} suggests this syntax as a shorthand for llength
{math}{...}	Sarnold (this page)	Word	No	Yes	Minimal	Twylite 2012-12-14: Proposed more as a distint little language than a global syntax for expr
$[...]	Larry Smith, jcw	Substitution	Yes	No	Minor?	Twylite 2012-12-14: See also discussion in {expand}. Considered unlikely to break existing code.
$(...)	Steve Bennett, others	Substitution	Yes	No	Major	Twylite 2012-12-14: Added to Jim in 2010. Known conflict with the empty named array (used e.g. in Stooop)
$((...))	This page	Substitution	Yes	No	Minor to Major	Twylite 2012-12-14: From earlier on this page: Used in Bash and ksh; not strictly backwards compatible but unlikely to break existing code.
(...)	NEM, others	Word	No	No	Major	Twylite 2012-12-14: Known conflict with unquoted 'regexp (a\|b\|...)'. Uncertainty over rules for parsing/nesting and identifying the end of the word. NEM created a Jacl patch that implements this proposal (2009).
((...))	Reinhard Max on tcl-core	Word	No	No	Minor to Major	Twylite 2012-12-14: Similar to shell scripts; strictly backwards compatible but unlikely to break existing code.
'...'	Martin Lemburg (this page)	Word?	No	No	Major	Twylite 2012-12-14: Earlier on this page Lars H calls this "too radical"
={...}	xbody, Twylite	Word	No	No	Minimal
Numbers as commands	FB on tcl-core	Word	Yes	Yes	None	A very simple proof of concept can be written using unknown
⟦ ... ⟧	FM Larry Smith	Substitution	Yes	Yes (allmost)	Minor ?

(*) Interpretation: 'Word' means the syntax is only recognised as a whole word, so the start of the proposed syntax must match the start of a Tcl word (same rules as the expand operator {*}). 'Substitution' means the syntax can word in any substituted context, including in the middle of a string.

(**) Words within string: A (slightly redundant) indicator of whether you can put an expression (using the proposed shorthand) in the middle of a string. e.g. set a "value=$(1 + $a)" will evaluate '1 + $a', but `set a "value={=}{1 + $a}" will substitute $a into the string.

(***) Parser impact: This is an estimation of how invasive the change is to the parser implementation (and indirectly how difficult it is to specify the change in the Dodekalogue). A minimal change is expected to leverage the existing support for {*} and retarget how the word is processed. A minor change involves slightly more work, possibly elaborating on a corner case. A major change involves a new nested parsing rule, and dealing with how it interacts with substitution and other nested parsing rules.

FB - 2012-12-14 14:31:17

Numbers as commands edit

This proposal needs no change to the Dodekalogue, but instead relies on the existing power of Tcl. Under this proposal, mathematical expressions become regular Tcl commands with the same syntactic and substitution rules as the rest of Tcl, unlike the current expr command.

For this to work, the interpreter must recognize numbers as implicitly defined commands. The first numeric operand is the "virtual command" name, remaining operators and operands are passed as arguments to this "virtual command". So operators and operands are words separated by spaces, and grouping is done with square brackets as subexpressions are regular Tcl commands themselves. Math functions are defined as Tcl commands too. This infix math syntax naturally blends into Tcl, unlike expr variants defining their own sublanguage.

For example:

set x [1 + 2]
# -> 3
set y [2 * [3 + 4]]
# -> 14

set distance [sqrt [[$x * $x] + [$y * $y]]]
# -> 14.317821063276353

An interesting side effect is that numbers evaluate to themselves recursively, e.g:

[[[1]]]
# -> 1

Operators allowing lazy evaluation, such as the ternary operator ?:, shall accept braced scripts as operands:

0 ? {1 + 2} : {3 + 4}
# -> 7

This proposal can be tested under current Tcl versions with a very simple proof-of-concept unknown handler that redirects to the existing expr command (note: doesn't work with lazy evaluation):

rename unknown _original_unknown
proc unknown {cmd args} {
     if {[string is double $cmd]} {
         uplevel 1 [list expr "$cmd $args"]
     } else {
         uplevel 1 [list _original_unknown $cmd] $args
     }
}

# Math functions must also exist as Tcl commands
proc sqrt i {expr {sqrt($i)}}

FM why limit the shorthand to be a sequence of ascii caracters ? Is Tcl not unicode compatible ?

For instance : ⟦ (\u27e6) and ⟧ (\u27e7)

lassign ⟦list(1+1,2+2,3+3,4+4)⟧ A B C D

Justifications for a global syntax

Concerns

Non-solutions

Syntax proposals

Numbers as commands edit

See also edit