Tackle

START OF Tackle Programming Language Specification - Version 1 Draft

Peter Newman 25 April 2004

INTRODUCTION

The Programming Language Tackle is an idea that arose from the April 2004 discussion on Tcl Common library.

The idea of a Tcl Standard Library was floated, and generated a lot of interest. But there were so many ideas, so many issues! Lots of people want a Tcl Standard Library. But there were many different ideas as to what that actually was.

During that discussion, I realised that though Tcl is great, there are still lots of outstanding issues that haven't been resolved. For example:-

The lack of a standard distribution system - like Perl's CPAN and RPM.
The Tiny/Minimal Core versus Batteries Included Core issue. We seem to have a compromise solution at the moment, that neither side is entirely happy with.
The stagnation of certain aspects of Tcl - like skinning/theming of the core widgets, for example.
The poor documentation of many of the core commands, widgets and concepts.
Outstanding issues regarding TEA and CVS and how to create packages and enhancements to Tcl. In other words, the architecture and mechanics of how one goes about adding extensions to Tcl.
The continual talk about/work on megawidgets and MegaWidget Librarys.
The continual calls for some object-oriented extension to be included in the core - and the debate about which one (if any).
And there may well be many more. If there is, just let me know.

These issues seem to have been discussed/debated for many years. But progress seems either painfully slow or non-existent.

But what struck me about the debate in Tcl Common Library was that though these issues are always thought of as separate and un-related - they are in fact all connected (with perhaps a few exceptions).

And that the reason there seems to be little progress in resolving them, is because you can't (in most cases,) do this - without changing other aspects of Tcl. And the changes required are in most cases so major, that hey... it's just not happening. Hence the lack of progress.

But it also occured to me that these issues could easily be resolved - to the satisfaction of all parties - no matter what their position on any given matter - if, instead of thinking of and implementing Tcl as:-

A "core" of script level commands - burnt into Tclsh and Wish - with optional packages that may be added to extend the "core" functionality provided,

we re-designed Tcl as:-

A "Standard Library" of script level commands and packages - where EVERYTHING - including the parser, the interpreter, and the current "core" level commands - is/are an optional and stand-alone extension.

This needs clarification, and will get it below. But it's basically the motivation behind Tackle.

BUT WHY A NEW PROGRAMMING LANGUAGE?

Quite obviously you can't just re-design Tcl at the drop of a hat. So I though a better way was to define a new programming language Tackle - which was what the re-designed Tcl would look like.

Then those that are interested can either evolve Tcl into Tackle - or keep the two separate - as they see fit.

THE TACKLE PROGRAMMING LANGUAGE

Tackle is a new programming language. It's a superset of Tcl.

In other words, Tcl is Tackle - with some restrictions (listed below,) applied.

Or put another way. Tackle is what Tcl could/would be - if those restrictions were removed.

WHY TACKLE?

Well...

Tcl -> Tickle
Tickle -> Tackle
And Tackle, like any other programming language, is a tool for tackling problems with.
Why not?

Larry Smith And besides, Tcl has its feather, and Tackle can have a fish hook.

THIS DOCUMENT

This document is the Tackle Programming Language Specification - Version 1 Draft.

Once it's been worked into the situation where it seems worthwhile to do so, we'll rename it to Version 1 Final - and begin work on Version 2 Draft.

TIME FRAME

It's anticipated that it will take at least three to twelve months - from the current start date of April 2004 - before Version 1 Final will be ready.

MAINTAINER

The initiator and current maintainer of this specification is Peter Newman.

COLLARORATION & CONSENSUS

It is intended that this specification be created collaboratively.

In other words, it's NOT Peter Newman's version of what the perfect programming language is. Instead, I'm assuming that this spec. will be a co-operative effort, and that anyone interested can contribute to it.

My only job is to write everything down - so that the resulting, hopefully consensus, opinion is clearly documented.

CONTRIBUTIONS

If you want to contribute to the spec, please append your contribution (or a link to it,) to the very bottom of this Wiki Page (or the relevant sub-page, linked from this Wiki Page) - in date order - in conventional Tcl'ers Wiki style.

MAINTAINER BIASES

Any maintainer has their own biases and opinions. Mine are:-

The belief in a collaborative, consensus approach.
The belief that Tcl should be re-designed where EVERYTHING - including the parser and the "core" scripting language - is an OPTIONAL and STAND-ALONE extension.
The belief that if we do this, then we create virtually the perfect language. Since any individual programer can select exactly the system they want. Discarding or modifying any feature they don't want. Or adding any new features they do.

But collaboration and consensus is the main one. So if the consensus is that my obsessions are de-railing the project, then I'm quite happy to get lost.

VOTING

For a truly consensus approach, some form of voting/polling system is presumably required. So that - where decisions have to be made (eg; "Is the spec version N ready?" or "Should we use Solution X or Solution Y to this problem" - a true community decision can be made (and can be seen to be made).

However, as of the current date (April 2004), no such voting/polling system exists - and I can't see the current maintainer having the time to fix that.

So I've delegated this problem to the [Tackle - Voting/Polling] page, for ideas, discussion, and implementation.

WHAT IS TCL?

1. T H E --- C O R E --- C O N C E P T S...

1.1 The C/C++ Programming Language

Tcl believes that C/C++ is the de-facto standard compiled programming language. And that for truly flexible, powerful, constraint-free applications development, the programmer needs the option to code in C/C++ (irrespective of what other scripted/interpreted language dialect they may also be programming in).
However, rather create its own C/C++ compiler, Tcl takes the position that the programmer should just use the existing tools - like gcc for example.

1.2 The C/C+++ Libraries

One of C/C++'s strengths, is that the core language can be extended with library functions. Tcl believes these should be available to the programmer too. This is both a motivation for, and the benefit of, using existing/external C/C++ compilers.

1.3 A Scripting Language

Tcl believes that, though you can almost anything in C, for many jobs it's overkill. And an interpreted scripting language, like Awk or Perl, or those supplied with the Unix shells, is quicker and easier.
In fact, Tcl believes that in many cases, even the most complicated application may be more easily implemented in a scripting language - with perhaps the occasional dip into C/C++ (for speed reasons, for example).

1.4 Summary - Scripting AND Compiling

At heart then, Tcl believes that programming's easiest in a scripted/interpreted environment. But that the programmer must also be able to code any part of the application - and possibly almost all of it - in C/C++, if they want to.

2. J O H N --- O S T E R H O U T ' S --- I M P L E M E N T A T I O N...

Tcl's not the first language to combine compiled C/C++ and scripting. Those ideas are part of Perl too. But Tcl and Perl both go about it in different ways. Tcl's approach is accredited to John Ousterhout, who defined things thus:-

2.1 A Script Is A Sequence Of Commands

In Tcl, a script is a sequence of commands, where those commands are separated by either new-lines and/or semi-colons.

2.2 A Command Is A Command Name Followed By Zero Or More Parameters

2.3 C/C++ And Scripted Commands

Commands can either be written in compiled C/C++ - or as "procs" written in the Tcl scripting language.

2.4 The (Tcl) Interpreter

With a script comprised of commands as defined above, you need an interpreter to parse that script - splitting it up into commands - and then executing each command in turn, feeding any arguments that may follow the command's name to the command as it does so.
That's what the standard Tcl interpreters - tclsh and wish do. These are generally single file binary executables - supported by some dll/so library files - and some library Tcl scripts.

I won't waffle on any more about Tcl. For two reasons:-

It's assumed that the people reading this spec. are Tcl'ers, and know all this stuff anyway. If you're not (a Tcl'er), you'll just have to read the relevant Tcl documentation.
It's at this point - the design and implementation of the interpreter - that Tcl and Tackle part company.

However, it's important to mention this stuff, because Tackle is a superset of Tcl. And thus the above is the foundation on which Tackle is built.

SO WHAT'S TACKLE?

Tcl and Tackle are the same - up to the point at which we start designing and implementing the interpreter that makes all the above happen.

The Tcl Interpreter

Although John Osterhout's original vision allows programmers to quickly and easily implement new (scripting) languages, Tcl as currently implemented (Tcl 0.0 to Tcl 8.5 inclusive,) doesn't take advantage of this.

Instead, there's just two monolithic Tcl interpreters - tclsh and wish. Where tclsh holds the core Tcl scripting language, on an all or nothing basis. To which wish adds the Tk GUI.

So although you can use Tcl to define new commands - these commands may only be used to EXTEND the core scripting language - not to re-define, extend, or bug-fix the core scripting language itself.

Tackle is built on the premise that this is A BIG MISTAKE!

The Tackle Interpreter

In Tackle, there are TWO (or more) interpreters:-

The Tackle Bootstrap Interpreter --- This is a very simple interpreter that the programmer uses to load up the core scripting language or languages that they wish to use. It will also burn the same to a single file executable, like tclsh or wish, if the programmer so desires.
The Language Interpreter(s) --- Each core scripting language that the programmer has loaded into his/her own, personal (typically application specific,) core, will generally have it's own language specfic parser/interpreter. For example; Tcl, Perl, Awk, Forth, Lisp, C/C++, etc, etc. These are all parsed completely differently. And so each needs it's own specific parser/interpreter.

Separation Of Duties

Another difference between the Tcl Interpreter, and a Tackle Language Interpreter, is that the Tcl Interpreter:-

Parses the source code,
Converts it into byte codes, and then;
Immediately executes those byte-codes.

ANOTHER BIG MISTAKE!

A Tackle Language Interpreter can do things this way if it wants to. But it's generally recommended that a Tackle Language Interpreter be implemented in such a way that the programmer can choose what's done with the parsed output from the script level.

The default, unless otherwise specified, is to immediately execute the parsed output. The alternative is for the parsed about to be supplied to the calling script - via callbacks, and/or data structures in memory (or on disk), etc - so that the calling script can for example:-

Syntax check it.
Strip comments.
Obfuscate it.
Write bytes codes to disk (if the language concerned supports them).

And for some languages, it may even be possible to convert the scripted source into C/C++ or assembler, for compilation, etc.

It's the Tackle Bootstrap Interpreter that makes this happen...

The (Tackle) Bootstrap Interpreter

This is a very simple interpreter - that parses an ASCII text file - and supports the following commands:-

bootWith dll/so_filespec --- bootWith works much like Tcl's load - and appends the commands in the specified dll/so to the (bootstrap) interpreter.
burn executable_filespec --- burn will typically use gcc and/or makefiles - to create an executable file that contains the commands you've bootWithed.
burndata [args] --- Forget about this for now.
<Parse [args] ... Parse> ---

Customised Compiled Multi-Language Interpreter - From A Script

The assumption is that the programmer uses bootWith and burn to create a custom interpreter containing the scripting language or languages of their choice. For example:-

 bootWith /Tackle/Tcl/CommandSets/Tcl84
 burn     /Tackle/MyInterpreters/MyTclsh

would create an executable that was basically the same as tclsh84. And:-

 bootWith /Tackle/Tcl/CommandSets/Tcl84
 bootWith /Tackle/Tcl/CommandSets/Tk84
 burn     /Tackle/MyInterpreters/Wish84

would give you wish84. And:-

 bootWith /Tackle/Tcl/CommandSets/UltraForth
 bootWith /Tackle/Tcl/CommandSets/Tk84
 burn     /Tackle/MyInterpreters/ForthGui

would give you a Forth Interpreter with a Tk GUI. And:-

 bootWith /Tackle/Tcl/CommandSets/Lex
 bootWith /Tackle/Tcl/CommandSets/Yacc
 bootWith /Tackle/Tcl/CommandSets/Perl5
 bootWith /Tackle/Tcl/CommandSets/SQL
 burn     /Tackle/MyInterpreters/

gives you Lex, Yacc, Perl 5 and SQL. And so on...

And note that you get a compiled executable without having to mess with C/C++, gcc or makefiles. A few lines of script is all it takes. This is desirable, because it allows the programmer to select the language or languages appropriate to the application.

Breaking Up/Customising/Enhancing/Improving Tcl

Now creating tclsh and/or wish this way isn't particularly useful (since those programs already exist).

But just as the programmer should be able to add whatever scripting languages he/she prefers, there are some languages where the programmer might want only certain features of the language.

C/C++ is one example. A programmer might select one or the other - depending on whether or not they wanted the object-oriented features that C++ adds to C.

And Tcl is another. If you want to:-

Get rid of namespace and interp - because you never use them, and/or;
Get rid of expr too - so that you can replace it with something more conventional, and/or;
Get rid of all the list functions - because in the application at hand, you'd rather use a more conventional, 1-indexed, array type thing, and/or;
Re-define set and get rid of global - and replace them with my/our type local variables, similar to Perl's my/local (because it's better), and/or;
Throw out upvar and uplevel - because they're little used (and what is all this stuff about stack frames and levels anyway?),
etc, etc,

shouldn't it be possible to do so?

Of course it should. Which means that the Tcl core should really be totally comprised of optional, stand-alone packages - so these can be added/subtracted at will.

END OF Tackle Programming Language Specification - Version 1 Draft

daapp 2004-04-24:I suggest to use following scheme:

the tcl standard library [TSL] should include only one object-oriented extension. This extension can be compiled or pure Tcl, it doesn't really matter from a specification point of view. From a performance and foundation point of view, it probably should be compiled, either initially or at least eventually. This extension should not be written in a way that other object-oriented extensions are made impossible to develop and use.
TSL should include a megawidget framework, based on the object-oriented extension
TSL should include the tools and modules for documentation creation and convertion into other formats
TSL should include the tools and modules to create skeletons for new modules and to work with a network archive of other code
TSL should include the tools and modules for easy compilation (where appropriate) and installation of extentions
[fill in other needs here]

My candidates:

XOTcl for the OO framework [anyone want to explain _why_ the choices here were made?]
Namespace MegaWidgets or Snit like Delegation in XOTcl or create something new using XOTcl (Sarnold : why not Xoins?)
doctools, but with xhtml output included :)
modules framework, which needs to be created from stratch - perhaps based on starkits however as well as TEA2 directory structure
CriTcl

Larry Smith Can Snit be implemented efficiently in XOTcl?

Well, it look like the Standard Tcl Library Specification :).

Peter Newman 25 April 2004: Thanks for your feedback. It's my intention that Tackle will give you EVERYTHING you've asked for above. However, it'll be at least a few more days work before I've explained this...

davidw I'd rather create a mailing list than use wiki's for discussion, but here are some of my thoughts on the matter:

Worse is Better! - let's get something working, and save the more contentious pieces until later.

Every module should have one or more people willing to take responsibility for it.

Every module should attempt to harmonize on a documentation standard.

Every module should have some tests demonstrating its functionality.

Every module (except where obviously not appropriate) should function on MacOS X, Linux/BSD/Unix and Windows.

Every module should have a BSD or compatible license.

Initial candidates: thread, TclX, XML stuff, tcl big number package, udp extension.

Peter Newman 25 April 2004: Thanks for your feedback. I agree with eveything you've said/suggested. However, it'll be at least a few more days work before I've explained how this magic is achieved...

I've never used mailing lists, but in principle (bearing in mind that I currently know nothing about them,) I don't mind adding that. I like the Wiki, because hopefully it documents the whole thing, even for people that join in at some later stage. But if you still feel that mailing lists are worthwhile, can you let me know how it all works - or where I can go to find out. Thanks.

SS: 25Apr2004: David, I fully agree with you, but I've to note that worse is better works well if we are ready to change interface from time to time, in an incremental way, without to worry about 5 years old code that no-one is using will stop to work. Corporate's code can just use old versions of Tcl if they don't want to update the source code. Opensource projects MUST update the source to be compatible with a recent release of Tcl, if they don't do it, they are probably dead projects that no one is using anyway. I see a very big inertia in the Tcl comunity to change interfaces, so for the fear that old code will not work, we get that new code will never be written because we can't improve (fast) enough Tcl. This is a bit polemic but I force a bit the things to make my point clear.

daapp: Using Starkit for distribute modules will be a good idea, but this time standard library must include support for Starkits.

About choice that was made, it was only my vision (not a recommendation); nothing else.

I agree with davidw; we need something that will work, after that we can polish it.

About license: I think because Tcl is under BSD license, every module must be under that license.

About other modules, mentioned by davidw: let's look at this variant - we have small TSL that helps to develop not applications, but other libraries, and make other libraries using TSL: megawidget library, network library, text processing library, etc...

What do you think about this?,

Peter Newman 25 April 2004: See above. Also, I'm at a very general, high-level stage at the moment. It'll probably be at least a few weeks before we start getting down to the lower level specifics. Hang in there.

TV Well, eehr, what is your angle, I mean where do you come from or think about proposing things like these? Maybe make a general (short?) page on yourself, so people can understand why you take this point of view?

The lack of a standard distribution system - like Perl's CPAN and RPM.

a g(zip) file ? tcl is simply text...

The Tiny/Minimal Core versus Batteries Included Core issue. We seem to have a compromise solution at the moment, that neither side is entirely happy with.

Hmm, a small core is desirable for obvious reasons, tcl and tk can be seperated, or do you mean the distribution?

The stagnation of certain aspects of Tcl - like skinning/theming of the core widgets, for example.

People seem to be looking into this, I gather your proposal would add something here? About things like bwidgets as a themable library and such issue, or a new Tk configuration possibility? Have you proposals on that?

The poor documentation of many of the core commands, widgets and concepts.

Fine, for that no new language is needed, or do you propose a tcl-lib built in man command?

Outstanding issues regarding TEA and CVS and how to create packages and enhancements to Tcl. In other words, the architecture and mechanics of how one goes about adding extensions to Tcl.

Seems like a big issue in the face of the various OS-es and make systems.

The continual talk about/work on megawidgets and MegaWidget Librarys.

The continual calls for some object-oriented extension to be included in the core - and the debate about which one (if any).

And what are you saying about this?

Peter Newman 25 April 2004: Thanks for your comments TV. I'm just another Tcl programmer. Prior to that it's been Pascal, Forth, 80x86 Assembler, Awk, Perl, and C - over the last 20 years or so. I been at Tcl only for just under a year. Tcl's great - but it also has some problems (listed above) that seem to have been endlessly discussed (in some cases for many years), and never resolved. During the discussion in Tcl Common Library it suddenly dawned on me how why these issues were occuring - and why they were never getting resolved - and how we could change Tcl so that individual Tcl programmers could fix things - so as to get what amounts to their own personal version of Tcl, exactly as THEY want it. But it's going to take a few weeks to document this clearly. And yes, I'll work through the issues listed above, and explain what I mean more clearly.

Lars H: You might want to create a [Peter Newman] page here on the Wiki with that information. Take a look at the Tcl'ers page.

Lars H: The claim above that "the parser and the interpreter" should be optional stand-alone extensions may look like moonshine, but in Tcl it really is possible! Or rather: nothing prevents you from adding another parser and interpreter (e.g. for Tackle) as extensions to a Tcl shell. (If they turn out to be good the next step would be to try and drop the Tcl parser and interpreters from the executable, just to shed the weight, but that is some time away.) I don't know if this has ever been done (most people probably prefer to just add commands to the existing language), but it is certainly possible. You could just go

 tackleproc mycmd {args} {
     ...
     The body is all Tackle
     ...
 }

and define a command in a language that doesn't have to have anything in common with Tcl -- not even the bytecode executor. Or you could go even further and not even create the mycmd as a command in the Tcl interpreter, but only make it available to the Tackle interpreter. This really is an interesting concept, even though I doubt I will do any work on it.

The most intriguing point is probably: How much can one drop? Would it be necessary for Tackle to support the same basic concept of values as Tcl does? Or at least contain that part of the Tcl C library that implements Tcl_Objs?

ulis, 2004-04-26: Maybe you're true in starting the definition of a superset of Tcl in the hope of resolving the main difficulties. Maybe it's better than waiting for 9.0.

pwq 27 Apr 04

If it takes 12 months to get a draft spec, then people will lose interest, plus how many years would it then take to code it up.
Everything changes, but nothing is different. Unless there is a clear objective for tackle (other than having a sickly cute name) it will end up as just another language. You can't be taken seriously as a programmer if you say you program in Tickle. FW: Er, there are plenty of professional Tcl'ers here who I assume almost all say it "tickle."
The concensus approach means that no-one gets what they want. A language designed by committee will please no-one.
Open source projects generally are low quality (look at the exponential increase in the number of bugs in mozilla, and open office.), inefficient and take a long time to produce results.
Most modern languages have been designed and coded by one person.
The TCT can't even agree on the direction of TCL , how will a disparate group of individuals do better.
TCL+ , TCL-NG, TCL plan 9, Tackle, all this is doing is splitting the development effort much like gcc2 and egcs did.

pikhq, 2004-04-27: I take it that you want to start a flame war. I mean, honestly, saying "Open source projects generally are low quality" on a Wiki that mostly involves an Open Source project...

RJM 22 Jul 04 -- admit pwq's list (and pikhq's corollary). Perhaps Peter should look at Lua prior to continuing the Tackle spec work. I believe that could be an alternative coming close to Peter's wishes.

Category Discussion | Category Language