Tcl Modules

To learn about the internal commands used to load a tcl module, read http://www.tcl.tk/man/tcl8.5/TclCmd/tm.htm

Note also the encoding requirement from Changes in Tcl/Tk 8.5.2.

What is a Tcl module?

In brief, "modules" is a vastly simplified approach to software packaging from 8.5. A Tcl module is a package in a single file that can be sourced. (DLLs may still be embedded in that). There is no pkgIndex.tcl file, the package name and version are expressed in the module filename itself, so that version 3.14 of package foo must be named

foo-3.14.tm

RS 2007-08-17

What are the benefits of a Tcl module?

simplified deployment of packages (with some necessary formalization of layout)
speed: no filesystem search in subdirectories of $auto_path directories (there are often very many such subdirectories)
speed: no need to read/parse pkgIndex.tcl
…

Purpose of this page: Discuss the purpose, and implementation, of TIP 189

, TIP 190

, and TIP 191

To understand the justification of the Tcl module facility, you need to read those TIPs.

Note this is something totally different from the Tcl Modules package.

JMN 2008-03-15 TIP 189 provides for user specific paths through the use of an environment variable $::env(TCLX.y_TM_PATH)

It seems however that in Apache, (supposedly for 'portability reasons') environment variables may contain only letters, numbers and the underscore character. Apache translates other characters to the underscore.

This means for example that TCL8.5_TM_PATH may appear as TCL8_5_TM_PATH for cgi scripts and so the tcl::tm::list is not adjusted.

I hacked my way around this by adjusting tcl::tm::Defaults (in lib/tcl8.5/tm.tcl )

2008-03-24 Note that Andreas Kupries fixed this ( bug 1914604 ), so versions 8.5.2 & later will accept both forms of the environment variable.

LV 2007 Aug 30

Why were Tcl modules created when starkits already existed? What is the advantage intended for modules over starkits?
What tools are included in the core Tcl for building Tcl modules and extracting the code inside of Tcl modules?
What does a developer have to take into consideration when developing code that would eventually be placed into a tcl module?

According to http://tip.tcl.tk/190

, a simple tcl script only package, consisting of only one file, is already a tcl module (though there are, I believe, some requirements about the filepath and file name for tcl to locate it). According to the same web page, for a simple package consisting only of tcl scripts in the same directory, all one has to do is concatenate the files into a single file (following, of course, the same distribution file naming conventions).

What special debugging and programming techniques have developers found useful when develop code residing in a tcl module?

LV I've seen people ask whether zip files will be supported as tcl modules; I myself am hoping to see starkits as tcl modules.

jmn 2005-09-13 ::tcl::tm::add can have unintuitive (IMO) results. tip189 clearly states that paths added last are 'looked at' first - but it doesn't make it clear that a package of the same version in a later path will 'override' the earlier entry in the 'package ifneeded' database.

This may be of concern if you are trying to create a local cache of packages with a fallback to your main repository on a (slower) network share - or if you are trying to force use of a particular module file during testing (without hacking at versions).

'package forget' won't help here - in contrast to the standard auto_path system where a path inserted into the head of $auto_path WILL allow loading of this package in preference to the one found later in the list (assuming either auto_path wasn't scanned prior to the addition, or you've called 'package forget' after the addition) .

Is this behaviour intended?

I think it'd be really nice to have some way (that works consistently for both module & auto_path systems) to override existing entries in the 'package ifneeded' database - for just those packages in a particular folder.

e.g. a 'package scan <dir>' command might add just the packages found in <dir> - replacing any previous entries in the database.

Otherwise - some better introspection of the 'package ifneeded' database would be nice.. It's currently quite confusing to debug package loading issues so if you could do something like a 'package ifneeded *' and see what it holds - that might help.

Lars H: Do you mean something like this?

 proc package_ifneeded_* {} {
    set res {}
    foreach pkg [package names] {
       set L {}
       foreach ver [package versions $pkg] {
          lappend L $ver [package ifneeded $pkg $ver]
       }
       lappend res $pkg $L
    }
    set res
 }

Example:

 % join [lrange [package_ifneeded_*] 0 5] \n
 rcs
 0.1 {source /Library/Tcl/tcllib1.8/rcs/rcs.tcl}
 logger
 0.6.1 {source /Library/Tcl/tcllib1.8/log/logger.tcl}
 SOAP::Utils
 1.0 {source /Library/Tcl/tclsoap1.6.7/utils.tcl}

jmn Thanks yes - of course you're right.. the package command does provide enough basics to build your own view of the database as you require - I just think something similar to your snippet would be a nice inclusion to the standard distribution. While I should have known better and just built such a thing - I think for new users (or even just anyone a little unsure of how it all works) the package mechanisms have enough tricky subtleties that any extra introspection help that can be provided would be good.

Anyway.. this is an aside to the main issue I'm having: that I can't get the 'package require' system to use a package earlier in the module path if the package versions are the same. The theory may go that if the package versions are the same it shouldn't matter - but there may be administrative reasons to need this to behave properly - such as repository access speed.. or maybe a c-coded version to take higher precedence than a tcl-only version.

MG Perhaps an addition to the way [package require] works would help. For instance, if you could do

  package require -path [list "/path/to/thisfile" "/path/to/thatfile"] myPackage 2.0

to either have only those paths checked, or to have those checked first, followed by all the usual places, it might solve your problem. That would also allow you to [package require] a package which you know resides in a certain place, on its own, without having to add that path into your $auto_path (which is checked for all packages, slowing things down a little, as I understand it).

Another simpler solution to the C vs. Tcl versions is to just add another level to the version number. For instance, call a C version of a package version 2.0.2 and the pure-Tcl (slower) version 2.0.1, so that a [package require myPackage 2.0] would give priority to the C one.

LV So, does this mean that the order of auto_path doesn't control the order that package require searches for a package to load? It does seem that it would be useful if one could say

 package require /path/to/specific/version/of/extension

and only that would be used. That way, someone could test a new version of a package without it going into the live environment with possible negative consequences. Or one could code the above if there appeared to be some sort of problem with a package, to determine for certain whether a particular directory was the one causing a problem.

Lars H, 15 Oct 2005: One way to make sure a particular instance of a package is loaded would be to first explicitly source its pkgIndex.tcl file, like so:

 set dir /path/to/specific/version/of/extension
 source [file join $dir pkgIndex.tcl]
 package require packageName exactVersion

But mostly that is going the wrong way. A major reason package is useful is that it relieves the programmer of having to know exactly where packages are located, so rolling that information back into its API would mess things up again.

escargo 14 Oct 2005 - Comparing the package mechanism to the executable search in Unix and Linux, maybe we need something like a which[1] or whereis[2] command (as subcommands of package).

In a way this is a kind of introspection of the execution environment.

tje Okay, I'm going nuts here...

I made a simple module 'foo-1.0.tm' under directory 'a', like this:

       /_path_/
               a/
                 foo-1.0.tm

I then started my tcl shell (8.5a4, a tclkit) and issued this command:

       ::tcl::tm::add /_path_/a

It shows up at the head of the list returned by ::tcl::tm::list now. So, what's the proper way to source this module without giving the absolute path? I've tried sourcing foo, foo-1, foo-1.0, foo.tm, foo-1.tm, and foo-1.0.tm, all to no avail.

ANSWER:

 package require foo 1

ZB 20100402 Is it proper answer? The manual page states: "a Tcl Module is always imported via 'source module_file'".

RT 12Jan2007 Would be nice having a wiki page that truly does explain modules but it's not this page.

LV If you feel that explanations are missing here, please add questions so that the people who understand tcl modules can provide that information here.

So I went to TIP #189 which does seem to explain pretty well. Then my attention was caught by this RE

  ([[:alpha:]_][:[:alnum:]_]*)-([[:digit:]].*)\.tm

which describes legal module file names. Thing is, it permits a colon in the name. Colons are totally illegal in DOS/Windows file names. I think this is a nasty booby trap for anyone wanting to bring a package from *nix world to Windows. Is a colon really worth giving up portability of module file names? - RS: TIP 189 says that the string :: in a Tcl module name is replaced with the platform's path separator, so module foo::bar 1.0 will on Unix/Linux resolve to the pathname foo/bar-1.0.tm

Not only that, but notice after the - that it specifies one digit followed by any character whatsoever. That's likely to present various unpleasant surprises... - RS: Note the other condition that the version string must be acceptable to package vcompare, which limits them to digits and periods.

[tb] 2008-08-06 - Hi, just came across this page. To me the concept of starkits fulfills any needs for modularization. If I require the packages of a starkit in the kit's main.tcl, then I have the complete module functionality with that kit. I simply source it and have the containing packages loaded, ready for usage. Allthough, having that lot of different semantics for starkits might be confusing? Maybe we should distinguish between:

modkit - A module - packages are required, when the kit is sourced
libkit - A Library - making packages available
appkit - A tclkit application
starpack - An executable

Otherwise it would be nice to have package require <package> from /path/to/kit and remove the need for module kits. Just my thoughts :)

Ro: I think breaking it down into those terms fractures the term starkit too much. Honestly though, I'm with you, starkits fulfill my modularization needs; I don't plan on using Tcl modules. Hell usually I don't even bother and just copy the necessary library into my application, for most of my apps disk space isn't that limited, and I just don't have the energy or time to debug library mismatches across applications. DLL Hell is over for me. That way I have a single app, with all it's necessary libraries, the exact interpreter binary that runs it, all together in one directory. I don't get bitten by interpreter or library bugs, changes or whatever. If it works today, it'll work tomorrow, the app is not in flux.

Lars H: Starkits and Tcl modules are different kinds of concepts — starkits are about packaging, whereas modules are about making the package contents obvious from its file system appearance. Indeed, TIP#190 has an example of how a starkit can be a Tcl module.

It appears that ActiveTcl is including tm.tcl in Tcl 8.4.17 (and likely other versions), providing the modules functionality under 8.4 even though this wasn't added until 8.5. I'd guess that it's included due to Teapot, whose files all seem to conform to the module naming scheme.

ZB 20100402 Well, surely there's something I'm missing, but if you want just to "source single file", you don't want do have pkgIndex.tcl, and you want its name to obey specific rules etc. - so, actually, why not just name the file in the specific manner (can be "foo-3.14.tcl"), and then just "source foo-3.14.tcl"? Below I can read about benefits, that all are there already, when using just ordinary source instead of "package require", so is it about that path manipulating commands only ("::tcl::tm::path")? Just to be able to source the ordinary TCL-file without giving absolute path ("source foo-3.14.tcl" instead of "source [file join $root $subdir foo-3.14.tcl]")?

Is it all about formality, I mean: some rules, that the files, recognized as "modules", should fulfill?

JHJL20110330 If I use Tcl Modules they don't seem to appear in 'package names' list. Is this by design or me ?

AK Effectively 'by design'. package names shows the list of packages the Tcl interpreter has in its in-memory database of packages, i.e. it knows about them. They are not necessarily loaded. When we had only regular packages a unknown package caused a sweep of the whole set of auto_path directories, reading their pkgIndex.tcl files, filling the in-memory database to the brim. Now, with Tcl Modules the package management searches first for a matching module, and this search is much more targeted. This is by design because the 'whole sweep' is what we wish to avoid, to keep startup times low. Another effect of the targeted search is that the in-memory database doesn't get filled as much. We do put in a bit more than just the package searched for (*), but not as much as the whole sweep. (Ad *) Because we do not know the version number of, for example 'math::fourier', we have to still search the whole <BASE>/math/ directory, and so we turn this effort into an opportunity and record all the modules under math/

JHJL Many thanks for the explanation and support, I thought this might be the case. FYI I hit upon this because I wanted to use modules with the pluginmgr list command which uses package names internally (the hope was to use modules so that I didn't have to maintain pkgIndex.tcl files and modules seem to be the way to go. Incidentally pluginmgr will load .tm style modules, just not list them as available).