Updated 2016-08-03 10:14:33 by Kroc ▲

Tclbench contains basic tools to run benchmarks, plus the benchmark suites for Tcl and Tk itself.

Disambiguation  edit

bench is a similar but separate module of tcllib

Obtaining  edit

development repository
unofficial starkit (version 1.15.2)

Synopsis  edit

runbench.tcl ?-option value ...? ?file ...?

Description  edit

The runbench.tcl application provides the overall management for the execution of benchmarks. If no specific benchmark files are provided on the command line itself the application will use all directory containing the application itself. If files are provided all files matching the glob pattern *tk* are assumed to be benchmarks for the Tk toolkit.

The following options are recognized by the application:
-help
Causes the application to print a list of the recognized option plus short explanations of their meaning
-delta ratio
Delta range for wiki highlight (default: 0.05)
-iterations count
The value of this option is a positive integer number and declares how often the body of benchmarks should be executed to get accurate timing results. This is a global maximum value which can be overidden by the individual benchmarks.
-minversion version
Declares the minimum version of tcl interpreters the application is allowed to use in the benchmarking.
-maxversion version
Declares the maximum version of tcl interpreters the application is allowed to use in the benchmarking.
-match pattern
Restricts the benchmarking to benchmarks whose description matches the glob pattern specified as the value of this option. If both -match and -rmatch are specified benchmarks have to fulfill both conditions to be executed.
-rmatch regexp
Restricts the benchmarking to benchmarks whose description matches the regexp pattern specified as the value of this option. If both -match and -rmatch are specified benchmarks have to fulfill both conditions to be executed.
-normalize version
Takes a version number as value and causes the application to normalize the timing values to the value of the given version.
-notcl
Forces the application to skip all tcl related benchmarks. Takes no additional value.
-notk
Forces the application to skip all tk related benchmarks. Takes no additional value.
-output style
Declares the style to use when generating the output for the benchmarks. Accepts text, csv, list and wiki. text is the default.
-threads count
Number of threads to use for running benchmarks (where possible).
-paths list
Accepts a list of paths to search for tcl interpreters. If no paths are specified the paths in the environment variable PATH are used.
-throwerrors
Causes errors in benchmark files to be propagated out.
-verbose
Activates the output of interim status info. Takes no additional value

Misc  edit

Kroc 2004-06-23: I compared two recents 8.4.6 Tclkit versions, a standard and an upx one, and an older 8.4.0. The bench were launched on three differents computers, running nothing else.

  • Laptop with Crusoe 600 MHz processor, 192 Mb RAM running linux 2.6.6 :
TCL tests : TK tests : Total
Diff. :
 Tclkit 8.4.0     ->      08 min 48 sec         00 min 41 sec         10 min 12 sec         + 24 %
 Tclkit 8.4.6     ->      07 min 33 sec         00 min 40 sec         08 min 13 sec         +  0 %
 Tclkit 8.4.6 upx ->      09 min 03 sec         02 min 36 sec         11 min 39 sec         + 42 %

  • PC with Celeron 600 MHz processor, 128 Mb RAM running Windows 98 SE :
TCL tests : TK tests : Total
Diff. :
 Tclkit 8.4.0     ->      08 min 01 sec         01 min 35 sec         09 min 36 sec         +  6 %
 Tclkit 8.4.6     ->      07 min 32 sec         01 min 31 sec         09 min 03 sec         +  0 %
 Tclkit 8.4.6 upx ->      10 min 06 sec         05 min 32 sec         15 min 38 sec         + 72 %

  • PC with Celeron 2400 MHz processor, 512 Mb RAM running Windows XP Family :
TCL tests : TK tests : Total
Diff. :
 Tclkit 8.4.0     ->      06 min 16 sec         01 min 03 sec         07 min 19 sec         + 33 %
 Tclkit 8.4.6     ->      04 min 31 sec         01 min 00 sec         05 min 31 sec         +  0 %
 Tclkit 8.4.6 upx ->      05 min 05 sec         02 min 04 sec         07 min 09 sec         + 30 %

First thing: a recent Tclkit version is always fastest than an older one, whatever platform and computer running it. So we could say TCT do their job ;^) jcw explained me upx results are bad because it is slower on startup. Anyway, the smaller the computer is, the slower upx was compared to standard Tclkit.

ferrieux 2010-10-20: Measuring perf effects in TEBC et al, I found it necessary to do some extra work to _denoise_ tclbench output:

  • Run it 10 times, and look at min, med, avg, max for each value. Verify that (med-min)/min stays small, so that there's strong accumulation on the min.
  • Vary spacing between functions in libtcl.so. A 16-byte "void foo(void){}" at the beginning of regcomp.c has a large impact on the recursive variant of STR.match.
  • Use the new autoscale feature, so that no test lasts below 0.1s. There may be undershoot due to clock granularity on the first measurement, so I added an extra step of re-evaluation of iteration count (with margin) in libbench.tcl:
if {$res * $iter < 100000} {
    set iter [expr {int(200000.0 / $res)}]
    set code [catch {uplevel #0 [list time $opts(-body) $iter]} res]
    set res [lindex $res 0]
}

The effect of these 3 tricks is a much greater stability. Remaining noise no longer occurs from one run to another, but rather exposes sensitivity to function offsets, instruction cache associativity, etc. Here again, other "spacers" sprinkled over the library have drastic effects. However, optimizing such things across processors and compilers is a daunting task. The bottom line is that tclbench can be used as a reliable speed comparison tool, but that an observed slowdown or speedup may be a very indirect consequence of code changes.

According to MS, Tcl chatroom, 2015-10-26, wha is not good about Tclbench is the way it invokes test scripts, and also that for the micro-benchmarks the signal is drowned in the overhead noise.

Category PackageCategory Performance