How to Measure Performance

How to Measure Performance

Description edit

Before anything else time it!

The wonderful time command can help you answer questions like "Is this code faster than that code?" Check out the man page. Or check out this example of Counting Elements in a List which includes a basic timing framework.

If you need to record time stamps (e.g., event driven Tk applications), then take a look at clock clicks which gives you a time stamp of the finest granularity available on the platform.

Sometimes you may want to profile an application to find out where all the time is spent. There are several solutions. See Profiling Tcl for some suggestions.

One idea is to build in time routines into an extension or application. For instance, md5pure::time lets anyone time the md5pure extension and compare the results to those of others (see the chart of timings at the md5pure).

Tools edit

PDQ: an analytical solver in which resources are represented by queues

Timing Example edit

Use time to compare to different coding techniques. For this simple example, we will compare list creation using the lappend list command and the append string command. But the technique can apply to many different situations.

Put your test code in a proc so that it gets byte-compiled. Run enough iterations of the test code to make a meaningful timed test. (Or maybe just use a slower computer.)

proc test1 {} {
    for {set i 0} {$i < 1000} {incr i} {
        lappend mylist $i
    }
}

proc test2 {} {
    for {set i 0} {$i < 1000} {incr i} {
        append mylist "$i "
    }
}
% time test1 100
349 microseconds per iteration
% time test2 100
1018 microseconds per iteration

DKF: You should be aware of the fact that UNIX has a much finer time granularity than Windows (certainly 95/[98] and quite probably NT too.)

KBK 2000-12-24: The problem with Windows clock resolution is fixed in the latest alpha of 8.4.

The best way to benchmark a tiny snippet of Tcl (especially where it is platform-independent code you're testing) is to use a slow UNIX box. The slowness of its operation allows for sensitivity and the fine granularity of its timer lets you pick up what differences there are. And then you have to pray that the relative speeds scale as you move to different platforms/architectures and/or faster machines. Which is usually a good assumption if you're not shimmering between numbers and strings (where performance seems to vary a lot according to the vendor's implementation of the C library.)

Unix has millisecond-sized granularity (i.e. around 1000Hz or 1kHz.) Windows has, for reasons relating to the hardware used in the original IBM-PC, 42.7Hz (or thereabouts) granularity. (KBK 2000-12-24: The latest 8.4 alpha uses the 1.193 MHz 8254 timer in place of the Windows 1- or 10-ms time quantum.) Obviously, this makes it harder to perform finer measurements. Good thing most operations in Tcl are actually platform-independent, so the relative differences in execution speed tend to carry across.

KBK: The patch for using the 8254 clock in earlier Tcl releases is at [1] (AMG: 404). Note that multi-CPU Windows systems are still going to be limited to the 1-ms resolution, because of a bug in the Hardware Abstraction Layer that Microsoft ships.

Remember, the actual numerical performance of computers varies widely with operating system and processor speed, and comparing actual figures is very tricky. If you're going to put up benchmarking info, at least do everyone the favour of Finding Out Your Processor and Operating System Configuration...

MAK: While working on optimizations for TkPNG, alternating between using time and gprof for various measurements, I was sometimes seeing my numbers change in odd ways with some going up and some going down for a change that should affect all images in one direction or the other. So I was interested in seeing error margins to prove to myself I wasn't insane. So I whipped up this "errortime" proc. It works similarly to time, but it runs the script through time with a given group size and does so the specified number of times, then calculates a new average and error margin.

proc errortime {script groupsize {iters 2}} {
    set i 0
    set times {} 
    if {$iters < 2} {set iters 2}

    for {set i 0} {$i < $iters} {incr i} {
        set result [uplevel [list time $script $groupsize]]
        lappend times [lindex $result 0]
    }

    set average 0.0
    set s2 0.0

    foreach time $times {
        set average [expr {$average + double($time)/$iters}]
    }

    foreach time $times {
        set s2 [expr {$s2 + (($time-$average)*($time-$average) / ($iters-1))}]
    }

    set sigma [expr {int(sqrt($s2))}]
    set average [expr int($average)]

    return "$average +/- $sigma microseconds per iteration"
}

Example:

proc test1 {} {
   for {set i 0} {$i < 1000} {incr i} {
       lappend mylist $i
   }
}

proc test2 {} {
    for {set i 0} {$ i <1000} {incr i} {
        append mylist "$i "
}

% time test1 10000
855 microseconds per iteration
% time test2 10000
2376 microseconds per iteration
% errortime test1 100 100
843 +/- 42 microseconds per iteration
% errortime test2 100 100
2316 +/- 29 microseconds per iteration

Category Performance

See Also edit

Description edit

Tools edit

Timing Example edit