Updated 2017-10-18 21:54:56 by ABU ▲

ABU 10-aug-2012 - TTXN-1.0.1 available.

Download:

TTXN 1.0 edit

TclTest eXtended Notation edit

SYNOPSIS edit

package require Tcl 8.5

package require TTXN ?1.0?

  • Name: id
  • Description: description
  • Only: keywordList|expression
  • Setup: script
  • Cleanup: script
  • Test: script
  • OutputChannel: expectedValue
  • ErrorChannel: expectedValue
  • Expected: ?expectedCodeList? ?matchMode? expectedValue

DESCRIPTION edit

TTXN is a package for the construction of test suites. TTXN development is the evolution of R.Seeger and P.Maker's ideas for alternative notations of tcltest scripts. These original works are available on the TclTk wiki-site at the following links:

Like the original Pretty Test Langage, TTXN is a wrapper of the standard tcltest package; it does not completely hide the tcltest commands, but provides an alternative notation for writing more readable tests. Being a wrapper of tcltest, TTXN commands can be inserted in a normal tcltest script. Therefore with TTXN you can write a tcltest suite (e.g. *.test) interspersing tcl commands, tcltest commands and TTXN commands. Aim of TTXN is not to replace tcltest commands but just to provide a simpler notation for test-case specs. Simpler test-cases are easy to read and easy to maintain. Just an example to get the flavor of how to use TTXN: First, let's consider a (structurally) complex test-case for tcltest
        tcltest::test demo-1.0.0 { 
          list / lappend 
        } -body { 
                set L {}
                lappend L a
                lappend L b
                lappend L c
        } -cleanup {
                unset L
        } -result [list a b c]

and now the same test-case written with the TTXN notation
        Name: demo-1.0.0
        Descr: list / lappend
        Test: {
                set L {}
                lappend L a
                lappend L b
                lappend L c
        }
        Cleanup: { unset L }
        Expected: [list a b c]

As you can see, whilst tcltest provides a single command (tcltest::test) with many options (-setup, -body, -result, ...), in TTXN the same test-case can be written with a sequence of simpler, more readable commands (Name:, Setup:, Test:, Expected:, ...)

TTXN Commands edit

A test-case in TTXN can be expressed as a sequence of commands. Although the recommended sequence for a test-case is the following:
    Name:
    Description: (optional)
    Only: (optional)
    Setup: (optional)
    Test:
    OutputChannel: (optional)
    ErrorChannel:  (optional)
    Cleanup: (optional)
    Expected:

commands can be specified in any order with just these exceptions:

  • Name: must be the first command of a test-case
  • Expected: must be the last command of a test-case.

Note that all the special TTXN commands start with an upper-case letter, and have a trailing ":" . The trailing colon is part of the command (it's not a token separator), therefore at least one space is required after it.
Name:FirstTest   ; # this is wrong
Name: FirstTest  ; # ok

TTXN supports the following commands:
Name: id
This command sets the test-case identifier.
Description: description
This command associates a description with the test-case (may be abbreviated with Descr:). If description fits on a single line and it does not contain special characters ( square brackets, curly braces, dollar-sign ) it may be written without enclosing curly braces. If description takes more than one line or if it contains the above listed special characters, it must be enclosed by braces.
Only: keywordList|expression
This optional command takes a list of one or more keywords or an expression. Each keywords should be the name of a built-in constraint or a constraint defined by a call to tcltest::testConstraint. If any of the listed constraints is false or does not exist, or if expression evaluates to false, then the test-case is skipped.
Setup: script
This optional command specifies a script to run before the test-case body (see Test:) If evaluation of script raises an error, the test will fail.
Cleanup: script
This optional command specifies a script to run after the test-case body (see Test:) If evaluation of script raises an error, the test will fail.
Test: script
This command indicates the script to run to carry out the test.
OutputChannel: expectedValue
This optional command supplies the expectedValue against which any output sent to stdout or outputChannel during evaluation of the Test: body will be compared. Note that only output printed using ::puts is used for comparison. If OutputChannel: is not specified, output sent to stdout and outputChannel is not processed for comparison. The actual output is compared with expectedValue using the matchMode specified (or implicit) with the Expected: command (see below).
ErrorChannel: expectedValue
Same as OutputChannel:, just for stderr instead of stdout.
Expected: ?expectedCodeList? ?matchMode? expectedValue
This command supplies the expectedValue against which the result of the evaluation of Test: will be compared. The optional argument expectedCodeList is a list whose elements are return codes known to the return command, in both numeric and symbolic form, including extended return codes. This expectedCodeList should be used only when you know that the script specified by Test: will throw an exception ( such as error ), insted of returning a 'normal' expectedValue. Default value is {ok return}. The optional argument matchMode determines how expectedValue is compared with the value returned by the evaluation of Test:. Valid values for matchMode are regexp, glob, exact, and any value registered by a prior call to tcltest::customMatch. The default value is exact. Note that matchMode determines how caught output (see OutputChannel:, ErrorChannel:) is compared, too.

Guided Tour edit

Simplest basic test

Let's start with a small test-suite made of 2 test-cases. Save the following script in a file named "hello.test".
        package require TTXN
        
         # load all the modules for the Software Under Testing
         # ---------------------------------------------------
         #  package require hello ; # fake loading
         #  init_hello            ; # fake init
        
         # begin of test-cases
         # --------------------              
        
        Name: helloworld
        Test: { string toupper "Hello world" }
        Expected: "HELLO WORLD"
        
        
         # Note: this test sometimes may fail...  deep analysis required !         
        Name: weatherForecast
        Test: { 
            #
            # ... forecast in progress ...
            #
           return "Rainy" 
        }
        Expected: "Sunny"
        
        
        # test-suite standard cleanup
        # ---------------------------
        ::tcltest::cleanupTests

then run
        tclsh hello.test

Note: be sure TTXN package is installed under a tcl-library path. Result should be something like this:
==== weatherForecast  FAILED
==== Contents of test case:
    #
    # ... forecast in progress ...
    #
   return "Rainy"
        
---- Result was:
Rainy
---- Result should have been (exact matching):
Sunny
==== weatherForecast FAILED
        
hello.test:  Total   2       Passed  1       Skipped 0       Failed  1

1 test-case passed, 1 failed ! Try to adjust "hello" package ( or better, "hello.test" ) !

Testing multi-value results

If test-case returns (a list of) two values, the Expected: command should specify a list of two values ...
    ...
    Name: test-split
    Test: { 
        set str "alpha:beta"
        return [split $str ":"]
    }
    # Expected:  alpha beta  ; # this is wrong !
    Expected: {alpha beta}
    ...

Testing the empty-list

When a test-case returns nothing, the Expected: value should be written as {} (or "")
    ...
    Name: test-lassign
    Descr: first element of an empty list is {}
    Test: {
        lassign {} x
        return $x 
    }
    Expected: {}
    ...

Changing the way results are compared : exact,glob,regexp,...

By default the value returned by the body of a test-case (Test:) is exactly compared with the expected-result, but you may specify other criteria for comparison. You can use one of the predefined criteria

  • exact, glob, regexp (default value is exact) or any value registered by a prior call to tcltest::customMatch ( see Adding new comparison criteria)

These comparison-criteria (matchMode) may be specified as the first optional parameter of the "Expected:" command:
  Expected:  ?matchMode? expectedValue

All the previosly seen "Expected:" commands such as
  Expected: expectedValue

have an implicit matchMode whose default is "exact" and are therefore equivalent to:
  Expected: exact expectedValue

Example - express expectedValue as a "glob" expression :
    ...
    Name: test-string-reverse
    Test: { string reverse "abcdefgz" }
    Expected: glob "z*a"  ; # note the "glob" match-mode"
    ...

Testing for "error" or other return-codes

Your test-case body can be designed not only for testing normal expectedValues, but for trapping exceptions, too. Let's write a test-case for checking the exception thrown when we try to open a non-existing file: Here is an old (discouraging) trick for such test-case
    Name: test-open-exception
    Test: {
        list [catch {open "not-existing-file.txt" r} f] $f
    }
    Expected: {1 {couldn't open "not-existing-file.txt": no such file or directory}}

and here is a more polite alternative making use of the extended syntax of the "Expected:" command:
    Name: test-open-exception
    Test: {
        set f [open "not-existing-file.txt" r]
    }
    Expected: error {couldn't open "not-existing-file.txt": no such file or directory}

Note that this testcase body is much more readable and that the Expected: command has been extended to support the following full syntax:
    Expected:  ?expectedCodeList? ?matchMode? expectedValue

All the previosly seen "Expected:" commands such as
        Expected: expectedValue

have an implicit ?expectedCodeList? and ?matchMode? and are therefore equivalent to:
        Expected: {ok return} exact expectedValue

As a final improvement, we could rewrite our last test-case, replacing the (implicit) "exact" matching with a simpler-to-test "glob" matching:
    Name: test-open-exception
    Test: {
        set f [open "not-existing-file.txt" r]
    }
    Expected: error glob {couldn't open *}

Testing for channel output

the following TTXN commands
        OutputChannel:  expectedValue
         ErrorChannel:  expectedValue

allow you to specify the expectedValue against which any output sent to stdout or stderr during evaluation of the test-case body script will be compared. Note that (as for tcltest) only output printed using ::puts is used for comparison. If OutputChannel: (or ErrorChannel:) is not specified, output sent to stdout (or stderr) is not processed for comparison.

Adding new comparison criteria edit

Let's suppose we have an instrument returning a decimal value close to 1.0 with a "random" error of +/- 0.0001. How can we compare the expected-value "1.0" with a "random" but very close value ? We should add a new comparison criteria, just by using some "classic" tcltest features
    tcltest::customMatch approx approxCompare
    
    proc approxCompare {expected actual} {
            expr {abs($expected - $actual) < 0.001 } 
    }

With this new matchMethod, we could write the following test-case:
    ...
    Name:  instrument-reset
    Test: { 
        # ... do something and compute a result ...
        return 0.999997    ;# of course this is not a random value. It's just a demo
    }
    Expected: approx 1.0
    ...

KEYWORDS edit

tcltest, testing

dcd - 2012-08-09 21:46:17

There's a bug in your last example, presumably a holdover from PTL:

Result: approx 1.0

s.b.

Expected: approx 1.0

ABU - 4 hours later Thanks, fixed with other small wiki-formatting errors. A new package 1.0.1 will be available tomorrow.

Category Testing