A Look at the Tcl Test Suite with gcov

gcov is the gcc coverage testing tool. It shows, how often each line of code executes and especially what lines of code are actually executed. It can be used, to see which code paths are not tested by the tcl test suite.

To instrument tcl for usage with gcov is easy. The gcov manual advises to compile without compiler optimization, to get better results. Therefore, I simply configured tcl with --enable-symbols (which disables optimization).

    ./configure --enable-symbols

Now, edit the generated Makefile, and add the special gcc options -fprofile-arcs -ftest-coverage to CFLAGS. Then, just call make test. I've done this with tcl8.4.7. Without modifying the Makefile the test suite runs without failed test for me (on linux). If I run the test suite with the described Makefile modifications I get 11 failed tests out of the files exec.test, io.test and unixInit.test. I haven't taken a deeper look, why this tests fail if tcl was compiled with that flags.

The compilation and the test run creates a bunch of new data files. Now, you're able to run gcov. You should run gcov in the same directory you've invoked the compiler for. 'gcov' is called for every source file in this way:

    gcov ../generic/tclBasic.c

For convenience, I did

    for file in ../generic/*.c; do
    gcov $file >> gcov.results 2>&1
    done

After that, gcov.results look like:

    regc_color.bbg:cannot open graph file
    regc_cvec.bbg:cannot open graph file
    regc_lex.bbg:cannot open graph file
    regc_locale.bbg:cannot open graph file
    regc_nfa.bbg:cannot open graph file
    File `../generic/regcomp.c'
    Lines executed:80.22% of 1097
    ../generic/regcomp.c:creating `regcomp.c.gcov'

    File `../generic/regc_lex.c'
    Lines executed:96.75% of 554
    ../generic/regc_lex.c:creating `regc_lex.c.gcov'

    File `../generic/regc_color.c'
    Lines executed:86.34% of 344
    ../generic/regc_color.c:creating `regc_color.c.gcov'

    File `../generic/regc_nfa.c'
    Lines executed:92.06% of 642
    ../generic/regc_nfa.c:creating `regc_nfa.c.gcov'

    File `../generic/regc_cvec.c'
    Lines executed:58.90% of 73
    ../generic/regc_cvec.c:creating `regc_cvec.c.gcov'

    File `../generic/regc_locale.c'
    Lines executed:65.13% of 195
    ../generic/regc_locale.c:creating `regc_locale.c.gcov'

    rege_dfa.bbg:cannot open graph file
    File `../generic/regerror.c'
    Lines executed:64.86% of 37
    ../generic/regerror.c:creating `regerror.c.gcov'

    File `../generic/regexec.c'
    Lines executed:76.47% of 476
    ../generic/regexec.c:creating `regexec.c.gcov'

    File `../generic/rege_dfa.c'
    Lines executed:89.59% of 365
    ../generic/rege_dfa.c:creating `rege_dfa.c.gcov'

    File `../generic/regfree.c'
    Lines executed:75.00% of 4
    ../generic/regfree.c:creating `regfree.c.gcov'

    regfronts.bbg:cannot open graph file
    File `../generic/tclAlloc.c'
    Lines executed:100.00% of 6
    ../generic/tclAlloc.c:creating `tclAlloc.c.gcov'

    File `../generic/tclAsync.c'
    Lines executed:96.72% of 61
    ../generic/tclAsync.c:creating `tclAsync.c.gcov'

    File `../generic/tclBasic.c'
    Lines executed:83.75% of 1495
    ../generic/tclBasic.c:creating `tclBasic.c.gcov'

    File `../generic/tclBinary.c'
    Lines executed:96.61% of 678
    ../generic/tclBinary.c:creating `tclBinary.c.gcov'

    File `../generic/tclCkalloc.c'
    Lines executed:39.62% of 53
    ../generic/tclCkalloc.c:creating `tclCkalloc.c.gcov'

    File `../generic/tclClock.c'
    Lines executed:92.06% of 126
    ../generic/tclClock.c:creating `tclClock.c.gcov'

etc.

The percent numbers show, how much lines of code with actual operations (that is: not counted pre-processor directives like defines, empty lines etc.) were actually called in the test run. A look at the generated *.gcov files provides much more detailed information. For example, take a look at this snippet out of tclCompile.c.gcov:

        -: 1850: *----------------------------------------------------------------------
        -: 1851: *
        -: 1852: * TclInitCompiledLocals --
        -: 1853: *
        -: 1854: *        This routine is invoked in order to initialize the compiled
        -: 1855: *        locals table for a new call frame.
        -: 1856: *
        -: 1857: * Results:
        -: 1858: *        None.
        -: 1859: *
        -: 1860: * Side effects:
        -: 1861: *        May invoke various name resolvers in order to determine which
        -: 1862: *        variables are being referenced at runtime.
        -: 1863: *
        -: 1864: *----------------------------------------------------------------------
        -: 1865: */
        -: 1866:
        -: 1867:void
        -: 1868:TclInitCompiledLocals(interp, framePtr, nsPtr)

-: 1869: Tcl_Interp *interp; /* Current interpreter. */
-: 1870: CallFrame *framePtr; /* Call frame to initialize. */
-: 1871: Namespace *nsPtr; /* Pointer to current namespace. */

  1025735: 1872:{
  1025735: 1873:    register CompiledLocal *localPtr;
  1025735: 1874:    Interp *iPtr = (Interp*) interp;
  1025735: 1875:    Tcl_ResolvedVarInfo *vinfo, *resVarInfo;
  1025735: 1876:    Var *varPtr = framePtr->compiledLocals;
  1025735: 1877:    Var *resolvedVarPtr;
  1025735: 1878:    ResolverScheme *resPtr;
  1025735: 1879:    int result;
        -: 1880:

-: 1881: /*
-: 1882: * Initialize the array of local variables stored in the call frame.
-: 1883: * Some variables may have special resolution rules. In that case,
-: 1884: * we call their "resolver" procs to get our hands on the variable,
-: 1885: * and we make the compiled local a link to the real variable.
-: 1886: */

        -: 1887:
  5676824: 1888:    for (localPtr = framePtr->procPtr->firstLocalPtr;

-: 1889: localPtr != NULL;
-: 1890: localPtr = localPtr->nextPtr) {

        -: 1891:

-: 1892: /*
-: 1893: * Check to see if this local is affected by namespace or
-: 1894: * interp resolvers. The resolver to use is cached for the
-: 1895: * next invocation of the procedure.
-: 1896: */

        -: 1897:
  4651089: 1898:        if (!(localPtr->flags & (VAR_ARGUMENT|VAR_TEMPORARY|VAR_RESOLVED))

-: 1899: && (nsPtr->compiledVarResProc || iPtr->resolverPtr)) {
#####: 1900: resPtr = iPtr->resolverPtr;

        -: 1901:

#####: 1902: if (nsPtr->compiledVarResProc) {
#####: 1903: result = (*nsPtr->compiledVarResProc)(nsPtr->interp,
-: 1904: localPtr->name, localPtr->nameLength,
-: 1905: (Tcl_Namespace *) nsPtr, &vinfo);
-: 1906: } else {
#####: 1907: result = TCL_CONTINUE;
-: 1908: }

        -: 1909:

#####: 1910: while ((result == TCL_CONTINUE) && resPtr) {
#####: 1911: if (resPtr->compiledVarResProc) {
#####: 1912: result = (*resPtr->compiledVarResProc)(nsPtr->interp,
-: 1913: localPtr->name, localPtr->nameLength,
-: 1914: (Tcl_Namespace *) nsPtr, &vinfo);
-: 1915: }
#####: 1916: resPtr = resPtr->nextPtr;
-: 1917: }
#####: 1918: if (result == TCL_OK) {
#####: 1919: localPtr->resolveInfo = vinfo;
#####: 1920: localPtr->flags |= VAR_RESOLVED;
-: 1921: }
-: 1922: }

        -: 1923:

-: 1924: /*
-: 1925: * Now invoke the resolvers to determine the exact variables that
-: 1926: * should be used.
-: 1927: */

        -: 1928:
  4651089: 1929:        resVarInfo = localPtr->resolveInfo;
  4651089: 1930:        resolvedVarPtr = NULL;
        -: 1931:
  4651089: 1932:        if (resVarInfo && resVarInfo->fetchProc) {

#####: 1933: resolvedVarPtr = (Var*) (*resVarInfo->fetchProc)(interp,
-: 1934: resVarInfo);
-: 1935: }

        -: 1936:
  4651089: 1937:        if (resolvedVarPtr) {

#####: 1938: varPtr->name = localPtr->name; /* will be just '\0' if temp var */
#####: 1939: varPtr->nsPtr = NULL;
#####: 1940: varPtr->hPtr = NULL;
#####: 1941: varPtr->refCount = 0;
#####: 1942: varPtr->tracePtr = NULL;
#####: 1943: varPtr->searchPtr = NULL;
#####: 1944: varPtr->flags = 0;
#####: 1945: TclSetVarLink(varPtr);
#####: 1946: varPtr->value.linkPtr = resolvedVarPtr;
#####: 1947: resolvedVarPtr->refCount++;
-: 1948: } else {

  4651089: 1949:            varPtr->value.objPtr = NULL;
  4651089: 1950:            varPtr->name = localPtr->name; /* will be just '\0' if temp var */
  4651089: 1951:            varPtr->nsPtr = NULL;
  4651089: 1952:            varPtr->hPtr = NULL;
  4651089: 1953:            varPtr->refCount = 0;
  4651089: 1954:            varPtr->tracePtr = NULL;
  4651089: 1955:            varPtr->searchPtr = NULL;
  4651089: 1956:            varPtr->flags = localPtr->flags;

-: 1957: }

  4651089: 1958:        varPtr++;

-: 1959: }

        -: 1960:}

The numbers before the line numbers show you, how often that line of code was called. This information may be helpful for optimization efforts. The lines with '####' are especially interesting: this lines of code are not called at all. This may help to find additional tests (or even to detect dead code).

See the 'gcov' man page for a few more output options.

Diving thru the results I got the impression, that the tcl test suite is already pretty good. Of course, some code paths are only called, if tcl was compiled with according flags (e.g.: tcl compiled with block allocator or not). Some debugging facilities seems not to be tested at all, but that may be intentionally. You've to search for relevant code paths, which are not called. One the other hand, gcov may be helpful, to spot that few code areas, which are currently not coveraged by the test suite. de

jmn 2004-08-04 It would be great to have something like this to use in conjunction with tcltest on Tcl scripts. Does such a thing exist? - (update: Coverage Analysis answers my question)

Category Debugging