Passing arrays of data

Arjen Markus (24 september 2002) I wish to create a test program for a library of C routines that I am developing. These C routines are meant to examine the contents of a file and to read particular data from this file. So, the test program I want to write, in Tcl, must manage these routines and test whether the information returned is correct (in my case: the C routines are new and more flexible versions of existing routines).

The C routines, however, fill arrays of strings, integers and floats and I need a suitable form to represent these arrays on the Tcl side. As I prefer to put as much code on the Tcl side as possible, I came up with the following solution (the basic idea was proposed in the Tcl chatroom): Translate Tcl lists into C arrays and vice versa.

The easiest method that I could imagine was:

Create a Tcl byte array from the list
This byte array can be filled with the binary format command in such a way that my C routines would "understand" the contents
The C routines can then fill these arrays (Tcl is responsable for allocating and freeing the memory).
When the C routines have done their job, the byte array is converted into a list via the binary scan command.

There are a few caveats:

You can not use Tcl_GetString() to get a pointer to the byte array, as this returns a UTF-8 string and not the raw sequence of bytes.
You must be careful to use little-endian or big-endian depending on the platform, when filling the byte array with integers.
Strings must be translated into (effectively) a two-dimensional array of characters, each string terminated with a NUL byte and of the same length. (Actually the "a" and the "A" format code help to pad with NUL bytes or with spaces - the latter being important when interfacing with Fortran routines!)
You should be careful with changing the byte arrays: it is possible to change the value of a variable holding the byte array without use of upvar. So, on the Tcl side you can not see if a variable is supposed to change or not.
To do the hard part, an experimental version of Critcl is used that recognises argument types int*, float*, double* for numerical arrays and rawchar* for arrays of character strings (like char string[10][20], not char* string[10]).

Most of these points can be solved easily as shown in the code below

The C code, as generated via Critcl, includes the following statements:

   _data = (int*) Tcl_GetByteArrayFromObj(ov[1], NULL);
   Tcl_InvalidateStringRep(ov[1]) ;

In essence: the "data" argument is a pointer to a bytearray. In C we can use it as a pointer to an array of integers.

 #
 # Use critcl to do the hard part
 #
 package require critcl
 #
 # Define the C routine to interface with
 #
 ::critcl::cproc getfibonaci {
    int      maxsize
    int*     data
 } ok {
    int     error  ;
    int     i      ;

    error  = TCL_OK ;

    data[0] = 1 ;
    data[1] = 1 ;
    for ( i = 2 ; i < maxsize ; i ++ ) {
       data[i] = data[i-2] +data[i-1] ;
    }

    return error ;
 }

 #
 # Tcl interface to the routine - for clean operation
 #
 proc getSeries { maxdata data } {
    upvar $data _data

    #
    # Create a list with the correct number of integer elements
    #
    for { set i 0 } { $i < $maxdata } { incr i } {
       lappend _data 0
    }

    #
    # Convert the list to a byte array
    #
    set c_data [intsToByteArray $_data]

    #
    # Call the C routine - that will fill the byte array
    #
    set error  [getfibonaci $maxdata $c_data]

    #
    # Convert the byte array into an ordinary list
    #
    set _data [byteArrayToInts $c_data]
 }

The above script requires a number of support procedures, here they are:

 #
 # Generic routine to convert a list into a bytearray
 #
 proc listToByteArray { valuetype list {elemsize 0} } {
    if { $valuetype == "i" || $valuetype == "I" } {
       if { $::tcl_platform(byteOrder) == "littleEndian" } {
          set valuetype "i"
       } else {
          set valuetype "I"
       }
    }

    switch -- $valuetype {
    f - d - i - I {
       set result [binary format ${valuetype}* $list]
    }
    s {
       set result {}
       foreach elem $list {
          append result [binary format a$elemsize $elem]
       }
    }
    default {
       error "Unknown value type: $valuetype"
    }
    }

    return $result
 }

 interp alias {} stringsToByteArray {} listToByteArray s
 interp alias {} intsToByteArray    {} listToByteArray i
 interp alias {} floatsToByteArray  {} listToByteArray f
 interp alias {} doublesToByteArray {} listToByteArray d

 #
 # Generic routine to convert a bytearray into a list
 #
 proc byteArrayToList { valuetype bytearray {elemsize 0} } {
    if { $valuetype == "i" || $valuetype == "I" } {
       if { $::tcl_platform(byteOrder) == "littleEndian" } {
          set valuetype "i"
       } else {
          set valuetype "I"
       }
    }

    switch -- $valuetype {
    f - d - i - I {
       binary scan $bytearray ${valuetype}* result
    }
    s {
       set result  {}
       set length  [string length $bytearray]
       set noelems [expr {$length/$elemsize}]
       for { set i 0 } { $i < $noelems } { incr i } {
          set elem    [string range $bytearray \
                         [expr {$i*$elemsize}] [expr {($i+1)*$elemsize-1}]]
          set posnull [string first "\000" $elem]
          if { $posnull != -1 } {
             set elem [string range $elem 0 [expr {$posnull-1}]]
          }
          lappend result $elem
       }
    }
    default {
       error "Unknown value type: $valuetype"
    }
    }
    return $result
 }

 interp alias {} byteArrayToStrings {} byteArrayToList s
 interp alias {} byteArrayToInts    {} byteArrayToList i
 interp alias {} byteArrayToFloats  {} byteArrayToList f
 interp alias {} byteArrayToDoubles {} byteArrayToList d

 #
 # Test the routine
 #
 getSeries 10 data
 puts $data

While I haven't fully understood the intention of Arjen's code above I found the Tcl_NewByteArrayObj() http://www.tcl.tk/man/tcl8.4/TclLib/ByteArrObj.htm

functions and related helpful when handling binary data in TCL objects in C/C++. ralfixx@gmx.de

Arts and Crafts of Tcl-Tk Programming