scan

scan, a built-in Tcl command, extracts values from a string according to the string representations indicated in a pattern specification.

Synopsis edit

: scan string format ?varName ...?

Description edit

If string matches format, which consists of literal values, whitespace, and conversion specifiers, each value matched by a conversion specifier is stored in the variable named by the next varname. If at least one varName is given, there must be at least one varName for each matching conversion specifier, and the number of matching conversion specifiers is returned. If no varName is given, a list containing the values that matched a conversion specifier is returned.

See the documentation for more details.

The following example produces a list of numbers found in the data:

scan {planet1 0.330 planet2 4.87} {planet1 %g planet2 %g}; #-> 0.330 4.87

Unicode Character to \u Sequence edit

Simple, but handy when examining Unicode output:

proc u2x {u} {
    scan $u %c t; format "\\u%04.4X" $t
} ;#RS

Stripping of Leading Zeroes edit

One important use is to possibly strip off leading zeroes from decimals:

 scan $possiblyZeroedDecimal %d cleanInteger

Context: Leading zeroes induce octal parsing in the expr command and in if and similar commands that invoke expr. Use of scan to remove leading zeros can avoid this often-reported problem.

Converting a number to an ASCII character edit

One very frequently-asked question is how one converts between display and numeric format for ASCII characters. Scan provides the usual answer:

foreach character {a A b B c} {
    scan $character %c numeric
    puts "ASCII character '$numeric' displays as '$character'."
}

This makes the table

ASCII character '97' displays as 'a'.
ASCII character '65' displays as 'A'.
ASCII character '98' displays as 'b'.
ASCII character '66' displays as 'B'.
ASCII character '99' displays as 'c'.

Unmatched Conversion Specifiers edit

Alastair Davies/PYK: When scan returns a list of matching values, it includes an empty string for each format specifier that wasn't matched. If there is only one format specifier, a list with one blank element is returned, and this is not the same thing as and empty string. For example, scan foo %d returns {}, which not an empty string, but a literal open-brace character followed by a literal close-brace character. The documentation states:

 In the inline case, an empty string is returned when the end of the input string
 is reached before any conversions have been performed.

you might think that scan foo %d would return an empty string. But Don Porter carefully explains this special case in [c.l.t]:

Back to the example:

 % scan foo %d
 {}

We're going to scan the string "foo", trying to parse a decimal integer from it, and when we're done, we're going to return the results as an inline list, since we provided no variables for field value storage.

While scanning, we see "f", and "f" can't be part of a decimal integer. In the non inline case, we would assign no value to the variable associated with this field. In the inline format, an empty string is stored in the corresponding list location to represent that situation.

Now we've exhausted the format spec string. But note that we haven't reached "the end of the input string." We're still sitting at the "f". So the "underflow" case where we run out of input before we run out of format spec does not apply, and we should not expect scan to return the empty string.

To see the underflow case in action, consider these examples:

 % scan a ab
 % scan {} %d
 % scan {   } %d
 %

In all these cases we run out of string to parse before we run out of format spec string to guide our parsing. There are more cases where this happens, but they're difficult to construct. Consult the sources and the test suite for details and more examples.

Part of the reason this does tend to be confusing is that the detection of the "underflow" case and returning the special magic empty value when it is detected is just about 100% worthless in Tcl. It's a feature of sscanf() in C that was apparently slavishly copied over without full consideration of whether it had any value in Tcl. (A common problem in built-in commands and features from Tcl's earliest days. Octal, anyone?)

scan's rules are arcane and weird, but they are what they are.

Matching a Set of Characters edit

To match specified characters use %[chars], to match un-specified characters use %[^chars]:

% scan 12a34 {%d%[abc]%d} x - y
3
% list $x $y
12 34

Notice that the above scan's format string says "some decimal number, followed by either an a, or a b or a c, followed by another decimal number". Then, the last 3 arguments are x, the name of the variable to be assigned the first decimal number, - , the name of the variable to be assigned the letter, and y, the name of the second decimal number. The name - was chosen with the intent to convey "oh, this value isn't going to be used".

Effect on the Internal Representation of a Value edit

AMG: Even a simple numeric scan always results in shimmering.

% set var [expr 123]; list
% tcl::unsupported::representation $var
value is a int with a refcount of 2, object pointer at 0317BDF8,
internal representation 0000007B:0317BD68, no string representation
% scan $var %d; list
% tcl::unsupported::representation $var
value is a int with a refcount of 2, object pointer at 0317BDF8,
internal representation 0000007B:0317BD68, string representation "123"

I'd think the single numeric conversions could be optimized to recognize when the input already has numeric representation, then simply return the input.

Interact and I/O Streams edit

AMG: Tcl's [scan] doesn't integrate terribly well with file I/O. In C, scanf() and fscanf() try to read as many characters as needed for the format, but in Tcl, you need to know in advance how many characters to read. I find this limitation to be very unfortunate, but I don't know a good solution.

Perhaps [scan] could be given a special option to get its data from a channel rather than a string, then the C-level guts of the [scan] and file I/O implementations can work everything out much more efficiently and correctly than can be done at the Tcl level.

Netstrings are one scenario where this would be helpful. You need to read a length prefix (not knowing in advance how many digits form the length), then confirm the next character is colon (for framing), then read the data (length in bytes indicated by the prefix), then confirm the final character is comma (for framing).

In C, you just do scanf("%d", &len) to get the length prefix. But in Tcl, you have to [read] one character at a time until you find a non-digit. That's a lot of looping and script which would be much better handled at the C level.

(Pedantic aside: the scanf() approach isn't technically correct because it allows negative lengths and leading zeroes, but you can check for negatives and can forgive leading zeroes.)

(And a question: with scanf("%d:", &len), how can you tell whether or not the colon was found in the input? I think the answer is to use %n. Trouble with that is inconsistency in whether successful scanf("%d:%n", &len, &pos) returns 1 or 2. So set pos to 0 first then check if it changed. What a mess.)

Let's say you have all the data read in advance, e.g. it came from a UDP datagram, thereby making Tcl [scan] suitable for reading the length prefix. [scan $datagram %d len] works, but how do you know how many digits were consumed? Guess you'd need %n for that too: [scan $datagram %d:%n len pos]. Now the data is [string range $datagram $pos [expr {$pos + $len - 1}], and [string index $datagram [expr {$pos + $len}]] should be a comma. So far, so good.

Buuuuuuuuut how do you now process the next netstring in the sequence? You have to use [string range] to chop off the consumed portion, which involves a lot of copying. I recently had a great deal of trouble when doing similar processing with strings many megabytes in size, wishing to scan a few fields from the front at a time. So if we're going around adding options to [scan], I also suggest adding a -start option to skip leading characters. [regexp] has such an option. Or go the [binary scan] route and offer %n@ to skip to an arbitrary location, though I prefer the option approach.

(By the way, [scan] also has a serious slowdown with large input strings, possibly due to conversion from UTF-8 to UCS-2 or whatever for random access, even when random access is not required. Fix that too.)

Tcl syntax

Arts and crafts of Tcl-Tk programming

Category Command

Category Binary Data

Category String Processing

Category Parsing