fickle

fickle -- the fast-ish lexical analyzer generator

fickle is a Tcl utility that generates Tcl code, much like flex does for C code. From the README:

 This is a scanner generator program much like flex(1) is to C.  If you
 have no desire to author Tcl programs, particularly those that
 manipulate text, fickle is not for you.  A passing knowledge of flex
 or some other lex-like program would be useful as that fickle uses
 nearly identical syntax and commands as flex.  Two good references are
 the flex(1) man page and the O'Reilly book 'lex & yacc' by Levine,
 Mason, and Brown.

fickle differs from other Tcl scanner generators. Whereas yeti dynamically generates the scanner fickle is designed to be used at compile time (and thus in theory faster). tcLex[1] is a C binary in contrast to fickle which is written in pure Tcl (tested under both Tcl 8.3 and 8.4). Like lex is to yacc, fickle is designed to co-exist with my LALR(1) parser generator taccle.

The fickle package comes with several examples derived from lex & yacc[2]. One example I wrote is tsa, a Tcl source code analyzer. It tries to calculate source lines of code (sloc).

Bezoar 2015 Dec13 Links to tar.gz files are dead. However Jason has put his taccle and fickle projects in git hub fickle[3]

LV 2007 June 27 The web site tcl.jtang.org (as well as www.jtang.org) is not, at this time, accessible. I've sent emails to see if I can reconnect with Jason, to see if this software is available from him. However, if the reader of this page happens to have copies to share, it might be nice to mention.

A full list of features is described in the README[4]. Check the section below for errata.

Download fickle from below:

fickle 2.04 at http://tcl.jtang.org/fickle/fickle-2.04.tar.gz. (also from here [5] - direct link [6])
fickle 2.03 at http://tcl.jtang.org/fickle/fickle-2.03.tar.gz.
fickle 2.02 at http://tcl.jtang.org/fickle/fickle-2.02.tar.gz.
fickle 2.01 at http://tcl.jtang.org/fickle/fickle-2.01.tar.gz.
fickle 2.00 at http://tcl.jtang.org/fickle/fickle-2.00.tar.gz.
fickle 1.00 at http://tcl.jtang.org/fickle/fickle-1.00.tar.gz.

fickle is protected by the GNU General Public License. Of course, any code derived from using fickle is protected by whatever copyright you impose.

Invoke fickle before running the program much akin to using flex:

 > tclsh fickle.tcl foo.fcl
 > tclsh foo.tcl

whereas with flex it would be:

 > flex foo.f
 > gcc -o foo lex.yy.c -lfl
 > ./foo

For comparison purposes here is a fickle specification file to match some English verbs:

 # Recognizes various English verbs in sentences.
 # This is based upon example 'ch1-02.l' from "lex & yacc" by John
 # R. Levine, Tony Mason, and Doug Brown (by O'Reilly & Associates, ISBN
 # 1-56592-000-7).  For more information on using lex and yacc, see
 # http://www.oreilly.com/catalog/lex/.
 %{
 #!/usr/bin/tclsh
 %}

 %%
 [\t ]+        # ignore whitespace
 is |
 am |
 are |
 were |
 was |
 will          puts "$yytext: is a verb"
 [a-zA-Z]+     puts "$yytext: is not a verb"
 .|\n          ECHO  ;# normal default anyway
 %%
 yylex

And the equivalent flex version:

 %{
 /*
  * this sample demonstrates (very) simple recognition:
  * a verb/not a verb.  this example derived from 'ch1-02.l'.
  */
 %}

 %%
 [\t ]+        /* ignore whitespace */
 is |
 am |
 are |
 were |
 was |
 will          { printf ("%s: is a verb\n", yytext); }
 [a-zA-Z]+     { printf ("%s: is not a verb\n", yytext); }
 .|\n          { ECHO;  /* normal default anyway */ }
 %%
 main()
 {
     yylex();
 }

ifickle

2005-06-03 jt I received from Detlef Groth a significant patch to fickle 2.04 to allow it to co-exist within an [incr Tcl] program. I have not fully reviewed the changes, but seeing his other works I have no doubt that it creates a proper scanner. When I have some time I will merge the changes into fickle baseline. Meanwhile get it here:

ifickle 2.04 at http://tcl.jtang.org/fickle/ifickle-2.04.tcl

Dr. Groth was even kind enough to supply a sample [incr Tcl] script for ifickle. Here it is:

 %{
 #!/usr/bin/tclsh8.4
 public variable nline 0
 public variable nword 0
 public variable nchar 0
 %}
 %buffersize 1024
 %%
 \n        { incr nline; incr nchar ; }
 [^ \t\n]+ { incr nword; incr nchar $yyleng ;}
 .         { incr nchar;}
 %%
 if {[llength $argv] == 0} {
     puts stderr "usage wc-fickle inputfile"
     exit 0
 }
 if {[catch {open [lindex $argv 0] r} yyin]} {
     puts stderr "Could not open [lindex $argv 0]"
     exit 0
 }

 set sc [iwcfickle \#auto -yyin $yyin]
 $sc yylex
 puts [format "%7d %7d %7d %s" [$sc cget -nline] [$sc cget -nword] [$sc cget -nchar] [lindex $argv 0]]
 close $yyin

Versioning and Errata

2010-07-29. unofficial by MHN: the BEGIN procedure has not worked in my v2.04. The error could be fixed in line 522 of fickle.tcl:

    eval set ::\${prefix}_state_stack \\\{\[lrange \[set ::\${prefix}_state_stack\] 0 end-1\]\\\} ;# <your bugfix note>

2005-03-11. There is an error in v2.04 on line 377 of fickle.tcl. Thanks to Jeff S. for finding it. The correct code is:

    set ::${::p}in \$new_file

version 2.04 is another bug fix:

yyless was missing a parameter within its internal [string match] command
updated README file

version 2.03 is a bug fix:

fixed problems when using both %option debug and %option prefix
fixed an error with patterns that have a vertical bar

version 2.02 has the following:

corrects error with %buffersize option
reformatted code and added some more comments

version 2.01 had the following:

interactive mode (with -I flag or %option interactive)
corrects errors in yy_scan_string and definitions containing backslashes
generated comments are now TclDoc friendly

version 2.00 was a near complete rewrite of the previous one. Its features are:

much faster than previous versions
start states, both inclusive and exclusive (with %option stack)
case sensitive and insensitive matching (-i flag, %option caseful, and %option caseless)
yy_flex_debug to aid debugging (with -d flag or %option debug)
yylineno to keep track of input line numbers (with %option yylineno)
yywrap, yyrestart, yy_scan_string to manipulate input buffers
input, unput, yyless, and YY_INPUT to modify how fickle reads from yyin
change default yy prefix via -P flag

27sep04 jcw - There's a small typo, I've changed a line containing:

    "%buffersize " {

to:

    "%buffersize" {

This showed up when running "make" in the examples/ directory.

jt Thanks for spotting this. Version 2.02 includes this correction.

Return to Jason Tang

Category Application

Category Dev. Tools

Category Parsing