dict extensions

Dict Extensions edit

Napier - 02/22/2016

Tcl provides the powerful namespace ensemble command allowing one to "extend" the functionality of built-in commands such as dict. ensemble extend is a page where you can see other examples of this being used. I really like ES6 Javascript's capabilities to work with objects such as "const { key1, key2 } = myObject", so I decided to give myself similar functionality with a "dict pull" command.

These commands have generally helped with building and passing around data in many ways. I am providing this as a dedicated page instead of on the ensemble extend page where they used to be because they likely require some instruction to understand what they are doing and how they can be used. Regardless I believe you will find benefits in using them so feel free if you like!

In tcllib 1.17 there is also the dicttool package, which provides some extra functionality and has some overlap. I have discussed merging the two feature sets with the creator in a future release.

First you will need the extend procedure provided on the ensemble extend wiki page. This is included in the script given at the bottom of this page.

Summary of Commands

: dict isDict dictionaryValue

checks to see if the given value is a Tcl Dictionary.

: dict get? dictionaryValue ?key...?

attempts to get the value found at ...key and responds with the value or an empty string. No error is raised should the value not exist.

: dict modify dictionaryVariable ?args...?

modify or create each key/value pair on the given variable name. Any current keys are left alone (merged).

: dict push dictionaryVariable ?variable name(s)...?

merges the variables given in ?variable name(s)...? into the dictionary with the variables name as the key and its value as the value.

: dict pull dictionaryVariable ?key name(s)...?

grab each key provided in ?key name(s)...? and assign their values to a variable of the key's name in the local scope.

: dict pullFrom dictionaryVariable ?key name(s)...?

similar to the dict pull command except allows for pulling nested values (see below).

: dict destruct dictionaryVariable ?key name(s)...?

same as the dict pull and dict pullFrom commands except it will also remove the keys that it pulls from the original dictionary.

: dict types dictionaryVariable
: dict sort <k* / v*> dictionaryVariable ?key name(s)...?

Tutorial and Examples edit

Ok so there is quite a lot of features packed into these commands that can be used to simplify the handling of data. There may be better ways to handle various parts of this API but it has worked great for our needs at Dash. We deal with a lot of API's that, in a perfect world, would tell us exactly how they will behave and/or respond to requests. Our scripts often need to interpret a lot of "what if's" such as a key in a JSON Response that was supposed to be guaranteed suddenly not being there. We also have to pass a lot of data around and format it, so these commands have simplified the process extensively.

Let's start with a simple "tempDict", which we will operate upon throughout. We are using a procedure in the examples because part of the examples will need to show how you can use these commands to return values in your procedures.

proc myProc {} {
     set tempDict [dict create Foo one Bar two Baz three]
}

Now that we have our initial tempDict we can start to use the dict extension tools on it.

In the example below, we use the dict get? command to grab "Grill" from the tempDict. Since it doesn't exist, rather than raising an error, the Grill variable simply will become an empty string.

proc myProc {} {
     set tempDict [dict create Foo one Bar two Baz three]
     set Grill [dict get? $tempDict Grill]
     puts $Grill;     # ""
}

MSH 22/08/2016 I have a similar function and find it useful to have an optional default value argument this allows using the return value without an extra isempty test.

proc myProc {} {
     set tempDict [dict create Foo one Bar two Baz three]
     set Flag [dict get? $tempDict isOn false]
     if {$isOn} {
          puts "IS ON"
     }
}

If we wanted to modify the contents of a dictionary, we can use the dict modify command. This is helpful when you want to modify one or more keys on a dictionary but don't want to affect the others. If we were to use 'dict set' it would either overwrite the entire key if we use it with 'dict create' or it would think the keys are nested and provide undesired results.

proc myProc {} {
     set tempDict [dict create Foo one Bar two Baz three]
     dict modify tempDict Foo oneTwo Grill four
     puts $tempDict     ;     # Foo oneTwo Bar two Baz three Grill four
}

If we want to modify a key that is nested within the dict, we must instead provide the first argument as a list (this is generally accepted with any command provided with the exception of 'dict pull', which requires the use of 'dict pullFrom' to get the same functionality; reasons given when we get to that):

proc myProc {} {
     set tempDict [dict create Foo one Bar two Baz three]
     dict modify [list tempDict Grill] Foo oneTwo Grill four
     puts $tempDict     ;     # Foo one Bar two Baz three Grill {Foo oneTwo Grill four}
}

Next we will provide convenience features for pushing and pulling data to/from dicts. Starting with 'dict push', this command will take each variable given as its arguments and "push" them into the dictionary, which makes the variable's name a key with its value the value. We have a few options here as well:

Note: In the example below we also show in the third part of the procedure that we can take a local variable and push it into the tempDict with a different key.

Note: We also show that dict push returns a dictionary with the pushed values, allowing us to return a dictionary of our local variables easily. Since it's a return value, the "name" of the variable we will use doesn't matter (we generally use vars)

proc myProc {} {
     set Foo one; set Bar two; set Baz three
     dict push tempDict Foo Bar Baz
     puts $tempDict     ;     # Foo one Bar two Baz three

     dict push [list tempDict Grill] Foo Bar Baz
     puts $tempDict     ;     # Foo one Bar two Baz three Grill {Foo one Bar two Baz three}

     dict push tempDict {Foo _Foo} {Bar _Bar} {Baz _Baz}
     puts $tempDict     ;    #  Foo one Bar two Baz three Grill {Foo one Bar two Baz three} _Foo one _Bar two _Baz three

     return [dict push vars Foo Baz]
}

set myDict [myProc]
puts $myDict     ;     # Foo one Baz three

When we want to grab values from a dictionary, we can use one of the 'dict pull', 'dict pullFrom', 'dict destruct' commands. dict pull and dict pullFrom are similar commands and both are provided to try to stay somewhat consistent with the standard way the tcl handles dictionaries. However, if we want to operate on a nested dictionary, we will need to use the pullFrom command. When using the dict pull command, you may either send the dictionary value in directly (similar to dict get $tempDict foo) or use the name of the variable instead (dict pull tempDict foo). One could change pullFrom to pull if you always wanted to use the local variable name. This is demonstrated below:

proc myProc {} {
     # We will create a nested dictionary this time to provide an example of operating against a nested dictionary.
     set tempDict [dict create Foo one Bar two Baz three Grill [dict create Yay value]]
     
     # Pull keys from tempDict and assign them to variables with the same name as the key.
     dict pull tempDict Foo Bar Grill     ;     # We could also have done dict pull $tempDict Foo Bar Grill
     puts $Foo            ;     #     one
     puts $Bar             ;     #     two
     puts $Grill            ;     #     Yay value
 
     # Grabbing keys from our dictionary and assigning them to a different variable name in our local scope
     dict pullFrom tempDict {Foo _Foo} {Bar myVar}
     puts $_Foo      ;     #     one
     puts $myVar       ;     #     two

     # Operating on a nested dictionary, capturing a key and assigning to a new variable name, and grabbing a non existent key in the nested area.
     dict pullFrom {tempDict Grill} {Yay nestedVar} RandomKey
     puts $nestedVar      ;     #     value
     puts $RandomKey       ;     # ""

     # dict pull and dict pullFrom return a dictionary with the pulled values:
     return [dict pullFrom {tempDict Grill} {Yay finalValue} RandomKey]
}

set myDict [myProc]
puts $myDict     ;     # finalValue value RandomKey {}

'dict destruct' is nearly identical to 'dict pullFrom' except it will also remove any values it grabs from the dictionary. We do not remove the nested keys should they become empty due to a destruct call. Destruct does not return any value.

proc myProc {} {
     # We will create a nested dictionary this time to provide an example of operating against a nested dictionary.
     set tempDict [dict create Foo one Bar two Baz three Grill [dict create Yay value]]
     
     dict destruct {tempDict Grill} {Yay _yay}
     puts $_yay      ;     #     value
     puts $tempDict      ;     #     Foo one Bar two Baz three Grill {}

     dict destruct tempDict Foo
     puts $Foo        ;      # one
     puts $tempDict     ;     # Bar two Baz three Grill {}
}

The Script edit

proc extend {ens script} {
    uplevel 1 [string map [list %ens [list $ens]] {
        namespace ensemble configure %ens -unknown [list ::apply [list {ens cmd args} {
            ::if {$cmd in [::namespace eval ::${ens} {::info commands}]} {
                ::set map [::namespace ensemble configure $ens -map]
                ::dict set map $cmd ::${ens}::$cmd
                ::namespace ensemble configure $ens -map $map
            }
            ::return {} ;# back to namespace ensemble dispatch
                  ;# which will error appropriately if the cmd doesn't exist
        } [namespace current]]]
    }]\;[list namespace eval $ens $script]
}

extend dict {

  proc isDict {var} { 
    if { [::catch {::dict size ${var}}] } {::return 0} else {::return 1} 
  }
  
  proc get? {tempDict key args} {
    if {[::dict exists $tempDict $key {*}$args]} {
      ::return [::dict get $tempDict $key {*}$args]
    }
  }
  
  proc pull {var args} {
    ::upvar 1 $var check
    if { [::info exists check] } {
      ::set d $check
    } else { ::set d $var }
    ::foreach v $args {
      ::set path [::lassign $v variable name default]
      ::if { $name eq {} } { ::set name $variable }
      ::upvar 1 $name value
      ::if { [::dict exists $d {*}$path $variable] } {
        ::set value [::dict get $d {*}$path $variable]
      } else { ::set value $default }
      ::dict set rd $name $value
    }
    ::return $rd
  }
  
  proc pullFrom {var args} {
    ::set mpath [::lassign $var var]
    ::upvar 1 $var check
    ::if { [::info exists check] } { 
      ::set d $check
    } else { ::set d $var }
    ::foreach v $args {
      ::set path [::lassign $v variable name default]
      ::if { $name eq {} } { ::set name $variable }
      ::upvar 1 $name value
      ::if { [::dict exists $d {*}$mpath $variable {*}$path] } {
        ::set value [::dict get $d {*}$mpath $variable {*}$path]
      } else { ::set value $default }
      ::dict set rd $name $value
    }
    ::return $rd
  }
  
  proc modify {var args} {
    ::upvar 1 $var d
    ::if { ! [info exists d] } { ::set d {} }
    ::if { [::llength $args] == 1 } { ::set args [::lindex $args 0] }
    ::dict for { k v } $args { ::dict set d $k $v }
    ::return $d
  }
  
  proc push {var args} {
    ::if {$var ne "->"} { ::upvar 1 $var d }
    ::if { ! [::info exists d] } { ::set d {} }
    ::foreach arg $args {
      ::set default [::lassign $arg variable name]
      ::upvar 1 $variable value
      ::if { [::info exists value] } {
        ::if { $name eq {} } { ::set name $variable }
        ::if { $value ne {} } {
          ::dict set d $name $value
        } else { ::dict set d $name $default }
      } else { ::throw error "$variable doesn't exist when trying to push $name into dict $var" }
    }
    ::return $d
  }
  
  proc pushIf {var args} {
    ::if {$var ne "->"} { ::upvar 1 $var d }
    ::if { ! [::info exists d] } { ::set d {} }
    ::foreach arg $args {
      ::set default [::lassign $arg variable name]
      ::upvar 1 $variable value
      ::if { ! [::info exists value] } { ::throw error "$variable doesn't exist when trying to pushIf $name into dict $var" }
      ::if { $name eq {} } { ::set name $variable }
      ::if { $value ne {} } {
        ::dict set d $name $value
      } elseif { $default ne {} } {
        ::dict set d $name $default
      }
    }
    ::return $d
  }
  
  proc pushTo {var args} {
    ::set mpath [::lassign $var var]
    ::if {$var ne "->"} { ::upvar 1 $var d }
    ::if { ! [::info exists d] } { ::set d {} }
    ::foreach arg $args {
      ::set path [::lassign $arg variable name]
      ::upvar 1 $variable value
      ::if { ! [::info exists value] } { ::throw error "$variable doesn't exist when trying to pushTo $name into dict $var at path $path" }
      ::if { $name eq {} } { ::set name $variable }
      ::dict set d {*}$mpath {*}$path $name $value
    }
    ::return $d
  }

  proc destruct {var args} {
    ::set opVar [::lindex $var 0]
    ::set dArgs [::lrange $var 1 end]
    ::upvar 1 $opVar theDict
    ::if { ! [::info exists theDict] } {
      ::set theDict {}
    }
    ::set returnDict {}
    ::foreach val $args {
      ::lassign $val val nVar def
      ::if {$nVar eq ""} {::set nVar $val}
      ::upvar 1 $nVar $nVar
      ::if {$def ne ""} {
        ::set $nVar [::if? [::dict get? $theDict {*}$dArgs $val] $def]
      } else {
        ::set $nVar [::dict get? $theDict {*}$dArgs $val]
      }
      ::dict set returnDict $nVar [set $nVar]
      ::catch {::dict unset theDict {*}$dArgs $val}
    }
    ::return $returnDict
  }
  
  proc pickIf {var args} { ::return [::dict pick $var {*}$args] }
  
  proc pick {var args} {
    ::set tempDict {}
    ::foreach arg $args {
      ::lassign $arg key as
      ::if { [::dict exists $var $key] } {
        ::if { $as eq {} } { ::set as $key }
        ::set v [::dict get $var $key]
        ::if { $v ne {} } { ::dict set tempDict $as $v }
      }
    }
    ::return $tempDict
  }
  
  proc withKey {var key} {
    ::set tempDict {}
    ::dict for {k v} $var {
      ::if { [::dict exists $v $key] } {
        ::dict set tempDict $k [::dict get $v $key]        
      }
    }
    ::return $tempDict
  }
  
  ::proc fromlist { lst {values {}} } {
    ::set tempDict {}
    ::append tempDict [::join $lst " [list $values] "] " [list $values]"
  }
  
  proc sort {what dict args} {
    ::set res {}
    ::if {$dict eq {}} { ::return }
    ::set dictKeys [::dict keys $dict]
    ::switch -glob -nocase -- $what {
      "v*" {
        ::set valuePositions [::dict values $dict]
        ::foreach value [ ::lsort {*}$args [::dict values $dict] ] {
          ::set position       [::lsearch $valuePositions $value]
          ::if {$position eq -1} { ::puts "Error for $value" }
          ::set key            [::lindex $dictKeys $position]
          ::set dictKeys       [::lreplace $dictKeys $position $position]
          ::set valuePositions [::lreplace $valuePositions $position $position]
          ::dict set res $key $value
        }
      }
      "k*" -
      default {
        ::foreach key [::lsort {*}$args $dictKeys] {
          ::dict set res $key [::dict get $dict $key] 
        }
      }
    }
    ::return $res
  }
  
  proc invert {var args} {
    ::set d {}
    ::dict for {k v} $var {
      ::if {"-overwrite" in $args} {
        ::dict set d $v $k
      } else {
        ::dict lappend d $v $k
      }
    }
    ::return $d
  }
  
  proc json {json dict {key {}}} {
    ::upvar 1 $dict convertFrom
    ::if {![info exists convertFrom] || $convertFrom eq {}} { ::return }
    ::set key [::if? $key $dict]
    $json map_key $key map_open
      ::dict for {k v} $convertFrom {
        ::if {$v eq {} || $k eq {}} { ::continue }
        ::if {[::string is entier -strict $v]} {   $json string $k number $v
        } elseif {[::string is bool -strict $v]} { $json string $k bool $v
        } else {                                   $json string $k string $v  
        }
      }
    $json map_close
    ::return
  }
  
  proc serialize { json dict } {
    ::dict for {k v} $dict {
      ::if {$v eq {} || $k eq {}} { ::continue }
      ::if {[::string is entier -strict $v]} {   $json string $k number $v
      } elseif {[::string is bool -strict $v]} { $json string $k bool $v
      } else {                                   $json string $k string $v  
      }
    }
  }
  
  proc types {tempDict} {
    ::set typeDict {}
    ::dict for {k v} $tempDict {
      ::if {[::string is entier -strict $v]} {     ::dict set typeDict $k number
        } elseif {[::string is bool -strict $v]} { ::dict set typeDict $k bool
        } else {                                   ::dict set typeDict $k string 
        }
    }
    ::return $typeDict
  }
  
}

Category Data Structure

Category Command