Regular expressions are a formal way to describe string patterns. They provide a powerful and compact way to specify patterns in your data. Even better, there is a very efficient implementation of the regular expression mechanism due to Henry Spenser. If your script does much string processing, it is worth the effort to learn about regular expressions. Your Tcl scripts will be compact and efficient. This chapter includes a large set of canned regular expressions you can pick up and use.
Regular expression substitution is a mechanism that lets you rewrite a string based on regular expression matching. This is another powerful tool, and this chapter includes several examples that do a lot of work in just a few Tcl commands. One of the key ideas is to transform your input data into a Tcl script, and then evaluate the script to process the data. The idea takes a moment to get used to, but it provides a very efficient way to process strings.
A number of examples of regular expressions follow. Any pattern that contains brackets, dollar sign, or spaces must be handled specially when used in a Tcl command. It is unfortunate, but there is a clash between the Tcl special characters and the regular expression special characters. The patterns in this section ignore this problem. The sample expressions in Table X are quoted for use within Tcl scripts.
..The matching character can be restricted to a set of characters with the [xyz] syntax. Any of the characters between the two brackets is allowed to match. For example, the following matches either Hello or hello:
[Hh]elloThe matching set can be specified as a range over the ASCII character set with the [x-y] syntax. There is also the ability to specify the complement of a set. That is, the matching character can be anything except what is in the set. This is achieved with the [^xyz] syntax. Ranges and complements can be combined. The following matches anything except the upper and lowercase letters:
[^a-zA-Z]Repetition is specified with *, for zero-or-more, +, for one-or-more, and ?, for zero-or-one. These operators apply to the previous item, which is either a matching character, which could involve the set syntax, or a subpattern grouped with parentheses. The following matches a string that contains b followed by zero or more a's:
ba*While the following matches a string that has one or more sequences of ab:
(ab)+The pattern that matches anything is:
.*Alternation is specified with |, a pipe symbol. Another way to match either Hello or hello is:
hello|HelloBy default a pattern does not have to match the whole string. There can be unmatched characters before and after the match. You can anchor the match to the beginning of the string by starting the pattern with ^, or to the end of the string by ending the pattern with $. You can force the pattern to match the whole string by using both. All strings that begin with spaces or tabs are matched with:
^[ \t]+If a pattern can match several parts of a string, the matcher takes the match that occurs earliest in the input string. Then, if there is more than one match from that same point, the matcher takes the longest possible match. The rule of thumb is: first, then longest.
http://www.sun.com:80/index.htmlThe regular expression is:
[^:]+://[^:/]+(:[0-9]+)?/.*Let's look at the pattern one piece at a time. The first part looks for the protocol, which is separated by a colon from the rest of the URL. The first part of the pattern is one or more characters that are not a colon, followed by a colon. This matches the http: part of the URL:
[^:]+:The next part of the pattern looks for the server name, which comes after two slashes. The server name is either followed by a colon and a port number, or by a slash. The next part of the pattern includes the slashes, and then has a subpattern of one or more characters that are not a colon or a slash. This matches the //www.sun.com part of the URL:
//[^:/]+The port number is optional, so a subpattern is delimited with parentheses and followed by a question mark. This matches the :80 part of the URL:
(:[0-9]+)?The last part of the pattern is everything else, starting with a slash. This matches the /index.html part of the URL:
/.*To make this pattern really useful we delimit several subpatterns with parentheses:
([^:]+)://([^:/]+)(:([0-9]+))?(/.*)These parentheses do not change the way the pattern matches. Only the optional port number really needs the parentheses in this example. However, the regexp command described next gives us access to the strings that match these subpatterns. In one step regexp can test for a valid URL and divide it into the protocol part, the server, the port, and the trailing path. This is shown later in Example 11-2 on page 121.
The regexp Command
The regexp command provides direct access to the regular expression matcher. Not only does it tell you if a string matches a pattern, it can extract one or more matching substrings. The return value is 1 if some part of the string matches the pattern, it is 0 otherwise. Its syntax is:
regexp ?flags? pattern string ?match sub1 sub2...?
The flags are optional and constrain the match as follows:
If string matches pattern, then the results of the match are stored in the variables named in the command. These match variable arguments are optional. If present, match is set to be the part of the string that matched the pattern. The remaining variables are set to be the substrings of string that matched the corresponding subpatterns in pattern.. The correspondence is based on the order of left parentheses in the pattern to avoid ambiguities that can arise from nested subpatterns.
Example 11-1 uses regexp to pick the hostname out of the DISPLAY environment variable, which has the form:
hostname:display.screen
set env(DISPLAY) corvina:0.1
regexp {([^:]*):} $env(DISPLAY) match host
=> 1
set match
=> corvina:
set host=> corvina
The pattern involves a complementary set, [^:], to match anything except a colon. It uses repetition, *, to repeat that zero or more times. It groups that part into a subexpression with parentheses. The literal colon ensures that the DISPLAY value matches the format we expect. The part of the string that matches the complete pattern is stored into the match variable. The part that matches the subpattern is stored into host. The whole pattern has been grouped with braces to quote the square brackets. Without braces it would be:
regexp (\[^:]*): $env(DISPLAY) match hostThis is quite a powerful statement, and it is efficient. If we only had the string command to work with, we would have needed to resort to the following, which takes roughly twice as long to interpret:
set i [string first : $env(DISPLAY)]
if {$i >= 0} {
set host [string range $env(DISPLAY) 0 [expr $i-1]]
}Example 11-2 demonstrates a pattern with several subpatterns that extract the different parts of a URL. There are lots of subpatterns, and you can determine which match variable is associated with which subpattern by counting left parenthesis. Note that the port number uses a nested subpattern. The subpattern (:([0-9]+))? uses the zero-or-more pattern character ? because the port specification is optional. However, this puts a colon into the match variable x, so another subpattern is used to just get the digits of the port number into the port match variable:
set url http://www.sun.com:80/index.html
regexp {([^:]+)://([^:/]+)(:([0-9]+))?(/.*)} $url \
match protocol server x port path
=> 1
set match
=> http://www.sun.com:80/index.html
set protocol
=> http
set server
=> www.sun.com
set x
=> :80
set port
=> 80
set path=> /index.html
The regsub Command
The regsub command does string substitution based on pattern matching. Its syntax is:
regsub ?switches? pattern string subspec varnameThe regsub command returns the number of matches and replacements, or 0 if there was no match. regsub copies string to varname, replacing occurrences of pattern with the substitution specified by subspec.
The optional switches include -all, which means to replace all occurrences of the pattern. Otherwise only the first occurrence is replaced. The -nocase switch means that upper-case characters in the string are converted to lowercase before matching. The -- switch separates the pattern from the switches, which is necessary if your pattern begins with a -.
The replacement pattern, subspec, can contain literal characters as well as the following special sequences:
regsub ^$env(HOME)/ $pathname ~/ newpathThe following constructs a C compile command line given a filename:
regsub {([^\.]*)\.c} file.c {cc -c & -o \1.o} ccCmd
The \. is used to specify a match against period. The & is replaced with file.c, and \1 is replaced with file, which matches the pattern between the parentheses. The value assigned to ccCmd is:
cc -c file.c -o file.o
These examples are primarily written by Stephen Uhler.
Replacing the + with spaces requires quoting the + because it is the one-or-more special character in regular expressions:
regsub -all {\+} $url { } url
The %xx are replaced with a format command that will generate the right character:
regsub -all {%([0-9a-hA-H][0-9a-hA-H])} $url \
{[format %c 0x\1]} url
The %c directive to format tells it to generate the character from a character code number. We force a hexadecimal interpretation with a leading 0x. The resulting string is passed to subst to get the format commands subsituted:
set url [subst $url]For example, if the input is %7ewelch, the result of the regsub is:
[format %c 0x7e]welchAnd then subst generates:
~welchExample 11-3 encapsulates this trick in the Url_Decode procedure.
proc Url_Decode {url} {
regsub -all {\+} $url { } url
regsub -all {%([0-9a-hA-H][0-9a-hA-H])} $url \
{[format %c 0x\1]} url
return [subst $url]
}
name1=value1&name2=value2&name3=value3Each of the names and values is URL encoded as described above. Example 11-4 uses split to chop up strings in this format and decodes them with Url_decode. It assign the values to an array named cgi. An HTML form can have several instances of the same name, and this can result in repeated names. This is why an lappend is used to assign to the array. This makes it a bit awkward to get the values back; either lindex or foreach can be used to get a single value or iterate through multiple values associated with the same name:
proc Cgi_Parse {} {
global cgi env
if [info exists env(CGI_QUERY)] {
# This comes from GET-style forms
set query $env(CGI_QUERY)
} else{
# This comes from POST-style forms
set query [read stdin]
}
foreach {name value} [split $query &=] {
lappend cgi([Url_Decode $name]) [Url_Decode $value]
}
}
proc Cgi_Value {name} {
global cgi
if ![info exists cgi($name)] {
return {}
} elseif {[llength $cgi($name)] == 1} {
# Strip a level of listing for convenience
return [lindex $cgi($name) 0]
} else {
# Return the whole list of values
return $cgi($name)
}
}
The entity decoder is similar to Url_Decode. In this case, however, we need to be more careful with subst. The text passed to the decoder could contain special characters like square bracket or dollar sign. With Url_Decode we can rely on those special characters being encoded as, for example %24. Entity encoding is different (do not ask me why URLs and HTML have different encoding standards), and dollar sign and square brackets are not necessarily encoded. This requires an additional pass to quote these characters. This regsub puts a backslash in front of all the brackets, dollar signs, and backslashes.
regsub -all {([][$\\])} $text {\\\1} new
The decimal encoding (e.g., ©) is also more awkward than the hexidecimal encoding used in URLs. We cannot force a decimal interpretation of a number in Tcl. In particular, if the entity has a leading zero (e.g. 
) then Tcl interprets the value (e.g., 010) as octal. The scan command is used to do a decimal interpretation:
regsub -all {&#([0-9][0-9]?[0-9]?);?} $new \
{[format %c [scan \1 %d tmp;set tmp]]} new
The named entities are converted with an array that maps from the entity names to the special character. The only detail is that unknown entitiy names (e.g., &foobar;) are not converted. This mapping is done inside HtmlMapEntity, which guards against invalid entities.
regsub -all {&([a-zA-Z]+)(;?)} $new \
{[HtmlMapEntity \1 \\\2 ]} new
If the input text contained:
[x < y]Then the regsubs would transform this into:
\[x [HtmlMapEntity lt \; ] y\]Finally, subst will result in:
[x < y]
proc Html_DecodeEntify {text} {
if {![regexp & $text]} {return $text}
regsub -all {([][$\\])} $text {\\\1} new
regsub -all {&#([0-9][0-9]?[0-9]?);?} $new {\
[format %c [scan \1 %d tmp;set tmp]]} new
regsub -all {&([a-zA-Z]+)(;?)} $new \
{[HtmlMapEntity \1 \\\2 ]} new
return [subst $new]
}
proc HtmlMapEntity {text {semi {}} {
global htmlEntityMap
set result $text$semi
catch {set result $htmlEntityMap($text)}
return $result
}
# Some of the htmlEntityMap
array set htmlEntityMap {
lt < gt > amp &
aring \xe5 atilde \xe3
copy \xa9 ecirc \xea egrave \xe8
}
proc Html_Parse {html cmd {start {}} {
# Map braces and backslashes into HTML entities
regsub -all \{ $html {\&ob;} html
regsub -all \} $html {\&cb;} html
regsub -all {\\} $html {\&bsl;} html
# This pattern matches the parts of an HTML tag
set w " \t\r\n" ;# white space
set exp <(/?)(\[^$w>]+)\[$w]*(\[^>]*)>
# This generates a call to cmd with HTML tag parts as args
# \1 is the leading /, if any
# \2 is the HTML tag name
# \3 is the parameters to the tag, if any
# The curly braces at either end group of all the free text
# after the HTML tag, which becomes the last arg to $cmd.
set sub "\}\n$cmd {\\2} {\\1} {\\3} \{"
regsub -all $exp $html $sub html
# This balances the curley braces in the generated script,
# and calls $cmd with $start as a pseudo-tag
# at the beginning and end of the script.
eval "$cmd {$start} {} {} {$html}"
eval "$cmd {$start} / {} {}"
}An example will help visualize the transformation. Given this HTML:
<Title>My Home Page</Title>
<Body bgcolor=white text=black>
<H1>My Home</H1>This is my <b>home</b> page.
And a call to Html_Parse that looks like this:
Html_Parse $html {Render .text} hmstart
Then the generated program is this:Render .text {hmstart} {} {} {}
Render .text {Title} {} {} {My Home Page}
Render .text {Title} {/} {} {
}
Render .text {Body} {} {bgcolor=white text=black} {
}
Render .text {H1} {} {} {My Home}
Render .text {H1} {/} {} {
This is my }
Render .text {b} {} {} {home}
Render .text {b} {/} {} { page.
}
Render .text {hmstart} / {} {}
One overall point to make about this example is the difference between using eval and subst with the generated script. The decoders shown in Example 11-3 and 11-5 use subst to selectively replace encoded characters while ignoring the rest of the text. In Html_Parse we must process all the text. The main trick is to replace the matching text (e.g., the HTML tag) with some Tcl code that ends in an open curly brace and starts with a close curly brace. This effectively groups all the unmatched text.
When eval is used this way you must do something with any braces and backslashes in the unmatched text. Otherwise the resulting script does not parse correctly. In this case these special characters are encoded as HTML entities. The cmd that is called must deal with encoded entities already. It is not possible to quote these special characters with backslashes because all this text is inside a curly braces so no backslash substitution is performed. The backslashes will be seen by the cmd callback.
eval "$cmd {$start} {} {} {$html}"
I always forget that $start and $html are substituted inspite of the braces. This is because double quotes are being used to group the argument, so the quoting effect of braces is turned off. Try this:
set x hmstart
set y "foo {$x} bar"
=> foo {hmstart} bar