In this beginner level article, Dave Cross look at the the Template Toolkit in a bit more detail.

This article was originally published in Linux Format in May 2004.

Last month we were processing one template at a time using ‘tpage’. Often you will want to process a set of associated templates at the same time. The Template Toolkit comes with a utility program called ‘ttree’ that allows you to do just that. It processes a set of files from an input directory and puts the processed versions into an output directory.

There are many other ways that ‘ttree’ is more powerful than ‘tpage’. It supports a huge number of options that control exactly how your templates are processed. You can get a preview of what all of those options are by typing ‘ttree -h’ or a more detailed description by typing “man ttree”. We’ll cover some of them in more detail later.

Networking Configuration Files

In this article we’ll create some configuration files that define a simple network. Specifically, we’ll generate an ‘/etc/hosts’ file and some of the files that are required to configure BIND. We’ll be looking at the simple network that which is described in the boxout “The Sunnydale Network”.

Getting Started with ‘ttree’

As I mentioned before, ‘ttree’ takes templates from an input directory, processes them, and writes the processed versions to an output directory. Therefore when starting with a new ‘ttree’ project, I like to create a project directory and subdirectories called ‘in’ and ‘out’.

You can configure ‘ttree’ in a number of ways. The easiest is probably to use a .ttreerc file. By default, ‘ttree’ looks for ~/.ttreerc, but you can override this by either using the -f option or by setting the $TTREERC environment variable. As I often like to have slightly different ttree configurations for different projects, I set $TTREERC to be ./.ttreerc and put a new .ttreerc file in each project directory.

One nice little touch is that if you run ‘ttree’ and you don’t have a .ttreerc file (either in your home directory or in the location defined by $TTREERC) then ‘ttree’ will offer to create a sample file for you. This file has most of the common ttree options in it together with copious comments that make it easy to edit. We won’t use that this time, we’ll edit our own .ttreerc from scratch. Here are the first three lines:

[text]
verbose
src = ./in
dest = ./out
[/text]

The second and third lines are pretty self-explainatory. They define the source and destination template directories. The first line puts ‘ttree’ into verbose mode where it tells you everything that it is doing.

With this in place we can test our first template. We’ll create a template that expands our network files into a hosts file. Here’s the template.

[text]
#
# /etc/hosts
#
[% USE networks = datafile(‘data/networks.txt’) -%]
[% FOREACH network = networks -%]
# Network: [% network.netname %]
# IP: [% network.number %]

[% USE hosts = datafile(“data/net_${network.netname}.txt”) -%]
[% FOREACH host = hosts -%]
[% host.IP %] [% host.hostname %] [% host.alias %]
[% END -%]
# End of network [% network.netname %]

[% END -%]

# End of networks
[/text]

If you put this in a file called ‘hosts’ in the ./in directory, you can then run ‘ttree’ like this

[text]
$ ttree
[/text]

If all is well, you will see output that looks something like this

[text]
ttree 2.75 (Template Toolkit version 2.13)

Source: ./in
Destination: ./out
Include Path: [ ]
Ignore: [ ]
Copy: [ ]
Accept: [ ]
Suffix: [ ]

+ hosts
[/text]

You’ll see that ‘ttree’ has reported on the source and destination directories that it has processed. It also reports on a number of other options which we will explain soon. At the end of the output you’ll see the name of our template file. The ‘+’ sign next to it indicates that the template was processed.

‘ttree’ has laziness built in. It will only process the templates that are necessary. It works out which templates to process by comparing the contents of the source and destination directories. A template is only processed if it doesn’t exist in the destination directory or if the source version is more recent than the destination version. In fact it works very much like ‘make’. If you try to run ‘ttree’ again immediately then the output will be identical to the first run except that the last line will be replaced with

[text]
– hosts (not modified)
[/text]

This indicates that as we haven’t updated ‘hosts’ there is no need to process it. If you want to process all of the templates without checking their timestamps, then just give ‘ttree’ the -a option.

The template itself is pretty simple. There’s not much in it that we didn’t cover last month. The only interesting thing is that we are using the ‘datafile’ plugin twice – once to open the mail ‘networks.txt’ file and then again within the loop to open each individual network’s data file. In this second usage, we have to use the more explicit syntax ‘${network.netname}’ to reference the network’s name. Without it the parser would have had little chance of realising what we meant. The Template Toolkit parser can usually work out what you want it to do, but in rare cases (like this one) it needs a bit of a hint.

If you look in the ./out directory, you will see the results of processing our template which should look like this.

[text]
#
# /etc/hosts
#
# Network: sunnydale
# IP: 192.168.1/24
192.168.1.1 buffy slayer
192.168.1.2 willow witch
192.168.1.3 xander
192.168.1.4 spike
# End of network sunnydale

# Network: los_angeles
# IP: 192.168.2/24
192.168.2.1 angel
192.168.2.2 cordelia cheerleader
192.168.2.3 wesley
192.168.2.4 spike
# End of network los_angeles

# End of networks
[/text]

Some More ‘ttree’ Options

Let’s fill in a few more options in our .ttreerc file, so it looks like this.

[text]
verbose
recurse
ignore = \b(CVS|RCS)\b
ignore = ^#
copy = \.png$
copy = \.gif$
src = ./in
dest = ./out
lib = ./lib
[/text]

The ‘recurse’ option tells ‘ttree’ to look in any subdirectories below your source directory and to recreate the same directory structure under the destination directory. The ‘ignore’ option lists files that should never be processed. The arguments to this option are interpreted as Perl regular expressions and filenames that match the regular expression are ignored. In this example the first ‘ignore’ line matches CVS or RCS thereby removing any sourcecode control files from consideration. The second line matches files that start with #, thereby ignoring emacs backup files. If you’re a ‘vi’ user you might like to replace that with ‘ignore = ~$’.

The next option is ‘copy’. This lists files that are simply copied from source to destination without processing. Here we are copying png and gif files. The final option defines a library directory. This is an additional directory where ‘ttree’ will look for templates. This is often used to store templates that are included in other templates and that aren’t intended to be processed on their own.

Two other useful options that we won’t be using in this example are ‘accept’ and ‘suffix’. ‘accept’ is the opposite of ‘ignore’ as it defines the set of files that will be processed. You normally only need to use one of ‘ignore’ or ‘accept’. ‘suffix’ gives you a way to change the extension of files as they are being processed. For example you might want to have a standard extension of .tt for templates, but convert that to .txt for the output files. In that case you could have a ‘suffix’ option which looked like ‘suffix tt=txt’.

Creating More Files

The main advantage that ‘ttree’ has over the simpler ‘tpage’ is that it processes a complete directory of templates in one go. So far our example only processes one template. So let’s add another.

Another file that could potentially be derived from our network definition data is a BIND configuration file, so here is a template that could be used to create such a file.

[text]
[% PROCESS config;
file = ‘db.’ _ main_domain;
FILTER redirect(file);
PROCESS soa domain=main_domain -%]

[% main_domain %] IN NS [% dns %].[% main_domain %]

; Hosts
[% USE networks = datafile(‘data/networks.txt’);
FOREACH network = networks;
USE hosts = datafile(“data/net_${network.netname}.txt”);
FOREACH host = hosts -%]
[% host.hostname _ ‘.’ _ main_domain _ ‘.’ | format(‘%-32s’) %] IN A \
[% host.IP %]
[% IF host.alias -%]
[% host.alias _ ‘.’ _ main_domain _ ‘.’ | format(‘%-32s’) %] IN CNAME \
[% host.hostname %].[% main_domain %].
[% END;
END;
END;
END -%]
[/text]

One thing that you’ll notice immediately is that because this template uses a lot of directives we have started to combine multiple directives within one tag set. The Template Toolkit parser allows you to do this as long as you separate the directives with semicolons.

This template is far more complex than anything that we’ve seen before so it’s worth going through it in some detail. It starts by processing another template called “config” which is shown below.

[text]
[% main_domain = ‘whedon.example.com’
ttl = ‘3h’
dns = ‘xander’
hostmaster = ‘hostmaster.whedon.example.com’
-%]
[/text]

This is a good example of a library template. All it does is defines some variables that we will need elsewhere. We don’t want to put it in the source directory as then it will be processed by ‘ttree’ and we will end up with an extra unnecessary output file. Therefore you should put the config template in the ./lib directory.

The next thing the template does is to create a new variable ‘file’ which contains the name of the required output file. In this example, ‘file’ will get the value ‘db.whedon.example.com’. We do this because we will eventually want to create a number of BIND configuration files and it will be nice to create them all using the same input template. To actually write the output to the correct file, we use the ‘redirect’ filter. This takes one parameter which is a filename and writes the output from the filter to that file. Everything from the opening FILTER directive to the matching END will end up in the new file. In this example, the END that matches our FILTER right at the end of the template, so everything is written to the given file.

The next directive processes another external template called ‘soa’. This template provides the “start of authority” block for the BIND file. Again, we’ve created a separate template as we would like to use it from several different templates. The template is shown below.

[text]
$TTL [% ttl %]
[% domain %]. IN SOA [% dns %].[% domain %]. [% hostmaster %]. (
[% serial %] ; serial
3h ; refresh
1h ; retry
1w ; expire
1h ) ; caching TTL
[/text]

This is another file which we don’t want ‘ttree’ to process, so once again we put it in the ./lib directory. Notice that the ‘soa’ template uses a variable called “domain” and that this is passed in as a named parameter in the PROCESS directive.

The next directive in the template simply adds the NS record to the output file. It uses simple variable expansion that we’ve seen many times before.

Then we come to the part of the template which creates the A and CNAME records for the hosts on the network. This uses the same kind of logic that we used for the hosts file to loop through the data contained in the various data files and display the correct records. One nice touch is that we use the “format” filter to ensure that the domain name part of the record is always padded to the same length. Here we use the short syntax for the FILTER directive where the FILTER keyword is replaced by the pipe character (‘|’). This makes it read a bit like a Unix filter command like ‘ls -l | sort’.

The template generates an A record for each host and a CNAME record for any aliases.

There’s one thing missing from this description of the template. Sharp-eyed readers (and BIND experts) will have noticed that the ‘soa’ template uses a variable called ‘serial’ and that hasn’t been defined anywhere. As the serial number needs to be incremented for each version of the configuration file I thought that it was pointless to include it in any of the template files. You could, of course, include it in the “config” template, but you would need to remember to update it each time you processed the template. In my opinion, it’s much easier to pass this value on the command line to ‘ttree’ and ‘ttree’ supports the same ‘–define var=value’ option as ‘tpage’ does. You can therefore process both of our templates with a command like this

[text]
$ ttree –define serial=1
[/text]

And you’ll see that both the ‘hosts’ template and the ‘db’ template are processed.

Other BIND Files

Of course one db file doesn’t make a complete BIND configuration. You’ll need to define reverse lookup files for the 1.168.192.in-addr.arpa and 2.168.192.in-addr.arpa domains as well as for the loopback domain. You’ll also need the actual ‘named.conf’ file that pulls all of these files together. I don’t have space to demonstrate creating all of these in this tutorial, but I hope I’ve given you some ideas on how you might go about it.

Template Complexity

This month’s templates have been a lot more complex than the ones that we saw last month. It would be easy to argue that they were too complex. Part of this complexity comes from trying to do too much in a template. Templates should really only be concerned with presentation logic.

We can simplify our templates significantly if we use Perl to gather the data that we want to display and only use the Template Toolkit language to control how we display that data. We’ll look at how you do that in next month’s tutorial.