At the heart of marmotte are two pipelines: one for processing requests, and one for preparing responses to these requests. Both pipelines are implemented in the same manner: a collection of transformations applied to a piece of data describing a request or a response. Each transformation is made up of a predicate and a function. To understand how this works, let's follow a simple exchange between a client and a server.

A client sends a request to the server, in the form of a selector (a string). Upon receving the request, the server simply goes through the collection of request transformations, asking each, via its associated predicate, if it should run, and if so, calls the function with the request. The modified request is then passed to the next transformation in the pipeline, and so on until all candidate transformations have got a chance to run. The final version of the request is then passed to the response pipeline the same way: each transformation lets its predicate decide whether it should run, in which case its function is run against the request and the current version of the response, and both are passed down the rest of the collection. The final response is what is returned to the client.

This generic approach allows to satisfy basic cases, where a request contains a path to a file, for example, and doesn't need any change, and where a response is handled by a single transformation, which simply loads the data from the corresponding file and stores it in the response. But the same architecture caters to more complex scenarios. One is that of selector rewriting: it is similar to URL rewriting, where a path in a selector is changed to map onto a real, physical directory, and is an easy way of implementing virtual directories. Another is transforming the data itself: adding headers, footers, modifying the data, compressing it… All this can be implemented via instances of marmotte's RequestTransformers and ResponseTransformers.

The current version of marmotte implements dynamic gophermaps that way: when handling a directory where no gophermap file is present, the ResponseTransformer simply generates one. Headers and Footers are also available and multiple instances can be defined: one can devise a Footer for all gophermaps, a Header for textfiles in a specific directory, another Header for files maching specific names, and these can be combined or remain exclusive. This is made possible thanks to having the functions being separate from the predicates: a given transformation function can be combined with any predicate and packaged in a ResponseTransformer any number of times. In the examples above, we would have three ResponseTransformers, each having the same functions (Header or Footer) and different predicates. Some would only match on gophermaps, some on files with a specific name, or in a specific path…

Over time, I will keep adding to these transformation functions and predicates to implement gateways to other protocols, file format conversion, file compression, responses that would change depending on the time of the day, and so on.

marmotte now also has a new ResponseTransformer, the ExecutableTransformer. As the name suggests, it allows marmotte to call external programs and return the data generated by these programs. The programs in question can be anything that marmotte can call: shell scripts, Perl scripts, binaries… The main contract between marmotte and a program is in two parts: the latter will receive a root path where to look for and generate files and directories, and the former will get in return a single string, which can be either a directory or a file. The response crafted by this transformation is then picked up by the rest of the pipeline and either presented as a gophermap (generated or loaded from a gophermap file the program would have placed in the directory), or loaded from a file on the disk.

Below is a simple example:

#!/bin/sh

# Generate the filename
FILENAME=`cat /dev/random | env LC_CTYPE=C tr -cd "[:alnum:]" | head -c 32`
FILEPATH=/tmp/$FILENAME

# Generate a file with some text contents
if [ ! -d $1/tmp ]; then
   mkdir $1/tmp
fi

echo "This is an automatically generated file, named $FILENAME." > $1/$FILEPATH

# Return the filepath to marmotte
echo $FILEPATH

This shell script generates a file with a random name (tapping into the power of /dev/random), places it in the subdirectory /tmp of the gopherhole and returns the path to it (relative to the server's root). Nothing fancy, but anything can happen between the first line and the last line…

Having the ability to run executables opens many, many doors: this is one of the main reasons the web exploded in the mid-90s, and is akin to the cgi-bin that were popular then (and still are everywhere today, in the more modern implementation of fast-cgi). A file can be altered before being served, or generated from scratch, and entire directory hierarchies can even be generated in one go and served to the client.

One thing that I will need to implement in the coming weeks is sandboxing, to restrict the programs to only safe portions of the filesystem, instead on relying on the executables to politely place everything under the server root given in argument.

The last addition in terms of dynamic rendering is templates. This is another feature that propelled the early Web forward in the 90s, and opens at least as many doors as the ability to call external executables. marmotte being written in Go, I've made the simple choice here to leverage the power of the template engine that comes with this language.

The current implementation is still in its very early stages, and is only used for dynamic error reporting (more on that in a later article). These error documents are designed as Go templates that accept from the server a structure of data containing arbitrary informations.

Here's an example below:

You have the wrong permissions to access the resources behind the selector {{ .Selector }}. If you believe you should have access to these resources, please contact the administrator.

Here, the selector originally sent by the client will be displayed, coming from the data structure provided by the server. This might seem fairly basic, but this opens all kinds of possibilities, as this structure can contain any type of data, from any provenance, and even data contributed to by other RequestTranformers or ResponseTransformers, and be used in any kind of document, not only error messages.

In the coming weeks, I will probably introduce these templates to the existing Headers and Footers facilities, to make these even more interesting, and simply generalise and promote the use of templates as a very simple way to prepare dynamic content. Again, ResponseTransformers will do the work here, simply running against any file with a .gt extension, for Gopher Template.