tl-who - Port of CL-WHO to TXR Lisp with fixes and improvements

What is TL-WHO?

TL-WHO is a translation of the Common Lisp library CL-WHO to TXR Lisp. It is a derived work of CL-WHO, copyrighted by the original authors and carrying their licensing notices.

TL-WHO also provides a macro called deftag, which is inspired by a same-named macro described in the documentation of another Lisp HTML generation library called Spinneret.

Unlike some HTML generating libraries, TL-WHO works by translating the syntax into code that writes HTML to a stream, rather that translating the syntax into code which constructs a nested data structure, which is then traversed to produce HTML. TL-WHO merges strings as much as possible.

For instance, here is what happens when we use it on syntax which is entirely static (contains no computations whose results are to be inserted into the HTML):

1> (tl-who:with-html-output (*stdout* nil :indent t)
     (:table (:tr (:td (:span "foo")))))

<table>
  <tr>
    <td>
      <span>foo
      </span>
    </td>
  </tr>
</table>

And here the code which produces the HTML:

2> (expand '(tl-who:with-html-output (*stdout* nil :indent t)
              (:table (:tr (:td (:span "foo"))))))
(let ((*stdout* (sys:dvbind *stdout*
                  *stdout*)))
  (put-string "\n<table>\n  <tr>\n    <td>\n      <span>foo\n      </span>\n    </td>\n  </tr>\n</table>"
              *stdout*))

What is noteworthy here is that a single string is produced out of the nested Lisp syntax, and that string is therefore sent to the *stdout* stream using a single call to the put-string function.

Another nice feature of TL-WHO is that it has a "quiet" syntax free of extraneous punctuation and nesting. A HTML tag is written as a list headed by a keyword symbol, for instance (:a) encodes <a></a>. If this keyword is followed by keyword-value properties, those are the attributes:

(:a :href "foo.html")
-> <a href="foo.html"></a>

(:a :href "foo.html" :target "__blank")
-> <a href="foo.html" target="__blank"></a>

Then any remaining material after the keyword-value pairs is interpreted as the tag content.

(:a :href "foo.html" :target "__blank" "foo")
-> <a href="foo.html">foo</a>

All material that is not a tag or attribute keyword is an evaluated Lisp expression, so no special syntax is needed to indicate evaluation.

When the value of an attribute is given as a Lisp expression, that value is interpolated automatically into the generated attribute syntax.

In the interior of a tag, when a Lisp expression is evaluated, its value is not turned into output. It is that expression's responsibility to generate output, either directly to the HTML stream object, or via the convenient local macros like fmt, esc or str.

Documentation

Users of TL-WHO may rely on the original CL-WHO API documentation.

Here are the differences to be aware of:

The symbol package is tl-who not cl-who. It is case-sensitive; the symbol TL-WHO:WITH-HTML-OUTPUT refers to a non-existent package, and so cannot even be read.
The *downcase-tokens-p* variable does not exist in TL-WHO. It is replaced by *upcase-tokens-p*, whose default value is nil. TXR Lisp is case sensitive and lower-case is preferred for symbols. A symbol like :table in TXR Lisp has the name "table" rather than "TABLE". And HTML is usually written in lower case nowadays, so no case conversion is required to turn :table into <table>. Therefore, in TL-WHO it is better to have a case conversion variable that is off by default.
Because TXR Lisp is case-sensitive, a symbol like :TABLE will produce a <TABLE> tag in upper case, whereas :table will produce <table>. They are different symbols. Thus it is possible to control case on a case by case basis, no pun intended. If the *upcase-tokens-p* variable is bound to t, then both will produce <TABLE>.
TL-WHO fixes the issue that CL-WHO doesn't HTML-escape the values of of constant string material, of attributes, and of material added by local macro fmt.

This is a potential security issue, because if an untrusted value is interpolated, it can be a vector for an injection attack. The special variable *cl-who-compat* can be set true to disable the escaping, but is not recommended.
```
[1]> (in-package :cl-who)
#<PACKAGE CL-WHO>
WHO[2]> (with-html-output-to-string (out)
          (:a :href "https://example.com'>malicious here</a><a href='blah" "click me"))
"<a href='https://example.com'>malicious here</a><a href='blah'>click me</a>"
```
Not escaping constant material is error-prone.

The CL-WHO user has to remember to write (:div "black&white") whereas the TL-WHO user just writes (:div "black&white"); the & escape is produced by TL-WHO.
TL-WHO provides a noesc syntax. When the value of an attribute is expressed as (noesc <expr>), escaping is disabled:
```
(:a :href (noesc trusted-url) "click me")
```
A more limited form of noesc can be used in tag bodies, to defeat the default escaping applied to constant material.

The following:
```
(:div (noesc "&amp;") " &lt;")
```
produces <div>& &lt;</div>.
TL-WHO provides a noesc-fmt which doesn't HTML-escape.
TL-WHO provides escq and escj local macros. escq is like esc but also HTML-escapes the ASCII apostrophe and double quote. escj escapes a string such that it can be safely interpolated into a Javascript string literal, which is itself embedded in a HTML script tag.
TL-WHO's esc accepts arguments which are not strings. They are turned to a string using tostringp, and HTML-escaped.
TL-WHO allows fmt as well as escj to be used to calculate attributes. These local macros have alternative definitions when used in attributes. The remaining local macros don't make sense in attributes, and make a mess of the output if used; unlike CL-WHO, TL-WHO warns when they are used in attributes.
The elaborate manual HTML escaping support featured in CL-WHO is absent in TL-WHO. These symbols do not exist: escape-string, escape-char, *escape-char-p*, escape-string-minimal, escape-string-minimal, escape-string-minimal-plus-quotes, escape-string-iso-8859-1, escape-string-all, escape-char-minimal, escape-char-minimal-plus-quotes, escape-char-iso-8859-1 and escape-char-all. TL-WHO uses the TXR Lisp standard functions html-encode, and html-encode*, which have no options to control its behavior. html-encode is used internally for escaping material to be inserted into attributes; html-encode* is used by esc and fmt.
TL-WHO provides a deftag macro for defining macro-expanding tags. This is inspired by a deftag described in the Spinneret documentation. Though ordinary macros can easily be used inside with-html-output, deftag is an expansion mechanism which transforms the tag markup, rather than embedded Lisp code. It is documented below.
The CL-WHO conc function is missing. TXR Lisp has a function like this, which is called join, and that is what is used in TL-WHO.
There is no generic function convert-tag-to-string-list. It is an ordinary function. There is no support for customizing its behavior. TXR Lisp doesn't have CLOS, and thus no generic functions or methods. A mechanism for customizing the conversion of tags to string lists may be adopted by TL-WHO in the future.
Numerous instances of internal unused code that in CL-WHO do not appear at all in TL-WHO; they have been pruned. Code depending on internal CL-WHO functions that are not present in TL-WHO will not work and must be ported in an alternative way.
TXR Lisp isn't Common Lisp and so CL-WHO examples which use ANSI CL idioms like the loop macro will not work. All Lisp code embedded in a CL-WHO expression has to be translated to TXR Lisp in order for that expression to work as TL-WHO; the TL-WHO library doesn't translate embedded Common Lisp to TXR Lisp; it only translates the markup syntax to HTML in the same way as CL-WHO.
The variable *attribute-quote-char* will not work if it is bound by let around a with-html-output-form. The reason is that TL-WHO interpolates this character at macro-expansion time, even for HTML attributes whose values are calculated at run-time. The binding construct expander-let must be used, or else the variable's global binding must be assigned.

CL-WHO has some bugs around attribute handling. When the value of an attribute is a constant expression, only the specific values T and NIL are treated properly, not constant expressions which evaluate to T and NIL. Then we see the mistaken attribute values 'NIL and 'T': This works properly in TL-WHO:

[1]> (in-package :cl-who)
#<PACKAGE CL-WHO>
WHO[2]> (with-html-output-to-string (str)
          (:foo :bar t))
"<foo bar='bar'></foo>"
WHO[3]> (with-html-output-to-string (str)
          (:foo :bar (quote t)))
"<foo bar='T'></foo>"
WHO[4]> (with-html-output-to-string (str)
          (:foo :bar nil))
"<foo></foo>"
WHO[5]> (with-html-output-to-string (str)
          (:foo :bar 'nil))
"<foo bar='NIL'></foo>"

Additionally, users (of both CL-WHO and TL-WHO alike) are advised to watch for the following issue: the CL-WHO documentation is not accurately maintained and makes some references to material that no longer exists in CL-WHO, such as the macro show-html-expansion, which was removed from CL-WHO in 2009.

The `deftag` macro

Syntax:

(deftag <keyword> <attr-param-list> <body-pattern>
  <body> ...)

<attr-param-list> ::= (<key-param>* [ . <rest-param>])

<key-param> := <symbol> | (<name> [<default> [<pres-var>]])

<body-pattern> := <macro-style-parameter-list>

Description:

A deftag macro rewrites HTML markup written in TL-WHO syntax into other markup.

Simple example:

(deftag :boldface (. attrs) body
  ^(:span :font "bold" ,*attrs ,*body))

(with-html-output-to-string (out)
  (:boldface :id "hello-id" "Hello!"))

--> <span font='bold' id='hello-id'>Hello!</span>

Complex example, adapted from Spinneret documentation:

(deftag :easy-input (label (name (gensym))
                     (id name) (type "text") . other-attrs) default
  ^(progn
     (:label :for ,name ,label)
     (:input :name ,name :id ,id :type ,type
             ,*other-attrs :value (progn ,*default))))

Note that progn here isn't the Lisp progn operator; it is recognized by the deftag expansion mechanism as a way of producing multiple elements. To invoke :easy-input, we might do this:

3> (with-html-output (*stdout* nil :indent t)
     (:div :class "cls"
       (:easy-input :name "foo" :id "foo-23"
                    :style "style" :label "lab" "123")))
<div class='cls'>
  <label for='foo'>lab
  </label>
  <input name='foo' id='foo-23' type='text' style='style' value='123' />
</div>

Note a small flexibility: the body argument "123" of :easy-input wasn't inserted as an element in the middle of the tag as in the simple example, but as an attribute value.

Note that the other-attrs rest parameter of :easy-input received a list which only contained the :style attribute; all the others were captured by their respective keyword parameters.

The following example shows the destructuring <body-pattern>, which gives the :atag non-attribute parameters.

 ::text
 (deftag :atag (. other-attrs) (url cls . rest)
   ^(:a :href ,url :class ,cls ,*other-attrs ,*rest))

This :atag is then used like this:

 ::text
 3> (with-html-output (*stdout*)
      (:atag "https://example.com" "ext" "link text"))
 <a href='https://example.com' class='ext'>link text</a>

Here, the "https://example.c" item from the body got matched as the url parameter, then cls took "ext" and rest got the rest of the body as a list of items, which includes "link text".

Other attributes tag can be passed through thanks to other-attributes parameter:

 ::text
 3> (with-html-output (*stdout*)
      (:atag :target "_blank" "https://example.com" "ext" "link text"))
 <a href='https://example.com' class='ext' target='_blank'>link text</a>

Parameters:

The attr-param-list is an implicit keyword parameter list which destructures attributes. It can specify default values for attributes that are not passed. Its rest parameter is bound to the remaining attributes that were not captured by the parameters.

Each key-param is an ordinary TXR Lisp key parameter, implemented by the TXR Lisp :key parameter list macro. It may be just a name, or a name with default expression giving a value for the parameter if the corresponds keyword argument is missing, possibly followed by another variable name which, if present, will be a Boolean value indicating, if true, that the keyword argument was present, or if false that it was missing (and thus defaulted).

The body-pattern is a destructuring pattern (macro-style parameter list). It captures the body that is passed to the tag: material after the attributes. Usually, this is specified as a single symbol, which captures the entire body. However, if a pattern is specified, the body material is destructurd, which effectively allows deftag tags to have parameters, as in the :atag example above.

Returns:

The deftag macro returns the <keyword> parameter.

Dependencies

TL-WHO has no external dependencies other than TXR itself.

It requires version 288 or newer.

Installation/Use

To use TL-WHO, you just have to put the code somewhere and load the tl-who.tl file (or its compiled version, if you compile the code).

(load "path/to/tl-who")

TXR Lisp's load is intelligent; the tl-who loader module easily loads the needed files, which it finds in the same directory as itself.

To compile all the TL-WHO files:

(load "path/to/tl-who" :compile)

Cleaning compiled files away is similar; use the load argument :clean instead of :compile.

TL-WHO symbols are in a package called tl-who, which is case sensitive.

Its private symbols are in tl-who-priv.

Two TL-WHO symbols have the same names as standard TXR library symbols, namely fmt and str. Code which references the tl-who package in preference to the usr package (for instance by putting tl-who first in the package :fallback list) must use usr:fmt and usr:str to refer to the TXR Lispfmt functions, because the unprefixed tokens will produce tl-who:fmt and tl-who:str, which are different symbols.

License

TL-WHO is distributed under the two-clause BSD license. LICENSE file.