path: root/README.md

## What is TL-WHO?

TL-WHO is a translation of the Common Lisp library
[CL-WHO](https://edicl.github.io/cl-who/) to TXR Lisp.
It is a derived work of CL-WHO, copyrighted by the original authors and
carrying their licensing notices.

TL-WHO also provides a macro called `deftag`, which is inspired
by a same-named macro described in the documentation of another Lisp HTML
generation library called [Spinneret](https://github.com/ruricolist/spinneret).

Unlike some HTML generating libraries, TL-WHO works by translating the
syntax into code that writes HTML to a stream, rather that translating
the syntax into code which constructs a nested data structure, which is then
traversed to produce HTML. TL-WHO merges strings as much as possible.

For instance, here is what happens when we use it on syntax which is
entirely static (contains no computations whose results are to be
inserted into the HTML):

    ::text
    1> (tl-who:with-html-output (*stdout* nil :indent t)
         (:table (:tr (:td (:span "foo")))))

    <table>
      <tr>
        <td>
          <span>foo
          </span>
        </td>
      </tr>
    </table>

And here the code which produces the HTML:

    ::text
    2> (expand '(tl-who:with-html-output (*stdout* nil :indent t)
                  (:table (:tr (:td (:span "foo"))))))
    (let ((*stdout* (sys:dvbind *stdout*
                      *stdout*)))
      (put-string "\n<table>\n  <tr>\n    <td>\n      <span>foo\n      </span>\n    </td>\n  </tr>\n</table>"
                  *stdout*))

What is noteworthy here is that a single string is produced out of the nested
Lisp syntax, and that string is therefore sent to the `*stdout*` stream
using a single call to the `put-string` function.

Another nice feature of TL-WHO is that it has a "quiet" syntax free
of extraneous punctuation and nesting. A HTML tag is
written as a list headed by a keyword symbol, for instance `(:a)`
encodes `<a></a>`. If this keyword is followed by keyword-value
properties, those are the attributes:

    ::text
    (:a :href "foo.html")
    -> <a href="foo.html"></a>

    (:a :href "foo.html" :target "__blank")
    -> <a href="foo.html" target="__blank"></a>

Then any remaining material after the keyword-value pairs is
interpreted as the tag content.

    ::text
    (:a :href "foo.html" :target "__blank" "foo")
    -> <a href="foo.html">foo</a>

All material that is not a tag or attribute keyword is an evaluated Lisp
expression, so no special syntax is needed to indicate evaluation.

When the value of an attribute is given as a Lisp expression, that
value is interpolated automatically into the generated attribute
syntax.

In the interior of a tag, when a Lisp expression is evaluated, its
value is **not** turned into output. It is that expression's
responsibility to generate output, either directly to the HTML
stream object, or via the convenient local macros like `fmt`,
`esc` or `str`.

## Documentation

Users of TL-WHO may rely on the original CL-WHO API documentation.

Here are the differences to be aware of:

*   The symbol package is `tl-who` not `cl-who`. It is case-sensitive;
    the symbol `TL-WHO:WITH-HTML-OUTPUT` refers to a non-existent
    package, and so cannot even be read.

*   The `*downcase-tokens-p*` variable does not exist in TL-WHO. It is
    replaced by `*upcase-tokens-p*`, whose default value is `nil`.
    TXR Lisp is case sensitive and lower-case is preferred for symbols.
    A symbol like `:table` in TXR Lisp has the name `"table"` rather
    than `"TABLE"`. And HTML is usually written in lower case nowadays,
    so no case conversion is required to turn `:table` into `<table>`.
    Therefore, in TL-WHO it is better to have a case conversion
    variable that is off by default.

*   Because TXR Lisp is case-sensitive, a symbol like `:TABLE` will
    produce a `<TABLE>` tag in upper case, whereas `:table` will
    produce `<table>`. They are different symbols. Thus it is possible to
    control case on a case by case basis, no pun intended. If the
    `*upcase-tokens-p*` variable is bound to `t`, then both will
    produce `<TABLE>`.

*   TL-WHO fixes the issue that CL-WHO doesn't HTML-escape the values of
    of constant string material, of attributes, and of material added
    by local macro `fmt`.

    This is a potential security issue, because if an untrusted value
    is interpolated, it can be a vector for an injection attack.
    The special variable `*cl-who-compat*` can be set true to disable the
    escaping, but is not recommended.

        ::text
        [1]> (in-package :cl-who)
        #<PACKAGE CL-WHO>
        WHO[2]> (with-html-output-to-string (out)
                  (:a :href "https://example.com'>malicious here</a><a href='blah" "click me"))
        "<a href='https://example.com'>malicious here</a><a href='blah'>click me</a>"

    Not escaping constant material is error-prone.

    The CL-WHO user has to remember to write `(:div "black&amp;white")`
    whereas the TL-WHO user just writes `(:div "black&white");` the
    `&amp;` escape is produced by TL-WHO.

*   TL-WHO provides a `noesc` syntax. When the value of an attribute is
    expressed as `(noesc <expr>)`, escaping is disabled:

        ::text
        (:a :href (noesc trusted-url) "click me")

    A more limited form of `noesc` can be used in tag bodies,
    to defeat the default escaping applied to constant material.

    The following:

        ::text
        (:div (noesc "&amp;") " &lt;")

    produces `<div>&amp; &amp;lt;</div>`.

*   TL-WHO provides a `noesc-fmt` which doesn't HTML-escape.

*   TL-WHO provides `escq` and `escj` local macros. `escq` is like `esc`
    but also HTML-escapes the ASCII apostrophe and double quote.
    `escj` escapes a string such that it can be safely interpolated into
    a Javascript string literal, which is itself embedded in a HTML
    `script` tag.

*   TL-WHO's `esc` accepts arguments which are not strings. They
    are turned to a string using `tostringp`, and HTML-escaped.

*   TL-WHO allows `fmt` as well as `escj` to be used to calculate attributes.
    These local macros have alternative definitions when used in attributes.
    The remaining local macros don't make sense in attributes, and make
    a mess of the output if used; unlike CL-WHO, TL-WHO warns when they
    are used in attributes.

*   The elaborate manual HTML escaping support featured in CL-WHO is
    absent in TL-WHO. These symbols do not exist: `escape-string`,
    `escape-char`, `*escape-char-p*`, `escape-string-minimal`,
    `escape-string-minimal`, `escape-string-minimal-plus-quotes`,
    `escape-string-iso-8859-1`, `escape-string-all`,
    `escape-char-minimal`, `escape-char-minimal-plus-quotes`,
    `escape-char-iso-8859-1` and `escape-char-all`.
    TL-WHO uses the TXR Lisp standard functions `html-encode`,
    and `html-encode*`, which have no options to control its behavior.
    `html-encode` is used internally for escaping material
    to be inserted into attributes; `html-encode*` is used by `esc`
    and `fmt`.

*   TL-WHO provides a `deftag` macro for defining macro-expanding tags.
    This is inspired by a `deftag` described in the
    [Spinneret](https://github.com/ruricolist/spinneret) documentation.
    Though ordinary macros can easily be used inside `with-html-output`,
    `deftag` is an expansion mechanism which transforms the tag
    markup, rather than embedded Lisp code. It is documented below.

*   The CL-WHO `conc` function is missing. TXR Lisp has a function
    like this, which is called `join`, and that is what is used in TL-WHO.

*   There is no generic function `convert-tag-to-string-list`.
    It is an ordinary function. There is no support for customizing
    its behavior. TXR Lisp doesn't have CLOS, and thus no generic functions
    or methods. A mechanism for customizing the conversion of tags to
    string lists may be adopted by TL-WHO in the future.

*   Numerous instances of internal unused code that in CL-WHO do
    not appear at all in TL-WHO; they have been pruned. Code depending
    on internal CL-WHO functions that are not present in TL-WHO
    will not work and must be ported in an alternative way.

*   TXR Lisp isn't Common Lisp and so CL-WHO examples which use ANSI CL
    idioms like the `loop` macro will not work. All Lisp code
    embedded in a CL-WHO expression has to be translated to TXR Lisp
    in order for that expression to work as TL-WHO; the TL-WHO
    library doesn't translate embedded Common Lisp to TXR Lisp;
    it only translates the markup syntax to HTML in the same way as CL-WHO.

*   The variable `*attribute-quote-char*` will not work if it is bound by
    `let` around a `with-html-output-form`. The reason is that TL-WHO
    interpolates this character at macro-expansion time, even for
    HTML attributes whose values are calculated at run-time.
    The binding construct `expander-let` must be used, or else the
    variable's global binding must be assigned.

*   CL-WHO has some bugs around attribute handling. When the value of an
    attribute is a constant expression, only the specific values `T`
    and `NIL` are treated properly, not constant expressions which evaluate
    to `T` and `NIL`. Then we see the mistaken attribute values `'NIL` and `'T'`:
    This works properly in TL-WHO:

        ::text
        [1]> (in-package :cl-who)
        #<PACKAGE CL-WHO>
        WHO[2]> (with-html-output-to-string (str)
                  (:foo :bar t))
        "<foo bar='bar'></foo>"
        WHO[3]> (with-html-output-to-string (str)
                  (:foo :bar (quote t)))
        "<foo bar='T'></foo>"
        WHO[4]> (with-html-output-to-string (str)
                  (:foo :bar nil))
        "<foo></foo>"
        WHO[5]> (with-html-output-to-string (str)
                  (:foo :bar 'nil))
        "<foo bar='NIL'></foo>"

Additionally, users (of both CL-WHO and TL-WHO alike) are advised to watch for the
following issue: the CL-WHO documentation is not accurately maintained and
makes some references to material that no longer exists in CL-WHO, such as the
macro `show-html-expansion`, which was removed from CL-WHO in 2009.

## The `deftag` macro

### Syntax:

    ::text
    (deftag <keyword> <attr-param-list> <body-pattern>
      <body> ...)

    <attr-param-list> ::= (<key-param>* [ . <rest-param>])

    <key-param> := <symbol> | (<name> [<default> [<pres-var>]])

    <body-pattern> := <macro-style-parameter-list>

### Description: 

A `deftag` macro rewrites HTML markup written in TL-WHO syntax
into other markup.

Simple example:

    ::text
    (deftag :boldface (. attrs) body
      ^(:span :font "bold" ,*attrs ,*body))

    (with-html-output-to-string (out)
      (:boldface :id "hello-id" "Hello!"))

    --> <span font='bold' id='hello-id'>Hello!</span>

Complex example, adapted from Spinneret documentation:

    ::text
    (deftag :easy-input (label (name (gensym))
                         (id name) (type "text") . other-attrs) default
      ^(progn
         (:label :for ,name ,label)
         (:input :name ,name :id ,id :type ,type
                 ,*other-attrs :value (progn ,*default))))

Note that `progn` here isn't the Lisp `progn` operator; it is recognized
by the `deftag` expansion mechanism as a way of producing multiple elements.
To invoke `:easy-input`, we might do this:

    ::text
    3> (with-html-output (*stdout* nil :indent t)
         (:div :class "cls"
           (:easy-input :name "foo" :id "foo-23"
                        :style "style" :label "lab" "123")))
    <div class='cls'>
      <label for='foo'>lab
      </label>
      <input name='foo' id='foo-23' type='text' style='style' value='123' />
    </div>

Note a small flexibility: the `body` argument `"123"` of `:easy-input` wasn't
inserted as an element in the middle of the tag as in the simple example, but
as an attribute value.

Note that the `other-attrs` rest parameter of `:easy-input` received
a list which only contained the `:style` attribute; all the others
were captured by their respective keyword parameters.

The following example shows the destructuring `<body-pattern>`,
which gives the `:atag` non-attribute parameters.

     ::text
     (deftag :atag (. other-attrs) (url cls . rest)
       ^(:a :href ,url :class ,cls ,*other-attrs ,*rest))

This `:atag` is then used like this:

     ::text
     3> (with-html-output (*stdout*)
          (:atag "https://example.com" "ext" "link text"))
     <a href='https://example.com' class='ext'>link text</a>

Here, the `"https://example.c"` item from the body got matched
as the `url` parameter, then `cls` took `"ext"` and `rest` got
the rest of the body as a list of items, which includes `"link text"`.

Other attributes tag can be passed through thanks to `other-attributes` parameter:

     ::text
     3> (with-html-output (*stdout*)
          (:atag :target "_blank" "https://example.com" "ext" "link text"))
     <a href='https://example.com' class='ext' target='_blank'>link text</a>

### Parameters:

The `attr-param-list` is an implicit keyword parameter list
which destructures attributes. It can specify default values for
attributes that are not passed. Its rest parameter is bound
to the remaining attributes that were not captured by the parameters.

Each `key-param` is an ordinary TXR Lisp key parameter, implemented
by the TXR Lisp `:key` parameter list macro. It may be just a
name, or a name with `default` expression giving a value for
the parameter if the corresponds keyword argument is missing,
possibly followed by another variable name which, if present,
will be a Boolean value indicating, if true, that the keyword
argument was present, or if false that it was missing (and
thus defaulted).

The `body-pattern` is a destructuring pattern (macro-style parameter list).
It captures the body that is passed to the tag: material after the attributes.
Usually, this is specified as a single symbol, which captures the
entire body. However, if a pattern is specified, the body material
is destructurd, which effectively allows `deftag` tags to
have parameters, as in the `:atag` example above.

### Returns:

The `deftag` macro returns the `<keyword>` parameter.

## Dependencies

TL-WHO has no external dependencies other than TXR itself.

It requires version 288 or newer.

## Installation/Use

To use TL-WHO, you just have to put the code somewhere and `load`
the `tl-who.tl` file (or its compiled version, if you compile the code).

    ::text
    (load "path/to/tl-who")

TXR Lisp's `load` is intelligent; the `tl-who` loader module easily loads the
needed files, which it finds in the same directory as itself.

To compile all the TL-WHO files:

    ::text
    (load "path/to/tl-who" :compile)

Cleaning compiled files away is similar; use the load argument `:clean`
instead of `:compile`.

TL-WHO symbols are in a package called `tl-who`, which is case sensitive.

Its private symbols are in `tl-who-priv`.

Two TL-WHO symbols have the same names as standard TXR library symbols,
namely `fmt` and `str`. Code which references the `tl-who` package in
preference to the `usr` package (for instance by putting `tl-who` first
in the package `:fallback` list) must use `usr:fmt` and `usr:str` to refer to
the TXR Lisp`fmt` functions, because the unprefixed tokens will produce
`tl-who:fmt` and `tl-who:str`, which are different symbols.

## License

TL-WHO is distributed under the two-clause BSD license.
[`LICENSE`](../tree/LICENSE) file.