From 5fc1472e9ce7ebf3d3d25898f0e6288ffe87d08e Mon Sep 17 00:00:00 2001 From: Kaz Kylheku Date: Sun, 23 Jun 2019 21:13:06 -0700 Subject: doc: move buffers out of dynamic lib section. * txr.1: Buffers are now documented after strings and vectors. Buffer streams are in the I/O section. Note: the diff between this and the previous commit needs to be generated with git diff --minimal. Otherwise it looks like over 30,000 lines of changes. --- txr.1 | 63318 ++++++++++++++++++++++++++++++++-------------------------------- 1 file changed, 31660 insertions(+), 31658 deletions(-) diff --git a/txr.1 b/txr.1 index 5700febe..d36b5235 100644 --- a/txr.1 +++ b/txr.1 @@ -23551,37396 +23551,37398 @@ It returns a single large vector formed by catenating those vectors together in order. -.SS* Structures +.SS* Buffers -\*(TX supports a structure data type. Structures are objects which -hold multiple storage locations called slots, which are named by symbols. -Structures can be related to each other by inheritance. +.coSS The @ buf type -The type of a structure is itself an object, of type -.codn struct-type . +Object of the type +.code buf +are +.IR buffers : +vector-like objects specialized for holding binary data represented as +a sequence of 8 bit bytes. Buffers support operations specialized toward the +encoding of Lisp values into machine-oriented data types, and decoding such +data types into Lisp values. -When the program defines a new structure type, it does so by creating a new -.code struct-type -instance, with properties which describe the new structure type: its -name, its list of slots, its initialization and "boa constructor" functions, -and the structure type it inherits from (the "super"). +Buffers are particularly useful in conjunction with the Foreign Function +Interface (FFI), since they can be used to prepare arbitrary data which +can be passed into and out of a function by pointer. They are also useful for +binary I/O. -The -.code struct-type -object is then used to generate instances. +.coNP Conventions Used by the @ buf-put- Functions -Structures instances are not only containers which hold named slots, but they -also indicate their struct type. Two structures which have the same number of -slots having the same names are not necessarily of the same type. +Buffers support a number of similar functions for converting Lisp numeric +values into common data types, which are placed into the buffer. These +functions are named starting with the +.code buf-put- +prefix, followed by an abbreviated type name. -Structure types and structures may be created and manipulated using -a programming interface based on functions. +Each of these functions takes three arguments: +.meta buf +specifies the buffer, +.meta pos +specifies the byte offset position into the buffer which receives +the low-order byte of the data transfer, and +.meta val +indicates the value. -For more convenient and clutter-free expression of structure-based -program code, macros are also provided. +If +.meta pos +has a value such that any portion of the data transfer would +like outside of the buffer, the buffer is automatically extended +in length to contain the data transfer. If this extension causes +any padding bytes to appear between the previous length of the +buffer and +.metn pos , +those bytes are initialized to zero. -Furthermore, concise and expressive slot access syntax is provided courtesy of -the referencing dot and unbound referencing dot syntax, a syntactic sugar -for the -.code qref +The argument +.meta val +giving the value to be stored must be an integer or character, +except in the case of the types +.meta float and -.code uref -macros. +.metn double (the +functions +.code buf-put-float +and +.codn buf-put-double ) +for which it is required to be of type +.codn float , +and in case of the function +.code buf-put-cptr +which expects the +.meta val +argument to be a +.code cptr +object. -Structure types have a name, which is a symbol. The -.code typeof -function, when applied to any struct type, returns the symbol -.codn struct-type . -When -.code typeof -is applied to a struct instance, it returns the name of -the struct type. Effectively, struct names are types. +The +.meta val +argument must be in range for the data type, or an exception +results. -The consequences are unspecified if an existing struct name is re-used for a -different struct type, or an existing type name is used for a struct type. +Unless otherwise indicated, the stored datum is in the local format +used by the machine with regard to byte order and other representational +details. -.NP* Static Slots +.coNP Conventions Used by the @ buf-get- Functions -Structure slots can be of two kinds: they can be the ordinary instance slots or -they can be static slots. The instances of a given structure type have their -own instance of a given instance slot. However, they all share a single -instance of a static slot. +Buffers support a number of similar functions for extracting +common data types, and converting them into Lisp values. +These functions are named starting with the +.code buf-get- +prefix, followed by an abbreviated type name. -Static slots are allocated in a global area associated with a structure type -and are initialized when the structure type is created. They are useful for -efficiently representing properties which have the same value for all instances -of a struct. These properties don't have to occupy space in each instance, and -time doesn't have to be wasted initializing them each time a new instance is -created. Static slots are also useful for struct-specific global variables. -Lastly, static slots are also useful for holding methods and functions. -Although structures can have methods and functions in their instances, usually, -all structures of the same type share the same functions. The -.code defstruct -macro supports a special syntax for defining methods and struct-specific -functions at the same time when a new structure type is defined. +Each of these functions takes two arguments: +.meta buf +specifies the buffer and +.meta pos +specifies the byte offset position into the buffer which holds +the low-order byte of the datum to be extracted. + +If any portion of requested datum lies outside of the boundaries +of the buffer, an error exception is thrown. + +The extracted value is converted to a Lisp datum. For the +majority of these functions, the returned value is of type +integer. The +.code buf-get-float +and +.code buf-get-double +return a floating-point value. The -.code defmeth -macro can be used for adding new methods and functions to an existing -structure and its descendants. +.code buf-get-cptr +function returns a value of type +.codn cptr . -Static slots may be assigned just like instance slots. Changing a static -slot, of course, changes that slot in every structure of the same type. +.coNP Function @ make-buf +.synb +.mets (make-buf < len >> [ init-val <> [ alloc-size ]]) +.syne +.desc +The +.code make-buf +function creates a new buffer object which holds +.meta len +bytes. This argument may be zero. -Static slots are not listed in the -.code #S(...) -notation when a structure is printed. When the structure notation is -read from a stream, if static slots are present, they will be processed -and their values stored in the static locations they represent, thus -changing their values for all instances. +If +.meta init-val +is present, it specifies the value with which every +byte of the buffer is initialized. If omitted, it +defaults to zero. +bytes. The value of +.meta init-val +must lie in the range 0 to 255. -Static slots are inherited just like instance slots. If a given structure -.meta B -has some static slot -.metn s , -and a new structure -.meta D -is derived from -.metn B , -using -.codn defstruct , -and does not define a slot -.metn s , -then -.meta D -inherits -.metn s . -This means that -.meta D -shares the static slot with -.metn B : -both types share a single instance of that slot. +The +.meta alloc-size +parameter indicates how much memory to actually allocate for +the buffer. If an argument is specified, its value must not +be less than +.metn len . -On the other hand if -.code D -defines a static slot -.meta s -then that slot will have its own instance in the -.meta D -structure type; -.meta D -will not inherit the -.meta B -instance of slot -.metn s . -Moreover, if the the definition of -.code D -omits the -.meta init-form -for slot -.metn s , -then that slot will be initialized with a copy of the current value of slot -.meta s -of the -.meta B -base type, which allows derived types to obtain the value of base type's -static slot, yet have that in their own instance. +.coNP Function @ bufp +.synb +.mets (bufp << object ) +.syne +.desc +The +.code bufp +function returns +.code t +if +.meta object +is a +.codn buf , +otherwise it returns +.codn nil . -The slot type can be overridden. A structure type deriving from another -type can introduce slots which have the same names as the supertype, -but are of a different kind: an instance slot in the supertype -can be replaced by a static slot in the derived type or vice versa. +.coNP Function @ length-buf +.synb +.mets (length-buf << buf ) +.syne +.desc +The +.code length-buf +function retrieves the buffer length: how many bytes are stored +in the buffer. -A structure type is associated with a static initialization function -which may be used to store initial values into static slots. This function -is invoked once in a type's life time, when the type is created. -The function is also inherited by derived struct types and invoked when -they are created. +Note: the generic +.code length +function is also applicable to buffers. -.NP* Dirty Flags -All structure instances contain a Boolean flag called the -.IR "dirty flag" . -This flag is not a slot, but rather a meta-data property that is exposed -to program access. When the flag is set, an object is said to be dirty; -otherwise it is clean. +.coNP Function @ buf-alloc-size +.synb +.mets (buf-alloc-size << buf ) +.syne +.desc +The +.code buf-alloc-size +function retrieves the allocation size of the buffer. -Newly constructed objects come into existence dirty. The dirty flag -state can be tested with the function -.codn test-dirty . -An object can be marked as clean by clearing its dirty flag with -.codn clear-dirty . -A combined operation -.code test-clear-dirty -is provided which clears the dirty flag, and -returns its previous value. +.coNP Function @ buf-trim +.synb +.mets (buf-trim << buf ) +.syne +.desc +The +.code buf-trim +function reduces the amount of memory allocated to the buffer +to the minimum required to hold it contents, effectively +setting the allocation size to the current length. -The dirty flag is set whenever a new value is stored into the instance -slot of an object. +The previous allocation size is returned. -Note: the dirty flag can be used to support support the caching of values -derived from an object's slots. The derived values don't have to be -re-computed while an object remains clean. +.coNP Function @ buf-set-length +.synb +.mets (buf-set-length < buf < len <> [ init-val ]) +.syne +.desc +The +.code buf-set-length +function changes the length of the buffer. If the buffer +is made longer, the newly added bytes appear at the end, +and are initialized to the value given by +.metn init-val . +If +.meta init-val +is specified, its value must be in the range 0 to 255. +It defaults to zero. -.NP* Equality Substitution +.coNP Function @ copy-buf +.synb +.mets (copy-buf << buf ) +.syne +.desc +The +.code copy-buf +function returns a duplicate of +.metn buf : +an object distinct from +.meta buf +which has the same length and contents, and compares +.code equal +to +.metn buf . -In object-based or object-oriented programming, sometimes it is necessary for a -new data type to provide its own notion of equality: its own requirements for -when two distinct instances of the type are considered equal. Furthermore, -types sometimes have to implement their own notion, also, of inequality: the -requirements for the manner in which one instance is considered lesser or -greater than another. +.coNP Accessor @ sub-buf +.synb +.mets (sub-buf < buf >> [ from <> [ to ]]) +.mets (set (sub-buf < buf >> [ from <> [ to ]]) << new-val ) +.syne +.desc +The +.code sub-buf +function has the same semantics as the +.code sub +function, except that the first argument must be a buffer. -\*(TL structures implement a concept called -.IR "equality substitution" -which provides a simple, unified way for the implementor of an object to -encode the requirements for both equality and inequality. -Equality substitution allows for objects to be used as keys in a hash -table according to the custom equality, without the programmer being -burdened with the responsibility of developing a custom hashing function. +The extracted sub-range of a buffer is itself a buffer object. -An object participates in equality substitution by implementing the -.code equal -method. The -.code equal -method takes no arguments other than the object itself. It returns -a representative value which is used in place of that object -for the purposes of -.code equal -comparison. +If +.code sub-buf +is used as a syntactic place, the argument expressions +.metn buf , +.metn from , +.meta to +and +.meta new-val +are evaluated just once. The prior value, if required, is accessed by calling +.code buf-sub +and +.meta new-val +is then stored via +.codn replace-buf . -Whenever an object which supports equality substitution is used as an argument -of any of the functions -.codn equal , -.codn nequal , -.codn greater , -.codn less , -.codn gequal , -.code lequal -or -.codn hash-equal , -the -.code equal -method of that object is invoked, and the return value of that method -is taken in place of that object. +.coNP Function @ replace-buf +.synb +.mets (replace-buf < buf < item-sequence >> [ from <> [ to ]]) +.syne +.desc +The +.code replace-buf +function has the same semantics as the +.code replace +function, except that the first argument must be a buffer. -The same is true if an object which supports equality substitution is used as a -key in an -.code :equal-based -hash table. +The elements of +.code item-sequence +are stored into +.meta buf +as if using the +.code buf-put-u8 +function and therefore must be suitable +.meta val +arguments for that function. -The substitution is applied repeatedly: if the return value of the object's -.code equal -method is an object which itself supports equality substitution, -than that returned object's method is invoked on that object -to fetch its equality substitute. This repeats as many times as necessary -until an object is determined which isn't a structure that supports -equality substitution. +The of the arguments, semantics and return value given for +.code replace +apply to +.codn replace-buf . -Once the equality substitute is determined, then the given function proceeds -with the replacement object. Thus for example -.code equal -compares the replacement object in place of the original, and an -.code :equal-based -hash table uses the replacement object as the key for the purposes of -hashing and comparison. +.coNP Function @ buf-put-i8 +.synb +.mets (buf-put-i8 < buf < pos << val ) +.syne +.desc +The +.code buf-put-i8 +converts +.meta val +into an eight bit signed integer, and stores it into the buffer at +the offset indicated by +.metn pos . -.coNP Macro @ defstruct +The return value is +.metn val . + +.coNP Function @ buf-put-u8 .synb -.mets (defstruct >> { name | >> ( name << arg *)} < super -.mets \ \ << slot-specifier *) +.mets (buf-put-u8 < buf < pos << val ) .syne .desc The -.code defstruct -macro defines a new structure type and registers it under -.metn name , -which must be a bindable symbol, according to the -.code bindable -function. Likewise, the name of every -.meta slot -must also be a bindable symbol. +.code buf-put-u8 +converts +.meta val +into an eight bit unsigned integer, and stores it into the buffer at +the offset indicated by +.metn pos . + +The return value is +.metn val . +.coNP Function @ buf-put-i16 +.synb +.mets (buf-put-i16 < buf < pos << val ) +.syne +.desc The -.meta super -argument must either be -.code nil -or a symbol which names an existing struct type. -The newly defined struct type will inherit all slots, -as well as initialization behaviors from this type. +.code buf-put-i16 +converts +.meta val +into a sixteen bit signed integer, and stores it into the buffer at +the offset indicated by +.metn pos . + +The return value is +.metn val . +.coNP Function @ buf-put-u16 +.synb +.mets (buf-put-u16 < buf < pos << val ) +.syne +.desc The -.code defstruct -macro is implemented using the -.code make-struct-type -function, which is more general. The macro analyzes the -.code defstruct -argument syntax, and synthesizes arguments which are then -used to call the function. Some remarks in the description of -.code defstruct -only apply to structure types defined using that macro. +.code buf-put-u16 +converts +.meta val +into a sixteen bit unsigned integer, and stores it into the buffer at +the offset indicated by +.metn pos . -Slots are specified using zero or more -.IR "slot specifiers" . -Slot specifiers come in the following variety: -.RS -.meIP < name -The simplest slot specifier is just a name, which must be a bindable -symbol, as defined by the -.code bindable -function. This form is a short form for the -.mono -.meti (:instance << name ) -.onom -syntax. -.meIP >> ( name << init-form ) -This syntax is a short form for the -.mono -.meti (:instance < name << init-form ) -.onom -syntax. -.meIP (:instance < name <> [ init-form ]) -This syntax specifies an instance slot called -.meta name -whose initial value is obtained by evaluating -.meta init-form -whenever a new instance of the structure is created. -This evaluation takes place in the original lexical environment in which the -.code defstruct -form occurs. If -.meta init-form -is omitted, the slot is initialized to -.codn nil . -.meIP (:static < name <> [ init-form ]) -This syntax specifies a static slot called -.meta name -whose initial value is obtained by evaluating -.meta init-form -once, during the evaluation of the -.code defstruct -form in which it occurs, if the -.meta init-form -is present. If -.meta init-form -is absent, and a static slot with the same name -exists in the -.meta super -base type, then this slot is initialized -with the value of that slot. -Otherwise it is initialized to -.codn nil . +The return value is +.metn val . -The definition of a static slot in a -.code defstruct -causes the new type to have its own instance -that slot, even if a same-named static -slot occurs in the -.meta super -base type, or its bases. -.meIP (:method < name <> ( param +) << body-form *) -This syntax creates a static slot called -.meta name -which is initialized with an anonymous function. -The anonymous function is created during the -evaluation of the -.code defstruct -form. The function takes the arguments specified -by the -.meta param -symbols, and its body consists of the -.metn body-form -s. -There must be at least one -.metn param . -When the function is invoked as a method, as intended, -the leftmost -.meta param -receives the structure instance. +.coNP Function @ buf-put-i32 +.synb +.mets (buf-put-i32 < buf < pos << val ) +.syne +.desc The -.metn body-form -s -are evaluated in a context in which a block named -.meta name -is visible. Consequently, -.code return-from -may be used to terminate the execution of a method -and return a value. -Methods are invoked -using the -.code "instance.(name arg ...)" -syntax, which implicitly inserts the instance into the argument list. +.code buf-put-i32 +converts +.meta val +into a 32 bit signed integer, and stores it into the buffer at +the offset indicated by +.metn pos . -Due to the semantics of static slots, methods are naturally -inherited from a base structure to a derived one, -and defining a method in a derived class which also exists -in a base class performs OOP-style overriding. -.meIP (:function < name <> ( param *) << body-form *) -This syntax creates a static slot called -.meta name -which is initialized with an anonymous function. -The anonymous function is created during the -evaluation of the -.code defstruct -form. The function takes the arguments specified -by the -.meta param -symbols, and its body consists of the -.metn body-form -s. -This specifier differs from -.code :method -only in one respect: there may be zero -parameters. A structure function defined this way is -intended to be used as a utility function which doesn't -receive the structure instance as an argument. +The return value is +.metn val . + +.coNP Function @ buf-put-u32 +.synb +.mets (buf-put-u32 < buf < pos << val ) +.syne +.desc The -.metn body-form -s -are evaluated in a context in which a block named -.meta name -is visible. Consequently, -.code return-from -may be used to terminate the execution of the function -and return a value. -Such functions are called using the -.code "instance.[name arg ...]" -syntax which doesn't insert the instance into -the argument list. +.code buf-put-u32 +converts +.meta val +into a 32 bit unsigned integer, and stores it into the buffer at +the offset indicated by +.metn pos . -The remarks about inheritance and overriding -in the description of -.code :method -also apply to -.codn :function . -.meIP (:init <> ( param ) << body-form *) +The return value is +.metn val . + +.coNP Function @ buf-put-i64 +.synb +.mets (buf-put-i64 < buf < pos << val ) +.syne +.desc The -.code :init -specifier doesn't describe a slot. Rather, it specifies code -which is executed when a structure is instantiated, after -the slot initializations specific to the structure type -are performed. The code consists of -.metn body-form -s -which are evaluated in order in a lexical scope in -which the variable -.meta param -is bound to the structure object. +.code buf-put-i64 +converts +.meta val +into a 64 bit signed integer, and stores it into the buffer at +the offset indicated by +.metn pos . + +The return value is +.metn val . +.coNP Function @ buf-put-u64 +.synb +.mets (buf-put-u64 < buf < pos << val ) +.syne +.desc The -.code :init -specifier may not appear more than once in a given -.code defstruct -form. +.code buf-put-u64 +converts the value +.meta val +into a 64 bit unsigned integer, and stores it into the buffer at +the offset indicated by +.metn pos . -When an object with one or more levels of inheritance -is instantiated, the -.code :init -code of a base structure type, if any, is executed -before any initializations specific to a derived -structure type. +The return value is +.metn val . +.coNP Function @ buf-put-char +.synb +.mets (buf-put-char < buf < pos << val ) +.syne +.desc The -.code :init -initializations are executed before any other -slot initializations. The argument values passed to the -.code new -or -.code lnew -operator or the -.code make-struct -function are not yet stored in the object's slots, -and are not accessible. Initialization code which needs -these values to be stable can be defined with -.codn :postinit . +.code buf-put-char +converts +.meta val +into a value of the C type +.code char +and stores it into the buffer at +the offset indicated by +.metn pos . -Initializers in base structures must be careful about assumptions about slot -kinds, because derived structures can alter static slots to instance slots or -vice versa. To avoid an unwanted initialization being applied to the -wrong kind of slot, initialization code can be made conditional on the -outcome of -.code static-slot-p -applied to the slot. -(Code generated by -.code defstruct -for initializing instance slots performs this kind of check). +The return value is +.metn val . + +Note that the +.code char +type may be signed or unsigned. +.coNP Function @ buf-put-uchar +.synb +.mets (buf-put-uchar < buf < pos << val ) +.syne +.desc The -.metn body-form -s -of an -.code :init -specifier are not surrounded by an implicit -.codn block . -.meIP (:postinit <> ( param ) << body-form *) +.code buf-put-uchar +converts +.meta val +into a value of the C type +.code "unsigned char" +and stores it into the buffer at +the offset indicated by +.metn pos . + +.coNP Function @ buf-put-short +.synb +.mets (buf-put-short < buf < pos << val ) +.syne +.desc The -.code :postinit -specifier is very similar to -.codn :init . -Both specify forms which are evaluated during object instantiation. -The difference is that the -.codn body-form -s -of a -.code :postinit -are evaluated after other initializations have taken -place, including the -.code :init -initializations, as a second pass. By the time -.code :postinit -initialization runs, the argument material from the -.codn make-struct , -.code new -or -.code lnew -invocation has already been processed and stored -into slots. -Like -.code :init -actions, -.code :postinit -actions registered at different levels of the type's -inheritance hierarchy are invoked in the base-to-derived -order. -.meIP (:fini <> ( param ) << body-form *) +.code buf-put-short +converts +.meta val +into a value of the C type +.code short +and stores it into the buffer at +the offset indicated by +.metn pos . + +.coNP Function @ buf-put-ushort +.synb +.mets (buf-put-ushort < buf < pos << val ) +.syne +.desc The -.code :fini -specifier doesn't describe a slot. Rather, it specifies -a finalization function which is associated with the -structure instance, as if by use of the -.code finalize -function. This finalization registration takes place -as the first step when an instance of the structure -is created, before the slots are initialized and -the -.code :init -code, if any, has been executed. The registration -takes place as if by the evaluation of the form -.mono -.meti (finalize < obj (lambda <> ( param ) << body-form ...) t) -.onom -where -.meta obj -denotes the structure instance. Note the -.code t -argument which requests reverse order of registration, ensuring that if an -object has multiple finalizers registered at different levels of inheritance -hierarchy, the finalizers specified for a derived structure type are called -before inherited finalizers. +.code buf-put-ushort +converts +.meta val +into a value of the C type +.code "unsigned short" +and stores it into the buffer at +the offset indicated by +.metn pos . +.coNP Function @ buf-put-int +.synb +.mets (buf-put-int < buf < pos << val ) +.syne +.desc The -.metn body-form -s -of a -.code :fini -specifier are not surrounded by an implicit -.codn block . +.code buf-put-int +converts +.meta val +into a value of the C type +.code int +and stores it into the buffer at +the offset indicated by +.metn pos . -Note that an object's finalizers can be called explicitly with -.codn call-finalizers . -.RE -.IP +.coNP Function @ buf-put-uint +.synb +.mets (buf-put-uint < buf < pos << val ) +.syne +.desc The -.code with-objects -macro arranges for finalizers to be called on objects when the execution -of a scope terminates by any means. +.code buf-put-uint +converts +.meta val +into a value of the C type +.code "unsigned int" +and stores it into the buffer at +the offset indicated by +.metn pos . -The slot names given in a -.code defstruct -must all be unique among themselves, but they -may match the names of existing slots in the -.meta super -base type. +.coNP Function @ buf-put-long +.synb +.mets (buf-put-long < buf < pos << val ) +.syne +.desc +The +.code buf-put-long +converts +.meta val +into a value of the C type +.code long +and stores it into the buffer at +the offset indicated by +.metn pos . -A given structure type can have only one slot under a given -symbolic name. If a newly specified slot matches the name of an existing slot -in the -.meta super -type or that type's chain of ancestors, it is called a -.IR "repeated slot" . +.coNP Function @ buf-put-ulong +.synb +.mets (buf-put-ulong < buf < pos << val ) +.syne +.desc +The +.code buf-put-ulong +converts +.meta val +into a value of the C type +.code "unsigned long" +and stores it into the buffer at +the offset indicated by +.metn pos . -The kind of the repeated slot (static or instance) is not inherited; it -is established by the -.code defstruct -and may be different from the type of the same-named slot in the -supertype or its ancestors. +.coNP Function @ buf-put-float +.synb +.mets (buf-put-float < buf < pos << val ) +.syne +.desc +The +.code buf-put-float +converts +.meta val +into a value of the C type +.code float +and stores it into the buffer at +the offset indicated by +.metn pos . -If a repeated slot is introduced as a static slot, and -has no -.meta init-form -then it receives the current of the a static of the same name from -the nearest supertype which has such a slot. +Note: the conversion of a \*(TL floating-point value +to the C type float may be inexact, reducing the +numeric precision. -If a repeated slot is an instance slot, no such inheritance of value -takes place; only the local -.meta init-form -applies to it; if it is absent, the slot it initialized to -.code nil -in each newly created instance of the new type. +.coNP Function @ buf-put-double +.synb +.mets (buf-put-double < buf < pos << val ) +.syne +.desc +The +.code buf-put-double +converts +.meta val +into a value of the C type +.code double +and stores it into the buffer at +the offset indicated by +.metn pos . -However, -.code :init -and -.code :postinit -initializations are inherited from a base type and they apply to -the repeated slots, regardless of their kind. These initializations -take place on the instantiated object, and the slot references -resolve accordingly. +.coNP Function @ buf-put-cptr +.synb +.mets (buf-put-cptr < buf < pos << val ) +.syne +.desc +The +.code buf-put-cptr +expects +.meta val +to be of type +.codn cptr . +It stores the object's pointer value into the buffer at +the offset indicated by +.metn pos . -The initialization for slots which are specified using the -.code :method -or -.code :function -specifiers is re-ordered with regard to -.code :static -slots. Regardless of their placement in the -.code defstruct -form, -.code :method -and -.code :function -slots are initialized before -.code :static -slots. This ordering is useful, because it means that when the initialization -expression for a given static slot constructs an instance of the struct type, -any instance initialization code executing for that instance can use -all functions and methods of the struct type. -However, note the static slots which follow that slot in the -.code defstruct -syntax are not yet initialized. If it is necessary for a structure's -initialization code to have access to all static slots, even when the -structure is instantiated during the initialization of a static slot, -a possible solution may be to use lazy instantiation using the -.code lnew -operator, rather than ordinary eager instantiation via -.codn new . -It is also necessary to ensure that that the instance isn't accessed until all -static initializations are complete, since access to the instance slots of a -lazily instantiated structure triggers its initialization. - -The structure name is specified using two forms, plain -.meta name -or the syntax -.mono -.meti >> ( name << arg *) -.onom -If the second form is used, then the structure type will support -"boa construction", where "boa" stands for "by order of arguments". +.coNP Function @ buf-get-i8 +.synb +.mets (buf-get-i8 < buf << pos ) +.syne +.desc The -.metn arg -s -specify the list of slot names which are to be initialized in the -by-order-of-arguments style. For instance, if three slot names -are given, then those slots can be optionally initialized by giving three -arguments in the -.code new -macro or the -.code make-struct -function. - -Slots are first initialized according to their -.metn init-form -s, -regardless of whether they are involved in boa construction - -A slot initialized in this style still has a -.meta init-form -which is processed independently of the existence of, and prior to, -boa construction. - -The boa constructor syntax can specify optional parameters, delimited -by a colon, similarly to the -.code lambda -syntax. However, the optional parameters may not be arbitrary symbols; -they must be symbols which name -slots. Moreover, the -.mono -.meti >> ( name < init-form <> [ present-p ]) -.onom -optional parameter syntax isn't supported. - -When boa construction is invoked with optional arguments missing, -the default values for those arguments come from the -.metn init-form -s -in the remaining -.code defstruct -syntax. - -.TP* Examples: -.verb - (defvar *counter* 0) +.code buf-get-i8 +function extracts and returns signed eight bit integer from +.meta buf +at the offset given by +.metn pos . - ;; New struct type foo with no super type: - ;; Slots a and b initialize to nil. - ;; Slot c is initialized by value of (inc *counter*). - (defstruct foo nil (a b (c (inc *counter*)))) +.coNP Function @ buf-get-u8 +.synb +.mets (buf-get-u8 < buf << pos ) +.syne +.desc +The +.code buf-get-u8 +function extracts and returns an unsigned eight bit integer from +.meta buf +at the offset given by +.metn pos . - (new foo) -> #S(foo a nil b nil c 1) - (new foo) -> #S(foo a nil b nil c 2) +.coNP Function @ buf-get-i16 +.synb +.mets (buf-get-i16 < buf << pos ) +.syne +.desc +The +.code buf-get-i16 +function extracts and returns a signed 16 bit integer from +.meta buf +at the offset given by +.metn pos . - ;; New struct bar inheriting from foo. - (defstruct bar foo (c 0) (d 100)) +.coNP Function @ buf-get-u16 +.synb +.mets (buf-get-u16 < buf << pos ) +.syne +.desc +The +.code buf-get-u16 +function extracts and returns an unsigned 16 bit integer from +.meta buf +at the offset given by +.metn pos . - (new bar) -> #S(bar a nil b nil c 0 d 100) - (new bar) -> #S(bar a nil b nil c 0 d 100) +.coNP Function @ buf-get-i32 +.synb +.mets (buf-get-i32 < buf << pos ) +.syne +.desc +The +.code buf-get-i32 +function extracts and returns a signed 32 bit integer from +.meta buf +at the offset given by +.metn pos . - ;; counter was still incremented during - ;; construction of d: - *counter* -> 4 +.coNP Function @ buf-get-u32 +.synb +.mets (buf-get-u32 < buf << pos ) +.syne +.desc +The +.code buf-get-u32 +function extracts and returns an unsigned 32 bit integer from +.meta buf +at the offset given by +.metn pos . - ;; override slots with new arguments - (new foo a "str" c 17) -> #S(foo a "str" b nil c 17) +.coNP Function @ buf-get-i64 +.synb +.mets (buf-get-i64 < buf << pos ) +.syne +.desc +The +.code buf-get-i64 +function extracts and returns a signed 64 bit integer from +.meta buf +at the offset given by +.metn pos . - *counter* -> 5 +.coNP Function @ buf-get-u64 +.synb +.mets (buf-get-u64 < buf << pos ) +.syne +.desc +The +.code buf-get-u64 +function extracts and returns an unsigned 64 bit integer from +.meta buf +at the offset given by +.metn pos . - ;; boa initialization - (defstruct (point x : y) nil (x 0) (y 0)) +.coNP Function @ buf-get-char +.synb +.mets (buf-get-char < buf << pos ) +.syne +.desc +The +.code buf-get-char +function extracts and returns a value of the C type +.code char +from +.meta buf +at the offset given by +.metn pos . +Note that +.code char +may be signed or unsigned. - (new point) -> #S(point x 0 y 0) - (new (point 1 1)) -> #S(point x 1 y 1) +.coNP Function @ buf-get-uchar +.synb +.mets (buf-get-uchar < buf << pos ) +.syne +.desc +The +.code buf-get-uchar +function extracts and returns a value of the C type +.code "unsigned char" +from +.meta buf +at the offset given by +.metn pos . - ;; property list style initialization - ;; can always be used: - (new point x 4 y 5) -> #S(point x 4 y 5) +.coNP Function @ buf-get-short +.synb +.mets (buf-get-short < buf << pos ) +.syne +.desc +The +.code buf-get-short +function extracts and returns a value of the C type +.code short +from +.meta buf +at the offset given by +.metn pos . - ;; boa applies last: - (new (point 1 1) x 4 y 5) -> #S(point x 1 y 1) +.coNP Function @ buf-get-ushort +.synb +.mets (buf-get-ushort < buf << pos ) +.syne +.desc +The +.code buf-get-ushort +function extracts and returns a value of the C type +.code "unsigned short" +from +.meta buf +at the offset given by +.metn pos . - ;; boa with optional argument omitted: - (new (point 1)) -> #S(point x 1 y 0) +.coNP Function @ buf-get-int +.synb +.mets (buf-get-int < buf << pos ) +.syne +.desc +The +.code buf-get-int +function extracts and returns a value of the C type +.code int +from +.meta buf +at the offset given by +.metn pos . - ;; boa with optional argument omitted and - ;; with property list style initialization: - (new (point 1) x 5 y 5) -> #S(point x 1 y 5) -.brev +.coNP Function @ buf-get-uint +.synb +.mets (buf-get-uint < buf << pos ) +.syne +.desc +The +.code buf-get-uint +function extracts and returns a value of the C type +.code "unsigned int" +from +.meta buf +at the offset given by +.metn pos . -.coNP Macro @ defmeth +.coNP Function @ buf-get-long .synb -.mets (defmeth < type-name < name < param-list << body-form *) +.mets (buf-get-long < buf << pos ) .syne .desc -Unless -.meta name -is one of the two symbols -.code :init -or -.codn :postinit , -the -.code defmeth -macro installs a function into the static slot named by the symbol -.meta name -in the struct type indicated by -.metn type-name . +The +.code buf-get-long +function extracts and returns a value of the C type +.code long +from +.meta buf +at the offset given by +.metn pos . -If the structure type doesn't already have such a static slot, it is -first added, as if by the -.code static-slot-ensure -function, subject to the same checks. +.coNP Function @ buf-get-ulong +.synb +.mets (buf-get-ulong < buf << pos ) +.syne +.desc +The +.code buf-get-ulong +function extracts and returns a value of the C type +.code "unsigned long" +from +.meta buf +at the offset given by +.metn pos . -If the function has at least one argument, it can be used as a method. In that -situation, the leftmost argument passes the structure instance on which the -method is being invoked. +.coNP Function @ buf-get-float +.synb +.mets (buf-get-float < buf << pos ) +.syne +.desc +The +.code buf-get-float +function extracts and returns a value of the C type +.code float +from +.meta buf +at the offset given by +.metn pos , +returning that value as a Lisp floating-point number. -The function takes the arguments specified -by the -.meta param-list -symbols, and its body consists of the -.metn body-form -s. +.coNP Function @ buf-get-double +.synb +.mets (buf-get-double < buf << pos ) +.syne +.desc +The +.code buf-get-double +function extracts and returns a value of the C type +.code double +from +.meta buf +at the offset given by +.metn pos , +returning that value as a Lisp floating-point number. +.coNP Function @ buf-get-cptr +.synb +.mets (buf-get-cptr < buf << pos ) +.syne +.desc The -.metn body-form -s -are placed into a -.code block -named -.codn name . +.code buf-get-cptr +function extracts a C pointer from +.meta buf +at the offset given by +.metn pos , +returning that value as a Lisp object of type +.codn cnum . -A method named -.code lambda -allows a structure to be used as if it were a function. When arguments -are applied to the structure as if it were a function, the -.code lambda -method is invoked with those arguments, with the object itself inserted -into the leftmost argument position. +.coNP Function @ put-buf +.synb +.mets (put-buf < buf >> [ pos <> [ stream ]]) +.syne +.desc +The +.code put-buf +function writes the contents of buffer +.metn buf , +starting at position +.meta pos +to a stream, through to the last byte, if possible. +Successive bytes from the buffer are written to the stream as if by a +.code put-byte +operation. If -.code defmeth -is used to redefine an existing method, the semantics can be inferred -from that of -.codn static-slot-ensure . -In particular, the method will be imposed into all subtypes which inherit -(do not override) the method. +.meta stream +is omitted, it defaults to +.codn *stdout* . If -.meta name -is the keyword symbol -.codn :init , -then instead of operating on a static slot, the macro redefines the -.meta initfun -of the given structure type, as if by a call to the function -.codn struct-set-initfun . - -Similarly, if -.meta name -is the keyword symbol -.codn :postinit , -then the macro redefines the -.meta postinitfun -of the given structure type, as if by a call to the function -.codn struct-set-postinitfun . +.meta pos +is omitted, it defaults to zero. -When redefining -.code :initfun -the admonishments given in the description of -.code struct-set-initfun -apply: if the type has an -.meta initfun -generated by the -.code defstruct -macro, then that -.meta initfun -is what implements all of the slot initializations given in the -slot specifier syntax. These initializations are lost if the -.meta initfun -is overwritten. +The stream must support the +.code put-byte +operation. Streams which support +.code put-byte +can be expected to support +.code put-buf +and, conversely, streams which do not support +.code put-byte +do not support +.codn put-buf . The -.code defmeth -macro returns a method name: a unit of syntax of the form -.mono -.meti (meth < type-name << name) -.onom -which can be used as an argument to the accessor -.code symbol-function -and other situations. +.code put-buf +function returns the position of the last byte that was successfully written. +If the buffer was written through to the end, then this value corresponds to +the length of the buffer. -.coNP Macros @ new and @ lnew +If an error occurs before any bytes are written, the function +throws an error. + +.coNP Functions @ fill-buf and @ fill-buf-adjust .synb -.mets (new >> { name | >> ( name << arg *)} >> { slot << init-form }*) -.mets (lnew >> { name | >> ( name << arg *)} >> { slot << init-form }*) +.mets (fill-buf < buf >> [ pos <> [ stream ]]) +.mets (fill-buf-adjust < buf >> [ pos <> [ stream ]]) .syne .desc The -.code new -macro creates a new instance of the structure type named by -.metn name . +.code fill-buf +reads bytes from +.meta stream +and writes them into consecutive locations in buffer +.meta buf +starting at position +.metn pos . +The bytes are read as if using the +.code get-byte +function. -If the structure supports "boa construction", then, optionally, the -arguments may be given using the syntax -.mono -.meti >> ( name << arg *) -.onom -instead of -.metn name . +If the +.meta stream +argument is omitted, it defaults to +.codn *stdin* . -Slot values may also be specified by the -.meta slot -and -.meta init-form -arguments. +If +.meta pos +is omitted, it defaults to zero. -Note: the evaluation order in -.code new -is surprising: namely, -.metn init-form -s -are evaluated before -.metn arg -s -if both are present. +The stream must support the +.code get-byte +operation. Buffers which support +.code get-byte +can be expected to support +.code fill-buf +and, conversely, streams which do not support +.code get-byte +do not support +.codn fill-buf . -When the object is constructed, all default initializations take place -first. If the object's structure type has a supertype, then the supertype -initializations take place. Then the type's initializations take -place, followed by the -.meta slot -.meta init-form -overrides from the -.code new -macro, and lastly the "boa constructor" overrides. +The +.code fill-buf +function returns the position of the last byte that was successfully read. +If an end-of-file or other error condition occurs before the buffer is filled +through to the end, then the value returned is smaller than the buffer length. +In this case, the area of the buffer beyond the read size retains its previous +content. -If any of the initializations abandon the evaluation of -.code new -by a non-local exit such as an exception throw, the object's -finalizers, if any, are invoked. +If an error situation occurs other than a premature end-of-file before +any bytes are read, then an exception is thrown. -The macro -.code lnew -differs from new in that it specifies the construction of a -lazy struct, as if by the -.code make-lazy-struct -function. -When -.code lnew -is used to construct an instance, a lazy struct is returned -immediately, without evaluating any of the the -.meta arg -and -.meta init-form -expressions. -The expressions are evaluated when any of the object's -instance slots is accessed for the first time. At that time, -these expressions are evaluated (in the same order as under -.codn new ) -and initialization proceeds in the same way. - -If any of the initializations abandon the delayed initializations steps -arranged by -.code lnew -by a non-local exit such as an exception throw, the object's -finalizers, if any, are invoked. - -Lazy initialization does not detect cycles. Immediately prior to the lazy -initialization of a struct, the struct is marked as no longer requiring -initialization. Thus, during initialization, its instance slots may be -freely accessed. Slots not yet initialized evaluate as -.codn nil . +If an end-of-file condition occurs before any bytes are read, then zero +is returned. -.coNP Macros @ new* and @ lnew* -.synb -.mets (new* >> { expr | >> ( expr << arg *)} >> { slot << init-form }*) -.mets (lnew* >> { expr | >> ( expr << arg *)} >> { slot << init-form }*) -.syne -.desc The -.code new* -and -.code lnew* -macros are variants, respectively, of -.code new -and -.codn lnew . - -The only difference in behavior in these macros relative to -.code new -and -.code lnew -is that the -.meta name -argument is replaced with an expression -.meta expr -which is evaluated. The value of -.meta expr -must be a struct type, or a symbol which is the name of a struct type. +.code fill-buf-adjust +differs usefully from +.code fill-buf +as follows. Whereas +.code fill-buf +doesn't manipulate the length of the buffer at any stage of the operation, the +.code fill-buf-adjust +begins by adjusting the length of the buffer to the underlying allocated size. +Then it performs the fill operation in +exactly the same manner as +.codn fill-buf . +Finally, if the operation succeeds, then +.code fill-buf-adjust +adjusts the length of the buffer to match the position that is returned. -.coNP Macro @ with-slots -.synb -.mets (with-slots >> ({ slot | >> ( sym << slot )}*) < struct-expr -.mets \ \ << body-form *) -.syne -.desc -The -.code with-slots -binds lexical macros to serve as aliases for the slots of a structure. +.SS* Structures -The -.meta struct-expr -argument is expected to be an expression which evaluates to a struct -object. It is evaluated once, and its value is retained. The aliases are then -established to the slots of the resulting struct value. +\*(TX supports a structure data type. Structures are objects which +hold multiple storage locations called slots, which are named by symbols. +Structures can be related to each other by inheritance. -The aliases are specified as zero or more expressions which consist of either -a single symbol -.meta slot -or a -.mono -.meti >> ( sym << slot ) -.onom -pair. The simple form binds a macro named -.meta slot -to a slot also named -.metn slot . -The pair form binds a macro named -.meta sym -to a slot named -.metn slot . +The type of a structure is itself an object, of type +.codn struct-type . -The lexical aliases are syntactic places: assigning to an alias causes -the value to be stored into the slot which it denotes. +When the program defines a new structure type, it does so by creating a new +.code struct-type +instance, with properties which describe the new structure type: its +name, its list of slots, its initialization and "boa constructor" functions, +and the structure type it inherits from (the "super"). -After evaluating -.meta struct-expr -the -.code with-slots -macro arranges for the evaluation of -.metn body-form -s -in the lexical scope in which the aliases are visible. +The +.code struct-type +object is then used to generate instances. -.TP* "Dialect Notes:" +Structures instances are not only containers which hold named slots, but they +also indicate their struct type. Two structures which have the same number of +slots having the same names are not necessarily of the same type. -The intent of the -.code with-slots -macro is to help reduce the verbosity of code which makes multiple -references to the same slot. Use of -.code with-slots -is less necessary in \*(TL than other Lisp dialects -thanks to the dot operator for accessing struct slots. +Structure types and structures may be created and manipulated using +a programming interface based on functions. -Lexical aliases to struct places can also be -arranged with considerable convenience using the -.code placelet -operator. However, -.code placelet -will not bind multiple aliases to multiple slots of the same object -such that the expression which produces the object is evaluated only -once. +For more convenient and clutter-free expression of structure-based +program code, macros are also provided. -.TP* Example: -.verb - (defstruct point nil x y) +Furthermore, concise and expressive slot access syntax is provided courtesy of +the referencing dot and unbound referencing dot syntax, a syntactic sugar +for the +.code qref +and +.code uref +macros. - ;; Here, with-slots introduces verbosity because - ;; each slot is accessed only once. The function - ;; is equivalent to: - ;; - ;; (defun point-delta (p0 p1) - ;; (new point x (- p1.x p0.x) y (- p1.y p0.y))) - ;; - ;; Also contrast with the use of placelet: - ;; - ;; (defun point-delta (p0 p1) - ;; (placelet ((x0 p0.x) (y0 p0.y) - ;; (x1 p1.x) (y1 p1.y)) - ;; (new point x (- x1 x0) y (- y1 y0))))) +Structure types have a name, which is a symbol. The +.code typeof +function, when applied to any struct type, returns the symbol +.codn struct-type . +When +.code typeof +is applied to a struct instance, it returns the name of +the struct type. Effectively, struct names are types. - (defun point-delta (p0 p1) - (with-slots ((x0 x) (y0 y)) p0 - (with-slots ((x1 x) (y1 y)) p1 - (new point x (- x1 x0) y (- y1 y0))))) +The consequences are unspecified if an existing struct name is re-used for a +different struct type, or an existing type name is used for a struct type. +.NP* Static Slots -.brev +Structure slots can be of two kinds: they can be the ordinary instance slots or +they can be static slots. The instances of a given structure type have their +own instance of a given instance slot. However, they all share a single +instance of a static slot. -.coNP Macro @ qref -.synb -.mets (qref < object-form -.mets \ \ >> { slot | >> ( slot << arg *) | >> [ slot << arg *]}+) -.syne -.desc +Static slots are allocated in a global area associated with a structure type +and are initialized when the structure type is created. They are useful for +efficiently representing properties which have the same value for all instances +of a struct. These properties don't have to occupy space in each instance, and +time doesn't have to be wasted initializing them each time a new instance is +created. Static slots are also useful for struct-specific global variables. +Lastly, static slots are also useful for holding methods and functions. +Although structures can have methods and functions in their instances, usually, +all structures of the same type share the same functions. The +.code defstruct +macro supports a special syntax for defining methods and struct-specific +functions at the same time when a new structure type is defined. The -.code qref -macro ("quoted reference") performs structure slot access. Structure slot -access is more conveniently expressed using the referencing dot notation, which -works by translating to qref -.code qref -syntax, according to the following equivalence: - -.verb - a.b.c.d <--> (qref a b c d) ;; a b c d must not be numbers -.brev - -(See the Referencing Dot section under Additional Syntax.) - -The leftmost argument of -.code qref -is an expression which is evaluated. This argument is followed by one or more -reference designators. -If there are two or more designators, the following equivalence applies: - -.verb - (qref obj d1 d2 ...) <---> (qref (qref obj d1) d2 ...) -.brev +.code defmeth +macro can be used for adding new methods and functions to an existing +structure and its descendants. -That is to say, -.code qref -is applied to the object and a single designator. This must yield -an object, which to which the next designator is applied as if by -another -.code qref -operation, and so forth. +Static slots may be assigned just like instance slots. Changing a static +slot, of course, changes that slot in every structure of the same type. -Thus, -.code qref -can be understood entirely in terms of the semantics of the -binary form -.mono -.meti (qref < object-form << designator ) -.onom +Static slots are not listed in the +.code #S(...) +notation when a structure is printed. When the structure notation is +read from a stream, if static slots are present, they will be processed +and their values stored in the static locations they represent, thus +changing their values for all instances. -Designators come in three forms: a lone symbol, an ordinary compound expression -consisting of a symbol followed by arguments, or a DWIM expression -consisting of a symbol followed by arguments. +Static slots are inherited just like instance slots. If a given structure +.meta B +has some static slot +.metn s , +and a new structure +.meta D +is derived from +.metn B , +using +.codn defstruct , +and does not define a slot +.metn s , +then +.meta D +inherits +.metn s . +This means that +.meta D +shares the static slot with +.metn B : +both types share a single instance of that slot. -A lone symbol designator indicates the slot of that name. That is to say, the -following equivalence applies: +On the other hand if +.code D +defines a static slot +.meta s +then that slot will have its own instance in the +.meta D +structure type; +.meta D +will not inherit the +.meta B +instance of slot +.metn s . +Moreover, if the the definition of +.code D +omits the +.meta init-form +for slot +.metn s , +then that slot will be initialized with a copy of the current value of slot +.meta s +of the +.meta B +base type, which allows derived types to obtain the value of base type's +static slot, yet have that in their own instance. -.verb - (qref o n) <--> (slot o 'n) -.brev +The slot type can be overridden. A structure type deriving from another +type can introduce slots which have the same names as the supertype, +but are of a different kind: an instance slot in the supertype +can be replaced by a static slot in the derived type or vice versa. -where -.code slot -is the structure slot accessor function. Because -.code slot -is an accessor, this form denotes the slot as a syntactic place; -slots can be modified via assignment to the -.code qref -form and the referencing dot syntax. +A structure type is associated with a static initialization function +which may be used to store initial values into static slots. This function +is invoked once in a type's life time, when the type is created. +The function is also inherited by derived struct types and invoked when +they are created. -The slot name being implicitly quoted is the basis of the term -"quoted reference", giving rise to the -.code qref -name. +.NP* Dirty Flags +All structure instances contain a Boolean flag called the +.IR "dirty flag" . +This flag is not a slot, but rather a meta-data property that is exposed +to program access. When the flag is set, an object is said to be dirty; +otherwise it is clean. -A compound designator indicates that the named slot is a function, -and arguments are to be applied to it. The following equivalence applies -in this case, except that -.code o -is evaluated only once: +Newly constructed objects come into existence dirty. The dirty flag +state can be tested with the function +.codn test-dirty . +An object can be marked as clean by clearing its dirty flag with +.codn clear-dirty . +A combined operation +.code test-clear-dirty +is provided which clears the dirty flag, and +returns its previous value. -.verb - (qref o (n arg ...)) <--> (call (slot o 'n) o arg ...) -.brev +The dirty flag is set whenever a new value is stored into the instance +slot of an object. -A DWIM designator indicates that the named slot is a function or an -indexable or callable object. The following equivalence applies: +Note: the dirty flag can be used to support support the caching of values +derived from an object's slots. The derived values don't have to be +re-computed while an object remains clean. -.verb - (qref obj [name arg ...]) <--> [(slot obj 'name) arg ...] -.brev +.NP* Equality Substitution -.TP* Example: +In object-based or object-oriented programming, sometimes it is necessary for a +new data type to provide its own notion of equality: its own requirements for +when two distinct instances of the type are considered equal. Furthermore, +types sometimes have to implement their own notion, also, of inequality: the +requirements for the manner in which one instance is considered lesser or +greater than another. -.verb - (defstruct foo nil - (array (vec 1 2 3)) - (increment (lambda (self index delta) - (inc [self.array index] delta)))) +\*(TL structures implement a concept called +.IR "equality substitution" +which provides a simple, unified way for the implementor of an object to +encode the requirements for both equality and inequality. +Equality substitution allows for objects to be used as keys in a hash +table according to the custom equality, without the programmer being +burdened with the responsibility of developing a custom hashing function. - (defvarl s (new foo)) +An object participates in equality substitution by implementing the +.code equal +method. The +.code equal +method takes no arguments other than the object itself. It returns +a representative value which is used in place of that object +for the purposes of +.code equal +comparison. - ;; access third element of s.array: - s.[array 2] --> 3 +Whenever an object which supports equality substitution is used as an argument +of any of the functions +.codn equal , +.codn nequal , +.codn greater , +.codn less , +.codn gequal , +.code lequal +or +.codn hash-equal , +the +.code equal +method of that object is invoked, and the return value of that method +is taken in place of that object. - ;; increment first element of array by 42 - s.(increment 0 42) --> 43 +The same is true if an object which supports equality substitution is used as a +key in an +.code :equal-based +hash table. - ;; access array member - s.array --> #(43 2 3) -.brev +The substitution is applied repeatedly: if the return value of the object's +.code equal +method is an object which itself supports equality substitution, +than that returned object's method is invoked on that object +to fetch its equality substitute. This repeats as many times as necessary +until an object is determined which isn't a structure that supports +equality substitution. -Note how -.code increment -behaves much like a single-argument-dispatch object-oriented method. -Firstly, the syntax -.mono -s.(increment 0 42) -.onom -effectively selects the -.code increment -function which is particular to the -.code s -object. Secondly, the object is passed to the selected function as the -leftmost argument, so that the function has access to the object. +Once the equality substitute is determined, then the given function proceeds +with the replacement object. Thus for example +.code equal +compares the replacement object in place of the original, and an +.code :equal-based +hash table uses the replacement object as the key for the purposes of +hashing and comparison. -.coNP Macro @ uref +.coNP Macro @ defstruct .synb -.mets (uref >> { slot | >> ( slot << arg *) | >> [ slot << arg *]}+) +.mets (defstruct >> { name | >> ( name << arg *)} < super +.mets \ \ << slot-specifier *) .syne .desc The -.code uref -macro ("unbound reference") expands to an expression which evaluates to a -function. The function takes exactly one argument: an object. -When the function is invoked on an object, it references slots -or methods relative to that object. - -Note: the -.code uref -syntax may be used directly, but it is also produced by the unbound referencing -dot syntactic sugar: - -.verb - .a --> (uref a) - .(f x) --> (uref (f x)) - .(f x).b --> (uref (f x) b) - .a.(f x).b --> (uref a (f x) b) -.brev - -The macro may be understood in terms of the following translation -scheme: - -.verb - (uref a b c ...) --> (lambda (o) (qref o a b c ...)) -.brev - -where -.code o -is understood to be a unique symbol (for instance, as produced by the -.code gensym -function). +.code defstruct +macro defines a new structure type and registers it under +.metn name , +which must be a bindable symbol, according to the +.code bindable +function. Likewise, the name of every +.meta slot +must also be a bindable symbol. -When only one -.code uref -argument is present, these equivalences also hold: +The +.meta super +argument must either be +.code nil +or a symbol which names an existing struct type. +The newly defined struct type will inherit all slots, +as well as initialization behaviors from this type. -.verb - (uref (f a b c ...)) <--> (umeth f a b c ...) - (uref s) <--> (usl s) -.brev +The +.code defstruct +macro is implemented using the +.code make-struct-type +function, which is more general. The macro analyzes the +.code defstruct +argument syntax, and synthesizes arguments which are then +used to call the function. Some remarks in the description of +.code defstruct +only apply to structure types defined using that macro. -The terminology "unbound reference" refers to the property that -.code uref -expressions produce a function which isn't bound to a structure -object. The function binds a slot or method; the call to that function then -binds an object to that function, as an argument. +Slots are specified using zero or more +.IR "slot specifiers" . +Slot specifiers come in the following variety: +.RS +.meIP < name +The simplest slot specifier is just a name, which must be a bindable +symbol, as defined by the +.code bindable +function. This form is a short form for the +.mono +.meti (:instance << name ) +.onom +syntax. +.meIP >> ( name << init-form ) +This syntax is a short form for the +.mono +.meti (:instance < name << init-form ) +.onom +syntax. +.meIP (:instance < name <> [ init-form ]) +This syntax specifies an instance slot called +.meta name +whose initial value is obtained by evaluating +.meta init-form +whenever a new instance of the structure is created. +This evaluation takes place in the original lexical environment in which the +.code defstruct +form occurs. If +.meta init-form +is omitted, the slot is initialized to +.codn nil . +.meIP (:static < name <> [ init-form ]) +This syntax specifies a static slot called +.meta name +whose initial value is obtained by evaluating +.meta init-form +once, during the evaluation of the +.code defstruct +form in which it occurs, if the +.meta init-form +is present. If +.meta init-form +is absent, and a static slot with the same name +exists in the +.meta super +base type, then this slot is initialized +with the value of that slot. +Otherwise it is initialized to +.codn nil . -.TP* Examples: +The definition of a static slot in a +.code defstruct +causes the new type to have its own instance +that slot, even if a same-named static +slot occurs in the +.meta super +base type, or its bases. +.meIP (:method < name <> ( param +) << body-form *) +This syntax creates a static slot called +.meta name +which is initialized with an anonymous function. +The anonymous function is created during the +evaluation of the +.code defstruct +form. The function takes the arguments specified +by the +.meta param +symbols, and its body consists of the +.metn body-form -s. +There must be at least one +.metn param . +When the function is invoked as a method, as intended, +the leftmost +.meta param +receives the structure instance. +The +.metn body-form -s +are evaluated in a context in which a block named +.meta name +is visible. Consequently, +.code return-from +may be used to terminate the execution of a method +and return a value. +Methods are invoked +using the +.code "instance.(name arg ...)" +syntax, which implicitly inserts the instance into the argument list. -Suppose that the objects in -.code list -have slots -.code a -and -.codn b . -Then, a list of the -.code a -slot values may be obtained using: +Due to the semantics of static slots, methods are naturally +inherited from a base structure to a derived one, +and defining a method in a derived class which also exists +in a base class performs OOP-style overriding. +.meIP (:function < name <> ( param *) << body-form *) +This syntax creates a static slot called +.meta name +which is initialized with an anonymous function. +The anonymous function is created during the +evaluation of the +.code defstruct +form. The function takes the arguments specified +by the +.meta param +symbols, and its body consists of the +.metn body-form -s. +This specifier differs from +.code :method +only in one respect: there may be zero +parameters. A structure function defined this way is +intended to be used as a utility function which doesn't +receive the structure instance as an argument. +The +.metn body-form -s +are evaluated in a context in which a block named +.meta name +is visible. Consequently, +.code return-from +may be used to terminate the execution of the function +and return a value. +Such functions are called using the +.code "instance.[name arg ...]" +syntax which doesn't insert the instance into +the argument list. -.verb - (mapcar .a list) -.brev +The remarks about inheritance and overriding +in the description of +.code :method +also apply to +.codn :function . +.meIP (:init <> ( param ) << body-form *) +The +.code :init +specifier doesn't describe a slot. Rather, it specifies code +which is executed when a structure is instantiated, after +the slot initializations specific to the structure type +are performed. The code consists of +.metn body-form -s +which are evaluated in order in a lexical scope in +which the variable +.meta param +is bound to the structure object. -because this is equivalent to +The +.code :init +specifier may not appear more than once in a given +.code defstruct +form. -.verb - (mapcar (lambda (o) o.a) list) -.brev +When an object with one or more levels of inheritance +is instantiated, the +.code :init +code of a base structure type, if any, is executed +before any initializations specific to a derived +structure type. -Because -.code uref -produces a function, its result can be operated upon by -functional combinators. For instance, we can use the -.code juxt -combinator to produce a list of two-element lists, -which hold the -.code a -and -.code b -slots from each object in -.codn list : +The +.code :init +initializations are executed before any other +slot initializations. The argument values passed to the +.code new +or +.code lnew +operator or the +.code make-struct +function are not yet stored in the object's slots, +and are not accessible. Initialization code which needs +these values to be stable can be defined with +.codn :postinit . -.verb - (mapcar (juxt .a .b) list) -.brev +Initializers in base structures must be careful about assumptions about slot +kinds, because derived structures can alter static slots to instance slots or +vice versa. To avoid an unwanted initialization being applied to the +wrong kind of slot, initialization code can be made conditional on the +outcome of +.code static-slot-p +applied to the slot. +(Code generated by +.code defstruct +for initializing instance slots performs this kind of check). -.coNP Macro @ meth -.synb -.mets (meth < struct < slot << curried-expr *) -.syne -.desc The -.code meth -macro allows indirection upon a method-like function stored -in a function slot. - +.metn body-form -s +of an +.code :init +specifier are not surrounded by an implicit +.codn block . +.meIP (:postinit <> ( param ) << body-form *) The -.code meth -macro binds -.meta struct -as the leftmost argument of the function stored in -.metn slot , -returning a function which takes the remaining arguments. -That is to say, it returns a function -.meta f -such that -.mono -.meti >> [ f < arg ... ] -.onom -calls -.mono -.meti >> [ struct.slot < struct < arg ... ] -.onom -except that -.meta struct -is evaluated only once. - -If one or more -.meta curried-expr -expressions are present, their values are bound inside -.meta f -also, and when -.meta f -is invoked, these are passed to the function stored in the slot. -Thus if -.meta f -is produced by -.code "(meth struct slot c1 c2 c3 ...)" -then -.mono -.meti >> [ f < arg ... ] -.onom -calls +.code :postinit +specifier is very similar to +.codn :init . +Both specify forms which are evaluated during object instantiation. +The difference is that the +.codn body-form -s +of a +.code :postinit +are evaluated after other initializations have taken +place, including the +.code :init +initializations, as a second pass. By the time +.code :postinit +initialization runs, the argument material from the +.codn make-struct , +.code new +or +.code lnew +invocation has already been processed and stored +into slots. +Like +.code :init +actions, +.code :postinit +actions registered at different levels of the type's +inheritance hierarchy are invoked in the base-to-derived +order. +.meIP (:fini <> ( param ) << body-form *) +The +.code :fini +specifier doesn't describe a slot. Rather, it specifies +a finalization function which is associated with the +structure instance, as if by use of the +.code finalize +function. This finalization registration takes place +as the first step when an instance of the structure +is created, before the slots are initialized and +the +.code :init +code, if any, has been executed. The registration +takes place as if by the evaluation of the form .mono -.meti >> [ struct.slot < struct < c1v < c2v < c3v ... arg ... ] +.meti (finalize < obj (lambda <> ( param ) << body-form ...) t) .onom -except that -.meta struct -is evaluated only once, and -.metn c1v , -.meta c2v -and -.meta c3v -are the values of expressions -.codn c1 , -.code c2 -and -.codn c3 . +where +.meta obj +denotes the structure instance. Note the +.code t +argument which requests reverse order of registration, ensuring that if an +object has multiple finalizers registered at different levels of inheritance +hierarchy, the finalizers specified for a derived structure type are called +before inherited finalizers. -The argument -.meta struct -must be an expression which evaluates to a struct. The -.meta slot -argument is not evaluated, and must be a symbol denoting a slot. -The syntax can be understood as a translation to a call of the -.code method -function: +.metn body-form -s +of a +.code :fini +specifier are not surrounded by an implicit +.codn block . -.verb - (meth a b) <--> (method a 'b) -.brev +Note that an object's finalizers can be called explicitly with +.codn call-finalizers . +.RE +.IP +The +.code with-objects +macro arranges for finalizers to be called on objects when the execution +of a scope terminates by any means. -If -.meta curried-arg -expressions are present, the translation may be be understood -as: +The slot names given in a +.code defstruct +must all be unique among themselves, but they +may match the names of existing slots in the +.meta super +base type. -.verb - (meth a b c1 c2 ...) <--> [(fun method) a 'b c1 c2 ...] -.brev +A given structure type can have only one slot under a given +symbolic name. If a newly specified slot matches the name of an existing slot +in the +.meta super +type or that type's chain of ancestors, it is called a +.IR "repeated slot" . -In other words the -.meta curried-arg -expressions are evaluated under the -.code dwim -operator evaluation rules. +The kind of the repeated slot (static or instance) is not inherited; it +is established by the +.code defstruct +and may be different from the type of the same-named slot in the +supertype or its ancestors. -.TP* Example: +If a repeated slot is introduced as a static slot, and +has no +.meta init-form +then it receives the current of the a static of the same name from +the nearest supertype which has such a slot. -.verb - ;; struct for counting atoms eq to key - (defstruct (counter key) nil - key - (count 0) - (:method increment (self key) - (if (eq self.key key) - (inc self.count)))) +If a repeated slot is an instance slot, no such inheritance of value +takes place; only the local +.meta init-form +applies to it; if it is absent, the slot it initialized to +.code nil +in each newly created instance of the new type. - ;; pass all atoms in tree to func - (defun map-tree (tree func) - (if (atom tree) - [func tree] - (progn (map-tree (car tree) func) - (map-tree (cdr tree) func)))) +However, +.code :init +and +.code :postinit +initializations are inherited from a base type and they apply to +the repeated slots, regardless of their kind. These initializations +take place on the instantiated object, and the slot references +resolve accordingly. - ;; count occurrences of symbol a - ;; using increment method of counter, - ;; passed as func argument to map-tree. - (let ((c (new (counter 'a))) - (tr '(a (b (a a)) c a d))) - (map-tree tr (meth c increment)) - c) - --> #S(counter key a count 4 - increment #) -.brev +The initialization for slots which are specified using the +.code :method +or +.code :function +specifiers is re-ordered with regard to +.code :static +slots. Regardless of their placement in the +.code defstruct +form, +.code :method +and +.code :function +slots are initialized before +.code :static +slots. This ordering is useful, because it means that when the initialization +expression for a given static slot constructs an instance of the struct type, +any instance initialization code executing for that instance can use +all functions and methods of the struct type. +However, note the static slots which follow that slot in the +.code defstruct +syntax are not yet initialized. If it is necessary for a structure's +initialization code to have access to all static slots, even when the +structure is instantiated during the initialization of a static slot, +a possible solution may be to use lazy instantiation using the +.code lnew +operator, rather than ordinary eager instantiation via +.codn new . +It is also necessary to ensure that that the instance isn't accessed until all +static initializations are complete, since access to the instance slots of a +lazily instantiated structure triggers its initialization. -.coNP Macro @ umeth -.synb -.mets (umeth < slot << curried-expr *) -.syne -.desc +The structure name is specified using two forms, plain +.meta name +or the syntax +.mono +.meti >> ( name << arg *) +.onom +If the second form is used, then the structure type will support +"boa construction", where "boa" stands for "by order of arguments". The -.code umeth -macro binds the symbol -.meta slot -to a function and returns that function. +.metn arg -s +specify the list of slot names which are to be initialized in the +by-order-of-arguments style. For instance, if three slot names +are given, then those slots can be optionally initialized by giving three +arguments in the +.code new +macro or the +.code make-struct +function. -The -.meta curried-expr -arguments, if present, are evaluated as if they were -arguments to the -.code dwim -operator. +Slots are first initialized according to their +.metn init-form -s, +regardless of whether they are involved in boa construction -When that function is called, it expects at least one argument. -The leftmost argument must be an object of struct type. +A slot initialized in this style still has a +.meta init-form +which is processed independently of the existence of, and prior to, +boa construction. -The slot named -.meta slot -is retrieved from that object, and is expected to be a function. -That function is called with the object, followed by the values -of the -.metn curried-expr -s, -if any, followed by that function's arguments. +The boa constructor syntax can specify optional parameters, delimited +by a colon, similarly to the +.code lambda +syntax. However, the optional parameters may not be arbitrary symbols; +they must be symbols which name +slots. Moreover, the +.mono +.meti >> ( name < init-form <> [ present-p ]) +.onom +optional parameter syntax isn't supported. -The syntax can be understood as a translation to a call of the -.code umethod -function: +When boa construction is invoked with optional arguments missing, +the default values for those arguments come from the +.metn init-form -s +in the remaining +.code defstruct +syntax. +.TP* Examples: .verb - (umeth s ...) <--> [umethod 's ...] -.brev + (defvar *counter* 0) -The macro merely provides the syntactic sugar of not having to quote the -symbol, and automatically treating the curried argument expressions -using Lisp-1 semantics of the -.code dwim -operator. + ;; New struct type foo with no super type: + ;; Slots a and b initialize to nil. + ;; Slot c is initialized by value of (inc *counter*). + (defstruct foo nil (a b (c (inc *counter*)))) -.TP* Example: + (new foo) -> #S(foo a nil b nil c 1) + (new foo) -> #S(foo a nil b nil c 2) -.verb - ;; seal and dog are variables which hold structures of - ;; different types. Both have a method called bark. + ;; New struct bar inheriting from foo. + (defstruct bar foo (c 0) (d 100)) - (let ((bark-fun (umeth bark))) - [bark-fun dog] ;; same effect as dog.(bark) - [bark-fun seal]) ;; same effect as seal.(bark) -.brev + (new bar) -> #S(bar a nil b nil c 0 d 100) + (new bar) -> #S(bar a nil b nil c 0 d 100) -The -.code u -in -.code umeth -stands for "unbound". The function produced by -.code umeth -is not bound to any specific object; it binds to an object whenever it is -invoked by retrieving the actual method from the object's slot at call time. + ;; counter was still incremented during + ;; construction of d: + *counter* -> 4 -.coNP Macro @ usl -.synb -.mets (usl << slot ) -.syne -.desc -The -.code usl -macro binds the symbol -.meta slot -to a function and returns that function. + ;; override slots with new arguments + (new foo a "str" c 17) -> #S(foo a "str" b nil c 17) -When that function is called, it expects exactly one argument. -That argument must be an object of struct type. -The slot named -.meta slot -is retrieved from that object and returned. + *counter* -> 5 -The name -.code usl -stands for "unbound slot". The term "unbound" refers to the returned -function not being bound to a particular object. The binding of the -slot to an object takes place whenever the function is called. + ;; boa initialization + (defstruct (point x : y) nil (x 0) (y 0)) -.coNP Function @ make-struct-type + (new point) -> #S(point x 0 y 0) + (new (point 1 1)) -> #S(point x 1 y 1) + + ;; property list style initialization + ;; can always be used: + (new point x 4 y 5) -> #S(point x 4 y 5) + + ;; boa applies last: + (new (point 1 1) x 4 y 5) -> #S(point x 1 y 1) + + ;; boa with optional argument omitted: + (new (point 1)) -> #S(point x 1 y 0) + + ;; boa with optional argument omitted and + ;; with property list style initialization: + (new (point 1) x 5 y 5) -> #S(point x 1 y 5) +.brev + +.coNP Macro @ defmeth .synb -.mets (make-struct-type < name < super < static-slots < slots -.mets \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ < static-initfun < initfun << boactor -.mets \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ < boactor << postinitfun ) +.mets (defmeth < type-name < name < param-list << body-form *) .syne .desc -The -.code make-struct-type -function creates a new struct type. - -The +Unless .meta name -argument must be a bindable symbol, according to the -.code bindable -function. It specifies the name property of the struct -type as well as the name under which the struct type -is globally registered. - -The -.meta super -argument indicates the supertype for the struct type. -It must be either a value of type -.codn struct-type , -a symbol which names a struct type, or else -.codn nil , -indicating that the newly created struct type has no supertype. +is one of the two symbols +.code :init +or +.codn :postinit , +the +.code defmeth +macro installs a function into the static slot named by the symbol +.meta name +in the struct type indicated by +.metn type-name . -The -.meta static-slots -argument is a list of symbol which specify static slots. -The symbols must be bindable and the list must not contain duplicates. +If the structure type doesn't already have such a static slot, it is +first added, as if by the +.code static-slot-ensure +function, subject to the same checks. -The -.meta slots -argument is a list of symbols which specifies the instance slots. -The symbols must be bindable and there must not be any duplicates -within the list, or against entries in the -.meta static-slots -list. +If the function has at least one argument, it can be used as a method. In that +situation, the leftmost argument passes the structure instance on which the +method is being invoked. -The new struct type's effective list of slots is formed by appending -together -.meta static-slots -and -.metn slots , -and then appending that to the list of the supertype's slots, and -de-duplicating the resulting list as if by the -.code uniq -function. Thus, any slots which are already present in the supertype are -removed. If the structure has no supertype, then the list of supertype -slots is taken to be empty. When a structure is instantiated, it shall have all -the slots specified in the effective list of slots. Each instance slot -shall be initialized to the value -.codn nil , -prior to the invocation of -.meta initfun -and -.metn boactor . +The function takes the arguments specified +by the +.meta param-list +symbols, and its body consists of the +.metn body-form -s. The -.meta static-initfun -argument either specifies an initialization function, or is -.codn nil , -which is equivalent to specifying a function which does nothing. +.metn body-form -s +are placed into a +.code block +named +.codn name . -Prior to the invocation of -.metn static-initfun , -each new static slot shall be initialized the value -.codn nil . -Inherited static slots retain their values from the supertype. +A method named +.code lambda +allows a structure to be used as if it were a function. When arguments +are applied to the structure as if it were a function, the +.code lambda +method is invoked with those arguments, with the object itself inserted +into the leftmost argument position. -If specified, -.meta static-initfun -function must -accept one argument. When the structure type is created (before -the -.code make-struct-type -function returns) the -.meta static-initfun -function is invoked, passed the newly created -structure type as its argument. +If +.code defmeth +is used to redefine an existing method, the semantics can be inferred +from that of +.codn static-slot-ensure . +In particular, the method will be imposed into all subtypes which inherit +(do not override) the method. -The -.meta initfun -argument either specifies an initialization function, or is -.codn nil , -which is equivalent to specifying a function which does nothing. -If specified, this function must -accept one argument. When a structure is instantiated, every +If +.meta name +is the keyword symbol +.codn :init , +then instead of operating on a static slot, the macro redefines the .meta initfun -in its chain of supertype ancestry is invoked, in order of inheritance, -so that the root supertype's +of the given structure type, as if by a call to the function +.codn struct-set-initfun . + +Similarly, if +.meta name +is the keyword symbol +.codn :postinit , +then the macro redefines the +.meta postinitfun +of the given structure type, as if by a call to the function +.codn struct-set-postinitfun . + +When redefining +.code :initfun +the admonishments given in the description of +.code struct-set-initfun +apply: if the type has an .meta initfun -is called first and the structure's own specific +generated by the +.code defstruct +macro, then that .meta initfun -is called last. These calls occur before the slots are initialized -from the -.meta arg -arguments -or the -.meta slot-init-plist -of -.codn make-struct . -Each function is passed the newly created structure -object, and may alter its slots. - -The -.meta boactor -argument either specifies a by-order-of-arguments initialization -function ("boa constructor") or is -.codn nil , -which is equivalent to specifying a constructor which does nothing. -If specified, it must be a function which takes at least one argument. -When a structure is instantiated, and boa arguments are given, the -.meta boactor -is invoked, with the structure as the leftmost argument, and -the boa arguments as additional arguments. This takes place -after the processing of +is what implements all of the slot initializations given in the +slot specifier syntax. These initializations are lost if the .meta initfun -functions, and after the processing of the -.meta slot-init-plist -specified in the -.code make-struct -call. Note that the -.meta boactor -functions of the supertypes are not called, only the -.meta boactor -specific to the type being constructed. +is overwritten. The -.meta postinitfun -argument either specifies an initialization function, or is -.codn nil , -which is equivalent to specifying a function which does nothing. -If specified, this function must accept one argument. -The -.meta postinitfun -function is similar to -.metn initfun . -The difference is that -.meta postinitfun -functions are called after all other initialization processing, -rather than before. They are are also called in order of -inheritance: the -.meta postinitfun -of a structure's supertype is called before its own. +.code defmeth +macro returns a method name: a unit of syntax of the form +.mono +.meti (meth < type-name << name) +.onom +which can be used as an argument to the accessor +.code symbol-function +and other situations. -.coNP Function @ find-struct-type +.coNP Macros @ new and @ lnew .synb -.mets (find-struct-type << name ) +.mets (new >> { name | >> ( name << arg *)} >> { slot << init-form }*) +.mets (lnew >> { name | >> ( name << arg *)} >> { slot << init-form }*) .syne .desc The -.code find-struct-type -returns a -.code struct-type -object corresponding to the symbol +.code new +macro creates a new instance of the structure type named by .metn name . -If no struct type is registered under -.metn name , -then it returns -.codn nil . +If the structure supports "boa construction", then, optionally, the +arguments may be given using the syntax +.mono +.meti >> ( name << arg *) +.onom +instead of +.metn name . -A -.code struct-type -object exists for each structure type and holds information about it. -These objects are not themselves structures and are all of the -same type, -.codn struct-type . +Slot values may also be specified by the +.meta slot +and +.meta init-form +arguments. -.coNP Function @ struct-type-p -.synb -.mets (struct-type-p << obj ) -.syne -.desc -The -.code struct-type-p -function returns t if -.meta obj -is a structure type, otherwise it returns -.codn nil . +Note: the evaluation order in +.code new +is surprising: namely, +.metn init-form -s +are evaluated before +.metn arg -s +if both are present. -A structure type is an object of type -.codn struct-type , -returned by -.codn find-struct-type . +When the object is constructed, all default initializations take place +first. If the object's structure type has a supertype, then the supertype +initializations take place. Then the type's initializations take +place, followed by the +.meta slot +.meta init-form +overrides from the +.code new +macro, and lastly the "boa constructor" overrides. -.coNP Function @ struct-type-name -.synb -.mets (struct-type-name << type ) -.syne -.desc -The -.code struct-type-name -function returns the symbol which serves as the name of -.metn type , -which must be either a struct type object (such as the return value of -a successful lookup via -.codn find-struct-type ), -or else a struct type name. +If any of the initializations abandon the evaluation of +.code new +by a non-local exit such as an exception throw, the object's +finalizers, if any, are invoked. -.coNP Function @ super +The macro +.code lnew +differs from new in that it specifies the construction of a +lazy struct, as if by the +.code make-lazy-struct +function. +When +.code lnew +is used to construct an instance, a lazy struct is returned +immediately, without evaluating any of the the +.meta arg +and +.meta init-form +expressions. +The expressions are evaluated when any of the object's +instance slots is accessed for the first time. At that time, +these expressions are evaluated (in the same order as under +.codn new ) +and initialization proceeds in the same way. + +If any of the initializations abandon the delayed initializations steps +arranged by +.code lnew +by a non-local exit such as an exception throw, the object's +finalizers, if any, are invoked. + +Lazy initialization does not detect cycles. Immediately prior to the lazy +initialization of a struct, the struct is marked as no longer requiring +initialization. Thus, during initialization, its instance slots may be +freely accessed. Slots not yet initialized evaluate as +.codn nil . + +.coNP Macros @ new* and @ lnew* .synb -.mets (super << type ) +.mets (new* >> { expr | >> ( expr << arg *)} >> { slot << init-form }*) +.mets (lnew* >> { expr | >> ( expr << arg *)} >> { slot << init-form }*) .syne .desc The -.code super -function returns the struct type object which is the -supertype of -.metn type , -or returns -.code nil -if -.meta type -has no supertype. +.code new* +and +.code lnew* +macros are variants, respectively, of +.code new +and +.codn lnew . -The -.meta type -argument must be either a struct type object, a -a symbol which names a struct type (which is resolved to that type), -or else a structure instance (which is resolved to its structure type). +The only difference in behavior in these macros relative to +.code new +and +.code lnew +is that the +.meta name +argument is replaced with an expression +.meta expr +which is evaluated. The value of +.meta expr +must be a struct type, or a symbol which is the name of a struct type. -.coNP Function @ make-struct +.coNP Macro @ with-slots .synb -.mets (make-struct < type < slot-init-plist << arg *) +.mets (with-slots >> ({ slot | >> ( sym << slot )}*) < struct-expr +.mets \ \ << body-form *) .syne .desc The -.code make-struct -function returns a new object which is an instance of the -structure type -.metn type . +.code with-slots +binds lexical macros to serve as aliases for the slots of a structure. The -.meta type -argument must either be a -.code struct-type -object, or else a symbol which is the name of a structure. +.meta struct-expr +argument is expected to be an expression which evaluates to a struct +object. It is evaluated once, and its value is retained. The aliases are then +established to the slots of the resulting struct value. -The -.meta slot-init-plist -argument gives a list of slot initializations in the -style of a property list, as defined by the -.code prop -function. It may be empty, in which case -it has no effect. Otherwise, it specifies slot names -and their values. Each slot name which is given must -be a slot of the structure type. The corresponding value -will be stored into the slot of the newly created object. If a slot is -repeated, it is unspecified which value takes effect. +The aliases are specified as zero or more expressions which consist of either +a single symbol +.meta slot +or a +.mono +.meti >> ( sym << slot ) +.onom +pair. The simple form binds a macro named +.meta slot +to a slot also named +.metn slot . +The pair form binds a macro named +.meta sym +to a slot named +.metn slot . -The optional -.metn arg -s -specify arguments to the structure type's boa constructor. -If the arguments are omitted, the boa constructor is not invoked. -Otherwise the boa constructor is invoked on the structure object -and those arguments. The argument list must match the trailing parameters of -the boa constructor (the remaining parameters which follow the leftmost -argument which passes the structure to the boa constructor). +The lexical aliases are syntactic places: assigning to an alias causes +the value to be stored into the slot which it denotes. -When a new structure is instantiated by -.codn make-struct , -its slot values are first initialized by the structure type's registered -functions as described under -.codn make-struct-type . -Then, the -.meta slot-init-plist -is processed, if not empty, and finally, the -.metn arg -s -are processed, if present, and passed to the boa constructor. +After evaluating +.meta struct-expr +the +.code with-slots +macro arranges for the evaluation of +.metn body-form -s +in the lexical scope in which the aliases are visible. -If any of the initializations abandon the evaluation of -.code make-struct -by a non-local exit such as an exception throw, the object's -finalizers, if any, are invoked. +.TP* "Dialect Notes:" -.coNP Function @ make-lazy-struct -.synb -.mets (make-lazy-struct < type << argfun ) -.syne -.desc -The -.code make-lazy-struct -function returns a new object which is an instance of the -structure type -.metn type . +The intent of the +.code with-slots +macro is to help reduce the verbosity of code which makes multiple +references to the same slot. Use of +.code with-slots +is less necessary in \*(TL than other Lisp dialects +thanks to the dot operator for accessing struct slots. -The -.meta type -argument must either be a -.code struct-type -object, or else a symbol which is the name of a structure. +Lexical aliases to struct places can also be +arranged with considerable convenience using the +.code placelet +operator. However, +.code placelet +will not bind multiple aliases to multiple slots of the same object +such that the expression which produces the object is evaluated only +once. -The -.meta argfun -argument should be a function which can be called with no parameters -and returns a cons cell. More requirements are specified below. +.TP* Example: +.verb + (defstruct point nil x y) -The object returned by -.code make-lazy-struct -is a lazily-initialized struct instance, or -.IR "lazy struct" . + ;; Here, with-slots introduces verbosity because + ;; each slot is accessed only once. The function + ;; is equivalent to: + ;; + ;; (defun point-delta (p0 p1) + ;; (new point x (- p1.x p0.x) y (- p1.y p0.y))) + ;; + ;; Also contrast with the use of placelet: + ;; + ;; (defun point-delta (p0 p1) + ;; (placelet ((x0 p0.x) (y0 p0.y) + ;; (x1 p1.x) (y1 p1.y)) + ;; (new point x (- x1 x0) y (- y1 y0))))) -A lazy struct remains uninitialized until just before the first access -to any of its instance slots. Just before an instance slot is -accessed, initialization -takes place as follows. The -.meta argfun -function is invoked with no arguments. Its return value must be a cons -cell. The -.code car -of the cons cell is taken to be a property list, as defined by the -.code prop -function. The -.code cdr -field is taken to be a list of arguments. These values are treated -as if they were, respectively, the -.meta slot-init-plist -and the boa constructor arguments given in a -.code make-struct -invocation. Initialization of the structure proceeds as described -in the description of -.codn make-struct . + (defun point-delta (p0 p1) + (with-slots ((x0 x) (y0 y)) p0 + (with-slots ((x1 x) (y1 y)) p1 + (new point x (- x1 x0) y (- y1 y0))))) -.coNP Functions @ struct-from-plist and @ struct-from-args + +.brev + +.coNP Macro @ qref .synb -.mets (struct-from-plist < type >> { slot << value }*) -.mets (struct-from-arg < type << arg *) +.mets (qref < object-form +.mets \ \ >> { slot | >> ( slot << arg *) | >> [ slot << arg *]}+) .syne .desc The -.code struct-from-plist -and -.code struct-from-arg -are interfaces to the -.code make-struct -function. +.code qref +macro ("quoted reference") performs structure slot access. Structure slot +access is more conveniently expressed using the referencing dot notation, which +works by translating to qref +.code qref +syntax, according to the following equivalence: -The -.code struct-from-plist -function passes its -.meta slot -and -.meta value -arguments as the -.meta slot-init-plist -argument of -.codn make-struct . -It passes no boa constructor arguments. +.verb + a.b.c.d <--> (qref a b c d) ;; a b c d must not be numbers +.brev -The -.code struct-from-plist -function calls -.meta make-struct -with an empty -.metn slot-init-plist , -passing down the list of -.metn arg -s. +(See the Referencing Dot section under Additional Syntax.) -The following equivalences hold: +The leftmost argument of +.code qref +is an expression which is evaluated. This argument is followed by one or more +reference designators. +If there are two or more designators, the following equivalence applies: .verb - (struct-from-plist a s0 v0 s1 v1 ...) - <--> (make-struct a (list s0 v0 s1 v1 ...)) - - (struct-from-args a v0 v1 v2 ...) - <--> (make-struct a nil v0 v1 v2 ...) + (qref obj d1 d2 ...) <---> (qref (qref obj d1) d2 ...) .brev -.coNP Function @ allocate-struct -.synb -.mets (allocate-struct << type ) -.syne -.desc -The -.code allocate-struct -provides a low-level allocator for structure objects. +That is to say, +.code qref +is applied to the object and a single designator. This must yield +an object, which to which the next designator is applied as if by +another +.code qref +operation, and so forth. -The -.meta type -argument must either be a -.code struct-type -object, or else a symbol which is the name of a structure. +Thus, +.code qref +can be understood entirely in terms of the semantics of the +binary form +.mono +.meti (qref < object-form << designator ) +.onom -The -.code allocate-struct -creates and returns a new instance of -.meta type -all of whose instance slots take on the value -.codn nil . -No initializations are performed. The struct type's -registered initialization functions are not invoked. +Designators come in three forms: a lone symbol, an ordinary compound expression +consisting of a symbol followed by arguments, or a DWIM expression +consisting of a symbol followed by arguments. -.coNP Function @ copy-struct -.synb -.mets (copy-struct << struct-obj ) -.syne -.desc -The -.code copy-struct -function creates and returns a new object which is a duplicate of -.metn struct-obj , -which must be a structure. +A lone symbol designator indicates the slot of that name. That is to say, the +following equivalence applies: -The duplicate object is a structure of the same type as -.meta struct-obj -and has the same slot values. +.verb + (qref o n) <--> (slot o 'n) +.brev -The creation of a duplicate does not involve calling any of the -struct type's initialization functions. +where +.code slot +is the structure slot accessor function. Because +.code slot +is an accessor, this form denotes the slot as a syntactic place; +slots can be modified via assignment to the +.code qref +form and the referencing dot syntax. -Only instance slots participate in the duplication. Since -the original structure and copy are of the same structure type, -they already share static slots. +The slot name being implicitly quoted is the basis of the term +"quoted reference", giving rise to the +.code qref +name. -This is a low-level, "shallow" copying mechanism. If an object design -calls for a higher level cloning mechanism with deep copying or other -additional semantics, one can be built on top of -.codn copy-struct . -For instance, a structure can have a -.code copy -method similar to the following: +A compound designator indicates that the named slot is a function, +and arguments are to be applied to it. The following equivalence applies +in this case, except that +.code o +is evaluated only once: .verb - (:method copy (me) - (let ((my-copy (copy-struct me))) - ;; inform the copy that it has been created - ;; by invoking its copied method. - my-copy.(copied) - my-copy)) + (qref o (n arg ...)) <--> (call (slot o 'n) o arg ...) .brev -since this logic is generic, it can be placed in a base -method. The -.code copied -method which it calls is the means by which the new object is notified that it -is a copy. This method takes on whatever special responsibilities are required -when a copy is produced, such as registering the object in various necessary -associations, or performing a deeper copy of some of the objects held -in the slots. +A DWIM designator indicates that the named slot is a function or an +indexable or callable object. The following equivalence applies: -The -.code copied -handler can be implemented at multiple levels of an inheritance hierarchy. The -initial call to -.code copied -from -.code copy -will call the most derived override of that method. +.verb + (qref obj [name arg ...]) <--> [(slot obj 'name) arg ...] +.brev -To call the corresponding method in the base class, a given derived method -can use the -.code call-super-fun -function, or else the -.code "(meth ...)" -syntax in the first position of a compound form, in place of a function name. -Examples of both are given in the documentation for -.codn call-super-fun . +.TP* Example: -Thus derived structs can inherit the copy handling logic from base structs, and -extend it with their own. +.verb + (defstruct foo nil + (array (vec 1 2 3)) + (increment (lambda (self index delta) + (inc [self.array index] delta)))) -.coNP Accessor @ slot -.synb -.mets (slot < struct-obj << slot-name ) -.mets (set (slot < struct-obj << slot-name ) << new-value ) -.syne -.desc -The -.code slot -function retrieves a structure's slot. The -.meta struct-obj -argument must be a structure, and -.meta slot-name -must be a symbol which names a slot in that structure. + (defvarl s (new foo)) -Because -.code slot -is an accessor, a -.code slot -form is a syntactic place which denotes the -slot's storage location. + ;; access third element of s.array: + s.[array 2] --> 3 -A syntactic place expressed by -.code slot -does not support deletion. + ;; increment first element of array by 42 + s.(increment 0 42) --> 43 -.coNP Function @ slotset + ;; access array member + s.array --> #(43 2 3) +.brev + +Note how +.code increment +behaves much like a single-argument-dispatch object-oriented method. +Firstly, the syntax +.mono +s.(increment 0 42) +.onom +effectively selects the +.code increment +function which is particular to the +.code s +object. Secondly, the object is passed to the selected function as the +leftmost argument, so that the function has access to the object. + +.coNP Macro @ uref .synb -.mets (slotset < struct-obj < slot-name << new-value ) +.mets (uref >> { slot | >> ( slot << arg *) | >> [ slot << arg *]}+) .syne .desc The -.code slotset -function stores a value in a structure's slot. - The -.meta struct-obj -argument must be a structure, and -.meta slot-name -must be a symbol which names a slot in that structure. +.code uref +macro ("unbound reference") expands to an expression which evaluates to a +function. The function takes exactly one argument: an object. +When the function is invoked on an object, it references slots +or methods relative to that object. -The -.meta new-value -argument specifies the value to be stored in the slot. +Note: the +.code uref +syntax may be used directly, but it is also produced by the unbound referencing +dot syntactic sugar: -If a successful store takes place to an instance slot of -.metn struct-obj , -then the dirty flag of that object is set, causing the -.code test-dirty -function to report true for that object. +.verb + .a --> (uref a) + .(f x) --> (uref (f x)) + .(f x).b --> (uref (f x) b) + .a.(f x).b --> (uref a (f x) b) +.brev -The -.code slotset -function returns -.metn new-value . +The macro may be understood in terms of the following translation +scheme: -.coNP Functions @, test-dirty @ clear-dirty and @ test-clear-dirty -.synb -.mets (test-dirty << struct-obj ) -.mets (clear-dirty << struct-obj ) -.mets (test-clear-dirty << struct-obj ) -.syne -.desc -The -.codn test-dirty , -.code clear-dirty -and -.code test-clear-dirty -functions comprise the interface for interacting with structure -dirty flags. +.verb + (uref a b c ...) --> (lambda (o) (qref o a b c ...)) +.brev -Each structure instance has a dirty flag. When this flag is set, the -structure instance is said to be dirty, otherwise it is said to be clean. A -newly created structure is dirty. A structure remains dirty until its dirty -flag is explicitly reset. If a structure is clean, and one of its instance -slots is overwritten with a new value, it becomes dirty. +where +.code o +is understood to be a unique symbol (for instance, as produced by the +.code gensym +function). -The -.code test-dirty -function returns the dirty flag of -.metn struct-obj : -.code t -if -.meta struct-obj -is dirty, otherwise -.codn nil . +When only one +.code uref +argument is present, these equivalences also hold: -The -.code clear-dirty -function clears the dirty flag of -.meta struct-obj -and returns -.meta struct-obj -itself. +.verb + (uref (f a b c ...)) <--> (umeth f a b c ...) + (uref s) <--> (usl s) +.brev -The -.code test-clear-dirty -flag combines these operations: it makes a note of the dirty flag of -.meta struct-obj -and clears it. Then it returns the noted value, -.code t -or -.codn nil . +The terminology "unbound reference" refers to the property that +.code uref +expressions produce a function which isn't bound to a structure +object. The function binds a slot or method; the call to that function then +binds an object to that function, as an argument. -.coNP Function @ structp -.synb -.mets (structp << obj ) -.syne -.desc -The -.code structp -function returns t if -.meta obj -is a structure, otherwise it returns -.codn nil . +.TP* Examples: -.coNP Function @ struct-type -.synb -.mets (struct-type << struct-obj ) -.syne -.desc -The -.code struct-type -function returns the structure type object which -represents the type of the structure object instance -.metn struct-obj . +Suppose that the objects in +.code list +have slots +.code a +and +.codn b . +Then, a list of the +.code a +slot values may be obtained using: -.coNP Function @ clear-struct -.synb -.mets (clear-struct < struct-obj <> [ value ]) -.syne -.desc -The -.code clear-struct -replaces all instance slots of -.meta struct-obj -with -.metn value , -which defaults to -.code nil -if omitted. +.verb + (mapcar .a list) +.brev -Note that finalizers are not executed prior to replacing -the slot values. +because this is equivalent to -.coNP Function @ reset-struct +.verb + (mapcar (lambda (o) o.a) list) +.brev + +Because +.code uref +produces a function, its result can be operated upon by +functional combinators. For instance, we can use the +.code juxt +combinator to produce a list of two-element lists, +which hold the +.code a +and +.code b +slots from each object in +.codn list : + +.verb + (mapcar (juxt .a .b) list) +.brev + +.coNP Macro @ meth .synb -.mets (reset-struct << struct-obj ) +.mets (meth < struct < slot << curried-expr *) .syne .desc The -.code reset-struct -function reinitializes the structure object -.meta struct-obj -as if it were being newly created. -First, all the slots are set to -.code nil -as if by the -.code clear-struct -function. Then the slots are initialized by invoking the -initialization functions, in order of the supertype ancestry, just as would be -done for a new structure object created by -.code make-struct -with an empty -.meta slot-init-plist -and no boa arguments. +.code meth +macro allows indirection upon a method-like function stored +in a function slot. -Note that finalizers registered against -.meta struct-obj -are not invoked prior to the reset operation, and remain registered. +The +.code meth +macro binds +.meta struct +as the leftmost argument of the function stored in +.metn slot , +returning a function which takes the remaining arguments. +That is to say, it returns a function +.meta f +such that +.mono +.meti >> [ f < arg ... ] +.onom +calls +.mono +.meti >> [ struct.slot < struct < arg ... ] +.onom +except that +.meta struct +is evaluated only once. -If the structure has state which is cleaned up by -finalizers, it is advisable to invoke them using -.code call-finalizers -prior to using -.codn reset-struct , -or to take other measures to deal with the -situation. +If one or more +.meta curried-expr +expressions are present, their values are bound inside +.meta f +also, and when +.meta f +is invoked, these are passed to the function stored in the slot. +Thus if +.meta f +is produced by +.code "(meth struct slot c1 c2 c3 ...)" +then +.mono +.meti >> [ f < arg ... ] +.onom +calls +.mono +.meti >> [ struct.slot < struct < c1v < c2v < c3v ... arg ... ] +.onom +except that +.meta struct +is evaluated only once, and +.metn c1v , +.meta c2v +and +.meta c3v +are the values of expressions +.codn c1 , +.code c2 +and +.codn c3 . -If the structure specifies -.code :fini -handlers, then the reinitialization will cause -these to registered, just like when a new object -it constructed. Thus if -.code call-finalizers -is not used prior to -.codn reset-struct , -this will result in the existence of duplicate registrations of the -finalization functions. +The argument +.meta struct +must be an expression which evaluates to a struct. +The +.meta slot +argument is not evaluated, and must be a symbol denoting a slot. +The syntax can be understood as a translation to a call of the +.code method +function: -Finalizers registered against -.meta struct-obj -.B are -invoked if an exception is thrown -during the reinitialization, just like when a new -structure is being constructed. +.verb + (meth a b) <--> (method a 'b) +.brev -.coNP Function @ replace-struct -.synb -.mets (replace-struct < target-obj << source-obj ) -.syne -.desc -The -.code replace-struct -function causes -.meta target-obj -to take on the attributes of -.meta source-obj -without changing its identity. +If +.meta curried-arg +expressions are present, the translation may be be understood +as: -The type of -.code target-obj -is changed to that of -.codn source-obj . +.verb + (meth a b c1 c2 ...) <--> [(fun method) a 'b c1 c2 ...] +.brev -All instance slots of -.code target-obj -are discarded, and it is given new slots, -which are copies of the instance slots of -.codn source-obj . +In other words the +.meta curried-arg +expressions are evaluated under the +.code dwim +operator evaluation rules. -Because of the type change, -.code target-obj -implicitly loses all of its original static slots, -and acquires those of -.codn "source obj" . +.TP* Example: -Note that finalizers registered against -.meta target-obj -are not invoked, and remain registered. -If -.meta target-obj -has state which is cleaned up by -finalizers, it is advisable to invoke them using -.code call-finalizers -prior to using -.codn replace-struct , -or to take other measures to handle the situation. +.verb + ;; struct for counting atoms eq to key + (defstruct (counter key) nil + key + (count 0) + (:method increment (self key) + (if (eq self.key key) + (inc self.count)))) -If the -.meta target-obj -and -.meta source-obj -arguments are the same object, -.code replace-struct -has no effect. + ;; pass all atoms in tree to func + (defun map-tree (tree func) + (if (atom tree) + [func tree] + (progn (map-tree (car tree) func) + (map-tree (cdr tree) func)))) -The return value is -.metn target-obj . + ;; count occurrences of symbol a + ;; using increment method of counter, + ;; passed as func argument to map-tree. + (let ((c (new (counter 'a))) + (tr '(a (b (a a)) c a d))) + (map-tree tr (meth c increment)) + c) + --> #S(counter key a count 4 + increment #) +.brev -.coNP Function @ method +.coNP Macro @ umeth .synb -.mets (method < struct-obj < slot-name << curried-arg *) +.mets (umeth < slot << curried-expr *) .syne .desc The -.code method -function retrieves a function -.meta m -from a structure's slot -and returns a new function which binds that function's -left argument. If -.meta curried-arg -arguments are present, then they are also stored in -the returned function. These are the -.IR "curried arguments" . +.code umeth +macro binds the symbol +.meta slot +to a function and returns that function. The -.meta struct-obj -argument must be a structure, and -.meta slot-name -must be a symbol denoting a slot in that structure. -The slot must hold a function of at least one -argument. - -The function -.meta f -which -.code method -function returns, when invoked, -calls the function -.meta m -previously retrieved from the object's -slot, passing to that function -.meta struct-obj -as the leftmost argument, followed by the curried -arguments, followed by all of -.metn f 's -own arguments. +.meta curried-expr +arguments, if present, are evaluated as if they were +arguments to the +.code dwim +operator. -Note: the -.code meth -macro is an alternative interface which is suitable if -the slot name isn't a computed value. +When that function is called, it expects at least one argument. +The leftmost argument must be an object of struct type. -.coNP Function @ super-method -.synb -.mets (super-method < struct-obj << slot-name ) -.syne -.desc -The -.code super-method -function retrieves a function from a static -slot belonging to the supertype of the structure type of -.metn struct-obj . +The slot named +.meta slot +is retrieved from that object, and is expected to be a function. +That function is called with the object, followed by the values +of the +.metn curried-expr -s, +if any, followed by that function's arguments. -It then returns a function which binds -that function's left argument to the structure. +The syntax can be understood as a translation to a call of the +.code umethod +function: -The -.meta struct-obj -argument must be a structure which has a supertype, and -.meta slot-name -must be a symbol denoting a static slot in that supertype. -The slot must hold a function of at least one -argument. +.verb + (umeth s ...) <--> [umethod 's ...] +.brev -The -.code super-method -function returns a function which, when invoked, -calls the function previously retrieved from -the supertype's static slot, passing to that function -.meta struct-obj -as the leftmost argument, followed by the function's -own arguments. +The macro merely provides the syntactic sugar of not having to quote the +symbol, and automatically treating the curried argument expressions +using Lisp-1 semantics of the +.code dwim +operator. -.coNP Function @ umethod -.synb -.mets (umethod << slot-name << curried-arg *) -.syne -.desc -The -.code umethod -returns a function which represents the set of all methods named by -the slot -.meta slot-name -in all structure types, including ones not yet defined. -The -.meta slot-name -argument must be a symbol. +.TP* Example: -If one or more -.meta curried-arg -argument are present, these values represent the -.I "curried arguments" -which are stored in the function object which is returned. +.verb + ;; seal and dog are variables which hold structures of + ;; different types. Both have a method called bark. -This returned function must be called with at least one argument. Its leftmost -argument must be an object of structure type, which has a slot named -.metn slot-name . -The function will retrieve the value of the slot from that object, -expecting it to be a function, and calls it, passing to it the following -arguments: the object itself; all of the curried arguments, if any; and -all of its remaining arguments. + (let ((bark-fun (umeth bark))) + [bark-fun dog] ;; same effect as dog.(bark) + [bark-fun seal]) ;; same effect as seal.(bark) +.brev -Note: the -.code umethod -name stands for "unbound method". Unlike the -.code method -function, -.code umethod -doesn't return a method whose leftmost argument is already bound to -an object; the binding occurs at call time. +The +.code u +in +.code umeth +stands for "unbound". The function produced by +.code umeth +is not bound to any specific object; it binds to an object whenever it is +invoked by retrieving the actual method from the object's slot at call time. -.coNP Function @ uslot +.coNP Macro @ usl .synb -.mets (uslot << slot-name ) +.mets (usl << slot ) .syne .desc The -.code uslot -returns a function which represents all slots named -.meta slot-name -in all structure types, including ones not yet defined. -The -.meta slot-name -argument must be a symbol. +.code usl +macro binds the symbol +.meta slot +to a function and returns that function. -The returned function must be called with exactly one argument. -The argument must be a structure which has a slot named -.metn slot-name . -The function will retrieve the value of the slot from that object -and return it. +When that function is called, it expects exactly one argument. +That argument must be an object of struct type. +The slot named +.meta slot +is retrieved from that object and returned. -Note: the -.code uslot -name stands for "unbound slot". The returned function -isn't bound to a particular object. The binding of -.code slot-name -to a slot in the structure object occurs when the function is called. +The name +.code usl +stands for "unbound slot". The term "unbound" refers to the returned +function not being bound to a particular object. The binding of the +slot to an object takes place whenever the function is called. -.coNP Function @ slots +.coNP Function @ make-struct-type .synb -.mets (slots << type ) +.mets (make-struct-type < name < super < static-slots < slots +.mets \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ < static-initfun < initfun << boactor +.mets \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ < boactor << postinitfun ) .syne .desc The -.code slots -function returns a list of all of the slots of struct type -.metn type . +.code make-struct-type +function creates a new struct type. The -.meta type -argument must be a structure type, or else a symbol -which names a structure type. -.coNP Function @ slotp -.synb -.mets (slotp < type << name ) -.syne -.desc -The -.code slotp -function returns -.code t -if name .meta name -is a symbol which names a slot in the structure type -.metn type . -Otherwise it returns -.codn nil . - -The -.meta type -argument must be a structure type, or else a symbol -which names a structure type. +argument must be a bindable symbol, according to the +.code bindable +function. It specifies the name property of the struct +type as well as the name under which the struct type +is globally registered. -.coNP Function @ static-slot-p -.synb -.mets (static-slot-p < type << name ) -.syne -.desc The -.code static-slot-p -function returns -.code t -if name -.meta name -is a symbol which names a slot in the structure type -.metn type , -and if that slot is a static slot. -Otherwise it returns -.codn nil . +.meta super +argument indicates the supertype for the struct type. +It must be either a value of type +.codn struct-type , +a symbol which names a struct type, or else +.codn nil , +indicating that the newly created struct type has no supertype. The -.meta type -argument must be a structure type, or else a symbol -which names a structure type. +.meta static-slots +argument is a list of symbol which specify static slots. +The symbols must be bindable and the list must not contain duplicates. -.coNP Function @ static-slot -.synb -.mets (static-slot < type << name ) -.syne -.desc The -.code static-slot -function retrieves the value of the static slot -named by symbol -.meta name -of the structure type -.metn type . +.meta slots +argument is a list of symbols which specifies the instance slots. +The symbols must be bindable and there must not be any duplicates +within the list, or against entries in the +.meta static-slots +list. -The -.meta type -argument must be a structure type or a symbol which names a -structure type, and -.meta name -must be a static slot of this type. +The new struct type's effective list of slots is formed by appending +together +.meta static-slots +and +.metn slots , +and then appending that to the list of the supertype's slots, and +de-duplicating the resulting list as if by the +.code uniq +function. Thus, any slots which are already present in the supertype are +removed. If the structure has no supertype, then the list of supertype +slots is taken to be empty. When a structure is instantiated, it shall have all +the slots specified in the effective list of slots. Each instance slot +shall be initialized to the value +.codn nil , +prior to the invocation of +.meta initfun +and +.metn boactor . -.coNP Function @ static-slot-set -.synb -.mets (static-slot-set < type < name << new-value ) -.syne -.desc The -.code static-slot-set -function stores -.meta new-value -into the static slot named by symbol -.meta name -of the structure type -.metn type . +.meta static-initfun +argument either specifies an initialization function, or is +.codn nil , +which is equivalent to specifying a function which does nothing. -It returns -.metn new-value . +Prior to the invocation of +.metn static-initfun , +each new static slot shall be initialized the value +.codn nil . +Inherited static slots retain their values from the supertype. -The -.meta type -argument must be a structure type or the name of a structure type, and -.meta name -must be a static slot of this type. +If specified, +.meta static-initfun +function must +accept one argument. When the structure type is created (before +the +.code make-struct-type +function returns) the +.meta static-initfun +function is invoked, passed the newly created +structure type as its argument. -.coNP Function @ static-slot-ensure -.synb -.mets (static-slot-ensure < type < name < new-value <> [ no-error-p ]) -.syne -.desc The -.code static-slot-ensure -ensures, if possible, that the struct type -.metn type , -as well as possibly one or more struct types derived from it, -have a static slot called -.metn name , -that this slot is not shared with a supertype, -and that the value stored in it is -.metn new-value . - -Note: this function supports the redefinition of methods, -as the implementation underlying the -.code defmeth -macro; its semantics is designed to harmonize with expected -behaviors in that usage. +.meta initfun +argument either specifies an initialization function, or is +.codn nil , +which is equivalent to specifying a function which does nothing. +If specified, this function must +accept one argument. When a structure is instantiated, every +.meta initfun +in its chain of supertype ancestry is invoked, in order of inheritance, +so that the root supertype's +.meta initfun +is called first and the structure's own specific +.meta initfun +is called last. These calls occur before the slots are initialized +from the +.meta arg +arguments +or the +.meta slot-init-plist +of +.codn make-struct . +Each function is passed the newly created structure +object, and may alter its slots. -The function operates as follows. +The +.meta boactor +argument either specifies a by-order-of-arguments initialization +function ("boa constructor") or is +.codn nil , +which is equivalent to specifying a constructor which does nothing. +If specified, it must be a function which takes at least one argument. +When a structure is instantiated, and boa arguments are given, the +.meta boactor +is invoked, with the structure as the leftmost argument, and +the boa arguments as additional arguments. This takes place +after the processing of +.meta initfun +functions, and after the processing of the +.meta slot-init-plist +specified in the +.code make-struct +call. Note that the +.meta boactor +functions of the supertypes are not called, only the +.meta boactor +specific to the type being constructed. -If -.meta type -itself already has an instance slot called -.meta name -then an error is thrown, and the function has no effect, unless a -true argument is specified for the -.meta no-error-p -Boolean parameter. In that case, in the same situation, the function -has no effect and simply returns -.metn new-value . +The +.meta postinitfun +argument either specifies an initialization function, or is +.codn nil , +which is equivalent to specifying a function which does nothing. +If specified, this function must accept one argument. +The +.meta postinitfun +function is similar to +.metn initfun . +The difference is that +.meta postinitfun +functions are called after all other initialization processing, +rather than before. They are are also called in order of +inheritance: the +.meta postinitfun +of a structure's supertype is called before its own. -If -.meta type -already has a non-inherited static slot called -.meta name -then this slot is overwritten with -.meta new-value -and the function returns -.metn new-value . -Types derived from -.meta type -may also have this slot, via inheritance; consequently, its value -changes in those types also. +.coNP Function @ find-struct-type +.synb +.mets (find-struct-type << name ) +.syne +.desc +The +.code find-struct-type +returns a +.code struct-type +object corresponding to the symbol +.metn name . -If -.meta type -already has an inherited static slot called -.meta name -then its inheritance is severed; the slot is converted -to a non-inherited static slot of -.meta type -and initialized with -.metn new-value . -Then all struct types derived from -.meta type -are scanned. In each such type, if the original inherited -static slot is found, it is replaced with the same -newly converted static slot that was just introduced into -.metn type , -so that all these types now inherit this new slot from -.meta type -rather than the original slot from some supertype of -.metn type . -These types all share a single instance of the slot with -.metn type , -but not with supertypes of -.metn type . +If no struct type is registered under +.metn name , +then it returns +.codn nil . -In the remaining case, -.meta type -has no slot called -.metn name . -The slot is added as a static slot to -.metn type . -Then it is added to every struct type derived from -.meta type -which does not already have a slot by that name, as if -by inheritance. That is to say, types to which this slot is introduced share a -single instance of that slot. The value of the new slot is -.metn new-value , -which is also returned from the function. Any subtypes of -.meta type -which already have a slot called -.meta name -are ignored, as are their subtypes. +A +.code struct-type +object exists for each structure type and holds information about it. +These objects are not themselves structures and are all of the +same type, +.codn struct-type . -.coNP Function @ static-slot-home +.coNP Function @ struct-type-p .synb -.mets (static-slot-home < type << name ) +.mets (struct-type-p << obj ) .syne .desc The -.code static-slot-home -method determines which structure type actually defines the -static slot -.meta name -present in struct type -.metn type . - -If -.meta type -isn't a struct type, or the name of a struct type, -the function throws an error. Likewise, if -.meta name -isn't a static slot of -.metn type . +.code struct-type-p +function returns t if +.meta obj +is a structure type, otherwise it returns +.codn nil . -If -.meta name -is a static slot of -.meta type -then the function returns a struct type name symbol which is either -then name of -.meta type -itself, if the slot is defined specifically for -.meta type -or else the most distant ancestor of -.meta type -from which the slot is inherited. +A structure type is an object of type +.codn struct-type , +returned by +.codn find-struct-type . -.coNP Function @ call-super-method +.coNP Function @ struct-type-name .synb -.mets (call-super-method < struct-obj < name << argument *) +.mets (struct-type-name << type ) .syne .desc The -.code call-super-method -function is deprecated. Solutions involving -.code call-super-method -should be reworked in terms of -.codn call-super-fun . +.code struct-type-name +function returns the symbol which serves as the name of +.metn type , +which must be either a struct type object (such as the return value of +a successful lookup via +.codn find-struct-type ), +or else a struct type name. +.coNP Function @ super +.synb +.mets (super << type ) +.syne +.desc The -.code call-super-method -retrieves the function stored in the slot -.meta name -of the supertype of -.meta struct-obj -and invokes it, passing to that function -.meta struct-obj -as the leftmost argument, followed by the given -.metn argument -s, -if any. +.code super +function returns the struct type object which is the +supertype of +.metn type , +or returns +.code nil +if +.meta type +has no supertype. The -.meta struct-obj -argument must be of structure type. Moreover, -that structure type must be derived from another structure type, -and -.meta name -must name a static slot of that structure type. - -The object retrieved from that static slot must be -callable as a function, and accept the arguments. - -Note that it is not correct for a method that is defined -against a particular type to use -.code call-super-method -to call the same method (or any other method) in the supertype -of that particular type. This is because -.code call-super-method -refers to the type of the object instance -.metn struct-obj , -not to the type against which the calling method is defined. +.meta type +argument must be either a struct type object, a +a symbol which names a struct type (which is resolved to that type), +or else a structure instance (which is resolved to its structure type). -.coNP Function @ call-super-fun +.coNP Function @ make-struct .synb -.mets (call-super-fun < type < name << argument *) +.mets (make-struct < type < slot-init-plist << arg *) .syne .desc The -.code call-super-fun -retrieves the function stored in the slot -.meta name -of the supertype of -.meta type -and invokes it, passing to that function the given -.metn argument -s, -if any. +.code make-struct +function returns a new object which is an instance of the +structure type +.metn type . The .meta type -argument must be a structure type. Moreover, -that structure type must be derived from another structure type, -and -.meta name -must name a static slot of that structure type. - -The object retrieved from that static slot must be -callable as a function, and accept the arguments. +argument must either be a +.code struct-type +object, or else a symbol which is the name of a structure. -.TP* Example: +The +.meta slot-init-plist +argument gives a list of slot initializations in the +style of a property list, as defined by the +.code prop +function. It may be empty, in which case +it has no effect. Otherwise, it specifies slot names +and their values. Each slot name which is given must +be a slot of the structure type. The corresponding value +will be stored into the slot of the newly created object. If a slot is +repeated, it is unspecified which value takes effect. -Print a message and call supertype method: +The optional +.metn arg -s +specify arguments to the structure type's boa constructor. +If the arguments are omitted, the boa constructor is not invoked. +Otherwise the boa constructor is invoked on the structure object +and those arguments. The argument list must match the trailing parameters of +the boa constructor (the remaining parameters which follow the leftmost +argument which passes the structure to the boa constructor). -.verb - (defstruct base nil) +When a new structure is instantiated by +.codn make-struct , +its slot values are first initialized by the structure type's registered +functions as described under +.codn make-struct-type . +Then, the +.meta slot-init-plist +is processed, if not empty, and finally, the +.metn arg -s +are processed, if present, and passed to the boa constructor. - (defstruct derived base) +If any of the initializations abandon the evaluation of +.code make-struct +by a non-local exit such as an exception throw, the object's +finalizers, if any, are invoked. - (defmeth base fun (obj arg) - (format t "base fun method called with arg ~s\en" arg)) +.coNP Function @ make-lazy-struct +.synb +.mets (make-lazy-struct < type << argfun ) +.syne +.desc +The +.code make-lazy-struct +function returns a new object which is an instance of the +structure type +.metn type . - (defmeth derived fun (obj arg) - (format t "derived fun method called with arg ~s\en" arg) - (call-super-fun 'derived 'fun obj arg)) +The +.meta type +argument must either be a +.code struct-type +object, or else a symbol which is the name of a structure. - ;; Interactive Listener: - 1> (new derived).(fun 42) - derived fun method called with arg 42 - base fun method called with arg 42 -.brev +The +.meta argfun +argument should be a function which can be called with no parameters +and returns a cons cell. More requirements are specified below. -Note that a static method or function in any structure type -can be invoked by using the -.code "(meth ...)" -name syntax in the first position of a compound form, as -a function name. Thus, the above -.code "derived fun" -can also be written: +The object returned by +.code make-lazy-struct +is a lazily-initialized struct instance, or +.IR "lazy struct" . -.verb - (defmeth derived fun (obj arg) - (format t "derived fun method called with arg ~s\en" arg) - ((meth base fun) obj arg)) -.brev +A lazy struct remains uninitialized until just before the first access +to any of its instance slots. Just before an instance slot is +accessed, initialization +takes place as follows. The +.meta argfun +function is invoked with no arguments. Its return value must be a cons +cell. The +.code car +of the cons cell is taken to be a property list, as defined by the +.code prop +function. The +.code cdr +field is taken to be a list of arguments. These values are treated +as if they were, respectively, the +.meta slot-init-plist +and the boa constructor arguments given in a +.code make-struct +invocation. Initialization of the structure proceeds as described +in the description of +.codn make-struct . -.coNP Functions @ struct-get-initfun and @ struct-get-postinitfun +.coNP Functions @ struct-from-plist and @ struct-from-args .synb -.mets (struct-get-initfun << type ) -.mets (struct-get-postinitfun << type ) +.mets (struct-from-plist < type >> { slot << value }*) +.mets (struct-from-arg < type << arg *) .syne .desc The -.code struct-get-initfun -and -.code struct-get-postinitfun -functions retrieve, respectively, a structure type's -.meta initfun +.code struct-from-plist and -.meta postinitfun -functions. These are the functions which are initially configured in the call to -.code make-struct-type -via the -.meta initfun +.code struct-from-arg +are interfaces to the +.code make-struct +function. + +The +.code struct-from-plist +function passes its +.meta slot and -.meta postinitfun -arguments. +.meta value +arguments as the +.meta slot-init-plist +argument of +.codn make-struct . +It passes no boa constructor arguments. -Either one may be -.codn nil , -indicating that the type has no -.meta initfun -or -.metn postinitfun . +The +.code struct-from-plist +function calls +.meta make-struct +with an empty +.metn slot-init-plist , +passing down the list of +.metn arg -s. -.coNP Functions @ struct-set-initfun and @ struct-set-postinitfun +The following equivalences hold: + +.verb + (struct-from-plist a s0 v0 s1 v1 ...) + <--> (make-struct a (list s0 v0 s1 v1 ...)) + + (struct-from-args a v0 v1 v2 ...) + <--> (make-struct a nil v0 v1 v2 ...) +.brev + +.coNP Function @ allocate-struct .synb -.mets (struct-set-initfun < type << function ) -.mets (struct-set-postinitfun < type << function ) +.mets (allocate-struct << type ) .syne .desc The -.code struct-set-initfun -and -.code struct-set-postinitfun -functions overwrite, respectively, a structure type's -.meta initfun -and -.meta postinitfun -functions. These are the functions which are initially configured in the call to -.code make-struct-type -via the -.meta initfun -and -.meta postinitfun -arguments. +.code allocate-struct +provides a low-level allocator for structure objects. The -.meta function -argument must either be -.code nil -or else a function which accepts one argument. +.meta type +argument must either be a +.code struct-type +object, or else a symbol which is the name of a structure. -Note that -.meta initfun -has the responsibility for all instance slot initializations. The -.code defstruct -syntax compiles the initializing expressions in the slot specifier syntax -into statements which are placed into a function, which becomes the -.meta initfun -of the struct type. +The +.code allocate-struct +creates and returns a new instance of +.meta type +all of whose instance slots take on the value +.codn nil . +No initializations are performed. The struct type's +registered initialization functions are not invoked. -.coNP Macro @ with-objects +.coNP Function @ copy-struct .synb -.mets (with-objects >> ({( sym << init-form )}*) << body-form *) +.mets (copy-struct << struct-obj ) .syne .desc The -.code with-objects -macro provides a binding construct very similar to -.codn let* . +.code copy-struct +function creates and returns a new object which is a duplicate of +.metn struct-obj , +which must be a structure. -Each -.meta sym -must be a symbol suitable for use as a variable name. +The duplicate object is a structure of the same type as +.meta struct-obj +and has the same slot values. -Each -.meta init-form -is evaluated in sequence, and a binding is established for its -corresponding -.meta sym -which is initialized with the value of that form. The binding -is visible to subsequent -.metn init-form -s. +The creation of a duplicate does not involve calling any of the +struct type's initialization functions. -Additionally, the values of the -.metn init-form -s -are noted as they are produced. When the -.code with-objects -form terminates, by any means, the -.code call-finalizers -function is invoked on each value which was returned by an -.meta init-form -and had been noted. These calls are performed in the -reverse order relative to the original evaluation of the forms. - -After the variables are established and initialized, the -.metn body-form -s -are evaluated in the scope of the variables. The value of the -last form is returned, or else -.code nil -if there are no forms. The invocations of -.code call-finalizers -take place just before the value of the last form is returned. - -.SS* Special Structure Functions - -Special structure functions are user-defined methods or structure functions -which are specially recognized by certain functions in \*(TL. They endow -structure objects with the ability to participate in certain usage scenarios, -or to participate in a customized way. - -Special functions are required to bound to static slots, which is the -case if the -.code defmeth -macro is used, or when methods or functions are defined using syntax -inside a -.code defstruct -form. If a special function or method is defined as an instance slot, -then the behavior of library functions which depend on this method is -unspecified. +Only instance slots participate in the duplication. Since +the original structure and copy are of the same structure type, +they already share static slots. -Special functions introduced below by the word "Method" receive an object -instance as an argument. Their syntax is indicated using the same notation -which may be used to invoke them, such as: +This is a low-level, "shallow" copying mechanism. If an object design +calls for a higher level cloning mechanism with deep copying or other +additional semantics, one can be built on top of +.codn copy-struct . +For instance, a structure can have a +.code copy +method similar to the following: .verb -.mets << object .(function-name < arg << ... ) + (:method copy (me) + (let ((my-copy (copy-struct me))) + ;; inform the copy that it has been created + ;; by invoking its copied method. + my-copy.(copied) + my-copy)) .brev -However, those introduced as "Function" do not operate on an instance. For -brevity, their syntax is nevertheless exemplified as - -.verb -.meti << object .'['function-name < arg << ... ']' -.brev +since this logic is generic, it can be placed in a base +method. The +.code copied +method which it calls is the means by which the new object is notified that it +is a copy. This method takes on whatever special responsibilities are required +when a copy is produced, such as registering the object in various necessary +associations, or performing a deeper copy of some of the objects held +in the slots. -If such a invocation is actually used, the -.meta object -instance only serves for identifying the struct type whose static slot -.code function-name -provides the function; -.meta object -doesn't participate in the call. An object is not required since -the function can be called using +The +.code copied +handler can be implemented at multiple levels of an inheritance hierarchy. The +initial call to +.code copied +from +.code copy +will call the most derived override of that method. -.verb -.meti [(static-slot < type 'function-name) < arg << ... ] -.brev +To call the corresponding method in the base class, a given derived method +can use the +.code call-super-fun +function, or else the +.code "(meth ...)" +syntax in the first position of a compound form, in place of a function name. +Examples of both are given in the documentation for +.codn call-super-fun . -which looks up the function in the struct -.meta type -directly. +Thus derived structs can inherit the copy handling logic from base structs, and +extend it with their own. -.coNP Method @ equal +.coNP Accessor @ slot .synb -.mets << object .(equal) +.mets (slot < struct-obj << slot-name ) +.mets (set (slot < struct-obj << slot-name ) << new-value ) .syne .desc -Normally, two struct values are not considered the same under the -.code equal -function unless they are the same object. - -However, if the -.code equal -method is defined for a structure type, then instances of -that structure type support -.IR "equality substitution" . - The -.code equal -method must not require any arguments other than -.metn object . -Moreover, the method must never return -.codn nil . - -When a struct which supports equality substitution is compared using -.codn equal , -.code less -or -.codn greater , -its -.code equal -method is invoked, and the return value is used in place of that -structure for the purposes of the comparison. +.code slot +function retrieves a structure's slot. The +.meta struct-obj +argument must be a structure, and +.meta slot-name +must be a symbol which names a slot in that structure. -The same applies when an struct is hashed using the -.code hash-equal -function, or implicitly by an -.code :equal-hash -hash table. +Because +.code slot +is an accessor, a +.code slot +form is a syntactic place which denotes the +slot's storage location. -Note: if an -.code equal -method is defined or redefined with different semantics for a struct -type whose instances have already been inserted as keys in an -.code :equal-based -hash table, the behavior of subsequent insertion and lookup operations -on that hash table becomes unspecified. +A syntactic place expressed by +.code slot +does not support deletion. -.coNP Method @ print +.coNP Function @ slotset .synb -.mets << object .(print < stream << pretty-p ) +.mets (slotset < struct-obj < slot-name << new-value ) .syne .desc -If a method named by the symbol -.code print -is defined for a structure type, then it is used for printing instances -of that type. - The -.meta stream -argument specifies the output stream to which the printed representation -is to be written. +.code slotset +function stores a value in a structure's slot. + The +.meta struct-obj +argument must be a structure, and +.meta slot-name +must be a symbol which names a slot in that structure. The -.meta pretty-p -argument is a Boolean flag indicating whether pretty-printing -is requested. Its value may simply be passed to recursive calls to -.codn print , -or used to select between -.code ~s -or -.code ~a -formatting if -.code format -is used. +.meta new-value +argument specifies the value to be stored in the slot. -The value returned by the -.code print -method is significant. If the special keyword symbol -.code : -(colon) is returned, then the system will print the object -in the default way, as if no -.code print -method existed: it is understood that the method declined -the responsibility for printing the object. +If a successful store takes place to an instance slot of +.metn struct-obj , +then the dirty flag of that object is set, causing the +.code test-dirty +function to report true for that object. -If any other value is returned, then it is understood -that the method -.code print -method accepted the responsibility for printing the object, -and the system consequently will generate into -.meta stream -any output output pertaining to -.metn object 's -representation. +The +.code slotset +function returns +.metn new-value . -.coNP Method @ lambda +.coNP Functions @, test-dirty @ clear-dirty and @ test-clear-dirty .synb -.mets << object .(lambda << arg *) +.mets (test-dirty << struct-obj ) +.mets (clear-dirty << struct-obj ) +.mets (test-clear-dirty << struct-obj ) .syne .desc -If a structure type provides a method called -.code lambda -then it can be used as a function. +The +.codn test-dirty , +.code clear-dirty +and +.code test-clear-dirty +functions comprise the interface for interacting with structure +dirty flags. -Of course, this method can be called by name, using the syntax given -in the above syntactic description. +Each structure instance has a dirty flag. When this flag is set, the +structure instance is said to be dirty, otherwise it is said to be clean. A +newly created structure is dirty. A structure remains dirty until its dirty +flag is explicitly reset. If a structure is clean, and one of its instance +slots is overwritten with a new value, it becomes dirty. -However, the intended use is that it allows the structure instance itself to be -used as a function. When arguments are applied to a structure object as if it -were a function, this is erroneous, unless that object has a -.code lambda -method. In that case, the arguments are passed to the lambda method. -Of course, the leftmost argument of the method is the structure instance +The +.code test-dirty +function returns the dirty flag of +.metn struct-obj : +.code t +if +.meta struct-obj +is dirty, otherwise +.codn nil . + +The +.code clear-dirty +function clears the dirty flag of +.meta struct-obj +and returns +.meta struct-obj itself. -That is to say, the following equivalences apply, except that -.code s -is evaluated only once: +The +.code test-clear-dirty +flag combines these operations: it makes a note of the dirty flag of +.meta struct-obj +and clears it. Then it returns the noted value, +.code t +or +.codn nil . -.verb - (call s args ...) <--> s.(lambda args ...) +.coNP Function @ structp +.synb +.mets (structp << obj ) +.syne +.desc +The +.code structp +function returns t if +.meta obj +is a structure, otherwise it returns +.codn nil . - [s args ...] <--> [s.lambda s args ...] +.coNP Function @ struct-type +.synb +.mets (struct-type << struct-obj ) +.syne +.desc +The +.code struct-type +function returns the structure type object which +represents the type of the structure object instance +.metn struct-obj . - (mapcar s list) <--> (mapcar (meth s lambda) list) -.brev +.coNP Function @ clear-struct +.synb +.mets (clear-struct < struct-obj <> [ value ]) +.syne +.desc +The +.code clear-struct +replaces all instance slots of +.meta struct-obj +with +.metn value , +which defaults to +.code nil +if omitted. -Note: a form such as -.code "[s args ...]" -where -.code s -is a structure can be treated as a place if the method -.code lambda-set -is also implemented. +Note that finalizers are not executed prior to replacing +the slot values. -.coNP Method @ lambda-set +.coNP Function @ reset-struct .synb -.mets << object .(lambda-set << arg * << new-value ) +.mets (reset-struct << struct-obj ) .syne .desc The -.code lambda-set -method, in conjunction with a -.code lambda -method, allows structures to be used as place accessors. If -structure -.code s -supports a -.meta lambda-set -with four arguments, then the following use of the -.code dwim -operator is possible: +.code reset-struct +function reinitializes the structure object +.meta struct-obj +as if it were being newly created. +First, all the slots are set to +.code nil +as if by the +.code clear-struct +function. Then the slots are initialized by invoking the +initialization functions, in order of the supertype ancestry, just as would be +done for a new structure object created by +.code make-struct +with an empty +.meta slot-init-plist +and no boa arguments. -.verb - (set [s a b c d] v) - (set (dwim s a b c d) v) ;; precisely equivalently -.brev +Note that finalizers registered against +.meta struct-obj +are not invoked prior to the reset operation, and remain registered. -This has an effect which can be described by the following code: +If the structure has state which is cleaned up by +finalizers, it is advisable to invoke them using +.code call-finalizers +prior to using +.codn reset-struct , +or to take other measures to deal with the +situation. -.verb - (progn - s s.(lambda-set a b c d v) - v) -.brev +If the structure specifies +.code :fini +handlers, then the reinitialization will cause +these to registered, just like when a new object +it constructed. Thus if +.code call-finalizers +is not used prior to +.codn reset-struct , +this will result in the existence of duplicate registrations of the +finalization functions. -except that -.code s -and -.code v -are evaluated only once, and -.code a -through -.code d -are evaluated using the Lisp-1 semantics due the -.code dwim -operator. +Finalizers registered against +.meta struct-obj +.B are +invoked if an exception is thrown +during the reinitialization, just like when a new +structure is being constructed. -If a place-mutating operator is used on this form which requires the prior -value, such as the -.code inc -macro, then the structure must support the -.code lambda -function also. +.coNP Function @ replace-struct +.synb +.mets (replace-struct < target-obj << source-obj ) +.syne +.desc +The +.code replace-struct +function causes +.meta target-obj +to take on the attributes of +.meta source-obj +without changing its identity. -If -.code lambda -takes -.I n -arguments, then -.code lambda-set -should take -.I n+1 -arguments. The first -.I n -arguments of these two methods are congruent; the extra rightmost argument -of -.code lambda-set -is the new value to be stored into the place denoted by the prior -arguments. +The type of +.code target-obj +is changed to that of +.codn source-obj . -The return value of -.code lambda-set -is ignored. +All instance slots of +.code target-obj +are discarded, and it is given new slots, +which are copies of the instance slots of +.codn source-obj . -Note: the -.code lambda-set -method is also used by the -.code rplaca -function, if no -.code rplaca -method exists. +Because of the type change, +.code target-obj +implicitly loses all of its original static slots, +and acquires those of +.codn "source obj" . -.TP* Example +Note that finalizers registered against +.meta target-obj +are not invoked, and remain registered. +If +.meta target-obj +has state which is cleaned up by +finalizers, it is advisable to invoke them using +.code call-finalizers +prior to using +.codn replace-struct , +or to take other measures to handle the situation. -The following defines a structure with a single instance -slot -.code hash -which holds a hash table, as well as -.code lambda +If the +.meta target-obj and -.code lambda-set -methods: - -.verb - (defstruct hash-wrapper nil - (hash (hash)) - - (:method lambda (self key) - [self.hash key]) - - (:method lambda-set (self key new-val) - (set [self.hash key] new-val) self)) -.brev - -An instance of this structure can now be used as follows: +.meta source-obj +arguments are the same object, +.code replace-struct +has no effect. -.verb - (let ((s (new hash-wrapper))) - (set [s "apple"] 3 - [s "orange] 4) - [s "apple"]) -> 3 -.brev +The return value is +.metn target-obj . -.coNP Method @ length +.coNP Function @ method .synb -.mets << object .(length) +.mets (method < struct-obj < slot-name << curried-arg *) .syne .desc -If a structure has -.code length -method, then it can be used as an argument to the -.code length -function. - -Structures which implement the methods -.codn lambda , -.code lambda-set -and -.code length -can be treated as abstract vector-like sequences, because such -structures support the -.codn ref , -.code refset -and -.code length -functions. +The +.code method +function retrieves a function +.meta m +from a structure's slot +and returns a new function which binds that function's +left argument. If +.meta curried-arg +arguments are present, then they are also stored in +the returned function. These are the +.IR "curried arguments" . -For instance, the -.code nreverse -function will operate on such objects. +The +.meta struct-obj +argument must be a structure, and +.meta slot-name +must be a symbol denoting a slot in that structure. +The slot must hold a function of at least one +argument. -Note: a structure which supports the -.code car -method also supports the -.code length -function, in a different way. Such a structure is treated by -.code length -as a list-like sequence, and its length is measured by walking the -sequence with -.code cdr -operations. If a structure supports both -.code length -and -.codn car , -preference is given to -.codn length , -which is likely to be much more efficient. +The function +.meta f +which +.code method +function returns, when invoked, +calls the function +.meta m +previously retrieved from the object's +slot, passing to that function +.meta struct-obj +as the leftmost argument, followed by the curried +arguments, followed by all of +.metn f 's +own arguments. -.coNP Methods @, car @ cdr and @ nullify +Note: the +.code meth +macro is an alternative interface which is suitable if +the slot name isn't a computed value. + +.coNP Function @ super-method .synb -.mets << object .(car) -.mets << object .(cdr) -.mets << object .(nullify) +.mets (super-method < struct-obj << slot-name ) .syne .desc -Structures may be treated as sequences if they define methods named -by the symbols -.codn car , -.codn cdr , -and -.codn nullify . - -If a structure supports these methods, then these methods are used -by the functions -.codn car , -.codn cdr , -.codn nullify , -.code empty -and various other sequence manipulating functions derived from them, when those -functions are applied to that object. - -An object which implements these three methods can be considered to represent a -.I list-like -abstract sequence. +The +.code super-method +function retrieves a function from a static +slot belonging to the supertype of the structure type of +.metn struct-obj . -The object's -.code car -method should return the first value in that abstract sequence, or else -.code nil -if that sequence is empty. +It then returns a function which binds +that function's left argument to the structure. -The object's -.code cdr -method should return an object denoting the remainder of the sequence, -or else -.code nil -if the sequence is empty or contains only one value. This returned object can -be of any type: it may be of the same structure type as that object, a -different structure type, a list, or whatever else. If a non-sequence object -is returned. +The +.meta struct-obj +argument must be a structure which has a supertype, and +.meta slot-name +must be a symbol denoting a static slot in that supertype. +The slot must hold a function of at least one +argument. The -.code nullify -method should return -.code nil -if the object is considered to denote an empty sequence. Otherwise it -should either return that object itself, or else return the sequence which -that object represents. +.code super-method +function returns a function which, when invoked, +calls the function previously retrieved from +the supertype's static slot, passing to that function +.meta struct-obj +as the leftmost argument, followed by the function's +own arguments. -.coNP Methods @ rplaca and @ rplacd +.coNP Function @ umethod .synb -.mets << object .(rplaca << new-car-value ) -.mets << object .(rplacd << new-cdr-value ) +.mets (umethod << slot-name << curried-arg *) .syne .desc -If a structure type defines the methods -.code rplaca -and -.code rplacd -then, respectively, the -.code rplaca -and -.code rplacd -functions will use these methods if they are applied to instances of that type. +The +.code umethod +returns a function which represents the set of all methods named by +the slot +.meta slot-name +in all structure types, including ones not yet defined. +The +.meta slot-name +argument must be a symbol. -That is to say, when the function call -.mono -.meti (rplaca < o << v ) -.onom -is evaluated, and -.meta o -is a structure type, the function inquires whether -.meta o -supports a -.code rplaca -method. If so, then, effectively, -.mono -.meti << o . (rplaca << v) -.onom -is invoked. The return value of this method call is ignored; -.code rplaca -returns -.metn o . -The analogous requirements apply to -.code rplacd -in relation to the -.code rplacd -method. +If one or more +.meta curried-arg +argument are present, these values represent the +.I "curried arguments" +which are stored in the function object which is returned. -Note: if the -.code rplaca -method doesn't exist, the -.code rplaca -function falls back on trying to store -.meta new-car-value -by means of the structure type's -.code lambda-set -method, using an index of zero. That is to say, if the type has no -.code rplaca -method, but does have a -.code lambda-set -method, then -.mono -.meti << o . (lambda-set 0 << v) -.onom -is invoked. +This returned function must be called with at least one argument. Its leftmost +argument must be an object of structure type, which has a slot named +.metn slot-name . +The function will retrieve the value of the slot from that object, +expecting it to be a function, and calls it, passing to it the following +arguments: the object itself; all of the curried arguments, if any; and +all of its remaining arguments. -.coNP Function @ from-list +Note: the +.code umethod +name stands for "unbound method". Unlike the +.code method +function, +.code umethod +doesn't return a method whose leftmost argument is already bound to +an object; the binding occurs at call time. + +.coNP Function @ uslot .synb -.mets << object .'['from-list << list ']' +.mets (uslot << slot-name ) .syne .desc -If a -.code from-list -structure function is defined for a structure type, it is called in certain -situations with an argument which is a list object. The function's purpose -is to construct a new instance of the structure type, derived from that -list. +The +.code uslot +returns a function which represents all slots named +.meta slot-name +in all structure types, including ones not yet defined. +The +.meta slot-name +argument must be a symbol. -The purpose of this function is to allow sequence processing operations -such as -.code mapcar -and -.code remove -to operate on a structure object as if it were a sequence, and return a -transformed sequence of the same type. This is analogous to the way such -functions can operate on a vector or string, and return a vector or string. +The returned function must be called with exactly one argument. +The argument must be a structure which has a slot named +.metn slot-name . +The function will retrieve the value of the slot from that object +and return it. -If a structure object behaves as a sequence thanks to providing -.codn car , -.code cdr -and -.code nullify -methods, but does not have a -.code from-list -function, then those sequence-processing operations which return a sequence -will always return a plain list of items. +Note: the +.code uslot +name stands for "unbound slot". The returned function +isn't bound to a particular object. The binding of +.code slot-name +to a slot in the structure object occurs when the function is called. -.coNP Function @ derived +.coNP Function @ slots .synb -.mets << object .'['derived < supertype << subtype ']' +.mets (slots << type ) .syne .desc -If a structure type supports a function called -.metn derived , -this function is called whenever a new type is defined which names -that type as its supertype. +The +.code slots +function returns a list of all of the slots of struct type +.metn type . -The function is called with two arguments which are both struct types. The -.meta supertype -argument gives the type that is being inherited from. +.meta type +argument must be a structure type, or else a symbol +which names a structure type. +.coNP Function @ slotp +.synb +.mets (slotp < type << name ) +.syne +.desc The -.meta subtype -gives the new type that is inheriting from -.metn supertype . - -The function is called at most once for the creation of a given -.metn subtype , -only for its immediate supertype, if and only if that supertype -has defined this function. - -The function is not retroactively invoked if it is defined for -a structure type from which subtypes have already been derived. +.code slotp +function returns +.code t +if name +.meta name +is a symbol which names a slot in the structure type +.metn type . +Otherwise it returns +.codn nil . -Note: the -.meta supertype -parameter exists because the -.code derived -function is itself inherited. If the same version of this function is shared by -multiple structure types due to inheritance, this argument informs the function -which of those types it is being invoked for. +The +.meta type +argument must be a structure type, or else a symbol +which names a structure type. -.SS* Sequence Manipulation -.coNP Function @ seqp +.coNP Function @ static-slot-p .synb -.mets (seqp << object ) +.mets (static-slot-p < type << name ) .syne .desc -The function -.code seqp -returns +The +.code static-slot-p +function returns .code t -if -.meta object -is a sequence, otherwise +if name +.meta name +is a symbol which names a slot in the structure type +.metn type , +and if that slot is a static slot. +Otherwise it returns .codn nil . -A sequence is defined as a list, vector or string. The object -.code nil -denotes -the empty list and so is a sequence. +The +.meta type +argument must be a structure type, or else a symbol +which names a structure type. -.coNP Functions @ length and @ len +.coNP Function @ static-slot .synb -.mets (length << object ) -.mets (len << object ) +.mets (static-slot < type << name ) .syne .desc -If -.meta object -is a sequence, the -.code length -function returns the number of items it -contains. - The -.code len -function is a synonym of -.codn length . - -The function supports these additional types: -.RS -.coIP hash -The value of -.code hash-count -is returned. -.coIP range -The length of the interval -represented by the range is returned. -The length of a range -.code r -is defined as -.codn "(- (to r) (from r))" , -and thus may be negative. -The length of -.code "#R(1 -10)" -is -11 and of -.codn "#R(0.5 3)" , -2.5. -.coIP buf -The buffer length calculated by -.code length-buf -is returned. -.coIP carray -The number of elements in -.meta object -calculated by -.code length-carray -is returned. -.RE +.code static-slot +function retrieves the value of the static slot +named by symbol +.meta name +of the structure type +.metn type . -.IP -For other types, -.code length -throws an error exception. +The +.meta type +argument must be a structure type or a symbol which names a +structure type, and +.meta name +must be a static slot of this type. -.coNP Function @ empty +.coNP Function @ static-slot-set .synb -.mets (empty << object ) +.mets (static-slot-set < type < name << new-value ) .syne .desc -If -.meta object -is a suitable argument for the -.code length -function, then the -.code empty -Returns -.code t -if -.mono -.meti (length << object ) -.onom -is zero, otherwise -.codn nil . +The +.code static-slot-set +function stores +.meta new-value +into the static slot named by symbol +.meta name +of the structure type +.metn type . -If -.meta object -is not a suitable argument for -.codn length , -then -.code empty -throws an error exception. +It returns +.metn new-value . -.coNP Function @ copy +The +.meta type +argument must be a structure type or the name of a structure type, and +.meta name +must be a static slot of this type. + +.coNP Function @ static-slot-ensure .synb -.mets (copy << object ) +.mets (static-slot-ensure < type < name < new-value <> [ no-error-p ]) .syne .desc The -.code copy -function duplicates objects of various supported types: sequences, hashes, -structures and random states. If -.meta object -is -.codn nil , -it -returns -.codn nil . -Otherwise, -.code copy -is equivalent to invoking a more specific copying function according to -the type of the argument, as follows: -.RS -.coIP cons -.meti (copy-list << object ) -.coIP str -.meti (copy-str << object ) -.coIP vec -.meti (copy-vec << object ) -.coIP hash -.meti (copy-hash << object ) -.IP "struct type" -.meti (copy-struct << object ) -.coIP fun -.meti (copy-fun << object ) -.coIP buf -.meti (copy-buf << object ) -.coIP carray -.meti (copy-carray << object ) -.coIP random-state -.meti (make-random-state << object ) -.RE +.code static-slot-ensure +ensures, if possible, that the struct type +.metn type , +as well as possibly one or more struct types derived from it, +have a static slot called +.metn name , +that this slot is not shared with a supertype, +and that the value stored in it is +.metn new-value . -.IP -For all other types of -.metn object , -the invocation is erroneous. +Note: this function supports the redefinition of methods, +as the implementation underlying the +.code defmeth +macro; its semantics is designed to harmonize with expected +behaviors in that usage. -Except in the case when -.meta sequence -is -.codn nil , -.code copy -returns a value that -is distinct from (not -.code eq -to) -.metn sequence . -This is different from -the behavior of -.mono -.meti >> [ sequence 0..t] -.onom -or -.mono -.meti (sub < sequence 0 t) -.onom -which recognize -that they need not make a copy of -.metn sequence , -and just return it. +The function operates as follows. -Note however, that the elements of the returned sequence may be -eq to elements of the original sequence. In other words, copy is -a deeper copy than just duplicating the -.code sequence -value itself, -but it is not a deep copy. +If +.meta type +itself already has an instance slot called +.meta name +then an error is thrown, and the function has no effect, unless a +true argument is specified for the +.meta no-error-p +Boolean parameter. In that case, in the same situation, the function +has no effect and simply returns +.metn new-value . -.coNP Accessor @ sub +If +.meta type +already has a non-inherited static slot called +.meta name +then this slot is overwritten with +.meta new-value +and the function returns +.metn new-value . +Types derived from +.meta type +may also have this slot, via inheritance; consequently, its value +changes in those types also. + +If +.meta type +already has an inherited static slot called +.meta name +then its inheritance is severed; the slot is converted +to a non-inherited static slot of +.meta type +and initialized with +.metn new-value . +Then all struct types derived from +.meta type +are scanned. In each such type, if the original inherited +static slot is found, it is replaced with the same +newly converted static slot that was just introduced into +.metn type , +so that all these types now inherit this new slot from +.meta type +rather than the original slot from some supertype of +.metn type . +These types all share a single instance of the slot with +.metn type , +but not with supertypes of +.metn type . + +In the remaining case, +.meta type +has no slot called +.metn name . +The slot is added as a static slot to +.metn type . +Then it is added to every struct type derived from +.meta type +which does not already have a slot by that name, as if +by inheritance. That is to say, types to which this slot is introduced share a +single instance of that slot. The value of the new slot is +.metn new-value , +which is also returned from the function. Any subtypes of +.meta type +which already have a slot called +.meta name +are ignored, as are their subtypes. + +.coNP Function @ static-slot-home .synb -.mets (sub < sequence >> [ from <> [ to ]]) -.mets (set (sub < sequence >> [ from <> [ to ]]) << new-val ) +.mets (static-slot-home < type << name ) .syne .desc The -.code sub -function extracts a slice from input sequence -.metn sequence . -The slice is -a sequence of the same type as -.metn sequence . - -If the -.meta from -argument is omitted, it defaults to -.codn 0 . -If the -.meta to -parameter is -omitted, it defaults to -.codn t . -Thus -.code "(sub a)" -means -.codn "(sub a 0 t)" . - -The following semantic equivalence exists between a call to the -.code sub -function and -the DWIM-bracket syntax, except that -.code sub -is an ordinary function call form, which doesn't apply the -Lisp-1 evaluation semantics to its arguments: - -.verb - ;; from is not a list - (sub seq from to) <--> [seq from..to] -.brev - -The description of the -.code dwim -operator\(emin particular, the section -on Range Indexing\(emexplains the semantics of the range specification. - -The output sequence may share structure with the input sequence. - -If -.meta sequence -is a -.code carray -object, then the function behaves like -.codn carray-sub . +.code static-slot-home +method determines which structure type actually defines the +static slot +.meta name +present in struct type +.metn type . If -.meta sequence -is a -.code buf -object, then the function behaves like -.codn buf-sub . +.meta type +isn't a struct type, or the name of a struct type, +the function throws an error. Likewise, if +.meta name +isn't a static slot of +.metn type . If -.meta sequence -is a structure, it must support the -.code lambda -method. The -.code sub -operation is transformed into a call to the -.code lambda -method according to the following equivalence: - -.verb - (sub o from to) <--> o.(lambda (rcons from to)) - (sub o : to) <--> o.(lambda (rcons : to)) - (sub o from) <--> o.(lambda (rcons from :)) - (sub o) <--> o.(lambda (rcons : :)) -.brev +.meta name +is a static slot of +.meta type +then the function returns a struct type name symbol which is either +then name of +.meta type +itself, if the slot is defined specifically for +.meta type +or else the most distant ancestor of +.meta type +from which the slot is inherited. -That is to say, the -.meta from -and -.code to -arguments are converted to range object. If either argument -is missing, the symbol -.code : -is used for the corresponding element of the range. +.coNP Function @ call-super-method +.synb +.mets (call-super-method < struct-obj < name << argument *) +.syne +.desc +The +.code call-super-method +function is deprecated. Solutions involving +.code call-super-method +should be reworked in terms of +.codn call-super-fun . -When a -.code sub -form is used as a syntactic place, that place denotes a slice of -.metn seq . The -.meta seq -argument must be itself be syntactic place, because -receives a new value, which may be different from its original value in -cases when -.meta seq -is a list. +.code call-super-method +retrieves the function stored in the slot +.meta name +of the supertype of +.meta struct-obj +and invokes it, passing to that function +.meta struct-obj +as the leftmost argument, followed by the given +.metn argument -s, +if any. -Overwriting that slice is equivalent to using the -.code replace -function. The following equivalences give the semantics, except that -.codn x , -.codn a , -.code b +The +.meta struct-obj +argument must be of structure type. Moreover, +that structure type must be derived from another structure type, and -.code v -are evaluated only once, in left-to-right order: - -.verb - (set (sub x a b) v) <--> (progn (set x (replace x v a b)) - v) +.meta name +must name a static slot of that structure type. - (del (sub x a b)) <--> (prog1 (sub x a b) - (set x (replace x nil a b))) -.brev +The object retrieved from that static slot must be +callable as a function, and accept the arguments. -Note that the value of -.code x -is overwritten with the value returned by -.codn replace . -If -.code x -is a vector or string, then the return value of -.code replace -is just -.codn x : -the identity of the object doesn't change under mutation. -However, if -.code x -is a list, its identity changes when items are added to or removed from -the front of the list, and in those cases -.code replace -will return a value different from its first argument. -Similarly, if -.code x -is an object with a -.code lambda-set -method, that method's return value becomes the return value of -.code replace -and must be taken into account. +Note that it is not correct for a method that is defined +against a particular type to use +.code call-super-method +to call the same method (or any other method) in the supertype +of that particular type. This is because +.code call-super-method +refers to the type of the object instance +.metn struct-obj , +not to the type against which the calling method is defined. -.coNP Function @ replace +.coNP Function @ call-super-fun .synb -.mets (replace < sequence < replacement-sequence >> [ from <> [ to ]]) -.mets (replace < sequence < replacement-sequence << index-list ) +.mets (call-super-fun < type < name << argument *) .syne .desc The -.meta replace -function modifies -.meta sequence -in the ways described below. +.code call-super-fun +retrieves the function stored in the slot +.meta name +of the supertype of +.meta type +and invokes it, passing to that function the given +.metn argument -s, +if any. -The operation is destructive: it may work "in place" by modifying -the original sequence. The caller should retain the return value -and stop relying on the original input sequence. +The +.meta type +argument must be a structure type. Moreover, +that structure type must be derived from another structure type, +and +.meta name +must name a static slot of that structure type. -The return value of -.code replace -is the modified -version of -.metn sequence . -This may be the same object as -.meta sequence -or it may be a newly allocated object. +The object retrieved from that static slot must be +callable as a function, and accept the arguments. -Note that the form: +.TP* Example: + +Print a message and call supertype method: .verb - (set seq (replace seq new fr to)) + (defstruct base nil) + + (defstruct derived base) + + (defmeth base fun (obj arg) + (format t "base fun method called with arg ~s\en" arg)) + + (defmeth derived fun (obj arg) + (format t "derived fun method called with arg ~s\en" arg) + (call-super-fun 'derived 'fun obj arg)) + + ;; Interactive Listener: + 1> (new derived).(fun 42) + derived fun method called with arg 42 + base fun method called with arg 42 .brev -has the same effect on the variable -.code seq -as the form: +Note that a static method or function in any structure type +can be invoked by using the +.code "(meth ...)" +name syntax in the first position of a compound form, as +a function name. Thus, the above +.code "derived fun" +can also be written: .verb - (set [seq fr..to] new) + (defmeth derived fun (obj arg) + (format t "derived fun method called with arg ~s\en" arg) + ((meth base fun) obj arg)) .brev -except that the former -.code set -form returns the entire modified sequence, whereas the latter -returns the value of the -.code new -argument. - +.coNP Functions @ struct-get-initfun and @ struct-get-postinitfun +.synb +.mets (struct-get-initfun << type ) +.mets (struct-get-postinitfun << type ) +.syne +.desc The -.code replace -function has two invocation styles, distinguished by the -type of the third argument. If the third argument is a list or vector, then it -is deemed to be the -.meta index-list -parameter of the second form. -Otherwise, if the third argument is missing, or is not a list, then -it is deemed to be the -.meta from -argument of the first form. - -The first form of the replace function replaces a contiguous subsequence of the -.meta sequence -with -.metn replacement-sequence . -The replaced subsequence may be empty, -in which case an insertion is performed. If -.meta replacement-sequence -is empty -(for example, the empty list -.codn nil ), -then a deletion is performed. - -If the -.meta from +.code struct-get-initfun and -.meta to -arguments are omitted, their values default -to -.code 0 +.code struct-get-postinitfun +functions retrieve, respectively, a structure type's +.meta initfun and -.code t -respectively. +.meta postinitfun +functions. These are the functions which are initially configured in the call to +.code make-struct-type +via the +.meta initfun +and +.meta postinitfun +arguments. -The description of the dwim operator\(emin particular, the section -on Range Indexing\(emexplains the semantics of the range specification. +Either one may be +.codn nil , +indicating that the type has no +.meta initfun +or +.metn postinitfun . -The second form of the replace function replaces a subsequence of -elements from -.meta sequence -given by -.metn index-list , -with their counterparts -from -.metn replacement-sequence . -This form of the replace function does not insert -or delete; it simply overwrites elements. If -.meta replacement-sequence +.coNP Functions @ struct-set-initfun and @ struct-set-postinitfun +.synb +.mets (struct-set-initfun < type << function ) +.mets (struct-set-postinitfun < type << function ) +.syne +.desc +The +.code struct-set-initfun and -.meta index-list -are of different lengths, then the shorter of the two determines -the maximum number of elements which are overwritten. -Whenever a negative value occurs in -.meta index-list -the length of -.meta sequence -is added to that value. -Furthermore, similar restrictions apply on -.meta index-list -as under the -select function. Namely, the replacement stops when an index value -in -.meta index-list -is encountered which is out of range for -.metn sequence . -furthermore, if -.meta sequence -is a list, then -.meta index-list -must -be monotonically increasing, after consideration of the -displacement of negative values. +.code struct-set-postinitfun +functions overwrite, respectively, a structure type's +.meta initfun +and +.meta postinitfun +functions. These are the functions which are initially configured in the call to +.code make-struct-type +via the +.meta initfun +and +.meta postinitfun +arguments. -If -.meta sequence -is a -.code carray -object, then -.code replace -behaves like -.codn carray-replace . +The +.meta function +argument must either be +.code nil +or else a function which accepts one argument. -If -.meta sequence -is a -.code buf -object, then -.code replace -behaves like -.codn buf-replace . +Note that +.meta initfun +has the responsibility for all instance slot initializations. The +.code defstruct +syntax compiles the initializing expressions in the slot specifier syntax +into statements which are placed into a function, which becomes the +.meta initfun +of the struct type. -If -.meta sequence -is a structure, then the structure must support the -.code lambda-set -method. The -.code replace -operation is translated into a call of the -.code lambda-set -method according to the following equivalences: +.coNP Macro @ with-objects +.synb +.mets (with-objects >> ({( sym << init-form )}*) << body-form *) +.syne +.desc +The +.code with-objects +macro provides a binding construct very similar to +.codn let* . -.verb - (replace o items from to) - <--> o.(lambda-set (rcons from to) items) +Each +.meta sym +must be a symbol suitable for use as a variable name. - (replace o items index-list) - <--> o.(lambda-set index-list items) +Each +.meta init-form +is evaluated in sequence, and a binding is established for its +corresponding +.meta sym +which is initialized with the value of that form. The binding +is visible to subsequent +.metn init-form -s. + +Additionally, the values of the +.metn init-form -s +are noted as they are produced. When the +.code with-objects +form terminates, by any means, the +.code call-finalizers +function is invoked on each value which was returned by an +.meta init-form +and had been noted. These calls are performed in the +reverse order relative to the original evaluation of the forms. + +After the variables are established and initialized, the +.metn body-form -s +are evaluated in the scope of the variables. The value of the +last form is returned, or else +.code nil +if there are no forms. The invocations of +.code call-finalizers +take place just before the value of the last form is returned. + +.SS* Special Structure Functions + +Special structure functions are user-defined methods or structure functions +which are specially recognized by certain functions in \*(TL. They endow +structure objects with the ability to participate in certain usage scenarios, +or to participate in a customized way. + +Special functions are required to bound to static slots, which is the +case if the +.code defmeth +macro is used, or when methods or functions are defined using syntax +inside a +.code defstruct +form. If a special function or method is defined as an instance slot, +then the behavior of library functions which depend on this method is +unspecified. + +Special functions introduced below by the word "Method" receive an object +instance as an argument. Their syntax is indicated using the same notation +which may be used to invoke them, such as: + +.verb +.mets << object .(function-name < arg << ... ) .brev -Thus, the -.meta from -and -.meta to -arguments are converted to single range object, -whereas an -.meta index-list -is passed as-is. -It is an error if the -.code from -argument is a sequence, indicating an -.metn index-list , -and a -.code to -argument is also given; the situation is diagnosed. If either -.code from -or -.code to -are omitted, the range object contains the -.code : -symbol in the corresponding place: +However, those introduced as "Function" do not operate on an instance. For +brevity, their syntax is nevertheless exemplified as .verb - (replace o items from) - <--> o.(lambda-set (rcons from :) items) +.meti << object .'['function-name < arg << ... ']' +.brev - (replace o items : to) - <--> o.(lambda-set (rcons : to) items) +If such a invocation is actually used, the +.meta object +instance only serves for identifying the struct type whose static slot +.code function-name +provides the function; +.meta object +doesn't participate in the call. An object is not required since +the function can be called using - (replace o items) - <--> o.(lambda-set (rcons : :) items) +.verb +.meti [(static-slot < type 'function-name) < arg << ... ] .brev -It is the responsibility of the object's -.code lambda-set -method to implement semantics consistent with the -description of -.codn replace . +which looks up the function in the struct +.meta type +directly. -.coNP Function @ take +.coNP Method @ equal .synb -.mets (take < count << sequence ) +.mets << object .(equal) .syne .desc -The -.code take -function returns -.meta sequence -with all except the first -.meta count -items removed. +Normally, two struct values are not considered the same under the +.code equal +function unless they are the same object. -If -.meta sequence -is a list, then -.code take -returns a lazy list which produces the first -.meta count -items of sequence. +However, if the +.code equal +method is defined for a structure type, then instances of +that structure type support +.IR "equality substitution" . -For other kinds of sequences, including lazy strings, -.code take -works eagerly. +The +.code equal +method must not require any arguments other than +.metn object . +Moreover, the method must never return +.codn nil . -If -.meta count -exceeds the length of -.meta sequence -then a sequence is returned which has all the items. -This object may be -.meta sequence -itself, or a copy. +When a struct which supports equality substitution is compared using +.codn equal , +.code less +or +.codn greater , +its +.code equal +method is invoked, and the return value is used in place of that +structure for the purposes of the comparison. -If -.meta count -is negative, it is treated as zero. +The same applies when an struct is hashed using the +.code hash-equal +function, or implicitly by an +.code :equal-hash +hash table. -.coNP Functions @ take-while and @ take-until +Note: if an +.code equal +method is defined or redefined with different semantics for a struct +type whose instances have already been inserted as keys in an +.code :equal-based +hash table, the behavior of subsequent insertion and lookup operations +on that hash table becomes unspecified. + +.coNP Method @ print .synb -.mets (take-while < predfun < sequence <> [ keyfun ]) -.mets (take-until < predfun < sequence <> [ keyfun ]) +.mets << object .(print < stream << pretty-p ) .syne .desc -The -.code take-while -and -.code take-until -functions return a prefix of -.meta sequence -whose items satisfy certain conditions. +If a method named by the symbol +.code print +is defined for a structure type, then it is used for printing instances +of that type. The -.code take-while -function returns the longest prefix of -.meta sequence -whose elements, accessed through -.meta keyfun -satisfy the function -.metn predfun . +.meta stream +argument specifies the output stream to which the printed representation +is to be written. The -.meta keyfun -argument defaults to the identity function: the elements -of -.meta sequence -are examined themselves. +.meta pretty-p +argument is a Boolean flag indicating whether pretty-printing +is requested. Its value may simply be passed to recursive calls to +.codn print , +or used to select between +.code ~s +or +.code ~a +formatting if +.code format +is used. -The -.code take-until -function returns the longest prefix of -.meta sequence -which consists of elements, accessed through -.metn keyfun , -that do -.B not -satisfy -.meta predfun -followed by an element which does satisfy -.metn predfun . -If -.meta sequence -has no such prefix, then an empty sequence -is returned of the same kind as -.metn sequence . +The value returned by the +.code print +method is significant. If the special keyword symbol +.code : +(colon) is returned, then the system will print the object +in the default way, as if no +.code print +method existed: it is understood that the method declined +the responsibility for printing the object. -If -.meta sequence -is a list, then these functions return a lazy list. +If any other value is returned, then it is understood +that the method +.code print +method accepted the responsibility for printing the object, +and the system consequently will generate into +.meta stream +any output output pertaining to +.metn object 's +representation. -.coNP Function @ drop +.coNP Method @ lambda .synb -.mets (drop < count << sequence ) +.mets << object .(lambda << arg *) .syne .desc -The -.code drop -function returns -.meta sequence -with the first -.meta count -items removed. - -If -.meta count -is negative, it is treated as zero. - -If -.meta count -is zero, then -.meta sequence -is returned. +If a structure type provides a method called +.code lambda +then it can be used as a function. -If -.meta count -exceeds the length of -.meta sequence -then an empty sequence is returned -of the same kind as -.metn sequence . +Of course, this method can be called by name, using the syntax given +in the above syntactic description. -.coNP Functions @ drop-while and @ drop-until -.synb -.mets (drop-while < predfun < sequence <> [ keyfun ]) -.mets (drop-until < predfun < sequence <> [ keyfun ]) -.syne -.desc -The -.code drop-while -and -.code drop-until -functions return -.meta sequence -with a prefix of that sequence removed, -according to conditions involving -.meta predfun -and -.metn keyfun . +However, the intended use is that it allows the structure instance itself to be +used as a function. When arguments are applied to a structure object as if it +were a function, this is erroneous, unless that object has a +.code lambda +method. In that case, the arguments are passed to the lambda method. +Of course, the leftmost argument of the method is the structure instance +itself. +That is to say, the following equivalences apply, except that +.code s +is evaluated only once: -The -.code drop-while -function removes the longest prefix of -.meta sequence -whose elements, accessed through -.meta keyfun -satisfy the function -.metn predfun , -and returns the remaining sequence. +.verb + (call s args ...) <--> s.(lambda args ...) -The -.meta keyfun -argument defaults to the identity function: the elements -of -.meta sequence -are examined themselves. + [s args ...] <--> [s.lambda s args ...] -The -.code drop-until -function removes the longest prefix of -.meta sequence -which consists of elements, accessed through -.metn keyfun , -that do -.B not -satisfy -.meta predfun -followed by an element which does satisfy -.metn predfun . -A sequence of the remaining elements is -returned. + (mapcar s list) <--> (mapcar (meth s lambda) list) +.brev -If -.meta sequence -has no such prefix, then a sequence -same as -.meta sequence -is returned, which may be -.meta sequence -itself or a copy. +Note: a form such as +.code "[s args ...]" +where +.code s +is a structure can be treated as a place if the method +.code lambda-set +is also implemented. -.coNP Accessor @ last +.coNP Method @ lambda-set .synb -.mets (last < seq <> [ num ]) -.mets (set (last < seq <> [ num ]) << new-value) +.mets << object .(lambda-set << arg * << new-value ) .syne .desc The -.meta last -function returns a subsequence of -.meta seq -consisting of the last -.meta num -of its elements, where -.meta num -defaults to 1. - -If -.meta num -is zero or negative, then an empty sequence is returned. -If -.meta num -is positive, and greater than or equal to the length of seq, -then seq -.meta seq -is returned. - -If a -.code last -form is used as a place, then -.code seq -must be a place. The following equivalence gives the semantics -of assignment to a -.codn last : +.code lambda-set +method, in conjunction with a +.code lambda +method, allows structures to be used as place accessors. If +structure +.code s +supports a +.meta lambda-set +with four arguments, then the following use of the +.code dwim +operator is possible: .verb - (set (last x n) v) <--> (set (sub x (- (max n 0)) t) v) + (set [s a b c d] v) + (set (dwim s a b c d) v) ;; precisely equivalently .brev -A -.code last -place is deletable. The semantics of deletion may be understood -in terms of the following equivalence: +This has an effect which can be described by the following code: .verb - (del (last x n)) <--> (del (sub x (- (max n 0)) t)) + (progn + s s.(lambda-set a b c d v) + v) .brev -.coNP Accessor @ butlast -.synb -.mets (butlast < sequence <> [ num ]) -.mets (set (butlast < sequence <> [ num ]) << new-value ) -.syne -.desc -The -.code butlast -function returns the prefix of -.meta sequence -consisting of a copy of it, with the last -.meta num -items removed. +except that +.code s +and +.code v +are evaluated only once, and +.code a +through +.code d +are evaluated using the Lisp-1 semantics due the +.code dwim +operator. -The parameter -.meta num -defaults to 1 -if an argument is omitted. +If a place-mutating operator is used on this form which requires the prior +value, such as the +.code inc +macro, then the structure must support the +.code lambda +function also. If -.meta sequence -is empty, an empty sequence is returned. +.code lambda +takes +.I n +arguments, then +.code lambda-set +should take +.I n+1 +arguments. The first +.I n +arguments of these two methods are congruent; the extra rightmost argument +of +.code lambda-set +is the new value to be stored into the place denoted by the prior +arguments. -If -.meta num -is zero or negative, then -.meta sequence -is returned. +The return value of +.code lambda-set +is ignored. -If -.meta num -is positive, and meets or exceeds the length of -.metn sequence , -then an empty sequence is returned. +Note: the +.code lambda-set +method is also used by the +.code rplaca +function, if no +.code rplaca +method exists. -If a -.code butlast -form is used as a place, then -.meta sequence -must itself be a place. The following equivalence gives the semantics -of assignment to a -.codn last : +.TP* Example + +The following defines a structure with a single instance +slot +.code hash +which holds a hash table, as well as +.code lambda +and +.code lambda-set +methods: .verb - (set (butlast x n) v) <--> (set (sub x 0 (- (max n 0))) v) -.brev + (defstruct hash-wrapper nil + (hash (hash)) -A -.code butlast -place is deletable. The semantics of deletion may be understood -in terms of the following equivalence: + (:method lambda (self key) + [self.hash key]) -.verb - (del (last x n)) <--> (del (sub x 0 (- (max n 0)))) + (:method lambda-set (self key new-val) + (set [self.hash key] new-val) self)) .brev -Note: the \*(TL -.code take -function also computes the prefix of a list; however, it counts items -from the beginning, and provides lazy semantics which allow it -to work with infinite lists. - -See also: the -.code butlastn -accessor, which operates on lists. That function has useful semantics for -improper lists and treats an atom as the terminator of a zero-length improper -list. +An instance of this structure can now be used as follows: -Dialect note: a destructive function similar to Common Lisp's -.code nbutlast -isn't provided. Of course, assignment to an -.code butlast -form is destructive; Common Lisp doesn't support -.code butlast -as a place. +.verb + (let ((s (new hash-wrapper))) + (set [s "apple"] 3 + [s "orange] 4) + [s "apple"]) -> 3 +.brev -.coNP Function @ ldiff +.coNP Method @ length .synb -.mets (ldiff < sequence << tail-sequence ) +.mets << object .(length) .syne .desc -The -.code ldiff -function is a somewhat generalized version of the same-named classic Lisp -function found in traditional Lisp dialects. +If a structure has +.code length +method, then it can be used as an argument to the +.code length +function. -The -.code ldiff -function supports the original -.code ldiff -semantics when both inputs are lists. It determines whether the -.meta tail-sequence -list is a structural suffix of -.metn sequence ; -which is to say: is -.meta tail-sequence -one of the -.code cons -cells which comprise -.metn sequence ? -If so, then a list is returned consisting of all the items of -.meta sequence -before -.metn tail-sequence : -a copy of -.meta sequence -with the -.meta tail-sequence -part removed, and replaced by the -.code nil -terminator. If -.meta tail-sequence -is -.code nil -or the lists are unrelated, then -.meta sequence -is returned. +Structures which implement the methods +.codn lambda , +.code lambda-set +and +.code length +can be treated as abstract vector-like sequences, because such +structures support the +.codn ref , +.code refset +and +.code length +functions. -The \*(TL -.code ldiff -function supports the following additional semantics. +For instance, the +.code nreverse +function will operate on such objects. -.RS -.IP 1. -The basic description of -.code ldiff -is extended to work with list-like sequences, not -merely lists; that is to say, objects which support the +Note: a structure which supports the .code car -method. -.IP 2. -If -.meta sequence -is any kind of sequence, and -.meta tail-sequence -is any kind of empty sequence, then -.meta sequence -is returned. -.IP 3. -If either argument is an atom that is not a sequence, -.code ldiff -returns -.metn sequence . -.IP 4. -If -.meta sequence -is a list-like sequence, and -.meta tail-sequence -isn't, then the terminating atom of -.meta sequence -is determined. This atom is compared using -.code equal -to the -.meta tail-sequence -object. If they are equal, then a proper list is -returned containing the items of -.meta sequence -excluding the terminating atom. -.IP 5. -If both arguments are vector-like sequences, then -.code ldiff -determines whether -.meta sequence -has a suffix which is -.code equal -to -.metn tail-sequence . -If this is the case, then a sequence is returned, of the same kind as -.metn sequence , -consisting of the items of -.meta sequence -before that suffix. -If -.meta tail-sequence -is not -.code equal -to a suffix of -.metn sequence , -then -.meta sequence -is returned. -.IP 6 -In all other cases, -.meta sequence +method also supports the +.code length +function, in a different way. Such a structure is treated by +.code length +as a list-like sequence, and its length is measured by walking the +sequence with +.code cdr +operations. If a structure supports both +.code length and -.meta tail-sequence -are compared with -.codn equal . -If the comparison is true, -.code nil -is returned, otherwise -.meta sequence -is returned. -.RE - -.TP* Examples: - -.verb - ;;; unspecified: the compiler could make - ;;; '(2 3) a suffix of '(1 2 3), - ;;; or they could be separate objects. - (ldiff '(1 2 3) '(2 3)) -> either (1) or (1 2 3) +.codn car , +preference is given to +.codn length , +which is likely to be much more efficient. - ;; b is the (1 2) suffix of a, so the ldiff is (1) - (let* ((a '(1 2 3)) (b (cdr a))) - (ldiff a b)) - -> (1) +.coNP Methods @, car @ cdr and @ nullify +.synb +.mets << object .(car) +.mets << object .(cdr) +.mets << object .(nullify) +.syne +.desc +Structures may be treated as sequences if they define methods named +by the symbols +.codn car , +.codn cdr , +and +.codn nullify . - ;; Rule 5: strings and vector - (ldiff "abc" "bc") -> "a" - (ldiff "abc" nil) -> "abc" - (ldiff #(1 2 3) #(3)) -> #(1 2) +If a structure supports these methods, then these methods are used +by the functions +.codn car , +.codn cdr , +.codn nullify , +.code empty +and various other sequence manipulating functions derived from them, when those +functions are applied to that object. - ;; Rule 5: mixed vector kinds - (ldiff "abc" #(#\eb #\ec)) -> "abc" +An object which implements these three methods can be considered to represent a +.I list-like +abstract sequence. - ;; Rule 6: - (ldiff #(1 2 3) '(3)) -> #(1 2 3) +The object's +.code car +method should return the first value in that abstract sequence, or else +.code nil +if that sequence is empty. - ;; Rule 4: - (ldiff '(1 2 3) #(3)) -> '(1 2 3) - (ldiff '(1 2 3 . #(3)) #(3)) -> '(1 2 3) - (ldiff '(1 2 3 . 4) #(3)) -> '(1 2 3 . 4) +The object's +.code cdr +method should return an object denoting the remainder of the sequence, +or else +.code nil +if the sequence is empty or contains only one value. This returned object can +be of any type: it may be of the same structure type as that object, a +different structure type, a list, or whatever else. If a non-sequence object +is returned. - ;; Rule 6 - (ldiff 1 2) -> 1 - (ldiff 1 1) -> nil -.brev +The +.code nullify +method should return +.code nil +if the object is considered to denote an empty sequence. Otherwise it +should either return that object itself, or else return the sequence which +that object represents. -.coNP Function @ search +.coNP Methods @ rplaca and @ rplacd .synb -.mets (search < haystack < needle >> [ testfun <> [ keyfun ]) +.mets << object .(rplaca << new-car-value ) +.mets << object .(rplacd << new-cdr-value ) .syne .desc -The -.code search -function determines whether the sequence -.meta needle -occurs as substring -within -.metn haystack , -under the given comparison function -.meta testfun +If a structure type defines the methods +.code rplaca and -key function -.metn keyfun . -If this is the case, then the zero-based position of -the leftmost occurrence of -.meta key -within -.meta haystack -is returned. Otherwise -.code nil -is returned to indicate that -.meta key -does not occur within -.metn haystack . -If -.meta key -is empty, then zero is always returned. - -The arguments -.meta haystack +.code rplacd +then, respectively, the +.code rplaca and -.meta needle -are sequences: lists, vectors -or strings, in any combination. +.code rplacd +functions will use these methods if they are applied to instances of that type. -If -.meta needle -is not empty, then occurs at some position N within -.meta haystack -if -the first element of -.meta needle -matches the element at position N of -.metn haystack , -the second element of -.meta needle -matches the element at position N+1 of -.meta haystack -and so forth, for all elements of -.metn needle . -A match between elements -is determined by passing each element through -.metn keyfun , -and then comparing the resulting values using -.metn testfun . +That is to say, when the function call +.mono +.meti (rplaca < o << v ) +.onom +is evaluated, and +.meta o +is a structure type, the function inquires whether +.meta o +supports a +.code rplaca +method. If so, then, effectively, +.mono +.meti << o . (rplaca << v) +.onom +is invoked. The return value of this method call is ignored; +.code rplaca +returns +.metn o . +The analogous requirements apply to +.code rplacd +in relation to the +.code rplacd +method. -If -.meta testfun -is supplied, it must be a function which can be -called with two arguments. If it is not supplied, it defaults to -.codn eql . +Note: if the +.code rplaca +method doesn't exist, the +.code rplaca +function falls back on trying to store +.meta new-car-value +by means of the structure type's +.code lambda-set +method, using an index of zero. That is to say, if the type has no +.code rplaca +method, but does have a +.code lambda-set +method, then +.mono +.meti << o . (lambda-set 0 << v) +.onom +is invoked. -If -.meta keyfun -is supplied, it must be a function which can be called -with one argument. If it is not supplied, it defaults to -.codn identity . +.coNP Function @ from-list +.synb +.mets << object .'['from-list << list ']' +.syne +.desc +If a +.code from-list +structure function is defined for a structure type, it is called in certain +situations with an argument which is a list object. The function's purpose +is to construct a new instance of the structure type, derived from that +list. -.TP* Examples: +The purpose of this function is to allow sequence processing operations +such as +.code mapcar +and +.code remove +to operate on a structure object as if it were a sequence, and return a +transformed sequence of the same type. This is analogous to the way such +functions can operate on a vector or string, and return a vector or string. -.verb - ;; fails because 3.0 doesn't match 3 - ;; under the default eql function - [search #(1.0 3.0 4.0 7.0) '(3 4)] -> nil +If a structure object behaves as a sequence thanks to providing +.codn car , +.code cdr +and +.code nullify +methods, but does not have a +.code from-list +function, then those sequence-processing operations which return a sequence +will always return a plain list of items. - ;; occurrence found at position 1: - ;; (3.0 4.0) matches (3 4) under = - [search #(1.0 3.0 4.0 7.0) '(3 4) =] -> 1 +.coNP Function @ derived +.synb +.mets << object .'['derived < supertype << subtype ']' +.syne +.desc +If a structure type supports a function called +.metn derived , +this function is called whenever a new type is defined which names +that type as its supertype. - ;; "even odd odd odd even" pattern - ;; matches at position 2 - [search #(1 1 2 3 5 7 8) '(2 1 1 1 2) : evenp] -> 2 +The function is called with two arguments which are both struct types. +The +.meta supertype +argument gives the type that is being inherited from. +The +.meta subtype +gives the new type that is inheriting from +.metn supertype . - ;; Case insensitive string search - [search "abcd" "CD" : chr-toupper] -> 2 +The function is called at most once for the creation of a given +.metn subtype , +only for its immediate supertype, if and only if that supertype +has defined this function. - ;; Case insensitive string search - ;; using vector of characters as key - [search "abcd" #(#\eC #\eD) : chr-toupper] -> 2 -.brev +The function is not retroactively invoked if it is defined for +a structure type from which subtypes have already been derived. -.coNP Function @ rsearch +Note: the +.meta supertype +parameter exists because the +.code derived +function is itself inherited. If the same version of this function is shared by +multiple structure types due to inheritance, this argument informs the function +which of those types it is being invoked for. + +.SS* Sequence Manipulation +.coNP Function @ seqp .synb -.mets (rsearch < haystack < needle >> [ testfun <> [ keyfun ]) +.mets (seqp << object ) .syne .desc -The -.code rsearch -function is like -.code search -except that if -.meta needle -matches -.meta haystack -in multiple places, -.code rsearch -returns the right-most matching position rather than -the leftmost. +The function +.code seqp +returns +.code t +if +.meta object +is a sequence, otherwise +.codn nil . -.coNP Functions @ ref and @ refset +A sequence is defined as a list, vector or string. The object +.code nil +denotes +the empty list and so is a sequence. + +.coNP Functions @ length and @ len .synb -.mets (ref < seq << index ) -.mets (refset < seq < index << new-value ) +.mets (length << object ) +.mets (len << object ) .syne .desc -The -.code ref -and -.code refset -functions perform array-like indexing into sequences, as well as -objects of type -.code buf -and -.codn carray . +If +.meta object +is a sequence, the +.code length +function returns the number of items it +contains. -If the -.meta seq -parameter is a hash, then these functions perform -has retrieval and storage; in that case -.meta index -isn't restricted to an integer value. +The +.code len +function is a synonym of +.codn length . -If -.meta seq -is a structure, it supports -.code ref -directly if it has a -.code lambda -method. The -.meta index -argument is passed to that method, and the resulting value is -returned. -If a structure lacks a -.code lambda -method, but has a -.code car -method, then -.code ref -treats it as a list, traversing the structure using -.cod3 car / cdr -operations. In the absence of support for these operations, -the function fails with an error exception. - -Similarly, a structure supports -.code refset -directly if it has a -.code lambda-set -method. This gets called with -.meta index -and -.meta new-value -as arguments. Then -.meta new-value +The function supports these additional types: +.RS +.coIP hash +The value of +.code hash-count is returned. -If a structure lacks a -.code lambda-set -method, then -.code refset -treats it as a list, traversing the structure using -.cod3 car / cdr -operations, and storing -.meta new-value -using -.codn rplaca . -In the absence of support for these operations, -the function fails with an error exception. - -The -.code ref -function retrieves an element of -.metn seq , -whereas -.code refset -overwrites an -element of -.meta seq -with a new value. - -If -.meta seq -is a sequence then -.meta index -argument must be an integer. The first element of the sequence -is indexed by zero. Negative values are permitted, -denoting backward indexing from the end of the sequence, such that -the last element is indexed by -1, the second last by -2 and so on. -See also the Range Indexing section under the -description of the -.code dwim -operator. - -If -.meta seq -is a list, then out-of-range indices, whether positive or negative, -are treated leniently by -.codn ref : -such accesses produce the value -.codn nil , -rather than an error. For other sequence types, such accesses -are erroneous. For hashes, accesses to nonexistent elements -are treated leniently, and produce -.codn nil . - -The -.code refset -function is strict for out-of-range indices over all sequences, -including lists. In the case of hashes, a -.code refset -of a nonexistent key creates the key. - -The -.code refset -function returns -.codn new-value . - -The following equivalences hold between -.code ref -and -.codn refset , -and the DWIM bracket syntax, provided that -.meta idx -is a scalar index and -.meta seq -is a sequence object, rather than a hash. - -.verb - (ref seq idx) <--> [seq idx] - - (refset seq idx new) <--> (set [seq idx] new) -.brev +.coIP range +The length of the interval +represented by the range is returned. +The length of a range +.code r +is defined as +.codn "(- (to r) (from r))" , +and thus may be negative. +The length of +.code "#R(1 -10)" +is -11 and of +.codn "#R(0.5 3)" , +2.5. +.coIP buf +The buffer length calculated by +.code length-buf +is returned. +.coIP carray +The number of elements in +.meta object +calculated by +.code length-carray +is returned. +.RE -The difference is that -.code ref -and -.code refset -are first class functions which -can be used in functional programming as higher order functions, whereas the -bracket notation is syntactic sugar, and -.code set -is an operator, not a function. -Therefore the brackets cannot replace all uses of -.code ref -and -.codn refset . +.IP +For other types, +.code length +throws an error exception. -.coNP Function @ update +.coNP Function @ empty .synb -.mets (update < sequence-or-hash << function ) +.mets (empty << object ) .syne .desc -The -.code update -function replaces each elements in a sequence, or each value -in a hash table, with the value of -.meta function -applied to that element -or value. +If +.meta object +is a suitable argument for the +.code length +function, then the +.code empty +Returns +.code t +if +.mono +.meti (length << object ) +.onom +is zero, otherwise +.codn nil . -The sequence or hash table is returned. +If +.meta object +is not a suitable argument for +.codn length , +then +.code empty +throws an error exception. -.coNP Functions @, remq @ remql and @ remqual +.coNP Function @ copy .synb -.mets (remq < object < list <> [ key-function ]) -.mets (remql < object < list <> [ key-function ]) -.mets (remqual < object < list <> [ key-function ]) +.mets (copy << object ) .syne .desc The -.codn remq , -.code remql -and -.code remqual -functions produce a new list based on -.metn list , -removing the elements whose associated keys are -.codn eq , -.code eql -or -.code equal -to -.metn object . +.code copy +function duplicates objects of various supported types: sequences, hashes, +structures and random states. If +.meta object +is +.codn nil , +it +returns +.codn nil . +Otherwise, +.code copy +is equivalent to invoking a more specific copying function according to +the type of the argument, as follows: +.RS +.coIP cons +.meti (copy-list << object ) +.coIP str +.meti (copy-str << object ) +.coIP vec +.meti (copy-vec << object ) +.coIP hash +.meti (copy-hash << object ) +.IP "struct type" +.meti (copy-struct << object ) +.coIP fun +.meti (copy-fun << object ) +.coIP buf +.meti (copy-buf << object ) +.coIP carray +.meti (copy-carray << object ) +.coIP random-state +.meti (make-random-state << object ) +.RE -The input -.meta list -is unmodified, but the returned list may share substructure -with it. If no items are removed, it is possible that the return value +.IP +For all other types of +.metn object , +the invocation is erroneous. + +Except in the case when +.meta sequence is -.meta list -itself. +.codn nil , +.code copy +returns a value that +is distinct from (not +.code eq +to) +.metn sequence . +This is different from +the behavior of +.mono +.meti >> [ sequence 0..t] +.onom +or +.mono +.meti (sub < sequence 0 t) +.onom +which recognize +that they need not make a copy of +.metn sequence , +and just return it. -If -.meta key-function -is omitted, then the element keys compared to -.meta object -are the elements themselves. -Otherwise, -.meta key-function -is applied to each element and the resulting value -is that element's key which is compared to -.metn object . +Note however, that the elements of the returned sequence may be +eq to elements of the original sequence. In other words, copy is +a deeper copy than just duplicating the +.code sequence +value itself, +but it is not a deep copy. -.coNP Functions @, remq* @ remql* and @ remqual* +.coNP Accessor @ sub .synb -.mets (remq* < object << list ) -.mets (remql* < object << list ) -.mets (remqual* < object << list ) +.mets (sub < sequence >> [ from <> [ to ]]) +.mets (set (sub < sequence >> [ from <> [ to ]]) << new-val ) .syne .desc The -.codn remq* , -.code remql* -and -.code remqual* -functions are lazy versions of -.codn remq , -.code remql -and -.codn remqual . -Rather than computing the entire new list -prior to returning, these functions return a lazy list. +.code sub +function extracts a slice from input sequence +.metn sequence . +The slice is +a sequence of the same type as +.metn sequence . -Caution: these functions can still get into infinite looping behavior. -For instance, in -.codn "(remql* 0 (repeat '(0)))" , -.code remql -will keep consuming -the -.code 0 -values coming out of the infinite list, looking for the first item that -does not have to be deleted, in order to instantiate the first lazy value. +If the +.meta from +argument is omitted, it defaults to +.codn 0 . +If the +.meta to +parameter is +omitted, it defaults to +.codn t . +Thus +.code "(sub a)" +means +.codn "(sub a 0 t)" . -.TP* Examples: -.verb - ;; Return a list of all the natural numbers, excluding 13, - ;; then take the first 100 of these. - ;; If remql is used, it will loop until memory is exhausted, - ;; because (range 1) is an infinite list. +The following semantic equivalence exists between a call to the +.code sub +function and +the DWIM-bracket syntax, except that +.code sub +is an ordinary function call form, which doesn't apply the +Lisp-1 evaluation semantics to its arguments: - [(remql* 13 (range 1)) 0..100] +.verb + ;; from is not a list + (sub seq from to) <--> [seq from..to] .brev -.coNP Functions @, keepq @ keepql and @ keepqual -.synb -.mets (keepq < object < list <> [ key-function ]) -.mets (keepql < object < list <> [ key-function ]) -.mets (keepqual < object < list <> [ key-function ]) -.syne -.desc -The -.codn keepq , -.code keepql -and -.code keepqual -functions produce a new list based on -.metn list , -removing the items whose keys are not -.codn eq , -.code eql -or -.code equal -to -.metn object . +The description of the +.code dwim +operator\(emin particular, the section +on Range Indexing\(emexplains the semantics of the range specification. -The input -.meta list -is unmodified, but the returned list may share substructure -with it. If no items are removed, it is possible that the return value -is -.meta list -itself. +The output sequence may share structure with the input sequence. -The optional -.meta key-function -is applied to each element from the -.meta list -to convert it to a key which is compared to -.metn object . If -.meta key-function -is omitted, then each element itself of -.meta list -is compared to -.metn object . +.meta sequence +is a +.code carray +object, then the function behaves like +.codn carray-sub . -.coNP Functions @, remove-if @, keep-if @ remove-if* and @ keep-if* -.synb -.mets (remove-if < predicate-function < list <> [ key-function ]) -.mets (keep-if < predicate-function < list <> [ key-function ]) -.mets (remove-if* < predicate-function < list <> [ key-function ]) -.mets (keep-if* < predicate-function < list <> [ key-function ]) -.syne -.desc -The -.code remove-if -function produces a list whose contents are those of -.meta list -but with those elements removed which satisfy -.metn predicate-function . -Those elements which are not removed appear in the same order. -The result list may share substructure with the input list, -and may even be the same list object if no items are removed. +If +.meta sequence +is a +.code buf +object, then the function behaves like +.codn buf-sub . -The optional -.meta key-function -specifies how each element from the -.meta list -is transformed to an argument to -.metn predicate-function . -If this argument is omitted -then the predicate function is applied to the elements directly, a behavior -which is identical to -.meta key-function -being -.codn "(fun identity)" . +If +.meta sequence +is a structure, it must support the +.code lambda +method. The +.code sub +operation is transformed into a call to the +.code lambda +method according to the following equivalence: -The -.code keep-if -function is exactly like -.codn remove-if , -except the sense of -the predicate is inverted. The function -.code keep-if -retains those items -which -.code remove-if -will delete, and removes those that -.code remove-if -will preserve. +.verb + (sub o from to) <--> o.(lambda (rcons from to)) + (sub o : to) <--> o.(lambda (rcons : to)) + (sub o from) <--> o.(lambda (rcons from :)) + (sub o) <--> o.(lambda (rcons : :)) +.brev -The -.code remove-if* +That is to say, the +.meta from and -.code keep-if* -functions are like -.code remove-if +.code to +arguments are converted to range object. If either argument +is missing, the symbol +.code : +is used for the corresponding element of the range. + +When a +.code sub +form is used as a syntactic place, that place denotes a slice of +.metn seq . +The +.meta seq +argument must be itself be syntactic place, because +receives a new value, which may be different from its original value in +cases when +.meta seq +is a list. + +Overwriting that slice is equivalent to using the +.code replace +function. The following equivalences give the semantics, except that +.codn x , +.codn a , +.code b and -.codn keep-if , -but produce lazy lists. +.code v +are evaluated only once, in left-to-right order: -.TP* Examples: .verb - ;; remove any element numerically equal to 3. - (remove-if (op = 3) '(1 2 3 4 3.0 5)) -> (1 2 4 5) - - ;; remove those pairs whose first element begins with "abc" - [remove-if (op equal [@1 0..3] "abc") - '(("abcd" 4) ("defg" 5)) - car] - -> (("defg" 5)) + (set (sub x a b) v) <--> (progn (set x (replace x v a b)) + v) - ;; equivalent, without test function - (remove-if (op equal [(car @1) 0..3] "abc") - '(("abcd" 4) ("defg" 5))) - -> (("defg" 5)) + (del (sub x a b)) <--> (prog1 (sub x a b) + (set x (replace x nil a b))) .brev -.coNP Functions @, countqual @ countql and @ countq -.synb -.mets (countq < object << list ) -.mets (countql < object << list ) -.mets (countqual < object << list ) -.syne -.desc -The -.codn countq , -.code countql -and -.code countqual -functions count the number of objects -in -.meta list -which are -.codn eq , -.code eql -or -.code equal -to -.metn object , -and return the count. +Note that the value of +.code x +is overwritten with the value returned by +.codn replace . +If +.code x +is a vector or string, then the return value of +.code replace +is just +.codn x : +the identity of the object doesn't change under mutation. +However, if +.code x +is a list, its identity changes when items are added to or removed from +the front of the list, and in those cases +.code replace +will return a value different from its first argument. +Similarly, if +.code x +is an object with a +.code lambda-set +method, that method's return value becomes the return value of +.code replace +and must be taken into account. -.coNP Function @ count-if +.coNP Function @ replace .synb -.mets (count-if < predicate-function < list <> [ key-function ]) +.mets (replace < sequence < replacement-sequence >> [ from <> [ to ]]) +.mets (replace < sequence < replacement-sequence << index-list ) .syne .desc The -.code count-if -function counts the number of elements of -.meta list -which satisfy -.meta predicate-function -and returns the count. +.meta replace +function modifies +.meta sequence +in the ways described below. -The optional -.meta key-function -specifies how each element from the -.meta list -is transformed to an argument to -.metn predicate-function . -If this argument is omitted -then the predicate function is applied to the elements directly, a behavior -which is identical to -.meta key-function -being -.codn "(fun identity)" . +The operation is destructive: it may work "in place" by modifying +the original sequence. The caller should retain the return value +and stop relying on the original input sequence. + +The return value of +.code replace +is the modified +version of +.metn sequence . +This may be the same object as +.meta sequence +or it may be a newly allocated object. + +Note that the form: + +.verb + (set seq (replace seq new fr to)) +.brev + +has the same effect on the variable +.code seq +as the form: + +.verb + (set [seq fr..to] new) +.brev + +except that the former +.code set +form returns the entire modified sequence, whereas the latter +returns the value of the +.code new +argument. -.coNP Functions @, posq @ posql and @ posqual -.synb -.mets (posq < object << list ) -.mets (posql < object << list ) -.mets (posqual < object << list ) -.syne -.desc The -.codn posq , -.code posql +.code replace +function has two invocation styles, distinguished by the +type of the third argument. If the third argument is a list or vector, then it +is deemed to be the +.meta index-list +parameter of the second form. +Otherwise, if the third argument is missing, or is not a list, then +it is deemed to be the +.meta from +argument of the first form. + +The first form of the replace function replaces a contiguous subsequence of the +.meta sequence +with +.metn replacement-sequence . +The replaced subsequence may be empty, +in which case an insertion is performed. If +.meta replacement-sequence +is empty +(for example, the empty list +.codn nil ), +then a deletion is performed. + +If the +.meta from and -.code posqual -functions return the zero-based position of the -first item in -.meta list -which is, respectively, -.codn eq , -.code eql -or -.code equal +.meta to +arguments are omitted, their values default to -.metn object . - -.coNP Functions @ pos and @ pos-if -.synb -.mets (pos < key < list >> [ testfun <> [ keyfun ]]) -.mets (pos-if < predfun < list <> [ keyfun ]) -.syne -.desc -The -.code pos +.code 0 and -.code pos-if -functions search through -.meta list -for an item which matches -.metn key , -or satisfies predicate function -.metn predfun , +.code t respectively. -They return the zero-based position of the matching item. -The -.meta keyfun -argument specifies a function which is applied to the elements -of -.meta list -to produce the comparison key. If this argument is omitted, -then the untransformed elements of -.meta list -are examined. +The description of the dwim operator\(emin particular, the section +on Range Indexing\(emexplains the semantics of the range specification. -The -.code pos -function's -.meta testfun -argument specifies the test function which -is used to compare the comparison keys from -.meta list -to -.metn key . -If this argument is omitted, then the -.code equal -function is used. -The position of the first element -.meta list -whose comparison key (as -retrieved by -.metn keyfun ) -matches the search (under -.metn testfun ) -is -returned. If no such element is found, -.code nil -is returned. +The second form of the replace function replaces a subsequence of +elements from +.meta sequence +given by +.metn index-list , +with their counterparts +from +.metn replacement-sequence . +This form of the replace function does not insert +or delete; it simply overwrites elements. If +.meta replacement-sequence +and +.meta index-list +are of different lengths, then the shorter of the two determines +the maximum number of elements which are overwritten. +Whenever a negative value occurs in +.meta index-list +the length of +.meta sequence +is added to that value. +Furthermore, similar restrictions apply on +.meta index-list +as under the +select function. Namely, the replacement stops when an index value +in +.meta index-list +is encountered which is out of range for +.metn sequence . +furthermore, if +.meta sequence +is a list, then +.meta index-list +must +be monotonically increasing, after consideration of the +displacement of negative values. -The -.code pos-if -function's -.meta predfun -argument specifies a predicate function -which is applied to the successive comparison keys taken from -.meta list -by applying -.meta keyfun -to successive elements. The position of -the first element for which -.meta predfun -yields true is returned. If -no such element is found, -.code nil -is returned. +If +.meta sequence +is a +.code carray +object, then +.code replace +behaves like +.codn carray-replace . -.coNP Functions @, rposq @, rposql @, rposqual @ rpos and @ rpos-if -.synb -.mets (rposq < object << list ) -.mets (rposql < object << list ) -.mets (rposqual < object << list ) -.mets (rpos < key < list >> [ testfun <> [ keyfun ]]) -.mets (rpos-if < predfun < list <> [ keyfun ]) -.syne -.desc -These functions are counterparts of -.codn rposq , -.codn rposql , -.codn rposqual , -.code rpos +If +.meta sequence +is a +.code buf +object, then +.code replace +behaves like +.codn buf-replace . + +If +.meta sequence +is a structure, then the structure must support the +.code lambda-set +method. The +.code replace +operation is translated into a call of the +.code lambda-set +method according to the following equivalences: + +.verb + (replace o items from to) + <--> o.(lambda-set (rcons from to) items) + + (replace o items index-list) + <--> o.(lambda-set index-list items) +.brev + +Thus, the +.meta from and -.code rpos-if -which report position of the right-most matching item, -rather than the left-most. +.meta to +arguments are converted to single range object, +whereas an +.meta index-list +is passed as-is. +It is an error if the +.code from +argument is a sequence, indicating an +.metn index-list , +and a +.code to +argument is also given; the situation is diagnosed. If either +.code from +or +.code to +are omitted, the range object contains the +.code : +symbol in the corresponding place: -.coNP Functions @ pos-max and @ pos-min +.verb + (replace o items from) + <--> o.(lambda-set (rcons from :) items) + + (replace o items : to) + <--> o.(lambda-set (rcons : to) items) + + (replace o items) + <--> o.(lambda-set (rcons : :) items) +.brev + +It is the responsibility of the object's +.code lambda-set +method to implement semantics consistent with the +description of +.codn replace . + +.coNP Function @ take .synb -.mets (pos-max < sequence >> [ testfun <> [ keyfun ]]) -.mets (pos-min < sequence >> [ testfun <> [ keyfun ]]) +.mets (take < count << sequence ) .syne .desc The -.code pos-min -and -.code pos-max -functions implement exactly the same algorithm; they -differ only in their defaulting behavior with regard to the -.meta testfun -argument. If -.meta testfun -is not given, then the pos-max function defaults -.meta testfun -to the -.code greater -function, whereas -.code pos-min -defaults it to the -.code less -function. +.code take +function returns +.meta sequence +with all except the first +.meta count +items removed. If .meta sequence -is empty, both functions return -.codn nil . +is a list, then +.code take +returns a lazy list which produces the first +.meta count +items of sequence. -Without a -.meta testfun -argument, the -.code pos-max -function finds the zero-based -position index of the numerically maximum value occurring in -.metn sequence , -whereas -.code pos-min -without a -.meta testfun -argument finds the index of the minimum -value. +For other kinds of sequences, including lazy strings, +.code take +works eagerly. -If a -.meta testfun -argument is given, the two functions are equivalent. -The -.meta testfun -function must be callable with two arguments. If -.meta testfun -behaves like a greater-than comparison, then -.code pos-max -and -.code pos-min -return the index of the maximum element. If -.meta testfun -behaves like a -.code less-than -comparison, then the functions return -the index of the minimum element. - -The -.meta keyfun -argument defaults to the -.code identity -function. Each element -from +.meta count +exceeds the length of .meta sequence -is passed through this one-argument function, and -the resulting value is used in its place. +then a sequence is returned which has all the items. +This object may be +.meta sequence +itself, or a copy. -If a sequence contains multiple equivalent maxima, -whether the position of the leftmost or rightmost such maximum is reported -depends on whether -.meta testfun -compares for strict inequality, or whether it reports true for -equal arguments also. Under the default -.metn testfun , -which is -.codn less , -the -.code pos-max -function will return the position leftmost of a duplicate set of maximum -elements. To find the rightmost of the maxima, the -.code lequal -function can be substituted. Analogous reasoning applies to other -test functions. +If +.meta count +is negative, it is treated as zero. -.coNP Function @ mismatch +.coNP Functions @ take-while and @ take-until .synb -.mets (mismatch < left-seq < right-seq >> [ testfun <> [ keyfun ]]) +.mets (take-while < predfun < sequence <> [ keyfun ]) +.mets (take-until < predfun < sequence <> [ keyfun ]) .syne .desc The -.code mismatch -function compares corresponding elements from the sequences -.meta left-seq +.code take-while and -.metn right-seq , -returning the position at which the first mismatch occurs. - -If the sequences are of the same length, and their corresponding -elements are the same, then -.code nil -is returned. +.code take-until +functions return a prefix of +.meta sequence +whose items satisfy certain conditions. -If one sequence is shorter than the other, and matches a prefix -of the other, then the mismatching position returned is one position -after the last element of the shorter sequence, the same value -as its length. An empty sequence is a prefix of every sequence. +The +.code take-while +function returns the longest prefix of +.meta sequence +whose elements, accessed through +.meta keyfun +satisfy the function +.metn predfun . The .meta keyfun -argument defaults to the -.code identity -function. Each element -from +argument defaults to the identity function: the elements +of .meta sequence -is passed to -.meta keyfun -and the resulting value is used in its place. +are examined themselves. -After being converted through +The +.code take-until +function returns the longest prefix of +.meta sequence +which consists of elements, accessed through .metn keyfun , -items are then compared using -.metn testfun , -which must accept two arguments, and defaults to -.codn equal . +that do +.B not +satisfy +.meta predfun +followed by an element which does satisfy +.metn predfun . +If +.meta sequence +has no such prefix, then an empty sequence +is returned of the same kind as +.metn sequence . -.coNP Function @ where +If +.meta sequence +is a list, then these functions return a lazy list. + +.coNP Function @ drop .synb -.mets (where < function << object ) +.mets (drop < count << sequence ) .syne .desc -If -.meta object -is a sequence, the -.code where +The +.code drop function returns -a lazy list of the numeric indices of those of its elements which satisfy -.metn function . -The numeric indices appear in increasing order. +.meta sequence +with the first +.meta count +items removed. If -.meta object -is a hash, the -.code where -function returns a lazy list of -of keys which have values which satisfy -.metn function . -These keys are not subject to an order. - -.meta function -must be a function that can be called with one argument. -For each element of -.metn object , -.meta function -is called with that element -as an argument. If a -.cod2 non- nil -value is returned, then the zero-based index of -that element is added to a list. Finally, the list is returned. - -.coNP Function @ rmismatch -.synb -.mets (rmismatch < left-seq < right-seq >> [ testfun <> [ keyfun ]]) -.syne -.desc -Similarly to -.codn mismatch , -the -.code rmismatch -function compares corresponding elements from the sequences -.meta left-seq -and -.metn right-seq , -returning the position at which the first mismatch occurs. -All of the arguments have the same semantics as that of -.codn mismatch . +.meta count +is negative, it is treated as zero. -Unlike -.codn mismatch , -.code rmismatch -compares the sequences right-to-left, finding the suffix -which they have in common, rather than prefix. +If +.meta count +is zero, then +.meta sequence +is returned. -If the sequences match, then -.code nil -is returned. Otherwise, a negative index is returned giving the -mismatching position, regarded from the end. If the sequences -match only in the rightmost element, then -1 is returned. If they -match in two elements then -2 and so forth. +If +.meta count +exceeds the length of +.meta sequence +then an empty sequence is returned +of the same kind as +.metn sequence . -.coNP Functions @ starts-with and @ ends-with +.coNP Functions @ drop-while and @ drop-until .synb -.mets (starts-with < short-seq < long-seq >> [ testfun <> [ keyfun ]]) -.mets (ends-with < short-seq < long-seq >> [ testfun <> [ keyfun ]]) +.mets (drop-while < predfun < sequence <> [ keyfun ]) +.mets (drop-until < predfun < sequence <> [ keyfun ]) .syne .desc The -.code starts-with +.code drop-while and -.code ends-with -functions compare corresponding elements from sequences -.meta short-seq +.code drop-until +functions return +.meta sequence +with a prefix of that sequence removed, +according to conditions involving +.meta predfun and -.metn long-seq . +.metn keyfun . -The -.code starts-with -function returns -.code t -if -.meta short-seq -is prefix of -.metn long-seq ; -otherwise, it returns -.codn nil . The -.code ends-with -function returns -.code t -if -.meta short-seq -is suffix of -.metn long-seq ; -otherwise, it returns -.codn nil . - -Element from both sequences are mapped to comparison keys using -.metn keyfun , -which defaults to -.codn identity . - -Comparison keys are compared using -.meta testfun -which defaults to -.codn equal . +.code drop-while +function removes the longest prefix of +.meta sequence +whose elements, accessed through +.meta keyfun +satisfy the function +.metn predfun , +and returns the remaining sequence. -.coNP Function @ select -.synb -.mets (select < object >> { index-list <> | function }) -.syne -.desc The -.code select -function returns an object, of the same kind as -.metn object , -which consists of those elements of -.meta object -which are identified by -the indices in -.metn index-list , -which may be a list or a vector. - -If -.meta function -is given instead of -.metn index-list , -then -.meta function -is invoked with -.meta object -as its argument. The return value is then taken as -if it were the -.meta index-list -argument . +.meta keyfun +argument defaults to the identity function: the elements +of +.meta sequence +are examined themselves. -If -.meta object -is a sequence, then -.meta index-list -consists of numeric -indices. The length of the sequence, as reported by the -.code length -function, is added to every -.meta index-list -value which is negative. The -.code select -function stops collecting values upon encountering an index value which is -greater than or equal to the length of the sequence. -(Rationale: without -this strict behavior, -.code select -would not be able to terminate if -.meta index-list -is infinite.) - -If -.meta object -is, more specifically, a list-like sequence, then -.meta index-list -must contain monotonically increasing -numeric values, even if no value is out of range, since the -.code select -function -makes a single pass through the list based on the assumption that indices -are ordered. (Rationale: optimization.) -This requirement for monotonicity applies to the values which -result after negative indices are displaced by the sequence length -Also, in this list-like sequence case, values taken from -.meta index-list -which are still negative after being displaced by the sequence length are -ignored. +.code drop-until +function removes the longest prefix of +.meta sequence +which consists of elements, accessed through +.metn keyfun , +that do +.B not +satisfy +.meta predfun +followed by an element which does satisfy +.metn predfun . +A sequence of the remaining elements is +returned. If -.meta object -is a hash, then -.meta index-list -is a list of keys. A new hash is -returned which contains those elements of -.meta object -whose keys appear -in -.metn index-list . -All of -.meta index-list -is processed, even if it contains -keys which are not in -.metn object . -The nonexistent keys are ignored. - -The -.code select -function also supports objects of type -.codn carray , -in a manner similar to vectors. The indicated elements are extracted -from the input object, and a new -.code carray -is returned whose storage is initialized by converting the extracted -values back to the foreign representation. +.meta sequence +has no such prefix, then a sequence +same as +.meta sequence +is returned, which may be +.meta sequence +itself or a copy. -.coNP Function @ relate +.coNP Accessor @ last .synb -.mets (relate < domain-seq < range-seq <> [ default-val ]) +.mets (last < seq <> [ num ]) +.mets (set (last < seq <> [ num ]) << new-value) .syne .desc The -.code relate -function returns a one-argument function which implements the relation formed -by mapping the elements of -.meta domain-seq -to the positionally corresponding elements of -.metn range-seq . -That is to say, the function searches through the sequence -.meta domain-seq -to determine the position where its argument occurs, using -.code equal -as the comparison function. -Then it returns the element from that position in the -.meta range-seq -sequence. This returned function is called the -.IR "relation function" . +.meta last +function returns a subsequence of +.meta seq +consisting of the last +.meta num +of its elements, where +.meta num +defaults to 1. -If the relation function's argument is not found in -.metn domain-seq , -then the behavior depends on the optional parameter -.metn default-val . -If an argument is given for -.metn default-val , -then the relation function returns that value. -Otherwise, the relation function returns its argument. +If +.meta num +is zero or negative, then an empty sequence is returned. +If +.meta num +is positive, and greater than or equal to the length of seq, +then seq +.meta seq +is returned. -Note: the -.code relate -function may be understood in terms of the following equivalences: +If a +.code last +form is used as a place, then +.code seq +must be a place. The following equivalence gives the semantics +of assignment to a +.codn last : .verb - (relate d r) <--> (lambda (arg) - (iflet ((p (posqual arg d))) - [r p] - arg)) - - (relate d r v) <--> (lambda (arg) - (iflet ((p (posqual arg d))) - [r p] - v)) + (set (last x n) v) <--> (set (sub x (- (max n 0)) t) v) .brev -.TP* Examples: +A +.code last +place is deletable. The semantics of deletion may be understood +in terms of the following equivalence: .verb - (mapcar (relate "_" "-") "foo_bar") -> "foo-bar" - - (mapcar (relate "0123456789" "ABCDEFGHIJ" "X") "139D-345") - -> "BJDXXDEF" - - (mapcar (relate '(nil) '(0)) '(nil 1 2 nil 4)) -> (0 1 2 0 4) + (del (last x n)) <--> (del (sub x (- (max n 0)) t)) .brev -.coNP Function @ in +.coNP Accessor @ butlast .synb -.mets (in < sequence < key >> [ testfun <> [ keyfun ]]) -.mets (in < hash << key ) +.mets (butlast < sequence <> [ num ]) +.mets (set (butlast < sequence <> [ num ]) << new-value ) .syne .desc The -.code in -function tests whether -.meta key -is found inside -.meta sequence -or -.metn hash . - -If the -.meta testfun -argument is specified, it specifies the function -which is used to comparison keys from the sequence -to -.metn key . -Otherwise the -.code equal -function is used. - -If the -.meta keyfun -argument is specified, it specifies a function which -is applied to the elements of +.code butlast +function returns the prefix of .meta sequence -to produce the comparison keys. Without this -argument, the elements themselves are taken -as the comparison keys. - -If the object being searched is a hash, then the -.meta keyfun -and -.meta testfun -arguments are ignored. +consisting of a copy of it, with the last +.meta num +items removed. -The -.code in -function returns -.code t -if it finds -.meta key -in -.meta sequence -or -.metn hash , -otherwise -.codn nil . +The parameter +.meta num +defaults to 1 +if an argument is omitted. -.coNP Function @ partition -.synb -.mets (partition < sequence >> { index-list | index | << function }) -.syne -.desc If .meta sequence -is empty, then -.code partition -returns an empty list, and the -second argument is ignored; if it is -.metn function , -it is not called. - -Otherwise, -.code partition -returns a lazy list of partitions of -.metn sequence . -Partitions are consecutive, non-overlapping, non-empty sub-strings of -.metn sequence , -of the same kind as -.metn sequence , -such that if these sub-strings are catenated together in their order -of appearance, a sequence -.code equal -to the original is produced. - -If the second argument is of the form -.metn index-list , -or if an -.meta index-list -was produced from the -.meta index -or -.meta function -arguments, each value in that list must be an integer. Each integer -value which is non-negative specifies the index position -given by its value. Each integer value which is negative -specifies an index position given by adding the length of -.meta sequence -to its value. The sequence index positions thus denoted by -.meta index-list -shall be strictly non-decreasing. Each successive element -is expected to designate an index position at least as high -as all previous elements, otherwise the behavior is unspecified. -Leading index positions which are (still) negative, or zero, are effectively -ignored. +is empty, an empty sequence is returned. If -.meta index-list -is empty then a one-element list containing the entire +.meta num +is zero or negative, then .meta sequence is returned. If -.meta index-list -is an infinite lazy list, the function shall terminate if that -list eventually produces an index position which is greater than or equal to -the length of -.metn sequence . - -If the second argument is a function, then this function is applied -to +.meta num +is positive, and meets or exceeds the length of .metn sequence , -and the return value of this call is then used in place of the -second argument, which must be a single index value, which is then -taken as if it were the -.meta index -argument, or else a list of indices, which are taken as the -.meta index-list -argument. - -If the second argument is an atom other than a function, it is assumed to be -an integer index, and is turned into an -.meta index-list -of one element. +then an empty sequence is returned. -After the -.meta index-list -is obtained as an argument, or determined from the -.meta index -or -.meta function -arguments, the -.code partition -function then divides +If a +.code butlast +form is used as a place, then .meta sequence -according to the indices given by that list. -The first partition begins with the first element of -.metn sequence . -The second partition begins at the first position in -.metn index-list , -and so on. Indices beyond the length of the sequence are ignored, -as are indices less than or equal to zero. +must itself be a place. The following equivalence gives the semantics +of assignment to a +.codn last : -.TP* Examples: .verb - (partition '(1 2 3) 1) -> ((1) (2 3)) + (set (butlast x n) v) <--> (set (sub x 0 (- (max n 0))) v) +.brev - ;; split the string where there is a "b" - (partition "abcbcbd" (op where (op eql #\eb))) -> ("a" "bc" - "bc" "bd") +A +.code butlast +place is deletable. The semantics of deletion may be understood +in terms of the following equivalence: + +.verb + (del (last x n)) <--> (del (sub x 0 (- (max n 0)))) .brev -.coNP Functions @ split and @ split* +Note: the \*(TL +.code take +function also computes the prefix of a list; however, it counts items +from the beginning, and provides lazy semantics which allow it +to work with infinite lists. + +See also: the +.code butlastn +accessor, which operates on lists. That function has useful semantics for +improper lists and treats an atom as the terminator of a zero-length improper +list. + +Dialect note: a destructive function similar to Common Lisp's +.code nbutlast +isn't provided. Of course, assignment to an +.code butlast +form is destructive; Common Lisp doesn't support +.code butlast +as a place. + +.coNP Function @ ldiff .synb -.mets (split < sequence >> { index-list | index | << function }) -.mets (split* < sequence >> { index-list | index | << function }) +.mets (ldiff < sequence << tail-sequence ) .syne .desc -If -.meta sequence -is empty, then both -.code split -and -.code split* -return an empty list, and the -second argument is ignored; if it is -.metn function , -it is not called. - -Otherwise, -.code split -returns a lazy list of pieces of -.metn sequence : -consecutive, non-overlapping, possibly empty sub-strings of -.metn sequence , -of the same kind as -.metn sequence . -A catenation of these pieces in the order they appear would produce -a sequence that is -.code equal -to the original sequence. - The -.code split* -function differs from -.code split -in that the elements indicated by the split indices are removed. +.code ldiff +function is a somewhat generalized version of the same-named classic Lisp +function found in traditional Lisp dialects. The -.metn index , -.metn index-list , -and -.meta function -arguments are subject to the same restrictions and treatment -as the corresponding arguments of the -.code partition -function, with the following difference: the index positions indicated by -.code index-list -are required to be strictly increasing, rather than non-decreasing. +.code ldiff +function supports the original +.code ldiff +semantics when both inputs are lists. It determines whether the +.meta tail-sequence +list is a structural suffix of +.metn sequence ; +which is to say: is +.meta tail-sequence +one of the +.code cons +cells which comprise +.metn sequence ? +If so, then a list is returned consisting of all the items of +.meta sequence +before +.metn tail-sequence : +a copy of +.meta sequence +with the +.meta tail-sequence +part removed, and replaced by the +.code nil +terminator. If +.meta tail-sequence +is +.code nil +or the lists are unrelated, then +.meta sequence +is returned. -If the second argument is of the form -.metn index-list , -or if an -.meta index-list -was produced from the -.meta index -or -.meta function -arguments, then the -.code split -function divides +The \*(TL +.code ldiff +function supports the following additional semantics. + +.RS +.IP 1. +The basic description of +.code ldiff +is extended to work with list-like sequences, not +merely lists; that is to say, objects which support the +.code car +method. +.IP 2. +If .meta sequence -according to the indices indicated in the list. The first piece always begins -with the first element of +is any kind of sequence, and +.meta tail-sequence +is any kind of empty sequence, then +.meta sequence +is returned. +.IP 3. +If either argument is an atom that is not a sequence, +.code ldiff +returns .metn sequence . -Each subsequent piece begins with the position indicated by -an element of -.metn index-list . -Negative indices are ignored. +.IP 4. If -.meta index-list -includes index zero, -then an empty first piece is generated. +.meta sequence +is a list-like sequence, and +.meta tail-sequence +isn't, then the terminating atom of +.meta sequence +is determined. This atom is compared using +.code equal +to the +.meta tail-sequence +object. If they are equal, then a proper list is +returned containing the items of +.meta sequence +excluding the terminating atom. +.IP 5. +If both arguments are vector-like sequences, then +.code ldiff +determines whether +.meta sequence +has a suffix which is +.code equal +to +.metn tail-sequence . +If this is the case, then a sequence is returned, of the same kind as +.metn sequence , +consisting of the items of +.meta sequence +before that suffix. If -.meta index-list -includes an index greater than or equal to the length of +.meta tail-sequence +is not +.code equal +to a suffix of +.metn sequence , +then .meta sequence -(equivalently, an index beyond the last element of the sequence) -then an additional empty last piece is generated. -The length of +is returned. +.IP 6 +In all other cases, .meta sequence -is added to any negative indices. An index which is still negative -after being thus displaced is discarded. - -Note: the principal difference between -.code split and -.code partition -is that -.code partition -does not produce empty pieces. +.meta tail-sequence +are compared with +.codn equal . +If the comparison is true, +.code nil +is returned, otherwise +.meta sequence +is returned. +.RE .TP* Examples: + .verb - (split '(1 2 3) 1) -> ((1) (2 3)) + ;;; unspecified: the compiler could make + ;;; '(2 3) a suffix of '(1 2 3), + ;;; or they could be separate objects. + (ldiff '(1 2 3) '(2 3)) -> either (1) or (1 2 3) - (split "abc" 0) -> ("" "abc") - (split "abc" 3) -> ("abc" "") - (split "abc" 1) -> ("a" "bc") - (split "abc" '(0 1 2 3)) -> ("" "a" "b" "c" "") - (split "abc" '(1 2)) -> ("a" "b" "c") + ;; b is the (1 2) suffix of a, so the ldiff is (1) + (let* ((a '(1 2 3)) (b (cdr a))) + (ldiff a b)) + -> (1) - (split "abc" '(-1 1 2 15)) -> ("a" "b" "c") + ;; Rule 5: strings and vector + (ldiff "abc" "bc") -> "a" + (ldiff "abc" nil) -> "abc" + (ldiff #(1 2 3) #(3)) -> #(1 2) - ;; triple split at makes two additional empty pieces - (split "abc" '(1 1 1)) -> ("a" "" "" "bc") + ;; Rule 5: mixed vector kinds + (ldiff "abc" #(#\eb #\ec)) -> "abc" - (split* "abc" 0) -> ("" "bc") ;; "a" is removed + ;; Rule 6: + (ldiff #(1 2 3) '(3)) -> #(1 2 3) - ;; all characters removed - (split* "abc" '(0 1 2)) -> ("" "" "" "") + ;; Rule 4: + (ldiff '(1 2 3) #(3)) -> '(1 2 3) + (ldiff '(1 2 3 . #(3)) #(3)) -> '(1 2 3) + (ldiff '(1 2 3 . 4) #(3)) -> '(1 2 3 . 4) + + ;; Rule 6 + (ldiff 1 2) -> 1 + (ldiff 1 1) -> nil .brev -.coNP Function @ partition* +.coNP Function @ search .synb -.mets (partition* < sequence >> { index-list >> | index <> | function }) +.mets (search < haystack < needle >> [ testfun <> [ keyfun ]) .syne .desc -If -.meta sequence -is empty, then -.code partition* -returns an empty list, and the -second argument is ignored; if it is -.metn function , -it is not called. - The -.metn index , -.metn index-list , +.code search +function determines whether the sequence +.meta needle +occurs as substring +within +.metn haystack , +under the given comparison function +.meta testfun and -.meta function -arguments are subject to the same restrictions and treatment -as the corresponding arguments of the -.code partition -function, with the following difference: the index positions indicated by -.code index-list -are required to be strictly increasing, rather than non-decreasing. +key function +.metn keyfun . +If this is the case, then the zero-based position of +the leftmost occurrence of +.meta key +within +.meta haystack +is returned. Otherwise +.code nil +is returned to indicate that +.meta key +does not occur within +.metn haystack . +If +.meta key +is empty, then zero is always returned. -If the second argument is of the form -.metn index-list , -then -.code partition* -produces a -lazy list of pieces taken from -.metn sequence . -The pieces are formed by deleting from -.meta sequence -the elements at the positions given -in -.metn index-list , -such that the pieces are the remaining non-empty sub-strings from -between the deleted elements, maintaining their order. +The arguments +.meta haystack +and +.meta needle +are sequences: lists, vectors +or strings, in any combination. If -.meta index-list -is empty then a one-element list containing the entire -.meta sequence -is returned. +.meta needle +is not empty, then occurs at some position N within +.meta haystack +if +the first element of +.meta needle +matches the element at position N of +.metn haystack , +the second element of +.meta needle +matches the element at position N+1 of +.meta haystack +and so forth, for all elements of +.metn needle . +A match between elements +is determined by passing each element through +.metn keyfun , +and then comparing the resulting values using +.metn testfun . -.TP* Examples: -.verb - (partition* '(1 2 3 4 5) '(0 2 4)) -> ((1) (3) (5)) +If +.meta testfun +is supplied, it must be a function which can be +called with two arguments. If it is not supplied, it defaults to +.codn eql . - (partition* "abcd" '(0 3)) -> "bc" +If +.meta keyfun +is supplied, it must be a function which can be called +with one argument. If it is not supplied, it defaults to +.codn identity . - (partition* "abcd" '(0 1 2 3)) -> nil -.brev +.TP* Examples: -.coNP Functions @ find and @ find-if -.synb -.mets (find < key < sequence >> [ testfun <> [ keyfun ]]) -.mets (find-if < predfun >> { sequence | << hash } <> [ keyfun ]) -.syne -.desc -The -.code find -and -.code find-if -functions search through a sequence for an item which -matches a key, or satisfies a predicate function, respectively. +.verb + ;; fails because 3.0 doesn't match 3 + ;; under the default eql function + [search #(1.0 3.0 4.0 7.0) '(3 4)] -> nil -The -.meta keyfun -argument specifies a function which is applied to the elements -of -.meta sequence -to produce the comparison key. If this argument is omitted, -then the untransformed elements of the -.meta sequence -are searched. + ;; occurrence found at position 1: + ;; (3.0 4.0) matches (3 4) under = + [search #(1.0 3.0 4.0 7.0) '(3 4) =] -> 1 -The -.code find -function's -.meta testfun -argument specifies the test function which -is used to compare the comparison keys from -.meta sequence -to the search key. -If this argument is omitted, then the -.code equal -function is used. -The first element from the list whose comparison key (as retrieved by -.metn keyfun ) -matches the search (under -.metn testfun ) -is returned. If no such element is found, -.code nil -is returned. + ;; "even odd odd odd even" pattern + ;; matches at position 2 + [search #(1 1 2 3 5 7 8) '(2 1 1 1 2) : evenp] -> 2 -The -.code find-if -function's -.meta predfun -argument specifies a predicate function -which is applied to the successive comparison keys pulled from the list -by applying -.meta keyfun -to successive elements. The first element -for which -.meta predfun -yields true is returned. If no such -element is found, -.code nil -is returned. + ;; Case insensitive string search + [search "abcd" "CD" : chr-toupper] -> 2 -In the case of -.codn find-if , -a hash table may be specified instead of a sequence. -The -.meta hash -is treated as if it were a sequence of hash key and hash -value pairs represented as cons cells, the -.code car -slots of which are the hash keys, and the -.code cdr -of which are the hash values. If the caller doesn't specify a -.meta keyfun -then these cells are taken as their keys. + ;; Case insensitive string search + ;; using vector of characters as key + [search "abcd" #(#\eC #\eD) : chr-toupper] -> 2 +.brev -.coNP Functions @ rfind and @ rfind-if +.coNP Function @ rsearch .synb -.mets (rfind < key < sequence >> [ testfun <> [ keyfun ]]) -.mets (rfind-if < predfun >> { sequence | << hash } <> [ keyfun ]) +.mets (rsearch < haystack < needle >> [ testfun <> [ keyfun ]) .syne .desc The -.code rfind -and -.code rfind-if -functions are almost exactly like -.code find -and -.code find-if -except that if there are multiple matches for -.meta key -in -.metn sequence , -they return the right-most element rather than +.code rsearch +function is like +.code search +except that if +.meta needle +matches +.meta haystack +in multiple places, +.code rsearch +returns the right-most matching position rather than the leftmost. -In the case of -.code rfind-if -when a -.meta hash -is specified instead of a -.metn sequence , -the function searches through the hash entries in the same order as -.codn find-if , -but finds the last match rather than the first. -Note: hashes are inherently not ordered; the relative order of items in -a hash table can change when other items are inserted or deleted. - -.coNP Functions @ find-max and @ find-min +.coNP Functions @ ref and @ refset .synb -.mets (find-max >> { sequence | << hash } >> [ testfun <> [ keyfun ]]) -.mets (find-min >> { sequence | << hash } >> [ testfun <> [ keyfun ]]) +.mets (ref < seq << index ) +.mets (refset < seq < index << new-value ) .syne .desc The -.code find-min +.code ref and -.code find-max -function implement exactly the same algorithm; they -differ only in their defaulting behavior with regard to the -.meta testfun -argument. If -.meta testfun -is not given, then the find-max function defaults it to -the -.code greater -function, whereas -.code find-min -defaults it to the -.code less -function. - -Without a -.meta testfun -argument, the -.code find-max -function finds the numerically -maximum value occurring in -.metn sequence , -whereas -.code pos-min -without a -.meta testfun -argument finds the minimum value. - -If a -.meta testfun -argument is given, the two functions are equivalent. -The -.meta testfun -function must be callable with two arguments. -If -.meta testfun -behaves like a greater-than comparison, then -.code find-max +.code refset +functions perform array-like indexing into sequences, as well as +objects of type +.code buf and -.code find-min -both return the maximum element. If -.meta testfun -behaves like a less-than comparison, then the functions return -the minimum element. +.codn carray . -The -.meta keyfun -argument defaults to the -.code identity -function. Each element -from -.meta sequence -is passed through this one-argument function, and -the resulting value is used in its place for the purposes of the -comparison. However, the original element is returned. +If the +.meta seq +parameter is a hash, then these functions perform +has retrieval and storage; in that case +.meta index +isn't restricted to an integer value. -A hash table may be specified instead of a sequence. -The -.meta hash -is treated as if it were a sequence of hash key and hash -value pairs represented as cons cells, the -.code car -slots of which are the hash keys, and the -.code cdr -of which are the hash values. If the caller doesn't specify a -.meta keyfun -then these cells are taken as their keys. To find the hash -table's key-value cell with the maximum key, the +If +.meta seq +is a structure, it supports +.code ref +directly if it has a +.code lambda +method. The +.meta index +argument is passed to that method, and the resulting value is +returned. +If a structure lacks a +.code lambda +method, but has a .code car -function can be specified as -.metn keyfun . -To find the entry holding the maximum value, the -.code cdr -function can be specified. - -If there are multiple equivalent maxima, then under the default -.metn testfun , -that being -.codn less , -the leftmost one is reported. See the notes under -.code pos-max -regarding duplicate maxima. - -.coNP Functions @, uni @, isec @ diff and @ symdiff -.synb -.mets (uni < seq1 < seq2 >> [ testfun <> [ keyfun ]]) -.mets (isec < seq1 < seq2 >> [ testfun <> [ keyfun ]]) -.mets (diff < seq1 < seq2 >> [ testfun <> [ keyfun ]]) -.mets (symdiff < seq1 < seq2 >> [ testfun <> [ keyfun ]]) -.syne -.desc -The functions -.codn uni , -.codn isec , -.code diff -and -.code symdiff -treat the sequences -.meta seq1 -and -.meta seq2 -as if they were sets. +method, then +.code ref +treats it as a list, traversing the structure using +.cod3 car / cdr +operations. In the absence of support for these operations, +the function fails with an error exception. -They, respectively, compute the set union, set intersection, -set difference and symmetric difference of -.meta seq1 +Similarly, a structure supports +.code refset +directly if it has a +.code lambda-set +method. This gets called with +.meta index and -.metn seq2 , -returning a new sequence. +.meta new-value +as arguments. Then +.meta new-value +is returned. +If a structure lacks a +.code lambda-set +method, then +.code refset +treats it as a list, traversing the structure using +.cod3 car / cdr +operations, and storing +.meta new-value +using +.codn rplaca . +In the absence of support for these operations, +the function fails with an error exception. -The arguments -.meta seq1 -and -.meta seq2 -need not be of the same kind. They may be hash tables. +The +.code ref +function retrieves an element of +.metn seq , +whereas +.code refset +overwrites an +element of +.meta seq +with a new value. -The returned sequence is of the same kind as -.metn seq1 . If -.meta seq1 -is a hash table, the returned sequence is a list. - -For the purposes of these functions, an input which is a hash table -is considered as if it were a sequence of hash key and hash -value pairs represented as cons cells, the -.code car -slots of which are the hash keys, and the -.code cdr -of which are the hash values. This means that if no -.meta keyfun -is specified, these pairs are taken as keys. - -Since the input sequences are defined as representing sets, they are assumed -not to contain duplicate elements. These functions are not required, but may, -de-duplicate the sequences. +.meta seq +is a sequence then +.meta index +argument must be an integer. The first element of the sequence +is indexed by zero. Negative values are permitted, +denoting backward indexing from the end of the sequence, such that +the last element is indexed by -1, the second last by -2 and so on. +See also the Range Indexing section under the +description of the +.code dwim +operator. -The union sequence produced by -.code uni -contains all of the elements which occur in both -.meta seq1 -and -.metn seq2 . -If a given element occurs exactly once only in -.meta seq1 -or exactly once only in -.metn seq2 , -or exactly once in both sequences, then it occurs exactly once in the union -sequence. If a given element occurs at least once in either -.metn seq1 , -.meta seq2 -or both, then it occurs at least once in the union sequence. +If +.meta seq +is a list, then out-of-range indices, whether positive or negative, +are treated leniently by +.codn ref : +such accesses produce the value +.codn nil , +rather than an error. For other sequence types, such accesses +are erroneous. For hashes, accesses to nonexistent elements +are treated leniently, and produce +.codn nil . -The intersection sequence produced by -.code isec -contains all of the elements which occur in both -.meta seq1 -and -.metn seq2 . -If a given element occurs exactly once in -.meta seq1 -and exactly once in -.metn seq2 , -then in occurs exactly once in the intersection sequence. -If a given element occurs at least once in -.meta seq1 -and at least once in -.metn seq2 , -then in occurs at least once in the intersection sequence. +The +.code refset +function is strict for out-of-range indices over all sequences, +including lists. In the case of hashes, a +.code refset +of a nonexistent key creates the key. -The difference sequence produced by -.code diff -contains all of the elements which occur in -.meta seq1 -but do not occur in -.metn seq2 . -If an element occurs exactly once in -.meta seq1 -and does not occur in -.metn seq2 , -then it occurs exactly once in the difference sequence. -If an element occurs at least once in -.meta seq1 -and does not occur in -.metn seq2 , -then it occurs at least once in the difference sequence. -If an element occurs at least once in -.metn seq2 , -then it does not occur in the difference sequence. +The +.code refset +function returns +.codn new-value . -The symmetric difference sequence produced by -.code symdiff -contains all of the elements of -.meta seq1 -which do not occur in -.meta seq2 +The following equivalences hold between +.code ref and -.IR "vice versa" : -it also contains all of the elements of -.meta seq2 -which do not occur in -.metn seq1 . +.codn refset , +and the DWIM bracket syntax, provided that +.meta idx +is a scalar index and +.meta seq +is a sequence object, rather than a hash. -Element equivalence is determined by a combination of -.meta testfun -and -.metn keyfun . -Elements are compared pairwise, and each element of a pair is passed through -.meta keyfun -function to produce a comparison value. The comparison values -are compared using -.metn testfun . -If -.meta keyfun -is omitted, then the -untransformed elements themselves are compared, and if -.meta testfun -is omitted, -then the -.code equal -function is used. +.verb + (ref seq idx) <--> [seq idx] -Note: a function similar to -.code diff -named -.code set-diff -exists. This became deprecated starting in \*(TX 184. -For the -.code set-diff -function, the requirement was specified to preserve the original -order of items from -.meta seq1 -that survive into the output sequence. -This requirement is not documented for the -.code diff -function, but is -.I "de facto" -honored by the implementation for at as long as the -.code set-diff -synonym continues to be available. -The -.code set-diff -function doesn't support hash tables and is inefficient for vectors and -strings. + (refset seq idx new) <--> (set [seq idx] new) +.brev -Note: these functions are not efficient for the processing of hash tables, -even when both inputs are hashes, the -.meta keyfun -argument is -.codn car , +The difference is that +.code ref and -.meta testfun -matches the equality used by both hash table inputs. -If applicable, the operations -.codn hash-uni , -.code hash-isec +.code refset +are first class functions which +can be used in functional programming as higher order functions, whereas the +bracket notation is syntactic sugar, and +.code set +is an operator, not a function. +Therefore the brackets cannot replace all uses of +.code ref and -.code hash-diff -should be used instead. +.codn refset . -.coNP Functions @, mapcar @, mappend @ mapcar* and @ mappend* +.coNP Function @ update .synb -.mets (mapcar < function << sequence *) -.mets (mappend < function << sequence *) -.mets (mapcar* < function << sequence *) -.mets (mappend* < function << sequence *) +.mets (update < sequence-or-hash << function ) .syne .desc -When given only one argument, the -.code mapcar -function returns -.codn nil . -.meta function -is never called. - -When given two arguments, the -.code mapcar -function applies -.meta function -to each elements of -.meta sequence -and returns a sequence of the resulting values -in the same order as the original values. -The returned sequence is the same kind as -.metn sequence , -if possible. If the accumulated values cannot be -elements of that type of sequence, then a list is returned. - -When additional sequences are given as arguments, this filtering behavior is -generalized in the following way: -.code mapcar -traverses the sequences in parallel, -taking a value from each sequence as an argument to the function. If there -are two lists, +The +.code update +function replaces each elements in a sequence, or each value +in a hash table, with the value of .meta function -is called with two arguments and so forth. -The traversal is limited by the length of the shortest sequence. -The return values of the function are collected into a new sequence which is -returned. The returned sequence is of the same kind as the leftmost -input sequence, unless the accumulated values cannot be elements of that type of -sequence, in which case a list is returned. +applied to that element +or value. -The -.code mappend -function works like -.codn mapcar , -with the following difference. -Rather than accumulating the values returned by the function into a sequence, -mappend expects the items returned by the function to be sequences which -are catenated with -.codn append , -and the resulting sequence is returned. The returned sequence is of the same -kind as the leftmost input sequence, unless the values cannot be elements -of that type of sequence, in which case a list is returned. +The sequence or hash table is returned. +.coNP Functions @, remq @ remql and @ remqual +.synb +.mets (remq < object < list <> [ key-function ]) +.mets (remql < object < list <> [ key-function ]) +.mets (remqual < object < list <> [ key-function ]) +.syne +.desc The -.code mapcar* -and -.code mappend* -functions work like -.code mapcar +.codn remq , +.code remql and -.codn mappend , -respectively. -However, they return lazy lists rather than generating the entire -output list prior to returning. - -.TP* Caveats: +.code remqual +functions produce a new list based on +.metn list , +removing the elements whose associated keys are +.codn eq , +.code eql +or +.code equal +to +.metn object . -Like -.codn mappend , -.code mappend* -must "consume" empty lists. For instance, -if the function being mapped puts out a sequence of -.codn nil -s, -then the result must be the empty list -.codn nil , -because -.code "(append nil nil nil nil ...)" +The input +.meta list +is unmodified, but the returned list may share substructure +with it. If no items are removed, it is possible that the return value is -.codn nil . - -But suppose that -.code mappend* -is used on inputs which are infinite lazy -lists, such that the function returns -.code nil -values indefinitely. -For instance: +.meta list +itself. -.verb - ;; Danger: infinite loop!!! - (mappend* (fun identity) (repeat '(nil))) -.brev +If +.meta key-function +is omitted, then the element keys compared to +.meta object +are the elements themselves. +Otherwise, +.meta key-function +is applied to each element and the resulting value +is that element's key which is compared to +.metn object . +.coNP Functions @, remq* @ remql* and @ remqual* +.synb +.mets (remq* < object << list ) +.mets (remql* < object << list ) +.mets (remqual* < object << list ) +.syne +.desc The -.code mappend* -function is caught in a loop trying to consume -and squash an infinite stream of -.codn nil -s, -and so doesn't return. +.codn remq* , +.code remql* +and +.code remqual* +functions are lazy versions of +.codn remq , +.code remql +and +.codn remqual . +Rather than computing the entire new list +prior to returning, these functions return a lazy list. + +Caution: these functions can still get into infinite looping behavior. +For instance, in +.codn "(remql* 0 (repeat '(0)))" , +.code remql +will keep consuming +the +.code 0 +values coming out of the infinite list, looking for the first item that +does not have to be deleted, in order to instantiate the first lazy value. .TP* Examples: .verb - ;; multiply every element by two - (mapcar (lambda (item) (* 2 item)) '(1 2 3)) -> (4 6 8) - - ;; "zipper" two lists together - (mapcar (lambda (le ri) (list le ri)) '(1 2 3) '(a b c)) - -> '((1 a) (2 b) (3 c))) - - ;; like append, mappend allows a lone atom or a trailing atom: - (mappend (fun identity) 3) -> (3) - (mappend (fun identity) '((1) 2)) -> (1 . 2) + ;; Return a list of all the natural numbers, excluding 13, + ;; then take the first 100 of these. + ;; If remql is used, it will loop until memory is exhausted, + ;; because (range 1) is an infinite list. - ;; take just the even numbers - (mappend (lambda (item) (if (evenp x) (list x))) '(1 2 3 4 5)) - -> (2 4) + [(remql* 13 (range 1)) 0..100] .brev -.coNP Functions @ maprod and @ maprend +.coNP Functions @, keepq @ keepql and @ keepqual .synb -.mets (maprod < function << sequence *) -.mets (maprend < function << sequence *) +.mets (keepq < object < list <> [ key-function ]) +.mets (keepql < object < list <> [ key-function ]) +.mets (keepqual < object < list <> [ key-function ]) .syne .desc The -.code maprod -and -.code maprend -functions resemble -.code mapcar +.codn keepq , +.code keepql and -.codn mappend , -respectively. When given no -.meta sequence -arguments or exactly one -.meta sequence -argument, they behave exactly like those two functions. +.code keepqual +functions produce a new list based on +.metn list , +removing the items whose keys are not +.codn eq , +.code eql +or +.code equal +to +.metn object . -When two or more -.meta sequence -arguments are present, -.code maprod -differs from -.code mapcar -in the following way. Whereas -.code mapcar -iterates over the -.meta sequence -values in parallel, taking successive tuples of element -values and passing them to -.metn function , -the -.code maprod -function iterates over all -.I combinations -of elements from the sequences: the Cartesian product. The -.code prod -suffix stands for "product". +The input +.meta list +is unmodified, but the returned list may share substructure +with it. If no items are removed, it is possible that the return value +is +.meta list +itself. -If one or more -.meta sequence -arguments specify an empty sequence, then the Cartesian product is empty. -In this situation, -.meta function -is not called. The result of the function is then -.code nil -converted to the same kind of sequence as the leftmost -.metn sequence . +The optional +.meta key-function +is applied to each element from the +.meta list +to convert it to a key which is compared to +.metn object . +If +.meta key-function +is omitted, then each element itself of +.meta list +is compared to +.metn object . +.coNP Functions @, remove-if @, keep-if @ remove-if* and @ keep-if* +.synb +.mets (remove-if < predicate-function < list <> [ key-function ]) +.mets (keep-if < predicate-function < list <> [ key-function ]) +.mets (remove-if* < predicate-function < list <> [ key-function ]) +.mets (keep-if* < predicate-function < list <> [ key-function ]) +.syne +.desc The -.code maprod -function collects the values into a list just as -.code mapcar -does. Just like -.codn mapcar , -it converts the resulting list into the same kind of sequence -as the leftmost -.meta sequence -argument, if possible. For instance, if the resulting list is -a list or vector of characters, and the leftmost -.meta sequence -is a character string, then the list or vector of characters -is converted to a character string and returned. +.code remove-if +function produces a list whose contents are those of +.meta list +but with those elements removed which satisfy +.metn predicate-function . +Those elements which are not removed appear in the same order. +The result list may share substructure with the input list, +and may even be the same list object if no items are removed. -The -.code maprend -function ("map product through function and append") iterates the -.meta sequence -element combinations exactly like -.codn maprod , -passing them as arguments to -.metn function . -The values returned by -.meta function -are then treated exactly as by the -.code mappend -function. The return values are expected to be sequences which -are appended together as if by -.codn append , -and the final result is converted to the same kind of sequence as the leftmost -.meta sequence -if possible. +The optional +.meta key-function +specifies how each element from the +.meta list +is transformed to an argument to +.metn predicate-function . +If this argument is omitted +then the predicate function is applied to the elements directly, a behavior +which is identical to +.meta key-function +being +.codn "(fun identity)" . -The combination iteration gives priority to the rightmost -.metn sequence , -which means that the rightmost element of each generated tuple varies -fastest: the tuples are traversed in "rightmost major" order. -This is made clear in the examples. +The +.code keep-if +function is exactly like +.codn remove-if , +except the sense of +the predicate is inverted. The function +.code keep-if +retains those items +which +.code remove-if +will delete, and removes those that +.code remove-if +will preserve. -.TP* Examples +The +.code remove-if* +and +.code keep-if* +functions are like +.code remove-if +and +.codn keep-if , +but produce lazy lists. +.TP* Examples: .verb - [maprod list '(0 1 2) '(a b) '(i ii iii)] - -> - ((0 a i) (0 a ii) (0 a iii) (0 b i) (0 b ii) (0 b iii) - (1 a i) (1 a ii) (1 a iii) (1 b i) (1 b ii) (1 b iii) - (2 a i) (2 a ii) (2 a iii) (2 b i) (2 b ii) (2 b iii)) - - ;; Vectors #(#\ea #\ex) #(#\ea #\ey) ... are appended - ;; together resulting in #(#\ea #\ex #\ea #\ey ...) - ;; which is converted to a string: + ;; remove any element numerically equal to 3. + (remove-if (op = 3) '(1 2 3 4 3.0 5)) -> (1 2 4 5) - [maprend vec "ab" "xy"] -> "axaybxby" + ;; remove those pairs whose first element begins with "abc" + [remove-if (op equal [@1 0..3] "abc") + '(("abcd" 4) ("defg" 5)) + car] + -> (("defg" 5)) - ;; One of the sequences is empty, so the product is an - ;; empty sequence of the same kind as the leftmost - ;; sequence argument, thus an empty string: - [maprend vec "ab" ""] -> "" + ;; equivalent, without test function + (remove-if (op equal [(car @1) 0..3] "abc") + '(("abcd" 4) ("defg" 5))) + -> (("defg" 5)) .brev -.coNP Function @ mapdo +.coNP Functions @, countqual @ countql and @ countq .synb -.mets (mapdo < function << sequence *) +.mets (countq < object << list ) +.mets (countql < object << list ) +.mets (countqual < object << list ) .syne .desc The -.code mapdo -function is similar to -.codn mapcar , -but always returns -.codn nil . -It is useful -when -.meta function -performs some kind of side effect, hence the "do" in the name, -which is a mnemonic for the execution of imperative actions. - -When only the -.meta function -argument is given, -.meta function -is never called, +.codn countq , +.code countql and -.code nil -is returned. - -If a single -.meta sequence -argument is given, then -.code mapdo -iterates over -.metn sequence , -invoking -.meta function -on each element. - -If two or more -.meta sequence -arguments are given, then -.code mapdo -iterates over -the sequences in parallel, extracting parallel tuples of items. These -tuples are passed as arguments to -.metn function , -which must accept as many -arguments as there are sequences. +.code countqual +functions count the number of objects +in +.meta list +which are +.codn eq , +.code eql +or +.code equal +to +.metn object , +and return the count. -.coNP Functions @ transpose and @ zip +.coNP Function @ count-if .synb -.mets (transpose << sequence ) -.mets (zip << sequence *) +.mets (count-if < predicate-function < list <> [ key-function ]) .syne .desc The -.code transpose -function performs a transposition on -.metn sequence . -This means that the -elements of -.meta sequence -must be sequences. These sequences are understood to be -columns; transpose exchanges rows and columns, returning a sequence of the rows -which make up the columns. The returned sequence is of the same kind as -.metn sequence , -and the rows are also the same kind of sequence as the first column -of the original sequence. The number of rows returned is limited by the -shortest column among the sequences. +.code count-if +function counts the number of elements of +.meta list +which satisfy +.meta predicate-function +and returns the count. -All of the input sequences (the elements of -.metn sequence ) -must have elements -which are compatible with the first sequence. This means that if the first -element of -.meta sequence -is a string, then the remaining sequences must be -strings, or else sequences of characters, or of strings. +The optional +.meta key-function +specifies how each element from the +.meta list +is transformed to an argument to +.metn predicate-function . +If this argument is omitted +then the predicate function is applied to the elements directly, a behavior +which is identical to +.meta key-function +being +.codn "(fun identity)" . +.coNP Functions @, posq @ posql and @ posqual +.synb +.mets (posq < object << list ) +.mets (posql < object << list ) +.mets (posqual < object << list ) +.syne +.desc The -.code zip -function takes variable arguments, and is equivalent to calling -.code transpose -on a list of the arguments. The following equivalences hold: - -.verb - (zip . x) <--> (transpose x) - - [apply zip x] <--> (transpose x) -.brev +.codn posq , +.code posql +and +.code posqual +functions return the zero-based position of the +first item in +.meta list +which is, respectively, +.codn eq , +.code eql +or +.code equal +to +.metn object . -.TP* Examples: -.verb - ;; transpose list of lists - (transpose '((a b c) (c d e))) -> ((a c) (b d) (c e)) +.coNP Functions @ pos and @ pos-if +.synb +.mets (pos < key < list >> [ testfun <> [ keyfun ]]) +.mets (pos-if < predfun < list <> [ keyfun ]) +.syne +.desc +The +.code pos +and +.code pos-if +functions search through +.meta list +for an item which matches +.metn key , +or satisfies predicate function +.metn predfun , +respectively. +They return the zero-based position of the matching item. - ;; transpose vector of strings: - ;; - string columns become string rows - ;; - vector input becomes vector output - (transpose #("abc" "def" "ghij")) -> #("adg" "beh" "cfi") +The +.meta keyfun +argument specifies a function which is applied to the elements +of +.meta list +to produce the comparison key. If this argument is omitted, +then the untransformed elements of +.meta list +are examined. - ;; error: transpose wants to make a list of strings - ;; but 1 is not a character - (transpose #("abc" "def" '(1 2 3))) ;; error! +The +.code pos +function's +.meta testfun +argument specifies the test function which +is used to compare the comparison keys from +.meta list +to +.metn key . +If this argument is omitted, then the +.code equal +function is used. +The position of the first element +.meta list +whose comparison key (as +retrieved by +.metn keyfun ) +matches the search (under +.metn testfun ) +is +returned. If no such element is found, +.code nil +is returned. - ;; String elements are catenated: - (transpose #("abc" "def" ("UV" "XY" "WZ"))) - -> #("adUV" "beXY" "cfWZ") +The +.code pos-if +function's +.meta predfun +argument specifies a predicate function +which is applied to the successive comparison keys taken from +.meta list +by applying +.meta keyfun +to successive elements. The position of +the first element for which +.meta predfun +yields true is returned. If +no such element is found, +.code nil +is returned. - (zip '(a b c) '(c d e)) -> ((a c) (b d) (c e)) -.brev +.coNP Functions @, rposq @, rposql @, rposqual @ rpos and @ rpos-if +.synb +.mets (rposq < object << list ) +.mets (rposql < object << list ) +.mets (rposqual < object << list ) +.mets (rpos < key < list >> [ testfun <> [ keyfun ]]) +.mets (rpos-if < predfun < list <> [ keyfun ]) +.syne +.desc +These functions are counterparts of +.codn rposq , +.codn rposql , +.codn rposqual , +.code rpos +and +.code rpos-if +which report position of the right-most matching item, +rather than the left-most. -.coNP Functions @, window-map @ window-mappend and @ window-mapdo +.coNP Functions @ pos-max and @ pos-min .synb -.mets (window-map < range < boundary < function << sequence ) -.mets (window-mappend < range < boundary < function << sequence ) -.mets (window-mapdo < range < boundary < function << sequence ) +.mets (pos-max < sequence >> [ testfun <> [ keyfun ]]) +.mets (pos-min < sequence >> [ testfun <> [ keyfun ]]) .syne .desc The -.code window-map +.code pos-min and -.code window-mappend -functions process the elements of -.meta sequence -by passing arguments derived from each successive element to -.metn function . -Both functions return, if possible, a sequence of the same kind as -.codn sequence , -otherwise a list. +.code pos-max +functions implement exactly the same algorithm; they +differ only in their defaulting behavior with regard to the +.meta testfun +argument. If +.meta testfun +is not given, then the pos-max function defaults +.meta testfun +to the +.code greater +function, whereas +.code pos-min +defaults it to the +.code less +function. -Under -.codn window-map , -values returned by -.meta function -are accumulated into a sequence of the same type as +If .meta sequence -and that sequence is returned. Under -.codn window-mappend , -the values returned by the calls to -.meta function -are expected to be sequence which are appended together to -form the output sequence. +is empty, both functions return +.codn nil . -These functions are analogous to -.code mapcar +Without a +.meta testfun +argument, the +.code pos-max +function finds the zero-based +position index of the numerically maximum value occurring in +.metn sequence , +whereas +.code pos-min +without a +.meta testfun +argument finds the index of the minimum +value. + +If a +.meta testfun +argument is given, the two functions are equivalent. +The +.meta testfun +function must be callable with two arguments. +If +.meta testfun +behaves like a greater-than comparison, then +.code pos-max and -.codn mappend . -Unlike these, they operate only on a single sequence, and over this sequence -they perform a -.IR "sliding window mapping" , -whose description follows. +.code pos-min +return the index of the maximum element. If +.meta testfun +behaves like a +.code less-than +comparison, then the functions return +the index of the minimum element. -The function -.code window-mappend -avoids accumulating a sequence, and instead returns -.codn nil ; -it is analogous to -.codn mapdo . +The +.meta keyfun +argument defaults to the +.code identity +function. Each element +from +.meta sequence +is passed through this one-argument function, and +the resulting value is used in its place. -The argument to the -.meta range -parameter must be a positive integer, not exceeding 512. -This parameter specifies the amount of ahead/behind context on either -side of each element which is processed. It indirectly determines -the window size for the mapping. The window size is twice -.metn range , -plus one. For instance if range is 2, then the window size is 5: -the element being processed lies at the center of the window, flanked -by two elements on either side, making five. +If a sequence contains multiple equivalent maxima, +whether the position of the leftmost or rightmost such maximum is reported +depends on whether +.meta testfun +compares for strict inequality, or whether it reports true for +equal arguments also. Under the default +.metn testfun , +which is +.codn less , +the +.code pos-max +function will return the position leftmost of a duplicate set of maximum +elements. To find the rightmost of the maxima, the +.code lequal +function can be substituted. Analogous reasoning applies to other +test functions. +.coNP Function @ mismatch +.synb +.mets (mismatch < left-seq < right-seq >> [ testfun <> [ keyfun ]]) +.syne +.desc The -.meta function -argument must specify a function which accepts a number of arguments -corresponding to the window size. For instance if -.meta range -is 2, -making the window size 5, -then -.meta function -must accept 5 arguments. These arguments constitute the sliding -window being processed. Each time -.meta function -is called, the middle argument is the element being processed, -and the arguments surrounding it are its window. +.code mismatch +function compares corresponding elements from the sequences +.meta left-seq +and +.metn right-seq , +returning the position at which the first mismatch occurs. -When an element is processed from somewhere in the interior of -a sequence, where it is flanked on either side by at least -.meta range -elements, then the window is populated by those flanking elements -taken from -.metn sequence . +If the sequences are of the same length, and their corresponding +elements are the same, then +.code nil +is returned. -The -.meta boundary -parameter specifies the window contents which are used for the -processing of elements which are closer than -.meta range -to either end of the sequence. The argument may be a sequence containing -at least twice -.meta range -number of elements (one less than the window size): if it has additional -elements, they are not used. If it is a list, it may be shorter than twice -.metn range . -The argument -may also be one of the two keyword symbols -.code :wrap -or -.codn :reflect , -described below. +If one sequence is shorter than the other, and matches a prefix +of the other, then the mismatching position returned is one position +after the last element of the shorter sequence, the same value +as its length. An empty sequence is a prefix of every sequence. -If -.meta boundary -is a sequence, it may be regarded as divided into two pieces of -.meta range -length. If it is a list of insufficient length, then missing elements -are supplied as -.code nil -to make two -.metn range 's -worth of elements. These two pieces then flank -.code sequence -on either end. The left half of -.meta boundary -is effectively prepended to the sequence, and the right half -effectively appended. -When the sliding window extends beyond the boundary of +The +.meta keyfun +argument defaults to the +.code identity +function. Each element +from .meta sequence -near its start or end, the window is populated from these -flanking elements obtained from -.metn boundary . +is passed to +.meta keyfun +and the resulting value is used in its place. + +After being converted through +.metn keyfun , +items are then compared using +.metn testfun , +which must accept two arguments, and defaults to +.codn equal . +.coNP Function @ where +.synb +.mets (where < function << object ) +.syne +.desc If -.meta boundary -is the keyword -.codn :wrap , -then the sequence is effectively flanked by copies of itself on both -ends, repeated enough times to satisfy the window. For instance if -the sequence is -.code "(1 2 3)" -and the window size is 9 due to the value of -.meta range -being 7, then the behavior of -.code :wrap -is as if a -.meta boundary -were specified consisting of -.codn "(3 1 2 3 1 2 3 1)" . -The left flank is -.code "(3 1 2 3)" -and the right flank is -.code "(1 2 3 4)" -formed by repetitions of -.code "(1 2 3)" -surrounding it on either side, extending out to infinity, and chopped to -.metn range . +.meta object +is a sequence, the +.code where +function returns +a lazy list of the numeric indices of those of its elements which satisfy +.metn function . +The numeric indices appear in increasing order. If -.meta boundary -is the keyword -.codn :reflect , -then the sequence is effectively flanked by reversed copies of itself -on both ends, repeated enough times to satisfy the window. -For instance if the sequence is -.code "(1 2 3)" -and the window size is 9, then the behavior of -.code :wrap -is as if a -.meta boundary -were specified consisting of -.codn "(1 3 2 1 3 2 1 3)" . +.meta object +is a hash, the +.code where +function returns a lazy list of +of keys which have values which satisfy +.metn function . +These keys are not subject to an order. -.coNP Function @ interpose +.meta function +must be a function that can be called with one argument. +For each element of +.metn object , +.meta function +is called with that element +as an argument. If a +.cod2 non- nil +value is returned, then the zero-based index of +that element is added to a list. Finally, the list is returned. + +.coNP Function @ rmismatch .synb -.mets (interpose < sep << sequence ) +.mets (rmismatch < left-seq < right-seq >> [ testfun <> [ keyfun ]]) +.syne +.desc +Similarly to +.codn mismatch , +the +.code rmismatch +function compares corresponding elements from the sequences +.meta left-seq +and +.metn right-seq , +returning the position at which the first mismatch occurs. +All of the arguments have the same semantics as that of +.codn mismatch . + +Unlike +.codn mismatch , +.code rmismatch +compares the sequences right-to-left, finding the suffix +which they have in common, rather than prefix. + +If the sequences match, then +.code nil +is returned. Otherwise, a negative index is returned giving the +mismatching position, regarded from the end. If the sequences +match only in the rightmost element, then -1 is returned. If they +match in two elements then -2 and so forth. + +.coNP Functions @ starts-with and @ ends-with +.synb +.mets (starts-with < short-seq < long-seq >> [ testfun <> [ keyfun ]]) +.mets (ends-with < short-seq < long-seq >> [ testfun <> [ keyfun ]]) .syne .desc The -.code interpose -function returns a sequence of the same type as -.metn sequence , -in which the elements from -.meta sequence -appear with the -.meta sep -value inserted -between them. +.code starts-with +and +.code ends-with +functions compare corresponding elements from sequences +.meta short-seq +and +.metn long-seq . -If -.meta sequence -is an empty sequence or a sequence of length 1, then a -sequence identical to -.meta sequence -is returned. It may be a copy of -.meta sequence -or it may be -.meta sequence -itself. +The +.code starts-with +function returns +.code t +if +.meta short-seq +is prefix of +.metn long-seq ; +otherwise, it returns +.codn nil . -If -.meta sequence -is a character string, then the value -.meta sep -must be a character. +The +.code ends-with +function returns +.code t +if +.meta short-seq +is suffix of +.metn long-seq ; +otherwise, it returns +.codn nil . -It is permissible for -.metn sequence , -or for a suffix of -.meta sequence -to be a lazy -list, in which case interpose returns a lazy list, or a list with a lazy -suffix. +Element from both sequences are mapped to comparison keys using +.metn keyfun , +which defaults to +.codn identity . -.TP* Examples: -.verb - (interpose #\e- "xyz") -> "x-y-z" - (interpose t nil) -> nil - (interpose t #()) -> #() - (interpose #\ea "") -> "" - (interpose t (range 0 0)) -> (0) - (interpose t (range 0 1)) -> (0 t 1) - (interpose t (range 0 2)) -> (0 t 1 t 2) -.brev +Comparison keys are compared using +.meta testfun +which defaults to +.codn equal . -.coNP Functions @ apply and @ iapply +.coNP Function @ select .synb -.mets (apply < function <> [ arg * << trailing-args ]) -.mets (iapply < function <> [ arg * << trailing-args ]) +.mets (select < object >> { index-list <> | function }) .syne .desc The -.code apply -function invokes -.metn function , -optionally passing to it an argument -list. The return value of the -.code apply -call is that of -.metn function . +.code select +function returns an object, of the same kind as +.metn object , +which consists of those elements of +.meta object +which are identified by +the indices in +.metn index-list , +which may be a list or a vector. -If no arguments are present after -.metn function , +If +.meta function +is given instead of +.metn index-list , then .meta function -is invoked without arguments. +is invoked with +.meta object +as its argument. The return value is then taken as +if it were the +.meta index-list +argument . -If one argument is present after -.metn function , -then it is interpreted as -.metn trailing-args . -If this is a sequence (a list, vector or string), -then the elements of the sequence are passed as individual arguments to -.metn function . If -.meta trailing-args -is not a sequence, then -.meta function -is invoked -with an improper argument list, terminated by the -.meta trailing-args -atom. +.meta object +is a sequence, then +.meta index-list +consists of numeric +indices. The length of the sequence, as reported by the +.code length +function, is added to every +.meta index-list +value which is negative. +The +.code select +function stops collecting values upon encountering an index value which is +greater than or equal to the length of the sequence. +(Rationale: without +this strict behavior, +.code select +would not be able to terminate if +.meta index-list +is infinite.) -If two or more arguments are present after -.metn function , -then the last of these arguments is interpreted as -.metn trailing-args . -The previous arguments represent leading arguments which are applied to -.metn function , -prior to the arguments taken from -.metn trailing-args . +If +.meta object +is, more specifically, a list-like sequence, then +.meta index-list +must contain monotonically increasing +numeric values, even if no value is out of range, since the +.code select +function +makes a single pass through the list based on the assumption that indices +are ordered. (Rationale: optimization.) +This requirement for monotonicity applies to the values which +result after negative indices are displaced by the sequence length +Also, in this list-like sequence case, values taken from +.meta index-list +which are still negative after being displaced by the sequence length are +ignored. -Note that if -.meta trailing-args -value is an atom or an improper list, the function is then -invoked with an improper argument list. Only a variadic -function may be invoked with an improper argument lists. -Moreover, all of the function's required and optional -parameters must be satisfied by elements of the -improper list, such that the terminating atom either -matches the -.meta rest-param -directly (see the -.code lambda -operator) or else the -.meta rest-param -receives an improper list terminated by that atom. -To treat the terminating atom of an improper list as an -ordinary element which can satisfy a required or optional -function parameter, the -.code iapply -function may be used, described next. +If +.meta object +is a hash, then +.meta index-list +is a list of keys. A new hash is +returned which contains those elements of +.meta object +whose keys appear +in +.metn index-list . +All of +.meta index-list +is processed, even if it contains +keys which are not in +.metn object . +The nonexistent keys are ignored. The -.code iapply -function ("improper apply") is similar to -.codn apply , -except with regard to the treatment of -.metn trailing-args . -Firstly, under -.codn iapply , -if -.meta trailing-args -is an atom other than -.code nil -(possibly a sequence, such as a vector or string), -then it is treated as an ordinary argument: -.meta function -is invoked with a proper argument list, whose last element is -.metn trailing-args . -Secondly, if -.meta trailing-args -is a list, but an improper list, then the terminating atom of -.meta trailing-args -becomes an individual argument. -This terminating atom is not split into multiple arguments, -even if it is a sequence. -Thus, in all possible cases, -.code iapply -treats an extra -.cod2 non- nil -atom as an argument, and never calls -.meta function -with an improper argument list. +.code select +function also supports objects of type +.codn carray , +in a manner similar to vectors. The indicated elements are extracted +from the input object, and a new +.code carray +is returned whose storage is initialized by converting the extracted +values back to the foreign representation. -.TP* Examples: -.verb - ;; '(1 2 3) becomes arguments to list, thus (list 1 2 3). - (apply (fun list) '(1 2 3)) -> (1 2 3) +.coNP Function @ relate +.synb +.mets (relate < domain-seq < range-seq <> [ default-val ]) +.syne +.desc +The +.code relate +function returns a one-argument function which implements the relation formed +by mapping the elements of +.meta domain-seq +to the positionally corresponding elements of +.metn range-seq . +That is to say, the function searches through the sequence +.meta domain-seq +to determine the position where its argument occurs, using +.code equal +as the comparison function. +Then it returns the element from that position in the +.meta range-seq +sequence. This returned function is called the +.IR "relation function" . - ;; this effectively invokes (list 1 2 3 4) - (apply (fun list) 1 2 '(3 4)) -> (1 2 3 4) +If the relation function's argument is not found in +.metn domain-seq , +then the behavior depends on the optional parameter +.metn default-val . +If an argument is given for +.metn default-val , +then the relation function returns that value. +Otherwise, the relation function returns its argument. - ;; this effectively invokes (list 1 2 . 3) - (apply (fun list) 1 2 3)) -> (1 2 . 3) +Note: the +.code relate +function may be understood in terms of the following equivalences: - ;; "abc" is separated into characters - ;; which become arguments of list - (apply (fun list) "abc") -> (#\ea #\eb #\ec) +.verb + (relate d r) <--> (lambda (arg) + (iflet ((p (posqual arg d))) + [r p] + arg)) + + (relate d r v) <--> (lambda (arg) + (iflet ((p (posqual arg d))) + [r p] + v)) .brev -.TP* "Dialect Note:" -Note that some uses of this function that are necessary in other Lisp dialects -are not necessary in \*(TL. The reason is that in \*(TL, improper list -syntax is accepted as a compound form, and performs application: +.TP* Examples: .verb - (foo a b . x) -.brev + (mapcar (relate "_" "-") "foo_bar") -> "foo-bar" -Here, the variables -.code a -and -.code b -supply the first two arguments for -.codn foo . -In -the dotted position, -.code x -must evaluate to a list or vector. The list or -vector's elements are pulled out and treated as additional arguments for -.codn foo . -Of course, this syntax can only be used if -.code x -is a symbolic form or an atom. It -cannot be a compound form, because -.code "(foo a b . (x))" -and -.code "(foo a b x)" -are equivalent structures. + (mapcar (relate "0123456789" "ABCDEFGHIJ" "X") "139D-345") + -> "BJDXXDEF" -.coNP Functions @ reduce-left and @ reduce-right -.synb -.mets (reduce-left < binary-function < list -.mets \ \ \ \ \ \ \ \ \ \ \ \ >> [ init-value <> [ key-function ]]) + (mapcar (relate '(nil) '(0)) '(nil 1 2 nil 4)) -> (0 1 2 0 4) +.brev -.mets (reduce-right < binary-function < list -.mets \ \ \ \ \ \ \ \ \ \ \ \ \ >> [ init-value <> [ key-function ]]) +.coNP Function @ in +.synb +.mets (in < sequence < key >> [ testfun <> [ keyfun ]]) +.mets (in < hash << key ) .syne .desc The -.code reduce-left -and -.code reduce-right -functions reduce lists of operands specified -by -.meta list -and -.meta init-value -to a single value by the repeated application of -.metn binary-function . +.code in +function tests whether +.meta key +is found inside +.meta sequence +or +.metn hash . -An effective list of operands is formed by combining -.meta list -and -.metn init-value . -If -.meta key-function -is specified, then the items of -.meta list -are -mapped to new values through -.metn key-function , -as if by -.codn mapcar . -If -.meta init-value -is supplied, -then in the case of -.codn reduce-left , -the effective list of operands is formed by -prepending -.meta init-value -to -.metn list . -In the case of -.codn reduce-right , -the effective operand list is produced by appending -.meta init-value +If the +.meta testfun +argument is specified, it specifies the function +which is used to comparison keys from the sequence to -.metn list . -The -.meta init-value -isn't mapped through -.metn key-function . +.metn key . +Otherwise the +.code equal +function is used. -The production of the effective list can be expressed like this, -though this is not to be understood as the actual implementation: +If the +.meta keyfun +argument is specified, it specifies a function which +is applied to the elements of +.meta sequence +to produce the comparison keys. Without this +argument, the elements themselves are taken +as the comparison keys. -.verb - (append (if init-value-present (list init-value)) - [mapcar (or key-function identity) list])))) -.brev +If the object being searched is a hash, then the +.meta keyfun +and +.meta testfun +arguments are ignored. -In the -.code reduce-right -case, the arguments to -.code append -are reversed. +The +.code in +function returns +.code t +if it finds +.meta key +in +.meta sequence +or +.metn hash , +otherwise +.codn nil . -If the effective list of operands is empty, then -.meta binary-function -is called -with no arguments at all, and its value is returned. This is the only -case in which -.meta binary-function -is called with no arguments; in all -remaining cases, it is called with two arguments. +.coNP Function @ partition +.synb +.mets (partition < sequence >> { index-list | index | << function }) +.syne +.desc +If +.meta sequence +is empty, then +.code partition +returns an empty list, and the +second argument is ignored; if it is +.metn function , +it is not called. -If the effective list contains one item, then that item is returned. +Otherwise, +.code partition +returns a lazy list of partitions of +.metn sequence . +Partitions are consecutive, non-overlapping, non-empty sub-strings of +.metn sequence , +of the same kind as +.metn sequence , +such that if these sub-strings are catenated together in their order +of appearance, a sequence +.code equal +to the original is produced. -Otherwise, the effective list contains two or more items, and is decimated as -follows. +If the second argument is of the form +.metn index-list , +or if an +.meta index-list +was produced from the +.meta index +or +.meta function +arguments, each value in that list must be an integer. Each integer +value which is non-negative specifies the index position +given by its value. Each integer value which is negative +specifies an index position given by adding the length of +.meta sequence +to its value. The sequence index positions thus denoted by +.meta index-list +shall be strictly non-decreasing. Each successive element +is expected to designate an index position at least as high +as all previous elements, otherwise the behavior is unspecified. +Leading index positions which are (still) negative, or zero, are effectively +ignored. -Note that an -.meta init-value -specified as -.code nil -is not the same as a missing -.metn init-value ; -this means that the initial value is the object -.codn nil . -Omitting -.meta init-value -is the same as specifying a value of -.code : -(the colon symbol). -It is possible to specify -.meta key-function -while omitting an -.meta init-value -argument. This is achieved by explicitly specifying -.code : -as the -.meta init-value +If +.meta index-list +is empty then a one-element list containing the entire +.meta sequence +is returned. + +If +.meta index-list +is an infinite lazy list, the function shall terminate if that +list eventually produces an index position which is greater than or equal to +the length of +.metn sequence . + +If the second argument is a function, then this function is applied +to +.metn sequence , +and the return value of this call is then used in place of the +second argument, which must be a single index value, which is then +taken as if it were the +.meta index +argument, or else a list of indices, which are taken as the +.meta index-list argument. -Under -.codn reduce-left , -the leftmost pair of operands is removed -from the list and passed as arguments to -.metn binary-function , -in the same order -that they appear in the list, and the resulting value initializes an -accumulator. Then, for each remaining item in the list, -.meta binary-function -is invoked on two arguments: the current accumulator value, and the next element -from the list. After each call, the accumulator is updated with the return -value of -.metn binary-function . -The final value of the accumulator is returned. +If the second argument is an atom other than a function, it is assumed to be +an integer index, and is turned into an +.meta index-list +of one element. -Under -.codn reduce-right , -the list is processed right to left. The rightmost -pair of elements in the effective list is removed, and passed as arguments to -.metn binary-function , -in the same order that they appear in the list. The -resulting value initializes an accumulator. Then, for each remaining item in -the list, -.meta binary-function -is invoked on two arguments: the -next element from the list, in right to left order, and the current -accumulator value. After each call, the accumulator is updated with the return -value of -.metn binary-function . -The final value of the accumulator is returned. +After the +.meta index-list +is obtained as an argument, or determined from the +.meta index +or +.meta function +arguments, the +.code partition +function then divides +.meta sequence +according to the indices given by that list. +The first partition begins with the first element of +.metn sequence . +The second partition begins at the first position in +.metn index-list , +and so on. Indices beyond the length of the sequence are ignored, +as are indices less than or equal to zero. .TP* Examples: .verb - ;;; effective list is (1) so 1 is returned - (reduce-left (fun +) () 1 nil) -> 1 - - ;;; computes (- (- (- 0 1) 2) 3) - (reduce-left (fun -) '(1 2 3) 0 nil) -> -6 - - ;;; computes (- 1 (- 2 (- 3 0))) - (reduce-right (fun -) '(1 2 3) 0 nil) -> 2 - - ;;; computes (* 1 2 3) - (reduce-left (fun *) '((1) (2) (3)) nil (fun first)) -> 6 + (partition '(1 2 3) 1) -> ((1) (2 3)) - ;;; computes 1 because the effective list is empty - ;;; and so * is called with no arguments, which yields 1. - (reduce-left (fun *) nil) + ;; split the string where there is a "b" + (partition "abcbcbd" (op where (op eql #\eb))) -> ("a" "bc" + "bc" "bd") .brev -.coNP Functions @, some @ all and @ none +.coNP Functions @ split and @ split* .synb -.mets (some < sequence >> [ predicate-fun <> [ key-fun ]]) -.mets (all < sequence >> [ predicate-fun <> [ key-fun ]]) -.mets (none < sequence >> [ predicate-fun <> [ key-fun ]]) +.mets (split < sequence >> { index-list | index | << function }) +.mets (split* < sequence >> { index-list | index | << function }) .syne .desc -The -.codn some , -.code all -and -.code none -functions apply a predicate test function -.meta predicate-fun -over a list of elements. If the argument -.meta key-fun -is -specified, then elements of +If .meta sequence -are passed into -.metn key-fun , +is empty, then both +.code split and -.meta predicate-fun -is -applied to the resulting values. If -.meta key-fun -is omitted, the behavior is -as if -.meta key-fun -is the identity function. If -.meta predicate-fun -is omitted, -the behavior is as if -.meta predicate-fun -is the identity function. - -These functions have short-circuiting semantics and return conventions similar -to the and and or operators. +.code split* +return an empty list, and the +second argument is ignored; if it is +.metn function , +it is not called. -The some function applies -.meta predicate-fun -to successive values -produced by retrieving elements of -.meta list -and processing them through -.metn key-fun . -If the list is empty, it returns -.codn nil . -Otherwise it returns the -first -.cod2 non- nil -return value returned by a call to -.meta predicate-fun -and -stops evaluating more elements. If -.meta predicate-fun -returns -.code nil -for all -elements, it returns -.metn nil . +Otherwise, +.code split +returns a lazy list of pieces of +.metn sequence : +consecutive, non-overlapping, possibly empty sub-strings of +.metn sequence , +of the same kind as +.metn sequence . +A catenation of these pieces in the order they appear would produce +a sequence that is +.code equal +to the original sequence. The -.code all -function applies -.meta predicate-fun -to successive values -produced by retrieving elements of -.meta list -and processing them through -.metn key-fun . -If the list is empty, it returns -.codn t . -Otherwise, if -.meta predicate-fun -yields -.code nil -for any value, the -.code all -function immediately -returns without invoking -.meta predicate-fun -on any more elements. -If all the elements are processed, then the all function returns -the value which -.meta predicate-fun -yielded for the last element. +.code split* +function differs from +.code split +in that the elements indicated by the split indices are removed. The -.code none -function applies -.meta predicate-fun -to successive values -produced by retrieving elements of -.meta list -and processing them through -.metn key-fun . -If the list is empty, it returns -.codn t . -Otherwise, if -.meta predicate-fun -yields -.cod2 non- nil -for any value, the none function -immediately returns nil. If -.meta predicate-fun -yields nil for all -values, the none function returns -.codn t . +.metn index , +.metn index-list , +and +.meta function +arguments are subject to the same restrictions and treatment +as the corresponding arguments of the +.code partition +function, with the following difference: the index positions indicated by +.code index-list +are required to be strictly increasing, rather than non-decreasing. -.TP* Examples: +If the second argument is of the form +.metn index-list , +or if an +.meta index-list +was produced from the +.meta index +or +.meta function +arguments, then the +.code split +function divides +.meta sequence +according to the indices indicated in the list. The first piece always begins +with the first element of +.metn sequence . +Each subsequent piece begins with the position indicated by +an element of +.metn index-list . +Negative indices are ignored. +If +.meta index-list +includes index zero, +then an empty first piece is generated. +If +.meta index-list +includes an index greater than or equal to the length of +.meta sequence +(equivalently, an index beyond the last element of the sequence) +then an additional empty last piece is generated. +The length of +.meta sequence +is added to any negative indices. An index which is still negative +after being thus displaced is discarded. + +Note: the principal difference between +.code split +and +.code partition +is that +.code partition +does not produce empty pieces. +.TP* Examples: .verb - ;; some of the integers are odd - [some '(2 4 6 9) oddp] -> t + (split '(1 2 3) 1) -> ((1) (2 3)) - ;; none of the integers are even - [none '(1 3 4 7) evenp] -> t + (split "abc" 0) -> ("" "abc") + (split "abc" 3) -> ("abc" "") + (split "abc" 1) -> ("a" "bc") + (split "abc" '(0 1 2 3)) -> ("" "a" "b" "c" "") + (split "abc" '(1 2)) -> ("a" "b" "c") + + (split "abc" '(-1 1 2 15)) -> ("a" "b" "c") + + ;; triple split at makes two additional empty pieces + (split "abc" '(1 1 1)) -> ("a" "" "" "bc") + + (split* "abc" 0) -> ("" "bc") ;; "a" is removed + + ;; all characters removed + (split* "abc" '(0 1 2)) -> ("" "" "" "") .brev -.coNP Function @ multi +.coNP Function @ partition* .synb -.mets (multi < function << sequence *) +.mets (partition* < sequence >> { index-list >> | index <> | function }) .syne .desc -The -.code multi -function distributes an arbitrary list processing function -.meta multi -over multiple sequences given by the -.meta list -arguments. - -The +If .meta sequence -arguments are first transposed into a single list of tuples. Each -successive element of this transposed list consists of a tuple of the -successive items from the lists. The length of the transposed list is that -of the shortest -.meta list -argument. - -The transposed list is then passed to -.meta function -as an argument. +is empty, then +.code partition* +returns an empty list, and the +second argument is ignored; if it is +.metn function , +it is not called. The +.metn index , +.metn index-list , +and .meta function -is expected to produce a list of tuples, which are transposed -again to produce a list of lists which is then returned. +arguments are subject to the same restrictions and treatment +as the corresponding arguments of the +.code partition +function, with the following difference: the index positions indicated by +.code index-list +are required to be strictly increasing, rather than non-decreasing. -Conceptually, the input sequences are columns and -.meta function -is invoked on -a list of the rows formed from these columns. The output of -.meta function -is a transformed list of rows which is reconstituted into a list of columns. +If the second argument is of the form +.metn index-list , +then +.code partition* +produces a +lazy list of pieces taken from +.metn sequence . +The pieces are formed by deleting from +.meta sequence +the elements at the positions given +in +.metn index-list , +such that the pieces are the remaining non-empty sub-strings from +between the deleted elements, maintaining their order. -.TP* Example: +If +.meta index-list +is empty then a one-element list containing the entire +.meta sequence +is returned. +.TP* Examples: .verb - ;; Take three lists in parallel, and remove from all of them - ;; them the element at all positions where the third list - ;; has an element of 20. - - (multi (op remove-if (op eql 20) @1 third) - '(1 2 3) - '(a b c) - '(10 20 30)) - - -> ((1 3) (a c) (10 30)) + (partition* '(1 2 3 4 5) '(0 2 4)) -> ((1) (3) (5)) - ;; The (2 b 20) "row" is gone from the three "columns". + (partition* "abcd" '(0 3)) -> "bc" - ;; Note that the (op remove if (op eql 20) @1 third) - ;; expression can be simplified using the ap operator: - ;; - ;; (op remove-if (ap eql @3 20)) + (partition* "abcd" '(0 1 2 3)) -> nil .brev -.coNP Function @ sort +.coNP Functions @ find and @ find-if .synb -.mets (sort < sequence >> [ lessfun <> [ keyfun ]]) +.mets (find < key < sequence >> [ testfun <> [ keyfun ]]) +.mets (find-if < predfun >> { sequence | << hash } <> [ keyfun ]) .syne .desc The -.code sort -function destructively sorts -.metn sequence , -producing a sequence -which is sorted according to the -.meta lessfun +.code find and -.meta keyfun -arguments. +.code find-if +functions search through a sequence for an item which +matches a key, or satisfies a predicate function, respectively. The .meta keyfun -argument specifies a function which is applied to elements -of the sequence to obtain the key values which are then compared -using the lessfun. If -.meta keyfun -is omitted, the identity function is used -by default: the sequence elements themselves are their own sort keys. - -The -.meta lessfun -argument specifies the comparison function which determines -the sorting order. It must be a binary function which can be invoked -on pairs of keys as produced by the key function. It must -return a -.cod2 non- nil -value if the left argument is considered to be lesser -than the right argument. For instance, if the numeric function -.code < -is used -on numeric keys, it produces an ascending sorted order. If the function -.code > -is used, then a descending sort is produced. If -.meta lessfun -is omitted, then it defaults to the generic -.code less -function. - -The -.code sort -function is stable for sequences which are lists. This means that the -original order of items which are considered identical is preserved. -For strings and vectors, -.code sort -is not stable. +argument specifies a function which is applied to the elements +of +.meta sequence +to produce the comparison key. If this argument is omitted, +then the untransformed elements of the +.meta sequence +are searched. -.coNP Function @ grade -.synb -.mets (grade < sequence >> [ lessfun <> [ keyfun ]]) -.syne -.desc The -.code grade -function returns a list of integer indices which indicate the position -of the elements of +.code find +function's +.meta testfun +argument specifies the test function which +is used to compare the comparison keys from .meta sequence -in sorted order. +to the search key. +If this argument is omitted, then the +.code equal +function is used. +The first element from the list whose comparison key (as retrieved by +.metn keyfun ) +matches the search (under +.metn testfun ) +is returned. If no such element is found, +.code nil +is returned. The -.meta lessfun -and +.code find-if +function's +.meta predfun +argument specifies a predicate function +which is applied to the successive comparison keys pulled from the list +by applying .meta keyfun -arguments behave like those of the -.code sort -function. +to successive elements. The first element +for which +.meta predfun +yields true is returned. If no such +element is found, +.code nil +is returned. +In the case of +.codn find-if , +a hash table may be specified instead of a sequence. The -.meta sequence -object is not modified. - -The internal sort performed by -.code grade -is not stable. The indices of any elements considered equivalent under -.code lessfun -may appear in any order in the returned index sequence. - -Note: the -.code grade -function is inspired by the "grade up" and "grade down" operators -in the APL language. - -.TP* Examples: - -.verb - ;; Order of the 2 3 positions of the "l" - ;; characters is not specified: - - [grade "Hello"] -> (0 1 2 3 4) - [grade "Hello" >] -> (4 2 3 1 0) -.brev +.meta hash +is treated as if it were a sequence of hash key and hash +value pairs represented as cons cells, the +.code car +slots of which are the hash keys, and the +.code cdr +of which are the hash values. If the caller doesn't specify a +.meta keyfun +then these cells are taken as their keys. -.coNP Function @ shuffle +.coNP Functions @ rfind and @ rfind-if .synb -.mets (shuffle << sequence ) +.mets (rfind < key < sequence >> [ testfun <> [ keyfun ]]) +.mets (rfind-if < predfun >> { sequence | << hash } <> [ keyfun ]) .syne .desc The -.code shuffle -function pseudo-randomly rearranges the elements of -.metn sequence . -This is performed in place: -.meta sequence -object is modified. - -The return value is -.meta sequence -itself. +.code rfind +and +.code rfind-if +functions are almost exactly like +.code find +and +.code find-if +except that if there are multiple matches for +.meta key +in +.metn sequence , +they return the right-most element rather than +the leftmost. -The rearrangement depends on pseudo-random numbers obtained from the -.code rand -function. +In the case of +.code rfind-if +when a +.meta hash +is specified instead of a +.metn sequence , +the function searches through the hash entries in the same order as +.codn find-if , +but finds the last match rather than the first. +Note: hashes are inherently not ordered; the relative order of items in +a hash table can change when other items are inserted or deleted. -.coNP Function @ sort-group +.coNP Functions @ find-max and @ find-min .synb -.mets (sort-group < sequence >> [ keyfun <> [ lessfun ]]) +.mets (find-max >> { sequence | << hash } >> [ testfun <> [ keyfun ]]) +.mets (find-min >> { sequence | << hash } >> [ testfun <> [ keyfun ]]) .syne .desc The -.code sort-group -function sorts -.meta sequence -according to the -.meta keyfun -and -.meta lessfun -arguments, and then breaks the resulting sequence into groups, -based on the equivalence of the elements under -.metn keyfun . - -The following equivalence holds: - -.verb - (sort-group sq lf kf) - - <--> - - (partition-by kf (sort (copy sq) kf lf)) -.brev - -Note the reversed order of -.meta keyfun -and -.meta lessfun -arguments between -.code sort +.code find-min and -.codn sort-group . +.code find-max +function implement exactly the same algorithm; they +differ only in their defaulting behavior with regard to the +.meta testfun +argument. If +.meta testfun +is not given, then the find-max function defaults it to +the +.code greater +function, whereas +.code find-min +defaults it to the +.code less +function. -.coNP Function @ uniq -.synb -.mets (uniq << sequence ) -.syne -.desc -The -.code uniq -function returns a sequence of the same kind as +Without a +.meta testfun +argument, the +.code find-max +function finds the numerically +maximum value occurring in .metn sequence , -but with -duplicates removed. Elements of -.meta sequence -are considered equal under -the -.code equal -function. The first occurrence of each element is retained, -and the subsequent duplicates of that element, of any, are suppressed, -such that the order of the elements is otherwise preserved. +whereas +.code pos-min +without a +.meta testfun +argument finds the minimum value. +If a +.meta testfun +argument is given, the two functions are equivalent. The -.code uniq -function is an alias for the one-argument case of -.codn unique . -That is to say, this equivalence holds: - -.verb - (uniq s) <--> (unique s) -.brev +.meta testfun +function must be callable with two arguments. +If +.meta testfun +behaves like a greater-than comparison, then +.code find-max +and +.code find-min +both return the maximum element. If +.meta testfun +behaves like a less-than comparison, then the functions return +the minimum element. -.coNP Function @ unique -.synb -.mets (unique < sequence >> [ keyfun <> { hash-arg }* ]) -.syne -.desc The -.code unique -function is a generalization of -.codn uniq . -It returns a sequence of the same kind as -.metn sequence , -but with duplicates removed. - -If neither -.meta keyfun -nor -.metn hash-arg -s -are specified, then elements of sequence are considered equal under the -.code eql -function. The first occurrence of each element is retained, -and the subsequent duplicates of that element, of any, are suppressed, -such that the order of the elements is otherwise preserved. - -If -.meta keyfun -is specified, then that function is applied to each element, -and the resulting values are compared for equality. -In other words, the behavior is as if .meta keyfun -were the +argument defaults to the .code identity -function. - -If one or more -.metn hash-arg -s -are present, these specify the arguments for the construction of -the internal hash table used by -.codn unique . -The arguments are like those of the -.code hash -function. - -.coNP Function @ tuples -.synb -.mets (tuples < length < sequence <> [ fill-value ]) -.syne -.desc -The -.code tuples -function produces a lazy list which represents a reorganization -of the elements of -.meta sequence -into tuples of -.metn length , -where -.meta length -must be a positive integer. - -The length of the sequence might not be evenly divisible by the tuple length. -In this case, if a -.meta fill-value -argument is specified, then the last tuple -is padded with enough repetitions of -.meta fill-value -to make it have -.meta length -elements. If -.meta fill-value -is not specified, then the last tuple is left -shorter than -.metn length . - -The output of the function is a list, but the tuples themselves are sequences -of the same kind as -.metn sequence . -If +function. Each element +from .meta sequence -is any kind of list, they -are lists, and not lazy lists. +is passed through this one-argument function, and +the resulting value is used in its place for the purposes of the +comparison. However, the original element is returned. -.TP* Examples: +A hash table may be specified instead of a sequence. +The +.meta hash +is treated as if it were a sequence of hash key and hash +value pairs represented as cons cells, the +.code car +slots of which are the hash keys, and the +.code cdr +of which are the hash values. If the caller doesn't specify a +.meta keyfun +then these cells are taken as their keys. To find the hash +table's key-value cell with the maximum key, the +.code car +function can be specified as +.metn keyfun . +To find the entry holding the maximum value, the +.code cdr +function can be specified. -.verb - (tuples 3 #(1 2 3 4 5 6 7 8) 0) -> (#(1 2 3) #(4 5 6) - #(7 8 0)) - (tuples 3 "abc") -> ("abc") - (tuples 3 "abcd") -> ("abc" "d") - (tuples 3 "abcd" #\ez) -> ("abc" "dzz") - (tuples 3 (list 1 2) #\ez) -> ((1 2 #\ez)) -.brev +If there are multiple equivalent maxima, then under the default +.metn testfun , +that being +.codn less , +the leftmost one is reported. See the notes under +.code pos-max +regarding duplicate maxima. -.coNP Function @ partition-by +.coNP Functions @, uni @, isec @ diff and @ symdiff .synb -.mets (partition-by < function << sequence ) +.mets (uni < seq1 < seq2 >> [ testfun <> [ keyfun ]]) +.mets (isec < seq1 < seq2 >> [ testfun <> [ keyfun ]]) +.mets (diff < seq1 < seq2 >> [ testfun <> [ keyfun ]]) +.mets (symdiff < seq1 < seq2 >> [ testfun <> [ keyfun ]]) .syne .desc -If -.meta sequence -is empty, then -.code partition-by -returns an empty list, +The functions +.codn uni , +.codn isec , +.code diff and -.meta function -is never called. +.code symdiff +treat the sequences +.meta seq1 +and +.meta seq2 +as if they were sets. -Otherwise, -.code partition-by -returns a lazy list of partitions of the sequence -.metn sequence . -Partitions are consecutive, non-empty sub-strings of -.metn sequence , -of the same kind as -.metn sequence . +They, respectively, compute the set union, set intersection, +set difference and symmetric difference of +.meta seq1 +and +.metn seq2 , +returning a new sequence. -The partitioning begins with the first element of -.meta sequence -being placed into a partition. +The arguments +.meta seq1 +and +.meta seq2 +need not be of the same kind. They may be hash tables. -The subsequent partitioning is done according to -.metn function , -which is applied -to each element of -.metn sequence . -Whenever, for the next element, the function -returns the same value as it returned for the previous element, the -element is placed into the same partition. Otherwise, the next element -is placed into, and begins, a new partition. +The returned sequence is of the same kind as +.metn seq1 . +If +.meta seq1 +is a hash table, the returned sequence is a list. -The return values of the calls to -.meta function -are compared using the -.code equal -function. +For the purposes of these functions, an input which is a hash table +is considered as if it were a sequence of hash key and hash +value pairs represented as cons cells, the +.code car +slots of which are the hash keys, and the +.code cdr +of which are the hash values. This means that if no +.meta keyfun +is specified, these pairs are taken as keys. -.TP* Examples: +Since the input sequences are defined as representing sets, they are assumed +not to contain duplicate elements. These functions are not required, but may, +de-duplicate the sequences. -.verb - [partition-by identity '(1 2 3 3 4 4 4 5)] -> ((1) (2) (3 3) - (4 4 4) (5)) - - (partition-by (op = 3) #(1 2 3 4 5 6 7)) -> (#(1 2) #(3) - #(4 5 6 7)) -.brev +The union sequence produced by +.code uni +contains all of the elements which occur in both +.meta seq1 +and +.metn seq2 . +If a given element occurs exactly once only in +.meta seq1 +or exactly once only in +.metn seq2 , +or exactly once in both sequences, then it occurs exactly once in the union +sequence. If a given element occurs at least once in either +.metn seq1 , +.meta seq2 +or both, then it occurs at least once in the union sequence. -.coNP Function @ make-like -.synb -.mets (make-like < list << ref-sequence ) -.syne -.desc -The -.meta list -argument must be a list. If -.meta ref-sequence -is a sequence type, -then -.meta list -is converted to the same type of sequence and returned. -Otherwise the original -.meta list -is returned. +The intersection sequence produced by +.code isec +contains all of the elements which occur in both +.meta seq1 +and +.metn seq2 . +If a given element occurs exactly once in +.meta seq1 +and exactly once in +.metn seq2 , +then in occurs exactly once in the intersection sequence. +If a given element occurs at least once in +.meta seq1 +and at least once in +.metn seq2 , +then in occurs at least once in the intersection sequence. -Conversion is supported to string and vector type. +The difference sequence produced by +.code diff +contains all of the elements which occur in +.meta seq1 +but do not occur in +.metn seq2 . +If an element occurs exactly once in +.meta seq1 +and does not occur in +.metn seq2 , +then it occurs exactly once in the difference sequence. +If an element occurs at least once in +.meta seq1 +and does not occur in +.metn seq2 , +then it occurs at least once in the difference sequence. +If an element occurs at least once in +.metn seq2 , +then it does not occur in the difference sequence. -Conversion to a structure type is possible for structures. If -.meta ref-sequence -is an object of a structure type which has a static function -.codn from-list , -then -.code make-like -calls that function, passing to it, and the resulting value is returned. -.meta list -and returns whatever value that function returns. +The symmetric difference sequence produced by +.code symdiff +contains all of the elements of +.meta seq1 +which do not occur in +.meta seq2 +and +.IR "vice versa" : +it also contains all of the elements of +.meta seq2 +which do not occur in +.metn seq1 . +Element equivalence is determined by a combination of +.meta testfun +and +.metn keyfun . +Elements are compared pairwise, and each element of a pair is passed through +.meta keyfun +function to produce a comparison value. The comparison values +are compared using +.metn testfun . If -.meta ref-sequence -is a -.codn carray , -then -.meta list -is passed to the -.code carray-list -function, and the resulting value is returned. The second argument in the -.code carray-list -call is the element type taken from -.metn ref-sequence . -The third argument is -.codn nil , -indicating that the resulting -.code carray -is not to be null terminated. +.meta keyfun +is omitted, then the +untransformed elements themselves are compared, and if +.meta testfun +is omitted, +then the +.code equal +function is used. -Note: the -.code make-like -function is a helper which supports the development of -unoptimized versions of a generic function that accepts any type of -sequence as input, and produces a sequence of the same type as output. -The implementation of such a function can internally accumulate a list, and -then convert the resulting list to the same type as an input value -by using -.codn make-like . +Note: a function similar to +.code diff +named +.code set-diff +exists. This became deprecated starting in \*(TX 184. +For the +.code set-diff +function, the requirement was specified to preserve the original +order of items from +.meta seq1 +that survive into the output sequence. +This requirement is not documented for the +.code diff +function, but is +.I "de facto" +honored by the implementation for at as long as the +.code set-diff +synonym continues to be available. +The +.code set-diff +function doesn't support hash tables and is inefficient for vectors and +strings. -.coNP Function @ nullify +Note: these functions are not efficient for the processing of hash tables, +even when both inputs are hashes, the +.meta keyfun +argument is +.codn car , +and +.meta testfun +matches the equality used by both hash table inputs. +If applicable, the operations +.codn hash-uni , +.code hash-isec +and +.code hash-diff +should be used instead. + +.coNP Functions @, mapcar @, mappend @ mapcar* and @ mappend* .synb -.mets (nullify << sequence ) +.mets (mapcar < function << sequence *) +.mets (mappend < function << sequence *) +.mets (mapcar* < function << sequence *) +.mets (mappend* < function << sequence *) .syne .desc -The -.code nullify +When given only one argument, the +.code mapcar function returns -.code nil -if -.meta sequence -is an empty sequence. -Otherwise it returns +.codn nil . +.meta function +is never called. + +When given two arguments, the +.code mapcar +function applies +.meta function +to each elements of .meta sequence -itself. +and returns a sequence of the resulting values +in the same order as the original values. +The returned sequence is the same kind as +.metn sequence , +if possible. If the accumulated values cannot be +elements of that type of sequence, then a list is returned. + +When additional sequences are given as arguments, this filtering behavior is +generalized in the following way: +.code mapcar +traverses the sequences in parallel, +taking a value from each sequence as an argument to the function. If there +are two lists, +.meta function +is called with two arguments and so forth. +The traversal is limited by the length of the shortest sequence. +The return values of the function are collected into a new sequence which is +returned. The returned sequence is of the same kind as the leftmost +input sequence, unless the accumulated values cannot be elements of that type of +sequence, in which case a list is returned. -Note: the -.code nullify -function is a helper to support unoptimized generic -programming over sequences. Thanks to the generic behavior of -.codn cdr , -any sequence can be traversed using -.code cdr -functions, checking for the -.code nil -value as a terminator. This, however, breaks for empty sequences which are not -lists, because they are not equal to -.codn nil : -to -.code car -and -.code cdr -they look like -a one-element sequence containing -.codn nil . The -.code nullify -function reduces all empty -sequences to +.code mappend +function works like +.codn mapcar , +with the following difference. +Rather than accumulating the values returned by the function into a sequence, +mappend expects the items returned by the function to be sequences which +are catenated with +.codn append , +and the resulting sequence is returned. The returned sequence is of the same +kind as the leftmost input sequence, unless the values cannot be elements +of that type of sequence, in which case a list is returned. + +The +.code mapcar* +and +.code mappend* +functions work like +.code mapcar +and +.codn mappend , +respectively. +However, they return lazy lists rather than generating the entire +output list prior to returning. + +.TP* Caveats: + +Like +.codn mappend , +.code mappend* +must "consume" empty lists. For instance, +if the function being mapped puts out a sequence of +.codn nil -s, +then the result must be the empty list .codn nil , -thereby correcting the behavior of code which traverses -sequences using -.codn cdr , -and tests for termination with +because +.code "(append nil nil nil nil ...)" +is .codn nil . -.SS* Open Sequence Traversal +But suppose that +.code mappend* +is used on inputs which are infinite lazy +lists, such that the function returns +.code nil +values indefinitely. +For instance: -Functions in this category perform efficient traversal of sequences -of various kinds. +.verb + ;; Danger: infinite loop!!! + (mappend* (fun identity) (repeat '(nil))) +.brev -.coNP Function @ seq-begin -.synb -.mets (seq-begin << object ) -.syne -.desc The -.code seq-begin -function returns an iterator object specialized to the task of traversing -the sequence represented by the input -.metn object . +.code mappend* +function is caught in a loop trying to consume +and squash an infinite stream of +.codn nil -s, +and so doesn't return. -If -.meta object -isn't a sequence, an exception is thrown. +.TP* Examples: +.verb + ;; multiply every element by two + (mapcar (lambda (item) (* 2 item)) '(1 2 3)) -> (4 6 8) -.coNP Function @ seq-next + ;; "zipper" two lists together + (mapcar (lambda (le ri) (list le ri)) '(1 2 3) '(a b c)) + -> '((1 a) (2 b) (3 c))) + + ;; like append, mappend allows a lone atom or a trailing atom: + (mappend (fun identity) 3) -> (3) + (mappend (fun identity) '((1) 2)) -> (1 . 2) + + ;; take just the even numbers + (mappend (lambda (item) (if (evenp x) (list x))) '(1 2 3 4 5)) + -> (2 4) +.brev + +.coNP Functions @ maprod and @ maprend .synb -.mets (seq-next < iter << end-value ) +.mets (maprod < function << sequence *) +.mets (maprend < function << sequence *) .syne .desc The -.code seq-next -function retrieves the next available item from the sequence iterated by -.metn iter , -which must be an object returned by -.codn seq-begin . +.code maprod +and +.code maprend +functions resemble +.code mapcar +and +.codn mappend , +respectively. When given no +.meta sequence +arguments or exactly one +.meta sequence +argument, they behave exactly like those two functions. -If the sequence has no more items to be traversed, then -.meta end-value -is returned instead. +When two or more +.meta sequence +arguments are present, +.code maprod +differs from +.code mapcar +in the following way. Whereas +.code mapcar +iterates over the +.meta sequence +values in parallel, taking successive tuples of element +values and passing them to +.metn function , +the +.code maprod +function iterates over all +.I combinations +of elements from the sequences: the Cartesian product. The +.code prod +suffix stands for "product". -Note: to avoid ambiguities, the application should provide an -.meta end-value -which is guaranteed distinct from any item in the sequence, such as a -freshly allocated object. +If one or more +.meta sequence +arguments specify an empty sequence, then the Cartesian product is empty. +In this situation, +.meta function +is not called. The result of the function is then +.code nil +converted to the same kind of sequence as the leftmost +.metn sequence . -.coNP Function @ seq-reset -.synb -.mets (seq-reset < iter << object ) -.syne -.desc The -.code seq-reset -re-initializes the existing iterator object -.meta iter -to begin a new traversal over the given -.metn object , -which must be a value of a kind that would be a suitable argument for -.codn seq-begin . +.code maprod +function collects the values into a list just as +.code mapcar +does. Just like +.codn mapcar , +it converts the resulting list into the same kind of sequence +as the leftmost +.meta sequence +argument, if possible. For instance, if the resulting list is +a list or vector of characters, and the leftmost +.meta sequence +is a character string, then the list or vector of characters +is converted to a character string and returned. The -.code seq-reset -function returns -.metn iter . +.code maprend +function ("map product through function and append") iterates the +.meta sequence +element combinations exactly like +.codn maprod , +passing them as arguments to +.metn function . +The values returned by +.meta function +are then treated exactly as by the +.code mappend +function. The return values are expected to be sequences which +are appended together as if by +.codn append , +and the final result is converted to the same kind of sequence as the leftmost +.meta sequence +if possible. -.SS* Procedural List Construction +The combination iteration gives priority to the rightmost +.metn sequence , +which means that the rightmost element of each generated tuple varies +fastest: the tuples are traversed in "rightmost major" order. +This is made clear in the examples. -\*(TL provides an a structure type called -.code list-builder -which encapsulates state and methods for constructing lists procedurally. -Among the advantages of using -.code list-builder -is that lists can be constructed in the left to right direction without -requiring multiple traversals or reversal. For example, -.code list-builder -naturally combines with iteration or recursion: items visited in an -iterative or recursive process can be collected easily using -.code list-builder -in the order they are visited. +.TP* Examples -The basic workflow begins with the instantiation of a -.code list-builder -object. This object may be initialized with a piece of list material which -begins the to-be-constructed list, or it may be initialized to begin with an -empty list. Methods such as -.code add -and -.code pend -are invoked on this object to extend the list with new elements. At any point, -the list constructed so far is available using the -.code get -method, which is also how the final version of the list is eventually -retrieved. +.verb + [maprod list '(0 1 2) '(a b) '(i ii iii)] + -> + ((0 a i) (0 a ii) (0 a iii) (0 b i) (0 b ii) (0 b iii) + (1 a i) (1 a ii) (1 a iii) (1 b i) (1 b ii) (1 b iii) + (2 a i) (2 a ii) (2 a iii) (2 b i) (2 b ii) (2 b iii)) -The -.code build -macro is provided which syntactically streamlines the process. -It implicitly creates a -.code list-builder -instance and binds it to a hidden lexical variable. -It then evaluates forms in a lexical scope in which -short-hand macros are available for building the list. + ;; Vectors #(#\ea #\ex) #(#\ea #\ey) ... are appended + ;; together resulting in #(#\ea #\ex #\ea #\ey ...) + ;; which is converted to a string: -.coNP Structure @ list-builder -.synb -.mets (defstruct list-builder nil -.mets \ \ head tail) -.syne -.desc -The -.code list-builder -structure encapsulates the state for a list building process. -Programs should use the -.code build-list -function for creating an instance of -.codn list-builder . -The -.code head -and -.code tail -slots should be regarded as internal variables. + [maprend vec "ab" "xy"] -> "axaybxby" -.coNP Function @ build-list + ;; One of the sequences is empty, so the product is an + ;; empty sequence of the same kind as the leftmost + ;; sequence argument, thus an empty string: + [maprend vec "ab" ""] -> "" +.brev + +.coNP Function @ mapdo .synb -.mets (build-list <> [ initial-list ]) +.mets (mapdo < function << sequence *) .syne .desc The -.code build-list -function instantiates and returns an object of struct type -.codn list-builder . - -If no -.meta initial-list -argument is supplied, then the object is implicitly -with an empty list. - -If the argument is supplied, then it is equivalent -to calling -.code build-list -without an argument to produce an object -.meta obj -the invoking the method call -.mono -.meti << obj .(ncon << list ) -.onom -on this object. The object produced by the expression -.meta list -is installed (without being copied) into the -object as the prefix of the list to be constructed. +.code mapdo +function is similar to +.codn mapcar , +but always returns +.codn nil . +It is useful +when +.meta function +performs some kind of side effect, hence the "do" in the name, +which is a mnemonic for the execution of imperative actions. -.TP* Example: +When only the +.meta function +argument is given, +.meta function +is never called, +and +.code nil +is returned. -.verb - ;; build the list (a b) trivially +If a single +.meta sequence +argument is given, then +.code mapdo +iterates over +.metn sequence , +invoking +.meta function +on each element. - (let ((lb (build-list '(a b)))) - lb.(get) - -> (a b) -.brev +If two or more +.meta sequence +arguments are given, then +.code mapdo +iterates over +the sequences in parallel, extracting parallel tuples of items. These +tuples are passed as arguments to +.metn function , +which must accept as many +arguments as there are sequences. -.coNP Methods @ add and @ add* +.coNP Functions @ transpose and @ zip .synb -.mets << list-builder .(add << element *) -.mets << list-builder .(add* << element *) +.mets (transpose << sequence ) +.mets (zip << sequence *) .syne .desc The -.code add -and -.code add* -methods extend the list being constructed by a -.code list-builder -object by adding individual -elements to it. The -.code add -method adds elements at the tail of the list, -whereas -.code add* -adds elements at the front. +.code transpose +function performs a transposition on +.metn sequence . +This means that the +elements of +.meta sequence +must be sequences. These sequences are understood to be +columns; transpose exchanges rows and columns, returning a sequence of the rows +which make up the columns. The returned sequence is of the same kind as +.metn sequence , +and the rows are also the same kind of sequence as the first column +of the original sequence. The number of rows returned is limited by the +shortest column among the sequences. -The return value of these methods is unspecified. +All of the input sequences (the elements of +.metn sequence ) +must have elements +which are compatible with the first sequence. This means that if the first +element of +.meta sequence +is a string, then the remaining sequences must be +strings, or else sequences of characters, or of strings. -.TP* Example: +The +.code zip +function takes variable arguments, and is equivalent to calling +.code transpose +on a list of the arguments. The following equivalences hold: .verb - ;; Build the list (1 2 3 4) + (zip . x) <--> (transpose x) - (let ((lb (build-list))) - lb.(add 3 4) - lb.(add* 1 2) - lb.(get)) - -> (1 2 3 4) + [apply zip x] <--> (transpose x) .brev -.coNP Methods @ pend and @ pend* -.synb -.mets << list-builder .(pend << list *) -.mets << list-builder .(pend* << list *) -.syne -.desc -The -.code pend -and -.code pend* -methods extend the list being constructed by a -.code list-builder -object by adding lists to it. The -.code pend -method catenates the -.code list -arguments together as if by the -.code append -function, then appends the resulting list to -the end of the list being constructed. -The -.code pend* -method is similar, except it prepends the -catenated lists to the front of the list -being constructed. - -Both methods have the property that the -constructed list does not share structure -with the input lists. +.TP* Examples: +.verb + ;; transpose list of lists + (transpose '((a b c) (c d e))) -> ((a c) (b d) (c e)) -The return value of these methods is unspecified. + ;; transpose vector of strings: + ;; - string columns become string rows + ;; - vector input becomes vector output + (transpose #("abc" "def" "ghij")) -> #("adg" "beh" "cfi") -.TP* Example: + ;; error: transpose wants to make a list of strings + ;; but 1 is not a character + (transpose #("abc" "def" '(1 2 3))) ;; error! -.verb - ;; Build the list (1 2 3 4) + ;; String elements are catenated: + (transpose #("abc" "def" ("UV" "XY" "WZ"))) + -> #("adUV" "beXY" "cfWZ") - (let ((lb (build-list))) - lb.(pend '(3 4)) - lb.(pend* '(1 2)) - lb.(get)) - -> (1 2 3 4) + (zip '(a b c) '(c d e)) -> ((a c) (b d) (c e)) .brev -.coNP Methods @ ncon and @ ncon* +.coNP Functions @, window-map @ window-mappend and @ window-mapdo .synb -.mets << list-builder .(ncon << list *) -.mets << list-builder .(ncon* << list *) +.mets (window-map < range < boundary < function << sequence ) +.mets (window-mappend < range < boundary < function << sequence ) +.mets (window-mapdo < range < boundary < function << sequence ) .syne .desc The -.code ncon +.code window-map and -.code ncon* -methods extend the list being constructed by a -.code list-builder -object by adding lists to it. The -.code ncon -method destructively catenates the -.meta list -arguments as if by the -.code nconc -function. The resulting list is appended -to the list being constructed. -The -.code ncon* -method is similar, except it prepends -the catenated lists to the front of the -list being constructed. - -These methods destructively manipulate -the input lists. Moreover, they cause the -list being constructed to share substructure -with the input lists. - -Additionally, the -.code ncon* -function can be called with a single argument -which is an atom. This atom will simply be -installed as the terminating atom of the -list being constructed. +.code window-mappend +functions process the elements of +.meta sequence +by passing arguments derived from each successive element to +.metn function . +Both functions return, if possible, a sequence of the same kind as +.codn sequence , +otherwise a list. -The return value of these methods is unspecified. +Under +.codn window-map , +values returned by +.meta function +are accumulated into a sequence of the same type as +.meta sequence +and that sequence is returned. Under +.codn window-mappend , +the values returned by the calls to +.meta function +are expected to be sequence which are appended together to +form the output sequence. -.TP* Example: +These functions are analogous to +.code mapcar +and +.codn mappend . +Unlike these, they operate only on a single sequence, and over this sequence +they perform a +.IR "sliding window mapping" , +whose description follows. -.verb - ;; Build the list (1 2 3 4 . 5) +The function +.code window-mappend +avoids accumulating a sequence, and instead returns +.codn nil ; +it is analogous to +.codn mapdo . - (let ((lb (build-list))) - lb.(ncon* (list 1 2)) - lb.(ncon (list 3 4)) - lb.(ncon 5) - lb.(get)) - -> (1 2 3 4 . 5) -.brev +The argument to the +.meta range +parameter must be a positive integer, not exceeding 512. +This parameter specifies the amount of ahead/behind context on either +side of each element which is processed. It indirectly determines +the window size for the mapping. The window size is twice +.metn range , +plus one. For instance if range is 2, then the window size is 5: +the element being processed lies at the center of the window, flanked +by two elements on either side, making five. -.coNP Method @ get -.synb -.mets << list-builder .(get) -.syne -.desc The -.code get -method retrieves the list constructed so far by a -.code list-builder -object. It doesn't change the state of the object. -The retrieved list may be passed as an argument -into the construction methods on the same object. +.meta function +argument must specify a function which accepts a number of arguments +corresponding to the window size. For instance if +.meta range +is 2, +making the window size 5, +then +.meta function +must accept 5 arguments. These arguments constitute the sliding +window being processed. Each time +.meta function +is called, the middle argument is the element being processed, +and the arguments surrounding it are its window. -.TP* Examples: +When an element is processed from somewhere in the interior of +a sequence, where it is flanked on either side by at least +.meta range +elements, then the window is populated by those flanking elements +taken from +.metn sequence . -.verb - ;; Build the circular list (1 1 1 1 ...) - ;; by appending (1) to itself destructively: +The +.meta boundary +parameter specifies the window contents which are used for the +processing of elements which are closer than +.meta range +to either end of the sequence. The argument may be a sequence containing +at least twice +.meta range +number of elements (one less than the window size): if it has additional +elements, they are not used. If it is a list, it may be shorter than twice +.metn range . +The argument +may also be one of the two keyword symbols +.code :wrap +or +.codn :reflect , +described below. - (let ((lb (build-list '(1)))) - lb.(ncon* lb.(get)) - lb.(get)) - -> (1 1 1 1 ...) +If +.meta boundary +is a sequence, it may be regarded as divided into two pieces of +.meta range +length. If it is a list of insufficient length, then missing elements +are supplied as +.code nil +to make two +.metn range 's +worth of elements. These two pieces then flank +.code sequence +on either end. The left half of +.meta boundary +is effectively prepended to the sequence, and the right half +effectively appended. +When the sliding window extends beyond the boundary of +.meta sequence +near its start or end, the window is populated from these +flanking elements obtained from +.metn boundary . - ;; build the list (1 2 1 2 1 2 1 2) - ;; by doubling (1 2) twice: +If +.meta boundary +is the keyword +.codn :wrap , +then the sequence is effectively flanked by copies of itself on both +ends, repeated enough times to satisfy the window. For instance if +the sequence is +.code "(1 2 3)" +and the window size is 9 due to the value of +.meta range +being 7, then the behavior of +.code :wrap +is as if a +.meta boundary +were specified consisting of +.codn "(3 1 2 3 1 2 3 1)" . +The left flank is +.code "(3 1 2 3)" +and the right flank is +.code "(1 2 3 4)" +formed by repetitions of +.code "(1 2 3)" +surrounding it on either side, extending out to infinity, and chopped to +.metn range . - (let ((lb (build-list))) - lb.(add 1 2) - lb.(pend lb.(get)) - lb.(pend lb.(get)) - lb.(get)) - -> (1 2 1 2 1 2 1 2) -.brev +If +.meta boundary +is the keyword +.codn :reflect , +then the sequence is effectively flanked by reversed copies of itself +on both ends, repeated enough times to satisfy the window. +For instance if the sequence is +.code "(1 2 3)" +and the window size is 9, then the behavior of +.code :wrap +is as if a +.meta boundary +were specified consisting of +.codn "(1 3 2 1 3 2 1 3)" . -.coNP Macro @ build +.coNP Function @ interpose .synb -.mets (build << form *) +.mets (interpose < sep << sequence ) .syne .desc The -.code build -macro provides a shorthand notation for constructing lists using the -.code list-builder -structure. It eliminates the explicit call to the -.code build-list -function to construct the object, and eliminates the explicit -references to the object. +.code interpose +function returns a sequence of the same type as +.metn sequence , +in which the elements from +.meta sequence +appear with the +.meta sep +value inserted +between them. -Instead, -.code build -creates a lexical environment in which a -.code list-builder -object is implicitly constructed and bound to a hidden variable. -Local functions which mimic the -.code list-builder -methods operate implicitly on this hidden variable, so that -the object need not be mentioned as an argument. +If +.meta sequence +is an empty sequence or a sequence of length 1, then a +sequence identical to +.meta sequence +is returned. It may be a copy of +.meta sequence +or it may be +.meta sequence +itself. -With the exception of -.codn get , -these local functions -.codn get , -have unspecified return values, like the same-named -.code list-builder -methods. +If +.meta sequence +is a character string, then the value +.meta sep +must be a character. -.TP* Examples: +It is permissible for +.metn sequence , +or for a suffix of +.meta sequence +to be a lazy +list, in which case interpose returns a lazy list, or a list with a lazy +suffix. +.TP* Examples: .verb - ;; Build the circular list (1 1 1 1 ...) - ;; by appending (1) to itself destructively: - - (build - (add 1) - (ncon* (get))) -> (1 1 1 1 ...) - - ;; build the list (1 2 1 2 1 2 1 2) - ;; by doubling (1 2) twice: - - (build - (add 1 2) - (pend (get)) - (pend (get))) -> (1 2 1 2 1 2 1 2) - - ;; build a list by mapping over the local - ;; add function: - - (build [mapdo add (range 1 3)]) -> (1 2 3) + (interpose #\e- "xyz") -> "x-y-z" + (interpose t nil) -> nil + (interpose t #()) -> #() + (interpose #\ea "") -> "" + (interpose t (range 0 0)) -> (0) + (interpose t (range 0 1)) -> (0 t 1) + (interpose t (range 0 2)) -> (0 t 1 t 2) .brev -.SS* Permutations and Combinations - -.coNP Function @ perm +.coNP Functions @ apply and @ iapply .synb -.mets (perm < seq <> [ len ]) +.mets (apply < function <> [ arg * << trailing-args ]) +.mets (iapply < function <> [ arg * << trailing-args ]) .syne .desc The -.code rperm -function returns a lazy list which consists of all -length -.meta len -permutations of formed by items taken from -.metn seq . -The permutations do not use any element of -.meta seq -more than once. - -Argument -.metn len , -if present, must be a positive integer, and -.meta seq -must be a sequence. +.code apply +function invokes +.metn function , +optionally passing to it an argument +list. The return value of the +.code apply +call is that of +.metn function . -If -.meta len -is not present, then its value defaults to the length of -.metn seq : -the list of the full permutations of the entire sequence is returned. - -The permutations in the returned list are sequences of the same kind as -.codn seq . +If no arguments are present after +.metn function , +then +.meta function +is invoked without arguments. +If one argument is present after +.metn function , +then it is interpreted as +.metn trailing-args . +If this is a sequence (a list, vector or string), +then the elements of the sequence are passed as individual arguments to +.metn function . If -.meta len -is zero, then a list containing one permutation is returned, and that -permutation is of zero length. +.meta trailing-args +is not a sequence, then +.meta function +is invoked +with an improper argument list, terminated by the +.meta trailing-args +atom. -If -.meta len -exceeds the length of -.metn seq , -then an empty list is returned, -since it is impossible to make a single non-repeating permutation that -requires more items than are available. +If two or more arguments are present after +.metn function , +then the last of these arguments is interpreted as +.metn trailing-args . +The previous arguments represent leading arguments which are applied to +.metn function , +prior to the arguments taken from +.metn trailing-args . -The permutations are lexicographically ordered. +Note that if +.meta trailing-args +value is an atom or an improper list, the function is then +invoked with an improper argument list. Only a variadic +function may be invoked with an improper argument lists. +Moreover, all of the function's required and optional +parameters must be satisfied by elements of the +improper list, such that the terminating atom either +matches the +.meta rest-param +directly (see the +.code lambda +operator) or else the +.meta rest-param +receives an improper list terminated by that atom. +To treat the terminating atom of an improper list as an +ordinary element which can satisfy a required or optional +function parameter, the +.code iapply +function may be used, described next. -.coNP Function @ rperm -.synb -.mets (rperm < seq << len ) -.syne -.desc The -.code rperm -function returns a lazy list which consists of all the repeating -permutations of length -.meta len -formed by items taken from -.metn seq . -"Repeating" means that the items from -.meta seq -can appear more than -once in the permutations. - -The permutations which are returned are sequences of the same kind as -.metn seq . +.code iapply +function ("improper apply") is similar to +.codn apply , +except with regard to the treatment of +.metn trailing-args . +Firstly, under +.codn iapply , +if +.meta trailing-args +is an atom other than +.code nil +(possibly a sequence, such as a vector or string), +then it is treated as an ordinary argument: +.meta function +is invoked with a proper argument list, whose last element is +.metn trailing-args . +Secondly, if +.meta trailing-args +is a list, but an improper list, then the terminating atom of +.meta trailing-args +becomes an individual argument. +This terminating atom is not split into multiple arguments, +even if it is a sequence. +Thus, in all possible cases, +.code iapply +treats an extra +.cod2 non- nil +atom as an argument, and never calls +.meta function +with an improper argument list. -Argument -.meta len -must be a nonnegative integer, and -.meta seq -must be a sequence. +.TP* Examples: +.verb + ;; '(1 2 3) becomes arguments to list, thus (list 1 2 3). + (apply (fun list) '(1 2 3)) -> (1 2 3) -If -.meta len -is zero, then a single permutation is returned, of zero length. -This is true regardless of whether -.meta seq -is itself empty. + ;; this effectively invokes (list 1 2 3 4) + (apply (fun list) 1 2 '(3 4)) -> (1 2 3 4) -If -.meta seq -is empty and -.meta len -is greater than zero, then no permutations are -returned, since permutations of a positive length require items, and the -sequence has no items. Thus there exist no such permutations. + ;; this effectively invokes (list 1 2 . 3) + (apply (fun list) 1 2 3)) -> (1 2 . 3) -The first permutation consists of -.meta le -repetitions of the first element of -.metn seq . -The next repetition, if there is one, differs from the first -repetition in that its last element is the second element of -.metn seq . -That is to say, the permutations are lexicographically ordered. + ;; "abc" is separated into characters + ;; which become arguments of list + (apply (fun list) "abc") -> (#\ea #\eb #\ec) +.brev -.TP* Examples: +.TP* "Dialect Note:" +Note that some uses of this function that are necessary in other Lisp dialects +are not necessary in \*(TL. The reason is that in \*(TL, improper list +syntax is accepted as a compound form, and performs application: .verb - (rperm "01" 4) -> ("000" "001" "010" "011" - "100" "101" "110" "111") - - (rperm #(1) 3) -> (#(1 1 1)) - - (rperm '(0 1 2) 2) -> ((0 0) (0 1) (0 2) (1 0) - (1 1) (1 2) (2 0) (2 1) (2 2)) + (foo a b . x) .brev -.coNP Function @ comb +Here, the variables +.code a +and +.code b +supply the first two arguments for +.codn foo . +In +the dotted position, +.code x +must evaluate to a list or vector. The list or +vector's elements are pulled out and treated as additional arguments for +.codn foo . +Of course, this syntax can only be used if +.code x +is a symbolic form or an atom. It +cannot be a compound form, because +.code "(foo a b . (x))" +and +.code "(foo a b x)" +are equivalent structures. + +.coNP Functions @ reduce-left and @ reduce-right .synb -.mets (comb < seq << len ) +.mets (reduce-left < binary-function < list +.mets \ \ \ \ \ \ \ \ \ \ \ \ >> [ init-value <> [ key-function ]]) + +.mets (reduce-right < binary-function < list +.mets \ \ \ \ \ \ \ \ \ \ \ \ \ >> [ init-value <> [ key-function ]]) .syne .desc The -.code comb -function returns a lazy list which consists of all -length -.meta len -non-repeating combinations formed by taking items taken from -.metn seq . -"Non-repeating combinations" means that the combinations do not use any -element of -.meta seq -more than once. If -.meta seq -contains no duplicates, then -the combinations contain no duplicates. - -Argument -.meta len -must be a nonnegative integer, and -.meta seq -must be a sequence or a hash table. - -The combinations in the returned list are objects of the same kind as -.metn seq . +.code reduce-left +and +.code reduce-right +functions reduce lists of operands specified +by +.meta list +and +.meta init-value +to a single value by the repeated application of +.metn binary-function . +An effective list of operands is formed by combining +.meta list +and +.metn init-value . If -.meta len -is zero, then a list containing one combination is returned, and that -combination is of zero length. - +.meta key-function +is specified, then the items of +.meta list +are +mapped to new values through +.metn key-function , +as if by +.codn mapcar . If -.meta len -exceeds the number of elements in -.metn seq , -then an empty list is returned, since it is impossible to make a single -non-repeating combination that requires more items than are available. +.meta init-value +is supplied, +then in the case of +.codn reduce-left , +the effective list of operands is formed by +prepending +.meta init-value +to +.metn list . +In the case of +.codn reduce-right , +the effective operand list is produced by appending +.meta init-value +to +.metn list . +The +.meta init-value +isn't mapped through +.metn key-function . -If -.meta seq -is a sequence, the returned combinations are lexicographically ordered. -This requirement is not applicable when -.meta seq -is a hash table. +The production of the effective list can be expressed like this, +though this is not to be understood as the actual implementation: -.TP* Example: .verb - ;; powerset function, in terms of comb. - ;; Yields a lazy list of all subsets of s, - ;; expressed as sequences of the same type as s. - - (defun powerset (s) - (mappend* (op comb s) (range 0 (length s)))) + (append (if init-value-present (list init-value)) + [mapcar (or key-function identity) list])))) .brev -.coNP Function @ rcomb -.synb -.mets (rcomb < seq << len ) -.syne -.desc -The -.code comb -function returns a lazy list which consists of all -length -.meta len -repeating combinations formed by taking items taken from -.metn seq . -"Repeating combinations" means that the combinations can use -an element of -.meta seq -more than once. - -Argument -.meta len -must be a nonnegative integer, and -.meta seq -must be a sequence. +In the +.code reduce-right +case, the arguments to +.code append +are reversed. -The combinations in the returned list are sequences of the same kind as -.metn seq . +If the effective list of operands is empty, then +.meta binary-function +is called +with no arguments at all, and its value is returned. This is the only +case in which +.meta binary-function +is called with no arguments; in all +remaining cases, it is called with two arguments. -If -.meta len -is zero, then a list containing one combination is returned, and that -combination is of zero length. This is true even if -.meta seq -is empty. +If the effective list contains one item, then that item is returned. -If -.meta seq -is empty, and -.meta len -is nonzero, then an empty list is returned. +Otherwise, the effective list contains two or more items, and is decimated as +follows. -The combinations are lexicographically ordered. +Note that an +.meta init-value +specified as +.code nil +is not the same as a missing +.metn init-value ; +this means that the initial value is the object +.codn nil . +Omitting +.meta init-value +is the same as specifying a value of +.code : +(the colon symbol). +It is possible to specify +.meta key-function +while omitting an +.meta init-value +argument. This is achieved by explicitly specifying +.code : +as the +.meta init-value +argument. +Under +.codn reduce-left , +the leftmost pair of operands is removed +from the list and passed as arguments to +.metn binary-function , +in the same order +that they appear in the list, and the resulting value initializes an +accumulator. Then, for each remaining item in the list, +.meta binary-function +is invoked on two arguments: the current accumulator value, and the next element +from the list. After each call, the accumulator is updated with the return +value of +.metn binary-function . +The final value of the accumulator is returned. -.SS* Macros -\*(TL supports structural macros. \*(TX's model of macroexpansion is that -\*(TL code is processed in two phases: the expansion phase and the -evaluation phase. The expansion phase is invoked on Lisp code early during the -processing of source code. For instance when a \*(TX file containing a -.code "@(do ...)" -directive -is loaded, expansion of the Lisp forms are its arguments takes place during the -parsing of the entire source file, and is complete before any of the code in -that file is executed. If the -.code "@(do ...)" -form is later executed, -the expanded forms are then evaluated. +Under +.codn reduce-right , +the list is processed right to left. The rightmost +pair of elements in the effective list is removed, and passed as arguments to +.metn binary-function , +in the same order that they appear in the list. The +resulting value initializes an accumulator. Then, for each remaining item in +the list, +.meta binary-function +is invoked on two arguments: the +next element from the list, in right to left order, and the current +accumulator value. After each call, the accumulator is updated with the return +value of +.metn binary-function . +The final value of the accumulator is returned. -\*(TL also supports symbol macros, which are symbolic forms that stand -for forms, with which they are replaced at macro expansion time. +.TP* Examples: +.verb + ;;; effective list is (1) so 1 is returned + (reduce-left (fun +) () 1 nil) -> 1 -When Lisp data is processed as code by the -.code eval -function, it is first expanded, -and so processed in its entirety in the expansion phase. Then it is processed -in the evaluation phase. + ;;; computes (- (- (- 0 1) 2) 3) + (reduce-left (fun -) '(1 2 3) 0 nil) -> -6 -.NP* Macro parameter lists + ;;; computes (- 1 (- 2 (- 3 0))) + (reduce-right (fun -) '(1 2 3) 0 nil) -> 2 -\*(TX macros support destructuring, similarly to Common Lisp macros. -This means that macro parameter lists are like function argument lists, -but support nesting. A macro parameter list can specify a nested parameter -list in every place where an argument symbol may appear. For instance, -consider this macro parameter list: + ;;; computes (* 1 2 3) + (reduce-left (fun *) '((1) (2) (3)) nil (fun first)) -> 6 -.verb - ((a (b c)) : (c frm) ((d e) frm2 de-p) . g) + ;;; computes 1 because the effective list is empty + ;;; and so * is called with no arguments, which yields 1. + (reduce-left (fun *) nil) .brev -The top-level of this nested form has the structure - -.mono -.meti \ \ >> ( I : < J < K . << L ) -.onom - -in which we can identify the major constituent positions as -.metn I , -.metn J , -.meta K +.coNP Functions @, some @ all and @ none +.synb +.mets (some < sequence >> [ predicate-fun <> [ key-fun ]]) +.mets (all < sequence >> [ predicate-fun <> [ key-fun ]]) +.mets (none < sequence >> [ predicate-fun <> [ key-fun ]]) +.syne +.desc +The +.codn some , +.code all and -.metn L . - -The constituent at position -.meta I -is the mandatory parameter -.codn "(a (b c))" . -Position -.meta J -holds the optional parameter -.code c -(with default init form -.codn frm ). -At -.meta K -is found the optional parameter -.code "(d e)" -(with default init form -.code frm2 -and presence-indicating variable -.codn de-p ). -Finally, the parameter in the dot position -.meta L +.code none +functions apply a predicate test function +.meta predicate-fun +over a list of elements. If the argument +.meta key-fun is -.codn g , -which captures trailing arguments. - -Obviously, some of the parameters are compound expressions rather -than symbols: -.code "(a (b c))" +specified, then elements of +.meta sequence +are passed into +.metn key-fun , and -.codn "(d e)" . -These compounds express nested macro parameter lists. +.meta predicate-fun +is +applied to the resulting values. If +.meta key-fun +is omitted, the behavior is +as if +.meta key-fun +is the identity function. If +.meta predicate-fun +is omitted, +the behavior is as if +.meta predicate-fun +is the identity function. -Nested macro parameter lists recursively match the corresponding structure -in the argument object. For instance if a simple argument would capture -the structure -.code "(1 (2 3))" -then we can replace the argument with the nested argument list -.code "(a (b c))" -which destructures the -.code "(1 (2 3))" -such that the parameters -.codn a , -.code b -and -.code c -will end up bound -to -.codn 1 , -.code 2 -and -.codn 3 , -respectively. +These functions have short-circuiting semantics and return conventions similar +to the and and or operators. -Nested macro parameter lists have all the features of the top-level -macro parameter lists: they can have optional arguments with default -values, use the dotted position, and contain the -.codn :env , -.code :whole -and -.code :form -special parameters, which are described below. -In nested parameter lists, the binding strictness is relaxed for optional -parameters. If -.code "(a (b c))" -is optional, and the argument is, say, -.codn (1) , -then -.code a -gets -.codn 1 , -and -.code b -and -.code c -receive +The some function applies +.meta predicate-fun +to successive values +produced by retrieving elements of +.meta list +and processing them through +.metn key-fun . +If the list is empty, it returns .codn nil . - -Macro parameter lists also supports three special keywords, namely -.codn :env , -.code :whole +Otherwise it returns the +first +.cod2 non- nil +return value returned by a call to +.meta predicate-fun and -.codn :form . - -The parameter list -.code "(:whole x :env y :form z)" -will bind parameter -.code x -to the entire -macro parameter list, bind parameter -.code y -to the macro environment and bind parameter -.code z -to the entire macro form (the original compound form used to invoke the -macro). +stops evaluating more elements. If +.meta predicate-fun +returns +.code nil +for all +elements, it returns +.metn nil . The -.codn :env , -.code :whole -and -.code :form -notations can occur anywhere in a macro parameter list, other than -to the right of the consing dot. They can be used in nested -macro parameter lists also. Note that in a nested macro -parameter list, -.code :form -and -.code :env -do not change meaning: they bind the same object as they would in -the top-level of the macro parameter list. -However the -.code :whole -parameter inside has a restricted scope in a nested parameter -list: its parameter will capture just that part of the argument material which -matches that parameter list, rather than the entire argument list. - -The processing of macro parameter lists omits the feature that when the -keyword symbol -.code : -(colon) given as the argument to an optional parameter, that argument is -treated as a missing argument. This special logic is implemented only -in the function argument passing mechanism, not in the binding of macro -parameters to object structure. If the colon symbol appears in the object -structure and is matched against an optional parameter, it is an -ordinary value. That parameter is considered present, and takes on -that -.code : -keyword symbol as its value. +.code all +function applies +.meta predicate-fun +to successive values +produced by retrieving elements of +.meta list +and processing them through +.metn key-fun . +If the list is empty, it returns +.codn t . +Otherwise, if +.meta predicate-fun +yields +.code nil +for any value, the +.code all +function immediately +returns without invoking +.meta predicate-fun +on any more elements. +If all the elements are processed, then the all function returns +the value which +.meta predicate-fun +yielded for the last element. -.TP* "Dialect Note:" +The +.code none +function applies +.meta predicate-fun +to successive values +produced by retrieving elements of +.meta list +and processing them through +.metn key-fun . +If the list is empty, it returns +.codn t . +Otherwise, if +.meta predicate-fun +yields +.cod2 non- nil +for any value, the none function +immediately returns nil. If +.meta predicate-fun +yields nil for all +values, the none function returns +.codn t . -In ANSI Common Lisp, the lambda list keyword -.code &whole -binds its corresponding variable to the entire macro form, whereas -\*(TL's -.code :whole -binds its variable only to the arguments of the macro form. +.TP* Examples: -Note, however, that ANSI CL distinguishes destructuring lambda lists -and macro lambda lists and the -.code &whole -parameter has a different behavior between the two. Under -.codn destructuring-bind , -the -.code &whole -parameter receives just the arguments, just like the behavior -of \*(TL's -.code :whole -parameter. +.verb + ;; some of the integers are odd + [some '(2 4 6 9) oddp] -> t -\*(TL does not distinguish destructuring and macro lambda lists; -they are the same and behave the same way. Thus -.code :whole -is treated the same way in macros as in -.code tree-bind -and related binding operators: it binds just the arguments -to the parameter. \*(TL has the special parameter -.code :form -by means of which macros can access their invoking form. -This parameter is also supported in -.code tree-bind -and binds to the entire -.code tree-bind -form. + ;; none of the integers are even + [none '(1 3 4 7) evenp] -> t +.brev -.coNP Operator @ macro-time +.coNP Function @ multi .synb -.mets (macro-time << form *) +.mets (multi < function << sequence *) .syne .desc The -.code macro-time -operator has a syntax similar to the -.code progn -operator. Each -.meta form -is evaluated from left to right, and the resulting value is that of the last -form. +.code multi +function distributes an arbitrary list processing function +.meta multi +over multiple sequences given by the +.meta list +arguments. -The special behavior of -.code macro-time -is that the evaluation takes place during -the expansion phase, rather than during the evaluation phase. +The +.meta sequence +arguments are first transposed into a single list of tuples. Each +successive element of this transposed list consists of a tuple of the +successive items from the lists. The length of the transposed list is that +of the shortest +.meta list +argument. -Also, -.code macro-time -macro-expands each -.meta form -and evaluates it before processing the next -.meta form -in the same way. Thus, for instance, if a -.meta form -introduces a global definition, that definition will be visible not -only during the evaluation of a subsequent -.metn form , -but also during its macro-expansion time. +The transposed list is then passed to +.meta function +as an argument. -During the expansion phase, all -.code macro-time -expressions which occur in a context -that calls for evaluation are evaluated, and replaced by their quoted values. -For instance -.code "(macro-time (list 1 2 3))" -evaluates -.code "(list 1 2 3)" -to the object -.code "(1 2 3)" -and the entire -.code macro-time -form is replaced by that value, quoted: -.codn "'(1 2 3)" . -If the form is evaluated again at evaluation-time, the resulting value will be -that of the quote, in this case -.codn "(1 2 3)" . +The +.meta function +is expected to produce a list of tuples, which are transposed +again to produce a list of lists which is then returned. -.code macro-time -forms do not see the surrounding lexical environment; the see only -global function and variable bindings and macros. +Conceptually, the input sequences are columns and +.meta function +is invoked on +a list of the rows formed from these columns. The output of +.meta function +is a transformed list of rows which is reconstituted into a list of columns. -Note 1: -.code macro-time -is intended for defining helper functions and variables that -are used by macros. A macro cannot "see" a -.code defun -function or -.code defvar -variable -because -.code defun -and -.code defvar -forms are evaluated at evaluation time, which occurs -after expansion time. The macro-time operator hoists the evaluation of these -forms to macro-expansion time. +.TP* Example: -Note 2: -.code defmacro -forms are implicitly in macro-time; they do not have to -be wrapped with the -.code macro-time -operator. The -.code macro-time -operator doesn't have -to be used in programs whose macros do not make references to variables -or functions. +.verb + ;; Take three lists in parallel, and remove from all of them + ;; them the element at all positions where the third list + ;; has an element of 20. -.coNP Operator @ defmacro + (multi (op remove-if (op eql 20) @1 third) + '(1 2 3) + '(a b c) + '(10 20 30)) + + -> ((1 3) (a c) (10 30)) + + ;; The (2 b 20) "row" is gone from the three "columns". + + ;; Note that the (op remove if (op eql 20) @1 third) + ;; expression can be simplified using the ap operator: + ;; + ;; (op remove-if (ap eql @3 20)) +.brev + +.coNP Function @ sort .synb -.mets (defmacro < name -.mets \ \ \ \ \ \ \ \ \ <> ( param * [: << opt-param * ] [. < rest-param ]) -.mets \ \ << body-form *) +.mets (sort < sequence >> [ lessfun <> [ keyfun ]]) .syne .desc The -.code defmacro -operator is evaluated at expansion time. It defines a -macro-expander function under the name -.metn name , -effectively creating a new operator. +.code sort +function destructively sorts +.metn sequence , +producing a sequence +which is sorted according to the +.meta lessfun +and +.meta keyfun +arguments. -Note that the above syntax synopsis describes only the canonical -parameter syntax which remains after parameter list macros are -expanded. See the section Parameter List Macros. +The +.meta keyfun +argument specifies a function which is applied to elements +of the sequence to obtain the key values which are then compared +using the lessfun. If +.meta keyfun +is omitted, the identity function is used +by default: the sequence elements themselves are their own sort keys. -Note that the parameter list is a macro parameter list, and not a -function parameter list. This means that each -.meta param -and -.meta opt-param -can be not only a symbol, but it can itself be a parameter list. -The corresponding argument is then treated as a structure which -matches that parameter list. This nesting of parameter lists -can be carried to an arbitrary depth. +The +.meta lessfun +argument specifies the comparison function which determines +the sorting order. It must be a binary function which can be invoked +on pairs of keys as produced by the key function. It must +return a +.cod2 non- nil +value if the left argument is considered to be lesser +than the right argument. For instance, if the numeric function +.code < +is used +on numeric keys, it produces an ascending sorted order. If the function +.code > +is used, then a descending sort is produced. If +.meta lessfun +is omitted, then it defaults to the generic +.code less +function. -A macro is called like any other operator, and resembles a function. Unlike in -a function call, the macro receives the argument expressions themselves, rather -than their values. Therefore it operates on syntax rather than on values. -Also, unlike a function call, a macro call occurs in the expansion phase, -rather than the evaluation phase. +The +.code sort +function is stable for sequences which are lists. This means that the +original order of items which are considered identical is preserved. +For strings and vectors, +.code sort +is not stable. -The return value of the macro is the macro expansion. It is substituted in -place of the entire macro call form. That form is then expanded again; -it may itself be another macro call, or contain more macro calls. +.coNP Function @ grade +.synb +.mets (grade < sequence >> [ lessfun <> [ keyfun ]]) +.syne +.desc +The +.code grade +function returns a list of integer indices which indicate the position +of the elements of +.meta sequence +in sorted order. -A global macro defined using -.code defmacro -may decline to expand a macro form. Declining to expand is achieved by -returning the original unexpanded form, which may be captured using the -.code :form -parameter. When a global macro declines to expand a form, the form is -taken as-is. At evaluation time, it will be treated as a function call. -Note: when a local macro defined by -.code macrolet -declines, more complicated requirements apply; see the description of -.codn macrolet . +The +.meta lessfun +and +.meta keyfun +arguments behave like those of the +.code sort +function. -.TP* "Dialect Notes:" -A macro in the global namespace introduced by -.code defmacro -may co-exist with a function of the same name introduced by -.codn defun . -This is not permitted in ANSI Common Lisp. +The +.meta sequence +object is not modified. -ANSI Common Lisp doesn't describe the concept of declining to expand, except in -the area of compiler macros. Since TXR Lisp allows global macros and functions -of the same name to co-exist, ordinary macros can be used to optimize functions -in a manner similar to Common Lisp compiler macros. A macro can be written -of the same name as a function, and can optimize certain cases of the function -call by expanding them to some alternative syntax. Cases which it doesn't -optimize are handled by declining to expand, in which case the form remains -as the original function call. +The internal sort performed by +.code grade +is not stable. The indices of any elements considered equivalent under +.code lessfun +may appear in any order in the returned index sequence. -.TP* Example: +Note: the +.code grade +function is inspired by the "grade up" and "grade down" operators +in the APL language. + +.TP* Examples: .verb - ;; dolist macro similar to Common Lisp's: - ;; - ;; The following will print 1, 2 and 3 - ;; on separate lines: - ;; and return 42. - ;; - ;; (dolist (x '(1 2 3) 42) - ;; (format t "~s\en")) + ;; Order of the 2 3 positions of the "l" + ;; characters is not specified: - (defmacro dolist ((var list : result) . body) - (let ((i (my-gensym))) - ^(for ((i ,list)) (i ,result) ((set i (cdr i))) - (let ((,var (car i))) - ,*body)))) + [grade "Hello"] -> (0 1 2 3 4) + [grade "Hello" >] -> (4 2 3 1 0) .brev -.coNP Operator @ macrolet +.coNP Function @ shuffle .synb -.mets (macrolet >> ({( name < macro-style-params -.mets \ \ \ \ \ \ \ \ \ \ \ \ \ \ << macro-body-form *)}*) -.mets \ \ << body-form *) +.mets (shuffle << sequence ) .syne .desc The -.code macrolet -binding operator extends the macro-time lexical environment -by making zero or more new local macros visible. +.code shuffle +function pseudo-randomly rearranges the elements of +.metn sequence . +This is performed in place: +.meta sequence +object is modified. + +The return value is +.meta sequence +itself. + +The rearrangement depends on pseudo-random numbers obtained from the +.code rand +function. +.coNP Function @ sort-group +.synb +.mets (sort-group < sequence >> [ keyfun <> [ lessfun ]]) +.syne +.desc The -.code macrolet -symbol is followed by a list of macro definitions. -Each definition is a form which begins with a -.metn name , -followed by -.meta macro-style-params -which is a macro parameter list, and zero or more -.metn macro-body-form -s. -These macro definitions are similar -to those globally defined by the -.code defmacro -operator, except that they -are in a local environment. +.code sort-group +function sorts +.meta sequence +according to the +.meta keyfun +and +.meta lessfun +arguments, and then breaks the resulting sequence into groups, +based on the equivalence of the elements under +.metn keyfun . -The macro definitions are followed by optional -.metn body-forms . -The macros specified in the definitions are visible to these -forms. +The following equivalence holds: -Forms inside the macro definitions such as the -.metn macro-body-form -s, -and initializer forms appearing in the -.meta macro-style-params -are subject -to macro-expansion in a scope in which none of the new macros being -defined are yet visible. Once the macro definitions are themselves -macro-expanded, they are placed into a new macro environment, which -is then used for macro expanding the -.metn body-form -s. +.verb + (sort-group sq lf kf) -A -.code macrolet -form is fully processed in the expansion phase of a form, and is -effectively replaced by -.code progn -form which contains expanded versions of -.metn body-form -s. -This expanded structure shows no evidence that any -macrolet forms ever existed in it. Therefore, it is impossible for the code -evaluated in the bodies and parameter lists of -.code macrolet -macros to have any visibility to any surrounding lexical variable bindings, -which are only instantiated in the evaluation phase, after expansion is done -and macros no longer exist. + <--> -A local macro defined using -.code defmacro -may decline to expand a macro form. Declining to expand is achieved by returning the original -unexpanded form, which may be captured using the -.code :form -parameter. When a local macro declines to expand a form, the macro definition -is temporarily hidden, as if it didn't exist in the lexical scope. If another -macro of the same name is thereby revealed (a global macro or another local macro -at a shallower nesting level), then an expansion is tried with that macro. If -no such macro is revealed, or if a lexical function binding of that name is -revealed, then no expansion takes place; the original form is taken as-is. -When another macro is tried, the process repeats, resulting in a search which -proceeds as far as possible through outer lexical scopes and finally the -global scope. + (partition-by kf (sort (copy sq) kf lf)) +.brev -.coNP Function @ macro-form-p +Note the reversed order of +.meta keyfun +and +.meta lessfun +arguments between +.code sort +and +.codn sort-group . + +.coNP Function @ uniq .synb -.mets (macro-form-p < obj <> [ env ]) +.mets (uniq << sequence ) .syne .desc The -.code macro-form-p -function returns -.code t -if -.meta obj -represents the syntax of -a form which is a macro form: either a compound macro or a symbol macro. -Otherwise it returns -.codn nil . +.code uniq +function returns a sequence of the same kind as +.metn sequence , +but with +duplicates removed. Elements of +.meta sequence +are considered equal under +the +.code equal +function. The first occurrence of each element is retained, +and the subsequent duplicates of that element, of any, are suppressed, +such that the order of the elements is otherwise preserved. -A macro form is one that will transform under -.code macroexpand-1 -or -.codn macroexpand ; -an object which isn't a macro form will not undergo expansion. +The +.code uniq +function is an alias for the one-argument case of +.codn unique . +That is to say, this equivalence holds: -The optional -.meta env -parameter is a macroexpansion environment. -A macroexpansion environment is passed down to macros and can be received -via their special -.code :env -parameter. +.verb + (uniq s) <--> (unique s) +.brev -.meta env -is used by -.code macro-form-p -to determine whether -.meta obj -is a macro in a lexical macro environment. +.coNP Function @ unique +.synb +.mets (unique < sequence >> [ keyfun <> { hash-arg }* ]) +.syne +.desc +The +.code unique +function is a generalization of +.codn uniq . +It returns a sequence of the same kind as +.metn sequence , +but with duplicates removed. + +If neither +.meta keyfun +nor +.metn hash-arg -s +are specified, then elements of sequence are considered equal under the +.code eql +function. The first occurrence of each element is retained, +and the subsequent duplicates of that element, of any, are suppressed, +such that the order of the elements is otherwise preserved. If -.meta env -is not specified or is -.codn nil , -then -.code macro-form-p -only recognizes global macros. +.meta keyfun +is specified, then that function is applied to each element, +and the resulting values are compared for equality. +In other words, the behavior is as if +.meta keyfun +were the +.code identity +function. -.TP* Example: +If one or more +.metn hash-arg -s +are present, these specify the arguments for the construction of +the internal hash table used by +.codn unique . +The arguments are like those of the +.code hash +function. -.verb - ;; macro which translates to 'yes if its - ;; argument is a macro from, or otherwise - ;; transforms to the form 'no. +.coNP Function @ tuples +.synb +.mets (tuples < length < sequence <> [ fill-value ]) +.syne +.desc +The +.code tuples +function produces a lazy list which represents a reorganization +of the elements of +.meta sequence +into tuples of +.metn length , +where +.meta length +must be a positive integer. - (defmacro ismacro (:env menv form) - (if (macro-form-p form menv) - ''yes ''no)) +The length of the sequence might not be evenly divisible by the tuple length. +In this case, if a +.meta fill-value +argument is specified, then the last tuple +is padded with enough repetitions of +.meta fill-value +to make it have +.meta length +elements. If +.meta fill-value +is not specified, then the last tuple is left +shorter than +.metn length . - (macrolet ((local ())) - (ismacro (local))) ;; yields yes +The output of the function is a list, but the tuples themselves are sequences +of the same kind as +.metn sequence . +If +.meta sequence +is any kind of list, they +are lists, and not lazy lists. - (ismacro (local)) ;; yields no +.TP* Examples: - (ismacro (ismacro foo)) ;; yields yes +.verb + (tuples 3 #(1 2 3 4 5 6 7 8) 0) -> (#(1 2 3) #(4 5 6) + #(7 8 0)) + (tuples 3 "abc") -> ("abc") + (tuples 3 "abcd") -> ("abc" "d") + (tuples 3 "abcd" #\ez) -> ("abc" "dzz") + (tuples 3 (list 1 2) #\ez) -> ((1 2 #\ez)) .brev -During macro expansion, the global macro -.code ismacro -is handed the macro-expansion environment -via -.codn ":env menv" . - -When the macro is invoked within the macrolet, -this environment includes the macro-time lexical scope in which the -.code local -macro is defined. So when global checks whether the argument form -.code (local) -is a macro, the conclusion is yes: the (local) form is a macro -call in that environment: -.code macro-form-p -yields -.codn t . - -When -.code "(global (local))" -is invoked outside of the macrolet, no local macro is visible is -there, and so -.code macro-form-p -yields -.codn nil . - -.coNP Functions @ macroexpand-1 and @ macroexpand +.coNP Function @ partition-by .synb -.mets (macroexpand-1 < obj <> [ env ]) -.mets (macroexpand < obj <> [ env ]) +.mets (partition-by < function << sequence ) .syne .desc If -.meta obj -is a macro form (an object for which -.code macro-form-p -returns -.codn t ), -these functions expand the macro form and return the expanded form. -Otherwise, they return -.metn obj . - -.code macroexpand-1 -performs a single expansion, expanding just the macro -that is referenced by the symbol in the first position of -.metn obj , -and returns the expansion. That expansion may itself be a macro form. +.meta sequence +is empty, then +.code partition-by +returns an empty list, +and +.meta function +is never called. -.code macroexpand -performs an expansion similar to -.codn macroexpand-1 . -If the result is -a macro form, then it expands that form, and keeps repeating this process -until the expansion yields a non-macro-form. That non-macro-form is then -returned. +Otherwise, +.code partition-by +returns a lazy list of partitions of the sequence +.metn sequence . +Partitions are consecutive, non-empty sub-strings of +.metn sequence , +of the same kind as +.metn sequence . -The optional -.meta env -parameter is a macroexpansion environment. -A macroexpansion environment is passed down to macros and can be received -via their special -.code :env -parameter. The environment they receive is their -lexically apparent macro-time environment in which local macros may be -visible. A macro can use this environment to "manually" expand some -form in the context of that environment. +The partitioning begins with the first element of +.meta sequence +being placed into a partition. -.TP* Example: +The subsequent partitioning is done according to +.metn function , +which is applied +to each element of +.metn sequence . +Whenever, for the next element, the function +returns the same value as it returned for the previous element, the +element is placed into the same partition. Otherwise, the next element +is placed into, and begins, a new partition. -.verb - ;; (foo x) expands x, and if x begins with a number, - ;; it removes the number and returns the resulting - ;; form. Otherwise, it returns the entire form. +The return values of the calls to +.meta function +are compared using the +.code equal +function. - (defmacro rem-num (:env menv some-form) - (let ((expanded (macroexpand some-form menv))) - (if (numberp (car expanded)) - (cdr expanded) - some-form))) +.TP* Examples: - ;; The following yields (42 a). +.verb + [partition-by identity '(1 2 3 3 4 4 4 5)] -> ((1) (2) (3 3) + (4 4 4) (5)) - (macrolet ((foo () '(1 list 42)) - (bar () '(list 'a))) - (list (rem-num (foo)) (rem-num (bar))))) + (partition-by (op = 3) #(1 2 3 4 5 6 7)) -> (#(1 2) #(3) + #(4 5 6 7)) .brev -The -.code rem-num -macro is able to expand the -.code (foo) -and -.code (bar) -forms it receives as -the -.code some-form -argument, even though these forms use local macro that are only -visible in their local scope. This is thanks to the macro -environment passed to -.codn rem-num . -It is correctly able to work with the -expansions -.code "(1 list 42)" -and -.code "(list 'a)" -to produce -.code "(list 42)" -and -.code "(list 'a)" -which evaluate to -.code 42 -and -.code a -respectively. - -.coNP Functions @ macroexpand-1-lisp1 and @ macroexpand-lisp1 +.coNP Function @ make-like .synb -.mets (macroexpand-1-lisp1 < obj <> [ env ]) -.mets (macroexpand-lisp1 < obj <> [ env ]) +.mets (make-like < list << ref-sequence ) .syne .desc The -.code macroexpand-1-lisp1 -and -.code macroexpand-lisp1 -functions closely resemble, respectively, -.code macroexpand-1 -and -.codn macroexpand . +.meta list +argument must be a list. If +.meta ref-sequence +is a sequence type, +then +.meta list +is converted to the same type of sequence and returned. +Otherwise the original +.meta list +is returned. -The argument and return value syntax and semantics is almost -identical, except for one difference. These functions consider argument -.meta obj -to be syntax in a Lisp-1 evaluation context, such as any argument -position of the -.code dwim -operator, or the equivalent DWIM Brackets notation. +Conversion is supported to string and vector type. -This makes a difference because in a Lisp-1 evaluation context, an -inner function binding is able to shadow an outer symbol macro binding -of the same name. +Conversion to a structure type is possible for structures. If +.meta ref-sequence +is an object of a structure type which has a static function +.codn from-list , +then +.code make-like +calls that function, passing to it, and the resulting value is returned. +.meta list +and returns whatever value that function returns. -The requirements about this language area are given in more -detail in the description of the -.code dwim -operator. +If +.meta ref-sequence +is a +.codn carray , +then +.meta list +is passed to the +.code carray-list +function, and the resulting value is returned. The second argument in the +.code carray-list +call is the element type taken from +.metn ref-sequence . +The third argument is +.codn nil , +indicating that the resulting +.code carray +is not to be null terminated. Note: the -.code macroexpand-lisp1 -function is useful to the implementor of a macro whose semantics requires -one or more argument forms to be treated in a Lisp-1 context, in situations -when such a macro needs to itself expand the material, rather than merely -insert it as-is into the output code template. +.code make-like +function is a helper which supports the development of +unoptimized versions of a generic function that accepts any type of +sequence as input, and produces a sequence of the same type as output. +The implementation of such a function can internally accumulate a list, and +then convert the resulting list to the same type as an input value +by using +.codn make-like . -.coNP Functions @ expand and @ *expand +.coNP Function @ nullify .synb -.mets (expand < form <> [ env ]) -.mets (expand* < form <> [ env ]) +.mets (nullify << sequence ) .syne .desc -The functions -.code expand -and -.code expand* -both perform a complete expansion of -.meta form -in the macro-environment -.metn env , -and return that expansion. - -If -.meta env -is omitted, the expansion takes place in the global environment in -which only global macros are visible. +The +.code nullify +function returns +.code nil +if +.meta sequence +is an empty sequence. +Otherwise it returns +.meta sequence +itself. -The returned object is a structure that -is devoid of any macro calls. Also, all -.code macrolet +Note: the +.code nullify +function is a helper to support unoptimized generic +programming over sequences. Thanks to the generic behavior of +.codn cdr , +any sequence can be traversed using +.code cdr +functions, checking for the +.code nil +value as a terminator. This, however, breaks for empty sequences which are not +lists, because they are not equal to +.codn nil : +to +.code car and -.code symacrolet -blocks in form -.meta form -are removed in the returned structure, replaced by their fully -expanded bodies. +.code cdr +they look like +a one-element sequence containing +.codn nil . +The +.code nullify +function reduces all empty +sequences to +.codn nil , +thereby correcting the behavior of code which traverses +sequences using +.codn cdr , +and tests for termination with +.codn nil . -The difference between -.code expand -and -.code expand* -is that -.code expand -suppresses any warning exceptions that are issued during expansion. +.SS* Open Sequence Traversal -.coNP Function @ expand-with-free-refs +Functions in this category perform efficient traversal of sequences +of various kinds. + +.coNP Function @ seq-begin .synb -.mets (expand-with-free-refs < form >> [ inner-env <> [ outer-env ]]) +.mets (seq-begin << object ) .syne .desc The -.code expand-with-free-refs -form performs a full expansion of -.metn form , -as if by the -.code expand -function and returns a list containing that expansion, plus four additional -items which provide information about variable and function references which -occur in -.metn form . - -If both -.meta inner-env -and -.meta outer-env -are provided, then it is expected that -.meta inner-env -is lexically nested within -.metn outer-env . +.code seq-begin +function returns an iterator object specialized to the task of traversing +the sequence represented by the input +.metn object . -Note: it is not required that -.meta outer-env -be the immediate parent of -.metn inner-env . +If +.meta object +isn't a sequence, an exception is thrown. -Note: a common usage situation is that -.meta outer-env -is the environment of the invocation of a "parent" macro which generates a form -that contains local macros. The bodies of those local macros use -.codn expand-with-free-refs , -specifying their own environment as -.meta inner-env -and that of their generating "parent" as -.metn outer-env . - -In detail, the five items of the returned list are -.mono -.meti >> ( expansion < fv-inner < ff-inner < fv-outer << ff-outer ) -.onom -whose descriptions are: -.RS -.meIP < expansion -The full expansion of -.metn form , -containing no macro invocations, or -.code symacrolet -or -.code macrolet -forms. -.meIP < fv-inner -A list of the free variables which occur in -.meta form -relative to the -.meta inner-env -environment. That is to say, variables that are not bound inside -.meta form -and are not also bound in -.metn inner-env . -If -.meta inner-env -is omitted, then these are the absolutely free variables -occurring in -.metn form . -.meIP < ff-inner -Exactly like -.meta fv-inner -but informing about function bindings rather than variables. -.meIP < fv-outer -A list of the variables which which occur in -.meta form -which would be free if the environments between -.meta inner-env -and -.meta outer-env -(including the former, excluding the latter) -were removed from consideration. A more detailed description of this semantics -is given below. If -.meta outer-env -is omitted, then these are the absolutely free variables -occurring in -.metn form , -ignoring the -.metn inner-env . -.meIP < ff-outer -Exactly like -.meta fv-outer -but informing about function bindings rather than variables. -.RE +.coNP Function @ seq-next +.synb +.mets (seq-next < iter << end-value ) +.syne +.desc +The +.code seq-next +function retrieves the next available item from the sequence iterated by +.metn iter , +which must be an object returned by +.codn seq-begin . -.IP -The semantics of the treatment of -.meta inner-env -and -.meta outer-env -in the calculation of -.meta fv-outer -and -.meta ff-outer -is as follows. A new environment -.meta diff-env -is calculated from these two environments, and -.meta form -is expanded in this environment. Variables and functions occurring in -.meta form -which are not bound in -.meta diff-env -are listed as -.meta fv-outer -and -.metn ff-outer . +If the sequence has no more items to be traversed, then +.meta end-value +is returned instead. -This -.meta diff-env -is calculated as follows. First -.meta diff-env -is initialized as a copy of -.metn outer-env . -Then, all environments below -.meta outer-env -down to -.meta inner-env -are examined for bindings which shadow bindings in -.metn diff-env . -Those shadows are removed from -.metn diff-env . -Therefore, what remains in -.meta diff-env -are those bindings from -.meta outer-env -that are -.I not -shadowed by the environments between -.meta inner-env -and -.metn outer-env . +Note: to avoid ambiguities, the application should provide an +.meta end-value +which is guaranteed distinct from any item in the sequence, such as a +freshly allocated object. -.TP* Example: +.coNP Function @ seq-reset +.synb +.mets (seq-reset < iter << object ) +.syne +.desc +The +.code seq-reset +re-initializes the existing iterator object +.meta iter +to begin a new traversal over the given +.metn object , +which must be a value of a kind that would be a suitable argument for +.codn seq-begin . -Suppose that -.code mac -is a macro which somehow has access to the two indicated lexical environments -in the following code snippet: +The +.code seq-reset +function returns +.metn iter . -.verb - (let (a c) ;; <- outer-env - (let (b) - (let (c) ;; <- inner-env - (mac (list a b c d)))) -.brev +.SS* Procedural List Construction -Suppose that -.code mac -invokes the -.code expand-with-free-refs -function, passing in the -.code "(list a b c d)" -argument form as -.code form -and two macro-time environment objects corresponding to the indicated -environments. +\*(TL provides an a structure type called +.code list-builder +which encapsulates state and methods for constructing lists procedurally. +Among the advantages of using +.code list-builder +is that lists can be constructed in the left to right direction without +requiring multiple traversals or reversal. For example, +.code list-builder +naturally combines with iteration or recursion: items visited in an +iterative or recursive process can be collected easily using +.code list-builder +in the order they are visited. -The return value of -.code expand-with-free-refs -shall be: +The basic workflow begins with the instantiation of a +.code list-builder +object. This object may be initialized with a piece of list material which +begins the to-be-constructed list, or it may be initialized to begin with an +empty list. Methods such as +.code add +and +.code pend +are invoked on this object to extend the list with new elements. At any point, +the list constructed so far is available using the +.code get +method, which is also how the final version of the list is eventually +retrieved. -.verb - ((list a b c d) (a d) (list) (b c d) (list)) -.brev +The +.code build +macro is provided which syntactically streamlines the process. +It implicitly creates a +.code list-builder +instance and binds it to a hidden lexical variable. +It then evaluates forms in a lexical scope in which +short-hand macros are available for building the list. +.coNP Structure @ list-builder +.synb +.mets (defstruct list-builder nil +.mets \ \ head tail) +.syne +.desc The -.meta fv-inner -list is -.code "(d)" -because this is the only variable that occurs in -.code "(list a b c d)" -which is free with regard to -.metn inner-env . +.code list-builder +structure encapsulates the state for a list building process. +Programs should use the +.code build-list +function for creating an instance of +.codn list-builder . The -.codn a , -.code b +.code head and -.code c -variables are not listed because they appear bound inside -.metn inner-env . +.code tail +slots should be regarded as internal variables. -The reported -.meta fv-outer -list is -.code "(b c d)" -because the form is considered against -.meta diff-env -which is formed by removing the shadowing bindings from -.metn outer-env . -The difference between -.code "(a c)" -and -.code "(b c)" -is -.code a -and so the form is considered in an environment containing the binding -.code a -which leaves -.code "(b c d)" -free. +.coNP Function @ build-list +.synb +.mets (build-list <> [ initial-list ]) +.syne +.desc +The +.code build-list +function instantiates and returns an object of struct type +.codn list-builder . -Note: this information is useful because a set difference can be calculated -between the two reported sets. The set difference between the -.meta fv-outer -variables -.code "(b c d)" -and the -.meta fv-inner -variables -.code "(d)" -is -.codn "(b c)" . +If no +.meta initial-list +argument is supplied, then the object is implicitly +with an empty list. -That set difference -.code "(b c)" -is significant because it precisely informs about the -.I bound -variables which occur in -.code "(list a b c d)" -which appear bound in -.metn inner-env , -but are not bound due to a binding coming from -.metn outer-env . +If the argument is supplied, then it is equivalent +to calling +.code build-list +without an argument to produce an object +.meta obj +the invoking the method call +.mono +.meti << obj .(ncon << list ) +.onom +on this object. The object produced by the expression +.meta list +is installed (without being copied) into the +object as the prefix of the list to be constructed. -The variable -.code d -is not listed in -.code "(b c)" -because it is not a bound variable. -The variable -.code a -is not in -.code "(b c)" -because though it is bound in -.metn inner-env , -that binding comes from -.metn outer-env . +.TP* Example: -The upshot of this logic is that it allows a macro to inspect a form in order -to discover the identities of the variables and functions which are used inside -that form, whose definitions come from a specific, bounded scope surrounding -that form. +.verb + ;; build the list (a b) trivially -.coNP Functions @ lexical-var-p and @ lexical-fun-p + (let ((lb (build-list '(a b)))) + lb.(get) + -> (a b) +.brev + +.coNP Methods @ add and @ add* .synb -.mets (lexical-var-p < env << form ) -.mets (lexical-fun-p < env << form ) +.mets << list-builder .(add << element *) +.mets << list-builder .(add* << element *) .syne .desc -These two functions are useful to macro writers. They are intended -to be called from the bodies of macro expanders, such as the bodies of -.code defmacro -or -.code macrolet -forms. The -.meta env -argument is a macro-time environment, which is available to macros -via the special -.code :env -parameter. Using these functions, a macro can enquire whether -a given -.meta form -is a symbol which has a variable binding or a function binding -in the lexical environment. -This information is known during macro expansion. The macro expander -recognizes lexical function and variable bindings, because these -bindings can shadow macros. +The +.code add +and +.code add* +methods extend the list being constructed by a +.code list-builder +object by adding individual +elements to it. The +.code add +method adds elements at the tail of the list, +whereas +.code add* +adds elements at the front. + +The return value of these methods is unspecified. .TP* Example: .verb - ;; - ;; this macro replaces itself with :lexical-var if its - ;; argument is a lexical variable, :lexical-fun if - ;; its argument is a lexical function, or with - ;; :not-lex-fun-var if neither is the case. - ;; - (defmacro classify (sym :env e) - (cond - ((lexical-var-p e sym) :lexical-var) - ((lexical-fun-p e sym) :lexical-fun) - (t :not-lex-fun-var))) + ;; Build the list (1 2 3 4) - ;; - ;; This returns: - ;; - ;; (:lexical-var :not-lex-fun-var :lexical-fun) - ;; - (let ((x 1) (y 2)) - (symacrolet ((y x)) - (flet ((f () (+ 2 2))) - (list (classify x) (classify y) (classify z))))) + (let ((lb (build-list))) + lb.(add 3 4) + lb.(add* 1 2) + lb.(get)) + -> (1 2 3 4) .brev -.TP* Note: - -These functions do not call -.code macroexpand -on the form. In most cases, it is necessary for the macro writers -to do so. Not that in the above example, symbol -.code y -is classified as neither a lexical function nor variable. -However, it can be macro-expanded to -.code x -which is a lexical variable. - -.coNP Function @ lexical-lisp1-binding +.coNP Methods @ pend and @ pend* .synb -.mets (lexical-lisp1-binding < env << symbol ) +.mets << list-builder .(pend << list *) +.mets << list-builder .(pend* << list *) .syne .desc The -.code lexical-lisp1-binding -function inspects the macro-time environment -.meta env -to determine what kind of binding, if any, does -.meta symbol -have in that environment, from a Lisp-1 perspective. +.code pend +and +.code pend* +methods extend the list being constructed by a +.code list-builder +object by adding lists to it. The +.code pend +method catenates the +.code list +arguments together as if by the +.code append +function, then appends the resulting list to +the end of the list being constructed. +The +.code pend* +method is similar, except it prepends the +catenated lists to the front of the list +being constructed. -That is to say, it considers function bindings, variable bindings -and symbol macro bindings to be in a single name space and finds -the innermost binding of one of these types for -.metn symbol . +Both methods have the property that the +constructed list does not share structure +with the input lists. -If such a binding is found, then the function returns one of -the three keyword symbols -.codn :var , -.codn :fun , -or -.codn :symacro . +The return value of these methods is unspecified. -If no such lexical binding is found, then the function -returns -.codn nil . +.TP* Example: -Note that a -.code nil -return doesn't mean that the symbol doesn't have a lexical binding. It could -have an operator macro lexical binding (a macro binding in the function -namespace established by -.codn macrolet ). +.verb + ;; Build the list (1 2 3 4) -.coNP Operator @ defsymacro + (let ((lb (build-list))) + lb.(pend '(3 4)) + lb.(pend* '(1 2)) + lb.(get)) + -> (1 2 3 4) +.brev + +.coNP Methods @ ncon and @ ncon* .synb -.mets (defsymacro < sym << form ) +.mets << list-builder .(ncon << list *) +.mets << list-builder .(ncon* << list *) .syne .desc -A -.code defsymacro -form introduces a symbol macro. A symbol macro consists of a binding -between a symbol -.meta sym -and and a -.metn form . -The binding denotes the form itself, rather than its value. How the -symbol macro works is that if -.meta sym -occurs as a form in a scope where the symbol macro definition is -in scope, -.meta sym -is replaced by -.metn form . -Immediately after this replacement takes place, -.meta form -itself is then processed for further replacement of macros and -symbol macros. - -Symbol macros are also recognized in contexts -where -.meta sym -denotes a place which is the target of an assignment operation -like -.code set -and similar. +The +.code ncon +and +.code ncon* +methods extend the list being constructed by a +.code list-builder +object by adding lists to it. The +.code ncon +method destructively catenates the +.meta list +arguments as if by the +.code nconc +function. The resulting list is appended +to the list being constructed. +The +.code ncon* +method is similar, except it prepends +the catenated lists to the front of the +list being constructed. -A -.code defsymacro -form is implicitly executed at expansion time, and thus need -not be wrapped in a -.code macro-time -form, just like -.codn defmacro . +These methods destructively manipulate +the input lists. Moreover, they cause the +list being constructed to share substructure +with the input lists. -Note: if a symbol macro expands to itself directly, expansion stops. However, -if a symbol macro expands to itself through a chain of expansions, -runaway expansion-time recursion will occur. +Additionally, the +.code ncon* +function can be called with a single argument +which is an atom. This atom will simply be +installed as the terminating atom of the +list being constructed. -If a global variable exists by the name -.metn sym , -then -.code defsymacro -first removes that variable from the global environment, and if that -variable is special, the symbol's special marking is removed. -.code defsymacro -doesn't alter the dynamic binding of a special variable. Any such -a binding remains intact. -If -.code defsymacro -is evaluated in a scope in which there is any lexical or dynamic binding -of -.meta sym -in the variable namespace, whether as a variable or macro, -the global symbol macro is shadowed by that binding. +The return value of these methods is unspecified. -.coNP Operator @ symacrolet +.TP* Example: + +.verb + ;; Build the list (1 2 3 4 . 5) + + (let ((lb (build-list))) + lb.(ncon* (list 1 2)) + lb.(ncon (list 3 4)) + lb.(ncon 5) + lb.(get)) + -> (1 2 3 4 . 5) +.brev + +.coNP Method @ get .synb -.mets (symacrolet >> ({( sym << form )}*) << body-form *) +.mets << list-builder .(get) .syne .desc The -.code symacrolet -operator binds local, lexically scoped macros that are -similar to the global symbol macros introduced by -.codn defsymacro . +.code get +method retrieves the list constructed so far by a +.code list-builder +object. It doesn't change the state of the object. +The retrieved list may be passed as an argument +into the construction methods on the same object. -Each -.meta sym -in the bindings list is bound to its corresponding form, creating a -new extension of the expansion-time lexical macro environment. +.TP* Examples: -Each -.meta body-form -is subsequently macro-expanded in this new environment -in which the new symbol macros are visible. +.verb + ;; Build the circular list (1 1 1 1 ...) + ;; by appending (1) to itself destructively: -Note: ordinary lexical bindings such as those introduced by let or by -function parameters lists shadow symbol macros. If a symbol -.code x -is bound by nested instances of -.code macrolet -and a -.codn let , -then the scope enclosed by both -constructs will see whichever of the two bindings is more inner, -even though the bindings are active in completely separate phases of -processing. + (let ((lb (build-list '(1)))) + lb.(ncon* lb.(get)) + lb.(get)) + -> (1 1 1 1 ...) -From the perspective of the arguments of a -.code dwim -form, lexical function bindings also shadow symbol macros. -This is consistent with the Lisp-1-style name resolution which -applies inside a -.code dwim -form. Of course, lexical operator macros do not shadow -symbol macros under any circumstances. + ;; build the list (1 2 1 2 1 2 1 2) + ;; by doubling (1 2) twice: -.coNP Macros @ placelet and @ placelet* + (let ((lb (build-list))) + lb.(add 1 2) + lb.(pend lb.(get)) + lb.(pend lb.(get)) + lb.(get)) + -> (1 2 1 2 1 2 1 2) +.brev + +.coNP Macro @ build .synb -.mets (placelet >> ({( sym << place )}*) << body-form *) -.mets (placelet* >> ({( sym << place )}*) << body-form *) +.mets (build << form *) .syne .desc The -.code placelet -macro binds lexically scoped symbol macros in such -a way that they behave as aliases for places -denoted by place forms. +.code build +macro provides a shorthand notation for constructing lists using the +.code list-builder +structure. It eliminates the explicit call to the +.code build-list +function to construct the object, and eliminates the explicit +references to the object. -Each -.meta place -must be an expression denoting a syntactic place. The -corresponding -.meta sym -is established as an alias for the storage location which that place denotes, -over the scope of the -.metn body-form -s. +Instead, +.code build +creates a lexical environment in which a +.code list-builder +object is implicitly constructed and bound to a hidden variable. +Local functions which mimic the +.code list-builder +methods operate implicitly on this hidden variable, so that +the object need not be mentioned as an argument. -This binding takes place in such a way that each -.meta place -is evaluated exactly once, only in order to determine its -storage location. The corresponding -.meta sym -then serves as an alias for that location, over the -scope of the -.metn body-form -s. -This means that whenever -.meta sym -is evaluated, it stands for the value of the storage -location, and whenever a value is apparently stored into -.metn sym , -it is actually the storage location which receives it. +With the exception of +.codn get , +these local functions +.codn get , +have unspecified return values, like the same-named +.code list-builder +methods. -The -.code placelet* -variant implements an alternative scoping rule, which allows a later -.meta place -form to refer to a -.meta sym -bound to an earlier -.meta place -form. In other words, a given -.meta sym -binding is visible not only to the -.metn body-form -s -but also to -.meta place -forms which occur later. +.TP* Examples: -Note: certain kinds of places, notably -.mono -.meti (force << promise ) -.onom -expressions, must be accessed before they can be stored, -and this restriction continues to hold when those -places are accessed through -.code placelet -aliases. +.verb + ;; Build the circular list (1 1 1 1 ...) + ;; by appending (1) to itself destructively: -Note: -.code placelet -differs from -.code symacrolet -in that the forms themselves are not aliased, but the storage -locations which they denote. -.code "(symacrolet ((x y)) z)" -performs the syntactic substitution of symbol -.code x -by form -.codn y , -wherever -.code x -appears inside -.code z -as an evaluated form, and is not shadowed by any inner binding. -Whereas -.code "(placelet ((x y)) z)" -generates code which arranges for -.code y -to be evaluated to a storage location, and syntactically replaces occurrences -of -.code x -with a form which directly denotes that storage location, -wherever -.code x -appears inside -.code z -as an evaluated form, and is not shadowed by any inner binding. -Also, -.code x -is not necessarily substituted by a single, fixed form, -as in the case of -.codn symacrolet . -Rather it may be substituted by one kind of form when it -is treated as a pure value, and another kind of form -when it is treated as a place. + (build + (add 1) + (ncon* (get))) -> (1 1 1 1 ...) -.TP* "Example:" + ;; build the list (1 2 1 2 1 2 1 2) + ;; by doubling (1 2) twice: -Implementation of -.code inc -using -.codn placelet : + (build + (add 1 2) + (pend (get)) + (pend (get))) -> (1 2 1 2 1 2 1 2) -.verb - (defmacro inc (place : (delta 1)) - (with-gensyms (p) - ^(placelet ((,p ,place)) - (set ,p (+ ,p ,delta))))) + ;; build a list by mapping over the local + ;; add function: + + (build [mapdo add (range 1 3)]) -> (1 2 3) .brev -The gensym -.code p -is used to avoid accidental capture of references -emanating from the -.code delta -form. +.SS* Permutations and Combinations -.coNP Macro @ equot +.coNP Function @ perm .synb -.mets (equot << form ) +.mets (perm < seq <> [ len ]) .syne .desc The -.code equot -macro ("expand and quote") performs a full expansion of -.code form -in the surrounding macro environment. Then it constructs a -.code quote -form whose argument is the expansion. This quote form is -then returned as the macro replacement for the original -.code equot -form. +.code rperm +function returns a lazy list which consists of all +length +.meta len +permutations of formed by items taken from +.metn seq . +The permutations do not use any element of +.meta seq +more than once. -.TP* Example: +Argument +.metn len , +if present, must be a positive integer, and +.meta seq +must be a sequence. -.verb - (symacrolet ((a (+ 2 2))) - (list (quote a) (equot a) a)) - --> (a (+ 2 2) 4) -.brev +If +.meta len +is not present, then its value defaults to the length of +.metn seq : +the list of the full permutations of the entire sequence is returned. -Above, the expansion of -.code a -is -.codn "(+ 2 2)" . -Thus the macro call -.code "(equot a)" -expands to -.codn "(quote (+ 2 2))" . -When that is evaluated, it yields -.codn "(+ 2 2)" . +The permutations in the returned list are sequences of the same kind as +.codn seq . If -.code a -is quoted, then of course the result is -.codn a : -no expansion or evaluation takes place. -Whereas if -.code a -is presented for evaluation, then not only is it expanded to -.codn "(+ 2 2)" , -but that expansion is reduced to 4. +.meta len +is zero, then a list containing one permutation is returned, and that +permutation is of zero length. -The -.code equot -operator is a mongrel of these two semantics: it permits expansion to proceed, -but then suppresses evaluation of the result. +If +.meta len +exceeds the length of +.metn seq , +then an empty list is returned, +since it is impossible to make a single non-repeating permutation that +requires more items than are available. -.coNP Operators @ tree-bind and @ mac-param-bind +The permutations are lexicographically ordered. + +.coNP Function @ rperm .synb -.mets (tree-bind < macro-style-params < expr << form *) -.mets (mac-param-bind < context-expr -.mets \ \ < macro-style-params < expr << form *) +.mets (rperm < seq << len ) .syne .desc The -.code tree-bind -operator evaluates -.codn expr , -and then uses the -resulting value as a counterpart to a macro-style parameter list. -If the value has a tree structure which matches the parameters, -then those parameters are established as bindings, and the -.metn form -s, -if any, are evaluated in the scope of those bindings. The value -of the last -.meta form -is returned. If there are no forms, -.code nil -is returned. +.code rperm +function returns a lazy list which consists of all the repeating +permutations of length +.meta len +formed by items taken from +.metn seq . +"Repeating" means that the items from +.meta seq +can appear more than +once in the permutations. -Note: this operator throws an exception if there is a -structural mismatch between the parameters and the value of -.codn expr . +The permutations which are returned are sequences of the same kind as +.metn seq . -One way to avoid this exception is to use -.codn tree-case . +Argument +.meta len +must be a nonnegative integer, and +.meta seq +must be a sequence. -The -.code mac-param-bind -operator is similar to -.code tree-bind -except that it takes an extra argument, -.metn context-expr . -This argument is an expression which is evaluated. It is expected to -evaluate to a compound form. If an error occurs during binding, the error -diagnostic message is based on information obtained from this form. -By contrast, the -.code tree-bind -operator's error diagnostic refers to the -.code tree-bind -form, which is cryptic if the binding is used for the implementation -of some other construct, hidden from the user of that construct. +If +.meta len +is zero, then a single permutation is returned, of zero length. +This is true regardless of whether +.meta seq +is itself empty. -.coNP Operator @ tree-case +If +.meta seq +is empty and +.meta len +is greater than zero, then no permutations are +returned, since permutations of a positive length require items, and the +sequence has no items. Thus there exist no such permutations. + +The first permutation consists of +.meta le +repetitions of the first element of +.metn seq . +The next repetition, if there is one, differs from the first +repetition in that its last element is the second element of +.metn seq . +That is to say, the permutations are lexicographically ordered. + +.TP* Examples: + +.verb + (rperm "01" 4) -> ("000" "001" "010" "011" + "100" "101" "110" "111") + + (rperm #(1) 3) -> (#(1 1 1)) + + (rperm '(0 1 2) 2) -> ((0 0) (0 1) (0 2) (1 0) + (1 1) (1 2) (2 0) (2 1) (2 2)) +.brev + +.coNP Function @ comb .synb -.mets (tree-case < expr >> {( macro-style-params << form *)}*) +.mets (comb < seq << len ) .syne .desc The -.code tree-case -operator evaluates -.meta expr -and matches it against a succession -of zero or more cases. Each case defines a pattern match, expressed as a macro -style parameter list -.metn macro-style-params . +.code comb +function returns a lazy list which consists of all +length +.meta len +non-repeating combinations formed by taking items taken from +.metn seq . +"Non-repeating combinations" means that the combinations do not use any +element of +.meta seq +more than once. If +.meta seq +contains no duplicates, then +the combinations contain no duplicates. -If the object produced by -.meta expr -matches -.metn macro-style-params , -then the parameters are bound, becoming local variables, and the -.metn form -s, -if any, are evaluated in order in the environment in which those variables are -visible. If there are forms, the value of the last -.meta form -becomes the result -value of the case, otherwise the result value of the case is nil. +Argument +.meta len +must be a nonnegative integer, and +.meta seq +must be a sequence or a hash table. -If the result value of a case is the object -.code : -(the colon symbol), then processing continues with the next case. Otherwise the -evaluation of -.code tree-case -terminates, returning the result value. +The combinations in the returned list are objects of the same kind as +.metn seq . -If the value of -.meta expr -does not match the -.meta macro-style-params -parameter list of a case, processing continues with the next case. +If +.meta len +is zero, then a list containing one combination is returned, and that +combination is of zero length. -If no cases match, then -.code tree-case -terminates, returning -.codn nil . +If +.meta len +exceeds the number of elements in +.metn seq , +then an empty list is returned, since it is impossible to make a single +non-repeating combination that requires more items than are available. -.TP* Example: +If +.meta seq +is a sequence, the returned combinations are lexicographically ordered. +This requirement is not applicable when +.meta seq +is a hash table. +.TP* Example: .verb - ;; reverse function implemented using tree-case + ;; powerset function, in terms of comb. + ;; Yields a lazy list of all subsets of s, + ;; expressed as sequences of the same type as s. - (defun tb-reverse (obj) - (tree-case obj - (() ()) ;; the empty list is just returned - ((a) obj) ;; one-element list returned - ((a . b) ^(,*(tb-reverse b) ,a)) ;; car/cdr recursion - (a a))) ;; atom is just returned + (defun powerset (s) + (mappend* (op comb s) (range 0 (length s)))) .brev -Note that in this example, the atom case is placed last, because an -argument list which consists of a symbol is a "catch all" match -that matches any object. We know that it matches an atom, because -the previous -.code "(a . b)" -case matches conses. In general, the order of the cases in -.code tree-case -is important: even more so than the order of cases in a -.code cond -or -.codn caseql . -The one-element list case is unnecessary; it can be removed. - -.coNP Macro @ tb +.coNP Function @ rcomb .synb -.mets (tb < macro-style-params << form *) +.mets (rcomb < seq << len ) .syne .desc The -.code tb -macro is similar to the -.code lambda -operator but its argument binding is based on a macro-style parameter list. -The name is an abbreviation of -.codn tree-bind . +.code comb +function returns a lazy list which consists of all +length +.meta len +repeating combinations formed by taking items taken from +.metn seq . +"Repeating combinations" means that the combinations can use +an element of +.meta seq +more than once. -A -.code tb -form evaluates to a function which takes a variable number of -arguments. +Argument +.meta len +must be a nonnegative integer, and +.meta seq +must be a sequence. -When that function is called, those arguments are taken as a list object which -is matched against -.meta macro-style-params -as if by -.metn tree-bind . -If the match is successful, then the parameters are bound to the -corresponding elements from the argument structure and each successive -.meta form -is evaluated an environment in which those bindings are visible. -The value of the last -.meta form -is the return value of the function. If there are no forms, -the function's return value is -.codn nil . +The combinations in the returned list are sequences of the same kind as +.metn seq . -The following equivalence holds, where -.code args -should be understood to be a globally unique symbol: +If +.meta len +is zero, then a list containing one combination is returned, and that +combination is of zero length. This is true even if +.meta seq +is empty. -.verb - (tb pattern body ...) <--> (lambda (. args) - (tree-bind pattern args body ...)) -.brev +If +.meta seq +is empty, and +.meta len +is nonzero, then an empty list is returned. -.coNP Macro @ tc -.synb -.mets (tc >> {( macro-style-params << form *)}*) -.syne -.desc -The -.code tc -macro produces an anonymous function whose behavior is closely -based on the -.code tree-case -operator. Its name is an abbreviation of -.codn tree-case . +The combinations are lexicographically ordered. -The anonymous function takes a variable number of arguments. -Its argument list is taken to be the value macro is tested -against the multiple pattern clauses of an implicit -.codn tree-case . -The return value of the function is that of the implied -.codn tree-case . -The following equivalence holds, where -.code args -should be understood to be a globally unique symbol: +.SS* Macros +\*(TL supports structural macros. \*(TX's model of macroexpansion is that +\*(TL code is processed in two phases: the expansion phase and the +evaluation phase. The expansion phase is invoked on Lisp code early during the +processing of source code. For instance when a \*(TX file containing a +.code "@(do ...)" +directive +is loaded, expansion of the Lisp forms are its arguments takes place during the +parsing of the entire source file, and is complete before any of the code in +that file is executed. If the +.code "@(do ...)" +form is later executed, +the expanded forms are then evaluated. -.verb - (tc clause1 clause2 ...) <--> (lambda (. args) - (tree-case args - clause1 clause2 ...)) -.brev +\*(TL also supports symbol macros, which are symbolic forms that stand +for forms, with which they are replaced at macro expansion time. -.coNP Macro @ with-gensyms -.synb -.mets (with-gensyms <> ( sym *) << body-form *) -.syne -.desc -The -.code with-gensyms -evaluates the -.metn body-form -s -in an environment in which each variable name symbol -.meta sym -is bound to a new uninterned symbol ("gensym"). +When Lisp data is processed as code by the +.code eval +function, it is first expanded, +and so processed in its entirety in the expansion phase. Then it is processed +in the evaluation phase. -.TP* "Example:" +.NP* Macro parameter lists -The code: +\*(TX macros support destructuring, similarly to Common Lisp macros. +This means that macro parameter lists are like function argument lists, +but support nesting. A macro parameter list can specify a nested parameter +list in every place where an argument symbol may appear. For instance, +consider this macro parameter list: .verb - (let ((x (gensym)) - (y (gensym)) - (z (gensym))) - ^(,x ,y ,z)) + ((a (b c)) : (c frm) ((d e) frm2 de-p) . g) .brev -may be expressed more conveniently using the -.code with-gensyms -shorthand: +The top-level of this nested form has the structure -.verb - (with-gensyms (x y z) - ^(,x ,y ,z)) -.brev +.mono +.meti \ \ >> ( I : < J < K . << L ) +.onom -.SS* Parameter List Macros +in which we can identify the major constituent positions as +.metn I , +.metn J , +.meta K +and +.metn L . -Parameter list macros, also more briefly called -.I "parameter macros" -are an original feature of \*(TL. +The constituent at position +.meta I +is the mandatory parameter +.codn "(a (b c))" . +Position +.meta J +holds the optional parameter +.code c +(with default init form +.codn frm ). +At +.meta K +is found the optional parameter +.code "(d e)" +(with default init form +.code frm2 +and presence-indicating variable +.codn de-p ). +Finally, the parameter in the dot position +.meta L +is +.codn g , +which captures trailing arguments. -If the first element of a function or macro parameter list is a keyword -symbol other than +Obviously, some of the parameters are compound expressions rather +than symbols: +.code "(a (b c))" +and +.codn "(d e)" . +These compounds express nested macro parameter lists. + +Nested macro parameter lists recursively match the corresponding structure +in the argument object. For instance if a simple argument would capture +the structure +.code "(1 (2 3))" +then we can replace the argument with the nested argument list +.code "(a (b c))" +which destructures the +.code "(1 (2 3))" +such that the parameters +.codn a , +.code b +and +.code c +will end up bound +to +.codn 1 , +.code 2 +and +.codn 3 , +respectively. + +Nested macro parameter lists have all the features of the top-level +macro parameter lists: they can have optional arguments with default +values, use the dotted position, and contain the .codn :env , -.codn :whole , +.code :whole +and .code :form -or -.code : -(the colon symbol), -it denotes a parameter macro. This keyword symbol is expected to -have a binding in the parameter macro namespace: a global namespace -which associates keyword symbols with parameter list expander -functions. - -Expansion of a parameter list macro occurs at macro-expansion -time, when a function's parameter list is traversed by the -macro expander. It takes place as follows. -First, the keyword is removed from the parameter list. -The keyword's binding in the parameter macro namespace is -retrieved. If it doesn't exist, an exception is thrown. -Otherwise, the remaining parameter list is first recursively -processed for more occurrences of parameter macros. -This expansion produces a transformed parameter list, -along with a transformed function body. These two artifacts -are then passed to the transformer function retrieved from -the keyword symbol's binding. The function returns a -further transformed version of the parameter list and -body. These are processed for more parameter macros. -The process terminates when no more expansion is -possible, because a parameter list has been produced -which does not begin with a parameter macro. This -final parameter list and its accompanying body are then -taken in place of the original parameter list and -body. +special parameters, which are described below. +In nested parameter lists, the binding strictness is relaxed for optional +parameters. If +.code "(a (b c))" +is optional, and the argument is, say, +.codn (1) , +then +.code a +gets +.codn 1 , +and +.code b +and +.code c +receive +.codn nil . -\*(TL provides a built-in parameter list macro bound to the symbol -.code :key -which endows a function keyword parameters. The implementation is -written entirely using this parameter list macro mechanism, by means -of the -.code define-param-expander -macro. +Macro parameter lists also supports three special keywords, namely +.codn :env , +.code :whole +and +.codn :form . -.coNP Special variable @ *param-macro* -.desc -The variable -.code *param-macro* -holds a hash table which associates keyword symbols with -parameter list expander functions. +The parameter list +.code "(:whole x :env y :form z)" +will bind parameter +.code x +to the entire +macro parameter list, bind parameter +.code y +to the macro environment and bind parameter +.code z +to the entire macro form (the original compound form used to invoke the +macro). -The functions are expected to conform to the following -syntax: +The +.codn :env , +.code :whole +and +.code :form +notations can occur anywhere in a macro parameter list, other than +to the right of the consing dot. They can be used in nested +macro parameter lists also. Note that in a nested macro +parameter list, +.code :form +and +.code :env +do not change meaning: they bind the same object as they would in +the top-level of the macro parameter list. +However the +.code :whole +parameter inside has a restricted scope in a nested parameter +list: its parameter will capture just that part of the argument material which +matches that parameter list, rather than the entire argument list. -.mono -.mets (lambda >> ( params < body < env << form ) << form *) -.onom +The processing of macro parameter lists omits the feature that when the +keyword symbol +.code : +(colon) given as the argument to an optional parameter, that argument is +treated as a missing argument. This special logic is implemented only +in the function argument passing mechanism, not in the binding of macro +parameters to object structure. If the colon symbol appears in the object +structure and is matched against an optional parameter, it is an +ordinary value. That parameter is considered present, and takes on +that +.code : +keyword symbol as its value. -The -.meta params -parameter receives the parameter list of the function -which is undergoing parameter expansion. All other -parameter macros have already been expanded. +.TP* "Dialect Note:" -The -.meta body -parameter receives the list of body forms. -The function is expected to return a -.code cons -cell whose -.code car -contains the transformed parameter list, and whose -.code cdr -contains the transformed list of body forms. -Parameter expansion takes place at macro expansion time. +In ANSI Common Lisp, the lambda list keyword +.code &whole +binds its corresponding variable to the entire macro form, whereas +\*(TL's +.code :whole +binds its variable only to the arguments of the macro form. -The -.meta env -parameter receives the macro-expansion-time environment -which surrounds the function being expanded. -Note that this environment doesn't take into account the -parameters themselves; therefore, it is not the correct environment -for expanding macros among the -.meta body -forms. For that purpose, it must be extended with -shadowing entries, the manner of doing which is -undocumented. However -.meta env -may be used directly for expanding init forms -for optional parameters occurring in -.metn params . +Note, however, that ANSI CL distinguishes destructuring lambda lists +and macro lambda lists and the +.code &whole +parameter has a different behavior between the two. Under +.codn destructuring-bind , +the +.code &whole +parameter receives just the arguments, just like the behavior +of \*(TL's +.code :whole +parameter. -The -.meta form -parameter receives the overall function-defining -form that is being processes, such as a -.code defun -or -.code lambda -form. This is intended for error reporting. +\*(TL does not distinguish destructuring and macro lambda lists; +they are the same and behave the same way. Thus +.code :whole +is treated the same way in macros as in +.code tree-bind +and related binding operators: it binds just the arguments +to the parameter. \*(TL has the special parameter +.code :form +by means of which macros can access their invoking form. +This parameter is also supported in +.code tree-bind +and binds to the entire +.code tree-bind +form. -.coNP Macro @ define-param-expander +.coNP Operator @ macro-time .synb -.mets (define-param-expander < name >> ( pvar < bvar : < evar << fvar ) -.mets \ \ << form *) +.mets (macro-time << form *) .syne .desc The -.code define-param-expander -macro provides syntax for defining parameter macros. Invocations -of this macro expand to code which constructs an anonymous -function and installs it into the -.code *param-macro* -hash table, under the key given by -.metn name . - -The -.meta name -parameter's argument should be a keyword symbol that is valid for use -as a parameter macro name. - -The -.metn pvar , -.metn bvar , -.meta evar -and -.meta fvar -arguments must be symbols suitable for variable -binding. These symbols define the parameters of the -expander function which shall, respectively, receive -the parameter list, body forms, macro environment -and function form. If -.meta evar -is omitted, a symbol generated by the -.code gensym -function is used. Likewise if -.meta fvar -is omitted. - -The +.code macro-time +operator has a syntax similar to the +.code progn +operator. Each .meta form -arguments constitute the body of the expander. - -The -.code define-param-expander -form returns -.metn name . +is evaluated from left to right, and the resulting value is that of the last +form. -.TP* Example: +The special behavior of +.code macro-time +is that the evaluation takes place during +the expansion phase, rather than during the evaluation phase. -The following example shows the implementation -of a parameter macro -.code :memo -which provides rudimentary memoization. -Using the macro is extremely easy. It is a matter -of simply inserting the -.code :memo -keyword at the front of a function's parameter list. -The function is then memoized. +Also, +.code macro-time +macro-expands each +.meta form +and evaluates it before processing the next +.meta form +in the same way. Thus, for instance, if a +.meta form +introduces a global definition, that definition will be visible not +only during the evaluation of a subsequent +.metn form , +but also during its macro-expansion time. -.verb - (defvarl %memo% (hash :weak-keys)) +During the expansion phase, all +.code macro-time +expressions which occur in a context +that calls for evaluation are evaluated, and replaced by their quoted values. +For instance +.code "(macro-time (list 1 2 3))" +evaluates +.code "(list 1 2 3)" +to the object +.code "(1 2 3)" +and the entire +.code macro-time +form is replaced by that value, quoted: +.codn "'(1 2 3)" . +If the form is evaluated again at evaluation-time, the resulting value will be +that of the quote, in this case +.codn "(1 2 3)" . - (defun ensure-memo (sym) - (or (gethash %memo% sym) - (sethash %memo% sym (hash)))) +.code macro-time +forms do not see the surrounding lexical environment; the see only +global function and variable bindings and macros. - (define-param-expander :memo (param body) - (let* ((memo-parm [param 0..(posq : param)]) - (hash (gensym)) - (key (gensym))) - ^(,param (let ((,hash (ensure-memo ',hash)) - (,key (list ,*memo-parm))) - (or (gethash ,hash ,key) - (sethash ,hash ,key (progn ,*body))))))) -.brev - -The above -.code :memo -macro may be used to define a memoized Fibonacci function -as follows: - -.verb - (defun fib (:memo n) - (if (< n 2) - (clamp 0 1 n) - (+ (fib (pred n)) (fib (ppred n))))) -.brev +Note 1: +.code macro-time +is intended for defining helper functions and variables that +are used by macros. A macro cannot "see" a +.code defun +function or +.code defvar +variable +because +.code defun +and +.code defvar +forms are evaluated at evaluation time, which occurs +after expansion time. The macro-time operator hoists the evaluation of these +forms to macro-expansion time. -All that is required is the insertion of the -.code :memo -keyword. +Note 2: +.code defmacro +forms are implicitly in macro-time; they do not have to +be wrapped with the +.code macro-time +operator. The +.code macro-time +operator doesn't have +to be used in programs whose macros do not make references to variables +or functions. -.coNP Parameter list macro @ :key +.coNP Operator @ defmacro .synb -.mets (:key << param * -.mets \ \ [ -- >> { sym | >> ( sym >> [ init-form <> [ p-sym ]])}* ] -.mets \ \ [ . rest-param ]) +.mets (defmacro < name +.mets \ \ \ \ \ \ \ \ \ <> ( param * [: << opt-param * ] [. < rest-param ]) +.mets \ \ << body-form *) .syne .desc -Parameter list macro -.code :key -injects keyword parameter support into functions and macros. +The +.code defmacro +operator is evaluated at expansion time. It defines a +macro-expander function under the name +.metn name , +effectively creating a new operator. -When -.code :key -appears as the first item in a function parameter list, a special syntax is -recognized in the parameter list. After any required and optional parameters, -the symbol -.code -- -(two dashes) may appear. Parameters after this symbol are interpreted -as keyword parameters. After the keyword parameters, a rest parameter -may appear in the usual way as a symbol in the dotted position. +Note that the above syntax synopsis describes only the canonical +parameter syntax which remains after parameter list macros are +expanded. See the section Parameter List Macros. -Keyword parameters use the same syntax as optional parameters, except -that if used in a macro parameter list, they do not support -destructuring whereas optional parameters do. That is to say, regardless -whether -.code :key -is used in a function or macro, keyword parameters are symbols. +Note that the parameter list is a macro parameter list, and not a +function parameter list. This means that each +.meta param +and +.meta opt-param +can be not only a symbol, but it can itself be a parameter list. +The corresponding argument is then treated as a structure which +matches that parameter list. This nesting of parameter lists +can be carried to an arbitrary depth. -A keyword parameter takes three possible forms: +A macro is called like any other operator, and resembles a function. Unlike in +a function call, the macro receives the argument expressions themselves, rather +than their values. Therefore it operates on syntax rather than on values. +Also, unlike a function call, a macro call occurs in the expansion phase, +rather than the evaluation phase. -.RS -.meIP < sym -A keyword parameter may be specified as a simple symbol -.metn sym . -If the argument for such a keyword parameter is missing, -it takes on the value -.codn nil . -.meIP >> ( sym << init-form ) -If the keyword parameter symbol -.meta sym -is enclosed in a list, then the second element of that list -specifies a default value, similarly to the default value for -an optional argument. If the function is called in such a way -that the argument for the parameter is missing, the -.meta init-form -is evaluated and the resulting value is bound to the keyword parameter. -The evaluation takes place in a lexical scope in which the -required and optional parameters are are already visible, -and their values are bound. If there is a -.meta rest-param -it is also visible in this scope, even though in the parameter -list it appears to the left. -.meIP >> ( sym < init-form << p-sym ) -The three-element form of the keyword parameter specifies -an additional symbol -.metn p-sym , -which names an argument that implicitly receives a Boolean -argument indicating the presence of the keyword argument. -If an argument is not passed for the keyword parameter -.metn sym , -then parameter -.meta sym-p -takes on the value -.codn nil . -If an argument is given for -.metn sym , -then the -.meta sym-p -argument takes on the value -.codn t . -This mechanism also closely resembles the analogous -one supported in optional arguments. See the previous -paragraph regarding the evaluation scope of -.metn init-form . -.RE +The return value of the macro is the macro expansion. It is substituted in +place of the entire macro call form. That form is then expanded again; +it may itself be another macro call, or contain more macro calls. -.IP -Arguments to keyword appear as a property list which begins -after the last required or optional argument. A property list -consists of interleaved indicators and values. The indicators -for keyword parameters are keyword symbols whose names match -the parameter names. For instance, the indicator-value pair -.code ":xyz 42" -passes the value -.code 42 -to a keyword parameter that may be named -.code xyz -in any package: it may be -.code usr:xyz -or -.code mypackage:xyz -and so forth. +A global macro defined using +.code defmacro +may decline to expand a macro form. Declining to expand is achieved by +returning the original unexpanded form, which may be captured using the +.code :form +parameter. When a global macro declines to expand a form, the form is +taken as-is. At evaluation time, it will be treated as a function call. +Note: when a local macro defined by +.code macrolet +declines, more complicated requirements apply; see the description of +.codn macrolet . -If the function has a -.meta rest-param -then that param receives the keyword parameter list. That is to say, the -.code :key -mechanism generates a regular variadic function which receives the keyword -parameters as the trailing arguments. The function is endowed with code which -parses these extra arguments out of the trailing list, and binds them to -the keyword parameter symbols. If a -.meta rest-param -argument present, then it specifies a symbol to be used as the name of -the rest parameter, making the entire keyword argument list available -under that name. If there is no -.meta rest-param -then a machine-generated rest parameter is substituted; the keyword argument -parsing logic refers to that instead. +.TP* "Dialect Notes:" +A macro in the global namespace introduced by +.code defmacro +may co-exist with a function of the same name introduced by +.codn defun . +This is not permitted in ANSI Common Lisp. -.TP* Example: +ANSI Common Lisp doesn't describe the concept of declining to expand, except in +the area of compiler macros. Since TXR Lisp allows global macros and functions +of the same name to co-exist, ordinary macros can be used to optimize functions +in a manner similar to Common Lisp compiler macros. A macro can be written +of the same name as a function, and can optimize certain cases of the function +call by expanding them to some alternative syntax. Cases which it doesn't +optimize are handled by declining to expand, in which case the form remains +as the original function call. -Define a function -.code fun -with two required arguments -.codn "a b" , -one optional argument -.codn c , -two keyword arguments -.code foo -and -.codn bar , -and a rest parameter -.codn klist : +.TP* Example: .verb - (defun fun (:key a b : c -- foo bar . klist) - (list a b c foo bar klist)) + ;; dolist macro similar to Common Lisp's: + ;; + ;; The following will print 1, 2 and 3 + ;; on separate lines: + ;; and return 42. + ;; + ;; (dolist (x '(1 2 3) 42) + ;; (format t "~s\en")) - (fun 1 2 3 :bar 4) -> (1 2 3 nil 4 (:bar 4)) + (defmacro dolist ((var list : result) . body) + (let ((i (my-gensym))) + ^(for ((i ,list)) (i ,result) ((set i (cdr i))) + (let ((,var (car i))) + ,*body)))) .brev -Define a function with only keyword arguments, with default expressions and -Boolean indicator params: +.coNP Operator @ macrolet +.synb +.mets (macrolet >> ({( name < macro-style-params +.mets \ \ \ \ \ \ \ \ \ \ \ \ \ \ << macro-body-form *)}*) +.mets \ \ << body-form *) +.syne +.desc +The +.code macrolet +binding operator extends the macro-time lexical environment +by making zero or more new local macros visible. -.verb - (defun keyfun (:key -- (a 10 a-p) (b 20 b-p)) - (list a a-p b b-p)) +The +.code macrolet +symbol is followed by a list of macro definitions. +Each definition is a form which begins with a +.metn name , +followed by +.meta macro-style-params +which is a macro parameter list, and zero or more +.metn macro-body-form -s. +These macro definitions are similar +to those globally defined by the +.code defmacro +operator, except that they +are in a local environment. - (keyfun :a 3) -> (3 t 20 nil) +The macro definitions are followed by optional +.metn body-forms . +The macros specified in the definitions are visible to these +forms. - (keyfun :b 4) -> (10 nil 4 t) +Forms inside the macro definitions such as the +.metn macro-body-form -s, +and initializer forms appearing in the +.meta macro-style-params +are subject +to macro-expansion in a scope in which none of the new macros being +defined are yet visible. Once the macro definitions are themselves +macro-expanded, they are placed into a new macro environment, which +is then used for macro expanding the +.metn body-form -s. - (keyfun :c 4) -> (10 nil 20 nil) +A +.code macrolet +form is fully processed in the expansion phase of a form, and is +effectively replaced by +.code progn +form which contains expanded versions of +.metn body-form -s. +This expanded structure shows no evidence that any +macrolet forms ever existed in it. Therefore, it is impossible for the code +evaluated in the bodies and parameter lists of +.code macrolet +macros to have any visibility to any surrounding lexical variable bindings, +which are only instantiated in the evaluation phase, after expansion is done +and macros no longer exist. - (keyfun) -> (10 nil 20 nil) -.brev +A local macro defined using +.code defmacro +may decline to expand a macro form. Declining to expand is achieved by returning the original +unexpanded form, which may be captured using the +.code :form +parameter. When a local macro declines to expand a form, the macro definition +is temporarily hidden, as if it didn't exist in the lexical scope. If another +macro of the same name is thereby revealed (a global macro or another local macro +at a shallower nesting level), then an expansion is tried with that macro. If +no such macro is revealed, or if a lexical function binding of that name is +revealed, then no expansion takes place; the original form is taken as-is. +When another macro is tried, the process repeats, resulting in a search which +proceeds as far as possible through outer lexical scopes and finally the +global scope. -.SS* Mutation of Syntactic Places -.coNP Macro @ set +.coNP Function @ macro-form-p .synb -.mets (set >> { place << new-value }*) +.mets (macro-form-p < obj <> [ env ]) .syne .desc The -.code set -operator stores the values of expressions in places. It must -be given an even number of arguments. - -If there are no arguments, then -.code set -does nothing and returns +.code macro-form-p +function returns +.code t +if +.meta obj +represents the syntax of +a form which is a macro form: either a compound macro or a symbol macro. +Otherwise it returns .codn nil . -If there are two arguments, -.meta place -and -.metn new-value , -then -.meta place -is evaluated to determine its storage location, then -.meta new-value -is evaluated to determine the value to be stored there, -and then the value is stored in that location. Finally, -the value is also returned as the result value. - -If there are more than two arguments, then -.code set -performs multiple assignments in left to right order. -Effectively, -.code "(set v1 e1 v2 e2 ... vn en)" -is precisely equivalent to -.codn "(progn (set v1 e1) (set v2 e2) ... (set vn en))" . +A macro form is one that will transform under +.code macroexpand-1 +or +.codn macroexpand ; +an object which isn't a macro form will not undergo expansion. -.coNP Macro @ pset -.synb -.mets (pset >> { place << new-value }*) -.syne -.desc -The syntax of -.code pset -is similar to that of -.codn set , -and the semantics is similar also in that zero or more places are -assigned zero or more values. In fact, if there are no arguments, or -if there is exactly one pair of arguments, -.code pset -is equivalent to -.codn set . +The optional +.meta env +parameter is a macroexpansion environment. +A macroexpansion environment is passed down to macros and can be received +via their special +.code :env +parameter. -If there are two or more argument pairs, then all of the arguments -are evaluated first, in left-to-right order. No store takes place -until after every -.meta place -is determined, and every -.meta new-value -is calculated. During the calculation, the values to be stored -are retained in hidden, temporary locations. Finally, these values -are moved into the determined places. The rightmost value is returned -as the form's value. +.meta env +is used by +.code macro-form-p +to determine whether +.meta obj +is a macro in a lexical macro environment. -The assignments thus appear to take place in parallel, and -.code pset -is capable of exchanging the values of a pair of places, or rotating -the values among three or more places. (However, there are more convenient -operators for this, namely -.code rotate -and -.codn swap ). +If +.meta env +is not specified or is +.codn nil , +then +.code macro-form-p +only recognizes global macros. .TP* Example: + .verb - ;; exchange x and y - (pset x y y x) + ;; macro which translates to 'yes if its + ;; argument is a macro from, or otherwise + ;; transforms to the form 'no. - ;; exchange elements 0 and 1; and 2 and 3 of vector v: - (let ((v (vec 0 10 20 30)) - (i -1)) - (pset [vec (inc i)] [vec (inc i)] - [vec (inc i)] [vec (inc i)]) - vec) - -> #(10 0 30 20) -.brev + (defmacro ismacro (:env menv form) + (if (macro-form-p form menv) + ''yes ''no)) -.coNP Macro @ zap -.synb -.mets (zap < place <> [ new-value ]) -.syne -.desc -The -.code zap -macro assigns -.meta new-value -to -.meta place -and returns the previous value of -.metn place . + (macrolet ((local ())) + (ismacro (local))) ;; yields yes -If -.meta new-value -is missing, then -.code nil -is used. + (ismacro (local)) ;; yields no -In more detail, first -.code place -is evaluated to determine the storage location. -Then, the location is accessed to retrieve the -previous value. Then, the -.code new-value -expression is evaluated, and that value is -placed into the storage location. -Finally, the previously retrieved value is returned. + (ismacro (ismacro foo)) ;; yields yes +.brev -.coNP Macro @ flip -.synb -.mets (flip << place ) -.syne -.desc -The -.code flip -macro toggles the Boolean value stored in -.metn place . +During macro expansion, the global macro +.code ismacro +is handed the macro-expansion environment +via +.codn ":env menv" . -If -.meta place -previously held -.codn nil , -it is set to -.codn t , -and if it previously held a value other than -.codn nil , -it is set to +When the macro is invoked within the macrolet, +this environment includes the macro-time lexical scope in which the +.code local +macro is defined. So when global checks whether the argument form +.code (local) +is a macro, the conclusion is yes: the (local) form is a macro +call in that environment: +.code macro-form-p +yields +.codn t . + +When +.code "(global (local))" +is invoked outside of the macrolet, no local macro is visible is +there, and so +.code macro-form-p +yields .codn nil . -.coNP Macros @ test-set and @ test-clear +.coNP Functions @ macroexpand-1 and @ macroexpand .synb -.mets (test-set << place ) -.mets (test-clear << place ) +.mets (macroexpand-1 < obj <> [ env ]) +.mets (macroexpand < obj <> [ env ]) .syne .desc -The -.code test-set -macro examines the value of -.metn place . -If it is -.code nil -then it stores -.code t -into the place, and returns -.codn t . -Otherwise it leaves -.meta place -unchanged and returns -.codn nil . +If +.meta obj +is a macro form (an object for which +.code macro-form-p +returns +.codn t ), +these functions expand the macro form and return the expanded form. +Otherwise, they return +.metn obj . -The -.code test-clear -macro examines the value of -.metn place . -If it is Boolean true (any value except -.codn nil ) -then it stores -.code nil -into the place, and returns -.codn t . -Otherwise it leaves -.meta place -unchanged and returns -.codn nil . +.code macroexpand-1 +performs a single expansion, expanding just the macro +that is referenced by the symbol in the first position of +.metn obj , +and returns the expansion. That expansion may itself be a macro form. -.coNP Macro @ compare-swap -.synb -.mets (compare-swap < place < cmp-fun < cmp-val << store-val ) -.syne -.desc -The -.code compare-swap -macro examines the value of -.meta place -and compares it to -.meta cmp-val -using the comparison function given by the function name -.metn cmp-fun . +.code macroexpand +performs an expansion similar to +.codn macroexpand-1 . +If the result is +a macro form, then it expands that form, and keeps repeating this process +until the expansion yields a non-macro-form. That non-macro-form is then +returned. -This comparison takes places as if by evaluating the expression -.meti >> ( cmp-fun < value << cmp-val ) -where -.meta value -denotes the current value of -.metn place . +The optional +.meta env +parameter is a macroexpansion environment. +A macroexpansion environment is passed down to macros and can be received +via their special +.code :env +parameter. The environment they receive is their +lexically apparent macro-time environment in which local macros may be +visible. A macro can use this environment to "manually" expand some +form in the context of that environment. -If the comparison is false, -.meta place -is not modified, the -.meta store-val -expression is not evaluated, and the macro returns -.codn nil . +.TP* Example: -If the comparison is true, then -.code compare-swap -evaluates the -.meta store-val -expression, stores the resulting value into -.meta place -and returns -.codn t . +.verb + ;; (foo x) expands x, and if x begins with a number, + ;; it removes the number and returns the resulting + ;; form. Otherwise, it returns the entire form. -.coNP Macros @ inc and @ dec -.synb -.mets (inc < place <> [ delta ]) -.mets (dec < place <> [ delta ]) -.syne -.desc -The -.code inc -macro increments -.meta place -by adding -.meta delta -to its value. -If -.meta delta -is missing, the value used in its place the integer 1. + (defmacro rem-num (:env menv some-form) + (let ((expanded (macroexpand some-form menv))) + (if (numberp (car expanded)) + (cdr expanded) + some-form))) -First the -.meta place -argument is evaluated as a syntactic place to determine the location. -Then, the value currently stored in that location is retrieved. -Next, the -.meta delta -expression is evaluated. Its value is added to the previously retrieved -value as if by the -.code + -function. The resulting value is stored in the place, and returned. + ;; The following yields (42 a). -The macro -.code dec -works exactly like -.code inc -except that addition is replaced by subtraction. The similarly defaulted -.meta delta -value is subtracted from the previous value of the place. + (macrolet ((foo () '(1 list 42)) + (bar () '(list 'a))) + (list (rem-num (foo)) (rem-num (bar))))) +.brev -.coNP Macros @ pinc and @ pdec -.synb -.mets (pinc < place <> [ delta ]) -.mets (pdec < place <> [ delta ]) -.syne -.desc -The macros -.code pinc +The +.code rem-num +macro is able to expand the +.code (foo) and -.code pdec -are very similar to -.code inc +.code (bar) +forms it receives as +the +.code some-form +argument, even though these forms use local macro that are only +visible in their local scope. This is thanks to the macro +environment passed to +.codn rem-num . +It is correctly able to work with the +expansions +.code "(1 list 42)" and -.codn dec . - -The only difference is that they return the previous value of -.meta place -rather than the incremented value. - -.coNP Macros @ test-inc and @ test-dec -.synb -.mets (test-inc < place >> [ delta <> [ from-val ]]) -.mets (test-dec < place >> [ delta <> [ to-val ]]) -.syne -.desc -The -.code test-inc +.code "(list 'a)" +to produce +.code "(list 42)" and -.code test-dec -macros provide combined operations which change the value of a place and -provide a test whether, respectively, a certain previous value was -overwritten, or a certain new value was attained. By default, this tested -value is zero. - -The -.code test-inc -macro notes the prior value of -.meta place -and then updates it with that value, plus -.metn delta , -which defaults to 1. If the prior value is -.code eql -to -.meta from-val -then it returns -.codn t , -otherwise -.codn nil . -The default value of -.meta from-val -is zero. - -The -.code test-dec -macro produces a new value by subtracting -.meta delta -from the value of -.metn place . -The argument -.meta delta -defaults to 1. The new value is stored into -.metn place . -If the new value is -.code eql -to -.meta to-val -then -.code t -is returned, otherwise -.codn nil . +.code "(list 'a)" +which evaluate to +.code 42 +and +.code a +respectively. -.coNP Macro @ swap +.coNP Functions @ macroexpand-1-lisp1 and @ macroexpand-lisp1 .synb -.mets (swap < left-place << right-place ) +.mets (macroexpand-1-lisp1 < obj <> [ env ]) +.mets (macroexpand-lisp1 < obj <> [ env ]) .syne .desc The -.code swap -macro exchanges the values of -.meta left-place +.code macroexpand-1-lisp1 and -.meta right-place -and returns the value which is thereby transferred to -.metn right-place . - -First, -.meta left-place +.code macroexpand-lisp1 +functions closely resemble, respectively, +.code macroexpand-1 and -.meta right-place -are evaluated, in that order, to determine their locations. -Then the prior values are retrieved, exchanged and stored back. -The value stored in -.meta right-place -is also returned. +.codn macroexpand . -If -.meta left-place -and -.meta right-place -are ranges of the same sequence, the behavior is not specified -if the ranges overlap or are of unequal length. +The argument and return value syntax and semantics is almost +identical, except for one difference. These functions consider argument +.meta obj +to be syntax in a Lisp-1 evaluation context, such as any argument +position of the +.code dwim +operator, or the equivalent DWIM Brackets notation. -Note: the -.code rotate -macro's behavior is somewhat more specified in this regard. -Thus, although any correct -.code swap -expression can be expressed using -.codn rotate , -but the reverse isn't true. +This makes a difference because in a Lisp-1 evaluation context, an +inner function binding is able to shadow an outer symbol macro binding +of the same name. -.coNP Macro @ push -.synb -.mets (push < item << place ) -.syne -.desc -The -.code push -macro places -.meta item -at the head of the list stored in -.meta place -and returns the updated list which is stored back in -.metn place . +The requirements about this language area are given in more +detail in the description of the +.code dwim +operator. -First, the expression -.meta item -is evaluated to produce the push value. -Then, -.meta place -is evaluated to determine its storage location. -Next, the storage location is accessed to retrieve the -list value which is stored there. A new object is -produced as if by invoking -.code cons -function on the push value and list value. -This object is stored into the location, -and returned. +Note: the +.code macroexpand-lisp1 +function is useful to the implementor of a macro whose semantics requires +one or more argument forms to be treated in a Lisp-1 context, in situations +when such a macro needs to itself expand the material, rather than merely +insert it as-is into the output code template. -.coNP Macro @ pop +.coNP Functions @ expand and @ *expand .synb -.mets (pop << place ) +.mets (expand < form <> [ env ]) +.mets (expand* < form <> [ env ]) .syne .desc -The -.code pop -macro removes an element from the list stored in -.meta place -and returns it. +The functions +.code expand +and +.code expand* +both perform a complete expansion of +.meta form +in the macro-environment +.metn env , +and return that expansion. -First, -.meta place -is evaluated to determine the place. The place is accessed to -retrieve the original value. Then a new value is calculated, -as if by applying the -.code cdr -function to the old value. This new value is stored. -Finally, a return value is calculated and returned, as if by applying the -.code car -function to the original value. +If +.meta env +is omitted, the expansion takes place in the global environment in +which only global macros are visible. -.coNP Macro @ pushnew -.synb -.mets (pushnew < item < place >> [ testfun <> [ keyfun ]]) -.syne -.desc -The -.code pushnew -macro inspects the list stored in -.metn place . -If the list already contains the item, then -it returns the list. Otherwise it creates a new list -with the item at the front and stores it back -into -.metn place , -and returns it. +The returned object is a structure that +is devoid of any macro calls. Also, all +.code macrolet +and +.code symacrolet +blocks in form +.meta form +are removed in the returned structure, replaced by their fully +expanded bodies. -First, the expression -.meta item -is evaluated to produce the push value. -Then, -.meta place -is evaluated to determine its storage location. -Next, the storage location is accessed to retrieve the -list value which is stored there. The list is -inspected to check whether it already contains the push -value, as if using the -.code member -function. If that is the case, the list -is returned and the operation finishes. -Otherwise, a new object is -produced as if by invoking -.code cons -function on the push value and list value. -This object is stored into the location -and returned. +The difference between +.code expand +and +.code expand* +is that +.code expand +suppresses any warning exceptions that are issued during expansion. -.coNP Macro @ shift +.coNP Function @ expand-with-free-refs .synb -.mets (shift << place + << shift-in-value) +.mets (expand-with-free-refs < form >> [ inner-env <> [ outer-env ]]) .syne .desc The -.code shift -macro treats one or more places as a "multi-place shift register". -The values of the places are shifted one place to the left. -The first (leftmost) place receives the value of the second place, -the second receives that of the third, and so on. -The last (rightmost) place receives -.meta shift-in-value -(which is not treated as a place, even if it is a syntactic place form). -The previous value of the first place is returned. - -More precisely, all of the argument forms are evaluated left to right, in the -process of which the storage locations of the places are determined, -.meta shift-in-value -is reduced to its value. - -The values stored in the places are sampled and saved. +.code expand-with-free-refs +form performs a full expansion of +.metn form , +as if by the +.code expand +function and returns a list containing that expansion, plus four additional +items which provide information about variable and function references which +occur in +.metn form . -Note that it is not specified whether the places are sampled in a separate -pass after the evaluation of the argument forms, or whether the -sampling is interleaved into the argument evaluation. This affects -the behavior in situations in which the evaluation of any of the -.meta place -forms, or of -.metn shift-in-value , -has the side effect of modifying later places. +If both +.meta inner-env +and +.meta outer-env +are provided, then it is expected that +.meta inner-env +is lexically nested within +.metn outer-env . -Next, the places are updated by storing the saved value of the second -place into the first place, the third place into the second and so forth, -and the value of -.meta shift-in-value -into the last place. +Note: it is not required that +.meta outer-env +be the immediate parent of +.metn inner-env . -Finally, the saved original value of the first place is returned. +Note: a common usage situation is that +.meta outer-env +is the environment of the invocation of a "parent" macro which generates a form +that contains local macros. The bodies of those local macros use +.codn expand-with-free-refs , +specifying their own environment as +.meta inner-env +and that of their generating "parent" as +.metn outer-env . -If any of the places are ranges which index into the same sequence, -and the behavior is not otherwise unspecified due to the issue -noted in an earlier paragraph, the effect upon the multiply-stored -sequence can be inferred from the above-described storage order. -Note that even if stores take place which change the length of -the sequence and move some elements, not-yet-processed stores whose ranges -to refer to these elements are not adjusted. +In detail, the five items of the returned list are +.mono +.meti >> ( expansion < fv-inner < ff-inner < fv-outer << ff-outer ) +.onom +whose descriptions are: +.RS +.meIP < expansion +The full expansion of +.metn form , +containing no macro invocations, or +.code symacrolet +or +.code macrolet +forms. +.meIP < fv-inner +A list of the free variables which occur in +.meta form +relative to the +.meta inner-env +environment. That is to say, variables that are not bound inside +.meta form +and are not also bound in +.metn inner-env . +If +.meta inner-env +is omitted, then these are the absolutely free variables +occurring in +.metn form . +.meIP < ff-inner +Exactly like +.meta fv-inner +but informing about function bindings rather than variables. +.meIP < fv-outer +A list of the variables which which occur in +.meta form +which would be free if the environments between +.meta inner-env +and +.meta outer-env +(including the former, excluding the latter) +were removed from consideration. A more detailed description of this semantics +is given below. If +.meta outer-env +is omitted, then these are the absolutely free variables +occurring in +.metn form , +ignoring the +.metn inner-env . +.meIP < ff-outer +Exactly like +.meta fv-outer +but informing about function bindings rather than variables. +.RE -With regard to the foregoing paragraph, a recommended practice is -that if subranges of the same sequence object are shifted, they be -given to the macro in ascending order of starting index. Furthermore, the -semantics is simpler if the ranges do not overlap. - -.coNP Macro @ rotate -.synb -.mets (rotate << place *) -.syne -.desc -Treats zero or more places as a "multi-place rotate register". -If there are no arguments, there is no effect and -.code nil -is returned. Otherwise, the last (rightmost) place receives -the value of the first (leftmost) place. The leftmost place -receives the value of the second place, and so on. -If there are two arguments, this equivalent to -.codn swap . -The prior value of the first place, which is the the value -rotated into the last place, is returned. - -More precisely, the -.meta place -arguments are evaluated left to right, -and the storage locations are thereby determined. The storage -locations are sampled, and then the sampled values are -stored back into the locations, but rotated by one place -as described above. The saved original value of the leftmost -.meta place -is returned. +.IP +The semantics of the treatment of +.meta inner-env +and +.meta outer-env +in the calculation of +.meta fv-outer +and +.meta ff-outer +is as follows. A new environment +.meta diff-env +is calculated from these two environments, and +.meta form +is expanded in this environment. Variables and functions occurring in +.meta form +which are not bound in +.meta diff-env +are listed as +.meta fv-outer +and +.metn ff-outer . -It is not specified whether the sampling of the original values -is a separate pass which takes place after the arguments -are evaluated, or whether this sampling it is interleaved into argument -evaluation. This affects -the behavior in situations in which the evaluation of any of the -.meta place -forms has the side effect of modifying the value stored in -a later -.meta place -form. +This +.meta diff-env +is calculated as follows. First +.meta diff-env +is initialized as a copy of +.metn outer-env . +Then, all environments below +.meta outer-env +down to +.meta inner-env +are examined for bindings which shadow bindings in +.metn diff-env . +Those shadows are removed from +.metn diff-env . +Therefore, what remains in +.meta diff-env +are those bindings from +.meta outer-env +that are +.I not +shadowed by the environments between +.meta inner-env +and +.metn outer-env . -If any of the places are ranges which index into the same sequence, -and the behavior is not otherwise unspecified due to the issue -noted in the preceding paragraph, the effect upon the multiply-stored -sequence can be inferred from the above-described storage order. -Note that even if stores take place which change the length of -the sequence and move some elements, not-yet-processed stores whose ranges -to refer to these elements are not adjusted. +.TP* Example: -With regard to the foregoing paragraph, a recommended practice is -that if subranges of the same sequence object are shifted, they be -given to the macro in ascending order of starting index. Furthermore, the -semantics is simpler if the ranges do not overlap. +Suppose that +.code mac +is a macro which somehow has access to the two indicated lexical environments +in the following code snippet: -.coNP Macro @ del -.synb -.mets (del << place ) -.syne -.desc -The -.code del -macro requests the deletion of -.codn place . -If -.code place -doesn't support deletion, an exception is thrown. +.verb + (let (a c) ;; <- outer-env + (let (b) + (let (c) ;; <- inner-env + (mac (list a b c d)))) +.brev -First -.code place -is evaluated, thereby determining its location. -Then the place is accessed to retrieve its value. -The place is then subject to deletion. Finally, the -previously retrieved value is returned. +Suppose that +.code mac +invokes the +.code expand-with-free-refs +function, passing in the +.code "(list a b c d)" +argument form as +.code form +and two macro-time environment objects corresponding to the indicated +environments. -Precisely what deletion means depends on the kind of place. -The built-in places in \*(TL have deletion semantics which are -intended to be unsurprising to the programmer familiar with the -data structure which holds the place. +The return value of +.code expand-with-free-refs +shall be: -Generally, if a place denotes the element of a sequence, then deletion of the -place implies deletion of the element, and deletion of the element implies that -the gap produced by the element is closed. The deleted element is effectively -replaced by its successor, that successor by its successor and so on. If a -place denotes a value stored in a dynamic data set such as a hash table, -then deletion of that place implies deletion of the entry which holds -that value. If the entry is identified by a key, that key is also removed. +.verb + ((list a b c d) (a d) (list) (b c d) (list)) +.brev -.coNP Macro @ lset -.synb -.mets (lset <> { place }+ << sequence-expr ) -.syne -.desc The -.code lset -operator's parameter list consists of one or more places followed -by an expression -.metn sequence-expr . - -The macro evaluates -.codn sequence-expr , -which is expected to produce a sequence. +.meta fv-inner +list is +.code "(d)" +because this is the only variable that occurs in +.code "(list a b c d)" +which is free with regard to +.metn inner-env . +The +.codn a , +.code b +and +.code c +variables are not listed because they appear bound inside +.metn inner-env . -Successive elements of the resulting list are then assigned to each -successive -.codn place . +The reported +.meta fv-outer +list is +.code "(b c d)" +because the form is considered against +.meta diff-env +which is formed by removing the shadowing bindings from +.metn outer-env . +The difference between +.code "(a c)" +and +.code "(b c)" +is +.code a +and so the form is considered in an environment containing the binding +.code a +which leaves +.code "(b c d)" +free. -If there are fewer elements in the sequence than places, the -unmatched places receive the value -.codn nil . +Note: this information is useful because a set difference can be calculated +between the two reported sets. The set difference between the +.meta fv-outer +variables +.code "(b c d)" +and the +.meta fv-inner +variables +.code "(d)" +is +.codn "(b c)" . -Excess elements in the sequence are ignored. +That set difference +.code "(b c)" +is significant because it precisely informs about the +.I bound +variables which occur in +.code "(list a b c d)" +which appear bound in +.metn inner-env , +but are not bound due to a binding coming from +.metn outer-env . -An error exception occurs if the sequence is an improper list with fewer -elements than places. +The variable +.code d +is not listed in +.code "(b c)" +because it is not a bound variable. +The variable +.code a +is not in +.code "(b c)" +because though it is bound in +.metn inner-env , +that binding comes from +.metn outer-env . -A -.code lset -form produces the value of -.meta sequence-expr -as its result value. +The upshot of this logic is that it allows a macro to inspect a form in order +to discover the identities of the variables and functions which are used inside +that form, whose definitions come from a specific, bounded scope surrounding +that form. -.coNP Macro @ upd +.coNP Functions @ lexical-var-p and @ lexical-fun-p .synb -.mets (upd < place << opip-arg *) +.mets (lexical-var-p < env << form ) +.mets (lexical-fun-p < env << form ) .syne .desc -The -.code upd -macro evaluates -.meta place -and passes the value as an argument to the operational pipeline -function formed, -as if by the -.code opip -macro, from the -.meta opip-arg -arguments. The result of this function is then stored back into -.metn place . +These two functions are useful to macro writers. They are intended +to be called from the bodies of macro expanders, such as the bodies of +.code defmacro +or +.code macrolet +forms. The +.meta env +argument is a macro-time environment, which is available to macros +via the special +.code :env +parameter. Using these functions, a macro can enquire whether +a given +.meta form +is a symbol which has a variable binding or a function binding +in the lexical environment. +This information is known during macro expansion. The macro expander +recognizes lexical function and variable bindings, because these +bindings can shadow macros. -The following equivalence holds, except that place -.code p -is evaluated only once: +.TP* Example: .verb - (upd p x y z ...) <--> (set p (call (opip x y z ...) p)) + ;; + ;; this macro replaces itself with :lexical-var if its + ;; argument is a lexical variable, :lexical-fun if + ;; its argument is a lexical function, or with + ;; :not-lex-fun-var if neither is the case. + ;; + (defmacro classify (sym :env e) + (cond + ((lexical-var-p e sym) :lexical-var) + ((lexical-fun-p e sym) :lexical-fun) + (t :not-lex-fun-var))) + + ;; + ;; This returns: + ;; + ;; (:lexical-var :not-lex-fun-var :lexical-fun) + ;; + (let ((x 1) (y 2)) + (symacrolet ((y x)) + (flet ((f () (+ 2 2))) + (list (classify x) (classify y) (classify z))))) .brev -.SS* User-Defined Places and Place Operators -\*(TL provides a number of place-modifying operators such as -.codn set , -.codn push , -and -.codn inc . -It also provides a variety of kinds of syntactic places -which may be used with these operators. +.TP* Note: -Both of these categories are open-ended: \*(TL programs may extend -the set of place-modifying operators, as well as the vocabulary of -forms which are recognized as syntactic places. +These functions do not call +.code macroexpand +on the form. In most cases, it is necessary for the macro writers +to do so. Not that in the above example, symbol +.code y +is classified as neither a lexical function nor variable. +However, it can be macro-expanded to +.code x +which is a lexical variable. -Regarding place operators, it might seem obvious that new place operators can -be developed, since they are macros, and macros can expand to uses -of existing place operators. As an example, it may seem that -.code inc -operator could be written as a macro which uses -.codn set : +.coNP Function @ lexical-lisp1-binding +.synb +.mets (lexical-lisp1-binding < env << symbol ) +.syne +.desc +The +.code lexical-lisp1-binding +function inspects the macro-time environment +.meta env +to determine what kind of binding, if any, does +.meta symbol +have in that environment, from a Lisp-1 perspective. -.verb - (defmacro new-inc (place : (delta 1)) - ^(set ,place (+ ,place ,delta))) -.brev +That is to say, it considers function bindings, variable bindings +and symbol macro bindings to be in a single name space and finds +the innermost binding of one of these types for +.metn symbol . -However, the above -.code new-inc -macro has a problem: the -.code place -argument form is inserted into two places in the expansion, which -leads to two evaluations. This is visibly incorrect if the place -form contains any side effects. It is also potentially inefficient. +If such a binding is found, then the function returns one of +the three keyword symbols +.codn :var , +.codn :fun , +or +.codn :symacro . -\*(TL provides a framework for writing place update macros which -evaluate their argument forms once, even if they have to access -and update the same places. +If no such lexical binding is found, then the function +returns +.codn nil . -The framework also supports the development of new kinds of place forms -as capsules of code which introduce the right kind of material into -the lexical environment of the body of an update macro, to enable -this special evaluation. +Note that a +.code nil +return doesn't mean that the symbol doesn't have a lexical binding. It could +have an operator macro lexical binding (a macro binding in the function +namespace established by +.codn macrolet ). -.NP* Place-Expander Functions -The central design concept in \*(TL syntactic places are -.IR "place-expander functions" . -Each compound place is defined by up to three place-expander functions, -which are associated with the place via the leftmost operator -symbol of the place form. One place-expander, the -.IR "update expander" , -is mandatory. Optionally, a place may also provide a -.I "clobber expander" -as well as a -.IR "delete expander" . -An update expander provides the expertise for evaluating a place form once -in its proper run-time context to determine its actual run-time storage -location, and to access and modify the storage location. -A clobber expander provides an optimized mechanism for uses that perform -a one-time store to a place without requiring its prior value. -If a place definition does not supply a clobber expander, then the syntactic -places framework uses the update expander to achieve the functionality. -A delete expander provides the expertise for determining the actual run-time -storage location corresponding to a place, and obliterating it, -returning its prior value. If a place does not supply a delete expander, then -the place does not support deletion. Operators which require deletion, such as -.code del -will raise an error when applied to that place. +.coNP Operator @ defsymacro +.synb +.mets (defsymacro < sym << form ) +.syne +.desc +A +.code defsymacro +form introduces a symbol macro. A symbol macro consists of a binding +between a symbol +.meta sym +and and a +.metn form . +The binding denotes the form itself, rather than its value. How the +symbol macro works is that if +.meta sym +occurs as a form in a scope where the symbol macro definition is +in scope, +.meta sym +is replaced by +.metn form . +Immediately after this replacement takes place, +.meta form +itself is then processed for further replacement of macros and +symbol macros. -The expanders operate independently, and it is expected that place-modifying -operators choose one of the three, and use only that expander. For example, -accessing a place with an update expander and then overwriting its value -with a clobber expander may result in incorrect code which contains multiple -evaluations of the place form. +Symbol macros are also recognized in contexts +where +.meta sym +denotes a place which is the target of an assignment operation +like +.code set +and similar. -The programmer who implements a new place does not write expanders directly, -but rather defines them via the -.codn defplace , -.code define-accessor -or -.code defset -macro. +A +.code defsymacro +form is implicitly executed at expansion time, and thus need +not be wrapped in a +.code macro-time +form, just like +.codn defmacro . -The programmer who implements a new place update macro likewise does not -call the expanders directly. Usually, they are invoked via the macros -.codn with-update-expander , -.code with-clobber-expander -and -.codn with-delete-expander . -These are sufficient for most kind of macros. -In certain complicated cases, expanders may be invoked using the wrapper -functions -.codn call-update-expander , -.code call-clobber-expander -and -.codn call-delete-expander . -These convenience macros and functions perform certain common chores, like -macro-expanding the place in the correct environment, and choosing the -appropriate function. +Note: if a symbol macro expands to itself directly, expansion stops. However, +if a symbol macro expands to itself through a chain of expansions, +runaway expansion-time recursion will occur. -The expanders are described in the following sections. +If a global variable exists by the name +.metn sym , +then +.code defsymacro +first removes that variable from the global environment, and if that +variable is special, the symbol's special marking is removed. +.code defsymacro +doesn't alter the dynamic binding of a special variable. Any such +a binding remains intact. +If +.code defsymacro +is evaluated in a scope in which there is any lexical or dynamic binding +of +.meta sym +in the variable namespace, whether as a variable or macro, +the global symbol macro is shadowed by that binding. -.NP* The Update Expander +.coNP Operator @ symacrolet .synb -.mets (lambda >> ( getter-sym < setter-sym < place-form -.mets \ \ \ \ \ \ \ \ << body-form ) ...) +.mets (symacrolet >> ({( sym << form )}*) << body-form *) .syne .desc -The update expander is a code-writer. It takes a -.meta body-form -argument, representing code, and returns a larger form which surrounds -this code with additional code. +The +.code symacrolet +operator binds local, lexically scoped macros that are +similar to the global symbol macros introduced by +.codn defsymacro . -This larger form returned by the update expander can be regarded as having two -abstract actions, when it is substituted and evaluated in the context where -.meta place-form -occurs. The first abstract action is to evaluate -.meta place-form -exactly one time, in order to determine the actual run-time location to which -that form refers. -The second abstract action is to evaluate the caller's -.metn body-form -s, -in a lexical environment in which bindings exist for some lexical -functions or (more usually) lexical macros. These lexical macros -are explicitly referenced by the -.metn body-form ; -the update expander just provides their definition, under the names -it is given via the -.meta getter-sym -and -.meta setter-sym -arguments. +Each +.meta sym +in the bindings list is bound to its corresponding form, creating a +new extension of the expansion-time lexical macro environment. -The update expander writes local functions or macros under these names: a -getter function and a setter function. Usually, update expanders write -macros rather than functions, possibly in combination with some lexical -anonymous variables which hold temporary objects. Therefore the getter -and setter are henceforth referred to as macros. +Each +.meta body-form +is subsequently macro-expanded in this new environment +in which the new symbol macros are visible. -The code being generated is with regard to some concrete instance of -.metn place-form . -This argument is the actual form which occurs in a program. For -instance, the update expander for the -.code car -place might be called with an arbitrary variant of the -.meta place-form -which might look like -.codn "(car (inc (third some-list)))" . +Note: ordinary lexical bindings such as those introduced by let or by +function parameters lists shadow symbol macros. If a symbol +.code x +is bound by nested instances of +.code macrolet +and a +.codn let , +then the scope enclosed by both +constructs will see whichever of the two bindings is more inner, +even though the bindings are active in completely separate phases of +processing. -In the abstract semantics, upfront code wrapped around the -.meta body-form -by the update expander provides the logic to evaluate this place to -a location, which is retained in some hidden local context. +From the perspective of the arguments of a +.code dwim +form, lexical function bindings also shadow symbol macros. +This is consistent with the Lisp-1-style name resolution which +applies inside a +.code dwim +form. Of course, lexical operator macros do not shadow +symbol macros under any circumstances. -The getter local macro named by -.meta getter-sym -must provide the logic for retrieving the value of this place. -The getter macro takes no arguments. +.coNP Macros @ placelet and @ placelet* +.synb +.mets (placelet >> ({( sym << place )}*) << body-form *) +.mets (placelet* >> ({( sym << place )}*) << body-form *) +.syne +.desc The -.meta body-form -makes free use of the getter function; they may call it multiple times, -which must not trigger multiple evaluations of the original place -form. +.code placelet +macro binds lexically scoped symbol macros in such +a way that they behave as aliases for places +denoted by place forms. -The setter local macro named by -.meta setter-sym -must generate the logic for storing a new value into the once-evaluated -version of -.metn place-form . -The setter function takes exactly one argument, whose -value specifies the value to be stored into the place. -It is the caller's responsibility to ensure that the -argument form which produces the value to be stored via the setter is evaluated -only once, and in the correct order. The setter does not concern itself with -this form. Multiple calls to the setter can be expected to result in multiple -evaluations of its argument. Thus, if necessary, the caller must supply the code -to evaluate the new value form to a temporary variable, and then pass the -temporary variable to the setter. This code can be embedded in -the -.meta body-form -or can be added to the code returned by a call to the update expander. +Each +.meta place +must be an expression denoting a syntactic place. The +corresponding +.meta sym +is established as an alias for the storage location which that place denotes, +over the scope of the +.metn body-form -s. -The setter local macro or function must return the new value which is stored. -That is to say, when -.meta body-form -invokes this local macro or function, it may rely on it yielding the -new value which was stored, as part of achieving its own semantics. +This binding takes place in such a way that each +.meta place +is evaluated exactly once, only in order to determine its +storage location. The corresponding +.meta sym +then serves as an alias for that location, over the +scope of the +.metn body-form -s. +This means that whenever +.meta sym +is evaluated, it stands for the value of the storage +location, and whenever a value is apparently stored into +.metn sym , +it is actually the storage location which receives it. -The update expander does not macro-expand -.codn place-form . -It is assumed that the expander is invoked in such a way that the -place has been expanded in the correct environment. In other words, the -form matches the type of place which the expander handles. -If the expander had to macro-expand the place form, it would sometimes have -to come to the conclusion that the place form must be handled by a different -expander. No such consideration is the case: when an expander is called on -a form, that is final; it is certain that it is the correct expander, which -matches the symbol in the -.code car -position of the form, which is not a macro in the context where it occurs. +The +.code placelet* +variant implements an alternative scoping rule, which allows a later +.meta place +form to refer to a +.meta sym +bound to an earlier +.meta place +form. In other words, a given +.meta sym +binding is visible not only to the +.metn body-form -s +but also to +.meta place +forms which occur later. -An update expander is free to assume that any place which is stored -(the setter local macro is invoked on it) is accessed at least once by -an invocation of the getter. A place update macro which relies on an update -expander, but uses only the store macro, might not work properly. -An example of an update expander which relies on this assumption is the -expander for the +Note: certain kinds of places, notably .mono .meti (force << promise ) .onom -place type. If -.meta promise -has not yet been forced, and only the setter is used, then -.meta promise -might remain unforced as its internal value location is updated. -A subsequent access to the place will incorrectly trigger a force, -which will overwrite the value. The expected behavior is that storing -a value in an unforced -.code force -place changes the place to forced state, preempting the evaluation of -the delayed form. Afterward, the promise exhibits the value which was -thus assigned. - -The update expander is not responsible for all issues of evaluation order. A -place update macro may consist of numerous places, as well as numerous -value-producing forms which are not places. Each of the places can provide its -registered update expander which provides code for evaluating just that place, -and a means of accessing and storing the values. The place update macro must -call the place expanders in the correct order, and generate any additional code -in the correct order, so that the macro achieves its required documented -evaluation order. - -.TP* "Example Update Expander Call:" - -.verb - ;; First, capture the update expander - ;; function for (car ...) places - ;; in a variable, for clarity. - - (defvar car-update-expander [*place-update-expander* 'car]) - - ;; Next, call it for the place (car [a 0]). - ;; The body form specifies logic for - ;; incrementing the place by one and - ;; returning the new value. - - (call car-update-expander 'getit 'setit '(car [a 0]) - '(setit (+ (getit) 1))) - - ;; --> Resulting code: - - (rlet ((#:g0032 [a 0])) - (macrolet ((getit nil - (append (list 'car) (list '#:g0032))) - (setit (val) - (append (list 'sys:rplaca) - (list '#:g0032) (list val)))) - (setit (+ (getit) 1)))) - - ;; Same expander call as above, with a call to expand added - ;; to show the fully expanded version of the returned code, - ;; in which the ;; setit and getit calls have disappeared, - ;; replaced by their macro-expansions. +expressions, must be accessed before they can be stored, +and this restriction continues to hold when those +places are accessed through +.code placelet +aliases. - (expand - (call car-update-expander 'getit 'setit '(car [a 0]) - '(setit (+ (getit) 1)))) +Note: +.code placelet +differs from +.code symacrolet +in that the forms themselves are not aliased, but the storage +locations which they denote. +.code "(symacrolet ((x y)) z)" +performs the syntactic substitution of symbol +.code x +by form +.codn y , +wherever +.code x +appears inside +.code z +as an evaluated form, and is not shadowed by any inner binding. +Whereas +.code "(placelet ((x y)) z)" +generates code which arranges for +.code y +to be evaluated to a storage location, and syntactically replaces occurrences +of +.code x +with a form which directly denotes that storage location, +wherever +.code x +appears inside +.code z +as an evaluated form, and is not shadowed by any inner binding. +Also, +.code x +is not necessarily substituted by a single, fixed form, +as in the case of +.codn symacrolet . +Rather it may be substituted by one kind of form when it +is treated as a pure value, and another kind of form +when it is treated as a place. - ;; --> Resulting code: +.TP* "Example:" - (let ((#:g0032 [a 0])) - (sys:rplaca #:g0032 (+ (car #:g0032) 1))) +Implementation of +.code inc +using +.codn placelet : +.verb + (defmacro inc (place : (delta 1)) + (with-gensyms (p) + ^(placelet ((,p ,place)) + (set ,p (+ ,p ,delta))))) .brev -The main noteworthy points about the generated code are: -.RS -.IP - -the -.code "(car [a 0])" -place is evaluated by evaluating the embedded form -.code "[a 0]" -and storing storing the resulting object into a hidden local variable. -That's as close a reference as we can make to the -.code car -field. -.IP - -the getter macro expands to code which simply calls the -.code car -function on the cell. -.IP - -the setter uses a system function called -.codn sys:rplaca , -which differs from -.code rplaca -in that it returns the stored value, rather than the cell. -.RE -.NP* The Clobber Expander +The gensym +.code p +is used to avoid accidental capture of references +emanating from the +.code delta +form. + +.coNP Macro @ equot .synb -.mets (lambda >> ( simple-setter-sym < place-form -.mets \ \ \ \ \ \ \ \ << body-form ) ...) +.mets (equot << form ) .syne .desc -The clobber expander is a code-writer similar to the update expander. -It takes a -.meta body-form -argument, and returns a larger form which surrounds this form -with additional program code. +The +.code equot +macro ("expand and quote") performs a full expansion of +.code form +in the surrounding macro environment. Then it constructs a +.code quote +form whose argument is the expansion. This quote form is +then returned as the macro replacement for the original +.code equot +form. -The returned block of code has one main abstract action. -It must arrange for the evaluation of -.meta body-form -in a lexical environment in which a lexical macro or lexical function -exists which has the name requested by the -.meta simple-setter-sym -argument. +.TP* Example: -The simple setter local macro written by the clobber expander is similar to the -local setter written by the update expander. It has exactly the -same interface, performs the same action of storing a value into -the place, and returns the new value. +.verb + (symacrolet ((a (+ 2 2))) + (list (quote a) (equot a) a)) + --> (a (+ 2 2) 4) +.brev -The difference is that its logic may be considerably simplified by the -assumption that the place is being subject to exactly one store, -and no access. +Above, the expansion of +.code a +is +.codn "(+ 2 2)" . +Thus the macro call +.code "(equot a)" +expands to +.codn "(quote (+ 2 2))" . +When that is evaluated, it yields +.codn "(+ 2 2)" . -A place update macro which uses a clobber expander, and calls it more than -once, break the assumption; doing so may result in multiple evaluations -of the -.metn place-form . +If +.code a +is quoted, then of course the result is +.codn a : +no expansion or evaluation takes place. +Whereas if +.code a +is presented for evaluation, then not only is it expanded to +.codn "(+ 2 2)" , +but that expansion is reduced to 4. -.NP* The Delete Expander +The +.code equot +operator is a mongrel of these two semantics: it permits expansion to proceed, +but then suppresses evaluation of the result. + +.coNP Operators @ tree-bind and @ mac-param-bind .synb -.mets (lambda >> ( deleter-sym < place-form -.mets \ \ \ \ \ \ \ \ << body-form ) ...) +.mets (tree-bind < macro-style-params < expr << form *) +.mets (mac-param-bind < context-expr +.mets \ \ < macro-style-params < expr << form *) .syne .desc -The delete expander is a code-writer similar to clobber expander. -It takes a -.meta body-form -arguments, and returns a larger form which surrounds this form -with additional program code. +The +.code tree-bind +operator evaluates +.codn expr , +and then uses the +resulting value as a counterpart to a macro-style parameter list. +If the value has a tree structure which matches the parameters, +then those parameters are established as bindings, and the +.metn form -s, +if any, are evaluated in the scope of those bindings. The value +of the last +.meta form +is returned. If there are no forms, +.code nil +is returned. -The returned block of code has one main abstract action. -It must arrange for the evaluation of -.meta body-form -in a lexical environment in which a lexical macro or lexical function -exists which has the name requested by the -.meta deleter-sym -argument. +Note: this operator throws an exception if there is a +structural mismatch between the parameters and the value of +.codn expr . -The deleter macro written by the clobber expander takes no arguments. -It may be called at most once. It returns the previous value of the -place, and arranges for its obliteration, whatever that means for -that particular kind of place. +One way to avoid this exception is to use +.codn tree-case . -.coNP Macro @ with-update-expander +The +.code mac-param-bind +operator is similar to +.code tree-bind +except that it takes an extra argument, +.metn context-expr . +This argument is an expression which is evaluated. It is expected to +evaluate to a compound form. If an error occurs during binding, the error +diagnostic message is based on information obtained from this form. +By contrast, the +.code tree-bind +operator's error diagnostic refers to the +.code tree-bind +form, which is cryptic if the binding is used for the implementation +of some other construct, hidden from the user of that construct. + +.coNP Operator @ tree-case .synb -.mets (with-update-expander >> ( getter << setter ) < place < env -.mets \ << body-form ) +.mets (tree-case < expr >> {( macro-style-params << form *)}*) .syne .desc The -.code with-update-expander -macro evaluates the -.meta body-form -argument, whose result is expected to be a Lisp form. -The macro adds additional code around this code, and the result is returned. -This additional code is called the -.IR "place-access code" . +.code tree-case +operator evaluates +.meta expr +and matches it against a succession +of zero or more cases. Each case defines a pattern match, expressed as a macro +style parameter list +.metn macro-style-params . -The -.meta getter -and -.meta setter -arguments must be symbols. Over the evaluation of the -.metn body-form , -these symbols are bound to the names of local functions which -are provided in the place-access code. +If the object produced by +.meta expr +matches +.metn macro-style-params , +then the parameters are bound, becoming local variables, and the +.metn form -s, +if any, are evaluated in order in the environment in which those variables are +visible. If there are forms, the value of the last +.meta form +becomes the result +value of the case, otherwise the result value of the case is nil. -The -.meta place -argument is a form which evaluates to a syntactic place. The generated -place-access code is based on this place. +If the result value of a case is the object +.code : +(the colon symbol), then processing continues with the next case. Otherwise the +evaluation of +.code tree-case +terminates, returning the result value. -The -.meta env -argument is a form which evaluates to a macro-expansion-time environment. -The -.code with-update-expander -macro uses this environment to perform macro-expansion on the value of the -.meta place -form, to obtain the correct update expander function for the fully -macro-expanded place. +If the value of +.meta expr +does not match the +.meta macro-style-params +parameter list of a case, processing continues with the next case. -The place-access code is generated by calling the update expander -for the expanded version of -.codn place . +If no cases match, then +.code tree-case +terminates, returning +.codn nil . -.TP* "Example:" +.TP* Example: -The following is an implementation of the -.code swap -macro, which exchanges the contents of two places. +.verb + ;; reverse function implemented using tree-case -Two places are involved, and, correspondingly, the -.code with-update-expander -macro is used twice, to add two instances of place-update code -to the macro's body. + (defun tb-reverse (obj) + (tree-case obj + (() ()) ;; the empty list is just returned + ((a) obj) ;; one-element list returned + ((a . b) ^(,*(tb-reverse b) ,a)) ;; car/cdr recursion + (a a))) ;; atom is just returned +.brev -.verb - (defmacro swap (place-0 place-1 :env env) - (with-gensyms (tmp) - (with-update-expander (getter-0 setter-0) place-0 env - (with-update-expander (getter-1 setter-1) place-1 env - ^(let ((,tmp (,getter-0))) - (,setter-0 (,getter-1)) - (,setter-1 ,tmp)))))) -.brev - -The basic logic for swapping two places is contained in the code template: +Note that in this example, the atom case is placed last, because an +argument list which consists of a symbol is a "catch all" match +that matches any object. We know that it matches an atom, because +the previous +.code "(a . b)" +case matches conses. In general, the order of the cases in +.code tree-case +is important: even more so than the order of cases in a +.code cond +or +.codn caseql . +The one-element list case is unnecessary; it can be removed. -.verb - ^(let ((,tmp (,getter-0))) - (,setter-0 (,getter-1)) - (,setter-1 ,tmp)) -.brev +.coNP Macro @ tb +.synb +.mets (tb < macro-style-params << form *) +.syne +.desc +The +.code tb +macro is similar to the +.code lambda +operator but its argument binding is based on a macro-style parameter list. +The name is an abbreviation of +.codn tree-bind . -The temporary variable named by the -.code gensym -symbol -.code tmp -is initialized by calling the getter function for -.metn place-0 . -Then the setter function of -.meta place-0 -is called in order to store the value of -.meta place-1 -into -.metn place-0 . -Finally, the setter for -.meta place-1 -is invoked to store the previously saved temporary value into -that place. +A +.code tb +form evaluates to a function which takes a variable number of +arguments. -The name for the temporary variable is provided by the -.code with-gensyms -macro, but establishing the variable is the caller's responsibility; -this is seen as an explicit -.code let -binding in the code template. +When that function is called, those arguments are taken as a list object which +is matched against +.meta macro-style-params +as if by +.metn tree-bind . +If the match is successful, then the parameters are bound to the +corresponding elements from the argument structure and each successive +.meta form +is evaluated an environment in which those bindings are visible. +The value of the last +.meta form +is the return value of the function. If there are no forms, +the function's return value is +.codn nil . -The names of the getter and setter functions are similarly provided -by the -.code with-update-expander -macros. However, binding those functions is the responsibility of that -macro. To achieve this, it adds the place-access code to the code generated by -the -.code "^(let ...)" -backquote template. In the following example macro-expansion, the additional -code added around the template is seen. It takes the form of two -.code macrolet -binding blocks, each added by an invocation of -.codn with-update-expander : +The following equivalence holds, where +.code args +should be understood to be a globally unique symbol: .verb - (macroexpand '(swap a b)) - - --> - - (macrolet ((#:g0036 () 'a) ;; getter macro for a - (#:g0037 (val-expr) ;; setter macro for a - (append (list 'sys:setq) (list 'a) - (list val-expr)))) - (macrolet ((#:g0038 () 'b) ;; getter macro for b - (#:g0039 (val-expr) ;; setter macro for b - (append (list 'sys:setq) (list 'b) - (list val-expr)))) - (let ((#:g0035 (#:g0036))) ;; temp <- a - (#:g0037 (#:g0038)) ;; a <- b - (#:g0039 #:g0035)))) ;; b <- temp + (tb pattern body ...) <--> (lambda (. args) + (tree-bind pattern args body ...)) .brev -In this expansion, for example -.code #:g0036 -is the generated symbol which forms the value of the -.code getter-0 -variable in the -.code swap -macro. The getter is a macro which simply expands to a -.codn a : -straightforward access to the variable a. -Of course, -.code #:g0035 -is nothing but the value of the -.code tmp -variable. Thus the swap macro's -.mono -^(let ((,tmp (,getter-0))) ...) -.onom -has turned into -.mono -^(let ((#:g0035 (#:g0036))) ...) -.onom - -A full expansion, with the -.code macrolet -local macros expanded out: +.coNP Macro @ tc +.synb +.mets (tc >> {( macro-style-params << form *)}*) +.syne +.desc +The +.code tc +macro produces an anonymous function whose behavior is closely +based on the +.code tree-case +operator. Its name is an abbreviation of +.codn tree-case . -.verb - (expand '(swap a b)) +The anonymous function takes a variable number of arguments. +Its argument list is taken to be the value macro is tested +against the multiple pattern clauses of an implicit +.codn tree-case . +The return value of the function is that of the implied +.codn tree-case . - --> +The following equivalence holds, where +.code args +should be understood to be a globally unique symbol: - (let ((#:g0035 a)) - (sys:setq a b) - (sys:setq b #:g0035)) +.verb + (tc clause1 clause2 ...) <--> (lambda (. args) + (tree-case args + clause1 clause2 ...)) .brev -In other words, the original syntax -.mono -(,getter-0) -.onom -became -.mono -(#:g0036) -.onom -and finally just -.codn a . - -Similarly, -.mono -(,setter-0 (,getter-1)) -.onom -became the -.code macrolet -invocations -.mono -(#:g0037 (#:g0038)) -.onom -which finally turned into: -.codn "(sys:setq a b)" . - -.coNP Macro @ with-clobber-expander +.coNP Macro @ with-gensyms .synb -.mets (with-clobber-expander <> ( simple-setter ) < place < env -.mets \ << body-form ) +.mets (with-gensyms <> ( sym *) << body-form *) .syne .desc The -.code with-clobber-expander -macro evaluates -.metn body-form , -whose result is expected to be a Lisp form. The macro adds additional code -around this form, and the result is returned. This additional code is called -the -.IR "place-access code" . - -The -.meta simple-setter -argument must be a symbol. Over the evaluation of the -.metn body-form , -this symbol is bound to the name of a functions which -are provided in the place-access code. - -The -.meta place -argument is a form which evaluates to a syntactic place. The generated -place-access code is based on this place. - -The -.meta env -argument is a form which evaluates to a macro-expansion-time environment. -The -.code with-clobber-expander -macro uses this environment to perform macro-expansion on the value of the -.meta place -form, to obtain the correct update expander function for the fully -macro-expanded place. - -The place-access code is generated by calling the update expander -for the expanded version of -.codn place . +.code with-gensyms +evaluates the +.metn body-form -s +in an environment in which each variable name symbol +.meta sym +is bound to a new uninterned symbol ("gensym"). .TP* "Example:" -The following implements a simple assignment statement, similar to -.code set -except that it only handles exactly two arguments: +The code: .verb - (defmacro assign (place new-value :env env) - (with-clobber-expander (setter) place env - ^(,setter ,new-value))) + (let ((x (gensym)) + (y (gensym)) + (z (gensym))) + ^(,x ,y ,z)) .brev -Note that the correct evaluation order of -.code place -and -.code new-value -is taken care of, because -.code with-clobber-expander -generates the code which performs all the necessary evaluations of -.codn place . -This evaluation occurs before the code which is generated by -.mono -^(,setter ,new-value) -.onom -part is evaluated, and that code is what evaluates -.codn new-value . - -Suppose that a macro were desired which allows assignment to be notated in a right to left -style, as in: +may be expressed more conveniently using the +.code with-gensyms +shorthand: .verb - (assign 42 a) ;; store 42 in variable a + (with-gensyms (x y z) + ^(,x ,y ,z)) .brev -Now, the new value must be evaluated prior to the place, if left to right -evaluation order is to be maintained. The standard -.code push -macro has this property: the push value is on the left, and the place -is on the right. - -Now, the code has to explicitly take care of the order, like this: +.SS* Parameter List Macros -.verb - ;; WRONG! We can't just swap the parameters; - ;; place is still evaluated first, then new-value: +Parameter list macros, also more briefly called +.I "parameter macros" +are an original feature of \*(TL. - (defmacro assign (new-value place :env env) - (with-clobber-expander (setter) place env - ^(,setter ,new-value))) +If the first element of a function or macro parameter list is a keyword +symbol other than +.codn :env , +.codn :whole , +.code :form +or +.code : +(the colon symbol), +it denotes a parameter macro. This keyword symbol is expected to +have a binding in the parameter macro namespace: a global namespace +which associates keyword symbols with parameter list expander +functions. - ;; Correct: arrange for evaluation of new-value first, - ;; then place: +Expansion of a parameter list macro occurs at macro-expansion +time, when a function's parameter list is traversed by the +macro expander. It takes place as follows. +First, the keyword is removed from the parameter list. +The keyword's binding in the parameter macro namespace is +retrieved. If it doesn't exist, an exception is thrown. +Otherwise, the remaining parameter list is first recursively +processed for more occurrences of parameter macros. +This expansion produces a transformed parameter list, +along with a transformed function body. These two artifacts +are then passed to the transformer function retrieved from +the keyword symbol's binding. The function returns a +further transformed version of the parameter list and +body. These are processed for more parameter macros. +The process terminates when no more expansion is +possible, because a parameter list has been produced +which does not begin with a parameter macro. This +final parameter list and its accompanying body are then +taken in place of the original parameter list and +body. - (defmacro assign (new-value place :env env) - (with-gensym (tmp) - ^(let ((,tmp ,new-value)) - ,(with-clobber-expander (setter) place env - ^(,setter ,tmp))))) -.brev +\*(TL provides a built-in parameter list macro bound to the symbol +.code :key +which endows a function keyword parameters. The implementation is +written entirely using this parameter list macro mechanism, by means +of the +.code define-param-expander +macro. -.coNP Macro @ with-delete-expander -.synb -.mets (with-delete-expander <> ( deleter ) < place < env -.mets \ << body-form ) -.syne +.coNP Special variable @ *param-macro* .desc -The -.code with-delete-expander -macro evaluates -.metn body-form , -whose result is expected to be a Lisp form. -The macro adds additional code -around this code, and the resulting code is returned. This additional code is -called the -.IR "place-access code" . +The variable +.code *param-macro* +holds a hash table which associates keyword symbols with +parameter list expander functions. + +The functions are expected to conform to the following +syntax: + +.mono +.mets (lambda >> ( params < body < env << form ) << form *) +.onom The -.meta deleter -argument must be a symbol. Over the evaluation of the -.metn body-form , -this symbol is bound to the name of a functions which -are provided in the place-access code. +.meta params +parameter receives the parameter list of the function +which is undergoing parameter expansion. All other +parameter macros have already been expanded. The -.meta place -argument is a form which evaluates to a syntactic place. The generated -place-access code is based on this place. +.meta body +parameter receives the list of body forms. +The function is expected to return a +.code cons +cell whose +.code car +contains the transformed parameter list, and whose +.code cdr +contains the transformed list of body forms. +Parameter expansion takes place at macro expansion time. The .meta env -argument is a form which evaluates to a macro-expansion-time environment. -The -.code with-delete-expander -macro uses this environment to perform macro-expansion on the value of the -.meta place -form, to obtain the correct update expander function for the fully -macro-expanded place. - -The place-access code is generated by calling the update expander -for the expanded version of -.codn place . - -.TP* "Example:" - -The following implements the -.code del -macro: +parameter receives the macro-expansion-time environment +which surrounds the function being expanded. +Note that this environment doesn't take into account the +parameters themselves; therefore, it is not the correct environment +for expanding macros among the +.meta body +forms. For that purpose, it must be extended with +shadowing entries, the manner of doing which is +undocumented. However +.meta env +may be used directly for expanding init forms +for optional parameters occurring in +.metn params . -.verb - (defmacro del (place :env env) - (with-delete-expander (deleter) place env - ^(,deleter))) -.brev +The +.meta form +parameter receives the overall function-defining +form that is being processes, such as a +.code defun +or +.code lambda +form. This is intended for error reporting. -.coNP Function @ call-update-expander +.coNP Macro @ define-param-expander .synb -.mets (call-update-expander < getter < setter < place < env -.mets \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ << body-form ) +.mets (define-param-expander < name >> ( pvar < bvar : < evar << fvar ) +.mets \ \ << form *) .syne .desc The -.code call-update-expander -function provides an alternative interface for making use of an update -expander, complementary to -.codn with-update-expander . - -Arguments -.meta getter -and -.meta setter -are symbols, provided by the caller. These are passed to the update -expander function, and are used for naming local functions in the -generated code which the update expander adds to -.metn body-form . - -The -.meta place -argument is a place which has not been subject to macro-expansion. -The -.code call-update-expander -function takes on the responsibility for macro-expanding the place. +.code define-param-expander +macro provides syntax for defining parameter macros. Invocations +of this macro expand to code which constructs an anonymous +function and installs it into the +.code *param-macro* +hash table, under the key given by +.metn name . The -.meta env -parameter is the macro-expansion environment object required to -correctly expand -.code place -in its original environment. +.meta name +parameter's argument should be a keyword symbol that is valid for use +as a parameter macro name. The -.meta body-form -argument represents the source code of a place update operation. -This code makes references to the local functions whose names -are given by -.meta getter +.metn pvar , +.metn bvar , +.meta evar and -.metn setter . -Those arguments allow the update expander to write these functions -with the matching names expected by -.metn body-form . +.meta fvar +arguments must be symbols suitable for variable +binding. These symbols define the parameters of the +expander function which shall, respectively, receive +the parameter list, body forms, macro environment +and function form. If +.meta evar +is omitted, a symbol generated by the +.code gensym +function is used. Likewise if +.meta fvar +is omitted. -The return value is an object representing source code which incorporates -the -.metn body-form , -augmenting it with additional code which evaluates -.code place -to determine its location, and provides place accessor local functions -expected by the -.metn body-form . +The +.meta form +arguments constitute the body of the expander. -.TP* "Example:" +The +.code define-param-expander +form returns +.metn name . -The following shows how to implement a -.code with-update-expander -macro using -.codn call-update-expander : +.TP* Example: -.verb - (defmacro with-update-expander ((getter setter) - unex-place env body) - ^(with-gensyms (,getter ,setter) - (call-update-expander ,getter ,setter - ,unex-place ,env ,body))) -.brev +The following example shows the implementation +of a parameter macro +.code :memo +which provides rudimentary memoization. +Using the macro is extremely easy. It is a matter +of simply inserting the +.code :memo +keyword at the front of a function's parameter list. +The function is then memoized. -Essentially, all that -.code with-update-expander -does is to choose the names for the local functions, and bind them -to the local variable names it is given as arguments. Then it -calls -.codn call-update-expander . +.verb + (defvarl %memo% (hash :weak-keys)) -.TP* "Example:" + (defun ensure-memo (sym) + (or (gethash %memo% sym) + (sethash %memo% sym (hash)))) -Implement the swap macro using -.codn call-update-expander : + (define-param-expander :memo (param body) + (let* ((memo-parm [param 0..(posq : param)]) + (hash (gensym)) + (key (gensym))) + ^(,param (let ((,hash (ensure-memo ',hash)) + (,key (list ,*memo-parm))) + (or (gethash ,hash ,key) + (sethash ,hash ,key (progn ,*body))))))) +.brev + +The above +.code :memo +macro may be used to define a memoized Fibonacci function +as follows: .verb - (defmacro swap (place-0 place-1 :env env) - (with-gensyms (tmp getter-0 setter-0 getter-1 setter-1) - (call-update-expander getter-0 setter-0 place-0 env - (call-update-expander getter-1 setter-1 place-1 env - ^(let ((,tmp (,getter-0))) - (,setter-0 (,getter-1)) - (,setter-1 ,tmp)))))) + (defun fib (:memo n) + (if (< n 2) + (clamp 0 1 n) + (+ (fib (pred n)) (fib (ppred n))))) .brev -.coNP Function @ call-clobber-expander +All that is required is the insertion of the +.code :memo +keyword. + +.coNP Parameter list macro @ :key .synb -.mets (call-clobber-expander < simple-setter < place < env -.mets \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ << body-form ) +.mets (:key << param * +.mets \ \ [ -- >> { sym | >> ( sym >> [ init-form <> [ p-sym ]])}* ] +.mets \ \ [ . rest-param ]) .syne .desc -The -.code call-clobber-expander -function provides an alternative interface for making use of a clobber -expander, complementary to -.codn with-clobber-expander . +Parameter list macro +.code :key +injects keyword parameter support into functions and macros. -Argument -.meta simple-setter -is a symbol, provided by the caller. It is passed to the clobber -expander function, and is used for naming a local function in the -generated code which the update expander adds to -.metn body-form . +When +.code :key +appears as the first item in a function parameter list, a special syntax is +recognized in the parameter list. After any required and optional parameters, +the symbol +.code -- +(two dashes) may appear. Parameters after this symbol are interpreted +as keyword parameters. After the keyword parameters, a rest parameter +may appear in the usual way as a symbol in the dotted position. -The -.meta place -argument is a place which has not been subject to macro-expansion. -The -.code call-clobber-expander -function takes on the responsibility for macro-expanding the place. +Keyword parameters use the same syntax as optional parameters, except +that if used in a macro parameter list, they do not support +destructuring whereas optional parameters do. That is to say, regardless +whether +.code :key +is used in a function or macro, keyword parameters are symbols. -The -.meta env -parameter is the macro-expansion environment object required to -correctly expand -.code place -in its original environment. +A keyword parameter takes three possible forms: -The -.meta body-form -argument represents the source code of a place update operation. -This code makes references to the local function whose name -is given by -.metn simple-setter . -That argument allows the update expander to write this function -with the matching name expected by -.metn body-form . +.RS +.meIP < sym +A keyword parameter may be specified as a simple symbol +.metn sym . +If the argument for such a keyword parameter is missing, +it takes on the value +.codn nil . +.meIP >> ( sym << init-form ) +If the keyword parameter symbol +.meta sym +is enclosed in a list, then the second element of that list +specifies a default value, similarly to the default value for +an optional argument. If the function is called in such a way +that the argument for the parameter is missing, the +.meta init-form +is evaluated and the resulting value is bound to the keyword parameter. +The evaluation takes place in a lexical scope in which the +required and optional parameters are are already visible, +and their values are bound. If there is a +.meta rest-param +it is also visible in this scope, even though in the parameter +list it appears to the left. +.meIP >> ( sym < init-form << p-sym ) +The three-element form of the keyword parameter specifies +an additional symbol +.metn p-sym , +which names an argument that implicitly receives a Boolean +argument indicating the presence of the keyword argument. +If an argument is not passed for the keyword parameter +.metn sym , +then parameter +.meta sym-p +takes on the value +.codn nil . +If an argument is given for +.metn sym , +then the +.meta sym-p +argument takes on the value +.codn t . +This mechanism also closely resembles the analogous +one supported in optional arguments. See the previous +paragraph regarding the evaluation scope of +.metn init-form . +.RE -The return value is an object representing source code which incorporates -the -.metn body-form , -augmenting it with additional code which evaluates -.code place -to determine its location, and provides the clobber local function -to the -.metn body-form . +.IP +Arguments to keyword appear as a property list which begins +after the last required or optional argument. A property list +consists of interleaved indicators and values. The indicators +for keyword parameters are keyword symbols whose names match +the parameter names. For instance, the indicator-value pair +.code ":xyz 42" +passes the value +.code 42 +to a keyword parameter that may be named +.code xyz +in any package: it may be +.code usr:xyz +or +.code mypackage:xyz +and so forth. -.coNP Function @ call-delete-expander +If the function has a +.meta rest-param +then that param receives the keyword parameter list. That is to say, the +.code :key +mechanism generates a regular variadic function which receives the keyword +parameters as the trailing arguments. The function is endowed with code which +parses these extra arguments out of the trailing list, and binds them to +the keyword parameter symbols. If a +.meta rest-param +argument present, then it specifies a symbol to be used as the name of +the rest parameter, making the entire keyword argument list available +under that name. If there is no +.meta rest-param +then a machine-generated rest parameter is substituted; the keyword argument +parsing logic refers to that instead. + +.TP* Example: + +Define a function +.code fun +with two required arguments +.codn "a b" , +one optional argument +.codn c , +two keyword arguments +.code foo +and +.codn bar , +and a rest parameter +.codn klist : + +.verb + (defun fun (:key a b : c -- foo bar . klist) + (list a b c foo bar klist)) + + (fun 1 2 3 :bar 4) -> (1 2 3 nil 4 (:bar 4)) +.brev + +Define a function with only keyword arguments, with default expressions and +Boolean indicator params: + +.verb + (defun keyfun (:key -- (a 10 a-p) (b 20 b-p)) + (list a a-p b b-p)) + + (keyfun :a 3) -> (3 t 20 nil) + + (keyfun :b 4) -> (10 nil 4 t) + + (keyfun :c 4) -> (10 nil 20 nil) + + (keyfun) -> (10 nil 20 nil) +.brev + +.SS* Mutation of Syntactic Places +.coNP Macro @ set .synb -.mets (call-delete-expander < deleter < place < env << body-form ) +.mets (set >> { place << new-value }*) .syne .desc The -.code call-delete-expander -function provides an alternative interface for making use of a delete -expander, complementary to -.codn with-delete-expander . +.code set +operator stores the values of expressions in places. It must +be given an even number of arguments. -Argument -.meta deleter -is a symbol, provided by the caller. It is passed to the delete -expander function, and is used for naming a local function in the -generated code which the update expander adds to -.metn body-form . +If there are no arguments, then +.code set +does nothing and returns +.codn nil . -The +If there are two arguments, .meta place -argument is a place which has not been subject to macro-expansion. -The -.code call-delete-expander -function takes on the responsibility for macro-expanding the place. - -The -.meta env -parameter is the macro-expansion environment object required to -correctly expand -.code place -in its original environment. - -The -.meta body-form -argument represents the source code of a place delete operation. -This code makes references to the local function whose name -is given by -.metn deleter . -That argument allows the update expander to write this function -with the matching name expected by -.metn body-form . +and +.metn new-value , +then +.meta place +is evaluated to determine its storage location, then +.meta new-value +is evaluated to determine the value to be stored there, +and then the value is stored in that location. Finally, +the value is also returned as the result value. -The return value is an object representing source code which incorporates -the -.metn body-form , -augmenting it with additional code which evaluates -.code place -to determine its location, and provides the delete local function -to the -.metn body-form . +If there are more than two arguments, then +.code set +performs multiple assignments in left to right order. +Effectively, +.code "(set v1 e1 v2 e2 ... vn en)" +is precisely equivalent to +.codn "(progn (set v1 e1) (set v2 e2) ... (set vn en))" . -.coNP Macro @ define-modify-macro +.coNP Macro @ pset .synb -.mets (define-modify-macro < name < parameter-list << function-name ) +.mets (pset >> { place << new-value }*) .syne .desc -The -.code define-modify-macro -macro provides a simplified way to write certain kinds of place update -macros. Specifically, it provides a way to write place update macros -which modify a place by retrieving the previous value, pass it through -a function (perhaps together with some additional arguments), and then store -the resulting value back into the place and return it. +The syntax of +.code pset +is similar to that of +.codn set , +and the semantics is similar also in that zero or more places are +assigned zero or more values. In fact, if there are no arguments, or +if there is exactly one pair of arguments, +.code pset +is equivalent to +.codn set . -The -.meta name -parameter specifies the name for the place update macro to be written. +If there are two or more argument pairs, then all of the arguments +are evaluated first, in left-to-right order. No store takes place +until after every +.meta place +is determined, and every +.meta new-value +is calculated. During the calculation, the values to be stored +are retained in hidden, temporary locations. Finally, these values +are moved into the determined places. The rightmost value is returned +as the form's value. -The -.meta function-name -parameter must specify a symbol: the name of the update function. +The assignments thus appear to take place in parallel, and +.code pset +is capable of exchanging the values of a pair of places, or rotating +the values among three or more places. (However, there are more convenient +operators for this, namely +.code rotate +and +.codn swap ). -The update macro and update function both take at least one parameter: -the place to be updated, and its value, respectively. +.TP* Example: +.verb + ;; exchange x and y + (pset x y y x) -The -.meta parameter-list -specifies the additional parameters for update function, which will also -become additional parameters of the macro. Because it is a -function parameter list, it cannot use the special destructuring features of -macro parameter lists, or the -.code :env -or -.code :whole -special parameters. It can use optional parameters. Of course, it may be empty. + ;; exchange elements 0 and 1; and 2 and 3 of vector v: + (let ((v (vec 0 10 20 30)) + (i -1)) + (pset [vec (inc i)] [vec (inc i)] + [vec (inc i)] [vec (inc i)]) + vec) + -> #(10 0 30 20) +.brev +.coNP Macro @ zap +.synb +.mets (zap < place <> [ new-value ]) +.syne +.desc The -.code define-modify-macro -macro writes a macro called -.metn name . -The leftmost parameter of this macro is a place, followed by the additional arguments -specified by -.metn parameter-list . -The macro will arrange for the evaluation of the place argument to determine -the place location. It will then retrieve and save the prior value of the -place, and evaluate the remaining arguments. The prior value of the -place, and the values of the additional arguments, are all passed to -.meta function -and the resulting value is then stored back into the location previously -determined for +.code zap +macro assigns +.meta new-value +to +.meta place +and returns the previous value of .metn place . -.TP* "Example:" +If +.meta new-value +is missing, then +.code nil +is used. -Some standard place update macros are implementable using -.codn define-modify-macro , -such as -.codn inc . +In more detail, first +.code place +is evaluated to determine the storage location. +Then, the location is accessed to retrieve the +previous value. Then, the +.code new-value +expression is evaluated, and that value is +placed into the storage location. +Finally, the previously retrieved value is returned. +.coNP Macro @ flip +.synb +.mets (flip << place ) +.syne +.desc The -.code inc -macro reads the old value of the place, then passes it through the -.code + -(plus) function, along with an extra argument: the delta value, which -defaults to one. The -.code inc -macro could be written using -.code define-modify-macro -as follows: - -.verb - (define-modify-macro inc (: (delta 1)) +) -.brev +.code flip +macro toggles the Boolean value stored in +.metn place . -Note that the argument list -.code "(: (delta 1))" -doesn't specify the place, because the place is the implicit leftmost -argument of the macro which isn't given a name. With the above definition -in place, when -.code "(inc (car a))" -is invoked, then -.code "(car a)" -is first reduced to a location, and that location's value is retrieved and -saved. Then the -.code delta -parameter s evaluated to its value, which has defaulted to 1, since -the argument was omitted. -Then these two values are passed to the -.code + -function, and so 1 is added to the value previously retrieved from -.codn "(car a)" . -The resulting sum is then stored back -.code "(car a)" -without, of course, evaluating -.code "(car a)" -again. +If +.meta place +previously held +.codn nil , +it is set to +.codn t , +and if it previously held a value other than +.codn nil , +it is set to +.codn nil . -.coNP Macro @ defplace +.coNP Macros @ test-set and @ test-clear .synb -.mets (defplace < place-destructuring-args < body-sym -.mets \ \ \ \ \ \ \ \ \ >> ( getter-sym < setter-sym << update-body ) -.mets \ \ \ \ \ \ \ \ \ >> [( ssetter-sym << clobber-body ) -.mets \ \ \ \ \ \ \ \ \ \ >> [( deleter-sym << delete-body )]]) +.mets (test-set << place ) +.mets (test-clear << place ) .syne .desc The -.code defplace -macro is used to introduce a new kind of syntactic place. -It writes the update expander, and optionally clobber and delete -expander functions, from a simpler, more compact specification, -and automatically registers the resulting functions. The compact specification -of a -.code defplace -call contains only code fragments for the expander functions. - -The name and syntax of the place is determined by the -.meta place-destructuring-args -argument, which is macro-style parameter list whose structure -mimics that of the the place. In particular, its leftmost symbol -gives the name under which the place is registered. -The -.code defplace -macro provides automatic destructuring of the syntactic place, -so that the expander code fragments can refer to the components -of a place by name. +.code test-set +macro examines the value of +.metn place . +If it is +.code nil +then it stores +.code t +into the place, and returns +.codn t . +Otherwise it leaves +.meta place +unchanged and returns +.codn nil . The -.meta body-sym -parameter must be be a symbol. This symbol will capture the -.meta body-forms -parameter which is passed to the update expander, clobber -expander or delete expander. The code fragments then have -access to the the body forms via this name. - -The -.metn getter-sym , -.metn setter-sym , -and -.meta update-body -parenthesized triplet specify the update expander fragment. -The -.code defplace -macro will bind -.meta getter-sym -and -.meta setter-sym -to symbols. The -.meta update-body -must then specify a template of code which evaluates the syntactic place to -determine its storage location, and provides a pair of local functions, using -these two symbols as their name. The template must also insert the -.meta body-sym -forms into the scope of these local functions, and the place determining code. +.code test-clear +macro examines the value of +.metn place . +If it is Boolean true (any value except +.codn nil ) +then it stores +.code nil +into the place, and returns +.codn t . +Otherwise it leaves +.meta place +unchanged and returns +.codn nil . +.coNP Macro @ compare-swap +.synb +.mets (compare-swap < place < cmp-fun < cmp-val << store-val ) +.syne +.desc The -.meta setter-sym -and -.meta clobber-body -arguments similarly specify an optional clobber expander fragment, -as a single optional argument. If specified, the -.meta clobber-body -must generate a local function named using -.meta setter-sym -wrapped around -.meta body-sym -forms. +.code compare-swap +macro examines the value of +.meta place +and compares it to +.meta cmp-val +using the comparison function given by the function name +.metn cmp-fun . -The -.meta deleter-sym -and -.meta deleter-body -likewise specify a delete expander fragment. If this is omitted, -then the place shall not support deletion. +This comparison takes places as if by evaluating the expression +.meti >> ( cmp-fun < value << cmp-val ) +where +.meta value +denotes the current value of +.metn place . -.TP* "Example:" +If the comparison is false, +.meta place +is not modified, the +.meta store-val +expression is not evaluated, and the macro returns +.codn nil . -Implementation of the place denoting the -.code car -field of -.code cons -cells: +If the comparison is true, then +.code compare-swap +evaluates the +.meta store-val +expression, stores the resulting value into +.meta place +and returns +.codn t . -.verb - (defplace (car cell) body +.coNP Macros @ inc and @ dec +.synb +.mets (inc < place <> [ delta ]) +.mets (dec < place <> [ delta ]) +.syne +.desc +The +.code inc +macro increments +.meta place +by adding +.meta delta +to its value. +If +.meta delta +is missing, the value used in its place the integer 1. - ;; the update expander fragment - (getter setter - (with-gensyms (cell-sym) ;; temporary symbol for cell - ^(let ((,cell-sym ,cell)) ;; evaluate place to cell - ;; getter and setter access cell via temp var - (macrolet ((,getter () - ^(car ,',cell-sym)) - (,setter (val) - ^(sys:rplaca ,',cell-sym ,val))) +First the +.meta place +argument is evaluated as a syntactic place to determine the location. +Then, the value currently stored in that location is retrieved. +Next, the +.meta delta +expression is evaluated. Its value is added to the previously retrieved +value as if by the +.code + +function. The resulting value is stored in the place, and returned. - ;; insert body form from place update macro - ,body)))) +The macro +.code dec +works exactly like +.code inc +except that addition is replaced by subtraction. The similarly defaulted +.meta delta +value is subtracted from the previous value of the place. - ;; clobber expander fragment: simpler: no need - ;; to evaluate cell to temporary variable. - (ssetter - ^(macrolet ((,ssetter (val) - ^(sys:rplaca ,',cell ,val))) - ,body)) +.coNP Macros @ pinc and @ pdec +.synb +.mets (pinc < place <> [ delta ]) +.mets (pdec < place <> [ delta ]) +.syne +.desc +The macros +.code pinc +and +.code pdec +are very similar to +.code inc +and +.codn dec . - ;; deleter: delegate to pop semantics: - ;; (del (car a)) == (pop a). - (deleter - ^(macrolet ((,deleter () ^(pop ,',cell))) - ,body))) -.brev +The only difference is that they return the previous value of +.meta place +rather than the incremented value. -.coNP Macro @ defset +.coNP Macros @ test-inc and @ test-dec .synb -.mets (defset < name < params < new-val-sym << set-form ) -.mets (defset < get-fun-sym << set-fun-sym ) +.mets (test-inc < place >> [ delta <> [ from-val ]]) +.mets (test-dec < place >> [ delta <> [ to-val ]]) .syne .desc The -.code defset -macro provides a mechanism for introducing a new kind of syntactic place. -It is simpler to use than -.code defplace -and more concise, but not as general. - -The -.code defset -macro is designed for situations in which a function or macro which evaluates -all of its arguments is required to serve as a syntactic place. -It provides two flavors of syntax: the long form, indicated by giving -.code defset -five arguments, and a short form, which uses two arguments. - -In the long form of -.codn defset , -the syntactic place is described by -.meta name +.code test-inc and -.metn params . -The -.code defset -form expresses the request that call to the function or operator named -.meta name -is to be treated as a syntactic place, which has arguments described by -the parameter list -.metn params . +.code test-dec +macros provide combined operations which change the value of a place and +provide a test whether, respectively, a certain previous value was +overwritten, or a certain new value was attained. By default, this tested +value is zero. The -.meta new-val-sym -parameter is the name of a symbol which will be bound to -an expression which calculates the new value being stored into -the syntactic place. This is intended to be referenced in the -.meta set-form -only, which should ensure that the expression that -.meta new-val-sym -holds is evaluated only once. +.code test-inc +macro notes the prior value of +.meta place +and then updates it with that value, plus +.metn delta , +which defaults to 1. If the prior value is +.code eql +to +.meta from-val +then it returns +.codn t , +otherwise +.codn nil . +The default value of +.meta from-val +is zero. The -.meta set-form -argument specifies an expression which generates the code for storing a new -value to the place. +.code test-dec +macro produces a new value by subtracting +.meta delta +from the value of +.metn place . +The argument +.meta delta +defaults to 1. The new value is stored into +.metn place . +If the new value is +.code eql +to +.meta to-val +then +.code t +is returned, otherwise +.codn nil . +.coNP Macro @ swap +.synb +.mets (swap < left-place << right-place ) +.syne +.desc The -.code defset -macro makes the necessary arrangements such that when an operator form -named by -.meta name -is treated as a syntactic place, then at macro-expansion time, code is -generated to evaluate all of its argument expressions into machine-generated -variables. The names of those variables are automatically bound to the -corresponding symbols given in the -.meta params -argument list of the -.code defset -syntax. Code is also generated to evaluate the expression which gives the -new value to be stored, and that is bound to a generated variable whose -name is bound to the -.code new-val-sym -symbol. Then arrangements are made to invoke the operator named by -.meta name -and to evaluate the -.code set-form -in an environment in which these symbol bindings are visible. -The operator named -.meta name -is invoked using an altered argument list which uses temporary symbols in place -of the original expressions. The task of -.code set-form -is to insert the values of the symbols from -.meta params +.code swap +macro exchanges the values of +.meta left-place and -.meta new-val-sym -into a suitable code templates that will perform the store actions. -The code generated by -.code set-form -must also take on the responsibility of yielding the new value as its result. - -If -.meta params -list contains optional parameters, the default value expressions of those -parameters shall be evaluated in the scope of the -.code defset -definition. +.meta right-place +and returns the value which is thereby transferred to +.metn right-place . -The -.meta params -list may specify a rest parameter. In the expansion, this parameter will -capture a list of temporary symbols, corresponding to the list of variadic -argument expressions. For instance if the -.code defset -parameter list for a place -.code g -is -.codn "(a b . c)" , -featuring the rest parameter -.codn c , -and its -.meta set-form -is -.code "^(s ,a ,b ,*c)" -and the place is invoked as -.code "(g (i) (j) (k) (l))" -then parameter -.code c -will be bound to a list of gensyms such as -.code "(#:g0123 #:g0124)" -so that the evaluation of -.meta set-form -will yield syntax resembling -.codn "(s #:g0121 #:g0122 #:g0123 #:g0124)" . -Here, gensyms -.code #:g0123 -and -.code #:g0124 -are understood to be bound to the values of the expressions -.code (k) +First, +.meta left-place and -.codn (l) , -the two trailing parameters corresponding to the rest parameter -.codn c . - -Syntactic places defined by -.code defset -that have a rest parameter may be invoked with improper syntax such as -.codn "(set (g x y . z) v)" . -In this situation, that rest parameter will be bound to the name of -a temporary variable which holds the value of -.code z -rather than to a list of temporary variable names holding the values -of trailing expressions. -The -.code set-form -must be prepared for this situation. In particular, the rest parameter's value -is an atom, then it cannot be spliced in the backquote syntax, except at the -last position of a list. - -Although syntactic places defined by -.code defset -perform macro-parameter-like destructuring of the place form, binding -unevaluated argument expressions to the parameter symbols, -nested macro parameter lists are not supported: -.meta params -specifies a function parameter list. - -The parameter list may use parameter macros, keeping in mind that -the parameter expansion is applied at the time the -.code defset -form is processed, specifying an expanded parameter list which -receives unevaluated expressions. The -.meta set-form -may refer to all symbols produced by parameter list expansion, other -than generated symbols. For instance, if a parameter list macro -.code :addx -exists which adds the parameter symbol -.code x -to the parameter list, and this -.code :addx -is invoked in the -.meta params -list of a -.codn defset , -then -.code x -will be visible to the -.metn set-form . - -The short, two-argument form of -.code defset -simply specifies the names of two functions or operators: -.code get-fun-sym -names the operator which accesses the place, and -.code set-fun-sym -names the operator which stores a new value into the place. -It is expected that all arguments of these operators are evaluated -expressions, and that the store operator takes one argument more -than the access operator. The operators are otherwise assumed to be -variadic: each instance of a place based on -.code get-fun-sym -individually determines how many arguments are passed to that operator -and to the one named by -.codn set-fun-sym . +.meta right-place +are evaluated, in that order, to determine their locations. +Then the prior values are retrieved, exchanged and stored back. +The value stored in +.meta right-place +is also returned. -The definition -.code "(defset g s)" -means that -.code "(inc (g x y))" -will generate code which ensures that -.code x -and -.code y -are evaluated exactly once, and then those two values are passed as -arguments to -.code g -which returns the current value of the place. That value is then incremented -by one, and stored into the place by calling the -.code s -function/operator with three arguments: the two values that were passed to -.code g -and the new value. The exact number of arguments is determined by each -individual use of -.code g -as a place; the -.code defset -form doesn't specify the arity of -.code g +If +.meta left-place and -.codn s , -only that -.code s -must accept one more argument relative to -.codn g . +.meta right-place +are ranges of the same sequence, the behavior is not specified +if the ranges overlap or are of unequal length. -The following equivalence holds between the short and long forms: +Note: the +.code rotate +macro's behavior is somewhat more specified in this regard. +Thus, although any correct +.code swap +expression can be expressed using +.codn rotate , +but the reverse isn't true. -.verb - (defset g s) <--> (defset g (. r) n ^(g ,*r) ^(s ,*r ,n)) -.brev +.coNP Macro @ push +.synb +.mets (push < item << place ) +.syne +.desc +The +.code push +macro places +.meta item +at the head of the list stored in +.meta place +and returns the updated list which is stored back in +.metn place . -Note: -the short form of -.code defset -is similar to the -.code define-accessor -macro. +First, the expression +.meta item +is evaluated to produce the push value. +Then, +.meta place +is evaluated to determine its storage location. +Next, the storage location is accessed to retrieve the +list value which is stored there. A new object is +produced as if by invoking +.code cons +function on the push value and list value. +This object is stored into the location, +and returned. -.TP* "Example:" +.coNP Macro @ pop +.synb +.mets (pop << place ) +.syne +.desc +The +.code pop +macro removes an element from the list stored in +.meta place +and returns it. -Implementation of +First, +.meta place +is evaluated to determine the place. The place is accessed to +retrieve the original value. Then a new value is calculated, +as if by applying the +.code cdr +function to the old value. This new value is stored. +Finally, a return value is calculated and returned, as if by applying the .code car -as a syntactic place using a long form -.codn defset : - -.verb - (defset car (cell) new - (let ((n (gensym))) - ^(rlet ((,n ,new)) - (progn (rplaca ,cell ,n) ,n)))) -.brev - -Given such a definition, the expression -.code "(inc (car (abc)))" -expands to code closely resembling: - -.verb - (let ((#:g0048 (abc))) - (let ((#:g0050 (succ (car #:g0048)))) - (rplaca #:g0048 #:g0050) - #:g0050)) -.brev +function to the original value. +.coNP Macro @ pushnew +.synb +.mets (pushnew < item < place >> [ testfun <> [ keyfun ]]) +.syne +.desc The -.code defset -macro has arranged for the argument expression -.code (abc) -of -.code car -to be evaluated to a temporary variable -.codn #:g0048 , -a -.codn gensym . -This, then, holds the +.code pushnew +macro inspects the list stored in +.metn place . +If the list already contains the item, then +it returns the list. Otherwise it creates a new list +with the item at the front and stores it back +into +.metn place , +and returns it. + +First, the expression +.meta item +is evaluated to produce the push value. +Then, +.meta place +is evaluated to determine its storage location. +Next, the storage location is accessed to retrieve the +list value which is stored there. The list is +inspected to check whether it already contains the push +value, as if using the +.code member +function. If that is the case, the list +is returned and the operation finishes. +Otherwise, a new object is +produced as if by invoking .code cons -cell being operated on. -At macro-expansion time, the variable -.code cell -from the parameter list specified by the -.code defset -is bound to this symbol. The access expression -.code "(car #:0048)" -to retrieve the prior value is automatically generated -by combining the name of the place -.code car -with the gensym to which its argument -.code (abc) -has been evaluated. -The -.code new -variable was bound to the expression giving the new value, namely -.codn "(succ (car #:g0048))" . -The -.meta set-form -is careful to evaluate this only one time, storing its value into -the temporary variable -.codn #:g0050 , -referenced by the variable -.codn n . -The -.metn set-form 's -.code "(rplaca ,cell ,n)" -fragment thus turned into -.code "(rplaca #:g0048 #:g0050)" -where -.code #:g0048 -references the cons cell being operated on, and -.code #:g0050 -the calculated new value to be stored into its -.code car -field. -The -.meta set-form -is careful to arrange for the new value -.code #:g0050 -to be returned. Those place-mutating operators which yield the new value, such -as -.code set -and -.code inc -rely on this behavior. +function on the push value and list value. +This object is stored into the location +and returned. -.coNP Macro @ define-place-macro +.coNP Macro @ shift .synb -.mets (define-place-macro < name < macro-style-params -.mets \ \ << body-form *) +.mets (shift << place + << shift-in-value) .syne .desc -In some situations, an equivalence exists between two forms, only one -of which is recognized as a place. The -.code define-place-macro -macro can be used to establish a form as a place in terms of a translation to -an equivalent form which is already a place. - The -.code define-place-macro -has the same syntax as -.codn defmacro . -It specifies a macro transformation for a compound form which has the -.meta name -symbol in its leftmost position. +.code shift +macro treats one or more places as a "multi-place shift register". +The values of the places are shifted one place to the left. +The first (leftmost) place receives the value of the second place, +the second receives that of the third, and so on. +The last (rightmost) place receives +.meta shift-in-value +(which is not treated as a place, even if it is a syntactic place form). +The previous value of the first place is returned. -Place macro expansion doesn't use an environment; place macros are in a single -global namespace, special to place macros. There are no lexically scoped place -macros. Such an effect can be achieved by having a place macro expand to -an a form which is the target of a global or local macro, as necessary. +More precisely, all of the argument forms are evaluated left to right, in the +process of which the storage locations of the places are determined, +.meta shift-in-value +is reduced to its value. -To support place macros, forms which are used as syntactic places are subject -to a modified macro-expansion algorithm: -.RS -.IP 1. -If a place macro exists for a form that is being used as a place, then the -that place macro is invoked to expand the -form, and the expansion is taken in place of the original form. This process -repeats until the form can no longer be expanded as a place macro, or -the place macro declines to expand the form by returning the unexpanded -input. -.IP 2. -A form that has been fully expanded as a place macro is then subject -to a single-round of macro-expansion, as if by -.codn macroexpand-1 , -which takes place in the original form's lexical environment. -If the form doesn't expand, or the result of expansion is -.code nil -or a -non-symbolic atom, then the process terminates. Otherwise, the -process is repeated from step 1. -.RE +The values stored in the places are sampled and saved. -.IP -The -.code define-place-macro -macro does not cause -.meta name -to become -.codn mboundp . +Note that it is not specified whether the places are sampled in a separate +pass after the evaluation of the argument forms, or whether the +sampling is interleaved into the argument evaluation. This affects +the behavior in situations in which the evaluation of any of the +.meta place +forms, or of +.metn shift-in-value , +has the side effect of modifying later places. -There can exist both an ordinary macro and a place macro of the same name. -In this situation, when the macro call appears as a place form, it is -expanded as a place macro, according to the above steps. When the macro call -appears as an evaluated form, not being used as a place, the form is -expanded using the ordinary macro. +Next, the places are updated by storing the saved value of the second +place into the first place, the third place into the second and so forth, +and the value of +.meta shift-in-value +into the last place. -.TP* "Example:" +Finally, the saved original value of the first place is returned. -Implementation of -.code first -in terms of -.codn car : +If any of the places are ranges which index into the same sequence, +and the behavior is not otherwise unspecified due to the issue +noted in an earlier paragraph, the effect upon the multiply-stored +sequence can be inferred from the above-described storage order. +Note that even if stores take place which change the length of +the sequence and move some elements, not-yet-processed stores whose ranges +to refer to these elements are not adjusted. -.verb - (define-place-macro first (obj) - ^(car ,obj)) -.brev +With regard to the foregoing paragraph, a recommended practice is +that if subranges of the same sequence object are shifted, they be +given to the macro in ascending order of starting index. Furthermore, the +semantics is simpler if the ranges do not overlap. -.coNP Macro @ rlet +.coNP Macro @ rotate .synb -.mets (rlet >> ({( sym << init-form )}*) << body-form *) +.mets (rotate << place *) .syne .desc -The macro -.code rlet -is similar to the -.code let -operator. It establishes bindings for one or more -.metn sym -s, -which are initialized using the values of -.metn init-form -s. - -Note that the simplified syntax for a variable which initializes to +Treats zero or more places as a "multi-place rotate register". +If there are no arguments, there is no effect and .code nil -by default is not supported by -.codn rlet ; -that is to say, the syntax -.meta sym -cannot be used in place of the -.meti >> ( sym << init-form ) -syntax when -.meta sym -is to be initialized to -.codn nil . +is returned. Otherwise, the last (rightmost) place receives +the value of the first (leftmost) place. The leftmost place +receives the value of the second place, and so on. +If there are two arguments, this equivalent to +.codn swap . +The prior value of the first place, which is the the value +rotated into the last place, is returned. -The -.code rlet -macro differs from -.code let -in that -.code rlet -assumes that those -.metn sym -s -whose -.metn init-form -s, -after macro expansion, -are constant expressions -(according to the -.code constantp -function) may be safely implemented as a symbol macro rather than a lexical -variable. +More precisely, the +.meta place +arguments are evaluated left to right, +and the storage locations are thereby determined. The storage +locations are sampled, and then the sampled values are +stored back into the locations, but rotated by one place +as described above. The saved original value of the leftmost +.meta place +is returned. -Therefore -.code rlet -is suitable in situations in which simpler code is desired from the output -of certain kinds of machine-generated code, which binds local symbols: -code with fewer temporary variables. +It is not specified whether the sampling of the original values +is a separate pass which takes place after the arguments +are evaluated, or whether this sampling it is interleaved into argument +evaluation. This affects +the behavior in situations in which the evaluation of any of the +.meta place +forms has the side effect of modifying the value stored in +a later +.meta place +form. -On the other hand, -.code rlet -is not suitable in some situations when true variables are required, which -are assignable, and provide temporary storage. +If any of the places are ranges which index into the same sequence, +and the behavior is not otherwise unspecified due to the issue +noted in the preceding paragraph, the effect upon the multiply-stored +sequence can be inferred from the above-described storage order. +Note that even if stores take place which change the length of +the sequence and move some elements, not-yet-processed stores whose ranges +to refer to these elements are not adjusted. -.TP* "Example:" +With regard to the foregoing paragraph, a recommended practice is +that if subranges of the same sequence object are shifted, they be +given to the macro in ascending order of starting index. Furthermore, the +semantics is simpler if the ranges do not overlap. -.verb - ;; WRONG! Real storage location needed. - (rlet ((flag nil)) - (flip flag)) ;; error: flag expands to nil +.coNP Macro @ del +.synb +.mets (del << place ) +.syne +.desc +The +.code del +macro requests the deletion of +.codn place . +If +.code place +doesn't support deletion, an exception is thrown. - ;; Demonstration of constant-propagation - (let ((a 42)) - (rlet ((x 1) - (y a)) - (+ x y))) --> 43 +First +.code place +is evaluated, thereby determining its location. +Then the place is accessed to retrieve its value. +The place is then subject to deletion. Finally, the +previously retrieved value is returned. - (expand - '(let ((a 42)) - (rlet ((x 1) - (y a)) - (+ x y)))) --> (let ((a 42)) - (let ((y a)) - (+ 1 y))) -.brev +Precisely what deletion means depends on the kind of place. +The built-in places in \*(TL have deletion semantics which are +intended to be unsurprising to the programmer familiar with the +data structure which holds the place. -The last example shows that the -.code x -variable has disappeared in the expansion. The -.code rlet -macro turned it into into a -.code symacrolet -denoting the constant 1, which then propagated to the use site, -turning the expression -.code "(+ x y)" -into -.codn "(+ 1 y)" . +Generally, if a place denotes the element of a sequence, then deletion of the +place implies deletion of the element, and deletion of the element implies that +the gap produced by the element is closed. The deleted element is effectively +replaced by its successor, that successor by its successor and so on. If a +place denotes a value stored in a dynamic data set such as a hash table, +then deletion of that place implies deletion of the entry which holds +that value. If the entry is identified by a key, that key is also removed. -.coNP Macro @ slet +.coNP Macro @ lset .synb -.mets (slet >> ({( sym << init-form )}*) << body-form *) +.mets (lset <> { place }+ << sequence-expr ) .syne .desc -The macro -.code slet -a weaker form of the -.code rlet -macro. Just like -.codn rlet , -.code slet -reduces bindings initialized by constant expressions -to symbol macros. In addition, unlike -.codn rlet , -.code slet -also reduces to symbol macros those bindings which -are initialized by symbol expressions (values of variables). +The +.code lset +operator's parameter list consists of one or more places followed +by an expression +.metn sequence-expr . -.coNP Macro @ alet -.synb -.mets (alet >> ({( sym << init-form )}*) << body-form *) -.syne -.desc -The macro -.code alet -("atomic" or "all") is a stronger form of the -.code slet -macro. All bindings initialized by constant expressions are -turned to symbol macros. Then, if all of the remaining bindings are -all initialized by symbol expressions, they are also turned to -symbol macros. Otherwise, none of the remaining bindings -are turned to symbol macros. +The macro evaluates +.codn sequence-expr , +which is expected to produce a sequence. -The -.code alet -macro can be used even in situations when it is possible that the initializing -forms the variables may have side effects through which they affect each -others' evaluations. In this situation -.code alet -still propagates constants via symbol macros, and can eliminate the -remaining temporaries if they can all be made symbol macros for -existing variables: i.e. there doesn't exist any initialization form -with interfering side effects. +Successive elements of the resulting list are then assigned to each +successive +.codn place . -.coNP Macro @ define-accessor +If there are fewer elements in the sequence than places, the +unmatched places receive the value +.codn nil . + +Excess elements in the sequence are ignored. + +An error exception occurs if the sequence is an improper list with fewer +elements than places. + +A +.code lset +form produces the value of +.meta sequence-expr +as its result value. + +.coNP Macro @ upd .synb -.mets (define-accessor < get-function << set-function ) +.mets (upd < place << opip-arg *) .syne .desc The -.code define-accessor -macro is used for turning a function into an accessor, -such that forms which call the function can be treated -as places. - -Arguments to -.code define-accessor -are two symbols, which must name functions. When the -.code define-accessor -call is evaluated, the -.meta get-function -symbol is registered as a syntactic place. Stores to the -place are handled via calls to -.metn set-function . +.code upd +macro evaluates +.meta place +and passes the value as an argument to the operational pipeline +function formed, +as if by the +.code opip +macro, from the +.meta opip-arg +arguments. The result of this function is then stored back into +.metn place . -If -.meta get-function -names a function which takes N -arguments, -.meta set-function -must name a function which takes N+1 arguments. +The following equivalence holds, except that place +.code p +is evaluated only once: -Moreover, in order for the accessor semantics to be correct -.meta set-function -must treat its rightmost argument as the value being stored, -and must also return that value. +.verb + (upd p x y z ...) <--> (set p (call (opip x y z ...) p)) +.brev -When a function call form targeting -.meta get-function -is treated as a place which is subject -to an update operation (for instance an increment via the -.code inc -macro), -the accessor definition created by -.code define-accessor -ensures that the arguments of -.meta get-function -are evaluated only once, even though the update involves -a call to -.meta get-function +.SS* User-Defined Places and Place Operators +\*(TL provides a number of place-modifying operators such as +.codn set , +.codn push , and -.meta set-function -with the same arguments. The argument forms are evaluated to -temporary variables, and these temporaries are used as the -arguments in the calls. - -No other assurances are provided by -.codn define-accessor . +.codn inc . +It also provides a variety of kinds of syntactic places +which may be used with these operators. -In particular, if -.meta get-function -and -.meta set-function -internally each perform some redundant calculation over their arguments, -this cannot be optimized. Moreover, if that calculation has a visible effect, -that effect is observed multiple times in an update operation. +Both of these categories are open-ended: \*(TL programs may extend +the set of place-modifying operators, as well as the vocabulary of +forms which are recognized as syntactic places. -If further optimization or suppression of multiple effects is required, -the more general -.code defplace -macro must be used to define the accessor. It may also be possible to -treat the situation in a satisfactory way using a -.code define-place-macro -definition, which effectively then supplies inline code whenever a certain form -is used as a place, and that code itself is treated as a place. +Regarding place operators, it might seem obvious that new place operators can +be developed, since they are macros, and macros can expand to uses +of existing place operators. As an example, it may seem that +.code inc +operator could be written as a macro which uses +.codn set : -Note: -.code define-accessor -is very similar to the short form of -.codn defset . +.verb + (defmacro new-inc (place : (delta 1)) + ^(set ,place (+ ,place ,delta))) +.brev -.coNP Special variables @, *place-update-expander* @ *place-clobber-expander* and @ *place-delete-expander* -.desc -These variables hold hash tables, by means of which update expanders, -clobber expanders and delete expanders are registered, as associations -between symbols and functions. +However, the above +.code new-inc +macro has a problem: the +.code place +argument form is inserted into two places in the expansion, which +leads to two evaluations. This is visibly incorrect if the place +form contains any side effects. It is also potentially inefficient. -If -.code "[*place-update-expander* 'sym]" -yields a function, then symbol -.code sym -is the basis for a syntactic place. If the expression yields -.codn nil , -then forms beginning with -.code sym -are not syntactic places. (The situation of a clobber accessor or delete -accessor being defined without an update expander is improper). +\*(TL provides a framework for writing place update macros which +evaluate their argument forms once, even if they have to access +and update the same places. -.coNP Special variable @ *place-macro* -.desc -The -.code *place-macro* -special variable holds the hash table of associations between -symbols and place macro expanders. +The framework also supports the development of new kinds of place forms +as capsules of code which introduce the right kind of material into +the lexical environment of the body of an update macro, to enable +this special evaluation. -If the expression -.code "[*place-macro* 'sym]" -yields a function, then symbol -.code sym -has a binding as a place macro. If that -expression yields -.codn nil , -then there is no such binding: compound forms beginning with -.code sym -do not undergo place macro expansion. +.NP* Place-Expander Functions +The central design concept in \*(TL syntactic places are +.IR "place-expander functions" . +Each compound place is defined by up to three place-expander functions, +which are associated with the place via the leftmost operator +symbol of the place form. One place-expander, the +.IR "update expander" , +is mandatory. Optionally, a place may also provide a +.I "clobber expander" +as well as a +.IR "delete expander" . +An update expander provides the expertise for evaluating a place form once +in its proper run-time context to determine its actual run-time storage +location, and to access and modify the storage location. +A clobber expander provides an optimized mechanism for uses that perform +a one-time store to a place without requiring its prior value. +If a place definition does not supply a clobber expander, then the syntactic +places framework uses the update expander to achieve the functionality. +A delete expander provides the expertise for determining the actual run-time +storage location corresponding to a place, and obliterating it, +returning its prior value. If a place does not supply a delete expander, then +the place does not support deletion. Operators which require deletion, such as +.code del +will raise an error when applied to that place. -.SS* Quasiquote Operator Syntax -.coNP Macro @ qquote +The expanders operate independently, and it is expected that place-modifying +operators choose one of the three, and use only that expander. For example, +accessing a place with an update expander and then overwriting its value +with a clobber expander may result in incorrect code which contains multiple +evaluations of the place form. + +The programmer who implements a new place does not write expanders directly, +but rather defines them via the +.codn defplace , +.code define-accessor +or +.code defset +macro. + +The programmer who implements a new place update macro likewise does not +call the expanders directly. Usually, they are invoked via the macros +.codn with-update-expander , +.code with-clobber-expander +and +.codn with-delete-expander . +These are sufficient for most kind of macros. +In certain complicated cases, expanders may be invoked using the wrapper +functions +.codn call-update-expander , +.code call-clobber-expander +and +.codn call-delete-expander . +These convenience macros and functions perform certain common chores, like +macro-expanding the place in the correct environment, and choosing the +appropriate function. + +The expanders are described in the following sections. + +.NP* The Update Expander .synb -.mets (qquote << form ) +.mets (lambda >> ( getter-sym < setter-sym < place-form +.mets \ \ \ \ \ \ \ \ << body-form ) ...) .syne .desc -The -.code qquote -(quasi-quote) macro operator implements a notation for convenient -list construction. If -.meta form -is an atom, or a list structure which -does not contain any -.code unquote -or -.code splice -operators, then -.mono -.meti (qquote << form ) -.onom -is equivalent to -.mono -.meti (qquote << form ). -.onom +The update expander is a code-writer. It takes a +.meta body-form +argument, representing code, and returns a larger form which surrounds +this code with additional code. -If -.metn form , -however, is a list structure which contains -.code unquote -or -.code splice -operators, then the substitutions implied by those operators are performed -on -.metn form , -and the -.code qquote -operator returns the resulting structure. +This larger form returned by the update expander can be regarded as having two +abstract actions, when it is substituted and evaluated in the context where +.meta place-form +occurs. The first abstract action is to evaluate +.meta place-form +exactly one time, in order to determine the actual run-time location to which +that form refers. +The second abstract action is to evaluate the caller's +.metn body-form -s, +in a lexical environment in which bindings exist for some lexical +functions or (more usually) lexical macros. These lexical macros +are explicitly referenced by the +.metn body-form ; +the update expander just provides their definition, under the names +it is given via the +.meta getter-sym +and +.meta setter-sym +arguments. -Note: how the qquote operator actually works is that it is compiled into -code. It becomes a Lisp expression which, when evaluated, computes the -resulting structure. +The update expander writes local functions or macros under these names: a +getter function and a setter function. Usually, update expanders write +macros rather than functions, possibly in combination with some lexical +anonymous variables which hold temporary objects. Therefore the getter +and setter are henceforth referred to as macros. -A -.code qquote -can contain another -.codn qquote . -If an -.code unquote -or -.code splice -operator occurs -within a nested -.codn qquote , -it belongs to that -.codn qquote , -and not to the outer one. +The code being generated is with regard to some concrete instance of +.metn place-form . +This argument is the actual form which occurs in a program. For +instance, the update expander for the +.code car +place might be called with an arbitrary variant of the +.meta place-form +which might look like +.codn "(car (inc (third some-list)))" . -However, an unquote operator which occurs inside another one belongs one level -higher. For instance in +In the abstract semantics, upfront code wrapped around the +.meta body-form +by the update expander provides the logic to evaluate this place to +a location, which is retained in some hidden local context. -.verb - (qquote (qquote (unquote (unquote x)))) -.brev +The getter local macro named by +.meta getter-sym +must provide the logic for retrieving the value of this place. +The getter macro takes no arguments. +The +.meta body-form +makes free use of the getter function; they may call it multiple times, +which must not trigger multiple evaluations of the original place +form. -the leftmost -.code qquote -belongs with the rightmost unquote, and the inner -.code qquote -and -.code unquote -belong together. When the outer -.code qquote -is evaluated, -it will insert the value of -.codn x , -resulting in the object -.codn "(qquote (unquote [value-of-x]))" . -If this resulting qquote value is evaluated again as Lisp syntax, then it will -yield -.codn [value-of-value-of-x] , -the value of -.code [value-of-x] -when treated as a Lisp expression and evaluated. +The setter local macro named by +.meta setter-sym +must generate the logic for storing a new value into the once-evaluated +version of +.metn place-form . +The setter function takes exactly one argument, whose +value specifies the value to be stored into the place. +It is the caller's responsibility to ensure that the +argument form which produces the value to be stored via the setter is evaluated +only once, and in the correct order. The setter does not concern itself with +this form. Multiple calls to the setter can be expected to result in multiple +evaluations of its argument. Thus, if necessary, the caller must supply the code +to evaluate the new value form to a temporary variable, and then pass the +temporary variable to the setter. This code can be embedded in +the +.meta body-form +or can be added to the code returned by a call to the update expander. -.TP* Examples: +The setter local macro or function must return the new value which is stored. +That is to say, when +.meta body-form +invokes this local macro or function, it may rely on it yielding the +new value which was stored, as part of achieving its own semantics. -.verb - (qquote a) -> a +The update expander does not macro-expand +.codn place-form . +It is assumed that the expander is invoked in such a way that the +place has been expanded in the correct environment. In other words, the +form matches the type of place which the expander handles. +If the expander had to macro-expand the place form, it would sometimes have +to come to the conclusion that the place form must be handled by a different +expander. No such consideration is the case: when an expander is called on +a form, that is final; it is certain that it is the correct expander, which +matches the symbol in the +.code car +position of the form, which is not a macro in the context where it occurs. - (qquote (a b c)) -> (a b c) +An update expander is free to assume that any place which is stored +(the setter local macro is invoked on it) is accessed at least once by +an invocation of the getter. A place update macro which relies on an update +expander, but uses only the store macro, might not work properly. +An example of an update expander which relies on this assumption is the +expander for the +.mono +.meti (force << promise ) +.onom +place type. If +.meta promise +has not yet been forced, and only the setter is used, then +.meta promise +might remain unforced as its internal value location is updated. +A subsequent access to the place will incorrectly trigger a force, +which will overwrite the value. The expected behavior is that storing +a value in an unforced +.code force +place changes the place to forced state, preempting the evaluation of +the delayed form. Afterward, the promise exhibits the value which was +thus assigned. - (qquote (1 2 3 (unquote (+ 2 2)) (+ 2 3))) -> (1 2 3 4 (+ 2 3)) +The update expander is not responsible for all issues of evaluation order. A +place update macro may consist of numerous places, as well as numerous +value-producing forms which are not places. Each of the places can provide its +registered update expander which provides code for evaluating just that place, +and a means of accessing and storing the values. The place update macro must +call the place expanders in the correct order, and generate any additional code +in the correct order, so that the macro achieves its required documented +evaluation order. - (qquote (unquote (+ 2 2))) -> 4 -.brev +.TP* "Example Update Expander Call:" -In the second-to-last example, the -.code "1 2 3" -and the -.code "(+ 2 3)" -are quoted verbatim. -Whereas the -.code "(unquote (+ 2 2))" -operator caused the evaluation of -.code "(+ 2 2)" -and the substitution of the resulting value. +.verb + ;; First, capture the update expander + ;; function for (car ...) places + ;; in a variable, for clarity. -The last example shows that -.meta form -can itself (the entire argument of -.codn qquote ) -can be an unquote operator. -However, note: -.code "(quote (splice form))" -is not valid. + (defvar car-update-expander [*place-update-expander* 'car]) -Note: a way to understand the nesting behavior is a via a possible model of -quasi-quote expansion which recursively compiles any nested quasi quotes first, -and then treats the result of their expansion. For instance, in the processing -of + ;; Next, call it for the place (car [a 0]). + ;; The body form specifies logic for + ;; incrementing the place by one and + ;; returning the new value. -.verb - (qquote (qquote (unquote (unquote x)))) -.brev + (call car-update-expander 'getit 'setit '(car [a 0]) + '(setit (+ (getit) 1))) -the -.code qquote -operator first encounters the -embedded -.code "(qquote ...)" -and compiles it to code. During that recursive -compilation, the syntax -.code "(unquote (unquote x))" -is encountered. The inner quote -processes the outer unquote which belongs to it, and the inner -.code "(unquote x)" -becomes material that is embedded verbatim in the compilation, which will then -be found when the recursion pops back to the outer quasiquote, which will -then traverse the result of the inner compilation and find the -.codn "(unquote x)" . + ;; --> Resulting code: -.TP* "Dialect note:" + (rlet ((#:g0032 [a 0])) + (macrolet ((getit nil + (append (list 'car) (list '#:g0032))) + (setit (val) + (append (list 'sys:rplaca) + (list '#:g0032) (list val)))) + (setit (+ (getit) 1)))) -In Lisp dialects which have a published quasiquoting operator syntax, there is -the expectation that the quasiquote read syntax corresponds to it. That is to -say, that for instance the read syntax -.code "^(a b ,c)" -is expected translated to -.codn "(qquote b (unquote c))" . + ;; Same expander call as above, with a call to expand added + ;; to show the fully expanded version of the returned code, + ;; in which the ;; setit and getit calls have disappeared, + ;; replaced by their macro-expansions. -In \*(TL, this is not true! Although -.code "^(b b ,c)" -is translated to a -quasiquoting macro, it is an internal one, not based on the public -.codn qquote , -.code unquote -and -.code splice -symbols being documented here. + (expand + (call car-update-expander 'getit 'setit '(car [a 0]) + '(setit (+ (getit) 1)))) -This idea exists for hygiene. The quasiquote read syntax is not confused -by the presence of the symbols -.codn qquote , -.code unquote -or -.code splice -in the template, since it doesn't treat them specially. + ;; --> Resulting code: -This also allows programmers to use the quasiquote read syntax to construct -quasiquote macros. For instance + (let ((#:g0032 [a 0])) + (sys:rplaca #:g0032 (+ (car #:g0032) 1))) -.verb - ^(qquote (unquote ,x)) ;; does not mean ^^,,x ! .brev +The main noteworthy points about the generated code are: +.RS +.IP - +the +.code "(car [a 0])" +place is evaluated by evaluating the embedded form +.code "[a 0]" +and storing storing the resulting object into a hidden local variable. +That's as close a reference as we can make to the +.code car +field. +.IP - +the getter macro expands to code which simply calls the +.code car +function on the cell. +.IP - +the setter uses a system function called +.codn sys:rplaca , +which differs from +.code rplaca +in that it returns the stored value, rather than the cell. +.RE -To the quasiquote reader, the -.code qquote -and -.code unquote -symbols mean nothing special, -and so this syntax simply means that if the value of -.code x -is -.codn foo , -the result of evaluating this expression will be -.codn "(qquote (unquote foo))" . +.NP* The Clobber Expander +.synb +.mets (lambda >> ( simple-setter-sym < place-form +.mets \ \ \ \ \ \ \ \ << body-form ) ...) +.syne +.desc +The clobber expander is a code-writer similar to the update expander. +It takes a +.meta body-form +argument, and returns a larger form which surrounds this form +with additional program code. -The form's expansion is actually this: +The returned block of code has one main abstract action. +It must arrange for the evaluation of +.meta body-form +in a lexical environment in which a lexical macro or lexical function +exists which has the name requested by the +.meta simple-setter-sym +argument. -.verb - (sys:qquote (qquote (unquote (sys:unquote x)))) -.brev +The simple setter local macro written by the clobber expander is similar to the +local setter written by the update expander. It has exactly the +same interface, performs the same action of storing a value into +the place, and returns the new value. -the -.code sys:qquote -macro recognizes -.code sys:unquote -embedded in the form, and -the other symbols not in the -.code sys: -package are just static template material. +The difference is that its logic may be considerably simplified by the +assumption that the place is being subject to exactly one store, +and no access. -The -.code sys:quote -macro and its associated -.code sys:unquote -and -.code sys:splice -operators work exactly like their ordinary counterparts. So in effect, \*(TX has -two nearly identical, independent quasi-quote implementations, one of which is -tied to the read syntax, and one of which isn't. This is useful for writing -quasiquotes which write quasiquotes. +A place update macro which uses a clobber expander, and calls it more than +once, break the assumption; doing so may result in multiple evaluations +of the +.metn place-form . -.coNP Operator @ unquote +.NP* The Delete Expander .synb -.mets (qquote (... (unquote << form ) ...)) -.mets (qquote (unquote << form )) +.mets (lambda >> ( deleter-sym < place-form +.mets \ \ \ \ \ \ \ \ << body-form ) ...) .syne .desc -The -.code unquote -operator is not an operator -.I per -.IR se . -The -.code unquote -symbol has no -binding in the global environment. It is a special syntax that is recognized -within a -.code qquote -form, to indicate forms within the quasiquote which are to be -evaluated and inserted into the resulting structure. +The delete expander is a code-writer similar to clobber expander. +It takes a +.meta body-form +arguments, and returns a larger form which surrounds this form +with additional program code. -The syntax -.mono -.meti (qquote (unquote << form )) -.onom -is equivalent to -.metn form : -the -.code qquote -and -.code unquote -"cancel out". +The returned block of code has one main abstract action. +It must arrange for the evaluation of +.meta body-form +in a lexical environment in which a lexical macro or lexical function +exists which has the name requested by the +.meta deleter-sym +argument. -.coNP Operator @ splice +The deleter macro written by the clobber expander takes no arguments. +It may be called at most once. It returns the previous value of the +place, and arranges for its obliteration, whatever that means for +that particular kind of place. + +.coNP Macro @ with-update-expander .synb -.mets (qquote (... (splice << form ) ...)) +.mets (with-update-expander >> ( getter << setter ) < place < env +.mets \ << body-form ) .syne .desc The -.code splice -operator is not an operator -.I per -.IR se . -The -.code splice -symbol has no -binding in the global environment. It is a special syntax that is recognized -within a -.code qquote -form, to indicate forms within the quasiquote which are to be -evaluated and inserted into the resulting structure. - -The syntax -.mono -.meti (qquote (splice << form )) -.onom -is not permitted and raises an exception if evaluated. The -.code splice -syntax must occur within a list, and not in the dotted position. - -The -.code splice -form differs from unquote in that -.mono -.meti (splice << form ) -.onom -requires that -.meta form -must evaluate to a list. That list is -integrated into the surrounding list. +.code with-update-expander +macro evaluates the +.meta body-form +argument, whose result is expected to be a Lisp form. +The macro adds additional code around this code, and the result is returned. +This additional code is called the +.IR "place-access code" . -.SS* Math Library -.coNP Functions @ + and @ - -.synb -.mets (+ << number *) -.mets (- < number << number *) -.mets (* << number *) -.syne -.desc The -.codn + , -.code - +.meta getter and -.code * -functions perform addition, subtraction and multiplication, -respectively. Additionally, the -.code - -function performs additive inverse. +.meta setter +arguments must be symbols. Over the evaluation of the +.metn body-form , +these symbols are bound to the names of local functions which +are provided in the place-access code. The -.code + -function requires zero or more arguments. When called with no -arguments, it produces 0 (the identity element for addition), otherwise it -produces the sum over all of the arguments. +.meta place +argument is a form which evaluates to a syntactic place. The generated +place-access code is based on this place. -Similarly, the -.code * -function requires zero or more arguments. When called -with no arguments, it produces 1 (the identity element for multiplication). -Otherwise it produces the product of all the arguments. +The +.meta env +argument is a form which evaluates to a macro-expansion-time environment. +The +.code with-update-expander +macro uses this environment to perform macro-expansion on the value of the +.meta place +form, to obtain the correct update expander function for the fully +macro-expanded place. -The semantics of -.code - -changes from subtraction to additive inverse -when there is only one argument. The argument is treated as a subtrahend, -against an implicit minuend of zero. When there are two or more -argument, the first one is the minuend, and the remaining are subtrahends. +The place-access code is generated by calling the update expander +for the expanded version of +.codn place . -When there are three or more operands, these operations are performed as if by -binary operations, in a left-associative way. That is to say, -.code "(+ a b c)" -means -.codn "(+ (+ a b) c)" . -The sum of -.code a -and -.code b -is computed first, and then this is added to -.codn c . -Similarly -.code "(- a b c)" -means -.codn "(- (- a b) c)" . -First, -.code b -is subtracted from -.codn a , -and then -.code c -is subtracted from that result. +.TP* "Example:" -The arithmetic inverse is performed as if it were subtraction from integer 0. -That is, -.code "(- x)" -means the same thing as -.codn "(- 0 x)" . +The following is an implementation of the +.code swap +macro, which exchanges the contents of two places. -The operands of -.codn + , -.code - -and -.code * -can be characters, integers (fixnum and bignum), and -floats, in nearly any combination. +Two places are involved, and, correspondingly, the +.code with-update-expander +macro is used twice, to add two instances of place-update code +to the macro's body. -If two operands have different types, then one of them is converted to the -type of the one with the higher rank, according to this ranking: -character < integer < float. For instance if one operand is integer, and the -other float, the integer is converted to a float. +.verb + (defmacro swap (place-0 place-1 :env env) + (with-gensyms (tmp) + (with-update-expander (getter-0 setter-0) place-0 env + (with-update-expander (getter-1 setter-1) place-1 env + ^(let ((,tmp (,getter-0))) + (,setter-0 (,getter-1)) + (,setter-1 ,tmp)))))) +.brev -.TP* Restrictions: +The basic logic for swapping two places is contained in the code template: -Characters are not considered numbers, and participate in these operations in -limited ways. Subtraction can be used to computed the displacement between the -Unicode values of characters, and an integer displacement can be added to a -character, or subtracted from a character. For instance -.codn "(- #\e9 #\e0) is 9" . -The Unicode value of a character -.code C -can be found using -.codn "(- C #\ex0)" : -the displacement from the NUL character. +.verb + ^(let ((,tmp (,getter-0))) + (,setter-0 (,getter-1)) + (,setter-1 ,tmp)) +.brev -The rules can be stated as a set of restrictions: -.RS -.IP 1 -Two characters may not be added together. -.IP 2 -A character may not be subtracted from an integer (which also rules out -the possibility of computing the additive inverse of a character). -.IP 3 -A character operand may not be opposite to a floating point operand -in any operation. -.IP 4 -A character may not be an operand of multiplication. -.RE +The temporary variable named by the +.code gensym +symbol +.code tmp +is initialized by calling the getter function for +.metn place-0 . +Then the setter function of +.meta place-0 +is called in order to store the value of +.meta place-1 +into +.metn place-0 . +Finally, the setter for +.meta place-1 +is invoked to store the previously saved temporary value into +that place. -.PP +The name for the temporary variable is provided by the +.code with-gensyms +macro, but establishing the variable is the caller's responsibility; +this is seen as an explicit +.code let +binding in the code template. -.coNP Function @ / -.synb -.mets (/ << divisor ) -.mets (/ < dividend << divisor *) -.syne -.desc -The -.code / -function performs floating-point division. Each operands is first -converted to floating-point type, if necessary. In the one-argument -form, the -.meta dividend -argument is omitted. An implicit dividend is present, whose value is -.codn 1.0 , -such that the one-argument form -.code "(/ x)" -is equivalent to the two-argument form -.codn "(/ 1.0 x)" . +The names of the getter and setter functions are similarly provided +by the +.code with-update-expander +macros. However, binding those functions is the responsibility of that +macro. To achieve this, it adds the place-access code to the code generated by +the +.code "^(let ...)" +backquote template. In the following example macro-expansion, the additional +code added around the template is seen. It takes the form of two +.code macrolet +binding blocks, each added by an invocation of +.codn with-update-expander : -If there are two or more arguments, explicitly or by the above equivalence, -then a cumulative division is performed. The -.meta divisor -value is taken into consideration, and divided by the first -.codn divisor . -If another -.code divisor -follows, then that value is divided by that subsequent divisor. -This process repeats until all divisors are exhausted, and the -value of the last division is returned. +.verb + (macroexpand '(swap a b)) -A division by zero throws an exception of type -.codn numeric-error . + --> -.coNP Functions @ sum and @ prod -.synb -.mets (sum < sequence <> [ keyfun ]) -.mets (prod < sequence <> [ keyfun ]) -.syne -.desc -The -.code sum -and -.code prod -functions operate on an effective sequence of numbers derived from -.metn sequence . + (macrolet ((#:g0036 () 'a) ;; getter macro for a + (#:g0037 (val-expr) ;; setter macro for a + (append (list 'sys:setq) (list 'a) + (list val-expr)))) + (macrolet ((#:g0038 () 'b) ;; getter macro for b + (#:g0039 (val-expr) ;; setter macro for b + (append (list 'sys:setq) (list 'b) + (list val-expr)))) + (let ((#:g0035 (#:g0036))) ;; temp <- a + (#:g0037 (#:g0038)) ;; a <- b + (#:g0039 #:g0035)))) ;; b <- temp +.brev -If the -.meta keyfun -argument is omitted, then the effective sequence is the -.meta sequence -argument itself. Otherwise, the effective sequence is understood to be -a projection mapping of the elements of -.meta sequence -through -.meta keyfun -as would be calculated by the +In this expansion, for example +.code #:g0036 +is the generated symbol which forms the value of the +.code getter-0 +variable in the +.code swap +macro. The getter is a macro which simply expands to a +.codn a : +straightforward access to the variable a. +Of course, +.code #:g0035 +is nothing but the value of the +.code tmp +variable. Thus the swap macro's .mono -.meti (mapcar < keyfun << sequence ) +^(let ((,tmp (,getter-0))) ...) +.onom +has turned into +.mono +^(let ((#:g0035 (#:g0036))) ...) .onom -expression. -The -.code sum -function returns the left-associative sum of the elements of -the effective sequence calculated as if using the -.code + -function. Similarly, the -.code prod -function calculates the left-associative product of the elements of -the sequence as if using the -.code * -function. +A full expansion, with the +.code macrolet +local macros expanded out: -If -.meta sequence -is empty then -.code sum -returns -.code 0 -and -.code prod -returns -.codn 1 . +.verb + (expand '(swap a b)) -If the effective sequence contains one number, then both functions -return that number. + --> -.coNP Functions @ wrap and @ wrap* + (let ((#:g0035 a)) + (sys:setq a b) + (sys:setq b #:g0035)) +.brev + +In other words, the original syntax +.mono +(,getter-0) +.onom +became +.mono +(#:g0036) +.onom +and finally just +.codn a . + +Similarly, +.mono +(,setter-0 (,getter-1)) +.onom +became the +.code macrolet +invocations +.mono +(#:g0037 (#:g0038)) +.onom +which finally turned into: +.codn "(sys:setq a b)" . + +.coNP Macro @ with-clobber-expander .synb -.mets (wrap < start < end << number ) -.mets (wrap* < start < end << number ) +.mets (with-clobber-expander <> ( simple-setter ) < place < env +.mets \ << body-form ) .syne .desc The -.code wrap -and -.code wrap* -functions reduce -.meta number -into the range specified by -.meta start -and -.metn end . +.code with-clobber-expander +macro evaluates +.metn body-form , +whose result is expected to be a Lisp form. The macro adds additional code +around this form, and the result is returned. This additional code is called +the +.IR "place-access code" . -Under -.code wrap -the range is inclusive of the -.meta end -value, whereas under -.code wrap* -it is exclusive. +The +.meta simple-setter +argument must be a symbol. Over the evaluation of the +.metn body-form , +this symbol is bound to the name of a functions which +are provided in the place-access code. -The following equivalence holds +The +.meta place +argument is a form which evaluates to a syntactic place. The generated +place-access code is based on this place. + +The +.meta env +argument is a form which evaluates to a macro-expansion-time environment. +The +.code with-clobber-expander +macro uses this environment to perform macro-expansion on the value of the +.meta place +form, to obtain the correct update expander function for the fully +macro-expanded place. + +The place-access code is generated by calling the update expander +for the expanded version of +.codn place . + +.TP* "Example:" + +The following implements a simple assignment statement, similar to +.code set +except that it only handles exactly two arguments: .verb - (wrap a b c) <--> (wrap* a (succ b) c) + (defmacro assign (place new-value :env env) + (with-clobber-expander (setter) place env + ^(,setter ,new-value))) .brev -The expression -.code "(wrap* x0 x1 x)" -performs the following calculation: - +Note that the correct evaluation order of +.code place +and +.code new-value +is taken care of, because +.code with-clobber-expander +generates the code which performs all the necessary evaluations of +.codn place . +This evaluation occurs before the code which is generated by .mono -.mets (+ (mod (- x x0) (- x1 x0)) x0) +^(,setter ,new-value) .onom +part is evaluated, and that code is what evaluates +.codn new-value . -In other words, first -.meta start -is subtracted from -.metn number . -Then the result is reduced modulo the displacement -between -.code start -and -.codn end . -Finally, -.meta start -is added back to that result, which is returned. +Suppose that a macro were desired which allows assignment to be notated in a right to left +style, as in: -.TP* Example: +.verb + (assign 42 a) ;; store 42 in variable a +.brev + +Now, the new value must be evaluated prior to the place, if left to right +evaluation order is to be maintained. The standard +.code push +macro has this property: the push value is on the left, and the place +is on the right. + +Now, the code has to explicitly take care of the order, like this: .verb - ;; perform ROT13 on the string "nop" - [mapcar (opip (+ 13) (wrap #\ea #\ez)) "nop"] -> "abc" + ;; WRONG! We can't just swap the parameters; + ;; place is still evaluated first, then new-value: + + (defmacro assign (new-value place :env env) + (with-clobber-expander (setter) place env + ^(,setter ,new-value))) + + ;; Correct: arrange for evaluation of new-value first, + ;; then place: + + (defmacro assign (new-value place :env env) + (with-gensym (tmp) + ^(let ((,tmp ,new-value)) + ,(with-clobber-expander (setter) place env + ^(,setter ,tmp))))) .brev -.coNP Functions @ gcd and @ lcm +.coNP Macro @ with-delete-expander .synb -.mets (gcd << number *) -.mets (lcm << number *) +.mets (with-delete-expander <> ( deleter ) < place < env +.mets \ << body-form ) .syne .desc The -.code gcd -function computes the greatest common divisor: the largest positive -integer which divides each -.metn number . +.code with-delete-expander +macro evaluates +.metn body-form , +whose result is expected to be a Lisp form. +The macro adds additional code +around this code, and the resulting code is returned. This additional code is +called the +.IR "place-access code" . The -.code lcm -function computes the lowest common multiple: the smallest positive -integer which is a multiple of -each -.metn number . +.meta deleter +argument must be a symbol. Over the evaluation of the +.metn body-form , +this symbol is bound to the name of a functions which +are provided in the place-access code. -Each -.meta number -must be an integer. +The +.meta place +argument is a form which evaluates to a syntactic place. The generated +place-access code is based on this place. -Negative integers are replaced by their absolute values, so -.code "(lcm -3 -4)" -is -.code 12 -and -.code "(gcd -12 -9)" -yields -.codn 3 . +The +.meta env +argument is a form which evaluates to a macro-expansion-time environment. +The +.code with-delete-expander +macro uses this environment to perform macro-expansion on the value of the +.meta place +form, to obtain the correct update expander function for the fully +macro-expanded place. -The value of -.code (gcd) -is -.code 0 -and that of -.code (lcm) -is 1 . +The place-access code is generated by calling the update expander +for the expanded version of +.codn place . -The value of -.code "(gcd x)" -and -.code "(lcm x)" -is -.codn "(abs x)" . +.TP* "Example:" -Any arguments of -.code gcd -which are zero are effectively ignored so that -.code "(gcd 0)" -and -.code "(gcd 0 0 0)" -are both the same as -.code (gcd) -and -.code "(gcd 1 0 2 0 3)" -is the same as -.codn "(gcd 1 2 3)" . +The following implements the +.code del +macro: -If -.code lcm -has any argument which is zero, it yields zero. +.verb + (defmacro del (place :env env) + (with-delete-expander (deleter) place env + ^(,deleter))) +.brev -.coNP Function @ divides +.coNP Function @ call-update-expander .synb -.mets (divides < d << n ) +.mets (call-update-expander < getter < setter < place < env +.mets \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ << body-form ) .syne .desc The -.code divides -function tests whether integer -.meta d -divides integer -.metn n . -If this is true, -.code t -is returned, otherwise -.codn nil . - -The integers 1 and -1 divide every other integer and themselves. -By established convention, every integer, except zero, divides zero. +.code call-update-expander +function provides an alternative interface for making use of an update +expander, complementary to +.codn with-update-expander . -For other values, -.meta d -divides -.meta n -if division of -.meta n -by -.meta d -leaves no remainder. +Arguments +.meta getter +and +.meta setter +are symbols, provided by the caller. These are passed to the update +expander function, and are used for naming local functions in the +generated code which the update expander adds to +.metn body-form . -.coNP Function @ abs -.synb -.mets (abs << number ) -.syne -.desc The -.code abs -function computes the absolute value of -.metn number . -If -.meta number -is positive, it is returned. If -.meta number -is negative, its additive inverse is -returned: a positive number of the same type with exactly the same magnitude. - -.coNP Function @ signum -.synb -.mets (signum << number ) -.syne -.desc +.meta place +argument is a place which has not been subject to macro-expansion. The -.code signum -function calculates a representation of the sign of -.meta number -as a numeric value. - -If -.meta number -is an integer, then -.code signum -returns -1 if the integer is negative, 1 if the integer is positive, -or else 0. +.code call-update-expander +function takes on the responsibility for macro-expanding the place. -If -.meta number -is a floating-point value then -.code signum -returns -1.0 if the value is negative, 1.0 if the value is positive or -else 0.0. +The +.meta env +parameter is the macro-expansion environment object required to +correctly expand +.code place +in its original environment. -.coNP Functions @, trunc @, floor @ ceil and @ round -.synb -.mets (trunc < dividend <> [ divisor ]) -.mets (floor < dividend <> [ divisor ]) -.mets (ceil < dividend <> [ divisor ]) -.mets (round < dividend <> [ divisor ]) -.syne -.desc The -.codn trunc , -.codn floor , -.code ceiling +.meta body-form +argument represents the source code of a place update operation. +This code makes references to the local functions whose names +are given by +.meta getter and -.code round -functions perform division of the -.meta dividend -by the -.metn divisor , -returning an integer quotient. +.metn setter . +Those arguments allow the update expander to write these functions +with the matching names expected by +.metn body-form . -If the -.meta divisor -is omitted, it defaults to 1. +The return value is an object representing source code which incorporates +the +.metn body-form , +augmenting it with additional code which evaluates +.code place +to determine its location, and provides place accessor local functions +expected by the +.metn body-form . -A zero -.meta divisor -results in an exception of type -.codn numeric-error . +.TP* "Example:" -If both inputs are integers, -the result is of type integer. +The following shows how to implement a +.code with-update-expander +macro using +.codn call-update-expander : -If all inputs are numbers and at least one of them is -floating-point, the others are converted to floating-point -and the result is floating-point. +.verb + (defmacro with-update-expander ((getter setter) + unex-place env body) + ^(with-gensyms (,getter ,setter) + (call-update-expander ,getter ,setter + ,unex-place ,env ,body))) +.brev -The -.code dividend -input may be a range. In this situation, the operation is -recursively distributed over the -.code from -and -.code to -fields of the range, individually matched against the -.metn divisor , -and the result is a range composed of these two individual -quotients. +Essentially, all that +.code with-update-expander +does is to choose the names for the local functions, and bind them +to the local variable names it is given as arguments. Then it +calls +.codn call-update-expander . -When the quotient is a scalar value, -.code trunc -returns the closest integer, in the zero direction, -from the value of the quotient. -The -.code floor -function returns the highest integer which does not exceed -the value of the quotient. That is to say, the division is -truncated to an integer value toward negative infinity. -The -.code ceil -function the lowest integer which is not below the value -of the quotient. -does not exceed the value of -.metn dividend . -That is to say, the division is truncated to an integer -value toward positive infinity. The -.code round -function returns the nearest integer to the quotient. -Exact halfway cases are rounded to the integer away from -zero so that -.code "(round -1 2)" -yields -.code -1 -and -.code "(round 1 2)" -yields 1, +.TP* "Example:" -Note that for large floating point values, due to the limited -precision, the integer value corresponding to the mathematical -floor or ceiling may not be available. +Implement the swap macro using +.codn call-update-expander : -.TP* "Dialect note:" -In ANSI Common Lisp, the -.code round -function chooses the nearest even integer, rather than -rounding halfway cases away from zero. \*(TX's choice -harmonizes with the semantics of the -.code round -function in the C language. +.verb + (defmacro swap (place-0 place-1 :env env) + (with-gensyms (tmp getter-0 setter-0 getter-1 setter-1) + (call-update-expander getter-0 setter-0 place-0 env + (call-update-expander getter-1 setter-1 place-1 env + ^(let ((,tmp (,getter-0))) + (,setter-0 (,getter-1)) + (,setter-1 ,tmp)))))) +.brev -.coNP Function @ mod +.coNP Function @ call-clobber-expander .synb -.mets (mod < dividend << divisor ) +.mets (call-clobber-expander < simple-setter < place < env +.mets \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ << body-form ) .syne .desc The -.code mod -function performs a modulus operation. Firstly, the absolute value -of -.meta divisor -is taken to be a modulus. Then a residue of -.meta dividend -with respect to -.meta modulus -is calculated. The residue's sign follows -that of the sign of -.metn divisor . -That is, it is the smallest magnitude -(closest to zero) residue of -.meta dividend -with respect to the absolute -value of -.metn divisor , -having the same sign as -.metn divisor . -If the operands are integer, the result is an integer. If either operand -is of type float, then the result is a float. The modulus operation is -then generalized into the floating point domain. For instance the expression -.code "(mod 0.75 0.5)" -yields a residue of 0.25 because 0.5 "goes into" 0.75 only -once, with a "remainder" of 0.25. +.code call-clobber-expander +function provides an alternative interface for making use of a clobber +expander, complementary to +.codn with-clobber-expander . -If -.meta divisor -is zero, -.code mod -throws an exception of type -.codn numeric-error . +Argument +.meta simple-setter +is a symbol, provided by the caller. It is passed to the clobber +expander function, and is used for naming a local function in the +generated code which the update expander adds to +.metn body-form . -.coNP Functions @, trunc-rem @, floor-rem @ ceil-rem and @ round-rem +The +.meta place +argument is a place which has not been subject to macro-expansion. +The +.code call-clobber-expander +function takes on the responsibility for macro-expanding the place. + +The +.meta env +parameter is the macro-expansion environment object required to +correctly expand +.code place +in its original environment. + +The +.meta body-form +argument represents the source code of a place update operation. +This code makes references to the local function whose name +is given by +.metn simple-setter . +That argument allows the update expander to write this function +with the matching name expected by +.metn body-form . + +The return value is an object representing source code which incorporates +the +.metn body-form , +augmenting it with additional code which evaluates +.code place +to determine its location, and provides the clobber local function +to the +.metn body-form . + +.coNP Function @ call-delete-expander .synb -.mets (trunc-rem < dividend <> [ divisor ]) -.mets (floor-rem < dividend <> [ divisor ]) -.mets (ceil-rem < dividend <> [ divisor ]) -.mets (round-rem < dividend <> [ divisor ]) +.mets (call-delete-expander < deleter < place < env << body-form ) .syne .desc -These functions, respectively, perform the same division operation -as -.codn trunc , -.codn floor , -.codn ceil , -and -.codn round , -referred to here as the respective target functions. +The +.code call-delete-expander +function provides an alternative interface for making use of a delete +expander, complementary to +.codn with-delete-expander . -If the -.meta divisor -is missing, it defaults to 1. +Argument +.meta deleter +is a symbol, provided by the caller. It is passed to the delete +expander function, and is used for naming a local function in the +generated code which the update expander adds to +.metn body-form . -Each function returns a list of two values: a -.meta quotient -and a -.metn remainder . The -.meta quotient -is exactly the same value as what would be returned by the -respective target function for the same inputs. - +.meta place +argument is a place which has not been subject to macro-expansion. The -.meta remainder -value obeys the following identity: +.code call-delete-expander +function takes on the responsibility for macro-expanding the place. -.mono -.mets (eql < remainder (- < dividend >> (* divisor << quotient ))) -.onom +The +.meta env +parameter is the macro-expansion environment object required to +correctly expand +.code place +in its original environment. -If -.meta divisor -is zero, these functions throw an exception of type -.codn numeric-error . +The +.meta body-form +argument represents the source code of a place delete operation. +This code makes references to the local function whose name +is given by +.metn deleter . +That argument allows the update expander to write this function +with the matching name expected by +.metn body-form . -.coNP Functions @, sin @, cos @, tan @, asin @, acos @ atan and @ atan2 -.synb -.mets (sin << radians ) -.mets (cos << radians ) -.mets (tan << radians ) -.mets (atan << slope ) -.mets (atan2 < y << x ) -.mets (asin << num ) -.mets (acos << num ) -.syne -.desc -These trigonometric functions convert their argument to floating point and -return a float result. The -.codn sin , -.code cos -and -.code tan -functions compute the sine and -cosine and tangent of the -.meta radians -argument which represents an angle -expressed in radians. The -.codn atan , -.code acos -and -.code asin -are their respective inverse -functions. The -.meta num -argument to -.code asin -and -.code acos -must be in the -range -1.0 to 1.0. The -.code atan2 -function converts the rectilinear coordinates -.meta x -and -.meta y -to an angle in polar coordinates in the range [0, 2\(*p). +The return value is an object representing source code which incorporates +the +.metn body-form , +augmenting it with additional code which evaluates +.code place +to determine its location, and provides the delete local function +to the +.metn body-form . -.coNP Functions @, exp @, log @ log10 and @ log2 +.coNP Macro @ define-modify-macro .synb -.mets (exp << arg ) -.mets (log << arg ) -.mets (log10 << arg ) -.mets (log2 << arg ) +.mets (define-modify-macro < name < parameter-list << function-name ) .syne .desc The -.code exp -function calculates the value of the transcendental number e raised to -the exponent -.metn arg . +.code define-modify-macro +macro provides a simplified way to write certain kinds of place update +macros. Specifically, it provides a way to write place update macros +which modify a place by retrieving the previous value, pass it through +a function (perhaps together with some additional arguments), and then store +the resulting value back into the place and return it. The -.code log -function calculates the base e logarithm of -.metn arg , -which must be a positive value. +.meta name +parameter specifies the name for the place update macro to be written. The -.code log10 -function calculates the base 10 logarithm of -.metn arg , -which must be a positive value. - -The -.code log2 -function calculates the base 2 logarithm of -.metn arg , -which must be a positive value. - -.coNP Functions @, expt @ sqrt and @ isqrt -.synb -.mets (expt < base << exponent *) -.mets (sqrt << arg ) -.mets (isqrt << arg ) -.syne -.desc -The -.code expt -function raises -.meta base -to zero or more exponents given -by the -.meta exponent -arguments. -.code "(expt x)" -is equivalent to -.codn "(expt x 1)" , -and yields -.code x -for all -.codn x . -For three or more arguments, the operation is right-associative. -That is to say, -.code "(expt x y z)" -is equivalent to -.codn "(expt x (expt y z))" , -similarly to the way nested exponents work in standard algebraic -notation. - -Exponentiation is done pairwise using a binary operation. -If both operands to this binary operation are non-negative integers, then the -result is an integer. - -If the exponent is negative, and the base is zero, the situation is -treated as a division by zero: an exception of type -.code numeric-error -is thrown. Otherwise, a negative exponent is converted to floating-point, -if it already isn't, and a floating-point exponentiation is performed. +.meta function-name +parameter must specify a symbol: the name of the update function. -If either operand is a float, then the other -operand is converted to a float, and a floating point exponentiation -is performed. Exponentiation that would produce a complex number is -not supported. +The update macro and update function both take at least one parameter: +the place to be updated, and its value, respectively. The -.code sqrt -function produces a floating-point square root of -.metn arg , -which is converted from integer to floating-point if necessary. Negative -operands are not supported. +.meta parameter-list +specifies the additional parameters for update function, which will also +become additional parameters of the macro. Because it is a +function parameter list, it cannot use the special destructuring features of +macro parameter lists, or the +.code :env +or +.code :whole +special parameters. It can use optional parameters. Of course, it may be empty. The -.code isqrt -function computes the integer square root of -.metn arg , -which must be an integer. -The integer square root is a value which is the -greatest integer that is no greater than the real square root of -.metn arg . -The input value must be an integer. +.code define-modify-macro +macro writes a macro called +.metn name . +The leftmost parameter of this macro is a place, followed by the additional arguments +specified by +.metn parameter-list . +The macro will arrange for the evaluation of the place argument to determine +the place location. It will then retrieve and save the prior value of the +place, and evaluate the remaining arguments. The prior value of the +place, and the values of the additional arguments, are all passed to +.meta function +and the resulting value is then stored back into the location previously +determined for +.metn place . -.coNP Function @ exptmod -.synb -.mets (exptmod < base < exponent << modulus ) -.syne -.desc -The -.code exptmod -function performs modular exponentiation and accepts only integer -arguments. Furthermore, -.meta exponent -must be a non-negative and -.meta modulus -must be positive. +.TP* "Example:" -The return value is -.meta base -raised to -.metn exponent , -and reduced to the -least positive residue modulo -.metn modulus . +Some standard place update macros are implementable using +.codn define-modify-macro , +such as +.codn inc . -.coNP Function @ square -.synb -.mets (square << argument ) -.syne -.desc The -.code square -function returns the product of -.meta argument -with itself. The following -equivalence holds, except that -.code x -is evaluated only once in the the -.code square -expression: +.code inc +macro reads the old value of the place, then passes it through the +.code + +(plus) function, along with an extra argument: the delta value, which +defaults to one. The +.code inc +macro could be written using +.code define-modify-macro +as follows: .verb - (square x) <--> (* x x) + (define-modify-macro inc (: (delta 1)) +) .brev -.coNP Function @ cum-norm-dist -.synb -.mets (cum-norm-dist << argument ) -.syne -.desc -The -.code cum-norm-dist -function calculates an approximation to the cumulative normal -distribution function: the integral, of the normal distribution function, from -negative infinity to the -.metn argument . +Note that the argument list +.code "(: (delta 1))" +doesn't specify the place, because the place is the implicit leftmost +argument of the macro which isn't given a name. With the above definition +in place, when +.code "(inc (car a))" +is invoked, then +.code "(car a)" +is first reduced to a location, and that location's value is retrieved and +saved. Then the +.code delta +parameter s evaluated to its value, which has defaulted to 1, since +the argument was omitted. +Then these two values are passed to the +.code + +function, and so 1 is added to the value previously retrieved from +.codn "(car a)" . +The resulting sum is then stored back +.code "(car a)" +without, of course, evaluating +.code "(car a)" +again. -.coNP Function @ inv-cum-norm +.coNP Macro @ defplace .synb -.mets (inv-cum-norm << argument ) +.mets (defplace < place-destructuring-args < body-sym +.mets \ \ \ \ \ \ \ \ \ >> ( getter-sym < setter-sym << update-body ) +.mets \ \ \ \ \ \ \ \ \ >> [( ssetter-sym << clobber-body ) +.mets \ \ \ \ \ \ \ \ \ \ >> [( deleter-sym << delete-body )]]) .syne .desc The -.code inv-cum-norm -function calculates an approximate to the inverse of the cumulative -normal distribution function. The argument, a value expected to lie -in the range [0, 1], represents the integral of the normal distribution -function from negative infinity to some domain point -.IR p . -The function calculates the approximate value of -.IR p . -The minimum value returned is -10, and the maximum value returned is 10, -regardless of how closely the argument approaches, respectively, -the 0 or 1 integral endpoints. For values less than zero, or exceeding 1, the -values returned, respectively, are -10 and 10. +.code defplace +macro is used to introduce a new kind of syntactic place. +It writes the update expander, and optionally clobber and delete +expander functions, from a simpler, more compact specification, +and automatically registers the resulting functions. The compact specification +of a +.code defplace +call contains only code fragments for the expander functions. -.coNP Functions @ n-choose-k and @ n-perm-k -.synb -.mets (n-choose-k < n << k ) -.mets (n-perm-k < n << k ) -.syne -.desc +The name and syntax of the place is determined by the +.meta place-destructuring-args +argument, which is macro-style parameter list whose structure +mimics that of the the place. In particular, its leftmost symbol +gives the name under which the place is registered. The -.code n-choose-k -function computes the binomial coefficient nCk which -expresses the number of combinations of -.meta k -items that can be chosen from -a set of -.metn n , -where combinations are subsets. +.code defplace +macro provides automatic destructuring of the syntactic place, +so that the expander code fragments can refer to the components +of a place by name. The -.code n-perm-k -function computes nPk: the number of permutations of size -.meta k -that can be drawn from a set of -.metn n , -where permutations are sequences, -whose order is significant. +.meta body-sym +parameter must be be a symbol. This symbol will capture the +.meta body-forms +parameter which is passed to the update expander, clobber +expander or delete expander. The code fragments then have +access to the the body forms via this name. -The calculations only make sense when -.meta n +The +.metn getter-sym , +.metn setter-sym , and -.meta k -are nonnegative integers, and -.meta k -does not exceed -.metn n . -The behavior is not specified if these conditions -are not met. - -.coNP Functions @, fixnump @, bignump @, integerp @ floatp and @ numberp -.synb -.mets (fixnump << object ) -.mets (bignump << object ) -.mets (integerp << object ) -.mets (floatp << object ) -.mets (numberp << object ) -.syne -.desc -These functions test the type of -.metn object , -returning -.code t -if it is an object -of the implied type, -.code nil -otherwise. The -.codn fixnump , -.code bignump +.meta update-body +parenthesized triplet specify the update expander fragment. +The +.code defplace +macro will bind +.meta getter-sym and -.code floatp -functions return -.code t -if the object is of the basic type -.codn fixnum , -.code bignum -or -.codn float . -The function -.code integerp -returns true of -.meta object -is either a -.code fixnum -or -a -.codn bignum . -The function -.code numberp -returns -.code t -if -.meta object -is either -a -.codn fixnum , -.code bignum -or -.codn float . +.meta setter-sym +to symbols. The +.meta update-body +must then specify a template of code which evaluates the syntactic place to +determine its storage location, and provides a pair of local functions, using +these two symbols as their name. The template must also insert the +.meta body-sym +forms into the scope of these local functions, and the place determining code. -.coNP Functions @ zerop and @ nzerop -.synb -.mets (zerop << number ) -.mets (nzerop << number ) -.syne -.desc The -.code zerop -function tests -.meta number -for equivalence to zero. The argument must be -a number or character. It returns -.code t -for the integer value -.code 0 -and for the floating-point -value -.codn 0.0 . -For other numbers, it returns -.codn nil . -It returns -.code t -for the null character -.code #\enul +.meta setter-sym and -.code nil -for all other characters. - -If -.meta number -is a range, then -.code zerop -returns -.code t -if both of the range endpoints individually satisfy -.codn zerop . +.meta clobber-body +arguments similarly specify an optional clobber expander fragment, +as a single optional argument. If specified, the +.meta clobber-body +must generate a local function named using +.meta setter-sym +wrapped around +.meta body-sym +forms. The -.code nzerop -function is the logical inverse of -.codn zerop : -it returns -.code t -for those arguments for which -.code zerop -returns -.code nil +.meta deleter-sym and -.IR "vice versa" . +.meta deleter-body +likewise specify a delete expander fragment. If this is omitted, +then the place shall not support deletion. -.coNP Functions @ plusp and @ minusp -.synb -.mets (plusp << number ) -.mets (minusp << number ) -.syne -.desc -These functions test whether a number is positive or negative, -returning -.code t -or -.codn nil , -as the case may be. +.TP* "Example:" -The argument may also be a character. All characters other than -the null character -.code #\enul -are positive. No character is negative. +Implementation of the place denoting the +.code car +field of +.code cons +cells: -.coNP Functions @ evenp and @ oddp +.verb + (defplace (car cell) body + + ;; the update expander fragment + (getter setter + (with-gensyms (cell-sym) ;; temporary symbol for cell + ^(let ((,cell-sym ,cell)) ;; evaluate place to cell + ;; getter and setter access cell via temp var + (macrolet ((,getter () + ^(car ,',cell-sym)) + (,setter (val) + ^(sys:rplaca ,',cell-sym ,val))) + + ;; insert body form from place update macro + ,body)))) + + ;; clobber expander fragment: simpler: no need + ;; to evaluate cell to temporary variable. + (ssetter + ^(macrolet ((,ssetter (val) + ^(sys:rplaca ,',cell ,val))) + ,body)) + + ;; deleter: delegate to pop semantics: + ;; (del (car a)) == (pop a). + (deleter + ^(macrolet ((,deleter () ^(pop ,',cell))) + ,body))) +.brev + +.coNP Macro @ defset .synb -.mets (evenp << integer ) -.mets (oddp << integer ) +.mets (defset < name < params < new-val-sym << set-form ) +.mets (defset < get-fun-sym << set-fun-sym ) .syne .desc The -.code evenp -and -.code oddp -functions require integer arguments. -.code evenp -returns -.code t -if -.meta integer -is even (divisible by two), otherwise it returns -.codn nil . -.code oddp -returns -.code t -if -.meta integer -is not divisible by two (odd), otherwise -it returns -.codn nil . +.code defset +macro provides a mechanism for introducing a new kind of syntactic place. +It is simpler to use than +.code defplace +and more concise, but not as general. -.coNP Functions @, succ @, ssucc @, sssucc @, pred @ ppred and @ pppred -.synb -.mets (succ << number ) -.mets (ssucc << number ) -.mets (sssucc << number ) -.mets (pred << number ) -.mets (ppred << number ) -.mets (pppred << number ) -.syne -.desc The -.code succ -function adds 1 to its argument and returns the resulting value. -If the argument is an integer, then the return value is the successor -of that integer, and if it is a character, then the return value -is the successor of that character according to Unicode. +.code defset +macro is designed for situations in which a function or macro which evaluates +all of its arguments is required to serve as a syntactic place. +It provides two flavors of syntax: the long form, indicated by giving +.code defset +five arguments, and a short form, which uses two arguments. +In the long form of +.codn defset , +the syntactic place is described by +.meta name +and +.metn params . The -.code pred -function subtracts 1 from its argument, and under similar considerations -as above, the result represents the predecessor. +.code defset +form expresses the request that call to the function or operator named +.meta name +is to be treated as a syntactic place, which has arguments described by +the parameter list +.metn params . The -.code ssucc -and -.code sssucc -functions add 2 and 3, respectively. Similarly, -.code ppred -and -.code pppred -subtract 2 and 3 from their argument. +.meta new-val-sym +parameter is the name of a symbol which will be bound to +an expression which calculates the new value being stored into +the syntactic place. This is intended to be referenced in the +.meta set-form +only, which should ensure that the expression that +.meta new-val-sym +holds is evaluated only once. -.coNP Functions @, > @, < @, >= @ <= and @ = -.synb -.mets (> < number << number *) -.mets (> < number << number *) -.mets (>= < number << number *) -.mets (<= < number << number *) -.mets (= < number << number *) -.syne -.desc -These relational functions compare characters and numbers for numeric equality -or inequality. The arguments must be one or more numbers or characters. - -If just one argument is given, then these functions all return -.codn t . +The +.meta set-form +argument specifies an expression which generates the code for storing a new +value to the place. -If two arguments are given then, they are compared as follows. -First, if the numbers do not have the same type, then the one -which has the lower ranking type is converted to the type of -the other, according to this ranking: character < integer < float. -For instance if a character and integer are compared, the character -is converted to integer. Then a straightforward numeric comparison -is applied. +The +.code defset +macro makes the necessary arrangements such that when an operator form +named by +.meta name +is treated as a syntactic place, then at macro-expansion time, code is +generated to evaluate all of its argument expressions into machine-generated +variables. The names of those variables are automatically bound to the +corresponding symbols given in the +.meta params +argument list of the +.code defset +syntax. Code is also generated to evaluate the expression which gives the +new value to be stored, and that is bound to a generated variable whose +name is bound to the +.code new-val-sym +symbol. Then arrangements are made to invoke the operator named by +.meta name +and to evaluate the +.code set-form +in an environment in which these symbol bindings are visible. +The operator named +.meta name +is invoked using an altered argument list which uses temporary symbols in place +of the original expressions. The task of +.code set-form +is to insert the values of the symbols from +.meta params +and +.meta new-val-sym +into a suitable code templates that will perform the store actions. +The code generated by +.code set-form +must also take on the responsibility of yielding the new value as its result. -Three or more arguments may be given, in which case the comparison proceeds -pairwise from left to right. For instance in -.codn "(< a b c)" , -the comparison -.code "(< a b)" -is performed in isolation. If the comparison is false, then -.code nil -is returned, otherwise -the comparison -.code "(< b c)" -is performed in isolation, and if that is false, -.code nil -is returned, otherwise -.code t -is returned. Note that it is possible for -.code b -to -undergo two different conversions. For instance in the -.mono -.meti (< < float < character << integer ) -.onom -comparison, -.meta character -will first convert to a floating-point representation -of its Unicode value so that it can be compared to -.metn float , -and if that comparison succeeds, then in the second comparison, -.meta character -will be converted to integer so that it can be compared to -.metn integer . +If +.meta params +list contains optional parameters, the default value expressions of those +parameters shall be evaluated in the scope of the +.code defset +definition. -.coNP Function @ /= -.synb -.mets (/= << number *) -.syne -.desc -The arguments to -.code /= -may be numbers or characters. The -.code /= -function returns -.code t -if no two of its arguments are numerically equal. That is to say, if there -exist some -.code a +The +.meta params +list may specify a rest parameter. In the expansion, this parameter will +capture a list of temporary symbols, corresponding to the list of variadic +argument expressions. For instance if the +.code defset +parameter list for a place +.code g +is +.codn "(a b . c)" , +featuring the rest parameter +.codn c , +and its +.meta set-form +is +.code "^(s ,a ,b ,*c)" +and the place is invoked as +.code "(g (i) (j) (k) (l))" +then parameter +.code c +will be bound to a list of gensyms such as +.code "(#:g0123 #:g0124)" +so that the evaluation of +.meta set-form +will yield syntax resembling +.codn "(s #:g0121 #:g0122 #:g0123 #:g0124)" . +Here, gensyms +.code #:g0123 and -.code b -which are distinct arguments such that -.code "(= a b)" -is true, then -the function returns -.codn nil . -Otherwise it returns -.codn t . +.code #:g0124 +are understood to be bound to the values of the expressions +.code (k) +and +.codn (l) , +the two trailing parameters corresponding to the rest parameter +.codn c . -.coNP Functions @ max and @ min -.synb -.mets (max < first-arg << arg *) -.mets (min < first-arg << args *) -.syne -.desc +Syntactic places defined by +.code defset +that have a rest parameter may be invoked with improper syntax such as +.codn "(set (g x y . z) v)" . +In this situation, that rest parameter will be bound to the name of +a temporary variable which holds the value of +.code z +rather than to a list of temporary variable names holding the values +of trailing expressions. The -.code max -and -.code min -functions determine and return the highest or lowest -value from among their arguments. +.code set-form +must be prepared for this situation. In particular, the rest parameter's value +is an atom, then it cannot be spliced in the backquote syntax, except at the +last position of a list. -If only -.meta first-arg -is given, that value is returned. +Although syntactic places defined by +.code defset +perform macro-parameter-like destructuring of the place form, binding +unevaluated argument expressions to the parameter symbols, +nested macro parameter lists are not supported: +.meta params +specifies a function parameter list. -These functions are type generic, since they compare arguments -using the same semantics as the -.code less -function. +The parameter list may use parameter macros, keeping in mind that +the parameter expansion is applied at the time the +.code defset +form is processed, specifying an expanded parameter list which +receives unevaluated expressions. The +.meta set-form +may refer to all symbols produced by parameter list expansion, other +than generated symbols. For instance, if a parameter list macro +.code :addx +exists which adds the parameter symbol +.code x +to the parameter list, and this +.code :addx +is invoked in the +.meta params +list of a +.codn defset , +then +.code x +will be visible to the +.metn set-form . -If two or more arguments are given, then -.code "(max a b)" -is equivalent to -.codn "(if (less a b) b a)" , -and -.code "(min a b)" -is equivalent to -.codn "(if (less a b) a b)" . -If the operands do not -have the same type, then one of them is converted to the type of the other; -however, the original unconverted values are returned. For instance -.code "(max 4 3.0)" -yields the integer -.codn 4 , -not -.codn 4.0 . +The short, two-argument form of +.code defset +simply specifies the names of two functions or operators: +.code get-fun-sym +names the operator which accesses the place, and +.code set-fun-sym +names the operator which stores a new value into the place. +It is expected that all arguments of these operators are evaluated +expressions, and that the store operator takes one argument more +than the access operator. The operators are otherwise assumed to be +variadic: each instance of a place based on +.code get-fun-sym +individually determines how many arguments are passed to that operator +and to the one named by +.codn set-fun-sym . -If three or more arguments are given, -.code max +The definition +.code "(defset g s)" +means that +.code "(inc (g x y))" +will generate code which ensures that +.code x and -.code min -reduce the arguments in a left-associative manner. -Thus -.code "(max a b c)" -means -.codn "(max (max a b) c)" . - -.coNP Function @ clamp -.synb -.mets (clamp < low < high << val ) -.syne -.desc -The -.code clamp -function clamps value -.meta val -into the range -.meta low -to -.metn high . +.code y +are evaluated exactly once, and then those two values are passed as +arguments to +.code g +which returns the current value of the place. That value is then incremented +by one, and stored into the place by calling the +.code s +function/operator with three arguments: the two values that were passed to +.code g +and the new value. The exact number of arguments is determined by each +individual use of +.code g +as a place; the +.code defset +form doesn't specify the arity of +.code g +and +.codn s , +only that +.code s +must accept one more argument relative to +.codn g . -The -.code clamp -function returns -.meta low -if -.meta val -is less than -.metn low . -If -.meta val -is greater than or equal to -.metn low , -but less than -.metn high , -then it returns -.metn val . -Otherwise it returns -.metn high . +The following equivalence holds between the short and long forms: -More precisely, -.code "(clamp a b c)" -is equivalent to -.codn "(max a (min b c))" . +.verb + (defset g s) <--> (defset g (. r) n ^(g ,*r) ^(s ,*r ,n)) +.brev -.coNP Function @ bracket -.synb -.mets (bracket < value << level *) -.syne -.desc -The -.code bracket -function's arguments consist of one required -.meta value -followed by -.I n -.meta level -arguments. -The -.meta level -arguments are optional; in other words, -.I n -may be zero. +Note: +the short form of +.code defset +is similar to the +.code define-accessor +macro. -The -.code bracket -function calculates the -.I bracket -of the -.meta value -argument: a zero-based positional index of the value, in relation to the -.meta level -arguments. +.TP* "Example:" -Each of the -.meta level -arguments, of which there may be none, is associated with -an integer index, starting at zero, in left to right order. The -.meta level -arguments are examined in that order. When a -.meta level -argument is encountered which exceeds -.metn value , -that -.meta level -argument's index is returned. -If -.meta value -exceeds all of the -.meta level -arguments, then -.I n -is returned. +Implementation of +.code car +as a syntactic place using a long form +.codn defset : -Determining whether -.meta value -exceeds a -.meta level -is performed using the -.code less -function. +.verb + (defset car (cell) new + (let ((n (gensym))) + ^(rlet ((,n ,new)) + (progn (rplaca ,cell ,n) ,n)))) +.brev -.TP* Examples: +Given such a definition, the expression +.code "(inc (car (abc)))" +expands to code closely resembling: .verb - (bracket 42) -> 0 - (bracket 5 10) -> 0 - (bracket 15 10) -> 1 - (bracket 15 10 20) -> 1 - (bracket 15 10 20 30) -> 1 - (bracket 20 10 20 30) -> 2 - (bracket 35 10 20 30) -> 3 - (bracket "a" "aardvark" "zebra") -> 0 - (bracket "ant" "aardvark" "zebra") -> 1 - (bracket "zebu" "aardvark" "zebra") -> 2 + (let ((#:g0048 (abc))) + (let ((#:g0050 (succ (car #:g0048)))) + (rplaca #:g0048 #:g0050) + #:g0050)) .brev -.coNP Functions @, int-str @ flo-str and @ num-str +The +.code defset +macro has arranged for the argument expression +.code (abc) +of +.code car +to be evaluated to a temporary variable +.codn #:g0048 , +a +.codn gensym . +This, then, holds the +.code cons +cell being operated on. +At macro-expansion time, the variable +.code cell +from the parameter list specified by the +.code defset +is bound to this symbol. The access expression +.code "(car #:0048)" +to retrieve the prior value is automatically generated +by combining the name of the place +.code car +with the gensym to which its argument +.code (abc) +has been evaluated. +The +.code new +variable was bound to the expression giving the new value, namely +.codn "(succ (car #:g0048))" . +The +.meta set-form +is careful to evaluate this only one time, storing its value into +the temporary variable +.codn #:g0050 , +referenced by the variable +.codn n . +The +.metn set-form 's +.code "(rplaca ,cell ,n)" +fragment thus turned into +.code "(rplaca #:g0048 #:g0050)" +where +.code #:g0048 +references the cons cell being operated on, and +.code #:g0050 +the calculated new value to be stored into its +.code car +field. +The +.meta set-form +is careful to arrange for the new value +.code #:g0050 +to be returned. Those place-mutating operators which yield the new value, such +as +.code set +and +.code inc +rely on this behavior. + +.coNP Macro @ define-place-macro .synb -.mets (int-str < string <> [ radix ]) -.mets (flo-str << string ) -.mets (num-str << string ) +.mets (define-place-macro < name < macro-style-params +.mets \ \ << body-form *) .syne .desc -These functions extract numeric values from character string -.metn string . -Leading whitespace in -.metn string , -if any, is skipped. If no digits can be successfully extracted, then -.code nil -is returned. Trailing material which does not contribute to the number is -ignored. +In some situations, an equivalence exists between two forms, only one +of which is recognized as a place. The +.code define-place-macro +macro can be used to establish a form as a place in terms of a translation to +an equivalent form which is already a place. The -.code int-str -function converts a string of digits in the specified -.meta radix -to an integer value. If -.meta radix -isn't specified, it defaults to 10. -Otherwise it must be an integer in the range 2 to 36, or else the character -.codn #\ec . +.code define-place-macro +has the same syntax as +.codn defmacro . +It specifies a macro transformation for a compound form which has the +.meta name +symbol in its leftmost position. -For radices above 10, letters of the alphabet -are used for digits: -.code A -represent a digit whose value is 10, -.code B -represents 11 and -so forth until -.codn Z . -Upper and lower case letters are recognized. -Any character which is not a digit of the specified radix is regarded -as the start of trailing junk at which the extraction of the digits stops. +Place macro expansion doesn't use an environment; place macros are in a single +global namespace, special to place macros. There are no lexically scoped place +macros. Such an effect can be achieved by having a place macro expand to +an a form which is the target of a global or local macro, as necessary. -When -.meta radix -is specified as the character object -.codn #\ec , -this indicates that a C-language-style integer constant should be -recognized. If, after any optional sign, the remainder of -.meta string -begins with the character pair -.code 0x -then that pair is considered removed from the string, and it is treated -as base 16 (hexadecimal). If, after any optional sign, the remainder of -.meta string -begins with a leading zero not followed by -.codn x , -then the radix is taken to be 8 (octal). In scanning these formats, -.code int-str -function is not otherwise constrained by C language representational -limitations. Specifically, the input values are taken to be the printed -representation of arbitrary-precision integers and treated accordingly. - -The -.code flo-str -function converts a floating-point decimal notation to a nearby -floating point value. The material which contributes to the value -is the longest match for optional leading space, followed by a -mantissa which consists of an optional sign followed by a mixture of at least -one digit, and at most one decimal point, optionally followed by an exponent -part denoted by the letter -.code E -or -.codn e , -an optional sign and one or more optional exponent digits. -If the value specified by -.meta string -is out of range of the floating-point representation, then +To support place macros, forms which are used as syntactic places are subject +to a modified macro-expansion algorithm: +.RS +.IP 1. +If a place macro exists for a form that is being used as a place, then the +that place macro is invoked to expand the +form, and the expansion is taken in place of the original form. This process +repeats until the form can no longer be expanded as a place macro, or +the place macro declines to expand the form by returning the unexpanded +input. +.IP 2. +A form that has been fully expanded as a place macro is then subject +to a single-round of macro-expansion, as if by +.codn macroexpand-1 , +which takes place in the original form's lexical environment. +If the form doesn't expand, or the result of expansion is .code nil -is returned. - -The -.code num-str -function converts a decimal notation to either an integer as if by -a radix 10 application of -.codn int-str , -or to a floating point value as if by -.codn flo-str . -The floating point interpretation is chosen if the possibly empty -initial sequence of digits (following any whitespace and optional sign) is -followed by a period, or by -.code e -or -.codn E . +or a +non-symbolic atom, then the process terminates. Otherwise, the +process is repeated from step 1. +.RE -.coNP Functions @ int-flo and @ flo-int -.synb -.mets (int-flo << float ) -.mets (flo-int << integer ) -.syne -.desc -These functions perform numeric conversion between integer and floating point -type. The -.code int-flo -function returns an integer by truncating toward zero. +.IP The -.code flo-int -function returns an exact floating point value corresponding to -.metn integer , -if possible, otherwise an approximation using a nearby -floating point value. - -.coNP Functions @ tofloat and @ toint -.synb -.mets (tofloat << value ) -.mets (toint < value <> [ radix ]) -.syne -.desc -These convenience functions convert -.meta value -to floating-point or integer, respectively. - -If a floating-point value is passed into tofloat, or an integer value into -toint, then the value is simply returned. - -If -.meta value -is a character, then it is treated as a string of length one -containing that character. +.code define-place-macro +macro does not cause +.meta name +to become +.codn mboundp . -If -.meta value -is a string, then it is converted by -.code tofloat -as if by the function -.metn flo-str , -, and by -.code toint -as if by the function -.codn int-str . +There can exist both an ordinary macro and a place macro of the same name. +In this situation, when the macro call appears as a place form, it is +expanded as a place macro, according to the above steps. When the macro call +appears as an evaluated form, not being used as a place, the form is +expanded using the ordinary macro. -If -.meta value -is an integer, then it is converted by -.code tofloat -as if by the function -.codn flo-int . +.TP* "Example:" -If -.meta value -is a floating-point number, then it is converted by -.code toint -as if by the function -.codn int-flo . +Implementation of +.code first +in terms of +.codn car : -.coNP Variables @ fixnum-min and @ fixnum-max -.desc -These variables hold, respectively, the most negative value of the -.code fixnum -integer type, and its most positive value. Integer values -from -.code fixnum-min -to -.code fixnum-max -are all of type -.codn fixnum . -Integers outside of this range are -.code bignum -integers. +.verb + (define-place-macro first (obj) + ^(car ,obj)) +.brev -.coNP Functions @ tofloatz and @ tointz +.coNP Macro @ rlet .synb -.mets (tofloatz << value ) -.mets (tointz < value <> [ radix ]) +.mets (rlet >> ({( sym << init-form )}*) << body-form *) .syne .desc -These functions are closely related to, respectively, -.code tofloat -and -.codn toint . -They differ in that these functions return a floating-point -or integer zero, respectively, in some situations -in which those functions would return +The macro +.code rlet +is similar to the +.code let +operator. It establishes bindings for one or more +.metn sym -s, +which are initialized using the values of +.metn init-form -s. + +Note that the simplified syntax for a variable which initializes to .code nil -or throw an error. +by default is not supported by +.codn rlet ; +that is to say, the syntax +.meta sym +cannot be used in place of the +.meti >> ( sym << init-form ) +syntax when +.meta sym +is to be initialized to +.codn nil . -Whereas those functions reject a -.meta value -argument of -.codn nil , -for that same argument -.code tofloatz -function returns 0.0 and -.code tointz -returns 0. +The +.code rlet +macro differs from +.code let +in that +.code rlet +assumes that those +.metn sym -s +whose +.metn init-form -s, +after macro expansion, +are constant expressions +(according to the +.code constantp +function) may be safely implemented as a symbol macro rather than a lexical +variable. -Likewise, in cases when -.code value -contains a string or character which cannot be -converted to a number, and -.code tofloat -and -.code toint -would return -.codn nil , -these functions return 0.0 and 0, respectively. +Therefore +.code rlet +is suitable in situations in which simpler code is desired from the output +of certain kinds of machine-generated code, which binds local symbols: +code with fewer temporary variables. -In other situations, these functions behave -exactly like -.code tofloat -and -.codn toint . +On the other hand, +.code rlet +is not suitable in some situations when true variables are required, which +are assignable, and provide temporary storage. -.coNP Variables @, flo-min @ flo-max and @ flo-epsilon -.desc -These variables hold, respectively: the smallest positive floating-point -value; the largest positive floating-point value; and the difference -between 1.0 and the smallest representable value greater than 1.0. +.TP* "Example:" -.code flo-min -and -.code flo-max -define the floating-point range, which consists -of three regions: values from -.code "(- flo-max)" -to -.codn "(- flo-min)" ; -the value 0.0, and values from -.code flo-min -to -.codn flo-max . +.verb + ;; WRONG! Real storage location needed. + (rlet ((flag nil)) + (flip flag)) ;; error: flag expands to nil -.coNP Variable @ flo-dig -.desc -This variable holds an integer representing the number of decimal digits -in a decimal floating-point number such that this number can be converted -to a \*(TX floating-point number, and back to decimal, without a change in any of -the digits. This holds regardless of the value of the number, provided that it -does not exceed the floating-point range. + ;; Demonstration of constant-propagation + (let ((a 42)) + (rlet ((x 1) + (y a)) + (+ x y))) --> 43 -.coNP Variable @ flo-max-dig -.desc -This variable holds an integer representing the maximum number of -decimal digits required to capture the value of a floating-point number -such that the resulting decimal form will convert back to the same -floating-point number. See also the -.code *print-flo-precision* -variable. + (expand + '(let ((a 42)) + (rlet ((x 1) + (y a)) + (+ x y)))) --> (let ((a 42)) + (let ((y a)) + (+ 1 y))) +.brev -.coNP Variables @ %pi% and @ %e% +The last example shows that the +.code x +variable has disappeared in the expansion. The +.code rlet +macro turned it into into a +.code symacrolet +denoting the constant 1, which then propagated to the use site, +turning the expression +.code "(+ x y)" +into +.codn "(+ 1 y)" . + +.coNP Macro @ slet +.synb +.mets (slet >> ({( sym << init-form )}*) << body-form *) +.syne .desc -These variables hold an approximation of the mathematical constants \(*p and e. -To four digits of precision, \(*p is 3.142 and e is 2.718. The -.code %pi% -and -.code %e% -approximations are accurate to -.code flo-dig -decimal digits. +The macro +.code slet +a weaker form of the +.code rlet +macro. Just like +.codn rlet , +.code slet +reduces bindings initialized by constant expressions +to symbol macros. In addition, unlike +.codn rlet , +.code slet +also reduces to symbol macros those bindings which +are initialized by symbol expressions (values of variables). -.coNP Function @ digits +.coNP Macro @ alet .synb -.mets (digits < number <> [ radix ]) +.mets (alet >> ({( sym << init-form )}*) << body-form *) .syne .desc -The -.code digits -function returns a list of the digits of -.meta number -represented in the base given by -.metn radix . +The macro +.code alet +("atomic" or "all") is a stronger form of the +.code slet +macro. All bindings initialized by constant expressions are +turned to symbol macros. Then, if all of the remaining bindings are +all initialized by symbol expressions, they are also turned to +symbol macros. Otherwise, none of the remaining bindings +are turned to symbol macros. The -.meta number -argument must be a non-negative integer, and -.meta radix -must be an integer greater than one. - -If -.meta radix -is omitted, it defaults to 10. - -The return value is a list of the digits in descending order of significance: -most significant to least significant. -The digits are integers. For instance, if -.meta radix -is 42, then the digits are integer values in the range 0 to 41. - -The returned list always contains at least one element, and -includes no leading zeros, except when -.meta number -is zero. In that case, a one-element list containing zero is returned. - -.TP* Examples: - -.verb - (digits 1234) -> (1 2 3 4) - (digits 1234567 1000) -> (1 234 567) - (digits 30 2) -> (1 1 1 1 0) - (digits 0) -> (0) -.brev +.code alet +macro can be used even in situations when it is possible that the initializing +forms the variables may have side effects through which they affect each +others' evaluations. In this situation +.code alet +still propagates constants via symbol macros, and can eliminate the +remaining temporaries if they can all be made symbol macros for +existing variables: i.e. there doesn't exist any initialization form +with interfering side effects. -.coNP Function @ digpow +.coNP Macro @ define-accessor .synb -.mets (digpow < number <> [ radix ]) +.mets (define-accessor < get-function << set-function ) .syne .desc The -.code digpow -function decomposes the -.meta number -argument into a power series whose terms add up to -.metn number . +.code define-accessor +macro is used for turning a function into an accessor, +such that forms which call the function can be treated +as places. -The -.meta number -argument must be a non-negative integer, and -.meta radix -must be an integer greater than one. +Arguments to +.code define-accessor +are two symbols, which must name functions. When the +.code define-accessor +call is evaluated, the +.meta get-function +symbol is registered as a syntactic place. Stores to the +place are handled via calls to +.metn set-function . -The returned power series consists of a list of nonnegative -integers. It is formed from the digits of -.meta number -in the given -.metn radix , -which serve as coefficients which multiply successive -powers of the -.metn radix , -starting at the zeroth power (one). +If +.meta get-function +names a function which takes N +arguments, +.meta set-function +must name a function which takes N+1 arguments. -The terms are given in decreasing order of significance: -the term corresponding to the most significant digit of -.metn number , -multiplying the highest power of -.metn radix , -is listed first. +Moreover, in order for the accessor semantics to be correct +.meta set-function +must treat its rightmost argument as the value being stored, +and must also return that value. -The returned list always contains at least one element, and -includes no leading zeros, except when -.meta number -is zero. In that case, a one-element list containing zero is returned. +When a function call form targeting +.meta get-function +is treated as a place which is subject +to an update operation (for instance an increment via the +.code inc +macro), +the accessor definition created by +.code define-accessor +ensures that the arguments of +.meta get-function +are evaluated only once, even though the update involves +a call to +.meta get-function +and +.meta set-function +with the same arguments. The argument forms are evaluated to +temporary variables, and these temporaries are used as the +arguments in the calls. -.verb - (digpow 1234) -> (1000 200 30 4) - (digpow 1234567 1000) -> (1000000 234000 567) - (digpow 30 2) -> (16 8 4 2 0) - (digpow 0) -> (0) -.brev +No other assurances are provided by +.codn define-accessor . -.coNP Functions @ poly and @ rpoly -.synb -.mets (poly < arg << coeffs ) -.mets (rpoly < arg << coeffs ) -.syne -.desc -The -.code poly +In particular, if +.meta get-function and -.code rpoly -functions evaluate a polynomial, for the given numeric argument value -.meta arg -and the coefficients given by -.metn coeffs , -a sequence of numbers. +.meta set-function +internally each perform some redundant calculation over their arguments, +this cannot be optimized. Moreover, if that calculation has a visible effect, +that effect is observed multiple times in an update operation. -If -.meta coeffs -is an empty sequence, it denotes the zero polynomial, whose value -is zero everywhere; the functions return zero in this case. +If further optimization or suppression of multiple effects is required, +the more general +.code defplace +macro must be used to define the accessor. It may also be possible to +treat the situation in a satisfactory way using a +.code define-place-macro +definition, which effectively then supplies inline code whenever a certain form +is used as a place, and that code itself is treated as a place. -Otherwise, the -.code poly -function considers -.meta coeffs -to hold the coefficients in the conventional order, namely in order -of decreasing degree of polynomial term. The first element of -.meta coeffs -is the leading coefficient, and the constant term appears as the last element. +Note: +.code define-accessor +is very similar to the short form of +.codn defset . -The -.code rpoly -function takes the coefficients in opposite order: the first element of -.meta coeffs -gives the constant term coefficient, and the last element gives the -leading coefficient. - -Note: except in the case of -.code rpoly -operating on a list or list-like sequence of coefficients, -Horner's method of evaluation is -used: a single result accumulator is initialized with zero, and then for each -successive coefficient, in order of decreasing term degree, the accumulator is -multiplied by the argument, and the coefficient is added. When -.code rpoly -operates on a list or list-like sequence, it makes a single -pass through the coefficients in order, thus taking them in increasing -term degree. It maintains two accumulators: one for successive powers of -.meta arg -and one for the resulting value. For each coefficient, the power -accumulator is updated by a multiplication by -.meta arg -and then this value is multiplied by the coefficient, and -that value is then added to the result accumulator. +.coNP Special variables @, *place-update-expander* @ *place-clobber-expander* and @ *place-delete-expander* +.desc +These variables hold hash tables, by means of which update expanders, +clobber expanders and delete expanders are registered, as associations +between symbols and functions. -.TP* Examples: +If +.code "[*place-update-expander* 'sym]" +yields a function, then symbol +.code sym +is the basis for a syntactic place. If the expression yields +.codn nil , +then forms beginning with +.code sym +are not syntactic places. (The situation of a clobber accessor or delete +accessor being defined without an update expander is improper). -.verb - ;; 2 - ;; evaluate x + 2x + 3 for x = 10. - (poly 10 '(1 2 3)) -> 123 +.coNP Special variable @ *place-macro* +.desc +The +.code *place-macro* +special variable holds the hash table of associations between +symbols and place macro expanders. - ;; 2 - ;; evaluate 3x + 2x + 1 for x = 10. - (rpoly 10 '(1 2 3)) -> 321 -.brev +If the expression +.code "[*place-macro* 'sym]" +yields a function, then symbol +.code sym +has a binding as a place macro. If that +expression yields +.codn nil , +then there is no such binding: compound forms beginning with +.code sym +do not undergo place macro expansion. -.coNP Function @ bignum-len +.SS* Quasiquote Operator Syntax +.coNP Macro @ qquote .synb -.mets (bignum-len << arg ) +.mets (qquote << form ) .syne .desc The -.code bignum-len -function reports the machine-specific -.I "bignum order" -of the integer or character argument -.metn arg . +.code qquote +(quasi-quote) macro operator implements a notation for convenient +list construction. If +.meta form +is an atom, or a list structure which +does not contain any +.code unquote +or +.code splice +operators, then +.mono +.meti (qquote << form ) +.onom +is equivalent to +.mono +.meti (qquote << form ). +.onom If -.meta arg -is a character or -.code fixnum -integer, the function returns zero. - -Otherwise -.meta arg -is expected to be a -.code bignum -integer, and the function returns the number of "limbs" used for its -representation, a positive integer. - -Note: the -.code bignum-len -function is intended to be of use in algorithms whose performance -benefits from ordering the operations on multiple integer operands -according to the magnitudes of those operands. The function provides an -estimate of magnitude which trades accuracy for efficiency. +.metn form , +however, is a list structure which contains +.code unquote +or +.code splice +operators, then the substitutions implied by those operators are performed +on +.metn form , +and the +.code qquote +operator returns the resulting structure. -.coNP Variables @, flo-near @, flo-down @ flo-up and @ flo-zero -.desc -These variables hold integer values suitable as arguments to the -.code flo-set-round-mode -function, which controls the rounding mode for the results of floating-point -operations. These variables are only defined on platforms which support -rounding control. +Note: how the qquote operator actually works is that it is compiled into +code. It becomes a Lisp expression which, when evaluated, computes the +resulting structure. -Their values have the following meanings: -.RS -.coIP flo-near -Round to nearest: the result of an operation is rounded to the nearest -representable value. -.coIP flo-down -Round down: the result of an operation is rounded to the nearest representable -value that lies in the direction of negative infinity. -.coIP flo-up -Round up: the result of an operation is rounded to the nearest representable -value that lies in the direction of positive infinity. -.coIP flo-up -Round to zero: the result of an operation is rounded to the nearest -representable value that lies in the direction of zero. -.RE -.IP +A +.code qquote +can contain another +.codn qquote . +If an +.code unquote +or +.code splice +operator occurs +within a nested +.codn qquote , +it belongs to that +.codn qquote , +and not to the outer one. -.coNP Functions @ flo-get-round-mode and @ flo-set-round-mode -.synb -.mets (flo-get-round-mode) -.mets (flo-set-round-mode << mode ) -.syne -.desc -Sometimes floating-point operations produce a result which -requires more bits of precision than the floating point representation -can provide. A representable floating-point value must be substituted -for the true result and yielded by the operation. +However, an unquote operator which occurs inside another one belongs one level +higher. For instance in -On platforms which support rounding control, these functions are provided for -selecting the decision procedure by which the floating-point representation -is taken. +.verb + (qquote (qquote (unquote (unquote x)))) +.brev -The -.code flo-get-round-mode -returns the current rounding mode. The rounding mode is represented by -an integer value which is either equal to one of the four variables -.codn flo-near , -.codn flo-down , -.code flo-up +the leftmost +.code qquote +belongs with the rightmost unquote, and the inner +.code qquote and -.codn flo-zero , -or else some other value specific to the host environment. Initially, -the value is that of -.codn flo-near . -Otherwise, the value returned is that which was stored by the most -recent successful call to -.codn flo-set-round-mode . +.code unquote +belong together. When the outer +.code qquote +is evaluated, +it will insert the value of +.codn x , +resulting in the object +.codn "(qquote (unquote [value-of-x]))" . +If this resulting qquote value is evaluated again as Lisp syntax, then it will +yield +.codn [value-of-value-of-x] , +the value of +.code [value-of-x] +when treated as a Lisp expression and evaluated. -The -.code flo-set-round-mode -function changes the rounding mode. The argument to its -.meta mode -parameter may be the value of one of the above four variables, -or else some other value supported by the host environment's -.code fesetround -C library function. +.TP* Examples: -The -.code flo-set-round-mode -function returns -.code t -if it is successful, otherwise the return value is -.code nil -and the rounding mode is not changed. +.verb + (qquote a) -> a -If a value is is passed to -.code flo-set-round-mode -which is not the value of one of the above -four rounding mode variables, and the function succeeds anyway, then the -rounding behavior of floating-point operations depends on the host -environment's interpretation of that value. + (qquote (a b c)) -> (a b c) -.SS* Bit Operations -In \*(TL, similarly to Common Lisp, bit operations on integers are based -on a concept that might be called "infinite two's-complement". -Under infinite two's complement, a positive number is regarded as having -a binary representation prefixed by an infinite stream of zero digits (for -example -.code 1 -is -.codn ...00001 ). -A negative number -in infinite two's complement is the bitwise negation of its positive counterpart, -plus one: it carries an infinite prefix of 1 digits. So for instance the number -.code -1 -is represented by -.codn ...11111111 : -an infinite sequence of -1 -bits. There -is no specific sign bit; any operation which produces such an infinite sequence -of 1 digits on the left gives rise to a negative number. For instance, consider the -operation of computing the bitwise complement of the number -.codn 1 . -Since the -number -.code 1 -is represented as -.codn ...0000001 , -its complement is -.codn ...11111110 . -Each one of the -.code 0 -digits in the infinite sequence is replaced by -.codn 1 , -And this leading sequence means that the number -is negative, in fact corresponding to the two's-complement representation of -the value -.codn -2 . -Hence, the infinite digit concept corresponds to an arithmetic -interpretation. + (qquote (1 2 3 (unquote (+ 2 2)) (+ 2 3))) -> (1 2 3 4 (+ 2 3)) -In fact \*(TL's bignum integers do not use a two's complement -representation internally. Numbers are represented as an array which holds a -pure binary number. A separate field indicates the sign: negative, -or non-negative. That negative numbers appear as two's-complement under the -bit operations is merely a carefully maintained illusion (which makes bit -operations on negative numbers more expensive). + (qquote (unquote (+ 2 2))) -> 4 +.brev -The -.code logtrunc -function, as well as a feature of the -.code lognot -function, allow bit -manipulation code to be written which works with positive numbers only, even if -complements are required. The trade off is that the application has to manage a -limit on the number of bits. +In the second-to-last example, the +.code "1 2 3" +and the +.code "(+ 2 3)" +are quoted verbatim. +Whereas the +.code "(unquote (+ 2 2))" +operator caused the evaluation of +.code "(+ 2 2)" +and the substitution of the resulting value. -.coNP Functions @, logand @ logior and @ logxor -.synb -.mets (logand << integer *) -.mets (logior << integer *) -.mets (logxor < int1 << int2 ) -.syne -.desc -These operations perform the familiar bitwise and, inclusive or, and exclusive -or operations, respectively. Positive values inputs are treated as -pure binary numbers. Negative inputs are treated as infinite-bit -two's-complement. +The last example shows that +.meta form +can itself (the entire argument of +.codn qquote ) +can be an unquote operator. +However, note: +.code "(quote (splice form))" +is not valid. -For example -.code "(logand -2 7)" -produces -.codn 6 . -This is because -.code -2 -is -.code ...111110 -in infinite-bit two's-complement. And-ing this value with -.code 7 -(or -.codn ...000111 ) -produces -.codn 110 . +Note: a way to understand the nesting behavior is a via a possible model of +quasi-quote expansion which recursively compiles any nested quasi quotes first, +and then treats the result of their expansion. For instance, in the processing +of -The -.code logand -and -.code logior -functions are variadic, and may be called with zero, one, -two, or more input values. If -.code logand -is called with no arguments, it produces -the value -1 (all bits 1). If -.code logior -is called with no arguments it produces -zero. In the one-argument case, the functions just return their argument value. +.verb + (qquote (qquote (unquote (unquote x)))) +.brev -In the two-argument case, one of the operands may be a character, if the other -operand is a fixnum integer. The character operand is taken to be an integer -corresponding to the character value's Unicode code point value. The resulting -value is regarded as a Unicode code point and converted to a character value -accordingly. +the +.code qquote +operator first encounters the +embedded +.code "(qquote ...)" +and compiles it to code. During that recursive +compilation, the syntax +.code "(unquote (unquote x))" +is encountered. The inner quote +processes the outer unquote which belongs to it, and the inner +.code "(unquote x)" +becomes material that is embedded verbatim in the compilation, which will then +be found when the recursion pops back to the outer quasiquote, which will +then traverse the result of the inner compilation and find the +.codn "(unquote x)" . -When three or more arguments are specified, the operation's semantics is -that of a left-associative reduction through two-argument invocations, -so that the three-argument case -.code "(logand a b c)" -is equivalent to the expression -.codn "(logand (logand a b) c)" , -which features two two-argument cases.. +.TP* "Dialect note:" -.coNP Function @ logtest -.synb -.mets (logtest < int1 << int2 ) -.syne -.desc -The -.code logtest -function returns true if -.meta int1 +In Lisp dialects which have a published quasiquoting operator syntax, there is +the expectation that the quasiquote read syntax corresponds to it. That is to +say, that for instance the read syntax +.code "^(a b ,c)" +is expected translated to +.codn "(qquote b (unquote c))" . + +In \*(TL, this is not true! Although +.code "^(b b ,c)" +is translated to a +quasiquoting macro, it is an internal one, not based on the public +.codn qquote , +.code unquote and -.meta int2 -have bits in -common. The following equivalence holds: +.code splice +symbols being documented here. + +This idea exists for hygiene. The quasiquote read syntax is not confused +by the presence of the symbols +.codn qquote , +.code unquote +or +.code splice +in the template, since it doesn't treat them specially. + +This also allows programmers to use the quasiquote read syntax to construct +quasiquote macros. For instance .verb - (logtest a b) <--> (not (zerop (logand a b))) + ^(qquote (unquote ,x)) ;; does not mean ^^,,x ! .brev -.coNP Functions @ lognot and @ logtrunc -.synb -.mets (lognot < value <> [ bits ]) -.mets (logtrunc < value << bits ) -.syne -.desc -The -.code lognot -function performs a bitwise complement of -.metn value . -When the one-argument form of lognot is used, then if -.meta value -is nonnegative, -then the result is negative, and vice versa, according to the infinite-bit -two's complement representation. For instance -.code "(lognot -2)" -is -.codn 1 , +To the quasiquote reader, the +.code qquote and -.code "(lognot 1)" +.code unquote +symbols mean nothing special, +and so this syntax simply means that if the value of +.code x is -.codn -2 . +.codn foo , +the result of evaluating this expression will be +.codn "(qquote (unquote foo))" . -The two-argument form of -.code lognot -produces a truncated complement. Conceptually, -a bitwise complement is first calculated, and then the resulting number is -truncated to the number of bits given by -.metn bits , -which must be a nonnegative integer. The following equivalence holds: +The form's expansion is actually this: .verb - (lognot a b) <--> (logtrunc (lognot a) b) + (sys:qquote (qquote (unquote (sys:unquote x)))) .brev +the +.code sys:qquote +macro recognizes +.code sys:unquote +embedded in the form, and +the other symbols not in the +.code sys: +package are just static template material. + The -.code logtrunc -function truncates the integer -.meta value -to the specified number -of bits. If -.meta value -is negative, then the two's-complement representation -is truncated. The return value of -.code logtrunc -is always a non-negative integer. +.code sys:quote +macro and its associated +.code sys:unquote +and +.code sys:splice +operators work exactly like their ordinary counterparts. So in effect, \*(TX has +two nearly identical, independent quasi-quote implementations, one of which is +tied to the read syntax, and one of which isn't. This is useful for writing +quasiquotes which write quasiquotes. -.coNP Function @ sign-extend +.coNP Operator @ unquote .synb -.mets (sign-extend < value << bits ) +.mets (qquote (... (unquote << form ) ...)) +.mets (qquote (unquote << form )) .syne .desc The -.code sign-extend -function first truncates the infinite-bit two's complement representation of -the integer -.meta value -to the specified number of bits, similarly to the -.code logtrunc -function. Then, this truncated value is regarded as a -.meta bits -wide two's complement integer. The value of this integer is -calculated and returned. +.code unquote +operator is not an operator +.I per +.IR se . +The +.code unquote +symbol has no +binding in the global environment. It is a special syntax that is recognized +within a +.code qquote +form, to indicate forms within the quasiquote which are to be +evaluated and inserted into the resulting structure. -.TP* Examples: - -.verb - (sign-extend 127 8) -> 127 - (sign-extend 128 8) -> -128 - (sign-extend 129 8) -> -127 - (sign-extend 255 8) -> -1 - (sign-extend 256 8) -> 0 - (sign-extend -1 8) -> -1 - (sign-extend -255 8) -> 0 -.brev +The syntax +.mono +.meti (qquote (unquote << form )) +.onom +is equivalent to +.metn form : +the +.code qquote +and +.code unquote +"cancel out". -.coNP Function @ ash +.coNP Operator @ splice .synb -.mets (ash < value << bits ) +.mets (qquote (... (splice << form ) ...)) .syne .desc The -.code ash -function shifts -.meta value -by the specified number of -.meta bits -producing a -new value. If -.meta bits -is positive, then a left shift takes place. If -.meta bits -is negative, then a right shift takes place. If -.meta bit -is zero, then -.meta value -is returned unaltered. For positive numbers, a left shift by n bits is -equivalent to a multiplication by two to the power of n, or -.codn "(expt 2 n)" . -A right shift by n bits of a positive integer is equivalent to integer -division by -.codn "(expt 2 n)" , -with truncation toward zero. -For negative numbers, the bit shift is performed as if on the two's-complement -representation. Under the infinite two's-complement representation, -a right shift does not exhaust the infinite sequence of -.code 1 -digits which -extends to the left. Thus if -.code -4 -is shifted right it becomes -.code -2 -because -the bitwise representations of these values are -.code ...111100 -and -.codn ...11110 . +.code splice +operator is not an operator +.I per +.IR se . +The +.code splice +symbol has no +binding in the global environment. It is a special syntax that is recognized +within a +.code qquote +form, to indicate forms within the quasiquote which are to be +evaluated and inserted into the resulting structure. -.coNP Function @ bit +The syntax +.mono +.meti (qquote (splice << form )) +.onom +is not permitted and raises an exception if evaluated. The +.code splice +syntax must occur within a list, and not in the dotted position. + +The +.code splice +form differs from unquote in that +.mono +.meti (splice << form ) +.onom +requires that +.meta form +must evaluate to a list. That list is +integrated into the surrounding list. + +.SS* Math Library +.coNP Functions @ + and @ - .synb -.mets (bit < value << bit ) +.mets (+ << number *) +.mets (- < number << number *) +.mets (* << number *) .syne .desc The -.code bit -function tests whether the integer or character -.meta value -has a 1 in bit position -.metn bit . +.codn + , +.code - +and +.code * +functions perform addition, subtraction and multiplication, +respectively. Additionally, the +.code - +function performs additive inverse. + The -.meta bit -argument must be a non-negative integer. A value of zero of -.meta bit -indicates the least significant bit position of -.metn value . +.code + +function requires zero or more arguments. When called with no +arguments, it produces 0 (the identity element for addition), otherwise it +produces the sum over all of the arguments. + +Similarly, the +.code * +function requires zero or more arguments. When called +with no arguments, it produces 1 (the identity element for multiplication). +Otherwise it produces the product of all the arguments. + +The semantics of +.code - +changes from subtraction to additive inverse +when there is only one argument. The argument is treated as a subtrahend, +against an implicit minuend of zero. When there are two or more +argument, the first one is the minuend, and the remaining are subtrahends. + +When there are three or more operands, these operations are performed as if by +binary operations, in a left-associative way. That is to say, +.code "(+ a b c)" +means +.codn "(+ (+ a b) c)" . +The sum of +.code a +and +.code b +is computed first, and then this is added to +.codn c . +Similarly +.code "(- a b c)" +means +.codn "(- (- a b) c)" . +First, +.code b +is subtracted from +.codn a , +and then +.code c +is subtracted from that result. + +The arithmetic inverse is performed as if it were subtraction from integer 0. +That is, +.code "(- x)" +means the same thing as +.codn "(- 0 x)" . + +The operands of +.codn + , +.code - +and +.code * +can be characters, integers (fixnum and bignum), and +floats, in nearly any combination. + +If two operands have different types, then one of them is converted to the +type of the one with the higher rank, according to this ranking: +character < integer < float. For instance if one operand is integer, and the +other float, the integer is converted to a float. + +.TP* Restrictions: + +Characters are not considered numbers, and participate in these operations in +limited ways. Subtraction can be used to computed the displacement between the +Unicode values of characters, and an integer displacement can be added to a +character, or subtracted from a character. For instance +.codn "(- #\e9 #\e0) is 9" . +The Unicode value of a character +.code C +can be found using +.codn "(- C #\ex0)" : +the displacement from the NUL character. + +The rules can be stated as a set of restrictions: +.RS +.IP 1 +Two characters may not be added together. +.IP 2 +A character may not be subtracted from an integer (which also rules out +the possibility of computing the additive inverse of a character). +.IP 3 +A character operand may not be opposite to a floating point operand +in any operation. +.IP 4 +A character may not be an operand of multiplication. +.RE + +.PP +.coNP Function @ / +.synb +.mets (/ << divisor ) +.mets (/ < dividend << divisor *) +.syne +.desc The -.code bit -function has a Boolean result, returning the symbol -.code t -if bit -.meta bit -of -.meta value -is set, otherwise -.codn nil . +.code / +function performs floating-point division. Each operands is first +converted to floating-point type, if necessary. In the one-argument +form, the +.meta dividend +argument is omitted. An implicit dividend is present, whose value is +.codn 1.0 , +such that the one-argument form +.code "(/ x)" +is equivalent to the two-argument form +.codn "(/ 1.0 x)" . -If -.meta value -is negative, it is treated as if it had an infinite-bit two's -complement representation. For instance, if value is -.codn -2 , -then the bit -function returns -.code nil -for a -.meta bit -value of zero, and -.code t -for all other values, -since the infinite bit two's complement representation of -.code -2 -is -.codn ...11110 . +If there are two or more arguments, explicitly or by the above equivalence, +then a cumulative division is performed. The +.meta divisor +value is taken into consideration, and divided by the first +.codn divisor . +If another +.code divisor +follows, then that value is divided by that subsequent divisor. +This process repeats until all divisors are exhausted, and the +value of the last division is returned. -.coNP Function @ mask +A division by zero throws an exception of type +.codn numeric-error . + +.coNP Functions @ sum and @ prod .synb -.mets (mask << integer *) +.mets (sum < sequence <> [ keyfun ]) +.mets (prod < sequence <> [ keyfun ]) .syne .desc The -.code mask -function takes zero or more integer arguments, and produces an integer -value which corresponds a bitmask made up of the bit positions specified by the -integer values. +.code sum +and +.code prod +functions operate on an effective sequence of numbers derived from +.metn sequence . -If -.code mask -is called with no arguments, then the return value is zero. +If the +.meta keyfun +argument is omitted, then the effective sequence is the +.meta sequence +argument itself. Otherwise, the effective sequence is understood to be +a projection mapping of the elements of +.meta sequence +through +.meta keyfun +as would be calculated by the +.mono +.meti (mapcar < keyfun << sequence ) +.onom +expression. -If -.code mask -is called with a single argument -.meta integer -then the return value is the same as -that of the expression -.codn "(ash 1 )" : -the value 1 shifted left by -.meta integer -bit positions. If -.meta integer -is zero, then the result is -.codn 1 ; -if -.meta integer -is -.codn 1 , -the -result is -.code 2 -and so forth. If -.meta value -is negative, then the result is zero. +The +.code sum +function returns the left-associative sum of the elements of +the effective sequence calculated as if using the +.code + +function. Similarly, the +.code prod +function calculates the left-associative product of the elements of +the sequence as if using the +.code * +function. If -.code mask -is called with two or more arguments, then the result is a bitwise or of -the masks individually computed for each of the values. - -In other words, the following equivalences hold: +.meta sequence +is empty then +.code sum +returns +.code 0 +and +.code prod +returns +.codn 1 . -.verb - (mask) <--> 0 - (mask a) <--> (ash 1 a) - (mask a b c ...) <--> (logior (mask a) (mask b) (mask c) ...) -.brev +If the effective sequence contains one number, then both functions +return that number. -.coNP Function @ bitset +.coNP Functions @ wrap and @ wrap* .synb -.mets (bitset << integer ) +.mets (wrap < start < end << number ) +.mets (wrap* < start < end << number ) .syne .desc The -.code bitset -function returns a list of the positions of bits which have a value of -1 in a positive -.meta integer -argument, or the positions of bits which have a value of zero in a negative -.meta integer -argument. The positions are ordered from least to greatest. The least -significant bit has position zero. If -.meta integer -is zero, the empty list -.code nil -is returned. - -A negative integer is treated as an infinite bit two's complement -representation. +.code wrap +and +.code wrap* +functions reduce +.meta number +into the range specified by +.meta start +and +.metn end . -The argument may be a character. +Under +.code wrap +the range is inclusive of the +.meta end +value, whereas under +.code wrap* +it is exclusive. -If -.meta integer -.code x -is non-negative, the following equivalence holds: +The following equivalence holds .verb - x <--> [apply mask (bitset x)] + (wrap a b c) <--> (wrap* a (succ b) c) .brev -That is to say, the value of -.code x -may be reconstituted by applying the bit positions returned by -.code bitset -as arguments to the -.code mask -function. +The expression +.code "(wrap* x0 x1 x)" +performs the following calculation: -The value of a negative -.code x -may be reconstituted from its -.code bitset -as follows: +.mono +.mets (+ (mod (- x x0) (- x1 x0)) x0) +.onom -.verb - x <--> (pred (- [apply mask (bitset x)])) -.brev +In other words, first +.meta start +is subtracted from +.metn number . +Then the result is reduced modulo the displacement +between +.code start +and +.codn end . +Finally, +.meta start +is added back to that result, which is returned. -also, more trivially, thus: +.TP* Example: .verb - x <--> (- [apply mask (bitset (- x))]) + ;; perform ROT13 on the string "nop" + [mapcar (opip (+ 13) (wrap #\ea #\ez)) "nop"] -> "abc" .brev -.coNP Function @ width +.coNP Functions @ gcd and @ lcm .synb -.mets (width << integer *) +.mets (gcd << number *) +.mets (lcm << number *) .syne .desc -A two's complement representation of an integer consists of a sign bit and a -mantissa field. The -.code width -function computes the minimum number of bits required for the mantissa portion -of the two's complement representation of the -.meta integer -argument. +.code gcd +function computes the greatest common divisor: the largest positive +integer which divides each +.metn number . -For a nonnegative argument, the width also corresponds to the number of bits -required for a natural binary representation of that value. +The +.code lcm +function computes the lowest common multiple: the smallest positive +integer which is a multiple of +each +.metn number . -Two integer values have a width of zero, namely 0 and -1. This means that these -two values can be represented in a one-bit two's complement, consisting of only -a sign bit: the one-bit two's complement bitfield 1 denotes -1, and 0 denotes -0. +Each +.meta number +must be an integer. -Similarly, two integer values have a width of 1: 1 and -2. The two-bit -two's complement bitfield 01 denotes 1, and 10 denotes -2. +Negative integers are replaced by their absolute values, so +.code "(lcm -3 -4)" +is +.code 12 +and +.code "(gcd -12 -9)" +yields +.codn 3 . -The argument may be a character. +The value of +.code (gcd) +is +.code 0 +and that of +.code (lcm) +is 1 . -.coNP Function @ logcount +The value of +.code "(gcd x)" +and +.code "(lcm x)" +is +.codn "(abs x)" . + +Any arguments of +.code gcd +which are zero are effectively ignored so that +.code "(gcd 0)" +and +.code "(gcd 0 0 0)" +are both the same as +.code (gcd) +and +.code "(gcd 1 0 2 0 3)" +is the same as +.codn "(gcd 1 2 3)" . + +If +.code lcm +has any argument which is zero, it yields zero. + +.coNP Function @ divides .synb -.mets (logcount << integer ) +.mets (divides < d << n ) .syne .desc The -.code logcount -function considers -.meta integer -to have a two's complement representation. If the integer is positive, -it returns the count of bits in that representation whose value is 1. -If -.meta integer -is negative, it returns the count of zero bits instead. If -.meta integer -is zero, the value returned is zero. +.code divides +function tests whether integer +.meta d +divides integer +.metn n . +If this is true, +.code t +is returned, otherwise +.codn nil . -The argument may be a character. +The integers 1 and -1 divide every other integer and themselves. +By established convention, every integer, except zero, divides zero. -.SS* User-Defined Arithmetic Types +For other values, +.meta d +divides +.meta n +if division of +.meta n +by +.meta d +leaves no remainder. -\*(TL makes it possible for the user application program to define structure -types which can participate in arithmetic operations as if they were numbers. -Under most arithmetic functions, a structure object may be used instead of a -number, if that structure object implements a specific method which is required -by that arithmetic function. +.coNP Function @ abs +.synb +.mets (abs << number ) +.syne +.desc +The +.code abs +function computes the absolute value of +.metn number . +If +.meta number +is positive, it is returned. If +.meta number +is negative, its additive inverse is +returned: a positive number of the same type with exactly the same magnitude. -The following paragraphs give general remarks about the method conventions. -Not all arithmetic and bit manipulation functions have a corresponding -method, and a small number of functions do not follow these conventions. +.coNP Function @ signum +.synb +.mets (signum << number ) +.syne +.desc +The +.code signum +function calculates a representation of the sign of +.meta number +as a numeric value. -In the simplest case of arithmetic functions which are unary, the method -takes no argument other than the object itself. Most unary arithmetic functions -expect a structure argument to have a method which has the same name as that -function. For instance, if -.code x -is a structure, then -.code "(cos x)" -will invoke -.codn "x.(cos)" . If -.code x -has no -.code cos -method, then an -.code error -exception is thrown. A few unary methods are not named after the corresponding function. -The unary case of the -.code - -function excepts an object to have a method named -.codn neg ; -thus, -.code "(- x)" -invokes -.codn "x.(neg)" . -Unary division requires a method called -.codn recip ; -thus, -.codn "(/ x)" , -invokes -.codn "x.(recip)" . - -When a structure object is used as an argument in a two-argument (binary) -arithmetic function, there are several cases to consider. If the left argument -to a binary function is an object, then that object is expected to support a -binary method. That method is called with two arguments: the object itself, of -course, and the right argument of the arithmetic operation. In this case, the -method is named after the function. For instance, if -.code x -is an object, then -.code "(+ x 3)" -invokes -.codn "x.(+ 3)" . -If the right argument, and only the right argument, of a binary operation is an -object, then the situation falls into two cases depending on whether the operation -is commutative. If the operation is commutative, then the same method is used -as in the case when the object is the left argument. The arguments are merely reversed. -Thus -.code "(+ 3 x)" -also invokes -.codn "x.(+ 3)" . -If the operation is not commutative, then the object must supply an alternative -method. For most functions, that method is named by a symbol whose name begins -with a -.code r- -prefix. For instance -.code "(mod x 5)" -invokes -.code "x.(mod 5)" -whereas -.code "(mod 5 x)" -invokes -.codn "x.(r-mod 5)" . -Note: the "r" may be remembered as indicating that the object is the -.B right -argument -of the binary operation or that the arguments are -.BR reversed . -Two functions do not follow the -.code r- -convention. These are -.code - -and -.codn / . -For these, the methods used for the object as a right argument, respectively, are -.code -- -and -.codn // . -Thus -.code "(/ 5 x)" -invokes -.code "x.(// 5)" -and -.code "(- 5 x)" -invokes -.codn "x.(-- 5)" . -Several binary functions do not support an object as the right argument. These are -.codn sign-extend , -.code ash -and -.codn bit . +.meta number +is an integer, then +.code signum +returns -1 if the integer is negative, 1 if the integer is positive, +or else 0. -Variadic arithmetic functions, when given three or more arguments, are regarded -as performing a left-associative decimation of the arguments through a binary -function. Thus for instance -.code "(- 1 x 4)" -is understood as -.code "(- (- 1 x) 4)" -where -.code "x.(-- 1)" -is evaluated first. If that method yields an object -.code o -then -.code "o.(- 4)" -is invoked. +If +.meta number +is a floating-point value then +.code signum +returns -1.0 if the value is negative, 1.0 if the value is positive or +else 0.0. -Certain variadic arithmetic functions, if invoked with one argument, just -return that argument: for instance, -.code + +.coNP Functions @, trunc @, floor @ ceil and @ round +.synb +.mets (trunc < dividend <> [ divisor ]) +.mets (floor < dividend <> [ divisor ]) +.mets (ceil < dividend <> [ divisor ]) +.mets (round < dividend <> [ divisor ]) +.syne +.desc +The +.codn trunc , +.codn floor , +.code ceiling and -.code * -are in this category. A special concession exists in these functions: if -their one and only argument is a structure, then that structure is returned -without any error checking, even if it implements no methods related -to arithmetic. +.code round +functions perform division of the +.meta dividend +by the +.metn divisor , +returning an integer quotient. -The following sections describe each of the methods that must be implemented -by an object for the associated arithmetic function to work with that object, -either at all, or in a specific argument position, as the case may be. -These methods are not provided by \*(TL; the application is required to provide -them. +If the +.meta divisor +is omitted, it defaults to 1. -.de bmc -. coNP Method @ \\$1 -. synb -. mets << obj .(\\$1 << arg ) -. syne -. desc -The -. code \\$1 -method is invoked when a structure is used as an argument to the -. code \\$1 -function. +A zero +.meta divisor +results in an exception of type +.codn numeric-error . -If an object -. meta obj -is combined with an argument -. metn arg , -either as -. mono -. meti (\\$1 < obj << arg ) -. onom -or as -. mono -. meti (\\$1 < arg << obj ) -. onom -then, effectively, the method call -. mono -. meti << obj .(\\$1 << arg ) -. onom -takes place, and its return value is taken as the result -of the operation. -.. +If both inputs are integers, +the result is of type integer. -.de bmcv -. coNP Method @ \\$1 -. synb -. mets << obj .(\\$1 << arg ) -. syne -. desc -The -. code \\$1 -method is invoked when a structure is used as an argument to the -. code \\$1 -function together with at least one other operand. +If all inputs are numbers and at least one of them is +floating-point, the others are converted to floating-point +and the result is floating-point. -If an object -. meta obj -is combined with an argument -. metn arg , -either as -. mono -. meti (\\$1 < obj << arg ) -. onom -or as -. mono -. meti (\\$1 < arg << obj ) -. onom -then, effectively, the method call -. mono -. meti << obj .(\\$1 << arg ) -. onom -takes place, and its return value is taken as the result -of the operation. -.. +The +.code dividend +input may be a range. In this situation, the operation is +recursively distributed over the +.code from +and +.code to +fields of the range, individually matched against the +.metn divisor , +and the result is a range composed of these two individual +quotients. -.de bmnl -. coNP Method @ \\$1 -. synb -. mets << obj .(\\$1 << arg ) -. syne -. desc +When the quotient is a scalar value, +.code trunc +returns the closest integer, in the zero direction, +from the value of the quotient. The -. code \\$1 -method is invoked when the structure -. meta obj -is used as the left argument of the -. code \\$1 -function. +.code floor +function returns the highest integer which does not exceed +the value of the quotient. That is to say, the division is +truncated to an integer value toward negative infinity. +The +.code ceil +function the lowest integer which is not below the value +of the quotient. +does not exceed the value of +.metn dividend . +That is to say, the division is truncated to an integer +value toward positive infinity. The +.code round +function returns the nearest integer to the quotient. +Exact halfway cases are rounded to the integer away from +zero so that +.code "(round -1 2)" +yields +.code -1 +and +.code "(round 1 2)" +yields 1, -If an object -. meta obj -is combined with an argument -. metn arg , -as -. mono -. meti (\\$1 < obj << arg ) -. onom -then, effectively, the method call -. mono -. meti << obj .(\\$1 << arg ) -. onom -takes place, and its return value is taken as the result -of the operation. -.. +Note that for large floating point values, due to the limited +precision, the integer value corresponding to the mathematical +floor or ceiling may not be available. -.de bmnr -. coNP Method @ \\$1 -. synb -. mets << obj .(\\$1 << arg ) -. syne -. desc +.TP* "Dialect note:" +In ANSI Common Lisp, the +.code round +function chooses the nearest even integer, rather than +rounding halfway cases away from zero. \*(TX's choice +harmonizes with the semantics of the +.code round +function in the C language. + +.coNP Function @ mod +.synb +.mets (mod < dividend << divisor ) +.syne +.desc The -. code \\$1 -method is invoked when the structure -. meta obj -is used as the right argument of the -. code \\$2 -function. +.code mod +function performs a modulus operation. Firstly, the absolute value +of +.meta divisor +is taken to be a modulus. Then a residue of +.meta dividend +with respect to +.meta modulus +is calculated. The residue's sign follows +that of the sign of +.metn divisor . +That is, it is the smallest magnitude +(closest to zero) residue of +.meta dividend +with respect to the absolute +value of +.metn divisor , +having the same sign as +.metn divisor . +If the operands are integer, the result is an integer. If either operand +is of type float, then the result is a float. The modulus operation is +then generalized into the floating point domain. For instance the expression +.code "(mod 0.75 0.5)" +yields a residue of 0.25 because 0.5 "goes into" 0.75 only +once, with a "remainder" of 0.25. -If an object -. meta obj -is combined with an argument -. metn arg , +If +.meta divisor +is zero, +.code mod +throws an exception of type +.codn numeric-error . + +.coNP Functions @, trunc-rem @, floor-rem @ ceil-rem and @ round-rem +.synb +.mets (trunc-rem < dividend <> [ divisor ]) +.mets (floor-rem < dividend <> [ divisor ]) +.mets (ceil-rem < dividend <> [ divisor ]) +.mets (round-rem < dividend <> [ divisor ]) +.syne +.desc +These functions, respectively, perform the same division operation as -. mono -. meti (\\$2 < arg << obj ) -. onom -then, effectively, the method call -. mono -. meti << obj .(\\$1 << arg ) -. onom -takes place, and its return value is taken as the result -of the operation. -.. +.codn trunc , +.codn floor , +.codn ceil , +and +.codn round , +referred to here as the respective target functions. -.de umv -. coNP Method @ \\$1 -. synb -. mets << obj .(\\$1) -. syne -. desc -The -. code \\$1 -method is invoked when the structure -. meta obj -is used as the sole argument to the -. code \\$2 -function. +If the +.meta divisor +is missing, it defaults to 1. -If an object -. meta obj -is passed to the function as -. mono -. meti (\\$2 << obj ) -. onom -then, effectively, the method call -. mono -. meti << obj .(\\$1) -. onom -takes place, and its return value is taken as the result -of the operation. -.. +Each function returns a list of two values: a +.meta quotient +and a +.metn remainder . +The +.meta quotient +is exactly the same value as what would be returned by the +respective target function for the same inputs. -.de bma -. coNP Method @ \\$1 -. synb -. mets << obj .(\\$1 << arg ) -. syne -. desc The -. code \\$1 -method is invoked when the -. code \\$1 -function is invoked with two operands, and the structure -. meta obj -is the left operand. -The method is also invoked when the -. code \\$2 -function is invoked with two operands, and -.meta obj -is the right operand. +.meta remainder +value obeys the following identity: -If an object -. meta obj -is combined with an argument -. metn arg , -either as -. mono -. meti (\\$1 < obj << arg ) -. onom -or as -. mono -. meti (\\$2 < arg << obj ) -. onom -then, effectively, the method call -. mono -. meti << obj .(\\$1 << arg ) -. onom -takes place, and its return value is taken as the result -of the operation. -.. +.mono +.mets (eql < remainder (- < dividend >> (* divisor << quotient ))) +.onom -.de um -. coNP Method @ \\$1 -. synb -. mets << obj .(\\$1) -. syne -. desc -The -. code \\$1 -method is invoked when a structure is used as the argument to the -. code \\$1 -function. +If +.meta divisor +is zero, these functions throw an exception of type +.codn numeric-error . -If an object -. meta obj -is passed to the function as -. mono -. meti (\\$1 << obj ) -. onom -then, effectively, the method call -. mono -. meti << obj .(\\$1) -. onom -takes place, and its return value is taken as the result -of the operation. -.. +.coNP Functions @, sin @, cos @, tan @, asin @, acos @ atan and @ atan2 +.synb +.mets (sin << radians ) +.mets (cos << radians ) +.mets (tan << radians ) +.mets (atan << slope ) +.mets (atan2 < y << x ) +.mets (asin << num ) +.mets (acos << num ) +.syne +.desc +These trigonometric functions convert their argument to floating point and +return a float result. The +.codn sin , +.code cos +and +.code tan +functions compute the sine and +cosine and tangent of the +.meta radians +argument which represents an angle +expressed in radians. The +.codn atan , +.code acos +and +.code asin +are their respective inverse +functions. The +.meta num +argument to +.code asin +and +.code acos +must be in the +range -1.0 to 1.0. The +.code atan2 +function converts the rectilinear coordinates +.meta x +and +.meta y +to an angle in polar coordinates in the range [0, 2\(*p). -.de tmnl -. coNP Method @ \\$1 -. synb -. mets << obj .(\\$1 < arg1 << arg2 ) -. syne -. desc +.coNP Functions @, exp @, log @ log10 and @ log2 +.synb +.mets (exp << arg ) +.mets (log << arg ) +.mets (log10 << arg ) +.mets (log2 << arg ) +.syne +.desc The -. code \\$1 -method is invoked when the structure -. meta obj -is used as the left argument of the -. code \\$1 -function. +.code exp +function calculates the value of the transcendental number e raised to +the exponent +.metn arg . -If an object -. meta obj -is combined with arguments -. meta arg1 -and -. metn arg2 , -as -. mono -. meti (\\$1 < obj < arg1 << arg2 ) -. onom -then, effectively, the method call -. mono -. meti << obj .(\\$1 < arg1 << arg2 ) -. onom -takes place, and its return value is taken as the result -of the operation. -.. +The +.code log +function calculates the base e logarithm of +.metn arg , +which must be a positive value. -.bmcv + -.bmnl - -.bmnr -- - -.umv neg - -.bmcv * -.bmnl / -.bmnr // / -.umv recip / -.um abs -.um signum -.bmnl trunc -.bmnr r-trunc trunc -.umv trunc1 trunc -.bmnl mod -.bmnr r-mod mod -.bmnl expt -.bmnr r-expt expt -.tmnl exptmod +The +.code log10 +function calculates the base 10 logarithm of +.metn arg , +which must be a positive value. -Note: the -.code exptmod -function doesn't support structure objects in the second and -third argument positions. The -.meta exponent -and -.meta base -arguments must be integers. +The +.code log2 +function calculates the base 2 logarithm of +.metn arg , +which must be a positive value. -.um isqrt -.um square -.bma > < -.bma < > -.bma >= <= -.bma <= >= -.bmc = -.um zerop -.um plusp -.um minusp -.um evenp -.um oddp -.bmnl floor -.bmnr r-floor floor -.umv floor1 floor -.bmnl ceil -.bmnr r-ceil ceil -.umv ceil1 ceil -.bmnl round -.bmnr r-round round -.umv round1 round -.um sin -.um cos -.um tan -.um asin -.um acos -.um atan -.bmnl atan2 -.bmnr r-atan2 atan2 -.um log -.um log2 -.um log10 -.um exp -.um sqrt -.bmcv logand -.bmcv logior -.bmnl lognot -.bmnr r-lognot lognot -.umv lognot1 lognot -.bmnl logtrunc -.bmnr r-logtrunc logtrunc -.bmnl sign-extend +.coNP Functions @, expt @ sqrt and @ isqrt +.synb +.mets (expt < base << exponent *) +.mets (sqrt << arg ) +.mets (isqrt << arg ) +.syne +.desc +The +.code expt +function raises +.meta base +to zero or more exponents given +by the +.meta exponent +arguments. +.code "(expt x)" +is equivalent to +.codn "(expt x 1)" , +and yields +.code x +for all +.codn x . +For three or more arguments, the operation is right-associative. +That is to say, +.code "(expt x y z)" +is equivalent to +.codn "(expt x (expt y z))" , +similarly to the way nested exponents work in standard algebraic +notation. -Note: the -.code sign-extend -function doesn't support a structure as the right argument, -.metn bits , -which must be an integer. +Exponentiation is done pairwise using a binary operation. +If both operands to this binary operation are non-negative integers, then the +result is an integer. -.bmnl ash +If the exponent is negative, and the base is zero, the situation is +treated as a division by zero: an exception of type +.code numeric-error +is thrown. Otherwise, a negative exponent is converted to floating-point, +if it already isn't, and a floating-point exponentiation is performed. -Note: the -.code ash -function doesn't support a structure as the right argument, -.metn bits , -which must be an integer. +If either operand is a float, then the other +operand is converted to a float, and a floating point exponentiation +is performed. Exponentiation that would produce a complex number is +not supported. -.bmnl bit +The +.code sqrt +function produces a floating-point square root of +.metn arg , +which is converted from integer to floating-point if necessary. Negative +operands are not supported. -Note: the -.code bit -function doesn't support a structure as the right argument, -.metn bit , +The +.code isqrt +function computes the integer square root of +.metn arg , which must be an integer. +The integer square root is a value which is the +greatest integer that is no greater than the real square root of +.metn arg . +The input value must be an integer. -.um width -.um logcount -.um bitset - -.SS* Exception Handling +.coNP Function @ exptmod +.synb +.mets (exptmod < base < exponent << modulus ) +.syne +.desc +The +.code exptmod +function performs modular exponentiation and accepts only integer +arguments. Furthermore, +.meta exponent +must be a non-negative and +.meta modulus +must be positive. -An -.I exception -in \*(TX is a special event in the execution of the program which -results in transfer of control. An exception is identified by a symbol, -known as the -.IR "exception type" , -and it carries zero or more arguments, called the -.IR "exception arguments" . +The return value is +.meta base +raised to +.metn exponent , +and reduced to the +least positive residue modulo +.metn modulus . -When an exception is initiated, it is said to be -.IR thrown . -When an exception is thrown, \*(TX enters into exception processing -mode. Exception processing mode terminates in one of several ways: -.IP - -A -.I catch -is found which matches the exception, and control is transferred -to the catch. Catches are defined by the -.code catch -macro. -.IP - -A handler accepts the exception by performing a non-local transfer. -Handlers are defined by the -.code handler-bind -operator or -.code handle -macro. -.IP - -If no catch or accepting handler is found, control is transferred -to the function stored in the -.code *unhandled-hook* -variable. If that function returns, then unwinding is performed -after which the process terminates (unless the unwinding actions -intercept the control to prevent that). -.IP - -If no catch or accepting handler is found and -.code *unhandled-hook* -is -.codn nil , -then a built-in strategy for handling the exception is invoked, -consisting of unwinding, and then printing some informational messages and -terminating. -.PP +.coNP Function @ square +.synb +.mets (square << argument ) +.syne +.desc +The +.code square +function returns the product of +.meta argument +with itself. The following +equivalence holds, except that +.code x +is evaluated only once in the the +.code square +expression: -.NP* Catches and Handlers +.verb + (square x) <--> (* x x) +.brev -There are two ways by which exceptions are handled: catches and handlers. -Catches and handlers are similar, but different. -A catch is an exit point associated with an active scope. When an exception is -handled by a catch, the form which threw the exception is abandoned, and unwinding -takes place to the catch site, which receives the exception type and arguments. -A handler is also associated with an active scope. However, it is a function, -and not a dynamic exit point. When an exception is passed to handler, -unwinding does not take place; rather, the function is called. The function then -either completes the exception handling by performing a non-local transfer, -or else declines the exception by performing an ordinary return. +.coNP Function @ cum-norm-dist +.synb +.mets (cum-norm-dist << argument ) +.syne +.desc +The +.code cum-norm-dist +function calculates an approximation to the cumulative normal +distribution function: the integral, of the normal distribution function, from +negative infinity to the +.metn argument . -Catches and handlers are identified by exception type symbols. A catch or -handler is eligible to process an exception if it handles a type which is -a supertype of the exception which is being processed. Handles and catches -are located in a combined search which proceeds from the innermost nesting -to the outermost. When an eligible handle is encountered, it is called. If -it returns, the search continues. When an eligible catch is encountered, -the search stops and a control transfer takes place to the catch site. +.coNP Function @ inv-cum-norm +.synb +.mets (inv-cum-norm << argument ) +.syne +.desc +The +.code inv-cum-norm +function calculates an approximate to the inverse of the cumulative +normal distribution function. The argument, a value expected to lie +in the range [0, 1], represents the integral of the normal distribution +function from negative infinity to some domain point +.IR p . +The function calculates the approximate value of +.IR p . +The minimum value returned is -10, and the maximum value returned is 10, +regardless of how closely the argument approaches, respectively, +the 0 or 1 integral endpoints. For values less than zero, or exceeding 1, the +values returned, respectively, are -10 and 10. -.NP* Handlers and Sandboxing +.coNP Functions @ n-choose-k and @ n-perm-k +.synb +.mets (n-choose-k < n << k ) +.mets (n-perm-k < n << k ) +.syne +.desc +The +.code n-choose-k +function computes the binomial coefficient nCk which +expresses the number of combinations of +.meta k +items that can be chosen from +a set of +.metn n , +where combinations are subsets. -Because handlers execute in the dynamic context of the exception origin, -without any unwinding having taken place, they expose a potential route -of sandbox escape via the package system, unless special steps are taken. -The threat is that code at the handler site could take advantage of -the current value of the -.code *package* -and -.code *package-alist* -variables established at the exception throw site to gain inappropriate access -to symbols. +The +.code n-perm-k +function computes nPk: the number of permutations of size +.meta k +that can be drawn from a set of +.metn n , +where permutations are sequences, +whose order is significant. -For this reason, when a handler is established, the current values of -.code *package* +The calculations only make sense when +.meta n and -.code *package-alist* -are recorded into the handler frame. -When that handler is later invoked, it executes in a dynamic environment -in which those variables are bound to the previously noted values. - -The catch mechanism doesn't do any such thing because the unwinding -which is performed prior to the invocation of a catch implicitly -restores the values of -.B all -special variables to the values they had at the time the frame was -established. - -.NP* Exception Type Hierarchy +.meta k +are nonnegative integers, and +.meta k +does not exceed +.metn n . +The behavior is not specified if these conditions +are not met. -Exception type symbols are arranged -in an inheritance hierarchy, at whose top the symbol +.coNP Functions @, fixnump @, bignump @, integerp @ floatp and @ numberp +.synb +.mets (fixnump << object ) +.mets (bignump << object ) +.mets (integerp << object ) +.mets (floatp << object ) +.mets (numberp << object ) +.syne +.desc +These functions test the type of +.metn object , +returning .code t -is is the supertype of every exception type, and the +if it is an object +of the implied type, .code nil -symbol is at the bottom, the subtype of every exception type. - -Keyword symbols may be used as exception types. +otherwise. The +.codn fixnump , +.code bignump +and +.code floatp +functions return +.code t +if the object is of the basic type +.codn fixnum , +.code bignum +or +.codn float . +The function +.code integerp +returns true of +.meta object +is either a +.code fixnum +or +a +.codn bignum . +The function +.code numberp +returns +.code t +if +.meta object +is either +a +.codn fixnum , +.code bignum +or +.codn float . -Every symbol is its own supertype and subtype. Thus whenever X is known to be a -subtype of Y, it is possible that X is exactly Y. +.coNP Functions @ zerop and @ nzerop +.synb +.mets (zerop << number ) +.mets (nzerop << number ) +.syne +.desc The -.code defex -macro registers exception supertype/subtype relationships among symbols. - -The following tree diagram shows the relationships among \*(TL's built-in -exception symbols. Not shown is the exception symbol -.codn nil , -subtype of every exception type: - -.verb - t ----+--- warning - | - +--- restart --- continue - | - +--- error ---+--- type-error - | - +--- internal-error - | - +--- panic - | - +--- numeric-error - | - +--- range-error - | - +--- query-error - | - +--- file-error -------+--- path-not-found - | | - | +--- path-exists - | | - | +--- path-permission - | - +--- process-error - | - +--- socket-error - | - +--- system-error - | - +--- alloc-error - | - +--- timeout-error - | - +--- assert - | - +--- syntax-error - | - +--- eval-error -.brev - -Program designers are encouraged to derive new error exceptions from the -.code error -type. The -.code restart -type is intended to be the root of a hierarchy of exception -types used for denoting restart points: designers are encouraged -to derive restarts from this type. - -.NP* Dialect Notes - -Exception handling in \*(TL provides capabilities similar to the condition -system in ANSI Common Lisp. The implementation and terminology differ. - -Most obviously, ANSI CL uses the "condition" term, whereas \*(TL uses "exception". - -In ANSI CL, a condition is "raised", whereas a \*(TL exception is "thrown". - -In ANSI CL, when a condition is raised, a condition object is created. Condition -object are similar to class objects, but are not required to be in the Common Lisp -Object System. They are related by inheritance and can have properties. \*(TL -exceptions are unencapsulated: they consist of a symbol, plus zero or more -arguments. The symbols are related by inheritance. +.code zerop +function tests +.meta number +for equivalence to zero. The argument must be +a number or character. It returns +.code t +for the integer value +.code 0 +and for the floating-point +value +.codn 0.0 . +For other numbers, it returns +.codn nil . +It returns +.code t +for the null character +.code #\enul +and +.code nil +for all other characters. -When a condition is raised in ANSI CL, the dynamic scope is searched for a -handler, which is an ordinary function which receives the condition. No -unwinding or non-local transfer takes place. The handler can return, in which -case the search continues. Matching the condition to the handler is by -inheritance. Handler functions are bound to exception type names. -If a handler chooses to actually handle a condition (thereby terminating -the search) it must itself perform some kind of dynamic control transfer, -rather than return normally. ANSI CL provides a dynamic control mechanism -known as restarts which is usually used for this purpose. A condition handler -may invoke a particular restart handler. Restart handlers are similar to -exception handlers: they are functions associated with symbols in the -dynamic environment. +If +.meta number +is a range, then +.code zerop +returns +.code t +if both of the range endpoints individually satisfy +.codn zerop . -In \*(TL exceptions are a unification of conditions and restarts. From an ANSI CL -perspective, \*(TL exceptions are a lot like CL restarts, except that the -symbols are arranged in an inheritance hierarchy. \*(TL exceptions are used -both as the equivalent of ANSI CL conditions and as restarts. +The +.code nzerop +function is the logical inverse of +.codn zerop : +it returns +.code t +for those arguments for which +.code zerop +returns +.code nil +and +.IR "vice versa" . -In \*(TL the terminology "catch" and "handle" is used in a very specific way. -To handle an exception means to receive it without unwinding, with the possibility -of declining to handle it, so that the search continues for another handler. -To catch an exception means to match an exception to a catch handler, terminate -the search, unwind and pass control to the handler. +.coNP Functions @ plusp and @ minusp +.synb +.mets (plusp << number ) +.mets (minusp << number ) +.syne +.desc +These functions test whether a number is positive or negative, +returning +.code t +or +.codn nil , +as the case may be. -\*(TL provides an operator called -.code handler-bind -for specifying handlers. It has a different syntax from ANSI CL's -.codn handler-bind . -\*(TL provides a macro called -.code handle -which simplifies the use of -.codn handler-bind . -This macro superficially resembles ANSI CL's -.codn handler-case , -but is semantically different. The most notable difference is that the bodies -of handlers established by -.code handler-bind -execute without any unwinding taking place and may return normally, thereby -declining to take the exception. In other words, -.code handle -has the same semantics as -.codn handler-bind , -providing only convenient syntax. +The argument may also be a character. All characters other than +the null character +.code #\enul +are positive. No character is negative. -\*(TL provides a macro called -.code catch -which has the same syntax as -.code handle -but specifies a catch point for exceptions. If, during an exception search, a -.code catch -clause matches an exception, a dynamic control transfer takes place -from the throw site to the catch site. Then the clause body is executed. +.coNP Functions @ evenp and @ oddp +.synb +.mets (evenp << integer ) +.mets (oddp << integer ) +.syne +.desc The -.code catch -macro resembles ANSI CL's -.code restart-case -or possibly -.codn handler-case , -depending on point of view. - -\*(TL provides unified introspection over handler and catch frames. -A program can programmatically discover what handler and catches are -available in a given dynamic scope. ANSI CL provides introspection -over restarts only; the standard doesn't specify any mechanism for -inquiring what condition handlers are bound at a given point in -the execution. - -.TP* Example: +.code evenp +and +.code oddp +functions require integer arguments. +.code evenp +returns +.code t +if +.meta integer +is even (divisible by two), otherwise it returns +.codn nil . +.code oddp +returns +.code t +if +.meta integer +is not divisible by two (odd), otherwise +it returns +.codn nil . -The following two examples express a similar approach implemented -using ANSI Common Lisp conditions and restarts, and then using \*(TL -exceptions. +.coNP Functions @, succ @, ssucc @, sssucc @, pred @ ppred and @ pppred +.synb +.mets (succ << number ) +.mets (ssucc << number ) +.mets (sssucc << number ) +.mets (pred << number ) +.mets (ppred << number ) +.mets (pppred << number ) +.syne +.desc +The +.code succ +function adds 1 to its argument and returns the resulting value. +If the argument is an integer, then the return value is the successor +of that integer, and if it is a character, then the return value +is the successor of that character according to Unicode. -.verb - ;; Common Lisp - (define-condition foo-error (error) - ((arg :initarg :arg :reader foo-error-arg))) +The +.code pred +function subtracts 1 from its argument, and under similar considerations +as above, the result represents the predecessor. - (defun raise-foo-error (arg) - (restart-case - (let ((c (make-condition 'foo-error :arg arg))) - (error c)) - (recover (recover-arg) - (format t "recover, arg: ~s~%" recover-arg)))) +The +.code ssucc +and +.code sssucc +functions add 2 and 3, respectively. Similarly, +.code ppred +and +.code pppred +subtract 2 and 3 from their argument. - (handler-bind ((foo-error - (lambda (cond) - (format t "handling foo-error, arg: ~s~%" - (foo-error-arg cond)) - (invoke-restart 'recover 100)))) - (raise-foo-error 200)) -.brev +.coNP Functions @, > @, < @, >= @ <= and @ = +.synb +.mets (> < number << number *) +.mets (> < number << number *) +.mets (>= < number << number *) +.mets (<= < number << number *) +.mets (= < number << number *) +.syne +.desc +These relational functions compare characters and numbers for numeric equality +or inequality. The arguments must be one or more numbers or characters. -The output of the above is: +If just one argument is given, then these functions all return +.codn t . -.verb - handling foo-error, arg: 200 - recover, arg: 100 -.brev +If two arguments are given then, they are compared as follows. +First, if the numbers do not have the same type, then the one +which has the lower ranking type is converted to the type of +the other, according to this ranking: character < integer < float. +For instance if a character and integer are compared, the character +is converted to integer. Then a straightforward numeric comparison +is applied. -The following is possible \*(TL equivalent for the above Common Lisp example. -It produces identical output. +Three or more arguments may be given, in which case the comparison proceeds +pairwise from left to right. For instance in +.codn "(< a b c)" , +the comparison +.code "(< a b)" +is performed in isolation. If the comparison is false, then +.code nil +is returned, otherwise +the comparison +.code "(< b c)" +is performed in isolation, and if that is false, +.code nil +is returned, otherwise +.code t +is returned. Note that it is possible for +.code b +to +undergo two different conversions. For instance in the +.mono +.meti (< < float < character << integer ) +.onom +comparison, +.meta character +will first convert to a floating-point representation +of its Unicode value so that it can be compared to +.metn float , +and if that comparison succeeds, then in the second comparison, +.meta character +will be converted to integer so that it can be compared to +.metn integer . -.verb - (defex foo-error error) +.coNP Function @ /= +.synb +.mets (/= << number *) +.syne +.desc +The arguments to +.code /= +may be numbers or characters. The +.code /= +function returns +.code t +if no two of its arguments are numerically equal. That is to say, if there +exist some +.code a +and +.code b +which are distinct arguments such that +.code "(= a b)" +is true, then +the function returns +.codn nil . +Otherwise it returns +.codn t . - (defex recover restart) ;; recommended practice +.coNP Functions @ max and @ min +.synb +.mets (max < first-arg << arg *) +.mets (min < first-arg << args *) +.syne +.desc +The +.code max +and +.code min +functions determine and return the highest or lowest +value from among their arguments. - (defun raise-foo-error (arg) - (catch - (throw 'foo-error arg) - (recover (recover-arg) - (format t "recover, arg: ~s\en" recover-arg)))) +If only +.meta first-arg +is given, that value is returned. - (handle - (raise-foo-error 200) - (foo-error (arg) - (format t "handling foo-error, arg: ~s\en" arg) - (throw 'recover 100))) -.brev +These functions are type generic, since they compare arguments +using the same semantics as the +.code less +function. -To summarize the differences: exceptions serve as both -conditions and restarts in \*(TX. The same -.code throw -function is used to initiate exception handling for -.code foo-error -and then to transfer control out of the handler -to the recovery code. The handler accepts one exception -by raising another. +If two or more arguments are given, then +.code "(max a b)" +is equivalent to +.codn "(if (less a b) b a)" , +and +.code "(min a b)" +is equivalent to +.codn "(if (less a b) a b)" . +If the operands do not +have the same type, then one of them is converted to the type of the other; +however, the original unconverted values are returned. For instance +.code "(max 4 3.0)" +yields the integer +.codn 4 , +not +.codn 4.0 . -When an exception symbol is used for restarting, it is -a recommended practice to insert it into the inheritance -hierarchy rooted at the -.code restart -symbol, either by inheriting directly from -.code restart -or from an exception subtype of that symbol. +If three or more arguments are given, +.code max +and +.code min +reduce the arguments in a left-associative manner. +Thus +.code "(max a b c)" +means +.codn "(max (max a b) c)" . -.coNP Functions @, throw @ throwf and @ error +.coNP Function @ clamp .synb -.mets (throw < symbol << arg *) -.mets (throwf < symbol < format-string << format-arg *) -.mets (error < format-string << format-arg *) +.mets (clamp < low < high << val ) .syne .desc -These functions generate an exception. The -.code throw -and -.code throwf -functions generate -an exception identified by -.metn symbol , -whereas -.code error -throws an exception of -type -.codn error . -The call -.code "(error ...)" -can be regarded as a shorthand for -.codn "(throwf 'error ...)" . - The -.code throw -function takes zero or more additional arguments. These arguments -become the arguments of a -.code catch -handler which takes the exception. The -handler will have to be capable of accepting that number of arguments. +.code clamp +function clamps value +.meta val +into the range +.meta low +to +.metn high . The -.code throwf -and -.code error -functions generate an exception which has a single -argument: a character string created by a formatted print to a string stream -using the -.code format -string and additional arguments. +.code clamp +function returns +.meta low +if +.meta val +is less than +.metn low . +If +.meta val +is greater than or equal to +.metn low , +but less than +.metn high , +then it returns +.metn val . +Otherwise it returns +.metn high . -.coNP Macros @ catch and @ catch* +More precisely, +.code "(clamp a b c)" +is equivalent to +.codn "(max a (min b c))" . + +.coNP Function @ bracket .synb -.mets (catch < try-expression -.mets \ \ >> {( symbol <> ( arg *) << body-form *)}*) -.mets (catch* < try-expression -.mets \ \ >> {( symbol >> ( type-arg << arg *) << body-form *)}*) -.mets (catch** < try-expression -.mets \ \ >> {( symbol < desc >> ( type-arg << arg *) << body-form *)}*) +.mets (bracket < value << level *) .syne .desc The -.code catch -macro establishes an exception catching block around -the -.metn try-expression . +.code bracket +function's arguments consist of one required +.meta value +followed by +.I n +.meta level +arguments. The -.meta try-expression -is followed by zero or more -catch clauses. Each catch clause consists of a symbol which denotes -an exception type, an argument list, and zero or more body forms. +.meta level +arguments are optional; in other words, +.I n +may be zero. -If -.meta try-expression -terminates normally, then the catch clauses -are ignored. The catch itself terminates, and its return value is -that of the -.metn try-expression . - -If -.meta try-expression -throws an exception which is a subtype of one or more of -the type symbols given in the exception clauses, then the first (leftmost) such -clause becomes the exit point where the exception is handled. -The exception is converted into arguments for the clause, and the clause -body is executed. When the clause body terminates, the catch terminates, -and the return value of the catch is that of the clause body. +The +.code bracket +function calculates the +.I bracket +of the +.meta value +argument: a zero-based positional index of the value, in relation to the +.meta level +arguments. +Each of the +.meta level +arguments, of which there may be none, is associated with +an integer index, starting at zero, in left to right order. The +.meta level +arguments are examined in that order. When a +.meta level +argument is encountered which exceeds +.metn value , +that +.meta level +argument's index is returned. If -.meta try-expression -throws an exception which is not a subtype of any of -the symbols given in the clauses, then the search for an exit point for -the exception continues through the enclosing forms. The catch clauses -are not involved in the handling of that exception. - -When a clause catches an exception, the number of arguments in the catch must -match the number of elements in the exception. A catch argument list -resembles a function or lambda argument list, and may be dotted. For instance -the clause -.code "(foo (a . b))" -catches an exception subtyped from -.codn foo , -with one or -more elements. The first element binds to parameter -.codn a , -and the rest, if any, -bind to parameter -.codn b . -If there is only one element, -.code b -takes on the value -.codn nil . +.meta value +exceeds all of the +.meta level +arguments, then +.I n +is returned. -The -.code catch* -macro is a variant of -.code catch -with the following difference: when -.code catch* -invokes a clause, it passes the exception symbol as the leftmost argument -.metn type-arg . -Then the exception arguments follow. In contrast, -only the exception arguments are passed to the clauses of -.codn catch . +Determining whether +.meta value +exceeds a +.meta level +is performed using the +.code less +function. -The -.code catch** -macro is a further variant, which differs from -.code catch* -by requiring each catch clause to provide a description -.metn desc , -an expression which evaluates to a character string. -The -.meta desc -expressions are evaluated in left-to-right order prior to the -evaluation of -.metn try-expression . +.TP* Examples: -Also see: the -.code unwind-protect -operator, and the functions -.codn throw , -.code throwf -and -.codn error , -as well as the -.code handler-bind -operator and -.code handler -macro. +.verb + (bracket 42) -> 0 + (bracket 5 10) -> 0 + (bracket 15 10) -> 1 + (bracket 15 10 20) -> 1 + (bracket 15 10 20 30) -> 1 + (bracket 20 10 20 30) -> 2 + (bracket 35 10 20 30) -> 3 + (bracket "a" "aardvark" "zebra") -> 0 + (bracket "ant" "aardvark" "zebra") -> 1 + (bracket "zebu" "aardvark" "zebra") -> 2 +.brev -.coNP Operator @ unwind-protect +.coNP Functions @, int-str @ flo-str and @ num-str .synb -.mets (unwind-protect < protected-form << cleanup-form *) +.mets (int-str < string <> [ radix ]) +.mets (flo-str << string ) +.mets (num-str << string ) .syne .desc -The -.code unwind-protect -operator evaluates -.meta protected-form -in such a way that no matter how the execution of -.meta protected-form -terminates, the -.metn cleanup-form -s -will be executed. +These functions extract numeric values from character string +.metn string . +Leading whitespace in +.metn string , +if any, is skipped. If no digits can be successfully extracted, then +.code nil +is returned. Trailing material which does not contribute to the number is +ignored. The -.metn cleanup-form -s, -however, are not protected. If a -.meta cleanup-form -terminates via -some non-local jump, the subsequent -.metn cleanup-form -s -are not evaluated. - -.metn cleanup-form -s -themselves can "hijack" a non-local control transfer such -as an exception. If a -.meta cleanup-form -is evaluated during the processing of -a dynamic control transfer such as an exception, and that -.meta cleanup-form -initiates its own dynamic control transfer, the original control transfer -is aborted and replaced with the new one. +.code int-str +function converts a string of digits in the specified +.meta radix +to an integer value. If +.meta radix +isn't specified, it defaults to 10. +Otherwise it must be an integer in the range 2 to 36, or else the character +.codn #\ec . -The exit points for dynamic control transfers are removed as unwinding takes -place. That is to say, at the start of a dynamic control transfer, a search -takes place for the target exit point. That search might skip other exit points -which aren't targets of the control transfer. Those skipped exit points are left -undisturbed and are still visible during unwinding until their individual -binding forms are abandoned. Thus at the time of execution of an -.code unwind-protect -.metn cleanup-form , -all of the exit points of dynamically surrounding forms are still visible, even -ones which are nearer than the targeted exit point. +For radices above 10, letters of the alphabet +are used for digits: +.code A +represent a digit whose value is 10, +.code B +represents 11 and +so forth until +.codn Z . +Upper and lower case letters are recognized. +Any character which is not a digit of the specified radix is regarded +as the start of trailing junk at which the extraction of the digits stops. -.TP* Example: -.verb - (block foo - (unwind-protect - (progn (return-from foo 42) - (format t "not reached!\en")) - (format t "cleanup!\en"))) -.brev +When +.meta radix +is specified as the character object +.codn #\ec , +this indicates that a C-language-style integer constant should be +recognized. If, after any optional sign, the remainder of +.meta string +begins with the character pair +.code 0x +then that pair is considered removed from the string, and it is treated +as base 16 (hexadecimal). If, after any optional sign, the remainder of +.meta string +begins with a leading zero not followed by +.codn x , +then the radix is taken to be 8 (octal). In scanning these formats, +.code int-str +function is not otherwise constrained by C language representational +limitations. Specifically, the input values are taken to be the printed +representation of arbitrary-precision integers and treated accordingly. -In this example, the protected -.code progn -form terminates by returning from -block -.codn foo . -Therefore the form does not complete and so the -output -.str not reached! -is not produced. However, the cleanup form -executes, producing the output -.strn cleanup! . +The +.code flo-str +function converts a floating-point decimal notation to a nearby +floating point value. The material which contributes to the value +is the longest match for optional leading space, followed by a +mantissa which consists of an optional sign followed by a mixture of at least +one digit, and at most one decimal point, optionally followed by an exponent +part denoted by the letter +.code E +or +.codn e , +an optional sign and one or more optional exponent digits. +If the value specified by +.meta string +is out of range of the floating-point representation, then +.code nil +is returned. +The +.code num-str +function converts a decimal notation to either an integer as if by +a radix 10 application of +.codn int-str , +or to a floating point value as if by +.codn flo-str . +The floating point interpretation is chosen if the possibly empty +initial sequence of digits (following any whitespace and optional sign) is +followed by a period, or by +.code e +or +.codn E . -.coNP Macro @ ignerr +.coNP Functions @ int-flo and @ flo-int .synb -.mets (ignerr << form *) +.mets (int-flo << float ) +.mets (flo-int << integer ) .syne .desc +These functions perform numeric conversion between integer and floating point +type. The +.code int-flo +function returns an integer by truncating toward zero. The -.code ignerr -macro operator evaluates each -.meta form -similarly to the -.code progn -operator. If no forms are present, it returns -.codn nil . -Otherwise it evaluates each -.meta form -in turn, yielding the value of the last one. - -If the evaluation of any -.meta form -is abandoned due to an exception of type -.codn error , -the code generated by the -.code ignerr -macro catches this exception. In this situation, -the execution of the -.code ignerr -form terminates without evaluating the remaining -forms, and yields -.codn nil . +.code flo-int +function returns an exact floating point value corresponding to +.metn integer , +if possible, otherwise an approximation using a nearby +floating point value. -.coNP Macro @ ignwarn +.coNP Functions @ tofloat and @ toint .synb -.mets (ignwarn << form *) +.mets (tofloat << value ) +.mets (toint < value <> [ radix ]) .syne .desc -The -.code ignwarn -macro resembles -.codn ignerr . -It arranges for the evaluation of each -.meta form -in left-to-right order. If all the forms are evaluated, then the -value of the last one is returned. If no forms are present, then -.code nil -is returned. +These convenience functions convert +.meta value +to floating-point or integer, respectively. -If any -.meta form -throws an exception of type -.code warning -then this exception is intercepted by a handler established by -.codn ignwarn . -This handler reacts by throwing an exception of type -.codn continue . +If a floating-point value is passed into tofloat, or an integer value into +toint, then the value is simply returned. -The effect is that the warning is ignored, since the handler -doesn't issue any diagnostic, and passes control to the warning's -continue point. +If +.meta value +is a character, then it is treated as a string of length one +containing that character. -Note: all sites within \*(TX which throw a -.code warning -also provide a nearby catch for a -.code continue -exception, for resuming evaluation at the point where the warning -was issued. +If +.meta value +is a string, then it is converted by +.code tofloat +as if by the function +.metn flo-str , +, and by +.code toint +as if by the function +.codn int-str . -.coNP Operator @ handler-bind +If +.meta value +is an integer, then it is converted by +.code tofloat +as if by the function +.codn flo-int . + +If +.meta value +is a floating-point number, then it is converted by +.code toint +as if by the function +.codn int-flo . + +.coNP Variables @ fixnum-min and @ fixnum-max +.desc +These variables hold, respectively, the most negative value of the +.code fixnum +integer type, and its most positive value. Integer values +from +.code fixnum-min +to +.code fixnum-max +are all of type +.codn fixnum . +Integers outside of this range are +.code bignum +integers. + +.coNP Functions @ tofloatz and @ tointz .synb -.mets (handler-bind < function-form < symbol-list << body-form *) +.mets (tofloatz << value ) +.mets (tointz < value <> [ radix ]) .syne .desc -The -.code handler-bind -operator establishes a handler for one or more -exception types, and evaluates zero or more -.metn body-form -s -in a dynamic scope in which that handler is visible. - -When the -.code handler-bind -form terminates normally, the handler is removed. The value of the -last -.meta body-form -is returned, or else +These functions are closely related to, respectively, +.code tofloat +and +.codn toint . +They differ in that these functions return a floating-point +or integer zero, respectively, in some situations +in which those functions would return .code nil -if there are no forms. +or throw an error. -The -.meta function-form -argument is an expression which must evaluate to a function. The function -must be capable of accepting the exception arguments. All exceptions functions -require at least one argument, since the leftmost argument in an exception handler -call is the exception type symbol. +Whereas those functions reject a +.meta value +argument of +.codn nil , +for that same argument +.code tofloatz +function returns 0.0 and +.code tointz +returns 0. -The -.meta symbol-list -argument is a list of symbols, not evaluated. If it is empty, then the handler -isn't eligible for any exceptions. Otherwise it is eligible for any exception -whose exception type is a subtype of any of the symbols. +Likewise, in cases when +.code value +contains a string or character which cannot be +converted to a number, and +.code tofloat +and +.code toint +would return +.codn nil , +these functions return 0.0 and 0, respectively. -If the evaluation of any -.meta body-form -throws an exception which is not handled within that form, and the handler -is eligible for that exception, then the function is invoked. It receives -the exception's type symbol as the leftmost argument. If the exception has -arguments, they appear as additional arguments in the function call. -If the function returns normally, then the exception search continues. -The handler remains established until the exception is handled in such a way -that a dynamic control transfer abandons the -.code handler-bind -form. +In other situations, these functions behave +exactly like +.code tofloat +and +.codn toint . -Note: while a handler's function is executing, the handler is disabled. -If the function throws an exception for which the handler is eligible, -the handler will not receive that exception; it will be skipped by the -exception search as if it didn't exist. When the handler function terminates, -either via a normal return or a nonlocal control transfer, then the handler is -re-enabled. +.coNP Variables @, flo-min @ flo-max and @ flo-epsilon +.desc +These variables hold, respectively: the smallest positive floating-point +value; the largest positive floating-point value; and the difference +between 1.0 and the smallest representable value greater than 1.0. -.coNP Macros @ handle and @ handle* +.code flo-min +and +.code flo-max +define the floating-point range, which consists +of three regions: values from +.code "(- flo-max)" +to +.codn "(- flo-min)" ; +the value 0.0, and values from +.code flo-min +to +.codn flo-max . + +.coNP Variable @ flo-dig +.desc +This variable holds an integer representing the number of decimal digits +in a decimal floating-point number such that this number can be converted +to a \*(TX floating-point number, and back to decimal, without a change in any of +the digits. This holds regardless of the value of the number, provided that it +does not exceed the floating-point range. + +.coNP Variable @ flo-max-dig +.desc +This variable holds an integer representing the maximum number of +decimal digits required to capture the value of a floating-point number +such that the resulting decimal form will convert back to the same +floating-point number. See also the +.code *print-flo-precision* +variable. + +.coNP Variables @ %pi% and @ %e% +.desc +These variables hold an approximation of the mathematical constants \(*p and e. +To four digits of precision, \(*p is 3.142 and e is 2.718. The +.code %pi% +and +.code %e% +approximations are accurate to +.code flo-dig +decimal digits. + +.coNP Function @ digits .synb -.mets (handle < try-expression -.mets \ \ >> {( symbol <> ( arg *) << body-form *)}*) -.mets (handle* < try-expression -.mets \ \ >> {( symbol >> ( type-arg << arg *) << body-form *)}*) +.mets (digits < number <> [ radix ]) .syne .desc The -.code handle -macro is a syntactic sugar for the -.code handler-bind -operator. Its syntax is exactly like that of -.codn catch . -The difference between -.code handle -and -.code catch -is that the clauses in -.code handle -are invoked without unwinding. That is to say, -.code handle -does not establish an exit point for an exception. When control passes to -a clause, it is by means of an ordinary function call and not a dynamic -control transfer. No evaluation frames are yet unwound when this takes place. +.code digits +function returns a list of the digits of +.meta number +represented in the base given by +.metn radix . The -.code handle -macro establishes a handler, by -.code handler-bind -whose -.meta symbol-list -consists of every -.meta symbol -gathered from every clause. +.meta number +argument must be a non-negative integer, and +.meta radix +must be an integer greater than one. -The handler function established in the generated -.code handler-bind -is synthesized from of all of the clauses, together with dispatch logic which -which passes the exception and its arguments to the first -eligible clause. +If +.meta radix +is omitted, it defaults to 10. -The -.meta try-expression -is evaluated in the context of this handler. +The return value is a list of the digits in descending order of significance: +most significant to least significant. +The digits are integers. For instance, if +.meta radix +is 42, then the digits are integer values in the range 0 to 41. -The clause of the -.code handle -syntax can return normally, like a function, in which case the handler -is understood to have declined the exception, and exception processing -continues. To handle an exception, the clause of the -.code handle -macro must perform a dynamic control transfer, such returning from a block -via -.code return -or throwing an exception. +The returned list always contains at least one element, and +includes no leading zeros, except when +.meta number +is zero. In that case, a one-element list containing zero is returned. -The -.code handle* -macro is a variant of -.code handle -with the following difference: when -.code handle* -invokes a clause, it passes the exception symbol as the leftmost argument -.metn type-arg . -Then the exception arguments follow. In contrast, -only the exception arguments are passed to the clauses of -.codn handle . +.TP* Examples: -.coNP Macro @ with-resources +.verb + (digits 1234) -> (1 2 3 4) + (digits 1234567 1000) -> (1 234 567) + (digits 30 2) -> (1 1 1 1 0) + (digits 0) -> (0) +.brev + +.coNP Function @ digpow .synb -.mets (with-resources >> ({ sym >> [ init-form <> [ cleanup-form ])}*) -.mets \ \ << body-form *) +.mets (digpow < number <> [ radix ]) .syne .desc The -.code with-resources -macro provides a sequential binding construct similar to -.codn let* . -Every -.meta sym -is established as a variable which is visible to the -.metn init-form -s -of subsequent variables, to all subsequent -.metn cleanup-form -s -including that of the same variable, -and to the -.metn body-form -s. - -If no -.meta init-form -is supplied, then -.meta sym -is bound to the value -.codn nil . - -If an -.meta init-form -is supplied, but no -.metn cleanup-form , -then -.meta sym -is bound to the value of the -.metn init-form . +.code digpow +function decomposes the +.meta number +argument into a power series whose terms add up to +.metn number . -If a -.meta cleanup-form -is supplied in addition to -.metn init-form , -it specifies code to be executed upon the termination of the -entire -.code with-resources -construct. +The +.meta number +argument must be a non-negative integer, and +.meta radix +must be an integer greater than one. -When an instance of -.code with-resources -terminates, all of the -.metn cleanup-form -s -specified in its binding clauses are evaluated, in reverse (right-to-left) -order. The value of the last -.meta body-form -is returned, or else -.code nil -if no -.metn body-form -s -are present. +The returned power series consists of a list of nonnegative +integers. It is formed from the digits of +.meta number +in the given +.metn radix , +which serve as coefficients which multiply successive +powers of the +.metn radix , +starting at the zeroth power (one). -.TP* "Example:" +The terms are given in decreasing order of significance: +the term corresponding to the most significant digit of +.metn number , +multiplying the highest power of +.metn radix , +is listed first. -The following expression opens a text file and reads a line from it, -returning that line, while ensuring that the stream is closed -immediately: +The returned list always contains at least one element, and +includes no leading zeros, except when +.meta number +is zero. In that case, a one-element list containing zero is returned. .verb - (with-resources ((f (open-file "/etc/motd") (close-stream f))) - (whilet ((l (get-line f))) - (put-line l))) + (digpow 1234) -> (1000 200 30 4) + (digpow 1234567 1000) -> (1000000 234000 567) + (digpow 30 2) -> (16 8 4 2 0) + (digpow 0) -> (0) .brev - - -.coNP Special variable @ *unhandled-hook* -The -.code *unhandled-hook* -variable is initialized with -.code nil -by default. - -It may instead be assigned a function which is capable of taking -three arguments. - -When an exception occurs which has no handler, this function is called, -with the following arguments: the exception type symbol, the exception object, -and a third value which is either -.code nil -or else the form which was being evaluated when the exception was thrown. -The call occurs before any unwinding takes place. - -If the variable is -.codn nil , -or isn't a function, or the function returns after being called, -then unwinding takes place, after which some informational messages are printed -about the exception, and the process exits with a failed termination status. - -In the case when the variable contains a object other than -.code nil -which isn't a function, a diagnostic message is printed on the -.code *stderr* -stream prior to unwinding. - -Prior to the function being called, the -.code *unhandled-hook* -variable is reset to -.codn nil . - -Note: the functions -.code source-loc -or -.code source-loc-str -may be applied to the third argument of the -.code *unhandled-hook* -function to obtain more information about the form. - -.coNP Macro @ defex +.coNP Functions @ poly and @ rpoly .synb -.mets (defex <> { symbol }*) +.mets (poly < arg << coeffs ) +.mets (rpoly < arg << coeffs ) .syne .desc -The macro -.code defex -records hierarchical relationships among symbols, for the purposes -of the use of those symbols as exceptions. It is closely related to the -.code @(defex) -directive in the \*(TX pattern language, performing the same function. - -All symbols are considered to be exception subtypes, and every symbol -is implicitly its own exception subtype. This macro does not introduce -symbols as exception types; it only introduces subtype-supertype -relationships. - -If -.code defex -is invoked with no arguments, it has no effect. - -If arguments are present, they must be symbols. +The +.code poly +and +.code rpoly +functions evaluate a polynomial, for the given numeric argument value +.meta arg +and the coefficients given by +.metn coeffs , +a sequence of numbers. If -.code defex -is invoked with only one symbol as its argument, it has no effect. - -At least two -symbols must be specified for a useful effect to take place. If exactly two -symbols are specified, then, subject to error checks, -.code defex -makes the left symbol an -.I exception subtype -of the right symbol. - -This behavior generalizes to three or more arguments: if three or more symbols -are specified, then each symbol other than the last is registered as a subtype of -the symbol which follows. - -If a -.code defex -has three or more arguments, they are processed from left to right. -If errors are encountered during the processing, the correct registrations -already made for prior arguments remain in place. +.meta coeffs +is an empty sequence, it denotes the zero polynomial, whose value +is zero everywhere; the functions return zero in this case. -It is erroneous to register a duplicate relationship. If symbol -.code a -is already a direct or indirect subtype of -.code b -then -.code "(defex a b)" -and -.code "(defex a x b)" -are erroneous. +Otherwise, the +.code poly +function considers +.meta coeffs +to hold the coefficients in the conventional order, namely in order +of decreasing degree of polynomial term. The first element of +.meta coeffs +is the leading coefficient, and the constant term appears as the last element. -Every symbol is implicitly considered to be its own exception subtype, -therefore it is erroneous to explicitly register a symbol as its -own subtype. +The +.code rpoly +function takes the coefficients in opposite order: the first element of +.meta coeffs +gives the constant term coefficient, and the last element gives the +leading coefficient. -The foregoing rules eliminate the possibility of creating cycles in the -exception subtype inheritance graph. +Note: except in the case of +.code rpoly +operating on a list or list-like sequence of coefficients, +Horner's method of evaluation is +used: a single result accumulator is initialized with zero, and then for each +successive coefficient, in order of decreasing term degree, the accumulator is +multiplied by the argument, and the coefficient is added. When +.code rpoly +operates on a list or list-like sequence, it makes a single +pass through the coefficients in order, thus taking them in increasing +term degree. It maintains two accumulators: one for successive powers of +.meta arg +and one for the resulting value. For each coefficient, the power +accumulator is updated by a multiplication by +.meta arg +and then this value is multiplied by the coefficient, and +that value is then added to the result accumulator. -The symbol -.code nil -is implicitly a subtype of every exception type. Therefore, it is erroneous -to attempt to specify it as a supertype in a registration. -Using -.code nil -as a subtype in a registration is silently permitted, but has no effect. -No explicit registration is recorded between -.code nil -and its successor in the argument list. +.TP* Examples: -The symbol -.code t -is implicitly the supertype of every exception type. Therefore, it -is erroneous to attempt to register it as an exception subtype. -Using -.code t -as a supertype in a registration is also erroneous. +.verb + ;; 2 + ;; evaluate x + 2x + 3 for x = 10. + (poly 10 '(1 2 3)) -> 123 -Keyword symbols may be used as exception types. + ;; 2 + ;; evaluate 3x + 2x + 1 for x = 10. + (rpoly 10 '(1 2 3)) -> 321 +.brev -.coNP Function @ register-exception-subtypes +.coNP Function @ bignum-len .synb -.mets (register-exception-subtypes <> { symbol }*) +.mets (bignum-len << arg ) .syne .desc The -.code register-exception-subtypes -function constitutes the underlying implementation for the -.code defex -macro. - -The following equivalence applies: +.code bignum-len +function reports the machine-specific +.I "bignum order" +of the integer or character argument +.metn arg . -.verb - (defex a b ...) <--> (register-exception-subtypes 'a 'b ...) -.brev +If +.meta arg +is a character or +.code fixnum +integer, the function returns zero. -That is, the -.code defex -macro works as if by generating a call to the function, with -the arguments quoted. +Otherwise +.meta arg +is expected to be a +.code bignum +integer, and the function returns the number of "limbs" used for its +representation, a positive integer. -The semantics of the function is precisely that of the macro. +Note: the +.code bignum-len +function is intended to be of use in algorithms whose performance +benefits from ordering the operations on multiple integer operands +according to the magnitudes of those operands. The function provides an +estimate of magnitude which trades accuracy for efficiency. -.coNP Function @ exception-subtype-p -.synb -.mets (exception-subtype-p < left-symbol << right-symbol ) -.syne +.coNP Variables @, flo-near @, flo-down @ flo-up and @ flo-zero .desc -The -.code exception-subtype-p -function tests whether two symbols are in a relationship as exception types, -such that -.meta left-symbol -is a direct or indirect exception subtype of -.metn right-symbol . +These variables hold integer values suitable as arguments to the +.code flo-set-round-mode +function, which controls the rounding mode for the results of floating-point +operations. These variables are only defined on platforms which support +rounding control. -If that is the case, then -.code t -is returned, otherwise -.codn nil . +Their values have the following meanings: +.RS +.coIP flo-near +Round to nearest: the result of an operation is rounded to the nearest +representable value. +.coIP flo-down +Round down: the result of an operation is rounded to the nearest representable +value that lies in the direction of negative infinity. +.coIP flo-up +Round up: the result of an operation is rounded to the nearest representable +value that lies in the direction of positive infinity. +.coIP flo-up +Round to zero: the result of an operation is rounded to the nearest +representable value that lies in the direction of zero. +.RE +.IP -.coNP Function @ exception-subtype-map +.coNP Functions @ flo-get-round-mode and @ flo-set-round-mode .synb -.mets (exception-subtype-map) +.mets (flo-get-round-mode) +.mets (flo-set-round-mode << mode ) .syne .desc -The -.code exception-subtype-map -function returns a tree structure which captures information -about all registered exception types. - -The map appears as an association list which contains an entry -for every exception symbol, paired with that type's supertype path. -The first element in the supertype path is the exception's immediate -supertype. The next element is that type's supertype and so on. The -last element in every path is the grand supertype -.codn t . - -For instance, if only the types -.codn a , -.code b -and -.code c -existed in the system, and were linked according to this inheritance graph: +Sometimes floating-point operations produce a result which +requires more bits of precision than the floating point representation +can provide. A representable floating-point value must be substituted +for the true result and yielded by the operation. -.verb - t ----+--- b --- a - | - +--- c -.brev +On platforms which support rounding control, these functions are provided for +selecting the decision procedure by which the floating-point representation +is taken. -such that the supertype of -.code b -and -.code c -is -.codn t , +The +.code flo-get-round-mode +returns the current rounding mode. The rounding mode is represented by +an integer value which is either equal to one of the four variables +.codn flo-near , +.codn flo-down , +.code flo-up and -.code a -has -.code b -as supertype, then the function might return: - -.verb - ((a b t) (b t) (c t) (t)) -.brev +.codn flo-zero , +or else some other value specific to the host environment. Initially, +the value is that of +.codn flo-near . +Otherwise, the value returned is that which was stored by the most +recent successful call to +.codn flo-set-round-mode . -or any other equivalent permutation. +The +.code flo-set-round-mode +function changes the rounding mode. The argument to its +.meta mode +parameter may be the value of one of the above four variables, +or else some other value supported by the host environment's +.code fesetround +C library function. -The returned list may share substructure, so that the -.code "(t)" -sublist is shared among all four entries, and -.code "(b t)" -between the first two. +The +.code flo-set-round-mode +function returns +.code t +if it is successful, otherwise the return value is +.code nil +and the rounding mode is not changed. -If the program alters the tree structure returned by -.codn exception-map-p , -the consequences are unspecified; this structure may in fact -be the very object which actually represents the type hierarchy, -and not a derived representation. +If a value is is passed to +.code flo-set-round-mode +which is not the value of one of the above +four rounding mode variables, and the function succeeds anyway, then the +rounding behavior of floating-point operations depends on the host +environment's interpretation of that value. -.coNP Structures @, frame @ catch-frame and @ handle-frame -.synb -.mets (defstruct frame nil) -.mets (defstruct catch-frame frame types desc jump) -.mets (defstruct handle-frame frame types fun) +.SS* Bit Operations +In \*(TL, similarly to Common Lisp, bit operations on integers are based +on a concept that might be called "infinite two's-complement". +Under infinite two's complement, a positive number is regarded as having +a binary representation prefixed by an infinite stream of zero digits (for +example +.code 1 +is +.codn ...00001 ). +A negative number +in infinite two's complement is the bitwise negation of its positive counterpart, +plus one: it carries an infinite prefix of 1 digits. So for instance the number +.code -1 +is represented by +.codn ...11111111 : +an infinite sequence of +1 +bits. There +is no specific sign bit; any operation which produces such an infinite sequence +of 1 digits on the left gives rise to a negative number. For instance, consider the +operation of computing the bitwise complement of the number +.codn 1 . +Since the +number +.code 1 +is represented as +.codn ...0000001 , +its complement is +.codn ...11111110 . +Each one of the +.code 0 +digits in the infinite sequence is replaced by +.codn 1 , +And this leading sequence means that the number +is negative, in fact corresponding to the two's-complement representation of +the value +.codn -2 . +Hence, the infinite digit concept corresponds to an arithmetic +interpretation. + +In fact \*(TL's bignum integers do not use a two's complement +representation internally. Numbers are represented as an array which holds a +pure binary number. A separate field indicates the sign: negative, +or non-negative. That negative numbers appear as two's-complement under the +bit operations is merely a carefully maintained illusion (which makes bit +operations on negative numbers more expensive). + +The +.code logtrunc +function, as well as a feature of the +.code lognot +function, allow bit +manipulation code to be written which works with positive numbers only, even if +complements are required. The trade off is that the application has to manage a +limit on the number of bits. + +.coNP Functions @, logand @ logior and @ logxor +.synb +.mets (logand << integer *) +.mets (logior << integer *) +.mets (logxor < int1 << int2 ) .syne .desc -The structure types -.codn frame , -.code catch-frame -and -.code handle-frame -are used by the -.code get-frames -and -.code find-frame -functions to represent information about the currently established -exception catches (see the -.code catch -macro) and handlers -(see -.code handler-bind -and -.codn handler ). +These operations perform the familiar bitwise and, inclusive or, and exclusive +or operations, respectively. Positive values inputs are treated as +pure binary numbers. Negative inputs are treated as infinite-bit +two's-complement. + +For example +.code "(logand -2 7)" +produces +.codn 6 . +This is because +.code -2 +is +.code ...111110 +in infinite-bit two's-complement. And-ing this value with +.code 7 +(or +.codn ...000111 ) +produces +.codn 110 . The -.code frame -type serves as the common base for -.code catch-frame +.code logand and -.codn handle-frame . +.code logior +functions are variadic, and may be called with zero, one, +two, or more input values. If +.code logand +is called with no arguments, it produces +the value -1 (all bits 1). If +.code logior +is called with no arguments it produces +zero. In the one-argument case, the functions just return their argument value. -Modifying any of the slots of these structures has no effect on the -actual frame from which they are derived; the frame structures are only -representation which provides information about frames. They are not -the actual frames themselves. +In the two-argument case, one of the operands may be a character, if the other +operand is a fixnum integer. The character operand is taken to be an integer +corresponding to the character value's Unicode code point value. The resulting +value is regarded as a Unicode code point and converted to a character value +accordingly. -Both -.code catch-frame -and -.code handle-frame -have a -.code types -slot. This holds the list of exception type symbols which are matched -by the catch or handler. +When three or more arguments are specified, the operation's semantics is +that of a left-associative reduction through two-argument invocations, +so that the three-argument case +.code "(logand a b c)" +is equivalent to the expression +.codn "(logand (logand a b) c)" , +which features two two-argument cases.. +.coNP Function @ logtest +.synb +.mets (logtest < int1 << int2 ) +.syne +.desc The -.code desc -slot of a -.code catch-frame -holds a list of the descriptions produced by the -.code catch** -macro. If there are no descriptions, then this member is -.codn nil , -otherwise it is a list whose elements are in correspondence -with the list in the -.code types -slot. +.code logtest +function returns true if +.meta int1 +and +.meta int2 +have bits in +common. The following equivalence holds: + +.verb + (logtest a b) <--> (not (zerop (logand a b))) +.brev +.coNP Functions @ lognot and @ logtrunc +.synb +.mets (lognot < value <> [ bits ]) +.mets (logtrunc < value << bits ) +.syne +.desc The -.code jump -slot of a -.code catch-frame -is an opaque -.code cptr -("C pointer") -object which is related to the stack address of the catch -frame. If it is altered, the catch frame object becomes invalid -for the purposes of -.codn invoke-catch . +.code lognot +function performs a bitwise complement of +.metn value . +When the one-argument form of lognot is used, then if +.meta value +is nonnegative, +then the result is negative, and vice versa, according to the infinite-bit +two's complement representation. For instance +.code "(lognot -2)" +is +.codn 1 , +and +.code "(lognot 1)" +is +.codn -2 . + +The two-argument form of +.code lognot +produces a truncated complement. Conceptually, +a bitwise complement is first calculated, and then the resulting number is +truncated to the number of bits given by +.metn bits , +which must be a nonnegative integer. The following equivalence holds: + +.verb + (lognot a b) <--> (logtrunc (lognot a) b) +.brev The -.code fun -slot of a -.code handle-frame -is the registered handler function. Note that all the clauses of a -.code handler -macro are compiled to a single function, which is established via -.codn handler-bind , -so an instance of the -.code handler -macro corresponds to a single -.codn handle-frame . +.code logtrunc +function truncates the integer +.meta value +to the specified number +of bits. If +.meta value +is negative, then the two's-complement representation +is truncated. The return value of +.code logtrunc +is always a non-negative integer. -.coNP Function @ get-frames +.coNP Function @ sign-extend .synb -.mets (get-frames) +.mets (sign-extend < value << bits ) .syne .desc The -.code get-frames -function inquires the current dynamic environment in order to retrieve -information about established exception catch and handler frames. -The function returns a list, ordered from the inner-most nesting -level to the outer-most nesting, of structure objects derived from the -.code frame -structure type. The list contains two kinds of objects: structures -of type -.code catch-frame -and of type -.codn handle-frame . - -These objects are not the frames themselves, but only provide information -about frames. Modifying the slots in these structures has no effect on -the original frames. Also, these structures have their own lifetime and -can endure after the original frames have disappeared. This has implications -for the use of the -.code invoke-catch -function. +.code sign-extend +function first truncates the infinite-bit two's complement representation of +the integer +.meta value +to the specified number of bits, similarly to the +.code logtrunc +function. Then, this truncated value is regarded as a +.meta bits +wide two's complement integer. The value of this integer is +calculated and returned. -The -.code handle-frame -structures have a -.code fun -slot, which holds a function. It may be invoked directly. +.TP* Examples: -A -.code catch-frame -structure may be passed as an argument to the -.code invoke-catch -function. +.verb + (sign-extend 127 8) -> 127 + (sign-extend 128 8) -> -128 + (sign-extend 129 8) -> -127 + (sign-extend 255 8) -> -1 + (sign-extend 256 8) -> 0 + (sign-extend -1 8) -> -1 + (sign-extend -255 8) -> 0 +.brev -.coNP Functions @ find-frame and @ find-frames +.coNP Function @ ash .synb -.mets (find-frame >> [ exception-symbol <> [ frame-type ]]) -.mets (find-frames >> [ exception-symbol <> [ frame-type ]]) +.mets (ash < value << bits ) .syne .desc The -.code find-frame -function locates the first (innermost) instance of a specific kind of -exception frame (a catch frame or a handler frame) which is eligible -for processing an exception of a specific type. If such a frame -is found, it is returned. The returned frame object is of the same kind as the -objects which comprise the list returned by the function -.codn get-frames . -If such a frame is not found, -.code nil -is returned. +.code ash +function shifts +.meta value +by the specified number of +.meta bits +producing a +new value. If +.meta bits +is positive, then a left shift takes place. If +.meta bits +is negative, then a right shift takes place. If +.meta bit +is zero, then +.meta value +is returned unaltered. For positive numbers, a left shift by n bits is +equivalent to a multiplication by two to the power of n, or +.codn "(expt 2 n)" . +A right shift by n bits of a positive integer is equivalent to integer +division by +.codn "(expt 2 n)" , +with truncation toward zero. +For negative numbers, the bit shift is performed as if on the two's-complement +representation. Under the infinite two's-complement representation, +a right shift does not exhaust the infinite sequence of +.code 1 +digits which +extends to the left. Thus if +.code -4 +is shifted right it becomes +.code -2 +because +the bitwise representations of these values are +.code ...111100 +and +.codn ...11110 . +.coNP Function @ bit +.synb +.mets (bit < value << bit ) +.syne +.desc The -.meta exception-symbol -argument specifies a match by exception type: the candidate frame -must specify in its list of matches at least one type which is an exception -supertype of -.metn exception-symbol . -If this argument is omitted, it defaults to -.code nil -which finds any handler that matches at least one type. There is no way to -search for handlers which match an empty set of types; the -.code find-frame -function skips such frames. - +.code bit +function tests whether the integer or character +.meta value +has a 1 in bit position +.metn bit . The -.meta frame-type -argument specifies which frame type to find. Useful values for this -argument are the structure type names -.code catch-frame -and -.code handle-frame -or the actual structure type objects which these type names denote. -If any other value is specified, the function returns -.codn nil . -If the argument is omitted, it defaults to the type of the -.code catch-frame -structure. That is to say, by default, the function looks for catch -frames. +.meta bit +argument must be a non-negative integer. A value of zero of +.meta bit +indicates the least significant bit position of +.metn value . -Thus, if -.code find-frame -is called with no arguments at all it finds the innermost catch frame, -if any exists, or else returns +The +.code bit +function has a Boolean result, returning the symbol +.code t +if bit +.meta bit +of +.meta value +is set, otherwise .codn nil . -The -.code find-frames -function is similar to -.code find-frame -except that it returns all matching frames, ordered from the inner-most nesting -level to the outer-most nesting. If called with no arguments, it returns a -list of the catch frames. +If +.meta value +is negative, it is treated as if it had an infinite-bit two's +complement representation. For instance, if value is +.codn -2 , +then the bit +function returns +.code nil +for a +.meta bit +value of zero, and +.code t +for all other values, +since the infinite bit two's complement representation of +.code -2 +is +.codn ...11110 . -.coNP Function @ invoke-catch +.coNP Function @ mask .synb -.mets (invoke-catch < catch-frame < symbol << argument *) +.mets (mask << integer *) .syne .desc The -.code invoke-catch -function abandons the current evaluation context to perform -a non-local control transfer directly to the catch -described by the -.meta catch-frame -argument, which must be a structure of type -.code catch-frame -obtained using any of the functions -.codn get-frames , -.code find-frames -or -.codn find-frame . - -The control transfer is possible only if the catch -frame represented by -.meta catch-frame -structure is still established, and if the structure -hasn't been tampered with. - -If a given -.code catch-frame -structure is usable with -.codn invoke-catch , -then a copy of that structure made with -.code copy-struct -is also usable, denoting the same catch frame. +.code mask +function takes zero or more integer arguments, and produces an integer +value which corresponds a bitmask made up of the bit positions specified by the +integer values. -The -.meta symbol -argument should be an exception symbol. It is passed to the -exception frame, as if it had appeared as the first argument of the -.code throw -function. Similarly, the -.metn argument -s -are passed to the catch frame as if they were the trailing arguments -of a -.codn throw . -The difference between -.code invoke-catch -and -.code throw -is that -.code invoke-catch -targets a specific catch frame as its exit point, rather than searching for a -matching catch or handler frame. That specific frame receives the control. -The frame receives control even if it it is not otherwise eligible for -catching the exception type denoted by -.metn symbol . +If +.code mask +is called with no arguments, then the return value is zero. -.SS* Static Error Diagnosis +If +.code mask +is called with a single argument +.meta integer +then the return value is the same as +that of the expression +.codn "(ash 1 )" : +the value 1 shifted left by +.meta integer +bit positions. If +.meta integer +is zero, then the result is +.codn 1 ; +if +.meta integer +is +.codn 1 , +the +result is +.code 2 +and so forth. If +.meta value +is negative, then the result is zero. -This section describes a number of features related to the diagnosis -of errors during the static processing of program code prior to evaluation. -The material is of interest to developers of macros intended for broad -reuse. +If +.code mask +is called with two or more arguments, then the result is a bitwise or of +the masks individually computed for each of the values. -.NP* Error Exceptions +In other words, the following equivalences hold: -\*(TL uses exceptions of type -.code eval-error -to identify erroneous situations during both transformation of code -and its evaluation. These exceptions have one argument, which is a -character string. If not handled by program code, -.code eval-error -exceptions are specially recognized and treated by the built-in handling logic. -The message is incorporated into diagnostic output which includes -more information which is deduced. +.verb + (mask) <--> 0 + (mask a) <--> (ash 1 a) + (mask a b c ...) <--> (logior (mask a) (mask b) (mask c) ...) +.brev -.NP* Warning Exceptions +.coNP Function @ bitset +.synb +.mets (bitset << integer ) +.syne +.desc +The +.code bitset +function returns a list of the positions of bits which have a value of +1 in a positive +.meta integer +argument, or the positions of bits which have a value of zero in a negative +.meta integer +argument. The positions are ordered from least to greatest. The least +significant bit has position zero. If +.meta integer +is zero, the empty list +.code nil +is returned. -\*(TL uses exceptions of type -.code warning -to identify certain situations of interest. Ordinary non-deferrable -warnings have a structure identical to errors, except for the exception -symbol. \*(TX's built-in handling of warnings expects these exceptions -to be continuable. What this means is that a -.code catch -for the -.code continue -exception is expected to be visible. The handler for a warning exception -issues a diagnostic which incorporates the warning message. Then the -handler throws a -.code continue -exception to resume the computation which generated the warning. +A negative integer is treated as an infinite bit two's complement +representation. -The generation of a warning thus conforms to the following pattern: +The argument may be a character. + +If +.meta integer +.code x +is non-negative, the following equivalence holds: .verb - (catch - (throw 'warning "message") - (continue ())) + x <--> [apply mask (bitset x)] .brev -.NP* Deferrable Warnings - -\*(TX supports a form of diagnostic known as a -.IR "deferrable warning" . -A deferrable warning is distinguished in two ways. Firstly, it is -either of the type -.code defr-warning -or subtyped from that type. The -.code defr-warning -type itself is a direct subtype of -.codn warning . +That is to say, the value of +.code x +may be reconstituted by applying the bit positions returned by +.code bitset +as arguments to the +.code mask +function. -Secondly, a deferrable warning carries an additional tag argument after the -exception message. A deferrable exception is thrown according to -this pattern: +The value of a negative +.code x +may be reconstituted from its +.code bitset +as follows: .verb - (catch - (throw 'defr-warning "message" . tag) - (continue ())) + x <--> (pred (- [apply mask (bitset x)])) .brev -\*(TX's built-in exception handling logic reacts specially to the -presence of the tag material in the exception. First, the global -.I "tentative definition list" -is searched for the presence of the tag, using -.code equal -equality. If the tag is found, then the warning is discarded. -If the tag is not found, then the exception argument list is added -to the global -.IR "deferred warning list" . -In either case, -the -.code continue -exception is thrown to resume the computation which threw the warning, -as in the case of an ordinary non-deferrable warning. - -The purpose of this mechanism is to suppress warnings which -become superfluous when more of the program code is examined. -For instance, a warning about a call to an undefined function is -superfluous if a definition of that function is supplied later, -yet before that function call is executed. +also, more trivially, thus: -Deferred warnings accumulate in the deferred warning list -from which they can be removed. The list is purged at various -times such as when a top-level load completes, and the -deferred warnings are released, as if by a call to the -.code release-deferred-warnings -function. +.verb + x <--> (- [apply mask (bitset (- x))]) +.brev -.coNP Functions @ compile-error and @ compile-warning +.coNP Function @ width .synb -.mets (compile-error < context-obj < fmt-string << fmt-arg *) -.mets (compile-warning < context-obj < fmt-string << fmt-arg *) +.mets (width << integer *) .syne .desc -The functions -.code compile-error -and -.code compile-warning -provide a convenient and uniform way for code transforming -functions such as macro-expanders to generate diagnostics. -The -.code compile-error -function throws an exception of type -.codn eval-error . -The -.code compile-warning -function throws an exception of type -.code warning -and internally provides the expected -.code catch -for the -.code continue -exception needed to resume after the warning. - -The argument conventions are the same for both functions. +A two's complement representation of an integer consists of a sign bit and a +mantissa field. The -.meta context-obj -is typically a compound form to which the diagnostic -applies. +.code width +function computes the minimum number of bits required for the mantissa portion +of the two's complement representation of the +.meta integer +argument. -The functions produce a diagnostic message which -incorporates the location information and symbol -obtained from -.meta context-obj -and the -.codn format -style -arguments -.meta fmt-string -and its -.metn fmt-arg -s. +For a nonnegative argument, the width also corresponds to the number of bits +required for a natural binary representation of that value. -.coNP Function @ compile-defr-warning -.synb -.mets (compile-defr-warning < context-obj < tag -.mets \ \ < fmt-string << fmt-arg *) -.syne -.desc -The -.code compile-defr-warning -function throws an exception of type -.code defr-warning -and internally provides the expected -.code catch -for the -.code continue -exception needed to resume after the warning. +Two integer values have a width of zero, namely 0 and -1. This means that these +two values can be represented in a one-bit two's complement, consisting of only +a sign bit: the one-bit two's complement bitfield 1 denotes -1, and 0 denotes +0. -The function produces a diagnostic message which -incorporates the location information and symbol -obtained from -.meta context-obj -and the -.codn format -style -arguments -.meta fmt-string -and its -.metn fmt-arg -s. -This diagnostic message constitutes the first -argument of the exception. The -.meta tag -argument is taken as the second argument. +Similarly, two integer values have a width of 1: 1 and -2. The two-bit +two's complement bitfield 01 denotes 1, and 10 denotes -2. -.coNP Function @ purge-deferred-warning -.synb -.mets (purge-deferred-warning << tag ) -.syne -.desc -The -.code purge-deferred-warning -removes all warnings marked with -.meta tag -from the deferred list. It also removes all tags -matching -.meta tag -from the tentative definition list. -Tags are compared using the -.code equal -function. +The argument may be a character. -.coNP Function @ register-tentative-def +.coNP Function @ logcount .synb -.mets (register-tentative-def << tag ) +.mets (logcount << integer ) .syne .desc The -.code register-tentative-def -function adds -.meta tag -to the list of tentative definitions which are -used to suppress deferrable warnings. +.code logcount +function considers +.meta integer +to have a two's complement representation. If the integer is positive, +it returns the count of bits in that representation whose value is 1. +If +.meta integer +is negative, it returns the count of zero bits instead. If +.meta integer +is zero, the value returned is zero. -The idea is that a definition of some construct has been seen, -but not yet executed. Thus the construct is not defined, but -it can reasonably be expected that it will be defined; -hence, warnings about its nonexistence can be suppressed. +The argument may be a character. -For example, in the following code, when the expression -.code "(foo)" -is being expanded and transformed, the -.code foo -function does not exist: +.SS* User-Defined Arithmetic Types -.verb - (progn (defun foo ()) (foo)) -.brev +\*(TL makes it possible for the user application program to define structure +types which can participate in arithmetic operations as if they were numbers. +Under most arithmetic functions, a structure object may be used instead of a +number, if that structure object implements a specific method which is required +by that arithmetic function. -The function won't be defined until the -.code progn -is evaluated. Thus a warning is generated that -.code "(foo)" -refers to an undefined function. -However, this warning is discarded, because the -expander for -.code defun -registers a tentative definition tag for -.codn foo . +The following paragraphs give general remarks about the method conventions. +Not all arithmetic and bit manipulation functions have a corresponding +method, and a small number of functions do not follow these conventions. -When the definition of -.code foo -takes place, the -.code defun -operator will call -.code purge-deferred-warning -which will remove not only all accumulated warnings -related to the undefinedness of -.code foo -but also remove the tentative definition. +In the simplest case of arithmetic functions which are unary, the method +takes no argument other than the object itself. Most unary arithmetic functions +expect a structure argument to have a method which has the same name as that +function. For instance, if +.code x +is a structure, then +.code "(cos x)" +will invoke +.codn "x.(cos)" . +If +.code x +has no +.code cos +method, then an +.code error +exception is thrown. A few unary methods are not named after the corresponding function. +The unary case of the +.code - +function excepts an object to have a method named +.codn neg ; +thus, +.code "(- x)" +invokes +.codn "x.(neg)" . +Unary division requires a method called +.codn recip ; +thus, +.codn "(/ x)" , +invokes +.codn "x.(recip)" . -Note: this mechanism isn't perfect because it will -still suppresses the warning in situations like +When a structure object is used as an argument in a two-argument (binary) +arithmetic function, there are several cases to consider. If the left argument +to a binary function is an object, then that object is expected to support a +binary method. That method is called with two arguments: the object itself, of +course, and the right argument of the arithmetic operation. In this case, the +method is named after the function. For instance, if +.code x +is an object, then +.code "(+ x 3)" +invokes +.codn "x.(+ 3)" . +If the right argument, and only the right argument, of a binary operation is an +object, then the situation falls into two cases depending on whether the operation +is commutative. If the operation is commutative, then the same method is used +as in the case when the object is the left argument. The arguments are merely reversed. +Thus +.code "(+ 3 x)" +also invokes +.codn "x.(+ 3)" . +If the operation is not commutative, then the object must supply an alternative +method. For most functions, that method is named by a symbol whose name begins +with a +.code r- +prefix. For instance +.code "(mod x 5)" +invokes +.code "x.(mod 5)" +whereas +.code "(mod 5 x)" +invokes +.codn "x.(r-mod 5)" . +Note: the "r" may be remembered as indicating that the object is the +.B right +argument +of the binary operation or that the arguments are +.BR reversed . +Two functions do not follow the +.code r- +convention. These are +.code - +and +.codn / . +For these, the methods used for the object as a right argument, respectively, are +.code -- +and +.codn // . +Thus +.code "(/ 5 x)" +invokes +.code "x.(// 5)" +and +.code "(- 5 x)" +invokes +.codn "x.(-- 5)" . +Several binary functions do not support an object as the right argument. These are +.codn sign-extend , +.code ash +and +.codn bit . -.verb - (progn (if nil (defun foo ())) (foo)) -.brev +Variadic arithmetic functions, when given three or more arguments, are regarded +as performing a left-associative decimation of the arguments through a binary +function. Thus for instance +.code "(- 1 x 4)" +is understood as +.code "(- (- 1 x) 4)" +where +.code "x.(-- 1)" +is evaluated first. If that method yields an object +.code o +then +.code "o.(- 4)" +is invoked. +Certain variadic arithmetic functions, if invoked with one argument, just +return that argument: for instance, +.code + +and +.code * +are in this category. A special concession exists in these functions: if +their one and only argument is a structure, then that structure is returned +without any error checking, even if it implements no methods related +to arithmetic. -.coNP Function @ tentative-def-exists -.synb -.mets (tentative-def-exists << tag ) -.syne -.desc -The -.code tentative-def-exists -function checks whether -.meta tag -has been registered via -.code register-tentative-def -and not yet purged by -.codn purge-deferred-warning . +The following sections describe each of the methods that must be implemented +by an object for the associated arithmetic function to work with that object, +either at all, or in a specific argument position, as the case may be. +These methods are not provided by \*(TL; the application is required to provide +them. -.coNP Function @ defer-warning -.synb -.mets (defer-warning << args ) -.syne -.desc +.de bmc +. coNP Method @ \\$1 +. synb +. mets << obj .(\\$1 << arg ) +. syne +. desc The -.code defer-warning -function attempts to register a deferred warning. The -.meta args -argument corresponds to the arguments which are passed to the -.code throw -function in order to generate a warning exception, not including the exception -symbol. +. code \\$1 +method is invoked when a structure is used as an argument to the +. code \\$1 +function. -Args is expected to have at least two elements, the second of which -is a deferred warning tag. +If an object +. meta obj +is combined with an argument +. metn arg , +either as +. mono +. meti (\\$1 < obj << arg ) +. onom +or as +. mono +. meti (\\$1 < arg << obj ) +. onom +then, effectively, the method call +. mono +. meti << obj .(\\$1 << arg ) +. onom +takes place, and its return value is taken as the result +of the operation. +.. +.de bmcv +. coNP Method @ \\$1 +. synb +. mets << obj .(\\$1 << arg ) +. syne +. desc The -.code defer-warning -function returns -.codn nil . - -Note: this function is intended for use in exception handlers. The -following example shows a handler which intercepts warnings. It defers -deferrable warnings, and prints ordinary warnings: +. code \\$1 +method is invoked when a structure is used as an argument to the +. code \\$1 +function together with at least one other operand. -.verb - (handle - (some-form ..) ;; some code which might generate warnings - (defr-warning (msg tag) ;; catch deferrable and defer - (defer-warning (cons msg tag)) - (throw 'continue)) ;; warning processed: resume execution - (warning (msg) - (put-line `warning: @msg`) ;; print non-deferrable - (throw 'continue))) ;; warning processed: resume execution -.brev +If an object +. meta obj +is combined with an argument +. metn arg , +either as +. mono +. meti (\\$1 < obj << arg ) +. onom +or as +. mono +. meti (\\$1 < arg << obj ) +. onom +then, effectively, the method call +. mono +. meti << obj .(\\$1 << arg ) +. onom +takes place, and its return value is taken as the result +of the operation. +.. -.coNP Function @ release-deferred-warnings -.synb -.mets (release-deferred-warnings) -.syne -.desc +.de bmnl +. coNP Method @ \\$1 +. synb +. mets << obj .(\\$1 << arg ) +. syne +. desc The -.code release-deferred-warnings -removes all warnings from the deferred list. -Then, it issues each deferred warning as an ordinary warning. +. code \\$1 +method is invoked when the structure +. meta obj +is used as the left argument of the +. code \\$1 +function. -Note: there is normally no need for user programs to use this -function since deferred warnings are issued automatically. +If an object +. meta obj +is combined with an argument +. metn arg , +as +. mono +. meti (\\$1 < obj << arg ) +. onom +then, effectively, the method call +. mono +. meti << obj .(\\$1 << arg ) +. onom +takes place, and its return value is taken as the result +of the operation. +.. -.coNP Function @ dump-deferred-warnings -.synb -.mets (dump-deferred-warning << stream ) -.syne -.desc +.de bmnr +. coNP Method @ \\$1 +. synb +. mets << obj .(\\$1 << arg ) +. syne +. desc The -.code dump-deferred-warnings -empties the list of deferred warnings, and converts each one -into a diagnostic message sent to -sent to -.metn stream . -After the diagnostics are printed, the list of pending warnings -is cleared. +. code \\$1 +method is invoked when the structure +. meta obj +is used as the right argument of the +. code \\$2 +function. -Note: there is normally no need for user programs to use this -function since deferred warnings are issued automatically. +If an object +. meta obj +is combined with an argument +. metn arg , +as +. mono +. meti (\\$2 < arg << obj ) +. onom +then, effectively, the method call +. mono +. meti << obj .(\\$1 << arg ) +. onom +takes place, and its return value is taken as the result +of the operation. +.. -.SS* Delimited Continuations +.de umv +. coNP Method @ \\$1 +. synb +. mets << obj .(\\$1) +. syne +. desc +The +. code \\$1 +method is invoked when the structure +. meta obj +is used as the sole argument to the +. code \\$2 +function. -\*(TL supports delimited continuations, which are integrated with the -.code block -feature. Any named or anonymous block, including the implicit blocks -created around function bodies, can be used as the delimiting -.I prompt -for the capture of a continuation. +If an object +. meta obj +is passed to the function as +. mono +. meti (\\$2 << obj ) +. onom +then, effectively, the method call +. mono +. meti << obj .(\\$1) +. onom +takes place, and its return value is taken as the result +of the operation. +.. -A delimited continuation is section of a possible future of the -computation, up to a delimiting prompt, -.I reified -as a first class function. +.de bma +. coNP Method @ \\$1 +. synb +. mets << obj .(\\$1 << arg ) +. syne +. desc +The +. code \\$1 +method is invoked when the +. code \\$1 +function is invoked with two operands, and the structure +. meta obj +is the left operand. +The method is also invoked when the +. code \\$2 +function is invoked with two operands, and +.meta obj +is the right operand. -.TP* Example: +If an object +. meta obj +is combined with an argument +. metn arg , +either as +. mono +. meti (\\$1 < obj << arg ) +. onom +or as +. mono +. meti (\\$2 < arg << obj ) +. onom +then, effectively, the method call +. mono +. meti << obj .(\\$1 << arg ) +. onom +takes place, and its return value is taken as the result +of the operation. +.. -.verb - (defun receive (cont) - (format t "cont returned ~a\en" (call cont 3))) +.de um +. coNP Method @ \\$1 +. synb +. mets << obj .(\\$1) +. syne +. desc +The +. code \\$1 +method is invoked when a structure is used as the argument to the +. code \\$1 +function. - (defun function () - (sys:capture-cont 'abcd (fun receive))) +If an object +. meta obj +is passed to the function as +. mono +. meti (\\$1 << obj ) +. onom +then, effectively, the method call +. mono +. meti << obj .(\\$1) +. onom +takes place, and its return value is taken as the result +of the operation. +.. - (block abcd - (format t "function returned ~a\en" (function)) - 4) +.de tmnl +. coNP Method @ \\$1 +. synb +. mets << obj .(\\$1 < arg1 << arg2 ) +. syne +. desc +The +. code \\$1 +method is invoked when the structure +. meta obj +is used as the left argument of the +. code \\$1 +function. - Output: +If an object +. meta obj +is combined with arguments +. meta arg1 +and +. metn arg2 , +as +. mono +. meti (\\$1 < obj < arg1 << arg2 ) +. onom +then, effectively, the method call +. mono +. meti << obj .(\\$1 < arg1 << arg2 ) +. onom +takes place, and its return value is taken as the result +of the operation. +.. - function returned 3 - cont returned 4 - function returned t -.brev +.bmcv + +.bmnl - +.bmnr -- - +.umv neg - +.bmcv * +.bmnl / +.bmnr // / +.umv recip / +.um abs +.um signum +.bmnl trunc +.bmnr r-trunc trunc +.umv trunc1 trunc +.bmnl mod +.bmnr r-mod mod +.bmnl expt +.bmnr r-expt expt +.tmnl exptmod -.PP +Note: the +.code exptmod +function doesn't support structure objects in the second and +third argument positions. The +.meta exponent +and +.meta base +arguments must be integers. -Evaluation begins with the -.code block -form. This form calls -.code function -which uses -.code sys:capture-cont -to capture a continuation up to the -.code abcd -prompt. The continuation is passed to the -.code receive -function as an argument. +.um isqrt +.um square +.bma > < +.bma < > +.bma >= <= +.bma <= >= +.bmc = +.um zerop +.um plusp +.um minusp +.um evenp +.um oddp +.bmnl floor +.bmnr r-floor floor +.umv floor1 floor +.bmnl ceil +.bmnr r-ceil ceil +.umv ceil1 ceil +.bmnl round +.bmnr r-round round +.umv round1 round +.um sin +.um cos +.um tan +.um asin +.um acos +.um atan +.bmnl atan2 +.bmnr r-atan2 atan2 +.um log +.um log2 +.um log10 +.um exp +.um sqrt +.bmcv logand +.bmcv logior +.bmnl lognot +.bmnr r-lognot lognot +.umv lognot1 lognot +.bmnl logtrunc +.bmnr r-logtrunc logtrunc +.bmnl sign-extend -This captured object represents the continuation of computation -up to that prompt. It appears as a one-argument function which, when called, -resumes the captured computation. Its argument emerges out of the -.code sys:capture-cont -call as a return value. When the computation eventually returns all -the way to the delimiting prompt, the return value of that prompt -will then appear as the return value of the continuation function. +Note: the +.code sign-extend +function doesn't support a structure as the right argument, +.metn bits , +which must be an integer. -In this example, the function -.code receive -immediately invokes the continuation function which it receives, passing -it the argument value -.codn 3 . -And so, -evaluation now continues in the resumed future represented by the -continuation. Inside the continuation, -.code sys:capture-cont -appears to return, yielding the value -.codn 3 . -This bubbles up through -.code function -up to the -.code "block abcd" -where a message is printed: -.strn "function returned 3" . +.bmnl ash -The -.code block -terminates, yielding the value 4. Thereby, the continuation ends, since -it is delimited up to that block. Control now returns to the -.code receive -function which invoked the continuation, where the function call form -.code "(call cont)" -terminates, yielding the value -.code 4 -that was returned by the continuation's delimiting -.code block -form. The message -.str "cont returned 4" -is printed. The -.code receive -function returns normally, returning the value -.code t -which emerged from the -.code format -call. Control is now back in -.code function -where the -.code sys:capture-cont -form terminates and returns the -.codn t . -This bubbles up to -.code block -which prints -.strn "function returned t" . +Note: the +.code ash +function doesn't support a structure as the right argument, +.metn bits , +which must be an integer. -In summary, a continuation represents, as a function, the subsequent -computation that is to take place starting at some point, up to some -recently established, dynamically enclosing delimiting prompt. When -the continuation is captured, that future doesn't have to take place; -an alternative future can carry out in which that continuation is -available as a function. That alternative future can invoke the continuation at -will. Invocations (resumptions) of the continuation appear as additional -returns from the capture operator. A resumption of a continuation terminates -when the delimiting prompt terminates, and the continuation yields the -value which emerges from the prompt. +.bmnl bit -Delimited continuations are implemented by capturing a segment of the -evaluation stack between the prompt and the capture point. When -a continuation is resumed, this saved copy of a stack segment is inserted on -top of the current stack and the procedure context is resumed such -that evaluation appears to emerge from the capture operator. -As the continuation runs to completion, it simply pops these inserted -stack frames naturally. Eventually it pops out of the delimiting prompt, -at which point control ends up at the point which invoked the continuation -function. +Note: the +.code bit +function doesn't support a structure as the right argument, +.metn bit , +which must be an integer. -The low-level operator for capturing a continuation is -.codn sys:capture-cont . -More expressive and convenient programming with continuations is -provided by the macros -.codn obtain , -.codn obtain-block , -.code yield-from -and -.codn yield , -which create an abstraction which models the continuation as a suspended -procedure supporting two-way communication of data. -A -.code suspend -operator is provided, which is more general. It is identical to the -.code shift -operator described in various computer science literature about -delimited continuations, except that it refers to a specific delimiting -prompt by name. +.um width +.um logcount +.um bitset -Continuations raise the issue of what to do about unwinding. -The language Scheme provides the much criticized -.code dynamic-wind -operator which can execute initialization and clean-up code as -a continuation is entered and abandoned. \*(TX takes a simpler, -albeit risky approach. It provides a non-unwinding escape operator -.code sys:abscond-from -for use with continuations. Code which has captured a continuation -can use this operator to escape from the delimiting block without -triggering any unwinding among the frames between the capture point and the -delimiter. When the continuation is restarted, it will then do so -with all of the resources associated with it frames intact. -When the continuation executes normal returns within its context, -the unwinding takes place then. Thus tidy, "thread-like" use -of continuations is possible with a small measure of coding discipline. -Unfortunately, the absconding operator is dangerous: its use -breaks the language guarantee that clean-up associated with a form is done no -matter how a form terminates. +.SS* Exception Handling -.NP* Comparison with Lexical Closures +An +.I exception +in \*(TX is a special event in the execution of the program which +results in transfer of control. An exception is identified by a symbol, +known as the +.IR "exception type" , +and it carries zero or more arguments, called the +.IR "exception arguments" . -Delimited continuations resemble lexical closures in some ways. Both -constructs provide a way to return to some context whose evaluation -has already been abandoned, and to access some aspects of that context. -However, lexical closures are statically scoped. Closures capture the lexically -apparent scope at a given point, and produce a function whose body has access -to that scope, as well as to some arbitrary arguments. Thus, a lexical scope -is reified as a first-class function. By contrast, a delimited continuation -is dynamic. It captures an an entire segment of a program activation chain, -up to the delimiting prompt. This segment includes scopes which are not -lexically visible at the capture point: the scopes of parent functions. -Moreover, the segment includes not only scopes, but also other aspects of -the evaluation context, such as the possibility of returning to callers, -and the (captured portion of) the original dynamic environment, such as -exception handlers. That is to say, a lexical closure's body cannot return to -the surrounding code or see any of its original dynamic environment; it can -only inspect the environment, and then return to its own caller. Whereas a -restarted delimited continuation can continue evaluation of the surrounding -code, return to surrounding forms and parent functions, and access the dynamic -environment. The continuation function returns to its caller when that entire -restarted context terminates, whereas a closure returns to its caller as soon -as the closure body terminates. +When an exception is initiated, it is said to be +.IR thrown . +When an exception is thrown, \*(TX enters into exception processing +mode. Exception processing mode terminates in one of several ways: +.IP - +A +.I catch +is found which matches the exception, and control is transferred +to the catch. Catches are defined by the +.code catch +macro. +.IP - +A handler accepts the exception by performing a non-local transfer. +Handlers are defined by the +.code handler-bind +operator or +.code handle +macro. +.IP - +If no catch or accepting handler is found, control is transferred +to the function stored in the +.code *unhandled-hook* +variable. If that function returns, then unwinding is performed +after which the process terminates (unless the unwinding actions +intercept the control to prevent that). +.IP - +If no catch or accepting handler is found and +.code *unhandled-hook* +is +.codn nil , +then a built-in strategy for handling the exception is invoked, +consisting of unwinding, and then printing some informational messages and +terminating. +.PP -.NP* Differences in Compiled vs. Interpreted Behavior +.NP* Catches and Handlers -Delimited continuations in \*(TX expose a behavioral difference between -compiled and interpreted code which mutates the values of lexical variables. +There are two ways by which exceptions are handled: catches and handlers. +Catches and handlers are similar, but different. +A catch is an exit point associated with an active scope. When an exception is +handled by a catch, the form which threw the exception is abandoned, and unwinding +takes place to the catch site, which receives the exception type and arguments. +A handler is also associated with an active scope. However, it is a function, +and not a dynamic exit point. When an exception is passed to handler, +unwinding does not take place; rather, the function is called. The function then +either completes the exception handling by performing a non-local transfer, +or else declines the exception by performing an ordinary return. -When a continuation is captured in compiled code, it captures not only the -bindings of lexical variables, but also potentially their current values -at the time of capture. What this means is that whenever the continuation -is resumed, those variables will appear to have the captured values, -regardless of any mutations that have taken place since. In other words, -the captured future includes those specific values. This is because in -compiled code, variables are allocated on the stack, which is copied as part of -creating a continuation. Those variables are effectively newly instantiated in -each resumption of the continuation, when the captured stack segment -is reinstated into the stack, and take on those original values. +Catches and handlers are identified by exception type symbols. A catch or +handler is eligible to process an exception if it handles a type which is +a supertype of the exception which is being processed. Handles and catches +are located in a combined search which proceeds from the innermost nesting +to the outermost. When an eligible handle is encountered, it is called. If +it returns, the search continues. When an eligible catch is encountered, +the search stops and a control transfer takes place to the catch site. -In contrast, interpretation of code only maintains an -environment pointer on the stack; the lexical environment is a dynamically -allocated object whose contents aren't included in the continuation's -stack segment capture. If the captured variables are modified after the -capture, the continuation will see the updated values: all resumptions of the -continuation share the same instance of the captured environment among -themselves, and with the original context where the capture took place. +.NP* Handlers and Sandboxing -An additional complication is that when compiled code captures lexical -closures, captured variables are moved into dynamic storage and then -they become shared: the semantics of the mutation of those variables -is then similar to the situation in interpreted code. Therefore, the -above described non-sharing capture behavior of compiled code is not required -to hold. +Because handlers execute in the dynamic context of the exception origin, +without any unwinding having taken place, they expose a potential route +of sandbox escape via the package system, unless special steps are taken. +The threat is that code at the handler site could take advantage of +the current value of the +.code *package* +and +.code *package-alist* +variables established at the exception throw site to gain inappropriate access +to symbols. -In continuation-based code which relies on mutation of lexical variables -created with -.code let -or -.codn let* , -the macros -.code hlet +For this reason, when a handler is established, the current values of +.code *package* and -.code hlet* -can be used instead. These macros create variable bindings whose storage is -always outside of the stack, and therefore the variables will exhibit -consistent +.code *package-alist* +are recorded into the handler frame. +When that handler is later invoked, it executes in a dynamic environment +in which those variables are bound to the previously noted values. -If the affected variables are other kinds of bindings such as -function parameters or variables created with specialized binding -constructs such as -.codn with-stream , -additional coding changes may be required to get interpreted code -working under compilation. +The catch mechanism doesn't do any such thing because the unwinding +which is performed prior to the invocation of a catch implicitly +restores the values of +.B all +special variables to the values they had at the time the frame was +established. -.coNP Function @ sys:capture-cont -.synb -.mets (sys:capture-cont < name < receive-fun <> [ context-form ]) -.syne -.desc -The -.code sys:capture-cont -function captures a continuation, and also serves as the resume point -for the resulting continuation. Which of these two situations is the -case (capture or resumption) is distinguished by the use of the -.meta receive-fun -argument, which must be a function capable of being called with one -argument. +.NP* Exception Type Hierarchy -A block named -.meta name -must be visible; the continuation is delimited by the closest -enclosing block of this name. +Exception type symbols are arranged +in an inheritance hierarchy, at whose top the symbol +.code t +is is the supertype of every exception type, and the +.code nil +symbol is at the bottom, the subtype of every exception type. -The optional -.meta context-form -argument should be a compound form. If -.code sys:capture-cont -reports an error, it reports it against this form, -and uses the form's operator symbol as the name of the function which -encountered the error. If the argument is omitted, -.code sys:capture-cont -uses its own name. +Keyword symbols may be used as exception types. +Every symbol is its own supertype and subtype. Thus whenever X is known to be a +subtype of Y, it is possible that X is exactly Y. The -.code sys:capture-cont -function captures a continuation, represented as a function. -It immediately calls -.metn receive-fun , -passing it it the continuation function as an argument. -If -.meta receive-fun -returns normally, then -.code sys:capture-cont -returns whatever value -.meta receive-fun -returns. +.code defex +macro registers exception supertype/subtype relationships among symbols. -Resuming a continuation is done by invoking the continuation function. -When this happens, the entire continuation context is restored by re-creating -its captured evaluation frames on top of the current stack. Inside the -continuation, the -.code sys:capture-cont -function call which captured the continuation now appears to return, -and yields a value. That value is precisely the value which was just -passed to the continuation function moments ago. +The following tree diagram shows the relationships among \*(TL's built-in +exception symbols. Not shown is the exception symbol +.codn nil , +subtype of every exception type: -The resumed continuation can terminate in one of three ways. Firstly, it can -simply keep executing until it discards all of its evaluation frames below the -delimiting block, and then allows that block to terminate naturally by -evaluating the last form contained in the block. Secondly, can use -.code return-from -against its delimiting block to explicitly abandon all evaluations in between -and terminate that block. Or it may perform -a non-local control transfer past the delimited block somewhere into the -evaluation frames of the caller. In the first two cases, the termination -of the block turns into an ordinary return from the continuation function, and -the result value of the terminated block becomes the return value of that -function call. In the last case, the call of the continuation function is -abandoned and unwinding continues through the caller. +.verb + t ----+--- warning + | + +--- restart --- continue + | + +--- error ---+--- type-error + | + +--- internal-error + | + +--- panic + | + +--- numeric-error + | + +--- range-error + | + +--- query-error + | + +--- file-error -------+--- path-not-found + | | + | +--- path-exists + | | + | +--- path-permission + | + +--- process-error + | + +--- socket-error + | + +--- system-error + | + +--- alloc-error + | + +--- timeout-error + | + +--- assert + | + +--- syntax-error + | + +--- eval-error +.brev -If the symbol -.code sys:cont-poison -is passed to the continuation function, the continuation will be -resumed in a different manner: its context will be restored as in the -ordinary resume case, whereupon it will be immediately abandoned by -a nonlocal exit, causing unwinding to take place across all of the -continuation's evaluation frames. The function then returns -.codn nil . +Program designers are encouraged to derive new error exceptions from the +.code error +type. The +.code restart +type is intended to be the root of a hierarchy of exception +types used for denoting restart points: designers are encouraged +to derive restarts from this type. -If the symbol -.code sys:cont-free -is passed to the continuation function, the continuation isn't -be resumed at all; rather, the buffer which holds the saved context -of the continuation is released. Thereafter, an attempt to resume -the continuation results in an error exception being thrown. -After releasing the buffer, the function returns -.codn nil . +.NP* Dialect Notes -.TP* Notes: +Exception handling in \*(TL provides capabilities similar to the condition +system in ANSI Common Lisp. The implementation and terminology differ. -The continuation function may be used any time after it is produced, and may be -called more than once, regardless of whether the originally captured dynamic -context is still executing. The continuation object may be communicated into -the resumed continuation, which can then use it to call itself, resulting -in multiple nested resumptions of the same continuation. A delimited -continuation is effectively a first class function. +Most obviously, ANSI CL uses the "condition" term, whereas \*(TL uses "exception". -The underlying continuation object produced by -.code sys:capture-cont -stores a copy of the captured dynamic context. Whenever the continuation -function is invoked, a copy of the captured is reinstated as if it were a new -context. Thus each apparent return from the -.code sys:capture-cont -inside a resumed continuation is not actually made in the original context, but -in a copy of that context. That context can be resumed multiple times -sequentially or recursively. +In ANSI CL, a condition is "raised", whereas a \*(TL exception is "thrown". -Just like lexical closures, continuations do not copy lexical environments; -they capture lexical environments by reference. If a continuation modifies -the values of captured lexical variables, those modifications are visible to -other resumptions of the same continuation, to other continuations which -capture the same environment, to lexical closures which capture the same -environment and to the original context which created that environment, if it -is still active. +In ANSI CL, when a condition is raised, a condition object is created. Condition +object are similar to class objects, but are not required to be in the Common Lisp +Object System. They are related by inheritance and can have properties. \*(TL +exceptions are unencapsulated: they consist of a symbol, plus zero or more +arguments. The symbols are related by inheritance. -Unlike lexical closures, continuations do capture the local bindings -of special variables. That is to say, if -.code *var* -is a special variable, then a lexical closure created inside a -.code "(let ((*var* 42)) ...)" -form will not capture the local re-binding of -.code *var* -which holds 42. When the closure is invoked and accesses -.codn *var* , -it accesses whatever value of -.code *var* -is dynamically current, as dictated by the environment which calls the -closure, rather than the capturing environment. +When a condition is raised in ANSI CL, the dynamic scope is searched for a +handler, which is an ordinary function which receives the condition. No +unwinding or non-local transfer takes place. The handler can return, in which +case the search continues. Matching the condition to the handler is by +inheritance. Handler functions are bound to exception type names. +If a handler chooses to actually handle a condition (thereby terminating +the search) it must itself perform some kind of dynamic control transfer, +rather than return normally. ANSI CL provides a dynamic control mechanism +known as restarts which is usually used for this purpose. A condition handler +may invoke a particular restart handler. Restart handlers are similar to +exception handlers: they are functions associated with symbols in the +dynamic environment. -With continuations, the behavior is different. If a continuation -is captured inside a -.code "(let ((*var* 42)) ...)" -form then it does capture the local binding. This is regardless whether -the delimited prompt of the capture is enclosed in this form, or -outside of the form. -The special variable has a binding in a dynamic environment. There is always a -reference to a current dynamic environment associated with every evaluation -context, and a continuation captures that reference. Because it is a -reference, it means that the binding is shared. That is to say, all -invocations of all continuations which capture the same dynamic environment in -which that -.code "(let ((*var* 42)) ...)" -binding was made share the same binding; if -.code *var* -is modified by assignment, the modification is visible to all those views. +In \*(TL exceptions are a unification of conditions and restarts. From an ANSI CL +perspective, \*(TL exceptions are a lot like CL restarts, except that the +symbols are arranged in an inheritance hierarchy. \*(TL exceptions are used +both as the equivalent of ANSI CL conditions and as restarts. -Inside a resumed continuation, a form which binds a special variable such as -.code "(let ((*var* 42)) ...)" -may terminate. As expected, this causes the binding to be removed, -revealing either another local binding of -.code *var* -or the global binding. However, this unbinding only affects only that -that executing continuation; it has no effect inside other instances of the -same continuation or other continuations which capture the same variable. -Unbinding isn't a mutation of the dynamic environment, but may be understood -as merely the restoration of an earlier dynamic environment reference. +In \*(TL the terminology "catch" and "handle" is used in a very specific way. +To handle an exception means to receive it without unwinding, with the possibility +of declining to handle it, so that the search continues for another handler. +To catch an exception means to match an exception to a catch handler, terminate +the search, unwind and pass control to the handler. -.TP* "Example:" +\*(TL provides an operator called +.code handler-bind +for specifying handlers. It has a different syntax from ANSI CL's +.codn handler-bind . +\*(TL provides a macro called +.code handle +which simplifies the use of +.codn handler-bind . +This macro superficially resembles ANSI CL's +.codn handler-case , +but is semantically different. The most notable difference is that the bodies +of handlers established by +.code handler-bind +execute without any unwinding taking place and may return normally, thereby +declining to take the exception. In other words, +.code handle +has the same semantics as +.codn handler-bind , +providing only convenient syntax. -The following example shows an implementation of the -.code suspend -operator. +\*(TL provides a macro called +.code catch +which has the same syntax as +.code handle +but specifies a catch point for exceptions. If, during an exception search, a +.code catch +clause matches an exception, a dynamic control transfer takes place +from the throw site to the catch site. Then the clause body is executed. +The +.code catch +macro resembles ANSI CL's +.code restart-case +or possibly +.codn handler-case , +depending on point of view. + +\*(TL provides unified introspection over handler and catch frames. +A program can programmatically discover what handler and catches are +available in a given dynamic scope. ANSI CL provides introspection +over restarts only; the standard doesn't specify any mechanism for +inquiring what condition handlers are bound at a given point in +the execution. + +.TP* Example: + +The following two examples express a similar approach implemented +using ANSI Common Lisp conditions and restarts, and then using \*(TL +exceptions. .verb - (defmacro suspend (:form form name var . body) - ^(sys:capture-cont ',name (lambda (,var) - (sys:abscond-from ,name ,*body)) - ',form)) -.brev + ;; Common Lisp + (define-condition foo-error (error) + ((arg :initarg :arg :reader foo-error-arg))) -.coNP Operator @ sys:abscond-from -.synb -.mets (sys:abscond-from < name <> [ value ]) -.syne -.desc -The -.code sys:abscond-from -operator closely resembles -.code return-from -and performs the same function: it causes an enclosing block -.meta name -to terminate with -.meta value -which defaults to -.codn nil . + (defun raise-foo-error (arg) + (restart-case + (let ((c (make-condition 'foo-error :arg arg))) + (error c)) + (recover (recover-arg) + (format t "recover, arg: ~s~%" recover-arg)))) -However, unlike -.codn return-from , -.code sys:abscond-from -does not perform any unwinding. + (handler-bind ((foo-error + (lambda (cond) + (format t "handling foo-error, arg: ~s~%" + (foo-error-arg cond)) + (invoke-restart 'recover 100)))) + (raise-foo-error 200)) +.brev -This operator should never be used for any purpose other than -implementing primitives for the use of delimited continuations. -It is used by the -.code yield-from -and -.code yield -operators to escape out of a block in which a continuation has -been captured. Neglecting to unwind is valid due to the expectation -that control will return into a restarted copy of that context. +The output of the above is: -.coNP Function @ sys:abscond* -.synb -.mets (sys:abscond* < name <> [ value ]) -.syne -.desc -The -.code sys:abscond* -function is similar to the the -.code sys:abscond-from -operator, except that -.code name -is an ordinary function parameter, and so when -.code return* -is used, an argument expression must be specified which evaluates -to a symbol. Thus -.code sys:abscond* -allows the target block of a return to be dynamically computed. +.verb + handling foo-error, arg: 200 + recover, arg: 100 +.brev -The following equivalence holds between the operator and function: +The following is possible \*(TL equivalent for the above Common Lisp example. +It produces identical output. .verb - (sys:abscond-from a b) <--> (sys:abscond* 'a b) + (defex foo-error error) + + (defex recover restart) ;; recommended practice + + (defun raise-foo-error (arg) + (catch + (throw 'foo-error arg) + (recover (recover-arg) + (format t "recover, arg: ~s\en" recover-arg)))) + + (handle + (raise-foo-error 200) + (foo-error (arg) + (format t "handling foo-error, arg: ~s\en" arg) + (throw 'recover 100))) .brev -Expressions used as -.meta name -arguments to -.code abscond* -which do not simply quote a symbol have no equivalent in -.codn abscond-from . +To summarize the differences: exceptions serve as both +conditions and restarts in \*(TX. The same +.code throw +function is used to initiate exception handling for +.code foo-error +and then to transfer control out of the handler +to the recovery code. The handler accepts one exception +by raising another. -.coNP Macros @ obtain and @ yield-from +When an exception symbol is used for restarting, it is +a recommended practice to insert it into the inheritance +hierarchy rooted at the +.code restart +symbol, either by inheriting directly from +.code restart +or from an exception subtype of that symbol. + +.coNP Functions @, throw @ throwf and @ error .synb -.mets (obtain << forms *) -.mets (yield-from < name <> [ form ]) +.mets (throw < symbol << arg *) +.mets (throwf < symbol < format-string << format-arg *) +.mets (error < format-string << format-arg *) .syne .desc -The -.code obtain +These functions generate an exception. The +.code throw and -.code yield-from -macros closely inter-operate. +.code throwf +functions generate +an exception identified by +.metn symbol , +whereas +.code error +throws an exception of +type +.codn error . +The call +.code "(error ...)" +can be regarded as a shorthand for +.codn "(throwf 'error ...)" . The -.code obtain -macro treats zero or more -.metn form -s -as a suspendable execution context called the -.IR "obtain block" . -It is expected that -.metn form -s -establish a block named -.meta name -and return its result value to -.codn obtain . - -Without evaluating any of the forms in the obtain block, -.code obtain -returns a function, which takes one optional argument. -This argument, called the -.IR "resume value" , -defaults to -.code nil -if it is omitted. - -The function represents the suspended execution context. +.code throw +function takes zero or more additional arguments. These arguments +become the arguments of a +.code catch +handler which takes the exception. The +handler will have to be capable of accepting that number of arguments. -The context is resumed whenever the function is called, and executes -until the next -.code yield-from -statement which references the block named -.metn name . -The function's reply argument is noted. - -If the -.code yield-from -specifies a -.meta form -argument, then the execution context suspends, and the resume function -terminates and returns the value of that form. When the function is called -again to resume the context, the -.code yield-from -returns the previously noted resume value (and the new resume -value just passed is noted in its place). - -If the -.code yield-from -specifies no -.meta form -argument, then it briefly suspends the execution context only -to retrieve the resume value, without producing an item. Since -no item is produced, the resume function does not return. -The execution context implicitly resumes. - -When execution reaches the last form in the obtain block, the -resume value is discarded. The execution context terminates, and -the most recent call to the resume function returns the value of -that last form. - -.TP* Notes: +The +.code throwf +and +.code error +functions generate an exception which has a single +argument: a character string created by a formatted print to a string stream +using the +.code format +string and additional arguments. +.coNP Macros @ catch and @ catch* +.synb +.mets (catch < try-expression +.mets \ \ >> {( symbol <> ( arg *) << body-form *)}*) +.mets (catch* < try-expression +.mets \ \ >> {( symbol >> ( type-arg << arg *) << body-form *)}*) +.mets (catch** < try-expression +.mets \ \ >> {( symbol < desc >> ( type-arg << arg *) << body-form *)}*) +.syne +.desc The -.code obtain -macro registers a finalizer against the returned resume function. -The finalizer invokes the function, passing it the symbol -.codn sys:cont-poison , -thereby triggering unwinding in the most recently captured -continuation. Thus, abandoned -.code obtain -blocks are subject to unwinding when they become garbage. - +.code catch +macro establishes an exception catching block around +the +.metn try-expression . The -.code yield-from -macro works by capturing a continuation and performing a nonlocal -exit to the nearest block called -.metn name . -It passes a special yield object to that block. The -.code obtain -macro generates code which knows what to do with this special yield -object. +.meta try-expression +is followed by zero or more +catch clauses. Each catch clause consists of a symbol which denotes +an exception type, an argument list, and zero or more body forms. -.TP* Examples: +If +.meta try-expression +terminates normally, then the catch clauses +are ignored. The catch itself terminates, and its return value is +that of the +.metn try-expression . -The following example shows a function which recursively -traverses a -.code cons -cell structure, yielding all the -.cod2 non- nil -atoms it encounters. Finally, it returns the object -.codn nil . -The function is invoked on a list, -and the invocation is wrapped in an -.code obtain -block to convert it to a generating function. +If +.meta try-expression +throws an exception which is a subtype of one or more of +the type symbols given in the exception clauses, then the first (leftmost) such +clause becomes the exit point where the exception is handled. +The exception is converted into arguments for the clause, and the clause +body is executed. When the clause body terminates, the catch terminates, +and the return value of the catch is that of the clause body. -The generating function is then called six times -to retrieve the five atoms from the list, -and the final -.code nil -value. These are collected into a list. +If +.meta try-expression +throws an exception which is not a subtype of any of +the symbols given in the clauses, then the search for an exit point for +the exception continues through the enclosing forms. The catch clauses +are not involved in the handling of that exception. -This example demonstrates the power of delimited -continuations to suspend and resume a recursive -procedure. +When a clause catches an exception, the number of arguments in the catch must +match the number of elements in the exception. A catch argument list +resembles a function or lambda argument list, and may be dotted. For instance +the clause +.code "(foo (a . b))" +catches an exception subtyped from +.codn foo , +with one or +more elements. The first element binds to parameter +.codn a , +and the rest, if any, +bind to parameter +.codn b . +If there is only one element, +.code b +takes on the value +.codn nil . -.verb - (defun yflatten (obj) - (labels ((flatten-rec (obj) - (cond - ((null obj)) - ((atom obj) (yield-from yflatten obj)) - (t (flatten-rec (car obj)) - (flatten-rec (cdr obj)))))) - (flatten-rec obj) - nil)) +The +.code catch* +macro is a variant of +.code catch +with the following difference: when +.code catch* +invokes a clause, it passes the exception symbol as the leftmost argument +.metn type-arg . +Then the exception arguments follow. In contrast, +only the exception arguments are passed to the clauses of +.codn catch . - (let ((f (obtain (yflatten '(a (b (c . d)) e))))) - (list [f] [f] [f] [f] [f] [f])) - --> (a b c d e nil) -.brev +The +.code catch** +macro is a further variant, which differs from +.code catch* +by requiring each catch clause to provide a description +.metn desc , +an expression which evaluates to a character string. +The +.meta desc +expressions are evaluated in left-to-right order prior to the +evaluation of +.metn try-expression . -The following interactive session log exemplifies two-way communication between -the main code and a suspending function. +Also see: the +.code unwind-protect +operator, and the functions +.codn throw , +.code throwf +and +.codn error , +as well as the +.code handler-bind +operator and +.code handler +macro. -Here, -.code mappend -is invoked on a list of symbols representing fruit and vegetable names. -The objective is to return a list containing only fruits. +.coNP Operator @ unwind-protect +.synb +.mets (unwind-protect < protected-form << cleanup-form *) +.syne +.desc The -.code lambda -function suspends execution and yields a question out of the -.code map -block. It then classifies -the item as a fruit or not according to the reply it receives. The reply -emerges as a the result value of the -.code yield-from -call. +.code unwind-protect +operator evaluates +.meta protected-form +in such a way that no matter how the execution of +.meta protected-form +terminates, the +.metn cleanup-form -s +will be executed. The -.code obtain -macro converts the block to a generating function. The first call to the -function is made with no argument, because the argument would be ignored -anyway. The function returns a question, asking whether the first item -in the list, the potato, is a fruit. -To answer negatively, the user calls the function again, passing in -.codn nil . -The function returns the next question, which is answered in the -same manner. +.metn cleanup-form -s, +however, are not protected. If a +.meta cleanup-form +terminates via +some non-local jump, the subsequent +.metn cleanup-form -s +are not evaluated. -When the question for the last item is answered, the function -call yields the final item: the ordinary result of the block, which is the list -of fruit names. +.metn cleanup-form -s +themselves can "hijack" a non-local control transfer such +as an exception. If a +.meta cleanup-form +is evaluated during the processing of +a dynamic control transfer such as an exception, and that +.meta cleanup-form +initiates its own dynamic control transfer, the original control transfer +is aborted and replaced with the new one. + +The exit points for dynamic control transfers are removed as unwinding takes +place. That is to say, at the start of a dynamic control transfer, a search +takes place for the target exit point. That search might skip other exit points +which aren't targets of the control transfer. Those skipped exit points are left +undisturbed and are still visible during unwinding until their individual +binding forms are abandoned. Thus at the time of execution of an +.code unwind-protect +.metn cleanup-form , +all of the exit points of dynamically surrounding forms are still visible, even +ones which are nearer than the targeted exit point. +.TP* Example: .verb - 1> (obtain - (block map - (mappend (lambda (item) - (if (yield-from map `is @item a fruit?`) - (list item))) - '(potato apple banana lettuce orange carrot)))) - # - 2> (call *1) - "is potato a fruit?" - 3> (call *1 nil) - "is apple a fruit?" - 4> (call *1 t) - "is banana a fruit?" - 5> (call *1 t) - "is lettuce a fruit?" - 6> (call *1 nil) - "is orange a fruit?" - 7> (call *1 t) - "is carrot a fruit?" - 8> (call *1 nil) - (apple banana orange) + (block foo + (unwind-protect + (progn (return-from foo 42) + (format t "not reached!\en")) + (format t "cleanup!\en"))) .brev -The following example demonstrates an accumulator. Values passed to the -resume function are added to a counter which is initially zero. -Each call to the function returns the updated value of the accumulator. -Note the use of -.code "(yield-from acc)" -with no arguments to receive the value passed to the first -call to the resume function, without yielding an item. -The very first return value -.code 1 -is produced by the -.code "(yield-from acc sum)" -form, not by -.codn "(yield-from acc)" . -The latter only obtains the initial value -.code 1 -and uses it to establish the seed value of the accumulator. Without causing -the resume function to terminate and return, control passes into the loop, -which yields the first item, causing the resume function call -.code "(call *1 1)" -to return -.codn 1 : +In this example, the protected +.code progn +form terminates by returning from +block +.codn foo . +Therefore the form does not complete and so the +output +.str not reached! +is not produced. However, the cleanup form +executes, producing the output +.strn cleanup! . -.verb - 1> (obtain - (block acc - (let ((sum (yield-from acc))) - (while t (inc sum (yield-from acc sum)))))) - # - 2> (call *1 1) - 1 - 3> (call *1 2) - 3 - 4> (call *1 3) - 6 - 5> (call *1 4) - 10 -.brev -.coNP Macro @ obtain-block +.coNP Macro @ ignerr .synb -.mets (obtain-block < name << forms *) +.mets (ignerr << form *) .syne .desc The -.code obtain-block -macro combines -.code block -and -.code obtain -into a single expression. +.code ignerr +macro operator evaluates each +.meta form +similarly to the +.code progn +operator. If no forms are present, it returns +.codn nil . +Otherwise it evaluates each +.meta form +in turn, yielding the value of the last one. + +If the evaluation of any +.meta form +is abandoned due to an exception of type +.codn error , +the code generated by the +.code ignerr +macro catches this exception. In this situation, +the execution of the +.code ignerr +form terminates without evaluating the remaining +forms, and yields +.codn nil . + +.coNP Macro @ ignwarn +.synb +.mets (ignwarn << form *) +.syne +.desc The -.metn form -s -are evaluated in a block named -.codn name . +.code ignwarn +macro resembles +.codn ignerr . +It arranges for the evaluation of each +.meta form +in left-to-right order. If all the forms are evaluated, then the +value of the last one is returned. If no forms are present, then +.code nil +is returned. -That is to say, the following equivalence holds: +If any +.meta form +throws an exception of type +.code warning +then this exception is intercepted by a handler established by +.codn ignwarn . +This handler reacts by throwing an exception of type +.codn continue . -.verb - (obtain-block n f ...) <--> (obtain (block n f ...)) -.brev +The effect is that the warning is ignored, since the handler +doesn't issue any diagnostic, and passes control to the warning's +continue point. -.coNP Macro @ yield +Note: all sites within \*(TX which throw a +.code warning +also provide a nearby catch for a +.code continue +exception, for resuming evaluation at the point where the warning +was issued. + +.coNP Operator @ handler-bind .synb -.mets (yield <> [ form ]) +.mets (handler-bind < function-form < symbol-list << body-form *) .syne .desc The -.code yield -macro is to -.code yield-from -as -.code return -is to -.codn return-from : -it yields from an anonymous block. +.code handler-bind +operator establishes a handler for one or more +exception types, and evaluates zero or more +.metn body-form -s +in a dynamic scope in which that handler is visible. -It is equivalent to calling -.code yield-from -using +When the +.code handler-bind +form terminates normally, the handler is removed. The value of the +last +.meta body-form +is returned, or else .code nil -as the block name. - -In other words, the following equivalence holds: - -.verb - (yield x) <--> (yield-from nil x) -.brev +if there are no forms. -.TP* Example: +The +.meta function-form +argument is an expression which must evaluate to a function. The function +must be capable of accepting the exception arguments. All exceptions functions +require at least one argument, since the leftmost argument in an exception handler +call is the exception type symbol. -.verb - ;; Yield the integers 0 to 4 from a for loop, taking - ;; advantage of its implicit anonymous block: +The +.meta symbol-list +argument is a list of symbols, not evaluated. If it is empty, then the handler +isn't eligible for any exceptions. Otherwise it is eligible for any exception +whose exception type is a subtype of any of the symbols. - (defvarl f (obtain (for ((i 0)) ((< i 5)) ((inc i)) - (yield i)))) +If the evaluation of any +.meta body-form +throws an exception which is not handled within that form, and the handler +is eligible for that exception, then the function is invoked. It receives +the exception's type symbol as the leftmost argument. If the exception has +arguments, they appear as additional arguments in the function call. +If the function returns normally, then the exception search continues. +The handler remains established until the exception is handled in such a way +that a dynamic control transfer abandons the +.code handler-bind +form. - [f] -> 0 - [f] -> 1 - [f] -> 2 - [f] -> 3 - [f] -> 4 - [f] -> nil - [f] -> nil -.brev +Note: while a handler's function is executing, the handler is disabled. +If the function throws an exception for which the handler is eligible, +the handler will not receive that exception; it will be skipped by the +exception search as if it didn't exist. When the handler function terminates, +either via a normal return or a nonlocal control transfer, then the handler is +re-enabled. -.coNP Macros @ obtain* and @ obtain*-block +.coNP Macros @ handle and @ handle* .synb -.mets (obtain* << forms *) -.mets (obtain*-block < name << forms *) +.mets (handle < try-expression +.mets \ \ >> {( symbol <> ( arg *) << body-form *)}*) +.mets (handle* < try-expression +.mets \ \ >> {( symbol >> ( type-arg << arg *) << body-form *)}*) .syne .desc The -.code obtain* -and -.code obtain*-block -macros implement a useful variation of -.code obtain +.code handle +macro is a syntactic sugar for the +.code handler-bind +operator. Its syntax is exactly like that of +.codn catch . +The difference between +.code handle and -.codn obtain-block . +.code catch +is that the clauses in +.code handle +are invoked without unwinding. That is to say, +.code handle +does not establish an exit point for an exception. When control passes to +a clause, it is by means of an ordinary function call and not a dynamic +control transfer. No evaluation frames are yet unwound when this takes place. The -.code obtain* -macro differs from -.code obtain -in exactly one regard: prior to returning the function, it invokes -it one time, with the argument value -.codn nil . +.code handle +macro establishes a handler, by +.code handler-bind +whose +.meta symbol-list +consists of every +.meta symbol +gathered from every clause. -Thus, the following equivalence holds +The handler function established in the generated +.code handler-bind +is synthesized from of all of the clauses, together with dispatch logic which +which passes the exception and its arguments to the first +eligible clause. -.verb - (obtain* forms ...) <--> (let ((f (obtain forms ...))) - (call f) - f) -.brev +The +.meta try-expression +is evaluated in the context of this handler. -In other words, the suspended block is immediately resumed, so that it executes -either to completion (in which case its value is discarded), or to its first -.code yield -or -.code yield-from -call (in which case the yielded value is discarded). - -Note: the -.code obtain* -macro is useful in creating suspensions which accept data rather than -produce data. +The clause of the +.code handle +syntax can return normally, like a function, in which case the handler +is understood to have declined the exception, and exception processing +continues. To handle an exception, the clause of the +.code handle +macro must perform a dynamic control transfer, such returning from a block +via +.code return +or throwing an exception. The -.code obtain*-block -macro combines -.code obtain* -and -.code block -in the same manner that -.code obtain-block -combines -.code obtain -and -.codn block . - -.TP* Example: - -.verb - ;; Pass three values into suspended block, - ;; which get accumulated into list. - (let ((f (obtain*-block nil - (list (yield nil) (yield nil) (yield nil))))) - (call f 1) - (call f 2) - (call f 3)) -> (1 2 3) - - ;; Under obtain, extra call is required: - (let ((f (obtain-block nil - (list (yield nil) (yield nil) (yield nil))))) - (call f nil) ;; execute block to first yield - (call f 1) ;; resume first yield with 1 - (call f 2) - (call f 3)) -> (1 2 3) -.brev +.code handle* +macro is a variant of +.code handle +with the following difference: when +.code handle* +invokes a clause, it passes the exception symbol as the leftmost argument +.metn type-arg . +Then the exception arguments follow. In contrast, +only the exception arguments are passed to the clauses of +.codn handle . -.coNP Macro @ suspend +.coNP Macro @ with-resources .synb -.mets (suspend < block-name < var-name << body-form *) +.mets (with-resources >> ({ sym >> [ init-form <> [ cleanup-form ])}*) +.mets \ \ << body-form *) .syne .desc The -.code suspend -operator captures a continuation up to the prompt given by the -symbol -.meta block-name -and binds it to the variable name given by -.metn var-name , -which must be a symbol suitable for binding variables with -.codn let . +.code with-resources +macro provides a sequential binding construct similar to +.codn let* . +Every +.meta sym +is established as a variable which is visible to the +.metn init-form -s +of subsequent variables, to all subsequent +.metn cleanup-form -s +including that of the same variable, +and to the +.metn body-form -s. -Each -.meta body-form -is then evaluated in the scope of the variable -.metn var-name . +If no +.meta init-form +is supplied, then +.meta sym +is bound to the value +.codn nil . -When the last -.meta body-form -is evaluated, a non-local exit takes place to the block -named by -.meta block-name -(using the -.code sys:abscond-from -operator, so that unwinding isn't performed). +If an +.meta init-form +is supplied, but no +.metn cleanup-form , +then +.meta sym +is bound to the value of the +.metn init-form . -When the continuation bound to -.meta var-name -is invoked, a copy of the entire block -.meta block-name -is re-started, and in that copy, the -.code suspend -call appears to return normally, yielding the value which had been -passed to the continuation. +If a +.meta cleanup-form +is supplied in addition to +.metn init-form , +it specifies code to be executed upon the termination of the +entire +.code with-resources +construct. -.TP* Example +When an instance of +.code with-resources +terminates, all of the +.metn cleanup-form -s +specified in its binding clauses are evaluated, in reverse (right-to-left) +order. The value of the last +.meta body-form +is returned, or else +.code nil +if no +.metn body-form -s +are present. -Define John McCarthy's -.code amb -function using -.code block -and -.codn suspend : +.TP* "Example:" -.verb - (defmacro amb-scope (. forms) - ^(block amb-scope ,*forms)) +The following expression opens a text file and reads a line from it, +returning that line, while ensuring that the stream is closed +immediately: - (defun amb (. args) - (suspend amb-scope cont - (each ((a args)) - (when (and a (call cont a)) - (return-from amb a))))) +.verb + (with-resources ((f (open-file "/etc/motd") (close-stream f))) + (whilet ((l (get-line f))) + (put-line l))) .brev -Use -.code amb -to bind the of -.code x -and -.code y -which satisfy the predicate -.mono -.meti (eql (* x y) 8) -.onom -non-deterministically: -.verb - (amb-scope - (let ((x (amb 1 2 3)) - (y (amb 4 5 6))) - (amb (eql (* x y) 8) - (list x y)))) - -> (2 4) -.brev -.coNP Macros @ hlet and @ hlet* -.synb -.mets (hlet >> ({ sym | >> ( sym << init-form )}*) << body-form *) -.mets (hlet* >> ({ sym | >> ( sym << init-form )}*) << body-form *) -.syne -.desc +.coNP Special variable @ *unhandled-hook* The -.code hlet -and -.code hlet* -macros behave exactly like -.code let -and -.codn let* , -respectively except that they guarantee that the variable bindings are -allocated in storage which isn't captured by delimited continuations. +.code *unhandled-hook* +variable is initialized with +.code nil +by default. -The -.code h -in the names stands for "heap", serving as a mnemonic based on the -implementation concept of these bindings being "heap allocated". +It may instead be assigned a function which is capable of taking +three arguments. -.SS* Regular Expression Library +When an exception occurs which has no handler, this function is called, +with the following arguments: the exception type symbol, the exception object, +and a third value which is either +.code nil +or else the form which was being evaluated when the exception was thrown. +The call occurs before any unwinding takes place. -\*(TX provides a "pure" regular expression implementation based on automata -theory, which equates regular expressions, finite automata and sets of strings. -A regular expression determines whether or not a string of input characters -belongs to a set. \*(TX regular expressions do not support features such -as as "anchoring" a match to the start or end of a string, or capture of -parenthesized sub-expression matches into registers. Parenthesis syntax -denotes only grouping, with no additional meaning. +If the variable is +.codn nil , +or isn't a function, or the function returns after being called, +then unwinding takes place, after which some informational messages are printed +about the exception, and the process exits with a failed termination status. -The semantics of whether a regular expression is used for a substring -search, prefix match, suffix match, spring splitting and so forth comes from -the functions which use regular expressions to perform these operations. +In the case when the variable contains a object other than +.code nil +which isn't a function, a diagnostic message is printed on the +.code *stderr* +stream prior to unwinding. -.NP* Regular Expressions as Functions -.synb -.mets >> [ regex >> [ start <> [ from-end ]] < string ] -.syne -.desc -A regular expression is callable as a function in \*(TL. -When used this way, it requires a string argument. It searches -the string for the leftmost match for itself, and returns -the matching substring, which could be empty. If no match is -found, it returns +Prior to the function being called, the +.code *unhandled-hook* +variable is reset to .codn nil . -A regex takes one, two, or three arguments. The required -.meta string -is always the rightmost argument. This allows for convenient -partial application of the optional arguments using -macros in the -.code op -family, and macros in which the -.code op -syntax is implicit. - -The optional arguments -.meta start -and -.meta from-end -are treated exactly as their like-named counterparts in the -.code search-regst -function. - -.TP* Example: -Keep those elements from a list of strings which match -the regular expression -.codn #/a.*b/ : - -.verb - (keep-if #/a.*b/ '#"abracadabra zebra hat adlib adobe deer") - --> ("abracadabra" "adlib" "adobe") -.brev +Note: the functions +.code source-loc +or +.code source-loc-str +may be applied to the third argument of the +.code *unhandled-hook* +function to obtain more information about the form. -.coNP Functions @, search-regex @ range-regex and @ search-regst +.coNP Macro @ defex .synb -.mets (search-regex < string < regex >> [ start <> [ from-end ]]) -.mets (range-regex < string < regex >> [ start <> [ from-end ]]) -.mets (search-regst < string < regex >> [ start <> [ from-end ]]) +.mets (defex <> { symbol }*) .syne .desc -The -.code search-regex -function searches through -.meta string -starting -at position -.meta start -for a match for -.metn regex . +The macro +.code defex +records hierarchical relationships among symbols, for the purposes +of the use of those symbols as exceptions. It is closely related to the +.code @(defex) +directive in the \*(TX pattern language, performing the same function. + +All symbols are considered to be exception subtypes, and every symbol +is implicitly its own exception subtype. This macro does not introduce +symbols as exception types; it only introduces subtype-supertype +relationships. If -.meta start -is omitted, the search starts at position 0. If -.meta from-end -is specified and has a -.cod2 non- nil -value, the search -proceeds in reverse, from the position just beyond the last character of -.metn string , -toward -.metn start . +.code defex +is invoked with no arguments, it has no effect. -if -.meta start -exceeds the length of the string, then -.code search-regex -returns -.codn nil . +If arguments are present, they must be symbols. If -.meta start -is negative then it indicates positions from the end of the string, -such that -1 is the last character, -2 the second last and so forth. -If the value is so negative that it refers beyond the start of -the string, then the starting position is deemed to be zero. +.code defex +is invoked with only one symbol as its argument, it has no effect. -If -.meta start -is equal to the length of -.metn string , -and thus refers to the position one character past its -length, then a match occurs at that position if -.meta regex -admits such a match. +At least two +symbols must be specified for a useful effect to take place. If exactly two +symbols are specified, then, subject to error checks, +.code defex +makes the left symbol an +.I exception subtype +of the right symbol. -The -.code search-regex -function returns -.code nil -if no match is found, otherwise it returns -a cons, whose -.code car -indicates the position of the match, and whose -.code cdr -indicates the length of the match. +This behavior generalizes to three or more arguments: if three or more symbols +are specified, then each symbol other than the last is registered as a subtype of +the symbol which follows. -If -.meta regex -is capable of matching empty strings, and no other kind of match -is found within -.metn string , -then search regex reports a zero length match. If -.meta from-end -is false, then this match is reported at -.metn start , -otherwise it is reported at the position one character beyond -the end of the string. +If a +.code defex +has three or more arguments, they are processed from left to right. +If errors are encountered during the processing, the correct registrations +already made for prior arguments remain in place. -The -.code range-regex -function is similar to -.codn search-regex , -except that when -a match is found, it returns a position range, rather than a position -and length. A range object is returned whose -.code from -field indicates the position -of the match, and whose -.code to -indicates the position one element past the -last character of the match. If the match is empty, the two integers -are equal. +It is erroneous to register a duplicate relationship. If symbol +.code a +is already a direct or indirect subtype of +.code b +then +.code "(defex a b)" +and +.code "(defex a x b)" +are erroneous. -Also see the -.code rr -function, which provides an alternative argument syntax for -the semantics of -.codn range-regex . +Every symbol is implicitly considered to be its own exception subtype, +therefore it is erroneous to explicitly register a symbol as its +own subtype. -The -.code search-regst -differs from -.code search-regex -in the representation of the return value in the matching case. -Rather than returning the position and length of the match, -it returns the matching substring of -.metn string . +The foregoing rules eliminate the possibility of creating cycles in the +exception subtype inheritance graph. -.coNP Functions @ match-regex and @ match-regst +The symbol +.code nil +is implicitly a subtype of every exception type. Therefore, it is erroneous +to attempt to specify it as a supertype in a registration. +Using +.code nil +as a subtype in a registration is silently permitted, but has no effect. +No explicit registration is recorded between +.code nil +and its successor in the argument list. + +The symbol +.code t +is implicitly the supertype of every exception type. Therefore, it +is erroneous to attempt to register it as an exception subtype. +Using +.code t +as a supertype in a registration is also erroneous. + +Keyword symbols may be used as exception types. + +.coNP Function @ register-exception-subtypes .synb -.mets (match-regex < string < regex <> [ position ]) -.mets (match-regst < string < regex <> [ position ]) +.mets (register-exception-subtypes <> { symbol }*) .syne .desc The -.code match-regex -function tests whether -.meta regex -matches at -.meta position -in -.metn string . +.code register-exception-subtypes +function constitutes the underlying implementation for the +.code defex +macro. -If -.meta position -is not specified, it is taken to be zero. Negative values -of -.meta position -index from the right end of the string such that -1 -refers to the last character. Excessively negative -values which index before the first character cause -.code nil -to be returned. +The following equivalence applies: -If the regex matches, then the length of the match is returned. -If it does not match, then -.code nil -is returned. +.verb + (defex a b ...) <--> (register-exception-subtypes 'a 'b ...) +.brev -The -.code match-regst -differs from -.code match-regex -in the representation of the return value in the matching case. -Rather than returning the length of the match, it returns -matching substring of -.metn string . +That is, the +.code defex +macro works as if by generating a call to the function, with +the arguments quoted. -.coNP Functions @ match-regex-right and @ match-regst-right +The semantics of the function is precisely that of the macro. + +.coNP Function @ exception-subtype-p .synb -.mets (match-regex-right < string < regex <> [ end-position ]) -.mets (match-regst-right < string < regex <> [ end-position ]) +.mets (exception-subtype-p < left-symbol << right-symbol ) .syne .desc The -.code match-regex-right -function tests whether some substring of -.meta string -which terminates at the character position just before -.meta end-position -matches -.metn regex . +.code exception-subtype-p +function tests whether two symbols are in a relationship as exception types, +such that +.meta left-symbol +is a direct or indirect exception subtype of +.metn right-symbol . -If -.meta end-position -is not specified, it defaults to the length of the string, and the function -performs a right-anchored regex match. - -The -.meta end-position -argument can be a negative integer, in which case it denotes -positions from the end of the string, such that -1 refers -to the last character. If the value is excessively negative such -that the position immediately before it is before the start -of the string, then -.code nil -is returned. - -If -.meta end-position -is a positive value beyond the length of -.metn string , -then, likewise, -.code nil -is returned. - -If a match is found, then the length of the match is returned. - -A more precise way of articulating the role of -.meta end-position -is that for the purposes of matching, -.code string -is considered to terminate just before -.metn end-position : -in other words, that -.meta end-position -is the length of the string. The match is then anchored to the -end of this effective string. +If that is the case, then +.code t +is returned, otherwise +.codn nil . +.coNP Function @ exception-subtype-map +.synb +.mets (exception-subtype-map) +.syne +.desc The -.code match-regst-right -differs from -.code match-regst-right -in the representation of the return value in the matching case. -Rather than returning the length of the match, it returns -the matching substring of -.metn string . +.code exception-subtype-map +function returns a tree structure which captures information +about all registered exception types. -.TP* Examples: +The map appears as an association list which contains an entry +for every exception symbol, paired with that type's supertype path. +The first element in the supertype path is the exception's immediate +supertype. The next element is that type's supertype and so on. The +last element in every path is the grand supertype +.codn t . -.verb - ;; Return matching portion rather than length thereof. +For instance, if only the types +.codn a , +.code b +and +.code c +existed in the system, and were linked according to this inheritance graph: - (defun match-regex-right-substring (str reg : end-pos) - (set end-pos (or end-pos (length str))) - (let ((len (match-regex-right str reg end-pos))) - (if len - [str (- end-pos len)..end-pos] - nil))) +.verb + t ----+--- b --- a + | + +--- c +.brev - (match-regex-right-substring "abc" #/c/) -> "" +such that the supertype of +.code b +and +.code c +is +.codn t , +and +.code a +has +.code b +as supertype, then the function might return: - (match-regex-right-substring "acc" #/c*/) -> "cc" +.verb + ((a b t) (b t) (c t) (t)) +.brev - ;; Regex matches starting at multiple positions, but all - ;; the matches extend past the limit. - (match-regex-right-substring "acc" #/c*/ 2) -> nil +or any other equivalent permutation. - ;; If the above behavior is not wanted, then - ;; we can extract the string up to the limiting - ;; position and do the match on that. - (match-regex-right-substring ["acc" 0..2] #/c*/) -> "c" +The returned list may share substructure, so that the +.code "(t)" +sublist is shared among all four entries, and +.code "(b t)" +between the first two. - ;; Equivalent of above call - (match-regex-right-substring "ac" #/c*/) -> "c" -.brev +If the program alters the tree structure returned by +.codn exception-map-p , +the consequences are unspecified; this structure may in fact +be the very object which actually represents the type hierarchy, +and not a derived representation. -.coNP Function @ regex-prefix-match +.coNP Structures @, frame @ catch-frame and @ handle-frame .synb -.mets (regex-prefix-match < regex < string <> [ position ]) +.mets (defstruct frame nil) +.mets (defstruct catch-frame frame types desc jump) +.mets (defstruct handle-frame frame types fun) .syne .desc -The -.code regex-prefix-match -determines whether the input string might -might be the prefix of a string which matches regular expression -.metn regex . - -The result is true if the input string matches -.meta regex -exactly. However, it is also true in situations in which -the input string doesn't match -.metn regex , -yet can be extended with one or more additional characters beyond the end such -that the extended string -.B does -match. +The structure types +.codn frame , +.code catch-frame +and +.code handle-frame +are used by the +.code get-frames +and +.code find-frame +functions to represent information about the currently established +exception catches (see the +.code catch +macro) and handlers +(see +.code handler-bind +and +.codn handler ). The -.meta string -argument must be a character string. The function takes the input string to be -the suffix of -.meta string -which starts at the character position indicated by the -.meta position -argument. If that argument is omitted, then -.meta string -is taken as the input in its entirety. Negative values index backwards from -the end of -.meta string -according to the usual conventions elsewhere in the library. - -Note: this function is not to be confused for the semantics -of a regex matching a prefix of a string: that capability is -provided by the functions -.codn match-regex , -.codn m^ , -.codn r^ , -.code f^ +.code frame +type serves as the common base for +.code catch-frame and -.codn fr^ . - -.TP* Examples: - -.verb - ;; The empty string is not a viable prefix match for - ;; a regex that matches no strings at all: - (regex-prefix-match #/~.*/ "") -> nil - (regex-prefix-match #/[]/ "") -> nil +.codn handle-frame . - ;; The empty string is a viable prefix of any regex - ;; which matches at least one string: - (regex-prefix-match #// "") -> t - (regex-prefix-match #/abc/ "") -> t +Modifying any of the slots of these structures has no effect on the +actual frame from which they are derived; the frame structures are only +representation which provides information about frames. They are not +the actual frames themselves. - ;; This string doesn't match the regex because - ;; it doesn't end in b, but is a viable prefix: - (regex-prefix-match #/a*b/ "aa") -> t +Both +.code catch-frame +and +.code handle-frame +have a +.code types +slot. This holds the list of exception type symbols which are matched +by the catch or handler. - (regex-prefix-match #/a*b/ "ab") -> t +The +.code desc +slot of a +.code catch-frame +holds a list of the descriptions produced by the +.code catch** +macro. If there are no descriptions, then this member is +.codn nil , +otherwise it is a list whose elements are in correspondence +with the list in the +.code types +slot. - (regex-prefix-match #/a*b/ "ac") -> nil +The +.code jump +slot of a +.code catch-frame +is an opaque +.code cptr +("C pointer") +object which is related to the stack address of the catch +frame. If it is altered, the catch frame object becomes invalid +for the purposes of +.codn invoke-catch . - (regex-prefix-match #/a*b/ "abc") -> nil -.brev +The +.code fun +slot of a +.code handle-frame +is the registered handler function. Note that all the clauses of a +.code handler +macro are compiled to a single function, which is established via +.codn handler-bind , +so an instance of the +.code handler +macro corresponds to a single +.codn handle-frame . -.coNP Function @ regsub +.coNP Function @ get-frames .synb -.mets (regsub >> { regex | << function } < replacement << string ) +.mets (get-frames) .syne .desc The -.code regsub -function operates in two modes, depending on whether -the second argument is a regular expression, -or function. +.code get-frames +function inquires the current dynamic environment in order to retrieve +information about established exception catch and handler frames. +The function returns a list, ordered from the inner-most nesting +level to the outer-most nesting, of structure objects derived from the +.code frame +structure type. The list contains two kinds of objects: structures +of type +.code catch-frame +and of type +.codn handle-frame . -If the second argument is a regular expression it searches -.meta string -for multiple occurrences of non-overlapping matches for that -.metn regex . -A new string is constructed -similar to -.meta string -but in which each matching region is replaced -with using -.meta replacement -as follows. +These objects are not the frames themselves, but only provide information +about frames. Modifying the slots in these structures has no effect on +the original frames. Also, these structures have their own lifetime and +can endure after the original frames have disappeared. This has implications +for the use of the +.code invoke-catch +function. The -.meta replacement -object may be a character or a string, in which -case it is simply taken to be the replacement for each match -of the regular expression. +.code handle-frame +structures have a +.code fun +slot, which holds a function. It may be invoked directly. -The -.meta replacement -object may be a function of one argument, in -which case for every match which is found, this function is invoked, -with the matching piece of text as an argument. The function's -return value is then taken to be the replacement text. +A +.code catch-frame +structure may be passed as an argument to the +.code invoke-catch +function. -If the second argument is a function, then it is called, with -.meta string -as its argument. The return value must be either a range -object (see the -.code rcons -function) which indicates the extent of -.meta string -to be replaced, or else +.coNP Functions @ find-frame and @ find-frames +.synb +.mets (find-frame >> [ exception-symbol <> [ frame-type ]]) +.mets (find-frames >> [ exception-symbol <> [ frame-type ]]) +.syne +.desc +The +.code find-frame +function locates the first (innermost) instance of a specific kind of +exception frame (a catch frame or a handler frame) which is eligible +for processing an exception of a specific type. If such a frame +is found, it is returned. The returned frame object is of the same kind as the +objects which comprise the list returned by the function +.codn get-frames . +If such a frame is not found, .code nil -which indicates that no replacement is to take place. - -.TP* Examples: - -.verb - ;; match every lower case e or o, and replace by filtering - ;; through the upcase-str function: +is returned. - [regsub #/[eo]/ upcase-str "Hello world!"] -> "HEllO wOrld!" +The +.meta exception-symbol +argument specifies a match by exception type: the candidate frame +must specify in its list of matches at least one type which is an exception +supertype of +.metn exception-symbol . +If this argument is omitted, it defaults to +.code nil +which finds any handler that matches at least one type. There is no way to +search for handlers which match an empty set of types; the +.code find-frame +function skips such frames. - ;; Replace Hello with Goodbye: - (regsub #/Hello/ "Goodbye" "Hello world!") -> "Goodbye world!" +The +.meta frame-type +argument specifies which frame type to find. Useful values for this +argument are the structure type names +.code catch-frame +and +.code handle-frame +or the actual structure type objects which these type names denote. +If any other value is specified, the function returns +.codn nil . +If the argument is omitted, it defaults to the type of the +.code catch-frame +structure. That is to say, by default, the function looks for catch +frames. - ;; Left-anchored replacement with r^ function: - (regsub (fr^ #/H/) "J" "Hello, hello!") -> "Jello, hello!" -.brev +Thus, if +.code find-frame +is called with no arguments at all it finds the innermost catch frame, +if any exists, or else returns +.codn nil . -.coNP Function @ regexp -.synb -.mets (regexp << obj ) -.syne -.desc The -.code regexp -function returns -.code t -if -.meta obj -is a compiled regular expression -object. For any other object type, it returns -.codn nil . +.code find-frames +function is similar to +.code find-frame +except that it returns all matching frames, ordered from the inner-most nesting +level to the outer-most nesting. If called with no arguments, it returns a +list of the catch frames. -.coNP Function @ regex-compile +.coNP Function @ invoke-catch .synb -.mets (regex-compile < form-or-string <> [ error-stream ]) +.mets (invoke-catch < catch-frame < symbol << argument *) .syne .desc The -.code regex-compile -function takes the source code of a regular expression, -expressed as a Lisp data structure representing an abstract syntax tree, or -else a regular expression specified as a character string, and compiles it to a -regular expression object. +.code invoke-catch +function abandons the current evaluation context to perform +a non-local control transfer directly to the catch +described by the +.meta catch-frame +argument, which must be a structure of type +.code catch-frame +obtained using any of the functions +.codn get-frames , +.code find-frames +or +.codn find-frame . -If -.meta form-or-string -is a character string, it is parsed to an -abstract syntax tree first, if by the -.code regex-parse -function. -If the parse is successful (the result is not -.codn nil ) -then -the resulting tree structure is compiled by a recursive call to -.codn regex-compile . +The control transfer is possible only if the catch +frame represented by +.meta catch-frame +structure is still established, and if the structure +hasn't been tampered with. -The optional -.meta error-stream -argument is passed down to -.code regex-parse -as well as in the recursive call to -.codn regex-compile , -if that call takes place. +If a given +.code catch-frame +structure is usable with +.codn invoke-catch , +then a copy of that structure made with +.code copy-struct +is also usable, denoting the same catch frame. -If -.meta error-stream -is specified, it must be a stream. Any error diagnostics are sent to that -stream. +The +.meta symbol +argument should be an exception symbol. It is passed to the +exception frame, as if it had appeared as the first argument of the +.code throw +function. Similarly, the +.metn argument -s +are passed to the catch frame as if they were the trailing arguments +of a +.codn throw . +The difference between +.code invoke-catch +and +.code throw +is that +.code invoke-catch +targets a specific catch frame as its exit point, rather than searching for a +matching catch or handler frame. That specific frame receives the control. +The frame receives control even if it it is not otherwise eligible for +catching the exception type denoted by +.metn symbol . -.TP* Examples: +.SS* Static Error Diagnosis -.verb - ;; the equivalent of #/[a-zA-Z0-9_/ - (regex-compile '(set (#\ea . #\ez) (#\eA . #\eZ) - (#\e0 . #\e9) #\e_)) +This section describes a number of features related to the diagnosis +of errors during the static processing of program code prior to evaluation. +The material is of interest to developers of macros intended for broad +reuse. - ;; the equivalent of #/.*/ and #/.+/ - (regex-compile '(0+ wild)) - (regex-compile '(1+ wild)) +.NP* Error Exceptions - ;; #/a|b|c/ - (regex-compile '(or (or #\ea #\eb) #\ec)) +\*(TL uses exceptions of type +.code eval-error +to identify erroneous situations during both transformation of code +and its evaluation. These exceptions have one argument, which is a +character string. If not handled by program code, +.code eval-error +exceptions are specially recognized and treated by the built-in handling logic. +The message is incorporated into diagnostic output which includes +more information which is deduced. - ;; string - (regex-compile "a|b|c") -.brev +.NP* Warning Exceptions -.coNP Function @ regex-source -.synb -.mets (regex-source << regex ) -.syne -.desc -The -.code regex-source -function returns the source code of compiled regular expression -.metn regex . +\*(TL uses exceptions of type +.code warning +to identify certain situations of interest. Ordinary non-deferrable +warnings have a structure identical to errors, except for the exception +symbol. \*(TX's built-in handling of warnings expects these exceptions +to be continuable. What this means is that a +.code catch +for the +.code continue +exception is expected to be visible. The handler for a warning exception +issues a diagnostic which incorporates the warning message. Then the +handler throws a +.code continue +exception to resume the computation which generated the warning. -The source code isn't the textual notation, but the Lisp -data structure representing the abstract syntax tree: the -same representation as what is returned by -.codn regex-parse . +The generation of a warning thus conforms to the following pattern: -.coNP Function @ regex-parse -.synb -.mets (regex-parse < string <> [ error-stream ]) -.syne -.desc -The -.code regex-parse -function parses a character string which contains a regular expression and -turns it into a Lisp data structure (the abstract syntax tree representation of -the regular expression). - -The regular expression syntax -.code #/RE/ -produces the same structure, but as a -literal which is processed at the time \*(TX source code is read; the -.code regex-parse -function performs this parsing at run-time. - -If there are parse errors, the function returns -.codn nil . - -The optional -.meta error-stream -argument specifies a stream to which error messages -are sent from the parser. By default, diagnostic output goes to the -.code *stdnull* -stream, which discards it. If -.meta error-stream -is specified as -.codn t , -then the diagnostic output goes to the -.code *stdout* -stream. +.verb + (catch + (throw 'warning "message") + (continue ())) +.brev -If -.code regex-parse -returns a -.cod2 non- nil -value, that structure is then something -which is suitable as input to -.codn regex-compile . +.NP* Deferrable Warnings -There is a small difference in the syntax accepted by -.code regex-parse -and the syntax of regular expression literals. Any -.code / -(slash) characters occurring in any position within -.meta string -are treated as ordinary characters, not as regular expression delimiters. -The call -.mono -(regex-parse "/a/") -.onom -matches three characters: a slash, followed by the letter "a", followed -by another slash. Note that the slashes are not escaped. +\*(TX supports a form of diagnostic known as a +.IR "deferrable warning" . +A deferrable warning is distinguished in two ways. Firstly, it is +either of the type +.code defr-warning +or subtyped from that type. The +.code defr-warning +type itself is a direct subtype of +.codn warning . -Note: if a -.code regex-parse -call is written using a string literal as the -.meta string -argument, then note that any backslashes which are to be processed -by the regular expression must be doubled up, otherwise they belong -to the string literal: +Secondly, a deferrable warning carries an additional tag argument after the +exception message. A deferrable exception is thrown according to +this pattern: .verb - (regex-parse "\e*") ;; error, invalid string literal escape - (regex-parse "\e\e*") ;; correct: the \e* literal match for * + (catch + (throw 'defr-warning "message" . tag) + (continue ())) .brev -The double backslash in the string literal produces a single backslash -in the resulting string object that is processed by -.codn regex-parse . +\*(TX's built-in exception handling logic reacts specially to the +presence of the tag material in the exception. First, the global +.I "tentative definition list" +is searched for the presence of the tag, using +.code equal +equality. If the tag is found, then the warning is discarded. +If the tag is not found, then the exception argument list is added +to the global +.IR "deferred warning list" . +In either case, +the +.code continue +exception is thrown to resume the computation which threw the warning, +as in the case of an ordinary non-deferrable warning. -.coNP Function @ read-until-match +The purpose of this mechanism is to suppress warnings which +become superfluous when more of the program code is examined. +For instance, a warning about a call to an undefined function is +superfluous if a definition of that function is supplied later, +yet before that function call is executed. + +Deferred warnings accumulate in the deferred warning list +from which they can be removed. The list is purged at various +times such as when a top-level load completes, and the +deferred warnings are released, as if by a call to the +.code release-deferred-warnings +function. + +.coNP Functions @ compile-error and @ compile-warning .synb -.mets (read-until-match < regex >> [ stream <> [ include-match ]]) +.mets (compile-error < context-obj < fmt-string << fmt-arg *) +.mets (compile-warning < context-obj < fmt-string << fmt-arg *) .syne .desc +The functions +.code compile-error +and +.code compile-warning +provide a convenient and uniform way for code transforming +functions such as macro-expanders to generate diagnostics. The -.code read-until-match -function reads characters from -.metn stream , -accumulating them into a string, which is returned. - -If an argument is not specified for -.metn stream , -then the -.code *std-input* -stream is used. - +.code compile-error +function throws an exception of type +.codn eval-error . The -.meta include-match -argument is Boolean, indicating whether the delimiting text -matched by -.meta regex -is included in the returned string. It defaults to -.codn nil . - -The accumulation of characters is terminated by a match on -.metn regex , -the end of the stream, or an error. - -This means that characters are read from the stream and accumulated while the -stream has more characters available, and while its prefix does not match -.metn regex . - -If -.meta regex -matches the stream before any characters are accumulated, -then an empty string is returned. +.code compile-warning +function throws an exception of type +.code warning +and internally provides the expected +.code catch +for the +.code continue +exception needed to resume after the warning. -If the stream ends or an non-exception-throwing error occurs before any -characters are accumulated, the function returns -.codn nil . +The argument conventions are the same for both functions. +The +.meta context-obj +is typically a compound form to which the diagnostic +applies. -When the accumulation of characters terminates by a match on -.metn regex , -the longest possible matching sequence of characters is -removed from the stream. If -.meta include-match -is true, that matching text is included in -the returned string. Otherwise, it is discarded. +The functions produce a diagnostic message which +incorporates the location information and symbol +obtained from +.meta context-obj +and the +.codn format -style +arguments +.meta fmt-string +and its +.metn fmt-arg -s. -.coNP Functions @ scan-until-match and @ count-until-match +.coNP Function @ compile-defr-warning .synb -.mets (scan-until-match < regex <> [ stream ]) -.mets (count-until-match < regex <> [ stream ]) +.mets (compile-defr-warning < context-obj < tag +.mets \ \ < fmt-string << fmt-arg *) .syne .desc -The functions -.code scan-until-match -and -.code count-until-match -read characters from -.meta stream -until a match occurs in the stream for regular expression -.metn regex , -the stream runs out of characters, or an error occurs. - -If the stream runs out of characters, or a non-exception-throwing error -occurs, before a match for -.meta regex -is identified, these functions return -.codn nil . +The +.code compile-defr-warning +function throws an exception of type +.code defr-warning +and internally provides the expected +.code catch +for the +.code continue +exception needed to resume after the warning. -If a match for -.meta regex -occurs in -.metn stream , -then -.code count-until-match -returns the number of characters that were read and discarded prior to -encountering the first matching character. -In the same situation, the -.code scan-until-match -function returns a -.code cons -cell whose -.code car -holds the count of discarded characters, that being the same value as what -would be returned by -.codn count-until-match , -and whose -.code cdr -holds a character string that comprises the text matched by -.metn regex . +The function produces a diagnostic message which +incorporates the location information and symbol +obtained from +.meta context-obj +and the +.codn format -style +arguments +.meta fmt-string +and its +.metn fmt-arg -s. +This diagnostic message constitutes the first +argument of the exception. The +.meta tag +argument is taken as the second argument. -.coNP Functions @, m^$ @ m^ and @ m$ +.coNP Function @ purge-deferred-warning .synb -.mets (m^$ < regex <> [ position ] << string ) -.mets (m^ < regex <> [ position ] << string ) -.mets (m$ < regex <> [ end-position ] << string ) +.mets (purge-deferred-warning << tag ) .syne .desc -These functions provide functionality similar to the -.meta match-regst -and -.meta match-regst-right -functions, but under alternative interfaces which are more -convenient. - The -.code ^ -and -.code $ -notation used in their names are an allusion to the -regular expression search anchoring operators found in -familiar POSIX utilities such as -.codn grep . +.code purge-deferred-warning +removes all warnings marked with +.meta tag +from the deferred list. It also removes all tags +matching +.meta tag +from the tentative definition list. +Tags are compared using the +.code equal +function. +.coNP Function @ register-tentative-def +.synb +.mets (register-tentative-def << tag ) +.syne +.desc The -.meta position -argument, if omitted, -defaults to zero, so that the -entire -.meta string -is operated upon. +.code register-tentative-def +function adds +.meta tag +to the list of tentative definitions which are +used to suppress deferrable warnings. -The -.meta end-position -argument defaults to the length of -.metn string , -so that the end position coincides with the end of the -string. +The idea is that a definition of some construct has been seen, +but not yet executed. Thus the construct is not defined, but +it can reasonably be expected that it will be defined; +hence, warnings about its nonexistence can be suppressed. -If the -.meta position -or -.meta end-position -arguments are negative, they index backwards -from the length of -.meta string -so that -1 denotes the last character. +For example, in the following code, when the expression +.code "(foo)" +is being expanded and transformed, the +.code foo +function does not exist: -A value in either parameter which is excessively -negative or positive, such that it indexes before -the start of the string or exceeds its length -results in a failed match and consequently -.code nil -being returned. +.verb + (progn (defun foo ()) (foo)) +.brev -The -.code m^$ -function tests whether the entire portion of -.meta string -starting at -.meta position -through to the end of the string is in the set of strings -matched by -.metn regex . -If this is true, then that portion of the string is -returned. Otherwise -.code nil -is returned. +The function won't be defined until the +.code progn +is evaluated. Thus a warning is generated that +.code "(foo)" +refers to an undefined function. +However, this warning is discarded, because the +expander for +.code defun +registers a tentative definition tag for +.codn foo . -The -.code m^ -function tests whether the portion of the -.meta string -starting at -.meta position -has a prefix which matches -.metn regex . -If so, then this matching prefix is returned. -Otherwise -.code nil -is returned. +When the definition of +.code foo +takes place, the +.code defun +operator will call +.code purge-deferred-warning +which will remove not only all accumulated warnings +related to the undefinedness of +.code foo +but also remove the tentative definition. -The -.code m$ -function tests whether the portion of -.meta string -ending just before -.meta end-position -has a suffix which matches -.metn regex . -If so, then this matching suffix is returned. -Otherwise -.code nil -is returned. +Note: this mechanism isn't perfect because it will +still suppresses the warning in situations like -.coNP Functions @, r^$ @, r^ @ r$ and @ rr +.verb + (progn (if nil (defun foo ())) (foo)) +.brev + + +.coNP Function @ tentative-def-exists .synb -.mets (r^$ < regex <> [ position ] << string ) -.mets (r^ < regex <> [ position ] << string ) -.mets (r$ < regex <> [ end-position ] << string ) -.mets (rr < regex >> [ position <> [ from-end ]] << string ) +.mets (tentative-def-exists << tag ) .syne .desc -The first three of these functions perform the same operations as, -respectively, -.codn m^$ , -.code m^ -and -.codn m$ , -with the same argument conventions. They differ -in return value. When a match is found, they -return a range value indicating the extent of -the matching substring within -.meta string -rather than the matching substring itself. - -The -.code rr -function performs the same operation as -.code range-regex -with different conventions with regard to argument -order, harmonizing with those of the other three functions above. - The -.meta position -argument, if omitted, -defaults to zero, so that the -entire -.meta string -is operated upon. +.code tentative-def-exists +function checks whether +.meta tag +has been registered via +.code register-tentative-def +and not yet purged by +.codn purge-deferred-warning . +.coNP Function @ defer-warning +.synb +.mets (defer-warning << args ) +.syne +.desc The -.meta end-position -argument defaults to the length of -.metn string , -so that the end position coincides with the end of the -string. +.code defer-warning +function attempts to register a deferred warning. The +.meta args +argument corresponds to the arguments which are passed to the +.code throw +function in order to generate a warning exception, not including the exception +symbol. -With one exception, a value in either parameter which is excessively negative -or positive, such that it indexes before the start of the string or exceeds its -length results in a failed match and consequently -.code nil -being returned. The exception is that the -.code rr -function permits a negative -.meta position -value which refers before the start of the string; this is effectively -treated as zero. +Args is expected to have at least two elements, the second of which +is a deferred warning tag. The -.meta from-end -argument defaults to +.code defer-warning +function returns .codn nil . -The -.code r^$ -function tests whether the entire portion of -.meta string -starting at -.meta position -through to the end of the string is in the set of strings -matched by -.metn regex . -If this is true, then the matching range is returned, -as a range object. +Note: this function is intended for use in exception handlers. The +following example shows a handler which intercepts warnings. It defers +deferrable warnings, and prints ordinary warnings: -The -.code r^ -function tests whether the portion of the -.meta string -starting at -.meta position -has a prefix which matches -.metn regex . -If so, then the matching range is returned, as a range object. -Otherwise -.code nil -is returned. +.verb + (handle + (some-form ..) ;; some code which might generate warnings + (defr-warning (msg tag) ;; catch deferrable and defer + (defer-warning (cons msg tag)) + (throw 'continue)) ;; warning processed: resume execution + (warning (msg) + (put-line `warning: @msg`) ;; print non-deferrable + (throw 'continue))) ;; warning processed: resume execution +.brev +.coNP Function @ release-deferred-warnings +.synb +.mets (release-deferred-warnings) +.syne +.desc The -.code r$ -function tests whether the portion of -.meta string -ending just before -.meta end-position -has a suffix which matches -.metn regex . -If so, then the matching range is returned. -Otherwise -.code nil -is returned. - -The -.code rr -function searches -.meta string -starting at -.meta position -for a match for -.codn regex . -If -.meta from-end -is specified and true, the rightmost -match is reported. -If a match is found, it is reported -as a range. - -A regular expression which matches empty -strings matches at the start position, -and every other position, including -the position just after the last -character, coinciding with the length of -.metn string . +.code release-deferred-warnings +removes all warnings from the deferred list. +Then, it issues each deferred warning as an ordinary warning. -Except for the different argument order such that -.meta string -is always the rightmost argument, the -.code rr -function is equivalent to the -.code range-regex -function, such that correspondingly named -arguments have the same semantics. +Note: there is normally no need for user programs to use this +function since deferred warnings are issued automatically. -.coNP Function @ rra +.coNP Function @ dump-deferred-warnings .synb -.mets (rra < regex >> [ start <> [ end ]] << string ) +.mets (dump-deferred-warning << stream ) .syne .desc The -.code rra -function searches -.meta string -between the -.meta start -and -.meta end -position for matches for the regular expression -.metn regex . +.code dump-deferred-warnings +empties the list of deferred warnings, and converts each one +into a diagnostic message sent to +sent to +.metn stream . +After the diagnostics are printed, the list of pending warnings +is cleared. -The matches are returned as a list of range objects. +Note: there is normally no need for user programs to use this +function since deferred warnings are issued automatically. -The -.meta start -argument defaults to zero, and -.meta end -defaults to the length of the string (the position one -past the last character). +.SS* Delimited Continuations -Negative values of -.meta start -and -.meta end -indicate positions from the end of the string, such that -1 -denotes the last character, -2 the second-to-last and so forth. +\*(TL supports delimited continuations, which are integrated with the +.code block +feature. Any named or anonymous block, including the implicit blocks +created around function bodies, can be used as the delimiting +.I prompt +for the capture of a continuation. -If -.meta start -is so negative that it refers before the start of -.metn string , -it is treated as zero. If this situation is true of the -.meta end -argument, then the function returns -.codn nil . +A delimited continuation is section of a possible future of the +computation, up to a delimiting prompt, +.I reified +as a first class function. -If -.meta start -refers to a character position beyond the length of -.meta string -(two characters or more beyond the end of the string), -then the function returns -.codn nil . -If this situation is true of -.metn end , -then -.meta end -is is curtailed to the the string length. +.TP* Example: -The -.code rra -function returns all non-overlapping matches, including -zero length matches. Zero length matches may occur before -the first character of the string, or after the last -character. If so, these are included. +.verb + (defun receive (cont) + (format t "cont returned ~a\en" (call cont 3))) -.coNP Functions @, f^$ @ f^ and @ f$ -.synb -.mets (f^$ < regex <> [ position ]) -.mets (f^ < regex <> [ position ]) -.mets (f$ < regex <> [ end-position ]) -.syne -.desc -These regular expression functions do not directly -perform regex operations. Rather, they each return -a function of one argument which performs a regex -operation. + (defun function () + (sys:capture-cont 'abcd (fun receive))) -The returned functions perform the same operations as, -respectively, -.codn m^$ , -.code m^ -and -.codn m$ . + (block abcd + (format t "function returned ~a\en" (function)) + 4) -The following equivalences nearly hold, except that the functions -on the right side produced by -.code op -can accept two arguments when only -.code r -is curried, whereas the functions on the left take only -one argument: + Output: -.verb - [f^$ r] <--> (op m^$ r) - [f^$ r p] <--> (op m^$ r p) - [f^ r] <--> (op m^ r) - [f^ r p] <--> (op m^ r p) - [f$ r] <--> (op m$ r) - [f$ r p] <--> (op m$ r p) + function returned 3 + cont returned 4 + function returned t .brev -That is to say, -.code f^$ -returns a function which binds -.meta regex -and possibly the optional -.metn position . -When this function is invoked, it must be given an argument -which is a string. It performs the same operation as -.code m^$ -being called on -.meta regex -and possibly -.metn position . -The same holds between -.code f^ -and -.codn m^ , -and between -.code f$ -and -.codn m$ . - -.TP* Examples: - -.verb - ;; produce list which contains only strings - ;; beginning with "cat": - (keep-if (f^ #/cat/) '#"dog catalog cat fox catapult") - --> ("catalog" "cat" "catapult") +.PP - ;; map all strings in a list to just their trailing - ;; digits. - (mapcar (f$ #/\ed*/) '#"a123 4 z bc465") - --> ("123" "4" "" "465") +Evaluation begins with the +.code block +form. This form calls +.code function +which uses +.code sys:capture-cont +to capture a continuation up to the +.code abcd +prompt. The continuation is passed to the +.code receive +function as an argument. - ;; check that all strings consist of digits after - ;; the third position. - (all '#"ABC123 DFE45 12379" (f^$ #/\ed*/ 3)) - --> "79" ; i.e. true - (all '#"ABC123 DFE45 12379A" (f^$ #/\ed*/ 3)) - --> nil -.brev +This captured object represents the continuation of computation +up to that prompt. It appears as a one-argument function which, when called, +resumes the captured computation. Its argument emerges out of the +.code sys:capture-cont +call as a return value. When the computation eventually returns all +the way to the delimiting prompt, the return value of that prompt +will then appear as the return value of the continuation function. -.coNP Functions @, fr^$ @, fr^ @ fr$ and @ frr -.synb -.mets (fr^$ < regex <> [ position ]) -.mets (fr^ < regex <> [ position ]) -.mets (fr$ < regex <> [ end-position ]) -.mets (frr < regex <> [[ start-position ] << from-end ]) -.syne -.desc -These regular expression functions do not directly -perform regex operations. Rather, they each return -a function of one argument which performs a regex -operation. +In this example, the function +.code receive +immediately invokes the continuation function which it receives, passing +it the argument value +.codn 3 . +And so, +evaluation now continues in the resumed future represented by the +continuation. Inside the continuation, +.code sys:capture-cont +appears to return, yielding the value +.codn 3 . +This bubbles up through +.code function +up to the +.code "block abcd" +where a message is printed: +.strn "function returned 3" . -The returned functions perform the same operations as, -respectively, -.codn r^$ , -.codn r^ , -.code r$ -and -.codn rr . +The +.code block +terminates, yielding the value 4. Thereby, the continuation ends, since +it is delimited up to that block. Control now returns to the +.code receive +function which invoked the continuation, where the function call form +.code "(call cont)" +terminates, yielding the value +.code 4 +that was returned by the continuation's delimiting +.code block +form. The message +.str "cont returned 4" +is printed. The +.code receive +function returns normally, returning the value +.code t +which emerged from the +.code format +call. Control is now back in +.code function +where the +.code sys:capture-cont +form terminates and returns the +.codn t . +This bubbles up to +.code block +which prints +.strn "function returned t" . -The following equivalences nearly hold, except that some of the -functions on the right side produced by op -.code op -can accept additional arguments after the input string, -whereas the functions on the left produced by -.code f^$ -.I "et al." -accept only one parameter: the input string. +In summary, a continuation represents, as a function, the subsequent +computation that is to take place starting at some point, up to some +recently established, dynamically enclosing delimiting prompt. When +the continuation is captured, that future doesn't have to take place; +an alternative future can carry out in which that continuation is +available as a function. That alternative future can invoke the continuation at +will. Invocations (resumptions) of the continuation appear as additional +returns from the capture operator. A resumption of a continuation terminates +when the delimiting prompt terminates, and the continuation yields the +value which emerges from the prompt. -.verb - [fr^$ r] <--> (op m^$ r) - [fr^$ r p] <--> (op m^$ r p) - [fr^ r] <--> (op m^ r) - [fr^ r p] <--> (op m^ r p) - [fr$ r] <--> (op m$ r) - [fr$ r p] <--> (op m$ r p) - [frr r] <--> (op m$ r) - [frr r s] <--> (op m$ r s) - [frr r s fe] <--> (op m$ r s fe) -.brev +Delimited continuations are implemented by capturing a segment of the +evaluation stack between the prompt and the capture point. When +a continuation is resumed, this saved copy of a stack segment is inserted on +top of the current stack and the procedure context is resumed such +that evaluation appears to emerge from the capture operator. +As the continuation runs to completion, it simply pops these inserted +stack frames naturally. Eventually it pops out of the delimiting prompt, +at which point control ends up at the point which invoked the continuation +function. -That is to say, -.code fr^$ -returns a function which binds -.meta regex -and possibly the optional -.metn position . -When this function is invoked, it must be given an argument -which is a string. It performs the same operation as -.code r^$ -being called on -.meta regex -and possibly -.metn position , -and the string. -The same holds between -.code fr^ -and -.codn r^ , -between -.code fr$ -and -.codn r$ , -and between -.code frr +The low-level operator for capturing a continuation is +.codn sys:capture-cont . +More expressive and convenient programming with continuations is +provided by the macros +.codn obtain , +.codn obtain-block , +.code yield-from and -.codn rr . - -.TP* Examples: +.codn yield , +which create an abstraction which models the continuation as a suspended +procedure supporting two-way communication of data. +A +.code suspend +operator is provided, which is more general. It is identical to the +.code shift +operator described in various computer science literature about +delimited continuations, except that it refers to a specific delimiting +prompt by name. -.verb - ;; Remove leading digits from "123A456", - ;; other than first digit: - (regsub (fr^ #/\ed+/ 1) "" "123A456") - --> "1A456" -.brev +Continuations raise the issue of what to do about unwinding. +The language Scheme provides the much criticized +.code dynamic-wind +operator which can execute initialization and clean-up code as +a continuation is entered and abandoned. \*(TX takes a simpler, +albeit risky approach. It provides a non-unwinding escape operator +.code sys:abscond-from +for use with continuations. Code which has captured a continuation +can use this operator to escape from the delimiting block without +triggering any unwinding among the frames between the capture point and the +delimiter. When the continuation is restarted, it will then do so +with all of the resources associated with it frames intact. +When the continuation executes normal returns within its context, +the unwinding takes place then. Thus tidy, "thread-like" use +of continuations is possible with a small measure of coding discipline. +Unfortunately, the absconding operator is dangerous: its use +breaks the language guarantee that clean-up associated with a form is done no +matter how a form terminates. -.SS* Hashing Library +.NP* Comparison with Lexical Closures -A hash table is an object which retains an association between pairs of -objects. Each pair consists of a key and value. Given an object which is -similar to a key in the hash table, it is possible to retrieve the -corresponding value. Entries in a hash table are not ordered in any way, and -lookup is facilitated by hashing: quickly mapping a key object to a numeric -value which is then used to index into one of many buckets where the matching -key will be found (if such a key is present in the hash table). +Delimited continuations resemble lexical closures in some ways. Both +constructs provide a way to return to some context whose evaluation +has already been abandoned, and to access some aspects of that context. +However, lexical closures are statically scoped. Closures capture the lexically +apparent scope at a given point, and produce a function whose body has access +to that scope, as well as to some arbitrary arguments. Thus, a lexical scope +is reified as a first-class function. By contrast, a delimited continuation +is dynamic. It captures an an entire segment of a program activation chain, +up to the delimiting prompt. This segment includes scopes which are not +lexically visible at the capture point: the scopes of parent functions. +Moreover, the segment includes not only scopes, but also other aspects of +the evaluation context, such as the possibility of returning to callers, +and the (captured portion of) the original dynamic environment, such as +exception handlers. That is to say, a lexical closure's body cannot return to +the surrounding code or see any of its original dynamic environment; it can +only inspect the environment, and then return to its own caller. Whereas a +restarted delimited continuation can continue evaluation of the surrounding +code, return to surrounding forms and parent functions, and access the dynamic +environment. The continuation function returns to its caller when that entire +restarted context terminates, whereas a closure returns to its caller as soon +as the closure body terminates. -In addition to keys and values, a hash table contains a storage location -which allows it to be associated with user data. +.NP* Differences in Compiled vs. Interpreted Behavior -Important to the operation of a hash table is the criterion by which keys are -considered same. By default, this similarity follows the eql function. A hash -table will search for a stored key which is -.code eql -to the given search key. -A hash table constructed with the -.codn equal -based -property compares keys using -the -.code equal -function instead. +Delimited continuations in \*(TX expose a behavioral difference between +compiled and interpreted code which mutates the values of lexical variables. -In addition to storing key-value pairs, a hash table can have a piece of -information associated with it, called the user data. +When a continuation is captured in compiled code, it captures not only the +bindings of lexical variables, but also potentially their current values +at the time of capture. What this means is that whenever the continuation +is resumed, those variables will appear to have the captured values, +regardless of any mutations that have taken place since. In other words, +the captured future includes those specific values. This is because in +compiled code, variables are allocated on the stack, which is copied as part of +creating a continuation. Those variables are effectively newly instantiated in +each resumption of the continuation, when the captured stack segment +is reinstated into the stack, and take on those original values. -\*(TX hash tables contain a seed value which permutes the hashing operation, -at least for keys of certain types. This feature, if the seed is randomized, -helps to prevent software from being susceptible to hash collision -denial-of-service attacks. However, by default, the seed is not randomized. -Newly created hash tables for which a seed value is not specified take their -seed value from the -.code *hash-seed* -special variable, which is initialized to zero. That includes hash tables -created by parsing hash literal syntax. -Security-sensitive programs -requiring protection against collision attacks may use -.code gen-hash-seed -to create a randomized hash seed, and, depending on their specific need, either -store that value in -.codn *hash-seed* , -or pass the value to hash table constructors like -.codn make-hash , -or both. -Note: randomization of hash seeding isn't a default behavior because it affects -program reproducibility. The seed value affects the order in which keys are -traversed, which can change the output of programs whose inputs have not -changed, and whose logic is is otherwise deterministic. +In contrast, interpretation of code only maintains an +environment pointer on the stack; the lexical environment is a dynamically +allocated object whose contents aren't included in the continuation's +stack segment capture. If the captured variables are modified after the +capture, the continuation will see the updated values: all resumptions of the +continuation share the same instance of the captured environment among +themselves, and with the original context where the capture took place. -A hash table can be traversed to visit all of the keys and data. The order of -traversal bears no relation to the order of insertion, or to any properties of -the key type. +An additional complication is that when compiled code captures lexical +closures, captured variables are moved into dynamic storage and then +they become shared: the semantics of the mutation of those variables +is then similar to the situation in interpreted code. Therefore, the +above described non-sharing capture behavior of compiled code is not required +to hold. -During an open traversal, new keys can be inserted into a hash table or deleted -from it while a a traversal is in progress. Insertion of a new key during -traversal will not cause any existing key to be visited twice or to be skipped; -however, it is not specified whether the new key will be traversed. Similarly, -if a key is deleted during traversal, and that key has not yet been visited, it -is not specified whether it will be visited during the remainder of the -traversal. These remarks apply not only to deletion via -.code remhash -or the -.code del -operator, but also to wholesale deletion of all keys via -.codn clearhash . +In continuation-based code which relies on mutation of lexical variables +created with +.code let +or +.codn let* , +the macros +.code hlet +and +.code hlet* +can be used instead. These macros create variable bindings whose storage is +always outside of the stack, and therefore the variables will exhibit +consistent -The garbage collection of hash tables supports weak keys and weak values. -If a hash table has weak keys, this means that from the point of view -of garbage collection, that table holds only weak references to the keys -stored in it. Similarly, if a hash table has weak values, it means that it -holds a weak reference to each value stored. A weak reference is one -which does not prevent the reclamation of an object by the garbage -collector. That is to say, when the garbage collector discovers that the only -references to some object are weak references, then that object is considered -garbage, just as if it had no references to it. The object is reclaimed, and -the weak references "lapse" in some way, which depends on what kind they are. -Hash table weak references lapse by entry removal. When an object used -as a key in in one or more weak-key hash tables becomes unreachable, those -hash entries disappear. Similarly, when an object appearing as a value in -one or more hash table entries in weak-value hash tables becomes unreachable, -those entries disappear. When a hash table has both weak keys and weak values, -then its entries are removed when either keys or values become unreachable. -In other words, both the key and value must be reachable in order to -retain the entry. - -An open traversal of a hash table is performed by the -.code maphash -function and the -.code dohash -operator. The traversal is open because code supplied by the program -is evaluated for each entry. - -The functions -.codn hash-keys , -.codn hash-values , -.codn hash-pairs , -and -.code hash-alist -also perform an open traversal, because they return -lazy lists. The traversal isn't complete until the returned lazy list -is fully instantiated. In the meanwhile, the -\*(TX program can mutate the hash table from which the lazy list -is being generated. - -Certain hash operations expose access to the internal key-value association -entries of a hash table, which are represented as ordinary -.code cons -cells. Modifying the -.code car -field of such a cell potentially violates the integrity of the hash table; -the behavior of subsequent lookup and insertion operations becomes unspecified. - -Similarly, if an object is used as a key in an -.codn equal -based -hash table, and that object is mutated in such a way that its equality to -other objects under the -.code equal -function is affected or its hash value under -.code hash-equal -is altered, the behavior of subsequent lookup and insertion operations on the -becomes unspecified. +If the affected variables are other kinds of bindings such as +function parameters or variables created with specialized binding +constructs such as +.codn with-stream , +additional coding changes may be required to get interpreted code +working under compilation. -.coNP Functions @ make-hash and @ hash +.coNP Function @ sys:capture-cont .synb -.mets (make-hash < weak-keys < weak-vals -.mets \ \ \ \ \ \ \ \ \ \ < equal-based <> [ hash-seed ]) -.mets (hash {:weak-keys | :weak-vals | -.mets \ \ \ \ \ \ :eql-based | :equal-based | -.mets \ \ \ \ \ \ :userdata << obj }*) +.mets (sys:capture-cont < name < receive-fun <> [ context-form ]) .syne .desc -These functions construct a new hash table. +The +.code sys:capture-cont +function captures a continuation, and also serves as the resume point +for the resulting continuation. Which of these two situations is the +case (capture or resumption) is distinguished by the use of the +.meta receive-fun +argument, which must be a function capable of being called with one +argument. -.code make-hash -takes three mandatory Boolean arguments. The -.meta weak-keys -argument specifies whether the hash table shall have weak keys. The -.meta weak-vals -argument specifies whether it shall have weak values, and -.meta equal-based -specifies whether it is -.codn equal -based. -The hash function defaults -all three of these properties to false, and allows them to be overridden to -true by the presence of keyword arguments. +A block named +.meta name +must be visible; the continuation is delimited by the closest +enclosing block of this name. The optional -.meta hash-seed -parameter must be an integer, if specified. Its value perturbs the hashing -function of the hash table, which affects -.code :equal-based -hash tables, when character strings and buffers are used as keys. -If -.meta hash-seed -is omitted, then the value of the -.code *hash-seed* -variable is used as the seed. - -It is an error to attempt to construct an -.codn equal -based -hash table which has weak keys. +.meta context-form +argument should be a compound form. If +.code sys:capture-cont +reports an error, it reports it against this form, +and uses the form's operator symbol as the name of the function which +encountered the error. If the argument is omitted, +.code sys:capture-cont +uses its own name. The -.code hash -function provides an alternative interface. It accepts optional -keyword arguments. The supported keyword symbols are: -.codn :weak-keys , -.codn :weak-vals , -.codn :equal-based , -.code :eql-based -and -.code :userdata -which can be specified in any order to turn on the corresponding properties in -the newly constructed hash table. +.code sys:capture-cont +function captures a continuation, represented as a function. +It immediately calls +.metn receive-fun , +passing it it the continuation function as an argument. +If +.meta receive-fun +returns normally, then +.code sys:capture-cont +returns whatever value +.meta receive-fun +returns. -Only one of -.code :equal-based -and -.code :eql-based -may be specified. If specified, then the hash table uses -.code equal -or -.code eql -equality, respectively, for considering two keys to be the same key. -If neither is specified, the -.code hash -function produces an -.code :equal-based -hash table by default. +Resuming a continuation is done by invoking the continuation function. +When this happens, the entire continuation context is restored by re-creating +its captured evaluation frames on top of the current stack. Inside the +continuation, the +.code sys:capture-cont +function call which captured the continuation now appears to return, +and yields a value. That value is precisely the value which was just +passed to the continuation function moments ago. -If -.code :weak-keys -is specified, then it automatically implies -.code :eql-based -and, therefore, -.code :equal-based -may not be specified. +The resumed continuation can terminate in one of three ways. Firstly, it can +simply keep executing until it discards all of its evaluation frames below the +delimiting block, and then allows that block to terminate naturally by +evaluating the last form contained in the block. Secondly, can use +.code return-from +against its delimiting block to explicitly abandon all evaluations in between +and terminate that block. Or it may perform +a non-local control transfer past the delimited block somewhere into the +evaluation frames of the caller. In the first two cases, the termination +of the block turns into an ordinary return from the continuation function, and +the result value of the terminated block becomes the return value of that +function call. In the last case, the call of the continuation function is +abandoned and unwinding continues through the caller. -If -.code :userdata -is present, it must be followed by an argument value; that value -specifies the user data for the hash table, which can be retrieved using the -.code hash-userdata -function. +If the symbol +.code sys:cont-poison +is passed to the continuation function, the continuation will be +resumed in a different manner: its context will be restored as in the +ordinary resume case, whereupon it will be immediately abandoned by +a nonlocal exit, causing unwinding to take place across all of the +continuation's evaluation frames. The function then returns +.codn nil . -Note: there doesn't exist a keyword for specifying the seed. -This omission is deliberate. These hash construction keywords may appear in the -hash literal -.code #H -syntax. A seed keyword would allow literals to specify their own seed, which -would allow malicious hash literals to be crafted that perpetrate a hash -collision attack against the parser. +If the symbol +.code sys:cont-free +is passed to the continuation function, the continuation isn't +be resumed at all; rather, the buffer which holds the saved context +of the continuation is released. Thereafter, an attempt to resume +the continuation results in an error exception being thrown. +After releasing the buffer, the function returns +.codn nil . -.coNP Functions @, hash-construct @ hash-from-pairs and @ hash-from-alist -.synb -.mets (hash-construct < hash-args << key-val-pairs ) -.mets (hash-from-pairs < key-val-pairs << hash-arg *) -.mets (hash-from-alist < alist << hash-arg *) -.syne -.desc -The -.code hash-construct -function constructs a populated hash in one step. The -.meta hash-args -argument specifies a list suitable as an argument list in a call to the hash -function. The -.meta key-val-pairs -is a sequence of pairs, which are two-element -lists representing key-value pairs. +.TP* Notes: -A hash is constructed as if by a call to -.mono -.meti (apply hash << hash-args ), -.onom -then populated -with the specified pairs, and returned. +The continuation function may be used any time after it is produced, and may be +called more than once, regardless of whether the originally captured dynamic +context is still executing. The continuation object may be communicated into +the resumed continuation, which can then use it to call itself, resulting +in multiple nested resumptions of the same continuation. A delimited +continuation is effectively a first class function. -The -.code hash-from-pairs -function is an alternative interface to the same semantics. The -.meta key-val-pairs -argument is first, and the -.meta hash-args -are passed as trailing variadic arguments, rather than a single list argument. +The underlying continuation object produced by +.code sys:capture-cont +stores a copy of the captured dynamic context. Whenever the continuation +function is invoked, a copy of the captured is reinstated as if it were a new +context. Thus each apparent return from the +.code sys:capture-cont +inside a resumed continuation is not actually made in the original context, but +in a copy of that context. That context can be resumed multiple times +sequentially or recursively. -The -.code hash-from-alist -function is similar to -.codn hash-from-pairs , -except that the -.meta alist -argument specifies they keys and values as an association list. -The elements of the list are -.code cons -cells, each of whose -.code car -is a key, and whose -.code cdr -is the value. +Just like lexical closures, continuations do not copy lexical environments; +they capture lexical environments by reference. If a continuation modifies +the values of captured lexical variables, those modifications are visible to +other resumptions of the same continuation, to other continuations which +capture the same environment, to lexical closures which capture the same +environment and to the original context which created that environment, if it +is still active. -.coNP Function @ hash-list -.synb -.mets (hash-list < key-list << hash-arg *) -.syne -.desc -The -.code hash-list -function constructs a hash as if by a call to -.mono -.meti (apply hash << hash-args ), -.onom -where -.meta hash-args -is a list of the individual -.meta hash-arg -variadic arguments. +Unlike lexical closures, continuations do capture the local bindings +of special variables. That is to say, if +.code *var* +is a special variable, then a lexical closure created inside a +.code "(let ((*var* 42)) ...)" +form will not capture the local re-binding of +.code *var* +which holds 42. When the closure is invoked and accesses +.codn *var* , +it accesses whatever value of +.code *var* +is dynamically current, as dictated by the environment which calls the +closure, rather than the capturing environment. -The hash is then populated with keys taken from -.meta key-list -and returned. +With continuations, the behavior is different. If a continuation +is captured inside a +.code "(let ((*var* 42)) ...)" +form then it does capture the local binding. This is regardless whether +the delimited prompt of the capture is enclosed in this form, or +outside of the form. +The special variable has a binding in a dynamic environment. There is always a +reference to a current dynamic environment associated with every evaluation +context, and a continuation captures that reference. Because it is a +reference, it means that the binding is shared. That is to say, all +invocations of all continuations which capture the same dynamic environment in +which that +.code "(let ((*var* 42)) ...)" +binding was made share the same binding; if +.code *var* +is modified by assignment, the modification is visible to all those views. -The value associated with each key is that key itself. +Inside a resumed continuation, a form which binds a special variable such as +.code "(let ((*var* 42)) ...)" +may terminate. As expected, this causes the binding to be removed, +revealing either another local binding of +.code *var* +or the global binding. However, this unbinding only affects only that +that executing continuation; it has no effect inside other instances of the +same continuation or other continuations which capture the same variable. +Unbinding isn't a mutation of the dynamic environment, but may be understood +as merely the restoration of an earlier dynamic environment reference. -.coNP Function @ hash-update -.synb -.mets (hash-update < hash << function ) -.syne -.desc -The -.code hash-update -function replaces each value in -.metn hash , -with the value of -.meta function -applied to that value. +.TP* "Example:" -The return value is -.metn hash . +The following example shows an implementation of the +.code suspend +operator. -.coNP Function @ hash-update-1 +.verb + (defmacro suspend (:form form name var . body) + ^(sys:capture-cont ',name (lambda (,var) + (sys:abscond-from ,name ,*body)) + ',form)) +.brev + +.coNP Operator @ sys:abscond-from .synb -.mets (hash-update-1 < hash < key < function <> [ init ]) +.mets (sys:abscond-from < name <> [ value ]) .syne .desc The -.code hash-update-1 -function operates on a single entry in the hash table. - -If -.meta key -exists in the hash table, then its corresponding value is passed -into -.metn function , -and the return value of -.meta function -is then installed -in place of the key's value. The value is then returned. - -If -.meta key -does not exist in the hash table, and no -.meta init -argument is given, -then -.code hash-update-1 -does nothing and returns +.code sys:abscond-from +operator closely resembles +.code return-from +and performs the same function: it causes an enclosing block +.meta name +to terminate with +.meta value +which defaults to .codn nil . -If -.meta key -does not exist in the hash table, and an -.meta init -argument is given, -then -.meta function -is applied to -.metn init , -and then -.meta key -is inserted into -.meta hash -with the value returned by -.meta function -as the datum. This value -is also returned. +However, unlike +.codn return-from , +.code sys:abscond-from +does not perform any unwinding. -.coNP Function @ group-by +This operator should never be used for any purpose other than +implementing primitives for the use of delimited continuations. +It is used by the +.code yield-from +and +.code yield +operators to escape out of a block in which a continuation has +been captured. Neglecting to unwind is valid due to the expectation +that control will return into a restarted copy of that context. + +.coNP Function @ sys:abscond* .synb -.mets (group-by < func < sequence << option *) +.mets (sys:abscond* < name <> [ value ]) .syne .desc The -.code group-by -function produces a hash table from -.metn sequence , -which is a -list or vector. Entries of the hash table are not elements of -.metn sequence , -but lists of elements of -.metn sequence . -The function -.meta func -is applied to -each element of -.meta sequence -to compute a key. That key is used to determine -which list the item is added to in the hash table. - -The trailing arguments -.mono -.meti << option * -.onom -if any, consist of the same keywords -that are understood by the hash function, and determine the properties -of the hash. +.code sys:abscond* +function is similar to the the +.code sys:abscond-from +operator, except that +.code name +is an ordinary function parameter, and so when +.code return* +is used, an argument expression must be specified which evaluates +to a symbol. Thus +.code sys:abscond* +allows the target block of a return to be dynamically computed. -.TP* Example: -Group the integers from 0 to 10 into three buckets keyed on 0, 1 and 2 -according to the modulo 3 congruence: +The following equivalence holds between the operator and function: .verb - (group-by (op mod @1 3) (range 0 10))) - - -> #H(() (0 (0 3 6 9)) (1 (1 4 7 10)) (2 (2 5 8))) + (sys:abscond-from a b) <--> (sys:abscond* 'a b) .brev -.coNP Function @ group-reduce +Expressions used as +.meta name +arguments to +.code abscond* +which do not simply quote a symbol have no equivalent in +.codn abscond-from . + +.coNP Macros @ obtain and @ yield-from .synb -.mets (group-reduce < hash < classify-fun < binary-fun < seq -.mets \ \ >> [ init-value <> [ filter-fun ]]) +.mets (obtain << forms *) +.mets (yield-from < name <> [ form ]) .syne .desc The -.code group-reduce -updates hash table -.meta hash -by grouping and reducing sequence -.metn seq . +.code obtain +and +.code yield-from +macros closely inter-operate. -The function regards the hash table as being populated with -keys denoting accumulator values. Missing accumulators which -need to be created in the hash table are initialized with -.meta init-value -which defaults to -.codn nil . +The +.code obtain +macro treats zero or more +.metn form -s +as a suspendable execution context called the +.IR "obtain block" . +It is expected that +.metn form -s +establish a block named +.meta name +and return its result value to +.codn obtain . -The function iterates over -.meta seq -and treats each element according to the following steps: -.RS -.IP 1. -Each element is mapped to a hash key through -.metn classify-fun . -.IP 2. -The value associated with the hash key (the accumulator for that -key) is retrieved. If it doesn't exist, -.meta init-value -is used. -.IP 3. -The function -.meta binary-fun -is invoked with two arguments: the accumulator from step 2, and the -original element from -.metn seq . -.IP 4. -The resulting value from step 3 is stored back into the hash table under the -key from step 2. -.RE +Without evaluating any of the forms in the obtain block, +.code obtain +returns a function, which takes one optional argument. +This argument, called the +.IR "resume value" , +defaults to +.code nil +if it is omitted. -.IP -After the above processing, one more step is performed if the -.meta filter-fun -argument is present. In this case, the hash table is destructively mapped through -.meta filter-fun -before being returned. That is to say, every value in the hash table is -projected through -.meta filter-fun -and stored back in the table under the same key, as if by an invocation the -.mono -.meti (hash-update < hash << filter-fun ) -.onom -expression. +The function represents the suspended execution context. -.IP -If -.code group-reduce -is invoked on an empty hash table, its net result closely resembles a -.code group-by -operation followed by separately performing a -.code reduce-left -on each value in the hash. +The context is resumed whenever the function is called, and executes +until the next +.code yield-from +statement which references the block named +.metn name . +The function's reply argument is noted. -.TP* Examples: +If the +.code yield-from +specifies a +.meta form +argument, then the execution context suspends, and the resume function +terminates and returns the value of that form. When the function is called +again to resume the context, the +.code yield-from +returns the previously noted resume value (and the new resume +value just passed is noted in its place). -Frequency histogram: +If the +.code yield-from +specifies no +.meta form +argument, then it briefly suspends the execution context only +to retrieve the resume value, without producing an item. Since +no item is produced, the resume function does not return. +The execution context implicitly resumes. -.verb - [group-reduce (hash) identity (do inc @1) - "fourscoreandsevenyearsago" 0] - --> #H(() (#\ea 3) (#\ec 1) (#\ed 1) (#\ee 4) (#\ef 1) - (#\eg 1) (#\en 2) (#\eo 3) (#\er 3) (#\es 3) - (#\eu 1) (#\ev 1) (#\ey 1)) -.brev +When execution reaches the last form in the obtain block, the +resume value is discarded. The execution context terminates, and +the most recent call to the resume function returns the value of +that last form. -Separate the integers 1-10 into even and odd, and sum these groups: +.TP* Notes: -.verb - [group-reduce (hash) evenp + (range 1 10) 0] - -> #H(() (t 30) (nil 25)) -.brev +The +.code obtain +macro registers a finalizer against the returned resume function. +The finalizer invokes the function, passing it the symbol +.codn sys:cont-poison , +thereby triggering unwinding in the most recently captured +continuation. Thus, abandoned +.code obtain +blocks are subject to unwinding when they become garbage. -.coNP Functions @ make-similar-hash and @ copy-hash -.synb -.mets (make-similar-hash << hash ) -.mets (copy-hash << hash ) -.syne -.desc The -.code make-similar-hash -and copy-hash functions create a new hash object based on -the existing -.meta hash +.code yield-from +macro works by capturing a continuation and performing a nonlocal +exit to the nearest block called +.metn name . +It passes a special yield object to that block. The +.code obtain +macro generates code which knows what to do with this special yield object. -.code make-similar-hash -produces an empty hash table which inherits all of the -attributes of -.metn hash . -It uses the same kind of key equality, the -same configuration of weak keys and values, and has the same user data (see -the -.code set-hash-userdata -function). - -The -.code copy-hash -function is like -.codn make-similar-hash , -except that instead of -producing an empty hash table, it produces one which has all the same elements -as -.metn hash : -it contains the same key and value objects. +.TP* Examples: -.coNP Function @ inhash -.synb -.mets (inhash < hash < key <> [ init ]) -.syne -.desc -The -.code inhash -function searches hash table -.meta hash -for -.metn key . -If -.meta key -is found, then it return the hash table's cons cell which -represents the association between -.meta hash -and -.metn key . -Otherwise, it returns +The following example shows a function which recursively +traverses a +.code cons +cell structure, yielding all the +.cod2 non- nil +atoms it encounters. Finally, it returns the object .codn nil . +The function is invoked on a list, +and the invocation is wrapped in an +.code obtain +block to convert it to a generating function. -If argument -.meta init -is specified, then the function will create -an entry for -.meta key -in -.meta hash -whose value is that of -.metn init . -The cons cell representing that association is returned. +The generating function is then called six times +to retrieve the five atoms from the list, +and the final +.code nil +value. These are collected into a list. -Note: for as long as the -.meta key -continues to exist inside -.metn hash . -modifying the -.code car -field of the returned cons has ramifications for the logical integrity of -the hash; doing so results in unspecified behavior for subsequent -insertion and lookup operations. +This example demonstrates the power of delimited +continuations to suspend and resume a recursive +procedure. -Modifying the -.code cdr -field has the effect of updating the association with a new value. +.verb + (defun yflatten (obj) + (labels ((flatten-rec (obj) + (cond + ((null obj)) + ((atom obj) (yield-from yflatten obj)) + (t (flatten-rec (car obj)) + (flatten-rec (cdr obj)))))) + (flatten-rec obj) + nil)) -.coNP Accessor @ gethash -.synb -.mets (gethash < hash < key <> [ alt ]) -.mets (set (gethash < hash < key <> [ alt ]) << new-value ) -.syne -.desc + (let ((f (obtain (yflatten '(a (b (c . d)) e))))) + (list [f] [f] [f] [f] [f] [f])) + --> (a b c d e nil) +.brev + +The following interactive session log exemplifies two-way communication between +the main code and a suspending function. + +Here, +.code mappend +is invoked on a list of symbols representing fruit and vegetable names. +The objective is to return a list containing only fruits. The -.code gethash -function searches hash table -.meta hash -for key -.metn key . -If the -key is found then the associated value is returned. Otherwise, if -the -.meta alt -argument was specified, it is returned. If the -.meta alt -argument -was not specified, -.code nil -is returned. +.code lambda +function suspends execution and yields a question out of the +.code map +block. It then classifies +the item as a fruit or not according to the reply it receives. The reply +emerges as a the result value of the +.code yield-from +call. -A valid -.code gethash -form serves as a place. It denotes either an existing value in a hash -table or a value that would be created by the evaluation of the form. The -.meta alt -argument is meaningful when -.code gethash -is used as a place, and, if present, is always evaluated whenever the place is -evaluated. -In place update operations, it provides the initial value, which defaults -to -.code nil -if the argument is not specified. For example -.code "(inc (gethash h k d))" -will increment the value stored under key -.code k -in hash table -.code h -by one. If the key does not exist in the hash table, then -the value -.code "(+ 1 d)" -is inserted into the table under that key. -The expression -.code d -is always evaluated, whether or not its value is needed. +.code obtain +macro converts the block to a generating function. The first call to the +function is made with no argument, because the argument would be ignored +anyway. The function returns a question, asking whether the first item +in the list, the potato, is a fruit. +To answer negatively, the user calls the function again, passing in +.codn nil . +The function returns the next question, which is answered in the +same manner. -If a -.code gethash -place is subject to a deletion, but doesn't exist, it is not an error. -The operation does nothing, and -.code nil -is considered the prior value of the place yielded -by the deletion. +When the question for the last item is answered, the function +call yields the final item: the ordinary result of the block, which is the list +of fruit names. -.coNP Function @ sethash +.verb + 1> (obtain + (block map + (mappend (lambda (item) + (if (yield-from map `is @item a fruit?`) + (list item))) + '(potato apple banana lettuce orange carrot)))) + # + 2> (call *1) + "is potato a fruit?" + 3> (call *1 nil) + "is apple a fruit?" + 4> (call *1 t) + "is banana a fruit?" + 5> (call *1 t) + "is lettuce a fruit?" + 6> (call *1 nil) + "is orange a fruit?" + 7> (call *1 t) + "is carrot a fruit?" + 8> (call *1 nil) + (apple banana orange) +.brev + +The following example demonstrates an accumulator. Values passed to the +resume function are added to a counter which is initially zero. +Each call to the function returns the updated value of the accumulator. +Note the use of +.code "(yield-from acc)" +with no arguments to receive the value passed to the first +call to the resume function, without yielding an item. +The very first return value +.code 1 +is produced by the +.code "(yield-from acc sum)" +form, not by +.codn "(yield-from acc)" . +The latter only obtains the initial value +.code 1 +and uses it to establish the seed value of the accumulator. Without causing +the resume function to terminate and return, control passes into the loop, +which yields the first item, causing the resume function call +.code "(call *1 1)" +to return +.codn 1 : + +.verb + 1> (obtain + (block acc + (let ((sum (yield-from acc))) + (while t (inc sum (yield-from acc sum)))))) + # + 2> (call *1 1) + 1 + 3> (call *1 2) + 3 + 4> (call *1 3) + 6 + 5> (call *1 4) + 10 +.brev + +.coNP Macro @ obtain-block .synb -.mets (sethash < hash < key << value ) +.mets (obtain-block < name << forms *) .syne .desc The -.code sethash -function places a value into -.meta hash -table under the given -.metn key . -If a similar key already exists in the hash table, then that key's -value is replaced by -.metn value . -Otherwise, the -.meta key +.code obtain-block +macro combines +.code block and -.meta value -pair is -newly inserted into -.metn hash . - +.code obtain +into a single expression. The -.code sethash -function returns the -.meta value -argument. +.metn form -s +are evaluated in a block named +.codn name . -.coNP Function @ pushhash -.synb -.mets (pushhash < hash < key << element ) -.syne -.desc -The -.code pushhash -function is useful when the values stored in a hash table -are lists. If the given -.meta key -does not already exist in -.metn hash , -then a list of -length one is made which contains -.metn element , -and stored in -.meta hash -table under -.metn key . -If the -.meta key -already exists in the hash table, then the corresponding -value must be a list. The -.meta element -value is added to the front of that list, -and the extended list then becomes the new value under -.metn key . +That is to say, the following equivalence holds: -The return value is Boolean. If true, indicates that the hash table entry was -newly created. If false, it indicates that the push took place on an existing -entry. +.verb + (obtain-block n f ...) <--> (obtain (block n f ...)) +.brev -.coNP Function @ remhash +.coNP Macro @ yield .synb -.mets (remhash < hash << key ) +.mets (yield <> [ form ]) .syne .desc The -.code remhash -function searches -.meta hash -for a key similar to the -.metn key . -If that key is found, then that key and its corresponding value are -removed from the hash table. +.code yield +macro is to +.code yield-from +as +.code return +is to +.codn return-from : +it yields from an anonymous block. -If the key is found and removal takes place, then the associated value -is returned. Otherwise +It is equivalent to calling +.code yield-from +using .code nil -is returned. +as the block name. -.coNP Function @ clearhash -.synb -.mets (clearhash << hash ) -.syne -.desc -The -.code clearhash -function removes all keys-value pairs from -.metn hash , -causing it to be empty. +In other words, the following equivalence holds: -If -.meta hash -is already empty prior to the operation, then -.codn nil , -is returned. +.verb + (yield x) <--> (yield-from nil x) +.brev -Otherwise an integer is returned indicating the number of entries -that were purged from -.metn hash . +.TP* Example: -.coNP Function @ hash-count -.synb -.mets (hash-count << hash ) -.syne -.desc -The -.code hash-count -function returns an integer representing the number of -key-value pairs stored in -.metn hash . +.verb + ;; Yield the integers 0 to 4 from a for loop, taking + ;; advantage of its implicit anonymous block: -.coNP Accessor @ hash-userdata + (defvarl f (obtain (for ((i 0)) ((< i 5)) ((inc i)) + (yield i)))) + + [f] -> 0 + [f] -> 1 + [f] -> 2 + [f] -> 3 + [f] -> 4 + [f] -> nil + [f] -> nil +.brev + +.coNP Macros @ obtain* and @ obtain*-block .synb -.mets (hash-userdata << hash ) -.mets (set (hash-userdata << hash ) << new-value ) +.mets (obtain* << forms *) +.mets (obtain*-block < name << forms *) .syne .desc The -.code hash-userdata -function retrieves the user data object associated with -.metn hash . +.code obtain* +and +.code obtain*-block +macros implement a useful variation of +.code obtain +and +.codn obtain-block . -A hash table can be created with user data using the -.code :userdata -keyword in a hash table literal or in a call to the -.code hash -function, directly, or via other hash-constructing functions which take the -hash construction keywords, such as -.codn group-by . -If a hash table is created without user data, its user -data is initialized to +The +.code obtain* +macro differs from +.code obtain +in exactly one regard: prior to returning the function, it invokes +it one time, with the argument value .codn nil . -Because -.code hash-userdata -is an accessor, a -.code hash-userdata -form can be used as a place. Assigning a value to this place -causes the user data of -.meta hash -to be replaced with that value. +Thus, the following equivalence holds -.coNP Function @ get-hash-userdata -.synb -.mets (get-hash-userdata << hash ) -.syne -.desc -The -.code get-hash-userdata -function is a deprecated synonym for -.codn hash-userdata . +.verb + (obtain* forms ...) <--> (let ((f (obtain forms ...))) + (call f) + f) +.brev -.coNP Function @ set-hash-userdata -.synb -.mets (set-hash-userdata < hash << object ) -.syne -.desc -The -.code set-hash-userdata -replaces, with the -.metn object , -the user data object -associated with -.metn hash . +In other words, the suspended block is immediately resumed, so that it executes +either to completion (in which case its value is discarded), or to its first +.code yield +or +.code yield-from +call (in which case the yielded value is discarded). -.coNP Function @ hashp -.synb -.mets (hashp << object ) -.syne -.desc -The -.code hashp -function returns -.code t -if the -.meta object -is a hash table, -otherwise it returns -.codn nil . +Note: the +.code obtain* +macro is useful in creating suspensions which accept data rather than +produce data. -.coNP Function @ maphash -.synb -.mets (maphash < hash << binary-function ) -.syne -.desc The -.code maphash -function successively invokes -.meta binary-function -for each entry stored in -.metn hash . -Each entry's key and value are passed as arguments -to -.codn binary-function . +.code obtain*-block +macro combines +.code obtain* +and +.code block +in the same manner that +.code obtain-block +combines +.code obtain +and +.codn block . -The function returns -.codn nil . +.TP* Example: -.coNP Function @ hash-revget +.verb + ;; Pass three values into suspended block, + ;; which get accumulated into list. + (let ((f (obtain*-block nil + (list (yield nil) (yield nil) (yield nil))))) + (call f 1) + (call f 2) + (call f 3)) -> (1 2 3) + + ;; Under obtain, extra call is required: + (let ((f (obtain-block nil + (list (yield nil) (yield nil) (yield nil))))) + (call f nil) ;; execute block to first yield + (call f 1) ;; resume first yield with 1 + (call f 2) + (call f 3)) -> (1 2 3) +.brev + +.coNP Macro @ suspend .synb -.mets (hash-revget < hash < value >> [ testfun <> [ keyfun ]]) +.mets (suspend < block-name < var-name << body-form *) .syne .desc The -.code hash-revget -function performs a reverse lookup on -.metn hash . +.code suspend +operator captures a continuation up to the prompt given by the +symbol +.meta block-name +and binds it to the variable name given by +.metn var-name , +which must be a symbol suitable for binding variables with +.codn let . -It searches through the entries stored in -.meta hash -for an entry whose value matches -.metn value . +Each +.meta body-form +is then evaluated in the scope of the variable +.metn var-name . -If such an entry is found, that entry's key is returned. -Otherwise -.code nil -is returned. +When the last +.meta body-form +is evaluated, a non-local exit takes place to the block +named by +.meta block-name +(using the +.code sys:abscond-from +operator, so that unwinding isn't performed). -If multiple matching entries exist, it is not specified which entry's -key is returned. +When the continuation bound to +.meta var-name +is invoked, a copy of the entire block +.meta block-name +is re-started, and in that copy, the +.code suspend +call appears to return normally, yielding the value which had been +passed to the continuation. -The -.meta keyfun -function is applied to each value in -.meta hash -and the resulting value is compared with -.metn value . -The default -.meta keyfun -is the -.code identity -function. +.TP* Example -The comparison is performed using -.metn testfun . +Define John McCarthy's +.code amb +function using +.code block +and +.codn suspend : -The default -.meta testfun -is the -.code eql -function. +.verb + (defmacro amb-scope (. forms) + ^(block amb-scope ,*forms)) -.coNP Functions @ hash-eql and @ hash-equal + (defun amb (. args) + (suspend amb-scope cont + (each ((a args)) + (when (and a (call cont a)) + (return-from amb a))))) +.brev + +Use +.code amb +to bind the of +.code x +and +.code y +which satisfy the predicate +.mono +.meti (eql (* x y) 8) +.onom +non-deterministically: + +.verb + (amb-scope + (let ((x (amb 1 2 3)) + (y (amb 4 5 6))) + (amb (eql (* x y) 8) + (list x y)))) + -> (2 4) +.brev + +.coNP Macros @ hlet and @ hlet* .synb -.mets (hash-eql << object ) -.mets (hash-equal < object <> [ hash-seed ]) +.mets (hlet >> ({ sym | >> ( sym << init-form )}*) << body-form *) +.mets (hlet* >> ({ sym | >> ( sym << init-form )}*) << body-form *) .syne .desc -These functions each compute an integer hash value from the internal -representation of -.metn object , -which satisfies the following properties. -If two objects -.code A -and -.code B -are the same under the -.code eql -function, then -.code "(hash-eql A)" -and -.code "(hash-eql B)" -produce the same integer hash value. Similarly, -if two objects -.code A +The +.code hlet and -.code B -are the same under the -.code equal -function, then -.code "(hash-equal A)" +.code hlet* +macros behave exactly like +.code let and -.code "(hash-equal B)" -each produce the same integer hash value. In all other -circumstances, the hash values of two distinct objects are unrelated, and -may or may not be the same. - -Object of struct type may support custom hashing by way of defining -an equality substitution via an -.code equal -method. See the Equality Substitution section under Structures. +.codn let* , +respectively except that they guarantee that the variable bindings are +allocated in storage which isn't captured by delimited continuations. -The optional -.meta hash-seed -value perturbs the hashing function used by -.code hash-equal -for strings and buffer objects. This seed value must be a non-negative integer -no wider than 32 bits: that is, in the range 0 to 4294967295. -If the value isn't specified, it defaults to zero. -Effectively, each possible value of the seed specifies a different hashing -function. If two objects -.code A -and -.code B -are the same under the -.code equal -function, then -.code "(hash-equal A S)" -and -.code "(hash-equal B S)" -each produce the same integer hash value for any valid seed value -.codn S . +The +.code h +in the names stands for "heap", serving as a mnemonic based on the +implementation concept of these bindings being "heap allocated". -.coNP Functions @, hash_keys @, hash_values @ hash_pairs and @ hash_alist -.synb -.mets (hash-keys << hash ) -.mets (hash-values << hash ) -.mets (hash-pairs << hash ) -.mets (hash-alist << hash ) -.syne -.desc -These functions retrieve the bulk key-value data of hash table -.meta hash -in various ways. -.code hash-keys -retrieves a list of the keys. -.code hash-values -retrieves a list of the values. -.code hash-pairs -retrieves a list of pairs, -which are two-element lists consisting of the key, followed by the value. -Finally, -.code hash-alist -retrieves the key-value pairs as a Lisp association list: -a list of cons cells whose -.code car -fields are keys, and whose -.code cdr -fields are the values. Note that -.code hash-alist -returns the actual entries from the hash table, which are -conses. Modifying the -.code cdr -fields of these conses constitutes modifying the hash values -in the original hash table. Modifying the -.code car -fields interferes with the integrity of the hash table, -resulting in unspecified behavior for subsequent hash insertion -and lookup operations. +.SS* Regular Expression Library -These functions all retrieve the keys and values in the -same order. For example, if the keys are retrieved with -.codn hash-keys , -and the values with -.codn hash-values , -then the corresponding entries from -each list pairwise correspond to the pairs in -.metn hash . +\*(TX provides a "pure" regular expression implementation based on automata +theory, which equates regular expressions, finite automata and sets of strings. +A regular expression determines whether or not a string of input characters +belongs to a set. \*(TX regular expressions do not support features such +as as "anchoring" a match to the start or end of a string, or capture of +parenthesized sub-expression matches into registers. Parenthesis syntax +denotes only grouping, with no additional meaning. -The list returned by each of these functions is lazy, and hence constitutes -an open traversal of the hash table. +The semantics of whether a regular expression is used for a substring +search, prefix match, suffix match, spring splitting and so forth comes from +the functions which use regular expressions to perform these operations. -.coNP Operator @ dohash +.NP* Regular Expressions as Functions .synb -.mets (dohash >> ( key-var < value-var < hash-form <> [ result-form ]) -.mets \ \ << body-form *) +.mets >> [ regex >> [ start <> [ from-end ]] < string ] .syne .desc -The -.code dohash -operator iterates over a hash table. The -.meta hash-form -expression must -evaluate to an object of hash table type. The -.meta key-var -and -.meta value-var -arguments must be symbols suitable for use as variable names. -Bindings are established for these variables over the scope of the -.metn body-form -s -and the optional -.metn result-form . +A regular expression is callable as a function in \*(TL. +When used this way, it requires a string argument. It searches +the string for the leftmost match for itself, and returns +the matching substring, which could be empty. If no match is +found, it returns +.codn nil . -For each element in the hash table, the -.meta key-var +A regex takes one, two, or three arguments. The required +.meta string +is always the rightmost argument. This allows for convenient +partial application of the optional arguments using +macros in the +.code op +family, and macros in which the +.code op +syntax is implicit. + +The optional arguments +.meta start and -.meta value-var -variables are set to the key and value of that entry, respectively, -and each -.metn body-form , -if there are any, is evaluated. +.meta from-end +are treated exactly as their like-named counterparts in the +.code search-regst +function. -When all of the entries of the table are thus processed, the -.meta result-form -is evaluated, and its return value becomes the return value of the dohash form. -If there is no -.metn result-form , -the return value is -.codn nil . +.TP* Example: +Keep those elements from a list of strings which match +the regular expression +.codn #/a.*b/ : -The -.meta result-form -and -.metn body-form -s -are in the scope of an implicit anonymous -block, which means that it is possible to terminate the execution of -dohash early using -.mono -.meti (return << value ) -.onom -or -.codn (return) . +.verb + (keep-if #/a.*b/ '#"abracadabra zebra hat adlib adobe deer") + --> ("abracadabra" "adlib" "adobe") +.brev -.coNP Functions @, hash-uni @, hash-diff @ hash-symdiff and @ hash-isec +.coNP Functions @, search-regex @ range-regex and @ search-regst .synb -.mets (hash-uni < hash1 < hash2 <> [ join-func ]) -.mets (hash-diff < hash1 << hash2 ) -.mets (hash-symdiff < hash1 << hash2 ) -.mets (hash-isec < hash1 < hash2 <> [ join-func ]) +.mets (search-regex < string < regex >> [ start <> [ from-end ]]) +.mets (range-regex < string < regex >> [ start <> [ from-end ]]) +.mets (search-regst < string < regex >> [ start <> [ from-end ]]) .syne .desc -These functions perform basic set operations on hash tables in a nondestructive -way, returning a new hash table without altering the inputs. The arguments -.meta hash1 -and -.meta hash2 -must be compatible hash tables. This means that their keys -must use the same kind of equality. +The +.code search-regex +function searches through +.meta string +starting +at position +.meta start +for a match for +.metn regex . -The resulting hash table inherits attributes from -.metn hash1 , -as if created by the -.code make-similar-hash -function. If -.meta hash1 -has userdata, the resulting hash table -has the same userdata. If -.meta hash1 -has weak keys, the resulting table has weak -keys, and so forth. +If +.meta start +is omitted, the search starts at position 0. If +.meta from-end +is specified and has a +.cod2 non- nil +value, the search +proceeds in reverse, from the position just beyond the last character of +.metn string , +toward +.metn start . + +if +.meta start +exceeds the length of the string, then +.code search-regex +returns +.codn nil . -The -.code hash-uni -function performs a set union. The resulting hash contains all of -the keys from -.meta hash1 -and all of the keys from -.metn hash2 , -and their corresponding -values. If a key occurs both in -.meta hash1 -and -.metn hash2 , -then it occurs only once -in the resulting hash. In this case, if the -.meta join-func -argument is not given, -the value associated with this key is the one from -.metn hash1 . If -.meta join-func -is specified then it is called with two arguments: the respective -data items from -.meta hash1 -and -.metn hash2 . -The return value of this function is used -as the value in the union hash. +.meta start +is negative then it indicates positions from the end of the string, +such that -1 is the last character, -2 the second last and so forth. +If the value is so negative that it refers beyond the start of +the string, then the starting position is deemed to be zero. + +If +.meta start +is equal to the length of +.metn string , +and thus refers to the position one character past its +length, then a match occurs at that position if +.meta regex +admits such a match. The -.code hash-diff -function performs a set difference. First, a copy of -.meta hash1 -is made as if by the -.code copy-hash -function. Then from this copy, all keys which occur -in -.code hash2 -are deleted. - -The -.code hash-symdiff -function performs a symmetric difference. A new hash is returned which -contains all of the keys from -.meta hash1 -that are not in -.meta hash2 -and -.IR "vice versa" : -all of the keys from -.meta hash2 -that are not in -.metn hash1 . -The keys carry their corresponding values from -.meta hash1 -and -.metn hash2 , -respectively. +.code search-regex +function returns +.code nil +if no match is found, otherwise it returns +a cons, whose +.code car +indicates the position of the match, and whose +.code cdr +indicates the length of the match. -The -.code hash-isec -function performs a set intersection. The resulting hash contains -only those keys which occur both in -.meta hash1 -and -.metn hash2 . -If -.meta join-func -is not -specified, the values selected for these common keys are those from -.metn hash1 . If -.meta join-func -is specified, then for each key which occurs in both -.meta hash1 -and -.metn hash2 , -it is called with two arguments: the respective data items. The return -value is then used as the data item in the intersection hash. +.meta regex +is capable of matching empty strings, and no other kind of match +is found within +.metn string , +then search regex reports a zero length match. If +.meta from-end +is false, then this match is reported at +.metn start , +otherwise it is reported at the position one character beyond +the end of the string. -.coNP Functions @ hash-subset and @ hash-proper-subset -.synb -.mets (hash-subset < hash1 << hash2 ) -.mets (hash-proper-subset < hash1 << hash2 ) -.syne -.desc The -.code hash-subset -function returns -.code t -if the keys in -.meta hash1 -are a subset of the keys in -.metn hash2 . +.code range-regex +function is similar to +.codn search-regex , +except that when +a match is found, it returns a position range, rather than a position +and length. A range object is returned whose +.code from +field indicates the position +of the match, and whose +.code to +indicates the position one element past the +last character of the match. If the match is empty, the two integers +are equal. -The -.code hash-proper-subset -function returns -.code t -if the keys in -.meta hash1 -are a proper subset of the keys in -.metn hash2 . -This means that -.meta hash2 -has all the keys which are in -.meta hash1 -and at least one which isn't. +Also see the +.code rr +function, which provides an alternative argument syntax for +the semantics of +.codn range-regex . -Note: the return value may not be mathematically meaningful if -.meta hash1 -and -.meta hash2 -use different equality. In any case, the actual behavior -may be understood as follows. The implementation of -.code hash-subset -tests whether each of the keys in -.meta hash1 -occurs in -.meta hash2 -using their respective equalities. -The implementation of -.code hash-proper-subset -applies -.code hash-subset -first, as above. If that is true, and the two hashes have the same number of -elements, the result is falsified. +The +.code search-regst +differs from +.code search-regex +in the representation of the return value in the matching case. +Rather than returning the position and length of the match, +it returns the matching substring of +.metn string . -.coNP Functions @, hash-begin @ hash-next and @ hash-peek +.coNP Functions @ match-regex and @ match-regst .synb -.mets (hash-begin << hash ) -.mets (hash-next << hash-iter ) -.mets (hash-peek << hash-iter ) +.mets (match-regex < string < regex <> [ position ]) +.mets (match-regst < string < regex <> [ position ]) .syne .desc The -.code hash-begin -function returns a an iterator object capable of retrieving the -entries in stored in -.meta hash -one by one. +.code match-regex +function tests whether +.meta regex +matches at +.meta position +in +.metn string . -The -.code hash-next -function's -.meta hash-iter -argument is a hash iterator returned by -.codn hash-begin . -If unvisited entries remain in -.metn hash , -then -.code hash-next -returns the next one as a cons cell whose -.code car -holds the key and whose -.code cdr -holds the value. That entry is then considered visited by the iterator. -If no more entries remain to be visited, -.code hash-next -returns -.codn nil . +If +.meta position +is not specified, it is taken to be zero. Negative values +of +.meta position +index from the right end of the string such that -1 +refers to the last character. Excessively negative +values which index before the first character cause +.code nil +to be returned. + +If the regex matches, then the length of the match is returned. +If it does not match, then +.code nil +is returned. The -.code hash-peek -function returns the same value that a subsequent call to -.code hash-next -will return for the same -.metn hash-iter , -without changing the state of -.metn hash-iter . -That is to say, if a cell representing a hash entry is returned, that entry -remains unvisited by the iterator. +.code match-regst +differs from +.code match-regex +in the representation of the return value in the matching case. +Rather than returning the length of the match, it returns +matching substring of +.metn string . -.coNP Macro @ with-hash-iter +.coNP Functions @ match-regex-right and @ match-regst-right .synb -.mets (with-hash-iter >> ( isym < hash-form >> [ ksym <> [ vsym ]]) -.mets \ \ << body-form *) +.mets (match-regex-right < string < regex <> [ end-position ]) +.mets (match-regst-right < string < regex <> [ end-position ]) .syne .desc The -.code with-hash-iter -macro evaluates -.metn body-form -s -in an environment in which a lexically scoped function is visible. +.code match-regex-right +function tests whether some substring of +.meta string +which terminates at the character position just before +.meta end-position +matches +.metn regex . -The function is named by -.meta isym -which must be a symbol suitable for naming functions with -.codn flet . +If +.meta end-position +is not specified, it defaults to the length of the string, and the function +performs a right-anchored regex match. The -.meta hash-form -argument must be a form which evaluates to a hash table object. - -Invocations of the function retrieve successive entries of the hash table -as cons cell pairs of keys and values. The function returns +.meta end-position +argument can be a negative integer, in which case it denotes +positions from the end of the string, such that -1 refers +to the last character. If the value is excessively negative such +that the position immediately before it is before the start +of the string, then .code nil -to indicate no more entries remain. - -If either of the -.meta ksym -or -.meta vsym -arguments are present, they must be symbols suitable as variable names. They -are bound as variables visible to -.metn body-form -s, -initialized to the value -.codn nil . +is returned. If -.meta ksym -is specified, then whenever the function -.meta isym -macro is invoked and retrieves a hash table entry, the -.meta ksym -variable is set to the key. If the function returns +.meta end-position +is a positive value beyond the length of +.metn string , +then, likewise, .code nil -then the value of -.meta ksym -is set to -.codn nil . +is returned. -Similarly, if -.meta vsym -is specified, then the function stores the retrieved -hash value in that variable, or else sets the variable -to -.code nil -if there is no next value. +If a match is found, then the length of the match is returned. + +A more precise way of articulating the role of +.meta end-position +is that for the purposes of matching, +.code string +is considered to terminate just before +.metn end-position : +in other words, that +.meta end-position +is the length of the string. The match is then anchored to the +end of this effective string. -.coNP Special variable @ *hash-seed* -.desc The -.code *hash-seed* -special variable is initialized with a value of zero. Whenever a new -hash table is explicitly or implicitly created, it takes its seed from -the value of the -.code *hash-seed* -variable in the current dynamic environment. +.code match-regst-right +differs from +.code match-regst-right +in the representation of the return value in the matching case. +Rather than returning the length of the match, it returns +the matching substring of +.metn string . -The only situation in which -.code *hash-seed* -is not used when creating a new hash table is when -.code make-hash -is called with an argument given for the optional -.meta hash-seed -argument. +.TP* Examples: -Only -.codn equal -based -hash tables make use of their seed, and only for keys which are strings and -buffers. The purpose of the seed is to scramble the hashing function, to make -a hash table resistant to a type of denial-of-service attack, whereby a -malicious input causes a hash table to be populated with a large number of keys -which all map to the same hash table chain, causing the performance to severely -degrade. +.verb + ;; Return matching portion rather than length thereof. -The value of -.code *hash-seed* -must be a non-negative integer, no wider than 32 bits. + (defun match-regex-right-substring (str reg : end-pos) + (set end-pos (or end-pos (length str))) + (let ((len (match-regex-right str reg end-pos))) + (if len + [str (- end-pos len)..end-pos] + nil))) -.coNP Function @ gen-hash-seed + (match-regex-right-substring "abc" #/c/) -> "" + + (match-regex-right-substring "acc" #/c*/) -> "cc" + + ;; Regex matches starting at multiple positions, but all + ;; the matches extend past the limit. + (match-regex-right-substring "acc" #/c*/ 2) -> nil + + ;; If the above behavior is not wanted, then + ;; we can extract the string up to the limiting + ;; position and do the match on that. + (match-regex-right-substring ["acc" 0..2] #/c*/) -> "c" + + ;; Equivalent of above call + (match-regex-right-substring "ac" #/c*/) -> "c" +.brev + +.coNP Function @ regex-prefix-match .synb -.mets (gen-hash-seed) +.mets (regex-prefix-match < regex < string <> [ position ]) .syne .desc The -.code gen-hash-seed -function returns an integer value suitable for the -.code *hash-seed* -variable, or as the -.code hash-seed -argument of the -.code make-hash +.code regex-prefix-match +determines whether the input string might +might be the prefix of a string which matches regular expression +.metn regex . + +The result is true if the input string matches +.meta regex +exactly. However, it is also true in situations in which +the input string doesn't match +.metn regex , +yet can be extended with one or more additional characters beyond the end such +that the extended string +.B does +match. + +The +.meta string +argument must be a character string. The function takes the input string to be +the suffix of +.meta string +which starts at the character position indicated by the +.meta position +argument. If that argument is omitted, then +.meta string +is taken as the input in its entirety. Negative values index backwards from +the end of +.meta string +according to the usual conventions elsewhere in the library. + +Note: this function is not to be confused for the semantics +of a regex matching a prefix of a string: that capability is +provided by the functions +.codn match-regex , +.codn m^ , +.codn r^ , +.code f^ and -.code hash-equal -functions. +.codn fr^ . -The value is derived from the host environment, from information such -as the process ID and time of day. +.TP* Examples: -.SS* Partial Evaluation and Combinators -.coNP Macros @ op and @ do +.verb + ;; The empty string is not a viable prefix match for + ;; a regex that matches no strings at all: + (regex-prefix-match #/~.*/ "") -> nil + (regex-prefix-match #/[]/ "") -> nil + + ;; The empty string is a viable prefix of any regex + ;; which matches at least one string: + (regex-prefix-match #// "") -> t + (regex-prefix-match #/abc/ "") -> t + + ;; This string doesn't match the regex because + ;; it doesn't end in b, but is a viable prefix: + (regex-prefix-match #/a*b/ "aa") -> t + + (regex-prefix-match #/a*b/ "ab") -> t + + (regex-prefix-match #/a*b/ "ac") -> nil + + (regex-prefix-match #/a*b/ "abc") -> nil +.brev + +.coNP Function @ regsub .synb -.mets (op << form +) -.mets (do << form +) +.mets (regsub >> { regex | << function } < replacement << string ) .syne .desc The -.code op -and -.code do -macro operators are similar. +.code regsub +function operates in two modes, depending on whether +the second argument is a regular expression, +or function. -Like the lambda operator, the -.code op -operator creates an anonymous function based on its syntax. -The difference is that the arguments of the function are implicit, or -optionally specified within the function body, rather than as a formal -parameter list before the body. +If the second argument is a regular expression it searches +.meta string +for multiple occurrences of non-overlapping matches for that +.metn regex . +A new string is constructed +similar to +.meta string +but in which each matching region is replaced +with using +.meta replacement +as follows. -Also, the -.meta form -arguments of -.code op -are implicitly turned into a DWIM expression, -which means that argument evaluation follows Lisp-1 rules. (See the -.code dwim -operator). +The +.meta replacement +object may be a character or a string, in which +case it is simply taken to be the replacement for each match +of the regular expression. The -.code do -operator is like the -.code op -operator with the following difference: -the -.meta form -arguments of -.code do -are not implicitly treated as DWIM expressions, -but as ordinary expressions. In particular, this means that operator -syntax is permitted. Note that the syntax -.code "(op @1)" -makes sense, since -the argument can be a function, which will be invoked, but -.code "(do @1)" -doesn't -make sense because it will produce a Lisp-2 form like -.code "(#:arg1 ...)" -referring -to nonexistent function -.codn #:arg1 . -Because it accepts operators, -.code do -can be used with imperative constructs -which are not functions, like set: like set: for instance -.code "(do set x)" -produces -an anonymous function which, if called with one argument, stores that argument -into -.codn x . +.meta replacement +object may be a function of one argument, in +which case for every match which is found, this function is invoked, +with the matching piece of text as an argument. The function's +return value is then taken to be the replacement text. -The argument forms are arbitrary expressions, within which a special -convention is permitted: -.RS -.meIP >> @ num -A number preceded by a -.code @ -is a metanumber. This is a special syntax -which denotes an argument. For instance -.code @2 -means that the second argument of -the anonymous function is to be substituted in place of the -.codn @2 . -.code op -generates a function which has a number of required arguments equal to the -highest value of -.meta num -appearing in a -.mono -.meti >> @ num -.onom -construct in the body. For instance -.code "(op car @3)" -generates a three-argument function (which passes its third -argument to -.codn car , -returning the result, and ignores its first two arguments). -There is no way to use -.code op -to generate functions which have optional arguments. -.meIP < @rest -If the meta-symbol -.meta @rest -appears in the -.code op -syntax, it explicitly denotes the list of trailing arguments, -allowing them to be placed anywhere in the expression. -.RE - -.IP -Functions generated by -.code op -are always variadic; they always take additional arguments after -any required ones, whether or not the -.meta @rest -syntax is used. - -If the body does not contain -any -.meta @num -or -.meta @rest -syntax, then -.code @rest -is implicitly inserted. What this means is that, for example, since -the form -.code "(op foo)" -does not contain any numeric positional arguments like -.codn @1 , -and does not contain -.codn @rest , -it is actually a shorthand for -.codn "(op foo . @rest)" : -a function which applies all of its arguments to -.codn foo . -If the body does contain at least one -.meta @num -or -.metn @rest , -then -.meta @rest -isn't implicitly inserted. The notation -.code "(op foo @1)" -denotes a function which takes any number of arguments, and ignores -all but the first one, which is passed to -.codn foo . +If the second argument is a function, then it is called, with +.meta string +as its argument. The return value must be either a range +object (see the +.code rcons +function) which indicates the extent of +.meta string +to be replaced, or else +.code nil +which indicates that no replacement is to take place. -The actions of -.code op -be understood by these examples, which show -how -.code op -is rewritten to lambda. However, note that the real translator -uses generated symbols for the arguments, which are not equal to any -symbols in the program. +.TP* Examples: .verb - (op) -> invalid - - (op +) -> (lambda rest [+ . rest]) - - (op + foo) -> (lambda rest [+ foo . rest]) - - (op @1 @2) -> (lambda (arg1 arg2 . rest) [arg1 arg2]) - - (op @1 . @rest) -> (lambda (arg1 . rest) [arg1 . @rest]) - - (op @1 @rest) -> (lambda (arg1 . rest) [arg1 @rest]) - - (op @1 @2) -> (lambda (arg1 arg2 . rest) [arg1 arg2]) - - (op foo @1 (@2) (bar @3)) -> (lambda (arg1 arg2 arg3 . rest) - [foo arg1 (arg2) (bar arg3)]) - - (op foo @rest @1) -> (lambda (arg1 . rest) [foo rest arg1]) - - (do + foo) -> (lambda rest (+ foo . rest)) - - (do @1 @2) -> (lambda (arg1 arg2 . rest) (arg1 arg2)) + ;; match every lower case e or o, and replace by filtering + ;; through the upcase-str function: - (do foo @rest @1) -> (lambda (arg1 . rest) (foo rest arg1)) -.brev + [regsub #/[eo]/ upcase-str "Hello world!"] -> "HEllO wOrld!" -Note that if argument -.meta @n -appears, it is not necessary -for arguments -.meta @1 -through -.meta @n-1 -to appear. The function will have -.code n -arguments: + ;; Replace Hello with Goodbye: + (regsub #/Hello/ "Goodbye" "Hello world!") -> "Goodbye world!" -.verb - (op @3) -> (lambda (arg1 arg2 arg3 . rest) [arg3]) + ;; Left-anchored replacement with r^ function: + (regsub (fr^ #/H/) "J" "Hello, hello!") -> "Jello, hello!" .brev +.coNP Function @ regexp +.synb +.mets (regexp << obj ) +.syne +.desc The -.code op -and -.code do -operators can be nested, in any combination. This raises the -question: if a metanumber like -.code @1 -or -.code @rest -occurs in an -.code op -that is nested -within an -.codn op , -what is the meaning? - -A metanumber always belongs with the inner-most op or do operator. So for -instance -.code "(op (op @1))" -means that an -.code "(op @1)" -expression is nested -within an -.code op -expression which itself contains no meta-syntax. -The -.code @1 -belongs with the inner op. - -There is a way for an inner -.code op -to refer to an outer op metanumber argument. This is -expressed by adding an extra -.code @ -prefix for every level of escape. For example in -.code "(op (op @@1))" -the -.code @@1 -belongs to the outer -.codn op : -it is the same as -.code @1 -appearing in the outer -.codn op . -That is to say, -in the expression -.codn "(op @1 (op @@1))" , -the -.code @1 -and -.code @@1 -are the same thing: -both are parameter 1 of the lambda function generated by the outer -.codn op . -By contrast, in the expression -.code "(op @1 (op @1))" -there are two different parameters: -the first -.code @1 -is argument of the outer function, and the second -.code @1 -is the first argument of the inner function. Of course, if there -are three levels of nesting, then three -.code @ -meta-prefixes are needed to insert -a parameter from the outermost -.code op -into the innermost -.codn op . - -Note that meta-numbers and meta-symbols belonging to an -.code op -can be used in the dot position of a function call, such as: - -.verb - [(op list 1 . @1) 2] -> (1 . 2) -.brev - -This is a consequence of the special transformations described -in the paragraph -.B "Dot Position in Function Calls" -in the subsection -.B "Additional Syntax" -of the -.BR "TXR Lisp" -section. +.code regexp +function returns +.code t +if +.meta obj +is a compiled regular expression +object. For any other object type, it returns +.codn nil . +.coNP Function @ regex-compile +.synb +.mets (regex-compile < form-or-string <> [ error-stream ]) +.syne +.desc The -.code op -syntax works in conjunction with quasiliterals which are nested within it. -The metanumber notation as well as -.code @rest -are recognized without requiring an additional -.code @ -escape, which is effectively optional: +.code regex-compile +function takes the source code of a regular expression, +expressed as a Lisp data structure representing an abstract syntax tree, or +else a regular expression specified as a character string, and compiles it to a +regular expression object. -.verb - (apply (op list `@1-@rest`) '(1 2 3)) -> "1-2 3" +If +.meta form-or-string +is a character string, it is parsed to an +abstract syntax tree first, if by the +.code regex-parse +function. +If the parse is successful (the result is not +.codn nil ) +then +the resulting tree structure is compiled by a recursive call to +.codn regex-compile . - (apply (op list `@@1-@@rest`) '(1 2 3)) -> "1-2 3" -.brev +The optional +.meta error-stream +argument is passed down to +.code regex-parse +as well as in the recursive call to +.codn regex-compile , +if that call takes place. -Though they produce the same result, the above two examples differ in that -.code @rest -embeds a metasymbol into the quasiliteral structure, whereas -.code @@rest -embeds the Lisp expression -.code @rest -into the quasiliteral. Either way, in the scope of -.codn op , -.code @rest -undergoes the macro-expansion which renames it to the machine-generated -function argument symbol of the implicit function denoted by the -.code op -macro form. +If +.meta error-stream +is specified, it must be a stream. Any error diagnostics are sent to that +stream. -This convenient omission of the -.code @ -character isn't supported for reaching the arguments of an outer -.code op -from a quasiliteral within a nested -.codn op : +.TP* Examples: .verb - ;; To reach @@1, @@@1 must be written. - ;; @@1 Lisp expression introduced by @. - (op ... (op ... `@@@1`)) -.brev + ;; the equivalent of #/[a-zA-Z0-9_/ + (regex-compile '(set (#\ea . #\ez) (#\eA . #\eZ) + (#\e0 . #\e9) #\e_)) -.TP* Example: + ;; the equivalent of #/.*/ and #/.+/ + (regex-compile '(0+ wild)) + (regex-compile '(1+ wild)) -.verb - ;; Take a list of pairs and produce a list in which those pairs - ;; are reversed. + ;; #/a|b|c/ + (regex-compile '(or (or #\ea #\eb) #\ec)) - (mapcar (op list @2 @1) '((1 2) (a b))) -> ((2 1) (b a)) + ;; string + (regex-compile "a|b|c") .brev -.coNP Macro @ lop +.coNP Function @ regex-source .synb -.mets (lop << form +) +.mets (regex-source << regex ) .syne .desc The -.code lop -macro is variant of -.code op -with special semantics. +.code regex-source +function returns the source code of compiled regular expression +.metn regex . + +The source code isn't the textual notation, but the Lisp +data structure representing the abstract syntax tree: the +same representation as what is returned by +.codn regex-parse . +.coNP Function @ regex-parse +.synb +.mets (regex-parse < string <> [ error-stream ]) +.syne +.desc The -.meta form -arguments support the same notation as those of the -.code op -operator. +.code regex-parse +function parses a character string which contains a regular expression and +turns it into a Lisp data structure (the abstract syntax tree representation of +the regular expression). -If only one -.meta form -is given then -.code lop -is equivalent to -.codn op . +The regular expression syntax +.code #/RE/ +produces the same structure, but as a +literal which is processed at the time \*(TX source code is read; the +.code regex-parse +function performs this parsing at run-time. -If two or more -.meta form -arguments are present, then -.code lop -generates a variadic function which inserts all of its trailing -arguments between the first and second -.metn form -s. +If there are parse errors, the function returns +.codn nil . -That is to say, trailing arguments coming into the anonymous function -become the left arguments of the function or function-like object -denoted by the first -.meta form -and the remaining -.metn form -s -give additional arguments. Hence the name -.codn lop , -which stands for \(dqleft-inserting -.codn op \(dq. +The optional +.meta error-stream +argument specifies a stream to which error messages +are sent from the parser. By default, diagnostic output goes to the +.code *stdnull* +stream, which discards it. If +.meta error-stream +is specified as +.codn t , +then the diagnostic output goes to the +.code *stdout* +stream. -This left insertion of the trailing arguments takes place regardless of whether -.code @rest -occurs in any -.metn form . +If +.code regex-parse +returns a +.cod2 non- nil +value, that structure is then something +which is suitable as input to +.codn regex-compile . -The -.meta form -syntax determines the number of required arguments of the -generated function, according to the highest-valued meta-number. The trailing -arguments which are inserted into the left position are any arguments in excess -of the required arguments. +There is a small difference in the syntax accepted by +.code regex-parse +and the syntax of regular expression literals. Any +.code / +(slash) characters occurring in any position within +.meta string +are treated as ordinary characters, not as regular expression delimiters. +The call +.mono +(regex-parse "/a/") +.onom +matches three characters: a slash, followed by the letter "a", followed +by another slash. Note that the slashes are not escaped. -The -.code lop -macro's expansion can be understood via the following equivalences, -except that in the real implementation, the symbols -.code rest -and -.code arg1 -through -.code arg3 -are replaced with hygienic, unique symbols. +Note: if a +.code regex-parse +call is written using a string literal as the +.meta string +argument, then note that any backslashes which are to be processed +by the regular expression must be doubled up, otherwise they belong +to the string literal: .verb - (lop f) <--> (op f) <--> (lambda (. rest) [f . rest]) - - (lop f x y) <--> (lambda (. rest) - [apply f (append rest [list x y])]) - - (lop f x @3 y) <--> (lambda (arg1 arg2 arg3 . rest) - [apply f - (append rest - [list x arg3 y])]) + (regex-parse "\e*") ;; error, invalid string literal escape + (regex-parse "\e\e*") ;; correct: the \e* literal match for * .brev -.TP* Examples: - -.verb - (mapcar (lop list 3) '(a b c)) --> ((a 3) (b 3) (c 3)) - - (mapcar (lop list @1) '(a b c)) --> ((a) (b) (c)) - - (mapcar (lop list @1) '(a b c) '(d e f)) - --> ((d a) (e b) (f c)) - -.brev +The double backslash in the string literal produces a single backslash +in the resulting string object that is processed by +.codn regex-parse . -.coNP Macros @, ap @, ip @ ado and @ ido. +.coNP Function @ read-until-match .synb -.mets (ap << form +) -.mets (ip << form +) -.mets (ado << form +) -.mets (ido << form +) +.mets (read-until-match < regex >> [ stream <> [ include-match ]]) .syne .desc The -.code ap -macro is based on the -.code op -macro and has identical argument -conventions. +.code read-until-match +function reads characters from +.metn stream , +accumulating them into a string, which is returned. + +If an argument is not specified for +.metn stream , +then the +.code *std-input* +stream is used. The -.code ap -macro analyzes its arguments and produces a function -.metn f , -in exactly the same same way as the -.code op -macro. However, instead of returning -.metn f , -directly, it returns a different function -.metn g , -which is a one-argument function which accepts a list, -and then applies the list as arguments to -.metn f . +.meta include-match +argument is Boolean, indicating whether the delimiting text +matched by +.meta regex +is included in the returned string. It defaults to +.codn nil . -In other words, the following equivalence holds: +The accumulation of characters is terminated by a match on +.metn regex , +the end of the stream, or an error. -.verb - (ap form ...) <--> (apf (op form ...)) -.brev +This means that characters are read from the stream and accumulated while the +stream has more characters available, and while its prefix does not match +.metn regex . -The -.code ap -macro nests properly with -.code op -and -.codn do , -in any combination, in regard to the -.meta ...@@n -notation. +If +.meta regex +matches the stream before any characters are accumulated, +then an empty string is returned. -The -.code ip -macro is very similar to the -.code ap -macro, except that it is based -on the semantics of the function -.code iapply -rather than -.codn apply , -according -to the following equivalence: +If the stream ends or an non-exception-throwing error occurs before any +characters are accumulated, the function returns +.codn nil . -.verb - (ip form ...) <--> (ipf (op form ...)) -.brev +When the accumulation of characters terminates by a match on +.metn regex , +the longest possible matching sequence of characters is +removed from the stream. If +.meta include-match +is true, that matching text is included in +the returned string. Otherwise, it is discarded. -The -.code ado -and -.code ido -macros are related to do macro in the same way that -.code ap +.coNP Functions @ scan-until-match and @ count-until-match +.synb +.mets (scan-until-match < regex <> [ stream ]) +.mets (count-until-match < regex <> [ stream ]) +.syne +.desc +The functions +.code scan-until-match and -.code ip -are related to -.codn op . -They produce a one-argument function which works -as if by applying its arguments to the function generated by do, -according to the following equivalence: - -.verb - (ado form ...) <--> (apf (do form ...)) +.code count-until-match +read characters from +.meta stream +until a match occurs in the stream for regular expression +.metn regex , +the stream runs out of characters, or an error occurs. - (ido form ...) <--> (ipf (do form ...)) -.brev +If the stream runs out of characters, or a non-exception-throwing error +occurs, before a match for +.meta regex +is identified, these functions return +.codn nil . -See also: the -.code apf -and -.code ipf -functions. +If a match for +.meta regex +occurs in +.metn stream , +then +.code count-until-match +returns the number of characters that were read and discarded prior to +encountering the first matching character. +In the same situation, the +.code scan-until-match +function returns a +.code cons +cell whose +.code car +holds the count of discarded characters, that being the same value as what +would be returned by +.codn count-until-match , +and whose +.code cdr +holds a character string that comprises the text matched by +.metn regex . -.coNP Macros @ opip and @ oand +.coNP Functions @, m^$ @ m^ and @ m$ .synb -.mets (opip << clause *) -.mets (oand << clause *) +.mets (m^$ < regex <> [ position ] << string ) +.mets (m^ < regex <> [ position ] << string ) +.mets (m$ < regex <> [ end-position ] << string ) .syne .desc +These functions provide functionality similar to the +.meta match-regst +and +.meta match-regst-right +functions, but under alternative interfaces which are more +convenient. + The -.code opip +.code ^ and -.code oand -operators make it possible to chain together functions which are expressed -using the -.code op -syntax. (See the -.code op -operator for more information). +.code $ +notation used in their names are an allusion to the +regular expression search anchoring operators found in +familiar POSIX utilities such as +.codn grep . -Both macros perform the same transformation except that -.code opip -translates its arguments to a call to the -.code chain -function, whereas -.code oand -translates its arguments in the same way to a call to the -.code chand -function. +The +.meta position +argument, if omitted, +defaults to zero, so that the +entire +.meta string +is operated upon. -More precisely, these macros perform the following rewrites: +The +.meta end-position +argument defaults to the length of +.metn string , +so that the end position coincides with the end of the +string. -.verb - (opip arg1 arg2 ... argn) -> [chain {arg1} {arg2} ... {argn}] - (oand arg1 arg2 ... argn) -> [chand {arg1} {arg2} ... {argn}] -.brev +If the +.meta position +or +.meta end-position +arguments are negative, they index backwards +from the length of +.meta string +so that -1 denotes the last character. -where the above -.code {arg} -notation denotes the following transformation applied to each argument: +A value in either parameter which is excessively +negative or positive, such that it indexes before +the start of the string or exceeds its length +results in a failed match and consequently +.code nil +being returned. -.verb - (function ...) -> (op function ...) - (operator ...) -> (do operator ...) - (macro ...) -> (do macro ...) - (dwim ...) -> (dwim ...) - [...] -> [...] - (qref ...) -> (qref ...) - (uref ...) -> (uref ...) - .slot -> .slot - .(method ...) -> .(method ...) - atom -> atom -.brev +The +.code m^$ +function tests whether the entire portion of +.meta string +starting at +.meta position +through to the end of the string is in the set of strings +matched by +.metn regex . +If this is true, then that portion of the string is +returned. Otherwise +.code nil +is returned. -In other words, compound forms whose leftmost symbol is a macro or operator -are translated to the -.code do -notation. Compound forms denoting function calls are translated to the -.code op -notation. Compound forms which are -.code dwim -invocations, either explicit or via the DWIM brackets notation, are -used without transformation. Used without transformation also are forms -denoting struct slot access, either explicitly using -.code uref -or -.code qref -or the respective dot notations, as well as any atom forms. +The +.code m^ +function tests whether the portion of the +.meta string +starting at +.meta position +has a prefix which matches +.metn regex . +If so, then this matching prefix is returned. +Otherwise +.code nil +is returned. -Note: the -.code opip +The +.code m$ +function tests whether the portion of +.meta string +ending just before +.meta end-position +has a suffix which matches +.metn regex . +If so, then this matching suffix is returned. +Otherwise +.code nil +is returned. + +.coNP Functions @, r^$ @, r^ @ r$ and @ rr +.synb +.mets (r^$ < regex <> [ position ] << string ) +.mets (r^ < regex <> [ position ] << string ) +.mets (r$ < regex <> [ end-position ] << string ) +.mets (rr < regex >> [ position <> [ from-end ]] << string ) +.syne +.desc +The first three of these functions perform the same operations as, +respectively, +.codn m^$ , +.code m^ and -.code oand -macros use their macro environment in determining whether a form is a -macro call, thereby respecting lexical scoping. +.codn m$ , +with the same argument conventions. They differ +in return value. When a match is found, they +return a range value indicating the extent of +the matching substring within +.meta string +rather than the matching substring itself. -.TP* Example: -Take each element from the list -.code "(1 2 3 4)" -and multiply it by three, then add 1. -If the result is odd, collect that into the resulting list: +The +.code rr +function performs the same operation as +.code range-regex +with different conventions with regard to argument +order, harmonizing with those of the other three functions above. -.mono -(mappend (opip (* 3) - (+ 1) - [iff oddp list]) - (range 1 4)) -.onom +The +.meta position +argument, if omitted, +defaults to zero, so that the +entire +.meta string +is operated upon. -The above is equivalent to: +The +.meta end-position +argument defaults to the length of +.metn string , +so that the end position coincides with the end of the +string. -.mono -(mappend (chain (op * 3) - (op + 1) - [iff oddp list]) - (range 1 4)) -.onom +With one exception, a value in either parameter which is excessively negative +or positive, such that it indexes before the start of the string or exceeds its +length results in a failed match and consequently +.code nil +being returned. The exception is that the +.code rr +function permits a negative +.meta position +value which refers before the start of the string; this is effectively +treated as zero. The -.code "(* 3)" -and -.code "(+ 1)" -terms are rewritten to -.code "(op * 3)" -and -.codn "(op + 1)" , -respectively, whereas -.code "[iff oddp list]" -is passed through untransformed. +.meta from-end +argument defaults to +.codn nil . -.coNP Macro @ ret -.synb -.mets (ret << form ) -.syne -.desc The -.code ret -macro's -.meta form -argument is treated similarly to the second and subsequent arguments of the -.code op -operator. +.code r^$ +function tests whether the entire portion of +.meta string +starting at +.meta position +through to the end of the string is in the set of strings +matched by +.metn regex . +If this is true, then the matching range is returned, +as a range object. The -.code ret -macro produces a function which takes any number of arguments, -and returns the value specified by -.metn form . +.code r^ +function tests whether the portion of the +.meta string +starting at +.meta position +has a prefix which matches +.metn regex . +If so, then the matching range is returned, as a range object. +Otherwise +.code nil +is returned. -.meta form -can contain -.code op -meta syntax like -.code @n -and -.codn @rest . +The +.code r$ +function tests whether the portion of +.meta string +ending just before +.meta end-position +has a suffix which matches +.metn regex . +If so, then the matching range is returned. +Otherwise +.code nil +is returned. -The following equivalence holds: +The +.code rr +function searches +.meta string +starting at +.meta position +for a match for +.codn regex . +If +.meta from-end +is specified and true, the rightmost +match is reported. +If a match is found, it is reported +as a range. -.verb - (ret x) <--> (op identity x)) -.brev +A regular expression which matches empty +strings matches at the start position, +and every other position, including +the position just after the last +character, coinciding with the length of +.metn string . -Thus the expression -.code "(ret @2)" -returns a function similar to -.codn "(lambda (x y . z) y)" , -and the expression -.code "(ret 42)" -returns a function similar to -.codn "(lambda (. rest) 42)" . +Except for the different argument order such that +.meta string +is always the rightmost argument, the +.code rr +function is equivalent to the +.code range-regex +function, such that correspondingly named +arguments have the same semantics. -.coNP Macro @ aret +.coNP Function @ rra .synb -.mets (aret << form ) +.mets (rra < regex >> [ start <> [ end ]] << string ) .syne .desc The -.code aret -macro's -.meta form -argument is treated similarly to the second and subsequent arguments of the -.code op -operator. +.code rra +function searches +.meta string +between the +.meta start +and +.meta end +position for matches for the regular expression +.metn regex . + +The matches are returned as a list of range objects. The -.code aret -macro produces a function which takes any number of arguments, -and returns the value specified by -.metn form . +.meta start +argument defaults to zero, and +.meta end +defaults to the length of the string (the position one +past the last character). -.meta form -can contain -.code ap -meta syntax like -.meta @n +Negative values of +.meta start and -.codn @rest . +.meta end +indicate positions from the end of the string, such that -1 +denotes the last character, -2 the second-to-last and so forth. -The following equivalence holds: +If +.meta start +is so negative that it refers before the start of +.metn string , +it is treated as zero. If this situation is true of the +.meta end +argument, then the function returns +.codn nil . -.verb - (aret x) <--> (ap identity x)) -.brev +If +.meta start +refers to a character position beyond the length of +.meta string +(two characters or more beyond the end of the string), +then the function returns +.codn nil . +If this situation is true of +.metn end , +then +.meta end +is is curtailed to the the string length. -Thus the expression -.code "(aret @2)" -returns a function similar to -.codn "(lambda (. rest) (second rest))" , -and the expression -.code "(aret 42)" -returns a function similar to -.codn "(lambda (. rest) 42)" . +The +.code rra +function returns all non-overlapping matches, including +zero length matches. Zero length matches may occur before +the first character of the string, or after the last +character. If so, these are included. -.coNP Function @ dup +.coNP Functions @, f^$ @ f^ and @ f$ .synb -.mets (dup << func ) +.mets (f^$ < regex <> [ position ]) +.mets (f^ < regex <> [ position ]) +.mets (f$ < regex <> [ end-position ]) .syne .desc -The -.code dup -function returns a one-argument function which calls the two-argument -function -.meta func -by duplicating its argument. +These regular expression functions do not directly +perform regex operations. Rather, they each return +a function of one argument which performs a regex +operation. -.TP* Example: +The returned functions perform the same operations as, +respectively, +.codn m^$ , +.code m^ +and +.codn m$ . + +The following equivalences nearly hold, except that the functions +on the right side produced by +.code op +can accept two arguments when only +.code r +is curried, whereas the functions on the left take only +one argument: .verb - ;; square the elements of a list - (mapcar [dup *] '(1 2 3)) -> (1 4 9) + [f^$ r] <--> (op m^$ r) + [f^$ r p] <--> (op m^$ r p) + [f^ r] <--> (op m^ r) + [f^ r p] <--> (op m^ r p) + [f$ r] <--> (op m$ r) + [f$ r p] <--> (op m$ r p) .brev -.coNP Function @ flipargs -.synb -.mets (flipargs << func ) -.syne -.desc -The -.code flipargs -function returns a two-argument function which calls the two-argument -function -.meta func -with reversed arguments. +That is to say, +.code f^$ +returns a function which binds +.meta regex +and possibly the optional +.metn position . +When this function is invoked, it must be given an argument +which is a string. It performs the same operation as +.code m^$ +being called on +.meta regex +and possibly +.metn position . +The same holds between +.code f^ +and +.codn m^ , +and between +.code f$ +and +.codn m$ . -.coNP Functions @ chain and @ chand -.synb -.mets (chain << func *) -.mets (chand << func *) -.syne -.desc -The -.code chain -function accepts zero or more functions as arguments, and returns -a single function, called the chained function, which represents the chained -application of those functions, in left to right order. +.TP* Examples: -If -.code chain -is given no arguments, then it returns a variadic function which -ignores all of its arguments and returns -.codn nil . +.verb + ;; produce list which contains only strings + ;; beginning with "cat": + (keep-if (f^ #/cat/) '#"dog catalog cat fox catapult") + --> ("catalog" "cat" "catapult") -Otherwise, the first function may accept any number of arguments. The second -and subsequent functions, if any, must accept one argument. + ;; map all strings in a list to just their trailing + ;; digits. + (mapcar (f$ #/\ed*/) '#"a123 4 z bc465") + --> ("123" "4" "" "465") -The chained function can be called with an argument list which is acceptable -to the first function. Those arguments are in fact passed to the first -function. The return value of that call is then passed to the second -function, and the return value of that call is passed to the third function -and so on. The final return value is returned to the caller. + ;; check that all strings consist of digits after + ;; the third position. + (all '#"ABC123 DFE45 12379" (f^$ #/\ed*/ 3)) + --> "79" ; i.e. true + (all '#"ABC123 DFE45 12379A" (f^$ #/\ed*/ 3)) + --> nil +.brev -The -.code chand -function is similar, except that it combines the functionality of -.code andf -into chaining. The difference between -.code chain +.coNP Functions @, fr^$ @, fr^ @ fr$ and @ frr +.synb +.mets (fr^$ < regex <> [ position ]) +.mets (fr^ < regex <> [ position ]) +.mets (fr$ < regex <> [ end-position ]) +.mets (frr < regex <> [[ start-position ] << from-end ]) +.syne +.desc +These regular expression functions do not directly +perform regex operations. Rather, they each return +a function of one argument which performs a regex +operation. + +The returned functions perform the same operations as, +respectively, +.codn r^$ , +.codn r^ , +.code r$ and -.code chand -is that -.code chand -immediately terminates and returns -.code nil -whenever any of the functions returns -.codn nil , -without calling the remaining functions. +.codn rr . -.TP* Example: +The following equivalences nearly hold, except that some of the +functions on the right side produced by op +.code op +can accept additional arguments after the input string, +whereas the functions on the left produced by +.code f^$ +.I "et al." +accept only one parameter: the input string. .verb - (call [chain + (op * 2)] 3 4) -> 14 + [fr^$ r] <--> (op m^$ r) + [fr^$ r p] <--> (op m^$ r p) + [fr^ r] <--> (op m^ r) + [fr^ r p] <--> (op m^ r p) + [fr$ r] <--> (op m$ r) + [fr$ r p] <--> (op m$ r p) + [frr r] <--> (op m$ r) + [frr r s] <--> (op m$ r s) + [frr r s fe] <--> (op m$ r s fe) .brev -In this example, a two-element chain is formed from the -.code + -function -and the function produced by -.code "(op * 2)" -which is a one-argument -function that returns the value of its argument multiplied by two. -(See the definition of the -.code op -operator). - -The chained function is invoked using the -.code call -function, with the arguments -.code 3 +That is to say, +.code fr^$ +returns a function which binds +.meta regex +and possibly the optional +.metn position . +When this function is invoked, it must be given an argument +which is a string. It performs the same operation as +.code r^$ +being called on +.meta regex +and possibly +.metn position , +and the string. +The same holds between +.code fr^ and -.codn 4 . -The chained evaluation begins by passing -.code 3 +.codn r^ , +between +.code fr$ and -.code 4 -to -.codn + , -which yields -.codn 7 . -This -.code 7 -is then passed to the -.code "(op * 2)" -doubling function, resulting in -.codn 14 . +.codn r$ , +and between +.code frr +and +.codn rr . -A way to write the above example without the use of the DWIM brackets and the -op operator is this: +.TP* Examples: .verb - (call (chain (fun +) (lambda (x) (* 2 x))) 3 4) + ;; Remove leading digits from "123A456", + ;; other than first digit: + (regsub (fr^ #/\ed+/ 1) "" "123A456") + --> "1A456" .brev -.coNP Function @ juxt -.synb -.mets (juxt << func *) -.syne -.desc -The -.code juxt -function accepts a variable number of arguments which are functions. It -combines these into a single function which, when invoked, passes its arguments -to each of these functions, and collects the results into a list. +.SS* Hashing Library -Note: the juxt function can be understood in terms of the following reference -implementation: +A hash table is an object which retains an association between pairs of +objects. Each pair consists of a key and value. Given an object which is +similar to a key in the hash table, it is possible to retrieve the +corresponding value. Entries in a hash table are not ordered in any way, and +lookup is facilitated by hashing: quickly mapping a key object to a numeric +value which is then used to index into one of many buckets where the matching +key will be found (if such a key is present in the hash table). -.verb - (defun juxt (funcs) - (lambda (. args) - (mapcar (lambda (fun) - (apply fun args)) - funcs))) -.brev +In addition to keys and values, a hash table contains a storage location +which allows it to be associated with user data. -The -.code callf -function generalizes -.code juxt -by allowing the combining function to be specified. +Important to the operation of a hash table is the criterion by which keys are +considered same. By default, this similarity follows the eql function. A hash +table will search for a stored key which is +.code eql +to the given search key. +A hash table constructed with the +.codn equal -based +property compares keys using +the +.code equal +function instead. -.TP* Example: +In addition to storing key-value pairs, a hash table can have a piece of +information associated with it, called the user data. -.verb - ;; separate list (1 2 3 4 5 6) into lists of evens and odds, - ;; which end up juxtaposed in the output list: +\*(TX hash tables contain a seed value which permutes the hashing operation, +at least for keys of certain types. This feature, if the seed is randomized, +helps to prevent software from being susceptible to hash collision +denial-of-service attacks. However, by default, the seed is not randomized. +Newly created hash tables for which a seed value is not specified take their +seed value from the +.code *hash-seed* +special variable, which is initialized to zero. That includes hash tables +created by parsing hash literal syntax. +Security-sensitive programs +requiring protection against collision attacks may use +.code gen-hash-seed +to create a randomized hash seed, and, depending on their specific need, either +store that value in +.codn *hash-seed* , +or pass the value to hash table constructors like +.codn make-hash , +or both. +Note: randomization of hash seeding isn't a default behavior because it affects +program reproducibility. The seed value affects the order in which keys are +traversed, which can change the output of programs whose inputs have not +changed, and whose logic is is otherwise deterministic. - [(op [juxt keep-if remove-if] evenp) - '(1 2 3 4 5 6)] -> ((2 4 6) (1 3 5)) +A hash table can be traversed to visit all of the keys and data. The order of +traversal bears no relation to the order of insertion, or to any properties of +the key type. - ;; call several functions on 1, collecting their results: - [[juxt (op + 1) (op - 1) evenp sin cos] 1]' - -> (2 0 nil 0.841470984807897 0.54030230586814) -.brev +During an open traversal, new keys can be inserted into a hash table or deleted +from it while a a traversal is in progress. Insertion of a new key during +traversal will not cause any existing key to be visited twice or to be skipped; +however, it is not specified whether the new key will be traversed. Similarly, +if a key is deleted during traversal, and that key has not yet been visited, it +is not specified whether it will be visited during the remainder of the +traversal. These remarks apply not only to deletion via +.code remhash +or the +.code del +operator, but also to wholesale deletion of all keys via +.codn clearhash . -.coNP Functions @ andf and @ orf +The garbage collection of hash tables supports weak keys and weak values. +If a hash table has weak keys, this means that from the point of view +of garbage collection, that table holds only weak references to the keys +stored in it. Similarly, if a hash table has weak values, it means that it +holds a weak reference to each value stored. A weak reference is one +which does not prevent the reclamation of an object by the garbage +collector. That is to say, when the garbage collector discovers that the only +references to some object are weak references, then that object is considered +garbage, just as if it had no references to it. The object is reclaimed, and +the weak references "lapse" in some way, which depends on what kind they are. +Hash table weak references lapse by entry removal. When an object used +as a key in in one or more weak-key hash tables becomes unreachable, those +hash entries disappear. Similarly, when an object appearing as a value in +one or more hash table entries in weak-value hash tables becomes unreachable, +those entries disappear. When a hash table has both weak keys and weak values, +then its entries are removed when either keys or values become unreachable. +In other words, both the key and value must be reachable in order to +retain the entry. + +An open traversal of a hash table is performed by the +.code maphash +function and the +.code dohash +operator. The traversal is open because code supplied by the program +is evaluated for each entry. + +The functions +.codn hash-keys , +.codn hash-values , +.codn hash-pairs , +and +.code hash-alist +also perform an open traversal, because they return +lazy lists. The traversal isn't complete until the returned lazy list +is fully instantiated. In the meanwhile, the +\*(TX program can mutate the hash table from which the lazy list +is being generated. + +Certain hash operations expose access to the internal key-value association +entries of a hash table, which are represented as ordinary +.code cons +cells. Modifying the +.code car +field of such a cell potentially violates the integrity of the hash table; +the behavior of subsequent lookup and insertion operations becomes unspecified. + +Similarly, if an object is used as a key in an +.codn equal -based +hash table, and that object is mutated in such a way that its equality to +other objects under the +.code equal +function is affected or its hash value under +.code hash-equal +is altered, the behavior of subsequent lookup and insertion operations on the +becomes unspecified. + +.coNP Functions @ make-hash and @ hash .synb -.mets (andf << func *) -.mets (orf << func *) +.mets (make-hash < weak-keys < weak-vals +.mets \ \ \ \ \ \ \ \ \ \ < equal-based <> [ hash-seed ]) +.mets (hash {:weak-keys | :weak-vals | +.mets \ \ \ \ \ \ :eql-based | :equal-based | +.mets \ \ \ \ \ \ :userdata << obj }*) .syne .desc +These functions construct a new hash table. + +.code make-hash +takes three mandatory Boolean arguments. The +.meta weak-keys +argument specifies whether the hash table shall have weak keys. The +.meta weak-vals +argument specifies whether it shall have weak values, and +.meta equal-based +specifies whether it is +.codn equal -based. +The hash function defaults +all three of these properties to false, and allows them to be overridden to +true by the presence of keyword arguments. + +The optional +.meta hash-seed +parameter must be an integer, if specified. Its value perturbs the hashing +function of the hash table, which affects +.code :equal-based +hash tables, when character strings and buffers are used as keys. +If +.meta hash-seed +is omitted, then the value of the +.code *hash-seed* +variable is used as the seed. + +It is an error to attempt to construct an +.codn equal -based +hash table which has weak keys. + The -.code andf +.code hash +function provides an alternative interface. It accepts optional +keyword arguments. The supported keyword symbols are: +.codn :weak-keys , +.codn :weak-vals , +.codn :equal-based , +.code :eql-based and -.code orf -functions are the functional equivalent of the -.code and +.code :userdata +which can be specified in any order to turn on the corresponding properties in +the newly constructed hash table. + +Only one of +.code :equal-based and -.code or -operators. These functions accept multiple functions and return a new function -which represents the logical combination of those functions. +.code :eql-based +may be specified. If specified, then the hash table uses +.code equal +or +.code eql +equality, respectively, for considering two keys to be the same key. +If neither is specified, the +.code hash +function produces an +.code :equal-based +hash table by default. -The input functions should have the same arity. Failing that, there should -exist some common argument arity with which each of these can be invoked. The -resulting combined function is then callable with that many arguments. +If +.code :weak-keys +is specified, then it automatically implies +.code :eql-based +and, therefore, +.code :equal-based +may not be specified. -The -.code andf -function returns a function which combines the input functions with -a short-circuiting logical conjunction. The resulting function passes its -arguments to the functions successively, in left to right order. As soon as any -of the functions returns -.codn nil , -then nil is returned immediately, and the -remaining functions are not called. Otherwise, if none of the functions return -.codn nil , -then the value returned by the last function is returned. If the list of -functions is empty, then -.code t -is returned. That is, -.code (andf) -returns a function -which accepts any arguments, and returns -.codn t . +If +.code :userdata +is present, it must be followed by an argument value; that value +specifies the user data for the hash table, which can be retrieved using the +.code hash-userdata +function. -The -.code orf -function combines the input functions with a short-circuiting logical -disjunction. The function produced by -.code orf -passes its arguments down to the -functions successively, in left to right order. As soon as any function -returns a -.cod2 non- nil -value, that value is returned and the remaining functions are -not called. If all functions return -.codn nil , -then -.code nil -is returned. The expression -.code (orf) -returns a function which accepts any arguments and returns -.codn nil . +Note: there doesn't exist a keyword for specifying the seed. +This omission is deliberate. These hash construction keywords may appear in the +hash literal +.code #H +syntax. A seed keyword would allow literals to specify their own seed, which +would allow malicious hash literals to be crafted that perpetrate a hash +collision attack against the parser. -.coNP Function @ notf +.coNP Functions @, hash-construct @ hash-from-pairs and @ hash-from-alist .synb -.mets (notf << function ) +.mets (hash-construct < hash-args << key-val-pairs ) +.mets (hash-from-pairs < key-val-pairs << hash-arg *) +.mets (hash-from-alist < alist << hash-arg *) .syne .desc The -.code notf -function returns a function which is the Boolean negation -of -.metn function . +.code hash-construct +function constructs a populated hash in one step. The +.meta hash-args +argument specifies a list suitable as an argument list in a call to the hash +function. The +.meta key-val-pairs +is a sequence of pairs, which are two-element +lists representing key-value pairs. -The returned function takes a variable number of arguments. When -invoked, it passes all of these arguments to -.meta function -and then inverts the result as if by application of the -.codn not . +A hash is constructed as if by a call to +.mono +.meti (apply hash << hash-args ), +.onom +then populated +with the specified pairs, and returned. -.coNP Functions @ iff and @ iffi -.synb -.mets (iff < cond-func >> [ then-func <> [ else-func ]]) -.mets (iffi < cond-func < then-func <> [ else-func ]) -.syne -.desc The -.code iff -function is the functional equivalent of the -.code if -operator. It accepts -functional arguments and returns a function. - -The resulting function takes its arguments, if any, and applies them to -.metn cond-func . -If -.meta cond-func -yields true, then the arguments are passed to -.meta then-func -and the -resulting value is returned. Otherwise the arguments are passed to -.meta else-func -and the resulting value is returned. - -If -.meta then-func -is omitted then -.code identity -is used as default. This omission is not permitted by -.codn iffi , -only -.codn iff . - -If -.meta else-func -needs to be called, but is omitted, then -.code nil -is returned. +.code hash-from-pairs +function is an alternative interface to the same semantics. The +.meta key-val-pairs +argument is first, and the +.meta hash-args +are passed as trailing variadic arguments, rather than a single list argument. The -.code iffi -function differs from -.code iff -only in the defaulting behavior with respect -to the -.meta else-func -argument. If -.meta else-func -is omitted in a call to -.code iffi -then the default function is -.codn identity . -This is useful in situations when one value is to be -replaced with another one when the condition is true, otherwise -preserved. - -The following equivalences hold between -.code iffi -and -.codn iff : - -.verb - (iffi a b c) <--> (iff a b c) - - (iffi a b) <--> (iff a b identity) - - [iffi a b nilf] <--> [iff a b] - - [iffi a identity nilf] <--> [iff a] -.brev - -The following equivalences illustrate -.code iff -with both optional arguments omitted: - -.verb - [iff a] <---> [iff a identity nilf] <---> a -.brev +.code hash-from-alist +function is similar to +.codn hash-from-pairs , +except that the +.meta alist +argument specifies they keys and values as an association list. +The elements of the list are +.code cons +cells, each of whose +.code car +is a key, and whose +.code cdr +is the value. -.coNP Functions @ tf and @ nilf +.coNP Function @ hash-list .synb -.mets (tf << arg *) -.mets (nilf << arg *) +.mets (hash-list < key-list << hash-arg *) .syne .desc The -.code tf -and -.code nilf -functions take zero or more arguments, and ignore them. -The -.code tf -function returns -.codn t , -and the -.code nilf -function returns -.codn nil . - -Note: the following equivalences hold between these functions and the -.code ret -operator, and -.code retf -function. - -.verb - (fun tf) <--> (ret t) <--> (retf t) - (fun nilf) <--> (ret nil) <--> (ret) <--> (retf nil) -.brev - -In Lisp-1-style code, -.code tf -and -.code nilf -behave like constants which can replace uses of -.code "(ret t)" -and -.codn "(ret nil)" : - -.verb - [mapcar (ret nil) list] <--> [mapcar nilf list] -.brev - -.TP* Example: - -.verb - ;; tf and nilf are useful when functions are chained together. - ;; test whether (trunc n 2) is odd. +.code hash-list +function constructs a hash as if by a call to +.mono +.meti (apply hash << hash-args ), +.onom +where +.meta hash-args +is a list of the individual +.meta hash-arg +variadic arguments. - (defun trunc-n-2-odd (n) - [[chain (op trunc @1 2) [iff oddp tf nilf]] n]) -.brev +The hash is then populated with keys taken from +.meta key-list +and returned. -In this example, two functions are chained together, and -.code n -is passed -through the chain such that it is first divided by two via the -function denoted by -.code "(op trunc @1 2)" -and then the result is passed into the -function denoted by -.codn "[iff oddp tf nilf]" . -The -.code iff -function passes its argument into -.codn oddp , -and if -.code oddp -yields true, it passes the same argument to -.codn tf . -Here -.code tf -proves its utility by ignoring that value and returning -.codn t . -If the argument (the divided value) passed into -.code iff -is even, then iff passes it into the -.code nilf -function, which ignores the value and returns -.codn nil . +The value associated with each key is that key itself. -.coNP Function @ retf +.coNP Function @ hash-update .synb -.mets (retf << value ) +.mets (hash-update < hash << function ) .syne .desc The -.code retf -function returns a function. That function can take zero or -more arguments. When called, it ignores its arguments and returns -.metn value . - -See also: the -.code ret -macro. - -.TP* Example: +.code hash-update +function replaces each value in +.metn hash , +with the value of +.meta function +applied to that value. -.verb - ;; the function returned by (retf 42) - ;; ignores 1 2 3 and returns 42. - (call (retf 42) 1 2 3) -> 42 -.brev +The return value is +.metn hash . -.coNP Functions @ apf and @ ipf +.coNP Function @ hash-update-1 .synb -.mets (apf << function ) -.mets (ipf << function ) +.mets (hash-update-1 < hash < key < function <> [ init ]) .syne .desc The -.code apf -function returns a one-argument function which accepts -a list. When the function is called, it treats the list as -arguments which are applied to -.meta function -as if by apply. It returns whatever -.meta function -returns. - -The -.code ipf -function is similar to -.codn apf , -except that the returned -function applies arguments as if by -.code iapply -rather than -.codn apply . - -See also: the -.code ap -macro. +.code hash-update-1 +function operates on a single entry in the hash table. -.TP* Example: +If +.meta key +exists in the hash table, then its corresponding value is passed +into +.metn function , +and the return value of +.meta function +is then installed +in place of the key's value. The value is then returned. -.verb - ;; Function returned by [apf +] accepts the - ;; (1 2 3) list and applies it to +, as - ;; if (+ 1 2 3) were called. +If +.meta key +does not exist in the hash table, and no +.meta init +argument is given, +then +.code hash-update-1 +does nothing and returns +.codn nil . - (call [apf +] '(1 2 3)) -> 6 -.brev +If +.meta key +does not exist in the hash table, and an +.meta init +argument is given, +then +.meta function +is applied to +.metn init , +and then +.meta key +is inserted into +.meta hash +with the value returned by +.meta function +as the datum. This value +is also returned. -.coNP Function @ callf +.coNP Function @ group-by .synb -.mets (callf < main-function << arg-function *) +.mets (group-by < func < sequence << option *) .syne .desc The -.code callf -function returns a function which applies its arguments to each -.metn arg-function , -juxtaposing the return values of these calls to form arguments -which are then passed to -.metn main-function . -The return value of -.meta main-function -is returned. - -The following equivalence holds, except for the order of evaluation of -arguments: +.code group-by +function produces a hash table from +.metn sequence , +which is a +list or vector. Entries of the hash table are not elements of +.metn sequence , +but lists of elements of +.metn sequence . +The function +.meta func +is applied to +each element of +.meta sequence +to compute a key. That key is used to determine +which list the item is added to in the hash table. -.verb - (callf fm f0 f1 f2 ...) <--> (chain (juxt f0 f1 f2 ...) - (apf fm)) -.brev +The trailing arguments +.mono +.meti << option * +.onom +if any, consist of the same keywords +that are understood by the hash function, and determine the properties +of the hash. .TP* Example: +Group the integers from 0 to 10 into three buckets keyed on 0, 1 and 2 +according to the modulo 3 congruence: .verb - ;; Keep those pairs which are two of a kind - - (keep-if [callf eql first second] '((1 1) (2 3) (4 4) (5 6))) - -> ((1 1) (4 4)) -.brev - -The following equivalence holds between -.code juxt -and -.codn callf : + (group-by (op mod @1 3) (range 0 10))) -.verb - [juxt f0 f1 f2 ...] <--> [callf list f0 f1 f2 ...]:w + -> #H(() (0 (0 3 6 9)) (1 (1 4 7 10)) (2 (2 5 8))) .brev -Thus, -.code juxt -may be regarded as a specialization of -.code callf -in which the main function is implicitly -.codn list . - -.coNP Function @ mapf +.coNP Function @ group-reduce .synb -.mets (mapf < main-function << arg-function *) +.mets (group-reduce < hash < classify-fun < binary-fun < seq +.mets \ \ >> [ init-value <> [ filter-fun ]]) .syne .desc The -.code mapf -function returns a function which distributes its arguments -into the -.metn arg-function -s. -That is to say, each successive argument of the returned -function is associated with a successive -.metn arg-function . +.code group-reduce +updates hash table +.meta hash +by grouping and reducing sequence +.metn seq . -Each -.meta arg-function -is called, passed the corresponding argument. The return -values of these functions are then passed as arguments -to -.meta main-function -and the resulting value is returned. +The function regards the hash table as being populated with +keys denoting accumulator values. Missing accumulators which +need to be created in the hash table are initialized with +.meta init-value +which defaults to +.codn nil . -If the returned function is called with fewer arguments than there -are -.metn arg-function -s, -then only that many functions are used. Conversely, if the function is -called with more arguments than there are -.metn arg-function -s, -then those arguments are ignored. +The function iterates over +.meta seq +and treats each element according to the following steps: +.RS +.IP 1. +Each element is mapped to a hash key through +.metn classify-fun . +.IP 2. +The value associated with the hash key (the accumulator for that +key) is retrieved. If it doesn't exist, +.meta init-value +is used. +.IP 3. +The function +.meta binary-fun +is invoked with two arguments: the accumulator from step 2, and the +original element from +.metn seq . +.IP 4. +The resulting value from step 3 is stored back into the hash table under the +key from step 2. +.RE -The following equivalence holds: +.IP +After the above processing, one more step is performed if the +.meta filter-fun +argument is present. In this case, the hash table is destructively mapped through +.meta filter-fun +before being returned. That is to say, every value in the hash table is +projected through +.meta filter-fun +and stored back in the table under the same key, as if by an invocation the +.mono +.meti (hash-update < hash << filter-fun ) +.onom +expression. -.verb - (mapf fm f0 f1 ...) <--> (lambda (. rest) - [apply fm [mapcar call - (list f0 f1 ...) - rest]]) -.brev +.IP +If +.code group-reduce +is invoked on an empty hash table, its net result closely resembles a +.code group-by +operation followed by separately performing a +.code reduce-left +on each value in the hash. -.TP* Example: +.TP* Examples: -.verb - ;; Add the squares of 2 and 3 - [[mapf + [dup *] [dup *]] 2 3] -> 13 +Frequency histogram: +.verb + [group-reduce (hash) identity (do inc @1) + "fourscoreandsevenyearsago" 0] + --> #H(() (#\ea 3) (#\ec 1) (#\ed 1) (#\ee 4) (#\ef 1) + (#\eg 1) (#\en 2) (#\eo 3) (#\er 3) (#\es 3) + (#\eu 1) (#\ev 1) (#\ey 1)) .brev -.SS* Input and Output (Streams) -\*(TL supports input and output streams of various kinds, with -generic operations that work across the stream types. +Separate the integers 1-10 into even and odd, and sum these groups: -In general, I/O errors are usually turned into exceptions. When the description -of error reporting is omitted from the description of a function, it can be -assumed that it throws an error. +.verb + [group-reduce (hash) evenp + (range 1 10) 0] + -> #H(() (t 30) (nil 25)) +.brev -.coNP Special variables @, *stdout* @, *stddebug* @, *stdin* @ *stderr* and @ *stdnull* +.coNP Functions @ make-similar-hash and @ copy-hash +.synb +.mets (make-similar-hash << hash ) +.mets (copy-hash << hash ) +.syne .desc -These variables hold predefined stream objects. The -.codn *stdin* , -.code *stdout* -and -.code *stderr* -streams closely correspond to the underlying operating system streams. -Various I/O functions require stream objects as arguments. - The -.code *stddebug* -stream goes to the same destination as -.codn *stdout* , -but is a separate object which can be redirected independently, allowing -debugging output to be separated from normal output. +.code make-similar-hash +and copy-hash functions create a new hash object based on +the existing +.meta hash +object. -The -.code *stdnull* -stream is a special kind of stream called a null stream. -This stream is not connected to any device or file. It is similar to +.code make-similar-hash +produces an empty hash table which inherits all of the +attributes of +.metn hash . +It uses the same kind of key equality, the +same configuration of weak keys and values, and has the same user data (see the -.code /dev/null -device on Unix, but does not involve the operating system. +.code set-hash-userdata +function). -.coNP Special variables @ *print-flo-format* and @ *pprint-flo-format* -.desc The -.code *print-flo-format* -variable determines the conversion format which is applied when -a floating-point value is converted to decimal text by the -functions -.codn print , -.codn prinl , -and -.codn tostring . - -The default value is -.codn "~s" . +.code copy-hash +function is like +.codn make-similar-hash , +except that instead of +producing an empty hash table, it produces one which has all the same elements +as +.metn hash : +it contains the same key and value objects. -The related variable -.code *pprint-flo-format* -similarly determines the conversion format applied to floating-point -values by the functions -.codn pprint , -.codn pprinl , +.coNP Function @ inhash +.synb +.mets (inhash < hash < key <> [ init ]) +.syne +.desc +The +.code inhash +function searches hash table +.meta hash +for +.metn key . +If +.meta key +is found, then it return the hash table's cons cell which +represents the association between +.meta hash and -.codn tostringp . +.metn key . +Otherwise, it returns +.codn nil . -The default value is -.codn "~a" . +If argument +.meta init +is specified, then the function will create +an entry for +.meta key +in +.meta hash +whose value is that of +.metn init . +The cons cell representing that association is returned. -The format string in either variable must specify the consumption of -exactly one -.code format -argument. +Note: for as long as the +.meta key +continues to exist inside +.metn hash . +modifying the +.code car +field of the returned cons has ramifications for the logical integrity of +the hash; doing so results in unspecified behavior for subsequent +insertion and lookup operations. -The conversion string may use embedded width and precision values: -for instance, -.code "~3,4f" -is a valid value for -.code *print-flo-format* -or -.codn *pprint-flo-format* . +Modifying the +.code cdr +field has the effect of updating the association with a new value. -.coNP Special variable @ *print-flo-precision* +.coNP Accessor @ gethash +.synb +.mets (gethash < hash < key <> [ alt ]) +.mets (set (gethash < hash < key <> [ alt ]) << new-value ) +.syne .desc The -.code *print-flo-precision* -special variable specifies the default floating-point printing -precision which is used when the -.code ~a -or -.code ~s -conversion specifier of the -.code format -function is used for printing a floating-point value, and no precision -is specified. - -Note that since the default value of the variable -.code *print-flo-format* -is the string -.codn "~s" , +.code gethash +function searches hash table +.meta hash +for key +.metn key . +If the +key is found then the associated value is returned. Otherwise, if the -.code *printf-flo-precision* -variable, by default, also determines the precision which applies when -floating-point values are converted to decimal text by the functions -.codn print , -.codn pprint , -.codn prinl , -.codn pprinl , -.code tostring -and -.codn tostringp . +.meta alt +argument was specified, it is returned. If the +.meta alt +argument +was not specified, +.code nil +is returned. -The default value of -.code *print-flo-precision* -is that of the -.code flo-dig -variable. +A valid +.code gethash +form serves as a place. It denotes either an existing value in a hash +table or a value that would be created by the evaluation of the form. +The +.meta alt +argument is meaningful when +.code gethash +is used as a place, and, if present, is always evaluated whenever the place is +evaluated. +In place update operations, it provides the initial value, which defaults +to +.code nil +if the argument is not specified. For example +.code "(inc (gethash h k d))" +will increment the value stored under key +.code k +in hash table +.code h +by one. If the key does not exist in the hash table, then +the value +.code "(+ 1 d)" +is inserted into the table under that key. +The expression +.code d +is always evaluated, whether or not its value is needed. -Note: to print floating-point values in such a way that their values -can be precisely recovered from the printed representation, it is -recommended to override -.code *print-flo-precision* -to the value of the -.code flo-max-dig -variable. +If a +.code gethash +place is subject to a deletion, but doesn't exist, it is not an error. +The operation does nothing, and +.code nil +is considered the prior value of the place yielded +by the deletion. -.coNP Special variable @ *print-flo-digits* +.coNP Function @ sethash +.synb +.mets (sethash < hash < key << value ) +.syne .desc The -.code *print-flo-precision* -special variable specifies the default floating-point printing -precision which is used when the -.code ~f -or -.code ~e -conversion specifier of the -.code format -function is used for printing a floating-point value, and no precision -is specified. +.code sethash +function places a value into +.meta hash +table under the given +.metn key . +If a similar key already exists in the hash table, then that key's +value is replaced by +.metn value . +Otherwise, the +.meta key +and +.meta value +pair is +newly inserted into +.metn hash . -Its default value is -.codn 3 . +The +.code sethash +function returns the +.meta value +argument. -.coNP Special variable @ *print-base* +.coNP Function @ pushhash +.synb +.mets (pushhash < hash < key << element ) +.syne .desc The -.code *print-base* -variable controls the base (radix) used for printing integer values. -It applies when the functions -.codn print , -.codn pprint , -.codn prinl , -.codn pprinl , -.code tostring -and -.code tostringp -process an integer value. -It also applies when the -.code ~a -and -.code ~s -conversion specifiers of the -.code format -function are used for printing an integer value. +.code pushhash +function is useful when the values stored in a hash table +are lists. If the given +.meta key +does not already exist in +.metn hash , +then a list of +length one is made which contains +.metn element , +and stored in +.meta hash +table under +.metn key . +If the +.meta key +already exists in the hash table, then the corresponding +value must be a list. The +.meta element +value is added to the front of that list, +and the extended list then becomes the new value under +.metn key . -The default value of the variable is -.codn 10 . +The return value is Boolean. If true, indicates that the hash table entry was +newly created. If false, it indicates that the push took place on an existing +entry. -Meaningful values are: -.codn 2 , -.codn 8 , -.code 10 -and -.codn 16 . +.coNP Function @ remhash +.synb +.mets (remhash < hash << key ) +.syne +.desc +The +.code remhash +function searches +.meta hash +for a key similar to the +.metn key . +If that key is found, then that key and its corresponding value are +removed from the hash table. -When base 16 is selected, hexadecimal digits are printed as upper-case -characters. +If the key is found and removal takes place, then the associated value +is returned. Otherwise +.code nil +is returned. -.coNP Special variable @ *print-circle* +.coNP Function @ clearhash +.synb +.mets (clearhash << hash ) +.syne .desc The -.code *print-circle* -variable is a Boolean which controls whether the circle notation is -in effect for printing aggregate objects: conses, ranges, vectors, hash tables -and structs. The initial value of this variable is -.codn nil : -circle notation printing is disabled. +.code clearhash +function removes all keys-value pairs from +.metn hash , +causing it to be empty. -The circle notation works for structs also, including structs which have -user-defined -.code print -methods. When a -.code print -method calls functions which print objects, such as -.codn print , -.code pprinl -or -.code format -on the same stream, the detection of circularity and substructure sharing -continues in these recursive invocations. +If +.meta hash +is already empty prior to the operation, then +.codn nil , +is returned. -However, there are limitations in the degree of support for circle notation -printing across -.code print -methods. Namely, a -.code print -method of a struct -.meta S -must not procure and submit for printing objects which are not part of the -ordinary structure that is reachable from the (static or instance) slots of -.metn S , -if those objects have already been printed prior to invoking the -.code print -method, and have been printed without a -.code #= -circle notation label. The "ordinary structure that is reachable from the -slots" denotes structure that is directly reachable by traversing conses, -ranges, vectors, hashes and struct slots: all printable aggregate objects. +Otherwise an integer is returned indicating the number of entries +that were purged from +.metn hash . -.coNP Function @ format +.coNP Function @ hash-count .synb -.mets (format < stream-designator < format-string << format-arg *) +.mets (hash-count << hash ) .syne .desc The -.code format -function performs output to a stream given by -.metn stream-designator , -by interpreting the actions implicit in a -.metn format-string , -incorporating material pulled from additional arguments given by -.mono -.meti << format-arg *. -.onom -Though the function is simple to invoke, there is complexity in format string -language, which is documented below. +.code hash-count +function returns an integer representing the number of +key-value pairs stored in +.metn hash . +.coNP Accessor @ hash-userdata +.synb +.mets (hash-userdata << hash ) +.mets (set (hash-userdata << hash ) << new-value ) +.syne +.desc The -.meta stream-designator -argument can be a stream object, or one of the values -.code t -or +.code hash-userdata +function retrieves the user data object associated with +.metn hash . + +A hash table can be created with user data using the +.code :userdata +keyword in a hash table literal or in a call to the +.code hash +function, directly, or via other hash-constructing functions which take the +hash construction keywords, such as +.codn group-by . +If a hash table is created without user data, its user +data is initialized to .codn nil . -The value -.code t -serves as a shorthand for -.codn *stdout* . -The value -.code nil -means that the function will send output into a newly instantiated string -output stream, and then return the resulting string. -.TP* "Format string syntax:" +Because +.code hash-userdata +is an accessor, a +.code hash-userdata +form can be used as a place. Assigning a value to this place +causes the user data of +.meta hash +to be replaced with that value. -Within -.metn format-string , -most characters represent themselves. Those -characters are simply output. The character -.code ~ -(tilde) introduces formatting -directives, which are denoted by a single character, usually a letter. +.coNP Function @ get-hash-userdata +.synb +.mets (get-hash-userdata << hash ) +.syne +.desc +The +.code get-hash-userdata +function is a deprecated synonym for +.codn hash-userdata . -The special sequence -.code ~~ -(tilde-tilde) encodes a single tilde. Nothing is -permitted between the two tildes. +.coNP Function @ set-hash-userdata +.synb +.mets (set-hash-userdata < hash << object ) +.syne +.desc +The +.code set-hash-userdata +replaces, with the +.metn object , +the user data object +associated with +.metn hash . -The syntax of a directive is generally as follows: +.coNP Function @ hashp +.synb +.mets (hashp << object ) +.syne +.desc +The +.code hashp +function returns +.code t +if the +.meta object +is a hash table, +otherwise it returns +.codn nil . -.mono -.mets <> ~[ width ] <> [, precision ] < letter -.onom +.coNP Function @ maphash +.synb +.mets (maphash < hash << binary-function ) +.syne +.desc +The +.code maphash +function successively invokes +.meta binary-function +for each entry stored in +.metn hash . +Each entry's key and value are passed as arguments +to +.codn binary-function . -In other words, the -.code ~ -(tilde) character, followed by a -.meta width -specifier, a -.meta precision -specifier introduced by a comma, -and a -.metn letter , -such that -.meta width -and -.meta precision -are independently optional: either or both may be omitted. -No whitespace is allowed between these elements. +The function returns +.codn nil . +.coNP Function @ hash-revget +.synb +.mets (hash-revget < hash < value >> [ testfun <> [ keyfun ]]) +.syne +.desc The -.meta letter -is a single alphabetic character which determines the -general action of the directive. The optional width and precision -are specified as follows: +.code hash-revget +function performs a reverse lookup on +.metn hash . -.RS -.meIP < width -The width specifier consists of an optional -.code < -(left angle bracket) character or -.code ^ -(caret) -character followed by an optional width specification. +It searches through the entries stored in +.meta hash +for an entry whose value matches +.metn value . -If the leading -.code < -character is present, then the printing will be left-adjusted within -this field. If the -.code ^ -character is present, the printing will be centered within the field. -Otherwise it will be right-adjusted by default. +If such an entry is found, that entry's key is returned. +Otherwise +.code nil +is returned. + +If multiple matching entries exist, it is not specified which entry's +key is returned. -The width can be specified as a decimal integer with an optional leading -minus sign, or as the character -.codn * . The -.code * -notation means that instead of digits, the value of the next argument is -consumed, and expected to be an integer which specifies the width. If the -width, specified either way, is negative, then the field will be left-adjusted. -If the value is positive, but either the -.code < -or -.code ^ -prefix character is present in the width -specifier, then the field is adjusted according to that character. +.meta keyfun +function is applied to each value in +.meta hash +and the resulting value is compared with +.metn value . +The default +.meta keyfun +is the +.code identity +function. -The padding calculations for alignment and centering take into account -character display width, as defined by the -.code display-width -function. For instance, a character string containing four Chinese -characters (kanji) has a display width of 8, not 4. - -The width specification does not restrict the printed portion of a datum. -Rather, for some kinds of conversions, it is the precision specification that -performs such truncation. A datum's display width (or that of its printed -portion, after such truncation is applied) can equal or exceed the specified -field width. In this situation it overflows the field: the printed portion is -rendered in its entirety without any padding applied on either side for -alignment or centering. - -.meIP < precision -The precision specifier is introduced by a leading comma. If this comma appears -immediately after the directive's -.code ~ -character, then it means that -.meta width -is being omitted; there is only a precision field. +The comparison is performed using +.metn testfun . -The precision specifier may begin with these optional characters: -.RS -.coIP 0 -(the "leading zero flag"), -.coIP + -(print a sign for positive values") -.IP space -(print a space in place of a positive sign). -.RE +The default +.meta testfun +is the +.code eql +function. -The precision specifier itself is either a decimal integer that does not -begin with a zero digit, or the -.code * -character. +.coNP Functions @ hash-eql and @ hash-equal +.synb +.mets (hash-eql << object ) +.mets (hash-equal < object <> [ hash-seed ]) +.syne +.desc +These functions each compute an integer hash value from the internal +representation of +.metn object , +which satisfies the following properties. +If two objects +.code A +and +.code B +are the same under the +.code eql +function, then +.code "(hash-eql A)" +and +.code "(hash-eql B)" +produce the same integer hash value. Similarly, +if two objects +.code A +and +.code B +are the same under the +.code equal +function, then +.code "(hash-equal A)" +and +.code "(hash-equal B)" +each produce the same integer hash value. In all other +circumstances, the hash values of two distinct objects are unrelated, and +may or may not be the same. -The precision field's components have a meaning which depends on the type of -object printed and the conversion specifier. +Object of struct type may support custom hashing by way of defining +an equality substitution via an +.code equal +method. See the Equality Substitution section under Structures. -For integer arguments, the precision value specifies the minimum number of digits -to print. If the precision field has a leading zero flag, then the integer is -padded with zeros to the required number of digits, otherwise the number is -padded with spaces instead of zeros. If zero or space padding is present, and -a leading positive or negative sign must be printed, then it is placed before -leading zeros, or after leading spaces, as the case may be. +The optional +.meta hash-seed +value perturbs the hashing function used by +.code hash-equal +for strings and buffer objects. This seed value must be a non-negative integer +no wider than 32 bits: that is, in the range 0 to 4294967295. +If the value isn't specified, it defaults to zero. +Effectively, each possible value of the seed specifies a different hashing +function. If two objects +.code A +and +.code B +are the same under the +.code equal +function, then +.code "(hash-equal A S)" +and +.code "(hash-equal B S)" +each produce the same integer hash value for any valid seed value +.codn S . -For floating-point values, the meaning of the precision value depends on which -specific conversion specifier -.cod1 ( f , -.codn e , -.code a -or -.codn s ) -is used. The details are -documented in the description of each of these, below. The leading zero flag is -also taken into account for floating-point values, and treated uniformly by -these directives. If the flag is present, then the printed value's integer -part will be padded with leading zeros up to the width of the field such that -one character of unused space remains in the field, in case a positive or -negative sign needs also to be rendered. +.coNP Functions @, hash_keys @, hash_values @ hash_pairs and @ hash_alist +.synb +.mets (hash-keys << hash ) +.mets (hash-values << hash ) +.mets (hash-pairs << hash ) +.mets (hash-alist << hash ) +.syne +.desc +These functions retrieve the bulk key-value data of hash table +.meta hash +in various ways. +.code hash-keys +retrieves a list of the keys. +.code hash-values +retrieves a list of the values. +.code hash-pairs +retrieves a list of pairs, +which are two-element lists consisting of the key, followed by the value. +Finally, +.code hash-alist +retrieves the key-value pairs as a Lisp association list: +a list of cons cells whose +.code car +fields are keys, and whose +.code cdr +fields are the values. Note that +.code hash-alist +returns the actual entries from the hash table, which are +conses. Modifying the +.code cdr +fields of these conses constitutes modifying the hash values +in the original hash table. Modifying the +.code car +fields interferes with the integrity of the hash table, +resulting in unspecified behavior for subsequent hash insertion +and lookup operations. -For integer or floating-point arguments, if the precision specifier has a -.code + -sign -among the special characters, then a -.code + -sign is printed for positive numbers. If -the precision specifier has a leading space instead of a -.code + -sign, then the -.code + -sign is rendered as a space for positive numbers. If there is no leading space -or -.codn + , -then a sign character is omitted for positive numbers. Negative -numbers are unconditionally prefixed with a -.code - -sign. +These functions all retrieve the keys and values in the +same order. For example, if the keys are retrieved with +.codn hash-keys , +and the values with +.codn hash-values , +then the corresponding entries from +each list pairwise correspond to the pairs in +.metn hash . -For all other objects, the precision specifies the maximum number of -print positions to occupy, taking into account the display width of each -character of the printed representation of the object, as according -to the -.code display-width -function. The object's printed representation is truncated, if necessary, to -the maximum number of characters which will not exceed the specified number of -print positions. +The list returned by each of these functions is lazy, and hence constitutes +an open traversal of the hash table. -.RE +.coNP Operator @ dohash +.synb +.mets (dohash >> ( key-var < value-var < hash-form <> [ result-form ]) +.mets \ \ << body-form *) +.syne +.desc +The +.code dohash +operator iterates over a hash table. The +.meta hash-form +expression must +evaluate to an object of hash table type. The +.meta key-var +and +.meta value-var +arguments must be symbols suitable for use as variable names. +Bindings are established for these variables over the scope of the +.metn body-form -s +and the optional +.metn result-form . -.TP* "Format directives:" -.RS -Format directives are case sensitive, so that for example -.code ~x +For each element in the hash table, the +.meta key-var and -.code ~X -have a -different effect, and -.code ~A -doesn't exist whereas -.code ~a -does. They are: +.meta value-var +variables are set to the key and value of that entry, respectively, +and each +.metn body-form , +if there are any, is evaluated. -.coIP a -Prints any object in an aesthetic way, as if by the -.code pprint -function. -The aesthetic notation violates read-print consistency: this notation -is not necessarily readable if it is implanted in \*(TX source code. -The field width specifier is honored, including the left-right adjustment -semantics. +When all of the entries of the table are thus processed, the +.meta result-form +is evaluated, and its return value becomes the return value of the dohash form. +If there is no +.metn result-form , +the return value is +.codn nil . -When this specifier is used for floating-point values, the precision specifies -the maximum number of total significant figures, which do not include any -digits in the exponent, if one is printed. Numbers are printed in exponential -notation if their magnitude is small, or else if their exponent exceeds their -precision. If the precision is not specified, then it is obtained from -the -.code *print-flo-precision* -special variable, whose default value is the same as that of the -.code flo-dig -variable. -Floating point values which are integers are -printed without a trailing -.code .0 -(point zero). The -.code + -flag in the precision is honored for rendering an explicit -.code + -sign on non-negative values. -If a leading zero is specified in the precision, and a nonzero width is -specified, then the printed value's integer part will be padded with leading -zeros up to one less than the field width. These zeros are placed before the -sign. - -.coIP s -Prints any object in a standard way, as if by the -.code print -function. Objects for -which read-print consistency is possible are printed in a way such that -if their notation is implanted in \*(TX source, they are readable. -The field width specifier is honored, including the left-right adjustment -semantics. The precision field is treated very similarly to the -.code ~a -format directive, except that non-exponentiated floating point numbers that -would be mistaken for integers include a trailing -.code .0 -for the sake of read-print -consistency. Objects truncated by precision may not have read-print -consistency. For instance, if a string object is truncated, it loses its -trailing closing quote, so that the resulting representation is no longer -a properly formed string object. For integer objects, the -.code *print-base* -variable is honored. Effectively, an integer is printed by the -.code s -directive as if by the -.codn b , -.codn o , -.codn d , +.meta result-form +and +.metn body-form -s +are in the scope of an implicit anonymous +block, which means that it is possible to terminate the execution of +dohash early using +.mono +.meti (return << value ) +.onom or -.code x -directive, depending on the value of the variable. - -.coIP d -Requires an argument of integer or character type type. The integer -value or character code is printed in decimal. - -.coIP x -Requires an argument of character or integer type. The integer value or -character code is printed in hexadecimal, using lower-case letters -for the digits -.code a -through -.codn f . -Width and precision semantics -are as described for the -.code a -format directive, for integers. - -.coIP X -Like the -.code x -directive, but the hexadecimal digits -.code a -through -.code f -are rendered in upper case. +.codn (return) . -.coIP o -Like the -.code x -directive, but octal is used instead of hexadecimal. +.coNP Functions @, hash-uni @, hash-diff @ hash-symdiff and @ hash-isec +.synb +.mets (hash-uni < hash1 < hash2 <> [ join-func ]) +.mets (hash-diff < hash1 << hash2 ) +.mets (hash-symdiff < hash1 << hash2 ) +.mets (hash-isec < hash1 < hash2 <> [ join-func ]) +.syne +.desc +These functions perform basic set operations on hash tables in a nondestructive +way, returning a new hash table without altering the inputs. The arguments +.meta hash1 +and +.meta hash2 +must be compatible hash tables. This means that their keys +must use the same kind of equality. -.coIP b -Like the -.code x -directive, but binary is used instead of hexadecimal. +The resulting hash table inherits attributes from +.metn hash1 , +as if created by the +.code make-similar-hash +function. If +.meta hash1 +has userdata, the resulting hash table +has the same userdata. If +.meta hash1 +has weak keys, the resulting table has weak +keys, and so forth. -.coIP f The -.code f -directive prints numbers in a fixed point decimal notation, with -a fixed number of digits after the decimal point. It requires a numeric -argument. (Unlike -.codn x , -.code X +.code hash-uni +function performs a set union. The resulting hash contains all of +the keys from +.meta hash1 +and all of the keys from +.metn hash2 , +and their corresponding +values. If a key occurs both in +.meta hash1 and -.codn o , -it does not allow an argument of character type). -The precision specifier gives the number of digits past the decimal point. -The number is rounded off to the specified precision, if necessary. -Furthermore, that many digits are always printed, regardless of the actual -precision of the number or its type. If it is omitted, then the value -is obtained from the special variable -.codn *print-flo-digits* , -whose default value is three: three digits past the decimal point. A precision -of zero means no digits pas the decimal point, and in this case the decimal -point is suppressed (regardless of whether the numeric argument is -floating-point or integer). - -.coIP e -The -.code e -directive prints numbers in exponential notation. It requires -a numeric argument. (Unlike -.codn x , -.code X +.metn hash2 , +then it occurs only once +in the resulting hash. In this case, if the +.meta join-func +argument is not given, +the value associated with this key is the one from +.metn hash1 . +If +.meta join-func +is specified then it is called with two arguments: the respective +data items from +.meta hash1 and -.codn o , -it does not allow an argument of character type). -The precision specifier gives the number of digits past the decimal point -printed in the exponential notation, not counting the digits in the exponent. -Exactly that many digits are printed, regardless of the precision of the -number. If the precision is omitted, then the number of digits after the -decimal point is obtained from the value of the special variable -.codn *print-flo-digits* , -whose default value is three. If the precision is zero, then a decimal portion -is truncated off entirely, including the decimal point. +.metn hash2 . +The return value of this function is used +as the value in the union hash. -.coIP p The -.code p -directive prints a numeric representation in hexadecimal of the bit pattern -of the object, which is meaningful to someone familiar with the internals -of \*(TX. If the object is a pointer to heaped data, that value -has a correspondence to its address. +.code hash-diff +function performs a set difference. First, a copy of +.meta hash1 +is made as if by the +.code copy-hash +function. Then from this copy, all keys which occur +in +.code hash2 +are deleted. -.coIP ! The -.code ! -directive establishes hanging indentation, and turns on the stream's -indentation mode. Subsequent lines printed within the execution of the -same -.code format -call will be automatically indented. If no width is specified, then -the directive sets the hanging indentation to the current printing -column position. If a width is specified, then it represents an offset -(positive or negative). If the -.code < -prefix character is present, the hanging indentation is set to the -specified offset relative to the current printing column. -If the -.code < -prefix is present on the width field, then the offset is applied -relative to the indentation which was saved on entry into the -.code format -function. - -The indentation mode and indentation column are automatically restored to their -previous values when -.code format -function terminates, naturally or via an exception or non-local jump. +.code hash-symdiff +function performs a symmetric difference. A new hash is returned which +contains all of the keys from +.meta hash1 +that are not in +.meta hash2 +and +.IR "vice versa" : +all of the keys from +.meta hash2 +that are not in +.metn hash1 . +The keys carry their corresponding values from +.meta hash1 +and +.metn hash2 , +respectively. -The effect of a precision field (even if zero) combined with the -.code ! -directive is currently not specified, and reserved for future extension. -The precision field is processed syntactically, and no error occurs, however. -.RE +The +.code hash-isec +function performs a set intersection. The resulting hash contains +only those keys which occur both in +.meta hash1 +and +.metn hash2 . +If +.meta join-func +is not +specified, the values selected for these common keys are those from +.metn hash1 . +If +.meta join-func +is specified, then for each key which occurs in both +.meta hash1 +and +.metn hash2 , +it is called with two arguments: the respective data items. The return +value is then used as the data item in the intersection hash. -.coNP Function @ fmt +.coNP Functions @ hash-subset and @ hash-proper-subset .synb -.mets (fmt < format-string << format-arg *) +.mets (hash-subset < hash1 << hash2 ) +.mets (hash-proper-subset < hash1 << hash2 ) .syne .desc The -.code fmt -function provides a shorthand for formatting to a string, according -to the following equivalence which holds between -.code fmt -and -.codn format : +.code hash-subset +function returns +.code t +if the keys in +.meta hash1 +are a subset of the keys in +.metn hash2 . -.verb - (fmt s arg ...) <--> (format nil s arg ...) -.brev +The +.code hash-proper-subset +function returns +.code t +if the keys in +.meta hash1 +are a proper subset of the keys in +.metn hash2 . +This means that +.meta hash2 +has all the keys which are in +.meta hash1 +and at least one which isn't. -.coNP Functions @, print @, pprint @, prinl @, pprinl @ tostring and @ tostringp +Note: the return value may not be mathematically meaningful if +.meta hash1 +and +.meta hash2 +use different equality. In any case, the actual behavior +may be understood as follows. The implementation of +.code hash-subset +tests whether each of the keys in +.meta hash1 +occurs in +.meta hash2 +using their respective equalities. +The implementation of +.code hash-proper-subset +applies +.code hash-subset +first, as above. If that is true, and the two hashes have the same number of +elements, the result is falsified. + +.coNP Functions @, hash-begin @ hash-next and @ hash-peek .synb -.mets (print < obj >> [ stream <> [ pretty-p ]]) -.mets (pprint < obj <> [ stream ]) -.mets (prinl < obj <> [ stream ]) -.mets (pprinl < obj <> [ stream ]) -.mets (tostring << obj ) -.mets (tostringp << obj ) +.mets (hash-begin << hash ) +.mets (hash-next << hash-iter ) +.mets (hash-peek << hash-iter ) .syne .desc The -.code print -and -.code pprint -functions render a printed character representation of the -.meta obj -argument into -.metn stream . - -If the -.meta stream -argument is not supplied, then -the destination is the stream currently stored in the -.code *stdout* -variable. - -If Boolean argument -.meta pretty-p -is not supplied or is explicitly specified as -.codn nil , -then the -.code print -function renders in a way which strives for read-print -consistency: an object is printed in a notation which is recognized as -a similar object of the same kind when it appears in \*(TX source code. -Floating-point objects are printed as if using the -.code format -function, with formatting controlled by the -.code *print-flo-format* -variable. - -If -.meta pretty-p -is true, then -.code print -does not strive for read-print consistency. -Strings are printed by sending their characters to the output -stream, as if by the -.code put-string -function, rather than being rendered in the string literal notation -consisting of double quotes, and escape sequences for control -characters. Likewise, character objects are printed via -.code put-char -rather than the -.code #\e -notation. Buffer objects are printed by sending their bytes to the -output stream using -.code put-byte -rather than being rendered in the -.code #b -notation. -Symbols are printed without their package prefix, except that -symbols from the keyword package are still printed with the leading colon. -Floating-point objects are printed as if using the -.code format -function, with formatting controlled by the -.code *pprint-flo-format* -variable. - -When aggregate objects like conses, ranges and vectors are printed, -the notations of these objects themselves are unaffected by the -.code pretty-p -flag; however, that flag is distributed to the elements. +.code hash-begin +function returns a an iterator object capable of retrieving the +entries in stored in +.meta hash +one by one. The -.code print -function returns -.metn obj . +.code hash-next +function's +.meta hash-iter +argument is a hash iterator returned by +.codn hash-begin . +If unvisited entries remain in +.metn hash , +then +.code hash-next +returns the next one as a cons cell whose +.code car +holds the key and whose +.code cdr +holds the value. That entry is then considered visited by the iterator. +If no more entries remain to be visited, +.code hash-next +returns +.codn nil . The -.code pprint -("pretty print") function is equivalent to -.codn print , -with the -.meta pretty-p -argument hard-coded true. +.code hash-peek +function returns the same value that a subsequent call to +.code hash-next +will return for the same +.metn hash-iter , +without changing the state of +.metn hash-iter . +That is to say, if a cell representing a hash entry is returned, that entry +remains unvisited by the iterator. +.coNP Macro @ with-hash-iter +.synb +.mets (with-hash-iter >> ( isym < hash-form >> [ ksym <> [ vsym ]]) +.mets \ \ << body-form *) +.syne +.desc The -.code prinl -function ("print and new line") behaves like a call to -.code print -with -.meta pretty-p -defaulting to -.codn nil , -followed by issuing a newline characters to the stream. +.code with-hash-iter +macro evaluates +.metn body-form -s +in an environment in which a lexically scoped function is visible. -The -.code pprinl -function ("pretty print and new line") behaves like -.code pprint -followed by issuing a newline to the stream. +The function is named by +.meta isym +which must be a symbol suitable for naming functions with +.codn flet . The -.code tostring -and -.code tostringp -functions are like -.code print -and -.codn pprint , -but they do not accept a stream argument. Instead they print to a freshly -instantiated string stream, and return the resulting string. - -The following equivalences hold between calls to the -.code format -function and calls to the above functions: +.meta hash-form +argument must be a form which evaluates to a hash table object. -.verb - (format stream "~s" obj) <--> (print obj stream) - (format t "~s" obj) <--> (print obj) - (format t "~s\en" obj) <--> (prinl obj) - (format nil "~s" obj) <--> (tostring obj) -.brev +Invocations of the function retrieve successive entries of the hash table +as cons cell pairs of keys and values. The function returns +.code nil +to indicate no more entries remain. -For -.codn pprint , -.code tostringp -and -.codn pprinl , -the equivalence is produced by using -.code ~a -in format rather than -.codn ~s . +If either of the +.meta ksym +or +.meta vsym +arguments are present, they must be symbols suitable as variable names. They +are bound as variables visible to +.metn body-form -s, +initialized to the value +.codn nil . -.TP* Notes: -For floating-point numbers, the above description of the behavior in -terms of the format specifiers -.code ~s -and -.code ~a -only applies with respect to the default values of the variables -.code *print-flo-format* -and -.codn *pprint-flo-format* . +If +.meta ksym +is specified, then whenever the function +.meta isym +macro is invoked and retrieves a hash table entry, the +.meta ksym +variable is set to the key. If the function returns +.code nil +then the value of +.meta ksym +is set to +.codn nil . -For characters, the print function behaves as follows: most control -characters in the Unicode -.code C0 -and -.code C1 -range are rendered using the -.code #\ex -notation, -using two hex digits. Codes in the range -.code D800 +Similarly, if +.meta vsym +is specified, then the function stores the retrieved +hash value in that variable, or else sets the variable to -.codn DFFF , -and the codes -.code FFFE -and -.code FFFF -are printed in the -.code #\exNNNN -with four hexadecimal digits, and -character above this range are printed using the same notation, but with six -hexadecimal digits. Certain characters in the -.code C0 -range are printed using -their names such as -.code #\enul -and -.codn #\ereturn , -which are documented -in the Character Literals section. -The -.code DC00 -character is printed as -.codn #\epnul . -All other characters are printed as -.mono -.meti >> #\e char -.onom -where -.meta char -is the actual character. - -Caution: read-print consistency is affected by trailing material. If additional -digits are printed immediately after a number without intervening whitespace, -they extend that number. If hex digits are printed after the character -.codn x , -which is rendered as -.codn #\ex , -they look like a hex character code. +.code nil +if there is no next value. -.coNP Function @ tprint -.synb -.mets (tprint < obj <> [ stream ]) -.syne +.coNP Special variable @ *hash-seed* .desc The -.code tprint -function prints a representation of -.meta obj -on -.metn stream . +.code *hash-seed* +special variable is initialized with a value of zero. Whenever a new +hash table is explicitly or implicitly created, it takes its seed from +the value of the +.code *hash-seed* +variable in the current dynamic environment. -If the stream argument is not supplied, then -the destination is the stream currently stored in the -.code *stdout* -variable. +The only situation in which +.code *hash-seed* +is not used when creating a new hash table is when +.code make-hash +is called with an argument given for the optional +.meta hash-seed +argument. -For all object types except lists and vectors, -.code tprint -behaves like -.codn pprinl . +Only +.codn equal -based +hash tables make use of their seed, and only for keys which are strings and +buffers. The purpose of the seed is to scramble the hashing function, to make +a hash table resistant to a type of denial-of-service attack, whereby a +malicious input causes a hash table to be populated with a large number of keys +which all map to the same hash table chain, causing the performance to severely +degrade. -If -.code obj -is a list or vector, then -.code tprint -recurses: the -.code tprint -function is applied to each element. An empty list or vector -results in no output at all. This effectively means that an arbitrarily nested -structure of lists and vectors is printed flattened, with one element on each -line. +The value of +.code *hash-seed* +must be a non-negative integer, no wider than 32 bits. -.coNP Function @ display-width +.coNP Function @ gen-hash-seed .synb -.mets (display-width << char ) -.mets (display-width << string ) +.mets (gen-hash-seed) .syne .desc The -.code display-width -function calculates the number of places occupied by the printed representation -of -.meta char -or -.meta string -on a monospace display which renders certain characters, such as the East Asian -kanji and other characters, using two places. - -For a -.meta string -argument, this value is the sum of the individual display width of the -string's constituent characters. The display width of an empty string is zero. - -Control characters are assigned a display width of zero, regardless of -their display control semantics, if any. - -Characters marked by Unicode as being wide or full width, have a display -width of two. Other characters have a display width of one. +.code gen-hash-seed +function returns an integer value suitable for the +.code *hash-seed* +variable, or as the +.code hash-seed +argument of the +.code make-hash +and +.code hash-equal +functions. -.coNP Function @ streamp -.synb -.mets (streamp << obj ) -.syne -.desc -The -.code streamp -function returns -.code t -if -.meta obj -is any type of stream. Otherwise it returns -.codn nil . +The value is derived from the host environment, from information such +as the process ID and time of day. -.coNP Function @ real-time-stream-p +.SS* Partial Evaluation and Combinators +.coNP Macros @ op and @ do .synb -.mets (real-time-stream-p << obj ) +.mets (op << form +) +.mets (do << form +) .syne .desc The -.code real-time-streamp-p -function returns -.code t -if -.meta obj -is a stream marked as -"real-time". If -.meta obj -is not a stream, or not a stream marked as "real-time", -then it returns -.codn nil . - -Only certain kinds of streams accept the real-time attribute: file streams and -tail streams. This attribute controls the semantics of the application of -.code lazy-stream-cons -to the stream. For a real-time stream, -.code lazy-stream-cons -returns a stream with "naive" semantics which returns data as soon as it is -available, at the cost of generating spurious -.code nil -item when the stream -terminates. The application has to recognize and discard that -.code nil -item. -The ordinary lazy streams read ahead by one line and suppress this extra -item, so their representation is more accurate. +.code op +and +.code do +macro operators are similar. -When \*(TX starts up, it automatically marks the -.code *std-input* -stream as real-time, if it is connected to a TTY device (a device for which -the POSIX function -.code isatty -reports true). This is only supported on platforms that have this function. -The behavior is overridden by the -.code -n -command line option. +Like the lambda operator, the +.code op +operator creates an anonymous function based on its syntax. +The difference is that the arguments of the function are implicit, or +optionally specified within the function body, rather than as a formal +parameter list before the body. -.coNP Function @ open-file -.synb -.mets (open-file < path <> [ mode-string ]) -.syne -.desc -The -.code open-file -function creates a stream connected to the file -which is located at the given -.metn path , -which is a string. +Also, the +.meta form +arguments of +.code op +are implicitly turned into a DWIM expression, +which means that argument evaluation follows Lisp-1 rules. (See the +.code dwim +operator). The -.meta mode-string -argument is a string which uses the same -conventions as the mode argument of the C language -.code fopen -function, with greater permissiveness, and some extensions. - -The syntax of mode-string is described by the following -grammar. Note that it permits no whitespace characters: +.code do +operator is like the +.code op +operator with the following difference: +the +.meta form +arguments of +.code do +are not implicitly treated as DWIM expressions, +but as ordinary expressions. In particular, this means that operator +syntax is permitted. Note that the syntax +.code "(op @1)" +makes sense, since +the argument can be a function, which will be invoked, but +.code "(do @1)" +doesn't +make sense because it will produce a Lisp-2 form like +.code "(#:arg1 ...)" +referring +to nonexistent function +.codn #:arg1 . +Because it accepts operators, +.code do +can be used with imperative constructs +which are not functions, like set: like set: for instance +.code "(do set x)" +produces +an anonymous function which, if called with one argument, stores that argument +into +.codn x . +The argument forms are arbitrary expressions, within which a special +convention is permitted: +.RS +.meIP >> @ num +A number preceded by a +.code @ +is a metanumber. This is a special syntax +which denotes an argument. For instance +.code @2 +means that the second argument of +the anonymous function is to be substituted in place of the +.codn @2 . +.code op +generates a function which has a number of required arguments equal to the +highest value of +.meta num +appearing in a .mono -.mets < mode-string := [ < mode ] [ < options ] -.mets < mode := { < selector [ + ] | + } -.mets < selector := { r | w | a } -.mets < options := { b | l | u | < digit } -.mets < digit := { 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 } +.meti >> @ num .onom - -If the -.meta mode-string -argument is omitted, the behavior is the same as an empty -mode string. - -The -.meta mode -part of the mode string generates the following possibilities: -.RS -.meIP empty -If the -.meta mode -is missing, then a default mode is implied. The default -is specific to the particular stream-opening function. In -the case of -.codn open-file , -the default mode is -.codn r . -.coIP + -A -.meta mode -consisting of just the -.code + -character is equivalent to -.codn r+ . -.coIP r -This -.meta mode -means that the file is opened for reading. -.coIP r+ -The file is opened for reading and writing. It is not created -if it doesn't exist. -.coIP w -The file is opened for writing. If it exists, it is truncated -to zero length. If it doesn't exist, it is created. -.coIP w+ -The file is opened for reading and writing. If it exists, it -is truncated to zero length. If it doesn't exist, it is created. -.coIP a -The file is opened for writing. If it doesn't exist, it is -created. If it exists, the current position is advanced to -one byte past the end of the file, so that newly written data -are appended. -.coIP a+ -The file is opened for reading and writing. If it doesn't exist, -it is created. The read position is at the beginning of the file, -but writes are appended to the end regardless of the position. +construct in the body. For instance +.code "(op car @3)" +generates a three-argument function (which passes its third +argument to +.codn car , +returning the result, and ignores its first two arguments). +There is no way to use +.code op +to generate functions which have optional arguments. +.meIP < @rest +If the meta-symbol +.meta @rest +appears in the +.code op +syntax, it explicitly denotes the list of trailing arguments, +allowing them to be placed anywhere in the expression. .RE + .IP -The meanings of the option characters are: -.RS -.coIP b -The file is opened in binary mode: no line ending translation takes place. -In the absence of this option, files are opened in text mode, in which newline -characters in the stream are an abstract indication of the end of a line, -translate to a system-specific way of terminating lines in text files. -.coIP l -Specifies that the stream will be line buffered. This means that an implicit -flush operation takes place whenever the newline character is output. -.coIP u -Specifies that the stream will be unbuffered. It is erroneous for both -.code l -and -.code u -to be specified. -.coIP i -Specifies that the stream will have the real-time -property set. For a description of the semantics, see the -.code real-time-stream-p -function. Briefly, this property affects the semantics of lazy lists which draw -input from the stream. -In addition, for a stream opened for writing or reading and writing, the -.code i -mode letter specifies that the stream will be line buffered, unless -specified as unbuffered with -.codn u . -.meIP digit -A decimal digit specifies the the stream buffer size -as binary exponential buffer size order, such that -.code 0 -specifies 1024 bytes, -.code 1 -specifies 2048 and so forth up to -.code 9 -specifying 524288 bytes. If no such digit is specified, then the -stream uses a default buffer size. It is erroneous for the -size order digit to be present together with the option -.codn u . -.RE +Functions generated by +.code op +are always variadic; they always take additional arguments after +any required ones, whether or not the +.meta @rest +syntax is used. -.coNP Function @ open-tail -.synb -.mets (open-tail < path >> [ mode-string <> [ seek-to-end-p ]]) -.syne -.desc -The -.code open-tail -function creates a tail stream connected to the file which is -located at the given -.metn path . -The -.meta mode-string -argument is a string which uses -the same conventions as the mode argument of the C language -.code fopen -function. If this argument is omitted, then -.str r -is used. -See the -.code open-file -function for a discussion of modes. +If the body does not contain +any +.meta @num +or +.meta @rest +syntax, then +.code @rest +is implicitly inserted. What this means is that, for example, since +the form +.code "(op foo)" +does not contain any numeric positional arguments like +.codn @1 , +and does not contain +.codn @rest , +it is actually a shorthand for +.codn "(op foo . @rest)" : +a function which applies all of its arguments to +.codn foo . +If the body does contain at least one +.meta @num +or +.metn @rest , +then +.meta @rest +isn't implicitly inserted. The notation +.code "(op foo @1)" +denotes a function which takes any number of arguments, and ignores +all but the first one, which is passed to +.codn foo . -The -.code seek-to-end-p -argument is a Boolean which determines whether the initial -read/write position is at the start of the file, or just past the end. -It defaults to -.codn nil . -This argument only makes a difference if the file exists -at the time -.code open-tail -is called. If the file does not exist, and is later -created, then the tail stream will follow that file from the beginning. In -other words, -.meta seek-to-end-p -controls whether the tail stream reads all the -existing data in the file, if any, or whether it reads only newly added data -from approximately the time the stream is created. +The actions of +.code op +be understood by these examples, which show +how +.code op +is rewritten to lambda. However, note that the real translator +uses generated symbols for the arguments, which are not equal to any +symbols in the program. -A tail stream has special semantics with regard to reading at the end -of file. A tail stream never reports an end-of-file condition; instead -it polls the file until more data is added. Furthermore, if the file -is truncated, or replaced with a smaller file, the tail stream follows -this change: it automatically opens the smaller file and starts reading from -the beginning (the -.meta seek-to-end-p -flag only applies to the initial open). -In this manner, a tail stream can dynamically growing rotating log files. +.verb + (op) -> invalid -Caveat: since a tail stream can re-open a new file which has the same -name as the original file, it behave incorrectly if the program -changes the current working directory, and the path name is relative. + (op +) -> (lambda rest [+ . rest]) -.coNP Function @ open-directory -.synb -.mets (open-directory << path ) -.syne -.desc -The -.code open-directory -function tries to create a stream which reads the -directory given by the string argument -.metn path . -If a filesystem object exists -under the path, is accessible, and is a directory, then the function -returns a stream. Otherwise, a file error exception is thrown. + (op + foo) -> (lambda rest [+ foo . rest]) + + (op @1 @2) -> (lambda (arg1 arg2 . rest) [arg1 arg2]) + + (op @1 . @rest) -> (lambda (arg1 . rest) [arg1 . @rest]) + + (op @1 @rest) -> (lambda (arg1 . rest) [arg1 @rest]) + + (op @1 @2) -> (lambda (arg1 arg2 . rest) [arg1 arg2]) + + (op foo @1 (@2) (bar @3)) -> (lambda (arg1 arg2 arg3 . rest) + [foo arg1 (arg2) (bar arg3)]) + + (op foo @rest @1) -> (lambda (arg1 . rest) [foo rest arg1]) + + (do + foo) -> (lambda rest (+ foo . rest)) + + (do @1 @2) -> (lambda (arg1 arg2 . rest) (arg1 arg2)) + + (do foo @rest @1) -> (lambda (arg1 . rest) (foo rest arg1)) +.brev + +Note that if argument +.meta @n +appears, it is not necessary +for arguments +.meta @1 +through +.meta @n-1 +to appear. The function will have +.code n +arguments: + +.verb + (op @3) -> (lambda (arg1 arg2 arg3 . rest) [arg3]) +.brev -The resulting stream supports the get-line operation. Each call to the -.code get-line -operation retrieves a string representing the next directory -entry. The value -.code nil -is returned when there are no more directory entries. The -.code . +.code op and -.code .. -entries in Unix filesystems are not skipped. +.code do +operators can be nested, in any combination. This raises the +question: if a metanumber like +.code @1 +or +.code @rest +occurs in an +.code op +that is nested +within an +.codn op , +what is the meaning? -.coNP Function @ make-string-input-stream -.synb -.mets (make-string-input-stream << string ) -.syne -.desc +A metanumber always belongs with the inner-most op or do operator. So for +instance +.code "(op (op @1))" +means that an +.code "(op @1)" +expression is nested +within an +.code op +expression which itself contains no meta-syntax. The -.code make-string-input-stream -function produces an input stream object. Character read operations on the -stream object read successive characters from -.metn string . -Output operations and byte operations are not supported. +.code @1 +belongs with the inner op. -.coNP Function @ make-string-byte-input-stream -.synb -.mets (make-string-byte-input-stream << string ) -.syne -.desc -The -.code make-string-byte-input-stream -function produces an input stream object. Byte read operations on -this stream object read successive byte values obtained by encoding -.meta string -into UTF-8. Character read operations are not supported, and neither -are output operations. +There is a way for an inner +.code op +to refer to an outer op metanumber argument. This is +expressed by adding an extra +.code @ +prefix for every level of escape. For example in +.code "(op (op @@1))" +the +.code @@1 +belongs to the outer +.codn op : +it is the same as +.code @1 +appearing in the outer +.codn op . +That is to say, +in the expression +.codn "(op @1 (op @@1))" , +the +.code @1 +and +.code @@1 +are the same thing: +both are parameter 1 of the lambda function generated by the outer +.codn op . +By contrast, in the expression +.code "(op @1 (op @1))" +there are two different parameters: +the first +.code @1 +is argument of the outer function, and the second +.code @1 +is the first argument of the inner function. Of course, if there +are three levels of nesting, then three +.code @ +meta-prefixes are needed to insert +a parameter from the outermost +.code op +into the innermost +.codn op . -.coNP Function @ make-strlist-input-stream -.synb -.mets (make-strlist-input-stream << list ) -.syne -.desc -The -.code make-strlist-input-stream -function produces an input stream object based on a list of strings. -Through the character read operations invoked on this stream, -the list of strings appears as a list of newline-terminated lines. -Output operations and byte operations are not supported. +Note that meta-numbers and meta-symbols belonging to an +.code op +can be used in the dot position of a function call, such as: -.coNP Function @ make-string-output-stream -.synb -.mets (make-string-output-stream) -.syne -.desc -The -.code make-string-output-stream -function, which takes no arguments, creates a string output stream. -Data sent to this stream is accumulated into a string object. -String output streams supports both character and byte output operations. -Bytes are assumed to represent a UTF-8 encoding, and are decoded in order -to form characters which are stored into the string. +.verb + [(op list 1 . @1) 2] -> (1 . 2) +.brev -If an incomplete UTF-8 code is output, and a character output operation then -takes place, that code is assumed to be terminated and is decoded as invalid -bytes. The UTF-8 decoding machine is reset and ready for the start of a new -code. +This is a consequence of the special transformations described +in the paragraph +.B "Dot Position in Function Calls" +in the subsection +.B "Additional Syntax" +of the +.BR "TXR Lisp" +section. The -.code get-string-from-stream -function is used to retrieve the accumulated string. +.code op +syntax works in conjunction with quasiliterals which are nested within it. +The metanumber notation as well as +.code @rest +are recognized without requiring an additional +.code @ +escape, which is effectively optional: -If the null character is written to a string output stream, the behavior -is unspecified. \*(TX strings cannot contain null bytes. A the pseudo-null -character -.codn #\exDC00 , -also notated -.codn #\epnul , -will produce a null byte when converted to UTF-8 and thus serves as an -effective internal representation of the null character in external data. +.verb + (apply (op list `@1-@rest`) '(1 2 3)) -> "1-2 3" -.coNP Function @ get-string-from-stream -.synb -.mets (get-string-from-stream << stream ) -.syne -.desc -The -.meta stream -argument must be a string output stream. This function finalizes -the data sent to the stream and retrieves the accumulated character string. + (apply (op list `@@1-@@rest`) '(1 2 3)) -> "1-2 3" +.brev -If a partial UTF-8 code has been written to -.metn stream , -and then this -function is called, the byte stream is considered complete and the partial -code is decoded as invalid bytes. +Though they produce the same result, the above two examples differ in that +.code @rest +embeds a metasymbol into the quasiliteral structure, whereas +.code @@rest +embeds the Lisp expression +.code @rest +into the quasiliteral. Either way, in the scope of +.codn op , +.code @rest +undergoes the macro-expansion which renames it to the machine-generated +function argument symbol of the implicit function denoted by the +.code op +macro form. -After this function is called, further output on the stream is not possible. +This convenient omission of the +.code @ +character isn't supported for reaching the arguments of an outer +.code op +from a quasiliteral within a nested +.codn op : -.coNP Function @ make-strlist-output-stream -.synb -.mets (make-strlist-output-stream) -.syne -.desc -The -.code make-strlist-output-stream -function is very similar to -.codn make-string-output-stream . -However, the stream object produced by this function does not produce a string, -but a list of strings. The data is broken into multiple strings by newline -characters written to the stream. Newline characters do not appear in the -string list. Also, byte output operations are not supported. +.verb + ;; To reach @@1, @@@1 must be written. + ;; @@1 Lisp expression introduced by @. + (op ... (op ... `@@@1`)) +.brev -.coNP Function @ get-list-from-stream +.TP* Example: + +.verb + ;; Take a list of pairs and produce a list in which those pairs + ;; are reversed. + + (mapcar (op list @2 @1) '((1 2) (a b))) -> ((2 1) (b a)) +.brev + +.coNP Macro @ lop .synb -.mets (get-list-from-stream << stream ) +.mets (lop << form +) .syne .desc The -.code get-list-from-stream -function returns the string list which has accumulated inside -a string output stream given by -.metn stream . -The string output stream is -finalized, so that further output is no longer possible. +.code lop +macro is variant of +.code op +with special semantics. -.coNP Macro @ with-in-string-stream -.synb -.mets (with-in-string-stream >> ( stream-var << string ) -.mets \ \ << body-form *) -.syne -.desc The -.code with-in-string-stream -macro binds the symbol -.meta stream-var -as a variable, initializing it with a newly created -string input stream. The string input stream is -constructed from -.meta string -as if by the -.mono -.meti (make-string-input-stream << string ) -.onom -expression. +.meta form +arguments support the same notation as those of the +.code op +operator. -Then it evaluates the -.metn body-form -s -in the scope of the variable. +If only one +.meta form +is given then +.code lop +is equivalent to +.codn op . -The value of the last -.meta body-form -is returned, or else -.code nil -if no forms are present. +If two or more +.meta form +arguments are present, then +.code lop +generates a variadic function which inserts all of its trailing +arguments between the first and second +.metn form -s. + +That is to say, trailing arguments coming into the anonymous function +become the left arguments of the function or function-like object +denoted by the first +.meta form +and the remaining +.metn form -s +give additional arguments. Hence the name +.codn lop , +which stands for \(dqleft-inserting +.codn op \(dq. + +This left insertion of the trailing arguments takes place regardless of whether +.code @rest +occurs in any +.metn form . The -.meta stream-var -argument must be a bindable symbol, -as defined by the -.code bindable -function. +.meta form +syntax determines the number of required arguments of the +generated function, according to the highest-valued meta-number. The trailing +arguments which are inserted into the left position are any arguments in excess +of the required arguments. The -.meta string -argument must be a form -which evaluates to a character string value. +.code lop +macro's expansion can be understood via the following equivalences, +except that in the real implementation, the symbols +.code rest +and +.code arg1 +through +.code arg3 +are replaced with hygienic, unique symbols. -.coNP Macro @ with-in-string-byte-stream +.verb + (lop f) <--> (op f) <--> (lambda (. rest) [f . rest]) + + (lop f x y) <--> (lambda (. rest) + [apply f (append rest [list x y])]) + + (lop f x @3 y) <--> (lambda (arg1 arg2 arg3 . rest) + [apply f + (append rest + [list x arg3 y])]) +.brev + +.TP* Examples: + +.verb + (mapcar (lop list 3) '(a b c)) --> ((a 3) (b 3) (c 3)) + + (mapcar (lop list @1) '(a b c)) --> ((a) (b) (c)) + + (mapcar (lop list @1) '(a b c) '(d e f)) + --> ((d a) (e b) (f c)) + +.brev + +.coNP Macros @, ap @, ip @ ado and @ ido. .synb -.mets (with-in-string-byte-stream >> ( stream-var << string ) -.mets \ \ << body-form *) +.mets (ap << form +) +.mets (ip << form +) +.mets (ado << form +) +.mets (ido << form +) .syne .desc The -.code with-in-string-byte-stream -macro binds the symbol -.meta stream-var -as a variable, initializing it with a newly created -string byte input stream. The string input stream is -constructed from -.meta string -as if by the -.mono -.meti (make-string-byte-input-stream << string ) -.onom -expression. +.code ap +macro is based on the +.code op +macro and has identical argument +conventions. -Then it evaluates the -.metn body-form -s -in the scope of the variable. +The +.code ap +macro analyzes its arguments and produces a function +.metn f , +in exactly the same same way as the +.code op +macro. However, instead of returning +.metn f , +directly, it returns a different function +.metn g , +which is a one-argument function which accepts a list, +and then applies the list as arguments to +.metn f . -The value of the last -.meta body-form -is returned, or else -.code nil -if no forms are present. +In other words, the following equivalence holds: -The -.meta string -argument must be a form -which evaluates to a character string value. +.verb + (ap form ...) <--> (apf (op form ...)) +.brev -.coNP Macro @ with-out-string-stream -.synb -.mets (with-out-string-stream <> ( stream-var ) << body-form *) -.syne -.desc The -.code with-out-string-stream -macro binds the symbol specified -by the -.meta stream-var -argument as a variable, initializing it -with a newly created string output stream. The output -stream is created as if by the -.code make-string-output-stream -function. +.code ap +macro nests properly with +.code op +and +.codn do , +in any combination, in regard to the +.meta ...@@n +notation. -Then it evaluates -.metn body-form -s -in the scope of that variable. +The +.code ip +macro is very similar to the +.code ap +macro, except that it is based +on the semantics of the function +.code iapply +rather than +.codn apply , +according +to the following equivalence: -After these forms are evaluated, the string is extracted -from the string output stream, as if by the -.code get-string-from-stream -function, and returned as the result value -of the form. +.verb + (ip form ...) <--> (ipf (op form ...)) +.brev -.coNP Macro @ with-out-strlist-stream -.synb -.mets (with-out-strlist-stream <> ( stream-var ) << body-form *) -.syne -.desc The -.code with-out-strlist-stream -macro binds the symbol specified -by the -.meta stream-var -argument as a variable, initializing it -with a newly created string list output stream. The output -stream is created as if by the -.code make-strlist-output-stream -function. +.code ado +and +.code ido +macros are related to do macro in the same way that +.code ap +and +.code ip +are related to +.codn op . +They produce a one-argument function which works +as if by applying its arguments to the function generated by do, +according to the following equivalence: -Then it evaluates -.metn body-form -s -in the scope of that variable. +.verb + (ado form ...) <--> (apf (do form ...)) -After these forms are evaluated, the string list is extracted -from the string output stream, as if by the -.code get-strlist-from-stream -function, and returned as the result value -of the form. + (ido form ...) <--> (ipf (do form ...)) +.brev -.coNP Function @ close-stream +See also: the +.code apf +and +.code ipf +functions. + +.coNP Macros @ opip and @ oand .synb -.mets (close-stream < stream <> [ throw-on-error-p ]) +.mets (opip << clause *) +.mets (oand << clause *) .syne .desc The -.code close-stream -function performs a close operation on -.metn stream , -whose meaning is depends on the type of the stream. For some types of streams, -such as string streams, it does nothing. For streams which are connected -to operating system files or devices, will perform a close of the underlying -file descriptor, and dissociate that descriptor from the stream. Any buffered -data is flushed first. +.code opip +and +.code oand +operators make it possible to chain together functions which are expressed +using the +.code op +syntax. (See the +.code op +operator for more information). -.code close-stream -returns a Boolean true value if the close has occurred without -errors, otherwise -.codn nil . +Both macros perform the same transformation except that +.code opip +translates its arguments to a call to the +.code chain +function, whereas +.code oand +translates its arguments in the same way to a call to the +.code chand +function. -For most streams, "without errors" means that any buffered output data is -flushed successfully. +More precisely, these macros perform the following rewrites: -For command and process pipes (see open-command and open-process), success also -means that the process terminates normally, with a successful error code, or an -unsuccessful one. An abnormal termination is considered an error, as -as is the inability to retrieve the termination status, as well as the situation -that the process continues running in spite of the close attempt. -Detecting these situations is platform specific. +.verb + (opip arg1 arg2 ... argn) -> [chain {arg1} {arg2} ... {argn}] + (oand arg1 arg2 ... argn) -> [chand {arg1} {arg2} ... {argn}] +.brev -If the -.meta throw-on-error-p -argument is specified, and isn't -.codn nil , -then the -function throws an exception if an error occurs during the close operation -instead of returning -.codn nil . +where the above +.code {arg} +notation denotes the following transformation applied to each argument: -.coNP Macro @ with-stream -.synb -.mets (with-stream >> ( stream-var << init-form ) -.mets \ \ << body-form *) -.syne -.desc -The -.code with-stream -binds the variable whose name is given by the -.meta stream-var -argument, and macro arranges for the evaluation of -.metn body-form -s -in the scope of that variable. +.verb + (function ...) -> (op function ...) + (operator ...) -> (do operator ...) + (macro ...) -> (do macro ...) + (dwim ...) -> (dwim ...) + [...] -> [...] + (qref ...) -> (qref ...) + (uref ...) -> (uref ...) + .slot -> .slot + .(method ...) -> .(method ...) + atom -> atom +.brev -The variable is initialized with the value produced -by the evaluation of -.meta init-form -which must be an expression which evaluates to a stream. +In other words, compound forms whose leftmost symbol is a macro or operator +are translated to the +.code do +notation. Compound forms denoting function calls are translated to the +.code op +notation. Compound forms which are +.code dwim +invocations, either explicit or via the DWIM brackets notation, are +used without transformation. Used without transformation also are forms +denoting struct slot access, either explicitly using +.code uref +or +.code qref +or the respective dot notations, as well as any atom forms. + +Note: the +.code opip +and +.code oand +macros use their macro environment in determining whether a form is a +macro call, thereby respecting lexical scoping. + +.TP* Example: +Take each element from the list +.code "(1 2 3 4)" +and multiply it by three, then add 1. +If the result is odd, collect that into the resulting list: -After each -.meta body-form -is evaluated, the stream is closed, as if by the .mono -.meti (close-stream << stream-var ) +(mappend (opip (* 3) + (+ 1) + [iff oddp list]) + (range 1 4)) .onom -expression. -The value of the last -.meta body-form -then becomes the result value of the form, -or else -.code nil -if these forms are absent. +The above is equivalent to: -If the evaluation of the -.metn body-form -s -is abandoned, the stream is still closed. That is to say, -the closure of the stream is a protected action, as if by -the -.code unwind-protect -operator. +.mono +(mappend (chain (op * 3) + (op + 1) + [iff oddp list]) + (range 1 4)) +.onom -.coNP Functions @, get-error @ get-error-str and @ clear-error +The +.code "(* 3)" +and +.code "(+ 1)" +terms are rewritten to +.code "(op * 3)" +and +.codn "(op + 1)" , +respectively, whereas +.code "[iff oddp list]" +is passed through untransformed. + +.coNP Macro @ ret .synb -.mets (get-error << stream ) -.mets (get-error-str << stream ) -.mets (clear-error << stream ) +.mets (ret << form ) .syne .desc -When a stream operation fails, the -.code get-error -and -.code get-error-str -functions may be used to inquire about a more detailed cause of the error. - -Not all streams support these functions to the same extent. For instance, -string input streams have no persistent state. The only error which occurs -is the condition when the string has no more data. +The +.code ret +macro's +.meta form +argument is treated similarly to the second and subsequent arguments of the +.code op +operator. The -.code get-error -inquires -.meta stream -about its error condition. +.code ret +macro produces a function which takes any number of arguments, +and returns the value specified by +.metn form . -The function returns -.code nil -to indicate there is no error condition, -.code t -to indicate an end-of-data condition, -or else a value which is specific to the stream type indicating the -specific error type. +.meta form +can contain +.code op +meta syntax like +.code @n +and +.codn @rest . -Note: for some streams, it is possible for the -.code t -value to be returned even though no operation has failed; that is to say, the -streams "know" they are at the end of the data even though no read operation -has failed. Code which depends on this will not work with streams which -do not thus indicate the end-of-data -.I a priori, -but by means of a read operation which fails. +The following equivalence holds: -The -.code get-error-str -function returns a text representation of the error code. The -.code nil -error code is represented as the string -.codn "no error" ; -the -.code t -error code as -.code "eof" -and other codes have a stream-specific representation. +.verb + (ret x) <--> (op identity x)) +.brev -The -.code clear-error -function removes the error situation from a stream. On some streams, it does -nothing. If an error has occurred on a stream, this function should be called -prior to re-trying any I/O or positioning operations. -The return value is the previous error code, or -.code nil -if there was no error, or the operation is not supported on the stream. +Thus the expression +.code "(ret @2)" +returns a function similar to +.codn "(lambda (x y . z) y)" , +and the expression +.code "(ret 42)" +returns a function similar to +.codn "(lambda (. rest) 42)" . -.coNP Functions @, get-line @ get-char and @ get-byte +.coNP Macro @ aret .synb -.mets (get-line <> [ stream ]) -.mets (get-char <> [ stream ]) -.mets (get-byte <> [ stream ]) +.mets (aret << form ) .syne .desc -These fundamental stream functions perform input. The -.meta stream -argument -is optional. If it is specified, it should be an input stream which supports -the given operation. If it is not specified, then the -.code *stdin* -stream is used. - The -.code get-char -function pulls a character from a stream which supports character -input. Streams which support character input also support the -.code get-line -function which extracts a line of text delimited by the end of the stream or a -newline character and returns it as a string. (The newline character does not -appear in the string which is returned). - -Character input from streams based on bytes requires UTF-8 decoding, so that -get-char actually may read several bytes from the underlying low level -operating system stream. +.code aret +macro's +.meta form +argument is treated similarly to the second and subsequent arguments of the +.code op +operator. The -.code get-byte -function bypasses UTF-8 decoding and reads raw bytes from -any stream which supports byte input. Bytes are represented as integer -values in the range 0 to 255. +.code aret +macro produces a function which takes any number of arguments, +and returns the value specified by +.metn form . -Note that if a stream supports both byte input and character input, then mixing -the two operations will interfere with the UTF-8 decoding. +.meta form +can contain +.code ap +meta syntax like +.meta @n +and +.codn @rest . -These functions return -.code nil -when the end of data is reached. Errors are -represented as exceptions. +The following equivalence holds: -See also: -.code get-lines +.verb + (aret x) <--> (ap identity x)) +.brev -.coNP Function @ get-string +Thus the expression +.code "(aret @2)" +returns a function similar to +.codn "(lambda (. rest) (second rest))" , +and the expression +.code "(aret 42)" +returns a function similar to +.codn "(lambda (. rest) 42)" . + +.coNP Function @ dup .synb -.mets (get-string >> [ stream >> [ count <> [ close-after-p ]]]) +.mets (dup << func ) .syne .desc The -.code get-string -function reads characters from a stream, and assembles them into -a string, which is returned. If the -.meta stream -argument is omitted, then the -.code *stdin* -stream is used. +.code dup +function returns a one-argument function which calls the two-argument +function +.meta func +by duplicating its argument. -The stream is closed after extracting the data, unless -.meta close-after-p -is specified as -.codn nil . -The default value of this argument is -.codn t . +.TP* Example: -If the -.meta count -argument is missing, then all of the characters from the -stream are read and assembled into a string. +.verb + ;; square the elements of a list + (mapcar [dup *] '(1 2 3)) -> (1 4 9) +.brev -If present, the -.meta count -argument should be a positive integer indicating -a limit on how many characters to read. The returned string will be no -longer than -.metn count , -but may be shorter. +.coNP Function @ flipargs +.synb +.mets (flipargs << func ) +.syne +.desc +The +.code flipargs +function returns a two-argument function which calls the two-argument +function +.meta func +with reversed arguments. -.coNP Functions @ unget-char and @ unget-byte +.coNP Functions @ chain and @ chand .synb -.mets (unget-char < char <> [ stream ]) -.mets (unget-byte < byte <> [ stream ]) +.mets (chain << func *) +.mets (chand << func *) .syne .desc -These functions put back, into a stream, a character or byte which was -previously read. The character or byte must match the one which was most -recently read. If the -.meta stream -argument is omitted, then the -.code *stdin* -stream is used. +The +.code chain +function accepts zero or more functions as arguments, and returns +a single function, called the chained function, which represents the chained +application of those functions, in left to right order. -If the operation succeeds, the byte or character value is returned. -A -.code nil -return indicates that the operation is unsupported. +If +.code chain +is given no arguments, then it returns a variadic function which +ignores all of its arguments and returns +.codn nil . -Some streams do not support these operations; some support -only one of them. In general, if a stream supports -.codn get-char , -it supports -.codn unget-char , -and likewise for -.code get-byte +Otherwise, the first function may accept any number of arguments. The second +and subsequent functions, if any, must accept one argument. + +The chained function can be called with an argument list which is acceptable +to the first function. Those arguments are in fact passed to the first +function. The return value of that call is then passed to the second +function, and the return value of that call is passed to the third function +and so on. The final return value is returned to the caller. + +The +.code chand +function is similar, except that it combines the functionality of +.code andf +into chaining. The difference between +.code chain and -.codn unget-byte . +.code chand +is that +.code chand +immediately terminates and returns +.code nil +whenever any of the functions returns +.codn nil , +without calling the remaining functions. -Streams may require a pushed back byte or character to match -the character which was previously read from that stream -position, and may not allow a byte or character to be pushed -back beyond the beginning of the stream. +.TP* Example: -Space may be available for only one byte of pushback under the -.code unget-byte -operation. +.verb + (call [chain + (op * 2)] 3 4) -> 14 +.brev -The number of characters that may be pushed back by -.code unget-char -is not limited. +In this example, a two-element chain is formed from the +.code + +function +and the function produced by +.code "(op * 2)" +which is a one-argument +function that returns the value of its argument multiplied by two. +(See the definition of the +.code op +operator). -Pushing both a byte and a character, in either order, is also unsupported. -Pushing a byte and then reading a character, or pushing a character and -reading a byte, are unsupported mixtures of operations. +The chained function is invoked using the +.code call +function, with the arguments +.code 3 +and +.codn 4 . +The chained evaluation begins by passing +.code 3 +and +.code 4 +to +.codn + , +which yields +.codn 7 . +This +.code 7 +is then passed to the +.code "(op * 2)" +doubling function, resulting in +.codn 14 . -If the stream is binary, then pushing back a byte decrements its position, -except if the position is already zero. At that point, the position becomes -indeterminate. +A way to write the above example without the use of the DWIM brackets and the +op operator is this: -The behavior of pushing back immediately after a -.code seek-stream -positioning operation is unspecified. +.verb + (call (chain (fun +) (lambda (x) (* 2 x))) 3 4) +.brev -.coNP Functions @, put-string @, put-line @ put-char and @ put-byte +.coNP Function @ juxt .synb -.mets (put-string < string <> [ stream ]) -.mets (put-line >> [ string <> [ stream ]]) -.mets (put-char < char <> [ stream ]) -.mets (put-byte < byte <> [ stream ]) +.mets (juxt << func *) .syne .desc -These functions perform output on an output stream. The -.meta stream -argument -must be an output stream which supports the given operation. If it is omitted, -then -.code *stdout* -is used. - The -.code put-char -function writes a character given by -.code char -to a stream. If the -stream is based on bytes, then the character is encoded into UTF-8 and multiple -bytes are written. Streams which support -.code put-char -also support put-line, and -.codn put-string . +.code juxt +function accepts a variable number of arguments which are functions. It +combines these into a single function which, when invoked, passes its arguments +to each of these functions, and collects the results into a list. -The -.code put-string -function writes the characters of a string out to -the stream as if by multiple calls to put-char. The -.meta string -argument -may be a symbol, in which case its name is used as the string. +Note: the juxt function can be understood in terms of the following reference +implementation: -The -.code put-line -function is like -.codn put-string , -but also writes an additional newline -character. The string is optional in -.codn put-line , -and defaults to the empty string. +.verb + (defun juxt (funcs) + (lambda (. args) + (mapcar (lambda (fun) + (apply fun args)) + funcs))) +.brev The -.code put-byte -function writes a raw byte given by the -.meta byte -argument -to -.metn stream , -if -.meta stream -supports a byte write operation. The byte -value is specified as an integer value in the range 0 to 255. +.code callf +function generalizes +.code juxt +by allowing the combining function to be specified. -All these functions return -.codn t . -On failure, they do not return, but throw exceptions of type -.codn file-error . +.TP* Example: -.coNP Functions @ put-strings and @ put-lines +.verb + ;; separate list (1 2 3 4 5 6) into lists of evens and odds, + ;; which end up juxtaposed in the output list: + + [(op [juxt keep-if remove-if] evenp) + '(1 2 3 4 5 6)] -> ((2 4 6) (1 3 5)) + + ;; call several functions on 1, collecting their results: + [[juxt (op + 1) (op - 1) evenp sin cos] 1]' + -> (2 0 nil 0.841470984807897 0.54030230586814) +.brev + +.coNP Functions @ andf and @ orf .synb -.mets (put-strings < sequence <> [ stream ]]) -.mets (put-lines < sequence <> [ stream ]]) +.mets (andf << func *) +.mets (orf << func *) .syne .desc -These functions assume -.meta sequence -to be a sequence of strings, or of -symbols, or a mixture thereof. These strings are sent to the stream. The -.meta stream -argument must be an output stream. If it is omitted, then -.code *stdout* -is used. - The -.code put-strings -function iterates over -.meta sequence -and writes each element -to the stream as if using the -.code put-string -function. +.code andf +and +.code orf +functions are the functional equivalent of the +.code and +and +.code or +operators. These functions accept multiple functions and return a new function +which represents the logical combination of those functions. -The -.code put-lines -function iterates over -.code sequence -and writes each element -to the stream as if using the -.code put-line -function. +The input functions should have the same arity. Failing that, there should +exist some common argument arity with which each of these can be invoked. The +resulting combined function is then callable with that many arguments. -Both functions return +The +.code andf +function returns a function which combines the input functions with +a short-circuiting logical conjunction. The resulting function passes its +arguments to the functions successively, in left to right order. As soon as any +of the functions returns +.codn nil , +then nil is returned immediately, and the +remaining functions are not called. Otherwise, if none of the functions return +.codn nil , +then the value returned by the last function is returned. If the list of +functions is empty, then +.code t +is returned. That is, +.code (andf) +returns a function +which accepts any arguments, and returns .codn t . -.coNP Function @ flush-stream +The +.code orf +function combines the input functions with a short-circuiting logical +disjunction. The function produced by +.code orf +passes its arguments down to the +functions successively, in left to right order. As soon as any function +returns a +.cod2 non- nil +value, that value is returned and the remaining functions are +not called. If all functions return +.codn nil , +then +.code nil +is returned. The expression +.code (orf) +returns a function which accepts any arguments and returns +.codn nil . + +.coNP Function @ notf .synb -.mets (flush-stream <> [ stream ]) +.mets (notf << function ) .syne .desc The -.code flush-stream -function is meaningful for output streams which accumulate data -which is passed on to the operating system in larger transfer units. -Calling -.code flush-stream -causes all accumulated data inside -.meta stream -to be passed -to the operating system. If called on streams for which this function is not -meaningful, it does nothing, and returns -.codn nil . +.code notf +function returns a function which is the Boolean negation +of +.metn function . -If -.meta stream -is omitted, the current value of -.code *stdout* -is used. +The returned function takes a variable number of arguments. When +invoked, it passes all of these arguments to +.meta function +and then inverts the result as if by application of the +.codn not . -.coNP Function @ seek-stream +.coNP Functions @ iff and @ iffi .synb -.mets (seek-stream < stream < offset << whence ) +.mets (iff < cond-func >> [ then-func <> [ else-func ]]) +.mets (iffi < cond-func < then-func <> [ else-func ]) .syne .desc The -.code seek-stream -function is meaningful for file streams. It changes the -current read/write position within -.metn stream . -It can also be used to determine the current position: see the notes about the -return value below. +.code iff +function is the functional equivalent of the +.code if +operator. It accepts +functional arguments and returns a function. -The -.meta offset -argument is a positive or negative integer which gives a -displacement that is measured from the point identified by the -.meta whence -argument. +The resulting function takes its arguments, if any, and applies them to +.metn cond-func . +If +.meta cond-func +yields true, then the arguments are passed to +.meta then-func +and the +resulting value is returned. Otherwise the arguments are passed to +.meta else-func +and the resulting value is returned. -Note that for text files, there isn't necessarily a 1:1 correspondence between -characters and positions due to line-ending conversions and conversions -to and from UTF-8. +If +.meta then-func +is omitted then +.code identity +is used as default. This omission is not permitted by +.codn iffi , +only +.codn iff . + +If +.meta else-func +needs to be called, but is omitted, then +.code nil +is returned. The -.meta whence -argument is one of three keywords: :from-start, :from-current -and :from-end. These denote the start of the file, the current position -and the end of the file. +.code iffi +function differs from +.code iff +only in the defaulting behavior with respect +to the +.meta else-func +argument. If +.meta else-func +is omitted in a call to +.code iffi +then the default function is +.codn identity . +This is useful in situations when one value is to be +replaced with another one when the condition is true, otherwise +preserved. -If -.meta offset -is zero, and -.meta whence -is -.codn :from-current , -then -.code seek-stream -returns the current absolute position within the -stream, if it can successfully obtain it. Otherwise, it -returns -.code t -if it is successful. +The following equivalences hold between +.code iffi +and +.codn iff : -If a character has been successfully put back into a text stream with -.code unget-char -and is still pending, then the position value is unspecified. If a -byte has been put back into a binary stream with -.codn unget-byte , -and the previous position wasn't zero, then the position is decremented by one. +.verb + (iffi a b c) <--> (iff a b c) -On failure, it throws an exception of type -.codn stream-error . + (iffi a b) <--> (iff a b identity) -.coNP Function @ truncate-stream + [iffi a b nilf] <--> [iff a b] + + [iffi a identity nilf] <--> [iff a] +.brev + +The following equivalences illustrate +.code iff +with both optional arguments omitted: + +.verb + [iff a] <---> [iff a identity nilf] <---> a +.brev + +.coNP Functions @ tf and @ nilf .synb -.mets (truncate-stream < stream <> [ length ]) +.mets (tf << arg *) +.mets (nilf << arg *) .syne .desc The -.code truncate-stream -causes the length of the underlying file associated with -.meta stream -to be set to -.meta length -bytes. - -The stream must be a file stream, and must be open for writing. - -If -.meta length -is omitted, then it defaults to the current position, retrieved -as if by invoking the -.code seek-stream -with an -.meta offset -argument of zero and -.meta whence -argument of -.codn :from-current . -Hence, after the -.code truncate-stream -operation, that position is one byte past the end of the file. - -.coNP Functions @ stream-get-prop and @ stream-set-prop -.synb -.mets (stream-get-prop < stream << indicator ) -.mets (stream-set-prop < stream < indicator << value ) -.syne -.desc -These functions get and set properties on a stream. Only certain properties -are meaningful with certain kinds of streams, and the meaning depends on -the stream. If two or more stream types support a property of the same name, it -is expected that the property has the same or very similar meaning for both -streams to the maximum extent that similarity is possible. - -The -.code stream-set-prop -function sets a property on a stream. The -.meta indicator -argument is a symbol, usually a keyword symbol, denoting the property, +.code tf and -.meta value -is the property value. If the stream understands and accepts the -property, the function returns -.codn t . -Otherwise it returns -.codn nil . - +.code nilf +functions take zero or more arguments, and ignore them. The -.code stream-get-prop -function inquires about the value of a property on a -stream. If the stream understands the property, then it returns its current -value. If the stream does not understand a property, nil is returned, which is -also returned if the property exists, but its value happens to be +.code tf +function returns +.codn t , +and the +.code nilf +function returns .codn nil . -The -.code :name -property is widely supported by streams of various types. It associates -the stream with a name. This property is not always modifiable. - -File, process and stream socket I/O streams have a -.code :fd -property which can be accessed, but not modified. It retrieves -the same value as the -.code stream-fd +Note: the following equivalences hold between these functions and the +.code ret +operator, and +.code retf function. -The "real time" -property supported by these streams, connected with the -.code real-time-stream-p -function, also appears as the -.code :real-time -property. - -I/O streams also have a property called -.code :byte-oriented -which, if set, suppresses the decoding of UTF-8 on character input. Rather, -each byte of the file corresponds directly to one character. Bytes -in the range 1 to 255 correspond to the character code points U+0001 -to U+00FF. Byte value 0 is mapped to the code point U+DC00. - -The logging priority of the -.code *stdlog* -syslog stream is controlled by the -.code :prio -property. - -If -.meta stream -is a catenated stream (see the function -.codn make-catenated-stream ) -then these functions transparently operate on the current head stream of the -catenation. - -.coNP Functions @ make-catenated-stream and @ cat-streams -.synb -.mets (make-catenated-stream << stream *) -.mets (cat-streams << stream-list ) -.syne -.desc -The -.code make-catenated-stream -function takes zero or more arguments which -are input streams of the same type, and combines -them into a single virtual stream called a catenated stream. - -The -.code cat-streams -function takes a single list of input streams of -the same type, and similarly combines them into a catenated stream. - -A catenated stream does not support seeking operations or output, -regardless of the capabilities of the streams in the list. - -If the stream list is not empty, then the leftmost element of the -list is called the head stream. +.verb + (fun tf) <--> (ret t) <--> (retf t) + (fun nilf) <--> (ret nil) <--> (ret) <--> (retf nil) +.brev -The -.codn get-char , -.codn get-byte , -.codn get-line , -.code unget-char +In Lisp-1-style code, +.code tf and -.code unget-byte -functions delegate -to the corresponding operations on the head stream, if it exists. -If the stream list is empty, they return -.code nil -to the caller. +.code nilf +behave like constants which can replace uses of +.code "(ret t)" +and +.codn "(ret nil)" : -If the -.codn get-char , -.code get-byte -or -.code get-line -operation on the head stream yields -.codn nil , -and there are more lists in the stream, then the stream is closed, removed from -the list, and the next stream, if any, becomes the head list. The operation is -then tried again. If any of these operations fail on the last list, it is not -removed from the list, so that a stream remains in place which can take the -.code unget-char -or -.code unget-byte -operations. +.verb + [mapcar (ret nil) list] <--> [mapcar nilf list] +.brev -In this manner, the catenated streams appear to be a single stream. +.TP* Example: -Note that the operations can fail due to being unsupported. It is -the caller's responsibility to make sure all of the streams in the list -are compatible with the intended operations. +.verb + ;; tf and nilf are useful when functions are chained together. + ;; test whether (trunc n 2) is odd. -If the stream list is empty then an empty catenated stream is produced. -Input operations on this stream yield -.codn nil , -and the -.code unget-char -and -.code unget-byte -operations throw an exception. + (defun trunc-n-2-odd (n) + [[chain (op trunc @1 2) [iff oddp tf nilf]] n]) +.brev -.coNP Function @ catenated-stream-p -.synb -.mets (catenated-stream-p << obj ) -.syne -.desc +In this example, two functions are chained together, and +.code n +is passed +through the chain such that it is first divided by two via the +function denoted by +.code "(op trunc @1 2)" +and then the result is passed into the +function denoted by +.codn "[iff oddp tf nilf]" . The -.code catenated-stream-p -function returns -.code t -if -.meta obj -is a catenated stream. Otherwise it returns +.code iff +function passes its argument into +.codn oddp , +and if +.code oddp +yields true, it passes the same argument to +.codn tf . +Here +.code tf +proves its utility by ignoring that value and returning +.codn t . +If the argument (the divided value) passed into +.code iff +is even, then iff passes it into the +.code nilf +function, which ignores the value and returns .codn nil . -.coNP Function @ catenated-stream-push +.coNP Function @ retf .synb -.mets (catenated-stream-push < new-stream << cat-stream ) +.mets (retf << value ) .syne .desc The -.code catenated-stream-push -function pushes -.meta new-stream -to the front of the stream list inside -.metn cat-stream . +.code retf +function returns a function. That function can take zero or +more arguments. When called, it ignores its arguments and returns +.metn value . -If an -.code unget-byte -or -.code unget-char -operation was successfully performed on -.meta cat-stream -previously to a call to -.codn catenated-stream-push , -those operations were forwarded to the front stream. -If those bytes or characters are still pending, -they are pending inside that stream, and thus -are logically preceded by the contents -of -.metn new-stream . +See also: the +.code ret +macro. -.coNP Functions @ open-files and @ open-files* +.TP* Example: + +.verb + ;; the function returned by (retf 42) + ;; ignores 1 2 3 and returns 42. + (call (retf 42) 1 2 3) -> 42 +.brev + +.coNP Functions @ apf and @ ipf .synb -.mets (open-files < path-list <> [ alternative-stream ]) -.mets (open-files* < path-list <> [ alternative-stream ]) +.mets (apf << function ) +.mets (ipf << function ) .syne .desc The -.code open-files -and -.code open-files* -functions create a list of streams by invoking -the open-file function on each element of -.metn path-list . -These streams are turned -into a catenated stream as if applied as arguments to -.codn make-catenated-stream . - -The effect is that multiple files appear to be catenated together into a single -input stream. - -If the optional -.meta alternative-stream -argument is supplied, then if -.meta path-list -is empty, -.meta alternative-stream -is returned instead of an empty catenated stream. +.code apf +function returns a one-argument function which accepts +a list. When the function is called, it treats the list as +arguments which are applied to +.meta function +as if by apply. It returns whatever +.meta function +returns. -The difference between -.code open-files -and -.code open-files* -is that -.code open-files -creates all of the -streams up-front. So if any of the paths cannot be opened, the operation throws. The -.code open-files* -variant is lazy: it creates a lazy list of streams out of the -path list. The streams are opened as needed: before the second stream is opened, -the program has to read the first stream to the end, and so on. +.code ipf +function is similar to +.codn apf , +except that the returned +function applies arguments as if by +.code iapply +rather than +.codn apply . -.TP* Example: +See also: the +.code ap +macro. -Collect lines from all files that are given as arguments on the command line. If -there are no files, then read from standard input: +.TP* Example: .verb - @(next (open-files *args* *stdin*)) - @(collect) - @line - @(end) + ;; Function returned by [apf +] accepts the + ;; (1 2 3) list and applies it to +, as + ;; if (+ 1 2 3) were called. + + (call [apf +] '(1 2 3)) -> 6 .brev -.coNP Function @ abs-path-p +.coNP Function @ callf .synb -.mets (abs-path-p << path ) +.mets (callf < main-function << arg-function *) .syne .desc The -.code abs-path-function -tests whether the argument -.meta path -is an absolute path, returning a -.code t -or -.code nil -indication. - -The function behaves in the same manner on all platforms, implementing -a platform-agnostic definition of -.IR "absolute path" , -as follows. +.code callf +function returns a function which applies its arguments to each +.metn arg-function , +juxtaposing the return values of these calls to form arguments +which are then passed to +.metn main-function . +The return value of +.meta main-function +is returned. -An absolute path is a string which either begins with a slash or backslash -character, or which begins with an alphanumeric word, followed by a colon, -followed by a slash or backslash. +The following equivalence holds, except for the order of evaluation of +arguments: -The empty string isn't an absolute path. +.verb + (callf fm f0 f1 f2 ...) <--> (chain (juxt f0 f1 f2 ...) + (apf fm)) +.brev -Examples of absolute paths: +.TP* Example: .verb - /etc - c:/tmp - ftp://user@server - disk0:/home - Z:\eUsers + ;; Keep those pairs which are two of a kind + + (keep-if [callf eql first second] '((1 1) (2 3) (4 4) (5 6))) + -> ((1 1) (4 4)) .brev -Examples of strings which are not absolute paths: +The following equivalence holds between +.code juxt +and +.codn callf : -.mono -.mets >> ( the < empty << string ) - . - abc - foo:bar/x - $:\eabc -.onom +.verb + [juxt f0 f1 f2 ...] <--> [callf list f0 f1 f2 ...]:w +.brev -.coNP Function @ pure-rel-path-p +Thus, +.code juxt +may be regarded as a specialization of +.code callf +in which the main function is implicitly +.codn list . + +.coNP Function @ mapf .synb -.mets (pure-rel-path-p << path ) +.mets (mapf < main-function << arg-function *) .syne .desc The -.code pure-rel-path-p -function tests whether the string -.meta path -represents a -.IR "pure relative path" , -which is defined as a path which isn't absolute according to -.codn abs-path-p , -which isn't the string -.str . -(single period), -which doesn't begin with a period followed by a slash or backslash, -and which doesn't begin with alphanumeric word -terminated by a colon. +.code mapf +function returns a function which distributes its arguments +into the +.metn arg-function -s. +That is to say, each successive argument of the returned +function is associated with a successive +.metn arg-function . -The empty string is a pure relative path. +Each +.meta arg-function +is called, passed the corresponding argument. The return +values of these functions are then passed as arguments +to +.meta main-function +and the resulting value is returned. -Other examples of pure relative paths: +If the returned function is called with fewer arguments than there +are +.metn arg-function -s, +then only that many functions are used. Conversely, if the function is +called with more arguments than there are +.metn arg-function -s, +then those arguments are ignored. + +The following equivalence holds: .verb - abc.d - .tmp/bar - 1234 - x - $:/xyz + (mapf fm f0 f1 ...) <--> (lambda (. rest) + [apply fm [mapcar call + (list f0 f1 ...) + rest]]) .brev -Examples of strings which are not pure relative paths: +.TP* Example: .verb - . - / - /etc - ./abc - .\e - foo: - $:\eabc + ;; Add the squares of 2 and 3 + [[mapf + [dup *] [dup *]] 2 3] -> 13 + .brev -.coNP Functions @ dir-name and @ base-name -.synb -.mets (dir-name << path ) -.mets (base-name << path ) -.syne -.desc -The -.code dir-name -and -.code base-name -functions calculate, respective, the directory part and -base name part of a path name. +.SS* Input and Output (Streams) +\*(TL supports input and output streams of various kinds, with +generic operations that work across the stream types. -The calculation is performed in a platform-dependent way, using the -characters in the variable -.code path-sep-chars -as path component separators. +In general, I/O errors are usually turned into exceptions. When the description +of error reporting is omitted from the description of a function, it can be +assumed that it throws an error. -Both functions first remove from any further consideration all superfluous -trailing occurrences of the directory separator characters from -.codn path . -Thus input such as -.str "a////" -is reduced to just -.strn "a" , +.coNP Special variables @, *stdout* @, *stddebug* @, *stdin* @ *stderr* and @ *stdnull* +.desc +These variables hold predefined stream objects. The +.codn *stdin* , +.code *stdout* and -.str "///" -is reduced to -.strn "/" . - -The resulting trimmed path is the -.I "effective path" . +.code *stderr* +streams closely correspond to the underlying operating system streams. +Various I/O functions require stream objects as arguments. -If the effective path is an empty string, then -.code dir-name -returns -.str "." -and -.code base-name -returns the empty string. - -If the effective path is not empty, and contains no path separator -characters, then -.code dir-name -returns -.str "." -and -.code base-name -returns the effective path. - -Otherwise, the effective path is divided into two parts: the -.I "raw directory prefix" -and the remainder. - -The raw directory path is the maximally long prefix of the effective -path which ends in a separator character. - -The -.code dir-name -function returns the raw directory prefix, if that prefix consists of -nothing but a single directory separator character. Otherwise it -returns the raw directory prefix, with the trailing path separator -removed. +The +.code *stddebug* +stream goes to the same destination as +.codn *stdout* , +but is a separate object which can be redirected independently, allowing +debugging output to be separated from normal output. The -.code base-name -function returns the remaining part of the effective path, after -the raw directory prefix. +.code *stdnull* +stream is a special kind of stream called a null stream. +This stream is not connected to any device or file. It is similar to +the +.code /dev/null +device on Unix, but does not involve the operating system. -.coNP Function @ path-cat -.synb -.mets (path-cat < dir-path << rel-path ) -.syne +.coNP Special variables @ *print-flo-format* and @ *pprint-flo-format* .desc The -.code path-cat -function joins the directory path name given by the character -string argument -.meta dir-path -with the relative path name given by -.metn rel-path , -returning the joined path. - -The function is related to the functions -.code dir-name +.code *print-flo-format* +variable determines the conversion format which is applied when +a floating-point value is converted to decimal text by the +functions +.codn print , +.codn prinl , and -.code base-name -in the following way: if -.meta p -is some path denoting an object in the file system, then -.code "(path-cat (dir-name p) (base-name p))" -produces a path -.meta p* -which denotes the same object. The paths -.meta p +.codn tostring . + +The default value is +.codn "~s" . + +The related variable +.code *pprint-flo-format* +similarly determines the conversion format applied to floating-point +values by the functions +.codn pprint , +.codn pprinl , and -.meta p* -might not be equivalent strings. +.codn tostringp . -The -.code path-cat -function ensures that paths are joined without superfluous -path separator characters, regardless of whether -.meta dir-path -ends in a separator. +The default value is +.codn "~a" . -If a separator must be added, the character -.code / -(forward slash) is always used, even on platforms where -.code \e -(backslash) is also a pathname separator, and even if either argument includes -backslashes. +The format string in either variable must specify the consumption of +exactly one +.code format +argument. + +The conversion string may use embedded width and precision values: +for instance, +.code "~3,4f" +is a valid value for +.code *print-flo-format* +or +.codn *pprint-flo-format* . +.coNP Special variable @ *print-flo-precision* +.desc The -.code path-cat -function eliminates trivial occurrences of the -.code . -(dot) path component. It preserves trailing separators in the following -way: if -.meta rel-path -ends in a path separator character, then the returned string shall -end in that character; and if -.meta rel-path -vanishes entirely because it is equivalent to the dot, then the returned -string is -.meta dir-name -itself. +.code *print-flo-precision* +special variable specifies the default floating-point printing +precision which is used when the +.code ~a +or +.code ~s +conversion specifier of the +.code format +function is used for printing a floating-point value, and no precision +is specified. -.TP* Examples: +Note that since the default value of the variable +.code *print-flo-format* +is the string +.codn "~s" , +the +.code *printf-flo-precision* +variable, by default, also determines the precision which applies when +floating-point values are converted to decimal text by the functions +.codn print , +.codn pprint , +.codn prinl , +.codn pprinl , +.code tostring +and +.codn tostringp . -.verb - (path-cat "" "") --> "" - (path-cat "" ".") --> "" - (path-cat "." "") --> "" - (path-cat "." ".") --> "" +The default value of +.code *print-flo-precision* +is that of the +.code flo-dig +variable. - (path-cat "abc" ".") --> "abc" - (path-cat "." "abc") --> "abc" +Note: to print floating-point values in such a way that their values +can be precisely recovered from the printed representation, it is +recommended to override +.code *print-flo-precision* +to the value of the +.code flo-max-dig +variable. - (path-cat "./" ".") --> "./" - (path-cat "." "./") --> "./" +.coNP Special variable @ *print-flo-digits* +.desc +The +.code *print-flo-precision* +special variable specifies the default floating-point printing +precision which is used when the +.code ~f +or +.code ~e +conversion specifier of the +.code format +function is used for printing a floating-point value, and no precision +is specified. - (path-cat "abc/" ".") --> "abc/" - (path-cat "./" "abc") --> "abc" +Its default value is +.codn 3 . - (path-cat "/" ".") --> "/" +.coNP Special variable @ *print-base* +.desc +The +.code *print-base* +variable controls the base (radix) used for printing integer values. +It applies when the functions +.codn print , +.codn pprint , +.codn prinl , +.codn pprinl , +.code tostring +and +.code tostringp +process an integer value. +It also applies when the +.code ~a +and +.code ~s +conversion specifiers of the +.code format +function are used for printing an integer value. - (path-cat "/" "abc") --> "/abc" +The default value of the variable is +.codn 10 . - (path-cat "ab/cd" "ef") --> "ab/cd/ef" -.brev +Meaningful values are: +.codn 2 , +.codn 8 , +.code 10 +and +.codn 16 . +When base 16 is selected, hexadecimal digits are printed as upper-case +characters. -.coNP Variable @ path-sep-chars +.coNP Special variable @ *print-circle* .desc The -.code path-sep-chars -variable holds a string consisting of the characters which the underlying -operating system recognizes as path name separators. +.code *print-circle* +variable is a Boolean which controls whether the circle notation is +in effect for printing aggregate objects: conses, ranges, vectors, hash tables +and structs. The initial value of this variable is +.codn nil : +circle notation printing is disabled. -If a particular of these characters is considered preferred on -the host platform, that character is placed in the first position of -.codn path-sep-chars . +The circle notation works for structs also, including structs which have +user-defined +.code print +methods. When a +.code print +method calls functions which print objects, such as +.codn print , +.code pprinl +or +.code format +on the same stream, the detection of circularity and substructure sharing +continues in these recursive invocations. -Altering the value of this variable has no effect on any \*(TL library -function. +However, there are limitations in the degree of support for circle notation +printing across +.code print +methods. Namely, a +.code print +method of a struct +.meta S +must not procure and submit for printing objects which are not part of the +ordinary structure that is reachable from the (static or instance) slots of +.metn S , +if those objects have already been printed prior to invoking the +.code print +method, and have been printed without a +.code #= +circle notation label. The "ordinary structure that is reachable from the +slots" denotes structure that is directly reachable by traversing conses, +ranges, vectors, hashes and struct slots: all printable aggregate objects. -.coNP Functions @ read and @ iread +.coNP Function @ format .synb -.mets (read >> [ source >> [ error-stream >> [ error-retval <> [ name ]]]]) -.mets (iread >> [ source >> [ error-stream >> [ error-retval <> [ name ]]]]) +.mets (format < stream-designator < format-string << format-arg *) .syne .desc The -.code read -function converts text denoting \*(TL structure, into the -corresponding data structure. The -.meta source -argument may be either a character -string, or a stream. If it is omitted, then -.code *stdin* -is used as the stream. +.code format +function performs output to a stream given by +.metn stream-designator , +by interpreting the actions implicit in a +.metn format-string , +incorporating material pulled from additional arguments given by +.mono +.meti << format-arg *. +.onom +Though the function is simple to invoke, there is complexity in format string +language, which is documented below. -The source must provide the text representation of one complete \*(TL object. +The +.meta stream-designator +argument can be a stream object, or one of the values +.code t +or +.codn nil . +The value +.code t +serves as a shorthand for +.codn *stdout* . +The value +.code nil +means that the function will send output into a newly instantiated string +output stream, and then return the resulting string. -Multiple calls to read on the same stream will extract successive objects -from the stream. To parse successive objects from a string, it is necessary -to convert it to a string stream. +.TP* "Format string syntax:" -The optional -.meta error-stream -argument can be used to specify a stream to which -parse errors diagnostics are sent. If absent, the diagnostics are suppressed. +Within +.metn format-string , +most characters represent themselves. Those +characters are simply output. The character +.code ~ +(tilde) introduces formatting +directives, which are denoted by a single character, usually a letter. -The optional -.meta name -argument can be used to specify the file name which is used for reporting -errors. If this argument is missing, the name is taken from the name -property of the -.meta source -argument if it is a stream, or else the word -.code string -is used as the name if -.meta source -is a string. +The special sequence +.code ~~ +(tilde-tilde) encodes a single tilde. Nothing is +permitted between the two tildes. -If there are no parse errors, the function returns the parsed data -structure. If there are parse errors, and the -.meta error-retval -parameter is -present, its value is returned. If the -.meta error-retval -parameter -is not present, then an exception of type -.code syntax-error -is thrown. +The syntax of a directive is generally as follows: + +.mono +.mets <> ~[ width ] <> [, precision ] < letter +.onom + +In other words, the +.code ~ +(tilde) character, followed by a +.meta width +specifier, a +.meta precision +specifier introduced by a comma, +and a +.metn letter , +such that +.meta width +and +.meta precision +are independently optional: either or both may be omitted. +No whitespace is allowed between these elements. The -.code iread -function ("interactive read") is similar to -.code read -except that it parses a modified version of the syntax. The modified -syntax does not support the application of the dot and dotdot operators -on a top-level expression. For instance, if the input is -.code a.b -or -.code "a .. b" -then -.code iread -will only read the -.code a -token whereas -.code read -will read the entire expression. +.meta letter +is a single alphabetic character which determines the +general action of the directive. The optional width and precision +are specified as follows: -This modified syntax allows -.code iread -to return immediately when an expression is recognized, which is the -expected behavior if the input is being read from an interactive terminal. -By contrast, -.code read -waits for more input after seeing a complete expression, because of the -possibility that the expression will be further extended by means of the dot or -dotdot operators. An explicit end-of-input signal must be given from the -terminal to terminate the expression. +.RS +.meIP < width +The width specifier consists of an optional +.code < +(left angle bracket) character or +.code ^ +(caret) +character followed by an optional width specification. -The special variable -.code *rec-source-loc* -controls whether these functions record source location info similarly to -.codn load . -Note: if these functions are used to scan data which is evaluated as Lisp code, -it may be useful to set -.code *rec-source-loc* -true in order to obtain better diagnostics. However, source location recording -incurs a performance and storage penalty. +If the leading +.code < +character is present, then the printing will be left-adjusted within +this field. If the +.code ^ +character is present, the printing will be centered within the field. +Otherwise it will be right-adjusted by default. -.coNP Function @ record-adapter -.synb -.mets (record-adapter < regex >> [ stream <> [ include-match ]]) -.syne -.desc +The width can be specified as a decimal integer with an optional leading +minus sign, or as the character +.codn * . The -.code record-adapter -function returns a new stream object which acts as an -.I adapter -to the existing -.metn stream . +.code * +notation means that instead of digits, the value of the next argument is +consumed, and expected to be an integer which specifies the width. If the +width, specified either way, is negative, then the field will be left-adjusted. +If the value is positive, but either the +.code < +or +.code ^ +prefix character is present in the width +specifier, then the field is adjusted according to that character. -If an argument is not specified for -.metn stream , -then the -.code *std-input* -stream is used. +The padding calculations for alignment and centering take into account +character display width, as defined by the +.code display-width +function. For instance, a character string containing four Chinese +characters (kanji) has a display width of 8, not 4. -With the exception of -.metn get-line , -all operations on the returned adapter transparently delegate to the original -.meta stream -object. +The width specification does not restrict the printed portion of a datum. +Rather, for some kinds of conversions, it is the precision specification that +performs such truncation. A datum's display width (or that of its printed +portion, after such truncation is applied) can equal or exceed the specified +field width. In this situation it overflows the field: the printed portion is +rendered in its entirety without any padding applied on either side for +alignment or centering. -When the -.code get-line -function is used on the adapter, it behaves differently. A string is -extracted from -.metn stream , -and returned. However, the string isn't a line delimited by a newline -character, but rather a record delimited by -.metn regex . -This record is extracted as if by a call to the -.code read-until-match -function, invoked with the -.metn regex , -.meta stream -and -.meta include-match -arguments. +.meIP < precision +The precision specifier is introduced by a leading comma. If this comma appears +immediately after the directive's +.code ~ +character, then it means that +.meta width +is being omitted; there is only a precision field. -All behavior which is built on the -.code get-lines -function is affected by the record-delimiting semantics of a record adapter's -.code get-line -implementation. Notably, the -.code get-lines -and -.code lazy-stream-cons -functions return a lazy list of delimited records rather than of lines. +The precision specifier may begin with these optional characters: +.RS +.coIP 0 +(the "leading zero flag"), +.coIP + +(print a sign for positive values") +.IP space +(print a space in place of a positive sign). +.RE -.SS* Stream Output Indentation -\*(TL streams provide support for establishing hanging indentations -in text output. Each stream which supports output has a built-in state variable -called indentation mode, and another variable indicating the current -indentation amount. When indentation mode is enabled, then prior to the -first character of every line, the stream prepends the indentation: space -characters equal in number to the current indentation value. -This logic is implemented by the -.code put-char -and -.code put-string -functions, and all functions based on these. The -.code put-byte -function does not interact with indentation. The column position tracking -will be incorrect if byte and character output are mixed, affecting -the placement of indentation. +The precision specifier itself is either a decimal integer that does not +begin with a zero digit, or the +.code * +character. -Indentation mode takes on four numeric values, given by the four -variables -.codn indent-off , -.codn indent-data , -.code indent-code -and -.codn indent-foff . -As far as stream output is concerned, the code and data modes -represented by -.code indent-code -and -.code indent-data -behave -the same way: both represent the "indentation turned on" state. -The difference between them influences the behavior of the -.code width-check -function. This function isn't used by any lower-level stream output -routines. It is used by the object printing functions like -.code print -and -.code pprint -to break up long lines. -The -.code indent-off -and -.code intent-foff -modes are also treated the same way by lower level stream output, -indicating "indentation turned off". The modes are distinguished -by -.code print -and -.code pprint -in the following way: -.code indent-off -is a "soft" disable which allows these object-printing routines -to temporarily turn on indentation while traversing aggregate objects. -Whereas the -.code indent-foff -("force off") value is a "hard" disable: the object-printing routines will not -enable indentation and will not break up long lines. +The precision field's components have a meaning which depends on the type of +object printed and the conversion specifier. -.coNP Variables @, indent-off @, indent-data @ indent-code and @ indent-foff -.desc -These variables hold integer values representing output stream -indentation modes. The value of -.code indent-off -is zero. +For integer arguments, the precision value specifies the minimum number of digits +to print. If the precision field has a leading zero flag, then the integer is +padded with zeros to the required number of digits, otherwise the number is +padded with spaces instead of zeros. If zero or space padding is present, and +a leading positive or negative sign must be printed, then it is placed before +leading zeros, or after leading spaces, as the case may be. -.coNP Functions @ get-indent-mode and @ set-indent-mode -.synb -.mets (get-indent-mode << stream ) -.mets (set-indent-mode < stream << new-mode ) -.mets (test-set-indent-mode < stream < compare-mode << new-mode ) -.syne -.desc -These functions retrieve and manipulate the stream indent mode. -The -.code get-indent-mode -retrieves the current indent mode of -.metn stream . -The -.code set-indent-mode -function sets the indent mode of -.meta stream -to -.meta new-mode -and returns the previous mode. +For floating-point values, the meaning of the precision value depends on which +specific conversion specifier +.cod1 ( f , +.codn e , +.code a +or +.codn s ) +is used. The details are +documented in the description of each of these, below. The leading zero flag is +also taken into account for floating-point values, and treated uniformly by +these directives. If the flag is present, then the printed value's integer +part will be padded with leading zeros up to the width of the field such that +one character of unused space remains in the field, in case a positive or +negative sign needs also to be rendered. -Note: it is encouraged to save and restore the indentation mode, -and in a way that is exception safe. -If a block of code sets up indentation on a stream such as -.code *stdout* -and is terminated by an exception, the indentation will remain in -effect and affect subsequent output. The -.code with-resources -macro or -.code unwind-protect -operator may be used. +For integer or floating-point arguments, if the precision specifier has a +.code + +sign +among the special characters, then a +.code + +sign is printed for positive numbers. If +the precision specifier has a leading space instead of a +.code + +sign, then the +.code + +sign is rendered as a space for positive numbers. If there is no leading space +or +.codn + , +then a sign character is omitted for positive numbers. Negative +numbers are unconditionally prefixed with a +.code - +sign. -.coNP Functions @ test-set-indent-mode and @ test-neq-set-indent-mode -.synb -.mets (test-set-indent-mode < stream < compare-mode << new-mode ) -.mets (test-neq-set-indent-mode < stream < compare-mode << new-mode ) -.syne -.desc -The -.code test-set-indent-mode -function sets the indent mode of -.meta stream -to -.meta new-mode -if, and only if, -its current mode is equal to -.metn compare-mode . -Whether or not it changes the mode, it returns the previous mode. +For all other objects, the precision specifies the maximum number of +print positions to occupy, taking into account the display width of each +character of the printed representation of the object, as according +to the +.code display-width +function. The object's printed representation is truncated, if necessary, to +the maximum number of characters which will not exceed the specified number of +print positions. -The -.code test-neq-set-indent-mode -only differs in that it sets -.meta stream -to -.meta new-mode -if, and only if, -the current mode is -.B not -equal to -.metn compare-mode . +.RE -.coNP Functions @, get-indent @ set-indent and @ inc-indent -.synb -.mets (get-indent << stream ) -.mets (set-indent < stream << new-indent ) -.mets (inc-indent < stream << indent-delta ) -.syne -.desc -These functions manipulate the indentation value of the stream. -The indentation takes effect the next time a character is output -following a newline character. +.TP* "Format directives:" +.RS +Format directives are case sensitive, so that for example +.code ~x +and +.code ~X +have a +different effect, and +.code ~A +doesn't exist whereas +.code ~a +does. They are: -The -.code get-indent -function retrieves the current indentation amount. +.coIP a +Prints any object in an aesthetic way, as if by the +.code pprint +function. +The aesthetic notation violates read-print consistency: this notation +is not necessarily readable if it is implanted in \*(TX source code. +The field width specifier is honored, including the left-right adjustment +semantics. +When this specifier is used for floating-point values, the precision specifies +the maximum number of total significant figures, which do not include any +digits in the exponent, if one is printed. Numbers are printed in exponential +notation if their magnitude is small, or else if their exponent exceeds their +precision. If the precision is not specified, then it is obtained from +the +.code *print-flo-precision* +special variable, whose default value is the same as that of the +.code flo-dig +variable. +Floating point values which are integers are +printed without a trailing +.code .0 +(point zero). The -.code set-indent -function sets -.metn stream 's -indentation to the value -.meta new-indent -and returns the previous value. -Negative values are clamped to zero. +.code + +flag in the precision is honored for rendering an explicit +.code + +sign on non-negative values. +If a leading zero is specified in the precision, and a nonzero width is +specified, then the printed value's integer part will be padded with leading +zeros up to one less than the field width. These zeros are placed before the +sign. -The -.code inc-indent -function sets -.metn stream 's -indentation relative to the current printing column position, -and returns the old value. -The indentation is calculated by adding -.meta indent-delta -to the current column position. -If a negative indentation results, it is clamped to zero. +.coIP s +Prints any object in a standard way, as if by the +.code print +function. Objects for +which read-print consistency is possible are printed in a way such that +if their notation is implanted in \*(TX source, they are readable. +The field width specifier is honored, including the left-right adjustment +semantics. The precision field is treated very similarly to the +.code ~a +format directive, except that non-exponentiated floating point numbers that +would be mistaken for integers include a trailing +.code .0 +for the sake of read-print +consistency. Objects truncated by precision may not have read-print +consistency. For instance, if a string object is truncated, it loses its +trailing closing quote, so that the resulting representation is no longer +a properly formed string object. For integer objects, the +.code *print-base* +variable is honored. Effectively, an integer is printed by the +.code s +directive as if by the +.codn b , +.codn o , +.codn d , +or +.code x +directive, depending on the value of the variable. -.coNP Function @ width-check -.synb -.mets (width-check < stream << alt-char ) -.syne -.desc -The -.code width-check -function examines the state of the stream, taking into consideration -the current printing column position, the indentation state, -the indentation amount and an internal "force break" flag. It makes a decision -either to introduce a line break by printing a newline character, or else to -print the -.meta alt-char -character. +.coIP d +Requires an argument of integer or character type type. The integer +value or character code is printed in decimal. -If a decision is made not to emit a line break, but -.meta alt-char -is -.codn nil , -then the function has no effect at all. +.coIP x +Requires an argument of character or integer type. The integer value or +character code is printed in hexadecimal, using lower-case letters +for the digits +.code a +through +.codn f . +Width and precision semantics +are as described for the +.code a +format directive, for integers. -The return value is -.code t -if the function has issued a line break, otherwise -.codn nil . +.coIP X +Like the +.code x +directive, but the hexadecimal digits +.code a +through +.code f +are rendered in upper case. -.coNP Function @ force-break -.synb -.mets (force-break << stream ) -.syne -.desc -If the -.code force-break -function is called on a stream, it sets an internal "force break" flag which -affects the future behavior of -.codn width-check . -The -.code width-check -function examines this flag. If the flag is set, -.code width-check -clears it, and issues a line break without considering any other -conditions. +.coIP o +Like the +.code x +directive, but octal is used instead of hexadecimal. -The -.metn stream 's -.code force-break -flag is also cleared whenever a newline character is output. +.coIP b +Like the +.code x +directive, but binary is used instead of hexadecimal. +.coIP f The -.code force-break -function returns -.codn stream . - -Note: the -.code force-break -is involved in line breaking decisions. Whenever a list or list-like syntax is -being printed, whenever an element of that syntax is broken into multiple -lines, a break is forced after that element, in order to avoid output -which resembles the following diagonally-creeping pattern: - -.verb - (a b c (d e f - g h i) j (k l - m n) o) -.brev - -but instead is rendered in a more horizontally compact pattern: - -.verb - (a b c (d e f - g h i) - j (k l - m n) - o) -.brev - -When the printer prints -.code "(d e f g h i)" -it uses the -.code width-check -function between the elements; that function issues the -break between the .code f +directive prints numbers in a fixed point decimal notation, with +a fixed number of digits after the decimal point. It requires a numeric +argument. (Unlike +.codn x , +.code X and -.codn g . -The printer monitors the return value of -.codn width-check ; -it knows that since one of the calls returned -.codn t , -the object had been broken into two or more lines. It then calls -.code force-break -after printing the last element -.code i -of that object. Then, due to the force flag, the outer recursion of the -printer which is printing -.code "(a b c ...)" -will experience a break when it calls -.code width-check -before printing -.codn j . +.codn o , +it does not allow an argument of character type). +The precision specifier gives the number of digits past the decimal point. +The number is rounded off to the specified precision, if necessary. +Furthermore, that many digits are always printed, regardless of the actual +precision of the number or its type. If it is omitted, then the value +is obtained from the special variable +.codn *print-flo-digits* , +whose default value is three: three digits past the decimal point. A precision +of zero means no digits pas the decimal point, and in this case the decimal +point is suppressed (regardless of whether the numeric argument is +floating-point or integer). -Custom -.code print -methods defined on structure objects can take advantage of -.code width-check +.coIP e +The +.code e +directive prints numbers in exponential notation. It requires +a numeric argument. (Unlike +.codn x , +.code X and -.code force-break -in the same way so that application-defined output integrates -with the formatting algorithm. +.codn o , +it does not allow an argument of character type). +The precision specifier gives the number of digits past the decimal point +printed in the exponential notation, not counting the digits in the exponent. +Exactly that many digits are printed, regardless of the precision of the +number. If the precision is omitted, then the number of digits after the +decimal point is obtained from the value of the special variable +.codn *print-flo-digits* , +whose default value is three. If the precision is zero, then a decimal portion +is truncated off entirely, including the decimal point. -.SS* Stream Output Limitation +.coIP p +The +.code p +directive prints a numeric representation in hexadecimal of the bit pattern +of the object, which is meaningful to someone familiar with the internals +of \*(TX. If the object is a pointer to heaped data, that value +has a correspondence to its address. -Streams have two properties which are used by the The \*(TL object printer to -optionally truncate the output generated by aggregate objects. +.coIP ! +The +.code ! +directive establishes hanging indentation, and turns on the stream's +indentation mode. Subsequent lines printed within the execution of the +same +.code format +call will be automatically indented. If no width is specified, then +the directive sets the hanging indentation to the current printing +column position. If a width is specified, then it represents an offset +(positive or negative). If the +.code < +prefix character is present, the hanging indentation is set to the +specified offset relative to the current printing column. +If the +.code < +prefix is present on the width field, then the offset is applied +relative to the indentation which was saved on entry into the +.code format +function. -A stream can specify a maximum length for aggregate objects via the -.code set-max-length -function. Using the -.code set-max-depth -function, the maximum depth can also be specified. +The indentation mode and indentation column are automatically restored to their +previous values when +.code format +function terminates, naturally or via an exception or non-local jump. -This feature is -useful when diagnostic output is being produced, and the objects involved are -so large that the diagnostic output overwhelms the output device or the user, -so as to become uninformative. Output limiting also prevents the printer's -non-termination on infinite, lazy structures. +The effect of a precision field (even if zero) combined with the +.code ! +directive is currently not specified, and reserved for future extension. +The precision field is processed syntactically, and no error occurs, however. +.RE -It is recommended that functions which operate on streams passed in as -parameters save and restore these parameters, if they need to manipulate them, -for instance using -.codn with-resources : +.coNP Function @ fmt +.synb +.mets (fmt < format-string << format-arg *) +.syne +.desc +The +.code fmt +function provides a shorthand for formatting to a string, according +to the following equivalence which holds between +.code fmt +and +.codn format : .verb - (defun output-function (arg stream) - ;; temporarily impose maximum width and depth - (with-resources ((ml (set-max-length stream 42) - (set-max-length stream ml)) - (mw (set-max-depth stream 12) - (set-max-depth stream mw))) - (prinl arg stream) - ...)) + (fmt s arg ...) <--> (format nil s arg ...) .brev -.coNP Function @ set-max-length +.coNP Functions @, print @, pprint @, prinl @, pprinl @ tostring and @ tostringp .synb -.mets (set-max-length < stream << value ) +.mets (print < obj >> [ stream <> [ pretty-p ]]) +.mets (pprint < obj <> [ stream ]) +.mets (prinl < obj <> [ stream ]) +.mets (pprinl < obj <> [ stream ]) +.mets (tostring << obj ) +.mets (tostringp << obj ) .syne .desc The -.code set-max-length -function establishes the maximum length for aggregate object printing. -It affects the printing of lists, vectors, hash tables, strings -as well as quasiliterals and quasiword list literals (QLLs). +.code print +and +.code pprint +functions render a printed character representation of the +.meta obj +argument into +.metn stream . -The default value is 0 and this value means that no limit is imposed. -Otherwise, the value must be a positive integer. - -When the list, vector or hash table object being printed has more -elements than the maximum length, then elements are printed only up to -the maximum count, and then the remaining elements are summarized by -printing the -.code ... -(three dots) character sequence as if it were an additional element. -This sequence is an invalid token; it cannot be read as input. - -When a character string is printed, any positive value of -the maximum length which is less than 15 is considered to be 15. -The maximum length specifies the number of characters of the -a string which are output. +If the +.meta stream +argument is not supplied, then +the destination is the stream currently stored in the +.code *stdout* +variable. -If a string which exceeds the maximum length is being printed -with read-print consistency, as by the +If Boolean argument +.meta pretty-p +is not supplied or is explicitly specified as +.codn nil , +then the .code print -function, then only a prefix of the string is printed, limited -to the maximum number of characters. Then, the literal syntax is -closed using the character sequence -.code \e...\(dq -(backslash, dot, dot, dot, double quote) -whose leading invalid escape sequence -.code \e. -(backslash, dot) ensures that the truncated object is not readable. - -If a string which exceeds the maximum length is being printed -without read-print consistency, as by the -.code pprint -function, then only a prefix of the string is printed, limited -to the maximum number of characters. Then the -character sequence -.code ... -is emitted. - -Quasiliterals are treated using a combination of behaviors. Elements of a -quasiliteral are literal sequence of text, and embedded variables and -expressions. The maximum length specifies both the maximum number of elements -in the quasiliteral, and the maximum number of characters in any element which -is a sequence of text. When either limit is exceeded, the quasiliteral -is immediately terminated with the sequence -.code \e...` -(escaped dot, dot, dot, backtick). The maximum limit is applied to -the units of text cumulatively, rather than individually. As in the case of -string literals, smaller limit values than 15 are treated as 15, -but only for the cumulative text length limit. For limiting the number of -elements, the count is used as-is. - -When a QLL is printed, the space-separated elements -of the literal are individually subject to the maximum length limit as if -they were independent quasiliterals. Furthermore, the sequence of these -elements is subject to the maximum length. If there are more elements in the -QLL, then the sequence -.code \e...` -(escaped dot, dot, dot, backtick) is emitted and thus the QLL ends. - -The -.code set-max-length -function returns the previous value. - -.coNP Function @ set-max-depth -.synb -.mets (set-max-depth < stream << value ) -.syne -.desc -The -.code set-max-length -function establishes the maximum depth for the printing of nested -objects. It affects the printing of lists, vectors, hash tables -and structures. The default value is 0 and this value means that no limit is -imposed. Otherwise, the value must be a positive integer. - -The depth of an object not enclosed in any object is zero. The depth of the -element of an aggregate is one greater than the depth of the aggregate itself. -For instance, given the list -.code "(1 (2 3))" -the list itself has depth 0, the atom -.code 1 -has depth 1, as does the sublist -.codn "(2 3)" , -and the -.code 2 -and -.code 3 -atoms have depth 2. +function renders in a way which strives for read-print +consistency: an object is printed in a notation which is recognized as +a similar object of the same kind when it appears in \*(TX source code. +Floating-point objects are printed as if using the +.code format +function, with formatting controlled by the +.code *print-flo-format* +variable. -When an object is printed whose depth exceeds the maximum depth, then three dot -character sequence -.code ... -is printed instead of that object. This notation is an invalid token; it cannot be -read as input. +If +.meta pretty-p +is true, then +.code print +does not strive for read-print consistency. +Strings are printed by sending their characters to the output +stream, as if by the +.code put-string +function, rather than being rendered in the string literal notation +consisting of double quotes, and escape sequences for control +characters. Likewise, character objects are printed via +.code put-char +rather than the +.code #\e +notation. Buffer objects are printed by sending their bytes to the +output stream using +.code put-byte +rather than being rendered in the +.code #b +notation. +Symbols are printed without their package prefix, except that +symbols from the keyword package are still printed with the leading colon. +Floating-point objects are printed as if using the +.code format +function, with formatting controlled by the +.code *pprint-flo-format* +variable. -Additionally, when a vector, list, hash table or structure is printed which itself -doesn't exceed the maximum depth, but whose elements do exceed, then that object -is summarized, respectively, as -.codn "(...)" , -.codn "#(...)" , -.code "H#(...)" -and -.codn "S#(...)" , -rather than repeating the -.code ... -sequence for each of its elements. +When aggregate objects like conses, ranges and vectors are printed, +the notations of these objects themselves are unaffected by the +.code pretty-p +flag; however, that flag is distributed to the elements. The -.code set-max-depth -function returns the previous value. - -.SS* Coprocesses -.coNP Functions @ open-command and @ open-process -.synb -.mets (open-command < system-command <> [ mode-string ]) -.mets (open-process < program < mode-string <> [ argument-list ]) -.syne -.desc -These functions spawn external programs which execute concurrently -with the \*(TX program. Both functions return a unidirectional stream for -communicating with these programs: either an output stream, or an input -stream, depending on the contents of -.metn mode-string . - -In -.codn open-command , -the -.meta mode-string -argument is optional, defaulting to -the value -.str r -if it is missing. See the -.code open-file -function for a discussion of modes. +.code print +function returns +.metn obj . The -.code open-command -function accepts, via the -.meta system-command -string parameter, a -system command, which is in a system-dependent syntax. On a POSIX system, this -would be in the POSIX Shell Command Language. +.code pprint +("pretty print") function is equivalent to +.codn print , +with the +.meta pretty-p +argument hard-coded true. The -.code open-process -function specifies a program to invoke via the -.meta command -argument. This is subject to the operating system's search strategy. -On POSIX systems, if it is an absolute or relative path, it is treated as -such, but if it is a simple base name, then it is subject to searching -via the components of the PATH environment variable. If open-process -is not able to find -.metn program , -or is otherwise unable to execute -the program, the child process will exit, using the value of the C variable -.code errno -as its exit status. This value can be retrieved via -.codn close-stream . +.code prinl +function ("print and new line") behaves like a call to +.code print +with +.meta pretty-p +defaulting to +.codn nil , +followed by issuing a newline characters to the stream. The -.meta mode-string -argument follows the convention used by the POSIX -.code popen -function. +.code pprinl +function ("pretty print and new line") behaves like +.code pprint +followed by issuing a newline to the stream. The -.meta argument-list -argument is a list of strings which specifies additional -optional arguments to be passed passed to the program. The -.meta program -argument -becomes the first argument, and -.meta argument-string -become the second and -subsequent arguments. If -.meta argument-strings -is omitted, it defaults to empty. - -If a coprocess is open for writing -.mono -.meti >> ( mode-string -.onom -is specified as -.strn w ), -then -writing on the returned stream feeds input to that program's standard input -file descriptor. Indicating the end of input is performed by closing the -stream. - -If a coprocess is open for reading -.mono -.meti >> ( mode-string -.onom -is specified as -.strn r ), -then -the program's output can be gathered by reading from the returned stream. -When the program finishes output, it will close the stream, which can be -detected as normal end of data. - -The standard input and error file descriptors of an input coprocess -are obtained from the streams stored in the -.code *stdin* +.code tostring and -.code *stderr* -special variables, respectively. Similarly, the standard output and error -file descriptors of an output coprocess are obtained from the -.code *stdout* +.code tostringp +functions are like +.code print and -.code *stderr* -special variables. These variables must contain streams on which the -.code fileno -function is meaningful, otherwise the operation will fail. -What this functionality means is that re-binding the special variables -for standard streams has the effect of redirection. For example, -the following two expressions achieve the same effect of creating -a stream which reads the output of the -.code cat -program, which reads and produces the contents of the file -.codn text-file . - -.verb - ;; redirect input by rebinding *stdin* - (let ((*stdin* (open-file "text-file"))) - (open-command "cat")) - - ;; redirect input using POSIX shell redirection syntax - (open-command "cat < text-file") -.brev +.codn pprint , +but they do not accept a stream argument. Instead they print to a freshly +instantiated string stream, and return the resulting string. -The following is erroneous: +The following equivalences hold between calls to the +.code format +function and calls to the above functions: .verb - ;; (let ((*stdin* (make-string-input-stream "abc"))) - (open-command "cat")) + (format stream "~s" obj) <--> (print obj stream) + (format t "~s" obj) <--> (print obj) + (format t "~s\en" obj) <--> (prinl obj) + (format nil "~s" obj) <--> (tostring obj) .brev -A string input or output stream doesn't have an operating system file -descriptor; it cannot be passed to a coprocess. - -The streams -.codn *stdin* , -.code *stdout* +For +.codn pprint , +.code tostringp and -.code *stderr* -are not synchronized with their underlying file descriptors prior to -the execution of a coprocess. It is up to the program to ensure that -previous output to -.code *stdout* -or -.code *stderr* -is flushed, so that the output of the coprocess isn't re-ordered -with regard to output produced by the program. Similarly, -input buffered in -.code *stdin* -is not available to the coprocess, even though it has not -yet been read by the program. The program is responsible for preventing this -situation also. - -If a coprocess terminates abnormally or unsuccessfully, an exception is raised. +.codn pprinl , +the equivalence is produced by using +.code ~a +in format rather than +.codn ~s . -.SS* I/O-Related Convenience Functions +.TP* Notes: +For floating-point numbers, the above description of the behavior in +terms of the format specifiers +.code ~s +and +.code ~a +only applies with respect to the default values of the variables +.code *print-flo-format* +and +.codn *pprint-flo-format* . -The functions in this group create a stream, perform an I/O operation -on it, and ensure that it is closed, in one convenient operation. They -operate on files or command streams. +For characters, the print function behaves as follows: most control +characters in the Unicode +.code C0 +and +.code C1 +range are rendered using the +.code #\ex +notation, +using two hex digits. Codes in the range +.code D800 +to +.codn DFFF , +and the codes +.code FFFE +and +.code FFFF +are printed in the +.code #\exNNNN +with four hexadecimal digits, and +character above this range are printed using the same notation, but with six +hexadecimal digits. Certain characters in the +.code C0 +range are printed using +their names such as +.code #\enul +and +.codn #\ereturn , +which are documented +in the Character Literals section. +The +.code DC00 +character is printed as +.codn #\epnul . +All other characters are printed as +.mono +.meti >> #\e char +.onom +where +.meta char +is the actual character. -Several other functions in this category exist, which operate with buffers. -They are documented in the Buffer Functions subsection under the -FOREIGN FUNCTION INTERFACE section. +Caution: read-print consistency is affected by trailing material. If additional +digits are printed immediately after a number without intervening whitespace, +they extend that number. If hex digits are printed after the character +.codn x , +which is rendered as +.codn #\ex , +they look like a hex character code. -.coNP Functions @, file-get @ file-get-string and @ file-get-lines +.coNP Function @ tprint .synb -.mets (file-get << name ) -.mets (file-get-string << name ) -.mets (file-get-lines << name ) +.mets (tprint < obj <> [ stream ]) .syne .desc The -.code file-get -function opens a text stream over the file indicated by the string argument -.meta name -for reading, reads the printed representation of a \*(TL object from it, -and returns that object, ensuring that the stream is closed. +.code tprint +function prints a representation of +.meta obj +on +.metn stream . -The -.code file-get-string -is similar to -.code file-get -except that it reads the entire file as a text stream and returns -its contents in a single character string. +If the stream argument is not supplied, then +the destination is the stream currently stored in the +.code *stdout* +variable. -The -.code file-get-lines -function opens a text stream over the file indicated by -.meta name -and returns produces a lazy list of strings representing the lines -of text of that file as if by a call to the -.code get-lines -function, and returns that list. The stream remains open until the -list is consumed to the end, as indicated in the description of -.codn get-lines . +For all object types except lists and vectors, +.code tprint +behaves like +.codn pprinl . -.coNP Functions @, file-put @ file-put-string and @ file-put-lines +If +.code obj +is a list or vector, then +.code tprint +recurses: the +.code tprint +function is applied to each element. An empty list or vector +results in no output at all. This effectively means that an arbitrarily nested +structure of lists and vectors is printed flattened, with one element on each +line. + +.coNP Function @ display-width .synb -.mets (file-put < name << obj ) -.mets (file-put-string < name << string ) -.mets (file-put-lines < name << list ) +.mets (display-width << char ) +.mets (display-width << string ) .syne .desc The -.codn file-put , -.code file-put-string -and -.code file-put-lines -functions open a text stream over the file indicated by the string argument -.metn name , -write the argument object into the file in their specific manner, -and then close the file. - -If the file doesn't exist, it is created. -If it exists, it is truncated to zero length and overwritten. - -The -.code file-put -function writes a printed representation of -.meta obj -using the -.code prinl -function. The return value is that of -.codn prinl . +.code display-width +function calculates the number of places occupied by the printed representation +of +.meta char +or +.meta string +on a monospace display which renders certain characters, such as the East Asian +kanji and other characters, using two places. -The -.code file-put-string -function writes +For a .meta string -to the stream using the -.code put-string -function. The return value is that of -.codn put-string . +argument, this value is the sum of the individual display width of the +string's constituent characters. The display width of an empty string is zero. -The -.code file-put-lines -function writes -.meta list -to the stream using the -.code put-lines -function. The return value is that of -.codn put-lines . +Control characters are assigned a display width of zero, regardless of +their display control semantics, if any. -.coNP Functions @, file-append @ file-append-string and @ file-append-lines +Characters marked by Unicode as being wide or full width, have a display +width of two. Other characters have a display width of one. + +.coNP Function @ streamp .synb -.mets (file-append < name << obj ) -.mets (file-append-string < name << string ) -.mets (file-append-lines < name << list ) +.mets (streamp << obj ) .syne .desc The -.codn file-append , -.code file-append-string -and -.code file-append-lines -functions open a text stream over the file indicated by the string argument -.metn name , -write the argument object into the stream in their specific manner, -and then close the stream. - -These functions are close counterparts of, respectively, -.codn file-get , -.code file-append-string -and -.codn file-append-lines . - -These functions behave differently when the indicated file -already exists. Rather than being truncated and overwritten, -the file is extended by appending the new data to its end. +.code streamp +function returns +.code t +if +.meta obj +is any type of stream. Otherwise it returns +.codn nil . -.coNP Functions @, command-get @ command-get-string and @ command-get-lines +.coNP Function @ real-time-stream-p .synb -.mets (command-get << cmd ) -.mets (command-get-string << cmd ) -.mets (command-get-lines << cmd ) +.mets (real-time-stream-p << obj ) .syne .desc The -.code command-get -function opens text stream over an input command pipe created for -the command string -.metn cmd , -as if by the -.code open-command -function. It reads the printed representation of a \*(TL object from it, and -returns that object, ensuring that the stream is closed. +.code real-time-streamp-p +function returns +.code t +if +.meta obj +is a stream marked as +"real-time". If +.meta obj +is not a stream, or not a stream marked as "real-time", +then it returns +.codn nil . -The -.code command-get-string -is similar to -.code command-get -except that it reads the entire file as a text stream and returns -its contents in a single character string. +Only certain kinds of streams accept the real-time attribute: file streams and +tail streams. This attribute controls the semantics of the application of +.code lazy-stream-cons +to the stream. For a real-time stream, +.code lazy-stream-cons +returns a stream with "naive" semantics which returns data as soon as it is +available, at the cost of generating spurious +.code nil +item when the stream +terminates. The application has to recognize and discard that +.code nil +item. +The ordinary lazy streams read ahead by one line and suppress this extra +item, so their representation is more accurate. -The -.code command-get-lines -function opens a text stream over an input command pipe created for the -command string -.meta cmd -and returns produces a lazy list of strings representing the lines -of text of that file as if by a call to the -.code get-lines -function, and returns that list. The stream remains open until the -list is consumed to the end, as indicated in the description of -.codn get-lines . +When \*(TX starts up, it automatically marks the +.code *std-input* +stream as real-time, if it is connected to a TTY device (a device for which +the POSIX function +.code isatty +reports true). This is only supported on platforms that have this function. +The behavior is overridden by the +.code -n +command line option. -.coNP Functions @, command-put @ command-put-string and @ command-put-lines +.coNP Function @ open-file .synb -.mets (command-put < cmd << obj ) -.mets (command-put-string < cmd << string ) -.mets (command-put-lines < cmd << list ) +.mets (open-file < path <> [ mode-string ]) .syne .desc The -.codn command-put , -.code command-put-string -and -.code command-put-lines -functions open an output text stream over an output command pipe created -for the command specified in the string argument -.metn cmd , -as if by the -.code open-command -function. -They write the argument object into the stream in their specific manner, -and then close the stream. - -The -.code command-put -function writes a printed representation of -.meta obj -using the -.code prinl -function. The return value is that of -.codn prinl . - -The -.code command-put-string -function writes -.meta string -to the stream using the -.code put-string -function. The return value is that of -.codn put-string . +.code open-file +function creates a stream connected to the file +which is located at the given +.metn path , +which is a string. The -.code command-put-lines -function writes -.meta list -to the stream using the -.code put-lines -function. The return value is that of -.codn put-lines . +.meta mode-string +argument is a string which uses the same +conventions as the mode argument of the C language +.code fopen +function, with greater permissiveness, and some extensions. -.SS* User-Defined Streams +The syntax of mode-string is described by the following +grammar. Note that it permits no whitespace characters: -In \*(TL, stream objects aren't structure types, and therefore lie outside of -the object-oriented programming system. However, \*(TL supports a delegation -mechanism which allows a structure which provides certain methods to be used as -a stream. +.mono +.mets < mode-string := [ < mode ] [ < options ] +.mets < mode := { < selector [ + ] | + } +.mets < selector := { r | w | a } +.mets < options := { b | l | u | < digit } +.mets < digit := { 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 } +.onom -The function -.code make-struct-delegate-stream -takes as an argument the instance of a structure, which is -referred to as the -.IR "stream interface object" . -The function returns a stream object such that when -stream operations are invoked on this stream, it delegates these -operations to methods of the stream interface object. +If the +.meta mode-string +argument is omitted, the behavior is the same as an empty +mode string. -A structure type called -.code stream-wrap -is provided, whose instances can serve as stream interface objects. -This structure has a slot called -.meta stream -which holds a stream, and it provides all of the methods required for -the delegation mechanism used by -.codn make-struct-delegate-stream . +The +.meta mode +part of the mode string generates the following possibilities: +.RS +.meIP empty +If the +.meta mode +is missing, then a default mode is implied. The default +is specific to the particular stream-opening function. In +the case of +.codn open-file , +the default mode is +.codn r . +.coIP + +A +.meta mode +consisting of just the +.code + +character is equivalent to +.codn r+ . +.coIP r This -.code stream-wrap -operations simply invoke the ordinary stream operations on the -.meta stream -slot. The -.code stream-wrap -type can be used as a base class for a derived class which intercepts -certain operations on a stream (by defining the corresponding methods) while -allowing other operations to transparently pass to the stream (via the -base methods inherited from -.codn stream-wrap ). +.meta mode +means that the file is opened for reading. +.coIP r+ +The file is opened for reading and writing. It is not created +if it doesn't exist. +.coIP w +The file is opened for writing. If it exists, it is truncated +to zero length. If it doesn't exist, it is created. +.coIP w+ +The file is opened for reading and writing. If it exists, it +is truncated to zero length. If it doesn't exist, it is created. +.coIP a +The file is opened for writing. If it doesn't exist, it is +created. If it exists, the current position is advanced to +one byte past the end of the file, so that newly written data +are appended. +.coIP a+ +The file is opened for reading and writing. If it doesn't exist, +it is created. The read position is at the beginning of the file, +but writes are appended to the end regardless of the position. +.RE +.IP +The meanings of the option characters are: +.RS +.coIP b +The file is opened in binary mode: no line ending translation takes place. +In the absence of this option, files are opened in text mode, in which newline +characters in the stream are an abstract indication of the end of a line, +translate to a system-specific way of terminating lines in text files. +.coIP l +Specifies that the stream will be line buffered. This means that an implicit +flush operation takes place whenever the newline character is output. +.coIP u +Specifies that the stream will be unbuffered. It is erroneous for both +.code l +and +.code u +to be specified. +.coIP i +Specifies that the stream will have the real-time +property set. For a description of the semantics, see the +.code real-time-stream-p +function. Briefly, this property affects the semantics of lazy lists which draw +input from the stream. +In addition, for a stream opened for writing or reading and writing, the +.code i +mode letter specifies that the stream will be line buffered, unless +specified as unbuffered with +.codn u . +.meIP digit +A decimal digit specifies the the stream buffer size +as binary exponential buffer size order, such that +.code 0 +specifies 1024 bytes, +.code 1 +specifies 2048 and so forth up to +.code 9 +specifying 524288 bytes. If no such digit is specified, then the +stream uses a default buffer size. It is erroneous for the +size order digit to be present together with the option +.codn u . +.RE -.coNP Function @ make-struct-delegate-stream +.coNP Function @ open-tail .synb -.mets (make-struct-delegate-stream << object ) +.mets (open-tail < path >> [ mode-string <> [ seek-to-end-p ]]) .syne .desc The -.code make-struct-delegate-stream -function returns a stream whose operations depend on the -.metn object , -a stream interface object. +.code open-tail +function creates a tail stream connected to the file which is +located at the given +.metn path . +The +.meta mode-string +argument is a string which uses +the same conventions as the mode argument of the C language +.code fopen +function. If this argument is omitted, then +.str r +is used. +See the +.code open-file +function for a discussion of modes. The -.meta object -argument must be a structure which implements certain -subsets of, or all of, the following methods: -.codn put-string , -.codn put-char , -.codn put-byte , -.codn get-line , -.codn get-char , -.codn get-byte , -.codn unget-char , -.codn unget-byte , -.codn put-buf , -.codn fill-buf , -.codn close , -.codn flush , -.codn seek , -.codn truncate , -.codn get-prop , -.codn set-prop , -.codn get-error , -.codn get-error-str , -.code clear-error -and -.codn get-fd . +.code seek-to-end-p +argument is a Boolean which determines whether the initial +read/write position is at the start of the file, or just past the end. +It defaults to +.codn nil . +This argument only makes a difference if the file exists +at the time +.code open-tail +is called. If the file does not exist, and is later +created, then the tail stream will follow that file from the beginning. In +other words, +.meta seek-to-end-p +controls whether the tail stream reads all the +existing data in the file, if any, or whether it reads only newly added data +from approximately the time the stream is created. -Implementing -.code get-prop -is mandatory, and that method must support the -.code :name -property. +A tail stream has special semantics with regard to reading at the end +of file. A tail stream never reports an end-of-file condition; instead +it polls the file until more data is added. Furthermore, if the file +is truncated, or replaced with a smaller file, the tail stream follows +this change: it automatically opens the smaller file and starts reading from +the beginning (the +.meta seek-to-end-p +flag only applies to the initial open). +In this manner, a tail stream can dynamically growing rotating log files. -Failure to implement some of the other methods will impair the use of certain -stream operations on the object. +Caveat: since a tail stream can re-open a new file which has the same +name as the original file, it behave incorrectly if the program +changes the current working directory, and the path name is relative. -.coNP Method @ put-string +.coNP Function @ open-directory .synb -.mets << stream .(put-string str) +.mets (open-directory << path ) .syne .desc The -.code put-string -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code put-string -stream I/O function. +.code open-directory +function tries to create a stream which reads the +directory given by the string argument +.metn path . +If a filesystem object exists +under the path, is accessible, and is a directory, then the function +returns a stream. Otherwise, a file error exception is thrown. -.coNP Method @ put-char -.synb -.mets << stream .(put-char chr) -.syne -.desc +The resulting stream supports the get-line operation. Each call to the +.code get-line +operation retrieves a string representing the next directory +entry. The value +.code nil +is returned when there are no more directory entries. The -.code put-char -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code put-char -stream I/O function. +.code . +and +.code .. +entries in Unix filesystems are not skipped. -.coNP Method @ put-byte +.coNP Function @ make-string-input-stream .synb -.mets << stream .(put-byte byte) +.mets (make-string-input-stream << string ) .syne .desc The -.code put-byte -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code put-byte -stream I/O function. +.code make-string-input-stream +function produces an input stream object. Character read operations on the +stream object read successive characters from +.metn string . +Output operations and byte operations are not supported. -.coNP Method @ get-line +.coNP Function @ make-string-byte-input-stream .synb -.mets << stream .(get-line) +.mets (make-string-byte-input-stream << string ) .syne .desc The -.code get-line -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code get-line -stream I/O function. +.code make-string-byte-input-stream +function produces an input stream object. Byte read operations on +this stream object read successive byte values obtained by encoding +.meta string +into UTF-8. Character read operations are not supported, and neither +are output operations. -.coNP Method @ get-char +.coNP Function @ make-strlist-input-stream .synb -.mets << stream .(get-char) +.mets (make-strlist-input-stream << list ) .syne .desc The -.code get-char -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code get-char -stream I/O function. +.code make-strlist-input-stream +function produces an input stream object based on a list of strings. +Through the character read operations invoked on this stream, +the list of strings appears as a list of newline-terminated lines. +Output operations and byte operations are not supported. -.coNP Method @ get-byte +.coNP Function @ make-string-output-stream .synb -.mets << stream .(get-byte) +.mets (make-string-output-stream) .syne .desc The -.code get-byte -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code get-byte -stream I/O function. +.code make-string-output-stream +function, which takes no arguments, creates a string output stream. +Data sent to this stream is accumulated into a string object. +String output streams supports both character and byte output operations. +Bytes are assumed to represent a UTF-8 encoding, and are decoded in order +to form characters which are stored into the string. -.coNP Method @ unget-char +If an incomplete UTF-8 code is output, and a character output operation then +takes place, that code is assumed to be terminated and is decoded as invalid +bytes. The UTF-8 decoding machine is reset and ready for the start of a new +code. + +The +.code get-string-from-stream +function is used to retrieve the accumulated string. + +If the null character is written to a string output stream, the behavior +is unspecified. \*(TX strings cannot contain null bytes. A the pseudo-null +character +.codn #\exDC00 , +also notated +.codn #\epnul , +will produce a null byte when converted to UTF-8 and thus serves as an +effective internal representation of the null character in external data. + +.coNP Function @ get-string-from-stream .synb -.mets << stream .(unget-char chr) +.mets (get-string-from-stream << stream ) .syne .desc The -.code unget-char -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code unget-char -stream I/O function. +.meta stream +argument must be a string output stream. This function finalizes +the data sent to the stream and retrieves the accumulated character string. -.coNP Method @ unget-byte +If a partial UTF-8 code has been written to +.metn stream , +and then this +function is called, the byte stream is considered complete and the partial +code is decoded as invalid bytes. + +After this function is called, further output on the stream is not possible. + +.coNP Function @ make-strlist-output-stream .synb -.mets << stream .(unget-byte byte) +.mets (make-strlist-output-stream) .syne .desc The -.code unget-byte -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code unget-byte -stream I/O function. +.code make-strlist-output-stream +function is very similar to +.codn make-string-output-stream . +However, the stream object produced by this function does not produce a string, +but a list of strings. The data is broken into multiple strings by newline +characters written to the stream. Newline characters do not appear in the +string list. Also, byte output operations are not supported. -.coNP Method @ put-buf +.coNP Function @ get-list-from-stream .synb -.mets << stream .(put-buf buf pos) +.mets (get-list-from-stream << stream ) .syne .desc The -.code put-buf -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code put-buf -stream I/O function. +.code get-list-from-stream +function returns the string list which has accumulated inside +a string output stream given by +.metn stream . +The string output stream is +finalized, so that further output is no longer possible. -.coNP Method @ fill-buf +.coNP Macro @ with-in-string-stream .synb -.mets << stream .(fill-buf buf pos) +.mets (with-in-string-stream >> ( stream-var << string ) +.mets \ \ << body-form *) .syne .desc The -.code fill-buf -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code fill-buf -stream I/O function. +.code with-in-string-stream +macro binds the symbol +.meta stream-var +as a variable, initializing it with a newly created +string input stream. The string input stream is +constructed from +.meta string +as if by the +.mono +.meti (make-string-input-stream << string ) +.onom +expression. + +Then it evaluates the +.metn body-form -s +in the scope of the variable. + +The value of the last +.meta body-form +is returned, or else +.code nil +if no forms are present. -.coNP Method @ close -.synb -.mets << stream .(close offs whence) -.syne -.desc The -.code close -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code close-stream -stream I/O function. +.meta stream-var +argument must be a bindable symbol, +as defined by the +.code bindable +function. -.coNP Method @ flush +The +.meta string +argument must be a form +which evaluates to a character string value. + +.coNP Macro @ with-in-string-byte-stream .synb -.mets << stream .(flush offs whence) +.mets (with-in-string-byte-stream >> ( stream-var << string ) +.mets \ \ << body-form *) .syne .desc The -.code flush -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code flush-stream -stream I/O function. +.code with-in-string-byte-stream +macro binds the symbol +.meta stream-var +as a variable, initializing it with a newly created +string byte input stream. The string input stream is +constructed from +.meta string +as if by the +.mono +.meti (make-string-byte-input-stream << string ) +.onom +expression. -.coNP Method @ seek +Then it evaluates the +.metn body-form -s +in the scope of the variable. + +The value of the last +.meta body-form +is returned, or else +.code nil +if no forms are present. + +The +.meta string +argument must be a form +which evaluates to a character string value. + +.coNP Macro @ with-out-string-stream .synb -.mets << stream .(seek offs whence) +.mets (with-out-string-stream <> ( stream-var ) << body-form *) .syne .desc The -.code seek -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code seek-stream -stream I/O function. +.code with-out-string-stream +macro binds the symbol specified +by the +.meta stream-var +argument as a variable, initializing it +with a newly created string output stream. The output +stream is created as if by the +.code make-string-output-stream +function. -.coNP Method @ truncate +Then it evaluates +.metn body-form -s +in the scope of that variable. + +After these forms are evaluated, the string is extracted +from the string output stream, as if by the +.code get-string-from-stream +function, and returned as the result value +of the form. + +.coNP Macro @ with-out-strlist-stream .synb -.mets << stream .(truncate len) +.mets (with-out-strlist-stream <> ( stream-var ) << body-form *) .syne .desc The -.code truncate -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code truncate-stream -stream I/O function. +.code with-out-strlist-stream +macro binds the symbol specified +by the +.meta stream-var +argument as a variable, initializing it +with a newly created string list output stream. The output +stream is created as if by the +.code make-strlist-output-stream +function. -.coNP Method @ get-prop +Then it evaluates +.metn body-form -s +in the scope of that variable. + +After these forms are evaluated, the string list is extracted +from the string output stream, as if by the +.code get-strlist-from-stream +function, and returned as the result value +of the form. + +.coNP Function @ close-stream .synb -.mets << stream .(get-prop sym) +.mets (close-stream < stream <> [ throw-on-error-p ]) .syne .desc The -.code get-prop -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code get-prop -stream I/O function. +.code close-stream +function performs a close operation on +.metn stream , +whose meaning is depends on the type of the stream. For some types of streams, +such as string streams, it does nothing. For streams which are connected +to operating system files or devices, will perform a close of the underlying +file descriptor, and dissociate that descriptor from the stream. Any buffered +data is flushed first. -.coNP Method @ set-prop +.code close-stream +returns a Boolean true value if the close has occurred without +errors, otherwise +.codn nil . + +For most streams, "without errors" means that any buffered output data is +flushed successfully. + +For command and process pipes (see open-command and open-process), success also +means that the process terminates normally, with a successful error code, or an +unsuccessful one. An abnormal termination is considered an error, as +as is the inability to retrieve the termination status, as well as the situation +that the process continues running in spite of the close attempt. +Detecting these situations is platform specific. + +If the +.meta throw-on-error-p +argument is specified, and isn't +.codn nil , +then the +function throws an exception if an error occurs during the close operation +instead of returning +.codn nil . + +.coNP Macro @ with-stream .synb -.mets << stream .(set-prop sym nval) +.mets (with-stream >> ( stream-var << init-form ) +.mets \ \ << body-form *) .syne .desc The -.code set-prop -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code set-prop -stream I/O function. +.code with-stream +binds the variable whose name is given by the +.meta stream-var +argument, and macro arranges for the evaluation of +.metn body-form -s +in the scope of that variable. -.coNP Method @ get-error +The variable is initialized with the value produced +by the evaluation of +.meta init-form +which must be an expression which evaluates to a stream. + +After each +.meta body-form +is evaluated, the stream is closed, as if by the +.mono +.meti (close-stream << stream-var ) +.onom +expression. + +The value of the last +.meta body-form +then becomes the result value of the form, +or else +.code nil +if these forms are absent. + +If the evaluation of the +.metn body-form -s +is abandoned, the stream is still closed. That is to say, +the closure of the stream is a protected action, as if by +the +.code unwind-protect +operator. + +.coNP Functions @, get-error @ get-error-str and @ clear-error .synb -.mets << stream .(get-error) +.mets (get-error << stream ) +.mets (get-error-str << stream ) +.mets (clear-error << stream ) .syne .desc -The +When a stream operation fails, the .code get-error -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the +and +.code get-error-str +functions may be used to inquire about a more detailed cause of the error. + +Not all streams support these functions to the same extent. For instance, +string input streams have no persistent state. The only error which occurs +is the condition when the string has no more data. + +The .code get-error -stream I/O function. +inquires +.meta stream +about its error condition. + +The function returns +.code nil +to indicate there is no error condition, +.code t +to indicate an end-of-data condition, +or else a value which is specific to the stream type indicating the +specific error type. + +Note: for some streams, it is possible for the +.code t +value to be returned even though no operation has failed; that is to say, the +streams "know" they are at the end of the data even though no read operation +has failed. Code which depends on this will not work with streams which +do not thus indicate the end-of-data +.I a priori, +but by means of a read operation which fails. -.coNP Method @ get-error-str -.synb -.mets << stream .(get-error-str) -.syne -.desc The .code get-error-str -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code get-error-str -stream I/O function. +function returns a text representation of the error code. The +.code nil +error code is represented as the string +.codn "no error" ; +the +.code t +error code as +.code "eof" +and other codes have a stream-specific representation. -.coNP Method @ clear-error -.synb -.mets << stream .(clear-error) -.syne -.desc The .code clear-error -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code clear-error -stream I/O function. +function removes the error situation from a stream. On some streams, it does +nothing. If an error has occurred on a stream, this function should be called +prior to re-trying any I/O or positioning operations. +The return value is the previous error code, or +.code nil +if there was no error, or the operation is not supported on the stream. -.coNP Method @ get-fd +.coNP Functions @, get-line @ get-char and @ get-byte .synb -.mets << stream .(get-fd) +.mets (get-line <> [ stream ]) +.mets (get-char <> [ stream ]) +.mets (get-byte <> [ stream ]) .syne .desc -The -.code get-fd -method is implemented on a stream interface object. -It should behave in a manner consistent with the -description of the -.code fileno -stream I/O function. +These fundamental stream functions perform input. The +.meta stream +argument +is optional. If it is specified, it should be an input stream which supports +the given operation. If it is not specified, then the +.code *stdin* +stream is used. -.coNP Structure @ stream-wrap -.synb -.mets (defstruct stream-wrap nil -.mets \ \ stream -.mets \ \ (:method put-string (me str) -.mets \ \ \ \ (put-string str me.stream)) -.mets \ \ (:method put-char (me chr) -.mets \ \ \ \ (put-char chr me.stream)) -.mets \ \ (:method put-byte (me byte) -.mets \ \ \ \ (put-byte byte me.stream)) -.mets \ \ (:method get-line (me) -.mets \ \ \ \ (get-line me.stream)) -.mets \ \ (:method get-char (me) -.mets \ \ \ \ (get-char me.stream)) -.mets \ \ (:method get-byte (me) -.mets \ \ \ \ (get-byte me.stream)) -.mets \ \ (:method unget-char (me chr) -.mets \ \ \ \ (unget-char chr me.stream)) -.mets \ \ (:method unget-byte (me byte) -.mets \ \ \ \ (unget-byte byte me.stream)) -.mets \ \ (:method put-buf (me buf pos) -.mets \ \ \ \ (put-buf buf pos me.stream)) -.mets \ \ (:method fill-buf (me buf pos) -.mets \ \ \ \ (fill-buf buf pos me.stream)) -.mets \ \ (:method close (me) -.mets \ \ \ \ (close-stream me.stream)) -.mets \ \ (:method flush (me) -.mets \ \ \ \ (flush-stream me.stream)) -.mets \ \ (:method seek (me offs whence) -.mets \ \ \ \ (seek-stream me.stream offs whence)) -.mets \ \ (:method truncate (me len) -.mets \ \ \ \ (truncate-stream me.stream len)) -.mets \ \ (:method get-prop (me sym) -.mets \ \ \ \ (stream-get-prop me.stream sym)) -.mets \ \ (:method set-prop (me sym nval) -.mets \ \ \ \ (stream-set-prop me.stream sym nval)) -.mets \ \ (:method get-error (me) -.mets \ \ \ \ (get-error me.stream)) -.mets \ \ (:method get-error-str (me) -.mets \ \ \ \ (get-error-str me.stream)) -.mets \ \ (:method clear-error (me) -.mets \ \ \ \ (clear-error me.stream)) -.mets \ \ (:method get-fd (me) -.mets \ \ \ \ (fileno me.stream))) -.syne -.desc The -.code stream-wrap -class provides a trivial implementation of a stream interface. -It has a single slot, -.code stream -which should be initialized with a stream object. -Each methods of -.metn stream-wrap , -shown in its entirety in the above Syntax section, simply -invoke the corresponding stream I/O library functions, passing -the method arguments, and the value of the -.code stream -slot to that function, and consequently returning whatever that function -returns. - -Note: the -.code stream-wrap -function is intended to useful as an inheritance base. A user-defined -structure can inherit from -.code stream-wrap -and provide its own versions of some of the methods, thereby intercepting -those operations to customize the behavior. - -For instance, a function equivalent to the -.code record-adapter -function could be implemented by constructing an object derived from -.code stream-wrap -which overrides the behavior of the +.code get-char +function pulls a character from a stream which supports character +input. Streams which support character input also support the .code get-line -method, and then using the -.code make-struct-delegate-stream -to return a stream based on this object. - -.TP* Example: -.verb - ;;; Implementation of my-record-adapter, - ;;; a function resembling - ;;; the record-adapter implementation +function which extracts a line of text delimited by the end of the stream or a +newline character and returns it as a string. (The newline character does not +appear in the string which is returned). - (defstruct rec-input stream-wrap - regex - include-match-p +Character input from streams based on bytes requires UTF-8 decoding, so that +get-char actually may read several bytes from the underlying low level +operating system stream. - ;; get-line overridden to use regex-based - ;; extraction using read-until-match - (:method get-line (me) - (read-until-match me.regex me.stream - me.include-match-p)))) +The +.code get-byte +function bypasses UTF-8 decoding and reads raw bytes from +any stream which supports byte input. Bytes are represented as integer +values in the range 0 to 255. - (defun my-record-adapter (regex stream include-match-p) - (let ((recin (new rec-input - stream stream - regex regex - include-match-p include-match-p))) - (make-struct-delegate-stream recin))) -.brev +Note that if a stream supports both byte input and character input, then mixing +the two operations will interfere with the UTF-8 decoding. -.SS* Symbols and Packages -\*(TL has a package system inspired by the salient features of ANSI Common -Lisp, but substantially simpler. +These functions return +.code nil +when the end of data is reached. Errors are +represented as exceptions. -Each symbol has a name, which is a string. +See also: +.code get-lines -A package is an object which serves as a container of symbols; the package -associates the name strings with symbols. +.coNP Function @ get-string +.synb +.mets (get-string >> [ stream >> [ count <> [ close-after-p ]]]) +.syne +.desc +The +.code get-string +function reads characters from a stream, and assembles them into +a string, which is returned. If the +.meta stream +argument is omitted, then the +.code *stdin* +stream is used. -A symbol which exists inside a package is said to be interned in that package. -A symbol can be interned in more than one package. +The stream is closed after extracting the data, unless +.meta close-after-p +is specified as +.codn nil . +The default value of this argument is +.codn t . -A symbol may also have a home package. A symbol which has a home package -is always interned in that package. +If the +.meta count +argument is missing, then all of the characters from the +stream are read and assembled into a string. -A symbol which has a home package is called an -.IR "interned symbol" . +If present, the +.meta count +argument should be a positive integer indicating +a limit on how many characters to read. The returned string will be no +longer than +.metn count , +but may be shorter. -A symbol which is interned in one or more packages, but has no home package, -is a -.IR "quasi-interned symbol" . -When a quasi-interned symbol is printed, if it is not interned in -the package currently held in the -.code *package* -variable, it will appear in uninterned notation denoted by a -.code #: -prefix, even though it is interned in one or more packages. -This is because in any situation when a symbol is printed with a package -prefix, that prefix corresponds to the name of its home package. -The reverse isn't true: when a symbol token is read bearing a package -prefix, the token denotes any interned symbol in the indicated package, -whether or not the package is the home package of that symbol. +.coNP Functions @ unget-char and @ unget-byte +.synb +.mets (unget-char < char <> [ stream ]) +.mets (unget-byte < byte <> [ stream ]) +.syne +.desc +These functions put back, into a stream, a character or byte which was +previously read. The character or byte must match the one which was most +recently read. If the +.meta stream +argument is omitted, then the +.code *stdin* +stream is used. -Packages are held in a global list which can be used to search for a package by -name. The -.code find-package -function performs this lookup. A package may be -deleted from the list with the -.code delete-package -function, but it continues -to exist until the program loses the last reference to that package. -When a package is deleted with -.codn delete-package , -its symbols are uninterned from all other packages. +If the operation succeeds, the byte or character value is returned. +A +.code nil +return indicates that the operation is unsupported. -An existing symbol can be brought into a package via the -.code use-sym -function, causing it to be interned in that package. A symbol which thus exists -inside a package which is not its home package is called a -.IR "foreign symbol" , -relative to that package. -The contrasting term with -.I "foreign symbol" -is -.IR "local symbol" , -which refers to a symbol, relative to a package, which is interned in that -package and that package is also its home. Every symbol interned in -a package is either foreign or local. +Some streams do not support these operations; some support +only one of them. In general, if a stream supports +.codn get-char , +it supports +.codn unget-char , +and likewise for +.code get-byte +and +.codn unget-byte . -If a foreign symbol is introduced into a package, and has the same name -as an existing local symbol, the local symbol continues to exist, but -is hidden: it is not accessible via a name lookup on that package. -While hidden, a symbol loses its home package and is thus -degraded to either quasi-interned or uninterned status, depending -on whether that symbol is interned in other packages. +Streams may require a pushed back byte or character to match +the character which was previously read from that stream +position, and may not allow a byte or character to be pushed +back beyond the beginning of the stream. -When a foreign symbol is removed from a package via -.codn unuse-sym , -then if a hidden symbol exists in that package of the same name, -that hidden symbol is re-interned in that package and re-acquires -that package as its home package, becoming an interned symbol again. +Space may be available for only one byte of pushback under the +.code unget-byte +operation. -Finally, packages have a -.IR "fallback package list" : -a list of associated packages, which may be empty. The fallback package -list is manipulated with the functions -.code package-fallback-list -and -.codn set-package-fallback-list , -and with the -.code :fallback -clause of the -.code defpackage -macro. The fallback package list plays a role only in three situations: -one in the \*(TL parser, one in the printer, and one in the interactive -listener. +The number of characters that may be pushed back by +.code unget-char +is not limited. -The parser situation involving the fallback list occurs when the -\*(TL parser resolves an unqualified symbol token: a symbol token not carrying -a package prefix. Such a symbol name is resolved against the current package -(the package currently stored in the -.code *package* -special variable). If a symbol matching the token -is not found in the current package, then the packages in its fallback -package list are searched for the symbol. The first matching symbol which is -found in the fallback list is returned. If no matching symbol is found in the -fallback list, then the token is interned as a new symbol is interned in the -current package. The packages in the current package's fallback list may -themselves have fallback lists. Those fallback lists are not involved; no such -recursion takes place. +Pushing both a byte and a character, in either order, is also unsupported. +Pushing a byte and then reading a character, or pushing a character and +reading a byte, are unsupported mixtures of operations. -The printer situation involving the fallback list is as follows. -If a symbol is being printed in a machine-readable way (not "pretty"), -has a home package and is not a keyword symbol, then a search takes place -through the current package and its fallback list. If the symbol is found -in any of those places, and if those places are devoid of any symbols -which have the same name, thus causing ambiguity, then the symbol is printed -without a package prefix. +If the stream is binary, then pushing back a byte decrements its position, +except if the position is already zero. At that point, the position becomes +indeterminate. -The listener situation involving the fallback list is a follows. -When tab completion is used on a symbol without a package -prefix, the listener searches for completions not only in the current -package, but in the fallback list also. +The behavior of pushing back immediately after a +.code seek-stream +positioning operation is unspecified. -.TP* "Dialect Notes:" +.coNP Functions @, put-string @, put-line @ put-char and @ put-byte +.synb +.mets (put-string < string <> [ stream ]) +.mets (put-line >> [ string <> [ stream ]]) +.mets (put-char < char <> [ stream ]) +.mets (put-byte < byte <> [ stream ]) +.syne +.desc +These functions perform output on an output stream. The +.meta stream +argument +must be an output stream which supports the given operation. If it is omitted, +then +.code *stdout* +is used. -The \*(TL package system doesn't support the ANSI Common Lisp -concept of package use, replacing that concept with fallback packages. +The +.code put-char +function writes a character given by +.code char +to a stream. If the +stream is based on bytes, then the character is encoded into UTF-8 and multiple +bytes are written. Streams which support +.code put-char +also support put-line, and +.codn put-string . -Though the -.code use-package -and -.code unuse-package -functions exist and are similar to the ones in ANSI CL, -they actually operate on individual foreign symbols, bringing -them in or removing them, respectively. These functions effectively -iterate over the local symbols of the used or unused package, and invoke -.code use-sym -or -.codn unuse-sym , -respectively. +The +.code put-string +function writes the characters of a string out to +the stream as if by multiple calls to put-char. The +.meta string +argument +may be a symbol, in which case its name is used as the string. -The \*(TL package system consequently doesn't support the concept -of shadowing symbols, and conflicts do not exist. When a foreign symbol is -introduced into a package which already has a symbol by that name, that symbol -is silently removed from that package if it is itself foreign, or else hidden -if it is local. +The +.code put-line +function is like +.codn put-string , +but also writes an additional newline +character. The string is optional in +.codn put-line , +and defaults to the empty string. -The \*(TL package system also doesn't feature the concept of -internal and external symbols. The rationale is that this distinction -divides symbols into subsets in a redundant way. Packages are already -subsets of symbols. A module can use two packages to simulate private -symbols. An example of this is given in the Package Examples section below. +The +.code put-byte +function writes a raw byte given by the +.meta byte +argument +to +.metn stream , +if +.meta stream +supports a byte write operation. The byte +value is specified as an integer value in the range 0 to 255. -The \*(TL fallback package list mechanism resembles ANSI CL package use, -and satisfies similar use scenarios. However, this mechanism does not cause -a symbol to be considered visible in a package. If a package -.code foo -contains no symbol -.codn bar , -but one of the packages in -.codn foo 's -fallback list does contain -.codn bar , -that symbol is nevertheless not considered visible in -.codn foo . -The syntax -.code foo:bar -will not resolve. -The fallback mechanism only comes into play when a package is installed -as the current package in the -.code *package* -variable. It then allows unqualified symbol references to refer across -the fallback list. +All these functions return +.codn t . +On failure, they do not return, but throw exceptions of type +.codn file-error . -.NP* Package Examples -The following example illustrates a simple scenario of a module -whose identifies are in a package, and which also has private identifiers -in a private package. +.coNP Functions @ put-strings and @ put-lines +.synb +.mets (put-strings < sequence <> [ stream ]]) +.mets (put-lines < sequence <> [ stream ]]) +.syne +.desc +These functions assume +.meta sequence +to be a sequence of strings, or of +symbols, or a mixture thereof. These strings are sent to the stream. The +.meta stream +argument must be an output stream. If it is omitted, then +.code *stdout* +is used. -.verb - ;; Define three packages. - (defpackage mod-priv - (:fallback usr)) +The +.code put-strings +function iterates over +.meta sequence +and writes each element +to the stream as if using the +.code put-string +function. - (defpackage mod) +The +.code put-lines +function iterates over +.code sequence +and writes each element +to the stream as if using the +.code put-line +function. - (defpackage client - (:fallback mod usr) - (:use-from mod-priv other-priv)) +Both functions return +.codn t . - ;; Switch to mod-priv package - (in-package mod-priv) +.coNP Function @ flush-stream +.synb +.mets (flush-stream <> [ stream ]) +.syne +.desc +The +.code flush-stream +function is meaningful for output streams which accumulate data +which is passed on to the operating system in larger transfer units. +Calling +.code flush-stream +causes all accumulated data inside +.meta stream +to be passed +to the operating system. If called on streams for which this function is not +meaningful, it does nothing, and returns +.codn nil . - (defun priv-fun (arg) - (list arg)) +If +.meta stream +is omitted, the current value of +.code *stdout* +is used. - ;; Another function with a name in the mod-priv package. - (defun other-priv (arg) - (cons arg arg)) +.coNP Function @ seek-stream +.synb +.mets (seek-stream < stream < offset << whence ) +.syne +.desc +The +.code seek-stream +function is meaningful for file streams. It changes the +current read/write position within +.metn stream . +It can also be used to determine the current position: see the notes about the +return value below. - ;; Define a function in mod; a public function. - ;; Note that we don't have to change to the mod package, - ;; to define functions with names in that package. - ;; We rely on interning being allowed for the qualified - ;; mod:public-fun syntax. - (defun mod:public-fun (arg) - (priv-fun arg)) ;; priv-fun here is mod-priv:priv-fun +The +.meta offset +argument is a positive or negative integer which gives a +displacement that is measured from the point identified by the +.meta whence +argument. - ;; Switch to client package - (in-package client) +Note that for text files, there isn't necessarily a 1:1 correspondence between +characters and positions due to line-ending conversions and conversions +to and from UTF-8. - (priv-fun) ;; ERROR: refers to client:priv-fun, not defined - (mod:priv-fun) ;; ERROR: mod-priv:priv-fun not used in mod - (mod-priv:priv-fun 3) ;; OK: direct reference via qualifier - (public-fun 3) ;; OK: mod:public-fun symbol via fallback - (other-priv 3) ;; OK: foreign symbol mod-priv:other-priv - ;; present in client due to :use-from -.brev +The +.meta whence +argument is one of three keywords: :from-start, :from-current +and :from-end. These denote the start of the file, the current position +and the end of the file. -The following example shows how to create a package called -.code custom -in which the -.code + -symbol from the -.code usr -package is replaced with a local symbol. A function is -then defined using the local symbol, which allows strings -to be catenated with -.codn + : +If +.meta offset +is zero, and +.meta whence +is +.codn :from-current , +then +.code seek-stream +returns the current absolute position within the +stream, if it can successfully obtain it. Otherwise, it +returns +.code t +if it is successful. -.verb - (defpackage custom - (:fallback usr) - (:local + - * /)) +If a character has been successfully put back into a text stream with +.code unget-char +and is still pending, then the position value is unspecified. If a +byte has been put back into a binary stream with +.codn unget-byte , +and the previous position wasn't zero, then the position is decremented by one. - (defmacro outside-macro (x) ^(+ ,x 42)) +On failure, it throws an exception of type +.codn stream-error . - (in-package custom) +.coNP Function @ truncate-stream +.synb +.mets (truncate-stream < stream <> [ length ]) +.syne +.desc +The +.code truncate-stream +causes the length of the underlying file associated with +.meta stream +to be set to +.meta length +bytes. - (defun binary-+ (: (left 0) (right 0)) - (if (and (numberp left) (numberp right)) - (usr:+ left right) - `@left@right`)) +The stream must be a file stream, and must be open for writing. - (defun + (. args) - [reduce-left binary-+ args]) +If +.meta length +is omitted, then it defaults to the current position, retrieved +as if by invoking the +.code seek-stream +with an +.meta offset +argument of zero and +.meta whence +argument of +.codn :from-current . +Hence, after the +.code truncate-stream +operation, that position is one byte past the end of the file. - (+) -> 0 - (+ 1) -> 1 - (+ 1 "a") -> "1a" - (+ 1 2) -> 3 - (+ "a") -> "a" - (+ "a" "b" "c") -> "abc" +.coNP Functions @ stream-get-prop and @ stream-set-prop +.synb +.mets (stream-get-prop < stream << indicator ) +.mets (stream-set-prop < stream < indicator << value ) +.syne +.desc +These functions get and set properties on a stream. Only certain properties +are meaningful with certain kinds of streams, and the meaning depends on +the stream. If two or more stream types support a property of the same name, it +is expected that the property has the same or very similar meaning for both +streams to the maximum extent that similarity is possible. - ;; macro expansions using usr:+ are not affected - (outside-macro "a") -> ;; error: + invalid operands "a" 42 -.brev +The +.code stream-set-prop +function sets a property on a stream. The +.meta indicator +argument is a symbol, usually a keyword symbol, denoting the property, +and +.meta value +is the property value. If the stream understands and accepts the +property, the function returns +.codn t . +Otherwise it returns +.codn nil . -.NP* Packages and the Extraction Language -The \*(TX extraction language has a syntax in which certain Lisp symbolic -expressions denoting directives -.code "@(collect ...)" -or -.code "@(end)" -behave as if they were the tokens of a phrase structure. As a matter of -implementation, these are processed specially in the parser and lexical -analyzer, and are not read in the same way as ordinary Lisp forms. +The +.code stream-get-prop +function inquires about the value of a property on a +stream. If the stream understands the property, then it returns its current +value. If the stream does not understand a property, nil is returned, which is +also returned if the property exists, but its value happens to be +.codn nil . -On the other hand, some directives are not this way. For instance the -.codn "@(bind ...)" , -syntax is processed as a true Lisp expression, in which the -.code bind -token is subject to the usual rules for interning a symbol, sensitive to -.code *package* -in the usual way. +The +.code :name +property is widely supported by streams of various types. It associates +the stream with a name. This property is not always modifiable. -The following notes describe the treatment of "special" directives that are -involved in phrase structure syntax. It applies to all directives which head -off a block that must be terminated by -.codn "@(end)" , -all "punctuation" directives like -.code "@(and)" -or -.code "@(end)" -and all sub-phrase indicators like -.code "@(last)" -or -.codn "@(elif)" . +File, process and stream socket I/O streams have a +.code :fd +property which can be accessed, but not modified. It retrieves +the same value as the +.code stream-fd +function. -Firstly, each such directive may have a package prefix on its main symbol, yet -is still recognized as the same token. That is to say, -.code "@(foo:collect)" -is still treated by the tokenizer and parser as the -.code "@(collect)" -token, regardless of the package prefix, and regardless of whether -.code foo:end -is the same symbol as the -.code usr:end -symbol. +The "real time" +property supported by these streams, connected with the +.code real-time-stream-p +function, also appears as the +.code :real-time +property. -However, this doesn't mean that any -.code foo:collect -is allowed to denote the -.code collect -directive. +I/O streams also have a property called +.code :byte-oriented +which, if set, suppresses the decoding of UTF-8 on character input. Rather, +each byte of the file corresponds directly to one character. Bytes +in the range 1 to 255 correspond to the character code points U+0001 +to U+00FF. Byte value 0 is mapped to the code point U+DC00. -A qualified symbol such as -.code foo:collect -must correspond to (be the same object as) precisely one of two symbols: -either the same-named symbol in the -.code usr -package, or else the same-named symbol in the -.code keyword -package. If this condition isn't satisfied, the situation is a syntax -error. Note that this check uses the original -.code usr -and -.code keyword -packages, not the packages which are currently named -.str "usr" -or -.str "keyword" -in the current -.codn *package-alist* . +The logging priority of the +.code *stdlog* +syslog stream is controlled by the +.code :prio +property. -A check is also performed for an unqualified symbol. -An unqualified symbol like -.code collect -must also resolve, in the context of the current value of the -.code *package* -variable, to the same named-symbol in either the original -.code usr -or -.code keyword -package. Thus if the current package isn't -.codn usr , -and -.code "@(collect)" -is being processed, the current package must be such that -.code collect -resolves to -.codn usr:collect . -either because that symbol is present in the current pack via -import, or else visible via the fallback list. +If +.meta stream +is a catenated stream (see the function +.codn make-catenated-stream ) +then these functions transparently operate on the current head stream of the +catenation. -These rules are designed to approximate what the behavior would be -if these directives were actually scanned as Lisp forms in the usual -way and then recognized as phrase structure tokens according to -the identity of their leading symbol. The additional restriction is added that -that the directive symbol names are treated as reserved. If there exists a -user-defined pattern function called -.code mypackage:end -it may not be invoked using the syntax -.codn "@(mypackage:end)" , -which is erroneous; though it is invokable indirectly via the -.code "@(call)" -directive. +.coNP Functions @ make-catenated-stream and @ cat-streams +.synb +.mets (make-catenated-stream << stream *) +.mets (cat-streams << stream-list ) +.syne +.desc +The +.code make-catenated-stream +function takes zero or more arguments which +are input streams of the same type, and combines +them into a single virtual stream called a catenated stream. -.NP* Package Library Conventions -Various functions in the package and symbol area of the library have a -.meta package -parameter. When the argument is optional, it defaults to the current -value of the -.code *package* -special variable. +The +.code cat-streams +function takes a single list of input streams of +the same type, and similarly combines them into a catenated stream. -If specified, the argument may be a character string, which is taken as the -name of a package. It may also be a symbol, in which case the symbol's name, -which is a character string, is used. Thus the objects -.codn :sys , -.codn usr:sys , -.code abc:sys -and -.str sys -all refer to the same package, the system package which is named -.strn sys . +A catenated stream does not support seeking operations or output, +regardless of the capabilities of the streams in the list. -A -.code package -parameter may also simply be a package object. +If the stream list is not empty, then the leftmost element of the +list is called the head stream. -Some functions, like -.code use-package +The +.codn get-char , +.codn get-byte , +.codn get-line , +.code unget-char and -.code unuse-package -functions accept a list of packages as their first argument. -This may be a list of objects which follow the above conventions: -strings, symbols or package objects. -Also, instead of a list, an atom may be passed: a string, symbol -or package object. It is treated as a singleton list consisting -of that object. +.code unget-byte +functions delegate +to the corresponding operations on the head stream, if it exists. +If the stream list is empty, they return +.code nil +to the caller. -.coNP Variables @, user-package @ keyword-package and @ system-package -.desc -These variables hold predefined packages. The -.code user-package -contains all of the public symbols in the \*(TL library. -The -.code keyword-package -holds keyword symbols, which are printed with -a leading colon. The -.code system-package -is for internal symbols, helping -the implementation avoid name clashes with user code in some situations. +If the +.codn get-char , +.code get-byte +or +.code get-line +operation on the head stream yields +.codn nil , +and there are more lists in the stream, then the stream is closed, removed from +the list, and the next stream, if any, becomes the head list. The operation is +then tried again. If any of these operations fail on the last list, it is not +removed from the list, so that a stream remains in place which can take the +.code unget-char +or +.code unget-byte +operations. -These variables shouldn't be modified. If they are modified, the consequences -are unspecified. +In this manner, the catenated streams appear to be a single stream. -The names of these packages, respectively, are -.strn usr , -.strn sys , +Note that the operations can fail due to being unsupported. It is +the caller's responsibility to make sure all of the streams in the list +are compatible with the intended operations. + +If the stream list is empty then an empty catenated stream is produced. +Input operations on this stream yield +.codn nil , +and the +.code unget-char and -.strn keyword . +.code unget-byte +operations throw an exception. -.coNP Special variable @ *package* +.coNP Function @ catenated-stream-p +.synb +.mets (catenated-stream-p << obj ) +.syne .desc -This variable holds the current package. The global value of this variable -is initialized to a package called -.strn pub . The -.code pub -package has the -.code usr -package in its fallback list; thus when -.code pub -is current, all of the -.code usr -symbols, comprising the content of the \*(TL library, are visible. - -All forms read and evaluated from the \*(TX command line, in the interactive listener, -from files via -.code load -or -.code compile-file -or from the \*(TX pattern language are processed in this default -.code pub -package, unless arrangement are made to change to a different package. - -The current package is used as the default package for interning symbol tokens -which do not carry the colon-delimited package prefix. - -The current package also affects printing. When a symbol is printed whose -home package matches the current package, it is printed without a package -prefix. (Keyword symbols are always printed with the colon prefix, even if the -keyword package is current.) +.code catenated-stream-p +function returns +.code t +if +.meta obj +is a catenated stream. Otherwise it returns +.codn nil . -.coNP Function @ make-sym +.coNP Function @ catenated-stream-push .synb -.mets (make-sym << name ) +.mets (catenated-stream-push < new-stream << cat-stream ) .syne .desc The -.code make-sym -function creates and returns a new symbol object. The argument -.metn name , -which must be a string, specifies the name of the symbol. The symbol -does not belong to any package (it is said to be "uninterned"). +.code catenated-stream-push +function pushes +.meta new-stream +to the front of the stream list inside +.metn cat-stream . -Note: an uninterned symbol can be interned into a package with the -.code rehome-sym -function. Also see the -.code intern -function. +If an +.code unget-byte +or +.code unget-char +operation was successfully performed on +.meta cat-stream +previously to a call to +.codn catenated-stream-push , +those operations were forwarded to the front stream. +If those bytes or characters are still pending, +they are pending inside that stream, and thus +are logically preceded by the contents +of +.metn new-stream . -.coNP Function @ gensym +.coNP Functions @ open-files and @ open-files* .synb -.mets (gensym <> [ prefix ]) +.mets (open-files < path-list <> [ alternative-stream ]) +.mets (open-files* < path-list <> [ alternative-stream ]) .syne .desc The -.code gensym -function is similar to make-sym. It creates and returns a new -symbol object. If the -.meta prefix -argument is omitted, it defaults to -.strn g . -Otherwise it must be a string. +.code open-files +and +.code open-files* +functions create a list of streams by invoking +the open-file function on each element of +.metn path-list . +These streams are turned +into a catenated stream as if applied as arguments to +.codn make-catenated-stream . + +The effect is that multiple files appear to be catenated together into a single +input stream. + +If the optional +.meta alternative-stream +argument is supplied, then if +.meta path-list +is empty, +.meta alternative-stream +is returned instead of an empty catenated stream. The difference between -.code gensym +.code open-files and -.code make-sym +.code open-files* is that -.code gensym -creates the name -by combining the prefix with a numeric suffix. +.code open-files +creates all of the +streams up-front. So if any of the paths cannot be opened, the operation throws. +The +.code open-files* +variant is lazy: it creates a lazy list of streams out of the +path list. The streams are opened as needed: before the second stream is opened, +the program has to read the first stream to the end, and so on. -The numeric suffix is a decimal digit string, taken from the value of -the variable -.codn *gensym-counter* , -after incrementing it. +.TP* Example: -Note: the variation in name is not the basis of the uniqueness assurance -offered by -.code make-sym -and -.codn gensym ; -the basis is that the returned symbol is a freshly instantiated object. -.code make-sym -still returns unique symbols even if repeatedly called with the same -string. +Collect lines from all files that are given as arguments on the command line. If +there are no files, then read from standard input: -.coNP Special variable @ *gensym-counter* -.desc -This variable is initialized to 0. Each time the -.code gensym -function is called, -it is incremented. The incremented value forms the basis of the numeric -suffix which -.code gensym -uses to form the name of the new symbol. +.verb + @(next (open-files *args* *stdin*)) + @(collect) + @line + @(end) +.brev -.coNP Function @ make-package +.coNP Function @ abs-path-p .synb -.mets (make-package << name ) +.mets (abs-path-p << path ) .syne .desc The -.code make-package -function creates and returns a package named -.metn name , -where -.meta name -is a string. It is an error if a package by that name exists already. -Note: ordinary creation of packages for everyday program modularization -should be performed with the -.code defpackage -macro rather than by direct use of -.codn make-package . +.code abs-path-function +tests whether the argument +.meta path +is an absolute path, returning a +.code t +or +.code nil +indication. -.coNP Function @ delete-package -.synb -.mets (make-package << package ) -.syne -.desc -The -.code delete-package -breaks the association between a package and its name. -After -.codn delete-package , -the -.meta package -object continues to exist, but cannot be found using -.codn find-package . +The function behaves in the same manner on all platforms, implementing +a platform-agnostic definition of +.IR "absolute path" , +as follows. + +An absolute path is a string which either begins with a slash or backslash +character, or which begins with an alphanumeric word, followed by a colon, +followed by a slash or backslash. + +The empty string isn't an absolute path. + +Examples of absolute paths: + +.verb + /etc + c:/tmp + ftp://user@server + disk0:/home + Z:\eUsers +.brev + +Examples of strings which are not absolute paths: -Furthermore, -.code delete-package -iterates over all remaining packages. For each remaining package -.metn p , -it performs the semantic action of the .mono -.meti (unuse-package < package << p) +.mets >> ( the < empty << string ) + . + abc + foo:bar/x + $:\eabc .onom -expression. That is to say, all of the remaining packages -are scrubbed of any foreign symbols which are the local symbols -of the deleted -.metn package . - -.coNP Function @ packagep -.synb -.mets (packagep << obj ) -.syne -.desc -The -.code packagep -function returns -.code t -if -.meta obj -is a package, otherwise it returns -.codn nil . -.coNP Function @ find-package +.coNP Function @ pure-rel-path-p .synb -.mets (find-package << name ) +.mets (pure-rel-path-p << path ) .syne .desc -The argument -.meta name -should be a string. If a package called -.meta name -exists, -then it is returned. Otherwise -.code nil -is returned. - -.coNP Special variable @ *package-alist* -.desc The -.code *package-alist* -variable contains the master association list -which contains an entry about each existing -package. +.code pure-rel-path-p +function tests whether the string +.meta path +represents a +.IR "pure relative path" , +which is defined as a path which isn't absolute according to +.codn abs-path-p , +which isn't the string +.str . +(single period), +which doesn't begin with a period followed by a slash or backslash, +and which doesn't begin with alphanumeric word +terminated by a colon. -Each element of the list is a cons cell -whose -.code car -field is the name of a package and whose -.code cdr -is a package object. +The empty string is a pure relative path. -Note: the \*(TL application can overwrite or re-bind this -variable to manipulate the active package list. This is -very useful for -.IR sandboxing : -safely evaluating code that is obtained as an input -from an untrusted source, or calculated from such an input. +Other examples of pure relative paths: -The contents of -.code *package-alist* -have security implications because textual source code -can refer to any symbol in any package by invoking -a package prefix. For instance, even if the -.code open -function's name is not available in the current package -(established by the -.code *package* -variable) that symbol can easily be obtained using the -syntax -.codn usr:open . +.verb + abc.d + .tmp/bar + 1234 + x + $:/xyz +.brev -However, the entire -.code usr -package itself can be removed from -.codn *package-alist* . -In that situation, the syntax -.code usr:open -is no longer valid. +Examples of strings which are not pure relative paths: -At the same time, selected symbols from the original -.code usr -can be nevertheless made available via some intermediate -package, which is present in -.code *package-alist* -and which contains a subset of the -.code usr -symbols that has been curated for safety. That curated package may even -be called -.codn usr , -so that if for instance -.code cons -is present in that package, it may be referred to as -.code usr:cons -in the usual way. +.verb + . + / + /etc + ./abc + .\e + foo: + $:\eabc +.brev -.coNP Function @ package-alist +.coNP Functions @ dir-name and @ base-name .synb -.mets (package-alist) +.mets (dir-name << path ) +.mets (base-name << path ) .syne .desc The -.code package-alist -function retrieves the value of -.codn *package-alist* . +.code dir-name +and +.code base-name +functions calculate, respective, the directory part and +base name part of a path name. -Note: this function is obsolescent. There is no reason to use it -in new code instead of just accessing -.code *package-alist* -directly. +The calculation is performed in a platform-dependent way, using the +characters in the variable +.code path-sep-chars +as path component separators. -.coNP Function @ package-name -.synb -.mets (package-name << package ) -.syne -.desc -The -.code package-name -function retrieves the name of a package. +Both functions first remove from any further consideration all superfluous +trailing occurrences of the directory separator characters from +.codn path . +Thus input such as +.str "a////" +is reduced to just +.strn "a" , +and +.str "///" +is reduced to +.strn "/" . -.coNP Function @ package-symbols -.synb -.mets (package-symbols <> [ package ]) -.syne -.desc -The -.code package-symbols -function returns a list of all the symbols -which are interned in -.metn package . +The resulting trimmed path is the +.I "effective path" . -.coNP Functions @ package-local-symbols and @ package-foreign-symbols -.synb -.mets (package-local-symbols <> [ package ]) -.mets (package-foreign-symbols <> [ package ]) -.syne -.desc -The -.code package-local-symbols -function returns a list of all the symbols -which are interned in -.metn package , -and whose home package is that package. +If the effective path is an empty string, then +.code dir-name +returns +.str "." +and +.code base-name +returns the empty string. -The -.code package-foreign-symbols -function returns a list of all the symbols which -are interned in -.metn package , -which do not have that package as their home package, -or do not have a home package at all. +If the effective path is not empty, and contains no path separator +characters, then +.code dir-name +returns +.str "." +and +.code base-name +returns the effective path. -The union of the local and foreign symbols contains exactly -the same elements as the list returned by -.codn package-symbols : -the symbols interned in a package are partitioned into -local and foreign. +Otherwise, the effective path is divided into two parts: the +.I "raw directory prefix" +and the remainder. -.coNP Functions @ package-fallback-list and @ set-package-fallback-list -.synb -.mets (package-fallback-list << package ) -.mets (set-package-fallback-list < package << package-list ) -.syne -.desc -The -.code package-fallback-list -returns the current -.I "fallback package list" -associated with -.metn package . +The raw directory path is the maximally long prefix of the effective +path which ends in a separator character. The -.code set-package-fallback-list -replaces the fallback package list of -.meta package -with -.metn package-list . +.code dir-name +function returns the raw directory prefix, if that prefix consists of +nothing but a single directory separator character. Otherwise it +returns the raw directory prefix, with the trailing path separator +removed. The -.meta package-list -argument must be a list which is a mixture of symbols, strings or -package objects. Strings are taken to be package names, which must -resolve to existing packages. Symbols are reduced to strings via -.codn symbol-name . +.code base-name +function returns the remaining part of the effective path, after +the raw directory prefix. -.coNP Function @ intern +.coNP Function @ path-cat .synb -.mets (intern < name <> [ package ]) +.mets (path-cat < dir-path << rel-path ) .syne .desc -The argument -.meta name -should be a symbol. The optional argument -.meta package -should be a package. If -.meta package -is not supplied, then the value -taken is that of -.codn *package* . - The -.code intern -function searches -.meta package -for a symbol called -.metn name . -If that symbol is found, it is returned. If that symbol is not found, -then a new symbol called -.meta name -is created and inserted into -.metn package , -and that symbol is returned. In this case, the package becomes the -symbol's home package. +.code path-cat +function joins the directory path name given by the character +string argument +.meta dir-path +with the relative path name given by +.metn rel-path , +returning the joined path. -.coNP Function @ unintern -.synb -.mets (unintern < symbol <> [ package ]) -.syne -.desc -The -.code unintern -function removes -.meta symbol -from -.metn package . +The function is related to the functions +.code dir-name +and +.code base-name +in the following way: if +.meta p +is some path denoting an object in the file system, then +.code "(path-cat (dir-name p) (base-name p))" +produces a path +.meta p* +which denotes the same object. The paths +.meta p +and +.meta p* +might not be equivalent strings. The -.code nil -symbol may not be removed from the -.code usr -package; an error exception is thrown in this case. +.code path-cat +function ensures that paths are joined without superfluous +path separator characters, regardless of whether +.meta dir-path +ends in a separator. -If -.code symbol -isn't -.codn nil , -then -.meta package -is searched to determine whether it contains -.meta symbol -as an interned symbol (either local or foreign), or a hidden symbol. +If a separator must be added, the character +.code / +(forward slash) is always used, even on platforms where +.code \e +(backslash) is also a pathname separator, and even if either argument includes +backslashes. -If -.meta symbol -is a hidden symbol, then it is removed from the hidden symbol store. -Thereafter, even if a same-named foreign symbol is removed from the -package via -.code unuse-sym -or -.codn unuse-package , -those operations will no longer restore the hidden symbol to interned -status. In this case, -.meta unintern -returns the hidden symbol that was removed from the hidden store. +The +.code path-cat +function eliminates trivial occurrences of the +.code . +(dot) path component. It preserves trailing separators in the following +way: if +.meta rel-path +ends in a path separator character, then the returned string shall +end in that character; and if +.meta rel-path +vanishes entirely because it is equivalent to the dot, then the returned +string is +.meta dir-name +itself. -If -.meta symbol -is a foreign symbol, then it is removed from the package. If the package -has a hidden symbol of the same name, that hidden symbol is re-interned -in the package, and the package once again becomes its home package. -In this case, -.meta symbol -is returned. +.TP* Examples: -If -.meta symbol -is a local symbol, the symbol is removed from the package. -In this case also, -.meta symbol -is returned. +.verb + (path-cat "" "") --> "" + (path-cat "" ".") --> "" + (path-cat "." "") --> "" + (path-cat "." ".") --> "" -If -.meta symbol -is not found in the package as either an interned or hidden -symbol, then the function has no effect and returns -.codn nil . + (path-cat "abc" ".") --> "abc" + (path-cat "." "abc") --> "abc" -.coNP Function @ rehome-sym -.synb -.mets (rehome-sym < symbol <> [ package ]) -.syne -.desc -The arguments -.meta symbol -and -.meta package -must be a symbol and package object, -respectively, and -.meta symbol -must not be the symbol -.codn nil . + (path-cat "./" ".") --> "./" + (path-cat "." "./") --> "./" -The -.code rehome-sym -function moves -.meta symbol -into -.metn package . -If -.meta symbol -is already interned in a package, it is first removed from that package. + (path-cat "abc/" ".") --> "abc/" + (path-cat "./" "abc") --> "abc" -If a symbol of the same name exists in -.metn package , -that symbol is first removed -from -.metn package . + (path-cat "/" ".") --> "/" -Also, if a symbol of the same name exists in the hidden symbol store of -.metn package , -that hidden symbol is removed. + (path-cat "/" "abc") --> "/abc" -Then -.code symbol -is interned into -.metn package , -and -.meta package -becomes its home package, making it a local symbol of -.metn package . + (path-cat "ab/cd" "ef") --> "ab/cd/ef" +.brev -Note: if -.code symbol -is currently the hidden symbol of some package, it is not removed -from the hidden symbol store of that package. This is a degenerate -case. The implication is that if that hidden symbol is ever -restored in that package, it will once again have that package as -its home package, and consequently it will turn into a foreign -symbol of -.metn package . -.coNP Function @ symbolp -.synb -.mets (symbolp << obj ) -.syne +.coNP Variable @ path-sep-chars .desc The -.code symbolp -function returns -.code t -if -.meta obj -is a symbol, otherwise it returns -.codn nil . +.code path-sep-chars +variable holds a string consisting of the characters which the underlying +operating system recognizes as path name separators. -.coNP Function @ symbol-name -.synb -.mets (symbol-name << symbol ) -.syne -.desc -The -.code symbol-name -function returns the name of -.metn symbol . +If a particular of these characters is considered preferred on +the host platform, that character is placed in the first position of +.codn path-sep-chars . -.coNP Function @ symbol-package -.synb -.mets (symbol-package << symbol ) -.syne -.desc -The -.code symbol-package -function returns the home package of -.metn symbol . -If -.meta symbol -has no home package, it returns -.codn nil . +Altering the value of this variable has no effect on any \*(TL library +function. -.coNP Function @ keywordp +.coNP Functions @ read and @ iread .synb -.mets (keywordp << obj ) +.mets (read >> [ source >> [ error-stream >> [ error-retval <> [ name ]]]]) +.mets (iread >> [ source >> [ error-stream >> [ error-retval <> [ name ]]]]) .syne .desc The -.code keywordp -function returns -.code t -if -.meta obj -is a keyword symbol, otherwise it -returns -.codn nil . +.code read +function converts text denoting \*(TL structure, into the +corresponding data structure. The +.meta source +argument may be either a character +string, or a stream. If it is omitted, then +.code *stdin* +is used as the stream. -.coNP Function @ bindable -.synb -.mets (bindable << obj ) -.syne -.desc -The -.code bindable -function returns -.code t -if -.meta obj -is a bindable symbol, otherwise it returns -.codn nil . +The source must provide the text representation of one complete \*(TL object. -All symbols are bindable, except for keyword symbols, and the -special symbols -.code t -and -.codn nil . +Multiple calls to read on the same stream will extract successive objects +from the stream. To parse successive objects from a string, it is necessary +to convert it to a string stream. -.coNP Function @ use-sym -.synb -.mets (use-sym < symbol <> [ package ]) -.syne -.desc -The -.code use-sym -function brings an existing -.code symbol -into -.metn package . +The optional +.meta error-stream +argument can be used to specify a stream to which +parse errors diagnostics are sent. If absent, the diagnostics are suppressed. -In all cases, the function returns -.codn symbol . +The optional +.meta name +argument can be used to specify the file name which is used for reporting +errors. If this argument is missing, the name is taken from the name +property of the +.meta source +argument if it is a stream, or else the word +.code string +is used as the name if +.meta source +is a string. -If -.meta symbol -is already interned in -.metn package , -then the function has no effect. +If there are no parse errors, the function returns the parsed data +structure. If there are parse errors, and the +.meta error-retval +parameter is +present, its value is returned. If the +.meta error-retval +parameter +is not present, then an exception of type +.code syntax-error +is thrown. -Otherwise -.meta symbol -is interned in -.metn package . +The +.code iread +function ("interactive read") is similar to +.code read +except that it parses a modified version of the syntax. The modified +syntax does not support the application of the dot and dotdot operators +on a top-level expression. For instance, if the input is +.code a.b +or +.code "a .. b" +then +.code iread +will only read the +.code a +token whereas +.code read +will read the entire expression. -If a symbol having the same name as -.meta symbol -already exists in -.metn package , -then it is replaced. -If that replaced symbol is a local symbol of -.metn package , -then the replaced symbol turns into a hidden symbol associated -with the package. It is placed into a special hidden symbol store -associated with -.meta package -and is stripped of its home package, becoming quasi-interned or uninterned. +This modified syntax allows +.code iread +to return immediately when an expression is recognized, which is the +expected behavior if the input is being read from an interactive terminal. +By contrast, +.code read +waits for more input after seeing a complete expression, because of the +possibility that the expression will be further extended by means of the dot or +dotdot operators. An explicit end-of-input signal must be given from the +terminal to terminate the expression. -An odd case is possible whereby -.meta symbol -is already a hidden symbol of -.metn package . -In this case, the hidden symbol replaces some foreign symbol and -is interned in -.metn package . -Thus it simultaneously exists as both an interned -foreign symbol and as a hidden symbol of -.metn package . +The special variable +.code *rec-source-loc* +controls whether these functions record source location info similarly to +.codn load . +Note: if these functions are used to scan data which is evaluated as Lisp code, +it may be useful to set +.code *rec-source-loc* +true in order to obtain better diagnostics. However, source location recording +incurs a performance and storage penalty. -.coNP Function @ unuse-sym +.coNP Function @ record-adapter .synb -.mets (unuse-sym < symbol <> [ package ]) +.mets (record-adapter < regex >> [ stream <> [ include-match ]]) .syne .desc The -.code unuse-sym -function removes -.meta symbol -from -.metn package . +.code record-adapter +function returns a new stream object which acts as an +.I adapter +to the existing +.metn stream . -If -.meta symbol -is not interned in -.metn package , -the function does nothing and returns -.codn nil . +If an argument is not specified for +.metn stream , +then the +.code *std-input* +stream is used. -If -.meta symbol -is a local symbol of -.metn package , -an error is thrown: a package cannot "unuse" its own symbol. Removing -a symbol from its own home package requires the -.code unintern -function. +With the exception of +.metn get-line , +all operations on the returned adapter transparently delegate to the original +.meta stream +object. -Otherwise -.meta symbol -is a foreign symbol interned in -.meta package -and is removed. +When the +.code get-line +function is used on the adapter, it behaves differently. A string is +extracted from +.metn stream , +and returned. However, the string isn't a line delimited by a newline +character, but rather a record delimited by +.metn regex . +This record is extracted as if by a call to the +.code read-until-match +function, invoked with the +.metn regex , +.meta stream +and +.meta include-match +arguments. -If the package has a hidden symbol of the same name as -.metn symbol , -that symbol is re-interned into -.meta package -as a local symbol. In this case, that previously hidden symbol is -returned. +All behavior which is built on the +.code get-lines +function is affected by the record-delimiting semantics of a record adapter's +.code get-line +implementation. Notably, the +.code get-lines +and +.code lazy-stream-cons +functions return a lazy list of delimited records rather than of lines. -If the package has no hidden symbol matching the removed -.metn symbol , -then -.meta symbol -itself is returned. +.SS* Stream Output Indentation +\*(TL streams provide support for establishing hanging indentations +in text output. Each stream which supports output has a built-in state variable +called indentation mode, and another variable indicating the current +indentation amount. When indentation mode is enabled, then prior to the +first character of every line, the stream prepends the indentation: space +characters equal in number to the current indentation value. +This logic is implemented by the +.code put-char +and +.code put-string +functions, and all functions based on these. The +.code put-byte +function does not interact with indentation. The column position tracking +will be incorrect if byte and character output are mixed, affecting +the placement of indentation. -.coNP Functions @ use-package and @ unuse-package +Indentation mode takes on four numeric values, given by the four +variables +.codn indent-off , +.codn indent-data , +.code indent-code +and +.codn indent-foff . +As far as stream output is concerned, the code and data modes +represented by +.code indent-code +and +.code indent-data +behave +the same way: both represent the "indentation turned on" state. +The difference between them influences the behavior of the +.code width-check +function. This function isn't used by any lower-level stream output +routines. It is used by the object printing functions like +.code print +and +.code pprint +to break up long lines. +The +.code indent-off +and +.code intent-foff +modes are also treated the same way by lower level stream output, +indicating "indentation turned off". The modes are distinguished +by +.code print +and +.code pprint +in the following way: +.code indent-off +is a "soft" disable which allows these object-printing routines +to temporarily turn on indentation while traversing aggregate objects. +Whereas the +.code indent-foff +("force off") value is a "hard" disable: the object-printing routines will not +enable indentation and will not break up long lines. + +.coNP Variables @, indent-off @, indent-data @ indent-code and @ indent-foff +.desc +These variables hold integer values representing output stream +indentation modes. The value of +.code indent-off +is zero. + +.coNP Functions @ get-indent-mode and @ set-indent-mode .synb -.mets (use-package < package-list <> [ package ]) -.mets (unuse-package < package-list <> [ package ]) +.mets (get-indent-mode << stream ) +.mets (set-indent-mode < stream << new-mode ) +.mets (test-set-indent-mode < stream < compare-mode << new-mode ) .syne .desc +These functions retrieve and manipulate the stream indent mode. The -.meta use-package -and -.meta unuse-package -are convenience functions which perform a mass import of symbols from one -package to another, or a mass removal, respectively. - +.code get-indent-mode +retrieves the current indent mode of +.metn stream . The -.code use-package -function iterates over all of the local symbols of the packages in -.metn package-list . -For each symbol -.metn s , -it performs the semantic action implied by the -.mono -.meti (use-sym < s << package ) -.onom -expression. - -Similarly -.code unuse-package -iterates -.meta package-list -in the same way, performing, effectively, the semantic action -of the -.mono -.meti (unuse-sym < s << package ) -.onom -expression. +.code set-indent-mode +function sets the indent mode of +.meta stream +to +.meta new-mode +and returns the previous mode. -The -.meta package-list -argument must be a list which is a mixture of symbols, strings or -package objects. Strings are taken to be package names, which must -resolve to existing packages. Symbols are reduced to strings via -.codn symbol-name . +Note: it is encouraged to save and restore the indentation mode, +and in a way that is exception safe. +If a block of code sets up indentation on a stream such as +.code *stdout* +and is terminated by an exception, the indentation will remain in +effect and affect subsequent output. The +.code with-resources +macro or +.code unwind-protect +operator may be used. -.coNP Macro @ defpackage +.coNP Functions @ test-set-indent-mode and @ test-neq-set-indent-mode .synb -.mets (defpackage < name << clause *) +.mets (test-set-indent-mode < stream < compare-mode << new-mode ) +.mets (test-neq-set-indent-mode < stream < compare-mode << new-mode ) .syne .desc The -.code defpackage -macro provides a convenient means to create a package and establish its -properties in a single construct. It is intended for the ordinary situations -in which packages support the organization of programs into modules. +.code test-set-indent-mode +function sets the indent mode of +.meta stream +to +.meta new-mode +if, and only if, +its current mode is equal to +.metn compare-mode . +Whether or not it changes the mode, it returns the previous mode. The -.code name -argument, giving the package name, may be a symbol or a character string. -If it is a symbol, then the symbol's name is taken to be name for the -package. +.code test-neq-set-indent-mode +only differs in that it sets +.meta stream +to +.meta new-mode +if, and only if, +the current mode is +.B not +equal to +.metn compare-mode . -If a package called -.code name -already exists, then -.code defpackage -selects that package for further operations. Otherwise, a new, -empty package is created. In either case, this package is referred -to as the -.I "present package" -in the following descriptions. +.coNP Functions @, get-indent @ set-indent and @ inc-indent +.synb +.mets (get-indent << stream ) +.mets (set-indent < stream << new-indent ) +.mets (inc-indent < stream << indent-delta ) +.syne +.desc +These functions manipulate the indentation value of the stream. +The indentation takes effect the next time a character is output +following a newline character. The -.code name -may be optionally followed by one or more clauses, which are processed -in the order that they appear. Each clause is a compound form headed -by a keyword. -The supported clauses are as follows: -.RS -.meIP (:fallback << package-name *) -The -.code :fallback -clause specifies the packages to comprise the fallback list of -the present package. If this clause is omitted, or if it is present -with not -.meta package-name -arguments, then the present package has an empty fallback list. -Each -.meta package-name -may be a string or symbol naming an existing package. It is permitted -for the present package itself to appear in its own fallback list. -This is useful for creating a package with a non-empty fallback list -which doesn't actually provide access to any other package. -.meIP (:use << package-name *) -The -.code :use -clause specifies packages whose local symbols are to be interned -into the present package as foreign symbols. Each -.meta package-name -may be a string or symbol naming an existing package. -The list of package names is processed as if by a call to -.codn use-package . -.meIP (:use-syms << symbol *) -The -.code :use-syms -clause specifies individual symbols to be interned in the present package. -The arguments are symbols. -.meIP (:use-from < package-name << symbol-name *) +.code get-indent +function retrieves the current indentation amount. + The -.code :use-from -clause specifies the names of local symbols in a package denoted by -.meta package-name -to be used in the present package. All arguments of -.code :use-from -are either strings or symbols which are reduced to strings by mapping -to their names. Each -.meta symbol-name -is interned in the package identified by -.metn package-name , -which may have the effect of creating that symbol. -This symbol is expected to be a local symbol of that package. If -that is so, the symbol is brought into the present package via -.codn use-symbol . -Otherwise if the symbol is foreign to package identified by -.metn package-name , -then an error exception is thrown. -.meIP (:local << symbol-name *) +.code set-indent +function sets +.metn stream 's +indentation to the value +.meta new-indent +and returns the previous value. +Negative values are clamped to zero. + The -.code :local -clause specifies the names of symbols to be interned in the new package -as local symbols. Each -.meta symbol-name -argument must be either a character string or a symbol. If it is a symbol, its -name is taken, thereby reducing the argument to a character string. -The arguments are processed in the order in which they appear. Each name is -first interned in the newly created package using the -.code intern -function. Then, if the resulting symbol is foreign to the package, it is -removed with -.code unuse-sym -and the name is interned again. -.RE +.code inc-indent +function sets +.metn stream 's +indentation relative to the current printing column position, +and returns the old value. +The indentation is calculated by adding +.meta indent-delta +to the current column position. +If a negative indentation results, it is clamped to zero. -.coNP Macro @ in-package +.coNP Function @ width-check .synb -.mets (in-package << name ) +.mets (width-check < stream << alt-char ) .syne .desc The -.code in-package -macro causes the -.code *package* -special variable to take on the package denoted by -.metn name . -The macro checks, at expansion time, that -.meta name -is either a string or symbol. An error is thrown if -this isn't the case. +.code width-check +function examines the state of the stream, taking into consideration +the current printing column position, the indentation state, +the indentation amount and an internal "force break" flag. It makes a decision +either to introduce a line break by printing a newline character, or else to +print the +.meta alt-char +character. -The -.meta name -argument expression isn't evaluated, and so must not be quoted. +If a decision is made not to emit a line break, but +.meta alt-char +is +.codn nil , +then the function has no effect at all. -The code generated by the macro performs a search for the -package. If the package is not found at the time when -the macro's expansion is evaluated, an error is thrown. +The return value is +.code t +if the function has issued a line break, otherwise +.codn nil . -.SS* Pseudo-random Numbers -.coNP Special variable @ *random-state* +.coNP Function @ force-break +.synb +.mets (force-break << stream ) +.syne .desc +If the +.code force-break +function is called on a stream, it sets an internal "force break" flag which +affects the future behavior of +.codn width-check . The -.code *random-state* -variable holds an object which encapsulates the state -of a pseudo-random number generator. This variable is the default argument -value for the -.code random-fixnum -and -.codn "random functions" , -for the convenience of writing programs which are not concerned about the -management of random state. - -On the other hand, programs can create and manage random states, making it -possible to obtain repeatable sequences of pseudo-random numbers which do not -interfere with each other. For instance objects or modules in a program can -have their own independent streams of random numbers which are repeatable, -independently of other modules making calls to the random number functions. +.code width-check +function examines this flag. If the flag is set, +.code width-check +clears it, and issues a line break without considering any other +conditions. -When \*(TX starts up, the -.code *random-state* -variable is initialized with -a newly created random state object, which is produced as if by -the call -.codn "(make-random-state 42)" . +The +.metn stream 's +.code force-break +flag is also cleared whenever a newline character is output. -.coNP Special variable @ *random-warmup* -.desc The -.code *random-warmup* -special variable specifies the value which is used by -.code make-random-state -in place of a missing -.meta warmup-period -argument. +.code force-break +function returns +.codn stream . -To "warm up" a pseudo-random number generator (PRNG) means to obtain some -values from it which are discarded, prior to use. The number of values -discarded is the -.IR "warm-up period" . +Note: the +.code force-break +is involved in line breaking decisions. Whenever a list or list-like syntax is +being printed, whenever an element of that syntax is broken into multiple +lines, a break is forced after that element, in order to avoid output +which resembles the following diagonally-creeping pattern: -The WELL PRNG used in \*(TX produces 32-bit values, natively. Thus each -warm-up iteration retrieves and discards a 32-bit value. The PRNG has -a state space consisting of a vector of sixteen 32-bit words, making -the state space 4096 bits wide. +.verb + (a b c (d e f + g h i) j (k l + m n) o) +.brev -Warm up is required because PRNG-s, in particular PRNG-s with large state -spaces and long periods, produce fairly predictable sequences of values in the -beginning, before transitioning into chaotic behavior. This problem is worse -for low complexity seeds, such as small integer values. +but instead is rendered in a more horizontally compact pattern: -The sequences are predictable in two ways. Firstly, some initial values -extracted from the PRNG may exhibit patterns ("problem 1"). Secondly, the initial values -from sequences produced from similar seeds (for instance consecutive integers) -may be similar or identical ("problem 2"). +.verb + (a b c (d e f + g h i) + j (k l + m n) + o) +.brev -.TP* Notes: +When the printer prints +.code "(d e f g h i)" +it uses the +.code width-check +function between the elements; that function issues the +break between the +.code f +and +.codn g . +The printer monitors the return value of +.codn width-check ; +it knows that since one of the calls returned +.codn t , +the object had been broken into two or more lines. It then calls +.code force-break +after printing the last element +.code i +of that object. Then, due to the force flag, the outer recursion of the +printer which is printing +.code "(a b c ...)" +will experience a break when it calls +.code width-check +before printing +.codn j . -The default value of -.code *random-warmup* -is only 8. This is insufficient to -ensure good initial PRNG behavior for seeds even as large as 64 bits or more. -That is to say, even if as many as eight bytes' worth of true random bits are -used as the seed, the PRNG will exhibit predictable behaviors, and a poor -distribution of values. +Custom +.code print +methods defined on structure objects can take advantage of +.code width-check +and +.code force-break +in the same way so that application-defined output integrates +with the formatting algorithm. -Applications which critically depend on good PRNG behavior should choose -large warm-up periods into the hundreds or thousands of iterations. -If a small warm-up period is used, it is recommended to use larger seeds -which initialize more of the 4096 bit state space. +.SS* Stream Output Limitation -\*(TX's PRNG implementation addresses "problem 1" first problem by padding the -unseeded portions of the state space with random values (from a static table -that doesn't change). For instance, if the integer 1 is used to seed the space, -then one 32 bit word of the space is set to the value 1. The remaining 15 are -populated from the random table. This helps to ensure that a good PRNG sequence -is obtained immediately. However, it doesn't address "problem 2": that -similar seed values generate similar sequences, when the warm-up period is -small. For instance, if 65536 different random state objects are created, from -each of the 16-bit seeds in the range [0, 65536), and then a random 16-bit -value is extracted from each state, only 1024 unique values result. +Streams have two properties which are used by the The \*(TL object printer to +optionally truncate the output generated by aggregate objects. -.coNP Function @ make-random-state -.synb -.mets (make-random-state >> [ seed <> [ warmup-period ]) -.syne -.desc -The -.code make-random-state -function creates and returns a new random state, -an object of the same kind as what is stored in the -.code *random-state* -variable. +A stream can specify a maximum length for aggregate objects via the +.code set-max-length +function. Using the +.code set-max-depth +function, the maximum depth can also be specified. -The seed, if specified, must be either an integer value, an -existing random state object, or a vector returned from a call -to the function -.codn random-state-get-vec . +This feature is +useful when diagnostic output is being produced, and the objects involved are +so large that the diagnostic output overwhelms the output device or the user, +so as to become uninformative. Output limiting also prevents the printer's +non-termination on infinite, lazy structures. -Note that the sign of the seed is ignored, so that negative seed -values are equivalent to their additive inverses. +It is recommended that functions which operate on streams passed in as +parameters save and restore these parameters, if they need to manipulate them, +for instance using +.codn with-resources : -If seed is not specified, then -.code make-random-state -produces a seed based -on some information in the process environment, such as current -time of day. It is not guaranteed that two calls to -.code (make-random-state) -that are separated by less than some minimum increment of real time produce -different seeds. The minimum time increment depends on the platform. +.verb + (defun output-function (arg stream) + ;; temporarily impose maximum width and depth + (with-resources ((ml (set-max-length stream 42) + (set-max-length stream ml)) + (mw (set-max-depth stream 12) + (set-max-depth stream mw))) + (prinl arg stream) + ...)) +.brev -On a platform with a millisecond-resolution real-time clock, the minimum -time increment is a millisecond. Calls to make-random-state less than -a millisecond apart may predictably produce the same seed. +.coNP Function @ set-max-length +.synb +.mets (set-max-length < stream << value ) +.syne +.desc +The +.code set-max-length +function establishes the maximum length for aggregate object printing. +It affects the printing of lists, vectors, hash tables, strings +as well as quasiliterals and quasiword list literals (QLLs). -If an integer seed is specified, then the integer value is mapped to a -pseudo-random sequence, in a platform-independent way. +The default value is 0 and this value means that no limit is imposed. +Otherwise, the value must be a positive integer. -If an existing random state is specified as a seed, then it is duplicated. The -returned random state object is a distinct object which is in the same -state as the input object. It will produce the same remaining pseudo-random -number sequence, as will the input object. +When the list, vector or hash table object being printed has more +elements than the maximum length, then elements are printed only up to +the maximum count, and then the remaining elements are summarized by +printing the +.code ... +(three dots) character sequence as if it were an additional element. +This sequence is an invalid token; it cannot be read as input. -If a vector is specified as a seed, then a random state is constructed -which duplicates the random state object which was captured in that vector -representation by the -.code random-state-get-vec -function. +When a character string is printed, any positive value of +the maximum length which is less than 15 is considered to be 15. +The maximum length specifies the number of characters of the +a string which are output. + +If a string which exceeds the maximum length is being printed +with read-print consistency, as by the +.code print +function, then only a prefix of the string is printed, limited +to the maximum number of characters. Then, the literal syntax is +closed using the character sequence +.code \e...\(dq +(backslash, dot, dot, dot, double quote) +whose leading invalid escape sequence +.code \e. +(backslash, dot) ensures that the truncated object is not readable. + +If a string which exceeds the maximum length is being printed +without read-print consistency, as by the +.code pprint +function, then only a prefix of the string is printed, limited +to the maximum number of characters. Then the +character sequence +.code ... +is emitted. + +Quasiliterals are treated using a combination of behaviors. Elements of a +quasiliteral are literal sequence of text, and embedded variables and +expressions. The maximum length specifies both the maximum number of elements +in the quasiliteral, and the maximum number of characters in any element which +is a sequence of text. When either limit is exceeded, the quasiliteral +is immediately terminated with the sequence +.code \e...` +(escaped dot, dot, dot, backtick). The maximum limit is applied to +the units of text cumulatively, rather than individually. As in the case of +string literals, smaller limit values than 15 are treated as 15, +but only for the cumulative text length limit. For limiting the number of +elements, the count is used as-is. + +When a QLL is printed, the space-separated elements +of the literal are individually subject to the maximum length limit as if +they were independent quasiliterals. Furthermore, the sequence of these +elements is subject to the maximum length. If there are more elements in the +QLL, then the sequence +.code \e...` +(escaped dot, dot, dot, backtick) is emitted and thus the QLL ends. The -.meta warm-up-period -argument specifies the number of values which are immediately obtained and -discarded from the newly-seeded generator before it is returned. -Warm-up is not performed when -.meta seed -is an existing random state object, and this argument is ignored in that -case. If the parameter is required, but the argument is missing, then -the value of the -.code *random-warmup* -special variable is used. This variable has a default value which may be too -small for serious applications of pseudo-random numbers; see the Notes under -.codn *random-warmup* . +.code set-max-length +function returns the previous value. -.coNP Function @ random-state-p +.coNP Function @ set-max-depth .synb -.mets (random-state-p << obj ) +.mets (set-max-depth < stream << value ) .syne .desc The -.code random-state-p -function returns -.code t -if -.meta obj -is a random state, otherwise it -returns -.codn nil . +.code set-max-length +function establishes the maximum depth for the printing of nested +objects. It affects the printing of lists, vectors, hash tables +and structures. The default value is 0 and this value means that no limit is +imposed. Otherwise, the value must be a positive integer. + +The depth of an object not enclosed in any object is zero. The depth of the +element of an aggregate is one greater than the depth of the aggregate itself. +For instance, given the list +.code "(1 (2 3))" +the list itself has depth 0, the atom +.code 1 +has depth 1, as does the sublist +.codn "(2 3)" , +and the +.code 2 +and +.code 3 +atoms have depth 2. + +When an object is printed whose depth exceeds the maximum depth, then three dot +character sequence +.code ... +is printed instead of that object. This notation is an invalid token; it cannot be +read as input. + +Additionally, when a vector, list, hash table or structure is printed which itself +doesn't exceed the maximum depth, but whose elements do exceed, then that object +is summarized, respectively, as +.codn "(...)" , +.codn "#(...)" , +.code "H#(...)" +and +.codn "S#(...)" , +rather than repeating the +.code ... +sequence for each of its elements. -.coNP Function @ random-state-get-vec -.synb -.mets (random-state-get-vec <> [ random-state ]) -.syne -.desc The -.code random-state-get-vec -function converts a random state into a vector of integer values. -If the -.meta random-state -argument, which must be a random state object, is omitted, -then the value of the -.code *random-state* -is used. +.code set-max-depth +function returns the previous value. -.coNP Functions @, random-fixnum @ random and @ rand +.SS* Coprocesses +.coNP Functions @ open-command and @ open-process .synb -.mets (random-fixnum <> [ random-state ]) -.mets (random < random-state << modulus ) -.mets (rand < modulus <> [ random-state ]) +.mets (open-command < system-command <> [ mode-string ]) +.mets (open-process < program < mode-string <> [ argument-list ]) .syne .desc -All three functions produce pseudo-random numbers, which are positive integers. +These functions spawn external programs which execute concurrently +with the \*(TX program. Both functions return a unidirectional stream for +communicating with these programs: either an output stream, or an input +stream, depending on the contents of +.metn mode-string . -The numbers are obtained from a WELL 512 PRNG, whose state is stored in the -random state object. +In +.codn open-command , +the +.meta mode-string +argument is optional, defaulting to +the value +.str r +if it is missing. See the +.code open-file +function for a discussion of modes. The -.code random-fixnum -function produces a random fixnum integer: a reduced range -integer which fits into a value that does not have to be heap-allocated. +.code open-command +function accepts, via the +.meta system-command +string parameter, a +system command, which is in a system-dependent syntax. On a POSIX system, this +would be in the POSIX Shell Command Language. The -.code random -and -.code rand -functions produce a value in the range [0, -.metn modulus ). -They differ only in the order of arguments. In the -.code rand -function, the random state -object is the second argument and is optional. If it is omitted, the global -.code *random-state* -object is used. +.code open-process +function specifies a program to invoke via the +.meta command +argument. This is subject to the operating system's search strategy. +On POSIX systems, if it is an absolute or relative path, it is treated as +such, but if it is a simple base name, then it is subject to searching +via the components of the PATH environment variable. If open-process +is not able to find +.metn program , +or is otherwise unable to execute +the program, the child process will exit, using the value of the C variable +.code errno +as its exit status. This value can be retrieved via +.codn close-stream . The -.meta modulus -argument must be a positive integer. If -.meta modulus -is 1, then the function returns zero without altering the state of the -pseudo-random number generator. +.meta mode-string +argument follows the convention used by the POSIX +.code popen +function. -.coNP Function @ random-float -.synb -.mets (random-float <> [ random-state ]) -.syne -.desc The -.code random-float -function produces a pseudo-random floating-point value in the range [0.0, 1.0). +.meta argument-list +argument is a list of strings which specifies additional +optional arguments to be passed passed to the program. The +.meta program +argument +becomes the first argument, and +.meta argument-string +become the second and +subsequent arguments. If +.meta argument-strings +is omitted, it defaults to empty. -The numbers are obtained from a WELL 512 PRNG, whose state is stored in the -random state object given by the argument to the optional -.meta random-state -parameter, which defaults to the value of -.codn *random-state* . +If a coprocess is open for writing +.mono +.meti >> ( mode-string +.onom +is specified as +.strn w ), +then +writing on the returned stream feeds input to that program's standard input +file descriptor. Indicating the end of input is performed by closing the +stream. -.SS* Time -.coNP Functions @ time and @ time-usec +If a coprocess is open for reading +.mono +.meti >> ( mode-string +.onom +is specified as +.strn r ), +then +the program's output can be gathered by reading from the returned stream. +When the program finishes output, it will close the stream, which can be +detected as normal end of data. + +The standard input and error file descriptors of an input coprocess +are obtained from the streams stored in the +.code *stdin* +and +.code *stderr* +special variables, respectively. Similarly, the standard output and error +file descriptors of an output coprocess are obtained from the +.code *stdout* +and +.code *stderr* +special variables. These variables must contain streams on which the +.code fileno +function is meaningful, otherwise the operation will fail. +What this functionality means is that re-binding the special variables +for standard streams has the effect of redirection. For example, +the following two expressions achieve the same effect of creating +a stream which reads the output of the +.code cat +program, which reads and produces the contents of the file +.codn text-file . + +.verb + ;; redirect input by rebinding *stdin* + (let ((*stdin* (open-file "text-file"))) + (open-command "cat")) + + ;; redirect input using POSIX shell redirection syntax + (open-command "cat < text-file") +.brev + +The following is erroneous: + +.verb + ;; (let ((*stdin* (make-string-input-stream "abc"))) + (open-command "cat")) +.brev + +A string input or output stream doesn't have an operating system file +descriptor; it cannot be passed to a coprocess. + +The streams +.codn *stdin* , +.code *stdout* +and +.code *stderr* +are not synchronized with their underlying file descriptors prior to +the execution of a coprocess. It is up to the program to ensure that +previous output to +.code *stdout* +or +.code *stderr* +is flushed, so that the output of the coprocess isn't re-ordered +with regard to output produced by the program. Similarly, +input buffered in +.code *stdin* +is not available to the coprocess, even though it has not +yet been read by the program. The program is responsible for preventing this +situation also. + +If a coprocess terminates abnormally or unsuccessfully, an exception is raised. + +.SS* I/O-Related Convenience Functions + +The functions in this group create a stream, perform an I/O operation +on it, and ensure that it is closed, in one convenient operation. They +operate on files or command streams. + +Several other functions in this category exist, which operate with buffers. +They are documented in the Buffer Functions subsection under the +FOREIGN FUNCTION INTERFACE section. + +.coNP Functions @, file-get @ file-get-string and @ file-get-lines .synb -.mets (time) -.mets (time-usec) +.mets (file-get << name ) +.mets (file-get-string << name ) +.mets (file-get-lines << name ) .syne .desc The -.code time -function returns the number of seconds that have elapsed since -midnight, January 1, 1970, in the UTC timezone: a point in -time called -.IR "the epoch" . +.code file-get +function opens a text stream over the file indicated by the string argument +.meta name +for reading, reads the printed representation of a \*(TL object from it, +and returns that object, ensuring that the stream is closed. The -.code time-usec -function returns a cons cell whose -.code car -field holds the seconds measured in the same way, and whose -.code cdr -field extends the precision by giving -number of microseconds as an integer value between 0 and 999999. +.code file-get-string +is similar to +.code file-get +except that it reads the entire file as a text stream and returns +its contents in a single character string. -.coNP Functions @ time-string-local and @ time-string-utc +The +.code file-get-lines +function opens a text stream over the file indicated by +.meta name +and returns produces a lazy list of strings representing the lines +of text of that file as if by a call to the +.code get-lines +function, and returns that list. The stream remains open until the +list is consumed to the end, as indicated in the description of +.codn get-lines . + +.coNP Functions @, file-put @ file-put-string and @ file-put-lines .synb -.mets (time-string-local < time << format ) -.mets (time-string-utc < time << format ) +.mets (file-put < name << obj ) +.mets (file-put-string < name << string ) +.mets (file-put-lines < name << list ) .syne .desc -These functions take the numeric time returned by the -.code time -function, and convert it to a textual representation in a flexible way, -according to the contents of the -.meta format -string. +The +.codn file-put , +.code file-put-string +and +.code file-put-lines +functions open a text stream over the file indicated by the string argument +.metn name , +write the argument object into the file in their specific manner, +and then close the file. + +If the file doesn't exist, it is created. +If it exists, it is truncated to zero length and overwritten. The -.code time-string-local -function converts the time to the local timezone of -the host system. The -.code time-string-utc -function produces time in UTC. +.code file-put +function writes a printed representation of +.meta obj +using the +.code prinl +function. The return value is that of +.codn prinl . The -.meta format -argument is a string, and follows exactly the same conventions as -the format string of the C library function -.codn strftime . +.code file-put-string +function writes +.meta string +to the stream using the +.code put-string +function. The return value is that of +.codn put-string . The -.meta time -argument is an integer representing seconds obtained from the -time function or from the -.code car -field of the cons returned by the -.code time-usec -function. +.code file-put-lines +function writes +.meta list +to the stream using the +.code put-lines +function. The return value is that of +.codn put-lines . -.coNP Functions @ time-fields-local and @ time-fields-utc +.coNP Functions @, file-append @ file-append-string and @ file-append-lines .synb -.mets (time-fields-local << time ) -.mets (time-fields-utc << time ) +.mets (file-append < name << obj ) +.mets (file-append-string < name << string ) +.mets (file-append-lines < name << list ) .syne .desc -These functions take the numeric time returned by the time function, -and convert it to a list of seven fields. - The -.code time-string-local -function converts the time to the local timezone of -the host system. The -.code time-string-utc -function produces time in UTC. +.codn file-append , +.code file-append-string +and +.code file-append-lines +functions open a text stream over the file indicated by the string argument +.metn name , +write the argument object into the stream in their specific manner, +and then close the stream. -The fields returned as a list consist of six integers, and a Boolean value. -The six integers represent the year, month, day, hour, minute and second. -The Boolean value indicates whether daylight savings time is in effect -(always -.code nil -in the case of -.codn time-fields-utc ). +These functions are close counterparts of, respectively, +.codn file-get , +.code file-append-string +and +.codn file-append-lines . -The -.meta time -argument is an integer representing seconds obtained from the -.code time -function or from the -.code time-usec -function. +These functions behave differently when the indicated file +already exists. Rather than being truncated and overwritten, +the file is extended by appending the new data to its end. -.coNP Structure @ time +.coNP Functions @, command-get @ command-get-string and @ command-get-lines .synb -.mets (defstruct time nil -.mets \ \ year month day hour min sec dst -.mets \ \ gmtoff zone) +.mets (command-get << cmd ) +.mets (command-get-string << cmd ) +.mets (command-get-lines << cmd ) .syne .desc The -.code time -structure represents a time broken down into individual fields. -The structure almost directly corresponds to the -.code "struct tm" -type in the ISO C language. There are differences. -Whereas the -.code "struct tm" -member -.code tm_year -represents a year since 1900, the -.code year -slot of the -.code time -structure represents the absolute year, not relative to 1900. -Furthermore, the -.code month -slot represents a one-based numeric month, such that 1 represents -January, whereas the C member -.code tm_mon -uses a zero-based month. The -.code dst -slot is a \*(TL Boolean value. The slots -.codn hour , -.codn min , -and -.code sec -correspond directly to -.codn tm_hour , -.codn tm_min , -and -.codn tm_sec . +.code command-get +function opens text stream over an input command pipe created for +the command string +.metn cmd , +as if by the +.code open-command +function. It reads the printed representation of a \*(TL object from it, and +returns that object, ensuring that the stream is closed. -The slot -.code gmtoff -represents the number of seconds east of UTC, and -.code zone -holds a string giving the abbreviated time zone name. -On platform where the C type -.code "struct tm" -has fields corresponding to these slots, values for -these slots are calculated and stored into them by the -.code time-struct-local -and -.code time-struct-utc -functions, and also the related -.code time-local -and -.code time-utc -methods. On platform where the corresponding fields are not -present in the C language -.codn "struct tm" , -these slots are unaffected by those functions, -retaining the default initial value -.code nil -or a previously stored value, if applicable. -Lastly, the values of -.code gmtoff -and -.code zone -are not ignored by functions which accept a -.code time -structure as a source of input values. +The +.code command-get-string +is similar to +.code command-get +except that it reads the entire file as a text stream and returns +its contents in a single character string. -.coNP Functions @ time-struct-local and @ time-struct-utc +The +.code command-get-lines +function opens a text stream over an input command pipe created for the +command string +.meta cmd +and returns produces a lazy list of strings representing the lines +of text of that file as if by a call to the +.code get-lines +function, and returns that list. The stream remains open until the +list is consumed to the end, as indicated in the description of +.codn get-lines . + +.coNP Functions @, command-put @ command-put-string and @ command-put-lines .synb -.mets (time-struct-local << time ) -.mets (time-struct-utc << time ) +.mets (command-put < cmd << obj ) +.mets (command-put-string < cmd << string ) +.mets (command-put-lines < cmd << list ) .syne .desc -These functions take the numeric time returned by the time function, -and convert it to an instance of the -.code time -structure. - The -.code time-struct-local -function converts the time to the local timezone of -the host system. The -.code time-struct-utc -function produces time in UTC. +.codn command-put , +.code command-put-string +and +.code command-put-lines +functions open an output text stream over an output command pipe created +for the command specified in the string argument +.metn cmd , +as if by the +.code open-command +function. +They write the argument object into the stream in their specific manner, +and then close the stream. The -.meta time -argument is an integer representing seconds obtained from the -.code time -function or from the -.code time-usec -function. +.code command-put +function writes a printed representation of +.meta obj +using the +.code prinl +function. The return value is that of +.codn prinl . -.coNP Functions @, time-parse @ time-parse-local and @ time-parse-utc -.synb -.mets (time-parse < format << string ) -.mets (time-parse-local < format << string ) -.mets (time-parse-utc < format << string ) -.syne -.desc The -.code time-parse -function scans a time description in +.code command-put-string +function writes .meta string -according to the specification given in the -.meta format -string. If the scan is successful, a structure -of type -.code time -is returned, otherwise -.codn nil . +to the stream using the +.code put-string +function. The return value is that of +.codn put-string . The -.meta format -argument follows the same conventions as the POSIX -C library function -.codn strptime . +.code command-put-lines +function writes +.meta list +to the stream using the +.code put-lines +function. The return value is that of +.codn put-lines . -Prior to obtaining the time from -.meta format -and -.meta string -the returned structure is created and initialized -with a time which represents time 0 ("the epoch") -if interpreted in the UTC timezone as by the -.meta time-utc -method. +.SS* Buffer streams +A stream type exists which allows +.code buf +objects to be manipulated through the stream interface. +A buffer stream is created using the +.code make-buf-stream +function, which can either attach the stream to an existing buffer, +or create a new buffer that can later be retrieved from the stream +using +.codn get-buf-from-stream . +Operations on the buffer stream treat the underlying buffer much like if it +were a memory-based file. Unless the underlying buffer is a "borrowed buffer" +referencing the storage belonging to another object +(such as the buffer object produced by the +.code buf-d +FFI type's get semantics) the stream operations can change the buffer's size. +Seeking beyond the end of the buffer an then writing one or more bytes +extends the buffer's length, filling the newly allocated area with zero bytes. The -.code time-parse-local -and -.code time-parse-utc -functions return an integer time value: the same value -that would be returned by the -.code time-local +.code truncate-stream +function is supported also. +Buffer streams also support the +.code :byte-oriented +property. + +Macros +.code with-out-buf-stream and -.code time-utc -methods, respectively, when applied to the structure -object returned by -.codn time-parse . -Thus, these equivalences hold: +.code with-in-buf-stream +are provided to simplify the steps involved in using buffer streams +in some common scenarios. Note that in spite of the naming of these +macros there is only one buffer stream type, which supports bidirectional I/O. -.verb - (time-parse-local f s) <--> (time-parse f s).(time-local) - (time-parse-utc f s) <--> (time-parse f s).(time-utc) -.brev +.coNP Function @ make-buf-stream +.synb +.mets (make-buf-stream <> [ buf ]) +.syne +.desc +The +.code make-buf-stream +function return a new buffer stream. If the +.meta buf +argument is supplied, it must be a +.code buf +object. The stream is then associated with this object. +If the argument is omitted, a buffer of length zero is created and associated +with the stream. -Note: the availability of these three functions -depends on the availability of -.codn strptime . +.coNP Function @ get-buf-from-stream +.synb +.mets (get-buf-from-stream << buf-stream ) +.syne +.desc +The +.code get-buf-from-stream +returns the buffer object associated with +.meta buf-stream +which must be a buffer stream. -.coNP Methods @ time-local and @ time-utc +.coNP Macros @ with-out-buf-stream and @ with-in-buf-stream .synb -.mets << time-struct .(time-local) -.mets << time-struct .(time-utc) +.mets (with-out-buf-stream >> ( var <> [ buf-expr ]) +.mets \ \ << body-form *) +.mets (with-in-buf-stream >> ( var << buf-expr ) +.mets \ \ << body-form *) .syne .desc The -.code time -structure has two methods called -.code time-local +.code with-out-buf-stream and -.codn time-utc . +.code with-in-buf-stream +macros both bind variable +.meta var +to an implicitly created buffer stream, and evaluate zero or more +.metn body-form -s +in the environment where the variable is visible. The -.code time-local -function considers the slots of the -.code time -structure instance -.meta time-struct -to be local time, and returns its integer representation -as the number of seconds since the epoch. +.meta buf-expr +argument, which may be omitted in the use of the +.code with-out-buf-stream +macro, must be an expression which evaluates to a +.code buf +object. The -.code time-utc -function is similar, except it considers -the slots of -.meta time-struct -to be in the UTC time zone. +.meta var +argument must be a symbol suitable for naming a variable. -Note: these functions work by converting the slots into arguments -which are applied to -.code make-time -or -.codn make-time-utc . +The implicitly allocated buffer stream is connected +to the buffer specified by +.meta buf-expr +or, when +.meta buf-expr +is omitted, to a newly allocated buffer. + +The code generated by the +.code with-out-buf-stream +macro, if it terminates normally, yields the buffer object +as its result value. -.coNP Method @ time-string -.synb -.mets << time-struct .(time-string << format ) -.syne -.desc The -.code time -structure has a method called -.codn time-string . +.code with-in-buf-stream +returns the value of the last +.metn body-form , +or else +.code nil +if no forms are specified. -This method accepts a -.meta format -string argument, which it uses to convert -the fields to a character string representation -which is returned. +.TP* Examples: +.verb + (with-out-buf-stream (*stdout* (make-buf 24)) + (put-string "Hello, world!")) + -> #b'48656c6c6f2c2077 6f726c6421000000 0000000000000000' + + (with-out-buf-stream (*stdout*) (put-string "Hello, world!")) + -> #b'48656c6c6f2c2077 6f726c6421' +.brev + +.coSS The @ cptr type + +Objects of type +.code cptr +are Lisp values which contain a C pointer. This data type is used by the +.code dlopen +function and is generally useful in conjunction with the Foreign Function +Interface (FFI). An arbitrary pointer emanating from a foreign function +can be captured as a +.code cptr +value, which can be passed back into foreign code. For this purpose, there +exits also a matching FFI type called +.codn cptr . The -.meta format -argument is a string, and follows exactly the same conventions as -the format string of the C library function -.codn strftime . +.code cptr +type supports a symbolic type tag, which defaults to +.codn nil . +The type tag plays a role in FFI. The FFI +.code cptr +type supports a tag attribute. When a +.code cptr +object is converted to a foreign pointer under the control of the FFI +type, and that FFI type has a tag other than +.codn nil , +the object's tag must exactly match that of the FFI type, or the conversion +throws an error. In the reverse direction, when a foreign pointer is +converted to a +.code cptr +object under control of the FFI +.code cptr +type, the object inherits the type tag from the FFI type. -.coNP Method @ time-parse +.coNP Function @ cptr-int .synb -.mets << time-struct .(time-parse < format << string ) +.mets (cptr-int < integer <> [ type-symbol ]) .syne .desc The -.code time-parse -method scans a time description in -.meta string -according to the specification given in the -.meta format -string. - -If the scan is successful, the structure -is updated with the parsed information, and -the remaining unmatched portion of -.meta string -is returned. If all of -.meta string -is matched, then an empty string is returned. -Slots of -.meta time-struct -which are originally -.code nil -are replaced with zero, even if these -zero values are not actually parsed from -.metn string . - -If the scan is unsuccessful, then -.code nil -is returned and the structure is not -altered. +.code cptr-int +function converts +.meta integer +into a pointer in a system-specific way +which is consistent with the system's addressing structure. Then it returns +that pointer contained in a +.code cptr +object. The -.meta format -argument follows the same conventions as the POSIX -C library function -.codn strptime . +.meta integer +parameter must be an integer which is in range for a pointer value. +Note: this range is wider than the +.code fixnum +range; a portion of the range of +.code bignum +integers can denote pointers. -Note: the -.code time-parse -method may be unavailable if the host system does not -provide the -.code strptime -function. In this case, the -.code time-parse -static slot of the -.code time -struct is +The +.meta type-symbol +argument should be a symbol. If omitted, it defaults to .codn nil . +This symbol becomes the +.code cptr +object's type tag. -.coNP Functions @ make-time and @ make-time-utc +.coNP Function @ cptr-obj .synb -.mets (make-time < year < month < day -.mets \ \ \ \ \ \ \ \ \ \ < hour < minute < second << dst-advice ) -.mets (make-time-utc < year < month < day -.mets \ \ \ \ \ \ \ \ \ \ \ \ \ \ < hour < minute < second << dst-advice ) +.mets (cptr-obj < object <> [ type-symbol ]) .syne .desc The -.code make-time -function returns a time value, similar to the one returned by the -.code time -function. The -.code time -value is constructed not from the system clock, but -from a date and time specified as arguments. The -.meta year -argument is a calendar year, like 2014. -The -.meta month -argument ranges from 1 to 12. -The -.meta hour -argument is a 24-hour time, ranging from 0 to 23. -These arguments represent a local time, in the current time zone. +.code cptr-obj +function converts +.meta object +object directly to a +.codn cptr . The -.meta dst-advice -argument specifies whether the time is expressed in -daylight savings time (DST). It takes on three possible values: -.codn nil , -the keyword -.codn :auto , -or else the symbol -.codn t . -Any other value has the same interpretation as -.codn t . +.meta object +argument may be of any type. -If -.meta dst-advice -is -.codn t , -then the time is assumed to be expressed in DST. -If the argument is -.codn nil , -then the time is assumed not to be in DST. -If -.meta dst-advice -is -.codn :auto , -then the function tries to determine whether -DST is in effect in the current time zone for the specified date and time. +The raw representation of +.meta object +is simply stored in a new instance of +.code cptr +and returned. The -.code make-time-utc -function is similar to -.codn make-time , -except that -it treats the time as UTC rather than in the local time zone. +.meta type-symbol +argument should be a symbol. If omitted, it defaults to +.codn nil . +This symbol becomes the +.code cptr +object's type tag. + +.coNP Function @ int-cptr +.synb +.mets (int-cptr << cptr ) +.syne +.desc The -.meta dst-advice -argument is supported by -.code make-time-utc -for function -call compatibility with -.codn make-time . -It may or may not have any effect -on the output (since the UTC zone by definition doesn't have daylight -savings time). +.code int-cptr +function retrieves the pointer value of the +.meta cptr +object as an integer. -.SS* The Awk Utility +If an integer +.meta n +is in a range convertible to +.code cptr +type, then the expression +.mono +.meti (int-cptr (cptr-int << n )) +.onom +reproduces +.metn n . -The \*(TL library provides a macro called -.code awk -which is inspired by the Unix utility Awk. The macro implements -a processing paradigm very similar to that of the utility: it scans -one or more input streams, which are divided into records or fields, -under the control of user-settable regular-expression-based delimiters. -The records and fields are matched against a sequence of programmer-defined -conditions (called "patterns" in the original Awk), which have associated -actions. Like in Awk, the default action is to print the current record. +.coNP Function @ cptr-cast +.synb +.mets (cptr-cast < type-symbol << cptr ) +.syne +.desc +The +.code cptr-cast +function produces a new +.code cptr +object which has the same pointer as +.meta cptr +but whose type is given by +.metn type-symbol . -Unlike Awk, the -.code awk -macro is a robust, self-contained language feature which can be used -anywhere where a \*(TL expression is called for, cleanly nests -with itself and can produce a return value when done. By contrast, -a function in the Awk language, or an action body, cannot instantiate -a local Awk processing machine. +Casting +.meta cptr +objects with +.code cptr-cast +circumvents the safety mechanism which +.code cptr +type tagging provides. +.coNP Function @ cptr-zap +.synb +.mets (cptr-zap << cptr ) +.syne +.desc The -.code awk -macro implements some of the most important Awk -conventions and semantics, in Lisp syntax, while eschewing others. -It does not implement implement the Awk convention that -variables become defined upon first mention; variables must be -defined to be used. It doesn't implement Awk's weak type system. -A character string which looks like a number isn't a number, -and an empty string or undefined variable doesn't serve as zero -in arithmetic expressions enclosed in the macro. -All expression evaluation within -.code awk -is the usual \*(TL evaluation. +.code cptr-zap +function changes the pointer value of the +.meta cptr +object to the null pointer. The -.code awk -macro also does not provide a library of functions corresponding to -those in the Awk library, nor does it provide counterparts various -global variables in Awk such as the -.code ENVIRON -and -.code PROCINFO -arrays, or -.code RSTART -and -.codn RLENGTH . -Such features of Awk are extraneous to its central paradigm. +.meta cptr +argument must be of +.code cptr +type. -.coNP Macro @ awk +The return value is +.meta cptr +itself. + +Note: it is recommended to use +.code cptr-zap +when the program has taken some action which invalidates the pointer value +stored in a +.code cptr +object, where a risk exists that the value may be subsequently misused. + +.coNP Function @ cptr-free .synb -.mets (awk >> {( condition << action *)}*) +.mets (cptr-free << cptr ) .syne .desc The -.code awk -macro processes one or more input sources, which may be streams or -files. Each input source is scanned into records, and each record -is broken into fields. For each record, the sequence of condition-action -clauses (except for certain special clauses) is processed. Every -.meta condition -is evaluated, and if it yields true, the corresponding -.metn action -s -are evaluated. +.code cptr-free +function passes the +.meta cptr +object's pointer to the C library +.code free +function. After this action, it behaves exactly like +.codn cptr-zap . The -.meta condition -and -.meta action -forms are understood to be in a scope in which certain local -identifiers exist in the variable namespace as well as in the function -namespace. These are called -.I "awk functions" -and -.IR "awk macros" . +.meta cptr +argument must be of +.code cptr +type. -If -.meta condition -is one of the following keyword symbols, then it is a special clause, -with special semantics: -.codn :name , -.codn :let , -.codn :inputs , -.codn :output , -.codn :begin , -.codn :set , -.codn :end , -.codn :begin-file , -.code :set-file -and -.codn :end-file . -These clause types are explained below. -In such a clause, the -.meta action -expressions are not necessarily forms to be evaluated; the treatment -of these expressions depends on the clause. Otherwise, if -.meta condition -is not one of the above keyword symbols, the clause is an ordinary -condition-action clause, and -.meta condition -is a \*(TL expression, evaluated to determine a Boolean value -which controls whether the -.meta action -forms are evaluated. In every ordinary condition-action clause which -contains no -.meta action -forms, the -.code awk -macro substitutes the single action equivalent to the form -.codn "(prn)" : -a call to the local awk function -.codn prn . -The behavior of this macro, when called with no arguments, as above, -is to print the current -record (contents of the variable -.codn rec ) -followed by the output record terminator from the variable -.codn ors . +The return value is +.meta cptr +itself. -While the processing loop in -.code awk -scans an input source, it also binds the special variable -.code *stdin* -to the open stream associated with that source. This binding is -in effect across all ordinary clauses, as well as across the -special clauses -.code :begin-file -and -.codn :end-file . +Note: this function is unsafe. If the pointer didn't originate from the +.code malloc +family of memory allocation functions, or has already been freed, or +copies of the pointer exist which are still in use, the consequences are +likely catastrophic. -The following is a description of the special clauses: -.RS -.meIP (:name << sym ) +.coNP Function @ cptrp +.synb +.mets (cptrp << value ) +.syne +.desc The -.code :name -clause establishes the name of the implicit block contained -within the expansion of the -.code awk -macro. Forms enclosed in the macro can use -.code return-from -to abandon the -.code awk -form, specifying this symbol as the argument. +.code cptrp +function tests whether +.meta value +is a +.codn cptr . +It returns +.code t +if this is the +case, +.code nil +otherwise. -If the -.code :name -form is omitted, the implicit block is named -.codn awk . +.coNP Function @ cptr-type +.synb +.mets (cptr-type << cptr ) +.syne +.desc +The +.code cptr-type +function retrieves the +.meta cptr +object's type tag. -It is an error for two or more -.code :name -forms to appear. +.coNP Variable @ cptr-null +.desc +The +.code cptr-null +variable holds a null pointer as a +.code cptr +instance. + +Two +.code cptr +objects may be compared for equality using the +.code equal +function, which tests whether their pointers are equal. The -.code :name -clause must have an argument which is a symbol; -the symbol -.code nil -is not permitted. +.code cptr-null +variable compares +.code equal +to values which have been subject to +.code cptr-zap +or +.codn cptr-free . -.meIP (:let >> { sym | >> ( sym << init-form )}*) -Regardless of what order they appear in relation to -other clauses in the same -.code awk -macro, -.code :let -clauses are evaluated first before the macro takes any other action. The -argument forms of this clause are variables or variable-init forms. They are -treated the same way as analogous forms in the -.code let* -special form. Note that these are not enclosed in an extra list -as they are in the that form. The bindings established by the -.code :let -clause have a scope which extends over all the other clauses in the -.code awk -macro. +A null +.code cptr +may be produced by the expression +.codn "(cptr-obj nil)" ; +however, this creates a freshly allocated object on each evaluation. -If multiple -.code :let -clauses are present, they are effectively consolidated into -a single clause, in the order they appear. +The expression +.code "(cptr-int 0)" +also produces a null pointer on all platforms where \*(TX is found. -Note that the lexical variables, functions and macros established by the -.code awk -macro -(called, respectively, -.IR "awk macros" , -.I "awk functions" -and -.IR "awk variables" ) -are in an inner scope relative to -.code :let -bindings. For instance if -.code :let -creates a binding for a variable called -.codn fs , -that variable will be visible only to subsequent forms appearing -in the same -.code :let -clause or later -.code :let -clauses, and also visible in -.code :inputs -and -.code :output -clauses. -In -.codn :begin , -.codn :set , -.codn :end , -and ordinary clauses, it will be shadowed by the -.code awk -variable -.codn fs , -which holds the field separator regular expression or string. -.meIP (:inputs << source-form *) +.coNP Function @ cptr-size-hint +.synb +.mets (cptr-size-hint < cptr << bytes ) +.syne +.desc The -.code :inputs -clause is evaluated by the -.code awk -macro after processing the -.code :let -clauses. Each -.meta source-form -is evaluated and the values of these forms are gathered into a list. -This list then comprises the list of input sources for the -.code awk -processing task. +.code cptr-size-hint +function indicates to the garbage collector that the given +.meta cptr +object is associated with +.meta bytes +of foreign memory that are otherwise invisible to the garbage collector. -Each input source must be one of three kinds of objects. -It may be a stream object, which must be capable of character -input. It may be a list of strings, which -.code awk -will convert to an input stream as if by the -.code make-strlist-input-stream -function. -Or else it must be a character -string, which denotes a filesystem path name which -.code awk -will open for reading. +Note: this function should be used if the foreign memory is indirectly +managed by the +.meta cptr +object in cooperation with the garbage collector. Specifically, +.meta cptr +should have a finalizer registered against it which will liberate the +foreign memory. -If the -.code :inputs -clause is omitted, then a defaulting behavior occurs for obtaining -the list of input sources. If the special variable -.code *args* -isn't the empty list, then -.code *args* -is taken as the input sources. Otherwise, the -.code *stdin* -stream is taken as the one and only input source. +.SS* User-Defined Streams -If the -.code awk -macro uses -.code *args* -via the above defaulting behavior, it copies -.code *args* -and sets that variable to -.codn nil . -This is done in order that if -.code awk -is used from the \*(TX command line, for example using the -.code -e -command line option, after -.code awk -terminates, \*(TX will not try to open the next argument -as a script file or treat it as an option. -Note: programs which want -.code awk -not to modify -.code *args* -can explicitly specify -.code *args* -as the argument to the -.code :inputs -keyword, rather than allow -.code *args* -to be used through the defaulting behavior. Only the -defaulting behavior consumes the arguments by overwriting -.code *args* -with -.codn nil . +In \*(TL, stream objects aren't structure types, and therefore lie outside of +the object-oriented programming system. However, \*(TL supports a delegation +mechanism which allows a structure which provides certain methods to be used as +a stream. -It is an error to specify more than one -.code :inputs -clause. -.meIP (:output << output-form ) -The -.code :output -clause is processed just after the -.code :inputs -clause. It must have exactly one argument, which is an expression -that evaluates to a string, or else to an output stream. -If it evaluates to a string, then that string is used as the name -of a file to open for writing, and the resulting stream -is taken in place of that string. +The function +.code make-struct-delegate-stream +takes as an argument the instance of a structure, which is +referred to as the +.IR "stream interface object" . +The function returns a stream object such that when +stream operations are invoked on this stream, it delegates these +operations to methods of the stream interface object. + +A structure type called +.code stream-wrap +is provided, whose instances can serve as stream interface objects. +This structure has a slot called +.meta stream +which holds a stream, and it provides all of the methods required for +the delegation mechanism used by +.codn make-struct-delegate-stream . +This +.code stream-wrap +operations simply invoke the ordinary stream operations on the +.meta stream +slot. The +.code stream-wrap +type can be used as a base class for a derived class which intercepts +certain operations on a stream (by defining the corresponding methods) while +allowing other operations to transparently pass to the stream (via the +base methods inherited from +.codn stream-wrap ). +.coNP Function @ make-struct-delegate-stream +.synb +.mets (make-struct-delegate-stream << object ) +.syne +.desc The -.code :output -clause, if present, has the effect of creating a local binding for the -.code *stdout* -special variable. -This new value of -.code *stdout* -is visible to all forms within the macro. -If a -.code :let -clause is present, it establishes bindings -in a scope which is nested within the scope established -by -.codn :output . -Therefore, -.metn init-form -s -in the -.code :let -may refer to the new value of -.code *stdout* -established by -.codn :output . -Furthermore, -.code :let -can rebind -.codn *stdout* , -causing the definition provided by -.code :output -to be shadowed. +.code make-struct-delegate-stream +function returns a stream whose operations depend on the +.metn object , +a stream interface object. -In the case when the -.code :output -argument is a string such that a new stream is opened -on the file, the -.code awk -macro will close that stream when it finishes executing. -Moreover, that stream is treated uniformly as a member of -the set of streams that are implicitly managed by the -redirection macros in the same -.code awk -macro invocation. In brief, the implication is that if -.code :output -creates a stream for the file path name -.str "out.txt" -and somewhere in the same -.code awk -macro, there is a redirection of the form, or equivalent to -.mono -(-> "out.txt") -.onom -then this redirection shall refer to the same stream -that was established by -.codn :output . -Note also that in this example situation, the expression -.mono -(-> "out.txt" :close) -.onom -has the effect of closing the -.code :output -stream. - -.meIP (:begin << form *) -All -.code :begin -clauses are processed in the order in which they appear, before -input processing begins. -Each -.code form -is evaluated. These forms have in their scope the awk local variables -and macros. -.meIP (:set >> { place << new-value }*) The -.code :set -clause provides a shorthand which allows the frequently occurring pattern -.code "(:begin (set ...))" -to be condensed to -.codn "(:set ...)" . -.meIP (:end << form *) -All -.code :end -clauses are processed, in the order in which they appear, -when the input processing loop terminates. -This termination occurs when all records -from all input sources are either processed or skipped, or else -by an explicit termination such -as a dynamic non-local transfer, such as -.codn return-from , -or the throwing of an exception. - -Upon termination, the end clauses are processed in the order they appear. Each -.code form -is evaluated, left to right. +.meta object +argument must be a structure which implements certain +subsets of, or all of, the following methods: +.codn put-string , +.codn put-char , +.codn put-byte , +.codn get-line , +.codn get-char , +.codn get-byte , +.codn unget-char , +.codn unget-byte , +.codn put-buf , +.codn fill-buf , +.codn close , +.codn flush , +.codn seek , +.codn truncate , +.codn get-prop , +.codn set-prop , +.codn get-error , +.codn get-error-str , +.code clear-error +and +.codn get-fd . -In the normal termination case, the value of the last -.meta form -of the last end clause appears as the return value of the -.code awk -macro. +Implementing +.code get-prop +is mandatory, and that method must support the +.code :name +property. -Note that only termination of the -.code awk -macro initiated from condition-action clauses, -.code :begin-file -clauses, or -.code :end-file -clauses triggers -.code :end -clause processing. -If termination of the -.code awk -macro is initiated from within a -.codn :let , -.codn :inputs , -.code :output -or -.code :begin -clause, then end -clauses are not processed. -If an -.code :end -clause performs a non-local transfer, the remaining -.code :end -forms in that clause and -.code :end -clauses which follow are not evaluated. -.meIP (:begin-file << form *) -All -.code :begin-file -clauses are processed in the order in which they appear, before -.code awk -switches to each new input. +Failure to implement some of the other methods will impair the use of certain +stream operations on the object. -If both -.code :begin -and -.code :begin-file -forms are specified, then before the first input is processed, -.code :begin -clauses are processed first, then the -.code :begin-file -clauses. -.meIP (:set-file >> { place << new-value }*) +.coNP Method @ put-string +.synb +.mets << stream .(put-string str) +.syne +.desc The -.code :set-file -clause is a shorthand which translates -.code "(:set-file ...)" -to -.codn "(:begin-file (set ...))" . -.meIP (:end-file << form *) -All -.code :end-file -clauses are processed after the processing of an input -source finishes. - -If both -.code :end -and -.code :end-file -forms are specified, then before after the last input is processed, -.code :end-file -clauses are processed first, then the -.code :end -clauses. +.code put-string +method is implemented on a stream interface object. +It should behave in a manner consistent with the +description of the +.code put-string +stream I/O function. +.coNP Method @ put-char +.synb +.mets << stream .(put-char chr) +.syne +.desc The -:end-file -clauses are processed unconditionally, no matter how -the processing of an input source terminates, whether terminated -naturally by running out of records, prematurely by invocation of the -.code next-file -macro, or via a dynamic non-local control transfer such as a block -return or exception throw. - -If a -.code :begin-file -clause performs a non-local transfer, -.code :end-file -processing is not triggered, because the processing of the input -source is deemed not to have taken place. -.meIP >> ( condition << action *) -Clauses which do not have one of the specially recognized keywords -in the first position are ordinary condition-action clauses. After -processing the -.code :begin -clauses, the awk enters a loop in which it extracts successive records -from the input sources according to the -.code rs -(record separator) variable. Each record is divided into fields according -to the -.code fs -(field separator) -variable, and various -.code awk -variables are updated. Then, the condition-action clauses are processed, in the order -in which they appear. Each -.meta condition -is evaluated. If the resulting value is a regular expression -or a function, then this regular expression or function is invoked on the value -stored in the record variable -.codn rec , -and the result is taken to be the truth value of -.metn condition . -Otherwise, if the resulting value of -.meta condition -is other than a function or regular expression, it is taken directly -to be the truth value. -If the condition is true, then its associated -.meta action -forms are evaluated. Either way, processing passes to the next condition -clause (unless an explicit step is taken in one of the -.metn action -s -to prevent this, for instance by invoking the -.code next -and -.code next-file -macros). -When an input source runs out of records, -.code awk -switches to the next input source. When there are no more input sources, -the macro terminates. -.RE +.code put-char +method is implemented on a stream interface object. +It should behave in a manner consistent with the +description of the +.code put-char +stream I/O function. -.coNP Variables @ rec and @ orec +.coNP Method @ put-byte +.synb +.mets << stream .(put-byte byte) +.syne .desc -The awk variable -.code rec -holds the current record. It is automatically updated prior to the -processing of the condition-pattern clauses. Prior to the extraction -of the first record, its value is -.codn nil . - -It is possible to assign to -.codn rec . -The value assigned to -.code rec -must be a character string. Immediately upon the assignment, the character -string is delimited into fields according to the field separator -awk variable -.codn fs , -and these fields are assigned to the field list -.codn f . -At the same time, the -.code nf -variable is updated to reflect the new number of fields. -Likewise, modification of these variables causes -.code rec -to be reconstructed by a catenation of the textual representation -of the fields in -.code f -separated by copies of the output field separator -.codn ofs . - The -.code orec -variable ("original record") also holds the current record. It is automatically -updated prior to the processing of the condition-clauses at the same time as -.code rec -with the same contents. Like -.codn rec , -it is initially -.code nil -before the first record is read. The -.code orec -variable is unaffected by modification of -the variables -.codn rec , -.code f -and -.codn nf . -It may be assigned. Doing so has no effect on any other -variable. +.code put-byte +method is implemented on a stream interface object. +It should behave in a manner consistent with the +description of the +.code put-byte +stream I/O function. -.coNP Variable @ f +.coNP Method @ get-line +.synb +.mets << stream .(get-line) +.syne .desc -The awk variable -.code f -holds the list of fields. Prior to the first record being read, -its value is -.codn nil . -Whenever a new record is read, it is divided into fields according -to the field separator variable -.codn fs , -and these fields are stored in -.code f -as a list of character strings. - -If the variable -.code f -is assigned, the new value must be a sequence. The variable -.code nf -is automatically updated to reflect the length of this sequence. -Furthermore, the -.code rec -variable is updated by catenating a string representation of the -elements of this sequence, separated by the contents of the -.code ofs -(output field separator) -awk variable. - -Note that assigning to a DWIM bracket form which indexes -.codn f , -such as for instance -.code "[f 0]" -constitutes an implicit modification of -.codn f , -and triggers the recalculation of -.codn rec . -Modifications of the -.code f -list which do not involve an implicit or explicit assignment to the variable -.code f -itself do not have this recalculating effect. - -Unlike in Awk, assigning to the nonexistent field -.mono -.meti [f << m ] -.onom -where -.meta m ->= -.code nf -is erroneous. +The +.code get-line +method is implemented on a stream interface object. +It should behave in a manner consistent with the +description of the +.code get-line +stream I/O function. -.coNP Variable @ nf +.coNP Method @ get-char +.synb +.mets << stream .(get-char) +.syne .desc -The awk variable -.code nf -holds the current number of fields in the sequence -.codn f . -Prior to the first record being read, it is initially zero. +The +.code get-char +method is implemented on a stream interface object. +It should behave in a manner consistent with the +description of the +.code get-char +stream I/O function. -If -.code nf -is assigned, then -.code f -is modified to reflect the new number of fields. Fields are deleted from -.code f -if the new value of -.code nf -is smaller. If the new value of -.code nf -is larger, then fields are added. The added fields are empty strings, -which means that -.code f -must be a sequence of a type capable of holding elements which are -strings. +.coNP Method @ get-byte +.synb +.mets << stream .(get-byte) +.syne +.desc +The +.code get-byte +method is implemented on a stream interface object. +It should behave in a manner consistent with the +description of the +.code get-byte +stream I/O function. -If -.code nf -is assigned, then -.code rec -is also recalculated, in the same way as described in the documentation for the -.code f -variable. +.coNP Method @ unget-char +.synb +.mets << stream .(unget-char chr) +.syne +.desc +The +.code unget-char +method is implemented on a stream interface object. +It should behave in a manner consistent with the +description of the +.code unget-char +stream I/O function. -.coNP Variable @ nr +.coNP Method @ unget-byte +.synb +.mets << stream .(unget-byte byte) +.syne .desc -The awk variable -.code nr -holds the current absolute record number. Record numbers start at 1. -Absolute means that this value does not reset to 1 when -.code awk -switches to a new input source; it keeps incrementing for each record. -See the -.code fnr -variable. +The +.code unget-byte +method is implemented on a stream interface object. +It should behave in a manner consistent with the +description of the +.code unget-byte +stream I/O function. -Prior to the first record being read, the value of -.code nr -is zero. +.coNP Method @ put-buf +.synb +.mets << stream .(put-buf buf pos) +.syne +.desc +The +.code put-buf +method is implemented on a stream interface object. +It should behave in a manner consistent with the +description of the +.code put-buf +stream I/O function. -.coNP Variable @ fnr +.coNP Method @ fill-buf +.synb +.mets << stream .(fill-buf buf pos) +.syne .desc -The awk variable -.code fnr -holds the current record number within the file. The first record is 1. +The +.code fill-buf +method is implemented on a stream interface object. +It should behave in a manner consistent with the +description of the +.code fill-buf +stream I/O function. -Prior to the first record being read from the first input source, -the value of -.code fnr -is zero. Thereafter, it resets to 1 for the first record of each input -source and increments for the remaining records of the same input -source. +.coNP Method @ close +.synb +.mets << stream .(close offs whence) +.syne +.desc +The +.code close +method is implemented on a stream interface object. +It should behave in a manner consistent with the +description of the +.code close-stream +stream I/O function. -.coNP Variable @ arg +.coNP Method @ flush +.synb +.mets << stream .(flush offs whence) +.syne .desc -The awk variable -.code arg -is an integer which indicates what input source is being processed. -Prior to input processing, it holds the value zero. When the first -record is extracted from the first input source, it is set to 1. -Thereafter, it is incremented whenever -.code awk -switches to a new input source. +The +.code flush +method is implemented on a stream interface object. +It should behave in a manner consistent with the +description of the +.code flush-stream +stream I/O function. -.coNP Variable @ fname +.coNP Method @ seek +.synb +.mets << stream .(seek offs whence) +.syne .desc -The awk variable -.code fname -provides access to a character string which, if the current input is -a file stream, is the name of the underlying file. Assigning to this -variable changes its value, but has no effect on the input stream. -Whenever a new input source is used by -.code awk -it sets this variable either from the file name on which it is opening -a stream.. When using an existing stream rather than opening a file, -.code awk -sets this variable from the -.code :name -property of the stream. - -Note that the redirection macros -.code <- -and -.code 0 + (+ 1) -> 1 + (+ 1 "a") -> "1a" + (+ 1 2) -> 3 + (+ "a") -> "a" + (+ "a" "b" "c") -> "abc" -.verb - DATA rng -rng rng- -rng- --rng --rng- rng+ -rng+ --rng+ - ---------------------------------------------------------- - PROLOG - H1 X X X - H2 X X X X X X - H3 X X X X X X - B1 X X X X X X X X X - B2 X X X X X X X X - T1 X X X X X X - T2 X X X - T3 X X X - EPILOG + ;; macro expansions using usr:+ are not affected + (outside-macro "a") -> ;; error: + invalid operands "a" 42 .brev -The prefix or suffix characters are mnemonic. A single -.code - -(dash) indicates the exclusion of one record. A double -.code -- -(dash dash) -indicates the exclusion of all leading records which match -.metn from-condition ; -this appears on the left side only. -The -.code + -character, appearing on the right only, indicates that -all consecutive records which match -.meta to-condition -are included in the range, not only the first one. +.NP* Packages and the Extraction Language +The \*(TX extraction language has a syntax in which certain Lisp symbolic +expressions denoting directives +.code "@(collect ...)" +or +.code "@(end)" +behave as if they were the tokens of a phrase structure. As a matter of +implementation, these are processed specially in the parser and lexical +analyzer, and are not read in the same way as ordinary Lisp forms. -Ranges are oblivious to the division between successive sources of input; a -range can start in one file of records and terminate in another. -To prevent a range from spanning input transitions, additional complexity -is required in the expression. +On the other hand, some directives are not this way. For instance the +.codn "@(bind ...)" , +syntax is processed as a true Lisp expression, in which the +.code bind +token is subject to the usual rules for interning a symbol, sensitive to +.code *package* +in the usual way. -Ranges expressed using the -.code rng -family macros may combine with other expressions, including -other ranges, and allow arbitrary nesting: the -.meta from-condition +The following notes describe the treatment of "special" directives that are +involved in phrase structure syntax. It applies to all directives which head +off a block that must be terminated by +.codn "@(end)" , +all "punctuation" directives like +.code "@(and)" or -.meta to-condition -can be a range, or an expression containing ranges. +.code "@(end)" +and all sub-phrase indicators like +.code "@(last)" +or +.codn "@(elif)" . -The expressions -.meta from-condition -and -.meta to-condition -are ordinary expressions which are evaluated. However, their -evaluation is unusual in two ways. +Firstly, each such directive may have a package prefix on its main symbol, yet +is still recognized as the same token. That is to say, +.code "@(foo:collect)" +is still treated by the tokenizer and parser as the +.code "@(collect)" +token, regardless of the package prefix, and regardless of whether +.code foo:end +is the same symbol as the +.code usr:end +symbol. -Firstly, if either expression -produces, as its result, a function or regular expression object, -then that function or regular expression object is applied to -the current record (value of the -.code rec -variable), and the result of that application is then taken -as the result of the condition. This allows for expressions like -.code "(rng (f^ #/start/) #/end/)" -which denotes a range which begins with a record which -begins with the prefix -.str start -and ends with a record which contains -.str end -as a substring. +However, this doesn't mean that any +.code foo:collect +is allowed to denote the +.code collect +directive. -Secondly, the conditions are evaluated -out of order with respect to the surrounding expression -in which they occur. Ranges and their constituent -.meta from-condition -and -.meta to-condition -are evaluated just prior to the processing of the condition-action clauses. -Each -.code rng -expression is reduced to a Boolean value. -Then, when the condition-action clauses are processed and their -.meta condition +A qualified symbol such as +.code foo:collect +must correspond to (be the same object as) precisely one of two symbols: +either the same-named symbol in the +.code usr +package, or else the same-named symbol in the +.code keyword +package. If this condition isn't satisfied, the situation is a syntax +error. Note that this check uses the original +.code usr and -.meta action -forms are evaluated, each occurrence of a -.code rng -expression simply denotes its previously evaluated Boolean value. +.code keyword +packages, not the packages which are currently named +.str "usr" +or +.str "keyword" +in the current +.codn *package-alist* . -Therefore, it is not possible for expressions to short circuit -the evaluation of ranges. Ranges cannot "miss" their starting or -terminating conditions; every range occurring anywhere in the condition-action -clauses is tested against every record that is processed. +A check is also performed for an unqualified symbol. +An unqualified symbol like +.code collect +must also resolve, in the context of the current value of the +.code *package* +variable, to the same named-symbol in either the original +.code usr +or +.code keyword +package. Thus if the current package isn't +.codn usr , +and +.code "@(collect)" +is being processed, the current package must be such that +.code collect +resolves to +.codn usr:collect . +either because that symbol is present in the current pack via +import, or else visible via the fallback list. -Because of this perturbed evaluation order, code which happens to place side -effects into ranges may produce surprising results. +These rules are designed to approximate what the behavior would be +if these directives were actually scanned as Lisp forms in the usual +way and then recognized as phrase structure tokens according to +the identity of their leading symbol. The additional restriction is added that +that the directive symbol names are treated as reserved. If there exists a +user-defined pattern function called +.code mypackage:end +it may not be invoked using the syntax +.codn "@(mypackage:end)" , +which is erroneous; though it is invokable indirectly via the +.code "@(call)" +directive. -For instance, the expression -.code "(if nil (rng (prinl 'hello) (prinl 'world)))" -will produce output even though the -.code if -condition is -.codn nil , -and, moreover, this output will happen before the clauses are processed in -which this -.code if -expression appears. At the time when the -.code if -itself is evaluated, the -.code rng -expression merely fetches a previously computed Boolean value which indicates -whether the range is active for this record. +.NP* Package Library Conventions +Various functions in the package and symbol area of the library have a +.meta package +parameter. When the argument is optional, it defaults to the current +value of the +.code *package* +special variable. -Also, the behavior is unspecified if range expressions attempt to modify -the awk-special variables. -.codn rec , -.codn f , -.codn fs , -.code ft -or -.codn kfs . -It is not recommended to place any side effects into range expressions. +If specified, the argument may be a character string, which is taken as the +name of a package. It may also be a symbol, in which case the symbol's name, +which is a character string, is used. Thus the objects +.codn :sys , +.codn usr:sys , +.code abc:sys +and +.str sys +all refer to the same package, the system package which is named +.strn sys . -A more detailed description of the range operators follows. -.RS -.meIP (rng < from << to ) -This type of range becomes active when a record is encountered for which the -.meta from -expression yields true. While the range is active, the expression evaluates -true. If, when the range is active, a record is encountered for which the -.meta to -expression yields true, the range remains active for that record and is -deactivated after the completion of processing for that record. If -the range is inactive and a record is encountered or which both -.meta from +A +.code package +parameter may also simply be a package object. + +Some functions, like +.code use-package and -.meta to -are true, then the range is activated for that record and then deactivated -when that record is processed. -Records for which -.meta from +.code unuse-package +functions accept a list of packages as their first argument. +This may be a list of objects which follow the above conventions: +strings, symbols or package objects. +Also, instead of a list, an atom may be passed: a string, symbol +or package object. It is treated as a singleton list consisting +of that object. + +.coNP Variables @, user-package @ keyword-package and @ system-package +.desc +These variables hold predefined packages. The +.code user-package +contains all of the public symbols in the \*(TL library. +The +.code keyword-package +holds keyword symbols, which are printed with +a leading colon. The +.code system-package +is for internal symbols, helping +the implementation avoid name clashes with user code in some situations. + +These variables shouldn't be modified. If they are modified, the consequences +are unspecified. + +The names of these packages, respectively, are +.strn usr , +.strn sys , and -.meta to -are not true do not affect the range's activation state. -.meIP (-rng < from << to ) -This type of range is active under the same conditions as the -.code rng -type. However, the expression yields a Boolean false value for the -first record which begins a range. That is to say, when the range is -inactive, and a record is scanned for which -.meta from -is true, the range activates, but the range expression yields -.codn nil . -This is true regardless of whether the -.meta to -expression yields true for that record. If there are additional records -in the range, the expression yields a true value for those records. -.meIP (rng- < from << to ) -This type of range is active under the same conditions as the -.code rng -type. However, the range expression yields -.code nil -for the record for which -.code to -yields true which terminates the range. This occurs even if that is -the same record which activated the range by triggering the -.meta from -condition. Note that if a range terminates abruptly due to no more records -being available, the range expression still yields true for the last record. -.meIP (-rng- < from << to ) -This type of range is active under the same conditions as the -.code rng -type. However, the range expression yields -.code nil -for the first record which activates the range, and for the last -record which deactivates the range by activating the -.code to -condition. If the range is active over fewer than three records, then -the expression never yields true for that range. If the range terminates -abruptly due to no more records being available, and if the last record -processed isn't the one which activated the range due to triggering the -.code from -condition, the expression yields true for that record. -.meIP (--rng < from << to ) -This type of range is active under the same conditions as -.codn rng . -However, the range expression yields -.code nil -for the entire leading sequence of consecutive records for which -.meta from -is true. If -.meta from -is true of the -.meta to -record which terminates the range, -.code nil -is returned for that record also. -.meIP (--rng- < from << to ) -This type of range is active under the same conditions as -.codn rng . -However, the range expression yields -.code nil -for the entire leading sequence of consecutive records for which -.meta from -is true, and also yields nil for the last record which triggers the -.meta to -condition. -.meIP (rng+ < from << to ) -This range is active under different conditions compared to -.codn rng . -Though it becomes active in the same way, when the -.meta from -expression yields true, the deactivation logic is different. -The range is deactivated when a record for which -.meta to -is true is followed by a record for which -.meta to -is not true. That record is excluded from the range; if the -.meta from -expression happens to be true for that record, a new range begins -at that record. Thus, effectively, the range is terminated not -by single record which triggers -.meta to -but by a sequence of one or more such consecutive records. -.meIP (-rng+ < from << to ) -This range is active under the same conditions as -.codn rng+ . -However, the range expression yields -.code nil -for the first record in the range. If the range contains only one record, then -it returns -.code nil -for that record. -.meIP (--rng+ < from << to ) -This range is active under the same conditions as -.codn rng+ . -However, the range expression yields -.code nil -for the entire leading sequence of consecutive records for which -.meta from -is true. This is the case even for those for which the -.meta to -expression is true. -.RE +.strn keyword . -.coNP Macro @ ff -.synb -.mets (ff < opip-arg *) -.syne +.coNP Special variable @ *package* .desc -The awk macro -.code ff -(filter fields) -provides a shorthand for filtering the field list -.code f -trough a pipeline of chained functions expressed using -.code opip -argument syntax. +This variable holds the current package. The global value of this variable +is initialized to a package called +.strn pub . +The +.code pub +package has the +.code usr +package in its fallback list; thus when +.code pub +is current, all of the +.code usr +symbols, comprising the content of the \*(TL library, are visible. -The following equivalence holds, except that -.code f -refers to the awk variable even if the -.code mf -invocation occurs in code which establishes -a binding which shadows -.codn f . +All forms read and evaluated from the \*(TX command line, in the interactive listener, +from files via +.code load +or +.code compile-file +or from the \*(TX pattern language are processed in this default +.code pub +package, unless arrangement are made to change to a different package. -.verb - (ff a b c ...) <--> (set f [(opip a b c ...) f]) -.brev +The current package is used as the default package for interning symbol tokens +which do not carry the colon-delimited package prefix. -.TP* Example: -.verb - ;; convert all fields from string to floating-point - (ff (mapcar flo-str)) -.brev +The current package also affects printing. When a symbol is printed whose +home package matches the current package, it is printed without a package +prefix. (Keyword symbols are always printed with the colon prefix, even if the +keyword package is current.) -.coNP Macro @ mf +.coNP Function @ make-sym .synb -.mets (mf < opip-arg *) +.mets (make-sym << name ) .syne .desc -The awk macro -.code mf -(map fields) -provides a shorthand for mapping each field -individually trough a pipeline of chained functions expressed using -.code opip -argument syntax. +The +.code make-sym +function creates and returns a new symbol object. The argument +.metn name , +which must be a string, specifies the name of the symbol. The symbol +does not belong to any package (it is said to be "uninterned"). -The following equivalence holds, except that -.code f -refers to the awk variable even if the -.code mf -invocation occurs in code which establishes -a binding which shadows -.codn f . +Note: an uninterned symbol can be interned into a package with the +.code rehome-sym +function. Also see the +.code intern +function. -.verb - (mf a b c ...) <--> (set f (mapcar (opip a b c ...) f)) -.brev +.coNP Function @ gensym +.synb +.mets (gensym <> [ prefix ]) +.syne +.desc +The +.code gensym +function is similar to make-sym. It creates and returns a new +symbol object. If the +.meta prefix +argument is omitted, it defaults to +.strn g . +Otherwise it must be a string. -.TP* Example: -.verb - ;; convert all fields from string to floating-point - (mf flo-str) -.brev +The difference between +.code gensym +and +.code make-sym +is that +.code gensym +creates the name +by combining the prefix with a numeric suffix. -.coNP Macro @ fconv +The numeric suffix is a decimal digit string, taken from the value of +the variable +.codn *gensym-counter* , +after incrementing it. + +Note: the variation in name is not the basis of the uniqueness assurance +offered by +.code make-sym +and +.codn gensym ; +the basis is that the returned symbol is a freshly instantiated object. +.code make-sym +still returns unique symbols even if repeatedly called with the same +string. + +.coNP Special variable @ *gensym-counter* +.desc +This variable is initialized to 0. Each time the +.code gensym +function is called, +it is incremented. The incremented value forms the basis of the numeric +suffix which +.code gensym +uses to form the name of the new symbol. + +.coNP Function @ make-package .synb -.mets (fconv >> { clause | : | - }*) +.mets (make-package << name ) .syne .desc -The awk macro -.code fconv -provides a succinct way to request conversions of the textual fields. -Conversions are expressed by clauses which correspond with fields. +The +.code make-package +function creates and returns a package named +.metn name , +where +.meta name +is a string. It is an error if a package by that name exists already. +Note: ordinary creation of packages for everyday program modularization +should be performed with the +.code defpackage +macro rather than by direct use of +.codn make-package . -Each -.meta clause -is an expression which must evaluate to a function. The clause is evaluated -in the same manner as the argument a -.code dwim -operator, using Lisp-1-style name lookup. Thus, functions may be -specified simply by using their name as a -.metn clause . +.coNP Function @ delete-package +.synb +.mets (make-package << package ) +.syne +.desc +The +.code delete-package +breaks the association between a package and its name. +After +.codn delete-package , +the +.meta package +object continues to exist, but cannot be found using +.codn find-package . -Furthermore, several local functions exist in the scope of each -.metn clause , -providing a short-hand notation. These are described below. +Furthermore, +.code delete-package +iterates over all remaining packages. For each remaining package +.metn p , +it performs the semantic action of the +.mono +.meti (unuse-package < package << p) +.onom +expression. That is to say, all of the remaining packages +are scrubbed of any foreign symbols which are the local symbols +of the deleted +.metn package . -Conversion proceeds by applying the function produced by -a clause to the field to which that clause corresponds, positionally. -The return value of the function applied to the field replaces -the field. +.coNP Function @ packagep +.synb +.mets (packagep << obj ) +.syne +.desc +The +.code packagep +function returns +.code t +if +.meta obj +is a package, otherwise it returns +.codn nil . -When a clause is specified as the symbol -.code - -(minus) -it has a special meaning: this minus clause occupies a field -position and corresponds to a field, but performs no conversion -on its field. +.coNP Function @ find-package +.synb +.mets (find-package << name ) +.syne +.desc +The argument +.meta name +should be a string. If a package called +.meta name +exists, +then it is returned. Otherwise +.code nil +is returned. +.coNP Special variable @ *package-alist* +.desc The -.code : -(colon) -symbol isn't a clause and does not correspond to a field position. -Rather, it acts as a separator among clauses. It need not appear at -all. If it appears, it may appear at most twice. Thus, the -clauses may be separated into up to three sequences. +.code *package-alist* +variable contains the master association list +which contains an entry about each existing +package. -If the colon does not appear, then all the clauses are -.IR "prefix clauses" . -Prefix clauses line up with fields from left to right. If there are fewer -fields than prefix clauses, the values of the excess clauses are evaluated, but -ignored. -.IR "Vice versa" , -if there are fewer prefix clauses than fields, then the excess -fields are not subject to conversions. +Each element of the list is a cons cell +whose +.code car +field is the name of a package and whose +.code cdr +is a package object. -If the colon appears once, then the clauses before the colon, if any, are -prefix clauses, as described in the previous paragraph. Clauses after the -colon, if any, are -.IR "interior clauses" . -Interior clauses apply to any fields which are left unconverted by the prefix -clauses. All interior clauses are evaluated. If there are fewer fields than -interior clauses, then the values of the excess interior clauses are ignored. -If there are more fields than clauses, then the clause values are cycled: -re-used from the beginning against the excess fields, enough times to convert -all the fields. +Note: the \*(TL application can overwrite or re-bind this +variable to manipulate the active package list. This is +very useful for +.IR sandboxing : +safely evaluating code that is obtained as an input +from an untrusted source, or calculated from such an input. -If the colon appears twice, then the clauses before the first colon, if any, -are prefix clauses, the clauses between the two clause are interior clauses, -and those after the second colon are -.IR "suffix clauses" . -The presence of suffix clauses change the behavior relative to the one-colon -case as follows. After the conversions are performed according to the prefix -clauses, the remaining fields are counted. If there are are only as many -fields as there are suffix clauses, or fewer, then the interior clauses are -evaluated, but ignored. The remaining fields are processed against the suffix -clauses. If after processing the prefix clauses there are more fields -remaining than suffix clauses, then a number of rightmost fields equal to the -number of suffix clauses is reserved for those clauses. The interior fields -are applied only to the unreserved middle fields which precede these reserved -rightmost fields, using the same repeating behavior as in the one-colon case. -Finally, the previously reserved rightmost fields are processed using -the suffix clauses. - -The following special convenience functions are in scope of the clauses, -effectively providing a short-hand for commonly-needed conversions: -.RS -.coIP i -Provides conversion to integer. It is identical to the -.code toint -function. -.coIP o -Converts a string value holding an octal representation -to the integer which it denotes. The expression -.code "(o str)" -is equivalent to -.codn "(toint str 8)" . -.coIP x -Converts a string value holding a hexadecimal representation -to the integer which it denotes. The expression -.code "(x str)" -is equivalent to -.codn "(toint str 16)" . -.coIP b -Converts a string value holding a binary (base two) representation -to the integer which it denotes. The expression -.code "(b str)" -is equivalent to -.codn "(toint str 2)" . -.coIP c -Converts a string value holding a C-language-style representation -to the integer which it denotes, meaning that the -.code 0x -prefix denotes a hexadecimal value, a leading zero octal, otherwise -decimal. These prefixes follow the -.code + -or -.code - -sign, if present. -The expression -.code "(c str)" -is equivalent to -.codn "(toint str #\ec)" . -.coIP r -Converts a string holding a floating-point representation to -the floating-point value which it denotes. The expression -.code "(r str)" -is equivalent to -.codn "(tofloat str)" . -.ccIP @, iz @, oz @, xz @, bz @ cz and @ rz -Conversion similar to -.codn i , -.codn o , -.codn x , -.codn b , -.code c -and -.codn r , -but using -.code tointz -and -.codn tofloatz . -Thus fields which are non-numeric strings or the object -.code nil -get converted to 0, or 0.0 in the case of -.codn rz . -.RE -.IP -Because -.code fconv -macro destructively operates on the elements of the field list -.codn f , -it has the same effect as an assignment to the fields: -the value of -.code rec -is updated. +The contents of +.code *package-alist* +have security implications because textual source code +can refer to any symbol in any package by invoking +a package prefix. For instance, even if the +.code open +function's name is not available in the current package +(established by the +.code *package* +variable) that symbol can easily be obtained using the +syntax +.codn usr:open . -The return value of -.code fconv -is -.codn f . +However, the entire +.code usr +package itself can be removed from +.codn *package-alist* . +In that situation, the syntax +.code usr:open +is no longer valid. -.TP* Examples: +At the same time, selected symbols from the original +.code usr +can be nevertheless made available via some intermediate +package, which is present in +.code *package-alist* +and which contains a subset of the +.code usr +symbols that has been curated for safety. That curated package may even +be called +.codn usr , +so that if for instance +.code cons +is present in that package, it may be referred to as +.code usr:cons +in the usual way. -.verb - ;; convert up to first three fields to integer: - (awk ((fconv i i i))) +.coNP Function @ package-alist +.synb +.mets (package-alist) +.syne +.desc +The +.code package-alist +function retrieves the value of +.codn *package-alist* . - ;; convert all fields to floating-point - (awk ((fconv : r :))) +Note: this function is obsolescent. There is no reason to use it +in new code instead of just accessing +.code *package-alist* +directly. - ;; convert first and second fields to integer - ;; from hexadecimal; - ;; convert last field to integer from octal; - ;; process pairs of fields in between - ;; these by leaving the first element of - ;; each pair unconverted and converting second - ;; to floating-point; - (awk ((fconv x x : - r : o))) +.coNP Function @ package-name +.synb +.mets (package-name << package ) +.syne +.desc +The +.code package-name +function retrieves the name of a package. - ;; convert all fields, except the first, - ;; from integer, turning empty strings - ;; and non-integer junk as zero; - ;; leave first field unconverted: - (awk ((fconv - : iz))) -.brev +.coNP Function @ package-symbols +.synb +.mets (package-symbols <> [ package ]) +.syne +.desc +The +.code package-symbols +function returns a list of all the symbols +which are interned in +.metn package . -.coNP Macros @, -> @, ->> @, <- @ !> and @ < path << form *) -.mets (->> < path << form *) -.mets (<- < path << form *) -.mets (!> < command << form *) -.mets ( [ package ]) +.mets (package-foreign-symbols <> [ package ]) .syne .desc -These awk macros provide convenient redirection of output and input to and from -files and commands. +The +.code package-local-symbols +function returns a list of all the symbols +which are interned in +.metn package , +and whose home package is that package. -When at least one -.meta form -argument is present, the functions -.codn -> , -.code ->> -and -.code !> -evaluate each -.meta form -in a dynamic environment in which the -.code *stdout* -variable is bound to a file output stream, for the first two -functions, or output command pipe in the case of the last one. +The +.code package-foreign-symbols +function returns a list of all the symbols which +are interned in +.metn package , +which do not have that package as their home package, +or do not have a home package at all. -Similarly, when at least -.meta form -argument is present, the remaining functions -.code <- -and -.code -macro indicates that the file named -.meta path -is to be opened for writing and overwritten, or created if it doesn't exist. +.code set-package-fallback-list +replaces the fallback package list of +.meta package +with +.metn package-list . + The -.code ->> -macro indicates that the file named by -.meta path -is to be opened in append mode, created if necessary. +.meta package-list +argument must be a list which is a mixture of symbols, strings or +package objects. Strings are taken to be package names, which must +resolve to existing packages. Symbols are reduced to strings via +.codn symbol-name . + +.coNP Function @ intern +.synb +.mets (intern < name <> [ package ]) +.syne +.desc +The argument +.meta name +should be a symbol. The optional argument +.meta package +should be a package. If +.meta package +is not supplied, then the value +taken is that of +.codn *package* . + The -.code <- -macro indicates that the file given by -.meta path -is to be opened for reading. +.code intern +function searches +.meta package +for a symbol called +.metn name . +If that symbol is found, it is returned. If that symbol is not found, +then a new symbol called +.meta name +is created and inserted into +.metn package , +and that symbol is returned. In this case, the package becomes the +symbol's home package. +.coNP Function @ unintern +.synb +.mets (unintern < symbol <> [ package ]) +.syne +.desc The -.code !> -macro indicates that -.meta command -is to be opened as an output command pipe. The -.code [ package ]) +.syne +.desc +The arguments +.meta symbol +and +.meta package +must be a symbol and package object, +respectively, and +.meta symbol +must not be the symbol +.codn nil . -.coNP Examples of @ awk Macro Usage -The following examples are -.code awk -macro equivalents of the examples of the POSIX -.code awk -utility given in IEEE Std 1003.1, 2013 Edition. +The +.code rehome-sym +function moves +.meta symbol +into +.metn package . +If +.meta symbol +is already interned in a package, it is first removed from that package. -.RS -.IP 1. -Print lines for which field 3 is greater than 5: +If a symbol of the same name exists in +.metn package , +that symbol is first removed +from +.metn package . -.verb - ;; print lines with fields separated by ofs, - ;; and [f 2] converted to integer: - (awk ((and [f 2] (fconv - - iz) (> [f 2] 5)))) +Also, if a symbol of the same name exists in the hidden symbol store of +.metn package , +that hidden symbol is removed. - ;; print strictly original lines from orec - (awk ((and [f 2] (fconv - - iz) (> [f 2] 5)) - (prn orec))) +Then +.code symbol +is interned into +.metn package , +and +.meta package +becomes its home package, making it a local symbol of +.metn package . -.brev -.IP 2. -Print every tenth line: +Note: if +.code symbol +is currently the hidden symbol of some package, it is not removed +from the hidden symbol store of that package. This is a degenerate +case. The implication is that if that hidden symbol is ever +restored in that package, it will once again have that package as +its home package, and consequently it will turn into a foreign +symbol of +.metn package . -.verb - (awk ((zerop (mod nr 10)))) -.brev -.IP 3. -Print any line with a substring matching a regex: +.coNP Function @ symbolp +.synb +.mets (symbolp << obj ) +.syne +.desc +The +.code symbolp +function returns +.code t +if +.meta obj +is a symbol, otherwise it returns +.codn nil . -.verb - (awk (#/(G|D)(2\ed[\ew]*)/)) -.brev -Note the subtle flaw here: the -.code [\ew]* -portion of the regular expression contributes nothing -to what lines are matched. The following example -has a similar flaw. -.IP 4. -Print any line with a substring beginning with a -.code G -or -.code D -followed by a sequence of digits and characters: - -.verb - (awk (#/(G|D)([\ed\ew]*)/)) -.brev -.IP 5. -Print lines where the second field matches a regex, -while the fourth one doesn't: - -.verb - (awk (:let (r #/xyz/)) - ((and [f 3] [r [f 1]] (not [r [f 3]])))) -.brev -.IP 6. -Print lines containing a backslash in the second field: - -.verb - (awk ((find #\e\e [f 1]))) -.brev -.IP 7. -Print lines containing a backslash using a regex constructed -from a string. Note that backslash escapes are interpreted -twice: once in the string literal, and once in the parsing -of the regex, requiring four backslashes to encode one: - -.verb - (awk (:let (r (regex-compile "\e\e\e\e"))) - ((and [f 1] [r [f 1]]))) -.brev -.IP 8. -Print penultimate and ultimate field in each record, -separating then by a colon: - -.verb - ;; original: {OFS=":";print $(NF-1), $NF} - ;; - (awk (t (set ofs ":") (prn [f -2] [f -1]))) -.brev -.IP -Note that the above behaves -more correctly than the original Awk example because in the -when there is only one field, -.code $(NF-1) -reduces to -.code $0 -which refers to the entire record, not to the field. -This sort of bug is why the \*(TL -.code awk -does not imitate the design decision to make the record -the first numbered field. -.IP 9. -Output the line number and number of fields separated by colon, -by producing a single string first: - -.verb - (awk (t (prn `@nr:@nf`))) -.brev -.IP 10. -Print lines longer than 72 characters: - -.verb - (awk ((> (len rec) 72))) -.brev -.IP 11. -Print first two fields in reverse order, separated by -.codn ofs : - -.verb - (awk (t (prn [f 1] [f 0]))) -.brev -.IP 12. -Same as 11, but with field separation consisting of a -comma, or spaces and tabs, or both in sequence: +.coNP Function @ symbol-name +.synb +.mets (symbol-name << symbol ) +.syne +.desc +The +.code symbol-name +function returns the name of +.metn symbol . -.verb - (awk (:set fs #/,[ \et]*|[ \et]+/) - (t (prn [f 1] [f 0]))) -.brev -.IP 13. -Add the values in the first column, then print sum and -average: +.coNP Function @ symbol-package +.synb +.mets (symbol-package << symbol ) +.syne +.desc +The +.code symbol-package +function returns the home package of +.metn symbol . +If +.meta symbol +has no home package, it returns +.codn nil . -.verb - ;; original: - ;; {s += $1} - ;; END {print "sum is ", s, " average is", s/NR} - ;; - (awk (:let (s 0) (n 0)) - ([f 0] (fconv r) (inc s [f 0]) (inc n)) - (:end (prn `sum is @s average is @(/ s n)`))) -.brev +.coNP Function @ keywordp +.synb +.mets (keywordp << obj ) +.syne +.desc +The +.code keywordp +function returns +.code t +if +.meta obj +is a keyword symbol, otherwise it +returns +.codn nil . -Note that the original is not robust against blank lines -in the input. Blank lines are treated as if they had a -first column field of zero, and are counted toward the -denominator in the calculation of the average. -.IP 14. -Print fields in reverse order, one per line: +.coNP Function @ bindable +.synb +.mets (bindable << obj ) +.syne +.desc +The +.code bindable +function returns +.code t +if +.meta obj +is a bindable symbol, otherwise it returns +.codn nil . -.verb - (awk (t (tprint (reverse f)))) -.brev -.IP 15. -Print all lines between occurrences of -.code start +All symbols are bindable, except for keyword symbols, and the +special symbols +.code t and -.codn stop : - -.verb - (awk ((rng #/start/ #/stop/))) -.brev -.IP 16. -Print lines whose first field is different from -the corresponding field in the previous line: +.codn nil . -.verb - (awk (:let prev) - ((nequal [f 0] prev) (prn) (set prev [f 0]))) -.brev -.IP 17. -Simulate the -.code echo -utility: +.coNP Function @ use-sym +.synb +.mets (use-sym < symbol <> [ package ]) +.syne +.desc +The +.code use-sym +function brings an existing +.code symbol +into +.metn package . -.verb - (awk (:begin (prn `@{*args* " "}`))) -.brev +In all cases, the function returns +.codn symbol . -Note: if this is evaluated in the command line, for instance with the -.code -e -option, an explicit exit is required to prevent the arguments from being -processed by -\*(TX after -.code awk -completes: +If +.meta symbol +is already interned in +.metn package , +then the function has no effect. -.verb - (awk (:begin (prn `@{*args* " "}`) (exit 0))) -.brev -.IP 18. -Pint the components of the -.code PATH -environment variable, one per line: +Otherwise +.meta symbol +is interned in +.metn package . -.verb - ;; Process variable as if it were a file: - (awk (:inputs (make-string-input-stream - (getenv "PATH"))) - (:set fs ":") - (t (tprint f))) +If a symbol having the same name as +.meta symbol +already exists in +.metn package , +then it is replaced. +If that replaced symbol is a local symbol of +.metn package , +then the replaced symbol turns into a hidden symbol associated +with the package. It is placed into a special hidden symbol store +associated with +.meta package +and is stripped of its home package, becoming quasi-interned or uninterned. - ;; Just get, split and print; awk macro is irrelevant - (awk (:begin (tprint (split-str (getenv "PATH") ":")))) -.brev -.IP 19. -Given a file called -.code input -which contains page headers of the format -.str "Page #" -and a \*(TL file called -.code prog.tl -which contains: +An odd case is possible whereby +.meta symbol +is already a hidden symbol of +.metn package . +In this case, the hidden symbol replaces some foreign symbol and +is interned in +.metn package . +Thus it simultaneously exists as both an interned +foreign symbol and as a hidden symbol of +.metn package . -.verb - (awk (:let (n (toint n))) - (#/Page/ (set [f 1] (pinc n))) - (t)) -.brev +.coNP Function @ unuse-sym +.synb +.mets (unuse-sym < symbol <> [ package ]) +.syne +.desc +The +.code unuse-sym +function removes +.meta symbol +from +.metn package . -the command line: +If +.meta symbol +is not interned in +.metn package , +the function does nothing and returns +.codn nil . -.verb - txr -Dn=5 prog.tl input -.brev +If +.meta symbol +is a local symbol of +.metn package , +an error is thrown: a package cannot "unuse" its own symbol. Removing +a symbol from its own home package requires the +.code unintern +function. -prints the file, filling in page numbers starting at 5. -.RE +Otherwise +.meta symbol +is a foreign symbol interned in +.meta package +and is removed. -.SS* Environment Variables and Command Line +If the package has a hidden symbol of the same name as +.metn symbol , +that symbol is re-interned into +.meta package +as a local symbol. In this case, that previously hidden symbol is +returned. -Note that environment variable names, their values, and command line -arguments are all regarded as being externally encoded in UTF-8. \*(TX performs -the encoding and decoding automatically. +If the package has no hidden symbol matching the removed +.metn symbol , +then +.meta symbol +itself is returned. -.coNP Special variables @, *args-full* @ *args-eff* and @ *args* +.coNP Functions @ use-package and @ unuse-package +.synb +.mets (use-package < package-list <> [ package ]) +.mets (unuse-package < package-list <> [ package ]) +.syne .desc The -.code *args-full* -variable holds the original, complete list of arguments passed -from the operating system, including the program executable -name. - -During command line option processing, \*(TX may transform the -argument list. The hash bang mechanism, and the -.code --args +.meta use-package and -.code --eargs -options can inject new command line arguments, as can code -which is executed during argument processing via the -.code -e -options and others. +.meta unuse-package +are convenience functions which perform a mass import of symbols from one +package to another, or a mass removal, respectively. The -.code *args-eff* -variable holds the list of -.I "effective arguments" , -which is the argument list after these transformations are applied. -This variable is established and set to the same value as -.code *args-full* -prior to command line processing, but is not updated with its final -value until after command line processing. +.code use-package +function iterates over all of the local symbols of the packages in +.metn package-list . +For each symbol +.metn s , +it performs the semantic action implied by the +.mono +.meti (use-sym < s << package ) +.onom +expression. -The -.code *args* -variable holds a list of strings representing the remaining -arguments which follow any options processed by the \*(TX executable, -and the script name. This list is a suffix of -.codn *args-eff* . -Thus, the arguments before -.code *args* -can be calculated using the expression -.codn "(ldiff *args-eff* *args*)" . +Similarly +.code unuse-package +iterates +.meta package-list +in the same way, performing, effectively, the semantic action +of the +.mono +.meti (unuse-sym < s << package ) +.onom +expression. The -.code *args* -variable is available to to \*(TL expressions invoked from the -command line via the -.codn -p , -.code -e -and other such options. During these evaluations, -.code *args* -holds all the remaining options, after the invoking option and its -argument expression. In other words, code executed from the command line -has access to the remaining arguments which follow it. -Furthermore, this code may modify the value of -.codn *args* . -Such a modification is visible to the option processing code. -That is to say code executed from the command line can rewrite the remaining -list of arguments, and that list takes effect. +.meta package-list +argument must be a list which is a mixture of symbols, strings or +package objects. Strings are taken to be package names, which must +resolve to existing packages. Symbols are reduced to strings via +.codn symbol-name . -.coNP Function @ env +.coNP Macro @ defpackage .synb -.mets (env) +.mets (defpackage < name << clause *) .syne .desc The -.code env -function retrieves the list of environment variables. Each -variable is represented by a single entry in the list: a string which -contains an -.code = -(equal) character somewhere, separating the variable name -from its value. - -See also: the -.code env-hash -function. +.code defpackage +macro provides a convenient means to create a package and establish its +properties in a single construct. It is intended for the ordinary situations +in which packages support the organization of programs into modules. -.coNP Function @ env-hash -.synb -.mets (env-hash) -.syne -.desc The -.code env-hash -function constructs and returns an -.code :equal-based -hash. The hash is -populated with the environment variables, represented as key-value pairs. +.code name +argument, giving the package name, may be a symbol or a character string. +If it is a symbol, then the symbol's name is taken to be name for the +package. -.coNP Functions @, getenv @ setenv and @ unsetenv -.synb -.mets (getenv << name ) -.mets (setenv < name < value <> [ overwrite-p ]) -.mets (unsetenv << name ) -.syne -.desc -These functions provide access to, as well as manipulation of, environment -variables. Of these three, -.code setenv -and -.code unsetenv -might not be available on some platforms, or -.code unsetenv -might be be present in a simulated form which sets the variable -.meta name -to the empty string rather than deleting it. +If a package called +.code name +already exists, then +.code defpackage +selects that package for further operations. Otherwise, a new, +empty package is created. In either case, this package is referred +to as the +.I "present package" +in the following descriptions. The -.code getenv -function searches the environment for the environment variable whose name -is -.metn name . -If the variable is found, its value is returned. Otherwise -.code nil -is returned. - +.code name +may be optionally followed by one or more clauses, which are processed +in the order that they appear. Each clause is a compound form headed +by a keyword. +The supported clauses are as follows: +.RS +.meIP (:fallback << package-name *) The -.code setenv -function creates or modifies the environment variable indicated by -.metn name . +.code :fallback +clause specifies the packages to comprise the fallback list of +the present package. If this clause is omitted, or if it is present +with not +.meta package-name +arguments, then the present package has an empty fallback list. +Each +.meta package-name +may be a string or symbol naming an existing package. It is permitted +for the present package itself to appear in its own fallback list. +This is useful for creating a package with a non-empty fallback list +which doesn't actually provide access to any other package. +.meIP (:use << package-name *) The -.meta value -string argument specifies the new value for the variable. -If -.meta value -is -.codn nil , -then -.code setenv -behaves like -.codn unsetenv , -except that it observes the -.meta overwrite-p -argument. That is to say, the meaning of a null -.meta value -is that the variable is to be removed. - -If the -.meta overwrite-p -argument is specified, and is true, -then the variable is overwritten if it already exists. -If the argument is false, then the variable is not modified if it -already exists. If the argument is not specified, it defaults -to the value -.metn t , -effectively giving rise to a two-argument form of -.code setenv -which creates or overwrites environment variables. - -A variable removal is deemed to be an overwrite. -Thus if both -.meta value -and -.meta overwrite-p -are -.codn nil , -then -.code setenv -does nothing. +.code :use +clause specifies packages whose local symbols are to be interned +into the present package as foreign symbols. Each +.meta package-name +may be a string or symbol naming an existing package. +The list of package names is processed as if by a call to +.codn use-package . +.meIP (:use-syms << symbol *) +The +.code :use-syms +clause specifies individual symbols to be interned in the present package. +The arguments are symbols. +.meIP (:use-from < package-name << symbol-name *) +The +.code :use-from +clause specifies the names of local symbols in a package denoted by +.meta package-name +to be used in the present package. All arguments of +.code :use-from +are either strings or symbols which are reduced to strings by mapping +to their names. Each +.meta symbol-name +is interned in the package identified by +.metn package-name , +which may have the effect of creating that symbol. +This symbol is expected to be a local symbol of that package. If +that is so, the symbol is brought into the present package via +.codn use-symbol . +Otherwise if the symbol is foreign to package identified by +.metn package-name , +then an error exception is thrown. +.meIP (:local << symbol-name *) +The +.code :local +clause specifies the names of symbols to be interned in the new package +as local symbols. Each +.meta symbol-name +argument must be either a character string or a symbol. If it is a symbol, its +name is taken, thereby reducing the argument to a character string. +The arguments are processed in the order in which they appear. Each name is +first interned in the newly created package using the +.code intern +function. Then, if the resulting symbol is foreign to the package, it is +removed with +.code unuse-sym +and the name is interned again. +.RE +.coNP Macro @ in-package +.synb +.mets (in-package << name ) +.syne +.desc The -.code setenv -function unconditionally returns -.meta value -regardless of whether or not it overwrites or removes an existing variable. +.code in-package +macro causes the +.code *package* +special variable to take on the package denoted by +.metn name . +The macro checks, at expansion time, that +.meta name +is either a string or symbol. An error is thrown if +this isn't the case. The -.code unsetenv -function removes the environment variable -specified by -.metn name , -if it exists. On some platforms, it instead sets the environment variable -to the empty string. +.meta name +argument expression isn't evaluated, and so must not be quoted. -Note: supporting removal semantics in -.code setenv -allows for the following simple save/modify/restore pattern: +The code generated by the macro performs a search for the +package. If the package is not found at the time when +the macro's expansion is evaluated, an error is thrown. -.verb - (let* ((old-val (getenv "SOME-VAR"))) - (unwind-protect - (progn (setenv "SOME-VAR" new-val) - ...) - (setenv "SOME-VAR" old-val))) -.brev +.SS* Pseudo-random Numbers +.coNP Special variable @ *random-state* +.desc +The +.code *random-state* +variable holds an object which encapsulates the state +of a pseudo-random number generator. This variable is the default argument +value for the +.code random-fixnum +and +.codn "random functions" , +for the convenience of writing programs which are not concerned about the +management of random state. -This works in the case when -.code SOME-VAR -exists, as well as in the case that it doesn't exist. -In both cases, its previous value or, respectively, non-existence, -is restored by the -.code unwind-protect -cleanup form. +On the other hand, programs can create and manage random states, making it +possible to obtain repeatable sequences of pseudo-random numbers which do not +interfere with each other. For instance objects or modules in a program can +have their own independent streams of random numbers which are repeatable, +independently of other modules making calls to the random number functions. -.SS* Command Line Option Processing +When \*(TX starts up, the +.code *random-state* +variable is initialized with +a newly created random state object, which is produced as if by +the call +.codn "(make-random-state 42)" . -\*(TL provides a support for recognizing, extracting and validating -the POSIX-style options from a list of command-line arguments. +.coNP Special variable @ *random-warmup* +.desc +The +.code *random-warmup* +special variable specifies the value which is used by +.code make-random-state +in place of a missing +.meta warmup-period +argument. -The supported options can be defined as a list of option descriptor -objects each of which is constructed by a call to the -.code opt -function. Each option can have a long name, a short name, -a type, and a description. +To "warm up" a pseudo-random number generator (PRNG) means to obtain some +values from it which are discarded, prior to use. The number of values +discarded is the +.IR "warm-up period" . -The -.code getopts -function takes a list of option descriptors, and a list of arguments, -producing a parse, or else throwing an exception of type -.code opt-error -if an error is detected. The returned object, an instance of struct type -.codn opts , -can then be queried for specific option values, or for the remaining non-option -arguments. +The WELL PRNG used in \*(TX produces 32-bit values, natively. Thus each +warm-up iteration retrieves and discards a 32-bit value. The PRNG has +a state space consisting of a vector of sixteen 32-bit words, making +the state space 4096 bits wide. -The -.code opthelp -function takes a list of option descriptors and an output stream, -and generates help text on that stream. A program supporting a -.code --help -option can use this to generate that portion of its help text which -describes the available options, as well as the conventions that they use. +Warm up is required because PRNG-s, in particular PRNG-s with large state +spaces and long periods, produce fairly predictable sequences of values in the +beginning, before transitioning into chaotic behavior. This problem is worse +for low complexity seeds, such as small integer values. -.NP* Command Line Option Conventions +The sequences are predictable in two ways. Firstly, some initial values +extracted from the PRNG may exhibit patterns ("problem 1"). Secondly, the initial values +from sequences produced from similar seeds (for instance consecutive integers) +may be similar or identical ("problem 2"). -A command line option can have a short or long name. A short name is always -one-character long, and treated specially in the command line syntax. Long -options have names two or more characters long. An option can have both a long -and short name. Options may not begin with the -.code - -(ASCII dash) character. A long option may not contain the -.code = -character. +.TP* Notes: -Short options are invoked by specifying an argument with a single leading -.code - -followed by the option character. Multiple short options which take -no argument can be "clumped": combined into a single argument consisting of -a single -.code - -followed by multiple short option characters. +The default value of +.code *random-warmup* +is only 8. This is insufficient to +ensure good initial PRNG behavior for seeds even as large as 64 bits or more. +That is to say, even if as many as eight bytes' worth of true random bits are +used as the seed, the PRNG will exhibit predictable behaviors, and a poor +distribution of values. -An option can take an argument, in which case the argument is required. -An option which takes no argument is Boolean, and a Boolean option -never takes an argument: "takes no argument" and "Boolean" effectively -mean the same thing. +Applications which critically depend on good PRNG behavior should choose +large warm-up periods into the hundreds or thousands of iterations. +If a small warm-up period is used, it is recommended to use larger seeds +which initialize more of the 4096 bit state space. -Long options are invoked as an argument which begins with a -.code -- -(double dash) -immediately followed by the name. When a long option takes an argument, -it is mandatory. It must be specified in the same argument, separated -from the name by the -.code = -character. If that is omitted, then the next command line argument -is taken as the argument. That argument is removed, and not recognized as -an option, even if it looks like one. +\*(TX's PRNG implementation addresses "problem 1" first problem by padding the +unseeded portions of the state space with random values (from a static table +that doesn't change). For instance, if the integer 1 is used to seed the space, +then one 32 bit word of the space is set to the value 1. The remaining 15 are +populated from the random table. This helps to ensure that a good PRNG sequence +is obtained immediately. However, it doesn't address "problem 2": that +similar seed values generate similar sequences, when the warm-up period is +small. For instance, if 65536 different random state objects are created, from +each of the 16-bit seeds in the range [0, 65536), and then a random 16-bit +value is extracted from each state, only 1024 unique values result. -A Boolean long option can be explicitly specified as false using the -.code --no- -prefix rather than the -.code -- -prefix. +.coNP Function @ make-random-state +.synb +.mets (make-random-state >> [ seed <> [ warmup-period ]) +.syne +.desc +The +.code make-random-state +function creates and returns a new random state, +an object of the same kind as what is stored in the +.code *random-state* +variable. -Short options may be invoked using long name syntax; if -.code a -is a short option, then it may be referenced on the command line as -.code --a -and treated as a long option in all other ways, including the use -of -.code --no- -to explicitly specify false for a Boolean option. +The seed, if specified, must be either an integer value, an +existing random state object, or a vector returned from a call +to the function +.codn random-state-get-vec . -If a short option takes an argument, it may not clump with other -short option. The following command line argument is taken as the -options argument. That argument is removed and is not recognized as -an option even if it looks like one. +Note that the sign of the seed is ignored, so that negative seed +values are equivalent to their additive inverses. -If the command line argument -.code -- -occurs in the command line where an option would otherwise be recognized, -it signifies the end of the options. The subsequent arguments are the -non-option arguments, even if they resemble options. +If seed is not specified, then +.code make-random-state +produces a seed based +on some information in the process environment, such as current +time of day. It is not guaranteed that two calls to +.code (make-random-state) +that are separated by less than some minimum increment of real time produce +different seeds. The minimum time increment depends on the platform. -.NP* Command Line Processing Example +On a platform with a millisecond-resolution real-time clock, the minimum +time increment is a millisecond. Calls to make-random-state less than +a millisecond apart may predictably produce the same seed. -The following example illustrates a complete \*(TL program which -parses command line options: +If an integer seed is specified, then the integer value is mapped to a +pseudo-random sequence, in a platform-independent way. -.verb - (defvarl options - (list (opt "v" "verbose" :dec - "Verbosity level. Higher values produce more chatter.") - (opt nil "help" :bool - "List this help text.") - (opt "x" nil :hex - "The X factor: a number with a mysterious\e \e - interpretation, affecting the program\e \e - behavior in strange ways.") - (opt "z" nil) ;; undocumented option - (opt nil "cee" :cint - "C style integer.") - (opt "g" "gravity" :float - "Gravitational constant. This gives\e \e - the gravitational field\e \e - strength at the Earth's surface.") - (opt "l" "lit" :str - "A character string given in TXR Lisp notation.") - (opt "c" nil 'upcase-str - "Custom treatment: ARG is converted to upper case.") - (opt "b" "bool" :bool - "A flag you can flip true."))) +If an existing random state is specified as a seed, then it is duplicated. The +returned random state object is a distinct object which is in the same +state as the input object. It will produce the same remaining pseudo-random +number sequence, as will the input object. - (defvarl prog-name *load-path*) +If a vector is specified as a seed, then a random state is constructed +which duplicates the random state object which was captured in that vector +representation by the +.code random-state-get-vec +function. - (let ((o (getopts options *args*))) - (when [o "help"] - (put-line "Usage:\en") - (put-line ` @{prog-name} [options] arg*`) - (opthelp options) - (exit 0)) - (put-line `args after opts are: @{o.out-args ", "}`)) -.brev +The +.meta warm-up-period +argument specifies the number of values which are immediately obtained and +discarded from the newly-seeded generator before it is returned. +Warm-up is not performed when +.meta seed +is an existing random state object, and this argument is ignored in that +case. If the parameter is required, but the argument is missing, then +the value of the +.code *random-warmup* +special variable is used. This variable has a default value which may be too +small for serious applications of pseudo-random numbers; see the Notes under +.codn *random-warmup* . -.coNP Structure @ opt-desc +.coNP Function @ random-state-p .synb -.mets (defstruct opt-desc -.mets \ \ short long helptext type -.mets \ \ ... < unspecified << slots ) +.mets (random-state-p << obj ) .syne .desc The -.code opt-desc -structure describes a single command line option. +.code random-state-p +function returns +.code t +if +.meta obj +is a random state, otherwise it +returns +.codn nil . +.coNP Function @ random-state-get-vec +.synb +.mets (random-state-get-vec <> [ random-state ]) +.syne +.desc The -.code short -and -.code long -slots are either -.code nil -or else hold strings. -The -.code short -slot gives the option's short name: a one-character-long -string which may not be the ASCII dash character -.codn - . -The -.code long -slot gives the option's long name: a string two or more -characters long which doesn't begin with a dash. -An option must have at least one of these names. +.code random-state-get-vec +function converts a random state into a vector of integer values. +If the +.meta random-state +argument, which must be a random state object, is omitted, +then the value of the +.code *random-state* +is used. -The -.code helptext -slot provides a descriptive string. This string may be long. The -.code opthelp -function displays this text, formatting into multiple lines as necessary. -If -.code helptext -is -.codn nil , -the option is considered undocumented. +.coNP Functions @, random-fixnum @ random and @ rand +.synb +.mets (random-fixnum <> [ random-state ]) +.mets (random < random-state << modulus ) +.mets (rand < modulus <> [ random-state ]) +.syne +.desc +All three functions produce pseudo-random numbers, which are positive integers. -The -.code type -slot may be a symbol naming a global function which takes one argument, -or it may be such a function object. Otherwise it must be one of the -following keyword symbols: -.RS -.coIP :bool -This indicates that the type of the option is Boolean. Such -an option doesn't take any argument. Its value is -.code t -or -.codn nil . -.coIP :dec -This indicates that the option requires an argument, which is a -decimal integer with an optional positive or negative sign. -This argument is converted to an integer object. -.coIP :hex -This type indicates that the option requires an argument consisting -of a hexadecimal integer with an optional positive or negative sign. -This is converted to an integer object. -.coIP :oct -This type indicates that the option requires an argument consisting -of a octal integer with an optional positive or negative sign. -This is converted to an integer object. -.coIP :cint -This type indicates that the option requires an integer argument -whose format conforms to one of three C language conventions in most respects, -other than that this integer may have an arbitrary range. -All forms may carry an optional positive or negative leading sign -at the very beginning. -The first convention consists of decimal digits, which must not have -a superfluous leading zero. The second convention consists of octal -digits which are introduced by an extra leading zero. -The third convention consists of hexadecimal digits introduced by the -.code 0x -prefix. -.coIP :float -This type indicates a decimal floating-point argument, which is converted to -a floating-point number. Its basic form is: an optional leading plus or -minus sign, followed by a sequence of one or more digits which may contain -a single decimal point anywhere, including the very beginning of the -sequence or at the end, optionally followed by the letter -.code e -or -.code E -followed by a decimal integer which may have a leading positive or negative -sign, and include leading zeros. -.coIP :text -This type indicates a simple textual argument. The argument is taken as -verbatim UTF-8 text, converted to a string without interpreting -the characters in any special way. -.coIP :str -This type indicates that the argument consists of the interior notation of -a TXR Lisp character string. It is processed by adding a double quote -at the beginning or end, and parsed as a string literal. This parsing must -successfully yield a string object, otherwise the argument is ill-formed. -.meIP (list << type ) -If the type is specified as a compound form headed by the -.code list -symbol, it indicates that the command line option's argument is a list -of elements. The argument appears on the command line as a single string -contained within one argument. It may contain commas, and is split into pieces -using the comma character as a separator. The pieces are then individually -treated as of type -.meta type -and converted accordingly. The option's argument is then a list object -whose elements are the converted pieces. For instance -.code "(list :dec)" -will convert a list of comma-separated decimal integer tokens into -a list of integer objects. -.RE +The numbers are obtained from a WELL 512 PRNG, whose state is stored in the +random state object. -.IP -If -.code type -is a function, then the option requires an argument. The argument string -is passed to the function, and the value is whatever the function returns. +The +.code random-fixnum +function produces a random fixnum integer: a reduced range +integer which fits into a value that does not have to be heap-allocated. The -.code opt-desc -structure may have additional slots which are not specified. +.code random +and +.code rand +functions produce a value in the range [0, +.metn modulus ). +They differ only in the order of arguments. In the +.code rand +function, the random state +object is the second argument and is optional. If it is omitted, the global +.code *random-state* +object is used. The -.code opt -convenience function is provided for constructing -.code opt-desc -objects. +.meta modulus +argument must be a positive integer. If +.meta modulus +is 1, then the function returns zero without altering the state of the +pseudo-random number generator. -.coNP Function @ opt +.coNP Function @ random-float .synb -.mets (opt < short < long >> [ type <> [ helptext ]]) +.mets (random-float <> [ random-state ]) .syne .desc The -.code opt -function provides a slightly condensed syntax for constructing -an object of type -.codn opt-desc . - -The required arguments -.meta short -and -.meta long -are strings, corresponding to -.code opt-desc -slots of the same name. - -The optional parameter -.meta type -corresponds to the same-named slot and defaults to -.codn :bool . +.code random-float +function produces a pseudo-random floating-point value in the range [0.0, 1.0). -The optional parameter -.meta helptext -corresponds to the same-named slot, and defaults to -.code nil -(no help text provided for the option). +The numbers are obtained from a WELL 512 PRNG, whose state is stored in the +random state object given by the argument to the optional +.meta random-state +parameter, which defaults to the value of +.codn *random-state* . +.SS* Time +.coNP Functions @ time and @ time-usec +.synb +.mets (time) +.mets (time-usec) +.syne +.desc The -.code opt -function follows this equivalence: +.code time +function returns the number of seconds that have elapsed since +midnight, January 1, 1970, in the UTC timezone: a point in +time called +.IR "the epoch" . -.verb - (opt a b c d) <--> (new opt-desc short a long b - type c helptext d) -.brev +The +.code time-usec +function returns a cons cell whose +.code car +field holds the seconds measured in the same way, and whose +.code cdr +field extends the precision by giving +number of microseconds as an integer value between 0 and 999999. -.coNP Structure @ opts +.coNP Functions @ time-string-local and @ time-string-utc .synb -.mets (defstruct opts nil -.mets \ \ in-args out-args -.mets \ \ ... < unspecified << slots ) +.mets (time-string-local < time << format ) +.mets (time-string-utc < time << format ) .syne .desc -The -.code opts -structure represents a parsed command line, containing decoded -information obtained from the options, and an indication where -the non-option arguments start. +These functions take the numeric time returned by the +.code time +function, and convert it to a textual representation in a flexible way, +according to the contents of the +.meta format +string. The -.code opts -structure supports direct indexing for option retrieval. -That is the only documented interface for accessing the parsed -options; the implementation of the information set describing -the parsed options is unspecified. +.code time-string-local +function converts the time to the local timezone of +the host system. The +.code time-string-utc +function produces time in UTC. The -.code in-args -slot holds the original argument list. +.meta format +argument is a string, and follows exactly the same conventions as +the format string of the C library function +.codn strftime . The -.code out-args -slot holds the tail of the argument list consisting of the non-option -arguments. - -The mechanism by means of which -.code out-args -is calculated, and by means of which the information about the -options is populated, is unspecified. The only interface to that -mechanism is the -.code getopts +.meta time +argument is an integer representing seconds obtained from the +time function or from the +.code car +field of the cons returned by the +.code time-usec function. -The -.code opts -object supports indexing, including indexed assignment. - -If -.code o -is an instance of -.code opts -returned by -.codn getopts , -then the expression -.code "[o \(dqv\(dq]" -tests whether the option -.str v -is available in -.codn o ; -that is, whether it has been specified in the command line. -If so, then its associated value is returned, otherwise -.code nil -is returned. This -.code nil -is ambiguous: for a Boolean option it indicates that either -the option was not specified, or that it was explicitly -specified as false. For a Boolean option that was specified -(positively), the value -.code t -is returned. - -The expression -.code "[o \(dqv\(dq dfl]" -yields the value of option -.str v -if that option has been specified. If the option hasn't -been specified, then the expression yields the value -.codn dfl . - -Assigning to -.code "[o \(dqv\(dq]" -is possible. This replaces the value associated with option -.strn v . -The assignment is erroneous if no such option was parsed -from the command line, even if it is a valid option. - -.coNP Function @ getopts +.coNP Functions @ time-fields-local and @ time-fields-utc .synb -.mets (getopts < option-desc-list << arg-list ) +.mets (time-fields-local << time ) +.mets (time-fields-utc << time ) .syne .desc -The -.code getopts -function takes a list of -.code opt-desc -structures and a list of strings -.meta arg-list -representing command line arguments. +These functions take the numeric time returned by the time function, +and convert it to a list of seven fields. The -.meta arg-list -is parsed. If the parse is unsuccessful, an exception of type -.code opt-error -is thrown, derived from -.codn error . +.code time-string-local +function converts the time to the local timezone of +the host system. The +.code time-string-utc +function produces time in UTC. -If there are problems in -.code option-desc-list -itself, then an exception of type -.code error -is thrown. +The fields returned as a list consist of six integers, and a Boolean value. +The six integers represent the year, month, day, hour, minute and second. +The Boolean value indicates whether daylight savings time is in effect +(always +.code nil +in the case of +.codn time-fields-utc ). -If the parse is successful, -.code getopts -returns an instance of the -.code opts -structure describing the parsed opts, and listing the non-option -arguments. +The +.meta time +argument is an integer representing seconds obtained from the +.code time +function or from the +.code time-usec +function. -.coNP Function @ opthelp +.coNP Structure @ time .synb -.mets (opthelp < opt-desc-list <> [ stream ]) +.mets (defstruct time nil +.mets \ \ year month day hour min sec dst +.mets \ \ gmtoff zone) .syne .desc The -.code opthelp -function processes the list of -.code opt-desc -structures -.meta opt-desc-list -and compiles a customized body of help text describing all of the -options, as well as general description of the command line option -conventions to guide the user in in the correct use of command -line options. - -The text is formatted to fit within 79 columns, and begins and ends with a -blank line. Its format consists of headings which begin in the first column, -and paragraphs and tables which feature a two space left margin. -A blank line follows each section heading. The heading begins with a capital -letter. Its remaining words are uncapitalized, and it ends with a colon. - -The text is sent to -.metn stream , -if specified. This argument defaults to -.codn *stdout* . +.code time +structure represents a time broken down into individual fields. +The structure almost directly corresponds to the +.code "struct tm" +type in the ISO C language. There are differences. +Whereas the +.code "struct tm" +member +.code tm_year +represents a year since 1900, the +.code year +slot of the +.code time +structure represents the absolute year, not relative to 1900. +Furthermore, the +.code month +slot represents a one-based numeric month, such that 1 represents +January, whereas the C member +.code tm_mon +uses a zero-based month. The +.code dst +slot is a \*(TL Boolean value. The slots +.codn hour , +.codn min , +and +.code sec +correspond directly to +.codn tm_hour , +.codn tm_min , +and +.codn tm_sec . -If there are problems in -.code option-desc-list -itself, then an exception of type -.code error -is thrown. +The slot +.code gmtoff +represents the number of seconds east of UTC, and +.code zone +holds a string giving the abbreviated time zone name. +On platform where the C type +.code "struct tm" +has fields corresponding to these slots, values for +these slots are calculated and stored into them by the +.code time-struct-local +and +.code time-struct-utc +functions, and also the related +.code time-local +and +.code time-utc +methods. On platform where the corresponding fields are not +present in the C language +.codn "struct tm" , +these slots are unaffected by those functions, +retaining the default initial value +.code nil +or a previously stored value, if applicable. +Lastly, the values of +.code gmtoff +and +.code zone +are not ignored by functions which accept a +.code time +structure as a source of input values. -.SS* System Programming -.coNP Accessor @ errno +.coNP Functions @ time-struct-local and @ time-struct-utc .synb -.mets (errno <> [ new-errno ]) -.mets (set (errno) << new-value ) +.mets (time-struct-local << time ) +.mets (time-struct-utc << time ) .syne .desc +These functions take the numeric time returned by the time function, +and convert it to an instance of the +.code time +structure. + The -.code errno -function retrieves the current value of the C library error variable -.codn errno . -If the argument -.meta new-errno -is present and is not -.codn nil , -then it -specifies a value which is stored into -.codn errno . -The value returned is the prior value. +.code time-struct-local +function converts the time to the local timezone of +the host system. The +.code time-struct-utc +function produces time in UTC. -The place form of -.code errno -does not take an argument. +The +.meta time +argument is an integer representing seconds obtained from the +.code time +function or from the +.code time-usec +function. -.coNP Function @ exit +.coNP Functions @, time-parse @ time-parse-local and @ time-parse-utc .synb -.mets (exit << status ) +.mets (time-parse < format << string ) +.mets (time-parse-local < format << string ) +.mets (time-parse-utc < format << string ) .syne .desc The -.code exit -function terminates the entire process (running \*(TX image), specifying -the termination status to the operating system. Values of -.meta status -may be -.codn nil , -.codn t , -or an integer value. The value -.code nil -corresponds to the C constant -.codn EXIT_FAILURE , -and -.code t -corresponds to -.codn EXIT_SUCCESS . -These are platform-independent -indicators of failed or successful termination. The numeric value 0 also -indicates success. +.code time-parse +function scans a time description in +.meta string +according to the specification given in the +.meta format +string. If the scan is successful, a structure +of type +.code time +is returned, otherwise +.codn nil . -.coNP Variables @, e2big @, eacces @, eaddrinuse @, eaddrnotavail @, eafnosupport @, eagain @, ealready @, ebadf @, ebadmsg @, ebusy @, ecanceled @, echild @, econnaborted @, econnrefused @, econnreset @, edeadlk @, edestaddrreq @, edom @, edquot @, eexist @, efault @, efbig @, ehostunreach @, eidrm @, eilseq @, einprogress @, eintr @, einval @, eio @, eisconn @, eisdir @, eloop @, emfile @, emlink @, emsgsize @, emultihop @, enametoolong @, enetdown @, enetreset @, enetunreach @, enfile @, enobufs @, enodata @, enodev @, enoent @, enoexec @, enolck @, enolink @, enomem @, enomsg @, enoprotoopt @, enospc @, enosr @, enostr @, enosys @, enotconn @, enotdir @, enotempty @, enotrecoverable @, enotsock @, enotsup @, enotty @, enxio @, eopnotsupp @, eoverflow @, eownerdead @, eperm @, epipe @, eproto @, eprotonosupport @, eprototype @, erange @, erofs @, espipe @, esrch @, estale @, etime @, etimedout @, etxtbsy @ ewouldblock and @ exdev -.desc -These variables correspond to the POSIX -.cod2 \(dq errno -constants\(dq, namely -.codn E2BIG , -.codn EACCES , -.code EADDRINUSE -and so forth. -Variables corresponding to all of the -.code "" -constants from the Issue 6 2004 edition of POSIX are included. -The variables -.code eownerdead +The +.meta format +argument follows the same conventions as the POSIX +C library function +.codn strptime . + +Prior to obtaining the time from +.meta format and -.code enotrecoverable -from Issue 7 2018 are subject to the availability of the corresponding constants -in the host platform. +.meta string +the returned structure is created and initialized +with a time which represents time 0 ("the epoch") +if interpreted in the UTC timezone as by the +.meta time-utc +method. -.coNP Function @ abort -.synb -.mets (abort) -.syne -.desc The -.code abort -function terminates the entire process (running \*(TX image), specifying -an abnormal termination status to the process. +.code time-parse-local +and +.code time-parse-utc +functions return an integer time value: the same value +that would be returned by the +.code time-local +and +.code time-utc +methods, respectively, when applied to the structure +object returned by +.codn time-parse . +Thus, these equivalences hold: -Note: -.code abort -calls the C library function -.code abort -which works by raising the -.code SIG_ABRT -signal, known in \*(TX as the -.code sig-abrt -variable. Abnormal termination of the process is this signal's -default action. +.verb + (time-parse-local f s) <--> (time-parse f s).(time-local) + (time-parse-utc f s) <--> (time-parse f s).(time-utc) +.brev -.coNP Functions @ at-exit-call and @ at-exit-do-not-call +Note: the availability of these three functions +depends on the availability of +.codn strptime . + +.coNP Methods @ time-local and @ time-utc .synb -.mets (at-exit-call << function ) -.mets (at-exit-do-not-call << function ) +.mets << time-struct .(time-local) +.mets << time-struct .(time-utc) .syne .desc The -.code at-exit-call -function registers -.meta function -to be called when the process terminates normally. -Multiple functions can be registered, and the same function -can be registered more than once. The registered -functions are called in reverse order of their -registrations. +.code time +structure has two methods called +.code time-local +and +.codn time-utc . The -.code at-exit-do-not-call -function removes all previous -.code at-exit-call -registrations of -.metn function . +.code time-local +function considers the slots of the +.code time +structure instance +.meta time-struct +to be local time, and returns its integer representation +as the number of seconds since the epoch. The -.code at-exit-call -function returns -.metn function . +.code time-utc +function is similar, except it considers +the slots of +.meta time-struct +to be in the UTC time zone. -The -.code at-exit-do-not-call -function returns -.code t -if it removed anything, -.code nil -if no registrations of -.meta function -were found. +Note: these functions work by converting the slots into arguments +which are applied to +.code make-time +or +.codn make-time-utc . -.coNP Function @ usleep +.coNP Method @ time-string .synb -.mets (usleep << usec ) +.mets << time-struct .(time-string << format ) .syne .desc The -.code usleep -function suspends the execution of the program for at least -.meta usec -microseconds. +.code time +structure has a method called +.codn time-string . -The return value is -.code t -if the sleep was successfully executed. A -.code nil -value indicates premature wakeup or complete failure. +This method accepts a +.meta format +string argument, which it uses to convert +the fields to a character string representation +which is returned. -Note: the actual sleep resolution is not guaranteed, and depends on granularity -of the system timer. Actual sleep times may be rounded up to the nearest 10 -millisecond multiple on a system where timed suspensions are triggered by a 100 -Hz tick. +The +.meta format +argument is a string, and follows exactly the same conventions as +the format string of the C library function +.codn strftime . -.coNP Functions @ mkdir and @ ensure-dir +.coNP Method @ time-parse .synb -.mets (mkdir < path <> [ mode ]) -.mets (ensure-dir < path <> [ mode ]) +.mets << time-struct .(time-parse < format << string ) .syne .desc -.code mkdir -tries to create the directory named -.meta path -using the POSIX -.code mkdir -function. -An exception of type -.code file-error -is thrown if the function fails. Returns -.code t -on success. - The -.meta mode -argument specifies the request numeric permissions -for the newly created directory. If omitted, the requested permissions are -.code #o777 -(511): readable and writable to everyone. The requested permissions -are subject to the system -.codn umask . - -The function -.code ensure-dir -is similar to -.code mkdir -except that it attempts to create all the missing parent directories -as necessary, and does not throw an error if the directory exists. +.code time-parse +method scans a time description in +.meta string +according to the specification given in the +.meta format +string. -.coNP Function @ chdir -.synb -.mets (chdir << path ) -.syne -.desc -.code chdir -changes the current working directory to -.metn path , -and returns -.metn t , -or else throws an exception of type -.codn file-error . +If the scan is successful, the structure +is updated with the parsed information, and +the remaining unmatched portion of +.meta string +is returned. If all of +.meta string +is matched, then an empty string is returned. +Slots of +.meta time-struct +which are originally +.code nil +are replaced with zero, even if these +zero values are not actually parsed from +.metn string . -.coNP Function @ pwd -.synb -.mets (pwd) -.syne -.desc -The -.code pwd -function retrieves the current working directory. -If the underlying -.code getcwd -C library function fails with an -.code errno -other than -.codn ERANGE , -an exception will be thrown. +If the scan is unsuccessful, then +.code nil +is returned and the structure is not +altered. -.coNP Function @ remove-path -.synb -.mets (remove-path < path <> [ throw-on-error-p ]) -.syne -.desc The -.code remove-path -function tries to remove the filesystem object named -by -.metn path , -which may be a file, directory or something else. - -If successful, it returns -.codn t . +.meta format +argument follows the same conventions as the POSIX +C library function +.codn strptime . -The optional Boolean parameter -.metn throw-on-error-p , -which defaults to +Note: the +.code time-parse +method may be unavailable if the host system does not +provide the +.code strptime +function. In this case, the +.code time-parse +static slot of the +.code time +struct is .codn nil . -A failure to remove the object results in an exception of type -.code file-error -being thrown, unless the failure reason is that the object indicated by -.meta path -doesn't exist. In this non-existence case, the behavior is controlled by the -.meta throw-on-error -argument. If that argument is true, the exception is thrown. Otherwise, -the function returns normally, producing the value -.code nil -to indicate that it didn't perform a removal. - -.coNP Function @ rename-path +.coNP Functions @ make-time and @ make-time-utc .synb -.mets (rename-path < from-path << to-path ) +.mets (make-time < year < month < day +.mets \ \ \ \ \ \ \ \ \ \ < hour < minute < second << dst-advice ) +.mets (make-time-utc < year < month < day +.mets \ \ \ \ \ \ \ \ \ \ \ \ \ \ < hour < minute < second << dst-advice ) .syne .desc The -.code remove-path -function tries to rename filesystem path -.metn from-path , -which may refer to a file, directory or something else, to the path -.metn to-path . +.code make-time +function returns a time value, similar to the one returned by the +.code time +function. The +.code time +value is constructed not from the system clock, but +from a date and time specified as arguments. The +.meta year +argument is a calendar year, like 2014. +The +.meta month +argument ranges from 1 to 12. +The +.meta hour +argument is a 24-hour time, ranging from 0 to 23. +These arguments represent a local time, in the current time zone. -If successful, it returns +The +.meta dst-advice +argument specifies whether the time is expressed in +daylight savings time (DST). It takes on three possible values: +.codn nil , +the keyword +.codn :auto , +or else the symbol +.codn t . +Any other value has the same interpretation as .codn t . -A failure results in an exception of type -.codn file-error . +If +.meta dst-advice +is +.codn t , +then the time is assumed to be expressed in DST. +If the argument is +.codn nil , +then the time is assumed not to be in DST. +If +.meta dst-advice +is +.codn :auto , +then the function tries to determine whether +DST is in effect in the current time zone for the specified date and time. -.coNP Functions @ sh and @ run -.synb -.mets (sh << system-command ) -.mets (run < program <> [ argument-list ]) -.syne -.desc The -.code sh -function executes -.meta system-command -using the system command interpreter. -The run function spawns a -.metn program , -searching for it using the -system PATH. Using either method, the executed process receives environment -variables from the parent. - -\*(TX blocks until the process finishes executing. If the program terminates -normally, then its integer exit status is returned. The value zero indicates -successful termination. - -The return value -.code nil -indicates an abnormal termination, or the inability -to run the process at all. +.code make-time-utc +function is similar to +.codn make-time , +except that +it treats the time as UTC rather than in the local time zone. +The +.meta dst-advice +argument is supported by +.code make-time-utc +for function +call compatibility with +.codn make-time . +It may or may not have any effect +on the output (since the UTC zone by definition doesn't have daylight +savings time). -In the case of the -.code run -function, if the child process is created successfully -but the program cannot be executed, then the exit status will be an -.code errno -value from the failed -.code exec -attempt. +.SS* The Awk Utility -The standard input, output and error file descriptors of an executed -command are obtained from the streams stored in the -.codn *stdin* , -.code *stdout* -and -.code *stderr* -special variables, respectively. For a detailed description of the -behavior and restrictions, see the -.code open-command -function, whose description of this mechanism applies to the -.code run -and -.code sh -function also. +The \*(TL library provides a macro called +.code awk +which is inspired by the Unix utility Awk. The macro implements +a processing paradigm very similar to that of the utility: it scans +one or more input streams, which are divided into records or fields, +under the control of user-settable regular-expression-based delimiters. +The records and fields are matched against a sequence of programmer-defined +conditions (called "patterns" in the original Awk), which have associated +actions. Like in Awk, the default action is to print the current record. -Note: as of \*(TX 120, the -.code sh -function is implemented using -.code run -and not by means of the -.code system -C library function, as previously. The -.code run -function is used to invoke the system interpreter by name. On Unix-like -systems, the string -.code /bin/sh -is assumed to denote the system interpreter, which is expected to -support a pair of arguments -.mono -.meti -c < command -.onom -to specify the command to be executed. On MS Windows, the interpreter -is assumed to be the relative path name -.code cmd.exe -and expected to support -.mono -.meti /C < command -.onom -as a way of specifying a command to execute. +Unlike Awk, the +.code awk +macro is a robust, self-contained language feature which can be used +anywhere where a \*(TL expression is called for, cleanly nests +with itself and can produce a return value when done. By contrast, +a function in the Awk language, or an action body, cannot instantiate +a local Awk processing machine. -.SS* Unix Filesystem Manipulation +The +.code awk +macro implements some of the most important Awk +conventions and semantics, in Lisp syntax, while eschewing others. +It does not implement implement the Awk convention that +variables become defined upon first mention; variables must be +defined to be used. It doesn't implement Awk's weak type system. +A character string which looks like a number isn't a number, +and an empty string or undefined variable doesn't serve as zero +in arithmetic expressions enclosed in the macro. +All expression evaluation within +.code awk +is the usual \*(TL evaluation. -.coNP Structure @ stat -.synb -.mets (defstruct stat nil -.mets \ \ dev ino mod nlink uid gid -.mets \ \ rdev size blksize blocks atime -.mets \ \ mtime ctime path) -.syne -.desc The -.code stat -structure defines the type of object which is returned -by the -.codn stat , -.codn lstat , +.code awk +macro also does not provide a library of functions corresponding to +those in the Awk library, nor does it provide counterparts various +global variables in Awk such as the +.code ENVIRON and -.code fstat -functions. Except for -.codn path , -the slots are the direct counterparts of the -members of POSIX C structure -.codn "struct stat" . -For instance the slot -.code dev -corresponds to -.codn st_dev . -The -.code path -slot is set by the functions -.code stat +.code PROCINFO +arrays, or +.code RSTART and -.codn lstat . -Its value is -.code nil -when the path is not available. +.codn RLENGTH . +Such features of Awk are extraneous to its central paradigm. -.coNP Functions @, stat @ lstat and @ fstat +.coNP Macro @ awk .synb -.mets (stat << path ) -.mets (lstat << path ) -.mets (fstat << stream ) +.mets (awk >> {( condition << action *)}*) .syne .desc The -.code stat -function inquires the filesystem about the existence of an object -denoted by the string -.metn path . -If the object is not found or cannot be -accessed, an exception is thrown. +.code awk +macro processes one or more input sources, which may be streams or +files. Each input source is scanned into records, and each record +is broken into fields. For each record, the sequence of condition-action +clauses (except for certain special clauses) is processed. Every +.meta condition +is evaluated, and if it yields true, the corresponding +.metn action -s +are evaluated. -Otherwise, information is retrieved about the object. The information takes -the form of a structure of type -.codn stat . +The +.meta condition +and +.meta action +forms are understood to be in a scope in which certain local +identifiers exist in the variable namespace as well as in the function +namespace. These are called +.I "awk functions" +and +.IR "awk macros" . If -.meta path -refers to a symbolic link, the -.code stat -function retrieves information about the target of the link, if it exists, -or else throws an exception of type -.codn file-error . +.meta condition +is one of the following keyword symbols, then it is a special clause, +with special semantics: +.codn :name , +.codn :let , +.codn :inputs , +.codn :output , +.codn :begin , +.codn :set , +.codn :end , +.codn :begin-file , +.code :set-file +and +.codn :end-file . +These clause types are explained below. +In such a clause, the +.meta action +expressions are not necessarily forms to be evaluated; the treatment +of these expressions depends on the clause. Otherwise, if +.meta condition +is not one of the above keyword symbols, the clause is an ordinary +condition-action clause, and +.meta condition +is a \*(TL expression, evaluated to determine a Boolean value +which controls whether the +.meta action +forms are evaluated. In every ordinary condition-action clause which +contains no +.meta action +forms, the +.code awk +macro substitutes the single action equivalent to the form +.codn "(prn)" : +a call to the local awk function +.codn prn . +The behavior of this macro, when called with no arguments, as above, +is to print the current +record (contents of the variable +.codn rec ) +followed by the output record terminator from the variable +.codn ors . -The -.code lstat -function behaves the same as -.code stat -or objects which are not symbolic links. For a symbolic link, it retrieves -information about the link itself, rather than its target. +While the processing loop in +.code awk +scans an input source, it also binds the special variable +.code *stdin* +to the open stream associated with that source. This binding is +in effect across all ordinary clauses, as well as across the +special clauses +.code :begin-file +and +.codn :end-file . +The following is a description of the special clauses: +.RS +.meIP (:name << sym ) The -.code fstat -function retrieves information about the file system object associated with -the open stream -.metn stream . -The stream must be of a kind from which the -.code fileno -function can retrieve a file descriptor, otherwise an exception of type -.code file-error -is thrown. +.code :name +clause establishes the name of the implicit block contained +within the expansion of the +.code awk +macro. Forms enclosed in the macro can use +.code return-from +to abandon the +.code awk +form, specifying this symbol as the argument. + +If the +.code :name +form is omitted, the implicit block is named +.codn awk . + +It is an error for two or more +.code :name +forms to appear. The -.code path -slot of the returned structure -holds a copy of their -.meta path -argument value. -In the case of -.codn fstat , -this slot is -.codn nil . +.code :name +clause must have an argument which is a symbol; +the symbol +.code nil +is not permitted. -.coNP Variables @, s-ifmt @, s-iflnk @, s-ifreg @, s-ifblk ... , @ s-ixoth -.desc -The following variables exist, having integer values. These are bitmasks -which can be applied against the value given by the -.code mode -slot of the -.code stat -structure returned by the function -.codn stat : -.codn s-ifmt , -.codn s-ifsock , -.codn s-iflnk , -.codn s-ifreg , -.codn s-ifblk , -.codn s-ifdir , -.codn s-ifchr , -.codn s-ififo , -.codn s-isuid , -.codn s-isgid , -.codn s-isvtx , -.codn s-irwxu , -.codn s-irusr , -.codn s-iwusr , -.codn s-ixusr , -.codn s-irwxg , -.codn s-irgrp , -.codn s-iwgrp , -.codn s-ixgrp , -.codn s-irwxo , -.codn s-iroth , -.code s-iwoth +.meIP (:let >> { sym | >> ( sym << init-form )}*) +Regardless of what order they appear in relation to +other clauses in the same +.code awk +macro, +.code :let +clauses are evaluated first before the macro takes any other action. The +argument forms of this clause are variables or variable-init forms. They are +treated the same way as analogous forms in the +.code let* +special form. Note that these are not enclosed in an extra list +as they are in the that form. The bindings established by the +.code :let +clause have a scope which extends over all the other clauses in the +.code awk +macro. + +If multiple +.code :let +clauses are present, they are effectively consolidated into +a single clause, in the order they appear. + +Note that the lexical variables, functions and macros established by the +.code awk +macro +(called, respectively, +.IR "awk macros" , +.I "awk functions" and -.codn s-ixoth . +.IR "awk variables" ) +are in an inner scope relative to +.code :let +bindings. For instance if +.code :let +creates a binding for a variable called +.codn fs , +that variable will be visible only to subsequent forms appearing +in the same +.code :let +clause or later +.code :let +clauses, and also visible in +.code :inputs +and +.code :output +clauses. +In +.codn :begin , +.codn :set , +.codn :end , +and ordinary clauses, it will be shadowed by the +.code awk +variable +.codn fs , +which holds the field separator regular expression or string. +.meIP (:inputs << source-form *) +The +.code :inputs +clause is evaluated by the +.code awk +macro after processing the +.code :let +clauses. Each +.meta source-form +is evaluated and the values of these forms are gathered into a list. +This list then comprises the list of input sources for the +.code awk +processing task. -These variables correspond to the C language constants from POSIX: -.codn S_IFMT , -.codn S_IFLNK , -.code S_IFREG -and so forth. +Each input source must be one of three kinds of objects. +It may be a stream object, which must be capable of character +input. It may be a list of strings, which +.code awk +will convert to an input stream as if by the +.code make-strlist-input-stream +function. +Or else it must be a character +string, which denotes a filesystem path name which +.code awk +will open for reading. + +If the +.code :inputs +clause is omitted, then a defaulting behavior occurs for obtaining +the list of input sources. If the special variable +.code *args* +isn't the empty list, then +.code *args* +is taken as the input sources. Otherwise, the +.code *stdin* +stream is taken as the one and only input source. +If the +.code awk +macro uses +.code *args* +via the above defaulting behavior, it copies +.code *args* +and sets that variable to +.codn nil . +This is done in order that if +.code awk +is used from the \*(TX command line, for example using the +.code -e +command line option, after +.code awk +terminates, \*(TX will not try to open the next argument +as a script file or treat it as an option. +Note: programs which want +.code awk +not to modify +.code *args* +can explicitly specify +.code *args* +as the argument to the +.code :inputs +keyword, rather than allow +.code *args* +to be used through the defaulting behavior. Only the +defaulting behavior consumes the arguments by overwriting +.code *args* +with +.codn nil . + +It is an error to specify more than one +.code :inputs +clause. +.meIP (:output << output-form ) The -.code logtest -function can be used to test these against values of mode. -For example -.code "(logtest mode s-irgrp)" -tests for the group read permission. +.code :output +clause is processed just after the +.code :inputs +clause. It must have exactly one argument, which is an expression +that evaluates to a string, or else to an output stream. +If it evaluates to a string, then that string is used as the name +of a file to open for writing, and the resulting stream +is taken in place of that string. -.coNP Function @ umask -.synb -.mets (umask <> [ mask ]) -.syne -.desc The -.code umask -function provides access to the Unix C library function of the same name, -which controls which permissions are denied -when files are newly created. +.code :output +clause, if present, has the effect of creating a local binding for the +.code *stdout* +special variable. +This new value of +.code *stdout* +is visible to all forms within the macro. +If a +.code :let +clause is present, it establishes bindings +in a scope which is nested within the scope established +by +.codn :output . +Therefore, +.metn init-form -s +in the +.code :let +may refer to the new value of +.code *stdout* +established by +.codn :output . +Furthermore, +.code :let +can rebind +.codn *stdout* , +causing the definition provided by +.code :output +to be shadowed. -If -.code umask -is called with no argument, it returns the current value of the mask. +In the case when the +.code :output +argument is a string such that a new stream is opened +on the file, the +.code awk +macro will close that stream when it finishes executing. +Moreover, that stream is treated uniformly as a member of +the set of streams that are implicitly managed by the +redirection macros in the same +.code awk +macro invocation. In brief, the implication is that if +.code :output +creates a stream for the file path name +.str "out.txt" +and somewhere in the same +.code awk +macro, there is a redirection of the form, or equivalent to +.mono +(-> "out.txt") +.onom +then this redirection shall refer to the same stream +that was established by +.codn :output . +Note also that in this example situation, the expression +.mono +(-> "out.txt" :close) +.onom +has the effect of closing the +.code :output +stream. -If the -.meta mask -argument is present, it must be an integer specifying the new mask to be -installed. The previous mask is returned. +.meIP (:begin << form *) +All +.code :begin +clauses are processed in the order in which they appear, before +input processing begins. +Each +.code form +is evaluated. These forms have in their scope the awk local variables +and macros. +.meIP (:set >> { place << new-value }*) +The +.code :set +clause provides a shorthand which allows the frequently occurring pattern +.code "(:begin (set ...))" +to be condensed to +.codn "(:set ...)" . +.meIP (:end << form *) +All +.code :end +clauses are processed, in the order in which they appear, +when the input processing loop terminates. +This termination occurs when all records +from all input sources are either processed or skipped, or else +by an explicit termination such +as a dynamic non-local transfer, such as +.codn return-from , +or the throwing of an exception. -If -.meta mask -is absent, then -.code umask -returns the previous mask. +Upon termination, the end clauses are processed in the order they appear. Each +.code form +is evaluated, left to right. -Note: the value of the -.meta mask -argument may be calculated as a bitwise or of the following constants: -.codn s-irwxu , -.codn s-irusr , -.codn s-iwusr , -.codn s-ixusr , -.codn s-irwxg , -.codn s-irgrp , -.codn s-iwgrp , -.codn s-ixgrp , -.codn s-irwxo , -.codn s-iroth , -.code s-iwoth -and -.codn s-ixoth , -which correspond to the POSIX C constants -.codn S_IRWXU , -.codn S_IRUSR , -.codn S_IWUSR , -.codn S_IXUSR , -.codn S_IRWXG , -.codn S_IRGRP , -.codn S_IWGRP , -.codn S_IXGRP , -.codn S_IRWXO , -.codn S_IROTH , -.code S_IWOTH -and -.codn S_IXOTH . +In the normal termination case, the value of the last +.meta form +of the last end clause appears as the return value of the +.code awk +macro. -Implementation note: since the -.code umask -C library function provides no way to retrieve the current mask without -overwriting with a new one, the \*(TX -.code umask -function, when given no argument, simulates the pure retrieval of the mask -by calling the C function with an argument of -.code #o777 -to temporarily install the maximally safe mask. The value returned is then -reinstated as the mask by another call to -.codn umask , -and that value is also returned. +Note that only termination of the +.code awk +macro initiated from condition-action clauses, +.code :begin-file +clauses, or +.code :end-file +clauses triggers +.code :end +clause processing. +If termination of the +.code awk +macro is initiated from within a +.codn :let , +.codn :inputs , +.code :output +or +.code :begin +clause, then end +clauses are not processed. +If an +.code :end +clause performs a non-local transfer, the remaining +.code :end +forms in that clause and +.code :end +clauses which follow are not evaluated. +.meIP (:begin-file << form *) +All +.code :begin-file +clauses are processed in the order in which they appear, before +.code awk +switches to each new input. + +If both +.code :begin +and +.code :begin-file +forms are specified, then before the first input is processed, +.code :begin +clauses are processed first, then the +.code :begin-file +clauses. +.meIP (:set-file >> { place << new-value }*) +The +.code :set-file +clause is a shorthand which translates +.code "(:set-file ...)" +to +.codn "(:begin-file (set ...))" . +.meIP (:end-file << form *) +All +.code :end-file +clauses are processed after the processing of an input +source finishes. -.coNP Functions @, makedev @ minor and @ major -.synb -.mets (makedev < minor << major ) -.mets (minor << dev ) -.mets (major << dev ) -.syne -.desc -The parameters -.metn minor , -.meta major +If both +.code :end and -.meta dev -are all integers. The -.code makedev -function constructs a combined device number from a minor and major pair (by -calling the Unix -.code makedev -function). This device number is suitable as an -argument to the -.code mknod -function (see below). Device numbers also appear as values of the -.code dev -slot of the -.code stat -structure. +.code :end-file +forms are specified, then before after the last input is processed, +.code :end-file +clauses are processed first, then the +.code :end +clauses. The -.code minor +:end-file +clauses are processed unconditionally, no matter how +the processing of an input source terminates, whether terminated +naturally by running out of records, prematurely by invocation of the +.code next-file +macro, or via a dynamic non-local control transfer such as a block +return or exception throw. + +If a +.code :begin-file +clause performs a non-local transfer, +.code :end-file +processing is not triggered, because the processing of the input +source is deemed not to have taken place. +.meIP >> ( condition << action *) +Clauses which do not have one of the specially recognized keywords +in the first position are ordinary condition-action clauses. After +processing the +.code :begin +clauses, the awk enters a loop in which it extracts successive records +from the input sources according to the +.code rs +(record separator) variable. Each record is divided into fields according +to the +.code fs +(field separator) +variable, and various +.code awk +variables are updated. Then, the condition-action clauses are processed, in the order +in which they appear. Each +.meta condition +is evaluated. If the resulting value is a regular expression +or a function, then this regular expression or function is invoked on the value +stored in the record variable +.codn rec , +and the result is taken to be the truth value of +.metn condition . +Otherwise, if the resulting value of +.meta condition +is other than a function or regular expression, it is taken directly +to be the truth value. +If the condition is true, then its associated +.meta action +forms are evaluated. Either way, processing passes to the next condition +clause (unless an explicit step is taken in one of the +.metn action -s +to prevent this, for instance by invoking the +.code next and -.code major -functions extract the minor and major device number -from a combined device number. +.code next-file +macros). +When an input source runs out of records, +.code awk +switches to the next input source. When there are no more input sources, +the macro terminates. +.RE -.coNP Function @ chmod -.synb -.mets (chmod < path << mode ) -.syne +.coNP Variables @ rec and @ orec .desc -The -.code chmod -function changes the permissions of the filesystem objects -specified by -.metn path . -It is a direct wrapper for the POSIX C library function of the same name. +The awk variable +.code rec +holds the current record. It is automatically updated prior to the +processing of the condition-pattern clauses. Prior to the extraction +of the first record, its value is +.codn nil . -The permissions are specified by -.metn mode , -an integer argument. +It is possible to assign to +.codn rec . +The value assigned to +.code rec +must be a character string. Immediately upon the assignment, the character +string is delimited into fields according to the field separator +awk variable +.codn fs , +and these fields are assigned to the field list +.codn f . +At the same time, the +.code nf +variable is updated to reflect the new number of fields. +Likewise, modification of these variables causes +.code rec +to be reconstructed by a catenation of the textual representation +of the fields in +.code f +separated by copies of the output field separator +.codn ofs . -The existing permissions may be obtained using the -.code stat -function. +The +.code orec +variable ("original record") also holds the current record. It is automatically +updated prior to the processing of the condition-clauses at the same time as +.code rec +with the same contents. Like +.codn rec , +it is initially +.code nil +before the first record is read. The +.code orec +variable is unaffected by modification of +the variables +.codn rec , +.code f +and +.codn nf . +It may be assigned. Doing so has no effect on any other +variable. -The function throws a -.code file-error -exception if an error occurs, otherwise it returns -.codn t . +.coNP Variable @ f +.desc +The awk variable +.code f +holds the list of fields. Prior to the first record being read, +its value is +.codn nil . +Whenever a new record is read, it is divided into fields according +to the field separator variable +.codn fs , +and these fields are stored in +.code f +as a list of character strings. -.TP* Example: -.verb - ;; Set permissions of foo.txt to "rw-r--r--" - ;; (owner can read and write; group owner - ;; and other users can only read). +If the variable +.code f +is assigned, the new value must be a sequence. The variable +.code nf +is automatically updated to reflect the length of this sequence. +Furthermore, the +.code rec +variable is updated by catenating a string representation of the +elements of this sequence, separated by the contents of the +.code ofs +(output field separator) +awk variable. - ;; numerically: - (chmod "foo.txt" #o644) +Note that assigning to a DWIM bracket form which indexes +.codn f , +such as for instance +.code "[f 0]" +constitutes an implicit modification of +.codn f , +and triggers the recalculation of +.codn rec . +Modifications of the +.code f +list which do not involve an implicit or explicit assignment to the variable +.code f +itself do not have this recalculating effect. - ;; symbolically: - (chmod "foo.txt" (logior s-irusr s-iwusr - s-irgrp - s-iroth)) -.brev +Unlike in Awk, assigning to the nonexistent field +.mono +.meti [f << m ] +.onom +where +.meta m +>= +.code nf +is erroneous. -.coNP Function @ mknod -.synb -.mets (mknod < path < mode <> [ dev ]) -.syne +.coNP Variable @ nf .desc -The -.code mknod -function tries to create an entry in the filesystem: a file, -FIFO, or a device special file, under the name -.metn path . -If it is successful, -it returns -.codn t , -otherwise it throws an exception of type -.codn file-error . - -The -.meta mode -argument is a bitwise or combination of the requested permissions, -and the type of object to create: one of the constants -.codn s-ifreg , -.codn s-ififo , -.codn s-ifchr , -.code s-ifblk -or -.codn s-ifsock . -The permissions are subject to the system -.codn umask . +The awk variable +.code nf +holds the current number of fields in the sequence +.codn f . +Prior to the first record being read, it is initially zero. -If a block or character special device -.cod2 ( s-ifchr -or -.codn s-ifblk ) -is being -created, then the -.meta dev -argument specifies the major and minor numbers -of the device. A suitable value can be constructed from a major and minor -pair using the -.code makedev -function. +If +.code nf +is assigned, then +.code f +is modified to reflect the new number of fields. Fields are deleted from +.code f +if the new value of +.code nf +is smaller. If the new value of +.code nf +is larger, then fields are added. The added fields are empty strings, +which means that +.code f +must be a sequence of a type capable of holding elements which are +strings. -.TP* Example: +If +.code nf +is assigned, then +.code rec +is also recalculated, in the same way as described in the documentation for the +.code f +variable. -.verb - ;; make a character device (8, 3) called /dev/foo - ;; requesting rwx------ permissions +.coNP Variable @ nr +.desc +The awk variable +.code nr +holds the current absolute record number. Record numbers start at 1. +Absolute means that this value does not reset to 1 when +.code awk +switches to a new input source; it keeps incrementing for each record. +See the +.code fnr +variable. - (mknod "dev/foo" (logior #o700 s-ifchr) (makedev 8 3)) -.brev +Prior to the first record being read, the value of +.code nr +is zero. -.coNP Functions @ symlink and @ link -.synb -.mets (symlink < target << path ) -.mets (link < target << path ) -.syne +.coNP Variable @ fnr .desc -The -.code symlink -function creates a symbolic link called -.meta path -whose contents -are the absolute or relative path -.metn target . -.meta target -does not actually have to exist. - -The link function creates a hard link. The object at -.meta target -is installed -into the filesystem at -.meta path -also. +The awk variable +.code fnr +holds the current record number within the file. The first record is 1. -If these functions succeed, they return -.codn t . -Otherwise they throw an exception -of type -.codn file-error . +Prior to the first record being read from the first input source, +the value of +.code fnr +is zero. Thereafter, it resets to 1 for the first record of each input +source and increments for the remaining records of the same input +source. -.coNP Function @ readlink -.synb -.mets (readlink << path ) -.syne +.coNP Variable @ arg .desc -If -.meta path -names a filesystem object which is a symbolic link, the -.code readlink -function reads the contents of that symbolic link and returns it -as a string. Otherwise, it fails by throwing an exception of type -.codn file-error . +The awk variable +.code arg +is an integer which indicates what input source is being processed. +Prior to input processing, it holds the value zero. When the first +record is extracted from the first input source, it is set to 1. +Thereafter, it is incremented whenever +.code awk +switches to a new input source. -.coNP Function @ realpath -.synb -.mets (realpath << path ) -.syne +.coNP Variable @ fname .desc -The -.code realpath -function provides access to the same-named POSIX function. -It processes the input string -.meta path -by expanding all symbolic links, removes all superfluous -.str ".." +The awk variable +.code fname +provides access to a character string which, if the current input is +a file stream, is the name of the underlying file. Assigning to this +variable changes its value, but has no effect on the input stream. +Whenever a new input source is used by +.code awk +it sets this variable either from the file name on which it is opening +a stream.. When using an existing stream rather than opening a file, +.code awk +sets this variable from the +.code :name +property of the stream. + +Note that the redirection macros +.code <- and -.str "." -path components, and extra path-separating slash characters, -to produce a canonical absolute path name. +.code (set f [(opip a b c ...) f]) +.brev + +.TP* Example: +.verb + ;; convert all fields from string to floating-point + (ff (mapcar flo-str)) +.brev -.coNP Structure @ group +.coNP Macro @ mf .synb -.mets (defstruct group nil -.mets \ \ name passwd gid mem) +.mets (mf < opip-arg *) .syne .desc -The -.code group -structure corresponds to the C type -.codn "struct group" . -Objects of this struct are produced by the password database -query functions -.codn getgrent , -.codn getgrgid , -and -.codn getgrnam . +The awk macro +.code mf +(map fields) +provides a shorthand for mapping each field +individually trough a pipeline of chained functions expressed using +.code opip +argument syntax. -.coNP Functions @, getgrent @ setgrent and @ endgrent -.synb -.mets (getgrent) -.mets (setgrent) -.mets (endgrent) -.syne -.desc -The first time -.code getgrent -function is called, it returns the first group database entry. -On subsequent calls it returns successive entries. -Entries are returned as instances of the -.code passwd -structure. If the function cannot retrieve an entry for any reason, -it returns -.codn nil . +The following equivalence holds, except that +.code f +refers to the awk variable even if the +.code mf +invocation occurs in code which establishes +a binding which shadows +.codn f . -The -.code setgrent -function rewinds the database scan. +.verb + (mf a b c ...) <--> (set f (mapcar (opip a b c ...) f)) +.brev -The -.code endgrent -function releases the resources associated with the scan. +.TP* Example: +.verb + ;; convert all fields from string to floating-point + (mf flo-str) +.brev -.coNP Function @ getgrgid +.coNP Macro @ fconv .synb -.mets (getgrgid << gid ) +.mets (fconv >> { clause | : | - }*) .syne .desc -The -.code getgrgid -searches the group database for an entry whose group ID field -is equal to the numeric -.metn gid . -If the search is successful, then a -.code group -structure representing the database entry is returned. -If the search fails, -.code nil -is returned. +The awk macro +.code fconv +provides a succinct way to request conversions of the textual fields. +Conversions are expressed by clauses which correspond with fields. -.coNP Function @ getgrnam -.synb -.mets (getgrnam << name ) -.syne -.desc -The -.code getgrnam -searches the group database for an entry whose group name -is equal to -.metn name . -If the search is successful, then a -.code group -structure representing the database entry is returned. -If the search fails, -.code nil -is returned. +Each +.meta clause +is an expression which must evaluate to a function. The clause is evaluated +in the same manner as the argument a +.code dwim +operator, using Lisp-1-style name lookup. Thus, functions may be +specified simply by using their name as a +.metn clause . -.SS* Unix Password Hashing -.coNP Function @ crypt -.synb -.mets (crypt < key << salt ) -.syne -.desc -The -.code crypt -function is a wrapper for the Unix C library function of the same name. -It calculates a hash over the -.meta key -and -.meta salt -arguments, which are strings. The hash is returned as a string. +Furthermore, several local functions exist in the scope of each +.metn clause , +providing a short-hand notation. These are described below. + +Conversion proceeds by applying the function produced by +a clause to the field to which that clause corresponds, positionally. +The return value of the function applied to the field replaces +the field. + +When a clause is specified as the symbol +.code - +(minus) +it has a special meaning: this minus clause occupies a field +position and corresponds to a field, but performs no conversion +on its field. The -.meta key -and -.meta salt -arguments are converted into UTF-8 prior to being passed into the underlying -platform function. The hash value is assumed to be UTF-8 and converted to -Unicode characters, though it is not expected to contain anything but 7 -bit ASCII characters. +.code : +(colon) +symbol isn't a clause and does not correspond to a field position. +Rather, it acts as a separator among clauses. It need not appear at +all. If it appears, it may appear at most twice. Thus, the +clauses may be separated into up to three sequences. -Note: the underlying C library function uses a static buffer for its return -value. The return value of the \*(TL function is a copy of that buffer. +If the colon does not appear, then all the clauses are +.IR "prefix clauses" . +Prefix clauses line up with fields from left to right. If there are fewer +fields than prefix clauses, the values of the excess clauses are evaluated, but +ignored. +.IR "Vice versa" , +if there are fewer prefix clauses than fields, then the excess +fields are not subject to conversions. -.SS* Unix Signal Handling +If the colon appears once, then the clauses before the colon, if any, are +prefix clauses, as described in the previous paragraph. Clauses after the +colon, if any, are +.IR "interior clauses" . +Interior clauses apply to any fields which are left unconverted by the prefix +clauses. All interior clauses are evaluated. If there are fewer fields than +interior clauses, then the values of the excess interior clauses are ignored. +If there are more fields than clauses, then the clause values are cycled: +re-used from the beginning against the excess fields, enough times to convert +all the fields. -On platforms where certain advanced features of POSIX signal handling are -available at the C API level, \*(TX exposes signal-handling functionality. +If the colon appears twice, then the clauses before the first colon, if any, +are prefix clauses, the clauses between the two clause are interior clauses, +and those after the second colon are +.IR "suffix clauses" . +The presence of suffix clauses change the behavior relative to the one-colon +case as follows. After the conversions are performed according to the prefix +clauses, the remaining fields are counted. If there are are only as many +fields as there are suffix clauses, or fewer, then the interior clauses are +evaluated, but ignored. The remaining fields are processed against the suffix +clauses. If after processing the prefix clauses there are more fields +remaining than suffix clauses, then a number of rightmost fields equal to the +number of suffix clauses is reserved for those clauses. The interior fields +are applied only to the unreserved middle fields which precede these reserved +rightmost fields, using the same repeating behavior as in the one-colon case. +Finally, the previously reserved rightmost fields are processed using +the suffix clauses. -A \*(TX program can install a \*(TL function (such as an anonymous. -.codn lambda , -or the function object associated with a named function) as the handler for -a signal. +The following special convenience functions are in scope of the clauses, +effectively providing a short-hand for commonly-needed conversions: +.RS +.coIP i +Provides conversion to integer. It is identical to the +.code toint +function. +.coIP o +Converts a string value holding an octal representation +to the integer which it denotes. The expression +.code "(o str)" +is equivalent to +.codn "(toint str 8)" . +.coIP x +Converts a string value holding a hexadecimal representation +to the integer which it denotes. The expression +.code "(x str)" +is equivalent to +.codn "(toint str 16)" . +.coIP b +Converts a string value holding a binary (base two) representation +to the integer which it denotes. The expression +.code "(b str)" +is equivalent to +.codn "(toint str 2)" . +.coIP c +Converts a string value holding a C-language-style representation +to the integer which it denotes, meaning that the +.code 0x +prefix denotes a hexadecimal value, a leading zero octal, otherwise +decimal. These prefixes follow the +.code + +or +.code - +sign, if present. +The expression +.code "(c str)" +is equivalent to +.codn "(toint str #\ec)" . +.coIP r +Converts a string holding a floating-point representation to +the floating-point value which it denotes. The expression +.code "(r str)" +is equivalent to +.codn "(tofloat str)" . +.ccIP @, iz @, oz @, xz @, bz @ cz and @ rz +Conversion similar to +.codn i , +.codn o , +.codn x , +.codn b , +.code c +and +.codn r , +but using +.code tointz +and +.codn tofloatz . +Thus fields which are non-numeric strings or the object +.code nil +get converted to 0, or 0.0 in the case of +.codn rz . +.RE +.IP +Because +.code fconv +macro destructively operates on the elements of the field list +.codn f , +it has the same effect as an assignment to the fields: +the value of +.code rec +is updated. -When that signal is delivered, \*(TX will intercept it with its own safe, -internal handler, mark the signal as deferred (in a \*(TX sense) and then -dispatch the registered function at a convenient time. +The return value of +.code fconv +is +.codn f . -Handlers currently are not permitted to interrupt the execution of most -\*(TX internal code. Immediate, asynchronous execution of handlers is -currently enabled only while \*(TX is blocked on I/O operations or sleeping. -Additionally, the -.code sig-check -function can be used to dispatch and clear deferred -signals. These handlers are then safely called if they were subroutines of -.codn sig-check , -and not asynchronous interrupts. +.TP* Examples: -.coNP Variables @, sig-hup @, sig-int @, sig-quit @, sig-ill @, sig-trap @, sig-abrt @, sig-bus @, sig-fpe @, sig-kill @, sig-usr1 @, sig-segv @, sig-usr2 @, sig-pipe @, sig-alrm @, sig-term @, sig-chld @, sig-cont @, sig-stop @, sig-tstp @, sig-ttin @, sig-ttou @, sig-urg @, sig-xcpu @, sig-xfsz @, sig-vtalrm @, sig-prof @, sig-poll @, sig-sys @, sig-winch @, sig-iot @, sig-stkflt @, sig-io @ sig-lost and @ sig-pwr -.desc -These variables correspond to the C signal constants -.codn SIGHUP , -.code SIGINT -and so forth. -The variables -.codn sig-winch , -.codn sig-iot , -.codn sig-stkflt , -.codn sig-io , -.code sig-lost -and -.code sig-pwr -may not be available since a system may lack the corresponding signal -constants. See notes for the function -.codn log-authpriv . +.verb + ;; convert up to first three fields to integer: + (awk ((fconv i i i))) -The highest signal number is 31. + ;; convert all fields to floating-point + (awk ((fconv : r :))) -.coNP Functions @ set-sig-handler and @ get-sig-handler + ;; convert first and second fields to integer + ;; from hexadecimal; + ;; convert last field to integer from octal; + ;; process pairs of fields in between + ;; these by leaving the first element of + ;; each pair unconverted and converting second + ;; to floating-point; + (awk ((fconv x x : - r : o))) + + ;; convert all fields, except the first, + ;; from integer, turning empty strings + ;; and non-integer junk as zero; + ;; leave first field unconverted: + (awk ((fconv - : iz))) +.brev + +.coNP Macros @, -> @, ->> @, <- @ !> and @ < path << form *) +.mets (->> < path << form *) +.mets (<- < path << form *) +.mets (!> < command << form *) +.mets ( , +.code ->> +and +.code !> +evaluate each +.meta form +in a dynamic environment in which the +.code *stdout* +variable is bound to a file output stream, for the first two +functions, or output command pipe in the case of the last one. + +Similarly, when at least +.meta form +argument is present, the remaining functions +.code <- +and +.code +macro indicates that the file named +.meta path +is to be opened for writing and overwritten, or created if it doesn't exist. +The +.code ->> +macro indicates that the file named by +.meta path +is to be opened in append mode, created if necessary. +The +.code <- +macro indicates that the file given by +.meta path +is to be opened for reading. -If -.meta handling-spec -is the symbol -.codn nil , -then the function previously associated -with the signal, if any, is removed, and the signal is disabled. For a signal -to be disabled means that the signal is set to the -.code SIG_IGN -disposition (refer to the C API). +The +.code !> +macro indicates that +.meta command +is to be opened as an output command pipe. The +.code [f 2] 5)))) -A signal handling function must take two arguments. It is of the form: + ;; print strictly original lines from orec + (awk ((and [f 2] (fconv - - iz) (> [f 2] 5)) + (prn orec))) -.mono -.mets (lambda >> ( signal << async-p ) ...) -.onom +.brev +.IP 2. +Print every tenth line: -The -.meta signal -argument is an integer indicating the signal number for which the -handler is being invoked. The -.meta asyncp-p -argument is a Boolean value. -If it is -.codn t , -it indicates that the handler is being invoked -asynchronously\(emdirectly in a signal handling context. If it is -.codn nil , -then it -is a deferred call. Handlers may do more things in a deferred call, such -as terminate by throwing exceptions, and perform I/O. +.verb + (awk ((zerop (mod nr 10)))) +.brev +.IP 3. +Print any line with a substring matching a regex: -The return value of a handler is normally ignored. However if it invoked -asynchronously (the -.meta async-p -argument is true), then if the handler returns -a -.cod2 non- nil -value, it is understood that the handler -requesting that it be deferred. This means that the signal will be marked -as deferred, and the handler will be called again at some later -time in a deferred context, whereby -.meta async-p -is -.codn nil . -This is not guaranteed, however; -it's possible that another signal will arrive before that happens, -possibly resulting in another async call, so the handler must -be prepared to deal with an async call at any time. +.verb + (awk (#/(G|D)(2\ed[\ew]*)/)) +.brev +Note the subtle flaw here: the +.code [\ew]* +portion of the regular expression contributes nothing +to what lines are matched. The following example +has a similar flaw. +.IP 4. +Print any line with a substring beginning with a +.code G +or +.code D +followed by a sequence of digits and characters: -If a handler is invoked synchronously, then its return value is ignored. +.verb + (awk (#/(G|D)([\ed\ew]*)/)) +.brev +.IP 5. +Print lines where the second field matches a regex, +while the fourth one doesn't: -In the current implementation, signals do not queue. If a signal is delivered -to the process again, while it is marked as deferred, it simply stays deferred; -there is no counter associated with a signal, only a Boolean flag. +.verb + (awk (:let (r #/xyz/)) + ((and [f 3] [r [f 1]] (not [r [f 3]])))) +.brev +.IP 6. +Print lines containing a backslash in the second field: -.coNP Function @ sig-check -.synb -.mets (sig-check) -.syne -.desc -The -.code sig-check -function tests whether any signals are deferred, and for each -deferred signal in turn, it executes the corresponding handler. For a signal to -be deferred means that the signal was caught by an internal handler in -\*(TX and the event was recorded by a flag. If a handler function is removed -while a signal is deferred, the deferred flag is cleared for that signal. +.verb + (awk ((find #\e\e [f 1]))) +.brev +.IP 7. +Print lines containing a backslash using a regex constructed +from a string. Note that backslash escapes are interpreted +twice: once in the string literal, and once in the parsing +of the regex, requiring four backslashes to encode one: -Calls to the -.code sig-check -function may be inserted into CPU-intensive code that -has no opportunity to be interrupted by signals, because it doesn't invoke any -I/O functions. +.verb + (awk (:let (r (regex-compile "\e\e\e\e"))) + ((and [f 1] [r [f 1]]))) +.brev +.IP 8. +Print penultimate and ultimate field in each record, +separating then by a colon: -.coNP Function @ raise -.synb -.mets (raise << signal ) -.syne -.desc -The -.code raise -function sends -.meta signal -to the process. -It is a wrapper for the C function of the same name. +.verb + ;; original: {OFS=":";print $(NF-1), $NF} + ;; + (awk (t (set ofs ":") (prn [f -2] [f -1]))) +.brev +.IP +Note that the above behaves +more correctly than the original Awk example because in the +when there is only one field, +.code $(NF-1) +reduces to +.code $0 +which refers to the entire record, not to the field. +This sort of bug is why the \*(TL +.code awk +does not imitate the design decision to make the record +the first numbered field. +.IP 9. +Output the line number and number of fields separated by colon, +by producing a single string first: -The return value is -.code t -if the function succeeds, otherwise -.codn nil . +.verb + (awk (t (prn `@nr:@nf`))) +.brev +.IP 10. +Print lines longer than 72 characters: -.coNP Function @ kill -.synb -.mets (kill < process-id <> [ signal ]) -.syne -.desc -The -.code kill -function is used for sending a signal to a process group or process. -It is a wrapper for the POSIX -.code kill -function. +.verb + (awk ((> (len rec) 72))) +.brev +.IP 11. +Print first two fields in reverse order, separated by +.codn ofs : -If the -.meta signal -argument is omitted, it defaults to the same value as -.codn sig-term . +.verb + (awk (t (prn [f 1] [f 0]))) +.brev +.IP 12. +Same as 11, but with field separation consisting of a +comma, or spaces and tabs, or both in sequence: -The return value is -.code t -if the function succeeds, otherwise -.codn nil . +.verb + (awk (:set fs #/,[ \et]*|[ \et]+/) + (t (prn [f 1] [f 0]))) +.brev +.IP 13. +Add the values in the first column, then print sum and +average: -.SS* Unix Processes +.verb + ;; original: + ;; {s += $1} + ;; END {print "sum is ", s, " average is", s/NR} + ;; + (awk (:let (s 0) (n 0)) + ([f 0] (fconv r) (inc s [f 0]) (inc n)) + (:end (prn `sum is @s average is @(/ s n)`))) +.brev -.coNP Functions @ fork and @ wait -.synb -.mets (fork) -.mets (wait >> [ pid <> [ flags ]]) -.syne -.desc -The -.code fork -and -.code wait -functions are interfaces to the Unix functions -.code fork +Note that the original is not robust against blank lines +in the input. Blank lines are treated as if they had a +first column field of zero, and are counted toward the +denominator in the calculation of the average. +.IP 14. +Print fields in reverse order, one per line: + +.verb + (awk (t (tprint (reverse f)))) +.brev +.IP 15. +Print all lines between occurrences of +.code start and -.codn waitpid . +.codn stop : -The -.code fork -function creates a child process which is a replica of the parent. Both -processes return from the function. In the child process, the return value is -zero. In the parent, it is an integer representing the process ID of the child. -If the function fails to create a child, it returns -.code nil -rather than an integer. In this case, the -.code errno -function can be used to inquire about the cause. +.verb + (awk ((rng #/start/ #/stop/))) +.brev +.IP 16. +Print lines whose first field is different from +the corresponding field in the previous line: -The -.code wait -function, if successful, returns a cons cell consisting of a pair of integers. -The -.code car -of the cons is the process ID of the process or group which was successfully -waited on, and the -.code cdr -is the status. If -.code wait -fails, it returns -.codn nil . -The -.code errno -function can be used to inquire about the cause. +.verb + (awk (:let prev) + ((nequal [f 0] prev) (prn) (set prev [f 0]))) +.brev +.IP 17. +Simulate the +.code echo +utility: -The -.meta process-id -argument, if not supplied, defaults to -1, which means that -.code wait -waits for any process, rather than a specific process. Certain other -values have special meaning, as documented in the POSIX standard -for the -.code waitpid -function. +.verb + (awk (:begin (prn `@{*args* " "}`))) +.brev -The -.meta flags -argument defaults to zero. If it is specified as nonzero, it should be -a bitwise combination (via the -.code logior -function) of the variables -.codn w-nohang , -.code w-untraced -and -.codn w-continued . -If -.code w-nohang -is used, then -.code wait -returns a cons cell whose -.code car -specifies a process ID value of zero in the situation that at least -one of the processes designated by -.code process-id -exist and are children of the calling process, but have not changed state. -In this case, the status value in the -.code cdr -is unspecified. +Note: if this is evaluated in the command line, for instance with the +.code -e +option, an explicit exit is required to prevent the arguments from being +processed by +\*(TX after +.code awk +completes: -Status values may be inspected with the functions -.codn w-ifexited , -.codn w-exitstatus , -.codn w-ifsignaled , -.codn w-termsig , -.codn w-coredump , -.codn w-ifstopped , -.code w-stopsig -and -.codn w-ifcontinued . +.verb + (awk (:begin (prn `@{*args* " "}`) (exit 0))) +.brev +.IP 18. +Pint the components of the +.code PATH +environment variable, one per line: -.coNP Functions @, w-ifexited @, w-exitstatus @, w-ifsignaled @, w-termsig @, w-coredump @ w-ifstopped and @ w-stopsig -.synb -.mets (w-ifexited << status ) -.mets (w-exitstatus << status ) -.mets (w-ifsignaled << status ) -.mets (w-termsig << status ) -.mets (w-coredump << status ) -.mets (w-ifstopped << status ) -.mets (w-stopsig << status ) -.mets (w-ifcontinued << status ) -.syne -.desc -These functions analyze process exit values produced by the -.code wait -function. +.verb + ;; Process variable as if it were a file: + (awk (:inputs (make-string-input-stream + (getenv "PATH"))) + (:set fs ":") + (t (tprint f))) -They are closely based on the -POSIX macros -.codn WIFEXITED , -.codn WEXITSTATUS , -and so on. + ;; Just get, split and print; awk macro is irrelevant + (awk (:begin (tprint (split-str (getenv "PATH") ":")))) +.brev +.IP 19. +Given a file called +.code input +which contains page headers of the format +.str "Page #" +and a \*(TL file called +.code prog.tl +which contains: -The -.meta status -value is either an integer, or a cons cell. In this case, the cons -cell is expected to have an integer in its -.code cdr -which is used as the status. +.verb + (awk (:let (n (toint n))) + (#/Page/ (set [f 1] (pinc n))) + (t)) +.brev -The -.codn w-ifexited , -.codn w-ifsignaled , -.codn w-coredump , -.code w-ifstopped -and -.code w-ifcontinued -functions have Lisp Boolean return semantics, unlike their C language -counterparts: they return -.code t -or -.codn nil , -rather than zero or nonzero. The others return integer values. +the command line: -.coNP Function @ exec -.synb -.mets (exec < file <> [ args ]) -.syne -.desc -The exec function replaces the process image with the executable specified -by string argument -.metn file . -The executable is found by searching the system path. +.verb + txr -Dn=5 prog.tl input +.brev -The -.meta file -argument becomes the first argument of the executable, argument zero. +prints the file, filling in page numbers starting at 5. +.RE -If -.meta args -is specified, it is a list of strings. These are passed as the additional -arguments of the executable. +.SS* Environment Variables and Command Line -If -.code exec -fails, an exception of type -.code file-error -is thrown. +Note that environment variable names, their values, and command line +arguments are all regarded as being externally encoded in UTF-8. \*(TX performs +the encoding and decoding automatically. -.coNP Function @ exit* -.synb -.mets (exit* << status ) -.syne +.coNP Special variables @, *args-full* @ *args-eff* and @ *args* .desc The -.code exit* -function terminates the entire process (running \*(TX image), specifying -the termination status to the operating system. The -.meta status -argument is treated exactly like that of the -.code exit -function. Unlike that function, this one exits the process immediately, -cleaning up only low-level operating system resources such as closing file -descriptors and releasing memory mappings, without performing user-space -cleanup. - -.code exit* -is implemented using a call to the POSIX function -.codn _exit . +.code *args-full* +variable holds the original, complete list of arguments passed +from the operating system, including the program executable +name. -.coNP Functions @ getpid and @ getppid -.synb -.mets (getpid) -.mets (getppid) -.syne -.desc -These functions retrieve the current process ID and the parent process ID -respectively. They are wrappers for the POSIX functions -.code getpid +During command line option processing, \*(TX may transform the +argument list. The hash bang mechanism, and the +.code --args and -.codn getppid . +.code --eargs +options can inject new command line arguments, as can code +which is executed during argument processing via the +.code -e +options and others. -.coNP Function @ daemon -.synb -.mets (daemon < nochdir-p << noclose-p ) -.syne -.desc -This is a wrapper for the function -.code daemon -which originated in BSD Unix. +The +.code *args-eff* +variable holds the list of +.I "effective arguments" , +which is the argument list after these transformations are applied. +This variable is established and set to the same value as +.code *args-full* +prior to command line processing, but is not updated with its final +value until after command line processing. -It returns -.code t -if successful, -.code nil -otherwise, and the -.code errno -variable is set in that case. +The +.code *args* +variable holds a list of strings representing the remaining +arguments which follow any options processed by the \*(TX executable, +and the script name. This list is a suffix of +.codn *args-eff* . +Thus, the arguments before +.code *args* +can be calculated using the expression +.codn "(ldiff *args-eff* *args*)" . -.SS* Unix File Descriptors +The +.code *args* +variable is available to to \*(TL expressions invoked from the +command line via the +.codn -p , +.code -e +and other such options. During these evaluations, +.code *args* +holds all the remaining options, after the invoking option and its +argument expression. In other words, code executed from the command line +has access to the remaining arguments which follow it. +Furthermore, this code may modify the value of +.codn *args* . +Such a modification is visible to the option processing code. +That is to say code executed from the command line can rewrite the remaining +list of arguments, and that list takes effect. -.coNP Function @ open-fileno +.coNP Function @ env .synb -.mets (open-fileno < file-descriptor <> [ mode-string ]) +.mets (env) .syne .desc The -.code open-fileno -function creates a \*(TX stream over a file descriptor. The -.meta file-descriptor -argument must be an integer denoting a valid file descriptor. +.code env +function retrieves the list of environment variables. Each +variable is represented by a single entry in the list: a string which +contains an +.code = +(equal) character somewhere, separating the variable name +from its value. -For a description of -.metn mode-string , -see the -.code open-file +See also: the +.code env-hash function. -.coNP Function @ fileno +.coNP Function @ env-hash .synb -.mets (fileno << stream ) +.mets (env-hash) .syne .desc The -.code fileno -function returns the underlying file descriptor of -.metn stream , -if it has one. Otherwise, it returns -.codn nil . - -This is equivalent to querying the stream using -.code stream-get-prop -for the -.code :fd -property. +.code env-hash +function constructs and returns an +.code :equal-based +hash. The hash is +populated with the environment variables, represented as key-value pairs. -.coNP Function @ dupfd +.coNP Functions @, getenv @ setenv and @ unsetenv .synb -.mets (dupfd < old-fileno <> [ new-fileno ]) +.mets (getenv << name ) +.mets (setenv < name < value <> [ overwrite-p ]) +.mets (unsetenv << name ) .syne .desc +These functions provide access to, as well as manipulation of, environment +variables. Of these three, +.code setenv +and +.code unsetenv +might not be available on some platforms, or +.code unsetenv +might be be present in a simulated form which sets the variable +.meta name +to the empty string rather than deleting it. + The -.code dupfd -function provides an interface to the POSIX functions -.code dup -or -.codn dup2 , -when called with one or two arguments, respectively. +.code getenv +function searches the environment for the environment variable whose name +is +.metn name . +If the variable is found, its value is returned. Otherwise +.code nil +is returned. -.coNP Function @ pipe -.synb -.mets (pipe) -.syne -.desc The -.code pipe -function, if successful, returns a pair of integer file descriptors -as a cons cell pair. The descriptor in the -.code car -field of the pair is the read end of the pipe. +.code setenv +function creates or modifies the environment variable indicated by +.metn name . The -.code cdr -holds the write end. +.meta value +string argument specifies the new value for the variable. +If +.meta value +is +.codn nil , +then +.code setenv +behaves like +.codn unsetenv , +except that it observes the +.meta overwrite-p +argument. That is to say, the meaning of a null +.meta value +is that the variable is to be removed. -If the function fails, it throws an exception of type -.codn file-error . +If the +.meta overwrite-p +argument is specified, and is true, +then the variable is overwritten if it already exists. +If the argument is false, then the variable is not modified if it +already exists. If the argument is not specified, it defaults +to the value +.metn t , +effectively giving rise to a two-argument form of +.code setenv +which creates or overwrites environment variables. -.coNP Function @ close -.synb -.mets (close < fileno <> [ throw-on-error-p ]) -.syne -.desc -The -.code close -function passes the integer descriptor -.meta fileno -to the POSIX -.code close -function. If the operation is successful, then -.code t -is returned. Otherwise an exception of type -.code file-error -is thrown, unless the -.meta throw-on-error-p -argument is present, with a true value. In that case, -.code close -indicates failure by returning -.codn nil . +A variable removal is deemed to be an overwrite. +Thus if both +.meta value +and +.meta overwrite-p +are +.codn nil , +then +.code setenv +does nothing. -.coNP Function @ poll -.synb -.mets (poll < poll-list <> [ timeout ]) -.syne -.desc The -.code poll -function suspends execution while monitoring one or more file descriptors -for specified events. It is a wrapper for the same-named POSIX function. +.code setenv +function unconditionally returns +.meta value +regardless of whether or not it overwrites or removes an existing variable. The -.meta poll-list -argument is a list of -.code cons -pairs. The -.code car -of each pair is either an integer file descriptor, or else a stream -object which has a file descriptor (the -.code fileno -function can be applied to that stream to retrieve a descriptor). +.code unsetenv +function removes the environment variable +specified by +.metn name , +if it exists. On some platforms, it instead sets the environment variable +to the empty string. + +Note: supporting removal semantics in +.code setenv +allows for the following simple save/modify/restore pattern: + +.verb + (let* ((old-val (getenv "SOME-VAR"))) + (unwind-protect + (progn (setenv "SOME-VAR" new-val) + ...) + (setenv "SOME-VAR" old-val))) +.brev + +This works in the case when +.code SOME-VAR +exists, as well as in the case that it doesn't exist. +In both cases, its previous value or, respectively, non-existence, +is restored by the +.code unwind-protect +cleanup form. + +.SS* Command Line Option Processing + +\*(TL provides a support for recognizing, extracting and validating +the POSIX-style options from a list of command-line arguments. + +The supported options can be defined as a list of option descriptor +objects each of which is constructed by a call to the +.code opt +function. Each option can have a long name, a short name, +a type, and a description. + The -.code cdr -of each pair is an integer bit mask specifying the events, whose -occurrence the file descriptor is to be monitored for. The variables -.codn poll-in , -.codn poll-out , -.code poll-err -and several others are available which hold bitmask values corresponding -to the constants -.codn POLLIN , -.codn POLLOUT , -.code POLLERR -used with the C language -.code poll -function. +.code getopts +function takes a list of option descriptors, and a list of arguments, +producing a parse, or else throwing an exception of type +.code opt-error +if an error is detected. The returned object, an instance of struct type +.codn opts , +can then be queried for specific option values, or for the remaining non-option +arguments. The -.meta timeout -argument, if absent, defaults to the value -1, which specifies an indefinite -wait. A nonnegative value specifies a wait with a timeout, measured in -milliseconds. +.code opthelp +function takes a list of option descriptors and an output stream, +and generates help text on that stream. A program supporting a +.code --help +option can use this to generate that portion of its help text which +describes the available options, as well as the conventions that they use. -The function returns a list of pairs representing the descriptors or streams -which were successfully polled. If the function times out, it returns an -empty list. If an error occurs, an exception is thrown. +.NP* Command Line Option Conventions -The returned list is similar in structure to the input list. However, it holds -only entries which polled positive. The -.code cdr -of every pair now holds a bitmask of the events which were to have occurred. +A command line option can have a short or long name. A short name is always +one-character long, and treated specially in the command line syntax. Long +options have names two or more characters long. An option can have both a long +and short name. Options may not begin with the +.code - +(ASCII dash) character. A long option may not contain the +.code = +character. -.SS* Unix File Control +Short options are invoked by specifying an argument with a single leading +.code - +followed by the option character. Multiple short options which take +no argument can be "clumped": combined into a single argument consisting of +a single +.code - +followed by multiple short option characters. -.coNP Variables @, o-accmode @, o-rdonly @, o-wronly @, o-rdwr @, o-creat @, o-noctty @, o-trunc @, o-append @, o-nonblock @, o-sync @, o-async @, o-directory @, o-nofollow @, o-cloexec @, o-direct @ o-noatime and @ o-path -.desc -These variables correspond to the POSIX file mode constants -.codn O_ACCMODE , -.codn O_RDONLY , -.codn O_WRONLY , -.codn O_RDWR , -.codn O_CREAT , -.codn O_NOCTTY , -and so forth. +An option can take an argument, in which case the argument is required. +An option which takes no argument is Boolean, and a Boolean option +never takes an argument: "takes no argument" and "Boolean" effectively +mean the same thing. -The availability of the variables -.codn o-async , -.codn o-directory , -.codn o-nofollow , -.codn o-cloexec , -.codn o-direct , -.code o-noatime -and -.code o-path -depends on the host platform. +Long options are invoked as an argument which begins with a +.code -- +(double dash) +immediately followed by the name. When a long option takes an argument, +it is mandatory. It must be specified in the same argument, separated +from the name by the +.code = +character. If that is omitted, then the next command line argument +is taken as the argument. That argument is removed, and not recognized as +an option, even if it looks like one. -Some of these flags may be set or cleared on an existing file descriptor -using the -.code f-setfl -command of the -.code fcntl -function, in accordance with POSIX and the host platform documentation. +A Boolean long option can be explicitly specified as false using the +.code --no- +prefix rather than the +.code -- +prefix. -.coNP Variables @, seek-set @ seek-cur and @ seek-end +Short options may be invoked using long name syntax; if +.code a +is a short option, then it may be referenced on the command line as +.code --a +and treated as a long option in all other ways, including the use +of +.code --no- +to explicitly specify false for a Boolean option. + +If a short option takes an argument, it may not clump with other +short option. The following command line argument is taken as the +options argument. That argument is removed and is not recognized as +an option even if it looks like one. + +If the command line argument +.code -- +occurs in the command line where an option would otherwise be recognized, +it signifies the end of the options. The subsequent arguments are the +non-option arguments, even if they resemble options. + +.NP* Command Line Processing Example + +The following example illustrates a complete \*(TL program which +parses command line options: + +.verb + (defvarl options + (list (opt "v" "verbose" :dec + "Verbosity level. Higher values produce more chatter.") + (opt nil "help" :bool + "List this help text.") + (opt "x" nil :hex + "The X factor: a number with a mysterious\e \e + interpretation, affecting the program\e \e + behavior in strange ways.") + (opt "z" nil) ;; undocumented option + (opt nil "cee" :cint + "C style integer.") + (opt "g" "gravity" :float + "Gravitational constant. This gives\e \e + the gravitational field\e \e + strength at the Earth's surface.") + (opt "l" "lit" :str + "A character string given in TXR Lisp notation.") + (opt "c" nil 'upcase-str + "Custom treatment: ARG is converted to upper case.") + (opt "b" "bool" :bool + "A flag you can flip true."))) + + (defvarl prog-name *load-path*) + + (let ((o (getopts options *args*))) + (when [o "help"] + (put-line "Usage:\en") + (put-line ` @{prog-name} [options] arg*`) + (opthelp options) + (exit 0)) + (put-line `args after opts are: @{o.out-args ", "}`)) +.brev + +.coNP Structure @ opt-desc +.synb +.mets (defstruct opt-desc +.mets \ \ short long helptext type +.mets \ \ ... < unspecified << slots ) +.syne .desc -These variables correspond to the ISO C constants -.codn SEEK_SET , -.code SEEK_CUR +The +.code opt-desc +structure describes a single command line option. + +The +.code short and -.codn SEEK_END . -These values, usually associated with the ISO C -.code fseek -function, are also used in the -.code fcntl -file locking interface as values of the -.code whence -member of the -.code flock -structure. +.code long +slots are either +.code nil +or else hold strings. +The +.code short +slot gives the option's short name: a one-character-long +string which may not be the ASCII dash character +.codn - . +The +.code long +slot gives the option's long name: a string two or more +characters long which doesn't begin with a dash. +An option must have at least one of these names. -.coNP Variables @, f-dupfd @, f-dupfd-cloexec @, f-getfd @, f-setfd @, f-getfl @, f-setfl @, f-getlk @ f-setlk and @ f-setlkw -.desc -These variables correspond to the POSIX -.code fcntl -command constants -.codn F_DUPFD , -.codn F_GETFD , -.codn F_SETFD , -and so forth. Availability of the -.code f-dupfd-cloexec -depends on the host platform. +The +.code helptext +slot provides a descriptive string. This string may be long. The +.code opthelp +function displays this text, formatting into multiple lines as necessary. +If +.code helptext +is +.codn nil , +the option is considered undocumented. -.coNP Variable @ fd-cloexec -.desc The -.code fd-cloexec -variable corresponds to the POSIX -.code FD_CLOEXEC -constant. It denotes the flag which may be set by the -.code fd-setfd -command of the -.code fcntl -function. +.code type +slot may be a symbol naming a global function which takes one argument, +or it may be such a function object. Otherwise it must be one of the +following keyword symbols: +.RS +.coIP :bool +This indicates that the type of the option is Boolean. Such +an option doesn't take any argument. Its value is +.code t +or +.codn nil . +.coIP :dec +This indicates that the option requires an argument, which is a +decimal integer with an optional positive or negative sign. +This argument is converted to an integer object. +.coIP :hex +This type indicates that the option requires an argument consisting +of a hexadecimal integer with an optional positive or negative sign. +This is converted to an integer object. +.coIP :oct +This type indicates that the option requires an argument consisting +of a octal integer with an optional positive or negative sign. +This is converted to an integer object. +.coIP :cint +This type indicates that the option requires an integer argument +whose format conforms to one of three C language conventions in most respects, +other than that this integer may have an arbitrary range. +All forms may carry an optional positive or negative leading sign +at the very beginning. +The first convention consists of decimal digits, which must not have +a superfluous leading zero. The second convention consists of octal +digits which are introduced by an extra leading zero. +The third convention consists of hexadecimal digits introduced by the +.code 0x +prefix. +.coIP :float +This type indicates a decimal floating-point argument, which is converted to +a floating-point number. Its basic form is: an optional leading plus or +minus sign, followed by a sequence of one or more digits which may contain +a single decimal point anywhere, including the very beginning of the +sequence or at the end, optionally followed by the letter +.code e +or +.code E +followed by a decimal integer which may have a leading positive or negative +sign, and include leading zeros. +.coIP :text +This type indicates a simple textual argument. The argument is taken as +verbatim UTF-8 text, converted to a string without interpreting +the characters in any special way. +.coIP :str +This type indicates that the argument consists of the interior notation of +a TXR Lisp character string. It is processed by adding a double quote +at the beginning or end, and parsed as a string literal. This parsing must +successfully yield a string object, otherwise the argument is ill-formed. +.meIP (list << type ) +If the type is specified as a compound form headed by the +.code list +symbol, it indicates that the command line option's argument is a list +of elements. The argument appears on the command line as a single string +contained within one argument. It may contain commas, and is split into pieces +using the comma character as a separator. The pieces are then individually +treated as of type +.meta type +and converted accordingly. The option's argument is then a list object +whose elements are the converted pieces. For instance +.code "(list :dec)" +will convert a list of comma-separated decimal integer tokens into +a list of integer objects. +.RE -.coNP Variables @, f-rdlck @ f-wrlck and @ f-unlck -.desc -These variables correspond to the POSIX lock type constants -.codn F_RDLCK , -.code F_WRLCK -and -.codn F_UNLCK . -They specify the possible values of the +.IP +If .code type -field of the -.code flock -structure. +is a function, then the option requires an argument. The argument string +is passed to the function, and the value is whatever the function returns. -.coNP Structure @ flock -.synb -.mets (defstruct flock nil -.mets \ \ type whence -.mets \ \ start len -.mets \ \ pid) -.syne -.desc The -.code flock -structure corresponds to the POSIX structure of the same name. -An instance of this structure must be specified as the third -argument of the -.code fcntl -function when the -.meta command -argument is one of the values -.codn f-getlk , -.code f-setlk -or -.codn f-setlkw . +.code opt-desc +structure may have additional slots which are not specified. -All slots must be initialized with appropriate values before -calling -.code fcntl -with the exception that the -.code f-getlk -command does not access the existing value of the -.code pid -slot. +The +.code opt +convenience function is provided for constructing +.code opt-desc +objects. -.coNP Function @ fcntl +.coNP Function @ opt .synb -.mets (fcntl < fileno < command <> [ arg ]) +.mets (opt < short < long >> [ type <> [ helptext ]]) .syne .desc The -.code fcntl -function corresponds to the same-named POSIX function. -The -.meta fileno +.code opt +function provides a slightly condensed syntax for constructing +an object of type +.codn opt-desc . + +The required arguments +.meta short and -.meta command -arguments must be integers. -The \*(TL -.code fileno -restricts the -.meta command -argument to the supported values for which symbolic variable names are provided. -Other integer -.meta command -values are rejected by returning -1 and setting the -.code errno -variable to -.codn EINVAL . -Whether the third argument is required, and what type it must be, depends on the -.meta command -value. Commands not requiring the third argument ignore it if it is passed. +.meta long +are strings, corresponding to +.code opt-desc +slots of the same name. -.code fcntl -commands for which POSIX requires an argument of type -.code long -require -.meta arg -to be an integer. +The optional parameter +.meta type +corresponds to the same-named slot and defaults to +.codn :bool . -The file locking commands -.codn f-getlk , -.code f-setlk -and -.code f-setlkw -require -.meta arg -to be a -.code flock -structure. +The optional parameter +.meta helptext +corresponds to the same-named slot, and defaults to +.code nil +(no help text provided for the option). The -.code fcntl -function doesn't throw an error if the underlying POSIX function indicates -failure; the underlying function's return value is converted to a Lisp integer -and returned. - -.SS* Unix Itimers -Itimers ("interval timers") can be used in combination with signal handling to -execute asynchronous actions. Itimers deliver delayed, one-time signals, -and also periodically recurring signals. For more information, consult the -POSIX specification. +.code opt +function follows this equivalence: -.coNP Variables @, itimer-real @ itimer-virtual and @ itimer-prof -.desc -These variables correspond to the POSIX constants -.codn ITIMER_REAL , -.code ITIMER_VIRTUAL -and -.codn ITIMER_PROF . -Their values are suitable as the -.meta timer -argument of the -.code getitimer -and -.code setitimer -functions. +.verb + (opt a b c d) <--> (new opt-desc short a long b + type c helptext d) +.brev -.coNP Functions @ getitimer and @ setitimer +.coNP Structure @ opts .synb -.mets (getitimer << timer ) -.mets (setitimer < timer < interval << value ) +.mets (defstruct opts nil +.mets \ \ in-args out-args +.mets \ \ ... < unspecified << slots ) .syne .desc The -.code getitimer -function returns the current value of the specified timer, -which must be -.codn itimer-real , -.code itimer-virtual -or -.codn itimer-prof . - -The current value consists of a list of two integer values, which -represents microseconds. The first value is the timer interval, -and the second value is the timer's current value. - -Like -.codn getitimer , -the -.code setitimer -function also retrieves the specified timer. -In addition, it stores a new value in the timer, -which is given by the two arguments, expressed in microseconds. - -.SS* Unix Syslog - -On platforms where a Unix-like syslog API is available, \*(TX exports this -interface. \*(TX programs can configure logging via the -.code openlog -function, -control the logging mask via -.code setlogmask -and generate logs via -.codn syslog , -or using special syslog streams. - -.coNP Variables @, log-pid @, log-cons @, log-ndelay @, log-odelay @ log-nowait and @ log-perror -.desc -These variables take on the values of the corresponding C preprocessor -constants from the -.code -header: -.codn LOG_PID , -.codn LOG_CONS , -etc. -These integer values represent logging options used in the -.meta options -argument to the -.code openlog -function. +.code opts +structure represents a parsed command line, containing decoded +information obtained from the options, and an indication where +the non-option arguments start. -Note: -.code LOG_PERROR -is not in POSIX, and so -.code log-perror -might not be available. -See notes about -.code LOG_AUTHPRIV -in the documentation for -.codn log-authpriv . +The +.code opts +structure supports direct indexing for option retrieval. +That is the only documented interface for accessing the parsed +options; the implementation of the information set describing +the parsed options is unspecified. -.coNP Special variables @, log-user @, log-daemon @ log-auth and @ log-authpriv -.desc -These variables take on the values of the corresponding C preprocessor -constants from the -.code -header: -.codn LOG_USER , -.codn LOG_DAEMON , -.code LOG_AUTH -and -.codn LOG_AUTHPRIV . -These are the integer facility codes specified in the -.code openlog -function. +The +.code in-args +slot holds the original argument list. -Note: -.code LOG_AUTHPRIV -is not in POSIX, and so -.code log-authpriv -might not be available. -For portability use code like -.code "(or (symbol-value 'log-authpriv) 0)" -to evaluate to 0 if -.code log-authpriv -doesn't exist, or else check for its existence -using -.codn "(boundp 'log-authpriv)" . +The +.code out-args +slot holds the tail of the argument list consisting of the non-option +arguments. -.coNP Variables @, log-emerg @, log-alert @, log-crit @, log-err @, log-warning @, log-notice @ log-info and @ log-debug -.desc -These variables take on the values of the corresponding C preprocessor -constants from the -.code -header: -.codn LOG_EMERG , -.codn LOG_ALERT , -etc. -These are the integer priority codes specified in the -.code syslog +The mechanism by means of which +.code out-args +is calculated, and by means of which the information about the +options is populated, is unspecified. The only interface to that +mechanism is the +.code getopts function. -.coNP The @ *stdlog* special variable -.desc The -.code *stdlog* -variable holds a special kind of stream: a syslog stream. Each -newline-terminated line of text sent to this stream becomes a log message. +.code opts +object supports indexing, including indexed assignment. -The stream internally maintains a priority value that is applied -when it generates messages. By default, this value is that of -.codn log-info . -The stream holds the priority as the value of the -.code :prio -stream property, which may be changed with the -.code stream-set-prop -function. +If +.code o +is an instance of +.code opts +returned by +.codn getopts , +then the expression +.code "[o \(dqv\(dq]" +tests whether the option +.str v +is available in +.codn o ; +that is, whether it has been specified in the command line. +If so, then its associated value is returned, otherwise +.code nil +is returned. This +.code nil +is ambiguous: for a Boolean option it indicates that either +the option was not specified, or that it was explicitly +specified as false. For a Boolean option that was specified +(positively), the value +.code t +is returned. -The latest priority value which has been configured on the stream is used -at the time the newline character is processed and the log message -is generated, not necessarily the value which was in effect at the time the -accumulation of a line began to take place. +The expression +.code "[o \(dqv\(dq dfl]" +yields the value of option +.str v +if that option has been specified. If the option hasn't +been specified, then the expression yields the value +.codn dfl . -Messages sent to -.code *stdlog* -are delimited by newline characters. That is to say, each line of -text written to the stream is a new log. +Assigning to +.code "[o \(dqv\(dq]" +is possible. This replaces the value associated with option +.strn v . +The assignment is erroneous if no such option was parsed +from the command line, even if it is a valid option. -.coNP Function @ openlog +.coNP Function @ getopts .synb -.mets (openlog < id-string >> [ options <> [ facility ]]) +.mets (getopts < option-desc-list << arg-list ) .syne .desc The -.code openlog -function is a wrapper for the -.code openlog -C function, and the -arguments have the same semantics. It is not necessary to use -.code openlog -in order -to call the -.code syslog -function or to write data to -.codn *stdlog* . -The call is necessary in order to override the default identifying string, to -set options, such as having the PID (process ID) recorded in log messages, and -to specify the facility. +.code getopts +function takes a list of +.code opt-desc +structures and a list of strings +.meta arg-list +representing command line arguments. The -.meta id-string -argument is mandatory. +.meta arg-list +is parsed. If the parse is unsuccessful, an exception of type +.code opt-error +is thrown, derived from +.codn error . -The -.meta options -argument is a bitwise mask (see the logior function) of option -values such as -.code log-pid -and -.codn log-cons . -If it is missing, then a value of 0 is -used, specifying the absence of any options. +If there are problems in +.code option-desc-list +itself, then an exception of type +.code error +is thrown. -The -.meta facility -argument is one of the values -.codn log-user , -.code log-daemon -or -.codn log-auth . -If it is missing, then -.code log-user -is assumed. +If the parse is successful, +.code getopts +returns an instance of the +.code opts +structure describing the parsed opts, and listing the non-option +arguments. -.coNP Function @ closelog +.coNP Function @ opthelp .synb -.mets (closelog) +.mets (opthelp < opt-desc-list <> [ stream ]) .syne .desc The -.code closelog -function is a wrapper for the C function -.codn closelog . +.code opthelp +function processes the list of +.code opt-desc +structures +.meta opt-desc-list +and compiles a customized body of help text describing all of the +options, as well as general description of the command line option +conventions to guide the user in in the correct use of command +line options. -.coNP Function @ setlogmask +The text is formatted to fit within 79 columns, and begins and ends with a +blank line. Its format consists of headings which begin in the first column, +and paragraphs and tables which feature a two space left margin. +A blank line follows each section heading. The heading begins with a capital +letter. Its remaining words are uncapitalized, and it ends with a colon. + +The text is sent to +.metn stream , +if specified. This argument defaults to +.codn *stdout* . + +If there are problems in +.code option-desc-list +itself, then an exception of type +.code error +is thrown. + +.SS* System Programming +.coNP Accessor @ errno .synb -.mets (setlogmask << bitmask-integer ) +.mets (errno <> [ new-errno ]) +.mets (set (errno) << new-value ) .syne .desc The -.code setlogmask -function interfaces to the corresponding C function, and has the -same argument and return value semantics. The -.meta bitmask-integer -argument is a mask of priority -values to enable. The return value is the prior value. Note that if the -argument is zero, then the function doesn't set the mask to zero; it only -returns the current value of the mask. - -Note that the priority values like -.code log-emerg -and -.code log-debug -are integer -enumerations, not bitmasks. These values cannot be combined directly to create -a bitmask. Rather, the -.code mask -function should be used on these values. - -.TP* Example: +.code errno +function retrieves the current value of the C library error variable +.codn errno . +If the argument +.meta new-errno +is present and is not +.codn nil , +then it +specifies a value which is stored into +.codn errno . +The value returned is the prior value. -.verb - ;; Enable LOG_EMERG and LOG_ALERT messages, - ;; suppressing all others - (setlogmask (mask log-emerg log-alert)) -.brev +The place form of +.code errno +does not take an argument. -.coNP Function @ syslog +.coNP Function @ exit .synb -.mets (syslog < priority < format << format-arg *) +.mets (exit << status ) .syne .desc The -.code syslog -function is the interface to the -.code syslog -C function. The -.code printf -formatting capabilities of the function are not used; -the -.meta format -argument follows the conventions of the \*(TL -.code format -function instead. Note in particular that -the -.code %m -convention for interpolating the value of strerror(errno) which is -available in some versions of the -.code syslog -C function is currently not supported. - -Note that syslog messages are not newline-terminated. - -.SS* Unix Path Globbing - -On platforms where the POSIX -.code glob -function is available \*(TX provides this functionality in -the form of a like-named function, and some numeric constants. -\*(TX also provides access the -.code fnmatch -function, where available. +.code exit +function terminates the entire process (running \*(TX image), specifying +the termination status to the operating system. Values of +.meta status +may be +.codn nil , +.codn t , +or an integer value. The value +.code nil +corresponds to the C constant +.codn EXIT_FAILURE , +and +.code t +corresponds to +.codn EXIT_SUCCESS . +These are platform-independent +indicators of failed or successful termination. The numeric value 0 also +indicates success. -.coNP Variables @, glob-err @, glob-mark @, glob-nosort @, glob-nocheck @, glob-noescape @, glob-period @, glob-altdirfunc @, glob-brace @, glob-nomagic @, glob-tilde @ glob-tilde-check and @ glob-onlydir +.coNP Variables @, e2big @, eacces @, eaddrinuse @, eaddrnotavail @, eafnosupport @, eagain @, ealready @, ebadf @, ebadmsg @, ebusy @, ecanceled @, echild @, econnaborted @, econnrefused @, econnreset @, edeadlk @, edestaddrreq @, edom @, edquot @, eexist @, efault @, efbig @, ehostunreach @, eidrm @, eilseq @, einprogress @, eintr @, einval @, eio @, eisconn @, eisdir @, eloop @, emfile @, emlink @, emsgsize @, emultihop @, enametoolong @, enetdown @, enetreset @, enetunreach @, enfile @, enobufs @, enodata @, enodev @, enoent @, enoexec @, enolck @, enolink @, enomem @, enomsg @, enoprotoopt @, enospc @, enosr @, enostr @, enosys @, enotconn @, enotdir @, enotempty @, enotrecoverable @, enotsock @, enotsup @, enotty @, enxio @, eopnotsupp @, eoverflow @, eownerdead @, eperm @, epipe @, eproto @, eprotonosupport @, eprototype @, erange @, erofs @, espipe @, esrch @, estale @, etime @, etimedout @, etxtbsy @ ewouldblock and @ exdev .desc -These variables take on the values of the corresponding C preprocessor -constants from the -.code -header: -.codn GLOB_ERR , -.codn GLOB_MARK , -.codn GLOB_NOSORT , -etc. - -These values are passed as the optional second argument of the -.code glob -function. They are bitmasks and so multiple values can be combined -using the -.code logior -function. - -Note that the -.codn glob-period , -.codn glob-altdirfunc , -.codn glob-brace , -.codn glob-nomagic , -.codn glob-tilde , -.code glob-tilde-check +These variables correspond to the POSIX +.cod2 \(dq errno +constants\(dq, namely +.codn E2BIG , +.codn EACCES , +.code EADDRINUSE +and so forth. +Variables corresponding to all of the +.code "" +constants from the Issue 6 2004 edition of POSIX are included. +The variables +.code eownerdead and -.code glob-onlydir -variables may not be available. They are extensions in the GNU C library -implementation of -.codn glob . +.code enotrecoverable +from Issue 7 2018 are subject to the availability of the corresponding constants +in the host platform. -.coNP Function @ glob +.coNP Function @ abort .synb -.mets (glob < pattern >> [ flags <> [ error-func ]]) +.mets (abort) .syne .desc The -.code glob -function is a interface to the Unix function of the same name. -The -.meta pattern -argument must be a string, which holds a glob pattern: a pattern which -matches zero or more path names, similar to a regular expression. -The function tries to expand the pattern and return a list of strings -representing the matching path names in the file system. +.code abort +function terminates the entire process (running \*(TX image), specifying +an abnormal termination status to the process. -If there are no matches, then an empty list is returned. +Note: +.code abort +calls the C library function +.code abort +which works by raising the +.code SIG_ABRT +signal, known in \*(TX as the +.code sig-abrt +variable. Abnormal termination of the process is this signal's +default action. -The optional -.meta flags -argument defaults to zero. If given, it may be a bitwise combination of the -values of the variables -.codn glob-err , -.codn glob-mark , -.code glob-nosort -and others. +.coNP Functions @ at-exit-call and @ at-exit-do-not-call +.synb +.mets (at-exit-call << function ) +.mets (at-exit-do-not-call << function ) +.syne +.desc +The +.code at-exit-call +function registers +.meta function +to be called when the process terminates normally. +Multiple functions can be registered, and the same function +can be registered more than once. The registered +functions are called in reverse order of their +registrations. -If the -.meta error-func -argument is specified, it gives a callback function which is invoked -when -.code glob -encounters errors accessing paths. The function takes two arguments: -the pathname and the -.code errno -value which occurred for that pathname. The function's return value is -Boolean. If the function returns true, then -.code glob -will terminate. +The +.code at-exit-do-not-call +function removes all previous +.code at-exit-call +registrations of +.metn function . The -.meta error-func -may terminate the traversal by a nonlocal exit, such as by throwing -an exception or performing a block return. +.code at-exit-call +function returns +.metn function . The -.meta error-func -may not re-enter the -.code glob -function. This situation is detected and diagnosed by an exception. +.code at-exit-do-not-call +function returns +.code t +if it removed anything, +.code nil +if no registrations of +.meta function +were found. +.coNP Function @ usleep +.synb +.mets (usleep << usec ) +.syne +.desc The -.meta error-func -may not capture a continuation across the error boundary. That is to say, -code invoked from the error may not capture a continuation up to a prompt -which surrounds the -.code glob -call. Such an attempt is detected and diagnosed by an exception. +.code usleep +function suspends the execution of the program for at least +.meta usec +microseconds. -Details of the semantics of the -.code glob -function, and the meaning of all the -.meta flags -arguments are given in the documentation for the C function. +The return value is +.code t +if the sleep was successfully executed. A +.code nil +value indicates premature wakeup or complete failure. -.coNP Variables @, fnm-pathname @, fnm-noescape @, fnm-period @, fnm-leading-dir @ fnm-casefold and @ fnm-extmatch -.desc -These variables take on the values of the corresponding C preprocessor -constants from the -.code -header: -.codn FNM_PATHNAME , -.codn FNM_NOESCAPE , -.codn FNM_PERIOD , -etc. +Note: the actual sleep resolution is not guaranteed, and depends on granularity +of the system timer. Actual sleep times may be rounded up to the nearest 10 +millisecond multiple on a system where timed suspensions are triggered by a 100 +Hz tick. -These values are bit masks which may be combined with the -.code logior -function to form the optional third -.meta flags -argument of the -.code fnmatch +.coNP Functions @ mkdir and @ ensure-dir +.synb +.mets (mkdir < path <> [ mode ]) +.mets (ensure-dir < path <> [ mode ]) +.syne +.desc +.code mkdir +tries to create the directory named +.meta path +using the POSIX +.code mkdir function. +An exception of type +.code file-error +is thrown if the function fails. Returns +.code t +on success. -Note that the -.codn fnm-leading-dir , -.code fnm-case-fold -and -.code fnm-extmatch -may not be available. They are GNU extensions, found in the GNU C library. +The +.meta mode +argument specifies the request numeric permissions +for the newly created directory. If omitted, the requested permissions are +.code #o777 +(511): readable and writable to everyone. The requested permissions +are subject to the system +.codn umask . -.coNP Function @ fnmatch +The function +.code ensure-dir +is similar to +.code mkdir +except that it attempts to create all the missing parent directories +as necessary, and does not throw an error if the directory exists. + +.coNP Function @ chdir .synb -.mets (fnmatch < pattern < string <> [ flags ]]) +.mets (chdir << path ) +.syne +.desc +.code chdir +changes the current working directory to +.metn path , +and returns +.metn t , +or else throws an exception of type +.codn file-error . + +.coNP Function @ pwd +.synb +.mets (pwd) .syne .desc The -.code fnmatch -function, if available, provides access -to the like-named POSIX C library function. +.code pwd +function retrieves the current working directory. +If the underlying +.code getcwd +C library function fails with an +.code errno +other than +.codn ERANGE , +an exception will be thrown. + +.coNP Function @ remove-path +.synb +.mets (remove-path < path <> [ throw-on-error-p ]) +.syne +.desc The -.meta pattern -argument specifies a POSIX-shell-style file pattern matching expression. -Its exact features and dialect are controlled by -.metn flags . -If -.meta string -matches -.meta pattern -then -.code t -is returned. If there is no match, then -.code nil -is returned. If the C function indicates that an error has occurred, -an exception is thrown. +.code remove-path +function tries to remove the filesystem object named +by +.metn path , +which may be a file, directory or something else. -.SS* Unix Filesystem Traversal +If successful, it returns +.codn t . -On platforms where the POSIX -.code nftw -function is available \*(TX provides this functionality in -the form of the analogous Lisp function -.codn ftw , -accompanied by some numeric constants. +The optional Boolean parameter +.metn throw-on-error-p , +which defaults to +.codn nil . -.coNP Variables @, ftw-phys @, ftw-mount @, ftw-chdir @ ftw-depth and @ ftw-actionretval +A failure to remove the object results in an exception of type +.code file-error +being thrown, unless the failure reason is that the object indicated by +.meta path +doesn't exist. In this non-existence case, the behavior is controlled by the +.meta throw-on-error +argument. If that argument is true, the exception is thrown. Otherwise, +the function returns normally, producing the value +.code nil +to indicate that it didn't perform a removal. + +.coNP Function @ rename-path +.synb +.mets (rename-path < from-path << to-path ) +.syne .desc -These variables hold numeric values that may be combined into a single -bitmask bitmask value using the -.code logior -function. This value is suitable as the -.meta flags -argument of the -.code ftw -function. +The +.code remove-path +function tries to rename filesystem path +.metn from-path , +which may refer to a file, directory or something else, to the path +.metn to-path . -These variables corresponds to the C constants -.codn FTW_PHYS , -.codn FTW_MOUNT , -et cetera. +If successful, it returns +.codn t . -Note that -.code ftw-actionretval -is a GNU extension that is not available on all platforms. If the platform's -.code nftw -function doesn't have this feature, then this variable is not defined. +A failure results in an exception of type +.codn file-error . -.coNP Variables @, ftw-f @, ftw-d @, ftw-dnr @, ftw-ns @, ftw-sl @ ftw-dp and @ ftw-sln +.coNP Functions @ sh and @ run +.synb +.mets (sh << system-command ) +.mets (run < program <> [ argument-list ]) +.syne .desc -These variables provide symbolic names for the integer values that are -passed as the -.code type -argument of the callback function called by -.codn ftw . -This argument classifies the kind of file system node visited, or -error condition encountered. +The +.code sh +function executes +.meta system-command +using the system command interpreter. +The run function spawns a +.metn program , +searching for it using the +system PATH. Using either method, the executed process receives environment +variables from the parent. -These variables correspond to the C constants -.codn FTW_F , -.codn FTW_D , -et cetera. +\*(TX blocks until the process finishes executing. If the program terminates +normally, then its integer exit status is returned. The value zero indicates +successful termination. -Not all of them are present. If the underlying platform doesn't have -a given constant, then the corresponding variable doesn't exist in \*(TX. +The return value +.code nil +indicates an abnormal termination, or the inability +to run the process at all. -.coNP Variables @, ftw-continue @, ftw-stop @ ftw-skip-subtree and @ ftw-skip-siblings -.desc -These variables are defined if the variable -.code ftw-actionretval -is defined. +In the case of the +.code run +function, if the child process is created successfully +but the program cannot be executed, then the exit status will be an +.code errno +value from the failed +.code exec +attempt. -If the value of -.code ftw-actionretval -is included in the -.meta flags -argument of -.codn ftw , -then the callback function can use the values of these variables -as return codes. Ordinarily, the callback returns zero to continue -the search and nonzero to stop. +The standard input, output and error file descriptors of an executed +command are obtained from the streams stored in the +.codn *stdin* , +.code *stdout* +and +.code *stderr* +special variables, respectively. For a detailed description of the +behavior and restrictions, see the +.code open-command +function, whose description of this mechanism applies to the +.code run +and +.code sh +function also. -These variables correspond to the C constants -.codn FTW_CONTINUE , -.codn FTW_STOP , -et cetera. +Note: as of \*(TX 120, the +.code sh +function is implemented using +.code run +and not by means of the +.code system +C library function, as previously. The +.code run +function is used to invoke the system interpreter by name. On Unix-like +systems, the string +.code /bin/sh +is assumed to denote the system interpreter, which is expected to +support a pair of arguments +.mono +.meti -c < command +.onom +to specify the command to be executed. On MS Windows, the interpreter +is assumed to be the relative path name +.code cmd.exe +and expected to support +.mono +.meti /C < command +.onom +as a way of specifying a command to execute. -.coNP Function @ ftw +.SS* Unix Filesystem Manipulation + +.coNP Structure @ stat .synb -.mets (ftw < path-or-list < callback-func >> [ flags <> [ nopenfd ]]) -.mets >> [ callback-func < path < type < stat-struct < level << base ] +.mets (defstruct stat nil +.mets \ \ dev ino mod nlink uid gid +.mets \ \ rdev size blksize blocks atime +.mets \ \ mtime ctime path) .syne .desc The -.code ftw -function provides access to the -.code nftw -POSIX C library function. - -Note that the -.meta flags +.code stat +structure defines the type of object which is returned +by the +.codn stat , +.codn lstat , and -.meta nopenfd -arguments are reversed with respect to the C language interface. -They are both optional; -.meta flags -defaults to zero, and -.meta nopenfd -defaults to 20. - +.code fstat +functions. Except for +.codn path , +the slots are the direct counterparts of the +members of POSIX C structure +.codn "struct stat" . +For instance the slot +.code dev +corresponds to +.codn st_dev . The -.meta path-or-list -argument may be a string specifying the top-level path name that -.code ftw -shall visit. Or else, -.meta path-or-list -may be a list. If it is a list, then -.code ftw -recursively invokes itself over each of the elements, taking -that element as the -.meta path-or-name -argument of the recursive call, passing down all other argument -values as-is. -The traversal stops when any recursive invocation of -.code ftw -returns a value other than -.code t -or -.codn nil , -and that value is returned. If -.code t -or -.code nil -is returned, the traversal continues with the -application of -.code ftw -to the next list element, if any. -If the list is completely traversed, and some recursive -invocations of -.code ftw -return -.codn t , -then the return value is -.codn t . -If all recursive invocations return -.code nil -then +.code path +slot is set by the functions +.code stat +and +.codn lstat . +Its value is .code nil -is returned. -If the list is empty, -.code t -is returned. +when the path is not available. +.coNP Functions @, stat @ lstat and @ fstat +.synb +.mets (stat << path ) +.mets (lstat << path ) +.mets (fstat << stream ) +.syne +.desc The -.code ftw -function walks the filesystem, as directed by the -.meta path-or-list -argument and -.meta flags -bitmask arguments. +.code stat +function inquires the filesystem about the existence of an object +denoted by the string +.metn path . +If the object is not found or cannot be +accessed, an exception is thrown. -For each visited entry, it calls the supplied -.meta callback-func -function, which receives five arguments. +Otherwise, information is retrieved about the object. The information takes +the form of a structure of type +.codn stat . -The -.code ftw -function can continue the traversal by returning any non-integer value, -or the integer value zero. If -.code ftw-actionretval -is included in the -.meta flags -bitmask, then the only integer code which continues the traversal without -any special semantics is -.code ftw-continue -and only -.code ftw-stop -stops the traversal. (Non-integer return values behave like -.codn ftw-continue ). - -The .meta path -argument of the callback function gives the path of the -visited filesystem object. - -The -.meta type -argument is an integer code which indicates the kind of -object that is visited, or an error situation in visiting -that filesystem entry. See the documentation for -.code ftw-f -and -.code ftw-d -for possible values. +refers to a symbolic link, the +.code stat +function retrieves information about the target of the link, if it exists, +or else throws an exception of type +.codn file-error . The -.meta stat-struct -argument provides information about the filesystem object -as a -.code stat -structure, the same kind of object as what is returned by the +.code lstat +function behaves the same as .code stat -function. +or objects which are not symbolic links. For a symbolic link, it retrieves +information about the link itself, rather than its target. The -.meta level -argument is an integer value representing the directory level -depth. This value is obtained from the C structure -.code FTW -in the -.code nftw -C API. +.code fstat +function retrieves information about the file system object associated with +the open stream +.metn stream . +The stream must be of a kind from which the +.code fileno +function can retrieve a file descriptor, otherwise an exception of type +.code file-error +is thrown. The -.meta base -argument indicates the length of the directory part of the .code path -argument. Characters in excess of this length are thus the base name of the -visited object, and the expression -.mono -.meti >> [ path << base ..:] -.onom -calculates the base name. +slot of the returned structure +holds a copy of their +.meta path +argument value. +In the case of +.codn fstat , +this slot is +.codn nil . -The -.code ftw -function returns -.code t -upon successful completion and -.code nil -on failure. If -.code ftw -is terminated by a return value from -.metn callback-func , -then that value is returned. Such a value is always a nonzero integer. +.coNP Variables @, s-ifmt @, s-iflnk @, s-ifreg @, s-ifblk ... , @ s-ixoth +.desc +The following variables exist, having integer values. These are bitmasks +which can be applied against the value given by the +.code mode +slot of the +.code stat +structure returned by the function +.codn stat : +.codn s-ifmt , +.codn s-ifsock , +.codn s-iflnk , +.codn s-ifreg , +.codn s-ifblk , +.codn s-ifdir , +.codn s-ifchr , +.codn s-ififo , +.codn s-isuid , +.codn s-isgid , +.codn s-isvtx , +.codn s-irwxu , +.codn s-irusr , +.codn s-iwusr , +.codn s-ixusr , +.codn s-irwxg , +.codn s-irgrp , +.codn s-iwgrp , +.codn s-ixgrp , +.codn s-irwxo , +.codn s-iroth , +.code s-iwoth +and +.codn s-ixoth . -The -.meta callback-func -may terminate the traversal by a nonlocal exit, such as by throwing -an exception or performing a block return. +These variables correspond to the C language constants from POSIX: +.codn S_IFMT , +.codn S_IFLNK , +.code S_IFREG +and so forth. The -.meta callback-func -may not re-enter the -.code ftw -function. This situation is detected and diagnosed by an exception. +.code logtest +function can be used to test these against values of mode. +For example +.code "(logtest mode s-irgrp)" +tests for the group read permission. +.coNP Function @ umask +.synb +.mets (umask <> [ mask ]) +.syne +.desc The -.meta callback-func -may not capture a continuation across the callback boundary. That is to say, -code invoked from the callback may not capture a continuation up to a prompt -which surrounds the -.code ftw -call. Such an attempt is detected and diagnosed by an exception. +.code umask +function provides access to the Unix C library function of the same name, +which controls which permissions are denied +when files are newly created. -.SS* Unix Sockets +If +.code umask +is called with no argument, it returns the current value of the mask. -On platforms where the underlying system interface is available, \*(TX provides -a sockets library for communicating over Internet networks, or over Unix -sockets. +If the +.meta mask +argument is present, it must be an integer specifying the new mask to be +installed. The previous mask is returned. -Stream as well as datagram sockets are supported. +If +.meta mask +is absent, then +.code umask +returns the previous mask. -The classic Version 4 of the Internet protocol is supported, as well -as IP Version 6. +Note: the value of the +.meta mask +argument may be calculated as a bitwise or of the following constants: +.codn s-irwxu , +.codn s-irusr , +.codn s-iwusr , +.codn s-ixusr , +.codn s-irwxg , +.codn s-irgrp , +.codn s-iwgrp , +.codn s-ixgrp , +.codn s-irwxo , +.codn s-iroth , +.code s-iwoth +and +.codn s-ixoth , +which correspond to the POSIX C constants +.codn S_IRWXU , +.codn S_IRUSR , +.codn S_IWUSR , +.codn S_IXUSR , +.codn S_IRWXG , +.codn S_IRGRP , +.codn S_IWGRP , +.codn S_IXGRP , +.codn S_IRWXO , +.codn S_IROTH , +.code S_IWOTH +and +.codn S_IXOTH . -Sockets are mapped to \*(TX streams. The -.code open-socket -function creates a socket of a specified type, in a particular address family. -This socket is actually a stream (always, even if it isn't used for -data transfer, but only as a passive contact point). +Implementation note: since the +.code umask +C library function provides no way to retrieve the current mask without +overwriting with a new one, the \*(TX +.code umask +function, when given no argument, simulates the pure retrieval of the mask +by calling the C function with an argument of +.code #o777 +to temporarily install the maximally safe mask. The value returned is then +reinstated as the mask by another call to +.codn umask , +and that value is also returned. -The functions -.codn sock-connect , -.codn sock-bind , -.codn sock-listen , -.code sock-accept +.coNP Functions @, makedev @ minor and @ major +.synb +.mets (makedev < minor << major ) +.mets (minor << dev ) +.mets (major << dev ) +.syne +.desc +The parameters +.metn minor , +.meta major and -.code sock-shutdown -are used for enacting socket communication scenarios. - -Stream sockets use ordinary streams, re-using the same underlying framework -that is used for file I/O and process types. +.meta dev +are all integers. The +.code makedev +function constructs a combined device number from a minor and major pair (by +calling the Unix +.code makedev +function). This device number is suitable as an +argument to the +.code mknod +function (see below). Device numbers also appear as values of the +.code dev +slot of the +.code stat +structure. -Datagram socket streams are implemented using special datagram socket streams. -Datagram socket streams eliminate the need for operations analogous to the -.code sendto +The +.code minor and -.code recvfrom -socket API functions, even in server programs which handle multiple -clients. An overview of datagrams is treated in the following section, -Datagram Socket Streams. +.code major +functions extract the minor and major device number +from a combined device number. +.coNP Function @ chmod +.synb +.mets (chmod < path << mode ) +.syne +.desc The -.code getaddrinfo -function is provided for resolving host names and services to IPv4 and IPv6 -addresses. +.code chmod +function changes the permissions of the filesystem objects +specified by +.metn path . +It is a direct wrapper for the POSIX C library function of the same name. -Several structure types are provided for representing socket addresses, -and options for -.codn getaddrinfo . +The permissions are specified by +.metn mode , +an integer argument. -Various numeric constants are also provided: -.codn af-unix , -.codn af-inet , -.codn af-inet6 , -.codn sock-stream , -.code sock-dgram -and others. +The existing permissions may be obtained using the +.code stat +function. -.NP* Datagram Socket Streams +The function throws a +.code file-error +exception if an error occurs, otherwise it returns +.codn t . -Datagram socket streams are a new paradigm unique to \*(TX which -attempts to unify the programming model of stream and datagram -sockets. +.TP* Example: +.verb + ;; Set permissions of foo.txt to "rw-r--r--" + ;; (owner can read and write; group owner + ;; and other users can only read). -A datagram socket stream is created by the -.code open-socket -function, when the -.code sock-dgram -socket type is specified. Another way in which a datagram socket -is created is when -.code sock-accept -is invoked on a datagram socket, and returns a new socket. + ;; numerically: + (chmod "foo.txt" #o644) -I/O is performed on datagram sockets using the regular I/O functions. -None of the functions take or return peer addresses. There are no I/O -functions which are analogous to the C library -.code recvfrom -and -.code sendto -functions which are usually used for datagram programming. -Datagram I/O assumes that the datagram datagram socket is connected to a -specific remote peer, and that peer is implicitly used for all I/O. + ;; symbolically: + (chmod "foo.txt" (logior s-irusr s-iwusr + s-irgrp + s-iroth)) +.brev -Datagram streams solves the message framing problem essentially by -considering a single datagram to be an entire stream. On input, a datagram -stream holds an entire datagram in a buffer. The stream ends -(experiences the EOF condition) after the last byte of this buffer -is removed by an input operation. Another datagram will be received and -buffered if the EOF condition is first explicitly cleared with the -.code clear-error -function, and then another input operation is attempted. -On output, a datagram stream gathers data into an ever-growing output buffer -which isn't subject to any automatic flushing. An explicit -.code flush-stream -operation sends the buffer contents to the connected peer as a new -datagram, and empties the buffer. Subsequent output operations prepare -data for a new datagram. The -.code close-stream -function implicitly flushes the stream in the same way, and thus also -potentially generates a datagram. +.coNP Function @ mknod +.synb +.mets (mknod < path < mode <> [ dev ]) +.syne +.desc +The +.code mknod +function tries to create an entry in the filesystem: a file, +FIFO, or a device special file, under the name +.metn path . +If it is successful, +it returns +.codn t , +otherwise it throws an exception of type +.codn file-error . -A client-style datagram stream can be explicitly connected to a peer with the -.code sock-connect -function. This is equivalent to connecting a -datagram socket using the C library -.code connect -function. Writes on the stream will be transmitted using the C library function -.codn send . -A client-style datagram stream can also be "soft-connected" to a -peer using the -.code sock-set-peer -function. Writes on the stream will transmit data using the C library function -.code sendto -to the peer address. +The +.meta mode +argument is a bitwise or combination of the requested permissions, +and the type of object to create: one of the constants +.codn s-ifreg , +.codn s-ififo , +.codn s-ifchr , +.code s-ifblk +or +.codn s-ifsock . +The permissions are subject to the system +.codn umask . -A datagram server program which needs -to communicate using multiple peers is implemented by means of the -.code sock-accept -function which, unlike the C library -.code accept -function, works with datagram sockets as well as stream sockets. -The server creates a datagram socket, and uses -.code sock-bind -to bind it to a local address. Optionally, it may also call -.code sock-listen -which is a no-op on datagram sockets. Supporting this function on datagram -sockets allows program code to be more easily changed between datagram and -stream operation. -The server then uses -.code sock-accept -to accept new clients. Note that this is not possible with the C -library function -.codn accept , -which only works with stream sockets. +If a block or character special device +.cod2 ( s-ifchr +or +.codn s-ifblk ) +is being +created, then the +.meta dev +argument specifies the major and minor numbers +of the device. A suitable value can be constructed from a major and minor +pair using the +.code makedev +function. + +.TP* Example: + +.verb + ;; make a character device (8, 3) called /dev/foo + ;; requesting rwx------ permissions + + (mknod "dev/foo" (logior #o700 s-ifchr) (makedev 8 3)) +.brev +.coNP Functions @ symlink and @ link +.synb +.mets (symlink < target << path ) +.mets (link < target << path ) +.syne +.desc The -.code sock-accept -function receives a datagram from a client, and creates a new datagram -socket stream which is connected to that client, and whose input buffer -contains the received datagram. Input operations on this stream consume -the datagram. Note that clearing the EOF condition and trying to receive -another datagram is not recommended on datagram streams returned -by -.codn sock-accept , -since they share the same underlying operating system socket, which is -not actually connected to a specific peer. The receive operation could -receive a datagram from any peer, without any indication which peer that is. -Datagram servers should issue a new -.code sock-accept -call should be issued for each client datagram, treating it as a new -stream. +.code symlink +function creates a symbolic link called +.meta path +whose contents +are the absolute or relative path +.metn target . +.meta target +does not actually have to exist. -Datagram sockets ignore almost all aspects of the -.meta mode-string -passed in -.code open-socket -and -.codn sock-accept . -The only attribute not ignored is the buffer size specified -with a decimal digit character; however, it cannot be the -only item in the mode string. The string must be syntactically -valid, as described under the -.code open-file -function. The buffer size attribute controls the size used by -the datagram socket for receiving a datagram: the capture size. -A datagram socket has obtains a default capture size if one isn't -specified by the -.metn mode-string . -The default capture size is 65536 bytes for a datagram socket created by -.codn open-socket . -If a size is not passed to -.code sock-accept -via its -.meta mode-string -argument when it is invoked on a datagram socket, -that socket's size is used as the capture size of the -newly created datagram socket which is returned. +The link function creates a hard link. The object at +.meta target +is installed +into the filesystem at +.meta path +also. + +If these functions succeed, they return +.codn t . +Otherwise they throw an exception +of type +.codn file-error . -.coNP Structure @ sockaddr +.coNP Function @ readlink .synb -.mets (defstruct sockaddr nil -.mets \ \ (:static family nil)) +.mets (readlink << path ) .syne .desc -The -.code sockaddr -structure represents the abstract base class for socket addresses, from which -several other types are derived: -.codn sockaddr-in , -.code sockaddr-in6 -and -.codn sockaddr-un . - -It has a single slot called -.code family -which is static, and initialized to -.codn nil . +If +.meta path +names a filesystem object which is a symbolic link, the +.code readlink +function reads the contents of that symbolic link and returns it +as a string. Otherwise, it fails by throwing an exception of type +.codn file-error . -.coNP Structure @ sockaddr-in +.coNP Function @ realpath .synb -.mets (defstruct sockaddr-in sockaddr -.mets \ \ (addr 0) (port 0) -.mets \ \ (:static family af-inet)) +.mets (realpath << path ) .syne .desc The -.code sockaddr-in -address represents a socket address used in the context of networking over -IP Version 4. It may be used with sockets in the -.code af-inet -address family. +.code realpath +function provides access to the same-named POSIX function. +It processes the input string +.meta path +by expanding all symbolic links, removes all superfluous +.str ".." +and +.str "." +path components, and extra path-separating slash characters, +to produce a canonical absolute path name. -The -.code addr -slot holds an integer denoting an abstract IPv4 address. For instance the hexadecimal -integer literal constant -.code #x7F000001 -or its decimal equivalent -.code 2130706433 -represents the loopback address, whose familiar "dot notation" is -.codn 127.0.0.1 . -Conversion of the abstract IP address to four bytes in network order, as -required, is handled internally. +If the underlying POSIX function indicates failure, then +.code nil +is returned. In that situation the +.code errno +value is available using the +.code errno +function. -The -.code port -slot holds the TCP or UDP port number, whose value ranges from 0 to 65535. -Zero isn't a valid port; the value is used for requesting an ephemeral port number -in active connections. Zero also appears in situations when the port number isn't required: -for instance, when the -.code getaddrinfo -function is used with the aim of looking up the address of a host, without -caring about the port number. +.SS* Unix Filesystem Object Existence, Type and Access Tests -The -.code family -static slot holds the value -.codn af-inet . +The following functions all accept, as the +.meta path +argument, either a character string, or a structure returned by the +.code stat +or +.code lstat +functions. -.coNP Structure @ sockaddr-in6 +If the +.meta path +argument is a string, then +.code stat +is used to retrieve information about it, except in the case of +the +.code path-symlink-p +function, which uses +.codn lstat . +The subsequent test is then based on the result of this call. + +If the +.meta path +argument is the result of a +.code stat +or +.code lstat +call, then the testing is based on that object. + +Some of the accessibility tests (functions which determine whether the +calling process has certain access rights) may not be perfectly accurate, since +they are based strictly on portable information available via +.codn stat , +together with the basic, portable POSIX APIs for inquiring about +security credentials, such as +.codn geteuid . +They ignoring any special permissions which may exist such as operating system +and file system specific extended attributes (for example, file immutability +connected to a "secure level" and such) and special process capabilities +not reflected in the basic credentials. + +.coNP Function @ path-exists-p .synb -.mets (defstruct sockaddr-in6 sockaddr -.mets \ \ (addr 0) (port 0) (flow-info 0) (scope-id 0) -.mets \ \ (:static family af-inet6)) +.mets (path-exists-p << path ) .syne .desc The -.code sockaddr-in6 -address represents a socket address used in the context of networking over -IP Version 6. It may be used with sockets in the -.code af-inet6 -address family. +.code path-exists-p +function returns +.code t +if +.meta path +is a string which resolves to a filesystem object. +Otherwise it returns +.codn nil . +If the +.meta path +names a dangling symbolic link, it is considered nonexistent. -The -.code addr -slot holds an integer denoting an abstract IPv6 address. IPv6 addresses are -pure binary integers up to 128 bits wide. +If +.meta path +is an object returned by +.code stat +or +.codn lstat , +.code path-exists-p +unconditionally returns +.codn t . -The -.code port -slot holds the TCP or UDP port number, whose value ranges from 0 to 65535. -In IPv6, the port number functions similarly to IPv6; see -.codn sockaddr-in . +.coNP Functions @, path-file-p @, path-dir-p @, path-symlink-p @, path-blkdev-p @, path-chrdev-p @ path-sock-p and @ path-pipe-p +.synb +.mets (path-file-p << path ) +.mets (path-dir-p << path ) +.mets (path-symlink-p << path ) +.mets (path-blkdev-p << path ) +.mets (path-chrdev-p << path ) +.mets (path-sock-p << path ) +.mets (path-pipe-p << path ) +.syne +.desc +.code path-file-p +tests whether +.meta path +exists and is a regular file. -The -.code flow-info -and -.code scope-id -are special IPv6 parameters corresponding to the -.code sin6_flowinfo -and -.code sin6_scope_id -slots of the -.code sockaddr_in6 -C language structure. Their meaning and use are beyond the scope of this document. +.code path-dir-p +tests whether +.meta path +exists and is a directory. -The -.code family -static slot holds the value -.codn af-inet . +.code path-symlink-p +tests whether +.meta path +exists and is a symbolic link. -.coNP Structure @ sockaddr-un +Similarly, +.code path-blkdev-p +tests for a block device, +.code path-chrdev-p +for a character device, +.code path-sock-p +for a socket and +.code path-pipe-p +for a named pipe. + +.coNP Functions @, path-setgid-p @ path-setuid-p and @ path-sticky-p .synb -.mets (defstruct sockaddr-un sockaddr -.mets \ \ path -.mets \ \ (:static family af-unix)) +.mets (path-setgid-p << path ) +.mets (path-setuid-p << path ) +.mets (path-sticky-p << path ) .syne .desc -The -.code sockaddr-un -address represents a socket address used for inter-process communication -within a single operating system node, using the "Unix domain" sockets -of the -.code af-unix -address family. - -This structure has only one slot, -.code path -which holds the rendezvous name for connecting pairs of socket endpoints. -This name appears in the filesystem. - -When the -.code sockaddr-un -structure is converted to the C structure -.codn "struct sockaddr_un" , -the -.code path -slot undergoes conversion to UTF-8. The resulting bytes are stored in the -.code sun_path -member of the C structure. If the resulting UTF-8 byte string -is larger than the -.code sun_path -array, it is silently truncated. +.code path-setgid-p +tests whether +.meta path +exists and has the set-group-ID permission set. -Note: Linux systems have support for "abstract" names which do not appear in -the filesystem. These abstract names are distinguished by starting with a null -byte. For more information, consult Linux documentation. -This convention is supported in the -.code path -slot of the -.code sockaddr-un -structure. If -.code path -contains occurrences of the pseudo-null character U+DC00, these translate -to null bytes in the -.code sun_path -member of the corresponding C structure -.codn "struct sockaddr_un" . -For example, the path -.str "\exDC00;foo" -is valid and represents an abstract address consisting of the three bytes -.str "foo" -followed by null padding bytes. +.code path-setuid-p +tests whether +.meta path +exists and has the set-user-ID permission set. -The -.code family -static slot holds the value -.codn af-inet . +.code path-sticky-p +tests whether +.meta path +exists and has the "sticky" permission bit set. -.coNP Structure @ addrinfo +.coNP Functions @ path-mine-p and @ path-my-group-p .synb -.mets (defstruct addrinfo nil -.mets \ \ (flags 0) (family 0) (socktype 0)) +.mets (path-mine-p << path ) +.mets (path-my-group-p << path ) .syne .desc -The -.code addrinfo -structure is used in conjunction with the -.code getaddrinfo -function. If that function's -.meta hints -argument is specified, it is of this type. -The purpose of the argument is to narrow down -or possibly alter the selection of addresses which -are returned. +.code path-mine-p +tests whether +.meta path +exists, and is effectively owned by the calling process; that is, +it has a user ID equal to the effective user ID of the process. -The -.code flags -slot holds a bitwise or combination (see the -.code logior -function) of -.code getaddrinfo -flags: values given by the variables. -.codn ai-passive , -.codn ai-numerichost , -.codn ai-v4mapped , -.codn ai-all , -.code ai-addrconfig -and -.codn ai-numericserv . -These correspond to the C constants -.codn AI_PASSIVE , -.code AI_NUMERICHOST -and so forth. +.code path-my-group-p +tests whether +.meta path +exists, and is effectively owned by a group to which the calling process +belongs. This means that the group owner is either the same as the +effective group ID of the calling process, or else is among the +supplementary group IDs of the calling process. -The -.code family -slot holds an address family, which may be the value of -.codn af-unspec , -.codn af-unix , -.code af-inet -or -.codn af-inet6 . +.coNP Function @ path-readable-to-me-p +.synb +.mets (path-readable-to-me-p << path ) +.syne +.desc +.code path-readable-to-me-p +tests whether the calling process can read the +object named by +.metn path . +If necessary, this test examines the effective user ID of the +calling process, the effective group ID, and the list of supplementary groups. -The -.code socktype -slot holds, a socket type. Socket types are given -by the variables -.code sock-dgram -and -.codn sock-stream . +.coNP Function @ path-writable-to-me-p +.synb +.mets (path-writable-to-me-p << path ) +.syne +.desc +.code path-writable-to-me-p +tests whether the calling process can write the +object named by +.metn path . +If necessary, this test examines the effective user ID of the +calling process, the effective group ID, and the list of supplementary groups. -.coNP Function @ getaddrinfo +.coNP Function @ path-read-writable-to-me-p .synb -.mets (getaddrinfo >> [ node >> [ service <> [ hints ]]]) +.mets (path-read-writable-to-me-p << path ) .syne .desc -The -.code getaddrinfo -returns a list of socket addresses based on search criteria expressed -in its arguments. -That is to say, the returned list, unless empty, contains objects of type -.code sockaddr-in -and -.codn sockaddr-in6 . +.code path-readable-to-me-p +tests whether the calling process can both read and write the +object named by +.metn path . +If necessary, this test examines the effective user ID of the +calling process, the effective group ID, and the list of supplementary groups. -The function is implemented directly in terms of the like-named C library -function. All parameters are optional. Omitting any argument causes a null -pointer to be passed for the corresponding parameter of the C library function. +.coNP Function @ path-executable-to-me-p +.synb +.mets (path-executable-to-me-p << path ) +.syne +.desc +.code path-executable-to-me-p +tests whether the calling process can execute the +object named by +.metn path , +or perform a search (name lookup, not implying sequential readability) on it, +if it is a directory. +If necessary, this test examines the effective user ID of the +calling process, the effective group ID, and the list of supplementary groups. +.coNP Functions @ path-private-to-me-p and @ path-strictly-private-to-me-p +.synb +.mets (path-private-to-me-p << path ) +.mets (path-strictly-private-to-me-p << path ) +.syne +.desc The -.meta node -and -.meta service -parameters may be character strings which specify a host name, and service. -The contents of these strings may be symbolic, like -.str www.example.com -and -.str ssh -or numeric, like -.str 10.13.1.5 +.code path-private-to-me-p and -.strn 80 . +.code path-strictly-private-to-me-p +functions report whether the calling process can rely on the +object indicated by +.code path +to be, respectively, private or strictly private to the security context +implied by its effective user ID. -If an argument is given for the -.code hints -parameter, it must be of type -.codn addrinfo . +"Private" means that beside the effective user ID of the calling process and +the superuser, no other user ID has write access to the object, and thus its +contents may be trusted to be be free from tampering by any other user. -The -.meta node -and -.meta service -parameters may also be given integer arguments. -An integer argument value in either of these parameters is converted to a null -pointer when calling the C -.code getaddrinfo -function. The integer values are then simply installed into every returned -address as the IP address or port number, respectively. However, if both -arguments are numeric, then no addresses are returned, since the C library -function is then called with a null node and service. +"Strictly private" means that not only is the object private, as above, +but users other than the effective user ID of the calling process +and superuser also not not have read access. -.coNP Variables @, af-unix @ af-inet and @ af-inet6 -.desc -These variables hold integers which give the values of address -families. They correspond to the C constants -.codn AF_UNIX , -.code AF_INET -and -.codn AF_INET6 . -Address family values are used in the -.meta hints -argument of the -.code getaddrinfo -function, and in the -.code socket-open -function. -Note that unlike the C language socket addressing structures, -the \*(TX socket addresses do not contain an address family slot. -That is because they indicate their family via their type. -That is to say, an object of type -.code sockaddr-in -is an address which is implicitly associated with the -.code af-inet -family via its type. +The rules which the function applies are as follows: -.coNP Variables @ sock-stream and @ sock-dgram -.desc -These variables hold integers which give the values of address -families. They correspond to the C constants -.code SOCK_STREAM -and -.codn SOCK_DGRAM . +A file to be examined is initially assumed to be strictly private. -.coNP Variables @, ai-passive @, ai-numerichost @, ai-v4mapped @, ai-all @ ai-addrconfig and @ ai-numericserv -.desc -These variables hold integers which are bitmasks that combine -together via bitwise or, to express the -.code flags -slot of the -.code addrinfo -structure. They correspond to the C constants -.codn AI_PASSIVE , -.codn AI_NUMERICHOST , -.code AI_V4MAPPED -and so forth. They influence the behavior of the -.code getaddrinfo -function. +If the file is not owned by the effective user ID of the caller, or +else by the superuser, then it is not private. -.coNP Variables @, inaddr-any @, inaddr-loopback @ in6addr-any and @ in6addr-loopback -.desc -These integer-valued variables provide constants for commonly used IPv4 -and IPv6 address values. +If the file grants write permission to "others", then it is not private. -The value of -.code inaddr-any -and -.code in6addr-any -is zero. This address is used in binding a passive socket to all of the -external interfaces of a host, so that it can accept connections or datagrams -from all attached networks. +If the file grants read permission to "others", then it is not strictly +private. -The -.code inaddr-loopback -variable is IPv4 loopback address, the same integer as the hexadecimal -constant -.code #x7F000001. +If the file grants write permission to the group owner, then it is not +private if the group contains names other than that of the file owner or the +superuser. -The -.code in6addr-loopback -is the IPv6 loopback address. Its value is 1. +If the file grants read permission to the group owner, then it is not +strictly private if the group contains names other than that of the file owner +or the superuser. -.TP* Example: +Note that this interpretation of "private" and "strictly private" is vulnerable +to the following time-of-check to time-of-use race condition with regard to the +group check. At the time of the check, the group might be empty or contain +only the caller as a member. But by the time the file is subsequently accessed, +the group might have been innocently extended by the system administrator to +include additional users, who can maliciously modify the file. -.verb - ;; Construct an IPv6 socket address suitable for binding - ;; a socket to the loopback network, port 1234: - (new sockaddr-in6 addr in6addr-loopback port 1234) +Also note that the function is vulnerable to a time-of-check to time-of-use +race if +.meta path +is a string rather than a +.code stat +structure. If any components of the +.meta path +are symbolic links or directories that can be manipulated by other +users, then the object named by +.meta path +file can pass the check, but can later +.meta path +can be subverted to refer to a different object. - ;; Mistake: IPv4 address used with IPv6 sockaddr. - (new sockaddr-in6 addr inaddr-loopback) -.brev +One way to guard against this race is to open the file, then use +.code fstat +on the stream to obtain a +.code stat +structure which is then used as an argument to +.code path-private-to-me-p +or +.codn path-strictly-private-to-me-p . -.coNP Function @ open-socket +.coNP Functions @ path-newer and @ path-older .synb -.mets (open-socket < family < type <> [ mode-string ]) +.mets (path-newer < left-path << right-path ) +.mets (path-older < left-path << right-path ) .syne .desc The -.code open-socket -function creates a socket, which is a kind of stream. +.code path-newer +function compares two paths or stat results by modification time. +It returns +.code t +if +.meta left-path +exists, and either +.meta right-path +does not exist, or has a modification time stamp in the past +relative to +.metn left-path . The -.meta family -parameter specifies the address family of the socket. One of the -values -.codn af-inet , -.code af-inet -or -.code af-inet6 -should be used to create a Unix domain, Internet IPv4 or Internet IPv6 -socket, respectively. +.code path-older +function is equivalent to +.code path-newer +with the arguments reversed. +.coNP Function @ path-same-object +.synb +.mets (path-same-object < left-path << right-path ) +.syne +.desc The -.meta type -parameter specifies the socket type, either -.code sock-stream -(stream socket) or -.code sock-dgram -(datagram socket). +.code path-same-object +function returns +.code t +if +.meta left-path +and +.meta right-path +resolve to the same filesystem object: the same inode number on the same +device. -The -.meta mode-string -specifies several properties of the stream; for a description of -.meta mode-string -parameters, refer to the -.code open-file -function. Note that the defaulting behavior for an omitted -.meta mode-string -argument is different under -.code open-socket -from other functions. Because sockets are almost always used for bidirectional -data flow, the default mode string is -.str r+b -rather than the usual -.strn r . +.SS* Unix Credentials -Rationale for including the -.str b -flag in the default mode string is that network protocols are usually defined -in a way that is independent of machine and operating system, down to the byte -level, even when they are textual. It doesn't make sense for the same \*(TX -program to see a network stream differently based on what platform it is -running on. Line ending conversion has to do with how a platform locally stores -text files, whereas network streams are almost always external formats. +.coNP Functions @, getuid @, geteuid @ getgid and @ getegid +.synb +.mets (getuid) +.mets (geteuid) +.mets (getgid) +.mets (getegid) +.syne +.desc +These functions directly correspond to the POSIX C library functions +of the same name. They retrieve the real user ID, effective user ID, +real group ID and effective group ID, respectively, of the calling +process. -Like other stream times, stream sockets are buffered and marked as no -non-real-time streams. Specifying the -.str i -mode in -.meta mode-string -marks a socket as a real-time-stream, and, if it is opened for writing -or reading and writing, changes it to use line buffering. +.coNP Functions @, setuid @, seteuid @ setgid and @ setegid +.synb +.mets (setuid << uid ) +.mets (seteuid << uid ) +.mets (setgid << gid ) +.mets (setegid << gid ) +.syne +.desc +These functions directly correspond to the POSIX C library functions +of the same name. They set the real user ID, effective user ID, +real group ID and effective group ID, respectively, of the calling +process. +On success, they return +.codn t . +On failure, they throw an exception of type +.codn system-error . -.coNP Function @ open-socket-pair +.coNP Function @ getgroups .synb -.mets (open-socket-pair < family < type <> [ mode-string ]) +.mets (getgroups) .syne .desc The -.code open-socket-pair -function provides an interface to the functionality of the -.code socketpair -C library function. +.code getgroups +function retrieves the list of supplementary group IDs of the calling +process by calling the same-named POSIX C library function. -If successful, it creates and returns a list of two stream objects, -which are sockets that are connected together. +Whether or not the effective group ID retrieved by +.code getegid +is included in this list is system-dependent. Programs should not +depend on its presence or absence. -Note: the Internet address families -.code af-inet -and -.code af-inet6 -are not supported. +.coNP Function @ setgroups +.synb +.mets (setgroups << gid-list ) +.syne +.desc +The +.code setgroups +function corresponds to a C library function found in some Unix +operating systems, complementary to the +.code getgroups +function. The argument to +.meta gid-list +must be a list of numeric group IDs. +If the function is successful, this list is installed as the list of +supplementary group IDs of the calling process, and the value +.code t +is returned. +On failure, it throws an exception of type +.codn system-error . +.coNP Functions @ getresuid and @ getresgid +.synb +.mets (getresuid) +.mets (getresgid) +.syne +.desc +These functions directly correspond to the POSIX C library functions +of the same names available in some Unix operating systems. +Each function retrieves a three element list of numeric IDs. The -.code mode-string -is applied to each stream. For a description, see -.code open-socket -and -.codn open-file . +.code getresuid +function retrieves the real, effective and saved user ID of +the calling process. +The +.code getresgid +function retrieves the real, effective and saved group ID of +the calling process. -.coNP Functions @ sock-family and @ sock-type +.coNP Functions @ setresuid and @ setresgid .synb -.mets (sock-family << socket ) -.mets (sock-type << socket ) +.mets (setresuid < real-uid < effective-uid << saved-uid ) +.mets (setresgid < real-gid < effective-gid << saved-gid ) .syne .desc -These functions retrieve the integer values representing the address family -and type of a socket. The argument to the -.meta socket -parameter must be a socket stream or a file or process stream. For a file stream, -both functions return -.codn nil . -An exception of type -.code type-error -is thrown for other stream types. +These functions directly correspond to the POSIX C library functions of the +same names available in some Unix operating systems. They change the real, +effective and saved user ID or group ID, respectively, of the calling process. -.coNP Accessor @ sock-peer +A value of -1 for any of the IDs specifies that the ID is not to be changed. + +Only privileged processes may arbitrarily change IDs to different values. + +Unprivileged processes are restricted in the following way: +each of the new IDs that is replaced must have a new value which is equal to +one of the existing three IDs. + +.SS* Unix Password Database + +.coNP Structure @ passwd .synb -.mets (sock-peer << socket ) -.mets (set (sock-peer << socket ) << address ) +.mets (defstruct passwd nil +.mets \ \ name passwd uid gid +.mets \ \ gecos dir shell) .syne .desc The -.code sock-peer -function retrieves the peer address -has most recently been assigned to -.metn socket . +.code passwd +structure corresponds to the C type +.codn "struct passwd" . +Objects of this struct are produced by the password database +query functions +.codn getpwent , +.codn getpwuid , +and +.codn getpwnam . -Sockets which are not connected initially -have a peer address value of +.coNP Functions @, getpwent @ setpwent and @ endpwent +.synb +.mets (getpwent) +.mets (setpwent) +.mets (endpwent) +.syne +.desc +The first time +.code getpwent +function is called, it returns the first password database entry. +On subsequent calls it returns successive entries. +Entries are returned as instances of the +.code passwd +structure. If the function cannot retrieve an entry for any reason, +it returns .codn nil . -A socket which is connected to a remote peer -receives that peer's address as its -.codn sock-peer . - -If a socket is connected to a remote peer via -a successful use of the -.code sock-connect -function, then its -.code sock-peer -address is set to match that of the peer. -Sockets returned by the -.code sock-accept -function are connected, and have the remote endpoint address as their -.code sock-peer -address. +The +.code setpwent +function rewinds the database scan. -Assigning an address to a -.code sock-peer -form is equivalent to using -.code sock-set-peer -to set the address. +The +.code endpwent +function releases the resources associated with the scan. -Implementation note: the -.code sock-peer -function does not use the -.code getpeername -C library function; the association between a stream and -.code sockaddr -struct is maintained by \*(TX. +.coNP Function @ getpwuid +.synb +.mets (getpwuid << uid ) +.syne +.desc +The +.code getpwuid +searches the password database for an entry whose user ID field +is equal to the numeric +.metn uid . +If the search is successful, then a +.code passwd +structure representing the database entry is returned. +If the search fails, +.code nil +is returned. -.coNP Function @ sock-set-peer +.coNP Function @ getpwnam .synb -.mets (sock-set-peer < socket << address ) +.mets (getpwnam << name ) .syne .desc The -.code sock-set-peer -function stores -.meta address -into -.meta socket -as that socket's peer. +.code getpwnam +searches the password database for an entry whose user name +is equal to +.metn name . +If the search is successful, then a +.code passwd +structure representing the database entry is returned. +If the search fails, +.code nil +is returned. -Subsequently, the -.code sock-peer -function will retrieve that address. +.SS* Unix Group Database -If -.meta address -is not an appropriate address object in the address family of -.metn socket , -the behavior is unspecified. +.coNP Structure @ group +.synb +.mets (defstruct group nil +.mets \ \ name passwd gid mem) +.syne +.desc +The +.code group +structure corresponds to the C type +.codn "struct group" . +Objects of this struct are produced by the password database +query functions +.codn getgrent , +.codn getgrgid , +and +.codn getgrnam . -.coNP Function @ sock-connect +.coNP Functions @, getgrent @ setgrent and @ endgrent .synb -.mets (sock-connect < socket < address <> [ timeout-usec ]) +.mets (getgrent) +.mets (setgrent) +.mets (endgrent) .syne .desc +The first time +.code getgrent +function is called, it returns the first group database entry. +On subsequent calls it returns successive entries. +Entries are returned as instances of the +.code passwd +structure. If the function cannot retrieve an entry for any reason, +it returns +.codn nil . + The -.code sock-connect -function connects a socket stream to a peer address. +.code setgrent +function rewinds the database scan. The -.meta address -argument must be a -.code sockaddr -object of type matching the address family of the socket. +.code endgrent +function releases the resources associated with the scan. -If the operation fails, an exception of type -.code socket-error -is thrown. Otherwise, the function returns -.metn socket . +.coNP Function @ getgrgid +.synb +.mets (getgrgid << gid ) +.syne +.desc +The +.code getgrgid +searches the group database for an entry whose group ID field +is equal to the numeric +.metn gid . +If the search is successful, then a +.code group +structure representing the database entry is returned. +If the search fails, +.code nil +is returned. -If the -.meta timeout-usec -argument is specified, it must be a fixnum integer. -It denotes a connection timeout period in microseconds. -If the connection doesn't succeed within the specified timeout, -an exception of type -.code timeout-error -is thrown. +.coNP Function @ getgrnam +.synb +.mets (getgrnam << name ) +.syne +.desc +The +.code getgrnam +searches the group database for an entry whose group name +is equal to +.metn name . +If the search is successful, then a +.code group +structure representing the database entry is returned. +If the search fails, +.code nil +is returned. -.coNP Function @ sock-bind +.SS* Unix Password Hashing +.coNP Function @ crypt .synb -.mets (sock-bind < socket << address ) +.mets (crypt < key << salt ) .syne .desc The -.code sock-connect -function binds a socket stream to a local address. +.code crypt +function is a wrapper for the Unix C library function of the same name. +It calculates a hash over the +.meta key +and +.meta salt +arguments, which are strings. The hash is returned as a string. The -.meta address -argument must be a -.code sockaddr -object of type matching the address family of the socket. +.meta key +and +.meta salt +arguments are converted into UTF-8 prior to being passed into the underlying +platform function. The hash value is assumed to be UTF-8 and converted to +Unicode characters, though it is not expected to contain anything but 7 +bit ASCII characters. -If the operation fails, an exception of type -.code socket-error -is thrown. Otherwise, the function returns -.codn t . +Note: the underlying C library function uses a static buffer for its return +value. The return value of the \*(TL function is a copy of that buffer. -Returns -.code t -if successful. +.SS* Unix Signal Handling + +On platforms where certain advanced features of POSIX signal handling are +available at the C API level, \*(TX exposes signal-handling functionality. + +A \*(TX program can install a \*(TL function (such as an anonymous. +.codn lambda , +or the function object associated with a named function) as the handler for +a signal. + +When that signal is delivered, \*(TX will intercept it with its own safe, +internal handler, mark the signal as deferred (in a \*(TX sense) and then +dispatch the registered function at a convenient time. + +Handlers currently are not permitted to interrupt the execution of most +\*(TX internal code. Immediate, asynchronous execution of handlers is +currently enabled only while \*(TX is blocked on I/O operations or sleeping. +Additionally, the +.code sig-check +function can be used to dispatch and clear deferred +signals. These handlers are then safely called if they were subroutines of +.codn sig-check , +and not asynchronous interrupts. + +.coNP Variables @, sig-hup @, sig-int @, sig-quit @, sig-ill @, sig-trap @, sig-abrt @, sig-bus @, sig-fpe @, sig-kill @, sig-usr1 @, sig-segv @, sig-usr2 @, sig-pipe @, sig-alrm @, sig-term @, sig-chld @, sig-cont @, sig-stop @, sig-tstp @, sig-ttin @, sig-ttou @, sig-urg @, sig-xcpu @, sig-xfsz @, sig-vtalrm @, sig-prof @, sig-poll @, sig-sys @, sig-winch @, sig-iot @, sig-stkflt @, sig-io @ sig-lost and @ sig-pwr +.desc +These variables correspond to the C signal constants +.codn SIGHUP , +.code SIGINT +and so forth. +The variables +.codn sig-winch , +.codn sig-iot , +.codn sig-stkflt , +.codn sig-io , +.code sig-lost +and +.code sig-pwr +may not be available since a system may lack the corresponding signal +constants. See notes for the function +.codn log-authpriv . -.coNP Function @ sock-listen -.synb -.mets (sock-listen < socket <> [ backlog ]) -.syne -.desc -The -.code sock-listen -function prepares -.meta socket -for listening for connections. The -.meta backlog -parameter, if specified, requires an integer -argument. The default value is 16. +The highest signal number is 31. -.coNP Function @ sock-accept +.coNP Functions @ set-sig-handler and @ get-sig-handler .synb -.mets (sock-accept < socket >> [ mode-string <> [ timeout-usec ]]) +.mets (set-sig-handler < signal-number << handling-spec ) +.mets (get-sig-handler << signal-number ) .syne .desc The -.code sock-accept -function waits for a client connection on -.metn socket , -which must have been prepared for listening for -connections using -.code sock-bind -and -.codn sock-listen . +.code set-sig-handler +function is used to specify the handling for a signal, such +as the installation of a handler function. It updates the signal handling for +a signal whose number is +.meta signal-number +(usually one of the constants like +.codn sig-hup , +.code sig-int +and so forth), and returns the previous value. The +.code get-sig-handler +function returns the current value. -If the operation fails, an exception of type -.code socket-error -is thrown. Otherwise, the function returns a new socket -which is connected to the remote peer. +The +.meta signal-number +must be an integer the range 1 to 31. -The peer's address may be retrieved from this socket using -.codn sock-peer . +Initially, all 31 signal handling specifications are set to the value +.codn t . The -.code mode-string -parameter is applied to the new socket just like the -similar argument in -.codn socket-open . -It defaults to -.strn r+ . +.meta handling-spec +parameter may be a function. If a function is specified, +then the signal is enabled and connected to that function until another +call to +.code set-sig-handler +changes the handling for that signal. -If the -.meta timeout-usec -argument is specified, it must be a fixnum integer. -It denotes a timeout period in microseconds. -If no peer connects for the specified timeout, -.code sock-accept -throws an exception of type -.codn timeout-error . +If +.meta handling-spec +is the symbol +.codn nil , +then the function previously associated +with the signal, if any, is removed, and the signal is disabled. For a signal +to be disabled means that the signal is set to the +.code SIG_IGN +disposition (refer to the C API). -.coNP Variables @, shut-rd @ shut-wr and @ shut-rdwr -.desc -The values of these variables are useful as the second argument to the -.code sock-shutdown -function. +If +.meta handling-spec +is the symbol +.codn t , +then the function previously associated +with the signal, if any, is removed, and the signal is set to its default +disposition. This means that it is set to +.code SIG_DFL +(refer to the C API). +Some signals terminate the process if they are generated while the +handling is configured to the default disposition. -.coNP Function @ sock-shutdown -.synb -.mets (sock-shutdown < sock <> [ direction ]) -.syne -.desc -The -.code sock-shutdown -function indicates that no further communication is to take place on -.meta socket -in the specified direction(s). +Note that the certain signals like +.code sig-quit +and +.code sig-kill +cannot be ignored or handled. +Please observe the signal documentation in the IEEE POSIX standard, and your +platform. -If the operation fails, an exception of type -.code socket-error -is thrown. Otherwise, the function returns -.codn t . +A signal handling function must take two arguments. It is of the form: + +.mono +.mets (lambda >> ( signal << async-p ) ...) +.onom The -.code direction -parameter is one of the values given by the variables -.codn shut-rd , -.code shut-rw -or -.codn shut-rdwr . -These values shut down communication in the read direction, write direction, -or both directions, respectively. +.meta signal +argument is an integer indicating the signal number for which the +handler is being invoked. The +.meta asyncp-p +argument is a Boolean value. +If it is +.codn t , +it indicates that the handler is being invoked +asynchronously\(emdirectly in a signal handling context. If it is +.codn nil , +then it +is a deferred call. Handlers may do more things in a deferred call, such +as terminate by throwing exceptions, and perform I/O. -If the argument is omitted, -.code sock-shutdown -defaults to closing the write direction. +The return value of a handler is normally ignored. However if it invoked +asynchronously (the +.meta async-p +argument is true), then if the handler returns +a +.cod2 non- nil +value, it is understood that the handler +requesting that it be deferred. This means that the signal will be marked +as deferred, and the handler will be called again at some later +time in a deferred context, whereby +.meta async-p +is +.codn nil . +This is not guaranteed, however; +it's possible that another signal will arrive before that happens, +possibly resulting in another async call, so the handler must +be prepared to deal with an async call at any time. -Notes: shutting down is most commonly requested in the write direction, to perform -a "half close". The communicating process thereby indicates that it has written -all the data which it intends to write. When the shutdown action is processed on the -remote end, that end is unblocked from waiting on any further data, and -effectively experiences an "end of stream" condition on its own socket or -socket-like endpoint, while continuing to be able to transmit data. -Shutting down in the reading direction is potentially abrupt. If it is executed -before an "end of stream" indication is received from a peer, it results in an -abortive close. +If a handler is invoked synchronously, then its return value is ignored. -.coNP Functions @ sock-recv-timeout and @ sock-send-timeout +In the current implementation, signals do not queue. If a signal is delivered +to the process again, while it is marked as deferred, it simply stays deferred; +there is no counter associated with a signal, only a Boolean flag. + +.coNP Function @ sig-check .synb -.mets (sock-recv-timeout < sock << usec ) -.mets (sock-send-timeout < sock << usec ) +.mets (sig-check) .syne .desc The -.code sock-recv-timeout -and -.code sock-send-timeout -functions configure, respectively, receive and send timeouts on socket -.metn sock . - -The -.meta usec -parameter specifies the value, in microseconds. It must be a -.code fixnum -integer. - -When a receive timeout is configured on a socket, then an -exception of type -.code timeout-error -is thrown when an input operation waits for at least -.code usec -microseconds without receiving input. +.code sig-check +function tests whether any signals are deferred, and for each +deferred signal in turn, it executes the corresponding handler. For a signal to +be deferred means that the signal was caught by an internal handler in +\*(TX and the event was recorded by a flag. If a handler function is removed +while a signal is deferred, the deferred flag is cleared for that signal. -Similarly, when a send timeout is configured, then an -exception of type -.code timeout-error -is thrown when an output operation waits for at least -.code usec -microseconds for the availability of buffer space in the socket. +Calls to the +.code sig-check +function may be inserted into CPU-intensive code that +has no opportunity to be interrupted by signals, because it doesn't invoke any +I/O functions. -.coNP Functions @ str-inaddr and @ str-in6addr +.coNP Function @ raise .synb -.mets (str-inaddr address <> [ port ]) -.mets (str-in6addr address <> [ port ]) +.mets (raise << signal ) .syne .desc The -.code str-inaddr -and -.code str-in6addr -functions convert an IPv4 and IPv6 address, respectively, to textual -notation which is returned as a character string. -The conversion is done in conformance with RFC 5952, section 4. +.code raise +function sends +.meta signal +to the process. +It is a wrapper for the C function of the same name. -IPv6 addresses representing IPv6-mapped IPv4 addresses are printed -in the hybrid notation exemplified by -.codn ::ffff:192.168.1.1 . +The return value is +.code t +if the function succeeds, otherwise +.codn nil . +.coNP Function @ kill +.synb +.mets (kill < process-id <> [ signal ]) +.syne +.desc The -.meta address -parameter must be a non-negative integer in the appropriate range -for the address type. +.code kill +function is used for sending a signal to a process group or process. +It is a wrapper for the POSIX +.code kill +function. If the -.meta port -number argument is supplied, it is included in the returned character string, -according to the requirements in section 6 of RFC 5952 pertaining to IPv6 -addresses (including IPv6-mapped IPv6 addresses) and section 3.2.3 of RFC 3986 -for IPv4 addresses. In brief, IPv6 addresses with ports are expressed as -.code [address]:port -and IPv6 addresses follow the traditional -.code address:port -pattern. +.meta signal +argument is omitted, it defaults to the same value as +.codn sig-term . -.coNP Functions @ str-inaddr-net and @ str-in6addr-net +The return value is +.code t +if the function succeeds, otherwise +.codn nil . + +.SS* Unix Processes + +.coNP Functions @ fork and @ wait .synb -.mets (str-inaddr-net < address <> [ width ]) -.mets (str-in6addr-net < address <> [ width ]) +.mets (fork) +.mets (wait >> [ pid <> [ flags ]]) .syne .desc -The functions -.code str-inaddr-net +The +.code fork and -.code str-in6addr-net -convert, respectively, IPv4 and IPv6 network prefix addresses to -the "slash notation". For IPv6 addresses, the requirements of section -2.3 of RFC 4291 are implemented. For IPv4, section 3.1 of RFC 4632 is followed. - -The condensed portion of the IP address is always determined by measuring -the contiguous extent of all-zero bits in the least significant position -of the address. For instance an IPv4 address which has at least 24 zero bits -in the least significant position, so that the only nonzero bits are in the -highest octet, is always condensed to a single decimal number: the value of -the first octet. - -If the -.meta width -parameter is specified, then its value is incorporated into the returned -textual notation as the width. No check is made whether this width -large enough to span all of the nonzero bits in the address. - -If -.meta width -is omitted, then it is calculated as the number of bits in the address, -excluding the contiguous all-zero bits in the least significant position: -how many times the address can be shifted to the right before a 1 appears -in the least significant bit. - -.SS* Unix Terminal Control +.code wait +functions are interfaces to the Unix functions +.code fork +and +.codn waitpid . -\*(TX provides access to the terminal control "termios" interfaces defined by -POSIX, and some of the extensions to it in Linux. By using termios, programs -can control serial devices, consoles and virtual terminals. Terminal control -in POSIX revolves around a C language structure called -.codn "struct termios" . -This is mirrored in a \*(TL structure also called -.codn termios . +The +.code fork +function creates a child process which is a replica of the parent. Both +processes return from the function. In the child process, the return value is +zero. In the parent, it is an integer representing the process ID of the child. +If the function fails to create a child, it returns +.code nil +rather than an integer. In this case, the +.code errno +function can be used to inquire about the cause. -Like-named \*(TL functions are provided which correspond to the C functions -.codn tcgetattr , -.codn tcsetattr , -.codn tcsendbreak , -.codn tcdrain , -.code tcflush -and -.codn tcflow . +The +.code wait +function, if successful, returns a cons cell consisting of a pair of integers. +The +.code car +of the cons is the process ID of the process or group which was successfully +waited on, and the +.code cdr +is the status. If +.code wait +fails, it returns +.codn nil . +The +.code errno +function can be used to inquire about the cause. -These have somewhat different argument conventions. The TTY device is specified last, -so that it can conveniently default to the -.code *stdin* -stream. A TTY device can be specified as either a stream object or a numeric -file descriptor. +The +.meta process-id +argument, if not supplied, defaults to -1, which means that +.code wait +waits for any process, rather than a specific process. Certain other +values have special meaning, as documented in the POSIX standard +for the +.code waitpid +function. -The functions -.codn cfgetispeed , -.codn cfgetospeed , -.code cfsetispeed +The +.meta flags +argument defaults to zero. If it is specified as nonzero, it should be +a bitwise combination (via the +.code logior +function) of the variables +.codn w-nohang , +.code w-untraced and -.code cfsetospeed -are not provided, because they are unnecessary. Device speed (informally, "baud rate") -is specified directly as a integer value in the -.code termios -structure. The \*(TL termios functions automatically convert between integer -values and the speed constants (like -.codn B38400 ) -used by the C API. +.codn w-continued . +If +.code w-nohang +is used, then +.code wait +returns a cons cell whose +.code car +specifies a process ID value of zero in the situation that at least +one of the processes designated by +.code process-id +exist and are children of the calling process, but have not changed state. +In this case, the status value in the +.code cdr +is unspecified. -All of the various termios-related constants are provided, including some non-standard -ones. They appear in lower case. For instance -.code IGNBRK -and -.code PARENB -are simply known as the predefined \*(TL variables -.code ignbrk +Status values may be inspected with the functions +.codn w-ifexited , +.codn w-exitstatus , +.codn w-ifsignaled , +.codn w-termsig , +.codn w-coredump , +.codn w-ifstopped , +.code w-stopsig and -.codn parenb . +.codn w-ifcontinued . -.coNP Structure @ termios +.coNP Functions @, w-ifexited @, w-exitstatus @, w-ifsignaled @, w-termsig @, w-coredump @ w-ifstopped and @ w-stopsig .synb -.mets (defstruct termios nil -.mets \ \ iflag oflag cflag lflag -.mets \ \ cc ispeed ospeed) +.mets (w-ifexited << status ) +.mets (w-exitstatus << status ) +.mets (w-ifsignaled << status ) +.mets (w-termsig << status ) +.mets (w-coredump << status ) +.mets (w-ifstopped << status ) +.mets (w-stopsig << status ) +.mets (w-ifcontinued << status ) .syne .desc -The -.code termios -structure represents the kernel level terminal device configuration. -It holds hardware related setting such as serial line speed, parity -and handshaking. It also holds software settings like translations, -and settings affecting input behaviors. The structure closely corresponds -to the C language -.code termios -structure which exists in the POSIX API. - -The -.codn iflag , -.codn oflag , -.code cflag -and -.code lflag -slots correspond to the -.codn c_iflag , -.codn c_oflag , -.code c_cflag -and -.code c_lflag -members of the C structure. They hold integer values representing -bit fields. - -The -.code cc -slot corresponds to the -.code c_cc -member of the C structure. Whereas the -C structure's -.code c_cc -member is an array of the C type -.codn cc_t , -the -.code cc -slot is a vector of integers, whose values must have the same range as the -.code cc_t -type. +These functions analyze process exit values produced by the +.code wait +function. -.coNP Variables @, ignbrk @, brkint @, ignpar @, parmrk @, inpck @, istrip @, inlcr @, igncr @, icrnl @, iuclc @, ixon @, ixany @, ixoff @ imaxbel and @ iutf8 -.desc -These variables specify bitmask values for the -.code iflag -slot of the -.code termios -structure. They correspond to the C language preprocessor symbols -.codn IGNBRK , -.codn BRKINT , -.code IGNPAR -and so forth. +They are closely based on the +POSIX macros +.codn WIFEXITED , +.codn WEXITSTATUS , +and so on. The -.code imaxbel -and -.code iutf8 -variables are specific to Linux and may not be present. -Portable code should test for their presence with -.codn boundp . +.meta status +value is either an integer, or a cons cell. In this case, the cons +cell is expected to have an integer in its +.code cdr +which is used as the status. The -.code iuclc -variable is a legacy feature not found on all systems. - -Note: the -.code termios -methods -.code set-iflags -and -.code clear-iflags -provide a convenient means for setting and clearing combinations of -these flags. +.codn w-ifexited , +.codn w-ifsignaled , +.codn w-coredump , +.code w-ifstopped +and +.code w-ifcontinued +functions have Lisp Boolean return semantics, unlike their C language +counterparts: they return +.code t +or +.codn nil , +rather than zero or nonzero. The others return integer values. -.coNP Variables @, opost @, olcuc @, onlcr @, ocrnl @, onocr @, onlret @, ofill @, ofdel @, vtdly @, vt0 @, vt1 @, nldly @, nl0 @, nl1 @, crdly @, cr0 @, cr1 @, cr2 @, cr3 @, tabdly @, tab0 @, tab1 @, tab2 @, tab3 @, bsdly @, bs0 @, bs1 @, ffdly @ ff0 and @ ff1 +.coNP Function @ exec +.synb +.mets (exec < file <> [ args ]) +.syne .desc -These variables specify bitmask values for the -.code oflag -slot of the -.code termios -structure. They correspond to the C language preprocessor symbols -.codn OPOST , -.codn OLCUC , -.code ONLCR -and so forth. - -The variable -.code ofdel -is Linux-specific. Portable programs should test for its presence using -.codn boundp . +The exec function replaces the process image with the executable specified +by string argument +.metn file . +The executable is found by searching the system path. The -.code olcuc -variable is a legacy feature not found on all systems. +.meta file +argument becomes the first argument of the executable, argument zero. -Likewise, whether the following groups of symbols are present is -platform-specific: -.codn nldly , -.code nl0 -and -.codn nl1 ; -.codn crdly , -.codn cr0 , -.codn cr1 , -.code cr2 -and -.codn cr3 ; -.codn tabdly , -.codn tab0 , -.codn tab1 , -.code tab2 -and -.codn tab3 ; -.codn bsdly , -.code bs0 -and -.codn bs1 ; -and -.codn ffdly , -.code ff0 -and -.codn ff1 . +If +.meta args +is specified, it is a list of strings. These are passed as the additional +arguments of the executable. -Note: the -.code termios -methods -.code set-oflags -and -.code clear-oflags -provide a convenient means for setting and clearing combinations of -these flags. +If +.code exec +fails, an exception of type +.code file-error +is thrown. -.coNP Variables @, csize @, cs5 @, cs6 @, cs7 @, cs8 @, cstopb @, cread @, parenb @, parodd @, hupcl @, clocal @, cbaud @, cbaudex @ cmspar and @ crtscts +.coNP Function @ exit* +.synb +.mets (exit* << status ) +.syne .desc -These variables specify bitmask values for the -.code cflag -slot of the -.code termios -structure. They correspond to the C language preprocessor symbols -.codn CSIZE , -.codn CS5 , -.code CS6 -and so forth. +The +.code exit* +function terminates the entire process (running \*(TX image), specifying +the termination status to the operating system. The +.meta status +argument is treated exactly like that of the +.code exit +function. Unlike that function, this one exits the process immediately, +cleaning up only low-level operating system resources such as closing file +descriptors and releasing memory mappings, without performing user-space +cleanup. -The following are present on Linux, and may not be available -on other platforms. Portable code should test for them using -.codn boundp : -.codn cbaud , -.codn cbaudex , -.code cmspar -and -.codn crtscts . +.code exit* +is implemented using a call to the POSIX function +.codn _exit . -Note: the -.code termios -methods -.code set-cflags +.coNP Functions @ getpid and @ getppid +.synb +.mets (getpid) +.mets (getppid) +.syne +.desc +These functions retrieve the current process ID and the parent process ID +respectively. They are wrappers for the POSIX functions +.code getpid and -.code clear-cflags -provide a convenient means for setting and clearing combinations of -these flags. +.codn getppid . -.coNP Variables @, isig @, icanon @, echo @, echoe @, echok @, echonl @, noflsh @, tostop @, iexten @, xcase @, echoctl @, echoprt @, echoke @, flusho @ pendin and @ extproc +.coNP Function @ daemon +.synb +.mets (daemon < nochdir-p << noclose-p ) +.syne .desc -These variables specify bitmask values for the -.code lflag -slot of the -.code termios -structure. They correspond to the C language preprocessor symbols -.codn ISIG , -.codn ICANON , -.code ECHO -and so forth. +This is a wrapper for the function +.code daemon +which originated in BSD Unix. -The following are present on Linux, and may not be available -on other platforms. Portable code should test for them using -.codn boundp : -.codn iexten , -.codn xcase , -.codn echoctl , -.codn echoprt , -.codn echoke , -.codn flusho , -.code pendin -and -.codn extproc . +It returns +.code t +if successful, +.code nil +otherwise, and the +.code errno +variable is set in that case. -Note: the -.code termios -methods -.code set-lflags -and -.code clear-lflags -provide a convenient means for setting and clearing combinations of -these flags. +.SS* Unix File Descriptors -.coNP Variables @, vintr @, vquit @, verase @, vkill @, veof @, vtime @, vmin @, vswtc @, vstart @, vstop @, vsusp @, veol @, vreprint @, vdiscard @, vwerase @ vlnext and @ veol2 +.coNP Function @ open-fileno +.synb +.mets (open-fileno < file-descriptor <> [ mode-string ]) +.syne .desc -These variables specify integer offsets into the vector stored in the -.code cc -slot of the -.code termios -structure. They correspond to the C language preprocessor symbols -.codn VINTR , -.codn VQUIT , -.code VERASE -and so forth. +The +.code open-fileno +function creates a \*(TX stream over a file descriptor. The +.meta file-descriptor +argument must be an integer denoting a valid file descriptor. -The following are present on Linux, and may not be available -on other platforms. Portable code should test for them using -.codn boundp : -.codn vswtc , -.codn vreprint , -.codn vdiscard , -.code vlnext -and -.codn veol2 . +For a description of +.metn mode-string , +see the +.code open-file +function. -.coNP Variables @, tcooff @, tcoon @ tcioff and @ tcion +.coNP Function @ fileno +.synb +.mets (fileno << stream ) +.syne .desc -These variables hold integer values suitable as the -.meta action -argument of the -.code tcflow -function. They correspond to the C language preprocessor symbols -.codn TCOOFF , -.codn TCOON , -.code TCIOFF -and -.codn TCION . +The +.code fileno +function returns the underlying file descriptor of +.metn stream , +if it has one. Otherwise, it returns +.codn nil . -.coNP Variables @, tciflush @ tcoflush and @ tcioflush -.desc -These variables hold integer values suitable as the -.meta queue -argument of the -.code tcflush -function. They correspond to the C language preprocessor symbols -.codn TCIFLUSH , -.code TCOFLUSH -and -.codn TCIOFLUSH . +This is equivalent to querying the stream using +.code stream-get-prop +for the +.code :fd +property. -.coNP Variables @, tcsanow @ tcsadrain and @ tcsaflush +.coNP Function @ dupfd +.synb +.mets (dupfd < old-fileno <> [ new-fileno ]) +.syne .desc -These variables hold integer values suitable as the -.meta actions -argument of the -.code tcsetattr -function. They correspond to the C language preprocessor symbols -.codn TCSANOW , -.code TCSADRAIN -and -.codn TCSAFLUSH . +The +.code dupfd +function provides an interface to the POSIX functions +.code dup +or +.codn dup2 , +when called with one or two arguments, respectively. -.coNP Functions @ tcgetattr and @ tcsetattr +.coNP Function @ pipe .synb -.mets (tcgetattr <> [ device ]) -.mets (tcsetattr < termios >> [ actions <> [ device ]]) +.mets (pipe) .syne .desc The -.code tcgetattr -and -.code tcsetattr -functions, respectively, retrieve and install the configuration -of the terminal driver associated with the specified device. +.code pipe +function, if successful, returns a pair of integer file descriptors +as a cons cell pair. The descriptor in the +.code car +field of the pair is the read end of the pipe. +The +.code cdr +holds the write end. -These functions are wrappers for the like-named POSIX C library functions, -but with different argument conventions, and operating using -a \*(TL structure. +If the function fails, it throws an exception of type +.codn file-error . +.coNP Function @ close +.synb +.mets (close < fileno <> [ throw-on-error-p ]) +.syne +.desc The -.code tcgetattr -function, if successful, returns a new instance of the -.code termios -structure. +.code close +function passes the integer descriptor +.meta fileno +to the POSIX +.code close +function. If the operation is successful, then +.code t +is returned. Otherwise an exception of type +.code file-error +is thrown, unless the +.meta throw-on-error-p +argument is present, with a true value. In that case, +.code close +indicates failure by returning +.codn nil . +.coNP Function @ poll +.synb +.mets (poll < poll-list <> [ timeout ]) +.syne +.desc The -.code tcsetattr -function requires an instance of a -.code termios -structure as an argument to its -.meta termios -parameter. +.code poll +function suspends execution while monitoring one or more file descriptors +for specified events. It is a wrapper for the same-named POSIX function. -A program may alter the settings of a terminal device by -retrieving them using -.codn tcgetattr , -manipulating the structure returned by this function, and -then using -.code tcsetattr -to install the modified structure into the device. +The +.meta poll-list +argument is a list of +.code cons +pairs. The +.code car +of each pair is either an integer file descriptor, or else a stream +object which has a file descriptor (the +.code fileno +function can be applied to that stream to retrieve a descriptor). +The +.code cdr +of each pair is an integer bit mask specifying the events, whose +occurrence the file descriptor is to be monitored for. The variables +.codn poll-in , +.codn poll-out , +.code poll-err +and several others are available which hold bitmask values corresponding +to the constants +.codn POLLIN , +.codn POLLOUT , +.code POLLERR +used with the C language +.code poll +function. The -.meta actions -argument of -.code tcsetattr -may be given as the value of one of the variables -.codn tcsanow , -.code tcsadrain -or -.codn tcsaflush . -If it is omitted, the default is -.codn tcsadrain . +.meta timeout +argument, if absent, defaults to the value -1, which specifies an indefinite +wait. A nonnegative value specifies a wait with a timeout, measured in +milliseconds. -If an argument is given for -.meta device -it must be either a stream, or an integer file descriptor. -In either case, it is expected to be associated with a -terminal (TTY) device. +The function returns a list of pairs representing the descriptors or streams +which were successfully polled. If the function times out, it returns an +empty list. If an error occurs, an exception is thrown. -If the argument is omitted, it defaults to the stream currently -stored in the -.code *stdin* -stream special variable, expected to be associated with -a terminal device. +The returned list is similar in structure to the input list. However, it holds +only entries which polled positive. The +.code cdr +of every pair now holds a bitmask of the events which were to have occurred. -.TP* Notes: +.SS* Unix File Control -The C -.code termios -structure usually does not have members for representing the input -and output speed. \*(TL does not use such members, in any case, even -if they are present. The speeds are encoded in the -.code cc_iflag -and -.code cc_lflag -bitmasks. When retrieving the settings, the -.code tcgetattr -function uses the POSIX functions -.code cfgetispeed -and -.code cfgetospeed -to retrieve the speed values from the C structure. These values -are installed as the -.code ispeed -and -.code ospeed -slots of the Lisp structure. A reverse conversion takes place -when setting are installed using -.codn tcsetattr : -the speed values are taken from the slots, and installed into -the C structure using -.code cfsetispeed +.coNP Variables @, o-accmode @, o-rdonly @, o-wronly @, o-rdwr @, o-creat @, o-noctty @, o-trunc @, o-append @, o-nonblock @, o-sync @, o-async @, o-directory @, o-nofollow @, o-cloexec @, o-direct @ o-noatime and @ o-path +.desc +These variables correspond to the POSIX file mode constants +.codn O_ACCMODE , +.codn O_RDONLY , +.codn O_WRONLY , +.codn O_RDWR , +.codn O_CREAT , +.codn O_NOCTTY , +and so forth. + +The availability of the variables +.codn o-async , +.codn o-directory , +.codn o-nofollow , +.codn o-cloexec , +.codn o-direct , +.code o-noatime and -.code cfsetospeed -before the structure is passed to the C -.code tcsetattr -function. +.code o-path +depends on the host platform. -On Linux, TTY devices do not have a separate input and output speed. -The C -.code termios -structure stores only one speed which is taken as both the input -and output speed, with a special exception. The input speed may be -programmed as zero. In that case, it is independently represented. -speed may be programmed as zero. +Some of these flags may be set or cleared on an existing file descriptor +using the +.code f-setfl +command of the +.code fcntl +function, in accordance with POSIX and the host platform documentation. -.coNP Function @ tcsendbreak -.synb -.mets (tcsendbreak >> [ duration <> [ device ]]) -.syne +.coNP Variables @, seek-set @ seek-cur and @ seek-end .desc -The -.code tcsendbreak -function generates a break signal on serial devices. The -.meta duration -parameter specifies the length of the break signal in milliseconds. -If the argument is omitted, the value 500 is used. +These variables correspond to the ISO C constants +.codn SEEK_SET , +.code SEEK_CUR +and +.codn SEEK_END . +These values, usually associated with the ISO C +.code fseek +function, are also used in the +.code fcntl +file locking interface as values of the +.code whence +member of the +.code flock +structure. + +.coNP Variables @, f-dupfd @, f-dupfd-cloexec @, f-getfd @, f-setfd @, f-getfl @, f-setfl @, f-getlk @ f-setlk and @ f-setlkw +.desc +These variables correspond to the POSIX +.code fcntl +command constants +.codn F_DUPFD , +.codn F_GETFD , +.codn F_SETFD , +and so forth. Availability of the +.code f-dupfd-cloexec +depends on the host platform. +.coNP Variable @ fd-cloexec +.desc The -.meta device -parameter is exactly the same as that of the -.code tcsetattr +.code fd-cloexec +variable corresponds to the POSIX +.code FD_CLOEXEC +constant. It denotes the flag which may be set by the +.code fd-setfd +command of the +.code fcntl function. -.coNP Function @ tcdrain +.coNP Variables @, f-rdlck @ f-wrlck and @ f-unlck +.desc +These variables correspond to the POSIX lock type constants +.codn F_RDLCK , +.code F_WRLCK +and +.codn F_UNLCK . +They specify the possible values of the +.code type +field of the +.code flock +structure. + +.coNP Structure @ flock .synb -.mets (tcdrain <> [ device ]) +.mets (defstruct flock nil +.mets \ \ type whence +.mets \ \ start len +.mets \ \ pid) .syne .desc The -.code tcdrain -function waits until all queued output on a terminal -device has been transmitted. It is a direct wrapper -for the like-named POSIX C function. +.code flock +structure corresponds to the POSIX structure of the same name. +An instance of this structure must be specified as the third +argument of the +.code fcntl +function when the +.meta command +argument is one of the values +.codn f-getlk , +.code f-setlk +or +.codn f-setlkw . -The -.meta device -parameter is exactly the same as that of the -.code tcsetattr -function. +All slots must be initialized with appropriate values before +calling +.code fcntl +with the exception that the +.code f-getlk +command does not access the existing value of the +.code pid +slot. -.coNP Function @ tcflush +.coNP Function @ fcntl .synb -.mets (tcflush < queue <> [ device ]) +.mets (fcntl < fileno < command <> [ arg ]) .syne .desc The -.code tcflush -function discards either untransmitted output data, -or received and yet unread input data, depending on the -value of the -.meta queue -argument. It is a direct wrapper for the like-named -POSIX C function. - +.code fcntl +function corresponds to the same-named POSIX function. The -.meta queue -argument should be the value of one of the variables -.codn tciflush , -.code tcoflush +.meta fileno and -.codn tcioflush , -which specify the flushing of input data, output -data or both. +.meta command +arguments must be integers. +The \*(TL +.code fileno +restricts the +.meta command +argument to the supported values for which symbolic variable names are provided. +Other integer +.meta command +values are rejected by returning -1 and setting the +.code errno +variable to +.codn EINVAL . +Whether the third argument is required, and what type it must be, depends on the +.meta command +value. Commands not requiring the third argument ignore it if it is passed. + +.code fcntl +commands for which POSIX requires an argument of type +.code long +require +.meta arg +to be an integer. + +The file locking commands +.codn f-getlk , +.code f-setlk +and +.code f-setlkw +require +.meta arg +to be a +.code flock +structure. The -.meta device -parameter is exactly the same as that of the -.code tcsetattr -function. +.code fcntl +function doesn't throw an error if the underlying POSIX function indicates +failure; the underlying function's return value is converted to a Lisp integer +and returned. -.coNP Function @ tcflow +.SS* Unix Itimers +Itimers ("interval timers") can be used in combination with signal handling to +execute asynchronous actions. Itimers deliver delayed, one-time signals, +and also periodically recurring signals. For more information, consult the +POSIX specification. + +.coNP Variables @, itimer-real @ itimer-virtual and @ itimer-prof +.desc +These variables correspond to the POSIX constants +.codn ITIMER_REAL , +.code ITIMER_VIRTUAL +and +.codn ITIMER_PROF . +Their values are suitable as the +.meta timer +argument of the +.code getitimer +and +.code setitimer +functions. + +.coNP Functions @ getitimer and @ setitimer .synb -.mets (tcflow < action <> [ device ]) +.mets (getitimer << timer ) +.mets (setitimer < timer < interval << value ) .syne .desc The -.code tcflow -function provides bi-directional flow control on the -specified terminal device. It is a direct wrapper -for the like-named POSIX C function. +.code getitimer +function returns the current value of the specified timer, +which must be +.codn itimer-real , +.code itimer-virtual +or +.codn itimer-prof . -The -.meta action -argument should be the value of one of the -variables -.codn tcooff , -.codn tcoon , -.code tcioff -and -.codn tcion . +The current value consists of a list of two integer values, which +represents microseconds. The first value is the timer interval, +and the second value is the timer's current value. -The -.meta device -parameter is exactly the same as that of the -.code tcsetattr -function. +Like +.codn getitimer , +the +.code setitimer +function also retrieves the specified timer. +In addition, it stores a new value in the timer, +which is given by the two arguments, expressed in microseconds. -.coNP Methods @, set-iflags @, set-oflags @, set-cflags @, set-lflags @, clear-iflags @, clear-oflags @ clear-cflags and @ clear-lflags -.synb -.mets << termios .(set-iflags << flags *) -.mets << termios .(set-oflags << flags *) -.mets << termios .(set-cflags << flags *) -.mets << termios .(set-lflags << flags *) -.mets << termios .(clear-iflags << flags *) -.mets << termios .(clear-oflags << flags *) -.mets << termios .(clear-cflags << flags *) -.mets << termios .(clear-lflags << flags *) -.syne +.SS* Unix Syslog + +On platforms where a Unix-like syslog API is available, \*(TX exports this +interface. \*(TX programs can configure logging via the +.code openlog +function, +control the logging mask via +.code setlogmask +and generate logs via +.codn syslog , +or using special syslog streams. + +.coNP Variables @, log-pid @, log-cons @, log-ndelay @, log-odelay @ log-nowait and @ log-perror .desc -These methods of the -.code termios -structure set or clear multiple flags of the four bitmask flag fields. +These variables take on the values of the corresponding C preprocessor +constants from the +.code +header: +.codn LOG_PID , +.codn LOG_CONS , +etc. +These integer values represent logging options used in the +.meta options +argument to the +.code openlog +function. -The -.meta flags -arguments specify zero or more integer values. These values -are combined together bitwise, as if by the -.code logior -function to form a single effective mask. -If there are no -.meta flags -arguments, then the effective mask is zero. +Note: +.code LOG_PERROR +is not in POSIX, and so +.code log-perror +might not be available. +See notes about +.code LOG_AUTHPRIV +in the documentation for +.codn log-authpriv . -The -.code set-iflags -method sets, in the -.code iflag -slot of the -.meta termios -structure, all of the bits which -are set in the effective mask. That is to say, -the effective mask is combined with the value in -.code iflag -by a -.code logior -operation, and the result is stored back into -.codn iflag . -Similarly, the -.codn set-oflags , -.code set-cflags -and -.code set-lflags -methods operate on the -.codn oflag , -.code cflag +.coNP Special variables @, log-user @, log-daemon @ log-auth and @ log-authpriv +.desc +These variables take on the values of the corresponding C preprocessor +constants from the +.code +header: +.codn LOG_USER , +.codn LOG_DAEMON , +.code LOG_AUTH and -.code lflag -slots of the structure. +.codn LOG_AUTHPRIV . +These are the integer facility codes specified in the +.code openlog +function. -The -.code clear-iflags -method clears, in the -.code iflag -slot of the -.meta termios -structure, all of the bits which are -set in the effective mask. That is to say, -the effective mask is bitwise inverted as if -by the -.code lognot -function, and then combined with the -existing value of the -.code iflag -slot using -.codn logand . -The resulting value is stored back into the -.code iflag -slot. -Similarly, the -.codn clear-oflags , -.code clear-cflags -and -.code clear-lflags -methods operate on the -.codn oflag , -.code cflag -and -.code lflag -slots of the structure. +Note: +.code LOG_AUTHPRIV +is not in POSIX, and so +.code log-authpriv +might not be available. +For portability use code like +.code "(or (symbol-value 'log-authpriv) 0)" +to evaluate to 0 if +.code log-authpriv +doesn't exist, or else check for its existence +using +.codn "(boundp 'log-authpriv)" . -Note: the methods -.codn go-raw , -.code go-cbreak -and -.code go-canon -are provided for changing the settings to raw, "cbreak" and canonical mode. -These methods should be preferred to directly manipulating the flag and -.code cc -slots. +.coNP Variables @, log-emerg @, log-alert @, log-crit @, log-err @, log-warning @, log-notice @ log-info and @ log-debug +.desc +These variables take on the values of the corresponding C preprocessor +constants from the +.code +header: +.codn LOG_EMERG , +.codn LOG_ALERT , +etc. +These are the integer priority codes specified in the +.code syslog +function. -.TP* Example +.coNP The @ *stdlog* special variable +.desc +The +.code *stdlog* +variable holds a special kind of stream: a syslog stream. Each +newline-terminated line of text sent to this stream becomes a log message. -In this example, -.code tio -is assumed to be a variable holding an instance of a -.code termios -struct: +The stream internally maintains a priority value that is applied +when it generates messages. By default, this value is that of +.codn log-info . +The stream holds the priority as the value of the +.code :prio +stream property, which may be changed with the +.code stream-set-prop +function. -.verb - ;; clear the ignbrk, brkint, and various other flags: - tio.(clear-iflags ignbrk brkint parmrk istrip - inlcr igncr icrnl ixon) +The latest priority value which has been configured on the stream is used +at the time the newline character is processed and the log message +is generated, not necessarily the value which was in effect at the time the +accumulation of a line began to take place. - ;; set the csize and parenb flags: - tio.(set-cflags csize parenb) -.brev +Messages sent to +.code *stdlog* +are delimited by newline characters. That is to say, each line of +text written to the stream is a new log. -.coNP Methods @ go-raw and @ go-cbreak +.coNP Function @ openlog .synb -.mets << termios .(go-raw) -.mets << termios .(go-cbreak) +.mets (openlog < id-string >> [ options <> [ facility ]]) .syne .desc The -.code go-raw -and -.code go-cbreak -methods of the -.code termios -structure manipulate the flag slots, as well as certain elements -of the -.code cc -slot, in order to prepare the terminal settings for, respectively, -"raw" and "cbreak" mode, described below. +.code openlog +function is a wrapper for the +.code openlog +C function, and the +arguments have the same semantics. It is not necessary to use +.code openlog +in order +to call the +.code syslog +function or to write data to +.codn *stdlog* . +The call is necessary in order to override the default identifying string, to +set options, such as having the PID (process ID) recorded in log messages, and +to specify the facility. -Note that manipulating the -.code termios -structure doesn't actually put these settings into effect in -the terminal device; the settings represented by the structure must -be installed into the device using -.codn tcsetattr . -There is no way to reverse the effect of these methods. -To precisely restore the previous terminal settings, the program -should retain a copy of the original -.code termios -structure. +The +.meta id-string +argument is mandatory. -"Raw" mode refers to a configuration of the terminal device driver in which input -and output is passed transparently and without accumulation, conversion or -interpretation. Input isn't buffered into lines; as soon as a single byte is -received, it is available to the program. No special input characters such as -commands for generating an interrupt or process suspend request are processed -by the terminal driver; all characters are treated as input data. Input isn't -echoed; the only output which takes place is that generated by program -output requests to the device. +The +.meta options +argument is a bitwise mask (see the logior function) of option +values such as +.code log-pid +and +.codn log-cons . +If it is missing, then a value of 0 is +used, specifying the absence of any options. -"Cbreak" mode is named after a concept and function in the "curses" terminal -control library. It refers to a configuration of the terminal device driver -which is less transparent than "raw" mode. Input isn't buffered into lines, -and line editing commands are ordinary input characters, allowing -character-at-a-time input. However, most input translations are preserved, -except that the conversion of CR characters to NL is disabled. The -signal-generating characters are processed in this mode. This latter feature of -the configuration is the likely inspiration for the word "cbreak". Unless -otherwise configured, the interrupt character corresponds to the Ctrl-C key, -and "break" is another term for an interactive interruption. +The +.meta facility +argument is one of the values +.codn log-user , +.code log-daemon +or +.codn log-auth . +If it is missing, then +.code log-user +is assumed. -.coNP Methods @ string-encode and @ string-decode +.coNP Function @ closelog .synb -.mets << termios .(string-encode) -.mets << termios .(string-decode << string ) +.mets (closelog) .syne .desc The -.code string-encode -method converts the terminal state stored in a -.code termios -structure into a textual format, returning that representation -as a character string. +.code closelog +function is a wrapper for the C function +.codn closelog . +.coNP Function @ setlogmask +.synb +.mets (setlogmask << bitmask-integer ) +.syne +.desc The -.code string-decode -method parses the character representation produced by -.code string-encode -and populates the -.meta termios -structure with the settings are encoded in that string. +.code setlogmask +function interfaces to the corresponding C function, and has the +same argument and return value semantics. The +.meta bitmask-integer +argument is a mask of priority +values to enable. The return value is the prior value. Note that if the +argument is zero, then the function doesn't set the mask to zero; it only +returns the current value of the mask. -If a string is passed to -.code string-decode -which wasn't produced by -.codn string-encode , -the behavior is unspecified. An exception may or may not be -thrown, and the contents of -.meta termios -may or may not be affected. +Note that the priority values like +.code log-emerg +and +.code log-debug +are integer +enumerations, not bitmasks. These values cannot be combined directly to create +a bitmask. Rather, the +.code mask +function should be used on these values. -Note: the textual representation produced by -.code string-encode -is intended to be identical to that produced by the -.code -g -option of the GNU Coreutils version of the -.code stty -utility, on the same platform. That is to say, the output of -.code "stty -g" -may be used as input into -.codn string-decode , -and the output of -.code string-encode -may be used as an argument to -.codn stty . +.TP* Example: -.SS* Unix System Identification +.verb + ;; Enable LOG_EMERG and LOG_ALERT messages, + ;; suppressing all others + (setlogmask (mask log-emerg log-alert)) +.brev -.coNP Structure @ utsname +.coNP Function @ syslog .synb -.mets (defstruct utsname nil -.mets \ \ sysname nodename release -.mets \ \ version machine domainname) +.mets (syslog < priority < format << format-arg *) .syne .desc The -.code utsname -structure corresponds to the POSIX structure of the same name. -An instance of this structure is returned by the -.code uname -function. +.code syslog +function is the interface to the +.code syslog +C function. The +.code printf +formatting capabilities of the function are not used; +the +.meta format +argument follows the conventions of the \*(TL +.code format +function instead. Note in particular that +the +.code %m +convention for interpolating the value of strerror(errno) which is +available in some versions of the +.code syslog +C function is currently not supported. -.coNP Function @ uname -.synb -.mets (uname) -.syne +Note that syslog messages are not newline-terminated. + +.SS* Unix Path Globbing + +On platforms where the POSIX +.code glob +function is available \*(TX provides this functionality in +the form of a like-named function, and some numeric constants. +\*(TX also provides access the +.code fnmatch +function, where available. + +.coNP Variables @, glob-err @, glob-mark @, glob-nosort @, glob-nocheck @, glob-noescape @, glob-period @, glob-altdirfunc @, glob-brace @, glob-nomagic @, glob-tilde @ glob-tilde-check and @ glob-onlydir .desc -The -.code uname -function corresponds to the POSIX -function of the same name. It returns an instance of the -.code utsname -structure. Each slot of the returned structure is -initialized with a character string that identifies the corresponding attribute -of the host system. +These variables take on the values of the corresponding C preprocessor +constants from the +.code +header: +.codn GLOB_ERR , +.codn GLOB_MARK , +.codn GLOB_NOSORT , +etc. -The host system might not support the reporting of the -NIS domain name. In this case, the -.code domainname -slot of the returned -.code utsname -structure will have the value -.codn nil . +These values are passed as the optional second argument of the +.code glob +function. They are bitmasks and so multiple values can be combined +using the +.code logior +function. -.SS* Web Programming Support +Note that the +.codn glob-period , +.codn glob-altdirfunc , +.codn glob-brace , +.codn glob-nomagic , +.codn glob-tilde , +.code glob-tilde-check +and +.code glob-onlydir +variables may not be available. They are extensions in the GNU C library +implementation of +.codn glob . -.coNP Functions @ url-encode and @ url-decode +.coNP Function @ glob .synb -.mets (url-encode < string <> [ space-plus-p ]) -.mets (url-decode < string <> [ space-plus-p ]) +.mets (glob < pattern >> [ flags <> [ error-func ]]) .syne .desc -These functions convert character strings to and from a form which is suitable -for embedding into the request portions of URL syntax. - -Encoding a string for URL use means identifying in it certain characters that -might have a special meaning in the URL syntax and representing it using -"percent encoding": the percent character, followed by the ASCII value of the -character. Spaces and control characters are also encoded, as are all byte -values greater than or equal to 127 (7F hex). The printable ASCII characters -which are percent-encoded consist of this set: +The +.code glob +function is a interface to the Unix function of the same name. +The +.meta pattern +argument must be a string, which holds a glob pattern: a pattern which +matches zero or more path names, similar to a regular expression. +The function tries to expand the pattern and return a list of strings +representing the matching path names in the file system. -.verb - :/?#[]@!$&'()*+,;=% -.brev +If there are no matches, then an empty list is returned. -More generally, strings can consists of Unicode characters, but the URL -encoding consists only of printable ASCII characters. Unicode characters in the -original string are encoded by expanding into UTF-8, and applying -percent-encoding the UTF-8 bytes, which are all in the range -.codn \exx80-\exxFF . +The optional +.meta flags +argument defaults to zero. If given, it may be a bitwise combination of the +values of the variables +.codn glob-err , +.codn glob-mark , +.code glob-nosort +and others. -Decoding is the reverse process: reconstituting the UTF-8 byte sequence -specified by the URL-encoding, and then decoding the UTF-8 sequence into the -string of Unicode characters. +If the +.meta error-func +argument is specified, it gives a callback function which is invoked +when +.code glob +encounters errors accessing paths. The function takes two arguments: +the pathname and the +.code errno +value which occurred for that pathname. The function's return value is +Boolean. If the function returns true, then +.code glob +will terminate. -There is an additional complication: whether or not to encode spaces as plus, -and to decode plus characters to spaces. In encoding, if spaces are not encoded -to the plus character, then they are encoded as -.codn %20 , -since spaces are reserved -characters that must be encoded. In decoding, if plus characters are not -decoded to spaces, then they are left alone: they become plus characters in the -decoded string. +The +.meta error-func +may terminate the traversal by a nonlocal exit, such as by throwing +an exception or performing a block return. The -.code url-encode -function performs the encoding process. If the -.code space-plus-p -argument is omitted or specified as -.codn nil , -then spaces are encoded as -.codn %20 . -If the argument is a value other than -.codn nil , -then spaces are encoded as the -character -.code + -.codn (plus) . +.meta error-func +may not re-enter the +.code glob +function. This situation is detected and diagnosed by an exception. The -.code url-decode -function performs the decoding process. If the -.code space-plus-p -argument is omitted or specified as -.codn nil , -then -.code + -.code (plus) -characters in the -encoded data are retained as -.code + -characters in the decoded strings. Otherwise, -plus characters are converted to spaces. +.meta error-func +may not capture a continuation across the error boundary. That is to say, +code invoked from the error may not capture a continuation up to a prompt +which surrounds the +.code glob +call. Such an attempt is detected and diagnosed by an exception. -.coNP Functions @, html-encode @ html-encode* and @ html-decode +Details of the semantics of the +.code glob +function, and the meaning of all the +.meta flags +arguments are given in the documentation for the C function. + +.coNP Variables @, fnm-pathname @, fnm-noescape @, fnm-period @, fnm-leading-dir @ fnm-casefold and @ fnm-extmatch +.desc +These variables take on the values of the corresponding C preprocessor +constants from the +.code +header: +.codn FNM_PATHNAME , +.codn FNM_NOESCAPE , +.codn FNM_PERIOD , +etc. + +These values are bit masks which may be combined with the +.code logior +function to form the optional third +.meta flags +argument of the +.code fnmatch +function. + +Note that the +.codn fnm-leading-dir , +.code fnm-case-fold +and +.code fnm-extmatch +may not be available. They are GNU extensions, found in the GNU C library. + +.coNP Function @ fnmatch .synb -.mets (html-encode << text-string ) -.mets (html-decode << html-string ) +.mets (fnmatch < pattern < string <> [ flags ]]) .syne .desc The -.code html-encode -and -.code html-decode -functions convert between an HTML and raw -representation of text. - +.code fnmatch +function, if available, provides access +to the like-named POSIX C library function. The -.code html-encode -function returns a string which is based on the content of -.metn text-string , -but in which all characters which have special meaning in HTML -have been replaced by HTML codes for representing those characters literally. -The returned string is the HTML-encoded verbatim representation of -.metn text-string . +.meta pattern +argument specifies a POSIX-shell-style file pattern matching expression. +Its exact features and dialect are controlled by +.metn flags . +If +.meta string +matches +.meta pattern +then +.code t +is returned. If there is no match, then +.code nil +is returned. If the C function indicates that an error has occurred, +an exception is thrown. -The -.code html-decode -function converts -.metn html-string , -which may contain HTML -character encodings, into a string which contains the actual characters -represented by those encodings. +.SS* Unix Filesystem Traversal -The function composition -.code "(html-decode (html-encode text))" -returns a string which is equal to -.codn text . +On platforms where the POSIX +.code nftw +function is available \*(TX provides this functionality in +the form of the analogous Lisp function +.codn ftw , +accompanied by some numeric constants. -The reverse composition -.code "(html-encode (html-decode html))" -does not necessarily return a string equal to -.codn html . +.coNP Variables @, ftw-phys @, ftw-mount @, ftw-chdir @ ftw-depth and @ ftw-actionretval +.desc +These variables hold numeric values that may be combined into a single +bitmask bitmask value using the +.code logior +function. This value is suitable as the +.meta flags +argument of the +.code ftw +function. -For instance if html is the string -.strn "

Hello, world!

" , -then -.code html-decode -produces -.strn "

Hello, world!

" . -From this, -.code html-encode -produces -.strn "<p>Hello, world!</p>" . +These variables corresponds to the C constants +.codn FTW_PHYS , +.codn FTW_MOUNT , +et cetera. -The -.code html-encode* -function is similar to -.code html-encode -except that it does not encode the single and double quote characters -(ASCII 39 and 34, respectively). Text prepared by this function may not -be suitable for insertion into a HTML template, depending on the -context of its insertion. It is suitable as text placed between -tags but not necessarily as tag attribute material. +Note that +.code ftw-actionretval +is a GNU extension that is not available on all platforms. If the platform's +.code nftw +function doesn't have this feature, then this variable is not defined. -.coNP Functions @, base64-encode @ base64-decode and @ base64-decode-buf -.synb -.mets (base64-encode >> [ string | << buf ] <> [ column-width ]) -.mets (base64-decode < string) -.mets (base64-decode-buf < string) -.syne +.coNP Variables @, ftw-f @, ftw-d @, ftw-dnr @, ftw-ns @, ftw-sl @ ftw-dp and @ ftw-sln .desc -The -.code base64-encode -function converts the UTF-8 representation of -.metn string , -or the contents of -.metn buf , -to Base64 and returns that representation as a string. - -The second argument must either be a character string, or -a buffer object. +These variables provide symbolic names for the integer values that are +passed as the +.code type +argument of the callback function called by +.codn ftw . +This argument classifies the kind of file system node visited, or +error condition encountered. -The -.code base64-decode -functions performs the opposite conversion; it extracts the -bytes encoded in a Base64 string, and decodes them as UTF-8 -to return a character string. +These variables correspond to the C constants +.codn FTW_F , +.codn FTW_D , +et cetera. -The -.code base64-decode-buf -extracts the bytes encoded in a Base64 string, and returns -a new buffer object containing these bytes. +Not all of them are present. If the underlying platform doesn't have +a given constant, then the corresponding variable doesn't exist in \*(TX. -The Base64 encoding divides the UTF-8 representation of -.meta string -or the bytes contained in -.meta buf -into groups of six bits, each representing the values 0 to 63. Each value is -then mapped to the characters -.code A -to -.codn Z , -.code a -to -.codn z , -the digits -.code 0 -to -.code 9 -and the characters -.code + -and -.codn / . -One or two consecutive occurrences of the character -.code = -are added as padding so that the number of -non-whitespace characters is divisible by four. These characters map to -the code 0, but are understood not to contribute to the length of the -encoded message. The -.code base64-encode -function enforces this convention, but -.code base64-decode -doesn't require these padding characters. +.coNP Variables @, ftw-continue @, ftw-stop @ ftw-skip-subtree and @ ftw-skip-siblings +.desc +These variables are defined if the variable +.code ftw-actionretval +is defined. -Base64-encoding an empty string or zero-length buffer results in an empty -string. +If the value of +.code ftw-actionretval +is included in the +.meta flags +argument of +.codn ftw , +then the callback function can use the values of these variables +as return codes. Ordinarily, the callback returns zero to continue +the search and nonzero to stop. -If the -.meta column-width -argument is passed to -.codn base64-encode , -then the Base64 encoded string, unless empty, contains newline -characters, which divide it into lines which are -.meta column-width -long, except possibly for the last line. +These variables correspond to the C constants +.codn FTW_CONTINUE , +.codn FTW_STOP , +et cetera. -.coNP Functions @ base64-stream-enc and @ base64-stream-dec +.coNP Function @ ftw .synb -.mets (base64-stream-enc < out < in >> [ nbytes <> [ column-width ]]) -.mets (base64-stream-dec < out << in ) +.mets (ftw < path-or-list < callback-func >> [ flags <> [ nopenfd ]]) +.mets >> [ callback-func < path < type < stat-struct < level << base ] .syne .desc The -.code base64-stream-enc +.code ftw +function provides access to the +.code nftw +POSIX C library function. + +Note that the +.meta flags and -.code base64-stream-dec -perform, respectively, bulk Base64 encoding and decoding between streams. +.meta nopenfd +arguments are reversed with respect to the C language interface. +They are both optional; +.meta flags +defaults to zero, and +.meta nopenfd +defaults to 20. The -.meta in -and -.meta out -arguments must be stream objects. +.meta path-or-list +argument may be a string specifying the top-level path name that +.code ftw +shall visit. Or else, +.meta path-or-list +may be a list. If it is a list, then +.code ftw +recursively invokes itself over each of the elements, taking +that element as the +.meta path-or-name +argument of the recursive call, passing down all other argument +values as-is. +The traversal stops when any recursive invocation of +.code ftw +returns a value other than +.code t +or +.codn nil , +and that value is returned. If +.code t +or +.code nil +is returned, the traversal continues with the +application of +.code ftw +to the next list element, if any. +If the list is completely traversed, and some recursive +invocations of +.code ftw +return +.codn t , +then the return value is +.codn t . +If all recursive invocations return +.code nil +then +.code nil +is returned. +If the list is empty, +.code t +is returned. + The -.meta out -stream must support output. In the decode operation, it must support -byte output. +.code ftw +function walks the filesystem, as directed by the +.meta path-or-list +argument and +.meta flags +bitmask arguments. + +For each visited entry, it calls the supplied +.meta callback-func +function, which receives five arguments. + The -.meta in -stream must support input. In in the encode operation it must support -byte input. +.code ftw +function can continue the traversal by returning any non-integer value, +or the integer value zero. +If +.code ftw-actionretval +is included in the +.meta flags +bitmask, then the only integer code which continues the traversal without +any special semantics is +.code ftw-continue +and only +.code ftw-stop +stops the traversal. (Non-integer return values behave like +.codn ftw-continue ). The -.code base64-stream-enc -function reads a sequence of bytes from the -.meta in -stream and writes characters to the -.meta out -stream comprising the Base64 encoding of that sequence. If the -.meta nbytes -argument is specified, it must be a non-negative integer. At most -.meta nbytes -bytes will be read from the -.meta in -stream. If -.meta nbytes -is omitted, then the operation will read from the -.meta in -stream without limit, until that stream indicates that no more bytes -are available. +.meta path +argument of the callback function gives the path of the +visited filesystem object. -The optional -.meta column-with -argument influences the formatting of Base64 output, in the same manner -as documented for the -.code base64-encode +The +.meta type +argument is an integer code which indicates the kind of +object that is visited, or an error situation in visiting +that filesystem entry. See the documentation for +.code ftw-f +and +.code ftw-d +for possible values. + +The +.meta stat-struct +argument provides information about the filesystem object +as a +.code stat +structure, the same kind of object as what is returned by the +.code stat function. The -.code base64-stream-dec -function reads the characters of a Base64 encoding from the -.meta in -stream and writes the corresponding byte sequence to the -.meta out -stream. It keeps reading and decoding until it encounters the end of the -stream, or a character not used in Base64: a character that is not whitespace -according to -.codn chr-isspace , -isn't any of the Base64 coding characters (not an alphanumeric character, -and not one of the characters -.codn + , -.code / -or -.codn = . -If the function stops due to a non-Base64 character, that character is -pushed back into the -.meta in -stream. +.meta level +argument is an integer value representing the directory level +depth. This value is obtained from the C structure +.code FTW +in the +.code nftw +C API. The -.code base64-stream-enc -function returns the number of bytes encoded; -the -.code base64-stream-dec -function returns the number of bytes decoded. +.meta base +argument indicates the length of the directory part of the +.code path +argument. Characters in excess of this length are thus the base name of the +visited object, and the expression +.mono +.meti >> [ path << base ..:] +.onom +calculates the base name. -.SS* Filter Module -The filter module provides a trie (pronounced "try") data structure, -which is suitable for representing dictionaries for efficient filtering. -Dictionaries are unordered collections of keys, which are strings, which -have associated values, which are also strings. A trie can be used to filter -text, such that keys appearing in the text are replaced by the corresponding -values. A trie supports this filtering operation by providing an efficient -prefix-based lookup method which only looks at each input character ones, and -which does not require knowledge of the length of the key in advance. +The +.code ftw +function returns +.code t +upon successful completion and +.code nil +on failure. If +.code ftw +is terminated by a return value from +.metn callback-func , +then that value is returned. Such a value is always a nonzero integer. -.coNP Function @ make-trie -.synb -.mets (make-trie) -.syne -.desc The -.code make-trie -function creates an empty trie. There is no special data type for -a trie; a trie is some existing type such as a hash table. +.meta callback-func +may terminate the traversal by a nonlocal exit, such as by throwing +an exception or performing a block return. -.coNP Function @ trie-add -.synb -.mets (trie-add < trie < key << value ) -.syne -.desc The -.code trie-add -function adds the string -.meta key -to the trie, associating -it with -.metn value . -If -.meta key -already exists in -.metn trie , -then the value is updated with -.metn value . +.meta callback-func +may not re-enter the +.code ftw +function. This situation is detected and diagnosed by an exception. The -.meta trie -must not have been compressed with -.metn trie-compress . +.meta callback-func +may not capture a continuation across the callback boundary. That is to say, +code invoked from the callback may not capture a continuation up to a prompt +which surrounds the +.code ftw +call. Such an attempt is detected and diagnosed by an exception. -A trie can contain keys which are prefixes of other keys. For instance -it can contain -.str dog +.SS* Unix Sockets + +On platforms where the underlying system interface is available, \*(TX provides +a sockets library for communicating over Internet networks, or over Unix +sockets. + +Stream as well as datagram sockets are supported. + +The classic Version 4 of the Internet protocol is supported, as well +as IP Version 6. + +Sockets are mapped to \*(TX streams. The +.code open-socket +function creates a socket of a specified type, in a particular address family. +This socket is actually a stream (always, even if it isn't used for +data transfer, but only as a passive contact point). + +The functions +.codn sock-connect , +.codn sock-bind , +.codn sock-listen , +.code sock-accept and -.strn dogma . -When a trie is used for matching -and substitution, the longest match is used. If the input presents -the text -.strn doggy , -then the match is -.strn dog . -If the input is -.strn dogmatic , -then -.str dogma -matches. +.code sock-shutdown +are used for enacting socket communication scenarios. -.coNP Function @ trie-compress -.synb -.mets (trie-compress << trie ) -.syne -.desc -The -.code trie-compress -function changes the representation of -.meta trie -to a representation which occupies less space and supports faster lookups. -The new representation is returned. +Stream sockets use ordinary streams, re-using the same underlying framework +that is used for file I/O and process types. -The compressed representation of a trie does not support the -.code trie-add -function. +Datagram socket streams are implemented using special datagram socket streams. +Datagram socket streams eliminate the need for operations analogous to the +.code sendto +and +.code recvfrom +socket API functions, even in server programs which handle multiple +clients. An overview of datagrams is treated in the following section, +Datagram Socket Streams. The -.code trie-compress -function destructively manipulates -.metn trie , -and may return an object -that is the same object as -.codn trie , -or it may return a different object, -while at the same time still modifying the internals of -.metn trie . -Consequently, the program should not retain the input object -.codn trie , -but use the returned object in its place. +.code getaddrinfo +function is provided for resolving host names and services to IPv4 and IPv6 +addresses. -.coNP Function @ trie-lookup-begin -.synb -.mets (trie-lookup-begin << trie ) -.syne -.desc -The -.code trie-lookup-begin -function returns a context object for performing -an open-coded lookup traversal of a trie. The -.meta tri -argument -is expected to be a trie that was created by the -.code make-trie -function. +Several structure types are provided for representing socket addresses, +and options for +.codn getaddrinfo . -.coNP Function @ trie-lookup-feed-char -.synb -.mets (trie-lookup-feed-char < trie-context << char ) -.syne -.desc -The -.code trie-lookup-feed-char -function performs a one character step in a trie -lookup. The -.meta trie-context -argument must be a trie context returned -by -.metn trie-lookup-begin , -or by some previous call to -.codn trie-lookup-feed-char . -The -.meta char -argument is the next character to match. +Various numeric constants are also provided: +.codn af-unix , +.codn af-inet , +.codn af-inet6 , +.codn sock-stream , +.code sock-dgram +and others. -If the lookup is successful (the match through the trie can continue -with the given character) then a new trie context object is returned. -The old trie context remains valid. +.NP* Datagram Socket Streams -If the lookup is unsuccessful, -.code nil -is returned. +Datagram socket streams are a new paradigm unique to \*(TX which +attempts to unify the programming model of stream and datagram +sockets. -Note: determining whether a given string is stored in a trie can be -performed looking up every character of the string successively -with -.codn trie-lookup-feed-char , -using the newly returned context -for each successive operation. If every character is found, it means -that either that exact string is found in the trie, or a prefix. -The ambiguity can be resolved by testing whether the trie has a value -at the last node using -.codn tree-value-at . -For instance, if -.str catalog -is inserted into an empty trie with value -.strn foo , -then -.str cat -will look up successfully, being a prefix of -.strn catalog ; -however, the value at -.str cat -is -.codn nil , -indicating that -.str cat -is only a prefix of one or more entries in the trie. +A datagram socket stream is created by the +.code open-socket +function, when the +.code sock-dgram +socket type is specified. Another way in which a datagram socket +is created is when +.code sock-accept +is invoked on a datagram socket, and returns a new socket. -.coNP Function @ tree-value-at -.synb -.mets (trie-value-at << trie-context ) -.syne -.desc -The -.code trie-value-at -function returns the value stored at the node in -in the trie given by -.metn trie-context . -Nodes which have not been given -a value hold the value -.codn nil . +I/O is performed on datagram sockets using the regular I/O functions. +None of the functions take or return peer addresses. There are no I/O +functions which are analogous to the C library +.code recvfrom +and +.code sendto +functions which are usually used for datagram programming. +Datagram I/O assumes that the datagram datagram socket is connected to a +specific remote peer, and that peer is implicitly used for all I/O. -.coNP Function @ filter-string-tree -.synb -.mets (filter-string-tree < filter << obj ) -.syne -.desc -The -.code filter-string-tree -a tree structure similar to -.metn obj , -in which all of the -string atoms have been filtered through -.metn filter . +Datagram streams solves the message framing problem essentially by +considering a single datagram to be an entire stream. On input, a datagram +stream holds an entire datagram in a buffer. The stream ends +(experiences the EOF condition) after the last byte of this buffer +is removed by an input operation. Another datagram will be received and +buffered if the EOF condition is first explicitly cleared with the +.code clear-error +function, and then another input operation is attempted. +On output, a datagram stream gathers data into an ever-growing output buffer +which isn't subject to any automatic flushing. An explicit +.code flush-stream +operation sends the buffer contents to the connected peer as a new +datagram, and empties the buffer. Subsequent output operations prepare +data for a new datagram. The +.code close-stream +function implicitly flushes the stream in the same way, and thus also +potentially generates a datagram. -The -.meta obj -argument is a string tree structure: either the symbol -.codn nil , -denoting an empty structure; a string; or a list of tree structures. If -.meta obj -is -.codn nil , -then -.code filter-string-tree -returns -.codn nil . +A client-style datagram stream can be explicitly connected to a peer with the +.code sock-connect +function. This is equivalent to connecting a +datagram socket using the C library +.code connect +function. Writes on the stream will be transmitted using the C library function +.codn send . +A client-style datagram stream can also be "soft-connected" to a +peer using the +.code sock-set-peer +function. Writes on the stream will transmit data using the C library function +.code sendto +to the peer address. -The -.meta filter -argument is a filter: it is either a trie, a function, or nil. -If -.meta filter -is -.codn nil , -then -.code filter-string-trie -just returns -.metn obj . +A datagram server program which needs +to communicate using multiple peers is implemented by means of the +.code sock-accept +function which, unlike the C library +.code accept +function, works with datagram sockets as well as stream sockets. +The server creates a datagram socket, and uses +.code sock-bind +to bind it to a local address. Optionally, it may also call +.code sock-listen +which is a no-op on datagram sockets. Supporting this function on datagram +sockets allows program code to be more easily changed between datagram and +stream operation. +The server then uses +.code sock-accept +to accept new clients. Note that this is not possible with the C +library function +.codn accept , +which only works with stream sockets. -If -.meta filter -is a function, it must be a function that can be called -with one argument. The strings of the string tree are filtered by passing -each one into the function and substituting the return value into the -corresponding place in the returned structure. +The +.code sock-accept +function receives a datagram from a client, and creates a new datagram +socket stream which is connected to that client, and whose input buffer +contains the received datagram. Input operations on this stream consume +the datagram. Note that clearing the EOF condition and trying to receive +another datagram is not recommended on datagram streams returned +by +.codn sock-accept , +since they share the same underlying operating system socket, which is +not actually connected to a specific peer. The receive operation could +receive a datagram from any peer, without any indication which peer that is. +Datagram servers should issue a new +.code sock-accept +call should be issued for each client datagram, treating it as a new +stream. -Otherwise if -.meta filter -is a trie, then this trie is used for filtering, -the string elements similarly to a function. For each string, a new -string is returned in which occurrences of the keys in the trie are -replaced by the values in the trie. +Datagram sockets ignore almost all aspects of the +.meta mode-string +passed in +.code open-socket +and +.codn sock-accept . +The only attribute not ignored is the buffer size specified +with a decimal digit character; however, it cannot be the +only item in the mode string. The string must be syntactically +valid, as described under the +.code open-file +function. The buffer size attribute controls the size used by +the datagram socket for receiving a datagram: the capture size. +A datagram socket has obtains a default capture size if one isn't +specified by the +.metn mode-string . +The default capture size is 65536 bytes for a datagram socket created by +.codn open-socket . +If a size is not passed to +.code sock-accept +via its +.meta mode-string +argument when it is invoked on a datagram socket, +that socket's size is used as the capture size of the +newly created datagram socket which is returned. -.coNP Function @ filter-equal +.coNP Structure @ sockaddr .synb -.mets (filter-equal < filter-1 < filter-2 < obj-1 << obj-2 ) +.mets (defstruct sockaddr nil +.mets \ \ (:static family nil)) .syne .desc The -.code filter-equal -function tests whether two string trees are equal -under the given filters. - -The precise semantics can be given by this expression: - -.mono -.mets (equal (filter-string-tree < filter-1 << obj-1 ) -.mets \ \ \ \ \ \ (filter-string-tree < filter-2 << obj-2 )) -.onom +.code sockaddr +structure represents the abstract base class for socket addresses, from which +several other types are derived: +.codn sockaddr-in , +.code sockaddr-in6 +and +.codn sockaddr-un . -The string tree -.meta obj-1 -is filtered through -.metn filter-1 , -as if by the -.code filter-string-tree -function, and similarly, -.meta obj-2 -is -filtered through -.metn filter-2 . -The resulting structures are compared -using -.codn equal , -and the result of that is returned. +It has a single slot called +.code family +which is static, and initialized to +.codn nil . -.coNP Function @ regex-from-trie +.coNP Structure @ sockaddr-in .synb -.mets (regex-from-trie << trie ) +.mets (defstruct sockaddr-in sockaddr +.mets \ \ (addr 0) (port 0) +.mets \ \ (:static family af-inet)) .syne .desc The -.code regex-from-trie -function returns a representation of -.meta trie -as regular expression abstract syntax, suitable for -processing by the -.code regex-compile -function. +.code sockaddr-in +address represents a socket address used in the context of networking over +IP Version 4. It may be used with sockets in the +.code af-inet +address family. -The values stored in the trie nodes are not represented in -the regular expression. +The +.code addr +slot holds an integer denoting an abstract IPv4 address. For instance the hexadecimal +integer literal constant +.code #x7F000001 +or its decimal equivalent +.code 2130706433 +represents the loopback address, whose familiar "dot notation" is +.codn 127.0.0.1 . +Conversion of the abstract IP address to four bytes in network order, as +required, is handled internally. The -.meta trie -may be one that has been compressed via -.codn trie-compress ; -in fact, a compressed -.meta trie -results in more compact syntax. +.code port +slot holds the TCP or UDP port number, whose value ranges from 0 to 65535. +Zero isn't a valid port; the value is used for requesting an ephemeral port number +in active connections. Zero also appears in situations when the port number isn't required: +for instance, when the +.code getaddrinfo +function is used with the aim of looking up the address of a host, without +caring about the port number. -Note: this function is useful for creating a compact, prefix-compressed -regular expression which matches a list of strings. +The +.code family +static slot holds the value +.codn af-inet . -.coNP Special variable @ *filters* +.coNP Structure @ sockaddr-in6 +.synb +.mets (defstruct sockaddr-in6 sockaddr +.mets \ \ (addr 0) (port 0) (flow-info 0) (scope-id 0) +.mets \ \ (:static family af-inet6)) +.syne .desc The -.code *filters* -special variable holds a hash table which associates symbols with -filters. This hash table defines the named filters used in the -\*(TX pattern language. The names are the hash table keys, and filter -objects are the values. Filter objects are one of three representations. -The value -.code nil -represents a null filter, which performs no filtering, passing the input -string through. A filter object may be a raw or compressed trie. -It may also be a Lisp function, which must be callable with one argument -of string type, and must return a string. +.code sockaddr-in6 +address represents a socket address used in the context of networking over +IP Version 6. It may be used with sockets in the +.code af-inet6 +address family. -The application may define new filters by associating symbolic keys in -.code *filters* -with values which conform to the above representation of filters. +The +.code addr +slot holds an integer denoting an abstract IPv6 address. IPv6 addresses are +pure binary integers up to 128 bits wide. -The behavior is unspecified if any of the predefined filters -are removed or redefined, and are subsequently used, or if the -.code *filters* -variable is replaced or rebound with a hash table value which omits -those keys, or associates them with different values. +The +.code port +slot holds the TCP or UDP port number, whose value ranges from 0 to 65535. +In IPv6, the port number functions similarly to IPv6; see +.codn sockaddr-in . -Note that functions -.codn html-encode , -.code html-encode* +The +.code flow-info and -.code html-decode -use, respectively, the HTML-related -.codn :tohtml , -.code :tohtml* +.code scope-id +are special IPv6 parameters corresponding to the +.code sin6_flowinfo and -.codn :fromhtml . - -.SS* Access To TXR Pattern Language From Lisp - -It is useful to be able to invoke the abilities of the \*(TX pattern Language -from \*(TL. An interface for doing this provided in the form of the -.code match-fun -function, which is used for invoking a \*(TX pattern function. +.code sin6_scope_id +slots of the +.code sockaddr_in6 +C language structure. Their meaning and use are beyond the scope of this document. The -.code match-fun -function has a cumbersome interface which requires the \*(TL program to -explicitly deal with the variable bindings emerging from the pattern match -in the form of an association list. - -To make it the interface easier to use, \*(TX provides -the macros -.codn txr-if , -.code txr-when -and -.codn txr-case . +.code family +static slot holds the value +.codn af-inet . -.coNP Function @ match-fun +.coNP Structure @ sockaddr-un .synb -.mets (match-fun < name < args >> [ input <> [ files ]]) +.mets (defstruct sockaddr-un sockaddr +.mets \ \ path +.mets \ \ (:static family af-unix)) .syne .desc The -.code match-fun -function invokes a \*(TX pattern function whose name is -given by -.metn name , -which must be a symbol. +.code sockaddr-un +address represents a socket address used for inter-process communication +within a single operating system node, using the "Unix domain" sockets +of the +.code af-unix +address family. -The -.meta args -argument is a list of expressions. The expressions may be symbols -which will be interpreted as pattern variables, and may be bound or unbound. -If they are not symbols, then they are treated as expressions (of the -pattern language, not \*(TL) and evaluated accordingly. +This structure has only one slot, +.code path +which holds the rendezvous name for connecting pairs of socket endpoints. +This name appears in the filesystem. -The -.meta input -argument is a list of strings, which may be lazy. It represents the -lines of the text stream to be processed. If omitted, it defaults to -.codn nil . +When the +.code sockaddr-un +structure is converted to the C structure +.codn "struct sockaddr_un" , +the +.code path +slot undergoes conversion to UTF-8. The resulting bytes are stored in the +.code sun_path +member of the C structure. If the resulting UTF-8 byte string +is larger than the +.code sun_path +array, it is silently truncated. -The -.meta files -argument is a list of filename specifications, which follow -the same conventions as files given on the \*(TX command line. If the pattern -function uses the -.code @(next) -directive, it can process these additional files. If this argument is -omitted, it defaults to -.codn nil . +Note: Linux systems have support for "abstract" names which do not appear in +the filesystem. These abstract names are distinguished by starting with a null +byte. For more information, consult Linux documentation. +This convention is supported in the +.code path +slot of the +.code sockaddr-un +structure. If +.code path +contains occurrences of the pseudo-null character U+DC00, these translate +to null bytes in the +.code sun_path +member of the corresponding C structure +.codn "struct sockaddr_un" . +For example, the path +.str "\exDC00;foo" +is valid and represents an abstract address consisting of the three bytes +.str "foo" +followed by null padding bytes. The -.code match-fun -function's return value falls into three cases. If there is a -match failure, it returns -.codn nil . -Otherwise it returns a cons cell. The -.code car -field -of the cons cell holds the list of captured bindings. The -.code cdr -of the cons cell is one of two values. If the entire input was processed, the -cdr field holds the symbol -.codn t . -Otherwise it holds another cons cell whose -.code car -is the remainder of the list of lines which were not matched, and whose -.code cdr -is the line number. - -.TP* Example: - -.verb - @(define foo (x y)) - @x:@y - @line - @(end) - @(do - (format t "~s\en" - (match-fun 'foo '(a b) - '("alpha:beta" "gamma" "omega") nil))) +.code family +static slot holds the value +.codn af-inet . - Output: - (((a . "alpha") (b . "beta")) ("omega") . 3) -.brev +.coNP Structure @ addrinfo +.synb +.mets (defstruct addrinfo nil +.mets \ \ (flags 0) (family 0) (socktype 0)) +.syne +.desc +The +.code addrinfo +structure is used in conjunction with the +.code getaddrinfo +function. If that function's +.meta hints +argument is specified, it is of this type. +The purpose of the argument is to narrow down +or possibly alter the selection of addresses which +are returned. -In the above example, the pattern function -.code foo -is called with arguments -.codn "(a b)" . -These are unbound variables, so they correspond to parameters -.code x -and -.code y -of the function. If -.code x -and -.code y -get bound, those values propagate to -.code a -and -.codn b . -The data being matched consists of the lines -.strn alpha:beta , -.str gamma -and -.strn omega . -Inside -.codn foo , -.code x -and -.code y -bind to -.str alpha +The +.code flags +slot holds a bitwise or combination (see the +.code logior +function) of +.code getaddrinfo +flags: values given by the variables. +.codn ai-passive , +.codn ai-numerichost , +.codn ai-v4mapped , +.codn ai-all , +.code ai-addrconfig and -.strn beta , -and then the line variable binds to -.strn gamma . -The input stream is left with -.strn omega . +.codn ai-numericserv . +These correspond to the C constants +.codn AI_PASSIVE , +.code AI_NUMERICHOST +and so forth. -Hence, the return value consists of the bindings of -.code x -and -.code y -transferred to -.code a +The +.code family +slot holds an address family, which may be the value of +.codn af-unspec , +.codn af-unix , +.code af-inet +or +.codn af-inet6 . + +The +.code socktype +slot holds, a socket type. Socket types are given +by the variables +.code sock-dgram and -.codn b , -and the second cons cell which gives information about the rest of the -stream: it is the part starting at -.strn omega , -which is line 3. Note that the binding for the -.code line -variable does not propagate -out of the pattern function -.codn foo ; -it is local inside it. +.codn sock-stream . -.coNP Macro @ txr-if +.coNP Function @ getaddrinfo .synb -.mets (txr-if < name <> ( argument *) < input -.mets \ \ \ \ \ \ \ < then-expr <> [ else-expr ]) +.mets (getaddrinfo >> [ node >> [ service <> [ hints ]]]) .syne .desc The -.code txr-if -macro invokes the \*(TX pattern matching function -.meta name -on some input given by the -.meta input -parameter, which is a list of strings, or a single string. +.code getaddrinfo +returns a list of socket addresses based on search criteria expressed +in its arguments. +That is to say, the returned list, unless empty, contains objects of type +.code sockaddr-in +and +.codn sockaddr-in6 . -If -.meta name -succeeds, then -.meta then-expr -is evaluated, and if it fails, -.meta else-expr -is evaluated instead. +The function is implemented directly in terms of the like-named C library +function. All parameters are optional. Omitting any argument causes a null +pointer to be passed for the corresponding parameter of the C library function. -In the successful case, -.meta then-expr -is evaluated in a scope in which the bindings emerging from the -.meta name -function are turned into \*(TL variables. -The result of -.code txr-if -is that of -.metn then-expr . +The +.meta node +and +.meta service +parameters may be character strings which specify a host name, and service. +The contents of these strings may be symbolic, like +.str www.example.com +and +.str ssh +or numeric, like +.str 10.13.1.5 +and +.strn 80 . -In the failed case, -.meta else-expr -is evaluated in a scope which does not have any new bindings. -The result of -.code txr-if -is that of -.metn else-expr . -If -.meta else-expr -is missing, the result is -.codn nil . +If an argument is given for the +.code hints +parameter, it must be of type +.codn addrinfo . The -.meta argument -forms supply arguments to the pattern function -.metn name . -There must be as many of these arguments as the function -has parameters. - -Any argument which is a symbol is treated, for the purposes -of calling the pattern function, as an unbound pattern variable. -The function may or may not produce a binding for that variable. -Also, every argument which is a symbol also denotes a local variable -that is established around -.meta then-expr -if the function succeeds. For any such pattern variable for which the function -produces a binding, the corresponding local variable will be initialized -with the value of that pattern variable. For any such pattern variable -which is left unbound by the function, the corresponding local variable -will be set to -.codn nil . +.meta node +and +.meta service +parameters may also be given integer arguments. +An integer argument value in either of these parameters is converted to a null +pointer when calling the C +.code getaddrinfo +function. The integer values are then simply installed into every returned +address as the IP address or port number, respectively. However, if both +arguments are numeric, then no addresses are returned, since the C library +function is then called with a null node and service. -Any -.meta argument -can be a form other than a symbol. In this situation, the argument is -evaluated, and will be passed to the pattern function as the value of -the binding for the corresponding argument. +.coNP Variables @, af-unix @ af-inet and @ af-inet6 +.desc +These variables hold integers which give the values of address +families. They correspond to the C constants +.codn AF_UNIX , +.code AF_INET +and +.codn AF_INET6 . +Address family values are used in the +.meta hints +argument of the +.code getaddrinfo +function, and in the +.code socket-open +function. +Note that unlike the C language socket addressing structures, +the \*(TX socket addresses do not contain an address family slot. +That is because they indicate their family via their type. +That is to say, an object of type +.code sockaddr-in +is an address which is implicitly associated with the +.code af-inet +family via its type. -.TP* Example: +.coNP Variables @ sock-stream and @ sock-dgram +.desc +These variables hold integers which give the values of address +families. They correspond to the C constants +.code SOCK_STREAM +and +.codn SOCK_DGRAM . -.verb - @(define date (year month day)) - @{year /\ed\ed\ed\ed/}-@{month /\ed\ed/}-@{day /\ed\ed/} - @(end) - @(do - (each ((date '("09-10-20" "2009-10-20" - "July-15-2014" "foo"))) - (txr-if date (y m d) date - (put-line `match: year @y, month @m, day @d`) - (put-line `no match for @date`)))) +.coNP Variables @, ai-passive @, ai-numerichost @, ai-v4mapped @, ai-all @ ai-addrconfig and @ ai-numericserv +.desc +These variables hold integers which are bitmasks that combine +together via bitwise or, to express the +.code flags +slot of the +.code addrinfo +structure. They correspond to the C constants +.codn AI_PASSIVE , +.codn AI_NUMERICHOST , +.code AI_V4MAPPED +and so forth. They influence the behavior of the +.code getaddrinfo +function. - Output: +.coNP Variables @, inaddr-any @, inaddr-loopback @ in6addr-any and @ in6addr-loopback +.desc +These integer-valued variables provide constants for commonly used IPv4 +and IPv6 address values. - no match for 09-10-20 - match: year 2009, month 10, day 20 - no match for July-15-2014 - no match for foo -.brev +The value of +.code inaddr-any +and +.code in6addr-any +is zero. This address is used in binding a passive socket to all of the +external interfaces of a host, so that it can accept connections or datagrams +from all attached networks. -.coNP Macro @ txr-when -.synb -.mets (txr-when < name <> ( argument *) < input << form *) -.syne -.desc The -.code txr-when -macro is based on -.codn txr-if . -It is equivalent to +.code inaddr-loopback +variable is IPv4 loopback address, the same integer as the hexadecimal +constant +.code #x7F000001. + +The +.code in6addr-loopback +is the IPv6 loopback address. Its value is 1. -.mono -.meti \ \ (txr-if < name <> ( argument *) < input (progn << form *)) -.onom +.TP* Example: -If the pattern function -.meta name -produces a match, then each -.meta form -is evaluated in the scope of the variables established by the -.meta argument -expressions. The result of the -.code txr-when -form is that of the last -.metn form . +.verb + ;; Construct an IPv6 socket address suitable for binding + ;; a socket to the loopback network, port 1234: + (new sockaddr-in6 addr in6addr-loopback port 1234) -If the pattern function fails then the forms are not evaluated, -and the result value is -.codn nil . + ;; Mistake: IPv4 address used with IPv6 sockaddr. + (new sockaddr-in6 addr inaddr-loopback) +.brev -.coNP Macro @ txr-case +.coNP Function @ open-socket .synb -.mets (txr-case < input-form -.mets \ \ >> {( name <> ( argument *) << form *)}* -.mets \ \ >> [( t << form *)]) +.mets (open-socket < family < type <> [ mode-string ]) .syne .desc The -.code txr-case -macro evaluates -.meta input-form -and then uses the value as an input to zero or more test clauses. -Each test clause invokes the pattern function named by that clause's -.meta name -argument. +.code open-socket +function creates a socket, which is a kind of stream. -If the function succeeds, then each -.meta form -is evaluated, and the value of the last -.meta form -is taken to be the result value of -.codn txr-case , -which terminates. If there are no forms, then -.code txr-case -terminates with a -.code nil -result. +The +.meta family +parameter specifies the address family of the socket. One of the +values +.codn af-inet , +.code af-inet +or +.code af-inet6 +should be used to create a Unix domain, Internet IPv4 or Internet IPv6 +socket, respectively. -The forms are evaluated in an environment in which variables are bound -based on the -.meta argument -forms, with values depending on the result of the -invocation of the -.meta name -pattern function, in the same manner as documented in detail for the -.code txr-if -macro. +The +.meta type +parameter specifies the socket type, either +.code sock-stream +(stream socket) or +.code sock-dgram +(datagram socket). -If the function fails, then the forms are not evaluated, and control passes to -the next clause. +The +.meta mode-string +specifies several properties of the stream; for a description of +.meta mode-string +parameters, refer to the +.code open-file +function. Note that the defaulting behavior for an omitted +.meta mode-string +argument is different under +.code open-socket +from other functions. Because sockets are almost always used for bidirectional +data flow, the default mode string is +.str r+b +rather than the usual +.strn r . -A clause which begins with the symbol -.code t -executes unconditionally and causes -.code txr-case -to terminate. If it has no forms, then -.code txr-case -yields -.codn nil , -otherwise the forms are evaluated in order and the value of the last -one specifies the result of -.codn txr-case . +Rationale for including the +.str b +flag in the default mode string is that network protocols are usually defined +in a way that is independent of machine and operating system, down to the byte +level, even when they are textual. It doesn't make sense for the same \*(TX +program to see a network stream differently based on what platform it is +running on. Line ending conversion has to do with how a platform locally stores +text files, whereas network streams are almost always external formats. -.SS* Debugging Functions -.coNP Functions @ source-loc and @ source-loc-str +Like other stream times, stream sockets are buffered and marked as no +non-real-time streams. Specifying the +.str i +mode in +.meta mode-string +marks a socket as a real-time-stream, and, if it is opened for writing +or reading and writing, changes it to use line buffering. + +.coNP Function @ open-socket-pair .synb -.mets (source-loc << form ) -.mets (source-loc-str < form <> [ alternative ]) +.mets (open-socket-pair < family < type <> [ mode-string ]) .syne .desc -These functions map an expression in a \*(TX program to the file name and -line number of the source code where that form came from. - The -.code source-loc -function returns the raw information as a cons cell -whose -.cod3 car / cdr -consist of the line number, and file name. +.code open-socket-pair +function provides an interface to the functionality of the +.code socketpair +C library function. -The -.code source-loc-str -function formats the information as a string. +If successful, it creates and returns a list of two stream objects, +which are sockets that are connected together. -Forms which were parsed from a file have source location info -tracking to their origin in that file. Forms which are the result -of macro-expansion are traced to the form whose evaluation produced -them. That is to say, they inherit that form's source location info. +Note: the Internet address families +.code af-inet +and +.code af-inet6 +are not supported. -More precisely, when a form is produced by macro-expansion, -it usually consists of material which was passed to the macro as arguments, -plus some original material allocated by the macro, and possibly -literal structure material which is part of the macro code. -After the expansion is produced, any of its constituent material -which already has source location info keeps that info. Those nodes -which are newly allocated by the macro-expansion process inherit -their source location info from the form which yields the expansion. +The +.code mode-string +is applied to each stream. For a description, see +.code open-socket +and +.codn open-file . -If -.meta form -is not a piece of the program source code that was constructed by the -\*(TX parser or by a macro, and thus it was neither attributed with -source location info, nor has it inherited such info, then -.code source-loc -returns +.coNP Functions @ sock-family and @ sock-type +.synb +.mets (sock-family << socket ) +.mets (sock-type << socket ) +.syne +.desc +These functions retrieve the integer values representing the address family +and type of a socket. The argument to the +.meta socket +parameter must be a socket stream or a file or process stream. For a file stream, +both functions return .codn nil . +An exception of type +.code type-error +is thrown for other stream types. -In the same situation, and if its -.meta alternative -argument is missing, the -.code source-loc-str -returns a string whose text conveys that the source location is not -available. If the -.meta alternative -argument is present, it is returned. - -.coNP Functions @ rlcp and @ rlcp-tree +.coNP Accessor @ sock-peer .synb -.mets (rlcp < dest-form << source-form ) -.mets (rlcp < dest-tree << source-form ) +.mets (sock-peer << socket ) +.mets (set (sock-peer << socket ) << address ) .syne .desc The -.code rlcp -function copies the source code location info ("rl" means "read location") -from the -.meta source-form -object to the -.meta dest-form -object. These objects -are pieces of list-based syntax. -If -.meta dest-form -already has source code location info, then no copying -takes place. - -The -.code rlcp-tree -function copies the source code location info from -.code rlcp -into every cons cell in the -.meta dest-tree -tree structure which doesn't already have location info. -It may be regarded as a recursive application of -.code rlcp -via -.cod3 car / cdr -recursion on the tree structure. However, the traversal performed by -.code rlcp-tree -gracefully handles circular structures. - -Note: these functions are intended to be used in certain kinds of macros. If a -macro transforms -.meta source-form -to -.metn dest-form , -this function can be used to propagate the -source code location info also, so that when the \*(TL evaluator -encounters errors in transformed code, it can give diagnostics which refer -to the original untransformed source code. +.code sock-peer +function retrieves the peer address +has most recently been assigned to +.metn socket . -The macro expander already performs this transfer. If a macro call form -has location info, the expander propagates that info to that form's -expansion. In some situations, it is useful for a macro or other code -transformer to perform this action explicitly. +Sockets which are not connected initially +have a peer address value of +.codn nil . +A socket which is connected to a remote peer +receives that peer's address as its +.codn sock-peer . -.coNP Special variable @ *rec-source-loc* -.desc -The Boolean special variable -.code *rec-source-loc* -controls whether the -.code read -and -.code iread -functions record source location info. The variable is -.code nil -by default, so that these functions do not record source location info. -If it is true, then these functions record source location info. +If a socket is connected to a remote peer via +a successful use of the +.code sock-connect +function, then its +.code sock-peer +address is set to match that of the peer. -Regardless of the value of this variable, source location info is -recorded for Lisp forms which are read from files or streams under the -.code load -function or specified on the \*(TX command line. Source location -info is also always recorded when reading the \*(TX pattern language syntax. +Sockets returned by the +.code sock-accept +function are connected, and have the remote endpoint address as their +.code sock-peer +address. -Note: recording and propagating location info incurs a memory and performance -penalty. The individual cons cells and certain other literal objects in the -structure which emerges from the parser are associated with source location -info via a global weak hash table. +Assigning an address to a +.code sock-peer +form is equivalent to using +.code sock-set-peer +to set the address. +Implementation note: the +.code sock-peer +function does not use the +.code getpeername +C library function; the association between a stream and +.code sockaddr +struct is maintained by \*(TX. -.coNP Function @ macro-ancestor +.coNP Function @ sock-set-peer .synb -.mets (macro-ancestor << form ) +.mets (sock-set-peer < socket << address ) .syne .desc The -.code macro-ancestor -function returns information about the macro-expansion ancestor of -.metn form . -The ancestor is the original form whose expansion produced -.metn form . +.code sock-set-peer +function stores +.meta address +into +.meta socket +as that socket's peer. + +Subsequently, the +.code sock-peer +function will retrieve that address. If -.meta form -is not the result of macro-expansion, or the ancestor information -is unavailable, the function returns -.codn nil . +.meta address +is not an appropriate address object in the address family of +.metn socket , +the behavior is unspecified. -.SS* Profiling -.coNP Operator @ prof +.coNP Function @ sock-connect .synb -.mets (prof << form *) +.mets (sock-connect < socket < address <> [ timeout-usec ]) .syne .desc The -.code prof -operator evaluates the enclosed forms from left to right similarly -to -.codn progn , -while determining the memory allocation requests and time -consumed by the evaluation of the forms. - -If there are no forms, the prof operator measures the smallest measurable -operation of evaluating nothing and producing -.codn nil . - -If the evaluation terminates normally (not abruptly by a non-local -control transfer), then -.code prof -yields a list consisting of: - -.mono -.mets >> ( value < malloc-bytes < gc-bytes << milliseconds ) -.onom +.code sock-connect +function connects a socket stream to a peer address. -where -.meta value -is the value returned by the rightmost -.metn form , -or -.code nil -if there are no forms, -.meta malloc-bytes -is the total number of bytes of all memory allocation -requests (or at least those known to the \*(TX runtime, such as those of all -internal objects), -.meta gc-bytes -is the total number of bytes drawn from the -garbage-collected heaps, and -.meta milliseconds -is the total processor time -consumed over the execution of those forms. +The +.meta address +argument must be a +.code sockaddr +object of type matching the address family of the socket. -Notes: +If the operation fails, an exception of type +.code socket-error +is thrown. Otherwise, the function returns +.metn socket . -The bytes allocated by the garbage collector from the C function -.code malloc -to create -heap areas are not counted as -.metn malloc-bytes . -.meta malloc-bytes -includes storage -such as the space used for dynamic strings, vectors and bignums (in addition to -their gc-heap allocated nodes), and the various structures used by the -.code cobj -type objects such as streams and hashes. Objects in external libraries that use -uninstrumented allocators are not counted: for instance the C -.code "FILE *" -streams. +If the +.meta timeout-usec +argument is specified, it must be a fixnum integer. +It denotes a connection timeout period in microseconds. +If the connection doesn't succeed within the specified timeout, +an exception of type +.code timeout-error +is thrown. -.coNP Macro @ pprof +.coNP Function @ sock-bind .synb -.mets (pprof << form *) +.mets (sock-bind < socket << address ) .syne .desc The -.code pprof -(pretty-printing -.codn prof ) -macro is similar to -.codn progn . -It evaluates -.metn form -s, -and returns the rightmost one, or -.code nil -if there are no forms. +.code sock-connect +function binds a socket stream to a local address. -Over the evaluation of -.metn form -s, -it counts memory allocations, and measures -CPU time. If -.metn form -s -terminate normally, then just prior to returning, -.code pprof -prints these statistics in a concise report on the -.codn *stdout* . +The +.meta address +argument must be a +.code sockaddr +object of type matching the address family of the socket. +If the operation fails, an exception of type +.code socket-error +is thrown. Otherwise, the function returns +.codn t . + +Returns +.code t +if successful. + +.coNP Function @ sock-listen +.synb +.mets (sock-listen < socket <> [ backlog ]) +.syne +.desc The -.code pprof -macro relies on the -.code prof -operator. +.code sock-listen +function prepares +.meta socket +for listening for connections. The +.meta backlog +parameter, if specified, requires an integer +argument. The default value is 16. -.SS* Garbage Collection -.coNP Function @ sys:gc +.coNP Function @ sock-accept .synb -.mets (sys:gc) +.mets (sock-accept < socket >> [ mode-string <> [ timeout-usec ]]) .syne .desc The -.code gc -function triggers garbage collection. Garbage collection means -that unreachable objects are identified and reclaimed, so that their -storage can be re-used. +.code sock-accept +function waits for a client connection on +.metn socket , +which must have been prepared for listening for +connections using +.code sock-bind +and +.codn sock-listen . -The function returns -.code nil -if garbage collection is disabled (and consequently nothing is done), otherwise -.codn t . +If the operation fails, an exception of type +.code socket-error +is thrown. Otherwise, the function returns a new socket +which is connected to the remote peer. -.coNP Function @ sys:gc-set-delta +The peer's address may be retrieved from this socket using +.codn sock-peer . + +The +.code mode-string +parameter is applied to the new socket just like the +similar argument in +.codn socket-open . +It defaults to +.strn r+ . + +If the +.meta timeout-usec +argument is specified, it must be a fixnum integer. +It denotes a timeout period in microseconds. +If no peer connects for the specified timeout, +.code sock-accept +throws an exception of type +.codn timeout-error . + +.coNP Variables @, shut-rd @ shut-wr and @ shut-rdwr +.desc +The values of these variables are useful as the second argument to the +.code sock-shutdown +function. + +.coNP Function @ sock-shutdown .synb -.mets (sys:gc-set-delta << bytes ) +.mets (sock-shutdown < sock <> [ direction ]) .syne .desc The -.code gc-set-delta -function sets the GC delta parameter. - -Note: This function may disappear in a future release of \*(TX or suffer -a backward-incompatible change in its syntax or behavior. +.code sock-shutdown +function indicates that no further communication is to take place on +.meta socket +in the specified direction(s). -When the amount of new dynamic memory allocated since the last garbage -collection equals or exceeds the GC delta, a garbage collection pass is -triggered. From that point, a new delta begins to be accumulated. +If the operation fails, an exception of type +.code socket-error +is thrown. Otherwise, the function returns +.codn t . -Dynamic memory is used for allocating heaps of small garbage-collected objects -such as cons cells, as well as the satellite data attached to some objects: -like the storage arrays of vectors, strings or bignum integers. Most garbage -collector behaviors are based on counting objects in the heaps. +The +.code direction +parameter is one of the values given by the variables +.codn shut-rd , +.code shut-rw +or +.codn shut-rdwr . +These values shut down communication in the read direction, write direction, +or both directions, respectively. -Sometimes a program works with a small number of objects which are very large, -frequently allocating new, large objects and turning old ones into garbage. -For instance a single large integer could be many megabytes long. In such a -situation, a small number of heap objects therefore control a large amount of -memory. This requires garbage collection to be triggered much more often than -when working with small objects, such as conses, to prevent runaway allocation -of memory. It is for this reason that the garbage collector uses the GC delta. +If the argument is omitted, +.code sock-shutdown +defaults to closing the write direction. -There is a default GC delta of 64 megabytes. This may be overridden in -special builds of \*(TX for small systems. +Notes: shutting down is most commonly requested in the write direction, to perform +a "half close". The communicating process thereby indicates that it has written +all the data which it intends to write. When the shutdown action is processed on the +remote end, that end is unblocked from waiting on any further data, and +effectively experiences an "end of stream" condition on its own socket or +socket-like endpoint, while continuing to be able to transmit data. +Shutting down in the reading direction is potentially abrupt. If it is executed +before an "end of stream" indication is received from a peer, it results in an +abortive close. -.coNP Function @ finalize +.coNP Functions @ sock-recv-timeout and @ sock-send-timeout .synb -.mets (finalize < object < function <> [ reverse-order-p ]) +.mets (sock-recv-timeout < sock << usec ) +.mets (sock-send-timeout < sock << usec ) .syne .desc The -.code finalize -function registers -.meta function -to be invoked in the situation when -.meta object -is identified by the garbage collector as unreachable. -A function registered in this way is called a finalizer. - -If and when this situation occurs, the finalizer -.meta function -will be called with -.meta object -as its only argument. - -Multiple finalizer functions can be registered for the same object. -They are all called when the object becomes unreachable. -Finalizers registered against an object may also be invoked -and removed using the -.code call-finalizers -function. - -If the -.meta reverse-order-p -argument isn't specified, or is -.codn nil , -then finalizer is registered at the end of the list. - -If -.meta reverse-order-p -is true, then the finalizer is registered at the front of -the list. +.code sock-recv-timeout +and +.code sock-send-timeout +functions configure, respectively, receive and send timeouts on socket +.metn sock . -Finalizers which are activated in the same finalization processing phase -are called in the order in which they appear in the -registration list. +The +.meta usec +parameter specifies the value, in microseconds. It must be a +.code fixnum +integer. -After a finalization call takes place, its registration is removed. However, -neither -.meta object -nor -.meta function -are reclaimed immediately; they are treated as if they were reachable objects -until at least the next garbage collection pass. -It is therefore safe for -.meta function -to store somewhere a persistent reference to -.meta object -or to itself, thereby reinstating these objects as reachable. +When a receive timeout is configured on a socket, then an +exception of type +.code timeout-error +is thrown when an input operation waits for at least +.code usec +microseconds without receiving input. -A finalizer is itself permitted to call -.code finalize -to register the original -.code object -or any other object for finalization. Such registrations made during -finalization execution are not eligible for the current phase of finalization -processing; they will be processed in a later garbage collection pass. +Similarly, when a send timeout is configured, then an +exception of type +.code timeout-error +is thrown when an output operation waits for at least +.code usec +microseconds for the availability of buffer space in the socket. -.coNP Function @ call-finalizers +.coNP Functions @ str-inaddr and @ str-in6addr .synb -.mets (call-finalizers << object ) +.mets (str-inaddr address <> [ port ]) +.mets (str-in6addr address <> [ port ]) .syne .desc The -.code call-finalizers -function invokes and removes the finalizers, if any, registered against -.metn object . -If any finalizers are called, it returns -.codn t , -otherwise -.codn nil . - -It is permissible for a finalizer function itself to call -.codn call-finalizers . -Such a call can happen in two possible contexts: during actual -reclamation driven by garbage collection, or under the scope of a -.code call-finalizers -invocation from application code. - -Under the scope of garbage-collection-driven reclamation, the -order of finalizer calls may not be what the application logic -expects. For instance even though a finalizer registered for some object -.code A -itself invokes -.codn "(call-finalizers B)" , -it may be the case during GC reclamation that both -.code A +.code str-inaddr and -.code B -are identified as unreachable objects at the same time, and some or all -finalizers registered against -.code B -have already been called before the given -.code A -finalizer performs the explicit -.code call-finalizers -invocation against -.codn B . -Thus the the call either has no effect at all, or only calls some remaining -.code B -finalizers that have not yet been processed, rather than all of them, -as the application expects. - -The application must avoid creating a dependency on the order of -finalization calls, to prevent the situation that the finalization actions are -only correct under an explicit -.code call-finalizers -but incorrect under spontaneous reclamation driven by garbage collection. - -.SS* Modularization -.coNP Variable @ self-path -.desc -This variable holds the invocation path name of the \*(TX program. -The value of -.code self-path -when \*(TL expressions are being evaluated in command line arguments -is the string -.strn cmdline-expr . -The value of -.code self-path -when a \*(TX query is supplied on the command line via the -.code -c -command line option is the string -.strn cmdline . +.code str-in6addr +functions convert an IPv4 and IPv6 address, respectively, to textual +notation which is returned as a character string. +The conversion is done in conformance with RFC 5952, section 4. -Note that for programs read from a file, -.code self-path -holds the resolved name, and not the invocation name. For instance if -.code foo.tl -is invoked using the name -.codn foo , -whereby \*(TX infers the suffix, then -.code self-path -holds the suffixed name. +IPv6 addresses representing IPv6-mapped IPv4 addresses are printed +in the hybrid notation exemplified by +.codn ::ffff:192.168.1.1 . -.coNP Variable @ stdlib -.desc The -.code stdlib -variable expands to the directory where the \*(TX standard library -is installed. It includes the trailing slash. +.meta address +parameter must be a non-negative integer in the appropriate range +for the address type. -Note: there is no need to use the value of this variable to load library -modules. Library modules are keyed to specific symbols, and lazily loaded. When -a \*(TL library function, macro or variable is referenced for the first time, -the library module which defines it is loaded. This includes references -which occur during the code expansion phase, at "macro time", so it works for -macros. In the middle of processing a syntax tree, the expander may encounter a -symbol that is registered for auto-loading, and trigger the load. When the load -completes, the symbol might now be defined as a macro, which the expander -can immediately use to expand the given form that is being traversed. +If the +.meta port +number argument is supplied, it is included in the returned character string, +according to the requirements in section 6 of RFC 5952 pertaining to IPv6 +addresses (including IPv6-mapped IPv6 addresses) and section 3.2.3 of RFC 3986 +for IPv4 addresses. In brief, IPv6 addresses with ports are expressed as +.code [address]:port +and IPv6 addresses follow the traditional +.code address:port +pattern. -.coNP Function @ load +.coNP Functions @ str-inaddr-net and @ str-in6addr-net .synb -.mets (load << target ) +.mets (str-inaddr-net < address <> [ width ]) +.mets (str-in6addr-net < address <> [ width ]) .syne .desc -The -.code load -function causes a file containing \*(TL or \*(TX code to be read and processed. -The -.meta target -argument is a string. The function can load \*(TL source files as well -as compiled files. +The functions +.code str-inaddr-net +and +.code str-in6addr-net +convert, respectively, IPv4 and IPv6 network prefix addresses to +the "slash notation". For IPv6 addresses, the requirements of section +2.3 of RFC 4291 are implemented. For IPv4, section 3.1 of RFC 4632 is followed. -Firstly, the value in -.meta target -is converted to a -.I "tentative pathname" -as follows. +The condensed portion of the IP address is always determined by measuring +the contiguous extent of all-zero bits in the least significant position +of the address. For instance an IPv4 address which has at least 24 zero bits +in the least significant position, so that the only nonzero bits are in the +highest octet, is always condensed to a single decimal number: the value of +the first octet. -If -.meta target -specifies a pure relative pathname, as defined by the -.code pure-rel-path-p -function, then a special behavior applies. -If an existing load operation is in progress, then the special variable -.code *load-path* -has a binding. In this case, -.code load -will assume that the relative pathname is a reference relative to the -directory portion of that path name. -If -.code *load-path* -has the value -.codn nil , -then a pure relative -.meta target -pathname is used as-is, and thus resolved relative to the current working -directory. +If the +.meta width +parameter is specified, then its value is incorporated into the returned +textual notation as the width. No check is made whether this width +large enough to span all of the nonzero bits in the address. -Once the tentative path name is determined, -.code load -determines whether the name is suffixed. The name is suffixed if it -ends in any of these four suffixes: -.codn .tlo , -.codn .tl , -.code .txr -or -.codn .txr_profile . +If +.meta width +is omitted, then it is calculated as the number of bits in the address, +excluding the contiguous all-zero bits in the least significant position: +how many times the address can be shifted to the right before a 1 appears +in the least significant bit. -Depending on whether the tentative path name is suffixed, -.code load -tries to make one or more attempts to open several variations of that name. -These variations are called -.I "actual paths" . -If any attempt fails due to an error other than non-existence, -such as a permission error, then no further attempts are made; the -error exception propagates to -.codn load 's -caller. +.SS* Unix Terminal Control -If the tentative path name is suffixed, then -.code load -tries to open a file by that actual path name. If that attempt -fails, no other names are tried. +\*(TX provides access to the terminal control "termios" interfaces defined by +POSIX, and some of the extensions to it in Linux. By using termios, programs +can control serial devices, consoles and virtual terminals. Terminal control +in POSIX revolves around a C language structure called +.codn "struct termios" . +This is mirrored in a \*(TL structure also called +.codn termios . -If the tentative path name is unsuffixed, then first the suffix -.code .tlo -is appended to the name, and an attempt is made to open a file -with this actual path. If that file is not found, then the suffix -.code .tl -is similarly tried. If that file is not found, then the unsuffixed -name is tried. +Like-named \*(TL functions are provided which correspond to the C functions +.codn tcgetattr , +.codn tcsetattr , +.codn tcsendbreak , +.codn tcdrain , +.code tcflush +and +.codn tcflow . -If an unsuffixed file is opened, its contents are treated as interpreted Lisp. -Files ending in -.code .txr_profile -are also treated as interpreted Lisp. Files ending in -.code .tlo -are treated as compiled Lisp, and those ending in -.code .txr -are treated as the \*(TX Pattern Language. +These have somewhat different argument conventions. The TTY device is specified last, +so that it can conveniently default to the +.code *stdin* +stream. A TTY device can be specified as either a stream object or a numeric +file descriptor. -If the file is treated as \*(TL, then Lisp forms are read from it in -succession. Each form is evaluated as if by the -.code eval -function, before the next form is read. -If a syntax error is encountered, an exception of type -.code eval-error -is thrown. +The functions +.codn cfgetispeed , +.codn cfgetospeed , +.code cfsetispeed +and +.code cfsetospeed +are not provided, because they are unnecessary. Device speed (informally, "baud rate") +is specified directly as a integer value in the +.code termios +structure. The \*(TL termios functions automatically convert between integer +values and the speed constants (like +.codn B38400 ) +used by the C API. -If a file is treated as a compiled \*(TL object file, then the compiled images -of top-level forms are read from it, converted into compiled objects, and -executed. +All of the various termios-related constants are provided, including some non-standard +ones. They appear in lower case. For instance +.code IGNBRK +and +.code PARENB +are simply known as the predefined \*(TL variables +.code ignbrk +and +.codn parenb . -If the file treated as \*(TX Pattern Language code, -then its contents are parsed in their entirety. -If the parse is successful, the query is executed. -Previous \*(TX pattern variable and function bindings are in -effect. If the query binds new variables and functions, -these emerge from the -.code load -and take effect. If the parse is unsuccessful, an exception of type -.code query-error -is thrown. +.coNP Structure @ termios +.synb +.mets (defstruct termios nil +.mets \ \ iflag oflag cflag lflag +.mets \ \ cc ispeed ospeed) +.syne +.desc +The +.code termios +structure represents the kernel level terminal device configuration. +It holds hardware related setting such as serial line speed, parity +and handshaking. It also holds software settings like translations, +and settings affecting input behaviors. The structure closely corresponds +to the C language +.code termios +structure which exists in the POSIX API. -Parser error messages are directed to the -.code *stderr* -stream. +The +.codn iflag , +.codn oflag , +.code cflag +and +.code lflag +slots correspond to the +.codn c_iflag , +.codn c_oflag , +.code c_cflag +and +.code c_lflag +members of the C structure. They hold integer values representing +bit fields. -Over the evaluation of either a \*(TL, compiled file, or \*(TX file, -.code load -establishes a new dynamic binding for several special -variables. The variable -.code *load-path* -is given a new binding containing the actual path name. The -.code *package* -variable is also given a new dynamic binding, whose value is the -same as the existing binding. Thus if the processing of the -loaded file has the effect of altering the value of -.codn *package* , -that effect will be undone when the binding is removed -after the load completes. +.code cc +slot corresponds to the +.code c_cc +member of the C structure. Whereas the +C structure's +.code c_cc +member is an array of the C type +.codn cc_t , +the +.code cc +slot is a vector of integers, whose values must have the same range as the +.code cc_t +type. -When the -.code load -function terminates normally after processing a file, -it returns -.codn nil . -If the file contains a \*(TX pattern query which is -processed to completion, the matching success or failure -of that query has no bearing on the return value of -.codn load . -Note that this behavior is different from the -.code @(load) -directive which itself fails if the loaded query -fails, causing subsequent directives not to be processed. +.coNP Variables @, ignbrk @, brkint @, ignpar @, parmrk @, inpck @, istrip @, inlcr @, igncr @, icrnl @, iuclc @, ixon @, ixany @, ixoff @ imaxbel and @ iutf8 +.desc +These variables specify bitmask values for the +.code iflag +slot of the +.code termios +structure. They correspond to the C language preprocessor symbols +.codn IGNBRK , +.codn BRKINT , +.code IGNPAR +and so forth. -A \*(TX pattern language file loaded with the Lisp -.code load -function does not have the usual implicit access to the -command line arguments, unlike a top-level \*(TX query. -If the directives in the file try to match input, they -work against the -.code *stdin* -stream. The -.code @(next) -directive behaves as it does when no more arguments -are available. +The +.code imaxbel +and +.code iutf8 +variables are specific to Linux and may not be present. +Portable code should test for their presence with +.codn boundp . -If the source or compiled file begins with the characters -.codn #! , -usually indicating "hash bang" script, -.code load -reads reads the first line of the file and discards it. -Processing of the file then begins with the first byte -following that line. +The +.code iuclc +variable is a legacy feature not found on all systems. + +Note: the +.code termios +methods +.code set-iflags +and +.code clear-iflags +provide a convenient means for setting and clearing combinations of +these flags. -.coNP Special variable @ *load-path* +.coNP Variables @, opost @, olcuc @, onlcr @, ocrnl @, onocr @, onlret @, ofill @, ofdel @, vtdly @, vt0 @, vt1 @, nldly @, nl0 @, nl1 @, crdly @, cr0 @, cr1 @, cr2 @, cr3 @, tabdly @, tab0 @, tab1 @, tab2 @, tab3 @, bsdly @, bs0 @, bs1 @, ffdly @ ff0 and @ ff1 .desc -The -.code *load-path* -special variable has a top-level value which is -.codn nil . +These variables specify bitmask values for the +.code oflag +slot of the +.code termios +structure. They correspond to the C language preprocessor symbols +.codn OPOST , +.codn OLCUC , +.code ONLCR +and so forth. -When a file is being loaded, it is dynamically bound to the -path name of that file. This value is visible to the forms -are evaluated in that file during the loading process. +The variable +.code ofdel +is Linux-specific. Portable programs should test for its presence using +.codn boundp . The -.code *load-path* -variable is is bound when a file is loaded from the command -line. +.code olcuc +variable is a legacy feature not found on all systems. -If the -.code -i -command line option is used to enter the interactive listener, -and a file to be loaded is also specified, then the -.code *load-path* -variable remains bound to the name of that file inside the -listener. +Likewise, whether the following groups of symbols are present is +platform-specific: +.codn nldly , +.code nl0 +and +.codn nl1 ; +.codn crdly , +.codn cr0 , +.codn cr1 , +.code cr2 +and +.codn cr3 ; +.codn tabdly , +.codn tab0 , +.codn tab1 , +.code tab2 +and +.codn tab3 ; +.codn bsdly , +.code bs0 +and +.codn bs1 ; +and +.codn ffdly , +.code ff0 +and +.codn ff1 . -The -.code load -function establishes a binding for -.code *load-path* -prior to processing and evaluating all the top-level forms -in the target file. When the forms are evaluated, the binding -is discarded and -.code load -returns. +Note: the +.code termios +methods +.code set-oflags +and +.code clear-oflags +provide a convenient means for setting and clearing combinations of +these flags. -The -.code compile-file -function also establishes a binding for -.codn *load-path* . +.coNP Variables @, csize @, cs5 @, cs6 @, cs7 @, cs8 @, cstopb @, cread @, parenb @, parodd @, hupcl @, clocal @, cbaud @, cbaudex @ cmspar and @ crtscts +.desc +These variables specify bitmask values for the +.code cflag +slot of the +.code termios +structure. They correspond to the C language preprocessor symbols +.codn CSIZE , +.codn CS5 , +.code CS6 +and so forth. -The -.code @(load) -directive, also similarly establishes a binding around the -parsing and processing of a loaded \*(TX source file. +The following are present on Linux, and may not be available +on other platforms. Portable code should test for them using +.codn boundp : +.codn cbaud , +.codn cbaudex , +.code cmspar +and +.codn crtscts . -Also, during the processing of the profile file (see Interactive Profile File), -the variable is bound to the name of that file. +Note: the +.code termios +methods +.code set-cflags +and +.code clear-cflags +provide a convenient means for setting and clearing combinations of +these flags. -.coNP Macro @ load-for -.synb -.mets (load-for >> {( kind < sym << target )}*) -.syne +.coNP Variables @, isig @, icanon @, echo @, echoe @, echok @, echonl @, noflsh @, tostop @, iexten @, xcase @, echoctl @, echoprt @, echoke @, flusho @ pendin and @ extproc .desc -The -.code load-for -macro takes multiple arguments, each of which is a three-element -clause. Each clause specifies that a given -.meta target -file is to be conditionally loaded based on whether a symbol -.meta sym -has a certain kind of binding. +These variables specify bitmask values for the +.code lflag +slot of the +.code termios +structure. They correspond to the C language preprocessor symbols +.codn ISIG , +.codn ICANON , +.code ECHO +and so forth. -Each argument clause has the syntax -.mono -.meti >> ( kind < sym << target ) -.onom -where -.meta kind -is one of the five symbols -.codn var , -.codn fun , -.codn macro , -.code struct -or -.codn pkg . -The -.meta sym -element is a symbol suitable for use as a variable, function -or structure name, and -.meta target -is an expression which is evaluated to produce a value that is suitable -as an argument to the -.code load -function. +The following are present on Linux, and may not be available +on other platforms. Portable code should test for them using +.codn boundp : +.codn iexten , +.codn xcase , +.codn echoctl , +.codn echoprt , +.codn echoke , +.codn flusho , +.code pendin +and +.codn extproc . -First, all -.code target -expressions in all clauses are unconditionally evaluated in left to right -order. Then the clauses are processed in that order. If the -.meta kind -symbol of a clause is -.codn var , -then -.code load-for -tests whether -.meta sym -has a binding in the variable namespace using the -.code boundp -function. If a binding does not exist, then the value of the -.meta target -expression is passed to the -.code load -function. Otherwise, -.code load -is not called. -Similarly, if -.meta kind -is the symbol -.codn fun , -then -.meta sym -is instead tested using -.codn fboundp , -if -.meta kind -is -.codn macro , -then -.meta sym -is tested using -.codn mboundp , -if -.meta kind -is -.codn struct , -then -.meta sym -is tested using -.codn find-struct-type , -and if -.meta kind -is -.codn pkg , -then -.meta sym -is tested using -.codn find-package . +Note: the +.code termios +methods +.code set-lflags +and +.code clear-lflags +provide a convenient means for setting and clearing combinations of +these flags. -When -.code load-for -invokes the -.code load -function, it confirms whether loading file has had the expected effect of -providing a definition of -.meta sym -of the right -.metn kind . -If this isn't the case, an error is thrown. +.coNP Variables @, vintr @, vquit @, verase @, vkill @, veof @, vtime @, vmin @, vswtc @, vstart @, vstop @, vsusp @, veol @, vreprint @, vdiscard @, vwerase @ vlnext and @ veol2 +.desc +These variables specify integer offsets into the vector stored in the +.code cc +slot of the +.code termios +structure. They correspond to the C language preprocessor symbols +.codn VINTR , +.codn VQUIT , +.code VERASE +and so forth. -The -.code load-for -function returns -.codn nil . +The following are present on Linux, and may not be available +on other platforms. Portable code should test for them using +.codn boundp : +.codn vswtc , +.codn vreprint , +.codn vdiscard , +.code vlnext +and +.codn veol2 . -.coNP Variable @ txr-exe-path +.coNP Variables @, tcooff @, tcoon @ tcioff and @ tcion .desc -This variable holds the absolute path name of the executable file -of the running \*(TX instance. +These variables hold integer values suitable as the +.meta action +argument of the +.code tcflow +function. They correspond to the C language preprocessor symbols +.codn TCOOFF , +.codn TCOON , +.code TCIOFF +and +.codn TCION . -.SS* Function Tracing +.coNP Variables @, tciflush @ tcoflush and @ tcioflush +.desc +These variables hold integer values suitable as the +.meta queue +argument of the +.code tcflush +function. They correspond to the C language preprocessor symbols +.codn TCIFLUSH , +.code TCOFLUSH +and +.codn TCIOFLUSH . -.coNP Special variable @ *trace-output* +.coNP Variables @, tcsanow @ tcsadrain and @ tcsaflush .desc -The -.code *trace-output* -special variable holds a stream to which all trace output -is sent. Trace output consists of diagnostics enabled by the -.code trace -macro. +These variables hold integer values suitable as the +.meta actions +argument of the +.code tcsetattr +function. They correspond to the C language preprocessor symbols +.codn TCSANOW , +.code TCSADRAIN +and +.codn TCSAFLUSH . -.coNP Macros @ trace and @ untrace +.coNP Functions @ tcgetattr and @ tcsetattr .synb -.mets (trace << function-name *) -.mets (untrace << function-name *) +.mets (tcgetattr <> [ device ]) +.mets (tcsetattr < termios >> [ actions <> [ device ]]) .syne .desc The -.code trace +.code tcgetattr and -.code untrace -macros control function tracing. +.code tcsetattr +functions, respectively, retrieve and install the configuration +of the terminal driver associated with the specified device. -When -.code trace -is called with one or more arguments, it considers each -argument to be the name of a global function. For each -function, it turns on tracing, if it is not already turned on. -If an argument denotes a nonexistent function, or is invalid -function name syntax, -.code trace -terminates by throwing an exception, without processing the -subsequent arguments, or undoing the effects already applied -due to processing the previous arguments. +These functions are wrappers for the like-named POSIX C library functions, +but with different argument conventions, and operating using +a \*(TL structure. -When -.code trace -is called with no arguments, it lists the names of functions -for which tracing is currently enabled. In other cases it -returns -.codn nil . +The +.code tcgetattr +function, if successful, returns a new instance of the +.code termios +structure. -When -.code untrace -is called with one or more arguments, it considers each -argument to be the name of a global function. For each -function, it turns off tracing, if tracing is enabled. +The +.code tcsetattr +function requires an instance of a +.code termios +structure as an argument to its +.meta termios +parameter. -When -.code untrace -is called with no arguments, it disables tracing for all -functions. +A program may alter the settings of a terminal device by +retrieving them using +.codn tcgetattr , +manipulating the structure returned by this function, and +then using +.code tcsetattr +to install the modified structure into the device. The -.code untrace -macro always returns -.code nil -and silently tolerates arguments which are not names of functions -currently being traced. +.meta actions +argument of +.code tcsetattr +may be given as the value of one of the variables +.codn tcsanow , +.code tcsadrain +or +.codn tcsaflush . +If it is omitted, the default is +.codn tcsadrain . -Tracing a function consists of printing a message prior to entry into the -function indicating its name and arguments, and another message upon leaving -the function indicating its return value, which is syntactically correlated -with the entry message, using a combination of matching and indentation. -These messages are posted to the -.code *trace-output* -stream. +If an argument is given for +.meta device +it must be either a stream, or an integer file descriptor. +In either case, it is expected to be associated with a +terminal (TTY) device. -When traced functions call each other or recurse, these trace messages -nest. The nesting is detected and translated into indentation levels. +If the argument is omitted, it defaults to the stream currently +stored in the +.code *stdin* +stream special variable, expected to be associated with +a terminal device. -Tracing works by replacing a function definition with a trace hook function, and -retaining the previous definition. The trace hook calls the previous definition -and produces the diagnostics around it. When -.code untrace -is used to disable tracing, the previous definition is restored. +.TP* Notes: -Methods can be traced; their names are given using -.mono -.meti (meth < struct << slot ) -.onom -syntax: see the -.code func-get-name +The C +.code termios +structure usually does not have members for representing the input +and output speed. \*(TL does not use such members, in any case, even +if they are present. The speeds are encoded in the +.code cc_iflag +and +.code cc_lflag +bitmasks. When retrieving the settings, the +.code tcgetattr +function uses the POSIX functions +.code cfgetispeed +and +.code cfgetospeed +to retrieve the speed values from the C structure. These values +are installed as the +.code ispeed +and +.code ospeed +slots of the Lisp structure. A reverse conversion takes place +when setting are installed using +.codn tcsetattr : +the speed values are taken from the slots, and installed into +the C structure using +.code cfsetispeed +and +.code cfsetospeed +before the structure is passed to the C +.code tcsetattr function. -Macros can be traced; their names are given using -.mono -.meti (macro << name ) -.onom -syntax. Note that -.code trace -will not show the destructured internal macro arguments, but only the -two arguments passed to the expander function: the whole form, and the -environment. +On Linux, TTY devices do not have a separate input and output speed. +The C +.code termios +structure stores only one speed which is taken as both the input +and output speed, with a special exception. The input speed may be +programmed as zero. In that case, it is independently represented. +speed may be programmed as zero. + +.coNP Function @ tcsendbreak +.synb +.mets (tcsendbreak >> [ duration <> [ device ]]) +.syne +.desc +The +.code tcsendbreak +function generates a break signal on serial devices. The +.meta duration +parameter specifies the length of the break signal in milliseconds. +If the argument is omitted, the value 500 is used. + +The +.meta device +parameter is exactly the same as that of the +.code tcsetattr +function. + +.coNP Function @ tcdrain +.synb +.mets (tcdrain <> [ device ]) +.syne +.desc +The +.code tcdrain +function waits until all queued output on a terminal +device has been transmitted. It is a direct wrapper +for the like-named POSIX C function. + +The +.meta device +parameter is exactly the same as that of the +.code tcsetattr +function. + +.coNP Function @ tcflush +.synb +.mets (tcflush < queue <> [ device ]) +.syne +.desc +The +.code tcflush +function discards either untransmitted output data, +or received and yet unread input data, depending on the +value of the +.meta queue +argument. It is a direct wrapper for the like-named +POSIX C function. The -.code trace +.meta queue +argument should be the value of one of the variables +.codn tciflush , +.code tcoflush and -.code untrace -functions return -.codn nil . +.codn tcioflush , +which specify the flushing of input data, output +data or both. -.SS* Dynamic Library Access +The +.meta device +parameter is exactly the same as that of the +.code tcsetattr +function. -.coNP Function @ dlopen +.coNP Function @ tcflow .synb -.mets (dlopen >> [{ lib-name | nil} <> [ flags ]) +.mets (tcflow < action <> [ device ]) .syne .desc The -.code dlopen -function provides access to the POSIX C library function of the -same name. +.code tcflow +function provides bi-directional flow control on the +specified terminal device. It is a direct wrapper +for the like-named POSIX C function. -The argument to the optional -.meta lib-name -parameter may be a character string, or -.codn nil . +The +.meta action +argument should be the value of one of the +variables +.codn tcooff , +.codn tcoon , +.code tcioff +and +.codn tcion . -If it is -.codn nil , -then the POSIX function is called with a null pointer for -its name argument, returning the handle for the main program, -if possible. +The +.meta device +parameter is exactly the same as that of the +.code tcsetattr +function. + +.coNP Methods @, set-iflags @, set-oflags @, set-cflags @, set-lflags @, clear-iflags @, clear-oflags @ clear-cflags and @ clear-lflags +.synb +.mets << termios .(set-iflags << flags *) +.mets << termios .(set-oflags << flags *) +.mets << termios .(set-cflags << flags *) +.mets << termios .(set-lflags << flags *) +.mets << termios .(clear-iflags << flags *) +.mets << termios .(clear-oflags << flags *) +.mets << termios .(clear-cflags << flags *) +.mets << termios .(clear-lflags << flags *) +.syne +.desc +These methods of the +.code termios +structure set or clear multiple flags of the four bitmask flag fields. The .meta flags -argument should be expressed as some bitwise combination of the values -of the variables -.codn rtld-lazy , -.codn rtld-now , -or other -.code rtld- -variables which give names to the -.codn dlopen -related -flags. If the +arguments specify zero or more integer values. These values +are combined together bitwise, as if by the +.code logior +function to form a single effective mask. +If there are no .meta flags -argument is omitted, the default value used is -.codn rtld-lazy . - -If the function succeeds, it returns an object of type -.code cptr -which represents the open library handle ("dlhandle"). +arguments, then the effective mask is zero. -Otherwise it throws an exception, whose message incorporates, if possible, -error text retrieved from the -.code dlerror -POSIX function. +The +.code set-iflags +method sets, in the +.code iflag +slot of the +.meta termios +structure, all of the bits which +are set in the effective mask. That is to say, +the effective mask is combined with the value in +.code iflag +by a +.code logior +operation, and the result is stored back into +.codn iflag . +Similarly, the +.codn set-oflags , +.code set-cflags +and +.code set-lflags +methods operate on the +.codn oflag , +.code cflag +and +.code lflag +slots of the structure. The -.code cptr -handle returned by -.code dlopen -will automatically be subject to -.code dlclose -when reclaimed by the garbage collector. +.code clear-iflags +method clears, in the +.code iflag +slot of the +.meta termios +structure, all of the bits which are +set in the effective mask. That is to say, +the effective mask is bitwise inverted as if +by the +.code lognot +function, and then combined with the +existing value of the +.code iflag +slot using +.codn logand . +The resulting value is stored back into the +.code iflag +slot. +Similarly, the +.codn clear-oflags , +.code clear-cflags +and +.code clear-lflags +methods operate on the +.codn oflag , +.code cflag +and +.code lflag +slots of the structure. -.coNP Function @ dlclose +Note: the methods +.codn go-raw , +.code go-cbreak +and +.code go-canon +are provided for changing the settings to raw, "cbreak" and canonical mode. +These methods should be preferred to directly manipulating the flag and +.code cc +slots. + +.TP* Example + +In this example, +.code tio +is assumed to be a variable holding an instance of a +.code termios +struct: + +.verb + ;; clear the ignbrk, brkint, and various other flags: + tio.(clear-iflags ignbrk brkint parmrk istrip + inlcr igncr icrnl ixon) + + ;; set the csize and parenb flags: + tio.(set-cflags csize parenb) +.brev + +.coNP Methods @ go-raw and @ go-cbreak .synb -.mets (dlclose << dlhandle ) +.mets << termios .(go-raw) +.mets << termios .(go-cbreak) .syne .desc The -.code dlclose -closes the library indicated by -.metn dlhandle , -which must be a -.code cptr -object previously returned by -.codn dlopen . +.code go-raw +and +.code go-cbreak +methods of the +.code termios +structure manipulate the flag slots, as well as certain elements +of the +.code cc +slot, in order to prepare the terminal settings for, respectively, +"raw" and "cbreak" mode, described below. -The handle is closed by passing the stored pointer to the POSIX -.code dlclose -function. The internal pointer contained in the -.code cptr -object is then reset to null. +Note that manipulating the +.code termios +structure doesn't actually put these settings into effect in +the terminal device; the settings represented by the structure must +be installed into the device using +.codn tcsetattr . +There is no way to reverse the effect of these methods. +To precisely restore the previous terminal settings, the program +should retain a copy of the original +.code termios +structure. -It is permissible to invoke -.code dlclose -more than once on a -.code cptr -object which was created by -.codn dlopen . -The first invocation resets the -.code cptr -object's pointer to null; the subsequent invocations -do nothing. +"Raw" mode refers to a configuration of the terminal device driver in which input +and output is passed transparently and without accumulation, conversion or +interpretation. Input isn't buffered into lines; as soon as a single byte is +received, it is available to the program. No special input characters such as +commands for generating an interrupt or process suspend request are processed +by the terminal driver; all characters are treated as input data. Input isn't +echoed; the only output which takes place is that generated by program +output requests to the device. -The -.code dlclose -function returns -.code t -if the POSIX function reports a successful result (zero), otherwise -it returns -.codn nil . -It also returns -.code nil -if invoked on a previously closed, and hence nulled-out -.code cptr -handle. +"Cbreak" mode is named after a concept and function in the "curses" terminal +control library. It refers to a configuration of the terminal device driver +which is less transparent than "raw" mode. Input isn't buffered into lines, +and line editing commands are ordinary input characters, allowing +character-at-a-time input. However, most input translations are preserved, +except that the conversion of CR characters to NL is disabled. The +signal-generating characters are processed in this mode. This latter feature of +the configuration is the likely inspiration for the word "cbreak". Unless +otherwise configured, the interrupt character corresponds to the Ctrl-C key, +and "break" is another term for an interactive interruption. -.coNP Functions @ dlsym and @ dlvsym +.coNP Methods @ string-encode and @ string-decode .synb -.mets (dlsym < dlhandle << sym-name ) -.mets (dlvsym < dlhandle < sym-name << ver-name ) +.mets << termios .(string-encode) +.mets << termios .(string-decode << string ) .syne .desc The -.code dlsym -function provides access to the same-named POSIX function. The -.code dlvsym -function provides access to the same-named GNU C Library function, -if available. +.code string-encode +method converts the terminal state stored in a +.code termios +structure into a textual format, returning that representation +as a character string. The -.meta dlhandle -argument must be a -.code cptr -handle previously returned by -.code dlopen -and not subsequently closed by -.code dlclose -or altered in any way. +.code string-decode +method parses the character representation produced by +.code string-encode +and populates the +.meta termios +structure with the settings are encoded in that string. -The -.meta sym-name -and -.meta ver-name -arguments are character strings. +If a string is passed to +.code string-decode +which wasn't produced by +.codn string-encode , +the behavior is unspecified. An exception may or may not be +thrown, and the contents of +.meta termios +may or may not be affected. -If these functions succeed, they return a -.code cptr -value which holds the address of the symbol which was found -in the library. +Note: the textual representation produced by +.code string-encode +is intended to be identical to that produced by the +.code -g +option of the GNU Coreutils version of the +.code stty +utility, on the same platform. That is to say, the output of +.code "stty -g" +may be used as input into +.codn string-decode , +and the output of +.code string-encode +may be used as an argument to +.codn stty . -If they fail, they return a -.code cptr -object containing a null pointer. +.SS* Unix System Identification -.coNP Functions @ dlsym-checked and @ dlvsym-checked +.coNP Structure @ utsname .synb -.mets (dlsym-checked < dlhandle << sym-name ) -.mets (dlvsym-checked < dlhandle < sym-name << ver-name ) +.mets (defstruct utsname nil +.mets \ \ sysname nodename release +.mets \ \ version machine domainname) .syne .desc The -.code dlsym-checked -and -.code dlvsym-checked -functions are alternatives to -.code dlsym -and -.codn dlvsym , -respectively. Instead of returning a null -.code cptr -on failure, these functions throw an exception. +.code utsname +structure corresponds to the POSIX structure of the same name. +An instance of this structure is returned by the +.code uname +function. -.coNP Variables @, rtld-lazy @, rtld-now @, rtld-global @, rtld-local @, rtld-nodelete @ rtld-noload and @ rtld-deepbind +.coNP Function @ uname +.synb +.mets (uname) +.syne .desc -These variables provide the same values as constants in the POSIX C library -header -.code "" -named -.codn RTLD_LAZY , -.codn RTLD_NOW , -.codn RTLD_LOCAL , -.IR "et cetera" . +The +.code uname +function corresponds to the POSIX +function of the same name. It returns an instance of the +.code utsname +structure. Each slot of the returned structure is +initialized with a character string that identifies the corresponding attribute +of the host system. -.coSS The @ buf type +The host system might not support the reporting of the +NIS domain name. In this case, the +.code domainname +slot of the returned +.code utsname +structure will have the value +.codn nil . -Object of the type -.code buf -are -.IR buffers : -vector-like objects specialized for holding binary data represented as -a sequence of 8 bit bytes. Buffers support operations specialized toward the -encoding of Lisp values into machine-oriented data types, and decoding such -data types into Lisp values. +.SS* Web Programming Support -Buffers are particularly useful in conjunction with the Foreign Function -Interface (FFI), since they can be used to prepare arbitrary data which -can be passed into and out of a function by pointer. They are also useful for -binary I/O. +.coNP Functions @ url-encode and @ url-decode +.synb +.mets (url-encode < string <> [ space-plus-p ]) +.mets (url-decode < string <> [ space-plus-p ]) +.syne +.desc +These functions convert character strings to and from a form which is suitable +for embedding into the request portions of URL syntax. -.coNP Conventions Used by the @ buf-put- Functions +Encoding a string for URL use means identifying in it certain characters that +might have a special meaning in the URL syntax and representing it using +"percent encoding": the percent character, followed by the ASCII value of the +character. Spaces and control characters are also encoded, as are all byte +values greater than or equal to 127 (7F hex). The printable ASCII characters +which are percent-encoded consist of this set: -Buffers support a number of similar functions for converting Lisp numeric -values into common data types, which are placed into the buffer. These -functions are named starting with the -.code buf-put- -prefix, followed by an abbreviated type name. +.verb + :/?#[]@!$&'()*+,;=% +.brev -Each of these functions takes three arguments: -.meta buf -specifies the buffer, -.meta pos -specifies the byte offset position into the buffer which receives -the low-order byte of the data transfer, and -.meta val -indicates the value. +More generally, strings can consists of Unicode characters, but the URL +encoding consists only of printable ASCII characters. Unicode characters in the +original string are encoded by expanding into UTF-8, and applying +percent-encoding the UTF-8 bytes, which are all in the range +.codn \exx80-\exxFF . -If -.meta pos -has a value such that any portion of the data transfer would -like outside of the buffer, the buffer is automatically extended -in length to contain the data transfer. If this extension causes -any padding bytes to appear between the previous length of the -buffer and -.metn pos , -those bytes are initialized to zero. +Decoding is the reverse process: reconstituting the UTF-8 byte sequence +specified by the URL-encoding, and then decoding the UTF-8 sequence into the +string of Unicode characters. -The argument -.meta val -giving the value to be stored must be an integer or character, -except in the case of the types -.meta float -and -.metn double (the -functions -.code buf-put-float -and -.codn buf-put-double ) -for which it is required to be of type -.codn float , -and in case of the function -.code buf-put-cptr -which expects the -.meta val -argument to be a -.code cptr -object. +There is an additional complication: whether or not to encode spaces as plus, +and to decode plus characters to spaces. In encoding, if spaces are not encoded +to the plus character, then they are encoded as +.codn %20 , +since spaces are reserved +characters that must be encoded. In decoding, if plus characters are not +decoded to spaces, then they are left alone: they become plus characters in the +decoded string. The -.meta val -argument must be in range for the data type, or an exception -results. - -Unless otherwise indicated, the stored datum is in the local format -used by the machine with regard to byte order and other representational -details. - -.coNP Conventions Used by the @ buf-get- Functions - -Buffers support a number of similar functions for extracting -common data types, and converting them into Lisp values. -These functions are named starting with the -.code buf-get- -prefix, followed by an abbreviated type name. - -Each of these functions takes two arguments: -.meta buf -specifies the buffer and -.meta pos -specifies the byte offset position into the buffer which holds -the low-order byte of the datum to be extracted. - -If any portion of requested datum lies outside of the boundaries -of the buffer, an error exception is thrown. - -The extracted value is converted to a Lisp datum. For the -majority of these functions, the returned value is of type -integer. The -.code buf-get-float -and -.code buf-get-double -return a floating-point value. +.code url-encode +function performs the encoding process. If the +.code space-plus-p +argument is omitted or specified as +.codn nil , +then spaces are encoded as +.codn %20 . +If the argument is a value other than +.codn nil , +then spaces are encoded as the +character +.code + +.codn (plus) . + The -.code buf-get-cptr -function returns a value of type -.codn cptr . +.code url-decode +function performs the decoding process. If the +.code space-plus-p +argument is omitted or specified as +.codn nil , +then +.code + +.code (plus) +characters in the +encoded data are retained as +.code + +characters in the decoded strings. Otherwise, +plus characters are converted to spaces. -.coNP Function @ make-buf +.coNP Functions @, html-encode @ html-encode* and @ html-decode .synb -.mets (make-buf < len >> [ init-val <> [ alloc-size ]]) +.mets (html-encode << text-string ) +.mets (html-decode << html-string ) .syne .desc The -.code make-buf -function creates a new buffer object which holds -.meta len -bytes. This argument may be zero. - -If -.meta init-val -is present, it specifies the value with which every -byte of the buffer is initialized. If omitted, it -defaults to zero. -bytes. The value of -.meta init-val -must lie in the range 0 to 255. +.code html-encode +and +.code html-decode +functions convert between an HTML and raw +representation of text. The -.meta alloc-size -parameter indicates how much memory to actually allocate for -the buffer. If an argument is specified, its value must not -be less than -.metn len . +.code html-encode +function returns a string which is based on the content of +.metn text-string , +but in which all characters which have special meaning in HTML +have been replaced by HTML codes for representing those characters literally. +The returned string is the HTML-encoded verbatim representation of +.metn text-string . -.coNP Function @ bufp -.synb -.mets (bufp << object ) -.syne -.desc The -.code bufp -function returns -.code t -if -.meta object -is a -.codn buf , -otherwise it returns -.codn nil . +.code html-decode +function converts +.metn html-string , +which may contain HTML +character encodings, into a string which contains the actual characters +represented by those encodings. -.coNP Function @ length-buf -.synb -.mets (length-buf << buf ) -.syne -.desc -The -.code length-buf -function retrieves the buffer length: how many bytes are stored -in the buffer. +The function composition +.code "(html-decode (html-encode text))" +returns a string which is equal to +.codn text . -Note: the generic -.code length -function is also applicable to buffers. +The reverse composition +.code "(html-encode (html-decode html))" +does not necessarily return a string equal to +.codn html . + +For instance if html is the string +.strn "

Hello, world!

" , +then +.code html-decode +produces +.strn "

Hello, world!

" . +From this, +.code html-encode +produces +.strn "<p>Hello, world!</p>" . -.coNP Function @ buf-alloc-size -.synb -.mets (buf-alloc-size << buf ) -.syne -.desc The -.code buf-alloc-size -function retrieves the allocation size of the buffer. +.code html-encode* +function is similar to +.code html-encode +except that it does not encode the single and double quote characters +(ASCII 39 and 34, respectively). Text prepared by this function may not +be suitable for insertion into a HTML template, depending on the +context of its insertion. It is suitable as text placed between +tags but not necessarily as tag attribute material. -.coNP Function @ buf-trim +.coNP Functions @, base64-encode @ base64-decode and @ base64-decode-buf .synb -.mets (buf-trim << buf ) +.mets (base64-encode >> [ string | << buf ] <> [ column-width ]) +.mets (base64-decode < string) +.mets (base64-decode-buf < string) .syne .desc The -.code buf-trim -function reduces the amount of memory allocated to the buffer -to the minimum required to hold it contents, effectively -setting the allocation size to the current length. +.code base64-encode +function converts the UTF-8 representation of +.metn string , +or the contents of +.metn buf , +to Base64 and returns that representation as a string. -The previous allocation size is returned. +The second argument must either be a character string, or +a buffer object. -.coNP Function @ buf-set-length -.synb -.mets (buf-set-length < buf < len <> [ init-val ]) -.syne -.desc The -.code buf-set-length -function changes the length of the buffer. If the buffer -is made longer, the newly added bytes appear at the end, -and are initialized to the value given by -.metn init-val . -If -.meta init-val -is specified, its value must be in the range 0 to 255. -It defaults to zero. +.code base64-decode +functions performs the opposite conversion; it extracts the +bytes encoded in a Base64 string, and decodes them as UTF-8 +to return a character string. -.coNP Function @ copy-buf -.synb -.mets (copy-buf << buf ) -.syne -.desc The -.code copy-buf -function returns a duplicate of -.metn buf : -an object distinct from +.code base64-decode-buf +extracts the bytes encoded in a Base64 string, and returns +a new buffer object containing these bytes. + +The Base64 encoding divides the UTF-8 representation of +.meta string +or the bytes contained in .meta buf -which has the same length and contents, and compares -.code equal +into groups of six bits, each representing the values 0 to 63. Each value is +then mapped to the characters +.code A to -.metn buf . +.codn Z , +.code a +to +.codn z , +the digits +.code 0 +to +.code 9 +and the characters +.code + +and +.codn / . +One or two consecutive occurrences of the character +.code = +are added as padding so that the number of +non-whitespace characters is divisible by four. These characters map to +the code 0, but are understood not to contribute to the length of the +encoded message. The +.code base64-encode +function enforces this convention, but +.code base64-decode +doesn't require these padding characters. -.coNP Accessor @ sub-buf +Base64-encoding an empty string or zero-length buffer results in an empty +string. + +If the +.meta column-width +argument is passed to +.codn base64-encode , +then the Base64 encoded string, unless empty, contains newline +characters, which divide it into lines which are +.meta column-width +long, except possibly for the last line. + +.coNP Functions @ base64-stream-enc and @ base64-stream-dec .synb -.mets (sub-buf < buf >> [ from <> [ to ]]) -.mets (set (sub-buf < buf >> [ from <> [ to ]]) << new-val ) +.mets (base64-stream-enc < out < in >> [ nbytes <> [ column-width ]]) +.mets (base64-stream-dec < out << in ) .syne .desc The -.code sub-buf -function has the same semantics as the -.code sub -function, except that the first argument must be a buffer. - -The extracted sub-range of a buffer is itself a buffer object. - -If -.code sub-buf -is used as a syntactic place, the argument expressions -.metn buf , -.metn from , -.meta to +.code base64-stream-enc and -.meta new-val -are evaluated just once. The prior value, if required, is accessed by calling -.code buf-sub +.code base64-stream-dec +perform, respectively, bulk Base64 encoding and decoding between streams. + +The +.meta in and -.meta new-val -is then stored via -.codn replace-buf . +.meta out +arguments must be stream objects. +The +.meta out +stream must support output. In the decode operation, it must support +byte output. +The +.meta in +stream must support input. In in the encode operation it must support +byte input. -.coNP Function @ replace-buf -.synb -.mets (replace-buf < buf < item-sequence >> [ from <> [ to ]]) -.syne -.desc The -.code replace-buf -function has the same semantics as the -.code replace -function, except that the first argument must be a buffer. +.code base64-stream-enc +function reads a sequence of bytes from the +.meta in +stream and writes characters to the +.meta out +stream comprising the Base64 encoding of that sequence. If the +.meta nbytes +argument is specified, it must be a non-negative integer. At most +.meta nbytes +bytes will be read from the +.meta in +stream. If +.meta nbytes +is omitted, then the operation will read from the +.meta in +stream without limit, until that stream indicates that no more bytes +are available. -The elements of -.code item-sequence -are stored into -.meta buf -as if using the -.code buf-put-u8 -function and therefore must be suitable -.meta val -arguments for that function. +The optional +.meta column-with +argument influences the formatting of Base64 output, in the same manner +as documented for the +.code base64-encode +function. -The of the arguments, semantics and return value given for -.code replace -apply to -.codn replace-buf . +The +.code base64-stream-dec +function reads the characters of a Base64 encoding from the +.meta in +stream and writes the corresponding byte sequence to the +.meta out +stream. It keeps reading and decoding until it encounters the end of the +stream, or a character not used in Base64: a character that is not whitespace +according to +.codn chr-isspace , +isn't any of the Base64 coding characters (not an alphanumeric character, +and not one of the characters +.codn + , +.code / +or +.codn = . +If the function stops due to a non-Base64 character, that character is +pushed back into the +.meta in +stream. -.coNP Function @ buf-put-i8 -.synb -.mets (buf-put-i8 < buf < pos << val ) -.syne -.desc The -.code buf-put-i8 -converts -.meta val -into an eight bit signed integer, and stores it into the buffer at -the offset indicated by -.metn pos . +.code base64-stream-enc +function returns the number of bytes encoded; +the +.code base64-stream-dec +function returns the number of bytes decoded. -The return value is -.metn val . +.SS* Filter Module +The filter module provides a trie (pronounced "try") data structure, +which is suitable for representing dictionaries for efficient filtering. +Dictionaries are unordered collections of keys, which are strings, which +have associated values, which are also strings. A trie can be used to filter +text, such that keys appearing in the text are replaced by the corresponding +values. A trie supports this filtering operation by providing an efficient +prefix-based lookup method which only looks at each input character ones, and +which does not require knowledge of the length of the key in advance. -.coNP Function @ buf-put-u8 +.coNP Function @ make-trie .synb -.mets (buf-put-u8 < buf < pos << val ) +.mets (make-trie) .syne .desc The -.code buf-put-u8 -converts -.meta val -into an eight bit unsigned integer, and stores it into the buffer at -the offset indicated by -.metn pos . - -The return value is -.metn val . +.code make-trie +function creates an empty trie. There is no special data type for +a trie; a trie is some existing type such as a hash table. -.coNP Function @ buf-put-i16 +.coNP Function @ trie-add .synb -.mets (buf-put-i16 < buf < pos << val ) +.mets (trie-add < trie < key << value ) .syne .desc The -.code buf-put-i16 -converts -.meta val -into a sixteen bit signed integer, and stores it into the buffer at -the offset indicated by -.metn pos . +.code trie-add +function adds the string +.meta key +to the trie, associating +it with +.metn value . +If +.meta key +already exists in +.metn trie , +then the value is updated with +.metn value . -The return value is -.metn val . +The +.meta trie +must not have been compressed with +.metn trie-compress . -.coNP Function @ buf-put-u16 +A trie can contain keys which are prefixes of other keys. For instance +it can contain +.str dog +and +.strn dogma . +When a trie is used for matching +and substitution, the longest match is used. If the input presents +the text +.strn doggy , +then the match is +.strn dog . +If the input is +.strn dogmatic , +then +.str dogma +matches. + +.coNP Function @ trie-compress .synb -.mets (buf-put-u16 < buf < pos << val ) +.mets (trie-compress << trie ) .syne .desc The -.code buf-put-u16 -converts -.meta val -into a sixteen bit unsigned integer, and stores it into the buffer at -the offset indicated by -.metn pos . +.code trie-compress +function changes the representation of +.meta trie +to a representation which occupies less space and supports faster lookups. +The new representation is returned. -The return value is -.metn val . +The compressed representation of a trie does not support the +.code trie-add +function. -.coNP Function @ buf-put-i32 +The +.code trie-compress +function destructively manipulates +.metn trie , +and may return an object +that is the same object as +.codn trie , +or it may return a different object, +while at the same time still modifying the internals of +.metn trie . +Consequently, the program should not retain the input object +.codn trie , +but use the returned object in its place. + +.coNP Function @ trie-lookup-begin .synb -.mets (buf-put-i32 < buf < pos << val ) +.mets (trie-lookup-begin << trie ) .syne .desc The -.code buf-put-i32 -converts -.meta val -into a 32 bit signed integer, and stores it into the buffer at -the offset indicated by -.metn pos . - -The return value is -.metn val . +.code trie-lookup-begin +function returns a context object for performing +an open-coded lookup traversal of a trie. The +.meta tri +argument +is expected to be a trie that was created by the +.code make-trie +function. -.coNP Function @ buf-put-u32 +.coNP Function @ trie-lookup-feed-char .synb -.mets (buf-put-u32 < buf < pos << val ) +.mets (trie-lookup-feed-char < trie-context << char ) .syne .desc The -.code buf-put-u32 -converts -.meta val -into a 32 bit unsigned integer, and stores it into the buffer at -the offset indicated by -.metn pos . +.code trie-lookup-feed-char +function performs a one character step in a trie +lookup. The +.meta trie-context +argument must be a trie context returned +by +.metn trie-lookup-begin , +or by some previous call to +.codn trie-lookup-feed-char . +The +.meta char +argument is the next character to match. -The return value is -.metn val . +If the lookup is successful (the match through the trie can continue +with the given character) then a new trie context object is returned. +The old trie context remains valid. + +If the lookup is unsuccessful, +.code nil +is returned. + +Note: determining whether a given string is stored in a trie can be +performed looking up every character of the string successively +with +.codn trie-lookup-feed-char , +using the newly returned context +for each successive operation. If every character is found, it means +that either that exact string is found in the trie, or a prefix. +The ambiguity can be resolved by testing whether the trie has a value +at the last node using +.codn tree-value-at . +For instance, if +.str catalog +is inserted into an empty trie with value +.strn foo , +then +.str cat +will look up successfully, being a prefix of +.strn catalog ; +however, the value at +.str cat +is +.codn nil , +indicating that +.str cat +is only a prefix of one or more entries in the trie. -.coNP Function @ buf-put-i64 +.coNP Function @ tree-value-at .synb -.mets (buf-put-i64 < buf < pos << val ) +.mets (trie-value-at << trie-context ) .syne .desc The -.code buf-put-i64 -converts -.meta val -into a 64 bit signed integer, and stores it into the buffer at -the offset indicated by -.metn pos . - -The return value is -.metn val . +.code trie-value-at +function returns the value stored at the node in +in the trie given by +.metn trie-context . +Nodes which have not been given +a value hold the value +.codn nil . -.coNP Function @ buf-put-u64 +.coNP Function @ filter-string-tree .synb -.mets (buf-put-u64 < buf < pos << val ) +.mets (filter-string-tree < filter << obj ) .syne .desc The -.code buf-put-u64 -converts the value -.meta val -into a 64 bit unsigned integer, and stores it into the buffer at -the offset indicated by -.metn pos . +.code filter-string-tree +a tree structure similar to +.metn obj , +in which all of the +string atoms have been filtered through +.metn filter . -The return value is -.metn val . +The +.meta obj +argument is a string tree structure: either the symbol +.codn nil , +denoting an empty structure; a string; or a list of tree structures. If +.meta obj +is +.codn nil , +then +.code filter-string-tree +returns +.codn nil . -.coNP Function @ buf-put-char -.synb -.mets (buf-put-char < buf < pos << val ) -.syne -.desc The -.code buf-put-char -converts -.meta val -into a value of the C type -.code char -and stores it into the buffer at -the offset indicated by -.metn pos . +.meta filter +argument is a filter: it is either a trie, a function, or nil. +If +.meta filter +is +.codn nil , +then +.code filter-string-trie +just returns +.metn obj . -The return value is -.metn val . +If +.meta filter +is a function, it must be a function that can be called +with one argument. The strings of the string tree are filtered by passing +each one into the function and substituting the return value into the +corresponding place in the returned structure. -Note that the -.code char -type may be signed or unsigned. +Otherwise if +.meta filter +is a trie, then this trie is used for filtering, +the string elements similarly to a function. For each string, a new +string is returned in which occurrences of the keys in the trie are +replaced by the values in the trie. -.coNP Function @ buf-put-uchar +.coNP Function @ filter-equal .synb -.mets (buf-put-uchar < buf < pos << val ) +.mets (filter-equal < filter-1 < filter-2 < obj-1 << obj-2 ) .syne .desc The -.code buf-put-uchar -converts -.meta val -into a value of the C type -.code "unsigned char" -and stores it into the buffer at -the offset indicated by -.metn pos . +.code filter-equal +function tests whether two string trees are equal +under the given filters. -.coNP Function @ buf-put-short +The precise semantics can be given by this expression: + +.mono +.mets (equal (filter-string-tree < filter-1 << obj-1 ) +.mets \ \ \ \ \ \ (filter-string-tree < filter-2 << obj-2 )) +.onom + +The string tree +.meta obj-1 +is filtered through +.metn filter-1 , +as if by the +.code filter-string-tree +function, and similarly, +.meta obj-2 +is +filtered through +.metn filter-2 . +The resulting structures are compared +using +.codn equal , +and the result of that is returned. + +.coNP Function @ regex-from-trie .synb -.mets (buf-put-short < buf < pos << val ) +.mets (regex-from-trie << trie ) .syne .desc The -.code buf-put-short -converts -.meta val -into a value of the C type -.code short -and stores it into the buffer at -the offset indicated by -.metn pos . +.code regex-from-trie +function returns a representation of +.meta trie +as regular expression abstract syntax, suitable for +processing by the +.code regex-compile +function. + +The values stored in the trie nodes are not represented in +the regular expression. -.coNP Function @ buf-put-ushort -.synb -.mets (buf-put-ushort < buf < pos << val ) -.syne -.desc The -.code buf-put-ushort -converts -.meta val -into a value of the C type -.code "unsigned short" -and stores it into the buffer at -the offset indicated by -.metn pos . +.meta trie +may be one that has been compressed via +.codn trie-compress ; +in fact, a compressed +.meta trie +results in more compact syntax. -.coNP Function @ buf-put-int -.synb -.mets (buf-put-int < buf < pos << val ) -.syne +Note: this function is useful for creating a compact, prefix-compressed +regular expression which matches a list of strings. + +.coNP Special variable @ *filters* .desc The -.code buf-put-int -converts -.meta val -into a value of the C type -.code int -and stores it into the buffer at -the offset indicated by -.metn pos . +.code *filters* +special variable holds a hash table which associates symbols with +filters. This hash table defines the named filters used in the +\*(TX pattern language. The names are the hash table keys, and filter +objects are the values. Filter objects are one of three representations. +The value +.code nil +represents a null filter, which performs no filtering, passing the input +string through. A filter object may be a raw or compressed trie. +It may also be a Lisp function, which must be callable with one argument +of string type, and must return a string. + +The application may define new filters by associating symbolic keys in +.code *filters* +with values which conform to the above representation of filters. + +The behavior is unspecified if any of the predefined filters +are removed or redefined, and are subsequently used, or if the +.code *filters* +variable is replaced or rebound with a hash table value which omits +those keys, or associates them with different values. + +Note that functions +.codn html-encode , +.code html-encode* +and +.code html-decode +use, respectively, the HTML-related +.codn :tohtml , +.code :tohtml* +and +.codn :fromhtml . + +.SS* Access To TXR Pattern Language From Lisp + +It is useful to be able to invoke the abilities of the \*(TX pattern Language +from \*(TL. An interface for doing this provided in the form of the +.code match-fun +function, which is used for invoking a \*(TX pattern function. -.coNP Function @ buf-put-uint -.synb -.mets (buf-put-uint < buf < pos << val ) -.syne -.desc The -.code buf-put-uint -converts -.meta val -into a value of the C type -.code "unsigned int" -and stores it into the buffer at -the offset indicated by -.metn pos . +.code match-fun +function has a cumbersome interface which requires the \*(TL program to +explicitly deal with the variable bindings emerging from the pattern match +in the form of an association list. -.coNP Function @ buf-put-long +To make it the interface easier to use, \*(TX provides +the macros +.codn txr-if , +.code txr-when +and +.codn txr-case . + +.coNP Function @ match-fun .synb -.mets (buf-put-long < buf < pos << val ) +.mets (match-fun < name < args >> [ input <> [ files ]]) .syne .desc The -.code buf-put-long -converts -.meta val -into a value of the C type -.code long -and stores it into the buffer at -the offset indicated by -.metn pos . +.code match-fun +function invokes a \*(TX pattern function whose name is +given by +.metn name , +which must be a symbol. -.coNP Function @ buf-put-ulong -.synb -.mets (buf-put-ulong < buf < pos << val ) -.syne -.desc The -.code buf-put-ulong -converts -.meta val -into a value of the C type -.code "unsigned long" -and stores it into the buffer at -the offset indicated by -.metn pos . +.meta args +argument is a list of expressions. The expressions may be symbols +which will be interpreted as pattern variables, and may be bound or unbound. +If they are not symbols, then they are treated as expressions (of the +pattern language, not \*(TL) and evaluated accordingly. -.coNP Function @ buf-put-float -.synb -.mets (buf-put-float < buf < pos << val ) -.syne -.desc The -.code buf-put-float -converts -.meta val -into a value of the C type -.code float -and stores it into the buffer at -the offset indicated by -.metn pos . +.meta input +argument is a list of strings, which may be lazy. It represents the +lines of the text stream to be processed. If omitted, it defaults to +.codn nil . -Note: the conversion of a \*(TL floating-point value -to the C type float may be inexact, reducing the -numeric precision. +The +.meta files +argument is a list of filename specifications, which follow +the same conventions as files given on the \*(TX command line. If the pattern +function uses the +.code @(next) +directive, it can process these additional files. If this argument is +omitted, it defaults to +.codn nil . -.coNP Function @ buf-put-double -.synb -.mets (buf-put-double < buf < pos << val ) -.syne -.desc The -.code buf-put-double -converts -.meta val -into a value of the C type -.code double -and stores it into the buffer at -the offset indicated by -.metn pos . +.code match-fun +function's return value falls into three cases. If there is a +match failure, it returns +.codn nil . +Otherwise it returns a cons cell. The +.code car +field +of the cons cell holds the list of captured bindings. The +.code cdr +of the cons cell is one of two values. If the entire input was processed, the +cdr field holds the symbol +.codn t . +Otherwise it holds another cons cell whose +.code car +is the remainder of the list of lines which were not matched, and whose +.code cdr +is the line number. -.coNP Function @ buf-put-cptr +.TP* Example: + +.verb + @(define foo (x y)) + @x:@y + @line + @(end) + @(do + (format t "~s\en" + (match-fun 'foo '(a b) + '("alpha:beta" "gamma" "omega") nil))) + + Output: + (((a . "alpha") (b . "beta")) ("omega") . 3) +.brev + +In the above example, the pattern function +.code foo +is called with arguments +.codn "(a b)" . +These are unbound variables, so they correspond to parameters +.code x +and +.code y +of the function. If +.code x +and +.code y +get bound, those values propagate to +.code a +and +.codn b . +The data being matched consists of the lines +.strn alpha:beta , +.str gamma +and +.strn omega . +Inside +.codn foo , +.code x +and +.code y +bind to +.str alpha +and +.strn beta , +and then the line variable binds to +.strn gamma . +The input stream is left with +.strn omega . + +Hence, the return value consists of the bindings of +.code x +and +.code y +transferred to +.code a +and +.codn b , +and the second cons cell which gives information about the rest of the +stream: it is the part starting at +.strn omega , +which is line 3. Note that the binding for the +.code line +variable does not propagate +out of the pattern function +.codn foo ; +it is local inside it. + +.coNP Macro @ txr-if .synb -.mets (buf-put-cptr < buf < pos << val ) +.mets (txr-if < name <> ( argument *) < input +.mets \ \ \ \ \ \ \ < then-expr <> [ else-expr ]) .syne .desc The -.code buf-put-cptr -expects -.meta val -to be of type -.codn cptr . -It stores the object's pointer value into the buffer at -the offset indicated by -.metn pos . +.code txr-if +macro invokes the \*(TX pattern matching function +.meta name +on some input given by the +.meta input +parameter, which is a list of strings, or a single string. + +If +.meta name +succeeds, then +.meta then-expr +is evaluated, and if it fails, +.meta else-expr +is evaluated instead. + +In the successful case, +.meta then-expr +is evaluated in a scope in which the bindings emerging from the +.meta name +function are turned into \*(TL variables. +The result of +.code txr-if +is that of +.metn then-expr . + +In the failed case, +.meta else-expr +is evaluated in a scope which does not have any new bindings. +The result of +.code txr-if +is that of +.metn else-expr . +If +.meta else-expr +is missing, the result is +.codn nil . -.coNP Function @ buf-get-i8 -.synb -.mets (buf-get-i8 < buf << pos ) -.syne -.desc The -.code buf-get-i8 -function extracts and returns signed eight bit integer from -.meta buf -at the offset given by -.metn pos . +.meta argument +forms supply arguments to the pattern function +.metn name . +There must be as many of these arguments as the function +has parameters. + +Any argument which is a symbol is treated, for the purposes +of calling the pattern function, as an unbound pattern variable. +The function may or may not produce a binding for that variable. +Also, every argument which is a symbol also denotes a local variable +that is established around +.meta then-expr +if the function succeeds. For any such pattern variable for which the function +produces a binding, the corresponding local variable will be initialized +with the value of that pattern variable. For any such pattern variable +which is left unbound by the function, the corresponding local variable +will be set to +.codn nil . + +Any +.meta argument +can be a form other than a symbol. In this situation, the argument is +evaluated, and will be passed to the pattern function as the value of +the binding for the corresponding argument. + +.TP* Example: + +.verb + @(define date (year month day)) + @{year /\ed\ed\ed\ed/}-@{month /\ed\ed/}-@{day /\ed\ed/} + @(end) + @(do + (each ((date '("09-10-20" "2009-10-20" + "July-15-2014" "foo"))) + (txr-if date (y m d) date + (put-line `match: year @y, month @m, day @d`) + (put-line `no match for @date`)))) -.coNP Function @ buf-get-u8 + Output: + + no match for 09-10-20 + match: year 2009, month 10, day 20 + no match for July-15-2014 + no match for foo +.brev + +.coNP Macro @ txr-when .synb -.mets (buf-get-u8 < buf << pos ) +.mets (txr-when < name <> ( argument *) < input << form *) .syne .desc The -.code buf-get-u8 -function extracts and returns an unsigned eight bit integer from -.meta buf -at the offset given by -.metn pos . +.code txr-when +macro is based on +.codn txr-if . +It is equivalent to -.coNP Function @ buf-get-i16 +.mono +.meti \ \ (txr-if < name <> ( argument *) < input (progn << form *)) +.onom + +If the pattern function +.meta name +produces a match, then each +.meta form +is evaluated in the scope of the variables established by the +.meta argument +expressions. The result of the +.code txr-when +form is that of the last +.metn form . + +If the pattern function fails then the forms are not evaluated, +and the result value is +.codn nil . + +.coNP Macro @ txr-case .synb -.mets (buf-get-i16 < buf << pos ) +.mets (txr-case < input-form +.mets \ \ >> {( name <> ( argument *) << form *)}* +.mets \ \ >> [( t << form *)]) .syne .desc The -.code buf-get-i16 -function extracts and returns a signed 16 bit integer from -.meta buf -at the offset given by -.metn pos . +.code txr-case +macro evaluates +.meta input-form +and then uses the value as an input to zero or more test clauses. +Each test clause invokes the pattern function named by that clause's +.meta name +argument. -.coNP Function @ buf-get-u16 +If the function succeeds, then each +.meta form +is evaluated, and the value of the last +.meta form +is taken to be the result value of +.codn txr-case , +which terminates. If there are no forms, then +.code txr-case +terminates with a +.code nil +result. + +The forms are evaluated in an environment in which variables are bound +based on the +.meta argument +forms, with values depending on the result of the +invocation of the +.meta name +pattern function, in the same manner as documented in detail for the +.code txr-if +macro. + +If the function fails, then the forms are not evaluated, and control passes to +the next clause. + +A clause which begins with the symbol +.code t +executes unconditionally and causes +.code txr-case +to terminate. If it has no forms, then +.code txr-case +yields +.codn nil , +otherwise the forms are evaluated in order and the value of the last +one specifies the result of +.codn txr-case . + +.SS* Debugging Functions +.coNP Functions @ source-loc and @ source-loc-str .synb -.mets (buf-get-u16 < buf << pos ) +.mets (source-loc << form ) +.mets (source-loc-str < form <> [ alternative ]) .syne .desc +These functions map an expression in a \*(TX program to the file name and +line number of the source code where that form came from. + The -.code buf-get-u16 -function extracts and returns an unsigned 16 bit integer from -.meta buf -at the offset given by -.metn pos . +.code source-loc +function returns the raw information as a cons cell +whose +.cod3 car / cdr +consist of the line number, and file name. -.coNP Function @ buf-get-i32 -.synb -.mets (buf-get-i32 < buf << pos ) -.syne -.desc The -.code buf-get-i32 -function extracts and returns a signed 32 bit integer from -.meta buf -at the offset given by -.metn pos . +.code source-loc-str +function formats the information as a string. -.coNP Function @ buf-get-u32 +Forms which were parsed from a file have source location info +tracking to their origin in that file. Forms which are the result +of macro-expansion are traced to the form whose evaluation produced +them. That is to say, they inherit that form's source location info. + +More precisely, when a form is produced by macro-expansion, +it usually consists of material which was passed to the macro as arguments, +plus some original material allocated by the macro, and possibly +literal structure material which is part of the macro code. +After the expansion is produced, any of its constituent material +which already has source location info keeps that info. Those nodes +which are newly allocated by the macro-expansion process inherit +their source location info from the form which yields the expansion. + +If +.meta form +is not a piece of the program source code that was constructed by the +\*(TX parser or by a macro, and thus it was neither attributed with +source location info, nor has it inherited such info, then +.code source-loc +returns +.codn nil . + +In the same situation, and if its +.meta alternative +argument is missing, the +.code source-loc-str +returns a string whose text conveys that the source location is not +available. If the +.meta alternative +argument is present, it is returned. + +.coNP Functions @ rlcp and @ rlcp-tree .synb -.mets (buf-get-u32 < buf << pos ) +.mets (rlcp < dest-form << source-form ) +.mets (rlcp < dest-tree << source-form ) .syne .desc The -.code buf-get-u32 -function extracts and returns an unsigned 32 bit integer from -.meta buf -at the offset given by -.metn pos . +.code rlcp +function copies the source code location info ("rl" means "read location") +from the +.meta source-form +object to the +.meta dest-form +object. These objects +are pieces of list-based syntax. +If +.meta dest-form +already has source code location info, then no copying +takes place. -.coNP Function @ buf-get-i64 -.synb -.mets (buf-get-i64 < buf << pos ) -.syne -.desc The -.code buf-get-i64 -function extracts and returns a signed 64 bit integer from -.meta buf -at the offset given by -.metn pos . +.code rlcp-tree +function copies the source code location info from +.code rlcp +into every cons cell in the +.meta dest-tree +tree structure which doesn't already have location info. +It may be regarded as a recursive application of +.code rlcp +via +.cod3 car / cdr +recursion on the tree structure. However, the traversal performed by +.code rlcp-tree +gracefully handles circular structures. -.coNP Function @ buf-get-u64 -.synb -.mets (buf-get-u64 < buf << pos ) -.syne +Note: these functions are intended to be used in certain kinds of macros. If a +macro transforms +.meta source-form +to +.metn dest-form , +this function can be used to propagate the +source code location info also, so that when the \*(TL evaluator +encounters errors in transformed code, it can give diagnostics which refer +to the original untransformed source code. + +The macro expander already performs this transfer. If a macro call form +has location info, the expander propagates that info to that form's +expansion. In some situations, it is useful for a macro or other code +transformer to perform this action explicitly. + +.coNP Special variable @ *rec-source-loc* .desc -The -.code buf-get-u64 -function extracts and returns an unsigned 64 bit integer from -.meta buf -at the offset given by -.metn pos . +The Boolean special variable +.code *rec-source-loc* +controls whether the +.code read +and +.code iread +functions record source location info. The variable is +.code nil +by default, so that these functions do not record source location info. +If it is true, then these functions record source location info. -.coNP Function @ buf-get-char +Regardless of the value of this variable, source location info is +recorded for Lisp forms which are read from files or streams under the +.code load +function or specified on the \*(TX command line. Source location +info is also always recorded when reading the \*(TX pattern language syntax. + +Note: recording and propagating location info incurs a memory and performance +penalty. The individual cons cells and certain other literal objects in the +structure which emerges from the parser are associated with source location +info via a global weak hash table. + + +.coNP Function @ macro-ancestor .synb -.mets (buf-get-char < buf << pos ) +.mets (macro-ancestor << form ) .syne .desc The -.code buf-get-char -function extracts and returns a value of the C type -.code char -from -.meta buf -at the offset given by -.metn pos . -Note that -.code char -may be signed or unsigned. +.code macro-ancestor +function returns information about the macro-expansion ancestor of +.metn form . +The ancestor is the original form whose expansion produced +.metn form . -.coNP Function @ buf-get-uchar +If +.meta form +is not the result of macro-expansion, or the ancestor information +is unavailable, the function returns +.codn nil . + +.SS* Profiling +.coNP Operator @ prof .synb -.mets (buf-get-uchar < buf << pos ) +.mets (prof << form *) .syne .desc The -.code buf-get-uchar -function extracts and returns a value of the C type -.code "unsigned char" -from -.meta buf -at the offset given by -.metn pos . +.code prof +operator evaluates the enclosed forms from left to right similarly +to +.codn progn , +while determining the memory allocation requests and time +consumed by the evaluation of the forms. -.coNP Function @ buf-get-short +If there are no forms, the prof operator measures the smallest measurable +operation of evaluating nothing and producing +.codn nil . + +If the evaluation terminates normally (not abruptly by a non-local +control transfer), then +.code prof +yields a list consisting of: + +.mono +.mets >> ( value < malloc-bytes < gc-bytes << milliseconds ) +.onom + +where +.meta value +is the value returned by the rightmost +.metn form , +or +.code nil +if there are no forms, +.meta malloc-bytes +is the total number of bytes of all memory allocation +requests (or at least those known to the \*(TX runtime, such as those of all +internal objects), +.meta gc-bytes +is the total number of bytes drawn from the +garbage-collected heaps, and +.meta milliseconds +is the total processor time +consumed over the execution of those forms. + +Notes: + +The bytes allocated by the garbage collector from the C function +.code malloc +to create +heap areas are not counted as +.metn malloc-bytes . +.meta malloc-bytes +includes storage +such as the space used for dynamic strings, vectors and bignums (in addition to +their gc-heap allocated nodes), and the various structures used by the +.code cobj +type objects such as streams and hashes. Objects in external libraries that use +uninstrumented allocators are not counted: for instance the C +.code "FILE *" +streams. + +.coNP Macro @ pprof .synb -.mets (buf-get-short < buf << pos ) +.mets (pprof << form *) .syne .desc The -.code buf-get-short -function extracts and returns a value of the C type -.code short -from -.meta buf -at the offset given by -.metn pos . +.code pprof +(pretty-printing +.codn prof ) +macro is similar to +.codn progn . +It evaluates +.metn form -s, +and returns the rightmost one, or +.code nil +if there are no forms. + +Over the evaluation of +.metn form -s, +it counts memory allocations, and measures +CPU time. If +.metn form -s +terminate normally, then just prior to returning, +.code pprof +prints these statistics in a concise report on the +.codn *stdout* . -.coNP Function @ buf-get-ushort -.synb -.mets (buf-get-ushort < buf << pos ) -.syne -.desc The -.code buf-get-ushort -function extracts and returns a value of the C type -.code "unsigned short" -from -.meta buf -at the offset given by -.metn pos . +.code pprof +macro relies on the +.code prof +operator. -.coNP Function @ buf-get-int +.SS* Garbage Collection +.coNP Function @ sys:gc .synb -.mets (buf-get-int < buf << pos ) +.mets (sys:gc) .syne .desc The -.code buf-get-int -function extracts and returns a value of the C type -.code int -from -.meta buf -at the offset given by -.metn pos . +.code gc +function triggers garbage collection. Garbage collection means +that unreachable objects are identified and reclaimed, so that their +storage can be re-used. -.coNP Function @ buf-get-uint +The function returns +.code nil +if garbage collection is disabled (and consequently nothing is done), otherwise +.codn t . + +.coNP Function @ sys:gc-set-delta .synb -.mets (buf-get-uint < buf << pos ) +.mets (sys:gc-set-delta << bytes ) .syne .desc The -.code buf-get-uint -function extracts and returns a value of the C type -.code "unsigned int" -from -.meta buf -at the offset given by -.metn pos . +.code gc-set-delta +function sets the GC delta parameter. -.coNP Function @ buf-get-long +Note: This function may disappear in a future release of \*(TX or suffer +a backward-incompatible change in its syntax or behavior. + +When the amount of new dynamic memory allocated since the last garbage +collection equals or exceeds the GC delta, a garbage collection pass is +triggered. From that point, a new delta begins to be accumulated. + +Dynamic memory is used for allocating heaps of small garbage-collected objects +such as cons cells, as well as the satellite data attached to some objects: +like the storage arrays of vectors, strings or bignum integers. Most garbage +collector behaviors are based on counting objects in the heaps. + +Sometimes a program works with a small number of objects which are very large, +frequently allocating new, large objects and turning old ones into garbage. +For instance a single large integer could be many megabytes long. In such a +situation, a small number of heap objects therefore control a large amount of +memory. This requires garbage collection to be triggered much more often than +when working with small objects, such as conses, to prevent runaway allocation +of memory. It is for this reason that the garbage collector uses the GC delta. + +There is a default GC delta of 64 megabytes. This may be overridden in +special builds of \*(TX for small systems. + +.coNP Function @ finalize .synb -.mets (buf-get-long < buf << pos ) +.mets (finalize < object < function <> [ reverse-order-p ]) .syne .desc The -.code buf-get-long -function extracts and returns a value of the C type -.code long -from -.meta buf -at the offset given by -.metn pos . +.code finalize +function registers +.meta function +to be invoked in the situation when +.meta object +is identified by the garbage collector as unreachable. +A function registered in this way is called a finalizer. + +If and when this situation occurs, the finalizer +.meta function +will be called with +.meta object +as its only argument. + +Multiple finalizer functions can be registered for the same object. +They are all called when the object becomes unreachable. +Finalizers registered against an object may also be invoked +and removed using the +.code call-finalizers +function. + +If the +.meta reverse-order-p +argument isn't specified, or is +.codn nil , +then finalizer is registered at the end of the list. + +If +.meta reverse-order-p +is true, then the finalizer is registered at the front of +the list. + +Finalizers which are activated in the same finalization processing phase +are called in the order in which they appear in the +registration list. + +After a finalization call takes place, its registration is removed. However, +neither +.meta object +nor +.meta function +are reclaimed immediately; they are treated as if they were reachable objects +until at least the next garbage collection pass. +It is therefore safe for +.meta function +to store somewhere a persistent reference to +.meta object +or to itself, thereby reinstating these objects as reachable. + +A finalizer is itself permitted to call +.code finalize +to register the original +.code object +or any other object for finalization. Such registrations made during +finalization execution are not eligible for the current phase of finalization +processing; they will be processed in a later garbage collection pass. -.coNP Function @ buf-get-ulong +.coNP Function @ call-finalizers .synb -.mets (buf-get-ulong < buf << pos ) +.mets (call-finalizers << object ) .syne .desc The -.code buf-get-ulong -function extracts and returns a value of the C type -.code "unsigned long" -from -.meta buf -at the offset given by -.metn pos . +.code call-finalizers +function invokes and removes the finalizers, if any, registered against +.metn object . +If any finalizers are called, it returns +.codn t , +otherwise +.codn nil . -.coNP Function @ buf-get-float -.synb -.mets (buf-get-float < buf << pos ) -.syne -.desc -The -.code buf-get-float -function extracts and returns a value of the C type -.code float -from -.meta buf -at the offset given by -.metn pos , -returning that value as a Lisp floating-point number. +It is permissible for a finalizer function itself to call +.codn call-finalizers . +Such a call can happen in two possible contexts: during actual +reclamation driven by garbage collection, or under the scope of a +.code call-finalizers +invocation from application code. -.coNP Function @ buf-get-double -.synb -.mets (buf-get-double < buf << pos ) -.syne +Under the scope of garbage-collection-driven reclamation, the +order of finalizer calls may not be what the application logic +expects. For instance even though a finalizer registered for some object +.code A +itself invokes +.codn "(call-finalizers B)" , +it may be the case during GC reclamation that both +.code A +and +.code B +are identified as unreachable objects at the same time, and some or all +finalizers registered against +.code B +have already been called before the given +.code A +finalizer performs the explicit +.code call-finalizers +invocation against +.codn B . +Thus the the call either has no effect at all, or only calls some remaining +.code B +finalizers that have not yet been processed, rather than all of them, +as the application expects. + +The application must avoid creating a dependency on the order of +finalization calls, to prevent the situation that the finalization actions are +only correct under an explicit +.code call-finalizers +but incorrect under spontaneous reclamation driven by garbage collection. + +.SS* Modularization +.coNP Variable @ self-path .desc -The -.code buf-get-double -function extracts and returns a value of the C type -.code double -from -.meta buf -at the offset given by -.metn pos , -returning that value as a Lisp floating-point number. +This variable holds the invocation path name of the \*(TX program. +The value of +.code self-path +when \*(TL expressions are being evaluated in command line arguments +is the string +.strn cmdline-expr . +The value of +.code self-path +when a \*(TX query is supplied on the command line via the +.code -c +command line option is the string +.strn cmdline . -.coNP Function @ buf-get-cptr -.synb -.mets (buf-get-cptr < buf << pos ) -.syne +Note that for programs read from a file, +.code self-path +holds the resolved name, and not the invocation name. For instance if +.code foo.tl +is invoked using the name +.codn foo , +whereby \*(TX infers the suffix, then +.code self-path +holds the suffixed name. + +.coNP Variable @ stdlib .desc The -.code buf-get-cptr -function extracts a C pointer from -.meta buf -at the offset given by -.metn pos , -returning that value as a Lisp object of type -.codn cnum . +.code stdlib +variable expands to the directory where the \*(TX standard library +is installed. It includes the trailing slash. -.coNP Function @ put-buf +Note: there is no need to use the value of this variable to load library +modules. Library modules are keyed to specific symbols, and lazily loaded. When +a \*(TL library function, macro or variable is referenced for the first time, +the library module which defines it is loaded. This includes references +which occur during the code expansion phase, at "macro time", so it works for +macros. In the middle of processing a syntax tree, the expander may encounter a +symbol that is registered for auto-loading, and trigger the load. When the load +completes, the symbol might now be defined as a macro, which the expander +can immediately use to expand the given form that is being traversed. + +.coNP Function @ load .synb -.mets (put-buf < buf >> [ pos <> [ stream ]]) +.mets (load << target ) .syne .desc The -.code put-buf -function writes the contents of buffer -.metn buf , -starting at position -.meta pos -to a stream, through to the last byte, if possible. -Successive bytes from the buffer are written to the stream as if by a -.code put-byte -operation. +.code load +function causes a file containing \*(TL or \*(TX code to be read and processed. +The +.meta target +argument is a string. The function can load \*(TL source files as well +as compiled files. -If -.meta stream -is omitted, it defaults to -.codn *stdout* . +Firstly, the value in +.meta target +is converted to a +.I "tentative pathname" +as follows. If -.meta pos -is omitted, it defaults to zero. +.meta target +specifies a pure relative pathname, as defined by the +.code pure-rel-path-p +function, then a special behavior applies. +If an existing load operation is in progress, then the special variable +.code *load-path* +has a binding. In this case, +.code load +will assume that the relative pathname is a reference relative to the +directory portion of that path name. +If +.code *load-path* +has the value +.codn nil , +then a pure relative +.meta target +pathname is used as-is, and thus resolved relative to the current working +directory. -The stream must support the -.code put-byte -operation. Streams which support -.code put-byte -can be expected to support -.code put-buf -and, conversely, streams which do not support -.code put-byte -do not support -.codn put-buf . +Once the tentative path name is determined, +.code load +determines whether the name is suffixed. The name is suffixed if it +ends in any of these four suffixes: +.codn .tlo , +.codn .tl , +.code .txr +or +.codn .txr_profile . + +Depending on whether the tentative path name is suffixed, +.code load +tries to make one or more attempts to open several variations of that name. +These variations are called +.I "actual paths" . +If any attempt fails due to an error other than non-existence, +such as a permission error, then no further attempts are made; the +error exception propagates to +.codn load 's +caller. + +If the tentative path name is suffixed, then +.code load +tries to open a file by that actual path name. If that attempt +fails, no other names are tried. + +If the tentative path name is unsuffixed, then first the suffix +.code .tlo +is appended to the name, and an attempt is made to open a file +with this actual path. If that file is not found, then the suffix +.code .tl +is similarly tried. If that file is not found, then the unsuffixed +name is tried. + +If an unsuffixed file is opened, its contents are treated as interpreted Lisp. +Files ending in +.code .txr_profile +are also treated as interpreted Lisp. Files ending in +.code .tlo +are treated as compiled Lisp, and those ending in +.code .txr +are treated as the \*(TX Pattern Language. + +If the file is treated as \*(TL, then Lisp forms are read from it in +succession. Each form is evaluated as if by the +.code eval +function, before the next form is read. +If a syntax error is encountered, an exception of type +.code eval-error +is thrown. + +If a file is treated as a compiled \*(TL object file, then the compiled images +of top-level forms are read from it, converted into compiled objects, and +executed. + +If the file treated as \*(TX Pattern Language code, +then its contents are parsed in their entirety. +If the parse is successful, the query is executed. +Previous \*(TX pattern variable and function bindings are in +effect. If the query binds new variables and functions, +these emerge from the +.code load +and take effect. If the parse is unsuccessful, an exception of type +.code query-error +is thrown. + +Parser error messages are directed to the +.code *stderr* +stream. +Over the evaluation of either a \*(TL, compiled file, or \*(TX file, +.code load +establishes a new dynamic binding for several special +variables. The variable +.code *load-path* +is given a new binding containing the actual path name. The -.code put-buf -function returns the position of the last byte that was successfully written. -If the buffer was written through to the end, then this value corresponds to -the length of the buffer. +.code *package* +variable is also given a new dynamic binding, whose value is the +same as the existing binding. Thus if the processing of the +loaded file has the effect of altering the value of +.codn *package* , +that effect will be undone when the binding is removed +after the load completes. -If an error occurs before any bytes are written, the function -throws an error. +When the +.code load +function terminates normally after processing a file, +it returns +.codn nil . +If the file contains a \*(TX pattern query which is +processed to completion, the matching success or failure +of that query has no bearing on the return value of +.codn load . +Note that this behavior is different from the +.code @(load) +directive which itself fails if the loaded query +fails, causing subsequent directives not to be processed. -.coNP Functions @ fill-buf and @ fill-buf-adjust -.synb -.mets (fill-buf < buf >> [ pos <> [ stream ]]) -.mets (fill-buf-adjust < buf >> [ pos <> [ stream ]]) -.syne +A \*(TX pattern language file loaded with the Lisp +.code load +function does not have the usual implicit access to the +command line arguments, unlike a top-level \*(TX query. +If the directives in the file try to match input, they +work against the +.code *stdin* +stream. The +.code @(next) +directive behaves as it does when no more arguments +are available. + +If the source or compiled file begins with the characters +.codn #! , +usually indicating "hash bang" script, +.code load +reads reads the first line of the file and discards it. +Processing of the file then begins with the first byte +following that line. + +.coNP Special variable @ *load-path* .desc The -.code fill-buf -reads bytes from -.meta stream -and writes them into consecutive locations in buffer -.meta buf -starting at position -.metn pos . -The bytes are read as if using the -.code get-byte -function. +.code *load-path* +special variable has a top-level value which is +.codn nil . -If the -.meta stream -argument is omitted, it defaults to -.codn *stdin* . +When a file is being loaded, it is dynamically bound to the +path name of that file. This value is visible to the forms +are evaluated in that file during the loading process. -If -.meta pos -is omitted, it defaults to zero. +The +.code *load-path* +variable is is bound when a file is loaded from the command +line. -The stream must support the -.code get-byte -operation. Buffers which support -.code get-byte -can be expected to support -.code fill-buf -and, conversely, streams which do not support -.code get-byte -do not support -.codn fill-buf . +If the +.code -i +command line option is used to enter the interactive listener, +and a file to be loaded is also specified, then the +.code *load-path* +variable remains bound to the name of that file inside the +listener. The -.code fill-buf -function returns the position of the last byte that was successfully read. -If an end-of-file or other error condition occurs before the buffer is filled -through to the end, then the value returned is smaller than the buffer length. -In this case, the area of the buffer beyond the read size retains its previous -content. - -If an error situation occurs other than a premature end-of-file before -any bytes are read, then an exception is thrown. +.code load +function establishes a binding for +.code *load-path* +prior to processing and evaluating all the top-level forms +in the target file. When the forms are evaluated, the binding +is discarded and +.code load +returns. -If an end-of-file condition occurs before any bytes are read, then zero -is returned. +The +.code compile-file +function also establishes a binding for +.codn *load-path* . The -.code fill-buf-adjust -differs usefully from -.code fill-buf -as follows. Whereas -.code fill-buf -doesn't manipulate the length of the buffer at any stage of the operation, the -.code fill-buf-adjust -begins by adjusting the length of the buffer to the underlying allocated size. -Then it performs the fill operation in -exactly the same manner as -.codn fill-buf . -Finally, if the operation succeeds, then -.code fill-buf-adjust -adjusts the length of the buffer to match the position that is returned. +.code @(load) +directive, also similarly establishes a binding around the +parsing and processing of a loaded \*(TX source file. -.SS* Buffer streams -A stream type exists which allows -.code buf -objects to be manipulated through the stream interface. -A buffer stream is created using the -.code make-buf-stream -function, which can either attach the stream to an existing buffer, -or create a new buffer that can later be retrieved from the stream -using -.codn get-buf-from-stream . +Also, during the processing of the profile file (see Interactive Profile File), +the variable is bound to the name of that file. -Operations on the buffer stream treat the underlying buffer much like if it -were a memory-based file. Unless the underlying buffer is a "borrowed buffer" -referencing the storage belonging to another object -(such as the buffer object produced by the -.code buf-d -FFI type's get semantics) the stream operations can change the buffer's size. -Seeking beyond the end of the buffer an then writing one or more bytes -extends the buffer's length, filling the newly allocated area with zero bytes. +.coNP Macro @ load-for +.synb +.mets (load-for >> {( kind < sym << target )}*) +.syne +.desc The -.code truncate-stream -function is supported also. -Buffer streams also support the -.code :byte-oriented -property. +.code load-for +macro takes multiple arguments, each of which is a three-element +clause. Each clause specifies that a given +.meta target +file is to be conditionally loaded based on whether a symbol +.meta sym +has a certain kind of binding. + +Each argument clause has the syntax +.mono +.meti >> ( kind < sym << target ) +.onom +where +.meta kind +is one of the five symbols +.codn var , +.codn fun , +.codn macro , +.code struct +or +.codn pkg . +The +.meta sym +element is a symbol suitable for use as a variable, function +or structure name, and +.meta target +is an expression which is evaluated to produce a value that is suitable +as an argument to the +.code load +function. + +First, all +.code target +expressions in all clauses are unconditionally evaluated in left to right +order. Then the clauses are processed in that order. If the +.meta kind +symbol of a clause is +.codn var , +then +.code load-for +tests whether +.meta sym +has a binding in the variable namespace using the +.code boundp +function. If a binding does not exist, then the value of the +.meta target +expression is passed to the +.code load +function. Otherwise, +.code load +is not called. +Similarly, if +.meta kind +is the symbol +.codn fun , +then +.meta sym +is instead tested using +.codn fboundp , +if +.meta kind +is +.codn macro , +then +.meta sym +is tested using +.codn mboundp , +if +.meta kind +is +.codn struct , +then +.meta sym +is tested using +.codn find-struct-type , +and if +.meta kind +is +.codn pkg , +then +.meta sym +is tested using +.codn find-package . -Macros -.code with-out-buf-stream -and -.code with-in-buf-stream -are provided to simplify the steps involved in using buffer streams -in some common scenarios. Note that in spite of the naming of these -macros there is only one buffer stream type, which supports bidirectional I/O. +When +.code load-for +invokes the +.code load +function, it confirms whether loading file has had the expected effect of +providing a definition of +.meta sym +of the right +.metn kind . +If this isn't the case, an error is thrown. -.coNP Function @ make-buf-stream -.synb -.mets (make-buf-stream <> [ buf ]) -.syne -.desc The -.code make-buf-stream -function return a new buffer stream. If the -.meta buf -argument is supplied, it must be a -.code buf -object. The stream is then associated with this object. -If the argument is omitted, a buffer of length zero is created and associated -with the stream. +.code load-for +function returns +.codn nil . -.coNP Function @ get-buf-from-stream -.synb -.mets (get-buf-from-stream << buf-stream ) -.syne +.coNP Variable @ txr-exe-path +.desc +This variable holds the absolute path name of the executable file +of the running \*(TX instance. + +.SS* Function Tracing + +.coNP Special variable @ *trace-output* .desc The -.code get-buf-from-stream -returns the buffer object associated with -.meta buf-stream -which must be a buffer stream. +.code *trace-output* +special variable holds a stream to which all trace output +is sent. Trace output consists of diagnostics enabled by the +.code trace +macro. -.coNP Macros @ with-out-buf-stream and @ with-in-buf-stream +.coNP Macros @ trace and @ untrace .synb -.mets (with-out-buf-stream >> ( var <> [ buf-expr ]) -.mets \ \ << body-form *) -.mets (with-in-buf-stream >> ( var << buf-expr ) -.mets \ \ << body-form *) +.mets (trace << function-name *) +.mets (untrace << function-name *) .syne .desc The -.code with-out-buf-stream +.code trace and -.code with-in-buf-stream -macros both bind variable -.meta var -to an implicitly created buffer stream, and evaluate zero or more -.metn body-form -s -in the environment where the variable is visible. +.code untrace +macros control function tracing. -The -.meta buf-expr -argument, which may be omitted in the use of the -.code with-out-buf-stream -macro, must be an expression which evaluates to a -.code buf -object. +When +.code trace +is called with one or more arguments, it considers each +argument to be the name of a global function. For each +function, it turns on tracing, if it is not already turned on. +If an argument denotes a nonexistent function, or is invalid +function name syntax, +.code trace +terminates by throwing an exception, without processing the +subsequent arguments, or undoing the effects already applied +due to processing the previous arguments. -The -.meta var -argument must be a symbol suitable for naming a variable. +When +.code trace +is called with no arguments, it lists the names of functions +for which tracing is currently enabled. In other cases it +returns +.codn nil . -The implicitly allocated buffer stream is connected -to the buffer specified by -.meta buf-expr -or, when -.meta buf-expr -is omitted, to a newly allocated buffer. +When +.code untrace +is called with one or more arguments, it considers each +argument to be the name of a global function. For each +function, it turns off tracing, if tracing is enabled. -The code generated by the -.code with-out-buf-stream -macro, if it terminates normally, yields the buffer object -as its result value. +When +.code untrace +is called with no arguments, it disables tracing for all +functions. The -.code with-in-buf-stream -returns the value of the last -.metn body-form , -or else +.code untrace +macro always returns .code nil -if no forms are specified. +and silently tolerates arguments which are not names of functions +currently being traced. -.TP* Examples: -.verb - (with-out-buf-stream (*stdout* (make-buf 24)) - (put-string "Hello, world!")) - -> #b'48656c6c6f2c2077 6f726c6421000000 0000000000000000' +Tracing a function consists of printing a message prior to entry into the +function indicating its name and arguments, and another message upon leaving +the function indicating its return value, which is syntactically correlated +with the entry message, using a combination of matching and indentation. +These messages are posted to the +.code *trace-output* +stream. - (with-out-buf-stream (*stdout*) (put-string "Hello, world!")) - -> #b'48656c6c6f2c2077 6f726c6421' -.brev +When traced functions call each other or recurse, these trace messages +nest. The nesting is detected and translated into indentation levels. -.coSS The @ cptr type +Tracing works by replacing a function definition with a trace hook function, and +retaining the previous definition. The trace hook calls the previous definition +and produces the diagnostics around it. When +.code untrace +is used to disable tracing, the previous definition is restored. -Objects of type -.code cptr -are Lisp values which contain a C pointer. This data type is used by the -.code dlopen -function and is generally useful in conjunction with the Foreign Function -Interface (FFI). An arbitrary pointer emanating from a foreign function -can be captured as a -.code cptr -value, which can be passed back into foreign code. For this purpose, there -exits also a matching FFI type called -.codn cptr . +Methods can be traced; their names are given using +.mono +.meti (meth < struct << slot ) +.onom +syntax: see the +.code func-get-name +function. + +Macros can be traced; their names are given using +.mono +.meti (macro << name ) +.onom +syntax. Note that +.code trace +will not show the destructured internal macro arguments, but only the +two arguments passed to the expander function: the whole form, and the +environment. The -.code cptr -type supports a symbolic type tag, which defaults to +.code trace +and +.code untrace +functions return .codn nil . -The type tag plays a role in FFI. The FFI -.code cptr -type supports a tag attribute. When a -.code cptr -object is converted to a foreign pointer under the control of the FFI -type, and that FFI type has a tag other than -.codn nil , -the object's tag must exactly match that of the FFI type, or the conversion -throws an error. In the reverse direction, when a foreign pointer is -converted to a -.code cptr -object under control of the FFI -.code cptr -type, the object inherits the type tag from the FFI type. -.coNP Function @ cptr-int +.SS* Dynamic Library Access + +.coNP Function @ dlopen .synb -.mets (cptr-int < integer <> [ type-symbol ]) +.mets (dlopen >> [{ lib-name | nil} <> [ flags ]) .syne .desc The -.code cptr-int -function converts -.meta integer -into a pointer in a system-specific way -which is consistent with the system's addressing structure. Then it returns -that pointer contained in a -.code cptr -object. - -The -.meta integer -parameter must be an integer which is in range for a pointer value. -Note: this range is wider than the -.code fixnum -range; a portion of the range of -.code bignum -integers can denote pointers. +.code dlopen +function provides access to the POSIX C library function of the +same name. -The -.meta type-symbol -argument should be a symbol. If omitted, it defaults to +The argument to the optional +.meta lib-name +parameter may be a character string, or .codn nil . -This symbol becomes the -.code cptr -object's type tag. -.coNP Function @ cptr-obj -.synb -.mets (cptr-obj < object <> [ type-symbol ]) -.syne -.desc -The -.code cptr-obj -function converts -.meta object -object directly to a -.codn cptr . +If it is +.codn nil , +then the POSIX function is called with a null pointer for +its name argument, returning the handle for the main program, +if possible. The -.meta object -argument may be of any type. +.meta flags +argument should be expressed as some bitwise combination of the values +of the variables +.codn rtld-lazy , +.codn rtld-now , +or other +.code rtld- +variables which give names to the +.codn dlopen -related +flags. If the +.meta flags +argument is omitted, the default value used is +.codn rtld-lazy . -The raw representation of -.meta object -is simply stored in a new instance of +If the function succeeds, it returns an object of type .code cptr -and returned. +which represents the open library handle ("dlhandle"). -The -.meta type-symbol -argument should be a symbol. If omitted, it defaults to -.codn nil . -This symbol becomes the -.code cptr -object's type tag. +Otherwise it throws an exception, whose message incorporates, if possible, +error text retrieved from the +.code dlerror +POSIX function. -.coNP Function @ int-cptr -.synb -.mets (int-cptr << cptr ) -.syne -.desc The -.code int-cptr -function retrieves the pointer value of the -.meta cptr -object as an integer. - -If an integer -.meta n -is in a range convertible to .code cptr -type, then the expression -.mono -.meti (int-cptr (cptr-int << n )) -.onom -reproduces -.metn n . +handle returned by +.code dlopen +will automatically be subject to +.code dlclose +when reclaimed by the garbage collector. -.coNP Function @ cptr-cast +.coNP Function @ dlclose .synb -.mets (cptr-cast < type-symbol << cptr ) +.mets (dlclose << dlhandle ) .syne .desc The -.code cptr-cast -function produces a new -.code cptr -object which has the same pointer as -.meta cptr -but whose type is given by -.metn type-symbol . - -Casting -.meta cptr -objects with -.code cptr-cast -circumvents the safety mechanism which +.code dlclose +closes the library indicated by +.metn dlhandle , +which must be a .code cptr -type tagging provides. - -.coNP Function @ cptr-zap -.synb -.mets (cptr-zap << cptr ) -.syne -.desc -The -.code cptr-zap -function changes the pointer value of the -.meta cptr -object to the null pointer. +object previously returned by +.codn dlopen . -The -.meta cptr -argument must be of +The handle is closed by passing the stored pointer to the POSIX +.code dlclose +function. The internal pointer contained in the .code cptr -type. - -The return value is -.meta cptr -itself. +object is then reset to null. -Note: it is recommended to use -.code cptr-zap -when the program has taken some action which invalidates the pointer value -stored in a +It is permissible to invoke +.code dlclose +more than once on a .code cptr -object, where a risk exists that the value may be subsequently misused. - -.coNP Function @ cptr-free -.synb -.mets (cptr-free << cptr ) -.syne -.desc -The -.code cptr-free -function passes the -.meta cptr -object's pointer to the C library -.code free -function. After this action, it behaves exactly like -.codn cptr-zap . - -The -.meta cptr -argument must be of +object which was created by +.codn dlopen . +The first invocation resets the .code cptr -type. - -The return value is -.meta cptr -itself. - -Note: this function is unsafe. If the pointer didn't originate from the -.code malloc -family of memory allocation functions, or has already been freed, or -copies of the pointer exist which are still in use, the consequences are -likely catastrophic. +object's pointer to null; the subsequent invocations +do nothing. -.coNP Function @ cptrp -.synb -.mets (cptrp << value ) -.syne -.desc The -.code cptrp -function tests whether -.meta value -is a -.codn cptr . -It returns +.code dlclose +function returns .code t -if this is the -case, +if the POSIX function reports a successful result (zero), otherwise +it returns +.codn nil . +It also returns .code nil -otherwise. +if invoked on a previously closed, and hence nulled-out +.code cptr +handle. -.coNP Function @ cptr-type +.coNP Functions @ dlsym and @ dlvsym .synb -.mets (cptr-type << cptr ) +.mets (dlsym < dlhandle << sym-name ) +.mets (dlvsym < dlhandle < sym-name << ver-name ) .syne .desc The -.code cptr-type -function retrieves the -.meta cptr -object's type tag. +.code dlsym +function provides access to the same-named POSIX function. The +.code dlvsym +function provides access to the same-named GNU C Library function, +if available. -.coNP Variable @ cptr-null -.desc The -.code cptr-null -variable holds a null pointer as a -.code cptr -instance. - -Two +.meta dlhandle +argument must be a .code cptr -objects may be compared for equality using the -.code equal -function, which tests whether their pointers are equal. +handle previously returned by +.code dlopen +and not subsequently closed by +.code dlclose +or altered in any way. The -.code cptr-null -variable compares -.code equal -to values which have been subject to -.code cptr-zap -or -.codn cptr-free . +.meta sym-name +and +.meta ver-name +arguments are character strings. -A null +If these functions succeed, they return a .code cptr -may be produced by the expression -.codn "(cptr-obj nil)" ; -however, this creates a freshly allocated object on each evaluation. +value which holds the address of the symbol which was found +in the library. -The expression -.code "(cptr-int 0)" -also produces a null pointer on all platforms where \*(TX is found. +If they fail, they return a +.code cptr +object containing a null pointer. -.coNP Function @ cptr-size-hint +.coNP Functions @ dlsym-checked and @ dlvsym-checked .synb -.mets (cptr-size-hint < cptr << bytes ) +.mets (dlsym-checked < dlhandle << sym-name ) +.mets (dlvsym-checked < dlhandle < sym-name << ver-name ) .syne .desc The -.code cptr-size-hint -function indicates to the garbage collector that the given -.meta cptr -object is associated with -.meta bytes -of foreign memory that are otherwise invisible to the garbage collector. +.code dlsym-checked +and +.code dlvsym-checked +functions are alternatives to +.code dlsym +and +.codn dlvsym , +respectively. Instead of returning a null +.code cptr +on failure, these functions throw an exception. -Note: this function should be used if the foreign memory is indirectly -managed by the -.meta cptr -object in cooperation with the garbage collector. Specifically, -.meta cptr -should have a finalizer registered against it which will liberate the -foreign memory. +.coNP Variables @, rtld-lazy @, rtld-now @, rtld-global @, rtld-local @, rtld-nodelete @ rtld-noload and @ rtld-deepbind +.desc +These variables provide the same values as constants in the POSIX C library +header +.code "" +named +.codn RTLD_LAZY , +.codn RTLD_NOW , +.codn RTLD_LOCAL , +.IR "et cetera" . .SH* FOREIGN FUNCTION INTERFACE -- cgit v1.2.3