Top Node
********

Fake top node.

Copyright © 2012 Free Software Foundation, Inc.

This is Edition 1 of Writing Extensions for Gawk: A Temporary Manual, for the 4.1.0 (or later) version of the GNU implementation of AWK.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with the Invariant Sections being ``GNU General Public License'', the Front-Cover texts being (a) (see below), and with the Back-Cover Texts being (b) (see below).  A copy of the license is included in the section entitled ``GNU Free Documentation License''.

 a. ``A GNU Manual''

 b. ``You have the freedom to copy and modify this GNU manual.  Buying copies from the FSF supports it in developing GNU and promoting software freedom.''

Writing Extensions for gawk
***************************

It is possible to add new built-in functions to gawk using dynamically loaded libraries. This facility is available on systems (such as GNU/Linux) that support the C dlopen() and dlsym() functions. This chapter describes how to do so using code written in C or C++.  If you don't know anything about C programming, you can safely skip this chapter, although you may wish to review the documentation on the extensions that come with gawk (see Extension Samples).

     When --sandbox is specified, extensions are disabled (see Options.

Introduction
============

An extension (sometimes called a plug-in) is a piece of external compiled code that gawk can load at runtime to provide additional functionality, over and above the built-in capabilities described in the rest of this book.

Extensions are useful because they allow you (of course) to extend gawk's functionality. For example, they can provide access to system calls (such as chdir() to change directory) and to other C library routines that could be of use.  As with most software, ``the sky is the limit;'' if you can imagine something that you might want to do and can write in C or C++, you can write an extension to do it!

Extensions are written in C or C++, using the Application Programming Interface (API) defined for this purpose by the gawk developers.  The rest of this chapter explains the design decisions behind the API, the facilities it provides and how to use them, and presents a small sample extension.  In addition, it documents the sample extensions included in the gawk distribution.

Extension Licensing
===================

Every dynamic extension should define the global symbol plugin_is_GPL_compatible to assert that it has been licensed under a GPL-compatible license.  If this symbol does not exist, gawk will emit a fatal error and exit.

The declared type of the symbol should be int.  It does not need to be in any allocated section, though.  The code merely asserts that the symbol exists in the global scope.  Something like this is enough:

     int plugin_is_GPL_compatible;

Extension API Design
====================

The first version of extensions for gawk was developed in the mid-1990s and released with gawk 3.1 in the late 1990s. The basic mechanisms and design remained unchanged for close to 15 years, until 2012.

The old extension mechanism used data types and functions from gawk itself, with a ``clever hack'' to install extension functions.

gawk included some sample extensions, of which a few were really useful.  However, it was clear from the outset that the extension mechanism was bolted onto the side and was not really thought out.

Problems With The Old Mechanism
-------------------------------

The old extension mechanism had several problems:

  * It depended heavily upon gawk internals.  Any time the NODE structure (A critical central data structure inside gawk.) changed, an extension would have to be recompiled. Furthermore, to really write extensions required understanding something about gawk's internal functions.  There was some documentation in this book, but it was quite minimal.

  * Being able to call into gawk from an extension required linker facilities that are common on Unix-derived systems but that did not work on Windows systems; users wanting extensions on Windows had to statically link them into gawk, even though Windows supports dynamic loading of shared objects.

  * The API would change occasionally as gawk changed; no compatibility between versions was ever offered or planned for.

Despite the drawbacks, the xgawk project developers forked gawk and developed several significant extensions. They also enhanced gawk's facilities relating to file inclusion and shared object access.

A new API was desired for a long time, but only in 2012 did the gawk maintainer and the xgawk developers finally start working on it together.  More information about the xgawk project is provided in gawkextlib.

Goals For A New Mechanism
-------------------------

Some goals for the new API were:

  * The API should be independent of gawk internals.  Changes in gawk internals should not be visible to the writer of an extension function.

  * The API should provide binary compatibility across gawk releases as long as the API itself does not change.

  * The API should enable extensions written in C to have roughly the same ``appearance'' to awk-level code as awk functions do. This means that extensions should have:

       - The ability to access function parameters.

       - The ability to turn an undefined parameter into an array (call by reference).

       - The ability to create, access and update global variables.

       - Easy access to all the elements of an array at once (``array flattening'') in order to loop over all the element in an easy fashion for C code.

  * The ability to create arrays (including gawk's true multi-dimensional arrays).

Some additional important goals were:

  * The API should use only features in ISO C 90, so that extensions can be written using the widest range of C and C++ compilers. The header should include the appropriate #ifdef __cplusplus and extern "C" magic so that a C++ compiler could be used.  (If using the C++, the runtime system has to be smart enough to call any constructors and destructors, as gawk is a C program. As of this writing, this has not been tested.)

  * The API mechanism should not require access to gawk's symbols (The symbols are the variables and functions defined inside gawk.  Access to these symbols by code external to gawk loaded dynamically at runtime is problematic on Windows.) by the compile-time or dynamic linker, in order to enable creation of extensions that will also work on Windows.

During development, it became clear that there were other features that should be available to extensions, which were also subsequently provided:

  * Extensions should have the ability to hook into gawk's I/O redirection mechanism.  In particular, the xgawk developers provided a so-called ``open hook'' to take over reading records.  During the development, this was generalized to allow extensions to hook into input processing, output processing, and two-way I/O.

  * An extension should be able to provide a ``call back'' function to perform clean up actions when gawk exits.

  * An extension should be able to provide a version string so that gawk's --version option can provide information about extensions as well.

Other Design Decisions
----------------------

As an ``arbitrary'' design decision, extensions can read the values of built-in variables and arrays (such as ARGV and FS), but cannot change them, with the exception of PROCINFO.

The reason for this is to prevent an extension function from affecting the flow of an awk program outside its control.  While a real awk function can do what it likes, that is at the discretion of the programmer.  An extension function should provide a service or make a C API available for use within awk, and not mess with FS or ARGC and ARGV.

In addition, it becomes easy to start down a slippery slope. How much access to gawk facilities do extensions need? Do they need getline?  What about calling gsub() or compiling regular expressions?  What about calling into awk functions? (That would be messy.)

In order to avoid these issues, the gawk developers chose to start with the simplest, most basic features that are still truly useful.

Another decision is that although gawk provides nice things like MPFR, and arrays indexed internally by integers, these features are not being brought out to the API in order to keep things simple and close to traditional awk semantics.  (In fact, arrays indexed internally by integers are so transparent that they aren't even documented!)

With time, the API will undoubtedly evolve; the gawk developers expect this to be driven by user needs. For now, the current API seems to provide a minimal yet powerful set of features for extension creation.

At A High Level How It Works
----------------------------

The requirement to avoid access to gawk's symbols is, at first glance, a difficult one to meet.

One design, apparently used by Perl and Ruby and maybe others, would be to make the mainline gawk code into a library, with the gawk program a small C main() function linked against the library.

This seemed like the tail wagging the dog, complicating build and installation and making a simple copy of the gawk executable from one system to another (or one place to another on the same system!) into a chancy operation.

Pat Rankin suggested the solution that was adopted. Communication between gawk and an extension is two-way.  First, when an extension is loaded, it is passed a pointer to a struct whose fields are function pointers.

     FIGURE 1

The extension can call functions inside gawk through these function pointers, at runtime, without needing (link-time) access to gawk's symbols.  One of these function pointers is to a function for ``registering'' new built-in functions.

     FIGURE 2

In the other direction, the extension registers its new functions with gawk by passing function pointers to the functions that provide the new feature (do_chdir(), for example).  gawk associates the function pointer with a name and can then call it, using a defined calling convention.

     FIGURE 3

The do_XXX() function, in turn, then uses the function pointers in the API struct to do its work, such as updating variables or arrays, printing messages, setting ERRNO, and so on.

Convenience macros in the gawkapi.h header file make calling through the function pointers look like regular function calls so that extension code is quite readable and understandable.

Although all of this sounds medium complicated, the result is that extension code is quite clean and straightforward. This can be seen in the sample extensions filefuncs.c and also the testext.c code for testing the APIs.

Some other bits and pieces:

  * The API provides access to gawk's do_XXX values, reflecting command line options, like do_lint, do_profiling and so on (see Extension API Variables). These are informational: an extension cannot affect these inside gawk.  In addition, attempting to assign to them produces a compile-time error.

  * The API also provides major and minor version numbers, so that an extension can check if the gawk it is loaded with supports the facilities it was compiled with.  (Version mismatches ``shouldn't'' happen, but we all know how that goes.) See Extension Versioning, for details.

  * An extension may register a version string with gawk; this allows gawk to dump extension version information when invoked with the --version option.

Room For Future Growth
----------------------

The API provides room for future growth, in two ways.

An ``extension id'' is passed into the extension when its loaded. This extension id is then passed back to gawk with each function call.  This allows gawk to identify the extension calling it, should it need to know.

A ``name space'' is passed into gawk when an extension function is registered.  This provides for a future mechanism for grouping extension functions and possibly avoiding name conflicts.

Of course, as of this writing, no decisions have been made with respect to any of the above.

API Description
===============

This (rather large) section describes the API in detail.

Introduction
------------

Access to facilities within gawk are made available by calling through function pointers passed into your extension.

API function pointers are provided for the following kinds of operations:

  * Registrations functions. You may register:

       - extension functions,

       - input parsers,

       - output wrappers,

       - two-way processors,

       - exit callbacks,

       - and a version string.

     All of these are discussed in detail, later in this chapter.

  * Printing fatal, warning, and lint warning messages.

  * Updating ERRNO, or unsetting it.

  * Accessing parameters, including converting an undefined parameter into an array.

  * Symbol table access: retrieving a global variable, creating one, or changing one.  This also includes the ability to create a scalar variable that will be constant within awk code.

  * Creating and releasing cached values; this provides an efficient way to use values for multiple variables and can be a big performance win.

  * Manipulating arrays:

       - Retrieving, adding, deleting, and modifying elements

       - Getting the count of elements in an array

       - Creating a new array

       - Clearing an array

       - Flattening an array for easy C style looping over an array

Some points about using the API:

  * You must include <sys/types.h> and <sys/stat.h> before including the gawkapi.h header file. In addition, you must include either <stddef.h> or <stdlib.h> to get the definition of size_t. If you wish to use the boilerplate dl_load_func() macro, you will need to include <stdio.h> as well. Finally, to pass reasonable integer values for ERRNO, you will need to include <errno.h>.

  * Although the API only uses ISO C 90 features, there is an exception; the ``constructor'' functions use the inline keyword. If your compiler does not support this keyword, you should either place -Dinline='' on your command line, or use the GNU Autotools and include a config.h file in your extensions.

  * All pointers filled in by gawk are to memory managed by gawk and should be treated by the extension as read-only.  Memory for all strings passed into gawk from the extension must come from malloc() and is managed by gawk from then on.

  * The API defines several simple structs that map values as seen from awk.  A value can be a double, a string, or an array (as in multidimensional arrays, or when creating a new array). Strings maintain both pointer and length since embedded NUL characters are allowed.

     By intent, strings are maintained using the current multibyte encoding (as defined by LC_XXX environment variables) and not using wide characters.  This matches how gawk stores strings internally and also how characters are likely to be input and output from files.

  * When retrieving a value (such as a parameter or that of a global variable or array element), the extension requests a specific type (number, string, scalars, value cookie, array, or ``undefined'').  When the request is ``undefined,'' the returned value will have the real underlying type.

     However, if the request and actual type don't match, the access function returns ``false'' and fills in the type of the actual value that is there, so that the extension can, e.g., print an error message (``scalar passed where array expected'').

While you may call the API functions by using the function pointers directly, the interface is not so pretty. To make extension code look more like regular code, the gawkapi.h header file defines a number of macros which you should use in your code.  This section presents the macros as if they were functions.

General Purpose Data Types
--------------------------

     I have a true love/hate relationship with unions.

     Arnold Robbins

     That's the thing about unions: the compiler will arrange things so they
     can accommodate both love and hate.

     Chet Ramey

The extension API defines a number of simple types and structures for general purpose use. Additional, more specialized, data structures, are introduced in subsequent sections, together with the functions that use them.

typedef void *awk_ext_id_t;
     A value of this type is received from gawk when an extension is loaded. That value must then be passed back to gawk as the first parameter of each API function.

#define awk_const ...
     This macro expands to const when compiling an extension, and to nothing when compiling gawk itself.  This makes certain fields in the API data structures unwritable from extension code, while allowing gawk to use them as it needs to.

typedef int awk_bool_t;
     A simple boolean type. As of this moment, the API does not define special ``true'' and ``false'' values, although perhaps it should.

typedef struct {
    char *str;      /* data */
    size_t len;     /* length thereof, in chars */
} awk_string_t;
     This represents a mutable string. gawk owns the memory pointed to if it supplied the value. Otherwise, it takes ownership of the memory pointed to. Such memory must come from malloc()!

     As mentioned earlier, strings are maintained using the current multibyte encoding.

typedef enum {
    AWK_UNDEFINED,
    AWK_NUMBER,
    AWK_STRING,
    AWK_ARRAY,
    AWK_SCALAR,         /* opaque access to a variable */
    AWK_VALUE_COOKIE   /* for updating a previously created value */
} awk_valtype_t;
     This enum indicates the type of a value. It is used in the following struct.

typedef struct {
    awk_valtype_t   val_type;
    union {
        awk_string_t       s;
        double             d;
        awk_array_t        a;
        awk_scalar_t       scl;
        awk_value_cookie_t vc;
    } u;
} awk_value_t;
     An ``awk value.'' The val_type member indicates what kind of value the union holds, and each member is of the appropriate type.

#define str_value      u.s
#define num_value      u.d
#define array_cookie   u.a
#define scalar_cookie  u.scl
#define value_cookie   u.vc
     These macros make accessing the fields of the awk_value_t more readable.

typedef void *awk_scalar_t;
     Scalars can be represented as an opaque type. These values are obtained from gawk and then passed back into it. This is discussed below.

typedef void *awk_value_cookie_t;
     A ``value cookie'' is an opaque type representing a cached value. This is also discussed below.

Scalar values in awk are either numbers or strings. The awk_value_t struct represents values.  The val_type member indicates what is in the union.

Representing numbers is easy---the API uses a C double.  Strings require more work. Since gawk allows embedded NUL bytes in string values, a string must be represented as a pair containing a data-pointer and length. This is the awk_string_t type.

Identifiers (i.e., the names of global variables) can be associated with either scalar values or with arrays.  In addition, gawk provides true arrays of arrays, where any given array element can itself be an array.  Discussion of arrays is delayed until Array Manipulation

The various macros listed earlier make it easier to use the elements of the union as if they were fields in a struct; this is a common coding practice in C.  Such code is easier to write and to read, however it remains your responsibility to make sure that the val_type member correctly reflects the type of the value in the awk_value_t.

Conceptually, the first three members of the union (number, string, and array) are all that is needed for working with awk values. However, since the API provides routines for accessing and changing the value of global scalar variables only by using the variable's name, there is a performance penalty: gawk must find the variable each time it is accessed and changed.  This turns out to be a real issue, not just a theoretical one.

Thus, if you know that your extension will spend considerable time reading and/or changing the value of one or more scalar variables, you can obtain a scalar cookie (See the ``cookie'' entry in the Jargon file (http://catb.org/jargon/html/C/cookie.html) for a definition of cookie, and the ``magic cookie'' entry in the Jargon file (http://catb.org/jargon/html/M/magic-cookie.html) for a nice example. See also the entry in the Glossary.) object for that variable, and then use the cookie for getting the variable's value for changing the variable's value. This is the awk_scalar_t type and scalar_cookie macro. Given a scalar cookie, gawk can directly retrieve or modify the value, as required, without having to first find it.

The awk_value_cookie_t type and value_cookie macro are similar. If you know that you wish to use the same numeric or string value for one or more variables, you can create the value once, retaining a value cookie for it, and then pass in that value cookie whenever you wish to set the value of a variable.  This saves both storage space within the running gawk process as well as the time needed to create the value.

Requesting Values
-----------------

All of the functions that return values from gawk work in the same way. You pass in an awk_valtype_t value to indicate what kind of value you want.  If the actual value matches what you requested, the function returns true and fills in the awk_value_t result. Otherwise, the function returns false, and the val_type member indicates the type of the actual value.  You may then print an error message, or reissue the request for the actual value type, as appropriate.  This behavior is summarized in table-value-types-returned.

Table: Value Types Returned

                             +-------------------------------------------------+
                             |                Type of Actual Value:            |
                             +------------+------------+-----------+-----------+
                             |   String   |   Number   | Array     | Undefined |
     +-----------+-----------+------------+------------+-----------+-----------+
     |           | String    |   String   |   String   | false     | false     |
     |           |-----------+------------+------------+-----------+-----------+
     |           | Number    | Number if  |   Number   | false     | false     |
     |           |           | can be     |            |           |           |
     |           |           | converted, |            |           |           |
     |           |           | else false |            |           |           |
     |           |-----------+------------+------------+-----------+-----------+
     |   Type    | Array     |   false    |   false    | Array     | false     |
     | Requested |-----------+------------+------------+-----------+-----------+
     |           | Scalar    |   Scalar   |   Scalar   | false     | false     |
     |           |-----------+------------+------------+-----------+-----------+
     |           | Undefined |  String    |   Number   | Array     | Undefined |
     |           |-----------+------------+------------+-----------+-----------+
     |           | Value     |   false    |   false    | false     | false     |
     |           | Cookie    |            |            |           |           |
     +-----------+-----------+------------+------------+-----------+-----------+

Constructor Functions and Convenience Macros
--------------------------------------------

The API provides a number of constructor functions for creating string and numeric values, as well as a number of convenience macros. This subsection presents them all as function prototypes, in the way that extension code would use them.

static inline awk_value_t *
make_const_string(const char *string, size_t length, awk_value_t *result)
     This function creates a string value in the awk_value_t variable pointed to by result. It expects string to be a C string constant (or other string data), and automatically creates a copy of the data for storage in result.

static inline awk_value_t *
make_malloced_string(const char *string, size_t length, awk_value_t *result)
     This function creates a string value in the awk_value_t variable pointed to by result. It expects string to be a char * value pointing to data previously obtained from malloc(). The idea here is that the data will be passed directly to gawk, which will assume responsibility for it.

static inline awk_value_t *
make_null_string(awk_value_t *result)
     This specialized function creates a null string (the ``undefined'' value) in the awk_value_t variable pointed to by result.

static inline awk_value_t *
make_number(double num, awk_value_t *result)
     This function simply creates a numeric value in the awk_value_t variable pointed to by result.

Two convenience macros may be used for allocating storage from malloc() and realloc(). If the allocation fails, they cause gawk to exit with a fatal error message.  They should be used as if they were procedure calls that do not return a value.

emalloc(pointer, type, size, message)
     The arguments to this macro are as follows:

     pointer
          The pointer variable to point at the allocated storage.

     type
          The type of the pointer variable, used to create a cast for the call to malloc().

     size
          The total number of bytes to be allocated.

     message
          A message to be prefixed to the fatal error message. Typically this is the name of the function using the macro.

     For example, you might allocate a string value like so:

          awk_value_t result;
          char *message;
          const char greet[] = "Don't Panic!";

          emalloc(message, char *, sizeof(greet), "myfunc");
          strcpy(message, greet);
          make_malloced_string(message, strlen(message), & result);

erealloc(pointer, type, size, message)
     The arguments are the same as for the emalloc() macro.

Registration Functions
----------------------

This section describes the API functions which let you register parts of your extension with gawk.

Registering An Extension Function
.................................

Extension functions are described by the following record:

     typedef struct {
         const char *name;
         awk_value_t *(*function)(int num_actual_args, awk_value_t *result);
         size_t num_expected_args;
     } awk_ext_func_t;

The fields are:

const char *name;
     The name of the new function. awk level code will call the function by this name.

awk_value_t *(*function)(int num_actual_args, awk_value_t *result);
     This is a pointer to the C function that provides the desired functionality. The function must fill in the result with either a number or a string. awk takes ownership of any string memory. As mentioned earlier, string memory must come from malloc().

     The function must return the value of result. This is for the convenience of the calling code inside gawk.

size_t num_expected_args;
     This is the number of arguments the function expects to receive. Each extension function may decide what to do if the number of arguments isn't what it expected.  Following awk functions, it is likely OK to ignore extra arguments.

Once you have a record representing your extension function, you register it with gawk using this API function:

awk_bool_t add_ext_func(const char *namespace, const awk_ext_func_t *func);
     This function returns true upon success, false otherwise. The namespace parameter is currently not used; you should pass in an empty string ("").  The func pointer is the address of a struct describing your function, as just described.

Customized Input Parsers
........................

By default, gawk reads text files as its input. It uses the value of RS to find the end of the record, and then uses FS (or FIELDWIDTHS) to split it into fields.  Additionally, it sets the value of RT (see Built-in Variables).

If you want, you can provide your own, custom, input parser.  An input parser's job is to return a record to the gawk record processing code, along with indicators for the value and length of the data to be used for RT, if any.

To provide an input parser, you must first provide two functions (where XXX is a prefix name for your extension):

awk_bool_t XXX_can_take_file(const awk_input_buf_t *iobuf)
     This function examines the information available in iobuf (which we discuss shortly).  Based on the information there, it decides if the input parser should be used for this file. If so, it should return true (non-zero). Otherwise, it should return false (zero).

awk_bool_t XXX_take_control_of(awk_input_buf_t *iobuf)
     When gawk decides to hand control of the file over to the input parser, it calls this function.  This function in turn must fill in certain fields in the awk_input_buf_t structure, and ensure that certain conditions are true.  It should then return true. If an error of some kind occurs, it should not fill in any fields, and should return false; then gawk will not use the input parser. The details are presented shortly.

Your extension should package these functions inside an awk_input_parser_t, which looks like this:

     typedef struct input_parser {
         const char *name;   /* name of parser */
         awk_bool_t (*can_take_file)(const awk_input_buf_t *iobuf);
         awk_bool_t (*take_control_of)(awk_input_buf_t *iobuf);
         awk_const struct input_parser *awk_const next;    /* for use by gawk */
     } awk_input_parser_t;

The steps are as follows:

 1. Create a static awk_input_parser_t variable and initialize it appropriately.

 2. When your extension is loaded, register your input parser with gawk using the register_input_parser() API function (described below).

An awk_input_buf_t looks like this:

     typedef struct awk_input {
         const char *name;       /* filename */
         int fd;                 /* file descriptor */
     #define INVALID_HANDLE (-1)
         void *opaque;           /* private data for input parsers */
         int (*get_record)(char **out, struct awk_input *, int *errcode,
                 char **rt_start, size_t *rt_len);
         void (*close_func)(struct awk_input *);
         struct stat sbuf;       /* stat buf */
     } awk_input_buf_t;

The fields can be divided into two categories: those for use (initially, at least) by XXX_can_take_file(), and those for use by XXX_take_control_of().  The first group of fields and their uses are as follows:

const char *name;
     The name of the file.

int fd;
     A file descriptor for the file.  If gawk was able to open the file, then it will not be equal to INVALID_HANDLE. Otherwise, it will.

struct stat sbuf;
     If file descriptor is valid, then gawk will have filled in this structure with a call to the fstat() system call.

The XXX_can_take_file() function should examine these fields and decide if the input parser should be used for the file. The decision can be made based upon gawk state (the value of a variable defined previously by the extension and set by awk code), the name of the file, whether or not the file descriptor is valid, the information in the struct stat, or any combination of the above.

Once XXX_can_take_file() has returned true, and gawk has decided to use your input parser, it will call XXX_take_control_of().  That function then fills in at least the get_record field of the awk_input_buf_t.  It must also ensure that fd is not set to INVALID_HANDLE.  All of the fields that may be filled by XXX_take_control_of() are as follows:

void *opaque;
     This is used to hold any state information needed by the input parser for this file.  It is ``opaque'' to gawk.  The input parser is not required to use this pointer.

int (*get_record)(char **out, struct awk_input *, int *errcode,
char **rt_start, size_t *rt_len);
     This is a function pointer that should be set to point to the function that creates the input records. Said function is the core of the input parser.  Its behavior is described below.

void (*close_func)(struct awk_input *);
     This is a function pointer that should be set to point to the function that does the ``tear down.'' It should release any resources allocated by XXX_take_control_of().  It may also close the file. If it does so, it shold set the fd field to INVALID_HANDLE.

     Having a ``tear down'' function is optional. If your input parser does not need it, do not set this field.  In that case, gawk will close the regular close() system call on the file descriptor, so it should be valid.

The XXX_get_record() function does the work of creating input records.  The parameters are as follows:

char **out
     This is a pointer to a char * variable which is set to point to the record.  gawk will make its own copy of the data, so the extension must manage this storage.

struct awk_input *iobuf
     This is the awk_input_buf_t for the file.  The fields should be used for reading data (fd) and for managing private state (opaque), if any.

int *errcode
     If an error occurs, *errcode should be set to an appropriate code from <errno.h>.

char **rt_start
size_t *rt_len
     If the concept of a ``record terminator'' makes sense, then *rt_start should be set to point to the data to be used for RT, and *rt_len should be set to the length of the data. Otherwise, *rt_len should be set to zero. gawk makes its own copy of this data, so the extension must manage the storage.

The return value is the length of the buffer pointed to by *out, or EOF if end-of-file was reached or an error occurred.

It is guaranteed that errcode is a valid pointer, so there is no need to test for a NULL value.  gawk sets *errcode to zero, so there is no need to set it unless an error occurs.

If an error does occur, the function should return EOF and set *errcode to a non-zero value.  In that case, if *errcode does not equal -1, gawk| will automatically update the ERRNO variable based on the value of *errcode (e.g., setting *errcode = errno should do the right thing).

gawk ships with a sample extension (see Extension Sample Readdir) that reads directories, returning records for each entry in the directory.  You may wish to use that code as a guide for writing your own input parser.

When writing an input parser, you should think about (and document) how it is expected to interact with awk code.  You may want it to always be called, and take effect as appropriate (as the readdir extension does).  Or you may want it to take effect based upon the value of an awk variable, as the XML extension from the gawkextlib project does (see gawkextlib). In the latter case, code in a BEGINFILE section (see BEGINFILE/ENDFILE). can look at FILENAME and ERRNO to decide whether or not to activate an input parser.

You register your input parser with the following function:

void register_input_parser(awk_input_parser_t *input_parser);
     Register the input parser pointed to by input_parser with gawk.

Customized Output Wrappers
..........................

An output wrapper is the mirror image of an input parser. It allows an extension to take over the output to a file (opened with the > or >> operators, see Redirection).

The output wrapper is very similar to the input parser structure:

     typedef struct output_wrapper {
         const char *name;   /* name of the wrapper */
         awk_bool_t (*can_take_file)(const awk_output_buf_t *outbuf);
         awk_bool_t (*take_control_of)(awk_output_buf_t *outbuf);
         awk_const struct output_wrapper *awk_const next;  /* for use by gawk */
     } awk_output_wrapper_t;

The members are as follows:

const char *name;
     This is the name of the output wrapper.

awk_bool_t (*can_take_file)(const awk_output_buf_t *outbuf);
     This points to a function that examines the information in the awk_output_buf_t structure pointed to by outbuf. It should return true if the output wrapper wants to take over the file, and false otherwise.  It should not change any state (variable values, etc.) within gawk.

awk_bool_t (*take_control_of)(awk_output_buf_t *outbuf);
     The function pointed to by this field is called when gawk decides to let the output wrapper take control of the file. It should fill in appropriate members of the awk_output_buf_t structure, as described below, and return true if successful, false otherwise.

awk_const struct output_wrapper *awk_const next;
     This is for use by gawk.

The awk_output_buf_t structure looks like this:

     typedef struct {
         const char *name;   /* name of output file */
         const char *mode;   /* mode argument to fopen */
         FILE *fp;           /* stdio file pointer */
         awk_bool_t redirected;  /* true if a wrapper is active */
         void *opaque;       /* for use by output wrapper */
         size_t (*gawk_fwrite)(const void *buf, size_t size, size_t count,
                     FILE *fp, void *opaque);
         int (*gawk_fflush)(FILE *fp, void *opaque);
         int (*gawk_ferror)(FILE *fp, void *opaque);
         int (*gawk_fclose)(FILE *fp, void *opaque);
     } awk_output_buf_t;

Here too, your extension will define XXX_can_take_file() and XXX_take_control_of() functions that examine and update data members in the awk_output_buf_t. The data members are as follows:

const char *name;
     The name of the output file.

const char *mode;
     The mode string (as would be used in the second argument to fopen() with which the file was opened.

FILE *fp;
     The FILE pointer from <stdio.h>. gawk opens the file before attempting to find an output wrapper.

awk_bool_t redirected;
     The field should be set to true in the XXX_take_control_of() function.

void *opaque;
     This pointer is opaque to gawk. The extension should use it to store a pointer to any private data associated with the file.

size_t (*gawk_fwrite)(const void *buf, size_t size, size_t count,
                      FILE *fp, void *opaque);
int (*gawk_fflush)(FILE *fp, void *opaque);
int (*gawk_ferror)(FILE *fp, void *opaque);
int (*gawk_fclose)(FILE *fp, void *opaque);
     These pointers should be set to point to functions that perform the equivalent function as the <stdio.h> functions do, if appropriate. gawk uses these function pointers for all output. gawk initializes the pointers to point to internal, ``pass through'' functions that just call the regular <stdio.h> functions, so an extension only needs to redefine those functions that are appropriate for what it does.

The XXX_can_take_file() function should make a decision based upon the name and mode fields, and any additional state (such as awk variable values) that is appropriate.

When gawk calls XXX_take_control_of(), it should fill in the other fields, as appropriate, except for fp, which it should just use normally.

You register your output wrapper with the following function:

void register_output_wrapper(awk_output_wrapper_t *output_wrapper);
     Register the output wrapper pointed to by output_wrapper with gawk.

Customized Two-way Processors
.............................

A two-way processor combines an input parser and an output wrapper for two-way I/O with the |& operator (see Redirection).  It makes identical use of the awk_input_parser_t and awk_output_buf_t structures, as described earlier.

A two-way processor is represented by the following structure:

     typedef struct two_way_processor {
         const char *name;   /* name of the two-way processor */
         awk_bool_t (*can_take_two_way)(const char *name);
         awk_bool_t (*take_control_of)(const char *name, awk_input_buf_t *inbuf, awk_output_buf_t *outbuf);
         awk_const struct two_way_processor *awk_const next;  /* for use by gawk */
     } awk_two_way_processor_t;

The fields are as follows:

const char *name;
     The name of the two-way processor.

awk_bool_t (*can_take_two_way)(const char *name);
     This function returns true if it wants to take over the two-way I/O for this filename.

awk_bool_t (*take_control_of)(const char *name, awk_input_buf_t *inbuf, awk_output_buf_t *outbuf);
     This function should fill in the awk_input_buf_t and awk_outut_buf_t structures pointed to by inbuf and outbuf, respectively.  These structures were described earlier.

awk_const struct two_way_processor *awk_const next;
     This is for use by gawk.

As with the input parser and output processor, you provide ``yes I can take this'' and ``take over for this'' functions, XXX_can_take_two_way() and XXX_take_control_of().

You register your two-way processor with the following function:

void register_two_way_processor(awk_two_way_processor_t *two_way_processor);
     Register the two-way processor pointed to by two_way_processor with gawk.

Registering An Exit Callback Function
.....................................

An exit callback function is a function that gawk calls before it exits. Such functions are useful if you have general ``clean up'' tasks that should be performed in your extension (such as closing data base connections or other resource deallocations). You can register such a function with gawk using the following function.

void awk_atexit(void (*funcp)(void *data, int exit_status),
                void *arg0);
     The parameters are:

     funcp
          Points to the function to be called before gawk exits. The data parameter will be the original value of arg0. The exit_status parameter is the exit status value that gawk will pass to the exit() system call.

     arg0
          A pointer to private data which gawk saves in order to pass to the function pointed to by funcp.

Exit callback functions are called in Last-In-First-Out (LIFO) order---that is, in the reverse order in which they are registered with gawk.

Registering An Extension Version String
.......................................

You can register a version string which indicates the name and version of your extension, with gawk, as follows:

void register_ext_version(const char *version);
     Register the string pointed to by version with gawk.

gawk prints all registered extension version strings when it is invoked with the --version option.

Printing Messages
-----------------

You can print different kinds of warning messages from your extension, as described below.  Note that for these functions, you must pass in the extension id received from gawk when the extension was loaded. (Because the API uses only ISO C 90 features, it cannot make use of the ISO C 99 variadic macro feature to hide that parameter. More's the pity.)

void fatal(awk_ext_id_t id, const char *format, ...);
     Print a message and then cause gawk to exit immediately.

void warning(awk_ext_id_t id, const char *format, ...);
     Print a warning message.

void lintwarn(awk_ext_id_t id, const char *format, ...);
     Print a ``lint warning.''  Normally this is the same as printing a warning message, but if gawk was invoked with --lint=fatal, then they become fatal error messages.

All of these functions are otherwise like the C printf() family of functions, where the format parameter is a string with literal characters and formatting codes intermixed.

Updating ERRNO
--------------

The following functions allow you to update the ERRNO variable.

void update_ERRNO_int(int errno_val);
     Set ERRNO to the string equivalent of the error code in errno_val. The value should be one of the defined error codes in <errno.h>, and gawk will turn it into a (possibly translated) string using the C strerror() function.

void update_ERRNO_string(const char *string);
     Set ERRNO directly to the string value of ERRNO. gawk will make a copy of the value of string.

void unset_ERRNO();
     Unset ERRNO.

Accessing and Updating Parameters
---------------------------------

Two functions give you access to the arguments (parameters) passed to your extension function. They are:

awk_bool_t get_argument(size_t count,
                        awk_valtype_t wanted,
                        awk_value_t *result);
     Fill in the awk_value_t structure pointed to by result with the count'th argument.  Counts are zero based---the first argument is numbered zero, the second one, and so on. wanted indicates the type of value expected.  Return true if the actual type matches wanted, false otherwise In the latter case, result->val_type indicates the actual type.

awk_bool_t set_argument(size_t count, awk_array_t array);
     Convert a parameter that was undefined into an array; this provides call-by-reference for arrays.  Return false if count is too big, or if the argument's type is not undefined.

Symbol Table Access
-------------------

Three sets of routines provide access to global variables.

Variable Access and Update by Name
..................................

The following routines provide the ability to access and update global awk-level variables by name.  In compiler terminology, identifiers of different kinds are termed symbols, thus the ``sym'' in the routines' names.  The data structure which stores information about symbols is termed a symbol table.

awk_bool_t sym_lookup(const char *name,
                      awk_valtype_t wanted,
                      awk_value_t *result);
     Fill in the awk_value_t structure pointed to by result with the value of the variable named by the string name, which is a regular C string.  wanted indicates the type of value expected. Return true if the actual type matches wanted, false otherwise In the latter case, result->val_type indicates the actual type.

awk_bool_t sym_update(const char *name, awk_value_t *value);
     Update the variable named by the string name, which is a regular C string.  The variable will be added to gawk's symbol table if it is not there.  Return true if everything worked, false otherwise.

     Changing types (scalar to array or vice versa) of an existing variable is not allowed, nor may this routine be used to update an array. This routine can also not be be used to update any of the predefined variables (such as ARGC or NF).

awk_bool_t sym_constant(const char *name, awk_value_t *value);
     Create a variable named by the string name, which is a regular C string, that has the constant value as given by value. awk-level code cannot change the value of this variable. (There (currently) is no awk-level feature that provides this ability.) The extension may change the value name's variable with subsequent calls to this routine, and may also convert a variable created by sym_update() into a constant.  However, once a variable becomes a constant it cannot later be reverted into a mutable variable.

Variable Access and Update by Cookie
....................................

A scalar cookie is an opaque handle that provide access to a global variable or array. It is an optimization that avoids looking up variables in gawk's symbol table every time access is needed. This was discussed earlier, in General Data Types.

The following functions let you work with scalar cookies.

awk_bool_t sym_lookup_scalar(awk_scalar_t cookie,
                             awk_valtype_t wanted,
                             awk_value_t *result);
     Retrieve the current value of a scalar cookie. Once you have obtained a scalar_cookie using sym_lookup(), you can use this function to get its value more efficiently. Return false if the value cannot be retrieved.

awk_bool_t sym_update_scalar(awk_scalar_t cookie, awk_value_t *value);
     Update the value associated with a scalar cookie. Return will be false if the new value is not one of AWK_STRING or AWK_NUMBER. Here too, the built-in variables may not be updated.

It is not obvious at first glance how to work with scalar cookies or what their raison d'etre really is.  In theory, the sym_lookup() and sym_update() routines are all you really need to work with variables.  For example, you might have code that looked up the value of a variable, evaluated a condition, and then possibly changed the value of the variable based on the result of that evaluation, like so:

     /*  do_magic --- do something really great */

     static awk_value_t *
     do_magic(int nargs, awk_value_t *result)
     {
         awk_value_t value;

         if (   sym_lookup("MAGIC_VAR", AWK_NUMBER, & value)
             && some_condition(value.num_value)) {
                 value.num_value += 42;
                 sym_update("MAGIC_VAR", & value);
         }

         return make_number(0.0, result);
     }

This code looks (and is) simple and straightforward. So what's the problem?

Consider what happens if awk-level code associated with your extension calls the magic() function (implemented in C by do_magic()), once per record, while processing hundreds of thousands or millions of records. The MAGIC_VAR variable is looked up in the symbol table once or twice per function call!

The symbol table lookup is really pure overhead; it is considerably more efficient to get a cookie that represents the variable, and use that to get the variable's value and update it as needed. (The difference is measurable and quite real. Trust us.)

Thus, the way to use cookies is as follows.  First, install your extension's variable in gawk's symbol table using sym_update(), as usual. Then get a scalar cookie for the variable using sym_lookup():

     static awk_scalar_t magic_var_cookie;    /* static global cookie for MAGIC_VAR */

     static void
     my_extension_init()
     {
         awk_value_t value;

         sym_update("MAGIC_VAR", make_number(42.0, & value));   /* install initial value */
         sym_lookup("MAGIC_VAR", AWK_SCALAR, & value);          /* get cookie */
         magic_var_cookie = value.scalar_cookie;                /* save the cookie */
         ...
     }

Next, use the routines in this section for retrieving and updating the value by way of the cookie.  Thus, do_magic() now becomes something like this:

     /*  do_magic --- do something really great */

     static awk_value_t *
     do_magic(int nargs, awk_value_t *result)
     {
         awk_value_t value;

         if (   sym_lookup_scalar(magic_var_cookie, AWK_NUMBER, & value)
             && some_condition(value.num_value)) {
                 value.num_value += 42;
                 sym_update_scalar(magic_var_cookie, & value);
         }
         ...

         return make_number(0.0, result);
     }

     The previous code omitted error checking for presentation purposes.  Your extension code should be more robust and check the return values from the API functions carefully.

Creating and Using Cached Values
................................

The routines in this section allow you to create and release cached values.  As with scalar cookies, in theory, cached values are not necessary. You can create numbers and strings using the functions in Constructor Functions. You can then assign those values to variables using sym_update() or sym_update_scalar(), as you like.

However, you can understand the point of cached values if you remember that every string value's storage must come from malloc(). If you have 20 variables, all of which have the same string value, you must create 20 identical copies of the string. (Numeric values are clearly less problematic, requiring only a C double to store.)

It is clearly more efficient, if possible, to create a value once, and then tell gawk to reuse the value for multiple variables. That is what the routines in this section let you do.  The functions are as follows:

awk_bool_t create_value(awk_value_t *value, awk_value_cookie_t *result);
     Create a cached string or numeric value from value for efficient later assignment. Only AWK_NUMBER and AWK_STRING values are allowed.  Any other type is rejected.  While AWK_UNDEFINED could be allowed, doing so would result in inferior performance.

awk_bool_t release_value(awk_value_cookie_t vc);
     Release the memory associated with a value cookie obtained from create_value().

You use value cookies in a fashion similar to the way you use scalar cookies. In the extension initialization routine, you create the value cookie:

     static awk_value_cookie_t answer_cookie;  /* static value cookie */

     static void
     my_extension_init()
     {
         awk_value_t value;
         char *long_string;
         size_t long_string_len;

         ...     /* code from earlier */
         /* ... fill in long_string and long_string_len ... */
         make_malloced_string(long_string, long_string_len, & value);
         create_value(& value, & answer_cookie);    /* create cookie */
         ...
     }

Once the value is created, you can use it as the value of any number of variables:

     static awk_value_t *
     do_magic(int nargs, awk_value_t *result)
     {
         awk_value_t new_value;

         ...    /* as earlier */

         value.val_type = AWK_VALUE_COOKIE;
         value.value_cookie = answer_cookie;
         sym_update("VAR1", & value);
         sym_update("VAR2", & value);
         ...
         sym_update("VAR100", & value);
         ...
     }

Using value cookies in this way saves considerable storage, since all of VAR1 through VAR100 share the same value.

You might be wondering, ``Is this sharing problematic? What happens if awk code assigns a new value to VAR1, will all the others be changed too?''

That's a great question. The answer is that no, it's not a problem. gawk is smart enough to avoid such problems.

Finally, as part of your clean up action (see Exit Callback Functions) you should release any cached values that you created using release_value().

Array Manipulation
------------------

The primary data structure (Okay, the only data structure.) in awk is the associative array (see Arrays). Extensions need to be able to manipulate awk arrays. The API provides a number of data structures for working with arrays, functions for working with individual elements, and functions for working with arrays as a whole. This includes the ability to ``flatten'' an array so that it is easy for C code to traverse every element in an array.  The array data structures integrate nicely with the data structures for values to make it easy to both work with and create true arrays of arrays (see General Data Types).

Array Data Types
................

The data types associated with arrays are listed below.

typedef void *awk_array_t;
     If you request the value of an array variable, you get back an awk_array_t value. This value is opaque (It is also a ``cookie,'' but the gawk developers did not wish to overuse this term.) to the extension; it uniquely identifies the array but can only be used by passing it into API functions or receiving it from API functions. This is very similar to way FILE * values are used with the <stdio.h> library routines.

typedef struct awk_element {
    /* convenience linked list pointer, not used by gawk */
    struct awk_element *next;
    enum {
        AWK_ELEMENT_DEFAULT = 0,  /* set by gawk */
        AWK_ELEMENT_DELETE = 1    /* set by extension if
                                    should be deleted */
    } flags;
    awk_value_t    index;
    awk_value_t    value;
} awk_element_t;
     The awk_element_t is a ``flattened'' array element. awk produces an array of these inside the awk_flat_array_t (see the next item). ALL memory pointed to belongs to gawk. Individual elements may be marked for deletion. New elements must be added individually, one at a time, using the separate API for that purpose. The next pointer is for the convenience of extension writers. It allows an extension to create a linked list of new elements which can then be added to array in a loop that traverses the list.

typedef struct awk_flat_array {
    awk_const void *awk_const opaque1;    /* private data for use by gawk */
    awk_const void *awk_const opaque2;    /* private data for use by gawk */
    awk_const size_t count;     /* how many elements */
    awk_element_t elements[1];  /* will be extended */
} awk_flat_array_t;
     This is a flattened array. When an extension gets one of these from gawk, the elements array will be of actual size count. The opaque1 and opaque2 pointers are for use by gawk; therefore they are marked awk_const so that the extension cannot modify them.

Array Functions
...............

The following functions relate to individual array elements.

awk_bool_t get_element_count(awk_array_t a_cookie, size_t *count);
     For the array represented by a_cookie, return in *count the number of elements it contains. A subarray counts as a single element. Return false if there is an error.

awk_bool_t get_array_element(awk_array_t a_cookie,
                             const awk_value_t *const index,
                             awk_valtype_t wanted,
                             awk_value_t *result);
     For the array represented by a_cookie, return in *result the value of the element whose index is index. The value for index can be numeric, in which case gawk will convert it to a string. Using non-integral values is possible, but requires that you understand how such values are converted to strings (see Conversion); thus using integral values is safest. wanted specifies the type of value you wish to retrieve. Return false if wanted does not match the actual type or if index is not in the array.

     As with all strings passed into gawk from an extension, the string value of index must come from malloc(), and gawk will release the storage.

awk_bool_t set_array_element(awk_array_t a_cookie,
                             const awk_value_t *const index,
                             const awk_value_t *const value);
     In the array represented by a_cookie, create or modify the element whose index is given by index. The ARGV and ENVIRON arrays may not be changed.

awk_bool_t set_array_element_by_elem(awk_array_t a_cookie,
                                     awk_element_t element);
     Like set_array_element(), but take the index and value from element. This is a convenience macro.

awk_bool_t del_array_element(awk_array_t a_cookie,
                             const awk_value_t* const index);
     Remove the element with the given index from the array represented by a_cookie. Return true if the element was removed, or false if the element did not exist in the array.

Functions related to arrays as a whole.

awk_array_t create_array();
     Create a new array to which elements may be added. See Creating Arrays, for a discussion of how to create a new array and add elements to it.

awk_bool_t clear_array(awk_array_t a_cookie);
     Clear the array represented by a_cookie. Return false if there was some kind of problem, true otherwise. The array remains an array, but after calling this function, it has no elements. This is equivalent to using the delete statement (see Delete).

awk_bool_t flatten_array(awk_array_t a_cookie, awk_flat_array_t **data);
     For the array represented by a_cookie, create an awk_flat_array_t structure and fill it in. Set the pointer whose address is passed as data to point to this structure. Return true upon success, or false otherwise. See Flattening Arrays, for a discussion of how to flatten an array and work with it.

awk_bool_t release_flattened_array(awk_array_t a_cookie,
                                   awk_flat_array_t *data);
     When done with a flattened array, release the storage using this function. You must pass in both the original array cookie, and the address of the created awk_flat_array_t structure. The function returns true upon success, false otherwise.

Working With All The Elements of an Array
.........................................

To flatten an array is create a structure that represents the full array in a fashion that makes it easy for C code to traverse the entire array.  Test code in extension/testext.c does this, and also serves as a nice example to show how to use the APIs.

First, the gawk script that drives the test extension:

     @load "testext"
     BEGIN {
         n = split("blacky rusty sophie raincloud lucky", pets)
         printf "pets has %d elements\n", length(pets)
         ret = dump_array_and_delete("pets", "3")
         printf "dump_array_and_delete(pets) returned %d\n", ret
         if ("3" in pets)
             printf("dump_array_and_delete() did NOT remove index \"3\"!\n")
         else
             printf("dump_array_and_delete() did remove index \"3\"!\n")
         print ""
     }

This code creates an array with split() (see String Functions) and then calls dump_and_delete(). That function looks up the array whose name is passed as the first argument, and deletes the element at the index passed in the second argument. It then prints the return value and checks if the element was indeed deleted.  Here is the C code that implements dump_array_and_delete().

The first part declares variables, sets up the default return value in result, and checks that the function was called with the correct number of arguments:

     static awk_value_t *
     dump_array_and_delete(int nargs, awk_value_t *result)
     {
         awk_value_t value, value2, value3;
         awk_flat_array_t *flat_array;
         size_t count;
         char *name;
         int i;

         assert(result != NULL);
         make_number(0.0, result);

         if (nargs != 2) {
             printf("dump_array_and_delete: nargs not right (%d should be 2)\n", nargs);
             goto out;
         }

The function then proceeds in steps, as follows. First, retrieve the name of the array, passed as the first argument. Then retrieve the array itself. If either operation fails, print error messages and return.

         /* get argument named array as flat array and print it */
         if (get_argument(0, AWK_STRING, & value)) {
             name = value.str_value.str;
             if (sym_lookup(name, AWK_ARRAY, & value2))
                 printf("dump_array_and_delete: sym_lookup of %s passed\n", name);
             else {
                 printf("dump_array_and_delete: sym_lookup of %s failed\n", name);
                 goto out;
             }
         } else {
             printf("dump_array_and_delete: get_argument(0) failed\n");
             goto out;
         }

For testing purposes and to make sure that the C code sees the same number of elements as the awk code, The second step is to get the count of elements in the array and print it:

         if (! get_element_count(value2.array_cookie, & count)) {
             printf("dump_array_and_delete: get_element_count failed\n");
             goto out;
         }

         printf("dump_array_and_delete: incoming size is %lu\n", (unsigned long) count);

The third step is to actually flatten the array, and then to double check that the count in the awk_flat_array_t is the same as the count just retrieved.

         if (! flatten_array(value2.array_cookie, & flat_array)) {
             printf("dump_array_and_delete: could not flatten array\n");
             goto out;
         }

         if (flat_array->count != count) {
             printf("dump_array_and_delete: flat_array->count (%lu) != count (%lu)\n",
                     (unsigned long) flat_array->count,
                     (unsigned long) count);
             goto out;
         }

The fourth step is to retrieve the index of the element to be deleted, which was passed as the second argument. Remember that argument counts passed to get_argument() are zero-based, thus the second argument is numbered one.

         if (! get_argument(1, AWK_STRING, & value3)) {
             printf("dump_array_and_delete: get_argument(1) failed\n");
             goto out;
         }

The fifth step is where the ``real work'' is done. The function loops over every element in the array, printing the index and element values. In addition, upon finding the element with the index that is supposed to be deleted, the function sets the AWK_ELEMENT_DELETE bit in the flags field of the element.  When the array is released, gawk traverses the flattened array, and deletes any element which have this flag bit set.

         for (i = 0; i < flat_array->count; i++) {
             printf("\t%s[\"%.*s\"] = %s\n",
                 name,
                 (int) flat_array->elements[i].index.str_value.len,
                 flat_array->elements[i].index.str_value.str,
                 valrep2str(& flat_array->elements[i].value));

             if (strcmp(value3.str_value.str, flat_array->elements[i].index.str_value.str) == 0) {
                 flat_array->elements[i].flags |= AWK_ELEMENT_DELETE;
                 printf("dump_array_and_delete: marking element \"%s\" for deletion\n",
                     flat_array->elements[i].index.str_value.str);
             }
         }

The sixth step is to release the flattened array. This tells gawk that the extension is no longer using the array, and that it should delete any elements marked for deletion. gawk will also free any storage that was allocated, so you should not use the pointer (flat_array in this code) once you have called release_flattened_array():

         if (! release_flattened_array(value2.array_cookie, flat_array)) {
             printf("dump_array_and_delete: could not release flattened array\n");
             goto out;
         }

Finally, since everything was successful, the function sets the return value to success, and returns.

         make_number(1.0, result);
     out:
         return result;
     }

Here is the output from running this part of the test:

     pets has 5 elements
     dump_array_and_delete: sym_lookup of pets passed
     dump_array_and_delete: incoming size is 5
     	pets["1"] = "blacky"
     	pets["2"] = "rusty"
     	pets["3"] = "sophie"
     dump_array_and_delete: marking element "3" for deletion
     	pets["4"] = "raincloud"
     	pets["5"] = "lucky"
     dump_array_and_delete(pets) returned 1
     dump_array_and_delete() did remove index "3"!

How To Create and Populate Arrays
.................................

Besides working with arrays created by awk code, you can create arrays and populate them as you see fit, and then awk code can access them and manipulate them.

There are two important points about creating arrays from extension code:

 1. You must install a new array into gawk's symbol table immediately upon creating it.  Once you have done so, you can then populate the array.

     Similarly, if installing a new array as a subarray of an existing array, you must add the new array to its parent before adding any elements to it.

     Thus, the correct way to build an array is to work ``top down.''  Create the array, and immediately install it in gawk's symbol table using sym_update(), or install it as an element in a previously existing array using set_element(). Example code is coming shortly.

 2. Due to gawk internals, after using sym_update() to install an array into gawk, you have to retrieve the array cookie from the value passed in to sym_update() before doing anything else with it, like so:

          awk_value_t index, value;
          awk_array_t new_array;

          make_const_string("an index", 9, & index);

          new_array = create_array();
          val.val_type = AWK_ARRAY;
          val.array_cookie = new_array;

          sym_update("array", &index, & val);    /* install array in the symbol table */

          new_array = val.array_cookie;    /* YOU MUST DO THIS */

     If installing an array as a subarray, you must also retrieve the value of the array_cookie after the call to set_element().

The following C code is a simple test extension to create an array with two regular elements and with a subarray. The leading #include directives and boilerplate variable declarations are omitted for brevity. The first step is to create a new array and then install it in the symbol table:

     /* create_new_array --- create a named array */

     static void
     create_new_array()
     {
         awk_array_t a_cookie;
         awk_array_t subarray;
         awk_value_t index, value;

         a_cookie = create_array();
         value.val_type = AWK_ARRAY;
         value.array_cookie = a_cookie;

         if (! sym_update("new_array", & value))
             printf("create_new_array: sym_update(\"new_array\") failed!\n");
         a_cookie = value.array_cookie;

Note how a_cookie is reset from the array_cookie field in the value structure.

The second step is to install two regular values into new_array:

         (void) make_const_string("hello", 5, & index);
         (void) make_const_string("world", 5, & value);
         if (! set_array_element(a_cookie, & index, & value)) {
             printf("fill_in_array:%d: set_array_element failed\n", __LINE__);
             return;
         }

         (void) make_const_string("answer", 6, & index);
         (void) make_number(42.0, & value);
         if (! set_array_element(a_cookie, & index, & value)) {
             printf("fill_in_array:%d: set_array_element failed\n", __LINE__);
             return;
         }

The third step is to create the subarray and install it:

         (void) make_const_string("subarray", 8, & index);
         subarray = create_array();
         value.val_type = AWK_ARRAY;
         value.array_cookie = subarray;
         if (! set_array_element(a_cookie, & index, & value)) {
             printf("fill_in_array:%d: set_array_element failed\n", __LINE__);
             return;
         }
         subarray = value.array_cookie;

The final step is to populate the subarray with its own element:

         (void) make_const_string("foo", 3, & index);
         (void) make_const_string("bar", 3, & value);
         if (! set_array_element(subarray, & index, & value)) {
             printf("fill_in_array:%d: set_array_element failed\n", __LINE__);
             return;
         }
     }

Here is sample script that loads the extension and then dumps the array:

     @load "subarray"

     function dumparray(name, array,     i)
     {
         for (i in array)
             if (isarray(array[i]))
                 dumparray(name "[\"" i "\"]", array[i])
             else
                 printf("%s[\"%s\"] = %s\n", name, i, array[i])
     }

     BEGIN {
         dumparray("new_array", new_array);
     }

Here is the result of running the script:

     $ AWKLIBPATH=$PWD ./gawk -f foo.awk
     -| new_array["subarray"]["foo"] = bar
     -| new_array["hello"] = world
     -| new_array["answer"] = 42

Variables
---------

The API provides two sets of variables.  The first provides information about the version of the API (both with which the extension was compiled, and with which gawk was compiled).  The second provides information about how gawk was invoked.

API Version Constants and Variables
...................................

The API provides both a ``major'' and a ``minor'' version number. The API versions are available at compile time as constants:

GAWK_API_MAJOR_VERSION
     The major version of the API.

GAWK_API_MINOR_VERSION
     The minor version of the API.

The minor version increases when new functions are added to the API. Such new functions are always added to the end of the API struct.

The major version increases (and the minor version is reset to zero) if any of the data types change size or member order, or if any of the existing functions change signature.

It could happen that an extension may be compiled against one version of the API but loaded by a version of gawk using a different version. For this reason, the major and minor API versions of the running gawk are included in the API struct as read-only constant integers:

api->major_version
     The major version of the running gawk.

api->minor_version
     The minor version of the running gawk.

It is up to the extension to decide if there are API incompatibilities. Typically a check like this is enough:

     if (api->major_version != GAWK_API_MAJOR_VERSION
         || api->minor_version < GAWK_API_MINOR_VERSION) {
             fprintf(stderr, "foo_extension: version mismatch with gawk!\n");
             fprintf(stderr, "\tmy version (%d, %d), gawk version (%d, %d)\n",
                     GAWK_API_MAJOR_VERSION, GAWK_API_MINOR_VERSION,
                     api->major_version, api->minor_version);
             exit(1);
     }

Such code is included in the boilerplate dl_load_func macro provided in gawkapi.h (discussed later, in Extension API Boilerplate).

Informational Variables
.......................

The API provides access to several variables that describe whether the corresponding command-line options were enabled when gawk was invoked.  The variables are:

do_lint
     This variable will be true if the --lint option was passed (see Options).

do_traditional
     This variable will be true if the --traditional option was passed.

do_profile
     This variable will be true if the --profile option was passed.

do_sandbox
     This variable will be true if the --sandbox option was passed.

do_debug
     This variable will be true if the --debug option was passed.

do_mpfr
     This variable will be true if the --bignum option was passed.

The value of do_lint can change if awk code modifies the LINT built-in variable (see Built-in Variables). The others should not change during execution.

Boilerplate Code
----------------

As mentioned earlier (see Extension Mechanism Outline), the function definitions as presented are really macros. To use these macros, your extension must provide a small amount of boilerplate code (variables and functions) using pre-defined names as described below.  The boilerplate needed is also provided in comments in the gawkapi.h header file:

     /* Boiler plate code: */
     int plugin_is_GPL_compatible;

     static gawk_api_t *const api;
     static awk_ext_id_t ext_id;
     static const char *ext_version = NULL; /* or ... = "some string" */

     static awk_ext_func_t func_table[] = {
         { "name", do_name, 1 },
         /* ... */
     };

     /* EITHER: */

     static awk_bool_t (*init_func)(void) = NULL;

     /* OR: */

     static awk_bool_t
     init_my_module(void)
     {
         ...
     }

     static awk_bool_t (*init_func)(void) = init_my_module;

     dl_load_func(func_table, some_name, "name_space_in_quotes")

These variables and functions are as follows:

int plugin_is_GPL_compatible;
     This asserts that the extension is compatible with the GNU GPL (see Copying). If your extension does not have this, gawk will not load it.

static gawk_api_t *const api;
     This global static variable should be set to point to the gawk_api_t pointer that gawk passes to your dl_load() function. This variable is used by all of the macros.

static awk_ext_id_t ext_id;
     This global static variable should be set to point to the the awk_ext_id_t value that gawk passes to your dl_load() function. This variable is used by all of the macros.

static const char *ext_version = NULL; /* or ... = "some string" */
     This global static variable should be set either to NULL, or to point to a string giving the name and version of your extension.

static awk_ext_func_t func_table[] = { ... };
     This is an array of one or more awk_ext_func_t structures as described earlier (see Extension Functions). It can then be looped over for multiple calls to add_ext_func().

static awk_bool_t (*init_func)(void) = NULL;
                   OR
static awk_bool_t init_my_module(void) { ... }
static awk_bool_t (*init_func)(void) = init_my_module;
     If you need to do some initialization work, you should define a function that does it (creates variables, opens files, etc.) and then define the init_func pointer to point to your function. The function should return zero (false) upon failure, non-zero (success) if everything goes well.

     If you don't need to do any initialization, define the pointer and initialize it to NULL.

dl_load_func(func_table, some_name, "name_space_in_quotes")
     This macro expands to a dl_load() function that performs all the necessary initializations.

The point of the all the variables and arrays is to let the dl_load() function do all the standard work. It does the following:

 1. Check the API versions. If the extension major version does not match gawk's, or if the extension minor version is greater than gawk's, it prints a fatal error message and exits.

 2. Load the functions defined in func_table. If any of them fails to load, it prints a warning message but continues on.

 3. If the init_func pointer is not NULL, call the function it points to. If it returns non-zero, print a warning message.

 4. If ext_version is not NULL, register the version string with gawk.

How gawk Finds Extensions
-------------------------

Compiled extensions have to be installed in a directory where gawk can find them.  If gawk is configured and built in the default fashion, the default directory in which to find extensions is /usr/local/lib/gawk.  You can also specify a search path with a list of directories to search for compiled extensions. See AWKLIBPATH Variable, for more information.

Example: Some File Functions
============================

     No matter where you go, there you are.

     Buckaroo Bonzai

Two useful functions that are not in awk are chdir() (so that an awk program can change its directory) and stat() (so that an awk program can gather information about a file). This section implements these functions for gawk in an external extension.

Using chdir() and stat()
------------------------

This section shows how to use the new functions at the awk level once they've been integrated into the running gawk interpreter.  Using chdir() is very straightforward. It takes one argument, the new directory to change to:

     @load "filefuncs"
     ...
     newdir = "/home/arnold/funstuff"
     ret = chdir(newdir)
     if (ret < 0) {
         printf("could not change to %s: %s\n",
                        newdir, ERRNO) > "/dev/stderr"
         exit 1
     }
     ...

The return value is negative if the chdir() failed, and ERRNO (see Built-in Variables) is set to a string indicating the error.

Using stat() is a bit more complicated.  The C stat() function fills in a structure that has a fair amount of information. The right way to model this in awk is to fill in an associative array with the appropriate information:

     file = "/home/arnold/.profile"
     ret = stat(file, fdata)
     if (ret < 0) {
         printf("could not stat %s: %s\n",
                  file, ERRNO) > "/dev/stderr"
         exit 1
     }
     printf("size of %s is %d bytes\n", file, fdata["size"])

The stat() function always clears the data array, even if the stat() fails.  It fills in the following elements:

"name"
     The name of the file that was stat()'ed.

"dev"
"ino"
     The file's device and inode numbers, respectively.

"mode"
     The file's mode, as a numeric value. This includes both the file's type and its permissions.

"nlink"
     The number of hard links (directory entries) the file has.

"uid"
"gid"
     The numeric user and group ID numbers of the file's owner.

"size"
     The size in bytes of the file.

"blocks"
     The number of disk blocks the file actually occupies. This may not be a function of the file's size if the file has holes.

"atime"
"mtime"
"ctime"
     The file's last access, modification, and inode update times, respectively.  These are numeric timestamps, suitable for formatting with strftime() (see Built-in).

"pmode"
     The file's ``printable mode.''  This is a string representation of the file's type and permissions, such as what is produced by ls -l---for example, "drwxr-xr-x".

"type"
     A printable string representation of the file's type.  The value is one of the following:

     "blockdev"
     "chardev"
          The file is a block or character device (``special file'').

     "directory"
          The file is a directory.

     "fifo"
          The file is a named-pipe (also known as a FIFO).

     "file"
          The file is just a regular file.

     "socket"
          The file is an AF_UNIX (``Unix domain'') socket in the filesystem.

     "symlink"
          The file is a symbolic link.

Several additional elements may be present depending upon the operating system and the type of the file.  You can test for them in your awk program by using the in operator (see Reference to Elements):

"blksize"
     The preferred block size for I/O to the file. This field is not present on all POSIX-like systems in the C stat structure.

"linkval"
     If the file is a symbolic link, this element is the name of the file the link points to (i.e., the value of the link).

"rdev"
"major"
"minor"
     If the file is a block or character device file, then these values represent the numeric device number and the major and minor components of that number, respectively.

C Code for chdir() and stat()
-----------------------------

Here is the C code for these extensions. (This version is edited slightly for presentation.  See extension/filefuncs.c in the gawk distribution for the complete version.)

The file includes a number of standard header files, and then includes the "gawkapi.h" header file which provides the API definitions. Those are followed by the necessary variable declarations to make use of the API macros and boilerplate code (see Extension API Boilerplate).

     #ifdef HAVE_CONFIG_H
     #include <config.h>
     #endif

     #include <stdio.h>
     #include <assert.h>
     #include <errno.h>
     #include <stdlib.h>
     #include <string.h>
     #include <unistd.h>

     #include <sys/types.h>
     #include <sys/stat.h>

     #include "gawkapi.h"

     #include "gettext.h"
     #define _(msgid)  gettext(msgid)
     #define N_(msgid) msgid

     #include "gawkfts.h"
     #include "stack.h"

     static const gawk_api_t *api;    /* for convenience macros to work */
     static awk_ext_id_t *ext_id;
     static awk_bool_t init_filefuncs(void);
     static awk_bool_t (*init_func)(void) = init_filefuncs;
     static const char *ext_version = "filefuncs extension: version 1.0";

     int plugin_is_GPL_compatible;

By convention, for an awk function foo(), the function that implements it is called do_foo().  The function should have two arguments: the first is an int usually called nargs, that represents the number of defined arguments for the function. The second is a pointer to an awk_result_t, usually named result.

     /*  do_chdir --- provide dynamically loaded chdir() builtin for gawk */

     static awk_value_t *
     do_chdir(int nargs, awk_value_t *result)
     {
         awk_value_t newdir;
         int ret = -1;

         assert(result != NULL);

         if (do_lint && nargs != 1)
             lintwarn(ext_id, _("chdir: called with incorrect number of arguments, expecting 1"));

The newdir variable represents the new directory to change to, retrieved with get_argument().  Note that the first argument is numbered zero.

If the argument is retrieved successfully, the function calls the chdir() system call. If the chdir() fails, ERRNO is updated.

         if (get_argument(0, AWK_STRING, & newdir)) {
             ret = chdir(newdir.str_value.str);
             if (ret < 0)
                 update_ERRNO_int(errno);
         }

Finally, the function returns the return value to the awk level:

         return make_number(ret, result);
     }

The stat() built-in is more involved.  First comes a function that turns a numeric mode into a printable representation (e.g., 644 becomes -rw-r--r--). This is omitted here for brevity:

     /* format_mode --- turn a stat mode field into something readable */

     static char *
     format_mode(unsigned long fmode)
     {
         ...
     }

Next comes a function for reading symbolic links, which is also omitted here for brevity:

     /* read_symlink --- read a symbolic link into an allocated buffer.
        ... */

     static char *
     read_symlink(const char *fname, size_t bufsize, ssize_t *linksize)
     {
         ...
     }

Two helper functions simplify entering values in the array that will contain the result of the stat():

     /* array_set --- set an array element */

     static void
     array_set(awk_array_t array, const char *sub, awk_value_t *value)
     {
         awk_value_t index;

         set_array_element(array,
                           make_const_string(sub, strlen(sub), & index),
                           value);

     }

     /* array_set_numeric --- set an array element with a number */

     static void
     array_set_numeric(awk_array_t array, const char *sub, double num)
     {
         awk_value_t tmp;

         array_set(array, sub, make_number(num, & tmp));
     }

The following function does most of the work to fill in the awk_array_t result array with values obtained from a valid struct stat. It is done in a separate function to support the stat() function for gawk and also to support the fts() extension which is included in the same file but whose code is not shown here (see Extension Sample File Functions).

The first part of the function is variable declarations, including a table to map file types to strings:

     /* fill_stat_array --- do the work to fill an array with stat info */

     static int
     fill_stat_array(const char *name, awk_array_t array, struct stat *sbuf)
     {
         char *pmode;    /* printable mode */
         const char *type = "unknown";
         awk_value_t tmp;
         static struct ftype_map {
             unsigned int mask;
             const char *type;
         } ftype_map[] = {
             { S_IFREG, "file" },
             { S_IFBLK, "blockdev" },
             { S_IFCHR, "chardev" },
             { S_IFDIR, "directory" },
     #ifdef S_IFSOCK
             { S_IFSOCK, "socket" },
     #endif
     #ifdef S_IFIFO
             { S_IFIFO, "fifo" },
     #endif
     #ifdef S_IFLNK
             { S_IFLNK, "symlink" },
     #endif
     #ifdef S_IFDOOR /* Solaris weirdness */
             { S_IFDOOR, "door" },
     #endif /* S_IFDOOR */
         };
         int j, k;

The destination array is cleared, and then code fills in various elements based on values in the struct stat:

         /* empty out the array */
         clear_array(array);

         /* fill in the array */
         array_set(array, "name", make_const_string(name, strlen(name), & tmp));
         array_set_numeric(array, "dev", sbuf->st_dev);
         array_set_numeric(array, "ino", sbuf->st_ino);
         array_set_numeric(array, "mode", sbuf->st_mode);
         array_set_numeric(array, "nlink", sbuf->st_nlink);
         array_set_numeric(array, "uid", sbuf->st_uid);
         array_set_numeric(array, "gid", sbuf->st_gid);
         array_set_numeric(array, "size", sbuf->st_size);
         array_set_numeric(array, "blocks", sbuf->st_blocks);
         array_set_numeric(array, "atime", sbuf->st_atime);
         array_set_numeric(array, "mtime", sbuf->st_mtime);
         array_set_numeric(array, "ctime", sbuf->st_ctime);

         /* for block and character devices, add rdev, major and minor numbers */
         if (S_ISBLK(sbuf->st_mode) || S_ISCHR(sbuf->st_mode)) {
             array_set_numeric(array, "rdev", sbuf->st_rdev);
             array_set_numeric(array, "major", major(sbuf->st_rdev));
             array_set_numeric(array, "minor", minor(sbuf->st_rdev));
         }

The latter part of the function makes selective additions to the destination array, depending upon the availability of certain members and/or the type of the file. In the returns zero, for success:

     #ifdef HAVE_ST_BLKSIZE
         array_set_numeric(array, "blksize", sbuf->st_blksize);
     #endif /* HAVE_ST_BLKSIZE */

         pmode = format_mode(sbuf->st_mode);
         array_set(array, "pmode", make_const_string(pmode, strlen(pmode), & tmp));

         /* for symbolic links, add a linkval field */
         if (S_ISLNK(sbuf->st_mode)) {
             char *buf;
             ssize_t linksize;

             if ((buf = read_symlink(name, sbuf->st_size,
                         & linksize)) != NULL)
                 array_set(array, "linkval", make_malloced_string(buf, linksize, & tmp));
             else
                 warning(ext_id, _("stat: unable to read symbolic link `%s'"), name);
         }

         /* add a type field */
         type = "unknown";   /* shouldn't happen */
         for (j = 0, k = sizeof(ftype_map)/sizeof(ftype_map[0]); j < k; j++) {
             if ((sbuf->st_mode & S_IFMT) == ftype_map[j].mask) {
                 type = ftype_map[j].type;
                 break;
             }
         }

         array_set(array, "type", make_const_string(type, strlen(type), &tmp));

         return 0;
     }

Finally, here is the do_stat() function. It starts with variable declarations and argument checking:

     /* do_stat --- provide a stat() function for gawk */

     static awk_value_t *
     do_stat(int nargs, awk_value_t *result)
     {
         awk_value_t file_param, array_param;
         char *name;
         awk_array_t array;
         int ret;
         struct stat sbuf;

         assert(result != NULL);

         if (do_lint && nargs != 2) {
             lintwarn(ext_id, _("stat: called with wrong number of arguments"));
             return make_number(-1, result);
         }

Then comes the actual work. First, the function gets the arguments. Next, it gets the information for the file. The code use lstat() (instead of stat()) to get the file information, in case the file is a symbolic link. If there's an error, it sets ERRNO and returns:

         /* file is first arg, array to hold results is second */
         if (   ! get_argument(0, AWK_STRING, & file_param)
             || ! get_argument(1, AWK_ARRAY, & array_param)) {
             warning(ext_id, _("stat: bad parameters"));
             return make_number(-1, result);
         }

         name = file_param.str_value.str;
         array = array_param.array_cookie;

         /* always empty out the array */
         clear_array(array);

         /* lstat the file, if error, set ERRNO and return */
         ret = lstat(name, & sbuf);
         if (ret < 0) {
             update_ERRNO_int(errno);
             return make_number(ret, result);
         }

The tedious work is done by fill_stat_array(), shown earlier.  When done, return the result from fill_stat_array():

         ret = fill_stat_array(name, array, & sbuf);

         return make_number(ret, result);
     }

Finally, it's necessary to provide the ``glue'' that loads the new function(s) into gawk.

The filefuncs extension also provides an fts() function, which we omit here. For its sake there is an initialization function:

     /* init_filefuncs --- initialization routine */

     static awk_bool_t
     init_filefuncs(void)
     {
         ...
     }

We are almost done. We need an array of awk_ext_func_t structures for loading each function into gawk:

     static awk_ext_func_t func_table[] = {
         { "chdir", do_chdir, 1 },
         { "stat",  do_stat, 2 },
         { "fts",   do_fts, 3 },
     };

Each extension must have a routine named dl_load() to load everything that needs to be loaded.  The simplest way is to use the dl_load_func macro in gawkapi.h:

     /* define the dl_load function using the boilerplate macro */

     dl_load_func(func_table, filefuncs, "")

And that's it!  As an exercise, consider adding functions to implement system calls such as chown(), chmod(), and umask().

Integrating the Extensions
--------------------------

Now that the code is written, it must be possible to add it at runtime to the running gawk interpreter.  First, the code must be compiled.  Assuming that the functions are in a file named filefuncs.c, and IDIR is the location of the gawkapi.h header file, the following steps (In practice, you would probably want to
use the GNU Autotools---Automake, Autoconf, Libtool, and Gettext---to
configure and build your libraries. Instructions for doing so are beyond
the scope of this book. See gawkextlib, for WWW links to
the tools.) create a GNU/Linux shared library:

     $ gcc -fPIC -shared -DHAVE_CONFIG_H -c -O -g -IIDIR filefuncs.c
     $ ld -o filefuncs.so -shared filefuncs.o -lc

Once the library exists, it is loaded by using the @load keyword.

     # file testff.awk
     @load "filefuncs"

     BEGIN {
         "pwd" | getline curdir  # save current directory
         close("pwd")

         chdir("/tmp")
         system("pwd")   # test it
         chdir(curdir)   # go back

         print "Info for testff.awk"
         ret = stat("testff.awk", data)
         print "ret =", ret
         for (i in data)
             printf "data[\"%s\"] = %s\n", i, data[i]
         print "testff.awk modified:",
             strftime("%m %d %y %H:%M:%S", data["mtime"])

         print "\nInfo for JUNK"
         ret = stat("JUNK", data)
         print "ret =", ret
         for (i in data)
             printf "data[\"%s\"] = %s\n", i, data[i]
         print "JUNK modified:", strftime("%m %d %y %H:%M:%S", data["mtime"])
     }

The AWKLIBPATH environment variable tells gawk where to find shared libraries (see Finding Extensions). We set it to the current directory and run the program:

     $ AWKLIBPATH=$PWD gawk -f testff.awk
     -| /tmp
     -| Info for testff.awk
     -| ret = 0
     -| data["blksize"] = 4096
     -| data["mtime"] = 1350838628
     -| data["mode"] = 33204
     -| data["type"] = file
     -| data["dev"] = 2053
     -| data["gid"] = 1000
     -| data["ino"] = 1719496
     -| data["ctime"] = 1350838628
     -| data["blocks"] = 8
     -| data["nlink"] = 1
     -| data["name"] = testff.awk
     -| data["atime"] = 1350838632
     -| data["pmode"] = -rw-rw-r--
     -| data["size"] = 662
     -| data["uid"] = 1000
     -| testff.awk modified: 10 21 12 18:57:08
     -|
     -| Info for JUNK
     -| ret = -1
     -| JUNK modified: 01 01 70 02:00:00

The Sample Extensions in the gawk Distribution
==============================================

This section provides brief overviews of the sample extensions that come in the gawk distribution. Some of them are intended for production use, such the filefuncs and readdir extensions. Others mainly provide example code that shows how to use the extension API.

File Related Functions
----------------------

The filefuncs extension provides three different functions, as follows: The usage is:

@load "filefuncs"
     This is how you load the extension.

result = chdir("/some/directory")
     The chdir() function is a direct hook to the chdir() system call to change the current directory.  It returns zero upon success or less than zero upon error.  In the latter case it updates ERRNO.

result = stat("/some/path", statdata)
     The stat() function provides a hook into the stat() system call. In fact, it uses lstat(). It returns zero upon success or less than zero upon error. In the latter case it updates ERRNO.

     In all cases, it clears the statdata array. When the call is successful, stat() fills the statdata array with information retrieved from the filesystem, as follows:

     statdata["name"]
          The name of the file.

     statdata["dev"]
          Corresponds to the st_dev field in the struct stat.

     statdata["ino"]
          Corresponds to the st_ino field in the struct stat.

     statdata["mode"]
          Corresponds to the st_mode field in the struct stat.

     statdata["nlink"]
          Corresponds to the st_nlink field in the struct stat.

     statdata["uid"]
          Corresponds to the st_uid field in the struct stat.

     statdata["gid"]
          Corresponds to the st_gid field in the struct stat.

     statdata["size"]
          Corresponds to the st_size field in the struct stat.

     statdata["atime"]
          Corresponds to the st_atime field in the struct stat.

     statdata["mtime"]
          Corresponds to the st_mtime field in the struct stat.

     statdata["ctime"]
          Corresponds to the st_ctime field in the struct stat.

     statdata["rdev"]
          Corresponds to the st_rdev field in the struct stat. This element is only present for device files.

     statdata["major"]
          Corresponds to the st_major field in the struct stat. This element is only present for device files.

     statdata["minor"]
          Corresponds to the st_minor field in the struct stat. This element is only present for device files.

     statdata["blksize"]
          Corresponds to the st_blksize field in the struct stat. if this field is present on your system. (It is present on all modern systems that we know of.)

     statdata["pmode"]
          A human-readable version of the mode value, such as printed by ls.  For example, "-rwxr-xr-x".

     statdata["linkval"]
          If the named file is a symbolic link, this element will exist and its value is the value of the symbolic link (where the symbolic link points to).

     statdata["type"]
          The type of the file as a string. One of "file", "blockdev", "chardev", "directory", "socket", "fifo", "symlink", "door", or "unknown". Not all systems support all file types.

flags = or(FTS_PHYSICAL, ...)
result = fts(pathlist, flags, filedata)
     Walk a the file trees provided in pathlist and fill in the filedata array as described below. flags is the bitwise OR of several predefined constant values, also as described below.

The fts() function provides a hook to the fts() set of routines for traversing file hierarchies.  Instead of returning data about one file at a time in a stream, it fills in a multi-dimensional array with data about each file and directory encountered in the requested hierarchies.

The arguments are as follows:

pathlist
     An array of filenames.  The element values are used; the index values are ignored.

flags
     This should be the bitwise OR of one or more of the following predefined constant flag values.  At least one of FTS_LOGICAL or FTS_PHYSICAL must be provided; otherwise fts() returns an error value and sets ERRNO. The flags are:

     FTS_LOGICAL
          Do a ``logical'' file traversal, where the information returned for a symbolic link refers to the linked-to file, and not to the symbolic link itself.  This flag is mutually exclusive with FTS_PHYSICAL.

     FTS_PHYSICAL
          Do a ``physical'' file traversal, where the information returned for a symbolic link refers to the symbolic link itself.  This flag is mutually exclusive with FTS_LOGICAL.

     FTS_NOCHDIR
          As a performance optimization, the C library fts() routines change directory as they traverse a file hierarchy.  This flag disables that optimization.

     FTS_COMFOLLOW
          Immediately follow a symbolic link named in pathlist, whether or not FTS_LOGICAL is set.

     FTS_SEEDOT
          By default, the fts() routines do not return entries for . and ...  This option causes entries for .. to also be included.  (This extension always includes an entry for ., see below.)

     FTS_XDEV
          During a traversal, do not cross onto a different mounted filesystem.

filedata
     The filedata array is first cleared.  Then, fts() creates an element in filedata for every element in pathlist. The index is the name of the directory or file given in pathlist. The element for this index is itself an array.  There are two cases.

     The path is a file.
          In this case, the array contains two or three elements:

          "path"
               The full path to this file, starting from the ``root'' that was given in the pathlist array.

          "stat"
               This element is itself an array, containing the same information as provided by the stat() function described earlier for its statdata argument.  The element may not be present if stat() system call for the file failed.

          "error"
               If some kind of error was encountered, the array will also contain an element named "error", which is a string describing the error.

     The path is a directory.
          In this case, the array contains one element for each entry in the directory.  If an entry is a file, that element is as for files, just described.  If the entry is a directory, that element is (recursively), an array describing the subdirectory.  If FTS_SEEDOT was provided in the flags, then there will also be an element named "..".  This element will be an array containing the data as provided by stat().

          In addition, there will be an element whose index is ".". This element is an array containing the same two or three elements as for a file: "path", "stat", and "error".

The fts() function returns 0 if there were no errors. Otherwise it returns -1.

     The fts() extension does not exactly mimic the interface of the C library fts() routines, choosing instead to provide an interface that is based on associative arrays, which should be more comfortable to use from an awk program.  This includes the lack of a comparison function, since gawk already provides powerful array sorting facilities.  While an fts_read()-like interface could have been provided, this felt less natural than simply creating a multi-dimensional array to represent the file hierarchy and its information.

See test/fts.awk in the gawk distribution for an example.

Interface To fnmatch()
----------------------

This extension provides and interface to the C library fnmatch() function.  The usage is:

     @load "fnmatch"

     result = fnmatch(pattern, string, flags)

The fnmatch extension adds a single function named fnmatch(), one constant (FNM_NOMATCH), and an array of flag values named FNM.

The arguments to fnmatch() are:

pattern
     The filename wildcard to match.

string
     The filename string,

flag
     Either zero, or the bitwise OR of one or more of the flags in the FNM array.

The return value is zero on success, FNM_NOMATCH if the string did not match the pattern, or a different non-zero value if an error occurred.

The flags are follows:

FNM["CASEFOLD"]
     Corresponds to the FNM_CASEFOLD flag as defined in fnmatch().

FNM["FILE_NAME"]
     Corresponds to the FNM_FILE_NAME flag as defined in fnmatch().

FNM["LEADING_DIR"]
     Corresponds to the FNM_LEADING_DIR flag as defined in fnmatch().

FNM["NOESCAPE"]
     Corresponds to the FNM_NOESCAPE flag as defined in fnmatch().

FNM["PATHNAME"]
     Corresponds to the FNM_PATHNAME flag as defined in fnmatch().

FNM["PERIOD"]
     Corresponds to the FNM_PERIOD flag as defined in fnmatch().

Here is an example:

     @load "fnmatch"
     ...
     flags = or(FNM["PERIOD"], FNM["NOESCAPE"])
     if (fnmatch("*.a", "foo.c", flags) == FNM_NOMATCH)
         print "no match"

Interface to fork(), wait() and waitpid()
-----------------------------------------

The fork extension adds three functions, as follows.

@load "fork"
     This is how you load the extension.

pid = fork()
     This function creates a new process. The return value is the zero in the child and the process-id number of the child in the parent, or -1 upon error. In the latter case, ERRNO indicates the problem. In the child, PROCINFO["pid"] and PROCINFO["ppid"] are updated to reflect the correct values.

ret = waitpid(pid)
     This function takes a numeric argument, which is the process-id to wait for. The return value is that of the waitpid() system call.

ret = wait()
     This function waits for the first child to die. The return value is that of the wait() system call.

There is no corresponding exec() function.

Here is an example:

     @load "fork"
     ...
     if ((pid = fork()) == 0)
         print "hello from the child"
     else
         print "hello from the parent"

Character and Numeric values: ord() and chr()
---------------------------------------------

The ordchr extension adds two functions, named ord() and chr(), as follows.

number = ord(string)
     This function takes a string argument, and returns the numeric value of the first character in the string.

char = chr(number)
     This function takes a numeric argument and returns a string whose first character is that represented by the number.

These functions are inspired by the Pascal language functions of the same name.

Here is an example:

     @load "ordchr"
     ...
     printf("The numeric value of 'A' is %d\n", ord("A"))
     printf("The string value of 65 is %s\n", chr(65))

Reading Directories
-------------------

The readdir extension adds an input parser for directories, and adds a single function named readdir_do_ftype(). The usage is as follows:

     @load "readdir"

     readdir_do_ftype("stat")    # or "dirent" or "never"

When this extension is in use, instead of skipping directories named on the command line (or with getline), they are read, with each entry returned as a record.

The record consists of at least two fields: the inode number and the filename, separated by a forward slash character. On systems where the directory entry contains the file type, the record has a third field which is a single letter indicating the type of the file: f for file, d for directory, b for a block device, c for a character device, p for a FIFO, l for a symbolic link, s for a socket, and u (unknown) for anything else.

On systems without the file type information, calling readdir_do_ftype("stat") causes the extension to use the lstat() system call to retrieve the appropriate information. This is not the default, since lstat() is a potentially expensive operation.  By calling readdir_do_ftype("never") one can ensure that the file type information is never displayed, even when readily available in the directory entry.

The third option, readdir_do_ftype("dirent"), takes file type information from the directory entry, if it is available.  This is the default on systems that supply this information.

The readdir_do_ftype() function sets ERRNO if called without arguments or with invalid arguments.

     On GNU/Linux systems, there are filesystems that don't support the d_type entry (see the readdir(3) manual page), and so the file type is always u.  Therefore, using readdir_do_ftype("stat") is advisable even on GNU/Linux systems.  In this case, the readdir extension will fall back to using lstat() when it encounters an unknown file type.

Here is an example:

     @load "readdir"
     ...
     BEGIN { FS = "/" }
     { print "file name is", $2 }

Reversing Output
----------------

The revoutput extension adds a simple output wrapper that reverses the characters in each output line.  It's main purpose is to show how to write an output wrapper, although it may be mildly amusing for the unwary.

     @load "revoutput"

     BEGIN {
         REVOUT = 1
         print "hello, world" > "/dev/stdout"
     }

The output from this program is: dlrow ,olleh.

Two-Way I/O Example
-------------------

The revtwoway extension adds a simple two-way processor that reverses the characters in each line sent to it for reading back by the awk program.  It's main purpose is to show how to write a two-way extension, although it may also be mildly amusing.

The following example shows how to use it:

     @load "revtwoway"

     BEGIN {
         cmd = "/magic/mirror"
         print "hello, world" |& cmd
         cmd |& getline result
         print result
         close(cmd)
     }

Dumping and Restoring An Array
------------------------------

The rwarray extension adds two functions, named writea() and reada(), as follows:

ret = writea(file, array)
     This function takes a string argument, which is the name of the file to which dump the array, and the array itself as the second argument. writea() understands multidimensional arrays.  It returns 1 on success, 0 on failure.

ret = reada(file, array)
     reada() is the inverse of writea(); it reads the file named as its first argument, filling in the array named as the second argument. It clears the array first. Here too, the return value is 1 on success and 0 on failure.

The array created by reada() is identical to that written by writea() in the sense that the contents are the same. However, due to implementation issues, the array traversal order of the recreated array will likely be different from that of the original array.  As array traversal order in awk is by default undefined, this is not (technically) a problem.  If you need to guarantee a particular traversal order, use the array sorting features in gawk to do so.

The file contains binary data.  All integral values are written in network byte order.  However, double precision floating-point values are written as native binary data.  Thus, arrays containing only string data can theoretically be dumped on systems with one byte order and restored on systems with a different one, but this has not been tried.

Here is an example:

     @load "rwarray"
     ...
     ret = writea("arraydump.bin", array)
     ...
     ret = reada("arraydump.bin", array)

Reading An Entire File
----------------------

The readfile extension adds a single function named readfile():

result = readfile("/some/path")
     The argument is the name of the file to read.  The return value is a string containing the entire contents of the requested file.  Upon error, the function returns the empty string and sets ERRNO.

Here is an example:

     @load "readfile"
     ...
     contents = readfile("/path/to/file");
     if (contents == "" && ERRNO != "") {
         print("problem reading file", ERRNO) > "/dev/stderr"
         ...
     }

API Tests
---------

The testext extension exercises parts of the extension API that are not tested by the other samples.  The extension/testext.c file contains both the C code for the extension and awk test code inside C comments that run the tests. The testing framework extracts the awk code and runs the tests.  See the source file for more information.

Time Functions
--------------

These functions can be used by either invoking gawk with a command-line argument of -l time or by inserting @load "time" in your script.

gettimeofday()
     This function returns the time that has elapsed since 1970-01-01 UTC as a floating point value.  It should have sub-second precision, but the actual precision will vary based on the platform.  If the time is unavailable on this platform, it returns -1 and sets ERRNO. If the standard C gettimeofday() system call is available on this platform, then it simply returns the value.  Otherwise, if on Windows, it tries to use GetSystemTimeAsFileTime().

sleep(SECONDS)
     This function attempts to sleep for SECONDS seconds. Note that SECONDS may be a floating-point (non-integral) value. If SECONDS is negative, or the attempt to sleep fails, then it returns -1 and sets ERRNO. Otherwise, the function should return 0 after sleeping for the indicated amount of time.  Implementation details: depending on platform availability, it tries to use nanosleep() or select() to implement the delay.

The gawkextlib Project
======================

The gawkextlib (http://sourceforge.net/projects/gawkextlib/) project provides a number of gawk extensions, including one for processing XML files.  This is the evolution of the original xgawk (XML gawk) project.

As of this writing, there are four extensions:

  * XML parser extension, using the Expat (http://expat.sourceforge.net) XML parsing library

  * Postgres SQL extension

  * GD graphics library extension

  * MPFR library extension. This provides access to a number of MPFR functions which gawk's native MPFR support does not.

The time extension described earlier (see Extension Sample Time) was originally from this project but has been moved in to the main gawk distribution.

You can check out the code for the gawkextlib project using the GIT (http://git-scm.com) distributed source code control system.  The command is as follows:

     git clone git://git.code.sf.net/p/gawkextlib/code gawkextlib-code

You will need to have the Expat (http://expat.sourceforge.net) XML parser library installed in order to build and use the XML extension.

In addition, you should have the GNU Autotools installed (Autoconf (http://www.gnu.org/software/autoconf), Automake (http://www.gnu.org/software/automake), Libtool (http://www.gnu.org/software/libtool), and Gettext (http://www.gnu.org/software/gettext)).

The simple recipe for building and testing gawkextlib is as follows. First, build and install gawk:

     cd .../path/to/gawk/code
     ./configure --prefix=/tmp/newgawk     Install in /tmp/newgawk for now
     make && make check                    Build and check that all is OK
     make install                          Install gawk

Next, build gawkextlib and test it:

     cd .../path/to/gawkextlib-code
     ./update-autotools                    Generate configure, etc. May have to run twice
     ./configure --with-gawk=/tmp/newgawk  Configure, point at ``installed'' gawk
     make && make check                    Build and check that all is OK

Fake Sections For Cross References
**********************************

Referring to an Array Element
=============================

Built-in Functions
==================

Built-in Variables
==================

Command-Line Options
====================

The AWKLIBPATH Environment Variable
===================================

The BEGINFILE and ENDFILE Special Patterns
==========================================

Redirecting Output of print and printf
======================================

Arrays in awk
=============

Conversion of Strings and Numbers
=================================

The delete Statement
====================

String-Manipulation Functions
=============================

Glossary
========

GNU General Public License
==========================