Move to gawk-3.1.0.

1 files changed, 5075 insertions, 0 deletions

diff --git a/doc/gawkinet.texi b/doc/gawkinet.texi
new file mode 100644
index 00000000..2ffb5814
--- /dev/null
+++ b/doc/gawkinet.texi
@@ -0,0 +1,5075 @@
+\input texinfo   @c -*-texinfo-*-      
+@c %**start of header (This is for running Texinfo on a region.)
+@setfilename gawkinet.info
+@settitle TCP/IP Internetworking With @command{gawk}
+@c %**end of header (This is for running Texinfo on a region.)
+
+@c inside ifinfo for older versions of texinfo.tex
+@ifinfo
+@dircategory GNU Packages
+@direntry
+* Gawkinet: (gawkinet).    TCP/IP Internetworking With @command{gawk}.
+@end direntry
+@end ifinfo
+
+@iftex
+@set DOCUMENT book
+@set CHAPTER chapter
+@set SECTION section
+@set DARKCORNER @inmargin{@image{lflashlight,1cm}, @image{rflashlight,1cm}}
+@end iftex
+@ifinfo
+@set DOCUMENT Info file
+@set CHAPTER major node
+@set SECTION node
+@set DARKCORNER (d.c.)
+@end ifinfo
+@ifhtml
+@set DOCUMENT web page
+@set CHAPTER chapter
+@set SECTION section
+@set DARKCORNER (d.c.)
+@end ifhtml
+
+@set FSF
+
+@set FN file name
+@set FFN File Name
+
+@c merge the function and variable indexes into the concept index
+@ifinfo
+@synindex fn cp
+@synindex vr cp
+@end ifinfo
+@iftex
+@syncodeindex fn cp
+@syncodeindex vr cp
+@end iftex
+
+@c If "finalout" is commented out, the printed output will show
+@c black boxes that mark lines that are too long.  Thus, it is
+@c unwise to comment it out when running a master in case there are
+@c overfulls which are deemed okay.
+
+@iftex
+@finalout
+@end iftex
+
+@smallbook
+
+@c Special files are described in chapter 6 Printing Output under
+@c 6.7 Special File Names in gawk. I think the networking does not
+@c fit into that chapter, thus this separate document. At over 50
+@c pages, I think this is the right decision.  ADR.
+
+@set TITLE TCP/IP Internetworking With @command{gawk}
+@set EDITION 1.1
+@set UPDATE-MONTH March, 2001
+@c gawk versions:
+@set VERSION 3.1
+@set PATCHLEVEL 0
+
+@ifinfo
+This file documents the networking features in GNU @command{awk}.
+
+This is Edition @value{EDITION} of @cite{@value{TITLE}},
+for the @value{VERSION}.@value{PATCHLEVEL} (or later) version of the GNU
+implementation of AWK.
+
+Copyright (C) 2000, 2001 Free Software Foundation, Inc.
+
+Permission is granted to copy, distribute and/or modify this document
+under the terms of the GNU Free Documentation License, Version 1.1 or
+any later version published by the Free Software Foundation; with the
+Invariant Sections being ``GNU General Public License'', the Front-Cover
+texts being (a) (see below), and with the Back-Cover Texts being (b)
+(see below).  A copy of the license is included in the section entitled
+``GNU Free Documentation License''.
+
+@enumerate a
+@item
+``A GNU Manual''
+
+@item
+``You have freedom to copy and modify this GNU Manual, like GNU
+software.  Copies published by the Free Software Foundation raise
+funds for GNU development.''
+@end enumerate
+@end ifinfo
+
+@setchapternewpage odd
+
+@titlepage
+@title @value{TITLE}
+@subtitle Edition @value{EDITION}
+@subtitle @value{UPDATE-MONTH}
+@author J@"urgen Kahrs
+@author with Arnold D. Robbins
+
+@c Include the Distribution inside the titlepage environment so
+@c that headings are turned off.  Headings on and off do not work.
+
+@page
+@vskip 0pt plus 1filll
+Copyright @copyright{} 2000, 2001 Free Software Foundation, Inc.
+@sp 1
+@b{User Friendly} Copyright @copyright{} 2000 J.D.@: ``Iliad'' Frazier.
+Reprinted by permission.
+@sp 2
+        
+This is Edition @value{EDITION} of @cite{@value{TITLE}},
+for the @value{VERSION}.@value{PATCHLEVEL} (or later) version of the GNU
+implementation of AWK.
+
+@sp 2
+Published by:
+@sp 1
+
+Free Software Foundation @*
+59 Temple Place --- Suite 330 @*
+Boston, MA 02111-1307 USA @*
+Phone: +1-617-542-5942 @*
+Fax: +1-617-542-2652 @*
+Email: @email{gnu@@gnu.org} @*
+URL: @uref{http://www.gnu.org/} @*
+
+ISBN 1-882114-93-0 @*
+
+Permission is granted to copy, distribute and/or modify this document
+under the terms of the GNU Free Documentation License, Version 1.1 or
+any later version published by the Free Software Foundation; with the
+Invariant Sections being ``GNU General Public License'', the Front-Cover
+texts being (a) (see below), and with the Back-Cover Texts being (b)
+(see below).  A copy of the license is included in the section entitled
+``GNU Free Documentation License''.
+
+@enumerate a
+@item
+``A GNU Manual''
+
+@item
+``You have freedom to copy and modify this GNU Manual, like GNU
+software.  Copies published by the Free Software Foundation raise
+funds for GNU development.''
+@end enumerate
+@c @sp 2
+@c Cover art by ?????.
+@end titlepage
+
+@iftex
+@headings off
+@evenheading @thispage@ @ @ @strong{@value{TITLE}} @| @|
+@oddheading  @| @| @strong{@thischapter}@ @ @ @thispage
+@end iftex
+
+@ifinfo
+@node Top, Preface, (dir), (dir)
+@top General Introduction
+@comment node-name, next,          previous, up
+
+This file documents the networking features in GNU Awk (@command{gawk})
+version 3.1 and later.
+@end ifinfo
+
+@menu
+* Preface::                          About this document.
+* Introduction::                     About networkiing.
+* Using Networking::                 Some examples.
+* Some Applications and Techniques:: More extended examples.
+* Links::                            Where to find the stuff mentioned in this
+                                     document.
+* GNU Free Documentation License::   The license for this document.
+* Index::                            The index.
+
+@detailmenu
+* Stream Communications::          Sending data streams.
+* Datagram Communications::        Sending self-contained messages.
+* The TCP/IP Protocols::           How these models work in the Internet.
+* Basic Protocols::                The basic protocols.
+* Ports::                          The idea behind ports.
+* Making Connections::             Making TCP/IP connections.
+* Gawk Special Files::             How to do @command{gawk} networking.
+* Special File Fields::            The fields in the special file name.
+* Comparing Protocols::            Differences between the protocols.
+* File /inet/tcp::                 The TCP special file.
+* File /inet/udp::                 The UDB special file.
+* File /inet/raw::                 The RAW special file.
+* TCP Connecting::                 Making a TCP connection.
+* Troubleshooting::                Troubleshooting TCP/IP connections.
+* Interacting::                    Interacting with a service.
+* Setting Up::                     Setting up a service.
+* Email::                          Reading email.
+* Web page::                       Reading a Web page.
+* Primitive Service::              A primitive Web service.
+* Interacting Service::            A Web service with interaction.
+* CGI Lib::                        A simple CGI library.
+* Simple Server::                  A simple Web server.
+* Caveats::                        Network programming caveats.
+* Challenges::                     Where to go from here.
+* PANIC::                          An Emergency Web Server.
+* GETURL::                         Retrieving Web Pages.
+* REMCONF::                        Remote Configuration Of Embedded Systems.
+* URLCHK::                         Look For Changed Web Pages.
+* WEBGRAB::                        Extract Links From A Page.
+* STATIST::                        Graphing A Statistical Distribution.
+* MAZE::                           Walking Through A Maze In Virtual Reality.
+* MOBAGWHO::                       A Simple Mobile Agent.
+* STOXPRED::                       Stock Market Prediction As A Service.
+* PROTBASE::                       Searching Through A Protein Database.
+@end detailmenu
+@end menu
+
+@contents
+
+@node Preface, Introduction, Top, Top
+@unnumbered Preface
+
+In May of 1997, J@"urgen Kahrs felt the need for network access
+from @command{awk}, and, with a little help from me, set about adding
+features to do this for @command{gawk}.  At that time, he
+wrote the bulk of this @value{DOCUMENT}.
+
+The code and documentation were added to the @command{gawk} 3.1 development
+tree, and languished somewhat until I could finally get
+down to some serious work on that version of @command{gawk}.
+This finally happened in the middle of 2000.
+
+Meantime, J@"urgen wrote an article about the Internet special
+files and @samp{|&} operator for @cite{Linux Journal}, and made a
+networking patch for the production versions of @command{gawk}
+available from his home page.
+In August of 2000 (for @command{gawk} 3.0.6), this patch
+also made it to the main GNU @command{ftp} distribution site.
+
+For release with @command{gawk}, I edited J@"urgen's prose
+for English grammar and style, as he is not a native English
+speaker.  I also
+rearranged the material somewhat for what I felt was a better order of
+presentation, and (re)wrote some of the introductory material.
+
+The majority of this document and the code are his work, and the
+high quality and interesting ideas speak for themselves.  It is my
+hope that these features will be of significant value to the @command{awk}
+community.
+
+@sp 1
+@noindent
+Arnold Robbins @*
+Nof Ayalon, ISRAEL @*
+March, 2001
+
+@node Introduction, Using Networking, Preface, Top
+@chapter Networking Concepts
+
+This @value{CHAPTER} provides a (necessarily) brief intoduction to
+computer networking concepts.  For many applications of @command{gawk}
+to TCP/IP networking, we hope that this is enough.  For more
+advanced tasks, you will need deeper background, and it may be necessary
+to switch to lower-level programming in C or C++.
+
+There are two real-life models for the way computers send messages
+to each other over a network.  While the analogies are not perfect,
+they are close enough to convey the major concepts.
+These two models are the phone system (reliable byte-stream communications),
+and the postal system (best-effort datagrams).
+
+@menu
+* Stream Communications::       Sending data streams.
+* Datagram Communications::     Sending self-contained messages.
+* The TCP/IP Protocols::        How these models work in the Internet.
+* Making Connections::          Making TCP/IP connections.
+@end menu
+
+@node Stream Communications, Datagram Communications, Introduction, Introduction
+@section Reliable Byte-streams (Phone Calls)
+
+When you make a phone call, the following steps occur:
+
+@enumerate
+@item
+You dial a number.
+
+@item
+The phone system connects to the called party, telling
+them there is an incoming call. (Their phone rings.)
+
+@item
+The other party answers the call, or, in the case of a
+computer network, refuses to answer the call.
+
+@item
+Assuming the other party answers, the connection between
+you is now a @dfn{duplex} (two-way), @dfn{reliable} (no data lost),
+sequenced (data comes out in the order sent) data stream.
+
+@item
+You and your friend may now talk freely, with the phone system
+moving the data (your voices) from one end to the other.
+From your point of view, you have a direct end-to-end
+connection with the person on the other end.
+@end enumerate
+
+The same steps occur in a duplex reliable computer networking connection.
+There is considerably more overhead in setting up the communications,
+but once it's done, data moves in both directions, reliably, in sequence.
+
+@node Datagram Communications, The TCP/IP Protocols, Stream Communications, Introduction
+@section Best-effort Datagrams (Mailed Letters)
+
+Suppose you mail three different documents to your office on the
+other side of the country on two different days.  Doing so
+entails the following.
+
+@enumerate
+@item
+Each document travels in its own envelope.
+
+@item
+Each envelope contains both the sender and the
+recipient address.
+
+@item
+Each envelope may travel a different route to its destination.
+
+@item
+The envelopes may arrive in a different order from the one
+in which they were sent.
+
+@item
+One or more may get lost in the mail.
+(Although, fortunately, this does not occur very often.)
+
+@item
+In a computer network, one or more @dfn{packets}
+may also arrive multiple times.  (This doesn't happen
+with the postal system!)
+
+@end enumerate
+
+The important characteristics of datagram communications, like
+those of the postal system are thus:
+
+@itemize @bullet
+@item
+Delivery is ``best effort;'' the data may never get there.
+
+@item
+Each message is self-contained, including the source and
+destination addresses.
+
+@item
+Delivery is @emph{not} sequenced; packets may arrive out
+of order, and/or multiple times.
+
+@item
+Unlike the phone system, overhead is considerably lower.
+It is not necessary to set up the call first.
+@end itemize
+
+The price the user pays for the lower overhead of datagram communications
+is exactly the lower reliability; it is often necessary for user-level
+protocols that use datagram communications to add their own reliabilty
+features on top of the basic communications.
+
+@node The TCP/IP Protocols, Making Connections, Datagram Communications, Introduction
+@section The Internet Protocols
+
+The Internet Protocol Suite (usually referred as just TCP/IP)@footnote{
+It should be noted that although the Internet seems to have conquered the
+world, there are other networking protocol suites in existence and in use.}
+consists of a number of different protocols at different levels or ``layers.''
+For our purposes, three protocols provide the fundamental communications
+mechanisms.  All other defined protocols are referred to as user-level
+protocols (e.g., HTTP, used later in this @value{DOCUMENT}).
+
+@menu
+* Basic Protocols::             The basic protocols.
+* Ports::                       The idea behind ports.
+@end menu
+
+@node Basic Protocols, Ports, The TCP/IP Protocols, The TCP/IP Protocols
+@subsection The Basic Internet Protocols
+
+@table @asis
+@item IP
+The Internet Protocol.  This protocol is almost never used directly by
+applications.  It provides the basic packet delivery and routing infrastructure
+of the Internet.  Much like the phone company's switching centers or the Post
+Office's trucks, it is not of much day-to-day interest to the regular user
+(or programmer).
+It happens to be a best effort datagram protocol.
+
+@item UDP
+The User Datagram Protocol.  This is a best effort datagram protocol.
+It provides a small amount of extra reliability over IP, and adds
+the notion of @dfn{ports}, described in @ref{Ports, ,TCP and UDP Ports}.
+
+@item TCP
+The Transmission Control Protocol.  This is a duplex, reliable, sequenced
+byte-stream protocol, again layered on top of IP, and also providing the
+notion of ports.  This is the protocol that you will most likely use
+when using @command{gawk} for network programming.
+@end table
+
+All other user-level protocols use either TCP or UDP to do their basic
+communications.  Examples are SMTP (Simple Mail Transfer Protocol),
+FTP (File Transfer Protocol) and HTTP (HyperText Transfer Protocol).
+@cindex SMTP
+@cindex FTP
+@cindex HTTP
+
+@node Ports, , Basic Protocols, The TCP/IP Protocols
+@subsection TCP and UDP Ports
+
+In the postal system, the address on an envelope indicates a physical
+location, such as a residence or office building.  But there may be
+more than one person at the location; thus you have to further quantify
+the recipient by putting a person or company name on the envelope.
+
+In the phone system, one phone number may represent an entire company,
+in which case you need a person's extension number in order to
+reach that individual directly.  Or, when you call a home, you have to
+say, ``May I please speak to ...'' before talking to the person directly.
+
+IP networking provides the concept of addressing.  An IP address represents
+a particular computer, but no more.  In order to reach the mail service
+on a system, or the FTP or WWW service on a system, you have to have some
+way to further specify which service you want.  In the Internet Protocol suite,
+this is done with @dfn{port numbers}, which represent the services, much
+like an extension number used with a phone number.
+
+Port numbers are 16-bit integers.  Unix and Unix-like systems reserve ports
+below 1024 for ``well known'' services, such as SMTP, FTP, and HTTP.
+Numbers above 1024 may be used by any application, although there is no
+promise made that a particular port number is always available.
+
+@node Making Connections, , The TCP/IP Protocols, Introduction
+@section Making TCP/IP Connections (And Some Terminology)
+
+Two terms come up repeatedly when discussing networking:
+@dfn{client} and @dfn{server}.  For now, we'll discuss these terms
+at the @dfn{connection level}, when first establishing connections
+between two processes on different systems over a network.
+(Once the connection is established, the higher level, or
+@dfn{application level} protocols,
+such as HTTP or FTP, determine who is the client and who is the
+server.  Often, it turns out that the client and server are the
+same in both roles.)
+
+@cindex server
+The @dfn{server} is the system providing the service, such as the
+web server or email server.  It is the @dfn{host} (system) which
+is @emph{connected to} in a transaction.
+For this to work though, the server must be expecting connections.
+Much as there has to be someone at the office building to answer
+the phone@footnote{In the days before voice mail systems!}, the
+server process (usually) has to be started first and waiting
+for a connection.
+
+@cindex client
+The @dfn{client} is the system requesting the service.
+It is the system @emph{initiating the connection} in a transaction.
+(Just as when you pick up the phone to call an office or store.)
+
+In the TCP/IP framework, each end of a connection is represented by a pair
+of (@var{address}, @var{port}) pairs.  For the duration of the connection,
+the ports in use at each end are unique, and cannot be used simultaneously
+by other processes on the same system.  (Only after closing a connection
+can a new one be built up on the same port. This is contrary to the usual
+behavior of fully developed web servers which have to avoid situations
+in which they are not reachable. We have to pay this price in order to
+enjoy the benefits of a simple communication paradigm in @command{gawk}.)
+
+@cindex blocking
+@cindex synchronous communications
+Furthermore, once the connection is established, communications
+are @dfn{synchronous}.  I.e., each end waits on the other to finish
+transmitting, before replying. This is much like two people in a phone
+conversation.  While both could talk simultaneously, doing so usually
+doesn't work too well.
+
+In the case of TCP, the synchronicity is enforced by the protocol when
+sending data.  Data writes @dfn{block} until the data have been received on the
+other end.  For both TCP and UDP, data reads block until there is incoming
+data waiting to be read.  This is summarized in the following table,
+where an ``X'' indicates that the given action blocks.
+
+@ifnottex
+@multitable {Protocol}  {Reads}  {Writes}
+@item TCP @tab X @tab X
+@item UDP @tab X @tab
+@item RAW @tab X @tab
+@end multitable
+@end ifnottex
+@tex
+\centerline{
+\vbox{\bigskip % space above the table (about 1 linespace)
+% Because we have vertical rules, we can't let TeX insert interline space
+% in its usual way.
+\offinterlineskip
+\halign{\hfil\strut# &\vrule #& \hfil#\hfil& \hfil#\hfil\cr
+Protocol&&\quad Reads\quad &Writes\cr
+\noalign{\hrule}
+\omit&height 2pt\cr
+\noalign{\hrule height0pt}% without this the rule does not extend; why?
+TCP&&X&X\cr
+UDP&&X&\cr
+RAW&&X&\cr
+}}}
+@end tex
+
+@node Using Networking, Some Applications and Techniques, Introduction, Top
+@comment node-name, next, previous, up
+@chapter Networking With @command{gawk}
+
+@cindex network
+The @command{awk} programming language was originally developed as a
+pattern-matching language for writing short programs to perform
+data manipulation tasks.
+@command{awk}'s strength is the manipulation of textual data
+that is stored in files.
+It was never meant to be used for networking purposes.
+To exploit its features in a
+networking context, it's necessary to use an access mode for network connections
+that resembles the access of files as closely as possible.
+
+@cindex Perl
+@cindex Python
+@cindex Tcl/Tk
+@command{awk} is also meant to be a prototyping language. It is used
+to demonstrate feasibility and to play with features and user interfaces.
+This can be done with file-like handling of network
+connections.
+@command{gawk} trades the lack
+of many of the advanced features of the TCP/IP family of protocols
+for the convenience of simple connection handling.
+The advanced
+features are available when programming in C or Perl. In fact, the
+network programming
+in this @value{CHAPTER}
+is very similar to what is described in books like
+@cite{Internet Programming with Python},
+@cite{Advanced Perl Programming},
+or
+@cite{Web Client Programming with Perl}.
+But it's done here without first having to learn object-oriented ideology, underlying
+languages such as Tcl/Tk, Perl, Python, or all of the libraries necessary to
+extend these languages before they are ready for the Internet.
+
+This @value{CHAPTER} demonstrates how to use the TCP protocol. The
+other protocols are much less important for most users (UDP) or even
+untractable (RAW).
+
+@menu
+* Gawk Special Files::          How to do @command{gawk} networking.
+* TCP Connecting::              Making a TCP connection.
+* Troubleshooting::             Troubleshooting TCP/IP connections.
+* Interacting::                 Interacting with a service.
+* Setting Up::                  Setting up a service.
+* Email::                       Reading email.
+* Web page::                    Reading a Web page.
+* Primitive Service::           A primitive Web service.
+* Interacting Service::         A Web service with interaction.
+* Simple Server::               A simple Web server.
+* Caveats::                     Network programming caveats.
+* Challenges::                  Where to go from here.
+@end menu
+
+@node Gawk Special Files, TCP Connecting, Using Networking, Using Networking
+@comment node-name,      next,  previous, up
+@section @command{gawk} Networking Mechanisms
+@cindex network
+
+The @samp{|&} operator introduced in @command{gawk} 3.1 for use in
+communicating with a @dfn{co-process} is described in
+@ref{Two-way I/O, ,Two-way Communications With Another Process, gawk, GAWK: Effective AWK Programming}.
+It shows how to do two-way I/O to a
+separate process, sending it data with @code{print} or @code{printf} and
+reading data with @code{getline}.  If you haven't read it already, you should
+detour there to do so.
+
+@command{gawk} transparently extends the two-way I/O mechanism to simple networking through
+the use of special @value{FN}s.  When a ``co-process'' is started that matches
+the special files we are about to describe, @command{gawk} creates the appropriate network
+connection, and then two-way I/O proceeds as usual.
+
+At the C, C++ (and basic Perl) level, networking is accomplished
+via @dfn{sockets}, an Application Programming Interface (API) originally
+developed at the University of California at Berkeley that is now used
+almost universally for TCP/IP networking.
+Socket level programming, while fairly straightforward, requires paying
+attention to a number of details, as well as using binary data.  It is not
+well-suited for use from a high-level language like @command{awk}.
+The special files provided in @command{gawk} hide the details from
+the programmer, making things much simpler and easier to use.
+@c Who sez we can't toot our own horn occasionally?
+
+The special @value{FN} for network access is made up of several fields, all
+of them mandatory, none of them optional:
+
+@example
+/inet/@var{protocol}/@var{localport}/@var{hostname}/@var{remoteport}
+@end example
+
+The @file{/inet/} field is, of course, constant when accessing the network.
+The @var{localport} and @var{remoteport} fields do not have a meaning
+when used with @file{/inet/raw} because ``ports'' only apply to
+TCP and UDP. So, when using @file{/inet/raw}, the port fields always have
+to be @samp{0}.
+
+@menu
+* Special File Fields::         The fields in the special file name.
+* Comparing Protocols::         Differences between the protocols.
+@end menu
+
+@node Special File Fields, Comparing Protocols, Gawk Special Files, Gawk Special Files
+@subsection The Fields of the Special @value{FFN}
+This @value{SECTION} explains the meaning of all the other fields,
+as well as the range of values and the defaults.
+All of the fields are mandatory.  To let the system pick a value,
+or if the field doesn't apply to the protocol, specify it as @samp{0}.
+
+@table @var
+@item protocol
+Determines which member of the TCP/IP
+family of protocols is selected to transport the data across the
+network. There are three possible values (always written in lowercase):
+@samp{tcp}, @samp{udp}, and @samp{raw}. The exact meaning of each is
+explained later in this @value{SECTION}.
+
+@item localport
+Determines which port on the local
+machine is used to communicate across the network. It has no meaning
+with @file{/inet/raw} and must therefore be @samp{0}.  Application level clients
+usually use @samp{0} to indicate they do not care which local port is
+used---instead they specify a remote port to connect to. It is vital for
+application level servers to use a number different from @samp{0} here
+because their service has to be available at a specific publicly-known
+port number. It is possible to use a name from @file{/etc/services} here.
+
+@item hostname
+Determines which remote host is to
+be at the other end of the connection. Application level servers must fill
+this field with a @samp{0} to indicate their being open for all other hosts
+to connect to them and enforce connection level server behavior this way.
+It is not possible for an application level server to restrict its
+availability to one remote host by entering a host name here.
+Application level clients must enter a name different from @samp{0}.
+The name can be either symbolic
+(e.g., @samp{jpl-devvax.jpl.nasa.gov}) or numeric (e.g., @samp{128.149.1.143}).
+
+@item remoteport
+Determines which port on the remote
+machine is used to communicate across the network. It has no meaning
+with @file{/inet/raw} and must therefore be 0.
+For @file{/inet/tcp} and @file{/inet/udp},
+application level clients @emph{must} use a number
+other than @samp{0} to indicate which port on the remote machine
+they want to connect to. Application level servers must not fill this field with
+a @samp{0}. Instead they specify a local port for clients to connect to.
+It is possible to use a name from @file{/etc/services} here.
+@end table
+
+Experts in network programming will notice that the usual
+client/server asymmetry found at the level of the socket API is not visible
+here. This is for the sake of simplicity of the high-level concept. If this
+asymmetry is necessary for your application,
+use another language.
+For @command{gawk}, it is
+more important to enable users to write a client program with a minimum
+of code. What happens when first accessing a network connection is seen
+in the following pseudo-code:
+
+@smallexample
+if ((name of remote host given) && (other side accepts connection)) @{
+  rendez-vous successful; transmit with getline or print
+@} else @{
+  if ((other side did not accept) && (localport == 0))
+    exit unsuccessful
+  if (TCP) @{
+    set up a server accepting connections
+    this means waiting for the client on the other side to connect
+  @} else
+    ready
+@}
+@end smallexample
+
+The exact behavior of this algorithm depends on the values of the
+fields of the special @value{FN}. When in doubt, the following table
+gives you the combinations of values and their meaning. If this
+table is too complicated, focus on the three lines printed in
+@strong{bold}. All the examples in
+@ref{Using Networking, ,Networking With @command{gawk}},
+use only the
+patterns printed in bold letters.
+
+@multitable {12345678901234} {123456} {123456} {1234567} {1234567890123456789012345}
+@item @sc{protocol} @tab @sc{local port} @tab @sc{host name}
+@tab @sc{remote port} @tab @sc{Resulting connection level behavior}
+@item @strong{tcp} @tab @strong{0} @tab @strong{x} @tab @strong{x} @tab
+      @strong{Dedicated client, fails if immediately connecting to a
+              server on the other side fails}
+@item udp      @tab 0 @tab x @tab x @tab Dedicated client
+@item raw      @tab 0 @tab x @tab 0 @tab Dedicated client, works only as @code{root}
+@item @strong{tcp, udp} @tab @strong{x} @tab @strong{x} @tab @strong{x} @tab
+      @strong{Client, switches to dedicated server if necessary}
+@item @strong{tcp, udp} @tab @strong{x} @tab @strong{0} @tab @strong{0} @tab
+      @strong{Dedicated server}
+@item raw      @tab 0 @tab 0 @tab 0 @tab Dedicated server, works only as @code{root}
+@item tcp, udp, raw @tab x @tab x @tab 0 @tab Invalid
+@item tcp, udp, raw @tab 0 @tab 0 @tab x @tab Invalid
+@item tcp, udp, raw @tab x @tab 0 @tab x @tab Invalid
+@item tcp, udp @tab 0 @tab 0 @tab 0 @tab Invalid
+@item tcp, udp @tab 0 @tab x @tab 0 @tab Invalid
+@item raw      @tab x @tab 0 @tab 0 @tab Invalid
+@item raw      @tab 0 @tab x @tab x @tab Invalid
+@item raw      @tab x @tab x @tab x @tab Invalid
+@end multitable
+
+In general, TCP is the preferred mechanism to use.  It is the simplest
+protocol to understand and to use.  Use the others only if circumstances
+demand low-overhead.
+
+@node Comparing Protocols, , Special File Fields, Gawk Special Files
+@subsection Comparing Protocols
+
+This @value{SECTION} develops a pair of programs (sender and receiver)
+that do nothing but send a timestamp from one machine to another. The
+sender and the receiver are implemented with each of the three protocols
+available and demonstrate the differences between them.
+
+@menu
+* File /inet/tcp::              The TCP special file.
+* File /inet/udp::              The UDB special file.
+* File /inet/raw::              The RAW special file.
+@end menu
+
+@node File /inet/tcp, File /inet/udp, Comparing Protocols, Comparing Protocols
+@subsubsection @file{/inet/tcp}
+@cindex @file{/inet/tcp} special files
+@cindex TCP
+Once again, always use TCP.
+(Use UDP when low-overhead is a necessity, and use RAW for
+network experimentation.)
+The first example is the sender
+program:
+
+@example
+# Server
+BEGIN @{
+  print strftime() |& "/inet/tcp/8888/0/0"
+  close("/inet/tcp/8888/0/0")
+@}
+@end example
+
+The receiver is very simple:
+
+@example
+# Client
+BEGIN @{
+  "/inet/tcp/0/localhost/8888" |& getline
+  print $0
+  close("/inet/tcp/0/localhost/8888")
+@}
+@end example
+
+TCP guarantees that the bytes arrive at the receiving end in exactly
+the same order that they were sent. No byte is lost
+(except for broken connections), doubled, or out of order. Some
+overhead is necessary to accomplish this, but this is the price to pay for
+a reliable service.
+It does matter which side starts first. The sender/server has to be started
+first, and it waits for the receiver to read a line.
+
+@node File /inet/udp, File /inet/raw, File /inet/tcp, Comparing Protocols
+@subsubsection @file{/inet/udp}
+@cindex @file{/inet/udp} special files
+@cindex UDP
+The server and client programs that use UDP are almost identical to their TCP counterparts;
+only the @var{protocol} has changed. As before, it does matter which side
+starts first. The receiving side blocks and waits for the sender.
+In this case, the receiver/client has to be started first:
+
+@page
+@example
+# Server
+BEGIN @{
+  print strftime() |& "/inet/udp/8888/0/0"
+  close("/inet/udp/8888/0/0")
+@}
+@end example
+
+The receiver is almost identical to the TCP receiver:
+
+@example
+# Client
+BEGIN @{
+  "/inet/udp/0/localhost/8888" |& getline
+  print $0
+  close("/inet/udp/0/localhost/8888")
+@}
+@end example
+
+UDP cannot guarantee that the datagrams at the receiving end will arrive in exactly
+the same order they were sent. Some datagrams could be
+lost, some doubled, and some out of order. But no overhead is necessary to
+accomplish this. This unreliable behavior is good enough for tasks
+such as data acquisition, logging, and even stateless services like NFS.
+
+@node File /inet/raw, , File /inet/udp, Comparing Protocols
+@subsubsection @file{/inet/raw}
+@cindex @file{/inet/raw} special files
+@cindex RAW
+
+This is an IP-level protocol. Only @code{root} is allowed to access this
+special file. It is meant to be the basis for implementing
+and experimenting with transport level protocols.@footnote{This special file
+is reserved, but not otherwise currently implemented.}
+In the most general case,
+the sender has to supply the encapsulating header bytes in front of the
+packet and the receiver has to strip the additional bytes from the message.
+
+@cindex dark corner
+RAW receivers cannot receive packets sent with TCP or UDP because the
+operating system does not deliver the packets to a RAW receiver. The
+operating system knows about some of the protocols on top of IP
+and decides on its own which packet to deliver to which process.
+@value{DARKCORNER}
+Therefore, the UDP receiver must be used for receiving UDP
+datagrams sent with the RAW sender. This is a dark corner, not only of
+@command{gawk}, but also of TCP/IP.
+
+@cindex SPAK utility
+For extended experimentation with protocols, look into
+the approach implemented in a tool called SPAK.
+This tool reflects the hierarchical layering of protocols (encapsulation)
+in the way data streams are piped out of one program into the next one.
+It shows which protocol is based on which other (lower-level) protocol
+by looking at the command-line ordering of the program calls.
+Cleverly thought out, SPAK is much better than @command{gawk}'s
+@file{/inet} for learning the meaning of each and every bit in the
+protocol headers.
+
+The next example uses the RAW protocol to emulate
+the behavior of UDP. The sender program is the same as above, but with some
+additional bytes that fill the places of the UDP fields:
+
+@example
+@group
+BEGIN @{
+  Message = "Hello world\n"
+  SourcePort = 0
+  DestinationPort = 8888
+  MessageLength = length(Message)+8
+  RawService = "/inet/raw/0/localhost/0"
+  printf("%c%c%c%c%c%c%c%c%s",
+      SourcePort/256, SourcePort%256,
+      DestinationPort/256, DestinationPort%256,
+      MessageLength/256, MessageLength%256,
+      0, 0, Message) |& RawService
+  fflush(RawService)
+  close(RawService)
+@}
+@end group
+@end example
+
+Since this program tries
+to emulate the behavior of UDP, it checks if
+the RAW sender is understood by the UDP receiver but not if the RAW receiver
+can understand the UDP sender. In a real network, the
+RAW receiver is hardly
+of any use because it gets every IP packet that
+comes across the network. There are usually so many packets that
+@command{gawk} would be too slow for processing them.
+Only on a network with little
+traffic can the IP-level receiver program be tested. Programs for analyzing
+IP traffic on modem or ISDN channels should be possible.
+
+Port numbers do not have a meaning when using @file{/inet/raw}. Their fields
+have to be @samp{0}. Only TCP and UDP use ports. Receiving data from
+@file{/inet/raw} is difficult, not only because of processing speed but also
+because data is usually binary and not restricted to ASCII. This
+implies that line separation with @code{RS} does not work as usual.
+
+@node TCP Connecting, Troubleshooting, Gawk Special Files, Using Networking
+@section Establishing a TCP Connection
+
+Let's observe a network connection at work. Type in the following program
+and watch the output. Within a second, it connects via TCP (@file{/inet/tcp})
+to the machine it is running on (@samp{localhost}), and asks the service
+@samp{daytime} on the machine what time it is:
+
+@cindex @code{|&} I/O operator
+@cindex @code{getline} built-in function
+@example
+BEGIN @{
+  "/inet/tcp/0/localhost/daytime" |& getline
+  print $0
+  close("/inet/tcp/0/localhost/daytime")
+@}
+@end example
+
+Even experienced @command{awk} users will find the second line strange in two
+respects:
+
+@itemize @bullet
+@item
+A special file is used as a shell command that pipes its output
+into @code{getline}. One would rather expect to see the special file
+being read like any other file (@samp{getline <
+"/inet/tcp/0/localhost/daytime")}.
+
+@item
+The operator @samp{|&} has not been part of any @command{awk}
+implementation (until now).
+It is actually the only extension of the @command{awk}
+language needed (apart from the special files) to introduce network access.
+@end itemize
+
+The @samp{|&} operator was introduced in @command{gawk} 3.1 in order to
+overcome the crucial restriction that access to files and pipes in
+@command{awk} is always unidirectional. It was formerly impossible to use
+both access modes on the same file or pipe. Instead of changing the whole
+concept of file access, the @samp{|&} operator
+behaves exactly like the usual pipe operator except for two additions:
+
+@itemize @bullet
+@item
+Normal shell commands connected to their @command{gawk} program with a @samp{|&}
+pipe can be accessed bidirectionally. The @samp{|&} turns out to be a quite
+general, useful, and natural extension of @command{awk}.
+
+@item
+Pipes that consist of a special @value{FN} for network connections are not
+executed as shell commands. Instead, they can be read and written to, just
+like a full-duplex network connection.
+@end itemize
+
+In the earlier example, the @samp{|&} operator tells @code{getline}
+to read a line from the special file @file{/inet/tcp/0/localhost/daytime}.
+We could also have printed a line into the special file. But instead we just
+read a line with the time, printed it, and closed the connection.
+(While we could just let @command{gawk} close the connection by finishing
+the program, in this @value{DOCUMENT}
+we are pedantic, and always explicitly close the connections.)
+
+@node Troubleshooting, Interacting, TCP Connecting, Using Networking
+@section Troubleshooting Connection Problems
+It may well be that for some reason the above program does not run on your
+machine. When looking at possible reasons for this, you will learn much
+about typical problems that arise in network programming. First of all,
+your implementation of @command{gawk} may not support network access
+because it is
+a pre-3.1 version or you do not have a network interface in your machine.
+Perhaps your machine uses some other protocol
+like DECnet or Novell's IPX. For the rest of this @value{CHAPTER},
+we will assume
+you work on a Unix machine that supports TCP/IP. If the above program does
+not run on such a machine, it may help to replace the name
+@samp{localhost} with the name of your machine or its IP address. If it
+does, you could replace @samp{localhost} with the name of another machine
+in your vicinity. This way, the program connects to another machine.
+Now you should see the date and time being printed by the program.
+Otherwise your machine may not support the @samp{daytime} service.
+Try changing the service to @samp{chargen} or @samp{ftp}. This way, the program
+connects to other services that should give you some response. If you are
+curious, you should have a look at your file @file{/etc/services}. It could
+look like this:
+
+@ignore
+@multitable {1234567890123} {1234567890123} {123456789012345678901234567890123456789012}
+@item Service @strong{name} @tab Service @strong{number}
+@item echo @tab 7/tcp @tab echo sends back each line it receivces
+@item echo @tab 7/udp @tab echo is good for testing purposes
+@item discard @tab 9/tcp @tab discard behaves like @file{/dev/null}
+@item discard @tab 9/udp @tab discard just throws away each line
+@item daytime @tab 13/tcp @tab daytime sends date & time once per connection
+@item daytime @tab 13/udp
+@item chargen @tab 19/tcp @tab chargen infinitely produces character sets
+@item chargen @tab 19/udp @tab chargen is good for testing purposes
+@item ftp @tab 21/tcp @tab ftp is the usual file transfer protocol
+@item telnet @tab 23/tcp @tab telnet is the usual login facility
+@item smtp @tab 25/tcp @tab smtp is the Simple Mail Transfer Protocol
+@item finger @tab 79/tcp @tab finger tells you who is logged in
+@item www @tab 80/tcp @tab www is the HyperText Transfer Protocol
+@item pop2 @tab 109/tcp @tab pop2 is an older version of pop3
+@item pop2 @tab 109/udp
+@item pop3 @tab 110/tcp @tab pop3 is the Post Office Protocol
+@item pop3 @tab 110/udp @tab pop3 is used for receiving email
+@item nntp @tab 119/tcp @tab nntp is the USENET News Transfer Protocol
+@item irc @tab 194/tcp @tab irc is the Internet Relay Chat
+@item irc @tab 194/udp
+@end multitable
+@end ignore
+
+@smallexample
+# /etc/services:
+#
+# Network services, Internet style
+#
+# Name     Number/Protcol  Alternate name # Comments
+
+echo        7/tcp
+echo        7/udp
+discard     9/tcp         sink null
+discard     9/udp         sink null
+daytime     13/tcp
+daytime     13/udp
+chargen     19/tcp        ttytst source
+chargen     19/udp        ttytst source
+ftp         21/tcp
+telnet      23/tcp
+smtp        25/tcp        mail
+finger      79/tcp
+www         80/tcp        http      # WorldWideWeb HTTP
+www         80/udp        # HyperText Transfer Protocol
+pop-2       109/tcp       postoffice    # POP version 2
+pop-2       109/udp
+pop-3       110/tcp       # POP version 3
+pop-3       110/udp
+nntp        119/tcp       readnews untp  # USENET News
+irc         194/tcp       # Internet Relay Chat
+irc         194/udp
+@dots{}
+@end smallexample
+
+@cindex Linux
+@cindex GNU/Linux
+@cindex Microsoft Windows
+Here, you find a list of services that traditional Unix machines usually
+support. If your GNU/Linux machine does not do so, it may be that these
+services are switched off in some startup script. Systems running some
+flavor of Microsoft Windows usually do @emph{not} support such services.
+Nevertheless, it @emph{is} possible to do networking with @command{gawk} on
+Microsoft
+Windows.@footnote{Microsoft prefered to ignore the TCP/IP
+family of protocols until 1995. Then came the rise of the Netscape browser
+as a landmark ``killer application.'' Microsoft added TCP/IP support and
+their own browser to Microsoft Windows 95 at the last minute. They even back-ported
+their TCP/IP implementation to Microsoft Windows for Workgroups 3.11, but it was
+a rather rudimentary and half-hearted implementation. Nevertheless,
+the equivalent of @file{/etc/services} resides under
+@file{c:\windows\services} on Microsoft Windows.}
+The first column of the file gives the name of the service,
+the second a unique number, and the protocol that one can use to connect to
+this service.
+The rest of the line is treated as a comment.
+You see that some services (@samp{echo}) support TCP as
+well as UDP.
+
+@node Interacting, Setting Up, Troubleshooting, Using Networking
+@section Interacting with a Network Service
+
+The next program makes use of the possibility to really interact with a
+network service by printing something into the special file. It asks the
+so-called @command{finger} service if a user of the machine is logged in. When
+testing this program, try to change @samp{localhost} to
+some other machine name in your local network:
+
+@c system if test ! -d eg         ; then mkdir eg         ; fi
+@c system if test ! -d eg/network ; then mkdir eg/network ; fi
+@example
+@c file eg/network/fingerclient.awk
+BEGIN @{
+  NetService = "/inet/tcp/0/localhost/finger"
+  print "@var{name}" |& NetService
+  while ((NetService |& getline) > 0)
+    print $0
+  close(NetService)
+@}
+@c endfile
+@end example
+
+After telling the service on the machine which user to look for,
+the program repeatedly reads lines that come as a reply. When no more
+lines are coming (because the service has closed the connection), the
+program also closes the connection. Try replacing @code{"@var{name}"} with your
+login name (or the name of someone else logged in).  For a list
+of all users currently logged in, replace @var{name} with an empty string
+@code{""}.
+
+@cindex Linux
+@cindex GNU/Linux
+The final @code{close} command could be safely deleted from
+the above script, because the operating system closes any open connection
+by default when a script reaches the end of execution. In order to avoid
+portability problems, it is best to always close connections explicitly.
+With the Linux kernel,
+for example, proper closing results in flushing of buffers. Letting
+the close happen by default may result in discarding buffers.
+
+@ignore
+@c Chuck comments that this seems out of place.  He's right.  I dunno
+@c where to put it though.
+@cindex @command{finger} utility
+@cindex RFC 1288
+In the early days of the Internet (up until about 1992), you could use
+such a program to check if some user in another country was logged in on
+a specific machine.
+RFC 1288@footnote{@uref{http://www.cis.ohio-state.edu/htbin/rfc/rfc1288.html}}
+provides the exact definition of the @command{finger} protocol.
+Every contemporary Unix system also has a command named @command{finger},
+which functions as a client for the protocol of the same name.
+Still today, some people maintain simple information systems
+with this ancient protocol. For example, by typing
+@samp{finger quake@@seismo.unr.edu}
+you get the latest @dfn{Earthquake Bulletin} for the state of Nevada.
+
+@cindex Earthquake Bulletin
+@smallexample
+$ finger quake@@seismo.unr.edu
+
+[@dots{}]
+
+DATE-(UTC)-TIME   LAT    LON      DEP   MAG    COMMENTS
+yy/mm/dd hh:mm:ss   deg.   deg.    km
+
+98/12/14 21:09:22  37.47N 116.30W   0.0 2.3Md  76.4 km   S of WARM SPRINGS, NEVA
+98/12/14 22:05:09  39.69N 120.41W  11.9 2.1Md  53.8 km WNW of RENO, NEVADA      
+98/12/15 14:14:19  38.04N 118.60W   2.0 2.3Md  51.0 km   S of HAWTHORNE, NEVADA 
+98/12/17 01:49:02  36.06N 117.58W  13.9 3.0Md  74.9 km  SE of LONE PINE, CALIFOR
+98/12/17 05:39:26  39.95N 120.87W   6.2 2.6Md 101.6 km WNW of RENO, NEVADA      
+98/12/22 06:07:42  38.68N 119.82W   5.2 2.3Md  50.7 km   S of CARSON CITY, NEVAD
+@end smallexample
+
+@noindent
+This output from @command{finger} contains the time, location, depth,
+magnitude, and a short comment about
+the earthquakes registered in that region during the last 10 days.
+In many places today the use of such services is restricted
+because most networks have firewalls and proxy servers between them
+and the Internet. Most firewalls are programmed to not let
+@command{finger} requests go beyond the local network.
+
+@cindex Coke machine
+Another (ab)use of the @command{finger} protocol are several Coke machines
+that are connected to the Internet. There is a short list of such
+Coke machines.@footnote{@uref{http://ca.yahoo.com/Computers_and_Internet/Internet/Devices_Connected_to_the_Internet/Soda_Machines/}}
+You can access them either from the command-line or with a simple
+@command{gawk} script. They usually tell you about the different
+flavors of Coke and beer available there. If you have an account there,
+you can even order some drink this way.
+@end ignore
+
+When looking at @file{/etc/services} you may have noticed that the
+@samp{daytime} service is also available with @samp{udp}. In the earlier
+example, change @samp{tcp} to @samp{udp},
+and change @samp{finger} to @samp{daytime}.
+After starting the modified program, you see the expected day and time message.
+The program then hangs, because it waits for more lines coming from the
+service. However, they never come. This behavior is a consequence of the
+differences between TCP and UDP. When using UDP, neither party is
+automatically informed about the other closing the connection.
+Continuing to experiment this way reveals many other subtle
+differences between TCP and UDP. To avoid such trouble, one should always
+remember the advice Douglas E.@: Comer and David Stevens give in
+Volume III of their series @cite{Internetworking With TCP}
+(page 14):
+
+@cindex TCP
+@cindex UDP
+@quotation
+When designing client-server applications, beginners are strongly
+advised to use TCP because it provides reliable, connection-oriented
+communication. Programs only use UDP if the application protocol handles
+reliability, the application requires hardware broadcast or multicast,
+or the application cannot tolerate virtual circuit overhead.
+@end quotation
+
+@node Setting Up, Email, Interacting, Using Networking
+@section Setting Up a Service
+The preceding programs behaved as clients that connect to a server somewhere
+on the Internet and request a particular service. Now we set up such a
+service to mimic the behavior of the @samp{daytime} service.
+Such a server does not know in advance who is going to connect to it over
+the network. Therefore we cannot insert a name for the host to connect to
+in our special @value{FN}.
+
+Start the following program in one window. Notice that the service does
+not have the name @samp{daytime}, but the number @samp{8888}.
+From looking at @file{/etc/services}, you know that names like @samp{daytime}
+are just mnemonics for predetermined 16-bit integers.
+Only the system administrator (@code{root}) could enter
+our new service into @file{/etc/services} with an appropriate name.
+Also notice that the service name has to be entered into a different field
+of the special @value{FN} because we are setting up a server, not a client:
+
+@cindex @command{finger} utility
+@cindex server
+@example
+BEGIN @{
+  print strftime() |& "/inet/tcp/8888/0/0"
+  close("/inet/tcp/8888/0/0")
+@}
+@end example
+
+Now open another window on the same machine.
+Copy the client program given as the first example 
+(@pxref{TCP Connecting, ,Establishing a TCP Connection})
+to a new file and edit it, changing the name @samp{daytime} to
+@samp{8888}.  Then start the modified client.  You should get a reply
+like this:
+
+@example
+Sat Sep 27 19:08:16 CEST 1997
+@end example
+
+@noindent
+Both programs explicitly close the connection.
+
+@cindex Microsoft Windows
+@cindex reserved ports
+Now we will intentionally make a mistake to see what happens when the name
+@samp{8888} (the so-called port) is already used by another service.
+Start the server
+program in both windows. The first one works, but the second one
+complains that it could not open the connection. Each port on a single
+machine can only be used by one server program at a time. Now terminate the
+server program and change the name @samp{8888} to @samp{echo}. After restarting it,
+the server program does not run any more and you know why: there already is
+an @samp{echo} service running on your machine. But even if this isn't true,
+you would not get
+your own @samp{echo} server running on a Unix machine,
+because the ports with numbers smaller
+than 1024 (@samp{echo} is at port 7) are reserved for @code{root}.
+On machines running some flavor of Microsoft Windows, there is no restriction
+that reserves ports 1 to 1024 for a privileged user; hence you can start
+an @samp{echo} server there.
+
+Turning this short server program into something really useful is simple.
+Imagine a server that first reads a @value{FN} from the client through the
+network connection, then does something with the file and
+sends a result back to the client. The server-side processing
+could be:
+
+@example
+BEGIN @{
+  NetService = "/inet/tcp/8888/0/0"
+  NetService |& getline
+  CatPipe    = ("cat " $1)    # sets $0 and the fields
+  while ((CatPipe | getline) > 0)
+    print $0 |& NetService
+  close(NetService)
+@}
+@end example
+
+@noindent
+and we would
+have a remote copying facility. Such a server reads the name of a file
+from any client that connects to it and transmits the contents of the
+named file across the net. The server-side processing could also be
+the execution of a command that is transmitted across the network. From this
+example, you can see how simple it is to open up a security hole on your
+machine. If you allow clients to connect to your machine and
+execute arbitrary commands, anyone would be free to do @samp{rm -rf *}.
+
+@node Email, Web page, Setting Up, Using Networking
+@section Reading Email
+@cindex POP
+@cindex SMTP
+@cindex RFC 1939
+@cindex RFC 821
+The distribution of email is usually done by dedicated email servers that
+communicate with your machine using special protocols. To receive email, we
+will use the Post Office Protocol (POP).  Sending can be done with the much
+older Simple Mail Transfer Protocol (SMTP).
+@ignore
+@footnote{RFC 1939 defines POP.
+RFC 821 defines SMTP.  See
+@uref{http://rfc.fh-koeln.de/doc/rfc/html/rfc.html, RFCs in HTML}.}
+@end ignore
+
+When you type in the following program, replace the @var{emailhost} by the
+name of your local email server. Ask your administrator if the server has a
+POP service, and then use its name or number in the program below.
+Now the program is ready to connect to your email server, but it will not
+succeed in retrieving your mail because it does not yet know your login
+name or password. Replace them in the program and it
+shows you the first email the server has in store:
+
+@example
+BEGIN @{
+  POPService  = "/inet/tcp/0/@var{emailhost}/pop3"
+  RS = ORS = "\r\n"
+  print "user @var{name}"            |& POPService
+  POPService                    |& getline
+  print "pass @var{password}"         |& POPService
+  POPService                    |& getline
+  print "retr 1"                |& POPService
+  POPService                    |& getline
+  if ($1 != "+OK") exit
+  print "quit"                  |& POPService
+  RS = "\r\n\\.\r\n"
+  POPService |& getline
+  print $0
+  close(POPService)
+@}
+@end example
+
+@cindex RFC 1939
+The record separators @code{RS} and @code{ORS} are redefined because the
+protocol (POP) requires CR-LF to separate lines. After identifying
+yourself to the email service, the command @samp{retr 1} instructs the
+service to send the first of all your email messages in line. If the service
+replies with something other than @samp{+OK}, the program exits; maybe there
+is no email. Otherwise, the program first announces that it intends to finish
+reading email, and then redefines @code{RS} in order to read the entire
+email as multiline input in one record. From the POP RFC, we know that the body
+of the email always ends with a single line containing a single dot.
+The program looks for this using @samp{RS = "\r\n\\.\r\n"}.
+When it finds this sequence in the mail message, it quits.
+You can invoke this program as often as you like; it does not delete the
+message it reads, but instead leaves it on the server.
+
+@node Web page, Primitive Service, Email, Using Networking
+@section Reading a Web Page
+@cindex HTTP 
+@cindex RFC 2068
+@cindex RFC 2616
+
+Retrieving a web page from a web server is as simple as
+retrieving email from an email server. We only have to use a
+similar, but not identical, protocol and a different port. The name of the
+protocol is HyperText Transfer Protocol (HTTP) and the port number is usually
+80.  As in the preceding @value{SECTION}, ask your administrator about the
+name of your local web server or proxy web server and its port number
+for HTTP requests.
+
+@ignore
+@c Chuck says this stuff isn't necessary
+More detailed information about HTTP can be found at
+the home of the web protocols,@footnote{@uref{http://www.w3.org/pub/WWW/Protocols}}
+including the specification of HTTP in RFC 2068. The protocol specification
+in RFC 2068 is concise and you can get it for free. If you need more
+explanation and you are willing to pay for a book, you might be
+interested in one of these books:
+
+@enumerate
+
+@item
+When we started writing web clients and servers with @command{gawk},
+the only book available with details about HTTP was the one by Paul Hethmon
+called
+@cite{Illustrated Guide to HTTP}.@footnote{@uref{http://www.browsebooks.com/Hethmon/?882}}
+Hethmon not only describes HTTP,
+he also implements a simple web server in C++.
+
+@item
+Since July 2000, O'Reilly offers the book by Clinton Wong called
+@cite{HTTP Pocket Reference}.@footnote{@uref{http://www.oreilly.com/catalog/httppr}}
+It only has 75 pages but its
+focus definitely is HTTP. This pocket reference is not a replacement
+for the RFC, but I wish I had had it back in 1997 when I started writing
+scripts to handle HTTP.
+
+@item
+Another small booklet about HTTP is the one by Toexcell Incorporated Staff,
+ISBN 1-58348-270-9, called
+@cite{Hypertext Transfer Protocol Http 1.0 Specifications}
+
+@end enumerate
+@end ignore
+
+The following program employs a rather crude approach toward retrieving a
+web page. It uses the prehistoric syntax of HTTP 0.9, which almost all
+web servers still support. The most noticeable thing about it is that the
+program directs the request to the local proxy server whose name you insert
+in the special @value{FN} (which in turn calls @samp{www.yahoo.com}):
+
+@example
+BEGIN @{
+  RS = ORS = "\r\n"
+  HttpService = "/inet/tcp/0/@var{proxy}/80"
+  print "GET http://www.yahoo.com"     |& HttpService
+  while ((HttpService |& getline) > 0)
+     print $0
+  close(HttpService)
+@}
+@end example
+
+@cindex RFC 1945
+@cindex HTML
+@cindex Yahoo!
+Again, lines are separated by a redefined @code{RS} and @code{ORS}.
+The @code{GET} request that we send to the server is the only kind of
+HTTP request that existed when the web was created in the early 1990s.
+HTTP calls this @code{GET} request a ``method,'' which tells the
+service to transmit a web page (here the home page of the Yahoo! search
+engine). Version 1.0 added the request methods @code{HEAD} and
+@code{POST}. The current version of HTTP is 1.1,@footnote{Version 1.0 of
+HTTP was defined in RFC 1945.  HTTP 1.1 was initially specified in RFC
+2068. In June 1999, RFC 2068 was made obsolete by RFC 2616. It is an update
+without any substantial changes.} and knows the additional request
+methods @code{OPTIONS}, @code{PUT}, @code{DELETE}, and @code{TRACE}.
+You can fill in any valid web address, and the program prints the
+HTML code of that page to your screen.
+
+Notice the similarity between the responses of the POP and HTTP
+services. First, you get a header that is terminated by an empty line, and
+then you get the body of the page in HTML.  The lines of the headers also
+have the same form as in POP. There is the name of a parameter,
+then a colon, and finally the value of that parameter.
+
+@cindex CGI
+@cindex @file{gif} image format
+@cindex @file{png} image format
+Images (@file{.png} or @file{.gif} files) can also be retrieved this way,
+but then you
+get binary data that should be redirected into a file. Another
+application is calling a CGI (Common Gateway Interface) script on some
+server. CGI scripts are used when the contents of a web page are not
+constant, but generated instantly at the moment you send a request
+for the page. For example, to get a detailed report about the current
+quotes of Motorola stock shares, call a CGI script at Yahoo! with
+the following:
+
+@example
+get = "GET http://quote.yahoo.com/q?s=MOT&d=t"
+print get |& HttpService
+@end example
+
+You can also request weather reports this way.
+@ignore
+@cindex Boutell, Thomas
+A good book to go on with is
+the
+@cite{HTML Source Book}.@footnote{@uref{http://www.utoronto.ca/webdocs/HTMLdocs/NewHTML/book.html}}
+There are also some books on CGI programming
+like @cite{CGI Programming in C & Perl},
+by Thomas Boutell@footnote{@uref{http://cseng.aw.com/bookdetail.qry?ISBN=0-201-42219-0&ptype=0}},
+and @cite{The CGI Book}.@footnote{@uref{http://www.cgibook.com}}
+Another good source is @cite{The CGI Resource Index}}.@footnote{@uref{http://www.cgi-resources.com}}
+@end ignore
+
+@node Primitive Service, Interacting Service, Web page, Using Networking
+@section A Primitive Web Service
+Now we know enough about HTTP to set up a primitive web service that just
+says @code{"Hello, world"} when someone connects to it with a browser.
+Compared
+to the situation in the preceding @value{SECTION}, our program changes the role. It
+tries to behave just like the server we have observed. Since we are setting
+up a server here, we have to insert the port number in the @samp{localport}
+field of the special @value{FN}. The other two fields (@var{hostname} and
+@var{remoteport}) have to contain a @samp{0} because we do not know in
+advance which host will connect to our service.
+
+In the early 1990s, all a server had to do was send an HTML document and
+close the connection. Here, we adhere to the modern syntax of HTTP.
+The steps are as follows:
+
+@enumerate 1
+@item
+Send a status line telling the web browser that everything
+is OK.
+
+@item
+Send a line to tell the browser how many bytes follow in the
+body of the message. This was not necessary earlier because both
+parties knew that the document ended when the connection closed. Nowadays
+it is possible to stay connected after the transmission of one web page.
+This is to avoid the network traffic necessary for repeatedly establishing
+TCP connections for requesting several images. Thus, there is the need to tell
+the receiving party how many bytes will be sent. The header is terminated
+as usual with an empty line.
+
+@item
+Send the @code{"Hello, world"} body
+in HTML.
+The useless @code{while} loop swallows the request of the browser.
+We could actually omit the loop, and on most machines the program would still
+work.
+First, start the following program:
+@end enumerate
+
+@example
+@c file eg/network/hello-serv.awk
+BEGIN @{
+  RS = ORS = "\r\n"
+  HttpService = "/inet/tcp/8080/0/0"
+  Hello = "<HTML><HEAD>" \
+          "<TITLE>A Famous Greeting</TITLE></HEAD>" \
+          "<BODY><H1>Hello, world</H1></BODY></HTML>"
+  Len = length(Hello) + length(ORS)
+  print "HTTP/1.0 200 OK"          |& HttpService
+  print "Content-Length: " Len ORS |& HttpService
+  print Hello                      |& HttpService
+  while ((HttpService |& getline) > 0)
+     continue;
+  close(HttpService)
+@}
+@c endfile
+@end example
+
+Now, on the same machine, start your favorite browser and let it point to
+@uref{http://localhost:8080} (the browser needs to know on which port
+our server is listening for requests). If this does not work, the browser
+probably tries to connect to a proxy server that does not know your machine.
+If so, change the browser's configuration so that the browser does not try to
+use a proxy to connect to your machine.
+
+@node Interacting Service, Simple Server, Primitive Service, Using Networking
+@section A Web Service with Interaction
+@cindex GUI
+@ifinfo
+This node shows how to set up a simple web server.
+The subnode is a library file that we will use with all the examples in
+@ref{Some Applications and Techniques}.
+@end ifinfo
+
+@menu
+* CGI Lib::                     A simple CGI library.
+@end menu
+
+Setting up a web service that allows user interaction is more difficult and
+shows us the limits of network access in @command{gawk}. In this @value{SECTION},
+we develop  a main program (a @code{BEGIN} pattern and its action)
+that will become the core of event-driven execution controlled by a
+graphical user interface (GUI).
+Each HTTP event that the user triggers by some action within the browser
+is received in this central procedure. Parameters and menu choices are
+extracted from this request and an appropriate measure is taken according to
+the user's choice.
+For example:
+
+@cindex HTTP server, core logic
+@example
+BEGIN @{
+  if (MyHost == "") @{
+     "uname -n" | getline MyHost
+     close("uname -n")
+  @}
+  if (MyPort ==  0) MyPort = 8080
+  HttpService = "/inet/tcp/" MyPort "/0/0"
+  MyPrefix    = "http://" MyHost ":" MyPort
+  SetUpServer()
+  while ("awk" != "complex") @{
+    # header lines are terminated this way
+    RS = ORS = "\r\n"
+    Status   = 200          # this means OK
+    Reason   = "OK"
+    Header   = TopHeader
+    Document = TopDoc
+    Footer   = TopFooter
+    if        (GETARG["Method"] == "GET") @{
+        HandleGET()
+    @} else if (GETARG["Method"] == "HEAD") @{
+        # not yet implemented
+    @} else if (GETARG["Method"] != "") @{
+        print "bad method", GETARG["Method"]
+    @}
+    Prompt = Header Document Footer
+    print "HTTP/1.0", Status, Reason       |& HttpService
+    print "Connection: Close"              |& HttpService
+    print "Pragma: no-cache"               |& HttpService
+    len = length(Prompt) + length(ORS)
+    print "Content-length:", len           |& HttpService
+    print ORS Prompt                       |& HttpService
+    # ignore all the header lines
+    while ((HttpService |& getline) > 0)
+        ;
+    # stop talking to this client
+    close(HttpService)
+    # wait for new client request
+    HttpService |& getline          
+    # do some logging
+    print systime(), strftime(), $0
+    # read request parameters
+    CGI_setup($1, $2, $3)
+  @}
+@}
+@end example
+
+This web server presents menu choices in the form of HTML links.
+Therefore, it has to tell the browser the name of the host it is
+residing on. When starting the server, the user may supply the name
+of the host from the command line with @samp{gawk -v MyHost="Rumpelstilzchen"}.
+If the user does not do this, the server looks up the name of the host it is
+running on for later use as a web address in HTML documents. The same
+applies to the port number. These values are inserted later into the
+HTML content of the web pages to refer to the home system.
+
+Each server that is built around this core has to initialize some
+application-dependent variables (such as the default home page) in a procedure
+@code{SetUpServer}, which is called immediately before entering the
+infinite loop of the server. For now, we will write an instance that
+initiates a trivial interaction.  With this home page, the client user
+can click on two possible choices, and receive the current date either
+in human-readable format or in seconds since 1970:
+
+@example
+function SetUpServer() @{
+  TopHeader = "<HTML><HEAD>"
+  TopHeader = TopHeader \
+     "<title>My name is GAWK, GNU AWK</title></HEAD>"
+  TopDoc    = "<BODY><h2>\
+    Do you prefer your date <A HREF=" MyPrefix \
+    "/human>human</A> or \
+    <A HREF=" MyPrefix "/POSIX>POSIXed</A>?</h2>" ORS ORS
+  TopFooter = "</BODY></HTML>"
+@}
+@end example
+
+On the first run through the main loop, the default line terminators are
+set and the default home page is copied to the actual home page. Since this
+is the first run, @code{GETARG["Method"]} is not initialized yet, hence the
+case selection over the method does nothing. Now that the home page is
+initialized, the server can start communicating to a client browser.
+
+@cindex RFC 2068
+@cindex CGI
+It does so by printing the HTTP header into the network connection
+(@samp{print @dots{} |& HttpService}). This command blocks execution of
+the server script until a client connects. If this server
+script is compared with the primitive one we wrote before, you will notice
+two additional lines in the header. The first instructs the browser
+to close the connection after each request. The second tells the
+browser that it should never try to @emph{remember} earlier requests
+that had identical web addresses (no caching). Otherwise, it could happen
+that the browser retrieves the time of day in the previous example just once,
+and later it takes the web page from the cache, always displaying the same
+time of day although time advances each second.
+
+Having supplied the initial home page to the browser with a valid document
+stored in the parameter @code{Prompt}, it closes the connection and waits
+for the next request.  When the request comes, a log line is printed that
+allows us to see which request the server receives. The final step in the
+loop is to call the function @code{CGI_setup}, which reads all the lines
+of the request (coming from the browser), processes them, and stores the
+transmitted parameters in the array @code{PARAM}. The complete
+text of these application-independent functions can be found in
+@ref{CGI Lib, ,A Simple CGI Library}.
+For now, we use a simplified version of @code{CGI_setup}:
+
+@example
+function CGI_setup(   method, uri, version, i) @{
+  delete GETARG;         delete MENU;        delete PARAM
+  GETARG["Method"] = $1
+  GETARG["URI"] = $2
+  GETARG["Version"] = $3
+  i = index($2, "?")
+  # is there a "?" indicating a CGI request?
+@group
+  if (i > 0) @{
+    split(substr($2, 1, i-1), MENU, "[/:]")
+    split(substr($2, i+1), PARAM, "&") 
+    for (i in PARAM) @{ 
+      j = index(PARAM[i], "=")
+      GETARG[substr(PARAM[i], 1, j-1)] = \
+                                  substr(PARAM[i], j+1)
+    @}
+  @} else @{    # there is no "?", no need for splitting PARAMs
+    split($2, MENU, "[/:]")
+  @}
+@end group
+@}
+@end example
+
+At first, the function clears all variables used for
+global storage of request parameters. The rest of the function serves
+the purpose of filling the global parameters with the extracted new values.
+To accomplish this, the name of the requested resource is split into
+parts and stored for later evaluation. If the request contains a @samp{?},
+then the request has CGI variables seamlessly appended to the web address.
+Everything in front of the @samp{?} is split up into menu items, and
+everything behind the @samp{?} is a list of @samp{@var{variable}=@var{value}} pairs
+(separated by @samp{&}) that also need splitting. This way, CGI variables are
+isolated and stored. This procedure lacks recognition of special characters
+that are transmitted in coded form@footnote{As defined in RFC 2068.}. Here, any
+optional request header and body parts are ignored. We do not need
+header parameters and the request body. However, when refining our approach or
+working with the @code{POST} and @code{PUT} methods, reading the header
+and body
+becomes inevitable. Header parameters should then be stored in a global
+array as well as the body.
+
+On each subsequent run through the main loop, one request from a browser is
+received, evaluated, and answered according to the user's choice. This can be
+done by letting the value of the HTTP method guide the main loop into
+execution of the procedure @code{HandleGET}, which evaluates the user's
+choice. In this case, we have only one hierarchical level of menus,
+but in the general case,
+menus are nested.
+The menu choices at each level are
+separated by @samp{/}, just as in @value{FN}s. Notice how simple it is to
+construct menus of arbitrary depth:
+
+@example
+function HandleGET() @{
+  if (       MENU[2] == "human") @{
+    Footer = strftime() TopFooter
+  @} else if (MENU[2] == "POSIX") @{
+    Footer = systime()  TopFooter
+  @}
+@}
+@end example
+
+@cindex CGI
+The disadvantage of this approach is that our server is slow and can
+handle only one request at a time. Its main advantage, however, is that
+the server
+consists of just one @command{gawk} program. No need for installing an
+@command{httpd}, and no need for static separate HTML files, CGI scripts, or
+@code{root} privileges. This is rapid prototyping.
+This program can be started on the same host that runs your browser.
+Then let your browser point to @uref{http://localhost:8080}.
+
+@cindex @file{xbm} image format
+@cindex image format
+@cindex GNUPlot utility
+It is also possible to include images into the HTML pages.
+Most browsers support the not very well-known
+@file{.xbm} format,
+which may contain only
+monochrome pictures but is an ASCII format. Binary images are possible but
+not so easy to handle. Another way of including images is to generate them
+with a tool such as GNUPlot,
+by calling the tool with the @code{system} function or through a pipe.
+
+@node CGI Lib, , Interacting Service, Interacting Service
+@subsection A Simple CGI Library
+@quotation
+@i{HTTP is like being married: you have to be able to handle whatever
+you're given, while being very careful what you send back.}@*
+Phil Smith III,@*
+@uref{http://www.netfunny.com/rhf/jokes/99/Mar/http.html}
+@end quotation
+
+In @ref{Interacting Service, ,A Web Service with Interaction},
+we saw the function @code{CGI_setup} as part of the web server
+``core logic'' framework. The code presented there handles almost
+everything necessary for CGI requests.
+One thing it doesn't do is handle encoded characters in the requests.
+For example, an @samp{&} is encoded as a percent sign followed by
+the hexadecimal value---@samp{%26}.  These encoded values should be
+decoded.
+Following is a simple library to perform these tasks.
+This code is used for all web server examples
+used throughout the rest of this @value{DOCUMENT}.
+If you want to use it for your own web server, store the source code
+into a file named @file{inetlib.awk}. Then you can include
+these functions into your code by placing the following statement
+into your program:
+
+@example
+@@include inetlib.awk
+@end example
+
+@noindent
+on the first line of your script. But beware, this mechanism is
+only possible if you invoke your web server script with @command{igawk}
+instead of the usual @command{awk} or @command{gawk}.
+Here is the code:
+
+@example
+@c file eg/network/coreserv.awk
+# CGI Library and core of a web server
+@c endfile
+@ignore
+@c file eg/network/coreserv.awk
+#
+# Juergen Kahrs, Juergen.Kahrs@@vr-web.de
+# with Arnold Robbins, arnold@@gnu.org
+# September 2000
+
+@c endfile
+@end ignore
+@c file eg/network/coreserv.awk
+# Global arrays
+#   GETARG --- arguments to CGI GET command
+#   MENU   --- menu items (path names)
+#   PARAM  --- parameters of form x=y
+
+# Optional variable MyHost contains host address
+# Optional variable MyPort contains port number
+# Needs TopHeader, TopDoc, TopFooter
+# Sets MyPrefix, HttpService, Status, Reason
+
+BEGIN @{
+  if (MyHost == "") @{
+     "uname -n" | getline MyHost
+     close("uname -n")
+  @}
+  if (MyPort ==  0) MyPort = 8080
+  HttpService = "/inet/tcp/" MyPort "/0/0"
+  MyPrefix    = "http://" MyHost ":" MyPort
+  SetUpServer()
+  while ("awk" != "complex") @{
+    # header lines are terminated this way
+    RS = ORS    = "\r\n"
+    Status      = 200             # this means OK
+    Reason      = "OK"
+    Header      = TopHeader
+    Document    = TopDoc
+    Footer      = TopFooter
+    if        (GETARG["Method"] == "GET") @{
+        HandleGET()
+    @} else if (GETARG["Method"] == "HEAD") @{
+        # not yet implemented
+    @} else if (GETARG["Method"] != "") @{
+        print "bad method", GETARG["Method"]
+    @}
+    Prompt = Header Document Footer
+    print "HTTP/1.0", Status, Reason     |& HttpService
+    print "Connection: Close"            |& HttpService
+    print "Pragma: no-cache"             |& HttpService
+    len = length(Prompt) + length(ORS)
+    print "Content-length:", len         |& HttpService
+    print ORS Prompt                     |& HttpService
+    # ignore all the header lines
+    while ((HttpService |& getline) > 0) 
+        continue
+    # stop talking to this client
+    close(HttpService)
+    # wait for new client request
+    HttpService |& getline
+    # do some logging
+    print systime(), strftime(), $0
+    CGI_setup($1, $2, $3)
+  @}
+@}
+
+function CGI_setup(   method, uri, version, i)
+@{
+    delete GETARG
+    delete MENU
+    delete PARAM
+    GETARG["Method"] = method
+    GETARG["URI"] = uri
+    GETARG["Version"] = version
+
+    i = index(uri, "?")
+    if (i > 0) @{  # is there a "?" indicating a CGI request?
+        split(substr(uri, 1, i-1), MENU, "[/:]")
+        split(substr(uri, i+1), PARAM, "&")
+        for (i in PARAM) @{
+            PARAM[i] = _CGI_decode(PARAM[i])
+            j = index(PARAM[i], "=")
+            GETARG[substr(PARAM[i], 1, j-1)] = \
+	                                 substr(PARAM[i], j+1)
+        @}
+    @} else @{ # there is no "?", no need for splitting PARAMs
+        split(uri, MENU, "[/:]")
+    @}
+    for (i in MENU)     # decode characters in path
+        if (i > 4)      # but not those in host name
+            MENU[i] = _CGI_decode(MENU[i])
+@}
+@c endfile
+@end example
+
+This isolates details in a single function, @code{CGI_setup}.
+Decoding of encoded characters is pushed off to a helper function,
+@code{_CGI_decode}. The use of the leading underscore (@samp{_}) in
+the function name is intended to indicate that it is an ``internal''
+function, although there is nothing to enforce this:
+
+@example
+@c file eg/network/coreserv.awk
+function _CGI_decode(str,   hexdigs, i, pre, code1, code2,
+                            val, result)
+@{
+   hexdigs = "123456789abcdef"
+
+   i = index(str, "%")
+   if (i == 0) # no work to do
+      return str
+
+   do @{
+      pre = substr(str, 1, i-1)   # part before %xx
+      code1 = substr(str, i+1, 1) # first hex digit
+      code2 = substr(str, i+2, 1) # second hex digit
+      str = substr(str, i+3)      # rest of string
+
+      code1 = tolower(code1)
+      code2 = tolower(code2)
+      val = index(hexdigs, code1) * 16 \
+            + index(hexdigs, code2)
+
+      result = result pre sprintf("%c", val)
+      i = index(str, "%")
+   @} while (i != 0)
+   if (length(str) > 0)
+      result = result str
+   return result
+@}
+@c endfile
+@end example
+
+This works by splitting the string apart around an encoded character.
+The two digits are converted to lowercase and looked up in a string
+of hex digits.  Note that @code{0} is not in the string on purpose;
+@code{index} returns zero when it's not found, automatically giving
+the correct value!  Once the hexadecimal value is converted from
+characters in a string into a numerical value, @code{sprintf}
+converts the value back into a real character.
+The following is a simple test harness for the above functions:
+
+@example
+@c file eg/network/testserv.awk
+BEGIN @{
+  CGI_setup("GET",
+  "http://www.gnu.org/cgi-bin/foo?p1=stuff&p2=stuff%26junk" \
+       "&percent=a %25 sign",
+  "1.0")
+  for (i in MENU)
+      printf "MENU[\"%s\"] = %s\n", i, MENU[i]
+  for (i in PARAM)
+      printf "PARAM[\"%s\"] = %s\n", i, PARAM[i]
+  for (i in GETARG)
+      printf "GETARG[\"%s\"] = %s\n", i, GETARG[i]
+@}
+@c endfile
+@end example
+
+And this is the result when we run it:
+
+@c artificial line wrap in last output line
+@example
+$ gawk -f testserv.awk 
+@print{} MENU["4"] = www.gnu.org
+@print{} MENU["5"] = cgi-bin
+@print{} MENU["6"] = foo
+@print{} MENU["1"] = http
+@print{} MENU["2"] = 
+@print{} MENU["3"] = 
+@print{} PARAM["1"] = p1=stuff
+@print{} PARAM["2"] = p2=stuff&junk
+@print{} PARAM["3"] = percent=a % sign
+@print{} GETARG["p1"] = stuff
+@print{} GETARG["percent"] = a % sign
+@print{} GETARG["p2"] = stuff&junk
+@print{} GETARG["Method"] = GET
+@print{} GETARG["Version"] = 1.0
+@print{} GETARG["URI"] = http://www.gnu.org/cgi-bin/foo?p1=stuff&
+p2=stuff%26junk&percent=a %25 sign
+@end example
+
+@node Simple Server, Caveats, Interacting Service, Using Networking
+@section A Simple Web Server
+@cindex GUI
+In the preceding @value{SECTION}, we built the core logic for event driven GUIs.
+In this @value{SECTION}, we finally extend the core to a real application.
+No one would actually write a commercial web server in @command{gawk}, but
+it is instructive to see that it is feasible in principle.
+
+@iftex
+@image{uf002331,4in}
+@end iftex
+
+@cindex ELIZA program
+@cindex Weizenbaum, Joseph
+The application is ELIZA, the famous program by Joseph Weizenbaum that
+mimics the behavior of a professional psychotherapist when talking to you.
+Weizenbaum would certainly object to this description, but this is part of
+the legend around ELIZA.
+Take the site-independent core logic and append the following code:
+
+@example
+@c file eg/network/eliza.awk
+function SetUpServer() @{
+  SetUpEliza()
+  TopHeader = \
+    "<HTML><title>An HTTP-based System with GAWK</title>\
+    <HEAD><META HTTP-EQUIV=\"Content-Type\"\
+    CONTENT=\"text/html; charset=iso-8859-1\"></HEAD>\
+    <BODY BGCOLOR=\"#ffffff\" TEXT=\"#000000\"\
+    LINK=\"#0000ff\" VLINK=\"#0000ff\"\
+    ALINK=\"#0000ff\"> <A NAME=\"top\">"
+  TopDoc    = "\
+   <h2>Please choose one of the following actions:</h2>\
+   <UL>\
+   <LI>\
+   <A HREF=" MyPrefix "/AboutServer>About this server</A>\
+   </LI><LI>\
+   <A HREF=" MyPrefix "/AboutELIZA>About Eliza</A></LI>\
+   <LI>\
+   <A HREF=" MyPrefix \
+      "/StartELIZA>Start talking to Eliza</A></LI></UL>"
+  TopFooter = "</BODY></HTML>"
+@}
+@c endfile
+@end example
+
+@code{SetUpServer} is similar to the previous example,
+except for calling another function, @code{SetUpEliza}.
+This approach can be used to implement other kinds of servers.
+The only changes needed to do so are hidden in the functions
+@code{SetUpServer} and @code{HandleGET}. Perhaps it might be necessary to
+implement other HTTP methods.
+The @command{igawk} program that comes with @command{gawk}
+may be useful for this process.
+
+When extending this example to a complete application, the first
+thing to do is to implement the function @code{SetUpServer} to
+initialize the HTML pages and some variables. These initializations
+determine the way your HTML pages look (colors, titles, menu
+items, etc.).
+
+@cindex GUI
+The function @code{HandleGET} is a nested case selection that decides
+which page the user wants to see next.  Each nesting level refers to a menu
+level of the GUI. Each case implements a certain action of the menu. On the
+deepest level of case selection, the handler essentially knows what the
+user wants and stores the answer into the variable that holds the HTML
+page contents:
+
+@smallexample
+@c file eg/network/eliza.awk
+function HandleGET() @{
+  # A real HTTP server would treat some parts of the URI as a file name.
+  # We take parts of the URI as menu choices and go on accordingly.
+  if(MENU[2] == "AboutServer") @{
+    Document    = "This is not a CGI script.\
+      This is an httpd, an HTML file, and a CGI script all \
+      in one GAWK script. It needs no separate www-server, \
+      no installation, and no root privileges.\
+      <p>To run it, do this:</p><ul>\
+      <li> start this script with \"gawk -f httpserver.awk\",</li>\
+      <li> and on the same host let your www browser open location\
+           \"http://localhost:8080\"</li>\
+      </ul>\<p>\ Details of HTTP come from:</p><ul>\
+            <li>Hethmon:  Illustrated Guide to HTTP</p>\
+            <li>RFC 2068</li></ul><p>JK 14.9.1997</p>"
+  @} else if (MENU[2] == "AboutELIZA") @{
+    Document    = "This is an implementation of the famous ELIZA\
+        program by Joseph Weizenbaum. It is written in GAWK and\
+/bin/sh: expad: command not found
+  @} else if (MENU[2] == "StartELIZA") @{
+    gsub(/\+/, " ", GETARG["YouSay"])
+    # Here we also have to substitute coded special characters
+    Document    = "<form method=GET>" \
+      "<h3>" ElizaSays(GETARG["YouSay"]) "</h3>\
+      <p><input type=text name=YouSay value=\"\" size=60>\
+      <br><input type=submit value=\"Tell her about it\"></p></form>"
+  @}
+@}
+@c endfile
+@end smallexample
+
+Now we are down to the heart of ELIZA, so you can see how it works.
+Initially the user does not say anything; then ELIZA resets its money
+counter and asks the user to tell what comes to mind open heartedly.
+The subsequent answers are converted to uppercase and stored for
+later comparison. ELIZA presents the bill when being confronted with
+a sentence that contains the phrase ``shut up.'' Otherwise, it looks for
+keywords in the sentence, conjugates the rest of the sentence, remembers
+the keyword for later use, and finally selects an answer from the set of
+possible answers:
+
+@smallexample
+@c file eg/network/eliza.awk
+function ElizaSays(YouSay) @{
+  if (YouSay == "") @{
+    cost = 0
+    answer = "HI, IM ELIZA, TELL ME YOUR PROBLEM"
+  @} else @{
+    q = toupper(YouSay)
+    gsub("'", "", q)
+    if(q == qold) @{
+      answer = "PLEASE DONT REPEAT YOURSELF !"
+    @} else @{
+      if (index(q, "SHUT UP") > 0) @{
+        answer = "WELL, PLEASE PAY YOUR BILL. ITS EXACTLY ... $"\
+                 int(100*rand()+30+cost/100)
+      @} else @{
+        qold = q
+        w = "-"                 # no keyword recognized yet
+        for (i in k) @{          # search for keywords
+          if (index(q, i) > 0) @{
+            w = i
+            break
+          @}
+        @}
+        if (w == "-") @{         # no keyword, take old subject
+          w    = wold
+          subj = subjold
+        @} else @{                # find subject 
+          subj = substr(q, index(q, w) + length(w)+1)
+          wold = w
+          subjold = subj        #  remember keyword and subject
+        @}
+        for (i in conj)
+           gsub(i, conj[i], q)   # conjugation
+        # from all answers to this keyword, select one randomly
+        answer = r[indices[int(split(k[w], indices) * rand()) + 1]]
+        # insert subject into answer
+        gsub("_", subj, answer)
+      @}
+    @}
+  @}
+  cost += length(answer) # for later payment : 1 cent per character
+  return answer
+@}
+@c endfile
+@end smallexample
+
+In the long but simple function @code{SetUpEliza}, you can see tables
+for conjugation, keywords, and answers.@footnote{The version shown
+here is abbreviated.  The full version comes with the @command{gawk}
+distribution.} The associative array @code{k}
+contains indices into the array of answers @code{r}. To choose an
+answer, ELIZA just picks an index randomly:
+
+@example
+@c file eg/network/eliza.awk
+function SetUpEliza() @{
+  srand()
+  wold = "-"
+  subjold = " "
+
+  # table for conjugation
+  conj[" ARE "     ] = " AM "
+  conj["WERE "     ] = "WAS "
+  conj[" YOU "     ] = " I "
+  conj["YOUR "     ] = "MY "
+  conj[" IVE "     ] =\
+  conj[" I HAVE "  ] = " YOU HAVE "
+  conj[" YOUVE "   ] =\
+  conj[" YOU HAVE "] = " I HAVE "
+  conj[" IM "      ] =\
+  conj[" I AM "    ] = " YOU ARE "
+  conj[" YOURE "   ] =\
+  conj[" YOU ARE " ] = " I AM "
+
+  # table of all answers
+  r[1]   = "DONT YOU BELIEVE THAT I CAN  _"
+  r[2]   = "PERHAPS YOU WOULD LIKE TO BE ABLE TO _ ?"
+@c endfile
+  @dots{}
+@end example
+@ignore
+@c file eg/network/eliza.awk
+  r[3]   = "YOU WANT ME TO BE ABLE TO _ ?"
+  r[4]   = "PERHAPS YOU DONT WANT TO _ "
+  r[5]   = "DO YOU WANT TO BE ABLE TO _ ?"
+  r[6]   = "WHAT MAKES YOU THINK I AM _ ?"
+  r[7]   = "DOES IT PLEASE YOU TO BELIEVE I AM _ ?"
+  r[8]   = "PERHAPS YOU WOULD LIKE TO BE _ ?"
+  r[9]   = "DO YOU SOMETIMES WISH YOU WERE _ ?"
+  r[10]  = "DONT YOU REALLY _ ?"
+  r[11]  = "WHY DONT YOU _ ?"
+  r[12]  = "DO YOU WISH TO BE ABLE TO _ ?"
+  r[13]  = "DOES THAT TROUBLE YOU ?"
+  r[14]  = "TELL ME MORE ABOUT SUCH FEELINGS"
+  r[15]  = "DO YOU OFTEN FEEL _ ?"
+  r[16]  = "DO YOU ENJOY FEELING _ ?"
+  r[17]  = "DO YOU REALLY BELIEVE I DONT _ ?"
+  r[18]  = "PERHAPS IN GOOD TIME I WILL _ "
+  r[19]  = "DO YOU WANT ME TO _ ?"
+  r[20]  = "DO YOU THINK YOU SHOULD BE ABLE TO _ ?"
+  r[21]  = "WHY CANT YOU _ ?"
+  r[22]  = "WHY ARE YOU INTERESTED IN WHETHER OR NOT I AM _ ?"
+  r[23]  = "WOULD YOU PREFER IF I WERE NOT _ ?"
+  r[24]  = "PERHAPS IN YOUR FANTASIES I AM _ "
+  r[25]  = "HOW DO YOU KNOW YOU CANT _ ?"
+  r[26]  = "HAVE YOU TRIED ?"
+  r[27]  = "PERHAPS YOU CAN NOW _ "
+  r[28]  = "DID YOU COME TO ME BECAUSE YOU ARE _ ?"
+  r[29]  = "HOW LONG HAVE YOU BEEN _ ?"
+  r[30]  = "DO YOU BELIEVE ITS NORMAL TO BE _ ?"
+  r[31]  = "DO YOU ENJOY BEING _ ?"
+  r[32]  = "WE WERE DISCUSSING YOU -- NOT ME"
+  r[33]  = "Oh, I _"
+  r[34]  = "YOU'RE NOT REALLY TALKING ABOUT ME, ARE YOU ?"
+  r[35]  = "WHAT WOULD IT MEAN TO YOU, IF YOU GOT _ ?"
+  r[36]  = "WHY DO YOU WANT _ ?"
+  r[37]  = "SUPPOSE YOU SOON GOT _"
+  r[38]  = "WHAT IF YOU NEVER GOT _ ?"
+  r[39]  = "I SOMETIMES ALSO WANT _"
+  r[40]  = "WHY DO YOU ASK ?"
+  r[41]  = "DOES THAT QUESTION INTEREST YOU ?"
+  r[42]  = "WHAT ANSWER WOULD PLEASE YOU THE MOST ?"
+  r[43]  = "WHAT DO YOU THINK ?"
+  r[44]  = "ARE SUCH QUESTIONS IN YOUR MIND OFTEN ?"
+  r[45]  = "WHAT IS IT THAT YOU REALLY WANT TO KNOW ?"
+  r[46]  = "HAVE YOU ASKED ANYONE ELSE ?"
+  r[47]  = "HAVE YOU ASKED SUCH QUESTIONS BEFORE ?"
+  r[48]  = "WHAT ELSE COMES TO MIND WHEN YOU ASK THAT ?"
+  r[49]  = "NAMES DON'T INTEREST ME"
+  r[50]  = "I DONT CARE ABOUT NAMES -- PLEASE GO ON"
+  r[51]  = "IS THAT THE REAL REASON ?"
+  r[52]  = "DONT ANY OTHER REASONS COME TO MIND ?"
+  r[53]  = "DOES THAT REASON EXPLAIN ANYTHING ELSE ?"
+  r[54]  = "WHAT OTHER REASONS MIGHT THERE BE ?"
+  r[55]  = "PLEASE DON'T APOLOGIZE !"
+  r[56]  = "APOLOGIES ARE NOT NECESSARY"
+  r[57]  = "WHAT FEELINGS DO YOU HAVE WHEN YOU APOLOGIZE ?"
+  r[58]  = "DON'T BE SO DEFENSIVE"
+  r[59]  = "WHAT DOES THAT DREAM SUGGEST TO YOU ?"
+  r[60]  = "DO YOU DREAM OFTEN ?"
+  r[61]  = "WHAT PERSONS APPEAR IN YOUR DREAMS ?"
+  r[62]  = "ARE YOU DISTURBED BY YOUR DREAMS ?"
+  r[63]  = "HOW DO YOU DO ... PLEASE STATE YOUR PROBLEM"
+  r[64]  = "YOU DON'T SEEM QUITE CERTAIN"
+  r[65]  = "WHY THE UNCERTAIN TONE ?"
+  r[66]  = "CAN'T YOU BE MORE POSITIVE ?"
+  r[67]  = "YOU AREN'T SURE ?"
+  r[68]  = "DON'T YOU KNOW ?"
+  r[69]  = "WHY NO _ ?"
+  r[70]  = "DON'T SAY NO, IT'S ALWAYS SO NEGATIVE"
+  r[71]  = "WHY NOT ?"
+  r[72]  = "ARE YOU SURE ?"
+  r[73]  = "WHY NO ?"
+  r[74]  = "WHY ARE YOU CONCERNED ABOUT MY _ ?"
+  r[75]  = "WHAT ABOUT YOUR OWN _ ?"
+  r[76]  = "CAN'T YOU THINK ABOUT A SPECIFIC EXAMPLE ?"
+  r[77]  = "WHEN ?"
+  r[78]  = "WHAT ARE YOU THINKING OF ?"
+  r[79]  = "REALLY, ALWAYS ?"
+  r[80]  = "DO YOU REALLY THINK SO ?"
+  r[81]  = "BUT YOU ARE NOT SURE YOU _ "
+  r[82]  = "DO YOU DOUBT YOU _ ?"
+  r[83]  = "IN WHAT WAY ?"
+  r[84]  = "WHAT RESEMBLANCE DO YOU SEE ?"
+  r[85]  = "WHAT DOES THE SIMILARITY SUGGEST TO YOU ?"
+  r[86]  = "WHAT OTHER CONNECTION DO YOU SEE ?"
+  r[87]  = "COULD THERE REALLY BE SOME CONNECTIONS ?"
+  r[88]  = "HOW ?"
+  r[89]  = "YOU SEEM QUITE POSITIVE"
+  r[90]  = "ARE YOU SURE ?"
+  r[91]  = "I SEE"
+  r[92]  = "I UNDERSTAND"
+  r[93]  = "WHY DO YOU BRING UP THE TOPIC OF FRIENDS ?"
+  r[94]  = "DO YOUR FRIENDS WORRY YOU ?"
+  r[95]  = "DO YOUR FRIENDS PICK ON YOU ?"
+  r[96]  = "ARE YOU SURE YOU HAVE ANY FRIENDS ?"
+  r[97]  = "DO YOU IMPOSE ON YOUR FRIENDS ?"
+  r[98]  = "PERHAPS YOUR LOVE FOR FRIENDS WORRIES YOU"
+  r[99]  = "DO COMPUTERS WORRY YOU ?"
+  r[100] = "ARE YOU TALKING ABOUT ME IN PARTICULAR ?"
+  r[101] = "ARE YOU FRIGHTENED BY MACHINES ?"
+  r[102] = "WHY DO YOU MENTION COMPUTERS ?"
+  r[103] = "WHAT DO YOU THINK MACHINES HAVE TO DO WITH YOUR PROBLEMS ?"
+  r[104] = "DON'T YOU THINK COMPUTERS CAN HELP PEOPLE ?"
+  r[105] = "WHAT IS IT ABOUT MACHINES THAT WORRIES YOU ?"
+  r[106] = "SAY, DO YOU HAVE ANY PSYCHOLOGICAL PROBLEMS ?"
+  r[107] = "WHAT DOES THAT SUGGEST TO YOU ?"
+  r[108] = "I SEE"
+  r[109] = "IM NOT SURE I UNDERSTAND YOU FULLY"
+  r[110] = "COME COME ELUCIDATE YOUR THOUGHTS"
+  r[111] = "CAN YOU ELABORATE ON THAT ?"
+  r[112] = "THAT IS QUITE INTERESTING"
+  r[113] = "WHY DO YOU HAVE PROBLEMS WITH MONEY ?"
+  r[114] = "DO YOU THINK MONEY IS EVERYTHING ?"
+  r[115] = "ARE YOU SURE THAT MONEY IS THE PROBLEM ?"
+  r[116] = "I THINK WE WANT TO TALK ABOUT YOU, NOT ABOUT ME"
+  r[117] = "WHAT'S ABOUT ME ?"
+  r[118] = "WHY DO YOU ALWAYS BRING UP MY NAME ?"
+@c endfile
+@end ignore
+
+@example
+@c file eg/network/eliza.awk
+  # table for looking up answers that
+  # fit to a certain keyword 
+  k["CAN YOU"]      = "1 2 3"
+  k["CAN I"]        = "4 5"
+  k["YOU ARE"]      =\
+  k["YOURE"]        = "6 7 8 9"
+@c endfile
+  @dots{}
+@end example
+@ignore
+@c file eg/network/eliza.awk
+  k["I DONT"]       = "10 11 12 13"
+  k["I FEEL"]       = "14 15 16"
+  k["WHY DONT YOU"] = "17 18 19"
+  k["WHY CANT I"]   = "20 21"
+  k["ARE YOU"]      = "22 23 24"
+  k["I CANT"]       = "25 26 27"
+  k["I AM"]         =\
+  k["IM "]          = "28 29 30 31"
+  k["YOU "]         = "32 33 34"
+  k["I WANT"]       = "35 36 37 38 39"
+  k["WHAT"]         =\
+  k["HOW"]          =\
+  k["WHO"]          =\
+  k["WHERE"]        =\
+  k["WHEN"]         =\
+  k["WHY"]          = "40 41 42 43 44 45 46 47 48"
+  k["NAME"]         = "49 50"
+  k["CAUSE"]        = "51 52 53 54"
+  k["SORRY"]        = "55 56 57 58"
+  k["DREAM"]        = "59 60 61 62"
+  k["HELLO"]        =\
+  k["HI "]          = "63"
+  k["MAYBE"]        = "64 65 66 67 68"
+  k[" NO "]         = "69 70 71 72 73"
+  k["YOUR"]         = "74 75"
+  k["ALWAYS"]       = "76 77 78 79"
+  k["THINK"]        = "80 81 82"
+  k["LIKE"]         = "83 84 85 86 87 88 89"
+  k["YES"]          = "90 91 92"
+  k["FRIEND"]       = "93 94 95 96 97 98"
+  k["COMPUTER"]     = "99 100 101 102 103 104 105"
+  k["-"]            = "106 107 108 109 110 111 112"
+  k["MONEY"]        = "113 114 115"
+  k["ELIZA"]        = "116 117 118"
+@c endfile
+@end ignore
+@example
+@c file eg/network/eliza.awk
+@}
+@c endfile
+@end example
+
+@cindex Humphrys, Mark
+@cindex ELIZA program
+@cindex Yahoo!
+Some interesting remarks and details (including the original source code
+of ELIZA) are found on Mark Humphrys' home page.  Yahoo!  also has a
+page with a collection of ELIZA-like programs. Many of them are written
+in Java, some of them disclosing the Java source code, and a few even
+explain how to modify the Java source code.
+
+@node Caveats, Challenges, Simple Server, Using Networking
+@section Network Programming Caveats
+
+By now it should be clear
+that debugging a networked application is more
+complicated than debugging a single-process single-hosted application.
+The behavior of a networked application sometimes looks non-causal because
+it is not reproducible in a strong sense. Whether a network application
+works or not sometimes depends on the following:
+
+@itemize @bullet
+@item
+How crowded the underlying network is.
+
+@item
+If the party at the other end is running or not.
+
+@item
+The state of the party at the other end.
+@end itemize
+
+@cindex network
+The most difficult problems for a beginner arise from the hidden states of the
+underlying network. After closing a TCP connection, it's often necessary to wait
+a short while before reopening the connection. Even more difficult is the
+establishment of a connection that previously ended with a ``broken pipe.''
+Those connections have to ``time out'' for a minute or so
+before they can reopen.
+Check this with the command @samp{netstat -a}, which
+provides a list of still ``active'' connections.
+
+@node Challenges, , Caveats, Using Networking
+@section Where To Go From Here
+
+@cindex Loebner, Hugh
+@cindex Contest
+Now, you have learned enough to build your own application. You could,
+for example, take part in the
+Loebner Contest
+to win $100,000.  The Loebner Prize is the first formal instantiation of
+a Turing Test. Hugh Loebner agreed with The Cambridge Center for Behavioral
+Studies to underwrite a contest designed to implement the Turing Test.
+Dr.@: Loebner pledged a Grand Prize of $100,000 for the first computer whose
+responses were indistinguishable from a human's. Each year an annual prize
+of $2000 and a bronze medal is awarded to the @emph{most} human computer.
+The winner of the annual contest is the best entry relative to other entries 
+that year, irrespective of how good it is in an absolute sense. Here is
+an example of a conversation with the winning program of 1997:
+
+@smallexample
+(c)1997 Cambridge Center For Behavioral Studies all rights reserved [CONVERSE]
+[Intelligent Research Ltd.] Start at: [1997/04/29 08:59:40]
+PROGRAM[10:39:20]Hi, my name is Catherine, what's yours?
+JUDGE02[10:39:36]My name is Nathan.
+PROGRAM[10:39:37]Well, it's nice to meet you, Nathan.
+PROGRAM[10:39:45]Did you see that story on CNN last night about the lesbian
+PROGRAM[10:40:03]couple who came out at a White House party on Sunday?
+JUDGE02[10:40:45]No.  I'm sorry I missed that.  How did the Prez take it?
+PROGRAM[10:40:47]Ellen Degeneres was one of them - she was kissing her lover
+PROGRAM[10:41:08]in front of Bill Clinton. Do you think he was very
+PROGRAM[10:41:21]embarrassed?
+JUDGE02[10:41:37]You tell me.  I didn't see it.
+PROGRAM[10:41:38]Are you still a Friend of Bill Clinton?
+JUDGE02[10:42:22]Yes.  I would like to talk about something else.  Do you know anything about sculpture.
+PROGRAM[10:42:24]Well, I suppose he's doing his best like most of us, but he
+PROGRAM[10:42:35]does seem to attract an UNUSUAL amount of bad publicity,
+PROGRAM[10:42:51]wouldnt you say--can it all be bad luck?
+JUDGE02[10:43:20]I asked you if you knew anything about sculpture.
+PROGRAM[10:43:21]OK, but what happens now--does it all drift on, with
+PROGRAM[10:43:36]everyone waiting to see how much his friends have said about
+PROGRAM[10:43:47]him under threat of lots and lots of jail-or will something
+PROGRAM[10:44:01]dramatic actually happen?
+JUDGE02[10:44:19]Do you know what sculpture is?
+PROGRAM[10:44:20]Well, youre more a fan of his than I am I guess---though I
+PROGRAM[10:44:34]cant SWEAR what Id have done in that locked hotel room, I
+PROGRAM[10:44:51]must say.
+JUDGE02[10:45:26]Have you ever been to the Salmagundi Club?
+PROGRAM[10:45:29]Never. to go back to the original discussion Time to get
+PROGRAM[10:45:43]off this one I think and onto something more down to earth!
+JUDGE02[10:46:20]What is more down to earth than art?
+PROGRAM[10:46:21]I don't know what -- to be honest. -
+PROGRAM[10:46:30]I suppose it depends on where you're coming from, but as the
+PROGRAM[10:46:46]song goes in My Fair Lady, "Why can't a woman be more like a
+PROGRAM[10:47:03]man? "
+JUDGE02[10:47:52]I liked that song but I prefer "Why can't children be more like we are"
+JUDGE02[10:48:10]from another play the name of which I cannot remember.
+PROGRAM[10:48:12]What, if anything, do you enjoy about being a man?
+JUDGE02[10:49:01]I would enjoy all aspects of life no matter what my gender.
+PROGRAM[10:49:02]Do you think of yourself as being attractive?
+@end smallexample
+
+@cindex Clinton, Bill
+This program insists on always speaking about the same story around Bill
+Clinton. You see, even a program with a rather narrow mind can behave so
+much like a human being that it can win this prize. It is quite common to
+let these programs talk to each other via network connections. But during the
+competition itself, the program and its computer have to be present at the
+place the competition is held. We all would love to see a @command{gawk}
+program win in such an event. Maybe it is up to you to accomplish this?
+
+Some other ideas for useful networked applications:
+@itemize @bullet
+@item
+Read the file @file{doc/awkforai.txt} in the @command{gawk} distribution.
+It was written by Ronald P.@: Loui (Associate Professor of
+Computer Science, at Washington University in St. Louis,
+@email{loui@@ai.wustl.edu}) and summarizes why
+he teaches @command{gawk} to students of Artificial Intelligence. Here are
+some passages from the text:
+
+@cindex AI
+@cindex PROLOG
+@cindex Loui, Ronald P.
+@cindex agent
+@quotation
+The GAWK manual can
+be consumed in a single lab session and the language can be mastered by
+the next morning by the average student.  GAWK's automatic
+initialization, implicit coercion, I/O support and lack of pointers
+forgive many of the mistakes that young programmers are likely to make.
+Those who have seen C but not mastered it are happy to see that GAWK
+retains some of the same sensibilities while adding what must be
+regarded as spoonsful of syntactic sugar.@*
+@dots{}@*
+@cindex robot
+There are further simple answers.  Probably the best is the fact that
+increasingly, undergraduate AI programming is involving the Web.  Oren
+Etzioni (University of Washington, Seattle) has for a while been arguing
+that the ``softbot'' is replacing the mechanical engineers' robot as the
+most glamorous AI testbed.  If the artifact whose behavior needs to be
+controlled in an intelligent way is the software agent, then a language
+that is well-suited to controlling the software environment is the
+appropriate language.  That would imply a scripting language.  If the
+robot is KAREL, then the right language is ``turn left; turn right.'' If
+the robot is Netscape, then the right language is something that can
+generate @samp{netscape -remote 'openURL(http://cs.wustl.edu/~loui)'} with
+elan.@*
+@dots{}@*
+AI programming requires high-level thinking.  There have always been a few
+gifted programmers who can write high-level programs in assembly language.
+Most however need the ambient abstraction to have a higher floor.@*
+@dots{}@*
+Second, inference is merely the expansion of notation.  No matter whether
+the logic that underlies an AI program is fuzzy, probabilistic, deontic,
+defeasible, or deductive, the logic merely defines how strings can be 
+transformed into other strings.  A language that provides the best
+support for string processing in the end provides the best support for
+logic, for the exploration of various logics, and for most forms of
+symbolic processing that AI might choose to call ``reasoning'' instead of
+``logic.''  The implication is that PROLOG, which saves the AI programmer
+from having to write a unifier, saves perhaps two dozen lines of GAWK
+code at the expense of strongly biasing the logic and representational
+expressiveness of any approach.
+@end quotation
+
+Now that @command{gawk} itself can connect to the Internet, it should be obvious
+that it is suitable for writing intelligent web agents.
+
+@item
+@command{awk} is strong at pattern recognition and string processing.
+So, it is well suited to the classic problem of language translation.
+A first try could be a program that knows the 100 most frequent English
+words and their counterparts in German or French. The service could be
+implemented by regularly reading email with the program above, replacing
+each word by its translation and sending the translation back via SMTP.
+Users would send English email to their translation service and get
+back a translated email message in return. As soon as this works,
+more effort can be spent on a real translation program.
+
+@item
+Another dialogue-oriented application (on the verge
+of ridicule) is the email ``support service.'' Troubled customers write an
+email to an automatic @command{gawk} service that reads the email. It looks
+for keywords in the mail and assembles a reply email accordingly. By carefully
+investigating the email header, and repeating these keywords through the
+reply email, it is rather simple to give the customer a feeling that
+someone cares. Ideally, such a service would search a database of previous
+cases for solutions. If none exists, the database could, for example, consist
+of all the newsgroups, mailing lists and FAQs on the Internet.
+@end itemize
+
+@node Some Applications and Techniques, Links, Using Networking, Top
+@comment node-name,    next,  previous,      up
+
+@chapter Some Applications and Techniques
+In this @value{CHAPTER}, we look at a number of self-contained
+scripts, with an emphasis on concise networking.  Along the way, we
+work towards creating building blocks that encapsulate often needed
+functions of the networking world, show new techniques that
+broaden the scope of problems that can be solved with @command{gawk}, and
+explore leading edge technology that may shape the future of networking.
+
+We often refer to the site-independent core of the server that
+we built in
+@ref{Simple Server, ,A Simple Web Server}.
+When building new and non-trivial servers, we
+always copy this building block and append new instances of the two
+functions @code{SetUpServer} and @code{HandleGET}.
+
+This makes a lot of sense, since
+this scheme of event-driven
+execution provides @command{gawk} with an interface to the most widely
+accepted standard for GUIs: the web browser. Now, @command{gawk} can even rival
+Tcl/Tk.
+
+@cindex Tcl/Tk
+@cindex JavaScript
+Tcl and @command{gawk} have much in common. Both are simple scripting languages
+that allow us to quickly solve problems with short programs. But Tcl has Tk
+on top of it and @command{gawk} had nothing comparable up to now. While Tcl
+needs a large and ever changing library (Tk, which was bound to the X Window
+System until recently), @command{gawk} needs just the networking interface
+and some kind of browser on the client's side. Besides better portability,
+the most important advantage of this approach (embracing well-established
+standards such HTTP and HTML) is that @emph{we do not need to change the
+language}. We let others do the work of fighting over protocols and standards.
+We can use HTML, JavaScript, VRML, or whatever else comes along to do our work.
+
+@menu
+* PANIC::                       An Emergency Web Server.
+* GETURL::                      Retrieving Web Pages.
+* REMCONF::                     Remote Configuration Of Embedded Systems.
+* URLCHK::                      Look For Changed Web Pages.
+* WEBGRAB::                     Extract Links From A Page.
+* STATIST::                     Graphing A Statistical Distribution.
+* MAZE::                        Walking Through A Maze In Virtual Reality.
+* MOBAGWHO::                    A Simple Mobile Agent.
+* STOXPRED::                    Stock Market Prediction As A Service.
+* PROTBASE::                    Searching Through A Protein Database.
+@end menu
+
+@node PANIC, GETURL, Some Applications and Techniques, Some Applications and Techniques
+@section PANIC: an Emergency Web Server
+@cindex PANIC program
+At first glance, the @code{"Hello, world"} example in
+@ref{Primitive Service, ,A Primitive Web Service},
+seems useless. By adding just a few lines, we can turn it into something useful.
+
+The PANIC program tells everyone who connects that the local
+site is not working. When a web server breaks down, it makes a difference
+if customers get a strange ``network unreachable'' message, or a short message
+telling them that the server has a problem. In such an emergency,
+the hard disk and everything on it (including the regular web service) may
+be unavailable. Rebooting the web server off a diskette makes sense in this
+setting.
+
+To use the PANIC program as an emergency web server, all you need are the
+@command{gawk} executable and the program below on a diskette. By default,
+it connects to port 8080. A different value may be supplied on the
+command line:
+
+@example
+@c file eg/network/panic.awk
+BEGIN @{
+  RS = ORS = "\r\n"
+  if (MyPort ==  0) MyPort = 8080
+  HttpService = "/inet/tcp/" MyPort "/0/0"
+  Hello = "<HTML><HEAD><TITLE>Out Of Service</TITLE>" \
+     "</HEAD><BODY><H1>" \
+     "This site is temporarily out of service." \
+     "</H1></BODY></HTML>"
+  Len = length(Hello) + length(ORS)
+  while ("awk" != "complex") @{
+    print "HTTP/1.0 200 OK"          |& HttpService
+    print "Content-Length: " Len ORS |& HttpService
+    print Hello                      |& HttpService
+    while ((HttpService |& getline) > 0)
+       continue;
+    close(HttpService)
+  @}
+@}
+@c endfile
+@end example
+
+@node GETURL, REMCONF, PANIC, Some Applications and Techniques
+@section GETURL: Retrieving Web Pages
+@cindex GETURL program
+@cindex robot
+GETURL is a versatile building block for shell scripts that need to retrieve
+files from the Internet. It takes a web address as a command-line parameter and
+tries to retrieve the contents of this address. The contents are printed
+to standard output, while the header is printed to @file{/dev/stderr}.
+A surrounding shell script
+could analyze the contents and extract the text or the links. An ASCII
+browser could be written around GETURL. But more interestingly, web robots are
+straightforward to write on top of GETURL. On the Internet, you can find
+several programs of the same name that do the same job. They are usually
+much more complex internally and at least 10 times longer.
+
+At first, GETURL checks if it was called with exactly one web address.
+Then, it checks if the user chose to use a special proxy server whose name
+is handed over in a variable. By default, it is assumed that the local
+machine serves as proxy. GETURL uses the @code{GET} method by default
+to access the web page. By handing over the name of a different method
+(such as @code{HEAD}), it is possible to choose a different behavior. With
+the @code{HEAD} method, the user does not receive the body of the page
+content, but does receive the header:
+
+@example
+@c file eg/network/geturl.awk
+BEGIN @{
+  if (ARGC != 2) @{
+    print "GETURL - retrieve Web page via HTTP 1.0"
+    print "IN:\n    the URL as a command-line parameter"
+    print "PARAM(S):\n    -v Proxy=MyProxy"
+    print "OUT:\n    the page content on stdout"
+    print "    the page header on stderr"
+    print "JK 16.05.1997"
+    print "ADR 13.08.2000"
+    exit
+  @}
+  URL = ARGV[1]; ARGV[1] = ""
+  if (Proxy     == "")  Proxy     = "127.0.0.1"
+  if (ProxyPort ==  0)  ProxyPort = 80
+  if (Method    == "")  Method    = "GET"
+  HttpService = "/inet/tcp/0/" Proxy "/" ProxyPort
+  ORS = RS = "\r\n\r\n"
+  print Method " " URL " HTTP/1.0" |& HttpService
+  HttpService                      |& getline Header
+  print Header > "/dev/stderr"
+  while ((HttpService |& getline) > 0)
+    printf "%s", $0
+  close(HttpService)
+@}
+@c endfile
+@end example
+
+This program can be changed as needed, but be careful with the last lines.
+Make sure transmission of binary data is not corrupted by additional line
+breaks. Even as it is now, the byte sequence @code{"\r\n\r\n"} would
+disappear if it were contained in binary data. Don't get caught in a
+trap when trying a quick fix on this one.
+
+@node REMCONF, URLCHK, GETURL, Some Applications and Techniques
+@section REMCONF: Remote Configuration of Embedded Systems
+@cindex REMCONF program
+@cindex Linux
+@cindex GNU/Linux
+@cindex Yahoo!
+Today, you often find powerful processors in embedded systems.  Dedicated
+network routers and controllers for all kinds of machinery are examples
+of embedded systems. Processors like the Intel 80x86 or the AMD Elan are
+able to run multitasking operating systems, such as XINU or GNU/Linux
+in embedded PCs.  These systems are small and usually do not have
+a keyboard or a display.  Therefore it is difficult to set up their
+configuration. There are several widespread ways to set them up:
+
+@itemize @bullet
+@item
+DIP switches
+
+@item
+Read Only Memories such as EPROMs
+
+@item
+Serial lines or some kind of keyboard
+
+@item
+Network connections via @command{telnet} or SNMP
+
+@item
+HTTP connections with HTML GUIs 
+@end itemize
+
+In this @value{SECTION}, we look at a solution that uses HTTP connections
+to control variables of an embedded system that are stored in a file.
+Since embedded systems have tight limits on resources like memory,
+it is difficult to employ advanced techniques such as SNMP and HTTP
+servers. @command{gawk} fits in quite nicely with its single executable
+which needs just a short script to start working.
+The following program stores the variables in a file, and a concurrent
+process in the embedded system may read the file. The program uses the
+site-independent part of the simple web server that we developed in
+@ref{Interacting Service, ,A Web Service with Interaction}.
+As mentioned there, all we have to do is to write two new procedures
+@code{SetUpServer} and @code{HandleGET}:
+
+@smallexample
+@c file eg/network/remconf.awk
+function SetUpServer() @{
+  TopHeader = "<HTML><title>Remote Configuration</title>"
+  TopDoc = "<BODY>\
+    <h2>Please choose one of the following actions:</h2>\
+    <UL>\
+      <LI><A HREF=" MyPrefix "/AboutServer>About this server</A></LI>\
+      <LI><A HREF=" MyPrefix "/ReadConfig>Read Configuration</A></LI>\
+      <LI><A HREF=" MyPrefix "/CheckConfig>Check Configuration</A></LI>\
+      <LI><A HREF=" MyPrefix "/ChangeConfig>Change Configuration</A></LI>\
+      <LI><A HREF=" MyPrefix "/SaveConfig>Save Configuration</A></LI>\
+    </UL>"
+  TopFooter  = "</BODY></HTML>"
+  if (ConfigFile == "") ConfigFile = "config.asc"
+@}
+@c endfile
+@end smallexample
+
+The function @code{SetUpServer} initializes the top level HTML texts
+as usual. It also initializes the name of the file that contains the
+configuration parameters and their values. In case the user supplies
+a name from the command line, that name is used. The file is expected to
+contain one parameter per line, with the name of the parameter in
+column one and the value in column two.
+
+The function @code{HandleGET} reflects the structure of the menu
+tree as usual. The first menu choice tells the user what this is all
+about. The second choice reads the configuration file line by line
+and stores the parameters and their values. Notice that the record
+separator for this file is @code{"\n"}, in contrast to the record separator
+for HTTP. The third menu choice builds an HTML table to show
+the contents of the configuration file just read. The fourth choice
+does the real work of changing parameters, and the last one just saves
+the configuration into a file:
+
+@smallexample
+@c file eg/network/remconf.awk
+function HandleGET() @{
+  if(MENU[2] == "AboutServer") @{
+    Document  = "This is a GUI for remote configuration of an\
+      embedded system. It is is implemented as one GAWK script."
+  @} else if (MENU[2] == "ReadConfig") @{
+    RS = "\n"
+    while ((getline < ConfigFile) > 0)
+       config[$1] = $2;
+    close(ConfigFile)
+    RS = "\r\n"
+    Document = "Configuration has been read."
+  @} else if (MENU[2] == "CheckConfig") @{
+    Document = "<TABLE BORDER=1 CELLPADDING=5>"
+    for (i in config)
+      Document = Document "<TR><TD>" i "</TD>" \
+        "<TD>" config[i] "</TD></TR>"
+    Document = Document "</TABLE>"
+  @} else if (MENU[2] == "ChangeConfig") @{
+    if ("Param" in GETARG) @{            # any parameter to set?
+      if (GETARG["Param"] in config) @{  # is  parameter valid?
+        config[GETARG["Param"]] = GETARG["Value"]
+        Document = (GETARG["Param"] " = " GETARG["Value"] ".")
+      @} else @{
+        Document = "Parameter <b>" GETARG["Param"] "</b> is invalid." 
+      @}
+    @} else @{
+      Document = "<FORM method=GET><h4>Change one parameter</h4>\
+        <TABLE BORDER CELLPADDING=5>\
+        <TR><TD>Parameter</TD><TD>Value</TD></TR>\
+        <TR><TD><input type=text name=Param value=\"\" size=20></TD>\
+            <TD><input type=text name=Value value=\"\" size=40></TD>\
+        </TR></TABLE><input type=submit value=\"Set\"></FORM>"
+    @}
+  @} else if (MENU[2] == "SaveConfig") @{
+    for (i in config)
+      printf("%s %s\n", i, config[i]) > ConfigFile
+    close(ConfigFile)
+    Document = "Configuration has been saved."
+  @}
+@}
+@c endfile
+@end smallexample
+
+@cindex MiniSQL
+We could also view the configuration file as a database. From this
+point of view, the previous program acts like a primitive database server.
+Real SQL database systems also make a service available by providing
+a TCP port that clients can connect to. But the application level protocols
+they use are usually proprietary and also change from time to time.
+This is also true for the protocol that
+MiniSQL uses.
+
+@node URLCHK, WEBGRAB, REMCONF, Some Applications and Techniques
+@section URLCHK: Look for Changed Web Pages
+@cindex URLCHK program
+Most people who make heavy use of Internet resources have a large
+bookmark file with pointers to interesting web sites. It is impossible
+to regularly check by hand if any of these sites have changed. A program
+is needed to automatically look at the headers of web pages and tell
+which ones have changed. URLCHK does the comparison after using GETURL
+with the @code{HEAD} method to retrieve the header.
+
+Like GETURL, this program first checks that it is called with exactly
+one command-line parameter. URLCHK also takes the same command-line variables
+@code{Proxy} and @code{ProxyPort} as GETURL,
+because these variables are handed over to GETURL for each URL
+that gets checked. The one and only parameter is the name of a file that
+contains one line for each URL. In the first column, we find the URL, and
+the second and third columns hold the length of the URL's body when checked
+for the two last times. Now, we follow this plan:
+
+@enumerate
+@item
+Read the URLs from the file and remember their most recent lengths
+
+@item
+Delete the contents of the file
+
+@item
+For each URL, check its new length and write it into the file
+
+@item
+If the most recent and the new length differ, tell the user
+@end enumerate
+
+It may seem a bit peculiar to read the URLs from a file together
+with their two most recent lengths, but this approach has several
+advantages. You can call the program again and again with the same
+file. After running the program, you can regenerate the changed URLs
+by extracting those lines that differ in their second and third columns:
+
+@c inspired by URLCHK in iX 5/97 166.
+@smallexample
+@c file eg/network/urlchk.awk
+BEGIN @{
+  if (ARGC != 2) @{
+    print "URLCHK - check if URLs have changed"
+    print "IN:\n    the file with URLs as a command-line parameter"
+    print "    file contains URL, old length, new length"
+    print "PARAMS:\n    -v Proxy=MyProxy -v ProxyPort=8080"
+    print "OUT:\n    same as file with URLs"
+    print "JK 02.03.1998"
+    exit
+  @}
+  URLfile = ARGV[1]; ARGV[1] = ""
+  if (Proxy     != "") Proxy     = " -v Proxy="     Proxy
+  if (ProxyPort != "") ProxyPort = " -v ProxyPort=" ProxyPort
+  while ((getline < URLfile) > 0)
+     Length[$1] = $3 + 0
+  close(URLfile)      # now, URLfile is read in and can be updated
+  GetHeader = "gawk " Proxy ProxyPort " -v Method=\"HEAD\" -f geturl.awk "
+  for (i in Length) @{
+    GetThisHeader = GetHeader i " 2>&1"
+    while ((GetThisHeader | getline) > 0)
+      if (toupper($0) ~ /CONTENT-LENGTH/) NewLength = $2 + 0
+    close(GetThisHeader)
+    print i, Length[i], NewLength > URLfile
+    if (Length[i] != NewLength)  # report only changed URLs
+      print i, Length[i], NewLength
+  @}
+  close(URLfile)
+@}
+@c endfile
+@end smallexample
+
+Another thing that may look strange is the way GETURL is called.
+Before calling GETURL, we have to check if the proxy variables need
+to be passed on. If so, we prepare strings that will become part
+of the command line later. In @code{GetHeader}, we store these strings
+together with the longest part of the command line. Later, in the loop
+over the URLs, @code{GetHeader} is appended with the URL and a redirection
+operator to form the command that reads the URL's header over the Internet.
+GETURL always produces the headers over @file{/dev/stderr}. That is
+the reason why we need the redirection operator to have the header
+piped in.
+
+This program is not perfect because it assumes that changing URLs
+results in changed lengths, which is not necessarily true. A more
+advanced approach is to look at some other header line that
+holds time information. But, as always when things get a bit more
+complicated, this is left as an exercise to the reader.
+
+@node WEBGRAB, STATIST, URLCHK, Some Applications and Techniques
+@section WEBGRAB: Extract Links from a Page
+@cindex WEBGRAB program
+@c Inspired by iX 1/98 157.
+@cindex robot
+Sometimes it is necessary to extract links from web pages.
+Browsers do it, web robots do it, and sometimes even humans do it.
+Since we have a tool like GETURL at hand, we can solve this problem with
+some help from the Bourne shell:
+
+@example
+@c file eg/network/webgrab.awk
+BEGIN @{ RS = "http://[#%&\\+\\-\\./0-9\\:;\\?A-Z_a-z\\~]*" @}
+RT != "" @{
+   command = ("gawk -v Proxy=MyProxy -f geturl.awk " RT \
+               " > doc" NR ".html")
+   print command
+@}
+@c endfile
+@end example
+
+Notice that the regular expression for URLs is rather crude. A precise
+regular expression is much more complex. But this one works
+rather well. One problem is that it is unable to find internal links of
+an HTML document.  Another problem is that
+@samp{ftp}, @samp{telnet}, @samp{news}, @samp{mailto}, and other kinds
+of links are missing in the regular expression.
+However, it is straightforward to add them, if doing so is necessary for other tasks.
+
+This program reads an HTML file and prints all the HTTP links that it finds.
+It relies on @command{gawk}'s ability to use regular expressions as record
+separators. With @code{RS} set to a regular expression that matches links,
+the second action is executed each time a non-empty link is found.
+We can find the matching link itself in @code{RT}.
+
+The action could use the @code{system} function to let another GETURL
+retrieve the page, but here we use a different approach.
+This simple program prints shell commands that can be piped into @command{sh}
+for execution.  This way it is possible to first extract
+the links, wrap shell commands around them, and pipe all the shell commands
+into a file. After editing the file, execution of the file retrieves
+exactly those files that we really need. In case we do not want to edit,
+we can retrieve all the pages like this:
+
+@smallexample
+gawk -f geturl.awk http://www.suse.de | gawk -f webgrab.awk | sh
+@end smallexample
+
+@cindex Microsoft Windows
+After this, you will find the contents of all referenced documents in
+files named @file{doc*.html} even if they do not contain HTML code.
+The most annoying thing is that we always have to pass the proxy to
+GETURL. If you do not like to see the headers of the web pages
+appear on the screen, you can redirect them to @file{/dev/null}.
+Watching the headers appear can be quite interesting, because
+it reveals
+interesting details such as which web server the companies use.
+Now, it is clear how the clever marketing people
+use web robots to determine the
+market shares
+of Microsoft and Netscape in the web server market.
+
+Port 80 of any web server is like a small hole in a repellent firewall.
+After attaching a browser to port 80, we usually catch a glimpse
+of the bright side of the server (its home page). With a tool like GETURL
+at hand, we are able to discover some of the more concealed
+or even ``indecent'' services (i.e., lacking conformity to standards of quality).
+It can be exciting to see the fancy CGI scripts that lie
+there, revealing the inner workings of the server, ready to be called:
+
+@itemize @bullet
+@item
+With a command such as:
+
+@example
+gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/
+@end example
+
+some servers give you a directory listing of the CGI files.
+Knowing the names, you can try to call some of them and watch
+for useful results. Sometimes there are executables in such directories
+(such as Perl interpreters) that you may call remotely. If there are
+subdirectories with configuration data of the web server, this can also
+be quite interesting to read.
+
+@item
+@cindex apache
+The well-known Apache web server usually has its CGI files in the
+directory @file{/cgi-bin}. There you can often find the scripts
+@file{test-cgi} and @file{printenv}. Both tell you some things
+about the current connection and the installation of the web server.
+Just call:
+
+@smallexample
+gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/test-cgi
+gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/printenv
+@end smallexample
+
+@item
+Sometimes it is even possible to retrieve system files like the web
+server's log file---possibly containing customer data---or even the file
+@file{/etc/passwd}.
+(We don't recommend this!)
+@end itemize
+
+@strong{Caution:}
+Although this may sound funny or simply irrelevant, we are talking about
+severe security holes. Try to explore your own system this way and make
+sure that none of the above reveals too much information about your system.
+
+@node STATIST, MAZE, WEBGRAB, Some Applications and Techniques
+@section STATIST: Graphing a Statistical Distribution
+@cindex STATIST program
+
+@cindex GNUPlot utility
+@cindex image format
+@cindex @file{gif} image format
+@cindex @file{png} image format
+@cindex @file{ps} image format
+@cindex Boutell, Thomas
+@iftex
+@image{statist,3in}
+@end iftex
+In the HTTP server examples we've shown thus far, we never present an image
+to the browser and its user. Presenting images is one task. Generating
+images that reflect some user input and presenting these dynamically
+generated images is another. In this @value{SECTION}, we use GNUPlot
+for generating @file{.png}, @file{.ps}, or @file{.gif}
+files.@footnote{Due to licensing problems, the default
+installation of GNUPlot disables the generation of @file{.gif} files.
+If your installed version does not accept @samp{set term gif},
+just download and install the most recent version of GNUPlot and the
+@uref{http://www.boutell.com/gd/, GD library}
+by Thomas Boutell.
+Otherwise you still have the chance to generate some
+ASCII-art style images with GNUPlot by using @samp{set term dumb}.
+(We tried it and it worked.)}
+
+The program we develop takes the statistical parameters of two samples
+and computes the t-test statistics. As a result, we get the probabilities
+that the means and the variances of both samples are the same. In order to
+let the user check plausibility, the program presents an image of the
+distributions. The statistical computation follows
+@cite{Numerical Recipes in C: The Art of Scientific Computing}
+by William H.@: Press, Saul A.@: Teukolsky, William T.@: Vetterling, and Brian P. Flannery.
+Since @command{gawk} does not have a built-in function
+for the computation of the beta function, we use the @code{ibeta} function
+of GNUPlot. As a side effect, we learn how to use GNUPlot as a
+sophisticated calculator. The comparison of means is done as in @code{tutest},
+paragraph 14.2, page 613, and the comparison of variances is done as in @code{ftest},
+page 611 in @cite{Numerical Recipes}.
+@cindex Numerical Recipes
+
+As usual, we take the site-independent code for servers and append
+our own functions @code{SetUpServer} and @code{HandleGET}:
+
+@smallexample
+@c file eg/network/statist.awk
+function SetUpServer() @{
+  TopHeader = "<HTML><title>Statistics with GAWK</title>"
+  TopDoc = "<BODY>\
+   <h2>Please choose one of the following actions:</h2>\
+   <UL>\
+    <LI><A HREF=" MyPrefix "/AboutServer>About this server</A></LI>\
+    <LI><A HREF=" MyPrefix "/EnterParameters>Enter Parameters</A></LI>\
+   </UL>"
+  TopFooter  = "</BODY></HTML>"
+  GnuPlot    = "gnuplot 2>&1"
+  m1=m2=0;    v1=v2=1;    n1=n2=10
+@}
+@c endfile
+@end smallexample
+
+Here, you see the menu structure that the user sees. Later, we
+will see how the program structure of the @code{HandleGET} function
+reflects the menu structure. What is missing here is the link for the
+image we generate. In an event-driven environment, request,
+generation, and delivery of images are separated.
+
+Notice the way we initialize the @code{GnuPlot} command string for
+the pipe. By default,
+GNUPlot outputs the generated image via standard output, as well as
+the results of @code{print}(ed) calculations via standard error.
+The redirection causes standard error to be mixed into standard
+output, enabling us to read results of calculations with @code{getline}.
+By initializing the statistical parameters with some meaningful
+defaults, we make sure the user gets an image the first time
+he uses the program.
+
+@cindex JavaScript
+Following is the rather long function @code{HandleGET}, which
+implements the contents of this service by reacting to the different
+kinds of requests from the browser. Before you start playing with
+this script, make sure that your browser supports JavaScript and that it also
+has this option switched on. The script uses a short snippet of
+JavaScript code for delayed opening of a window with an image.
+A more detailed explanation follows:
+
+@smallexample
+@c file eg/network/statist.awk
+function HandleGET() @{
+  if(MENU[2] == "AboutServer") @{
+    Document  = "This is a GUI for a statistical computation.\
+      It compares means and variances of two distributions.\
+      It is implemented as one GAWK script and uses GNUPLOT."
+  @} else if (MENU[2] == "EnterParameters") @{
+    Document = ""
+    if ("m1" in GETARG) @{     # are there parameters to compare?
+      Document = Document "<SCRIPT LANGUAGE=\"JavaScript\">\
+        setTimeout(\"window.open(\\\"" MyPrefix "/Image" systime()\
+         "\\\",\\\"dist\\\", \\\"status=no\\\");\", 1000); </SCRIPT>"
+      m1 = GETARG["m1"]; v1 = GETARG["v1"]; n1 = GETARG["n1"]
+      m2 = GETARG["m2"]; v2 = GETARG["v2"]; n2 = GETARG["n2"]
+      t = (m1-m2)/sqrt(v1/n1+v2/n2)
+      df = (v1/n1+v2/n2)*(v1/n1+v2/n2)/((v1/n1)*(v1/n1)/(n1-1) \
+           + (v2/n2)*(v2/n2) /(n2-1))
+      if (v1>v2) @{
+          f = v1/v2
+          df1 = n1 - 1
+          df2 = n2 - 1
+      @} else @{
+          f = v2/v1
+          df1 = n2 - 1
+          df2 = n1 - 1
+      @}
+      print "pt=ibeta(" df/2 ",0.5," df/(df+t*t) ")"  |& GnuPlot
+      print "pF=2.0*ibeta(" df2/2 "," df1/2 "," \
+            df2/(df2+df1*f) ")"                    |& GnuPlot
+      print "print pt, pF"                         |& GnuPlot
+      RS="\n"; GnuPlot |& getline; RS="\r\n"    # $1 is pt, $2 is pF
+      print "invsqrt2pi=1.0/sqrt(2.0*pi)"          |& GnuPlot
+      print "nd(x)=invsqrt2pi/sd*exp(-0.5*((x-mu)/sd)**2)" |& GnuPlot
+      print "set term png small color"             |& GnuPlot
+      #print "set term postscript color"           |& GnuPlot
+      #print "set term gif medium size 320,240"    |& GnuPlot
+      print "set yrange[-0.3:]"                    |& GnuPlot
+      print "set label 'p(m1=m2) =" $1 "' at 0,-0.1 left"  |& GnuPlot
+      print "set label 'p(v1=v2) =" $2 "' at 0,-0.2 left"  |& GnuPlot
+      print "plot mu=" m1 ",sd=" sqrt(v1) ", nd(x) title 'sample 1',\
+        mu=" m2 ",sd=" sqrt(v2) ", nd(x) title 'sample 2'" |& GnuPlot
+      print "quit"                                         |& GnuPlot
+      GnuPlot |& getline Image
+      while ((GnuPlot |& getline) > 0)
+          Image = Image RS $0
+      close(GnuPlot)
+    @}
+    Document = Document "\
+    <h3>Do these samples have the same Gaussian distribution?</h3>\
+    <FORM METHOD=GET> <TABLE BORDER CELLPADDING=5>\
+    <TR>\
+    <TD>1. Mean    </TD>
+    <TD><input type=text name=m1 value=" m1 " size=8></TD>\
+    <TD>1. Variance</TD>
+    <TD><input type=text name=v1 value=" v1 " size=8></TD>\
+    <TD>1. Count   </TD>
+    <TD><input type=text name=n1 value=" n1 " size=8></TD>\
+    </TR><TR>\
+    <TD>2. Mean    </TD>
+    <TD><input type=text name=m2 value=" m2 " size=8></TD>\
+    <TD>2. Variance</TD>
+    <TD><input type=text name=v2 value=" v2 " size=8></TD>\
+    <TD>2. Count   </TD>
+    <TD><input type=text name=n2 value=" n2 " size=8></TD>\
+    </TR>                   <input type=submit value=\"Compute\">\      
+    </TABLE></FORM><BR>"
+  @} else if (MENU[2] ~ "Image") @{     
+    Reason = "OK" ORS "Content-type: image/png"
+    #Reason = "OK" ORS "Content-type: application/x-postscript"
+    #Reason = "OK" ORS "Content-type: image/gif"
+    Header = Footer = ""
+    Document = Image
+  @}
+@}
+@c endfile
+@end smallexample
+
+@cindex PostScript
+As usual, we give a short description of the service in the first
+menu choice. The third menu choice shows us that generation and
+presentation of an image are two separate actions. While the latter
+takes place quite instantly in the third menu choice, the former
+takes place in the much longer second choice. Image data passes from the
+generating action to the presenting action via the variable @code{Image}
+that contains a complete @file{.png} image, which is otherwise stored
+in a file. If you prefer @file{.ps} or @file{.gif} images over the
+default @file{.png} images, you may select these options by uncommenting
+the appropriate lines. But remember to do so in two places: when
+telling GNUPlot which kind of images to generate, and when transmitting the
+image at the end of the program.
+
+Looking at the end of the program,
+the way we pass the @samp{Content-type} to the browser is a bit unusual.
+It is appended to the @samp{OK} of the first header line
+to make sure the type information becomes part of the header.
+The other variables that get transmitted across the network are
+made empty, because in this case we do not have an HTML document to
+transmit, but rather raw image data to contain in the body.
+
+Most of the work is done in the second menu choice. It starts with a
+strange JavaScript code snippet. When first implementing this server,
+we used a short @code{@w{"<IMG SRC="} MyPrefix "/Image>"} here. But then
+browsers got smarter and tried to improve on speed by requesting the
+image and the HTML code at the same time. When doing this, the browser
+tries to build up a connection for the image request while the request for
+the HTML text is not yet completed. The browser tries to connect
+to the @command{gawk} server on port 8080 while port 8080 is still in use for
+transmission of the HTML text. The connection for the image cannot be
+built up, so the image appears as ``broken'' in the browser window.
+We solved this problem by telling the browser to open a separate window
+for the image, but only after a delay of 1000 milliseconds.
+By this time, the server should be ready for serving the next request.
+
+But there is one more subtlety in the JavaScript code.
+Each time the JavaScript code opens a window for the image, the
+name of the image is appended with a timestamp (@code{systime}).
+Why this constant change of name for the image? Initially, we always named
+the image @code{Image}, but then the Netscape browser noticed the name
+had @emph{not} changed since the previous request and displayed the
+previous image (caching behavior). The server core
+is implemented so that browsers are told @emph{not} to cache anything.
+Obviously HTTP requests do not always work as expected. One way to
+circumvent the cache of such overly smart browsers is to change the
+name of the image with each request. These three lines of JavaScript
+caused us a lot of trouble.
+
+The rest can be broken
+down into two phases. At first, we check if there are statistical
+parameters. When the program is first started, there usually are no
+parameters because it enters the page coming from the top menu.
+Then, we only have to present the user a form that he can use to change
+statistical parameters and submit them. Subsequently, the submission of
+the form causes the execution of the first phase because @emph{now}
+there @emph{are} parameters to handle.
+
+Now that we have parameters, we know there will be an image available.
+Therefore we insert the JavaScript code here to initiate the opening
+of the image in a separate window. Then,
+we prepare some variables that will be passed to GNUPlot for calculation
+of the probabilities. Prior to reading the results, we must temporarily
+change @code{RS} because GNUPlot separates lines with newlines.
+After instructing GNUPlot to generate a @file{.png} (or @file{.ps} or
+@file{.gif}) image, we initiate the insertion of some text,
+explaining the resulting probabilities. The final @samp{plot} command
+actually generates the image data. This raw binary has to be read in carefully
+without adding, changing, or deleting a single byte. Hence the unusual
+initialization of @code{Image} and completion with a @code{while} loop.
+
+When using this server, it soon becomes clear that it is far from being
+perfect. It mixes source code of six scripting languages or protocols:
+
+@itemize @bullet
+@item GNU @command{awk} implements a server for the protocol:
+@item HTTP which transmits:
+@item HTML text which contains a short piece of:
+@item JavaScript code opening a separate window.
+@item A Bourne shell script is used for piping commands into:
+@item GNUPlot to generate the image to be opened.
+@end itemize
+
+After all this work, the GNUPlot image opens in the JavaScript window
+where it can be viewed by the user.
+
+It is probably better not to mix up so many different languages.
+The result is not very readable.  Furthermore, the
+statistical part of the server does not take care of invalid input.
+Among others, using negative variances will cause invalid results.
+
+@node MAZE, MOBAGWHO, STATIST, Some Applications and Techniques
+@section MAZE: Walking Through a Maze In Virtual Reality
+@cindex MAZE
+@cindex VRML
+@c VRML in iX 11/96 134.
+@quotation
+@cindex Perlis, Alan
+@i{In the long run, every program becomes rococo, and then rubble.}@*
+Alan Perlis
+@end quotation
+
+By now, we know how to present arbitrary @samp{Content-type}s to a browser.
+In this @value{SECTION}, our server will present a 3D world to our browser.
+The 3D world is described in a scene description language (VRML,
+Virtual Reality Modeling Language) that allows us to travel through a
+perspective view of a 2D maze with our browser. Browsers with a
+VRML plugin enable exploration of this technology. We could do
+one of those boring @samp{Hello world} examples here, that are usually
+presented when introducing novices to
+VRML. If you have never written
+any VRML code, have a look at
+the VRML FAQ.
+Presenting a static VRML scene is a bit trivial; in order to expose
+@command{gawk}'s new capabilities, we will present a dynamically generated
+VRML scene. The function @code{SetUpServer} is very simple because it
+only sets the default HTML page and initializes the random number
+generator. As usual, the surrounding server lets you browse the maze.
+
+@smallexample
+@c file eg/network/maze.awk
+function SetUpServer() @{
+  TopHeader = "<HTML><title>Walk through a maze</title>"
+  TopDoc = "\
+    <h2>Please choose one of the following actions:</h2>\
+    <UL>\
+      <LI><A HREF=" MyPrefix "/AboutServer>About this server</A>\
+      <LI><A HREF=" MyPrefix "/VRMLtest>Watch a simple VRML scene</A>\
+    </UL>"
+  TopFooter  = "</HTML>"
+  srand()
+@}
+@c endfile
+@end smallexample
+
+The function @code{HandleGET} is a bit longer because it first computes
+the maze and afterwards generates the VRML code that is sent across
+the network. As shown in the STATIST example
+(@pxref{STATIST}),
+we set the type of the
+content to VRML and then store the VRML representation of the maze as the
+page content. We assume that the maze is stored in a 2D array. Initially,
+the maze consists of walls only. Then, we add an entry and an exit to the
+maze and let the rest of the work be done by the function @code{MakeMaze}.
+Now, only the wall fields are left in the maze. By iterating over the these
+fields, we generate one line of VRML code for each wall field.
+
+@smallexample
+@c file eg/network/maze.awk
+function HandleGET() @{
+  if (MENU[2] == "AboutServer") @{
+    Document  = "If your browser has a VRML 2 plugin,\
+      this server shows you a simple VRML scene."
+  @} else if (MENU[2] == "VRMLtest") @{
+    XSIZE = YSIZE = 11              # initially, everything is wall
+    for (y = 0; y < YSIZE; y++)
+       for (x = 0; x < XSIZE; x++)
+          Maze[x, y] = "#"
+    delete Maze[0, 1]              # entry is not wall
+    delete Maze[XSIZE-1, YSIZE-2]  # exit  is not wall
+    MakeMaze(1, 1)
+    Document = "\
+#VRML V2.0 utf8\n\
+Group @{\n\
+  children [\n\
+    PointLight @{\n\
+      ambientIntensity 0.2\n\
+      color 0.7 0.7 0.7\n\
+      location 0.0 8.0 10.0\n\
+    @}\n\
+    DEF B1 Background @{\n\
+      skyColor [0 0 0, 1.0 1.0 1.0 ]\n\
+      skyAngle 1.6\n\
+      groundColor [1 1 1, 0.8 0.8 0.8, 0.2 0.2 0.2 ]\n\
+      groundAngle [ 1.2 1.57 ]\n\
+    @}\n\
+    DEF Wall Shape @{\n\
+      geometry Box @{size 1 1 1@}\n\
+      appearance Appearance @{ material Material @{ diffuseColor 0 0 1 @} @}\n\
+    @}\n\
+    DEF Entry Viewpoint @{\n\
+      position 0.5 1.0 5.0\n\
+      orientation 0.0 0.0 -1.0 0.52\n\
+    @}\n"
+    for (i in Maze) @{
+      split(i, t, SUBSEP)
+      Document = Document "    Transform @{ translation "
+      Document = Document t[1] " 0 -" t[2] " children USE Wall @}\n"
+    @}
+    Document = Document "  ] # end of group for world\n@}"
+    Reason = "OK" ORS "Content-type: model/vrml"
+    Header = Footer = ""
+  @}
+@}
+@c endfile
+@end smallexample
+
+Finally, we have a look at @code{MakeMaze}, the function that generates
+the @code{Maze} array. When entered, this function assumes that the array
+has been initialized so that each element represents a wall element and
+the maze is initially full of wall elements. Only the entrance and the exit
+of the maze should have been left free. The parameters of the function tell
+us which element must be marked as not being a wall. After this, we take
+a look at the four neighbouring elements and remember which we have already
+treated. Of all the neighbouring elements, we take one at random and
+walk in that direction. Therefore, the wall element in that direction has
+to be removed and then, we call the function recursively for that element.
+The maze is only completed if we iterate the above procedure for
+@emph{all} neighbouring elements (in random order) and for our present
+element by recursively calling the function for the present element. This
+last iteration could have been done in a loop,
+but it is done much simpler recursively.
+
+Notice that elements with coordinates that are both odd are assumed to be
+on our way through the maze and the generating process cannot terminate
+as long as there is such an element not being @code{delete}d. All other
+elements are potentially part of the wall.
+
+@smallexample
+@c file eg/network/maze.awk
+function MakeMaze(x, y) @{
+  delete Maze[x, y]     # here we are, we have no wall here
+  p = 0                 # count unvisited fields in all directions
+  if (x-2 SUBSEP y   in Maze) d[p++] = "-x"
+  if (x   SUBSEP y-2 in Maze) d[p++] = "-y"
+  if (x+2 SUBSEP y   in Maze) d[p++] = "+x"
+  if (x   SUBSEP y+2 in Maze) d[p++] = "+y"
+  if (p>0) @{            # if there are univisited fields, go there
+    p = int(p*rand())   # choose one unvisited field at random
+    if        (d[p] == "-x") @{ delete Maze[x - 1, y]; MakeMaze(x - 2, y)
+    @} else if (d[p] == "-y") @{ delete Maze[x, y - 1]; MakeMaze(x, y - 2)
+    @} else if (d[p] == "+x") @{ delete Maze[x + 1, y]; MakeMaze(x + 2, y)
+    @} else if (d[p] == "+y") @{ delete Maze[x, y + 1]; MakeMaze(x, y + 2)
+    @}                   # we are back from recursion
+    MakeMaze(x, y);     # try again while there are unvisited fields
+  @}
+@}
+@c endfile
+@end smallexample
+
+@node MOBAGWHO, STOXPRED, MAZE, Some Applications and Techniques
+@section MOBAGWHO: a Simple Mobile Agent
+@cindex MOBAGWHO program
+@cindex agent
+@quotation
+@cindex Hoare, C.A.R.
+@i{There are two ways of constructing a software design: One way is to
+make it so simple that there are obviously no deficiencies, and the
+other way is to make it so complicated that there are no obvious
+deficiencies.} @*
+C. A. R. Hoare
+@end quotation
+
+A @dfn{mobile agent} is a program that can be dispatched from a computer and
+transported to a remote server for execution. This is called @dfn{migration},
+which means that a process on another system is started that is independent
+from its originator. Ideally, it wanders through
+a network while working for its creator or owner. In places like
+the UMBC Agent Web,
+people are quite confident that (mobile) agents are a software engineering
+paradigm that enables us to significantly increase the efficiency
+of our work. Mobile agents could become the mediators between users and
+the networking world. For an unbiased view at this technology,
+see the remarkable paper @cite{Mobile Agents: Are they a good
+idea?}.@footnote{@uref{http://www.research.ibm.com/massive/mobag.ps}}
+
+@ignore
+@c Chuck says to take all of this out.
+@cindex Tcl/Tk
+A good instance of this paradigm is
+@cite{Agent Tcl},@footnote{@uref{http://agent.cs.dartmouth.edu/software/agent2.0/}}
+an extension of the Tcl language. After introducing a typical
+development environment, the aforementioned paper shows a nice little
+example application that we will try to rebuild in @command{gawk}. The
+@command{who} agent takes a list of servers and wanders from one server
+to the next one, always looking to see who is logged in.
+Having reached the last
+one, it sends back a message with a list of all users it found on each
+machine.
+
+But before implementing something that might or might not be a mobile
+agent, let us clarify the concept and some important terms. The agent
+paradigm in general is such a young scientific discipline that it has
+not yet developed a widely-accepted terminology. Some authors try to
+give precise definitions, but their scope is often not wide enough
+to be generally accepted. Franklin and Graesser ask
+@cite{Is it an Agent or just a Program: A Taxonomy for Autonomous
+Agents}@footnote{@uref{http://www.msci.memphis.edu/~franklin/AgentProg.html}}
+and give even better answers than Caglayan and Harrison in their
+@cite{Agent Sourcebook}.@footnote{@uref{http://www.aminda.com/mazzu/sourcebook/}}
+
+@itemize @minus
+@item
+@i{An autonomous agent is a system situated within and a part of
+an environment that senses that environment and acts on it, over time, in
+pursuit of its own agenda and so as to effect what it senses in the future.}
+(Quoted from Franklin and Graesser.)
+@item
+A mobile agent is able to transport itself from one machine to another.
+@item
+The term @dfn{migration} often denotes this process of moving.
+But neither of the two sources above even mentions this term, while others
+use it regularly.
+@end itemize
+
+Before delving into the (rather demanding) details of
+implementation, let us give just one more quotation as a final
+motivation. Steven Farley published an excellent paper called
+@cite{Mobile Agent System Architecture},@footnote{This often
+cited text originally appeared as a conference paper here:
+@uref{http://www.sigs.com/publications/docs/java/9705/farley.html}
+Many bibliographies on the Internet point to this dead link. Meanwhile,
+the paper appeared as a contribution to a book called More Java Gems here:
+@uref{http://uk.cambridge.org/computerscience/object/catalogue/0521774772/default.htm}}
+in which he asks ``Why use an agent architecture?''
+
+@quotation
+If client-server systems are the currently established norm and distributed
+object systems such as CORBA are defining the future standards, why bother
+with agents? Agent architectures have certain advantages over these other
+types. Three of the most important advantages are:
+@cindex CORBA
+
+@enumerate
+@item
+An agent performs much processing at the server where local bandwidth
+is high, thus reducing the amount of network bandwidth consumed and increasing
+overall performance. In contrast, a CORBA client object with the equivalent
+functionality of a given agent must make repeated remote method calls to
+the server object because CORBA objects cannot move across the network
+at runtime.
+
+@item
+An agent operates independently of the application from which the
+agent was invoked. The agent operates asynchronously, meaning that the
+client application does not need to wait for the results. This is especially
+important for mobile users who are not always connected to the network.
+
+@item
+The use of agents allows for the injection of new functionality into
+a system at run time. An agent system essentially contains its own automatic
+software distribution mechanism. Since CORBA has no built-in support for
+mobile code, new functionality generally has to be installed manually.
+
+@end enumerate
+
+Of course a non-agent system can exhibit these same features with some
+work. But the mobile code paradigm supports the transfer of executable
+code to a remote location for asynchronous execution from the start. An
+agent architecture should be considered for systems where the above features
+are primary requirements.
+@end quotation
+@end ignore
+
+When trying to migrate a process from one system to another,
+a server process is needed on the receiving side. Depending on the kind
+of server process, several ways of implementation come to mind.
+How the process is implemented depends upon the kind of server process:
+
+@itemize @bullet
+@item
+HTTP can be used as the protocol for delivery of the migrating
+process. In this case, we use a common web
+server as the receiving server process. A universal CGI script
+mediates between migrating process and web server.
+Each server willing to accept migrating agents makes this universal
+service available. HTTP supplies the @code{POST} method to transfer
+some data to a file on the web server. When a CGI script is called
+remotely with the @code{POST} method instead of the usual @code{GET} method,
+data is transmitted from the client process to the standard input
+of the server's CGI script. So, to implement a mobile agent,
+we must not only write the agent program to start on the client
+side, but also the CGI script to receive the agent on the server side.
+
+@cindex CGI
+@cindex apache
+@item
+The @code{PUT} method can also be used for migration. HTTP does not
+require a CGI script for migration via @code{PUT}. However, with common web
+servers there is no advantage to this solution, because web servers such as
+Apache
+require explicit activation of a special @code{PUT} script.
+
+@item
+@cite{Agent Tcl} pursues a different course; it relies on a dedicated server
+process with a dedicated protocol specialized for receiving mobile agents.
+@end itemize
+
+Our agent example abuses a common web server as a migration tool. So, it needs a
+universal CGI script on the receiving side (the web server). The receiving script is
+activated with a @code{POST} request when placed into a location like
+@file{/httpd/cgi-bin/PostAgent.sh}. Make sure that the server system uses a
+version of @command{gawk} that supports network access (Version 3.1 or later;
+verify with @samp{gawk --version}).
+
+@example
+@c file eg/network/PostAgent.sh
+#!/bin/sh
+MobAg=/tmp/MobileAgent.$$
+# direct script to mobile agent file
+cat > $MobAg
+# execute agent concurrently
+gawk -f $MobAg $MobAg > /dev/null &
+# HTTP header, terminator and body
+gawk 'BEGIN @{ print "\r\nAgent started" @}'
+rm $MobAg      # delete script file of agent
+@c endfile
+@end example
+
+By making its process id (@code{$$}) part of the unique @value{FN}, the
+script avoids conflicts between concurrent instances of the script.
+First, all lines
+from standard input (the mobile agent's source code) are copied into
+this unique file. Then, the agent is started as a concurrent process
+and a short message reporting this fact is sent to the submitting client.
+Finally, the script file of the mobile agent is removed because it is
+no longer needed. Although it is a short script, there are several noteworthy
+points:
+
+@table @asis
+@item Security
+@emph{There is none}. In fact, the CGI script should never
+be made available on a server that is part of the Internet because everyone
+would be allowed to execute arbitrary commands with it. This behavior is
+acceptable only when performing rapid prototyping.
+
+@item Self-Reference
+Each migrating instance of an agent is started
+in a way that enables it to read its own source code from standard input
+and use the code for subsequent
+migrations. This is necessary because it needs to treat the agent's code
+as data to transmit. @command{gawk} is not the ideal language for such
+a job. Lisp and Tcl are more suitable because they do not make a distinction
+between program code and data.
+
+@item Independence
+After migration, the agent is not linked to its
+former home in any way. By reporting @samp{Agent started}, it waves
+``Goodbye'' to its origin. The originator may choose to terminate or not.
+@end table
+
+@cindex Lisp
+The originating agent itself is started just like any other command-line
+script, and reports the results on standard output.  By letting the name
+of the original host migrate with the agent, the agent that migrates
+to a host far away from its origin can report the result back home.
+Having arrived at the end of the journey, the agent establishes
+a connection and reports the results.  This is the reason for
+determining the name of the host with @samp{uname -n} and storing it
+in @code{MyOrigin} for later use.  We may also set variables with the
+@option{-v} option from the command line. This interactivity is only
+of importance in the context of starting a mobile agent; therefore this
+@code{BEGIN} pattern and its action do not take part in migration:
+
+@smallexample
+@c file eg/network/mobag.awk
+BEGIN @{
+  if (ARGC != 2) @{
+    print "MOBAG - a simple mobile agent"
+    print "CALL:\n    gawk -f mobag.awk mobag.awk"
+    print "IN:\n    the name of this script as a command-line parameter"
+    print "PARAM:\n    -v MyOrigin=myhost.com"
+    print "OUT:\n    the result on stdout"
+    print "JK 29.03.1998 01.04.1998"
+    exit
+  @}
+  if (MyOrigin == "") @{
+     "uname -n" | getline MyOrigin
+     close("uname -n")
+  @}
+@}
+@c endfile
+@end smallexample
+
+Since @command{gawk} cannot manipulate and transmit parts of the program
+directly, the source code is read and stored in strings.
+Therefore, the program scans itself for
+the beginning and the ending of functions.
+Each line in between is appended to the code string until the end of
+the function has been reached. A special case is this part of the program
+itself. It is not a function.
+Placing a similar framework around it causes it to be treated
+like a function. Notice that this mechanism works for all the
+functions of the source code, but it cannot guarantee that the order
+of the functions is preserved during migration:
+
+@smallexample
+@c file eg/network/mobag.awk
+#ReadMySelf
+/^function /                     @{ FUNC = $2 @}
+/^END/ || /^#ReadMySelf/         @{ FUNC = $1 @}
+FUNC != ""                       @{ MOBFUN[FUNC] = MOBFUN[FUNC] RS $0 @}
+(FUNC != "") && (/^@}/ || /^#EndOfMySelf/) \
+                                 @{ FUNC = "" @}
+#EndOfMySelf
+@c endfile
+@end smallexample
+
+The web server code in
+@ref{Interacting Service, ,A Web Service with Interaction},
+was first developed as a site-independent core. Likewise, the
+@command{gawk}-based mobile agent
+starts with an agent-independent core, to which can be appended
+application-dependent functions.  What follows is the only
+application-independent function needed for the mobile agent:
+
+@smallexample
+@c file eg/network/mobag.awk
+function migrate(Destination, MobCode, Label) @{
+  MOBVAR["Label"] = Label
+  MOBVAR["Destination"] = Destination
+  RS = ORS = "\r\n"
+  HttpService = "/inet/tcp/0/" Destination
+  for (i in MOBFUN)
+     MobCode = (MobCode "\n" MOBFUN[i])
+  MobCode = MobCode  "\n\nBEGIN @{"
+  for (i in MOBVAR)
+     MobCode = (MobCode "\n  MOBVAR[\"" i "\"] = \"" MOBVAR[i] "\"")
+  MobCode = MobCode "\n@}\n"
+  print "POST /cgi-bin/PostAgent.sh HTTP/1.0"  |& HttpService
+  print "Content-length:", length(MobCode) ORS |& HttpService
+  printf "%s", MobCode                         |& HttpService
+  while ((HttpService |& getline) > 0)
+     print $0
+  close(HttpService)
+@}
+@c endfile
+@end smallexample
+
+The @code{migrate} function prepares the
+aforementioned strings containing the program code and transmits them to a
+server. A consequence of this modular approach is that the @code{migrate}
+function takes some parameters that aren't needed in this application,
+but that will be in future ones. Its mandatory parameter @code{Destination} holds the
+name (or IP address) of the server that the agent wants as a host for its
+code. The optional parameter @code{MobCode} may contain some @command{gawk}
+code that is inserted during migration in front of all other code.
+The optional parameter @code{Label} may contain
+a string that tells the agent what to do in program execution after
+arrival at its new home site. One of the serious obstacles in implementing
+a framework for mobile agents is that it does not suffice to migrate the
+code. It is also necessary to migrate the state of execution of the agent. In
+contrast to @cite{Agent Tcl}, this program does not try to migrate the complete set
+of variables. The following conventions are used:
+
+@itemize @bullet
+@item
+Each variable in an agent program is local to the current host and does
+@emph{not} migrate.
+
+@item
+The array @code{MOBFUN} shown above is an exception. It is handled
+by the function @code{migrate} and does migrate with the application.
+
+@item
+The other exception is the array @code{MOBVAR}. Each variable that
+takes part in migration has to be an element of this array.
+@code{migrate} also takes care of this.
+@end itemize
+
+Now it's clear what happens to the @code{Label} parameter of the
+function @code{migrate}. It is copied into @code{MOBVAR["Label"]} and
+travels alongside the other data. Since travelling takes place via HTTP,
+records must be separated with @code{"\r\n"} in @code{RS} and
+@code{ORS} as usual. The code assembly for migration takes place in
+three steps:
+
+@itemize @bullet
+@item
+Iterate over @code{MOBFUN} to collect all functions verbatim.
+
+@item
+Prepare a @code{BEGIN} pattern and put assignments to mobile
+variables into the action part.
+
+@item
+Transmission itself resembles GETURL: the header with the request
+and the @code{Content-length} is followed by the body. In case there is
+any reply over the network, it is read completely and echoed to
+standard output to avoid irritating the server.
+@end itemize
+
+The application-independent framework is now almost complete. What follows
+is the @code{END} pattern that is executed  when the mobile agent has
+finished reading its own code. First, it checks whether it is already
+running on a remote host or not. In case initialization has not yet taken
+place, it starts @code{MyInit}. Otherwise (later, on a remote host), it
+starts @code{MyJob}:
+
+@smallexample
+@c file eg/network/mobag.awk
+END @{
+  if (ARGC != 2) exit    # stop when called with wrong parameters
+  if (MyOrigin != "")    # is this the originating host?
+    MyInit()             # if so, initialize the application
+  else                   # we are on a host with migrated data
+    MyJob()              # so we do our job
+@}
+@c endfile
+@end smallexample
+
+All that's left to extend the framework into a complete application
+is to write two application-specific functions: @code{MyInit} and
+@code{MyJob}. Keep in mind that the former is executed once on the
+originating host, while the latter is executed after each migration:
+
+@smallexample
+@c file eg/network/mobag.awk
+function MyInit() @{
+  MOBVAR["MyOrigin"] = MyOrigin
+  MOBVAR["Machines"] = "localhost/80 max/80 moritz/80 castor/80"
+  split(MOBVAR["Machines"], Machines)           # which host is the first?
+  migrate(Machines[1], "", "")                  # go to the first host
+  while (("/inet/tcp/8080/0/0" |& getline) > 0) # wait for result
+    print $0                                    # print result
+  close("/inet/tcp/8080/0/0")
+@}
+@c endfile
+@end smallexample
+
+As mentioned earlier, this agent takes the name of its origin
+(@code{MyOrigin}) with it. Then, it takes the name of its first
+destination and goes there for further work. Notice that this name has
+the port number of the web server appended to the name of the server,
+because the function @code{migrate} needs it this way to create
+the @code{HttpService} variable. Finally, it waits for the result to arrive.
+The @code{MyJob} function runs on the remote host:
+
+@smallexample
+@c file eg/network/mobag.awk
+function MyJob() @{
+  # forget this host
+  sub(MOBVAR["Destination"], "", MOBVAR["Machines"])
+  MOBVAR["Result"]=MOBVAR["Result"] SUBSEP SUBSEP MOBVAR["Destination"] ":"
+  while (("who" | getline) > 0)               # who is logged in?
+    MOBVAR["Result"] = MOBVAR["Result"] SUBSEP $0
+  close("who")
+  if (index(MOBVAR["Machines"], "/") > 0) @{   # any more machines to visit?
+    split(MOBVAR["Machines"], Machines)       # which host is next?
+    migrate(Machines[1], "", "")              # go there
+  @} else @{                                    # no more machines
+    gsub(SUBSEP, "\n", MOBVAR["Result"])      # send result to origin
+    print MOBVAR["Result"] |& "/inet/tcp/0/" MOBVAR["MyOrigin"] "/8080"
+    close("/inet/tcp/0/" MOBVAR["MyOrigin"] "/8080")
+  @}
+@}
+@c endfile
+@end smallexample
+
+After migrating, the first thing to do in @code{MyJob} is to delete
+the name of the current host from the list of hosts to visit. Now, it
+is time to start the real work by appending the host's name to the
+result string, and reading line by line who is logged in on this host.
+A very annoying circumstance is the fact that the elements of
+@code{MOBVAR} cannot hold the newline character (@code{"\n"}). If they
+did, migration of this string did not work because the string didn't
+obey the syntax rule for a string in @command{gawk}.
+@code{SUBSEP} is used as a temporary replacement.
+If the list of hosts to visit holds
+at least one more entry, the agent migrates to that place to go on
+working there. Otherwise, we replace the @code{SUBSEP}s
+with a newline character in the resulting string, and report it to
+the originating host, whose name is stored in @code{MOBVAR["MyOrigin"]}.
+
+@node STOXPRED, PROTBASE, MOBAGWHO, Some Applications and Techniques
+@section STOXPRED: Stock Market Prediction As A Service
+@cindex STOXPRED program
+@cindex Yahoo
+@quotation
+@i{Far out in the uncharted backwaters of the unfashionable end of
+the Western Spiral arm of the Galaxy lies a small unregarded yellow sun.}
+
+@i{Orbiting this at a distance of roughly ninety-two million miles is an
+utterly insignificant little blue-green planet whose ape-descendent life
+forms are so amazingly primitive that they still think digital watches are
+a pretty neat idea.}
+
+@i{This planet has --- or rather had --- a problem, which was this:
+most of the people living on it were unhappy for pretty much of the time.
+Many solutions were suggested for this problem, but most of these were
+largely concerned with the movements of small green pieces of paper,
+which is odd because it wasn't the small green pieces of paper that
+were unhappy.} @*
+Douglas Adams, @cite{The Hitch Hiker's Guide to the Galaxy}
+@end quotation
+
+@cindex @command{cron}
+Valuable services on the Internet are usually @emph{not} implemented
+as mobile agents. There are much simpler ways of implementing services.
+All Unix systems provide, for example, the @command{cron} service.
+Unix system users can write a list of tasks to be done each day, each
+week, twice a day, or just once. The list is entered into a file named
+@file{crontab}.  For example, to distribute a newsletter on a daily
+basis this way, use @command{cron} for calling a script each day early
+in the morning.
+
+@example
+# run at 8 am on weekdays, distribute the newsletter
+0 8 * * 1-5   $HOME/bin/daily.job >> $HOME/log/newsletter 2>&1
+@end example
+
+The script first looks for interesting information on the Internet,
+assembles it in a nice form and sends the results via email to
+the customers.
+
+The following is an example of a primitive
+newsletter on stock market prediction. It is a report which first
+tries to predict the change of each share in the Dow Jones Industrial
+Index for the particular day. Then it mentions some especially
+promising shares as well as some shares which look remarkably bad
+on that day. The report ends with the usual disclaimer which tells
+every child @emph{not} to try this at home and hurt anybody.
+@cindex Dow Jones Industrial Index
+
+@smallexample
+Good morning Uncle Scrooge,
+
+This is your daily stock market report for Monday, October 16, 2000.
+Here are the predictions for today:
+
+        AA      neutral
+        GE      up
+        JNJ     down
+        MSFT    neutral
+        @dots{}
+        UTX     up
+        DD      down
+        IBM     up
+        MO      down
+        WMT     up
+        DIS     up
+        INTC    up
+        MRK     down
+        XOM     down
+        EK      down
+        IP      down
+
+The most promising shares for today are these:
+
+        INTC            http://biz.yahoo.com/n/i/intc.html
+
+The stock shares to avoid today are these:
+
+        EK              http://biz.yahoo.com/n/e/ek.html
+        IP              http://biz.yahoo.com/n/i/ip.html
+        DD              http://biz.yahoo.com/n/d/dd.html
+        @dots{}
+@end smallexample
+
+@ignore
+@c Chuck suggests removing this paragraph
+If you are not into stock market prediction but want to earn money
+with a more humane service, you might prefer to send out horoscopes
+to your customers. Or, once every refrigerator in every household on this side
+of the Chinese Wall is connected to the Internet, such a service could
+inspect the contents of your customer's refrigerators each day and
+advise them on nutrition. Big Brother is watching them.
+@end ignore
+
+The script as a whole is rather long. In order to ease the pain of
+studying other people's source code, we have broken the script
+up into meaningful parts which are invoked one after the other.
+The basic structure of the script is as follows:
+
+@example
+@c file eg/network/stoxpred.awk
+BEGIN @{
+  Init()
+  ReadQuotes()
+  CleanUp()
+  Prediction()
+  Report()
+  SendMail()
+@}
+@c endfile
+@end example
+
+The earlier parts store data into variables and arrays which are
+subsequently used by later parts of the script. The @code{Init} function
+first checks if the script is invoked correctly (without any parameters).
+If not, it informs the user of the correct usage. What follows are preparations
+for the retrieval of the historical quote data. The names of the 30 stock
+shares are stored in an array @code{name} along with the current date
+in @code{day}, @code{month}, and @code{year}.
+
+All users who are separated
+from the Internet by a firewall and have to direct their Internet accesses
+to a proxy must supply the name of the proxy to this script with the
+@samp{-v Proxy=@var{name}} option. For most users, the default proxy and
+port number should suffice.
+
+@example
+@c file eg/network/stoxpred.awk
+function Init() @{
+  if (ARGC != 1) @{
+    print "STOXPRED - daily stock share prediction"
+    print "IN:\n    no parameters, nothing on stdin"
+    print "PARAM:\n    -v Proxy=MyProxy -v ProxyPort=80"
+    print "OUT:\n    commented predictions as email"
+    print "JK 09.10.2000"
+    exit
+  @}
+  # Remember ticker symbols from Dow Jones Industrial Index
+  StockCount = split("AA GE JNJ MSFT AXP GM JPM PG BA HD KO \
+    SBC C HON MCD T CAT HWP MMM UTX DD IBM MO WMT DIS INTC \
+    MRK XOM EK IP", name);
+  # Remember the current date as the end of the time series
+  day   = strftime("%d")
+  month = strftime("%m")
+  year  = strftime("%Y")
+  if (Proxy     == "")  Proxy     = "chart.yahoo.com"
+  if (ProxyPort ==  0)  ProxyPort = 80
+  YahooData = "/inet/tcp/0/" Proxy "/" ProxyPort
+@}
+@c endfile
+@end example
+
+@cindex CSV format
+There are two really interesting parts in the script. One is the
+function which reads the historical stock quotes from an Internet
+server. The other is the one that does the actual prediction. In
+the following function we see how the quotes are read from the
+Yahoo server. The data which comes from the server is in
+CSV format (comma-separated values):
+
+@example
+@c file eg/network/stoxdata.txt
+Date,Open,High,Low,Close,Volume
+9-Oct-00,22.75,22.75,21.375,22.375,7888500
+6-Oct-00,23.8125,24.9375,21.5625,22,10701100
+5-Oct-00,24.4375,24.625,23.125,23.50,5810300
+@c endfile
+@end example
+
+Lines contain values of the same time instant, whereas columns are
+separated by commas and contain the kind of data that is described
+in the header (first) line. At first, @command{gawk} is instructed to
+separate columns by commas (@samp{FS = ","}). In the loop that follows,
+a connection to the Yahoo server is first opened, then a download takes
+place, and finally the connection is closed. All this happens once for
+each ticker symbol. In the body of this loop, an Internet address is
+built up as a string according to the rules of the Yahoo server. The
+starting and ending date are chosen to be exactly the same, but one year
+apart in the past. All the action is initiated within the @code{printf}
+command which transmits the request for data to the Yahoo server.
+
+In the inner loop, the server's data is first read and then scanned
+line by line. Only lines which have six columns and the name of a month
+in the first column contain relevant data. This data is stored
+in the two-dimensional array @code{quote}; one dimension
+being time, the other being the ticker symbol. During retrieval of the
+first stock's data, the calendar names of the time instances are stored
+in the array @code{day} because we need them later.
+
+@smallexample
+@c file eg/network/stoxpred.awk
+function ReadQuotes() @{
+  # Retrieve historical data for each ticker symbol
+  FS = ","
+  for (stock = 1; stock <= StockCount; stock++) @{
+    URL = "http://chart.yahoo.com/table.csv?s=" name[stock] \
+          "&a=" month "&b=" day   "&c=" year-1 \
+          "&d=" month "&e=" day   "&f=" year \
+          "g=d&q=q&y=0&z=" name[stock] "&x=.csv"
+    printf("GET " URL " HTTP/1.0\r\n\r\n") |& YahooData
+    while ((YahooData |& getline) > 0) @{
+      if (NF == 6 && $1 ~ /Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec/) @{
+        if (stock == 1)
+          days[++daycount] = $1;
+        quote[$1, stock] = $5
+      @}
+    @}
+    close(YahooData)
+  @}
+  FS = " "
+@}
+@c endfile
+@end smallexample
+
+Now that we @emph{have} the data, it can be checked once again to make sure
+that no individual stock is missing or invalid, and that all the stock quotes are
+aligned correctly. Furthermore, we renumber the time instances. The
+most recent day gets day number 1 and all other days get consecutive
+numbers. All quotes are rounded toward the nearest whole number in US Dollars.
+
+@smallexample  
+@c file eg/network/stoxpred.awk
+function CleanUp() @{
+  # clean up time series; eliminate incomplete data sets
+  for (d = 1; d <= daycount; d++) @{
+    for (stock = 1; stock <= StockCount; stock++)
+      if (! ((days[d], stock) in quote))
+          stock = StockCount + 10
+    if (stock > StockCount + 1)
+        continue
+    datacount++
+    for (stock = 1; stock <= StockCount; stock++)
+      data[datacount, stock] = int(0.5 + quote[days[d], stock])
+  @}
+  delete quote
+  delete days
+@}
+@c endfile
+@end smallexample
+
+Now we have arrived at the second really interesting part of the whole affair.
+What we present here is a very primitive prediction algorithm:
+@emph{If a stock fell yesterday, assume it will also fall today; if
+it rose yesterday, assume it will rise today}.  (Feel free to replace this
+algorithm with a smarter one.) If a stock changed in the same direction
+on two consecutive days, this is an indication which should be highlighted.
+Two-day advances are stored in @code{hot} and two-day declines in
+@code{avoid}.
+
+The rest of the function is a sanity check. It counts the number of
+correct predictions in relation to the total number of predictions
+one could have made in the year before.
+
+@smallexample
+@c file eg/network/stoxpred.awk
+function Prediction() @{
+  # Predict each ticker symbol by prolonging yesterday's trend
+  for (stock = 1; stock <= StockCount; stock++) @{
+    if         (data[1, stock] > data[2, stock]) @{
+      predict[stock] = "up"
+    @} else if  (data[1, stock] < data[2, stock]) @{ 
+      predict[stock] = "down" 
+    @} else @{
+      predict[stock] = "neutral"
+    @}
+    if ((data[1, stock] > data[2, stock]) && (data[2, stock] > data[3, stock]))
+      hot[stock] = 1
+    if ((data[1, stock] < data[2, stock]) && (data[2, stock] < data[3, stock]))
+      avoid[stock] = 1  
+  @}
+  # Do a plausibility check: how many predictions proved correct?
+  for (s = 1; s <= StockCount; s++) @{
+    for (d = 1; d <= datacount-2; d++) @{
+      if         (data[d+1, s] > data[d+2, s]) @{
+        UpCount++
+      @} else if  (data[d+1, s] < data[d+2, s]) @{
+        DownCount++
+      @} else @{
+        NeutralCount++
+      @}   
+      if (((data[d, s]  > data[d+1, s]) && (data[d+1, s]  > data[d+2, s])) ||
+          ((data[d, s]  < data[d+1, s]) && (data[d+1, s]  < data[d+2, s])) ||
+          ((data[d, s] == data[d+1, s]) && (data[d+1, s] == data[d+2, s])))
+        CorrectCount++
+    @}   
+  @}       
+@}
+@c endfile
+@end smallexample
+
+At this point the hard work has been done: the array @code{predict}
+contains the predictions for all the ticker symbols. It is up to the
+function @code{Report} to find some nice words to introduce the
+desired information.
+
+@smallexample
+@c file eg/network/stoxpred.awk
+function Report() @{
+  # Generate report
+  report =        "\nThis is your daily "
+  report = report "stock market report for "strftime("%A, %B %d, %Y")".\n"
+  report = report "Here are the predictions for today:\n\n"
+  for (stock = 1; stock <= StockCount; stock++)  
+    report = report "\t" name[stock] "\t" predict[stock] "\n"
+  for (stock in hot) @{
+    if (HotCount++ == 0)
+      report = report "\nThe most promising shares for today are these:\n\n"
+    report = report "\t" name[stock] "\t\thttp://biz.yahoo.com/n/" \
+      tolower(substr(name[stock], 1, 1)) "/" tolower(name[stock]) ".html\n"
+  @}
+  for (stock in avoid) @{
+    if (AvoidCount++ == 0)
+      report = report "\nThe stock shares to avoid today are these:\n\n"
+    report = report "\t" name[stock] "\t\thttp://biz.yahoo.com/n/" \
+      tolower(substr(name[stock], 1, 1)) "/" tolower(name[stock]) ".html\n"
+  @}   
+  report = report "\nThis sums up to " HotCount+0 " winners and " AvoidCount+0
+  report = report " losers. When using this kind\nof prediction scheme for"
+  report = report " the 12 months which lie behind us,\nwe get " UpCount
+  report = report " 'ups' and " DownCount " 'downs' and " NeutralCount
+  report = report " 'neutrals'. Of all\nthese " UpCount+DownCount+NeutralCount
+  report = report " predictions " CorrectCount " proved correct next day.\n"
+  report = report "A success rate of "\
+             int(100*CorrectCount/(UpCount+DownCount+NeutralCount)) "%.\n"
+  report = report "Random choice would have produced a 33% success rate.\n"
+  report = report "Disclaimer: Like every other prediction of the stock\n"
+  report = report "market, this report is, of course, complete nonsense.\n"
+  report = report "If you are stupid enough to believe these predictions\n"
+  report = report "you should visit a doctor who can treat your ailment."
+@}     
+@c endfile
+@end smallexample
+
+The function @code{SendMail} goes through the list of customers and opens
+a pipe to the @code{mail} command for each of them. Each one receives an
+email message with a proper subject heading and is addressed with his full name.
+
+@smallexample
+@c file eg/network/stoxpred.awk
+function SendMail() @{ 
+  # send report to customers
+  customer["uncle.scrooge@@ducktown.gov"] = "Uncle Scrooge"
+  customer["more@@utopia.org"           ] = "Sir Thomas More"
+  customer["spinoza@@denhaag.nl"        ] = "Baruch de Spinoza"
+  customer["marx@@highgate.uk"          ] = "Karl Marx"
+  customer["keynes@@the.long.run"       ] = "John Maynard Keynes"
+  customer["bierce@@devil.hell.org"     ] = "Ambrose Bierce"
+  customer["laplace@@paris.fr"          ] = "Pierre Simon de Laplace"
+  for (c in customer) @{
+    MailPipe = "mail -s 'Daily Stock Prediction Newsletter'" c
+    print "Good morning " customer[c] "," | MailPipe
+    print report "\n.\n" | MailPipe
+    close(MailPipe)
+  @}
+@}   
+@c endfile
+@end smallexample
+
+Be patient when running the script by hand.
+Retrieving the data for all the ticker symbols and sending the emails
+may take several minutes to complete, depending upon network traffic
+and the speed of the available Internet link.
+The quality of the prediction algorithm is likely to be disappointing.
+Try to find a better one.
+Should you find one with a success rate of more than 50%, please tell
+us about it! It is only for the sake of curiosity, of course. @code{:-)}
+
+@ignore
+@c chuck says to remove this
+Let us give you one final indication as to what one can expect from
+a prediction of stock data, which is sometimes said to contain much
+randomness. One theory says that all relevant information to be taken
+into account when estimating the price of a stock is contained in the
+stock quotes. Every bit of useful information has influenced the
+fair price. Therefore (the theory says) temporary changes (i.e., fluctuations
+within a minute) have to be purely random. But what is the cause of
+short-term changes in stock prices?
+
+Stock prices are fixed when supply and demand meet each other.
+What people are willing to pay reflects human expectations.
+Human expectations are not necessarily random. On the Internet,
+you can find an elucidating paper about predictability and human
+expectations: 
+@uref{http://it.ucsd.edu/IT/Newsletter/archives/meir/05meir.html,
+@cite{Reflections on ``Universal Prediction of Individual Sequences''}}
+The authors (Feder, Merhav, Gutman) introduce the reader to the subject
+by telling a thrilling anecdote.
+@cindex Shannon, Claude
+@quotation
+In the early 50's, at Bell Laboratories, David Hagelbarger built a
+simple ``mind reading'' machine, whose purpose was to play the ``penny
+matching'' game. In this game, a player chooses head or tail, while a
+``mind reading'' machine tries to predict and match his choice.
+Surprisingly, as Robert Lucky tells in his book ``Silicon Dreams'',
+Hagelbarger's simple, 8-state machine, was able to match the ``pennies''
+of its human opponent 5,218 times over the course of 9,795 plays.
+Random guessing would lead to such a high success rate with a probability
+less than one out of 10 billion! Shannon, who was interested in prediction,
+information, and thinking machines, closely followed Hagelbarger's
+machine, and eventually built his own stripped-down version of the machine,
+having the same states, but one that used a simpler strategy at each state.
+As the legend goes, in a duel between the two machines, Shannon's machine
+won by a slight margin! No one knows if this was due to a superior algorithm
+or just a chance happening associated with the specific sequence at that game.
+In any event, the success of both these machines against ``untrained'' human
+opponents was explained by the fact that the human opponents cannot draw
+completely random
+bits.
+@end quotation 
+@end ignore
+
+@node PROTBASE, , STOXPRED, Some Applications and Techniques
+@section PROTBASE: Searching Through A Protein Database
+@cindex PROTBASE
+@cindex NCBI, National Center for Biotechnology Information
+@cindex BLAST, Basic Local Alignment Search Tool
+@cindex Hoare, C.A.R.
+@quotation
+@i{Hoare's Law of Large Problems: Inside every large problem is a small
+   problem struggling to get out.}
+@end quotation
+
+Yahoo's database of stock market data is just one among the many large
+databases on the Internet. Another one is located at NCBI
+(National Center for Biotechnology
+Information). Established in 1988 as a national resource for molecular
+biology information, NCBI creates public databases, conducts research
+in computational biology, develops software tools for analyzing genome
+data, and disseminates biomedical information. In this section, we
+look at one of NCBI's public services, which is called BLAST
+(Basic Local Alignment Search Tool).
+
+You probably know that the information necessary for reproducing living
+cells is encoded in the genetic material of the cells. The genetic material
+is a very long chain of four base nucleotides. It is the order of
+appearance (the sequence) of nucleotides which contains the information
+about the substance to be produced. Scientists in biotechnology often
+find a specific fragment, determine the nucleotide sequence, and need
+to know where the sequence at hand comes from. This is where the large
+databases enter the game. At NCBI, databases store the knowledge
+about which sequences have ever been found and where they have been found.
+When the scientist sends his sequence to the BLAST service, the server
+looks for regions of genetic material in its database which
+look the most similar to the delivered nucleotide sequence. After a
+search time of some seconds or minutes the server sends an answer to
+the scientist. In order to make access simple, NCBI chose to offer
+their database service through popular Internet protocols. There are
+four basic ways to use the so-called BLAST services:
+
+@itemize @bullet
+@item
+The easiest way to use BLAST is through the web. Users may simply point
+their browsers at the NCBI home page
+and link to the BLAST pages.
+NCBI provides a stable URL that may be used to perform BLAST searches
+without interactive use of a web browser. This is what we will do later
+in this section.
+A demonstration client
+and a @file{README} file demonstrate how to access this URL.
+
+@item
+Currently,
+@command{blastcl3} is the standard network BLAST client.
+You can download @command{blastcl3} from the
+anonymous FTP location.
+
+@item
+BLAST 2.0 can be run locally as a full executable and can be used to run
+BLAST searches against private local databases, or downloaded copies of the
+NCBI databases. BLAST 2.0 executables may be found on the NCBI
+anonymous FTP server.
+
+@item
+The NCBI BLAST Email server is the best option for people without convenient
+access to the web. A similarity search can be performed by sending a properly
+formatted mail message containing the nucleotide or protein query sequence to
+@email{blast@@ncbi.nlm.nih.gov}. The query sequence is compared against the
+specified database using the BLAST algorithm and the results are returned in
+an email message. For more information on formulating email BLAST searches,
+you can send a message consisting of the word ``HELP'' to the same address,
+@email{blast@@ncbi.nlm.nih.gov}.
+@end itemize
+
+Our starting point is the demonstration client mentioned in the first option.
+The @file{README} file that comes along with the client explains the whole
+process in a nutshell. In the rest of this section, we first show
+what such requests look like. Then we show how to use @command{gawk} to
+implement a client in about 10 lines of code. Finally, we show how to
+interpret the result returned from the service.
+
+Sequences are expected to be represented in the standard
+IUB/IUPAC amino acid and nucleic acid codes,
+with these exceptions:  lower-case letters are accepted and are mapped
+into upper-case; a single hyphen or dash can be used to represent a gap
+of indeterminate length; and in amino acid sequences, @samp{U} and @samp{*}  
+are acceptable letters (see below).  Before submitting a request, any numerical
+digits in the query sequence should either be removed or replaced by   
+appropriate letter codes (e.g., @samp{N} for unknown nucleic acid residue
+or @samp{X} for unknown amino acid residue).
+The nucleic acid codes supported are: 
+
+@example
+A --> adenosine               M --> A C (amino)
+C --> cytidine                S --> G C (strong)
+G --> guanine                 W --> A T (weak)
+T --> thymidine               B --> G T C
+U --> uridine                 D --> G A T
+R --> G A (purine)            H --> A C T
+Y --> T C (pyrimidine)        V --> G C A
+K --> G T (keto)              N --> A G C T (any)
+                              -  gap of indeterminate length       
+@end example
+
+Now you know the alphabet of nucleotide sequences. The last two lines
+of the following example query show you such a sequence, which is obviously
+made up only of elements of the alphabet just described. Store this example
+query into a file named @file{protbase.request}. You are now ready to send
+it to the server with the demonstration client.
+  
+@example
+@c file eg/network/protbase.request
+PROGRAM blastn
+DATALIB month
+EXPECT 0.75
+BEGIN
+>GAWK310 the gawking gene GNU AWK
+tgcttggctgaggagccataggacgagagcttcctggtgaagtgtgtttcttgaaatcat
+caccaccatggacagcaaa
+@c endfile 
+@end example
+
+@cindex FASTA/Pearson format
+The actual search request begins with the mandatory parameter @samp{PROGRAM}
+in the first column followed by the value @samp{blastn} (the name of the
+program) for searching nucleic acids.  The next line contains the mandatory
+search parameter @samp{DATALIB} with the value @samp{month} for the newest
+nucleic acid sequences.  The third line contains an optional @samp{EXPECT}
+parameter and the value desired for it. The fourth line contains the
+mandatory @samp{BEGIN} directive, followed by the query sequence in
+FASTA/Pearson format.
+Each line of information must be less than 80 characters in length.
+
+The ``month'' database contains all new or revised sequences released in the 
+last 30 days and is useful for searching against new sequences.
+There are five different blast programs, @command{blastn} being the one that
+compares a nucleotide  query  sequence  against a nucleotide sequence database.
+
+The last server directive that must appear in every request is the
+@samp{BEGIN} directive. The query sequence should immediately follow the
+@samp{BEGIN} directive and must appear in FASTA/Pearson format.
+A sequence in
+FASTA/Pearson format begins with a single-line description.
+The description line, which is required, is distinguished from the lines of
+sequence data that follow it by having a greater-than (@samp{>}) symbol
+in the first column.  For the purposes of the BLAST server, the text of
+the description is arbitrary.
+
+If you prefer to use a client written in @command{gawk}, just store the following
+10 lines of code into a file named @file{protbase.awk} and use this client
+instead. Invoke it with @samp{gawk -f protbase.awk protbase.request}.
+Then wait a minute and watch the result coming in. In order to replicate
+the demonstration client's behaviour as closely as possible, this client
+does not use a proxy server. We could also have extended the client program
+in @ref{GETURL, ,Retrieving Web Pages}, to implement the client request from
+@file{protbase.awk} as a special case.
+
+@smallexample
+@c file eg/network/protbase.awk
+@{ request = request "\n" $0 @}
+
+END @{
+  BLASTService     = "/inet/tcp/0/www.ncbi.nlm.nih.gov/80"
+  printf "POST /cgi-bin/BLAST/nph-blast_report HTTP/1.0\n" |& BLASTService
+  printf "Content-Length: " length(request) "\n\n"         |& BLASTService
+  printf request                                           |& BLASTService      
+  while ((BLASTService |& getline) > 0)
+      print $0
+  close(BLASTService)
+@}
+@c endfile
+@end smallexample
+
+The demonstration client from NCBI is 214 lines long (written in C) and
+it is not immediately obvious what it does. Our client is so short that
+it @emph{is} obvious what it does. First it loops over all lines of the
+query and stores the whole query into a variable. Then the script
+establishes an Internet connection to the NCBI server and transmits the
+query by framing it with a proper HTTP request. Finally it receives
+and prints the complete result coming from the server.
+
+Now, let us look at the result. It begins with an HTTP header, which you
+can ignore. Then there are some comments about the query having been
+filtered to avoid spuriously high scores. After this, there is a reference
+to the paper that describes the software being used for searching the data
+base. After a repitition of the original query's description we find the
+list of significant alignments:
+
+@smallexample
+@c file eg/network/protbase.result
+Sequences producing significant alignments:                        (bits)  Value
+
+gb|AC021182.14|AC021182 Homo sapiens chromosome 7 clone RP11-733...    38  0.20
+gb|AC021056.12|AC021056 Homo sapiens chromosome 3 clone RP11-115...    38  0.20
+emb|AL160278.10|AL160278 Homo sapiens chromosome 9 clone RP11-57...    38  0.20
+emb|AL391139.11|AL391139 Homo sapiens chromosome X clone RP11-35...    38  0.20
+emb|AL365192.6|AL365192 Homo sapiens chromosome 6 clone RP3-421H...    38  0.20
+emb|AL138812.9|AL138812 Homo sapiens chromosome 11 clone RP1-276...    38  0.20
+gb|AC073881.3|AC073881 Homo sapiens chromosome 15 clone CTD-2169...    38  0.20
+@c endfile
+@end smallexample
+
+This means that the query sequence was found in seven human chromosomes.
+But the value 0.20 (20%) means that the probability of an accidental match
+is rather high (20%) in all cases and should be taken into account.
+You may wonder what the first column means. It is a key to the specific
+database in which this occurence was found.  The unique sequence identifiers
+reported in the search results can be used as sequence retrieval keys
+via the NCBI server. The syntax of sequence header lines used by the NCBI
+BLAST server depends on the database from which each sequence was obtained.
+The table below lists the identifiers for the databases from which the
+sequences were derived.
+ 
+@ifinfo
+@example
+Database Name                     Identifier Syntax
+============================      ========================
+GenBank                           gb|accession|locus
+EMBL Data Library                 emb|accession|locus
+DDBJ, DNA Database of Japan       dbj|accession|locus
+NBRF PIR                          pir||entry
+Protein Research Foundation       prf||name
+SWISS-PROT                        sp|accession|entry name
+Brookhaven Protein Data Bank      pdb|entry|chain
+Kabat's Sequences of Immuno@dots{}    gnl|kabat|identifier
+Patents                           pat|country|number 
+GenInfo Backbone Id               bbs|number 
+@end example
+@end ifinfo
+
+@ifnotinfo
+@multitable {Kabat's Sequences of Immuno@dots{}} {@code{@w{sp|accession|entry name}}}
+@item GenBank @tab @code{gb|accession|locus}
+@item EMBL Data Library @tab @code{emb|accession|locus}
+@item DDBJ, DNA Database of Japan @tab @code{dbj|accession|locus}
+@item NBRF PIR @tab @code{pir||entry}
+@item Protein Research Foundation @tab @code{prf||name}
+@item SWISS-PROT @tab @code{@w{sp|accession|entry name}}
+@item Brookhaven Protein Data Bank @tab @code{pdb|entry|chain}
+@item Kabat's Sequences of Immuno@dots{} @tab @code{gnl|kabat|identifier}
+@item Patents @tab @code{pat|country|number}
+@item GenInfo Backbone Id @tab @code{bbs|number}
+@end multitable
+@end ifnotinfo
+
+ 
+For example, an identifier might be @samp{gb|AC021182.14|AC021182}, where the
+@samp{gb} tag indicates that the identifier refers to a GenBank sequence,
+@samp{AC021182.14} is its GenBank ACCESSION, and @samp{AC021182} is the GenBank LOCUS.
+The identifier contains no spaces, so that a space indicates the end of the
+identifier.
+
+Let us continue in the result listing. Each of the seven alignments mentioned
+above is subsequently described in detail. We will have a closer look at
+the first of them.
+
+@smallexample
+>gb|AC021182.14|AC021182 Homo sapiens chromosome 7 clone RP11-733N23, WORKING DRAFT SEQUENCE, 4
+             unordered pieces
+          Length = 176383
+
+ Score = 38.2 bits (19), Expect = 0.20
+ Identities = 19/19 (100%)
+ Strand = Plus / Plus
+
+Query: 35    tggtgaagtgtgtttcttg 53
+             |||||||||||||||||||
+Sbjct: 69786 tggtgaagtgtgtttcttg 69804
+@end smallexample 
+
+This alignment was located on the human chromosome 7. The fragment on which
+part of the query was found had a total length of 176383. Only 19 of the
+nucleotides matched and the matching sequence ran from character 35 to 53
+in the query sequence and from 69786 to 69804 in the fragment on chromosome 7.
+If you are still reading at this point, you are probably interested in finding
+out more about Computational Biology and you might appreciate the following
+hints.
+
+@cindex Computational Biology
+@cindex Bioinformatics
+@enumerate
+@item
+There is a book called @cite{Introduction to Computational Biology}
+by Michael S. Waterman, which is worth reading if you are seriously
+interested. You can find a good
+book review
+on the Internet.
+
+@item
+While Waterman's book can explain to you the algorithms employed internally
+in the database search engines, most practicioners prefer to approach
+the subject differently. The applied side of Computational Biology is
+called Bioinformatics, and emphasizes the tools available for day-to-day
+work as well as how to actually @emph{use} them. One of the very few affordable
+books on Bioinformatics is
+@cite{Developing Bioinformatics Computer Skills}.
+
+@item
+The sequences @emph{gawk} and @emph{gnuawk} are in widespread use in
+the genetic material of virtually every earthly living being. Let us
+take this as a clear indication that the divine creator has intended
+@code{gawk} to prevail over other scripting languages such as @code{perl},
+@code{tcl}, or @code{python} which are not even proper sequences. (:-)
+@end enumerate
+
+@node Links, GNU Free Documentation License, Some Applications and Techniques, Top
+@chapter Related Links
+
+This section lists the URLs for various items discussed in this @value{CHAPTER}.
+They are presented in the order in which they appear.
+
+@table @asis
+
+@item @cite{Internet Programming with Python}
+@uref{http://www.fsbassociates.com/books/python.htm}
+
+@item @cite{Advanced Perl Programming}
+@uref{http://www.oreilly.com/catalog/advperl}
+
+@item @cite{Web Client Programming with Perl}
+@uref{http://www.oreilly.com/catalog/webclient}
+
+@item Richard Stevens's home page and book
+@uref{http://www.kohala.com/~rstevens}
+
+@item The SPAK home page
+@uref{http://www.userfriendly.net/linux/RPM/contrib/libc6/i386/spak-0.6b-1.i386.html}
+
+@item Volume III of @cite{Internetworking with TCP/IP}, by Comer and Stevens
+@uref{http://www.cs.purdue.edu/homes/dec/tcpip3s.cont.html}
+
+@item XBM Graphics File Format
+@uref{http://www.wotsit.org/download.asp?f=xbm}
+
+@item GNUPlot
+@uref{http://www.cs.dartmouth.edu/gnuplot_info.html}
+
+@item Mark Humphrys' Eliza page 
+@uref{http://www.compapp.dcu.ie/~humphrys/eliza.html}
+
+@item Yahoo! Eliza Information 
+@uref{http://dir.yahoo.com/Recreation/Games/Computer_Games/Internet_Games/Web_Games/Artificial_Intelligence}
+
+@item Java versions of Eliza 
+@uref{http://www.tjhsst.edu/Psych/ch1/eliza.html}
+
+@item Java versions of Eliza with source code 
+@uref{http://home.adelphia.net/~lifeisgood/eliza/eliza.htm}
+
+@item Eliza Programs with Explanations 
+@uref{http://chayden.net/chayden/eliza/Eliza.shtml}
+
+@item Loebner Contest
+@uref{http://acm.org/~loebner/loebner-prize.htmlx}
+
+@item Tck/Tk Information 
+@uref{http://www.scriptics.com/}
+
+@item Intel 80x86 Processors
+@uref{http://developer.intel.com/design/platform/embedpc/what_is.htm}
+
+@item AMD Elan Processors
+@uref{http://www.amd.com/products/epd/processors/4.32bitcont/32bitcont/index.html}
+
+@item XINU 
+@uref{http://willow.canberra.edu.au/~chrisc/xinu.html }
+
+@item GNU/Linux
+@uref{http://uclinux.lineo.com/}
+
+@item Embedded PCs
+@uref{http://dir.yahoo.com/Business_and_Economy/Business_to_Business/Computers/Hardware/Embedded_Control/}
+
+@item MiniSQL 
+@uref{http://www.hughes.com.au/library/}
+
+@item Market Share Surveys
+@uref{http://www.netcraft.com/survey}
+
+@item @cite{Numerical Recipes in C: The Art of Scientific Computing}
+@uref{http://www.nr.com}
+
+@item VRML
+@uref{http://www.vrml.org}
+
+@item The VRML FAQ
+@uref{http://www.vrml.org/technicalinfo/specifications/specifications.htm#FAQ}
+
+@item The UMBC Agent Web 
+@uref{http://www.cs.umbc.edu/agents }
+
+@item Apache Web Server
+@uref{http://www.apache.org}
+
+@item National Center for Biotechnology Information (NCBI)
+@uref{http://www.ncbi.nlm.nih.gov}
+
+@item Basic Local Alignment Search Tool (BLAST)
+@uref{http://www.ncbi.nlm.nih.gov/BLAST/blast_overview.html}
+
+@item NCBI Home Page
+@uref{http://www.ncbi.nlm.nih.gov}
+
+@item BLAST Pages
+@uref{http://www.ncbi.nlm.nih.gov/BLAST}
+
+@item BLAST Demonstration Client
+@uref{ftp://ncbi.nlm.nih.gov/blast/blasturl/}
+
+@item BLAST anonymous FTP location
+@uref{ftp://ncbi.nlm.nih.gov/blast/network/netblast/}
+
+@item BLAST 2.0 Executables
+@uref{ftp://ncbi.nlm.nih.gov/blast/executables/}
+
+@item IUB/IUPAC Amino Acid and Nucleic Acid Codes
+@uref{http://www.uthscsa.edu/geninfo/blastmail.html#item6}
+
+@item FASTA/Pearson Format
+@uref{http://www.ncbi.nlm.nih.gov/BLAST/fasta.html}
+
+@item Fasta/Pearson Sequence in Java
+@uref{http://www.kazusa.or.jp/java/codon_table_java/}
+
+@item Book Review of @cite{Introduction to Computational Biology}
+@uref{http://www.acm.org/crossroads/xrds5-1/introcb.html}
+
+@item @cite{Developing Bioinformatics Computer Skills}
+@uref{http://www.oreilly.com/catalog/bioskills/}
+
+@end table
+
+@node GNU Free Documentation License, Index, Links, Top
+@unnumbered GNU Free Documentation License
+@center Version 1.1, March 2000
+
+@display
+Copyright (C) 2000  Free Software Foundation, Inc.
+59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+Everyone is permitted to copy and distribute verbatim copies
+of this license document, but changing it is not allowed.
+@end display
+@sp 1
+@enumerate 0
+@item
+PREAMBLE
+
+The purpose of this License is to make a manual, textbook, or other
+written document ``free'' in the sense of freedom: to assure everyone
+the effective freedom to copy and redistribute it, with or without
+modifying it, either commercially or noncommercially.  Secondarily,
+this License preserves for the author and publisher a way to get
+credit for their work, while not being considered responsible for
+modifications made by others.
+
+This License is a kind of ``copyleft'', which means that derivative
+works of the document must themselves be free in the same sense.  It
+complements the GNU General Public License, which is a copyleft
+license designed for free software.
+
+We have designed this License in order to use it for manuals for free
+software, because free software needs free documentation: a free
+program should come with manuals providing the same freedoms that the
+software does.  But this License is not limited to software manuals;
+it can be used for any textual work, regardless of subject matter or
+whether it is published as a printed book.  We recommend this License
+principally for works whose purpose is instruction or reference.
+
+@sp 1
+@item
+APPLICABILITY AND DEFINITIONS
+
+This License applies to any manual or other work that contains a
+notice placed by the copyright holder saying it can be distributed
+under the terms of this License.  The ``Document'', below, refers to any
+such manual or work.  Any member of the public is a licensee, and is
+addressed as ``you''.
+
+A ``Modified Version'' of the Document means any work containing the
+Document or a portion of it, either copied verbatim, or with
+modifications and/or translated into another language.
+
+A ``Secondary Section'' is a named appendix or a front-matter section of
+the Document that deals exclusively with the relationship of the
+publishers or authors of the Document to the Document's overall subject
+(or to related matters) and contains nothing that could fall directly
+within that overall subject.  (For example, if the Document is in part a
+textbook of mathematics, a Secondary Section may not explain any
+mathematics.)  The relationship could be a matter of historical
+connection with the subject or with related matters, or of legal,
+commercial, philosophical, ethical or political position regarding
+them.
+
+The ``Invariant Sections'' are certain Secondary Sections whose titles
+are designated, as being those of Invariant Sections, in the notice
+that says that the Document is released under this License.
+
+The ``Cover Texts'' are certain short passages of text that are listed,
+as Front-Cover Texts or Back-Cover Texts, in the notice that says that
+the Document is released under this License.
+
+A ``Transparent'' copy of the Document means a machine-readable copy,
+represented in a format whose specification is available to the
+general public, whose contents can be viewed and edited directly and
+straightforwardly with generic text editors or (for images composed of
+pixels) generic paint programs or (for drawings) some widely available
+drawing editor, and that is suitable for input to text formatters or
+for automatic translation to a variety of formats suitable for input
+to text formatters.  A copy made in an otherwise Transparent file
+format whose markup has been designed to thwart or discourage
+subsequent modification by readers is not Transparent.  A copy that is
+not ``Transparent'' is called ``Opaque''.
+
+Examples of suitable formats for Transparent copies include plain
+ASCII without markup, Texinfo input format, LaTeX input format, SGML
+or XML using a publicly available DTD, and standard-conforming simple
+HTML designed for human modification.  Opaque formats include
+PostScript, PDF, proprietary formats that can be read and edited only
+by proprietary word processors, SGML or XML for which the DTD and/or
+processing tools are not generally available, and the
+machine-generated HTML produced by some word processors for output
+purposes only.
+
+The ``Title Page'' means, for a printed book, the title page itself,
+plus such following pages as are needed to hold, legibly, the material
+this License requires to appear in the title page.  For works in
+formats which do not have any title page as such, ``Title Page'' means
+the text near the most prominent appearance of the work's title,
+preceding the beginning of the body of the text.
+@sp 1
+@item
+VERBATIM COPYING
+
+You may copy and distribute the Document in any medium, either
+commercially or noncommercially, provided that this License, the
+copyright notices, and the license notice saying this License applies
+to the Document are reproduced in all copies, and that you add no other
+conditions whatsoever to those of this License.  You may not use
+technical measures to obstruct or control the reading or further
+copying of the copies you make or distribute.  However, you may accept
+compensation in exchange for copies.  If you distribute a large enough
+number of copies you must also follow the conditions in section 3.
+
+You may also lend copies, under the same conditions stated above, and
+you may publicly display copies.
+@sp 1
+@item
+COPYING IN QUANTITY
+
+If you publish printed copies of the Document numbering more than 100,
+and the Document's license notice requires Cover Texts, you must enclose
+the copies in covers that carry, clearly and legibly, all these Cover
+Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on
+the back cover.  Both covers must also clearly and legibly identify
+you as the publisher of these copies.  The front cover must present
+the full title with all words of the title equally prominent and
+visible.  You may add other material on the covers in addition.
+Copying with changes limited to the covers, as long as they preserve
+the title of the Document and satisfy these conditions, can be treated
+as verbatim copying in other respects.
+
+If the required texts for either cover are too voluminous to fit
+legibly, you should put the first ones listed (as many as fit
+reasonably) on the actual cover, and continue the rest onto adjacent
+pages.
+
+If you publish or distribute Opaque copies of the Document numbering
+more than 100, you must either include a machine-readable Transparent
+copy along with each Opaque copy, or state in or with each Opaque copy
+a publicly-accessible computer-network location containing a complete
+Transparent copy of the Document, free of added material, which the
+general network-using public has access to download anonymously at no
+charge using public-standard network protocols.  If you use the latter
+option, you must take reasonably prudent steps, when you begin
+distribution of Opaque copies in quantity, to ensure that this
+Transparent copy will remain thus accessible at the stated location
+until at least one year after the last time you distribute an Opaque
+copy (directly or through your agents or retailers) of that edition to
+the public.
+
+It is requested, but not required, that you contact the authors of the
+Document well before redistributing any large number of copies, to give
+them a chance to provide you with an updated version of the Document.
+@sp 1
+@item
+MODIFICATIONS
+
+You may copy and distribute a Modified Version of the Document under
+the conditions of sections 2 and 3 above, provided that you release
+the Modified Version under precisely this License, with the Modified
+Version filling the role of the Document, thus licensing distribution
+and modification of the Modified Version to whoever possesses a copy
+of it.  In addition, you must do these things in the Modified Version:
+
+@enumerate A
+@item
+Use in the Title Page (and on the covers, if any) a title distinct
+from that of the Document, and from those of previous versions
+(which should, if there were any, be listed in the History section
+of the Document).  You may use the same title as a previous version
+if the original publisher of that version gives permission.
+
+@item
+List on the Title Page, as authors, one or more persons or entities
+responsible for authorship of the modifications in the Modified
+Version, together with at least five of the principal authors of the
+Document (all of its principal authors, if it has less than five).
+
+@item
+State on the Title page the name of the publisher of the
+Modified Version, as the publisher.
+
+@item
+Preserve all the copyright notices of the Document.
+
+@item
+Add an appropriate copyright notice for your modifications
+adjacent to the other copyright notices.
+
+@item
+Include, immediately after the copyright notices, a license notice
+giving the public permission to use the Modified Version under the
+terms of this License, in the form shown in the Addendum below.
+
+@item
+Preserve in that license notice the full lists of Invariant Sections
+and required Cover Texts given in the Document's license notice.
+
+@item
+Include an unaltered copy of this License.
+
+@item
+Preserve the section entitled ``History'', and its title, and add to
+it an item stating at least the title, year, new authors, and
+publisher of the Modified Version as given on the Title Page.  If
+there is no section entitled ``History'' in the Document, create one
+stating the title, year, authors, and publisher of the Document as
+given on its Title Page, then add an item describing the Modified
+Version as stated in the previous sentence.
+
+@item
+Preserve the network location, if any, given in the Document for
+public access to a Transparent copy of the Document, and likewise
+the network locations given in the Document for previous versions
+it was based on.  These may be placed in the ``History'' section.
+You may omit a network location for a work that was published at
+least four years before the Document itself, or if the original
+publisher of the version it refers to gives permission.
+
+@item
+In any section entitled ``Acknowledgements'' or ``Dedications'',
+preserve the section's title, and preserve in the section all the
+substance and tone of each of the contributor acknowledgements
+and/or dedications given therein.
+
+@item
+Preserve all the Invariant Sections of the Document,
+unaltered in their text and in their titles.  Section numbers
+or the equivalent are not considered part of the section titles.
+
+@item
+Delete any section entitled ``Endorsements''.  Such a section
+may not be included in the Modified Version.
+
+@item
+Do not retitle any existing section as ``Endorsements''
+or to conflict in title with any Invariant Section.
+@end enumerate
+
+If the Modified Version includes new front-matter sections or
+appendices that qualify as Secondary Sections and contain no material
+copied from the Document, you may at your option designate some or all
+of these sections as invariant.  To do this, add their titles to the
+list of Invariant Sections in the Modified Version's license notice.
+These titles must be distinct from any other section titles.
+
+You may add a section entitled ``Endorsements'', provided it contains
+nothing but endorsements of your Modified Version by various
+parties--for example, statements of peer review or that the text has
+been approved by an organization as the authoritative definition of a
+standard.
+
+You may add a passage of up to five words as a Front-Cover Text, and a
+passage of up to 25 words as a Back-Cover Text, to the end of the list
+of Cover Texts in the Modified Version.  Only one passage of
+Front-Cover Text and one of Back-Cover Text may be added by (or
+through arrangements made by) any one entity.  If the Document already
+includes a cover text for the same cover, previously added by you or
+by arrangement made by the same entity you are acting on behalf of,
+you may not add another; but you may replace the old one, on explicit
+permission from the previous publisher that added the old one.
+
+The author(s) and publisher(s) of the Document do not by this License
+give permission to use their names for publicity for or to assert or
+imply endorsement of any Modified Version.
+@sp 1
+@item
+COMBINING DOCUMENTS
+
+You may combine the Document with other documents released under this
+License, under the terms defined in section 4 above for modified
+versions, provided that you include in the combination all of the
+Invariant Sections of all of the original documents, unmodified, and
+list them all as Invariant Sections of your combined work in its
+license notice.
+
+The combined work need only contain one copy of this License, and
+multiple identical Invariant Sections may be replaced with a single
+copy.  If there are multiple Invariant Sections with the same name but
+different contents, make the title of each such section unique by
+adding at the end of it, in parentheses, the name of the original
+author or publisher of that section if known, or else a unique number.
+Make the same adjustment to the section titles in the list of
+Invariant Sections in the license notice of the combined work.
+
+In the combination, you must combine any sections entitled ``History''
+in the various original documents, forming one section entitled
+``History''; likewise combine any sections entitled ``Acknowledgements'',
+and any sections entitled ``Dedications''.  You must delete all sections
+entitled ``Endorsements.''
+@sp 1
+@item
+COLLECTIONS OF DOCUMENTS
+
+You may make a collection consisting of the Document and other documents
+released under this License, and replace the individual copies of this
+License in the various documents with a single copy that is included in
+the collection, provided that you follow the rules of this License for
+verbatim copying of each of the documents in all other respects.
+
+You may extract a single document from such a collection, and distribute
+it individually under this License, provided you insert a copy of this
+License into the extracted document, and follow this License in all
+other respects regarding verbatim copying of that document.
+@sp 1
+@item
+AGGREGATION WITH INDEPENDENT WORKS
+
+A compilation of the Document or its derivatives with other separate
+and independent documents or works, in or on a volume of a storage or
+distribution medium, does not as a whole count as a Modified Version
+of the Document, provided no compilation copyright is claimed for the
+compilation.  Such a compilation is called an ``aggregate'', and this
+License does not apply to the other self-contained works thus compiled
+with the Document, on account of their being thus compiled, if they
+are not themselves derivative works of the Document.
+
+If the Cover Text requirement of section 3 is applicable to these
+copies of the Document, then if the Document is less than one quarter
+of the entire aggregate, the Document's Cover Texts may be placed on
+covers that surround only the Document within the aggregate.
+Otherwise they must appear on covers around the whole aggregate.
+@sp 1
+@item
+TRANSLATION
+
+Translation is considered a kind of modification, so you may
+distribute translations of the Document under the terms of section 4.
+Replacing Invariant Sections with translations requires special
+permission from their copyright holders, but you may include
+translations of some or all Invariant Sections in addition to the
+original versions of these Invariant Sections.  You may include a
+translation of this License provided that you also include the
+original English version of this License.  In case of a disagreement
+between the translation and the original English version of this
+License, the original English version will prevail.
+@sp 1
+@item
+TERMINATION
+
+You may not copy, modify, sublicense, or distribute the Document except
+as expressly provided for under this License.  Any other attempt to
+copy, modify, sublicense or distribute the Document is void, and will
+automatically terminate your rights under this License.  However,
+parties who have received copies, or rights, from you under this
+License will not have their licenses terminated so long as such
+parties remain in full compliance.
+@sp 1
+@item
+FUTURE REVISIONS OF THIS LICENSE
+
+The Free Software Foundation may publish new, revised versions
+of the GNU Free Documentation License from time to time.  Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.  See
+@uref{http://www.gnu.org/copyleft/}.
+
+Each version of the License is given a distinguishing version number.
+If the Document specifies that a particular numbered version of this
+License ``or any later version'' applies to it, you have the option of
+following the terms and conditions either of that specified version or
+of any later version that has been published (not as a draft) by the
+Free Software Foundation.  If the Document does not specify a version
+number of this License, you may choose any version ever published (not
+as a draft) by the Free Software Foundation.
+
+@end enumerate
+
+@c fakenode --- for prepinfo
+@unnumberedsec ADDENDUM: How to use this License for your documents
+
+To use this License in a document you have written, include a copy of
+the License in the document and put the following copyright and
+license notices just after the title page:
+
+@smallexample
+@group
+
+  Copyright (C)  @var{year}  @var{your name}.
+  Permission is granted to copy, distribute and/or modify this document
+  under the terms of the GNU Free Documentation License, Version 1.1
+  or any later version published by the Free Software Foundation;
+  with the Invariant Sections being @var{list their titles}, with the
+  Front-Cover Texts being @var{list}, and with the Back-Cover Texts being @var{list}.
+  A copy of the license is included in the section entitled ``GNU
+  Free Documentation License''.
+@end group
+@end smallexample
+If you have no Invariant Sections, write ``with no Invariant Sections''
+instead of saying which ones are invariant.  If you have no
+Front-Cover Texts, write ``no Front-Cover Texts'' instead of
+``Front-Cover Texts being @var{list}''; likewise for Back-Cover Texts.
+
+If your document contains nontrivial examples of program code, we
+recommend releasing these examples in parallel under your choice of
+free software license, such as the GNU General Public License,
+to permit their use in free software.
+
+@node Index, , GNU Free Documentation License, Top
+@comment node-name,    next,  previous,      up
+
+@unnumbered Index
+@printindex cp
+@bye
+
+Conventions:
+1. Functions, built-in or otherwise, do NOT have () after them.
+2. Gawk built-in vars and functions are in @code.  Also program vars and
+   functions.
+3. HTTP method names are in @code.
+4. Protocols such as echo, ftp, etc are in @samp.
+5. URLs are in @url.
+6. All RFC's in the index.  Put a space between `RFC' and the number.