You are looking at historical revision 8462 of this page. It may differ significantly from its current revision.

http

Description

An easy to use HTTP client and server package. The server is fully multithreaded and supports persistent connections.

Author

Felix Winkelmann

Requirements

Requires the regex-case egg at compile-time.

http-client requires the url egg at runtime.

Both http-client and http-server require the http-utils module at runtime.

Documentation

What follows is a description of the http extension, which is separated into three sub-extensions: http-client (HTTP client functionality), http-server (serving HTTP requests) and http-utils (utility functions common to client and server code).

If you want to run a web-server, you should also take a look at the more featureful spiffy extension.

http-client
Usage

(require-extension http-client)

http:send-request
 
[procedure] (http:send-request REQUEST [INPUT-PORT OUTPUT-PORT])

Sends an HTTP request represented by REQUEST, which may either be a HTTP request object, or a string that specifies an URL. The request is sent to it's destination and four values are returned: a string containing the first line of the response received from the server, an a-list that maps HTTP headers to values (strings) and the input- and output-port of the connection. Note that the connection is still open, and the returned ports can be passed in subsequent invocations of http:send-request to achieve a persistent connection. The ports are closed automatically, if no longer referenced.

If an URL is given instead of a request record, a request is sent of the form

GET url HTTP/1.0
Connection: close
http:GET
 
[procedure] (http:GET REQUEST)

Sends a GET request represented by REQUEST, which may be a string (an URL) or a HTTP request object, and returns a string containing the body of the servers response. The REQUEST keeps a Cookie header that is set by Set-Cookie headers in a response, unsafely (hack), regardless of the storing policy. Be aware of it when reusing REQUEST.

http:POST
 
[procedure] (http:POST REQUEST [ARGUMENTS] #!key [headers: LIST] [type: CTYPE] [delim: DELIM])

Sends a POST request represented by REQUEST, which may be a string (an URL) or a HTTP request object, and returns a string containing the body of the servers response. ARGUMENTS may be one of several forms depending on the value of the Content-Type attribute. If ARGUMENTS is unspecified, the body will be blank (i.e., a headers-only request). DELIM is an arbitrary separator string defaulting to the null-string. The behaviour of DELIM is dependent on Content-Type. HEADERS should be either null or a list composed of (ATTRIBUTE . VALUE) pairs. Connection and Content-Type, if unspecified, will be added automatically. Attributes are not case sensitive. Any attributes explicitly given in HEADERS are used, even if normally autogenerated. CTYPE is the Content-Type header attribute, and defaults to application/x-www-form-urlencoded. http:POST knows several values of Content-Type and will create the body appropriately, as given below:

Content-Type Handler
application/x-www-form-urlencoded ARGUMENTS may be a string or a list. The list may contain either (NAME . VALUE) pairs or strings of the form "name=value". The message body is generated as "name1=val1&name2;=val2...". DELIM is ignored.
multipart/form-data ARGUMENTS may be a string or a list. DELIM is used as the boundary string, if given. If DELIM is not specified, it defaults to ----chicken-scheme---. The Content-Type header is automatically annotated with the boundary metadata. Each bounded part is generated, including the whitespace separators. The ARGUMENTS list must be composed of elements of the following types, where each element of the top-level list is a separate multipart:
NAME name attrib of this part is set to NAME. Body is empty.
(NAME . VALUE) name attrib of this part is NAME. Body is VALUE.
(NAME (ATTRIB . AVAL)... VALUE) name attrib of this part is NAME. Every following (ATTRIB . AVAL) pair is added to the part header as ATTRIB="AVAL" and separated with semicolons. If ATTRIB ends with a colon, it is whitespace separated and added to the body to allow file transmission headers. VALUE is added to the body after all attribs are processed, and must NOT be a pair.

</td></tr> <tr><td>everything else</td> <td>ARGUMENTS may be either a string or a list of strings. Lists of strings are concatenated with DELIM as a separator.</td></tr> </table>

The above alterations are performed only when ARGUMENTS is a list. If given as a string, the body is set to the string value without any alteration. All values other than strings or lists generate an error.

The REQUEST keeps a Cookie header, as with http:GET.

http:close-all-connections!
 
[procedure] (http:close-all-connections!)

Close all persistent connections kept in the current thread.

http:read-body
 
[procedure] (http:read-body ARGUMENTS PORT)

Read the HTTP Body from the input port PORT, according to the Content-Length header or the Transfer-Encoding header.

http:add-proxy!
 
[procedure] (http:add-proxy! PROXY-HOST PROXY-PORT [SERV-PATTERN] [HOST-PATTERN] [PORT-PATTERN] [PATH-PATTERN])

Add a http proxy. PATTERNs are either a string, a regex, or #t. The first added proxy has the least priority.

Note: CONNECT method for SSL is not supported (yet).

http:remove-all-proxies!
 
[procedure] (http:remove-all-proxies!)

Remove all http proxies.

http-server
Usage

(require-extension http-server)

General operation

The server maps URLs to resource-handlers, which are procedures that process incoming client-requests. A resource-handler is responsible for generating a response by writing output to the value of (current-output-port).

The data contained in a client-request is parsed by a so-called content-parser, which is a procedure that reads the request-body from the port given by (current-input-port). Content-types are encoded as symbols.

A parser for application/x-www-form-urlencoded is predefined, other content-parsers have to be defined by application-code using the procedure http:content-parser, or the default parser will be invoked (which reads the content as a plain string).

The content-parser for text returns the request-body as a string. The content-parser for urlencoded data returns the request-body as an a-list that maps variables to strings.

http:make-server
 
[procedure] (http:make-server PORTNUMBER #!key NAME PROTOCOL BACKLOG ACCEPT INIT)

Creates and returns a server-procedure. NAME defaults to something silly, PROTOCOL to #f (ignored), BACKLOG to 40 and ACCEPT to #f. INIT should be a procedure of no arguments that will be called after the networking initialisation has taken place (specifically, after the invocation of tcp-listen).

To run the server-loop, invoke the returned procedure, which takes an optional boolean argument (passing #t will generate debugging output). PORTNUMBER, BACKLOG, and ACCEPT are directly passed on to tcp-listen (see Unit tcp for more information about those parameters).

http:content-parser
 
[procedure] (http:content-parser CONTENTTYPE [PROC])

Returns or sets the parser-procedure (a procedure of three arguments: the size of the content (may be #f), the a-list of request headers and an input port) for CONTENTTYPE, which should be a symbol.

The content-parser procedure PROC should return two values: the parsed and the unparsed (raw) request body.

http:write-response-header
 
[procedure] (http:write-response-header REQ [CODE MSG [ALIST [PORT [PROTOCOL]]]])

Writes a HTTP response header for request REQ to PORT, containing the server-name. CODE and MSG default to 200 and OK, respectively. The optional ALIST may contain pairs with header-names and -values. If given, PROTOCOL specifes the HTTP protocol to use for the reply, which should be a symbol (either HTTP/1.0 or HTTP/1.1) and defaults to the protocol given in http:make-server.

http:write-error-response
 
[procedure] (http:write-error-response CODE MESSAGE [PORT])

Writes a HTTP error-response to PORT, which defaults to the value of (current-output-port). Uses the result returned by the value of (http:error-response-handler).

http:request-method-handler
 
[procedure] (http:request-method-handler METHOD [PROC])

Reads ot sets the handler procedure PROC for the request-method METHOD (which should be a symbol). PROC should accept one argument, a request-object. During execution of the handler, the current input- and output ports are bound to ports connected to the client.

Method-handlers for GET and POST requests are predefined.

http:add-resource
 
[procedure] (http:add-resource URL HANDLER)

Defines a new resource for URL (which may be a string, a symbol or a list of strings/symbols). HANDLER should be a procedure of two arguments: a request structure, and an a-list mapping urlencoded arguments to values.

During execution of the handler the current input- and an output-ports are bound to ports communicating with the client.

http:remove-resource
 
[procedure] (http:remove-resource URL)

Removes the resource defined under URL (string, symbol or list).

http:find-resource
 
[procedure] (http:find-resource URL)

Returns the handler-procedure for the resource defined under URL or #f if no such resource is registered.

http:fallback-handler
 
[parameter] http:fallback-handler

Contains a procedure that is is invoked on requests for resources that could not be found. This procedure is then called with the original request object. The default handler generates a 404 response.

http:error-response-handler
 
[parameter] http:error-response-handler

Contains a procedure that will be called when an HTTP error-response should be generated. The procedure is called with the error-code and message and should return a string containing HTML that will be sent in the body of the error-response.

http:current-request-count
 
[procedure] (http:current-request-count)

Returns the number of requests handled since startup.

http:request-count-limit
 
[parameter] http:request-count-limit

Maximum number of concurrently handled requests.

http:startup-hook
 
[parameter] http:startup-hook

A procedure that will be called on server startup, before accepting first request. The default procedure does nothing.

http:error-hook
 
[parameter] http:error-hook

A procedure that will be called when a thread triggers an unhandled exception. The exception is passed as an argument to the hook procedure.

http:log-hook
 
[parameter] http:log-hook

A procedure that will be called upon completion of a HTTP request. The procedure will be called with two arguments: the request object and the IP address of the client. The default value of this parameter does nothing.

http:url-transformation
 
[parameter] http:url-transformation

A procedure that allows arbitrary transformations of the URL part of each incoming request. The procedure is called with a single argument (the URL string) and should return the same URL, or a transformed one. The default transformation returns the original url unchanged.

http:listen-procedure
 
[parameter] http:listen-procedure

Holds a procedure that will be used to create a socket-listener - defaults to tcp-listen.

http:accept-procedure
 
[parameter] http:accept-procedure

Holds a procedure that will be used to accept a socket-connection - defaults to tcp-accept.

http:get-addresses-procedure
 
[parameter] http:get-addresses-procedure

Holds a procedure that will be used to obtain peer addresses - defaults to tcp-addresses.

http:hard-close-procedure
 
[parameter] http:hard-close-procedure

Holds a procedure that will be used to close a connection prematurely - defaults to tcp-abandon-port.

http-utils
Usage

(require-extension http-utils)

http:decode-url
 
[procedure] (http:decode-url URL)

Canonicalizes the string passed in URL and returns two values: the location-path and an alist containing argument <-> value pairs.

http:make-request
 
[procedure] (http:make-request METHOD URL [ATTRIBUTES [BODY [PROTOCOL [IP]]]])

Returns a freshly created HTTP request object (see below for the meaning of the arguments).

http:request?
 
[procedure] (http:request? X)

Returns #t if X is a request object, or #f otherwise.

http-request accessor methods
 
[procedure] (http:request-url REQUEST) -> URL
[procedure] (http:request-protocol REQUEST) -> PROTOCOL
[procedure] (http:request-attributes REQUEST) -> ATTRIBUTES
[procedure] (http:request-body REQUEST) -> X
[procedure] (http:request-method REQUEST) -> METHOD
[procedure] (http:request-ip REQUEST) -> STRING
[procedure] (http:request-sslctx REQUEST) -> <ssl-client-context>
[procedure] (http:request-url-set! REQUEST URL)
[procedure] (http:request-protocol-set! REQUEST PROTOCOL)
[procedure] (http:request-attributes-set! REQUEST ATTRIBUTES)
[procedure] (http:request-body-set! REQUEST X)
[procedure] (http:request-method-set! REQUEST METHOD)
[procedure] (http:request-ip-set! REQUEST STRING)
[procedure] (http:request-sslctx-set! REQUEST <ssl-client-context>)

Accessor procedures for the components of a HTTP request structure. URL is a string, METHOD and PROTOCOL are symbols, BODY is either #f or a string and ATTRIBUTES is an a-list where each pair contains an attribute string and a value string.

http:request attribute methods
 
[procedure] (http:request-attribute-get REQUEST ATTRIB [DEFAULT {{#f}}])

Accessor procedure to get the value of ATTRIB in REQUEST. If ATTRIB is not present in request, DEFAULT is returned. The search is case-insensitive. Note that this only returns the value; the attribute name is NOT returned.

[procedure] (http:request-attribute-add! REQUEST ATTRIB AVAL)

Adds ATTRIB to REQUEST's attribute list with its value set to AVAL. If ATTRIB already appears in the list (case-insensitive), its value is set to AVAL and it retains its position in the list; otherwise, the (ATTRIB . AVAL) pair is added to the end.

[procedure] (http:request-attribute-del! REQUEST ATTRIB)

Removes ATTRIB from the attribute list in REQUEST, if it exists (case-insensitive search). The order of the attribute list is not altered aside from the removal. It is not an error for ATTRIB to not appear in the list.

http:read-line-limit
 
[parameter] http:read-line-limit

The maximum length of a header-line.

http:read-request-attributes
 
[procedure] (http:read-request-attributes PORT)

Reads MIME type headers from PORT until end of file is reached or a line is not a valid MIME header and returns an a-list where each pair holds the header (converted to lowercase) and the value (both strings).

http:canonicalize-string
 
[procedure] (http:canonicalize-string STRING)

Canonicalizes STRING by substituting %XX and + sequences.

Examples

Server example

A simple "Hello, world" server:

(require-extension http-server)

(http:add-resource '("/" "/index.html")
  (lambda (r a)
    (let ([msg "&lt;h1>Hello, world!&lt;/h1>"])
      (http:write-response-header 
        r
        `(("Content-type" . "test/html")
          ("Content-length" . ,(string-length msg))))
      (display msg) ) ) )

((http:make-server 4242) #t)

To try it out, simply load the code into the interpreter and point your browser to localhost:4242.

Client example

 
(require-extension http-client)

(define-values (h a i o) (http:send-request "localhost:4242/"))

(pretty-print (read-lines i))
(close-input-port i)
(close-output-port o)

Loading this into the interpreter will print

("<h1>Hello, world!</h1>")

(provided the hello-world server is running)

Changelog

License

 
Copyright (c) 2003, Felix L. Winkelmann
All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following
conditions are met:

  Redistributions of source code must retain the above copyright notice, this list of conditions and the following
    disclaimer. 
  Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
    disclaimer in the documentation and/or other materials provided with the distribution. 
  Neither the name of the author nor the names of its contributors may be used to endorse or promote
    products derived from this software without specific prior written permission. 

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS
OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.