You are looking at historical revision 12098 of this page. It may differ significantly from its current revision.

Intarweb

Description

Intarweb is an advanced http library. It parses all headers into more useful Scheme values.

Author

Peter Bex

Requirements

Requires the defstruct, base64 and uri-generic extensions.

Documentation

The intarweb egg is set up to be used from a variety of situations. For this reason, it does not try to be a full HTTP client or server. If you need that kind of functionality, see eggs like spiffy.

Requests

A request object (a defstruct-type record) can be created using the following procedure:

[procedure] (make-request #!key uri port (method 'GET) (major 1) (minor 1) (headers (make-headers '())))

An existing request can be picked apart using the following procedures: <procedure>(request-uri REQUEST) => URI</procedure> <procedure>(request-port REQUEST) => PORT</procedure> <procedure>(request-method REQUEST) => SYMBOL</procedure> <procedure>(request-major REQUEST) => NUMBER</procedure> <procedure>(request-minor REQUEST) => NUMBER</procedure> <procedure>(request-headers REQUEST) => HEADERS</procedure>

The uri defines the entity to retrieve on the server, which should be a uri-generic-type URI object. The port is the port where the request is written to or read from. The method is a symbol that defines the HTTP method to use (case sensitive). major and minor identify the major and minor version of HTTP to use. Currently, 0.9, 1.0 and 1.1 are supported (but be careful with 0.9, it has some weird consequences and is not widely supported). Headers must be a headers object, which is described below.

The client will generally write requests, while the server will read them. To write a request, use the following procedure:

[procedure] (write-request REQUEST) => REQUEST

This will write a request line with headers to the server. In case it is a request type that has any body data, this should be written to the the request's port. Beware that this port can be modified by write-request, so be sure to write to the port as it is returned by the write-request procedure!

[procedure] (read-request PORT) => REQUEST

Reads a request object from the given input-port. An optional request body can be read from the request-port after calling this procedure.

Requests are parsed using parse procedures, which can be customized by overriding this parameter:

[parameter] (request-parsers [LIST])

The list is one of procedures which accept a request line string, which produce a request object from that, or #f if the request is not of the type handled by that procedure.

Requests are written using unparse procedures, which can be customized by overriding this parameter:

[parameter] (request-unparsers [LIST])

The list is one of procedures which accept a request object and write to the request's output port and return the new, possibly updated request object. If the request object is not unparsed by this handler, it returns #f.

Responses

A response is also a defstruct-type record, much like a request:

[procedure] (make-response #!key port (code 200) (reason "OK") (major 1) (minor 1) (headers (make-headers '())))

An existing response can be picked apart using the following procedures: <procedure>(response-port RESPONSE) => PORT</procedure> <procedure>(response-code RESPONSE) => NUMBER</procedure> <procedure>(response-reason RESPONSE) => STRING</procedure> <procedure>(response-major RESPONSE) => NUMBER</procedure> <procedure>(response-minor RESPONSE) => NUMBER</procedure> <procedure>(response-headers RESPONSE) => HEADERS</procedure>

The port, major, minor and headers are the same as for requests. code and reason are an integer status code and the short message that belongs to it, as defined in the spec (examples include: 200 OK, 301 Moved Permanently, etc).

A server will usually write a response, a client will read it. To write a response, use the following procedure:

[procedure] (write-response RESPONSE) => RESPONSE

If there is a response body, this must be written to the response-port after sending the response headers.

[procedure] (read-response PORT) => RESPONSE

Reads a response object from the port. An optional response body can be read from the response-port after calling this procedure.

Responses are parsed using parse procedures, which can be customized by overriding this parameter:

[parameter] (response-parsers [LIST])

The list is one of procedures which accept a response line string, which produce a response object from that, or #f if the response is not of the type handled by that procedure.

Responses are written using unparse procedures, which can be customized by overriding this parameter:

[parameter] (response-unparsers [LIST])

The list is one of procedures which accept a response object and write to the response's output port and return the new, possibly updated response object. If the response object is not unparsed by this handler, it returns #f.

Headers

Requests and responses contain HTTP headers wrapped in a special header-object to ensure they are properly normalized.

[procedure] (headers ALIST [HEADERS]) => HEADERS

This creates headers based on an input list. This list has the header-name as a symbol key, and a list of values as value:

<example> <expr> (headers `((host ("example.com" . 8080))

          (accept #(text/html ((q . 0.5)))
                  #(text/xml ((q . 0.1)))))
         old-headers)

</expr> </example>

This adds the named headers to the existing headers in old-headers. The host header is either a string with the hostname or a pair of hostname/port. The accept header is a list of allowed mime-type symbols. As can be seen here, optional parameters or "attributes" can be added to a header value by wrapping the value in a vector of length 2. The first entry in the vector is the header value, the second is an alist of attribute name/value pairs.

The headers all have their own different types. Here follows a list of headers with their value types:

Header name Value type Example value
accept List of mime-types (symbols), with optional q attribute indicating "quality" (preference level) (text/html #(text/xml ((q . 0.1))))
accept-charset List of charset-names (symbols), with optional q attribute (utf-8 #(iso-8859-5 ((q . 0.1))))
accept-encoding List of encoding-names (symbols), with optional q attribute (gzip #(identity ((q . 0))))
accept-language List of language-names (symbols), with optional q attribute (en-gb #(nl ((q . 0.5))))
accept-ranges List of range types acceptable (symbols). The spec only defines bytes and none. (bytes)
age Age in seconds (number) (3600)
allow List of methods that are allowed (symbols). (GET POST PUT DELETE)
authorization Authorization information. This consists of a symbol identifying the authentication scheme, with scheme-specific attributes. (digest #((username . "foo")))
cache-control An alist of key/value pairs. If no value is applicable, it is #t ((public . #t) (max-stale . 10) (no-cache . (max-age set-cookie)))
connection A list of connection options (symbols) (close)
content-encoding A list of encodings (symbols) applied to the entity-body. (deflate gzip)
content-language The natural language(s) of the "intended audience" (symbols) (de nl en-gb)
content-length The number of bytes (an exact number) in the entity-body (10)
content-location A location that the content can be retrieved from (a uri-generic object) (<#uri-generic# ...>)
content-md5 The MD5 checksum (a string) of the entity-body ("12345ABCDEF")
content-range Content range (pair with start- and endpoint) of the entity-body, if partially sent ((25 . 120))
content-type The mime type of the entity-body (a symbol) (text/html)
date The date at which the message originated TODO
etag An entity-tag (pair, car being either the symbol weak or strong, cdr being a symbol) that uniquely identifies the resource contents. ((strong . foo123))
expect Expectations of the server's behaviour (alist of symbol-string pairs), possibly with parameters. (#(((100-continue . #t)) ()))
expires Expiry timestamp for the entity TODO
from The e-mail address (a string) of the human user who controls the client ("info@example.com")
host The host to use (for virtual hosting). This is a pair of hostname and port (("example.com" . 80))
if-match Entity-tags (pair, weak/strong symbol and unique entity identifier symbol) which must match. ((strong . foo123) (strong . bar123))
if-modified-since Timestamp which indicates since when the entity must have been modified. TODO
if-none-match Entity tags (pair, weak/strong symbol and unique entity identifier symbol) which must not match. ((strong . foo123) (strong . bar123))
if-range The range to request, if the entity was unchanged TODO
if-unmodified-since A timestamp since which the entity must not have been modified TODO
last-modified A timestamp when the entity was last modified TODO
location A location (an URI object) to which to redirect (<#uri-object ...>)
max-forwards The maximum number of proxies that can forward a request (2)
pragma An alist of symbols containing implementation-specific directives. ((no-cache . #t) (my-extension . my-value))
proxy-authenticate Proxy authentication options (authentication scheme symbol, with parameters) (digest #((username . "foo")))
proxy-authorization Same as the above, only request-side instead of response-side (digest #((username . "foo")))
range The range of bytes (a pair of start and end) to request from the server. ((25 . 120))
referer The referring URL (uri-generic object) that linked to this one. (<#uri-object ...>)
retry-after Timestamp after which to retry the request if unavailable now. TODO
server Information about the server (a string) TODO
te Allowed transfer-encodings (symbols, with optional q attribute) for the response (deflate #(gzip ((q . 0.2))))
trailer Names of header fields (symbols) available in the trailer/after body (range etag)
transfer-encoding The encodings (symbols) used in the body (chunked)
upgrade Product names to which must be upgraded (strings) TODO
user-agent Product name of the user agent being used (string) TODO
vary The names of headers that define variation in the resource body, to determine cachability (symbols) (range etag)
via The intermediate hops through which the message is forwarded (strings) TODO
warning Warning code for special status TODO
www-authenticate If unauthorized, a challenge to authenticate (symbol, with attributes) (digest #((username . "foo")))
set-cookie Cookies to set (name/value string pair, with attributes) (#(("foo" . "bar") ((max-age . 10))))
cookie Cookies that were set (name/value string pair, with attributes) (#(("foo" . "bar") (($path . "/"))))

The parsers used to read and write header values can be customized with the following parameters:

[parameter] (header-parsers [ALIST])
[parameter] (header-unparsers [ALIST])

Whether a header is allowed once or multiple times in a request or response is determined by this parameter:

[parameter] (single-headers [LIST])

Changelog

License

 Copyright (c) 2008, Peter Bex
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are
 met:
 
 Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.
 
 Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in the
 documentation and/or other materials provided with the distribution.
 
 Neither the name of the author nor the names of its contributors may
 be used to endorse or promote products derived from this software
 without specific prior written permission.
 
 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
 FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
 COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
 INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
 (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
 SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
 STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
 OF THE POSSIBILITY OF SUCH DAMAGE.