1. lexgen
    1. Description
    2. Library Procedures
      1. Basic procedures
      2. Token procedure
      3. Convenience procedures
      4. Lexer procedure
    3. Examples
      1. A pattern to match floating point numbers
    4. Repository
    5. Version History
    6. License

lexgen

Description

lexgen is a lexer generator comprised in its core of only five small procedures that can be combined to form pattern matchers.

A pattern matcher procedure takes an input stream, and returns a new stream advanced by the pattern.

A stream is defined as a list that contains a list of characters consumed by the pattern matcher, and a list of characters not yet consumed. E.g., the list

 ((#\a) (#\b #\c #\d #\e))

represents a stream that contains the consumed character a, and the unconsumed characters b c d e.

A pattern matcher has the form of a procedure that takes a success continuation, which is invoked when the pattern matches and the stream is advanced, an error continuation, which is invoked when the pattern does not match, and an input stream.

Library Procedures

Every combinator procedure in this library returns a procedure that takes in a success continuation, error continuation and input stream as arguments.

Basic procedures

[procedure] (seq MATCHER1 MATCHER2) => MATCHER

seq builds a matcher that matches a sequence of patterns.

[procedure] (bar MATCHER1 MATCHER2) => MATCHER

bar matches either of two patterns. It's analogous to patterns separated by | in traditional regular expressions.

[procedure] (star MATCHER) => MATCHER

star is an implementation of the Kleene closure. It is analogous to * in traditional regular expressions.

Token procedure

[procedure] (tok TOKEN PROC) => MATCHER

Procedure tok builds pattern matchers based on character comparison operations. It is intended for matching input sequences that are SRFI-127 lazy streams.

For each stream given, tok applies a procedure to the given token TOKEN and an input character. If the procedure returns a true value, that value is prepended to the list of consumed elements, and the input character is removed from the stream of input elements.

[procedure] (char CHAR) => MATCHER

Matches a single character.

[procedure] (set CHAR-SET) => MATCHER

Matches any of a SRFI-14 set of characters.

[procedure] (range CHAR CHAR) => MATCHER

Matches a range of characters. Analogous to character class [].

[procedure] (lit STRING) => MATCHER

Matches a literal string s.

Convenience procedures

These procedures are built from the basic procedures and are provided for convenience.

[procedure] (try PROC) => PROC

Converts a binary predicate procedure to a binary procedure that returns its right argument when the predicate is true, and false otherwise.

[procedure] (lst MATCHER-LIST) => MATCHER

Constructs a matcher for the sequence of matchers in MATCHER-LIST.

[procedure] (pass) => MATCHER

This matcher returns without consuming any input.

[procedure] (pos MATCHER) => MATCHER

Positive closure. Analogous to +.

[procedure] (opt MATCHER) => MATCHER

Optional pattern. Analogous to ?.

[procedure] (bind F P) => MATCHER

Given a rule P and function F, returns a matcher that first applies P to the input stream, then applies F to the returned list of consumed tokens, and returns the result and the remainder of the input stream.

Note: this combinator will signal failure if the input stream is empty.

[procedure] (bind* F P) => MATCHER

The same as bind, but will signal success if the input stream is empty.

[procedure] (rebind F G P) => MATCHER

Given a rule P and procedures F and G, returns a matcher that first applies F to the input stream, then applies P to the resulting stream, then applies G to the resulting list of consumed elements and returns the result along with the remainder of the input stream.

Note: this combinator will signal failure if the input stream is empty.

[procedure] (rebind* F G P) => MATCHER

The same as rebind, but will signal success if the input stream is empty.

[procedure] (drop P) => MATCHER

Given a rule P, returns a matcher that always returns an empty list of consumed tokens when P succeeds.

Lexer procedure

[procedure] (lex MATCHER ERROR STRING) => CHAR-LIST

lex takes a pattern and a string, turns the string into a list of streams (containing one stream), applies the pattern, and returns the first possible match. Argument ERROR is a single-argument procedure called when the pattern does not match anything.

Examples

A pattern to match floating point numbers

 

;;  A pattern to match floating point numbers. 
;;  "-"?(([0-9]+(\\.[0-9]+)?)|(\\.[0-9]+))([eE][+-]?[0-9]+)? 

(define numpat
  (let* ((digit        (range #\0 #\9))
	 (digits       (pos digit))
	 (fraction     (seq (char #\.) digits))
	 (significand  (bar (seq digits (opt fraction)) fraction))
	 (exp          (seq (set "eE") (seq (opt (set "+-")) digits)))
	 (sign         (opt (char #\-))))
    (seq sign (seq significand (opt exp)))))
 
 (define (err s)
  (print "lexical error on stream: " s)
  (list))

 (lex numpat err "-123.45e-6")

Repository

https://github.com/iraikov/chicken-lexgen

Version History

License

Based on the SML lexer generator by Thant Tessman.

 Copyright 2009-2021 Ivan Raikov.


 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
 the Free Software Foundation, either version 3 of the License, or
 (at your option) any later version.

 This program is distributed in the hope that it will be useful, but
 WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 General Public License for more details.

 A full copy of the GPL license can be found at
 <http://www.gnu.org/licenses/>.