SRFI-121: Generators

This SRFI defines utility procedures that create, transform, and consume generators. A generator is simply a procedure with no arguments that works as a source of a series of values. Every time it is called, it yields a value. Generators may be finite or infinite; a finite generator returns an end-of-file object to indicate that it is exhausted. For example, read-char, read-line, and read are generators that generate characters, lines, and objects from the current input port. Generators provide lightweight laziness.

  1. SRFI-121: Generators
  2. Installation
  3. SRFI Description
  4. Generators
    1. Constructors
    2. Operations
    3. Consuming Generated Values
  5. Repository
  6. Version History
  7. License

Installation

 
$ chicken-install srfi-121

or

$ chicken-install srfi-121 -test

if you want to run the tests for the egg in addition.

SRFI Description

For a full description of this SRFI, see the full SRFI document. This documentation covers the API only.

Generators

Generators can be divided into two classes, finite and infinite. Both kinds of generators can be invoked an indefinite number of times. After a finite generator has generated all its values, it will return an end-of-file object for all subsequent calls. A generator is said to be exhausted if calling it will return an end-of-file object. By definition, infinite generators can never be exhausted.

A generator is said to be in an undefined state if it cannot be determined exactly how many values it has generated. This arises because it is impossible to tell by inspecting a generator whether it is exhausted or not. For example, (generator-fold + 0 (generator 1 2 3) (generator 1 2)) will compute 0 + 1 + 1 + 2 + 2 = 6, at which time the second generator will be exhausted. If the first generator is invoked, however, it may return either 3 or an end-of-file object, depending on whether the implementation of generator-fold has invoked it or not. Therefore, the first generator is said to be in an undefined state.

Constructors

The result of a generator constructor is just a procedure, so printing it doesn't show much. In the examples in this section we use generator->list to convert the generator to a list.

These procedures have names ending with generator.

[procedure] (generator arg ...)

The simplest finite generator. Generates each of its arguments in turn. When no arguments are provided, it returns an empty generator that generates no values.

[procedure] (make-iota-generator count [start [step]])

Creates a finite generator of a sequence of count numbers. The sequence begins with start (which defaults to 0) and increases by step (which defaults to 1). If both start and step are exact, it generates exact numbers; otherwise it generates inexact numbers. The exactness of count doesn't affect the exactness of the results.

(generator->list (make-iota-generator 3 8))
 ⇒ (8 9 10)

(generator->list (make-iota-generator 3 8 2))
 ⇒ (8 10 12)
[procedure] (make-range-generator start [end [step]])

Creates a generator of a sequence of numbers. The sequence begins with start, increases by step (default 1), and continues while the number is less than end, or forever if end is omitted. If both start and step are exact, it generates exact numbers; otherwise it generates inexact numbers. The exactness of end doesn't affect the exactness of the results.

(generator->list (make-range-generator 3) 4)
 ⇒ (3 4 5 6)
(generator->list (make-range-generator 3 8))
 ⇒ (3 4 5 6 7)
(generator->list (make-range-generator 3 8 2))
 ⇒ (3 5 7)
[procedure] (make-coroutine-generator proc)

Creates a generator from a coroutine.

The proc argument is a procedure that takes one argument, yield. When called, make-coroutine-generator immediately returns a generator g. When g is called, proc runs until it calls yield. Calling yield causes the execution of proc to be suspended, and g returns the value passed to yield.

Whether this generator is finite or infinite depends on the behavior of proc. If proc returns, it is the end of the sequence — g returns an end-of-file object from then on. The return value of proc is ignored.

The following code creates a generator that produces a series 0, 1, and 2 (effectively the same as (make-range-generator 0 3) and binds it to g.

(define g
  (make-coroutine-generator
   (lambda (yield)
     (let loop ((i 0))
       (when (< i 3) (yield i) (loop (+ i 1)))))))

(generator->list g) ⇒ (0 1 2)
[procedure] (list->generator lis)
[procedure] (vector->generator vec [start [end]])
[procedure] (reverse-vector->generator vec [start [end]])
[procedure] (string->generator str [start [end]])
[procedure] (bytevector->generator bytevector [start [end]])

These procedures return generators that yield each element of the given argument. Mutating the underlying object will affect the results of the generator.

(generator->list (list->generator '(1 2 3 4 5)))
 ⇒ (1 2 3 4 5)
(generator->list (vector->generator '#(1 2 3 4 5)))
 ⇒ (1 2 3 4 5)
(generator->list (reverse-vector->generator '#(1 2 3 4 5)))
 ⇒ (5 4 3 2 1)
(generator->list (string->generator "abcde"))
 ⇒ (#\a #\b #\c #\d #\e)

The generators returned by the constructors are exhausted once all elements are retrieved; the optional start-th and end-th arguments can limit the range the generator walks across.

For reverse-vector->generator, the first value is the element right before the end-th element, and the last value is the start-th element. For all the other constructors, the first value the generator yields is the start-th element, and it ends right before the end-th element.

(generator->list (vector->generator '#(a b c d e) 2))
 ⇒ (c d e)
(generator->list (vector->generator '#(a b c d e) 2 4))
 ⇒ (c d)
(generator->list (reverse-vector->generator '#(a b c d e) 2))
 ⇒ (e d c)
(generator->list (reverse-vector->generator '#(a b c d e) 2 4))
 ⇒ (d c)
(generator->list (reverse-vector->generator '#(a b c d e) 0 2))
 ⇒ (b a)
[procedure] (make-for-each-generator for-each obj)

A generator constructor that converts any collection obj to a generator that returns its elements using a for-each procedure appropriate for obj. This must be a procedure that when called as (for-each proc obj) calls proc on each element of obj. Examples of such procedures are for-each, string-for-each, and vector-for-each from R7RS. The value returned by for-each is ignored. The generator is finite if the collection is finite, which would typically be the case.

The collections need not be conventional ones (lists, strings, etc.) as long as for-each can invoke a procedure on everything that counts as a member. For example, the following procedure allows for-each-generator to generate the digits of an integer from least to most significant:

(define (for-each-digit proc n)
  (when (> n 0)
    (let-values (((div rem) (truncate/ n 10)))
      (proc rem)
      (for-each-digit proc div))))
[procedure] (make-unfold-generator stop? mapper successor seed)

A generator constructor similar to SRFI 1's unfold.

The stop? predicate takes a seed value and determines whether to stop. The mapper procedure calculates a value to be returned by the generator from a seed value. The successor procedure calculates the next seed value from the current seed value.

For each call of the resulting generator, stop? is called with the current seed value. If it returns true, then the generator returns an end-of-file object. Otherwise, it applies mapper to the current seed value to get the value to return, and uses successor to update the seed value.

This generator is finite unless stop? never returns true.

(generator->list (make-unfold-generator
                      (lambda (s) (> s 5))
                      (lambda (s) (* s 2))
                      (lambda (s) (+ s 1))
                      0))
 ⇒ (0 2 4 6 8 10)

Operations

The following procedures accept one or more generators and return a new generator without consuming any elements from the source generator(s). In general, the result will be a finite generator if the arguments are.

The names of these procedures are prefixed with g.

[procedure] (gcons* item ... gen)

Returns a generator that adds items in front of gen. Once the items have been consumed, the generator is guaranteed to tail-call gen.

(generator->list (gcons* 'a 'b (make-range-generator 0 2)))
 ⇒ (a b 0 1)
[procedure] (gappend gen ...)

Returns a generator that yields the items from the first given generator, and once it is exhausted, from the second generator, and so on.

(generator->list (gappend (make-range-generator 0 3) (make-range-generator 0 2)))
 ⇒ (0 1 2 0 1)
(generator->list (gappend))
 ⇒ ()
[procedure] (gcombine proc seed gen gen2 ...)

A generator for mapping with state. It yields a sequence of sub-folds over proc.

The proc argument is a procedure that takes as many arguments as the input generators plus one. It is called as (proc v1 v2 … seed), where v1, v2, ... are the values yielded from the input generators, and seed is the current seed value. It must return two values, the yielding value and the next seed. The result generator is exhausted when any of the genn generators is exhausted, at which time all the others are in an undefined state.

[procedure] (gfilter pred gen)
[procedure] (gremove pred gen)

Return generators that yield the items from the source generator, except those on which pred answers false or true respectively.

[procedure] (gtake gen k [padding])
[procedure] (gdrop gen k)

These are generator analogues of SRFI 1 take and drop. gtake returns a generator that yields (at most) the first k items of the source generator, while gdrop returns a generator that skips the first k items of the source generator.

These won't complain if the source generator is exhausted before generating k items. By default, the generator returned by gtake terminates when the source generator does, but if you provide the padding argument, then the returned generator will yield exactly k items, using the padding value as needed to provide sufficient additional values.

[procedure] (gtake-while pred gen)
[procedure] (gdrop-while pred gen)

The generator analogues of SRFI-1 take-while and drop-while. The generator returned from gtake-while yields items from the source generator as long as pred returns true for each. The generator returned from gdrop-while first reads and discards values from the source generator while pred returns true for them, then starts yielding items returned by the source.

[procedure] (gdelete item gen [=])

Creates a generator that returns whatever gen returns, except for any items that are the same as item in the sense of =, which defaults to equal?. The = predicate is passed exactly two arguments, of which the first was generated by gen before the second.

(generator->list (gdelete 3 (generator 1 2 3 4 5 3 6 7)))
 ⇒ (1 2 4 5 6 7)
[procedure] (gdelete-neighbor-dups gen [=])

Creates a generator that returns whatever gen returns, except for any items that are equal to the preceding item in the sense of =, which defaults to equal?. The = predicate is passed exactly two arguments, of which the first was generated by gen before the second.

(generator->list (gdelete-neighbor-dups (list->generator '(a a b c a a a d c))))
  ⇒ (a b c a d c)
[procedure] (gindex value-gen index-gen)

Creates a generator that returns elements of value-gen specified by the indices (non-negative exact integers) generated by index-gen. It is an error if the indices are not strictly increasing, or if any index exceeds the number of elements generated by value-gen. The result generator is exhausted when either generator is exhausted, at which time the other is in an undefined state.

(generator->list (gindex (list->generator '(a b c d e f))
                         (list->generator '(0 2 4))))
 ⇒ (a c e)
[procedure] (gselect value-gen truth-gen)

Creates a generator that returns elements of value-gen that correspond to the values generated by truth-gen. If the current value of truth-gen is true, the current value of value-gen is generated, but otherwise not. The result generator is exhausted when either generator is exhausted, at which time the other is in an undefined state.

(generator->list (gselect (list->generator '(a b c d e f))
                          (list->generator '(#t #f #f #t #t #f))))
 ⇒ (a d e)

Consuming Generated Values

Unless otherwise noted, these procedures consume all the values available from the generator that is passed to them, and therefore will not return if one or more generator arguments are infinite. They have names prefixed with generator.

[procedure] (generator->list generator [k])

Reads items from generator and returns a newly allocated list of them. By default, it reads until the generator is exhausted.

If an optional argument k is given, it must be a non-negative integer, and the list ends when either k items are consumed, or generator is exhausted; therefore generator can be infinite in this case.

[procedure] (generator->reverse-list generator [k])

Reads items from generator and returns a newly allocated list of them in reverse order. By default, this reads until the generator is exhausted.

If an optional argument k is given, it must be a non-negative integer, and the list ends when either k items are read, or generator is exhausted; therefore generator can be infinite in this case.

[procedure] (generator->vector generator [k])

Reads items from generator and returns a newly allocated vector of them. By default, it reads until the generator is exhausted.

If an optional argument k is given, it must be a non-negative integer, and the list ends when either k items are consumed, or generator is exhausted; therefore generator can be infinite in this case.

[procedure] (generator->vector! vector at generator)

Reads items from generator and puts them into vector starting at index at, until vector is full or generator is exhausted. Generator can be infinite. The number of elements generated is returned.

[procedure] (generator->string generator [k])

Reads items from generator and returns a newly allocated string of them. It is an error if the items are not characters. By default, it reads until the generator is exhausted.

If an optional argument k is given, it must be a non-negative integer, and the string ends when either k items are consumed, or generator is exhausted; therefore generator can be infinite in this case.

[procedure] (generator-fold proc seed gen1 gen2 ...)

Works like SRFI 1 fold on the values generated by the generator arguments.

When one generator is given, for each value v generated by gen, proc is called as (proc v r), where r is the current accumulated result; the initial value of the accumulated result is seed, and the return value from proc becomes the next accumulated result. When gen is exhausted, the accumulated result at that time is returned from generator-fold.

When more than one generator is given, proc is invoked on the values returned by all the generator arguments followed by the current accumulated result. The procedure terminates when any of the genn generators is exhausted, at which time all the others are in an undefined state.

(with-input-from-string "a b c d e"
  (lambda () (generator-fold cons 'z read)))
 ⇒ (e d c b a . z)
[procedure] (generator-for-each proc gen gen2 …)

A generator analogue of for-each that consumes generated values using side effects. Repeatedly applies proc on the values yielded by gen, gen2 ... until any one of the generators is exhausted. The values returned from proc are discarded. The procedure terminates when any of the genn generators is exhausted, at which time all the others are in an undefined state. Returns an unspecified value.

[procedure] (generator-find pred gen)

Returns the first item from the generator gen that satisfies the predicate pred, or #f if no such item is found before gen is exhausted. If gen is infinite, generator-find will not return if it cannot find an appropriate item.

[procedure] (generator-count pred gen)

Returns the number of items available from the generator gen that satisfy the predicate pred.

[procedure] (generator-any pred gen)

Applies pred to each item from gen. As soon as it yields a true value, the value is returned without consuming the rest of gen. If gen is exhausted, returns #f.

[procedure] (generator-every pred gen)

Applies pred to each item from gen. As soon as it yields a false value, the value is returned without consuming the rest of gen. If gen is exhausted, returns the last value returned by pred, or #t if pred was never called.

[procedure] (generator-unfold gen unfold arg ...)

Equivalent to (unfold eof-object? (lambda (x) x) (lambda (x) (gen)) (gen) arg ...). The values of gen are unfolded into the collection that unfold creates.

The signature of the unfold procedure is (unfold stop? mapper successor seed args ...). Note that the vector-unfold and vector-unfold-right of SRFI 43 and SRFI 133 do not have this signature and cannot be used with this procedure. To unfold into a vector, use SRFI 1's unfold and then apply list->vector to the result.

;; Iterates over string and unfolds into a list using SRFI 1 unfold
(generator-unfold (make-for-each-generator string-for-each "abc") unfold)
 ⇒ (#\a #\b #\c)

Repository

Github

Version History
1.8
Incorporate upstream bug-fixes from Mark Weaver (see mailing list for details).
1.7
CHICKEN 5 support
1.5
Updates to the tests (see mailing list for details).
1.4
Adds Kevin Wortman's fix to generator->list and generator->reverse-list
1.3
Removes hardcoded .so in setup
1.2
Adds standard README.org to repo
1.1
Fixes generator->list from upstream issue
1.0
Initial release

License

Copyright (C) John Cowan (2015-2016). All Rights Reserved.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.