testbase

  1. testbase
    1. Description
    2. Author
    3. Requirements
    4. Download
    5. Documentation
      1. Test File
      2. Macro and Procedure Naming Conventions
      3. Option Expansion Control
        1. test::timing
        2. test::filtering
        3. test::catching
        4. test::selecting
        5. test::limiting
      4. Multiline String Construction
        1. MULTILINE-STRING
      5. Immediate Test
        1. test
      6. Test Procedure Definition
        1. define-test
        2. declare-test
      7. Test Procedure Operations
        1. test::procedure?
        2. test::formal-arguments
        3. test::declarations
        4. test::procedure
        5. test::name
        6. test::procedure-variable
        7. test::styler
        8. test::styler-set!
        9. test::tolerance
        10. test::tolerance-set!
        11. test::reset!
        12. test::structure
        13. test::add-filter!
        14. test::remove-filter!
        15. test::add-echo!
        16. test::remove-echo!
        17. test::set-echo-option!
        18. test::run-mode
        19. test::selection-mode
        20. test::select!
        21. test::selections
        22. test::resource-limit
        23. test::resource-limit-set!
        24. test::procedures
        25. test::for-each
        26. test::map
        27. test::forget!
        28. test::exit-on-failure?
        29. test::run
      8. Run Tests
        1. run-test
      9. Test Macros
      10. Test Containers
        1. test-suite
        2. test-case
        3. test-collect
      11. Test Flags
        1. test-case-evaluate-thru-failure
        2. test-timing
        3. test-evaluation-mode?
      12. Test Binding Forms
        1. test-letrec
        2. test-let*
        3. test-let
      13. Test Expectations
        1. insist
        2. expect-unary
        3. expect-binary
        4. expect-zero
        5. expect-nonzero
        6. expect-positive
        7. expect-negative
        8. expect-near
        9. expect-true
        10. expect-false
        11. expect-success
        12. expect-failure
        13. expect-not-false
        14. expect-not-null
        15. expect-eq
        16. expect-eqv
        17. expect-equal
        18. expect-unary/values
        19. expect-near/values
        20. expect-near/values
        21. expect-eq/values
        22. expect-eqv/values
        23. expect-equal/values
        24. expect-exception
        25. expect-ec
        26. expect-set!
      14. Define New Expectations
        1. define-expect-unary
        2. define-expect-binary
        3. define-expect-unary/values
        4. define-expect-binary/values
        5. define-expect-nary
      15. Testeez Synonyms
        1. test-eval
        2. test/eq
        3. test/eqv
        4. test/equal
      16. Test Setup (Side-Effect)
        1. side-effect
        2. setup
      17. Test Teardown (Destructor)
        1. destructor-atexit!
        2. teardown
        3. destructor-clear!
        4. destructor-dump
      18. Gloss
        1. gloss
      19. Skip
        1. skip
      20. Termination
        1. terminate
      21. Todo
        1. todo
      22. Test Result Object
      23. Any Test Result Object
        1. test::result?
        2. test::ignore-result?
        3. test::result-passed?
        4. test::result-finding
        5. test::result-name
        6. test::result-message
        7. test::result-warning?
        8. test::result-warning
        9. test::result-timing?
        10. test::result-timing
        11. test::result-timing-message
        12. test::result-kind
      24. Test Container Result Object
      25. Test Suite Result Object
        1. test::test-suite-result?
        2. test::test-suite-result-passed?
        3. test::test-suite-result-finding
        4. test::test-suite-result-name
        5. test::test-suite-result-list
        6. test::test-suite-result-warning?
        7. test::test-suite-result-warning
        8. test::test-suite-result-timing?
        9. test::test-suite-result-timing
        10. test::test-suite-results-passed?
      26. Test Case Result Object
        1. test::test-case-result?
        2. test::test-case-result-passed?
        3. test::test-case-result-finding
        4. test::test-case-result-name
        5. test::test-case-result-list
        6. test::test-case-result-warning?
        7. test::test-case-result-warning
        8. test::test-case-result-timing?
        9. test::test-case-result-timing
        10. test::test-case-results-passed?
      27. Expectation Result Objects
      28. Unary Expectation
        1. test::expect-result?
        2. test::expect-result-finding
        3. test::expect-result-kind
        4. test::expect-result-name
        5. test::expect-result-unevaled
        6. test::expect-result-evaled
        7. test::expect-result-warning?
        8. test::expect-result-warning
        9. test::expect-result-timing?
        10. test::expect-result-timing
      29. Binary Expectation
        1. test::expect-equivalence-result?
        2. test::expect-equivalence-result-finding
        3. test::expect-equivalence-result-kind
        4. test::expect-equivalence-result-name
        5. test::expect-equivalence-result-lhs-evaled
        6. test::expect-equivalence-result-rhs-unevaled
        7. test::expect-equivalence-result-rhs-evaled
        8. test::expect-equivalence-result-warning?
        9. test::expect-equivalence-result-warning
        10. test::expect-equivalence-result-timing?
        11. test::expect-equivalence-result-timing
      30. Tolerance Style Expectation
        1. test::expect-tolerance-result?
        2. test::expect-tolerance-result-finding
        3. test::expect-tolerance-result-kind
        4. test::expect-tolerance-result-name
        5. test::expect-tolerance-result-lhs-evaled
        6. test::expect-tolerance-result-lhs-tol-evaled
        7. test::expect-tolerance-result-rhs-unevaled
        8. test::expect-tolerance-result-rhs-evaled
        9. test::expect-tolerance-result-warning?
        10. test::expect-tolerance-result-warning
        11. test::expect-tolerance-result-timing?
        12. test::expect-tolerance-result-timing
        13. Gloss Result Object
        14. test::gloss-result?
        15. test::gloss-result-message
        16. test::gloss-result-warning?
        17. test::gloss-result-warning
        18. Skip Result Object
        19. test::skip-result?
        20. test::skip-result-message
        21. test::skip-result-warning?
        22. test::skip-result-warning
        23. Termination Result Object
        24. test::terminate-result?
        25. test::terminate-result-finding
        26. test::terminate-result-scope
        27. test::terminate-result-container
        28. test::terminate-result-message
        29. Todo Result Object
        30. test::todo-result?
        31. test::todo-result-message
        32. test::todo-result-warning?
        33. test::todo-result-warning
      31. Miscellaneous
        1. test::walk-structure
      32. Statistics Generation
        1. test::stat-result-statistics
        2. test::stat-test-suites
        3. test::stat-test-suite-warnings
        4. test::stat-test-suites-passed
        5. test::stat-test-suites-failed
        6. test::stat-test-suites-timing
        7. test::stat-test-suites-terminated
        8. test::stat-test-cases
        9. test::stat-test-case-warnings
        10. test::stat-test-cases-passed
        11. test::stat-test-cases-failed
        12. test::stat-test-cases-timing
        13. test::stat-test-cases-terminated
        14. test::stat-all-expectations
        15. test::stat-all-expectation-warnings
        16. test::stat-all-expectations-passed
        17. test::stat-all-expectations-failed
        18. test::stat-all-expectations-timing
        19. test::stat-single-expectations
        20. test::stat-single-expectation-warnings
        21. test::stat-single-expectations-passed
        22. test::stat-single-expectations-failed
        23. test::stat-single-expectations-timing
        24. test::stat-tol-expectations
        25. test::stat-tol-expectation-warnings
        26. test::stat-tol-expectations-passed
        27. test::stat-tol-expectations-failed
        28. test::stat-tol-expectations-timing
        29. test::stat-equiv-expectations
        30. test::stat-equiv-expectation-warnings
        31. test::stat-equiv-expectations-passed
        32. test::stat-equiv-expectations-failed
        33. test::stat-equiv-expectations-timing
        34. test::stat-todos
        35. test::stat-todo-warnings
        36. test::stat-skips
        37. test::stat-skip-warnings
        38. test::stat-glosses
        39. test::stat-gloss-warnings
        40. test::stat-terminations
      33. Printing Routines
        1. test::write-object
        2. test::display-objects
        3. test::display-objects-newline
        4. test::display-indent
        5. test::display-indented-objects
        6. test::display-indented-objects-newline
        7. test::display-underlined
        8. test::display-structure
      34. Test Report Generation
        1. test::output-style-compact
        2. test::output-style-human
        3. test::output-style-html
        4. test::output-style-minimal
        5. test::with-output-style
      35. Notes & Caveats
      36. Examples
      37. Tips, Tricks, And Idioms
        1. General Tips
        2. Automated Egg Tests
      38. Migration from test-infrastructure
      39. Bugs & Limitations
    6. Contributions
    7. Changelog
    8. License

Description

Macro based unit-testing facility

Author

Kon Lovett, with portions from test-infrastructure 1.0 by Peter Keller

Requirements

Download

testbase.egg

Documentation

This extension provides a macro based unit testing facility based upon expectations concerning evaluations of expressions. Expectations are grouped into test containers, such as test-suite and test/case. In addition to expectation and container test forms a set of annotation forms are supplied. All test forms, be they containers, annotations, or expectations, return a test result object. Result objects generated by the test forms enclosed by a test container are composed by the container into the container's test result object. The set of test results forms a tree.

This result tree is then passed to either user defined functions which traverse the tree manipulating it in any way desired. API functions to deal with the result types are supplied. Examples of existing functions that traverse the result are the test results output routines.

Test File

A test-file is a Scheme source file consisting of testbase forms and necessary auxiliary definitions. The general template is:

(require-extension testbase)
.. other extensions, includes, imports, etc. as needed ..

#| TestBase
.. see "testbase-driver" documentation ..
|#

.. any expressions necessary to testing ..

(define-test foo-test ... )

(run-test "Foo Tests")

While possible to use testbase forms without invoking run-test command line processing is not performed.

Command Line Options:

---failure-exit CODE
Specifiy an exit code & set the early-exit-upon-failure-option. See test::exit-on-failure?, test::run, and run-test.
-b ---batch
Specify batch mode. See run-test.
-i ---interactive
Specify interactive (or driven) mode. See run-test.
-V ---version
Display the version.
-h ---help
Display help message.

Macro and Procedure Naming Conventions

A distinction is made between forms that are to be used within a test, and those that are to be used outside of a test. In general, those forms that are for use within a test are macros, and those for use outside a test are procedures.

Procedures have a test:: namespace prefix, such as test::timing

Macros do not have a test:: namespace prefix. Since they can only pollute the namespace of a single file a looser naming convention is used.

Option Expansion Control

The expanded source of the test code is controlled by a set of syntax constants. To drop or add an option place the appropriate syntax form below in a test file.

FLAG is a boolean, #t or #f.

test::timing
[syntax] (define-for-syntax test::timing FLAG)

test::timing controls the expectation timing option. Default is #f.

test::filtering
[syntax] (define-for-syntax test::filtering FLAG)

test::filtering controls the test result filtering option. Default is #f.

test::catching
[syntax] (define-for-syntax test::catching FLAG)

test::catching controls the expectation error/exception trapping option. Default is #t.

Dropping error/exception trapping is not suggested in the general case, since any intermediate errors will abort the entire test. Does not affect expect-exception.

test::selecting
[syntax] (define-for-syntax test::selecting FLAG)

test::selecting controls the test selective evaluation option. Default is #f.

test::limiting
[syntax] (define-for-syntax test::limiting FLAG)

test::limiting controls the test resource limit option. Default is #f.

Multiline String Construction

Due to a bug the Chicken multiline string forms cannot be used. This macro is provided as a workaround. For use in a test-file.

MULTILINE-STRING
[syntax] (MULTILINE-STRING STRING ...)

Yes, the macro name is uppercase.

STRING is any scheme string. The strings are suffixed with a newline and concatenated.

Immediate Test

This macro will evaluate in a left to right fashion the clauses inside it and output the results. The clauses are a sequence of (Expression Expectation) forms. A clause is evaluated as (expect-equal EXPRESSION EXPECTED). For use with a REPL (Read-Eval-Print-Loop).

The results are not kept and no other forms are allowed in the body.

test
[syntax] (test (EXPECTED EXPRESSION) ...)

EXPRESSION is any scheme expression.

EXPECTED is any scheme expression, the expected result of EXPRESSION.

Test Procedure Definition

All test forms are grouped into a test procedure. Test forms cannot be used outside of a test procedure without generating an error.

define-test
[syntax] (define-test PROCEDURE-VARIABLE [TEST-NAME] [(initial FORM ...)] [(final FORM ...)] CLAUSE ...)
[syntax] (define-test (PROCEDURE-VARIABLE VARIABLE ...) [TEST-NAME] [(initial FORM ...)] [(final FORM ...)] CLAUSE ...)

Defines a variable, PROCEDURE-VARIABLE, with a TEST-PROCEDURE value.

TEST-NAME must evaluate to a Scheme atom. Must be unique in the test-file. When missing a string name is formed from the PROCEDURE-VARIABLE.

VARIABLE is a lambda variable, a formal argument to the test-procedure.

When supplied the (initial FORM ...) will cause each FORM to be expanded at the beginning of the test procedure. FORM can be any Scheme form legal at the beginning of a lambda body. The initial section will be evaluated before the first CLAUSE. If define forms occur in this section, the variables will be in scope for the whole test.

When supplied the (final FORM ...) will cause each FORM to be expanded at the end of the test procedure. FORM can be any Scheme form legal at the end of a lambda body. The final section will be evaluated after the last CLAUSE and before the destructor is activated!

The forms in the initial and final sections do not need to return test result objects; they are not considered part of the test body. In other words, a FORM is not a CLAUSE.

CLAUSE is any of the test API elements. Actually CLAUSE can be any Scheme FORM but be careful. A CLAUSE must return a test result object. Unless this can be guaranteed for a FORM a runtime error can be generated.

The DTOR-NAME will be test-dtor. See test-suite.

The ESCR-NAME will be test-escr. See test-suite.

Can be called directly, as (PROCEDURE-VARIABLE) or (PROCEDURE-VARIABLE ARGUMENT ...), and returns a test-result.

A parameterized test definition should be complemented by one or more test declarations, a declare-test form. Otherwise the test cannot be invoked by test::run.

declare-test
[syntax] (declare-test PROCEDURE-VARIABLE TEST-NAME ARGUMENT ...)

Declares an actual test for the PROCEDURE-VARIABLE.

ARGUMENT is a scheme object. An actual argument to the corresponding test-procedure formal argument.

TEST-NAME must evaluate to a Scheme atom.

Must occur lexically after the PROCEDURE-VARIABLE definition in the test-file!

An actual-test is not a test-procedure and cannot be called directly. The actual-test is tied to the PROCEDURE-VARIABLE, and can only be invoked by test::run.

Test Procedure Operations

TEST-IDENTIFIER is either a PROCEDURE-VARIABLE bound to a TEST-PROCEDURE or a TEST-NAME. When a TEST-NAME the corresponding TEST-PROCEDURE will be substituted.

test::procedure?
[procedure] (test::procedure? OBJECT)

Is the OBJECT a defined test-procedure object?

test::formal-arguments
[procedure] (test::formal-arguments TEST-IDENTIFIER)

Returns the formal argument list of the test-procedure.

test::declarations
[procedure] (test::declarations TEST-IDENTIFIER)

Returns a list of (test-procedure actual-test-name actual-test-arguments) for the test-procedure.

TEST-IDENTIFIER should identify a parameterized test-procedure.

test::procedure
[procedure] (test::procedure TEST-IDENTIFIER)

Returns a test-procedure or #f.

test::name
[procedure] (test::name TEST-IDENTIFIER)

Returns the test-name of the test-procedure.

test::procedure-variable
[procedure] (test::procedure-variable TEST-IDENTIFIER)

Returns the defined symbol of the test-procedure.

test::styler
[procedure] (test::styler TEST-IDENTIFIER)

Returns the default output style procedure for the test-procedure, or #f when no default styler is set.

test::styler-set!
[procedure] (test::styler-set! TEST-IDENTIFIER [STYLER])

Sets the default output style procedure for the test-procedure. When missing or #f then no default styler is set.

test::tolerance
[procedure] (test::tolerance TEST-IDENTIFIER)

Returns the current in-exact number comparison tolerance of the test-procedure.

test::tolerance-set!
[procedure] (test::tolerance-set! TEST-IDENTIFIER TOLERANCE)

Sets the current in-exact number comparison tolerance of the test-procedure to TOLERANCE.

test::reset!
[procedure] (test::reset! TEST-IDENTIFIER)

Sets the state of the test-procedure to the initial settings.

test::structure
[procedure] (test::structure TEST-IDENTIFIER)

Returns the internal structure of test-procedure as a tree.

The tree is the test hierarchy, where each node of the tree is (suite | case | expect TEST-NAME ...), where ... are any sub-trees.

The top-level suite element value refers to the test-procedure.

test::add-filter!
[procedure] (test::add-filter! TEST-IDENTIFIER FILTER-PROCEDURE)

Places FILTER-PROCEDURE as the head of the result filter chain for the test-procedure.

FILTER-PROCEDURE must be a procedure of one argument, a test result object, and it must return a test result object. Not necessarily the same one.

Filtering occurs during test element evaluation.

test::remove-filter!
[procedure] (test::remove-filter! TEST-IDENTIFIER FILTER-PROCEDURE)

Removes FILTER-PROCEDURE from the result filter chain for the test-procedure.

test::add-echo!
[procedure] (test::add-echo! TEST-IDENTIFIER [PORT] [KEY VALUE ...])

Causes all test results for the test-procedure to be passed to the echo-procedure. The signature of an echo-procedure is (result-object port association-list -> unspecified). The association-list holds options for the echo-procedure.

When PORT missing the (current-output-port) is assumed.

The PORT is unique. Only one echo per PORT.

KEY is a symbol. The reserved keys are:

When the KEY echo-procedure is missing the default echo-procedure is used. The test results are echoed to the PORT as an association list. One list per line.

The echo-procedure is invoked after each test element evaluation. As such it must deal with test-container result objects in an incomplete state. Specifically the result-list field will be #f when a test-container is entered. It also must deal with ignore-results, which are stripped from the test-container result-list upon exit from the container.

test::remove-echo!
[procedure] (test::remove-echo! TEST-IDENTIFIER [PORT])

Removes the PORT echo for the test-procedure.

test::set-echo-option!
[procedure] (test::set-echo-option! TEST-IDENTIFIER [PORT] [KEY VALUE ...])

Modifies the echo specification for the specified PORT.

The PORT, KEY, and VALUE parameters have the same interpretation as the test::add-echo! procedure.

test::run-mode
[procedure] (test::run-mode)

Returns the test run mode, either batch, interactive or driven.

test::selection-mode
[procedure] (test::selection-mode TEST-IDENTIFIER)

Returns the test selection mode, either take or skip.

When the mode is take then unless a test is to be specifically skipped, it is taken.

When the mode is skip then unless a test is to be specifically taken, it is skipped.

The test procedure is always taken.

test::select!
[procedure] (test::select! SELECTION-MODE TEST-IDENTIFIER [TEST-NAME | (TEST-TYPE TEST-NAME)] ...)

SELECTION-MODE is one of 'take or 'skip.

TEST-TYPE is one of 'suite, 'case, or 'expect.

Select the test element(s) identified by (TEST-TYPE TEST-NAME) of the test-procedure. When only TEST-NAME is provided any test type will match.

When no tests are supplied all existing selections are removed and the test selection mode is set to the SELECTION-MODE.

(test::select! 'skip my-test-proc)
;; Now every test in 'my-test-proc' will be skipped.

(test::select! 'take my-test-proc '(case "Top Case") "Foo")
;; Now every test in 'my-test-proc' except the test-case "Top Case",
;; and any test named "Foo".

When a test container is selected all contained tests are similarly selected.

Independent of the (skip CLAUSE ...) form.

test::selections
[procedure] (test::selections TEST-IDENTIFIER)

Returns a list of the test selections, or #f when there are no selections.

The list is composed of sublists - (#f|test|suite|case|expect TEST-NAME take|skip).

test::resource-limit
[procedure] (test::resource-limit TEST-IDENTIFIER RLIMIT)

Returns the current value of RLIMIT for the test-procedure. The return value will either be a number or #f, indicating no set limit.

RLIMIT is one of 'cpu, 'fsize, 'nofile, 'nproc, 'data.

Only operational on *NIX platforms.

test::resource-limit-set!
[procedure] (test::resource-limit-set! TEST-IDENTIFIER RLIMIT VALUE)

Sets the current value of RLIMIT for the test-procedure to VALUE.

RLIMIT is one of 'cpu, 'fsize, 'nofile, 'nproc, 'data.

VALUE is a positive integer or #f, indicating no set limit.

When a 'cpu or 'fsize limit is reached the test will terminate with a termination result indicating the resource limit exceeded.

When a 'nofile, 'nproc, or 'data limit is reached the test will terminate with a termination result.

Only operational on *NIX platforms.

test::procedures
[procedure] (test::procedures)

Returns a list of all the test-procedures.

test::for-each
[procedure] (test::for-each PROCEDURE [TEST-IDENTIFIER ...])

Calls PROCEDURE with every supplied test procedure, or all test procedures when nothing supplied.

PROCEDURE takes one argument, a test-procedure.

test::map
[procedure] (test::map PROCEDURE [TEST-IDENTIFIER ...])

Calls PROCEDURE with every supplied test procedure, or all test procedures when nothing supplied. Returns a list of the procedure results.

PROCEDURE takes one argument, a test-procedure.

test::forget!
[procedure] (test::forget! [TEST-IDENTIFIER ...])

Causes the specified test-procedures, or all when nothing specified, to be dropped.

Really only for use during test development with a REPL.

test::exit-on-failure?
[procedure] (test::exit-on-failure? [CODE-OR-FLAG])

When CODE-OR-FLAG is supplied sets the early-exit-upon-failure-option. Otherwise returns the current setting.

CODE-OR-FLAG is either #f, to reset the option, or an integer, to set the option.

The initial setting is #f.

test::run
[procedure] (test::run [MESSAGE [TEST-IDENTIFIER ...]])

MESSAGE is some identity for the set of the test-procedures. When missing or #f the process command name is used.

Displays a header with MESSAGE, and outputs the result of calling each test-procedure with the test::styler. When a test::styler is not set a simple success/failure message is given. The test-procedures are called in left to right order of the TEST-IDENTIFIER arguments. When a TEST-IDENTIFIER is not specified all the defined test-procedures are called in the order of definition in the test-file.

For parameterized test-procedures the corresponding test declarations are run.

When (test::exit-on-failure?) and some test fails then testbase will exit with the exit-code.

When interactive mode without a console there is no output by default. See test::add-echo!.

Run Tests

Run test-procedures and output test results.

run-test
[syntax] (run-test [MESSAGE] [FORM ...])

Usually the last expression in a test file. Performs command-line processing.

MESSAGE is some identity for the test. When missing the process command name is used. Use "" for no message.

FORM ... are instances of Scheme forms. When missing (test::run MESSAGE) is used.

This macro will check the process command line for a batch mode (-b or --batch) or interactive mode (-i --interactive) option. When interactive mode without an attached console the mode is called driven. The default is batch mode.

When batch mode the FORM ... will be evaluated. When FORM ... are missing then <em>every</em> test-procedure is run with test result output using the test-procedure styler. See test::run.

When interactive mode a REPL will be entered.

When driven mode a REPL will be entered. Test results will be reported as for test::add-echo!.

Test macros are not automatically available in the REPL. They must be explicitly loaded.

Test Macros

Test macros are used to generate a test result object. A test procedure body is composed of a sequence of test macros.

The test macros are useful only within a test procedure or a test container. A runtime error will result when used outside of a test procedure.

Test Containers

Test containers are test forms that group test forms. Test container are available in long and short forms. The short form provides fewer options. A test container imposes an evaluation termination mode on the contained tests. Test containers may nest.

test-suite
[syntax] (test-suite TEST-NAME DTOR-NAME ESCR-NAME [(warn MESSAGE)] [((VARIABLE VALUE) ...)] CLAUSE ...)
[syntax] (test/suite TEST-NAME [((VARIABLE VALUE) ...)] CLAUSE ...)

Evaluates all contained test forms, CLAUSE, in a left to right order, thru any failures.

TEST-NAME must evaluate to a Scheme atom. Note that #f is interpreted as a missing name. Further, #f is assumed when a name is missing. Thus, the name of all unnamed test elements is the same - #f.

DTOR-NAME is an unquoted symbol for an automatic destructor object that gets called when the test suite completes for any reason. This symbol is bound to a destructor object and is available to you in the CLAUSE ... section of the test suite. See below for the description of the destructor object interface.

test/suite-dtor is the test/suite destructor name.

ESCR-NAME is an unquoted symbol for an escape procedure available in the body of the test suite. Usually this escape procedure is passed to (terminate ...) which calls it for you and performs other tasks. It is not recommended to call the escape procedure directly.

test/suite-escr is the test/suite escape procedure name.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the test suite. The warn function name is actually a syntax reserved word in the macro.

((VARIABLE VALUE) ...) are let-style bindings that you may create and exist in the lexical scope of the test suite.

CLAUSE ... are uses of a test container, annotation, or expectation macros. The clauses are evaluated in a left to right fashion.

While you may use the expectation macros directly in a test suite, doing so has a drawback. If the expectation fails, the test suite macro will continue evaluating until all clauses are evaluated or the escape procedure mechanism is activated. This is different from a test-case macro where upon discovery of a failed expectation, evaluation stops immediately.

test-case
[syntax] (test-case TEST-NAME DTOR-NAME ESCR-NAME [(warn MESSAGE)] [((VARIABLE VALUE) ...)] CLAUSE ...)
[syntax] (test/case TEST-NAME (warn MESSAGE)] [((VARIABLE VALUE) ...)] CLAUSE ...)

Evaluates all contained test forms, CLAUSE, in a left to right order, until failure.

TEST-NAME must evaluate to a Scheme atom.

DTOR-NAME is an unquoted symbol for an automatic destructor object that gets called when the test case completes for any reason. This symbol is bound to a destructor object and is available to you in the CLAUSE ... section of the test case. See below for the description of the destructor object interface.

test/case-dtor is the test/case destructor name.

ESCR-NAME is an unquoted symbol for an escape procedure available in the body of the test case. Usually this escape procedure is passed to (terminate ...) which calls it for you and performs other tasks. It is not recommended to call the escape procedure directly.

test/case-escr is the test/case escape procedure name.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the test case. The warn function name is actually a syntax reserved word in the macro.

((VARIABLE VALUE) ...) are let-style bindings that you may create and exist in the lexical scope of the test case.

CLAUSE ... are uses of a test container, annotation, or expectation macros. The clauses are evaluated in a left to right fashion.

Note: Upon discovery of a failed expectation, the test case stops its evaluation and returns with the test results up thru the failed expectation. This behavior can be changed using the test-case-evaluate-thru-failure macro. Reverts to default at the start of each test-case.

test-collect
[syntax] (test-collect TEST-NAME DTOR-NAME ESCR-NAME [(warn MESSAGE)] [((VARIABLE VALUE) ...)] EXPRESSION ...)
[syntax] (test/collect TEST-NAME (warn MESSAGE)] [((VARIABLE VALUE) ...)] EXPRESSION ...)

A special version of test-case. Since every test form must evaluate to a test-result the use of looping forms to generate test values is very difficult. The collect container provides a method to gather the results of such generated tests.

TEST-NAME must evaluate to a Scheme atom.

DTOR-NAME is an unquoted symbol for an automatic destructor object that gets called when the test collect completes for any reason. This symbol is bound to a destructor object and is available to you in the CLAUSE ... section of the test collect. See below for the description of the destructor object interface.

test/collect-dtor is the test/collect destructor name.

ESCR-NAME is an unquoted symbol for an escape procedure available in the body of the test collect. Usually this escape procedure is passed to (terminate ...) which calls it for you and performs other tasks. It is not recommended to call the escape procedure directly.

test/collect-escr is the test/collect escape procedure name.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the test collect. The warn function name is actually a syntax reserved word in the macro.

((VARIABLE VALUE) ...) are let-style bindings that you may create and exist in the lexical scope of the test collect.

EXPRESSION is either an arbitrary Scheme expression or the form (collect-test CLAUSE), where CLAUSE is a use of a test container, annotation, or expectation macro. The expressions are evaluated in a left to right fashion.

The collect-test macro "appends" the test result of the CLAUSE to the results of the enclosing test collect container.

Note: Upon discovery of a failed expectation, the test collect stops its evaluation and returns with the test results up thru the failed expectation. This behavior can be changed using the test-case-evaluate-thru-failure macro. Reverts to default at the start of each test-case.

Test Flags

These macros set test execution flags.

test-case-evaluate-thru-failure
[syntax] (test-case-evaluate-thru-failure BOOLEAN)

This macro will set the test-case evaluation continuation flag.

The starting value is #f.

test-timing
[syntax] (test-timing BOOLEAN)

This macro will set the expectation timing flag.

The starting value is #f.

Note that if timing expansion is turned off setting this flag has no effect.

test-evaluation-mode?
[syntax] (test-evaluation-mode?)

This macro will expand into a test for the evaluation mode.

Potentially useful within test procedure initial and final forms.

Test Binding Forms

test-letrec
[syntax] (test-letrec ((VARIABLE VALUE) ...) CLAUSE ...)

Defines VARIABLE, in a new lexical scope, as the result of evaluating EXPRESSION.

CLAUSE ... are as for test-suite. Evaluated in the scope of the definitions.

The last CLAUSE ... in the clause body <em>must</em> be a test form of some kind, such as an expectation.

test-let*
[syntax] (test-let* ((VARIABLE VALUE) ...) CLAUSE ...)

Defines VARIABLE, in a new lexical scope, as the result of evaluating EXPRESSION.

CLAUSE ... are as for test-suite. Evaluated in the scope of the definitions.

The last CLAUSE ... in the clause body must be a test form of some kind, such as an expectation.

test-let
[syntax] (test-let ((VARIABLE VALUE) ...) CLAUSE ...)

Defines VARIABLE, in a new lexical scope, as the result of evaluating EXPRESSION.

CLAUSE ... are as for test-suite. Evaluated in the scope of the definitions.

The last CLAUSE ... in the clause body <em>must</em> be a test form of some kind, such as an expectation.

Test Expectations

An expectation at its core simply evaluates its arguments and check to see if it matches the expectation. The positive or negative result is encapsulated, along with other things such as the unevaluated expressions being checked and some messages supplied with each expectation into a particular type of black box object that one can query with the appropriate API calls (detailed below).

Expectations all have a descriptive message that can be bound to them, along with an optional warning syntax detailed below. A design decision was made to supply expectation macros for the usual types of expectations a user needs because this reduced the abstractness of an expectation into something more manageable.

Expectations are normally evaluated with an exception catcher. An unexpected exception is treated as expectation failure. The option may be defeated by redefining test::if-catching.

Any exceptions in setup or teardown expressions will be caught and treated as an abnormal termination of the entire test.

insist
[syntax] (insist [TEST-NAME] [(warn MESSAGE)] EXPRESSION => EXPECTED)
[syntax] (insist [TEST-NAME] [(warn MESSAGE)] EXPRESSION (=> PREDICATE) EXPECTED)

This expectation checks to see if the evaluated expression passed to it meets the predicate condition.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

PREDICATE is a procedure of two arguments to perform the test. The default is equal?.

EXPECTED is evaluated and represents the value the EXPRESSION <em>must</em> match via PREDICATE in order for this expectation to return a positive result.

EXPRESSION should return a value meeting the conditions of the predicate.

expect-unary
[syntax] (expect-unary [TEST-NAME] [(warn MESSAGE)] KIND PREDICATE EXPRESSION)

This expectation checks to see if the evaluated expression passed to it meets the predicate condition. Allows unary and binary predicates.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

KIND is a string stating the kind of test.

PREDICATE is a procedure of one argument to perform the test.

EXPRESSION should return a value meeting the conditions of the predicate.

expect-binary
[syntax] (expect-binary [TEST-NAME] [(warn MESSAGE)] KIND PREDICATE EXPECTED EXPRESSION)

This expectation checks to see if the evaluated expression passed to it meets the predicate condition. Allows unary and binary predicates.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

KIND is a string stating the kind of test.

PREDICATE is a procedure of two arguments to perform the test.

EXPECTED is evaluated and represents the value the EXPRESSION <em>must</em> match via PREDICATE in order for this expectation to return a positive result.

EXPRESSION should return a value meeting the conditions of the predicate.

expect-zero
[syntax] (expect-zero [TEST-NAME] [(warn MESSAGE)] EXPRESSION)

This expectation checks to see if the evaluated expression passed to it is numerically equal to zero.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPRESSION should return a number.

expect-nonzero
[syntax] (expect-nonzero [TEST-NAME] [(warn MESSAGE)] EXPRESSION)

This expectation checks to see if the evaluated expression passed to it is numerically not equal to zero.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPRESSION should return a number.

expect-positive
[syntax] (expect-positive [TEST-NAME] [(warn MESSAGE)] EXPRESSION)

This expectation checks to see if the evaluated expression passed to it is a positive value greater than zero.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPRESSION should return a number.

expect-negative
[syntax] (expect-negative [TEST-NAME] [(warn MESSAGE)] EXPRESSION)

This expectation checks to see if the evaluated expression passed to it is a negative value less than zero.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPRESSION should return a number.

expect-near
[syntax] (expect-near [TEST-NAME] [(warn MESSAGE)] EXPECTED [TOL] EXPRESSION)

This expectation checks to see if (< (abs (- EXPECTED EXPRESSION)) TOL))) is true.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPECTED is evaluated and represents the value the EXPRESSION <em>must</em> be "near" to in order for this expectation to return a positive result.

EXPRESSION should return a number.

TOL is a single expression which, when evaluated, must return a tolerance value, a small inexact number. When missing the current tolerance value is used, usually 0.0001.

The (expect-near EXPECTED TOL EXPRESSION) variant is not supported!

expect-true
[syntax] (expect-true [TEST-NAME] [(warn MESSAGE)] EXPRESSION)

This expectation checks to see if the evaluated expression passed to it is the value #t.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPRESSION should return #t.

expect-false
[syntax] (expect-false [TEST-NAME] [(warn MESSAGE)] EXPRESSION)

This expectation checks to see if the evaluated expression passed to it is the value #f.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPRESSION should return #f.

expect-success
[syntax] (expect-success [TEST-NAME] [(warn MESSAGE)] EXPRESSION)

This expectation checks to see if the evaluated expression passed to it is not the value #f, or a non-error exception occurred.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPRESSION should return #t, or generate a non-error exception.

expect-failure
[syntax] (expect-failure [TEST-NAME] [(warn MESSAGE)] EXPRESSION)

This expectation checks to see if the evaluated expression passed to it is the value #f, or an error exception occurred.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPRESSION should return #f, or generate an error exception.

expect-not-false
[syntax] (expect-not-false [TEST-NAME] [(warn MESSAGE)] EXPRESSION)

This expectation checks to see if the evaluated expression passed to it is not the value #f.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPRESSION should not return #f, any other value is accepted.

expect-not-null
[syntax] (expect-not-null [TEST-NAME] [(warn MESSAGE)] EXPRESSION)

This expectation checks to see if the evaluated expression passed to it is not the value #f.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPRESSION should not return #f, any other value is accepted.

expect-eq
[syntax] (expect-eq [TEST-NAME] [(warn MESSAGE)] EXPECTED EXPRESSION)

This expectation checks to see if (eq? EXPECTED EXPRESSION) is true.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPECTED is evaluated and represents the value the EXPRESSION must be eq? to in order for this expectation to return a positive result.

EXPRESSION is a single expression which, when evaluated must return an object where an eq? of this result and the EXPECTED expression is #t.

The result object this macro produce shall contain the unevaluated EXPRESSION expression as a field, but not an unevaluated EXPECTED expression.

expect-eqv
[syntax] (expect-eqv [TEST-NAME] [(warn MESSAGE)] EXPECTED EXPRESSION)

This expectation checks to see if (eqv? EXPECTED EXPRESSION) is true.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPECTED is evaluated and represents the value the EXPRESSION <em>must</em> be eqv? to in order for this expectation to return a positive result.

EXPRESSION is a single expression which, when evaluated must return an object where an eqv? of this result and the EXPECTED expression is #t.

The result object this macro produce shall contain the unevaluated EXPRESSION expression as a field, but not an unevaluated EXPECTED expression.

expect-equal
[syntax] (expect-equal [TEST-NAME] [(warn MESSAGE)] EXPECTED EXPRESSION)

This expectation checks to see if (equal? EXPECTED EXPRESSION) is true.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPECTED is evaluated and represents the value the EXPRESSION <em>must</em> be equal? to in order for this expectation to return a positive result.

EXPRESSION is a single expression which, when evaluated must return an object where an equal? of this result and the EXPECTED expression is #t.

The result object this macro produce shall contain the unevaluated EXPRESSION expression as a field, but not an unevaluated EXPECTED expression.

expect-unary/values
[syntax] (expect-unary/values [TEST-NAME] [(warn MESSAGE)] KIND (PREDICATE ...) EXPRESSION)

This expectation checks to see if the evaluated multi-valued expression passed to it meets the predicate condition. Allows unary and binary predicates.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

KIND is a string stating the kind of test.

PREDICATE is a procedure taking one argument and returning #t or #f.

EXPRESSION has a multi-valued result.

expect-near/values
[syntax] (expect-near/values [TEST-NAME] [(warn MESSAGE)] KIND (PREDICATE ...) VALUES EXPRESSION)

This expectation checks to see if the evaluated multi-valued expression passed to it meets the predicate condition. Allows unary and binary predicates.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

KIND is a string stating the kind of test.

PREDICATE is a procedure taking two arguments and returning #t or #f.

VALUES are the expected values.

EXPRESSION has a multi-valued result.

expect-near/values
[syntax] (expect-near/values [TEST-NAME] [(warn MESSAGE)] EXPECTED [TOL] EXPRESSION)

This expectation checks to see if (< (abs (- EXPECTED[i] EXPRESSION[i])) TOL))) is true.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPECTED is evaluated and represents the values the EXPRESSION <em>must</em> be "near" to in order for this expectation to return a positive result.

EXPRESSION should return one or more inexact or exact number.

TOL is a single expression which, when evaluated, must return a tolerance value, a small inexact number. When missing the current tolerance value is used, usually 0.0001.

The (expect-near/values EXPECTED TOL EXPRESSION) variant is not supported!

expect-eq/values
[syntax] (expect-eq/values [TEST-NAME] [(warn MESSAGE)] VALUES EXPRESSION)

This expectation checks to see if the result of the evaluated multi-valued expression passed to it is element-wise eq? to the VALUES list. The count of returned multiple values must match the length of the VALUES list.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

VALUES is the expected values list.

EXPRESSION has a multi-valued result.

expect-eqv/values
[syntax] (expect-eqv/values [TEST-NAME] [(warn MESSAGE)] VALUES EXPRESSION)

This expectation checks to see if the result of the evaluated multi-valued expression passed to it is element-wise eqv? to the VALUES list. The count of returned multiple values must match the length of the VALUES list.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

VALUES is the expected values list.

EXPRESSION has a multi-valued result.

expect-equal/values
[syntax] (expect-equal/values [TEST-NAME] [(warn MESSAGE)] VALUES EXPRESSION)

This expectation checks to see if the result of the evaluated multi-valued expression passed to it is element-wise equal? to the VALUES list. The count of returned multiple values must match the length of the VALUES list.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

VALUES is the expected values list.

EXPRESSION has a multi-valued result.

expect-exception
[syntax] (expect-exception [TEST-NAME] [(warn MESSAGE)] EXCEPTION-PATTERN EXPRESSION)

This expectation checks to see if the evaluated expression passed to it signals an exception that matches the supplied exception pattern EXCEPTION-PATTERN.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXCEPTION-PATTERN is a specialized logical expression describing the expected exception, a condition as defined by SRFI-12.

EXPRESSION should signal an exception matching EXCEPTION-PATTERN.

 EXCEPTION-PATTERN : COND-EXN
                   | SIMP-EXN
                   | ()
 
 SIMP-EXN : KINDKEY
          | (property KINDKEY PROP-EXPR ...)
          | (SIMP-EXN ...)
 
 COND-EXN : (and EXN ...)
          | (or EXN ...)
          | (not EXN)
 
 PROP-EXPR : COND-PROP-EXPR
           | SIMP-PROP-EXPR
           | ()
 
 SIMP-PROP-EXPR : PROPKEY
                | (PROPKEY ...)
                | (PROPKEY VALUE)
                | (PREDICATE PROPKEY VALUE)
 
 COND-PROP-EXPR : (and PROP-EXPR ...)
                | (or PROP-EXPR ...)
                | (not PROP-EXPR)

An example:

(define-test ...
  ...
  (test-case "exn1" d e (
      [exn1 (make-composite-condition
              (make-property-condition 'abc 'cbs "pbs")
              (make-property-condition 'foo 'bar "zip"))])
  
    (expect-exception "1" (foo abc) (signal exn1))
  
      ;; These are the same test, the 2nd has an explicit predicate
    (expect-exception "2" ((property foo (bar "zip"))
                           (property abc (cbs "pbs"))) (signal exn1))
    (expect-exception "3" ((property foo (equal? bar "zip"))
                           (property abc (equal? cbs "pbs"))) (signal exn1))
  )
  ...
)
expect-ec
[syntax] (expect-ec [TEST-NAME] [(warn MESSAGE)] QUALIFIER ... EXPECTED [=> PREDICATE] EXPRESSION)

This expectation checks to see if (PREDICATE EXPECTED EXPRESSION) is true for every state produced by the set of QUALIFIER.

TEST-NAME must evaluate to a Scheme atom.

(warn MESSAGE) allows you to specify a warning object, usually a string, that gets associated with the expectation. The warn function name is actually a syntax reserved word in the macro.

EXPRESSION and EXPECTED are expressions.

PREDICATE is a two argument procedure returning a boolean, equal? is used when missing.

QUALIFIER is a SRFI-42 <qualifier>. An eager comprehension implementation, as described by SRFI-42 must be provided to use this expectation.

expect-set!
[syntax] (expect-set! [TEST-NAME] VARIABLE EXPRESSION)

Evaluates EXPRESSION as in expect-success. If successful the variable VARIABLE is bound to the result of evaluating the EXPRESSION.

EXPRESSION cannot evaluate to #f or a condition.

Define New Expectations

An API to define new expectation macros. May only be used at the toplevel.

define-expect-unary
[syntax] (define-expect-unary PREDICATE [SUFFIX [KIND]])

Defines a new unary expectation.

PREDICATE is the name of a procedure of one argument to perform the test.

SUFFIX is a symbol to create the expectation macro name. When missing the PREDICATE is used, without a ? suffix, if any.

KIND is a string stating the kind of test. When missing the string form of the SUFFIX is used.

define-expect-binary
[syntax] (define-expect-binary PREDICATE [SUFFIX [KIND]])

Defines a new binary expectation.

PREDICATE is the name of a procedure of two arguments to perform the test.

SUFFIX is a symbol to create the expectation macro name. When missing the PREDICATE is used, without a ? suffix, if any.

KIND is a string stating the kind of test. When missing the string form of the SUFFIX is used.

define-expect-unary/values
[syntax] (define-expect-unary/values PREDICATE [SUFFIX [KIND]])

Defines a new multivalued unary expectation.

PREDICATE is the name of a procedure of one list argument to perform the test.

SUFFIX is a symbol to create the expectation macro name. When missing the PREDICATE is used, without a ? suffix, if any.

KIND is a string stating the kind of test. When missing the string form of the SUFFIX is used.

define-expect-binary/values
[syntax] (define-expect-binary/values PREDICATE [SUFFIX [KIND]])

Defines a new multivalued binary expectation.

PREDICATE is the name of a procedure of two list arguments to perform the test.

SUFFIX is a symbol to create the expectation macro name. When missing the PREDICATE is used, without a ? suffix, if any.

KIND is a string stating the kind of test. When missing the string form of the SUFFIX is used.

define-expect-nary
[syntax] (define-expect-nary (SUFFIX ARG ...) EXPR ...)

Defines a new n-ary expectation with the name expect-SUFFIX.

(expect-SUFFIX MESSAGE [(warning WARNING)] VAL ...). Unlike other expectations either the MESSAGE or a WARNING must be supplied.

SUFFIX is a symbol.

ARG ... are the parameter variable symbols.

EXPR ... is the body, which must return #f for failure.

Testeez Synonyms

Syntax forms mimicking some of the testeez test framework forms. Expanded into the corresponding expect-* macros.

The formal argument names match those in the testeez documentation.

Following the testeez semantics all test forms are always evaluated as having possible multiple valued returns.

test-eval
[syntax] (test-eval [TEST-NAME] EXPRESSION)
test/eq
[syntax] (test/eq [TEST-NAME] EXPRESSION EXPECTED)
test/eqv
[syntax] (test/eqv [TEST-NAME] EXPRESSION EXPECTED)
test/equal
[syntax] (test/equal [TEST-NAME] EXPRESSION EXPECTED)

Test Setup (Side-Effect)

The side effecting evaluates all of its arguments as in a (begin ...) form, it returns a result that is completely ignored by the system and unavailable to the output analysis code.

side-effect
[syntax] (side-effect CLAUSES)

This macro expands into a begin form with the clauses in order. When it finishes evaluating them, a result is returned that is silently ignored by the testbase system.

Usually this is used in conjunction with (set! ...) or with complicated situations where a lot of setup work must happen for an expectation to be performed.

Define forms cannot be expressed here.

setup
[syntax] (setup CLAUSES)

A synonym for side-effect.

Test Teardown (Destructor)

The destructor object allows for you to create helper functions which clean up in case of aborting from a test container.

For example, suppose you are testing whether or not file writing to a file works correctly in a test case, so, you'd perform an expectation to open the file, and then queue a function in the destructor to remove the file, and then perform the expectation of the write. If the write (or subsequent) expectation fails, then the test case will automatically invoke the helper cleanup function specified in the destructor object that removes the file.

destructor-atexit!
[syntax] (destructor-atexit! [DTOR-NAME] CALL ...)
[syntax] (destructor-atexit! DTOR-NAME PROC ARGUMENT ...)

DTOR-NAME is the destructor to use, test-dtor when missing.

A CALL is a list where the first element is a PROC and the tail elements are the argument list; i.e. (PROC ARGUMENT ...).

Each CALL is added to the destructor queue, promised to be invoked when the DTOR-NAME destructor object is activated.

The second form will insert a promise to invoke (PROC ARGUMENT ...) into a queue in the DTOR-NAME destructor object.

Multiple invocations of this API call will continue to queue up (PROC ARGS ...) promises indefinitely.

Note: The PROC and ARGUMENT ... parameters are evaluated at the point of enqueue; i.e. the current lexical, and dynamic, environment of the macro expansion. This means a variable referenced will have this value when the destructor is activated, and not the "current" value! So no "forward" references.

This macro returns a special ignore-result type that is ignored by the testbase system.

teardown
[syntax] (teardown [DTOR-NAME] (FUNC ARGUMENT ...) ...)

A (FUNC ARGUMENT ...) for is also refered to as a CALL.

A synonym of destructor-atexit!.

[syntax] (destructor-activate! [DTOR-NAME])

DTOR-NAME is the destructor to use, test-dtor when missing.

This macro will call, in order of queueing, all the promises embedded into this destructor object, and then delete the queue. The destructor is always called at the completion of a test container; so be careful that the destructor object doesn't contain anything harmful.

This macro returns a special ignore-result type that is ignored by the testbase system.

destructor-clear!
[syntax] (destructor-clear! [DTOR-NAME])

DTOR-NAME is the destructor to use, test-dtor when missing.

This macro completely removes all of the promises associated with the destructor object DTOR-NAME.

This macro returns a special ignore-result type that is ignored by the testbase system.

destructor-dump
[syntax] (destructor-dump [DTOR-NAME])

DTOR-NAME is the destructor to use, test-dtor when missing.

This macro, mostly used for debugging purposes, prints out a simple representation of the queued atexit functions to the current port.

This macro returns a special ignore-result type that is ignored by the testbase system.

Gloss

The purpose of the gloss API is to allow the author of a test suite the ability to record messages into the result tree purely for documentation purposes.

gloss
[syntax] (gloss MESSAGE [(warn WARNING)])

MESSAGE can be any scheme object, though usually it is a string.

(warn WARNING) allows you to specify a warning object, usually a string, that gets associated with the gloss. The warn function name is actually a syntax reserved word in the macro.

Skip

The purpose of the skip API is to allow the author of a test suite to completely skip evaluation of a set of expressions.

skip
[syntax] (skip MESSAGE [(warn WARNING)] CLAUSES)

MESSAGE can be any scheme object, though usually it is a string.

(warn WARNING) allows you to specify a warning object, usually a string, that gets associated with the gloss. The warn function name is actually a syntax reserved word in the macro.

CLAUSES can be more than one expression (as in a lambda form) that does <em>NOT</em> get evaluated at any time.

Termination

When executing in a test suite or a test case, one might discover some catastrophic failure of such proportions that it is utterly impossible to continue executing the test case or test suite. When that happens you can use the termination facility to exit the test case or test suite. Of course, no more expressions will be evaluated in the scope of the termination. It is recommended that you use this method of terminating the test procedure, test case or test suite evaluation since it wraps some contextual information up into the termination result so you can figure out what happened (and where) later when analyzing the result tree.

terminate
[syntax] (terminate MESSAGE)
[syntax] (terminate MESSAGE FINDING)
[syntax] (terminate ESCR-NAME MESSAGE)
[syntax] (terminate ESCR-NAME MESSAGE FINDING)

This is the recommended termination method for a test procedure, test case, or a test suite.

ESCR-NAME is the name of the termination procedure that was specified in a test procedure, test case or test suite. You may pass any test suite or test case termination function available to you in the lexical scope in which you call this function. The termination will take effect in the scope of the created termination function.

MESSAGE can be any scheme object, though usually it is a string.

FINDING can be any scheme object, though usually it is a boolean.

Todo

The purpose of the todo API is to allow the author of a test suite the ability to record into the result tree for later analysis that something still needs to be done. This way you can count/manipulate this information at a later date.

todo
[syntax] (todo MESSAGE [(warn WARNING)])

MESSAGE can be any scheme object, though usually it is a string.

(warn WARNING) allows you to specify a warning object, usually a string, that gets associated with the todo. The warn function name is actually a syntax reserved word in the macro.

Test Result Object

Test procedures, test containers, expectations, and helper macros (gloss, todo, etc) all return an object that contains the results and other various aspects of the action performed which ultimately get wired together to form the result tree. This collection of functions forming the rest of the testbase API allows manipulation of these results in an abstracted way as to allow changing of the representation in the future.

Any Test Result Object

This section contains a few functions whose purpose is to simplify certain kinds of manipulations of result objects. Will generate an error for an invalid result object.

test::result?
[procedure] (test::result? RESULT-OBJ)

This function will return #t if RESULT-OBJ is any kind of an evaluated result object. Otherwise, it shall return #f.

test::ignore-result?
[procedure] (test::ignore-result? RESULT-OBJ)

This function will return #t if RESULT-OBJ is an ignore result object. Otherwise, it shall return #f. An ignore object is not considered an evaluated result object.

test::result-passed?
[procedure] (test::result-passed? RESULT-OBJ)

This function will return whether the boolean> finding of any kind of RESULT-OBJ passed to it if it is indeed a true result object.

test::result-finding
[procedure] (test::result-finding RESULT-OBJ)

This function will return the finding of any kind of RESULT-OBJ passed to it if it is indeed a true result object.

The finding is a variant type: boolean | condition.

test::result-name
[procedure] (test::result-name RESULT-OBJ)

This function will return the TEST-NAME of any kind of RESULT-OBJ passed to it if it is indeed a true result object.

test::result-message
[procedure] (test::result-message RESULT-OBJ)

This function will return the MESSAGE of any kind of RESULT-OBJ passed to it if it is indeed a true result object.

test::result-warning?
[procedure] (test::result-warning? RESULT-OBJ)

This function will return whether a warning is defined if RESULT-OBJ is any kind of an evaluated result object.

test::result-warning
[procedure] (test::result-warning RESULT-OBJ)

This function will return the warning of any kind of RESULT-OBJ passed to it if it is indeed a true result object.

test::result-timing?
[procedure] (test::result-timing? RESULT-OBJ)

This function will return whether a timing is defined if RESULT-OBJ is any kind of an evaluated result object.

test::result-timing
[procedure] (test::result-timing RESULT-OBJ)

This function will return the timing values of any kind of RESULT-OBJ passed to it if it is indeed a true result object.

The timing values will be a pair of the milliseconds spent in user code and in system code, for the test corresponding to this test::result-object.

For example, the list (6 . 1) indicates 6 milliseconds spent in user code and 1 millisecond spent in system code.

test::result-timing-message
[procedure] (test::result-timing-message RESULT-OBJ)

This function will return the timing string representation of any kind of RESULT-OBJ passed to it if it is indeed a true result object.

test::result-kind
[procedure] (test::result-kind RESULT-OBJ)

This function will return the kind of any RESULT-OBJ passed to it if it is indeed a true result object.

Test Container Result Object

Test Suite Result Object

If any of these API functions, except test::test-suite-result?, are passed something that isn't a test suite result object, they will return an undefined value.

test::test-suite-result?
[procedure] (test::test-suite-result? RESULT)

If RESULT is a result object from the invocation of a test suite macro, then this function will return #t. Otherwise, it will return #f.

test::test-suite-result-passed?
[procedure] (test::test-suite-result-passed? RESULT)

Returns the boolean result associated with the test suite RESULT object.

test::test-suite-result-finding
[procedure] (test::test-suite-result-finding RESULT)

Returns the finding result associated with the test suite RESULT object.

test::test-suite-result-name
[procedure] (test::test-suite-result-name RESULT)

Returns the message object associated with the test suite RESULT object.

test::test-suite-result-list
[procedure] (test::test-suite-result-list RESULT)

Returns the list of result objects associated with the test suite RESULT object.

test::test-suite-result-warning?
[procedure] (test::test-suite-result-warning? RESULT)

If a warning had been attached to this test suite, this function will return #t, otherwise it will be #f.

test::test-suite-result-warning
[procedure] (test::test-suite-result-warning RESULT)

If a warning had been attached to this test suite, this function will return the warning message object supplied by the user, otherwise it shall return '().

test::test-suite-result-timing?
[procedure] (test::test-suite-result-timing? RESULT)

If a timing had been attached to this test suite, this function will return #t, otherwise it will be #f.

test::test-suite-result-timing
[procedure] (test::test-suite-result-timing RESULT)

If a timing had been attached to this test suite, this function will return the timing pair, otherwise it shall return '(0 . 0).

test::test-suite-results-passed?
[procedure] (test::test-suite-results-passed? RESULT-LIST)

Returns #t if every single contained result in the RESULT-LIST passed, or #f otherwise.

Test Case Result Object

If any of these API functions, except test::test-case-result?, are passed something that isn't a test case result object, they will return an undefined value.

test::test-case-result?
[procedure] (test::test-case-result? RESULT)

If RESULT is a result object from the invocation of a test case macro, then this function will return #t. Otherwise, it will return #f.

test::test-case-result-passed?
[procedure] (test::test-case-result-passed? RESULT)

Returns the boolean result associated with the test case RESULT object.

test::test-case-result-finding
[procedure] (test::test-case-result-finding RESULT)

Returns the finding result associated with the test case RESULT object.

test::test-case-result-name
[procedure] (test::test-case-result-name RESULT)

Returns the message object associated with the test case RESULT object.

test::test-case-result-list
[procedure] (test::test-case-result-list RESULT)

Returns the list of result objects associated with the test case RESULT object.

test::test-case-result-warning?
[procedure] (test::test-case-result-warning? RESULT)

If a warning had been attached to this test case, this function will return #t, otherwise it will be #f.

test::test-case-result-warning
[procedure] (test::test-case-result-warning RESULT)

If a warning had been attached to this test case, this function will return the warning message object supplied by the user, otherwise it shall return '().

test::test-case-result-timing?
[procedure] (test::test-case-result-timing? RESULT)

If a timing had been attached to this test case, this function will return #t, otherwise it will be #f.

test::test-case-result-timing
[procedure] (test::test-case-result-timing RESULT)

If a timing had been attached to this test case, this function will return the timing pair, otherwise it shall return '(0 . 0).

test::test-case-results-passed?
[procedure] (test::test-case-results-passed? RESULT-LIST)

Returns #t if every single contained result in the RESULT-LIST passed, or #f otherwise.

Expectation Result Objects

Procedures used to access the various kinds of expectation result object.

Unary Expectation

These expectations all take the form of passing a single expression to them to see if they match some a priori expectation. If any of these API functions, except test::expect-result?, are passed something that isn't a single clause style expectation result object, they will return 'not-an-expect-result.

test::expect-result?
[procedure] (test::expect-result? RESULT)

If RESULT is a single clause style result object from the invocation of an expectation macro, then this function will return #t. Otherwise, it will return #f.

test::expect-result-finding
[procedure] (test::expect-result-finding RESULT)

Returns the boolean result associated with the single clause style expectation RESULT object.

test::expect-result-kind
[procedure] (test::expect-result-kind RESULT)

This retrieves the "kind" field of a particular single clause style expectation. For example, if you had a result object from an invocation of a (expect-zero "foobar" (- 1 1)) expectation, then the "kind" field of the expectation result object will be the string "zero". Here is a table describing what the "kind" fields are for each kind of single clause style expectation:

Unary Expectation Associated Specific String
expect-zero "zero"
expect-nonzero "nonzero"
expect-true "true"
expect-false "false"
expect-not-false "not-false"
expect-not-null "not-null"
expect-positive "positive"
expect-negative "negative"
expect-unary KIND
expect-unary/values KIND
test::expect-result-name
[procedure] (test::expect-result-name RESULT)

Returns the message object associated with the single clause style expectation RESULT object.

test::expect-result-unevaled
[procedure] (test::expect-result-unevaled RESULT)

Returns the unevaluated expression supplied to a single clause style expectation macro.

test::expect-result-evaled
[procedure] (test::expect-result-evaled RESULT)

Returns the evaluated expression supplied to a single clause style expectation macro.

test::expect-result-warning?
[procedure] (test::expect-result-warning? RESULT)

If a warning had been attached to this expectation, this function will return #t, otherwise it will be #f.

test::expect-result-warning
[procedure] (test::expect-result-warning RESULT)

If a warning had been attached to this expectation, this function will return the warning message object supplied by the user, otherwise it shall return '().

test::expect-result-timing?
[procedure] (test::expect-result-timing? RESULT)

If a timing had been attached to this expectation, this function will return #t, otherwise it will be #f.

test::expect-result-timing
[procedure] (test::expect-result-timing RESULT)

If a timing had been attached to this expectation, this function will return the timing pair, otherwise it shall return '(0 . 0).

Binary Expectation

These expectations all take the form of passing a two expressions, the "left hand side" and the "right hand side" to them to see if they match some a priori equivalence. The left hand side is that which you expect the right hand side to be equivalent. If any of these API functions, except test::expect-equivalence-result?, are passed something that isn't a single clause style expectation result object, they will return 'not-an-expect-equivalence-result.

test::expect-equivalence-result?
[procedure] (test::expect-equivalence-result? RESULT)

If RESULT is a comparison style result object from the invocation of an expectation macro, then this function will return #t. Otherwise, it will return #f.

test::expect-equivalence-result-finding
[procedure] (test::expect-equivalence-result-finding RESULT)

Returns the boolean result associated with the comparison style expectation RESULT object.

test::expect-equivalence-result-kind
[procedure] (test::expect-equivalence-result-kind RESULT)

This retrieves the "kind" field of a particular equivalence style expectation. For example, if you had a result object from an invocation of a (expect-equal? "foobar" 0 (- 1 1)) expectation, then the "kind" field of the expectation result object will be the string "equal". Here is a table describing what the "kind" fields are for each kind of equivalence style expectation:

Binary Expectation Associated Specific String
expect-eq "eq"
expect-eqv "eqv"
expect-equal "equal"
expect-exception "exception"
expect-binary KIND
expect/values "values"
expect-eq/values "eq/values"
expect-eqv/values "eqv/values"
expect-equal/values "equal/values"
expect-binary/values KIND
test::expect-equivalence-result-name
[procedure] (test::expect-equivalence-result-name RESULT)

Returns the message object associated with the equivalence style expectation RESULT object.

test::expect-equivalence-result-lhs-evaled
[procedure] (test::expect-equivalence-result-lhs-evaled RESULT)

Returns the evaluated "left hand side" expression supplied to an equivalence style expectation.

test::expect-equivalence-result-rhs-unevaled
[procedure] (test::expect-equivalence-result-rhs-unevaled RESULT)

Returns the unevaluated "right hand side" expression supplied to an equivalence style expectation.

test::expect-equivalence-result-rhs-evaled
[procedure] (test::expect-equivalence-result-rhs-evaled RESULT)

Returns the evaluated "right hand side" expression supplied to an equivalence style expectation.

test::expect-equivalence-result-warning?
[procedure] (test::expect-equivalence-result-warning? RESULT)

If a warning had been attached to this expectation, this function will return #t, otherwise it will be #f.

test::expect-equivalence-result-warning
[procedure] (test::expect-equivalence-result-warning RESULT)

If a warning had been attached to this expectation, this function will return the warning message object supplied by the user, otherwise it shall return '().

test::expect-equivalence-result-timing?
[procedure] (test::expect-equivalence-result-timing? RESULT)

If a timing had been attached to this expectation, this function will return #t, otherwise it will be #f.

test::expect-equivalence-result-timing
[procedure] (test::expect-equivalence-result-timing RESULT)

If a timing had been attached to this expectation, this function will return the timing pair, otherwise it shall return '(0 . 0).

Tolerance Style Expectation

This is a specialized expectation which accepts three expressions and checks to see if the "right hand side" is within a "tolerance" of the "left hand side". There is only one expectation in the tolerance style currently. If any of these API functions, except test::expect-tolerance-result?, are passed something that isn't a tolerance style expectation result object the result is unspecified.

test::expect-tolerance-result?
[procedure] (test::expect-tolerance-result? RESULT)

If RESULT is a tolerance style result object from the invocation of an expectation macro, then this function will return #t. Otherwise, it will return #f.

test::expect-tolerance-result-finding
[procedure] (test::expect-tolerance-result-finding RESULT)

Returns the boolean result associated with the tolerance style expectation RESULT object.

test::expect-tolerance-result-kind
[procedure] (test::expect-tolerance-result-kind RESULT)

This retrieves the "kind" field of a particular tolerance style expectation. For example, if you had a result object from an invocation of a (expect-near? "foobar" 100 .01 100.001) expectation, then the "kind" field of the expectation result object will be the string "near". Here is a table describing what the "kind" fields are for each kind of tolerance style expectation:

Tolerance Style Expectation Associated Specific String
expect-near "near"
test::expect-tolerance-result-name
[procedure] (test::expect-tolerance-result-name RESULT)

Returns the message object associated with a tolerance style expectation RESULT object.

test::expect-tolerance-result-lhs-evaled
[procedure] (test::expect-tolerance-result-lhs-evaled RESULT)

Returns the evaluated "left hand side" expression supplied to a tolerance style expectation.

test::expect-tolerance-result-lhs-tol-evaled
[procedure] (test::expect-tolerance-result-lhs-tol-evaled RESULT)

Returns the evaluated "tolerance" expression supplied to a tolerance style expectation.

test::expect-tolerance-result-rhs-unevaled
[procedure] (test::expect-tolerance-result-rhs-unevaled RESULT)

Returns the unevaluated "right hand side" expression supplied to a tolerance style expectation.

test::expect-tolerance-result-rhs-evaled
[procedure] (test::expect-tolerance-result-rhs-evaled RESULT)

Returns the evaluated "right hand side" expression supplied to a tolerance style expectation.

test::expect-tolerance-result-warning?
[procedure] (test::expect-tolerance-result-warning? RESULT)

If a warning had been attached to this expectation, this function will return #t, otherwise it will be #f.

test::expect-tolerance-result-warning
[procedure] (test::expect-tolerance-result-warning RESULT)

If a warning had been attached to this expectation, this function will return the warning message object supplied by the user, otherwise it shall return '().

test::expect-tolerance-result-timing?
[procedure] (test::expect-tolerance-result-timing? RESULT)

If a timing had been attached to this expectation, this function will return #t, otherwise it will be #f.

test::expect-tolerance-result-timing
[procedure] (test::expect-tolerance-result-timing RESULT)

If a timing had been attached to this expectation, this function will return the timing pair, otherwise it shall return '(0 . 0).

Gloss Result Object
test::gloss-result?
[procedure] (test::gloss-result? RESULT)

If RESULT is a gloss result object from the invocation of the gloss macro, then this function will return #t. Otherwise, it will return #f.

test::gloss-result-message
[procedure] (test::gloss-result-message RESULT)

Returns the message object associated with the gloss RESULT object.

test::gloss-result-warning?
[procedure] (test::gloss-result-warning? RESULT)

If a warning had been attached to this gloss, this function will return #t, otherwise it will be #f.

test::gloss-result-warning
[procedure] (test::gloss-result-warning RESULT)

If a warning had been attached to this gloss, this function will return the warning message object supplied by the user, otherwise it shall return '().

Skip Result Object
test::skip-result?
[procedure] (test::skip-result? RESULT)

If RESULT is a skip result object from the invocation of the skip macro, then this function will return #t. Otherwise, it will return #f.

test::skip-result-message
[procedure] (test::skip-result-message RESULT)

Returns the message object associated with the skip RESULT object. Hopefully, it was stated why this set of clauses had been skipped.

test::skip-result-warning?
[procedure] (test::skip-result-warning? RESULT)

If a warning had been attached to this skip, this function will return #t, otherwise it will be #f.

test::skip-result-warning
[procedure] (test::skip-result-warning RESULT)

If a warning had been attached to this skip, this function will return the warning message object supplied by the user, otherwise it shall return '().

Termination Result Object

When using the manipulation API for a terminate result, if you pass a result to one of these function that is not a terminate result, it will return an unspecified value.

test::terminate-result?
[procedure] (test::terminate-result? RESULT)

If RESULT is a termination result object from the invocation of a termination function, then this function will return #t. Otherwise, it will return #f.

test::terminate-result-finding
[procedure] (test::terminate-result-finding RESULT)

Returns the result associated with the termination function RESULT object.

test::terminate-result-scope
[procedure] (test::terminate-result-scope RESULT)

The "scope" of the termination result is exactly the TEST-NAME parameter supplied to the test case or test suite associated with the TERMFUNC.

test::terminate-result-container
[procedure] (test::terminate-result-container RESULT)

The "container" of the termination result is going to be either 'test-suite or 'test-case depending upon which the TERMFUNC was associated.

test::terminate-result-message
[procedure] (test::terminate-result-message RESULT)

Returns the message object associated with the termination RESULT object.

Todo Result Object
test::todo-result?
[procedure] (test::todo-result? RESULT)

If RESULT is a todo result object from the invocation of a todo macro, then this function will return #t. Otherwise, it will return #f.

test::todo-result-message
[procedure] (test::todo-result-message RESULT)

Returns the message object associated with the todo RESULT object.

test::todo-result-warning?
[procedure] (test::todo-result-warning? RESULT)

If a warning had been attached to this todo, this function will return #t, otherwise it will be #f.

test::todo-result-warning
[procedure] (test::todo-result-warning RESULT)

If a warning had been attached to this todo, this function will return the warning message object supplied by the user, otherwise it shall return '().

Miscellaneous

test::walk-structure
[procedure] (test::walk-structure WALKER TEST-STRUCTURE)

This procedure will enumerate the TEST-STRUCTURE association list, invoking the WALKER for each element. The hierarchy level value is the tree depth at the point of call.

WALKER is a three argument procedure. The first argument is the hierarchy level, counting from 0, the outermost level. The second argument is the type of the test element. The third argument is the name of the test element.

Statistics Generation

test::stat-result-statistics
[procedure] (test::stat-result-statistics RESULT-TREE)

This function will compute and return a vector of result STATISTICS. Use the procedures below to access the elements by name.

test::stat-test-suites
[procedure] (test::stat-test-suites STATISTICS)
test::stat-test-suite-warnings
[procedure] (test::stat-test-suite-warnings STATISTICS)
test::stat-test-suites-passed
[procedure] (test::stat-test-suites-passed STATISTICS)
test::stat-test-suites-failed
[procedure] (test::stat-test-suites-failed STATISTICS)
test::stat-test-suites-timing
[procedure] (test::stat-test-suites-timing STATISTICS)
test::stat-test-suites-terminated
[procedure] (test::stat-test-suites-terminated STATISTICS)
test::stat-test-cases
[procedure] (test::stat-test-cases STATISTICS)
test::stat-test-case-warnings
[procedure] (test::stat-test-case-warnings STATISTICS)
test::stat-test-cases-passed
[procedure] (test::stat-test-cases-passed STATISTICS)
test::stat-test-cases-failed
[procedure] (test::stat-test-cases-failed STATISTICS)
test::stat-test-cases-timing
[procedure] (test::stat-test-cases-timing STATISTICS)
test::stat-test-cases-terminated
[procedure] (test::stat-test-cases-terminated STATISTICS)
test::stat-all-expectations
[procedure] (test::stat-all-expectations STATISTICS)
test::stat-all-expectation-warnings
[procedure] (test::stat-all-expectation-warnings STATISTICS)
test::stat-all-expectations-passed
[procedure] (test::stat-all-expectations-passed STATISTICS)
test::stat-all-expectations-failed
[procedure] (test::stat-all-expectations-failed STATISTICS)
test::stat-all-expectations-timing
[procedure] (test::stat-all-expectations-timing STATISTICS)
test::stat-single-expectations
[procedure] (test::stat-single-expectations STATISTICS)
test::stat-single-expectation-warnings
[procedure] (test::stat-single-expectation-warnings STATISTICS)
test::stat-single-expectations-passed
[procedure] (test::stat-single-expectations-passed STATISTICS)
test::stat-single-expectations-failed
[procedure] (test::stat-single-expectations-failed STATISTICS)
test::stat-single-expectations-timing
[procedure] (test::stat-single-expectations-timing STATISTICS)
test::stat-tol-expectations
[procedure] (test::stat-tol-expectations STATISTICS)
test::stat-tol-expectation-warnings
[procedure] (test::stat-tol-expectation-warnings STATISTICS)
test::stat-tol-expectations-passed
[procedure] (test::stat-tol-expectations-passed STATISTICS)
test::stat-tol-expectations-failed
[procedure] (test::stat-tol-expectations-failed STATISTICS)
test::stat-tol-expectations-timing
[procedure] (test::stat-tol-expectations-timing STATISTICS)
test::stat-equiv-expectations
[procedure] (test::stat-equiv-expectations STATISTICS)
test::stat-equiv-expectation-warnings
[procedure] (test::stat-equiv-expectation-warnings STATISTICS)
test::stat-equiv-expectations-passed
[procedure] (test::stat-equiv-expectations-passed STATISTICS)
test::stat-equiv-expectations-failed
[procedure] (test::stat-equiv-expectations-failed STATISTICS)
test::stat-equiv-expectations-timing
[procedure] (test::stat-equiv-expectations-timing STATISTICS)
test::stat-todos
[procedure] (test::stat-todos STATISTICS)
test::stat-todo-warnings
[procedure] (test::stat-todo-warnings STATISTICS)
test::stat-skips
[procedure] (test::stat-skips STATISTICS)
test::stat-skip-warnings
[procedure] (test::stat-skip-warnings STATISTICS)
test::stat-glosses
[procedure] (test::stat-glosses STATISTICS)
test::stat-gloss-warnings
[procedure] (test::stat-gloss-warnings STATISTICS)
test::stat-terminations
[procedure] (test::stat-terminations STATISTICS)

Printing Routines

These procedures are used in the generation of test result reports.

test::write-object
[procedure] (test::write-object OBJECT)
test::display-objects
[procedure] (test::display-objects [OBJECT ...])
test::display-objects-newline
[procedure] (test::display-objects-newline [OBJECT ...])
test::display-indent
[procedure] (test::display-indent INDENT)
test::display-indented-objects
[procedure] (test::display-indented-objects INDENT [OBJECT ...])
test::display-indented-objects-newline
[procedure] (test::display-indented-objects-newline INDENT [OBJECT ...])

INDENT is the number of spaces to the left.

test::display-underlined
[procedure] (test::display-underlined [OBJECT ...])

Prints each supplied OBJECT to the (current-output-port), with a #\space interspersed, on a single line. Then prints a line of underscores, #\_.

test::display-structure
[procedure] (test::display-structure TEST-STRUCTURE)

Prints the supplied TEST-STRUCTURE tree to the (current-output-port) with indentation.

Test Report Generation

These functions will display a rendering of the RESULT-TREE and returns the RESULT-TREE.

test::output-style-compact

<procedure>(test::output-style-compact RESULT-TREE [PORT|FILE-NAME|#t])

Prints a simple listing of the test results. Details are only printed for failed tests.

test::output-style-human
[procedure] (test::output-style-human RESULT-TREE [PORT|FILE-NAME|#t])

Prints a verbose listing of the test results.

test::output-style-html
[procedure] (test::output-style-html RESULT-TREE [PORT|FILE-NAME|#t])

Prints a verbose listing of the test results as HTML.

test::output-style-minimal
[procedure] (test::output-style-minimal RESULT-TREE [PORT|FILE-NAME|#t])

Prints a summary of the test result statistics.

The output style procedures are not loaded automatically. They must be loaded explicitly, as in:

(require-extension testbase-output-human)

(require-extension testbase-output-html)

(require-extension testbase-output-compact)

(require-extension testbase-output-minimal)

This procedure is useful with a test report procedure. The actual report procedures of the above report generators are examples.

test::with-output-style
[procedure] (test::with-output-style STYLER RESULT-TREE [PORT|FILE-NAME|#t])

STYLER is a procedure with a single argument, a RESULT-TREE, to print a report.

The default destination is the (current-output-port). Otherwise the destination specifies how the output is targeted.

A PORT destination will print to the port.

A FILE-NAME destination will print to the specified file. Should the file exist the result is implementation dependent.

A #t destination will return the output as a string.

The STYLER will be called with RESULT-TREE as the only actual argument.

Notes & Caveats

To generate the tree structure of a test procedure the test body is evaluated using a special test mode, report. The expectation expressions, test container let bindings, and test-letrec bindings <em>are not</em> evaluated. This means there should not be any expressions in the test body that will be evaluated, such as the construction of a TEST-NAME, that depend on these bindings.

However, test procedure initial and final forms are evaluated.

Be careful when using test selection that dependencies between tests are not broken. For example, an expectation expression could depend on the result of an expect-set!, but the associated variable may never be bound because the expect-set! was skipped.

Examples

Assume the following are the contents of the file "testbase-example.scm":

(require-extension testbase testbase-output-human)

(define-expect-binary =)

(define-test example "Arithmetic Operators"
  (test/case "Testing '+'"
    (expect-= "Adding two positive numbers" 2 (+ 1 1))
    (expect-= "Adding two negative numbers" -2 (+ -1 -1))
    (expect-zero "Adding positive and negative" (+ -1 1)) )
  (test/case "Testing '-'"
    (expect-zero "Subtracting two positive numbers" (- 1 1))
    (expect-zero "Subtracting two negative numbers" (- -1 -1))
    (expect-= "Subtracting positive and negative" -2 (- -1 1)) ) )

(test::styler-set! example test::output-style-human)
(run-test "TestBase Example")

Then the following entered from a command line interface:

 csi -n -s testbase-example.scm

Produces the following output:

 ** Running TestBase Example **
 
 Test Suite: Arithmetic Operators
 
   Test Case: Testing '+'
 
     Expectation: Adding two positive numbers
     Expect =
        Expected: 2
     Unevaluated: (+ 1 1)
       Evaluated: 2
     Pass: Adding two positive numbers
 
     Expectation: Adding two negative numbers
     Expect =
        Expected: -2
     Unevaluated: (+ -1 -1)
       Evaluated: -2
     Pass: Adding two negative numbers
 
     Expectation: Adding positive and negative
     Expect zero
     Unevaluated: (+ -1 1)
       Evaluated: 0
     Pass: Adding positive and negative
 
   Pass: Testing '+'
 
   Test Case: Testing '-'
 
     Expectation: Subtracting two positive numbers
     Expect zero
     Unevaluated: (- 1 1)
       Evaluated: 0
     Pass: Subtracting two positive numbers
 
     Expectation: Subtracting two negative numbers
     Expect zero
     Unevaluated: (- -1 -1)
       Evaluated: 0
     Pass: Subtracting two negative numbers
 
     Expectation: Subtracting positive and negative
     Expect =
        Expected: -2
     Unevaluated: (- -1 1)
       Evaluated: -2
     Pass: Subtracting positive and negative
 
   Pass: Testing '-'
 
 Pass: Arithmetic Operators
 
 ALL TESTS SUCCESSFUL!

And the following entered from a command line interface:

 echo "(test::resource-limit-set! example 'cpu 0) (example)" | csi -n -s testbase-example.scm -- -i

Produces the following output:

 (begin (id tr0) (name "Arithmetic Operators") (result #f) (kind "suite"))
 (begin (id tr1) (name "Testing '+'") (result #f) (kind "case"))
 (expectation (id tr2) (name "Adding two positive numbers") (result #t) (kind "=") (unevaluated (+ 1 1)) (expected 2) (actual 2))
 (expectation (id tr3) (name "Adding two negative numbers") (result #t) (kind "=") (unevaluated (+ -1 -1)) (expected -2) (actual -2))
 (expectation (id tr4) (name "Adding positive and negative") (result #t) (kind "zero") (unevaluated (+ -1 1)) (actual 0))
 (end (id tr1) (name "Testing '+'") (result #t) (kind "case"))
 (begin (id tr5) (name "Testing '-'") (result #f) (kind "case"))
 (expectation (id tr6) (name "Subtracting two positive numbers") (result #t) (kind "zero") (unevaluated (- 1 1)) (actual 0))
 (error (id tr7) (message "Resource Limit Exceeded: Processor-Time") (result #f) (kind "terminate") (container test-suite) (location "Arithmetic Operators"))
 (end (id tr0) (name "Arithmetic Operators") (result #f) (kind "suite"))

And the following entered from a command line interface:

 echo "(test::run)" | csi -n -s testbase-example.scm -- -i

Produces the following output:

 (begin (id tr0) (name "Arithmetic Operators") (result #f) (kind "suite"))
 (begin (id tr1) (name "Testing '+'") (result #f) (kind "case"))
 (expectation (id tr2) (name "Adding two positive numbers") (result #t) (kind "=") (unevaluated (+ 1 1)) (expected 2) (actual 2))
 (expectation (id tr3) (name "Adding two negative numbers") (result #t) (kind "=") (unevaluated (+ -1 -1)) (expected -2) (actual -2))
 (expectation (id tr4) (name "Adding positive and negative") (result #t) (kind "zero") (unevaluated (+ -1 1)) (actual 0))
 (end (id tr1) (name "Testing '+'") (result #t) (kind "case"))
 (begin (id tr5) (name "Testing '-'") (result #f) (kind "case"))
 (expectation (id tr6) (name "Subtracting two positive numbers") (result #t) (kind "zero") (unevaluated (- 1 1)) (actual 0))
 (expectation (id tr7) (name "Subtracting two negative numbers") (result #t) (kind "zero") (unevaluated (- -1 -1)) (actual 0))
 (expectation (id tr8) (name "Subtracting positive and negative") (result #t) (kind "=") (unevaluated (- -1 1)) (expected -2) (actual -2))
 (end (id tr5) (name "Testing '-'") (result #t) (kind "case"))
 (end (id tr0) (name "Arithmetic Operators") (result #t) (kind "suite"))

A test-collect Example:

(require-extension testbase testbase-output-human)

(define-test tcx "A test-collect example"
  (test/collect "Are 0 - 9 integers?"
    (do ([i 0 (add1 i)])
        [(= i 10)]
      (collect-test (expect-true (integer? i))) ) ) )

(test::styler-set! tcx test::output-style-human)
(run-test "TestBase Example")

Produces the following output:

 ** Running TestBase Example **
 Test Suite: A test-collect example
 
   Test Case: Are 0 - 9 integers?
 
     Expectation:
     Expect true
     Unevaluated: (integer? i)
       Evaluated: #t
     Passed:
 
     Expectation:
     Expect true
     Unevaluated: (integer? i)
       Evaluated: #t
     Passed:
 
     Expectation:
     Expect true
     Unevaluated: (integer? i)
       Evaluated: #t
     Passed:
 
     Expectation:
     Expect true
     Unevaluated: (integer? i)
       Evaluated: #t
     Passed:
 
     Expectation:
     Expect true
     Unevaluated: (integer? i)
       Evaluated: #t
     Passed:
 
     Expectation:
     Expect true
     Unevaluated: (integer? i)
       Evaluated: #t
     Passed:
 
     Expectation:
     Expect true
     Unevaluated: (integer? i)
       Evaluated: #t
     Passed:
 
     Expectation:
     Expect true
     Unevaluated: (integer? i)
       Evaluated: #t
     Passed:
 
     Expectation:
     Expect true
     Unevaluated: (integer? i)
       Evaluated: #t
     Passed:
 
     Expectation:
     Expect true
     Unevaluated: (integer? i)
       Evaluated: #t
     Passed:
 
   Passed: Are 0 - 9 integers?
 
 Passed: A test-collect example
 
 ALL TESTS SUCCESSFUL!
 
 ** Passed TestBase Example **

Tips, Tricks, And Idioms

This documentation is pretty large, so put here any things that weren't obvious to you that seem like they might be useful to others.

General Tips
Automated Egg Tests

chicken-setup, at least when run with the -t flag, will treat the "tests" directory as a directory full of tests, and run the "run.scm" file in that directory. Here's a run.scm file, modified from one provided by Kon, that seems to work well. You'll need testbase-driver for it to work. If any tests fail, the setup will abort (although by that time it's already installed the egg, but at least it'll be quite cranky about the failure). This run.scm will run everything in the tests directory named "*/test.scm". It will put the results in "results/" under the tests directory.

;;;; run-testbase.scm (run.scm)
;
; Thanks to Kon Lovett for the code.

(use utils posix)

;;

(define *egg-name* (car (command-line-arguments)))

(define *verbose* #t)

(define-constant TEST-DRIVER "chicken-testbase-driver")
(define-constant TEST-DRIVER-OPTIONS "--indent 2 --test-interpret --failure-exit=1")

(define *test-driver-arguments*
  (conc
    TEST-DRIVER-OPTIONS
    " --results-repository="
    (current-directory)
    "/results "
    (if
      *verbose* " -v" "")))

;;

(system* "~A ~A ~A" TEST-DRIVER *test-driver-arguments* (string-append (current-directory) "/*-test.scm"))

Migration from test-infrastructure

Changes:

Forms like:

 (test::apply test::output-style-compact
     (skip ...)
     test-proc)

become

 (test::take! test-proc '(skip ...))
 (test::styler-set! test-proc test::output-style-compact)
 (test-proc)

Forms like

 (define (test-something)
   (test-suite "Something" pd pe
     ... test elements ...
   )
 )

become

 (define-test test-something
   (test-suite "Something" pd pe
     ... test elements ...
   )
 )

or even

 (define-test test-something "Something"
   ... test elements ...
   <where any use of 'pd' and 'pe' becomes 'test-dtor' and 'test-escr'>
 )

Forms like

 (let ((result
         (test-suite "Something" pd pe
           ... test elements ...)))
   ...)

become

 (define-test something
   (test-suite "Something" pd pe
     ... test elements ...
   )
 )
 
 (let ((result (something)))
   ...)

Bugs & Limitations

Because a sharp-syntax reader is created to handle "unprintable" results the "#<<TAG" and "#<#TAG" multiline string constant read syntax cannot be used within a test-file. Use (MULTILINE-STRING STRING ...).

Contributions

output-text-compact by Patrick Brannan.

Changelog

License

Portions Copyright (c) 2007, Kon Lovett Portions Copyright (c) 2000-2005, Peter Keller - All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

 Redistributions of source code must retain the above copyright notice, this list of conditions and the following
   disclaimer.
 Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
   disclaimer in the documentation and/or other materials provided with the distribution.
 Neither the name of the author nor the names of its contributors may be used to endorse or promote
   products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.