test.js

Combinational test framework.

This is not meant as a replacement for more advanced and feature-rich testing frameworks, instead this is a minimalist and complete experimental implementation of a specific approach to testing, i.e. combinational testing

Note, this module is experimental and can change quite allot within a short to mid time frame, use at your own risk, though ideas, feedback and suggestions are welcome.

Features

• Simple / minimalist implementation
• Supports combinational as well as unit testing paradigms
• Multiple modifier chaining

Planned

• Replaceable/extensible assertion library

Contents

• test.js
  • Features
    • Planned
  • Contents
  • Architecture
    • Combinational testing
    • Unit testing
  • Installation
  • Basic usage
  • CLI
  • Components
    • DEFAULT_TEST_FILES
    • IGNORE_TEST_FILES
    • Merged
      • <merged>.create(..)
      • <merged>.members
      • <merged>.size / <merged>.usize
      • <merged>.add(..) / <merged>.remove(..)
      • <merged>.clear()
      • <merged>.keys(..) / <merged>.values(..) / <merged>.entries(..)
      • <merged>.toObject(..)
      • <merged>.checkShadowing(..)
      • <merged>.handleShadowing(..)
      • <member>.filename
    • TestSet
    • BASE_TEST_SET
    • Setups(..) / Setup(..) (Merged)
    • Modifiers(..) / Modifier(..) (Merged)
    • Tests(..) / Test(..) (Merged)
    • Cases(..) / Case(..) (Merged)
    • Assert(..)
    • run(..)
  • Advanced components
    • runner(..)
    • parser(..)
  • Utilities
    • getCallerFilename()
  • License

Architecture

This package implements two testing schemes:

• Combinational testing
  Here the user sets up a set of Setup, Modifier and Test functions and the system chains different combinations of the three and runs them.
• Unit testing
  Simple independent tests.

Combinational testing

In general the idea here is that you define three things:

• Setups that build a test context and objects (the setup),
• Modifiers that modify the setup in some way,
• Tests that test some aspect of a setup.

The system builds chains in the form:

setup -> modifier* -> test

Where modifier is optional and can be a chain of zero or more modifiers.

A setup and modifier can also include assertions/tests for direct testing and sanity checks.

The system defines a blank pass-through modifier and test, to alleviate the requirement on the user to define at least one modifier and test, so in cases where it makes no sense only a setup is required.

Note that each element can reference and re-use other elements so for example a modifier can call (re-use) other modifiers to avoid code duplication.

This makes it simple to define procedural/generative tests.

Unit testing

This is the traditional self-contained test approach.

Installation

$ npm install -i ig-test

And to install the global CLI interface

$ npm install -g ig-test

Basic usage

Create a test script

$ touch test.js
$ chmod +x test.js

Note that the test script should be named either "test.js" or "<something>-test.js" for the system to find it automatically.

The code:

#!/usr/bin/env node

var test = require('ig-test')

// XXX add assert examples...

test.Setups({
    state: function(assert){
        return {
            a: 123,
            b: 321,
        } },
})

test.Modifiers({
    inc: function(assert, state){
        Object.keys(state)
            .forEach(function(k){
                state[k] += 1 })
        return state },    
})

test.Tests({
    
})

test.Cases({
    
})

// make the test runnable as a standalone script...
__filename == (require.main || {}).filename
    && tests.run()

Run the tests

$ node ./test.js

or

$ runtests

CLI

$ npm install -g ig-test

Basic help

$ runtests --help 
Usage: test.js [OPTIONS] [CHAIN] ...

Run tests.

Tests run by test.js can be specified in one of the 
following formats:

        <case>
        <setup>:<test>
        <setup>:<modifier>:<test>

Each of the items in the test spec can be a "*" indicating
that all relevant items should be used, for example:

        $ ./test.js basic:*:*

Here test.js is instructed to run all tests and modifiers
only on the basic setup.

Zero or more sets of tests can be specified.

If no tests are specified or '**' is given test.js will run
all found tests:
        
        $ ./test.js
or:
        $ ./test.js '**'

Note that using '*' or '**' may require escaping as they may 
get expanded by the shell.

Options:
	-h,  --help		- print this message and exit
	     --version		- show test.js version and exit
	-l,  --list=PATH	- list available tests;
				  note that if passing files via -f explicitly they
				  must precede the -l/-list flag;
				  this has the same defaults as -f
	     --list-found=PATH	- like -list but print found test modules and exit
	-m,  --max-modifier-chain=NUMBER
				- Maximum number of modifiers to use in chain
				  (default: 1)
	-f,  --test-file=PATH	- test script or filename pattern, supports glob;
				  this flag can be given multiple times for
				  multiple paths/patterns
				  (default: "**/?(*-)test.js")
	-i,  --ignore=PATH	- path/pattern to ignore in test file search
				  (default: ["node_modules/**"])
	-v,  --verbose		- verbose mode
				  (env: $VERBOSE)

Examples:
	$ ./test.js		- run all tests.
	$ ./test.js basic:*:*	- run all tests and modifiers on "basic" setup.
				  (see test.js -l for more info)
	$ ./test.js -v example	- run "example" test in verbose mode.
	$ ./test.js native:gen3:methods init:gen3:methods
				- run two tests/patterns.

Written by: Alex A. Naanou <alex.nanou@gmail.com>
Version: 1.6.7 / License: BSD-3-Clause

List available test components

$ runtests --list 

XXX chains

XXX notes on coverage

Components

DEFAULT_TEST_FILES

glob pattern(s) used to find test files by default.

DEFAULT_TEST_FILES =
    undefined
    | <path>
    | [ <path>, .. ]

Default value: "**/?(*-)test.js"

IGNORE_TEST_FILES

A list of glob patterns to ignore while searching for tests.

IGNORE_TEST_FILES =
    undefined
    | [ <path>, .. ]

Default value: ['node_modules/**']

Merged

Implements a merged collection of instances (members).

Create a new collection:

Merged.create(<name>)
    -> <merged>

Add members to collection:

<merged>({ <key>: <func>, .. })
    -> <member>

<merged>(<key>, <func>)
    -> <member>

On construction this will assign the input object / <key>-<func> into the resulting <member>/instance object.

Each <member>/instance created is added to the constructor as a member (i.e. added into .members)

Provides a set of methods and properties to access/introspect the merged (hence the name) attributes of the members (i.e. .keys(..), .values(..), .entries(..), .size/.usize and .members).

Note that though Merged itself is a collection, it is not designed to be used directly as a collection, use it to create new collections or as a prototype for inheritance.

<merged>.create(..)

Create a new Merged collection (member constructor).

Merged.create(<name>)
    -> <merged>

<merged>.members

List of members / instances of Merged in order of creation.

<merged>.size / <merged>.usize

Number of members including the "-" members and not including respectively.

<merged>.add(..) / <merged>.remove(..)

Add / remove a member.

<merged>.add(<member>)
    -> <merged>

<merged>.remove(<member>)
    -> <merged>

<merged>.clear()

Remove (clear) all the members.

<merged>.keys(..) / <merged>.values(..) / <merged>.entries(..)

<merged>.keys()
<merged>.keys(<merged>)
    -> <list>

<merged>.values()
<merged>.values(<merged>)
    -> <list>

<merged>.entries()
<merged>.entries(<merged>)
    -> <list>

These are similar to Object.keys(..) / Object.values(..) / Object.entries(..) but will also if called without arguments return a list of the callers member keys/values/entries respectively.

Note that members' attributes can shadow previous member attributes, only one value per key will be returned. <merged> will warn when adding a member of its attributes will shadow already existing members' attributes (see: <merged>.checkShadowing(..) and <merged>.handleShadowing(..));
Also note that the check for shadowing is performed when the <member> is created and not when new attributes are added manually.

<merged>.toObject(..)

Create an object containing all visible member attributes.

<merged>.toObject()
    -> <object>

<merged>.checkShadowing(..)

Find all shadowed attributes within <merged>.

<merged>.checkShadowing()

Find all attributes in <merged> that will be shadowed by <member>

<merged>.checkShadowing(<member>)
    -> <list>

<merged>.handleShadowing(..)

Will be called on <member> construction when attribute shadowing is detected.

`<merged>.handleShadowing(<attr>)`
    -> <merged>

By default this will print a warning and continue, but can be overloaded by the user to react to shadowing in a different manner.

<member>.filename

The filename where the <member> was defined.

TestSet

XXX

BASE_TEST_SET

XXX

Setups(..) / Setup(..) (Merged)

XXX

A subclass or rather sub-constructor of Merged.

Note that Setups and Setup are references to the same object, they exists for better readability in cases when we add a single element (<key>-<func> pair) or a bunch of elements (object), for example:

// single element...
test.Setup('some-setup', 
    function(){
        // ...
    })

// arbitrary number of elements...
test.Setups({
    'some-other-setup': function(){
        // ...
    },

    // ...
})

Modifiers(..) / Modifier(..) (Merged)

XXX

A sub-constructor of Merged.

Tests(..) / Test(..) (Merged)

XXX

A sub-constructor of Merged.

Cases(..) / Case(..) (Merged)

XXX

A sub-constructor of Merged.

Assert(..)

XXX this may still change...

run(..)

Run the test system.

run()
run(<tests>)
run(<default-files>)
run(<default_files>, <tests>)
    -> <parse-result>

This will:

• parse process.argv
• locate and run tests
• report basic stats

<tests> format:

{
    setups: <stups>,
    modifiers: <modifiers>,
    tests: <tests>,
    cases: <cases>,
}

Advanced components

runner(..)

The default test combinator and runner.

parser(..)

The default ig-argv parser setup.

Utilities

getCallerFilename()

Returns the filename of the module where getCallerFilename() is called.

• Multiple modifier chaining

License

BSD 3-Clause License

Copyright (c) 2020-2023, Alex A. Naanou,
All rights reserved.