NAME

testlib_sh - bash/ksh test library


SYNOPSIS

  . testlib.sh
  # initialize test suite
  test_initialize
  # set test suite configuration variable
  test_set variable value
  # something to run after every test
  test_set_hook result function_name
  # status comparison
  test_ok $status $test_name
  test_nok $badstatus $test_name
  test_notok $badstatus $test_name
  # numerical (integer) comparison
  test_num_is $value $exp $test_name
  test_num_isnt $value $exp $test_name
  # numerical (real) comparison using ndiff
  test_ndiff_is   [options] $value $exp $test_name
  test_ndiff_isnt [options] $value $exp $test_name
  # string comparison
  test_is $value $exp $test_name
  test_isnt $value $exp $test_name
  # file comparison
  test_file_is   [options] $file $exp_file $test_name
  test_file_isnt [options] $file $exp_file $test_name
  # numeric file comparison using ndiff
  test_numfile_is   [options] $file $exp_file $test_name
  test_numfile_isnt [options] $file $exp_file $test_name
  # file greps
  test_file_grep_is   [options] "$pattern" $file $test_name
  test_file_grep_isnt [options] "$pattern" $file $test_name
  # just doit
  test_fail $test_name
  test_pass $test_name
  # skipping tests
  if (( skip )); then
    test_skip "because" 2
  else
    test_ok $status "foo"
    test_ok $status "bar"
  fi
  # tests you'll one day get to work
  test_todo_begin
  test_ok  $status $test_name
  test_ok  $status $test_name
  test_end_todo
  test_begin_subtests
  test_end_subtests
  test_summary
  test_exit


DESCRIPTION

testlib_sh is a library for writing standardized tests in bash or ksh. It provides a set of functions which will generate uniform reporting on tests, accumulate statistics, and generate an exit value based upon the results of the tests. testlib_sh is modelled after the Perl Test::More module.

All testlib_sh functions begin with test_. testlib_sh uses a number of internal functions and variables to keep track of things. These all begin with _test. Please do not alter them! Some variables are made public for use by the calling application. See Testlib Variables for more information.


USER'S GUIDE

Test Suite Setup

To begin a test suite, first source the testlib.sh library, and then initialize the suite:

     . testlib.sh
     test_initialize

This will initialize the internal state variables and make available the variable test_id which may be used by the tests as a basis for filenames or the like.

Test Suite configuration

After calling test_initialize, the behavior of the library may be changed via several configuration variables. Use the test_set command to set their values. CONFIGURATION VARIABLES lists the available variables.

Hooking into the Test Suite Mechanisms

testlib_sh provides the ability for code to be called at specific points in its processing flow. A hook is a point where the user code may be called. Tests have access to hooks via test_set_hook. See HOOKS for more information.

Recording test results

There are a variety of functions which can be used to record the result of a test. These include

test_ok, test_nok, test_notok

tests of status values

test_is, test_isnt

comparison of string values

test_num_is, test_num_isnt

comparison of numerical values (according to the shell's idea about what is a numerical quantity)

test_ndiff_is, test_ndiff_isnt

comparison of numerical values using ndiff

test_file_is, test_file_isnt

file comparison tests using cmp

test_numfile_is, test_numfile_isnt

file comparison tests using ndiff

test_pass
test_fail

pass/fail for tests which are difficult to categorize

Typically, the _is and _isnt tests should be used, as these will give more useful diagnostics upon error. _ok and _notok tests are next best. pass and fail should be used when testing is so complicated that none of the other functions suffice.

Note that test_ok regards a status value of 0 as success. This allows it to easily be used with the exit status variable in the shell:

  cmp file1 file2
  test_ok $? "comparison"

Skipping Tests

Tests may be also be skipped. This is useful if some resource is not available. This requires a bit of code to ensure that that the tests are not run, and that the total number of tests remains constant.

  # set variable skip to true if skip condition is met
  if (( skip )); then
    test_skip $reason_for_skipping $number_of_tests_to_skip
  else
    # tests to run if not skipping.  
    # number must be $number_of_tests_to_skip
    test_ok ...
  fi

Todo Tests

Consider marking tests as ``todo'' if they test parts of the system which are not yet ready for prime time but should be tested. This provides incentive for finishing them. To mark them as such, surround them with test_begin_todo and test_end_todo. Such tests may also be skipped if running the todo tests is impossible.

  test_begin_todo
  test_ok $status $test_name
  if (( skip )); then
     test_skip $reason $number
  else
     test_ok $status $test_name
  fi
  test_end_todo

Tests which are marked ``todo'' and which fail are not counted against the exit status of the test script. Tests which succeed should obviously be moved out of todo sections!

Subtests

If the test suite is using the test id (see Testlib Variables) to name output files and it may be necessary to perform multiple tests on a single output file, the test id for the tests will be out of sync with that used to generate the output files. For example, consider this suite:

     logfile="$test_id.log"
     program > $logfile
     grep -q foo $logfile
     test_ok $? "foo in logfile"
     grep -q boo $logfile
     test_ok $? "boo in logfile"

If boo is not in the logfile, then the second test will report the problem in test 2, and if you are using the file cleanup conventions in the EXAMPLES section, the wrong file will be saved, as it is named using test 1's id. The way around this is to declare that you will be performing subtests:

     test_begin_subtests
     logfile="$test_id.log"
     program > $logfile
     grep -q foo $logfile
     test_ok $? "foo in logfile"
     grep -q boo $logfile
     test_ok $? "boo in logfile"
     
     test_end_subtests
     
The test id remains constant throughout the subtest block, so things
remain in sync.  You can get the subtest number via the C<test_subnum>
variable.  This is defined to be C<0> if not within a subtest block.
See L</Testlib Variables> for more information.

Note: If a result callback function has been specified with test_set_hook, that function will be called for each subtest, as well as an additional call after all of the subtests have been completed. The argument to the final call will be true only if all of the subtests have completed. This can be used to delay cleanup of test output until all of the subtests have been complete. Simply check if test_subnum is zero to determine if all of the subtests are complete.

Test summary and exit

The last statements in your tests should typically be

  test_summary
  test_exit

The first will generate the final summary of the test results, the last will exit with a status value indicating whether any tests failed.


FUNCTIONS

test_exit
  test_exit

This is should be the last statement in the test suite, as it will exit with a status reflecting the results of the tests.

test_pass
test_fail
  test_pass $test_name
  test_fail $test_name

If all other test_* routines are useless to you, you can use these to indicate the status of a test.

test_pass indicates that the named test has passed.

test_fail indicates that the named test has failed.

Consider using the _is, _isnt, _ok and _notok tests if possible.

test_initialize
  test_initialize

This should be the first statment in the test suite. It initializes the test environment.

test_ok
  test_ok $status $test_name

This reports success if a $status is 0.

test_nok
test_notok
  test_nok $status $test_name
  test_notok $status $test_name

These reports success if a $status is 1.

test_is
test_isnt
  test_is $value $expected $test_name
  test_isnt $value $expected $test_name

These are string comparison tests.

test_is reports success if the two strings are identical.

test_isnt reports success if the two strings differ.

test_num_is
test_num_isnt
  test_num_is $value $expected $test_name
  test_num_isnt $value $expected $test_name

These are numerical comparison tests, according to what the shell believes a number to be.

test_num_is reports success if the two numbers are identical.

test_num_isnt reports success if the two numbers differ.

test_ndiff_is
test_ndiff_isnt
  test_ndiff_is   [options] $value $expected $test_name
  test_ndiff_isnt [options] $value $expected $test_name

These are numerical comparison tests, using the ndiff program.

test_ndiff_is reports success if the two numbers are identical.

test_ndiff_isnt reports success if the two numbers differ.

The test will fail if ndiff is not available.

Both tests take the following options:

--

This indicates there are no further options. This must be used if $value might begin with the - character.

-a ndiff options

Options to be passed to ndiff. It is important that they be passed as a single string value (quote all spaces).

test_file_is
test_file_isnt
  test_file_is   [options] $file $exp_file $test_name
  test_file_isnt [options] $file $exp_file $test_name

These are file comparison tests, using the cmp command.

test_file_is reports success if the two files are identical.

test_file_isnt reports success if the two files differ.

Both tests take the following options:

--

This indicates there are no further options. This must be used if $file might begin with the - character.

-f failure file name

This file name will be output instead of $file on error. This is useful if $file will be renamed upon failure and the error message should track that.

test_numfile
test_numfile_is
test_numfile_isnt
  test_numfile_is   [options] $file $exp_file $test_name
  test_numfile_isnt [options] $file $exp_file $test_name

These are file comparison tests, using the ndiff command, which does a numerical difference of the files

test_numfile_is reports success if the two files are identical. test_numfile is identical to test_file_is.

test_numfile_isnt reports success if the two files differ.

Both tests take the following options:

--

This indicates there are no further options. This must be used if $file might begin with the - character.

-a ndiff options

Options to be passed to ndiff. It is important that they be passed as a single string value (quote all spaces).

-f failure file name

This file name will be output instead of $file on error. This is useful if $file will be renamed upon failure and the error message should track that.

test_file_grep_is
test_file_grep_isnt
  test_file_grep_is   [options] $pattern $file $test_name
  test_file_grep_isnt [options] $pattern $file $test_name

These tests grep the specified file for the specified pattern. The pattern should probably be escaped from the shell.

test_file_grep_is reports success if the pattern was found; test_file_grep_isnt reports success if the pattern was not found.

Both tests take the following options:

--

This indicates there are no further options. This must be used if $file might begin with the - character.

-a grep options

Options to be passed to grep. It is important that they be passed as a single string value (quote all spaces).

-f outputfile

A file to which grep's standard output should be written. It is normally discarded.

-E

Use egrep.

-F

Use fgrep.

test_set variable value

Set the named configuration variable to the value. If the variable does not exist, a message will be printed and the function will return 1.

See CONFIGURATION VARIABLES for more information.

test_set_hook hook function_to_call

This directs testlib_sh to call the given function function_to_call at the processing point identified by hook. If hook is not a recognized name, an error message is printed and the function will return 1.

See HOOKS for more information.

test_set_status status

This is a convenience function to override the shell's idea of the status of the most recently executed command. This is useful when status checking must be postponed for some reason, and $? is to be consulted for the status. For example,

  test_set check_exit_status 1
  output=$(run_test_exe)
  status=$?
  ...do stuff to generate $value from $output..
  # depend upon check_exit_status to check $? for us
  test_set_status $status
  test_is $value $expected $test_name
test_skip
  test_skip $reason $number

This indicates that $number tests should be recorded as being skipped. It is up to the test suite to ensure that those tests are not run, and that the number of tests not run is consistent with $number. See Skipping Tests.

test_summary
  test_summary

This generates a summary of the tests which were run or skipped.

test_begin_todo
test_end_todo
  test_begin_todo
    [... todo tests ...]
  test_end_todo

Begin and end a block of tests marked as ``todo''. Failure of these tests does not count towards the exit status of the suite. test_end_todo must be called to end the ``todo'' block. The variable test_todo will be set to 1 while a ``todo'' block is active.

test_begin_subtests
test_end_subtests
  test_begin_subtests
    [... subtests ...]
  test_end_subtests

Begin and end a block of subtests. See Subtests. The block must be closed with a test_end_subtests command.


CONFIGURATION VARIABLES

The behavior of testlb_sh can be changed by setting various configuration variables with the test_set command. The available variables are:

check_exit_status 0 | 1 | -1

If non-zero, tests (excluding test_pass and test_fail) will first check the exit status of the last command run before the actual test specified and will generate an error if the status is not what is expected; the ``real'' test will not be performed. The expected exit status depends upon the value of check_exit_status:

  1. The exit status must be zero (success). For example, the following will fail:

      test_set check_exit_status 1
      false
      test_ok 0 "normally should pass"
  2. -1

    The exit status must be non-zero (failure). For example, the following will fail:

      test_set check_exit_status -1
      true
      test_ok 0 "normally should pass"

check_exit_status can be used to simplify tests which need to check both the exit value of a command and its output (or some other side effect). For example, rather than write

  result=$(my_command)
  test_ok $? "my_command execute"
  test_is result "valid" "my_command results"

one can just write

  result=$(my_command)
  test_is result "valid" "my_command results"

In many cases, this may remove the requirement to use subtests.

NOTE!! Be sure not to inadvertently run other commands before the testlib routine is called. For example:

  some_program
  test_is "$result" $(date) "NO!"

will check the exit status of date, not some_program!

make_cmp_copy boolean

If true, file comparison tests will copy $file to $exp_file if the latter does not exist. This allows one to easily create the set of expected output on the first run through the test. Of course, one should verify that the output is correct! This should not be set for production use! A message is printed if the copying occurs.


HOOKS

Hooks are points in the test suite software where user code may be executed. The test_set_hook function is used to assign a function to a hook.

Currently the following hooks are available:

result

The result hook is called after each test result is determined. The function is passed a single boolean argument indicating whether the test has passed.

If subtests are enabled, the result hook will be called after each subtest, with an additional call after the subtests have been completed. The argument to this last call will be true only if all of the subtests have completed. To distinguish it from the preceeding calls, test the test_subnum variable; it will have a value of 0 for the final call.

This can be used to delay cleanup of test output until all of the subtests have been complete. Simply check if test_subnum is zero to determine if all of the subtests are complete.

See the Managing test output section for more information. The function described there is part of testlib.sh and may be used via

  test_set_hook result test_result_hook_save_output_on_failure

Note that previously test_result_hook was used to set this hook. This function is now deprecated.


VARIABLES

The following variables are available for use by the test suite:

test_id

This contains a zero prefixed and padded representation of the current test number.

test_num

This contains the raw test number (no padding)

test_subid

This contains a zero prefixed and padded representation of the current sub test number. It is 00 if not in a subtest block.

test_subnum

This contains the raw subtest number (no padding). It is 0 if not in a subtest block.

test_label

This contains the label which will be printed by testlib_sh when identifying the current test.

test_status

This indicates whether the last test succeeded; 0 means it did, non-zero means it didn't.

test_todo

This is a boolean value indicating whether the test suite is currently in a ``todo'' block.


EXAMPLES

Using shell exit status codes

Often the exit status of a program will indicate whether a test succeeded. For example:

  program
  test_ok $? "comparison"

Managing test output

One should always clean up test output if a test is successful, but it should be kept around if the test fails, to provide diagnostic information. Here's one way of handling this using a result hook.

  function result_hook {
    typeset pass=$1
    if (( pass )); then
      rm ${test_id}-*
    else
      echo "# Failed results are in:" ${test_id}*
    fi
  }
  test_set result result_hook

This example assumes that all output is written to files which begin with $test_id.

Another way is to use the process id as a prefix and to automatically delete all files with that prefix upon exit from the script. The benefit of this approach is that the shell handles cleanup even when there's an error in the script, and you can be somewhat less strict in naming intermediate files (say $$-temp) knowing that the shell will catch all of them.

In this case, however, failed results must be renamed to avoid the deletion.

  trap "rm -f $$-*" EXIT
  function test_result_hook_save_output_on_failure {
    typeset pass=$1
  
    # if not in a subtest
    if (( test_subnum == 0 )) ; then
  
      # the test has succeeded, remove the test files
      if (( pass )); then
        rm $$-${test_id}*
  
      # the test has failed:  save the test files by stripping the
      # $$- prefix and renaming it to fail-$(testprog)-$(testnum)
      else
  
        typeset exe=$(basename $0)
  
        for src in $$-${test_id}*; do
          if [ -f "$src" ]; then
            dst="fail-${exe}-${src#$$-}"
            mv "$src" "$dst"
            echo "# Failed results are in: $dst"
          fi
        done
      fi
  
    fi
  }

This function is part of testlib.sh, and may be activated via

  test_set_hook result test_result_hook_save_output_on_failure


VERSION

This documents version @VERSION@ of =testlib_sh=.


COPYRIGHT AND LICENSE

This software is Copyright by the Smithsonian Astrophysical Observatory. It is released under the GNU General Public License. You may find a copy at http://www.fsf.org/copyleft/gpl.html.


AUTHOR

Diab Jerius <djerius@cfa.harvard.edu>