testlib_sh - bash/ksh test library
. testlib.sh
# initialize test suite test_initialize
# set test suite configuration variable test_set variable value
# something to run after every test test_set_hook result function_name
# status comparison test_ok $status $test_name test_nok $badstatus $test_name test_notok $badstatus $test_name
# numerical (integer) comparison test_num_is $value $exp $test_name test_num_isnt $value $exp $test_name
# numerical (real) comparison using ndiff test_ndiff_is [options] $value $exp $test_name test_ndiff_isnt [options] $value $exp $test_name
# string comparison test_is $value $exp $test_name test_isnt $value $exp $test_name
# file comparison test_file_is [options] $file $exp_file $test_name test_file_isnt [options] $file $exp_file $test_name
# numeric file comparison using ndiff test_numfile_is [options] $file $exp_file $test_name test_numfile_isnt [options] $file $exp_file $test_name
# file greps test_file_grep_is [options] "$pattern" $file $test_name test_file_grep_isnt [options] "$pattern" $file $test_name
# just doit test_fail $test_name test_pass $test_name
# skipping tests if (( skip )); then
test_skip "because" 2
else
test_ok $status "foo" test_ok $status "bar"
fi
# tests you'll one day get to work test_todo_begin
test_ok $status $test_name test_ok $status $test_name
test_end_todo
test_begin_subtests
test_end_subtests
test_summary test_exit
testlib_sh is a library for writing standardized tests in bash or ksh. It provides a set of functions which will generate uniform reporting on tests, accumulate statistics, and generate an exit value based upon the results of the tests. testlib_sh is modelled after the Perl Test::More module.
All testlib_sh functions begin with test_
. testlib_sh uses a
number of internal functions and variables to keep track of things.
These all begin with _test
. Please do not alter them! Some
variables are made public for use by the calling application. See
Testlib Variables for more information.
To begin a test suite, first source the testlib.sh library, and then initialize the suite:
. testlib.sh test_initialize
This will initialize the internal state variables and make available the variable test_id which may be used by the tests as a basis for filenames or the like.
After calling test_initialize, the behavior of the library may be changed via several configuration variables. Use the test_set command to set their values. CONFIGURATION VARIABLES lists the available variables.
testlib_sh provides the ability for code to be called at specific points in its processing flow. A hook is a point where the user code may be called. Tests have access to hooks via test_set_hook. See HOOKS for more information.
There are a variety of functions which can be used to record the result of a test. These include
tests of status values
comparison of string values
comparison of numerical values (according to the shell's idea about what is a numerical quantity)
comparison of numerical values using ndiff
file comparison tests using cmp
file comparison tests using ndiff
pass/fail for tests which are difficult to categorize
Typically, the _is
and _isnt
tests should be used, as these will
give more useful diagnostics upon error. _ok
and _notok
tests
are next best. pass and fail should be used when
testing is so complicated that none of the other functions suffice.
Note that test_ok regards a status value of 0
as success. This
allows it to easily be used with the exit status variable in the
shell:
cmp file1 file2 test_ok $? "comparison"
Tests may be also be skipped. This is useful if some resource is not available. This requires a bit of code to ensure that that the tests are not run, and that the total number of tests remains constant.
# set variable skip to true if skip condition is met if (( skip )); then
test_skip $reason_for_skipping $number_of_tests_to_skip
else
# tests to run if not skipping. # number must be $number_of_tests_to_skip test_ok ...
fi
Consider marking tests as ``todo'' if they test parts of the system which are not yet ready for prime time but should be tested. This provides incentive for finishing them. To mark them as such, surround them with test_begin_todo and test_end_todo. Such tests may also be skipped if running the todo tests is impossible.
test_begin_todo
test_ok $status $test_name
if (( skip )); then
test_skip $reason $number
else
test_ok $status $test_name
fi
test_end_todo
Tests which are marked ``todo'' and which fail are not counted against the exit status of the test script. Tests which succeed should obviously be moved out of todo sections!
If the test suite is using the test id (see Testlib Variables) to name output files and it may be necessary to perform multiple tests on a single output file, the test id for the tests will be out of sync with that used to generate the output files. For example, consider this suite:
logfile="$test_id.log" program > $logfile
grep -q foo $logfile test_ok $? "foo in logfile"
grep -q boo $logfile test_ok $? "boo in logfile"
If boo
is not in the logfile, then the second test will report the
problem in test 2, and if you are using the file cleanup conventions
in the EXAMPLES section, the wrong file will be saved, as it is
named using test 1's id. The way around this is to declare that you
will be performing subtests:
test_begin_subtests
logfile="$test_id.log" program > $logfile
grep -q foo $logfile test_ok $? "foo in logfile"
grep -q boo $logfile test_ok $? "boo in logfile" test_end_subtests The test id remains constant throughout the subtest block, so things remain in sync. You can get the subtest number via the C<test_subnum> variable. This is defined to be C<0> if not within a subtest block. See L</Testlib Variables> for more information.
Note: If a result callback function has been specified with test_set_hook, that function will be called for each subtest, as well as an additional call after all of the subtests have been completed. The argument to the final call will be true only if all of the subtests have completed. This can be used to delay cleanup of test output until all of the subtests have been complete. Simply check if test_subnum is zero to determine if all of the subtests are complete.
The last statements in your tests should typically be
test_summary test_exit
The first will generate the final summary of the test results, the last will exit with a status value indicating whether any tests failed.
test_exit
This is should be the last statement in the test suite, as it will exit with a status reflecting the results of the tests.
test_pass $test_name test_fail $test_name
If all other test_* routines are useless to you, you can use these to indicate the status of a test.
test_pass indicates that the named test has passed.
test_fail indicates that the named test has failed.
Consider using the _is
, _isnt
, _ok
and _notok
tests if
possible.
test_initialize
This should be the first statment in the test suite. It initializes the test environment.
test_ok $status $test_name
This reports success if a $status
is 0.
test_nok $status $test_name test_notok $status $test_name
These reports success if a $status
is 1.
test_is $value $expected $test_name test_isnt $value $expected $test_name
These are string comparison tests.
test_is reports success if the two strings are identical.
test_isnt reports success if the two strings differ.
test_num_is $value $expected $test_name test_num_isnt $value $expected $test_name
These are numerical comparison tests, according to what the shell believes a number to be.
test_num_is reports success if the two numbers are identical.
test_num_isnt reports success if the two numbers differ.
test_ndiff_is [options] $value $expected $test_name test_ndiff_isnt [options] $value $expected $test_name
These are numerical comparison tests, using the ndiff program.
test_ndiff_is reports success if the two numbers are identical.
test_ndiff_isnt reports success if the two numbers differ.
The test will fail if ndiff is not available.
Both tests take the following options:
This indicates there are no further options. This must be used
if $value
might begin with the -
character.
Options to be passed to ndiff. It is important that they be passed as a single string value (quote all spaces).
test_file_is [options] $file $exp_file $test_name test_file_isnt [options] $file $exp_file $test_name
These are file comparison tests, using the cmp command.
test_file_is reports success if the two files are identical.
test_file_isnt reports success if the two files differ.
Both tests take the following options:
This indicates there are no further options. This must be used
if $file
might begin with the -
character.
This file name will be output instead of $file on error. This is
useful if $file
will be renamed upon failure and the error message
should track that.
test_numfile_is [options] $file $exp_file $test_name test_numfile_isnt [options] $file $exp_file $test_name
These are file comparison tests, using the ndiff command, which does a numerical difference of the files
test_numfile_is reports success if the two files are identical. test_numfile is identical to test_file_is.
test_numfile_isnt reports success if the two files differ.
Both tests take the following options:
This indicates there are no further options. This must be used
if $file
might begin with the -
character.
Options to be passed to ndiff. It is important that they be passed as a single string value (quote all spaces).
This file name will be output instead of $file on error. This is
useful if $file
will be renamed upon failure and the error message
should track that.
test_file_grep_is [options] $pattern $file $test_name test_file_grep_isnt [options] $pattern $file $test_name
These tests grep the specified file for the specified pattern. The pattern should probably be escaped from the shell.
test_file_grep_is reports success if the pattern was found; test_file_grep_isnt reports success if the pattern was not found.
Both tests take the following options:
This indicates there are no further options. This must be used
if $file
might begin with the -
character.
Options to be passed to grep. It is important that they be passed as a single string value (quote all spaces).
A file to which grep's standard output should be written. It is normally discarded.
Use egrep.
Use fgrep.
Set the named configuration variable to the value. If the variable
does not exist, a message will be printed and the function will return
1
.
See CONFIGURATION VARIABLES for more information.
This directs testlib_sh to call the given function
function_to_call at the processing point identified by hook. If
hook is not a recognized name, an error message is printed and the
function will return 1
.
See HOOKS for more information.
This is a convenience function to override the shell's idea of the
status of the most recently executed command. This is useful when
status checking must be postponed for some reason, and $?
is to
be consulted for the status. For example,
test_set check_exit_status 1 output=$(run_test_exe) status=$?
...do stuff to generate $value from $output..
# depend upon check_exit_status to check $? for us test_set_status $status test_is $value $expected $test_name
test_skip $reason $number
This indicates that $number
tests should be recorded as being
skipped. It is up to the test suite to ensure that those tests are
not run, and that the number of tests not run is consistent with
$number
. See Skipping Tests.
test_summary
This generates a summary of the tests which were run or skipped.
test_begin_todo [... todo tests ...] test_end_todo
Begin and end a block of tests marked as ``todo''. Failure of these tests does
not count towards the exit status of the suite. test_end_todo must
be called to end the ``todo'' block. The variable test_todo will be
set to 1
while a ``todo'' block is active.
test_begin_subtests [... subtests ...] test_end_subtests
Begin and end a block of subtests. See Subtests. The block must be closed with a test_end_subtests command.
The behavior of testlb_sh can be changed by setting various configuration variables with the test_set command. The available variables are:
If non-zero, tests (excluding test_pass and test_fail) will first check the exit status of the last command run before the actual test specified and will generate an error if the status is not what is expected; the ``real'' test will not be performed. The expected exit status depends upon the value of check_exit_status:
The exit status must be zero (success). For example, the following will fail:
test_set check_exit_status 1 false test_ok 0 "normally should pass"
The exit status must be non-zero (failure). For example, the following will fail:
test_set check_exit_status -1 true test_ok 0 "normally should pass"
check_exit_status can be used to simplify tests which need to check both the exit value of a command and its output (or some other side effect). For example, rather than write
result=$(my_command) test_ok $? "my_command execute" test_is result "valid" "my_command results"
one can just write
result=$(my_command) test_is result "valid" "my_command results"
In many cases, this may remove the requirement to use subtests.
NOTE!! Be sure not to inadvertently run other commands before the testlib routine is called. For example:
some_program test_is "$result" $(date) "NO!"
will check the exit status of date
, not some_program
!
If true, file comparison tests will copy $file
to $exp_file
if
the latter does not exist. This allows one to easily create the set
of expected output on the first run through the test. Of course, one
should verify that the output is correct! This should not be set
for production use! A message is printed if the copying occurs.
Hooks are points in the test suite software where user code may be executed. The test_set_hook function is used to assign a function to a hook.
Currently the following hooks are available:
The result hook is called after each test result is determined. The function is passed a single boolean argument indicating whether the test has passed.
If subtests are enabled, the result hook will be called after each
subtest, with an additional call after the subtests have been
completed. The argument to this last call will be true only if all of
the subtests have completed. To distinguish it from the preceeding
calls, test the test_subnum variable; it will have a value of 0
for the final call.
This can be used to delay cleanup of test output until all of the subtests have been complete. Simply check if test_subnum is zero to determine if all of the subtests are complete.
See the Managing test output section for more information. The function described there is part of testlib.sh and may be used via
test_set_hook result test_result_hook_save_output_on_failure
Note that previously test_result_hook was used to set this hook. This function is now deprecated.
The following variables are available for use by the test suite:
This contains a zero prefixed and padded representation of the current test number.
This contains the raw test number (no padding)
This contains a zero prefixed and padded representation of the current
sub test number. It is 00
if not in a subtest block.
This contains the raw subtest number (no padding). It is 0
if not
in a subtest block.
This contains the label which will be printed by testlib_sh when identifying the current test.
This indicates whether the last test succeeded; 0
means it did,
non-zero
means it didn't.
This is a boolean value indicating whether the test suite is currently in a ``todo'' block.
Often the exit status of a program will indicate whether a test succeeded. For example:
program test_ok $? "comparison"
One should always clean up test output if a test is successful, but it should be kept around if the test fails, to provide diagnostic information. Here's one way of handling this using a result hook.
function result_hook { typeset pass=$1
if (( pass )); then rm ${test_id}-* else echo "# Failed results are in:" ${test_id}* fi }
test_set result result_hook
This example assumes that all output is written to files which begin
with $test_id
.
Another way is to use the process id as a prefix and to automatically
delete all files with that prefix upon exit from the script. The
benefit of this approach is that the shell handles cleanup even when
there's an error in the script, and you can be somewhat less strict
in naming intermediate files (say $$-temp
) knowing that the shell
will catch all of them.
In this case, however, failed results must be renamed to avoid the deletion.
trap "rm -f $$-*" EXIT
function test_result_hook_save_output_on_failure { typeset pass=$1 # if not in a subtest if (( test_subnum == 0 )) ; then # the test has succeeded, remove the test files if (( pass )); then rm $$-${test_id}* # the test has failed: save the test files by stripping the # $$- prefix and renaming it to fail-$(testprog)-$(testnum) else typeset exe=$(basename $0) for src in $$-${test_id}*; do if [ -f "$src" ]; then dst="fail-${exe}-${src#$$-}" mv "$src" "$dst" echo "# Failed results are in: $dst" fi done fi fi }
This function is part of testlib.sh, and may be activated via
test_set_hook result test_result_hook_save_output_on_failure
This documents version @VERSION@ of =testlib_sh=.
This software is Copyright by the Smithsonian Astrophysical Observatory. It is released under the GNU General Public License. You may find a copy at http://www.fsf.org/copyleft/gpl.html.
Diab Jerius <djerius@cfa.harvard.edu>