README file for Singular test-suite
===================================
This is the test-suite for Singular. This directory contains
regress.cmd -- perl script which drives tests
Short -- directory containing "short" tests
Long -- directory containing "long" tests
Old -- directory containing "old" (TST) tests
recommended set of tests:
./regress.cmd Old/universal.lst Buch/buch.lst Plural/short.lst Plural/dmod.lst Short/ok_s.lst Long/ok_l.lst
Run
perl ./regress.cmd -h
for a summary of the comaand-line options of regress.cmd.
To test(s) correctness of Singular:
-----------------------------------
1.) Put the Singular binary in this directory
(prefered solution: ln -s <path_to_Singular_binary> Singular)
2.) Run
perl ./regress.cmd [*.tst] [*.lst]
with the desired tst (i.e., Singular test scripts) or lst (list
of tst files -- see Old/universal.lst for an example) file(s).
(prefered test scenario:
perl ./regress.cmd Old/universal.lst Short/ok_s.lst Long/ok_l.lst
)
3.) If no differences between the new results and the
previously obtained results are found, regress.cmd exits with
status 0.
4.) If differences between the new results and the
previously obtained results are found, regress.cmd exits with
status != 0, and, for each file xx.tst which lead to a
difference, the coresponding files xx.new.res, xx.res, and xx.diff
are kept, and differences are written to stdout.
5.) The Singular executable which is used for testing can
alternatively be specified using the -s <Singular> option.
To test memory/timing performance of Singular:
----------------------------------------------
1.) Put the Singular binary in this directory
2.) Run
perl ./regress.cmd -e [*.tst] [*.lst]
with the desired tst (i.e., Singular test scripts) or lst (list
of tst files -- see Old/universal.lst for an example) file(s).
3.) If no result differences occur, and if timing/memory performance
differences are less than $error_val (currently set to 5)
per-cent, then regress.cmd exits with status 0.
4.) Otherwise, exit status is != 0, and the corresponding files
xx.new.res, xx.res, xx.diff, and xx.stat.diff (which logs the
differences of timing/memory performance) are kept, and
differences are written to stdout.
5.) Using the -r instead of the -e option only reports about
memory/timings differences, i.e., it does not cause an exit with
status != 0 if timing/memory performance is above specified
per-cent.
6.) The timing/memory performance differenes which trigger an error
(or, report) can be also be set: Either use
./regress.cmd -[e,r] all%<val>
which sets tolerance of all criteria to <val> per-cent, or, use
./regress.cmd -[e,r] <crit>%<val>
which sets tolerance of all <crit> to <val> per-cent.
To generate new result file(s):
-------------------------------
Run
perl ./regress.cmd -g [*.tst] [*.lst]
with the desired tst (i.e., Singular test scripts) or lst (list
of tst files).
NOTE: Running regress.cmd with the -g option re-generates the result
and stat files, i.e., all previous results and statisitics are
overwritten and lost!
To include timing/memory statistics for a machine:
---------------------------------------------------
Run
perl ./regress.cmd -m [*.tst] [*.lst]
with the desired tst (i.e., Singular test scripts) or lst (list
of tst files).
NOTE: Running regress.cmd with the -m option merges the
timings/memory results of the current machine into the respective
xx.stat file and overwrites previous timings/memory results of the
current machine.
To add a new test for a library/commando to the test-suite:
-----------------------------------------------------------
* The following files need to be provided:
short/xx_s.tst : Singular code for short and basic tests
long/xx_l.tst : Singular code for long and extended tests
or, alternatively:
short/xx.tst: Singular code for short tests, only
short/xx.res : Result output of xx.tst
* xx_s.tst should test the essential functionality of the
library/commando in a relatively short time (say, in no more than
30s).
* xx_l.tst should test the functionality of the
library/commando in detail so that, if possible, all relevant
cases/results are tested. Nevertheless, such a test should not run
longer than, say, 10 minutes.
* If useful tests do generally execute in a short time, have xx.tst,
only.
* Rules for providing tst files:
1.) tst files always start with the following three commandos as
preamble:
LIB "tst.lib";
tst_init();
tst_ignore("CVS ID $Id$"); // or, write own version number here
tst_init() writes some general info to stdout (like date,
uname, hostname, version, etc.). The library tst.lib (contained in
the Singular distribution) provides, among others, the routines
tst_init() and tst_ignore().
2.) tst files should end with the following statements:
tst_status(1); $
which enables (automatic) checks of the timing/memory performance of
Singular.
3.) All system-dependent output (like run-times, memory usages,
pathnames, dates, etc.) should generally be avoided.
4.) After time/memory critical sections of the tst files, the
command
tst_status();
should be inserted. This enables (automatic) checks of the
timing/memory performance of Singular since the last call to
tst_status (resp. since the start-up of Singular).
5.) If system-dependent output can not be avodied, the routine
tst_ignore(...) should be used:
tst_ignore(val [, keyword]): 'val' can have arbitrary type for which
a string conversion exists;
if present, keyword must be one of the
following strings: "time", "memory"
tst_ignore() outputs 'val' by prepending the following prefix:
no keyword -- // tst_ignore:
"time" keyword -- // tst_ignore: time:
"memory" keyword -- // tst_ignore: memory:
which causes automatic tests to ignore these lines when doing a diff
on result files.
* The command for generating xx.res.gz.uu and xx.stat is:
cd <directopry where xx.tst resides>; perl ../regress.cmd -g xx.tst
* You can download the newest version of tst.lib from
ftp://www.mathematik.uni-kl.de/pub/Math/Singular/singular-cd/LIBRARIES/tst.lib