The OTP Test Server Framework

MODULE

MODULE SUMMARY

Test Server Framework for testing OTP

DESCRIPTION

This is a framework for testing OTP. The ts module implements the interface to all the functionality in the framework.

The framwork is built on top of the Test Server Controller, test_server_ctrl, and provides a high level operator interface. The main fatures added by the framework are:

Automatic compilation of test suites and data directories

Collection of files in central directories and creation of additional HTML pages for better overview.

Single command interface for running all available tests

Spawns a new node with correct parameters before starting the test server

Atomatically creates the parameter file needed when running tests on remote target

More information about the Test Server Framework and how to run test cases can be found in the Test Server User's Guide.

For writing you own test server framework, please turn to the reference manual for the Test Server Controller and chapter named "Writing your own test server framework" in the Test Server User's Guide. guide.

EXPORTS

install() -> ok | {error, Reason}
install(TargetSystem) -> ok | {error, Reason}
install(Opts) -> ok | {error, Reason}
install(TargetSystem,Opts) -> ok | {error, Reason}

TargetSystem = {Architecture, TargetHost}

Architecture = atom() or string()

e.g. "ose" or "vxworks_ppc603"

TargetHost = atom() or string()

The name of the target host

Opts = list()

Installs and configures the Test Server Framework for running test suites. If a remote host is to be used, the TargetSystem argument must be given so that "cross installation" can be done. This should be used for testing on VxWorks or OSE/Delta. Installation is required for any of the functions in ts to work.

Opts may be one or more of

{longnames, Bool}: Use fully qualified hostnames for test_server and slave nodes. Bool is true or false (default).
{verbose, Level}: Verbosity level for test server output, set to 0, 1 or 2, where 0 is quiet(default).
{hosts, Hosts}: This is a list of available hosts on which to start slave nodes. It is used when the {remote, true} option is given to the test_server:start_node/3 function. Also, if {require_nodenames, Num} is contained in a test specification file, the generated nodenames will be spread over all hosts given in this Hosts list. The hostnames are given as atoms or strings.
{slavetargets, SlaveTarges}: For VxWorks and OSE/Delta only. This is a list of available hosts where slave nodes can be started. This is necessary because only one node can run per host in the VxWorks and OSE/Delta environments. This is not the same as {hosts, Hosts} because it is used for all slave nodes - not only the ones started with {remote, true}. The hostnames are given as atoms or strings.
{crossroot, TargetErlRoot}: Erlang root directory on target host
This option is mandatory for remote targets
{master, {MasterHost, MasterCookie}}: If target is remote and the target node is started as a slave node (which is the case for OSE/Delta), this option indicates which master and cookie to use. The given master will also be used as master for slave nodes started with test_server:start_node/3. It is expected that the erl_boot_server is started on the master node before the test is run. If this option is not given, the test server controller node is used as master and the erl_boot_server is automatically started.
{erl_start_args, ArgString}: Additional arguments to be used when starting the test server controller node. ArgString will be appended to the command line when starting the erlang node. Note that this will only affect the startup of the controller node, i.e. not the target node or any slave nodes startet from a test case.
{ipv6_hosts, HostList}: This option will be inserted in the Config parameter for each test case. HostList is a list of hosts supporting IPv6.

help() -> ok

Presents simple help on the functions in ts. Useful for quick reference.

tests() -> Tests

Returns the list of available tests. This is actually just a list of all test specification files found by looking up "../*_test/*.spec".

In each ../Name_test/ directory there should be one test specification file named Name.spec.

run() -> ok | {error, Reason}
run([all_tests|Opts])
run(Specs)
run(Specs, Opts)
run(Spec, Module)
run(Spec, Module, Opts)
run(Spec, Module, Case)
run(Spec, Module, Case, Opts)

Specs = Spec | [Spec]

Spec = atom()

Module = atom()

Case = atom()

Opts = [Opt]

Opt = batch | verbose | {verbose, Level} | {vars, Vars} | keep_topcase | cover | cover_details |{cover,CoverFile} | {cover_details,CoverFile} | {trace, TraceSpec}

Level = integer(); 0 means silent

Vars = list() of key-value tuples

CoverFile = string(); name of file listing modules to exclude from or include in cover compilation. The name must include full path to the file.

Reason = term()

This function runs test suite(s)/case(s). To be able to run any tests, ts:install must first be called to create the variables file needed. To run a whole test specification, only specify the name of the test specification, and all test suite modules belonging to that test spec will be run. To run a single module in a test specification, use the Module argument to specify the name of the module to run and all test cases in that module will be run, and to run a specified test case, specify the name of the test case using the Case argument. If called with no argument, all test specifications availiable will be run. Use ts:tests/0 to see the available test specifications.

If the batch option is not given, a new xterm is started (unix) when ts:run is called.

The verbose option sets the verbosity level for test server output. This has the same effect as if given to ts:install/1/2

The vars option can be used for adding configuration variables that are not in the variables file generated during installation. Can be any of the Opts valid for ts:install/1/2.

The keep_topcase option forces ts to keep the topcase in your test specification file as is. This option can only be used if you don't give the Module or Case parameters to ts:run. The keep_topcase option is necessary if your topcase contains anything other than {dir,"../<Name>_test"}. If the option is not used, ts will modify your topcase.

The cover and cover_details options indicates that the test shall be run with code coverage analysis. cover_details means that analysis shall be done on the most detailed level. If the test is run with a remote target, this option creates a list of uncovered lines in each cover compiled module. If the test is run with a local target, each cover compiled module will be analysed with cover:analyse_to_file/1. The cover options will only create an overview of all cover compiled modules with the number of covered and not covered lines.

The CoverFile which can be given with the cover and cover_details options must be the filename of a file listing modules to be excluded from or included in the cover compilation. By default, ts believes that Spec is the name of an OTP application and that all modules in this application shall be cover compiled. The CoverFile can exclude modules that belong to the application and add modules that don't belong to the application. The file can have the following entries:

{exclude, all | ExcludeModuleList}.
{include, IncludeModuleList}.

Note that each line must end with a full stop. ExcludeModuleList and IncludeModuleList are lists of atoms, where each atom is a module name.

If the cover or cover_details options are given on their own, the directory ../<Spec>_test is searched for a CoverFile named <Spec>.cover. If this file is not found, Spec is assumed to be the name of an OTP application, and all modules in the ebin directory for the application are cover compiled. The ebin directory is found by adding ebin to code:lib_dir(Spec).

The same cover compiled code will be loaded on all slave or peer nodes started with test_server:start_node/3. If the loading fails, e.g. if the node runs an old version of OTP, the node will simply not be a part of the coverage analysis. Note that slave and peer nodes must be stopped with test_server:stop_node/1 for the node to be part of the coverage analysis, else the test server will not be able to fetch coverage data from the node.

The trace option is used to turn on call trace on target and on slave or peer nodes started with test_server:start_node/3. TraceSpec can be the name of a trace information file, or a list of elememnts like the ones in a trace information file. Please turn to the reference manual for test_server_ctrl:trc/1 for details about the trace information file.

cross_cover_analyse(Level) -> ok
cross_cover_analyse([Level]) -> ok

Analyse cover data collected from all tests.

See test_server_ctrl:cross_cover_analyse/1

r() -> ok
r(Opts) -> ok
r(SpecOrSuite) -> ok
r(SpecOrSuite,Opts) -> ok
r(Suite,Case) -> ok
r(Suite,Case,Opts) -> ok

SpecOrSuite = Spec | Suite

Spec = string()

"Name.spec" or "Name.spec.OsType", where OsType is ose or vxworks

Suite = atom()

Case = atom()

Opts = [Opt]

Opt = {Cover,AppOrCoverFile} | {Cover,Application,CoverFile}

Cover = cover | cover_details

AppOrCoverFile = Application | CoverFile

Application = atom()

OTP application to cover compile

CoverFile = string()

Name of file listing modules to exclude from or include in cover compilation

This function can be used to run a test suites or test cases directly, without any of the additional features added by the test server framework. It is simply a wrapper function for the add_dir, add_spec, add_module and add_case functions in test_server_ctrl:

r() -> add_dir(".")
r(Spec) -> add_spec(Spec)
r(Suite) -> add_module(Suite)
r(Suite,Case) -> add_case(Suite,Case)

To use this function, it is required that the test suite is compiled and in the code path of the node where the function is called. The function can be used without having ts installed.

For information about the cover and cover_details options, see test_server_ctrl:cover/2/3.

index() -> ok | {error, Reason}

Reason = term()

This function updates the local index page. This can be useful if a previous test run was not completed and the index is incomplete.

clean() -> ok
clean(all) -> ok

This function cleans up log directories created when running test cases. clean/0 cleans up all but the last run of each application. clean/1 cleans up all test runs found.

estone() -> ok | {error, Reason}
estone(Opts) -> ok

This function runs the EStone test. It is a shortcut for running the test suite estone_SUITE in the kernel application.

Opts is the same as the Opts argument for the ts:run functions.

Makfile.src in Data Directory

If a data directory contains code which must be compiled before the test suite is run, a makefile source called Makefile.src can be placed in the data directory. This file will be converted to a valid makefile by ts:run/0/1/2/3/4.

The reason for generating the makefile is that you can use variables from the variables file which was generated by ts:install/0/1/2. All occurencies of @Key@ in Makefile.src is substituted by the Value from {Key,Value} found in the variables file. Example:

Cut from variables:

    ...
    {'EMULATOR',"beam"}.
    {'CFLAGS',"-g -O2"}.
    {'LD',"$(CC) $(CFLAGS)"}.
    {'CC',"gcc"}.
    ...

Makefile.src for compiling erlang code could look something like this:

    EFLAGS=+debug_info
    
    all: ordsets1.@EMULATOR@
    
    ordsets1.@EMULATOR@: ordsets1.erl
           erlc $(EFLAGS) ordsets1.erl

Makefile.src for compiling c code could look something like this: