ct_suite — common_test v1.31

Search documentation of common_test

# ct_suite behaviour (common_test v1.31)

Copy Markdown View Source

The following section describes the mandatory and optional test suite functions that `Common Test` calls during test execution. For more details, see section Writing Test Suites in the User's Guide.

## Summary

### Types

ct_config()

The configuration data that can be modified.

ct_group_def()

The test group definition, as returned by `Module:groups/0`.

ct_group_props()
ct_group_props_ref()
ct_group_ref()
ct_group_repeat_type()
ct_groupname()

The name of the test group.

ct_hooks()
ct_info()

The test suite information, as returned by `Module:suite/0`, `Module:group/1` and `Module:Testcase/0`.

ct_info_required()
ct_info_required_subkeys()
ct_info_timetrap()
ct_info_timetrap_fun()
ct_status()

The status value for a nested subgroup.

ct_subgroups_def()
ct_test_def()

The test suite definition, as returned by `Module:all/0`.

ct_test_repeat()
ct_testcase_ref()
ct_testcase_repeat_prop()
ct_testname()

The name of the testcase function.

### Callbacks

all()

Returns the list of all test cases and test case groups in the test suite module to be executed.

end_per_group(GroupName, Config)

This function is called after the execution of a test case group is finished. It is meant to be used for cleaning up after `Module:init_per_group/2`.

end_per_suite(Config)

This function is called as the last test case in the suite. It is meant to be used for cleaning up after `Module:init_per_suite/1`.

end_per_testcase(TestCase, Config)

This function is called after each test case, and can be used to clean up after `Module:init_per_testcase/2` and the test case.

group(GroupName)

The test case group information function. It is supposed to return a list of tagged tuples that specify various properties related to the execution of a test case group (that is, its test cases and subgroups). Properties set by `Module:group/1` override properties with the same key that have been set previously by `Module:suite/0`.

groups()

Defines test case groups. For details, see section Test Case Groups in the User's Guide.

init_per_group(GroupName, Config)

This configuration function is called before execution of a test case group. It typically contains initializations that are common for all test cases and subgroups in the group, and that must only be performed once.

init_per_suite(Config)

This configuration function is called as the first function in the suite. It typically contains initializations that are common for all test cases in the suite, and that must only be done once.

init_per_testcase(TestCase, Config)

This function is called before each test case.

suite()

The test suite information function. Returns a list of tagged tuples specifying various properties related to the execution of this test suite (common for all test cases in the suite).

'Testcase'()

The test case information function. It is supposed to return a list of tagged tuples that specify various properties related to the execution of this particular test case. Properties set by `Module:Testcase/0` override properties set previously for the test case by `Module:group/1` or `Module:suite/0`.

'Testcase'(Config)

The implementation of a test case. Call the functions to test and check the result. If something fails, ensure the function causes a runtime error or call `ct:fail/1,2` (which also causes the test case process to terminate).

## Types

# ct_config()

    -type ct_config() :: [{Key :: atom(), Value :: term()}].

The configuration data that can be modified.

# ct_group_def()

    -type ct_group_def() ::
              {ct_groupname(),
               ct_group_props(),
               [ct_testname() | ct_group_def() | {group, ct_groupname()} | ct_testcase_ref()]}.

The test group definition, as returned by `Module:groups/0`.

# ct_group_props()

(not exported)

    -type ct_group_props() ::
              [parallel | sequence | shuffle |
               {shuffle, Seed :: {integer(), integer(), integer()}} |
               {ct_group_repeat_type(), ct_test_repeat()}].

# ct_group_props_ref()

(not exported)

    -type ct_group_props_ref() :: ct_group_props() | default.

# ct_group_ref()

(not exported)

    -type ct_group_ref() ::
              {group, ct_groupname()} |
              {group, ct_groupname(), ct_group_props_ref()} |
              {group, ct_groupname(), ct_group_props_ref(), ct_subgroups_def()}.

# ct_group_repeat_type()

(not exported)

    -type ct_group_repeat_type() ::
              repeat | repeat_until_all_ok | repeat_until_all_fail | repeat_until_any_ok |
              repeat_until_any_fail.

# ct_groupname()

    -type ct_groupname() :: atom().

The name of the test group.

# ct_hooks()

(not exported)

    -type ct_hooks() ::
              [CTHModule ::
                   atom() |
                   {CTHModule :: atom(), CTHInitArgs :: term()} |
                   {CTHModule :: atom(), CTHInitArgs :: term(), CTHPriority :: integer()}].

# ct_info()

    -type ct_info() ::
              {timetrap, ct_info_timetrap()} |
              {require, ct_info_required()} |
              {require, Name :: atom(), ct_info_required()} |
              {userdata, UserData :: term()} |
              {silent_connections, Conns :: [atom()]} |
              {stylesheet, CSSFile :: string()} |
              {ct_hooks, CTHs :: ct_hooks()}.

The test suite information, as returned by `Module:suite/0`, `Module:group/1` and `Module:Testcase/0`.

# ct_info_required()

(not exported)

    -type ct_info_required() ::
              Key ::
                  atom() |
                  {Key :: atom(), SubKeys :: ct_info_required_subkeys()} |
                  {Key :: atom(), SubKey :: atom()} |
                  {Key :: atom(), SubKey :: atom(), SubKeys :: ct_info_required_subkeys()}.

# ct_info_required_subkeys()

(not exported)

    -type ct_info_required_subkeys() :: SubKey :: atom() | [SubKey :: atom()].

# ct_info_timetrap()

(not exported)

    -type ct_info_timetrap() ::
              timeout() |
              {seconds, integer()} |
              {minutes, integer()} |
              {hours, integer()} |
              {Mod :: atom(), Func :: atom(), Args :: list()} |
              ct_info_timetrap_fun().

# ct_info_timetrap_fun()

(not exported)

    -type ct_info_timetrap_fun() :: fun().

# ct_status()

    -type ct_status() :: ok | skipped | failed.

The status value for a nested subgroup.

# ct_subgroups_def()

(not exported)

    -type ct_subgroups_def() ::
              {ct_groupname(), ct_group_props_ref()} |
              {ct_groupname(), ct_group_props_ref(), ct_subgroups_def()}.

# ct_test_def()

    -type ct_test_def() :: ct_testname() | ct_group_ref() | ct_testcase_ref().

The test suite definition, as returned by `Module:all/0`.

# ct_test_repeat()

(not exported)

    -type ct_test_repeat() :: integer() | forever.

# ct_testcase_ref()

(not exported)

    -type ct_testcase_ref() :: {testcase, ct_testname(), ct_testcase_repeat_prop()}.

# ct_testcase_repeat_prop()

(not exported)

    -type ct_testcase_repeat_prop() ::
              [{repeat, ct_test_repeat()} |
               {repeat_until_ok, ct_test_repeat()} |
               {repeat_until_fail, ct_test_repeat()}].

# ct_testname()

    -type ct_testname() :: atom().

The name of the testcase function.

## Callbacks

# all()

    -callback all() -> [TestDef :: ct_test_def()] | {skip, Reason :: term()}.

Returns the list of all test cases and test case groups in the test suite module to be executed.

This list also specifies the order the cases and groups are executed by `Common Test`. A test case is represented by an atom, the name of the test case function, or a `testcase` tuple indicating that the test case shall be repeated. A test case group is represented by a `group` tuple, where `GroupName`, an atom, is the name of the group (defined in `Module:groups/0`). Execution properties for groups can also be specified, both for a top-level group and for any of its subgroups. Group execution properties specified here override properties in the group definition (see `Module:groups/0`). (With value `default`, the group definition properties are used).

If `{skip, Reason}` is returned, all test cases in the module are skipped and `Reason` is printed on the HTML result page.

For details on groups, see section Test Case Groups in the User's Guide.

# end_per_group(GroupName, Config)

(optional)

    -callback end_per_group(GroupName :: ct_groupname(), Config :: ct_config()) ->
                               term() | {return_group_result, Status :: ct_status()}.

This function is called after the execution of a test case group is finished. It is meant to be used for cleaning up after `Module:init_per_group/2`.

A status value for a nested subgroup can be returned with `{return_group_result, Status}`. The status can be retrieved in `Module:end_per_group/2` for the group on the level above. The status is also used by `Common Test` for deciding if execution of a group is to proceed if property `sequence` or `repeat_until_*` is set.

For details about test case groups, see section Test Case Groups in the User's Guide.

If this function is defined, then `Module:init_per_group/2` must also be defined.

# end_per_suite(Config)

(optional)

    -callback end_per_suite(Config :: ct_config()) -> term() | {save_config, SaveConfig :: ct_config()}.

This function is called as the last test case in the suite. It is meant to be used for cleaning up after `Module:init_per_suite/1`.

For information on `save_config`, see section Saving Configuration Data in the User's Guide.

If this function is defined, then `Module:init_per_suite/1` must also be defined.

# end_per_testcase(TestCase, Config)

(optional)

    -callback end_per_testcase(TestCase :: ct_testname(), Config :: ct_config()) ->
                                  term() | {fail, Reason :: term()} | {save_config, SaveConfig :: ct_config()}.

This function is called after each test case, and can be used to clean up after `Module:init_per_testcase/2` and the test case.

Any return value (besides `{fail, Reason}` and `{save_config, SaveConfig}`) is ignored. By returning `{fail, Reason}`, `TestCase` is marked as faulty (even though it was successful in the sense that it returned a value instead of terminating).

For information on `save_config`, see section Saving Configuration Data in the User's Guide.

If this function is defined, then `Module:init_per_testcase/2` must also be defined.

# group(GroupName)

(since OTP R15B) (optional)

    -callback group(GroupName :: ct_groupname()) -> [Info :: ct_info()].

The test case group information function. It is supposed to return a list of tagged tuples that specify various properties related to the execution of a test case group (that is, its test cases and subgroups). Properties set by `Module:group/1` override properties with the same key that have been set previously by `Module:suite/0`.

Tag `timetrap` sets the maximum time that each test case is allowed to execute (including `Module:init_per_testcase/2` and `Module:end_per_testcase/2`). If the timetrap time is exceeded, the test case fails with reason `timetrap_timeout`. A `TimeFunc` function can be used to set a new timetrap by returning a `TimeVal`. It can also be used to trigger a timetrap time-out by, at some point, returning a value other than a `TimeVal`. For details, see section Timetrap Time-Outs in the User's Guide.

Tag `require` specifies configuration variables required by test cases (or configuration functions) in the suite. If the required configuration variables are not found in any of the configuration files, all test cases in this group are skipped. For details about the `require` functionality, see function `ct:require/1,2`.

With `userdata`, the user can specify any test case group related information that can be read by calling `ct:userdata/2`.

Tag `ct_hooks` specifies the Common Test Hooks to be run with this suite.

Other tuples than the ones defined are ignored.

For details about the test case group information function, see section Group Information Function in the User's Guide.

# groups()

(optional)

    -callback groups() -> [GroupDef :: ct_group_def()].

Defines test case groups. For details, see section Test Case Groups in the User's Guide.

# init_per_group(GroupName, Config)

(optional)

    -callback init_per_group(GroupName :: ct_groupname(), Config :: ct_config()) ->
                                NewConfig :: ct_config() | {skip, Reason :: term()}.

This configuration function is called before execution of a test case group. It typically contains initializations that are common for all test cases and subgroups in the group, and that must only be performed once.

`GroupName` is the name of the group, as specified in the group definition (see `Module:groups/0`). Parameter `Config` is the configuration data that can be modified. The return value of this function is given as `Config` to all test cases and subgroups in the group.

If `{skip, Reason}` is returned, all test cases in the group are skipped and `Reason` is printed in the overview log for the group.

For information about test case groups, see section Test Case Groups in the User's Guide.

If this function is defined, then `Module:end_per_group/2` must also be defined.

# init_per_suite(Config)

(optional)

    -callback init_per_suite(Config :: ct_config()) ->
                                NewConfig ::
                                    ct_config() |
                                    {skip, Reason :: term()} |
                                    {skip_and_save, Reason :: term(), SaveConfig :: ct_config()}.

This configuration function is called as the first function in the suite. It typically contains initializations that are common for all test cases in the suite, and that must only be done once.

Parameter `Config` is the configuration data that can be modified. Whatever is returned from this function is specified as `Config` to all configuration functions and test cases in the suite.

If `{skip, Reason}` is returned, all test cases in the suite are skipped and `Reason` is printed in the overview log for the suite.

For information on `save_config` and `skip_and_save`, see section Saving Configuration Data in the User's Guide.

If this function is defined, then `Module:end_per_suite/1` must also be defined.

# init_per_testcase(TestCase, Config)

(optional)

    -callback init_per_testcase(TestCase :: ct_testname(), Config :: ct_config()) ->
                                   NewConfig ::
                                       ct_config() | {fail, Reason :: term()} | {skip, Reason :: term()}.

This function is called before each test case.

Argument `TestCase` is the test case name, and `Config` (list of key-value tuples) is the configuration data that can be modified. The `NewConfig` list returned from this function is given as `Config` to the test case.

If `{fail, Reason}` is returned, the test case is marked as failed without being executed.

If `{skip, Reason}` is returned, the test case is skipped and `Reason` is printed in the overview log for the suite.

If this function is defined, then `Module:end_per_testcase/2` must also be defined.

# suite()

(optional)

    -callback suite() -> [Info :: ct_info()].

The test suite information function. Returns a list of tagged tuples specifying various properties related to the execution of this test suite (common for all test cases in the suite).

Tag `timetrap` sets the maximum time that each test case is allowed to execute (including `Module:init_per_testcase/2` and `Module:end_per_testcase/2`). If the timetrap time is exceeded, the test case fails with reason `timetrap_timeout`. A `TimeFunc` function can be used to set a new timetrap by returning a `TimeVal`. It can also be used to trigger a timetrap time-out by, at some point, returning a value other than a `TimeVal`. For details, see section Timetrap Time-Outs in the User's Guide.

Tag `require` specifies configuration variables required by test cases (or configuration functions) in the suite. If the required configuration variables are not found in any of the configuration files, all test cases are skipped. For details about the `require` functionality, see function `ct:require/1,2`.

With `userdata`, the user can specify any test suite-related information, which can be read by calling `ct:userdata/2`.

Tag `ct_hooks` specifies the Common Test Hooks to be run with this suite.

Other tuples than the ones defined are ignored.

For details about the test suite information function, see section Test Suite Information Function in the User's Guide.

# 'Testcase'()

(since OTP R14B) (optional)

    -callback 'Testcase'() -> [ct_info()].

The test case information function. It is supposed to return a list of tagged tuples that specify various properties related to the execution of this particular test case. Properties set by `Module:Testcase/0` override properties set previously for the test case by `Module:group/1` or `Module:suite/0`.

Tag `timetrap` sets the maximum time that the test case is allowed to execute. If the timetrap time is exceeded, the test case fails with reason `timetrap_timeout`. `Module:init_per_testcase/2` and `Module:end_per_testcase/2` are included in the timetrap time. A `TimeFunc` function can be used to set a new timetrap by returning a `TimeVal`. It can also be used to trigger a timetrap time-out by, at some point, returning a value other than a `TimeVal`. For details, see section Timetrap Time-Outs in the User's Guide.

Tag `require` specifies configuration variables that are required by the test case (or `init_per_testcase/2` or `end_per_testcase/2`). If the required configuration variables are not found in any of the configuration files, the test case is skipped. For details about the `require` functionality, see function `ct:require/1,2`.

If `timetrap` or `require` is not set, the default values specified by `Module:suite/0` (or `Module:group/1`) are used.

With `userdata`, the user can specify any test case-related information that can be read by calling `ct:userdata/3`.

Other tuples than the ones defined are ignored.

For details about the test case information function, see section Test Case Information Function in the User's Guide.

# 'Testcase'(Config)

(since OTP R14B) (optional)

    -callback 'Testcase'(Config) ->
                            term() |
                            {skip, Reason} |
                            {fail, Reason} |
                            {comment, Comment} |
                            {save_config, SaveConfig} |
                            {skip_and_save, Reason, SaveConfig}
                            when
                                Config :: ct_config(),
                                SaveConfig :: ct_config(),
                                Reason :: term(),
                                Comment :: string().

The implementation of a test case. Call the functions to test and check the result. If something fails, ensure the function causes a runtime error or call `ct:fail/1,2` (which also causes the test case process to terminate).

Elements from the `Config` list can, for example, be read with `proplists:get_value/2` in STDLIB.

Possible return values are:

- `{fail, Reason}` - The test case is considered failed, and the `Reason` will be logged.

- `{skip, Reason}` - The test case is considered skipped, and the `Reason` will be logged.

- `{comment, Comment}` - The test case is considered successful, and the `Comment` will be logged.

- `{save_config, SaveConfig}` - The test case is considered successful, and the `SaveConfig` will be stored in the `Config` (see section Saving Configuration Data in the User's Guide).

- `{skip_and_save, Reason, SaveConfig}` - The test case is considered skipped, the `Reason` will be logged, and the `SaveConfig` will be stored in the `Config` (see section Saving Configuration Data in the User's Guide).

If the function returns any other term, the test case is considered successful.

If the test case function crashes, it is considered failed.

For details about test case implementation, see section Test Cases in the User's Guide.