utest testing framework

utest Introduction

utest (unit test) is a unit test framework developed by RT-Thread. The original intention of designing utest is to facilitate RT-Thread developers to use a unified framework interface to write test programs to achieve the purpose of unit testing, coverage testing and integration testing.

Test case definition

A test case (TC) is a single test performed to achieve a specific test objective. It is a specification that includes test input, execution conditions, test process, and expected results. It is a finite loop with clear end conditions and clear test results.

The utest (unit test) testing framework defines the test programs written by users as test cases . A test case contains only one testcase function (similar to the main function) and can contain multiple test unit functions.

Specifically, the test code for a certain function completed through the API provided by the utest testing framework is a test case.

Test unit definition

A test unit is a test point after the tested function is subdivided. Each test point can be the smallest measurable unit of the tested function. Of course, different classification methods will result in different test units.

utest application diagram

utest application diagram

As shown in the figure above, the test case is designed based on the service interface provided by the test framework utest, which supports compiling multiple test cases together for testing. In addition, as can be seen from the figure, a test case corresponds to a unique testcase function, which contains multiple test units.

utest API

In order to achieve a unified format of test case code, the test framework utest provides a set of common API interfaces for test case writing.

assert macro

注意：
这里的 assert 仅记录通过和失败的数量，不会产生断言并终止程序运行。其功能不等同于 RT_ASSERT。copymistakeCopy Success

assert macro	illustrate
uassert_true(value)	If value is true, the test passes, otherwise the test fails
uassert_false(value)	If value is false, the test passes, otherwise the test fails
uassert_null(value)	If value is null, the test passes, otherwise the test fails
uassert_not_null(value)	If value is non-null, the test passes, otherwise the test fails
uassert_int_equal(a, b)	If the values ​​of a and b are equal, the test passes, otherwise the test fails
uassert_int_not_equal(a, b)	If the values ​​of a and b are not equal, the test passes, otherwise the test fails
uassert_str_equal(a, b)	If string a and string b are the same, the test passes, otherwise the test fails
uassert_str_not_equal(a, b)	If string a and string b are not the same, the test passes, otherwise the test fails
uassert_in_range(value, min, max)	If the value is within the range of min and max, the test passes, otherwise the test fails
uassert_not_in_range(value, min, max)	If the value is not within the range of min and max, the test passes, otherwise the test fails

Test unit run macro

UTEST_UNIT_RUN(test_unit_func)copymistakeCopy Success

In a test case, use UTEST_UNIT_RUNthe macro to execute the specified test unit function test_unit_func. The test unit must UTEST_UNIT_RUNbe executed using the macro.

Test case export macro

UTEST_TC_EXPORT(testcase, name, init, cleanup, timeout)copymistakeCopy Success

parameter	describe
testcase	Test case main carrier function ( specified to use a function named static void testcase(void)
name	Test case name (unique). Specifies the naming format testcases 目录to use relative paths .to connect to the test case.
init	Initialization function before the test case starts
cleanup	Cleanup function after the test case ends
timeout	The estimated test time required for the test case (in seconds)

Test case naming requirements:

Test cases need to be named in the specified format. The naming format is specified to use testcases 目录the relative path of the current test case for .connection, and the name contains the file name of the current test case file (excluding the suffix).

Test case naming example:

Assuming that there is a test case file in the test case testcases directory testcases\components\filesystem\dfs\dfs_api_tc.c, the name of the test case in dfs_api_tc.c is named components.filesystem.dfs.dfs_api_tc.

Test case LOG output interface

The utest test framework relies on the ulog log module for log output, and the utest test framework has already set the log output level. Therefore, you only need to add it to the test case #include "utest.h"to use all the level interfaces (LOG_D/LOG_I/LOG_E) of the ulog log module.

In addition, the utest test framework adds additional log control interfaces as follows:

#define UTEST_LOG_ALL    (1u)
#define UTEST_LOG_ASSERT (2u)

void utest_log_lv_set(rt_uint8_t lv);copymistakeCopy Success

Users can use the interface to control the log output level in the test case utest_log_lv_set. UTEST_LOG_ALLConfigure to output all logs, or UTEST_LOG_ASSERTconfigure to output only the logs after uassert fails.

Configuration Enable

To use the utest test framework, you need to use menuconfig in the ENV tool to configure as follows:

RT-Thread Kernel  --->
    Kernel Device Object  --->
        (256) the buffer size for console log printf /* utest 日志需要的最小 buffer */
RT-Thread Components  --->
    Utilities  --->
        -*- Enable utest (RT-Thread test framework) /* 使能 utest 测试框架 */
        (4096) The utest thread stack size          /* 设置 utest 线程堆栈（-thread 模式需要） */
        (20)   The utest thread priority            /* 设置 utest 线程优先级（-thread 模式需要） */copymistakeCopy Success

Application Paradigm

The utest test framework and related APIs were introduced earlier. Here we introduce the basic test case code structure.

The required code blocks of the test case file are as follows:

/*
 * Copyright (c) 2006-2019, RT-Thread Development Team
 *
 * SPDX-License-Identifier: Apache-2.0
 *
 * Change Logs:
 * Date           Author       Notes
 * 2019-01-16     MurphyZhao   the first version
 */

#include <rtthread.h>
#include "utest.h"

static void test_xxx(void)
{
    uassert_true(1);
}

static rt_err_t utest_tc_init(void)
{
    return RT_EOK;
}

static rt_err_t utest_tc_cleanup(void)
{
    return RT_EOK;
}

static void testcase(void)
{
    UTEST_UNIT_RUN(test_xxx);
}
UTEST_TC_EXPORT(testcase, "components.utilities.utest.sample.sample_tc", utest_tc_init, utest_tc_cleanup, 10);copymistakeCopy Success

A basic test case must include the following:

• File Comment Header (Copyright)

  The test case file must contain a file comment header, including Copyrighttime, author, and description information.

• utest_tc_init(void)

  The initialization function before the test runs is generally used to initialize the environment required for the test.

• utest_tc_cleanup(void)

  The cleanup function after the test is completed is used to clean up the resources requested during the test (such as memory, threads, semaphores, etc.).

• testcase(void)

  Test body function: A test case implementation can only contain one testcase function (similar to the main function). Usually this function is only used to run the test unit execution function UTEST_UNIT_RUN.

  A testcase can contain multiple test units, each of which UTEST_UNIT_RUNis executed by .

• UTEST_UNIT_RUN

  Test unit execution function.

• test_xxx(void)

  Test implementation of each functional unit. Users determine the function name and function implementation based on their needs.

• uassert_true

  Assertion macro used to determine the test result (this assertion macro does not terminate the program). Test cases must use uassert_xxxmacros to determine the test result, otherwise the test framework will not know whether the test passes.

  The entire test case is considered to have passed only after all uassert_xxxmacros have passed.

• UTEST_TC_EXPORT

  Export the testcase function to the test framework.

Test case running requirements

The test framework utest exports all test cases to UtestTcTabthe code segment. In IAR and MDK compilers, there is no need to define UtestTcTabthe segment in the link script, but when compiling with GCC, the segment needs to be explicitly set in the link script UtestTcTab.

Therefore, in order for the test case to be compiled and run under GCC, the code segment must first be defined in the GCC link scriptUtestTcTab .

In the GCC linker script .text, add UtestTcTabthe definition of the segment, the format is as follows:

/* section information for utest */
. = ALIGN(4);
__rt_utest_tc_tab_start = .;
KEEP(*(UtestTcTab))
__rt_utest_tc_tab_end = .;copymistakeCopy Success

Running Test Cases

The test framework provides the following commands to facilitate users to run test cases in the RT-Thread MSH command line. The commands are as follows:

utest_list command

Lists the test cases supported by the current system, including the name of the test case and the time required for the test. This command has no parameters.

utest_run command

Test case execution command, the command format is as follows:

utest_run [-thread or -help] [testcase name] [loop num]copymistakeCopy Success

utest_run command parameters	describe
-thread	Running the test framework in threaded mode
-help	Print help information
testcase name	Specify the test case name. Supports the use of wildcards *and the first part of the test case name
loop num	Specify the number of test case loops

Test command usage example:

msh />utest_list
[14875] I/utest: Commands list :
[14879] I/utest: [testcase name]:components.filesystem.dfs.dfs_api_tc; [run timeout]:30
[14889] I/utest: [testcase name]:components.filesystem.posix.posix_api_tc; [run timeout]:30
[14899] I/utest: [testcase name]:packages.iot.netutils.iperf.iperf_tc; [run timeout]:30
msh />
msh />utest_run components.filesystem.dfs.dfs_api_tc
[83706] I/utest: [==========] [ utest    ] started
[83712] I/utest: [----------] [ testcase ] (components.filesystem.dfs.dfs_api_tc) started
[83721] I/testcase: in testcase func...
[84615] D/utest: [    OK    ] [ unit     ] (test_mkfs:26) is passed
[84624] D/testcase: dfs mount rst: 0
[84628] D/utest: [    OK    ] [ unit     ] (test_dfs_mount:35) is passed
[84639] D/utest: [    OK    ] [ unit     ] (test_dfs_open:40) is passed
[84762] D/utest: [    OK    ] [ unit     ] (test_dfs_write:74) is passed
[84770] D/utest: [    OK    ] [ unit     ] (test_dfs_read:113) is passed
[85116] D/utest: [    OK    ] [ unit     ] (test_dfs_close:118) is passed
[85123] I/utest: [  PASSED  ] [ result   ] testcase (components.filesystem.dfs.dfs_api_tc)
[85133] I/utest: [----------] [ testcase ] (components.filesystem.dfs.dfs_api_tc) finished
[85143] I/utest: [==========] [ utest    ] finished
msh />copymistakeCopy Success

Test results analysis

utest log display

As shown in the figure above, the test case running log is divided into four columns from left to right, namely log header information, result column, attribute column, and detailed information display column. resultThe attribute is used in the log to identify the test result (PASSED or FAILED) of the test case.

Test case running process

Test case running process

From the above flowchart, we can get the following:

• The utest test framework executes all test units in the testcase function sequentially.

• If an assert occurs in the last UTEST_UNIT_RUN macro, all subsequent UTEST_UNIT_RUN macros will be skipped.

Precautions

Before compiling with GCC, make sure that the link script adds UtestTcTaba segment - Before compiling, make sure that RT-Thread Kernel -> Kernel Device Object -> (256) the buffer size for console log printfis at least 256 bytes - The resources (threads, semaphores, timers, memory, etc.) created in the test case need to be released before the end of the test - A test case implementation can only use UTEST_TC_EXPORTexport a test body function (testcase function) - Write a README.md document for the written test case program to guide users to configure the test environment