Testing is an important part of developing mbed devices - and we have made it fast and easy. We created ARM® mbed™ Greentea (Generic Regression Environment for Test Automation) for board-agnostic automated testing of mbed Enabled devices. Greentea automates test firmware flashing, mbed device reset, test execution, test result collection and result presentation in human readable form for review and sign-off.
Greentea uses a duplex serial port connection between the host (your computer) and the mbed device. Tests run on the device under the supervision of host test scripts provided by Greentea without any user interaction. The scripts generate a series of prints and sscanf calls, which are displayed in human-readable format. Test results are simple to understand and evaluated as pass or fail. Testers only need to initiate the process and review the results.
Greentea uses two tools, which are distributed as Python scripts:
- mbed-htrun: the engine behind Greentea; it monitors the test execution and controls test workflow.
- mbed-ls: detects and lists mbed-enabled devices and their configuration parameters (such as their ports).
Please note that Greentea requires yotta. For more information, see the yotta documentation.
Why was the Greentea testing framework needed?
Automated testing enables rapid development of working prototypes. Manual testing is time consuming, inefficient and error-prone when performing multiple firmware updates on multiple mbed devices. Greentea automates the creation of tests suits for mbed enabled devices. It allows you to concentrate on test development rather than test framework building. It also allows you to reuse existing host tests or create new host tests for new functional and interoperability domains.
How does the Greentea framework work?
Greentea is a test coordinator – it calls different tools in sequence with your configuration parameters, runs the tests and displays the results. These tools include yotta, which builds the tests as binaries; mbed-ls, which identifies all connected boards; and mbed-htrun, which runs and supervises the tests and then returns results to Greentea. All you need to do is write the tests and point Greentea to your test targets.
To use the Greentea framework:
- Write your tests, in C or C++. These tests should include both the test procedure and the definition of test results
- If needed, write host tests in Python.
- Call Greentea and point to where your tests are (for information on the directory structure, check the documentation on the Greentea repository).
- Greentea gives yotta building instructions: the directory and targets you specified in the previous test. yotta generates a binary for each test.
- Greentea uses mbed-ls to discover boards connected to the host computer (your server or PC).
- For every target identified in the previous step, an mbed-htrun script flashes one test binary at a time and resets the device.
- The test runs automatically after the device reset. An mbed-htrun script generates a test report (as text or JSON file).
- Greentea displays the results in a human-readable format. It can also display a graph comparing the results of different test runs.
- The test framework only supports mbed Enabled devices.
- Distributed resource allocation (DRA) is not currently supplied as part of Greentea. Please use a stand-alone tool such as Jenkins. We will add DRA support in the future.
- The test framework works only with software packaged with yotta. yotta must have the tests in the location Greentea expects them to be. For instructions, see here.
Support for third-party testing tools
You can use third-party testing tools, for example CppUTest. These tools are listed in a directory of dependencies for yotta’s build process. More information on third-party testing tools can be found in the quick start guide.
You can also use Greentea to run a sample application on the target board. Greentea can display device output on your host console, so that you can monitor your application.
mbed-htrun is a script that runs on the host computer. It runs tests on the boards by flashing the test binaries to the board and then resetting it (the test runs automatically when the board is reset). It then monitors the board for a result, and sends that result to Greentea. You can also use mbed-htrun to evaluate test results or mock different scenarios.
You can use mbed-htrun to:
- Check the validity of values returned from a sensor.
- Determine success or failure for simple API calls.
- Act as a TCP server that responds to the board's queries and judges packet loss.
When you run Greentea, it calls mbed-htrun automatically. mbed-htrun then runs the Python scripts that it finds in the yotta test directory on the PC, at the same time that the test binary runs on the board.
Greentea uses mbed-ls to discover targets on which to run tests. It compares the connected devices to the list of targets you supplied and determines which targets need to be tested. It eliminates the manual entry of a mount point list every time you run your tests, and the need to unplug incompatible boards.
mbed-ls works on Windows 7, Ubuntu and Max OS X (Darwin).