Unit Tests Proposal

Proposal !
Topic: Unit Tests

Proposal type: Hardware [ ] , Software [ ] , Other [ ] : _________________
Mostly software, but possibly might need additional testing server like the autotest or coverity server once implemented. Python has some excellent unittest libraries, and since we use waf for building( which is python ), it would probably be best that the unit test system be python based. Many unit-tests themselves will be compiled code, so they will clearly be C/C++, but the wrapper-tools around this will be python.

Description: Problem: The autotest server currently tests the “well worn” code paths that are used for flying an aircraft, and does an excellent job of that, but the “corner cases” and other code paths are unfortunately not tested so well. Solution: The unit tests proposal aims to fix this by building up a library of code path tests that should “always work”, but aren’t validated well. Initially we can test that individual libraries give their expected output/s when given expected inputs, and/or we can choose to test higher-level functionality if desired.

Details / Plan for implementation: It’s hoped that we will be able to implement a new Unit Testing “system”, and integrate a fixed number of tests ( say 20?) of varying complexity that give a representation of what is possible with the system, and create a base on which others can build additional tests.

Planned amount: estimated $2000

Completion metrics:

  • a general unit-testing framework
  • ability to automatically execute the test suite.
  • 10 new unit tests covering 2 subsystems ( probably based on existing “example” code that some libraries have being made to execute in a repeatable / testable method ) - integration of at least 5 existing unit tests (such as in AP_Math/tests)
  • enough developer documentation that other developers can extend the framework easily.

Estimated time for completion: 2 months part-time

Status: DISCUSSION

1 Like

An input to this proposal is that with waf we already support Google Test and already have some tests for that - for example, https://github.com/ArduPilot/ardupilot/blob/master/libraries/AP_GPS/tests/test_gps.cpp

What really is needed is to add much more tests.

more tests is definitely part of the proposal, but also increasing developer exposure to the testing, and hopefully having the ability to run these test automatically ( either as part of the standard build, or otherwise ).

Personally, I was thinking of something like one of these:
https://pypi.python.org/pypi/autotest/0.16.2
http://doc.pytest.org/en/latest/
but using something like https://github.com/google/googletest , particularly for the C/C++ parts seems to make sense.

We already run the tests for generic Linux board in Travis/Semaphore and anyone can run them with waf. We need to improve/add support for other boards though.
Our flight code is C++, so I think it makes sense to use unit testing for that.

I like this proposal, but I’d note that we really need to also use gcov (or equivalent) to get us test coverage information, so we know which parts of the code aren’t yet covered by tests. I think we could extend our current autotest.ardupilot.org output to include coverage graphs

Thanks for the comment/s Tridge.
Despite this being opened in Oct '16, I still think this Proposal holds merit. Yes, it’s about improving the coverage of code being validated/tested/executed across the entire stack, especially where individual libraries don’t have their own unit tests, and/or where the high-level autotest suite doesn’t cover it already, or only cover/s one path.

I don’t know what I need to do to take this proposal to the next level…? Please funding committee, can u approve it.? :slight_smile:

the new committee is actively looking at it - they asked me about it today, which is why I commented :slight_smile:

Thanks Buzz.

We are actively looking at ALL outstanding proposals, and getting advice from whoever might be the appropriate member on the specific topic.

It is our intention to have every existing proposal with a clear answer as soon as possible, so that every new incoming proposal gets answered within a very short timeframe, as stated on the new charter for the funding committee.

brgds

Luis

just a quick update: I have a WIP that does ‘gcov’ code analysis here: https://github.com/davidbuzz/ardupilot/tree/GCOV_buzz , which is one of the things that will make this proposal possible moving forward.