[riot-notifications] [RIOT-OS/RIOT] makefiles/murdock.inc.mk: change policy to run tests by default (#11680)

Gaƫtan Harter notifications at github.com
Wed Jun 12 15:43:27 CEST 2019


### Contribution description

Change the policy to run the tests by default on murdock.

Introduce `TEST_ON_CI_BLACKLIST` for describing tests that must not be
run in CI.

By default if a test is available and not blacklisted it will be run by
murdock.

This will move from a whitelisting everything that works, to
blacklisting what fails. This way, failing tests will be easier to
track instead of being silently just not run.

It could even allow introducing failing tests before a fix is available.

### TODO

* [ ] This pull request will require blacklisting all what fails in a commit before changing the policy.
* [ ] Update the remaining `TEST_ON_CI_WHITELIST` with the reason.
    ```
    git grep 'TEST_ON_CI_WHITELIST' '*' ':!makefiles/murdock.inc.mk'
    tests/gnrc_ipv6_ext/Makefile:# TEST_ON_CI_WHITELIST += all
    tests/gnrc_rpl_srh/Makefile:# TEST_ON_CI_WHITELIST += all
    tests/gnrc_sock_dns/Makefile:# TEST_ON_CI_WHITELIST += all
    tests/libfixmath_unittests/Makefile:TEST_ON_CI_WHITELIST += native
    tests/pkg_libcose/Makefile:TEST_ON_CI_WHITELIST += native
    tests/pkg_tweetnacl/Makefile:TEST_ON_CI_WHITELIST += native
    tests/rng/Makefile:TEST_ON_CI_WHITELIST += native
    tests/xtimer_now64_continuity/Makefile:TEST_ON_CI_WHITELIST += samr21-xpro
    tests/xtimer_usleep/Makefile:TEST_ON_CI_WHITELIST += samr21-xpro
    ```

### Testing procedure

#### New behavior

If there is a test and nothing is set the board is enabled

```
BOARD=samr21-xpro make --no-print-directory -C tests/pkg_fatfs_vfs/ test/available test-on-ci-enabled 
```
##### Blacklisting

If all is blacklisted, no test will be run

```
BOARD=native make --no-print-directory -C tests/pkg_fatfs_vfs/ test/available test-on-ci-enabled 
/home/harter/work/git/RIOT/makefiles/murdock.inc.mk:47: recipe for target 'test-on-ci-enabled' failed
make: *** [test-on-ci-enabled] Error 1
BOARD=samr21-xpro make --no-print-directory -C tests/pkg_fatfs_vfs/ test/available test-on-ci-enabled 
/home/harter/work/git/RIOT/makefiles/murdock.inc.mk:47: recipe for target 'test-on-ci-enabled' failed
make: *** [test-on-ci-enabled] Error 1
```

``` diff
diff --git a/tests/pkg_fatfs_vfs/Makefile b/tests/pkg_fatfs_vfs/Makefile
index edbf14bf4..babd1db3c 100644
--- a/tests/pkg_fatfs_vfs/Makefile
+++ b/tests/pkg_fatfs_vfs/Makefile
@@ -50,6 +50,8 @@ BOARD_WHITELIST := airfy-beacon arduino-due arduino-duemilanove \
 
 TEST_DEPS += image
 
+TEST_ON_CI_BLACKLIST += all
+
 include $(RIOTBASE)/Makefile.include
 
 image:
```

If one board is blacklisted only this one will not run

```
BOARD=native make --no-print-directory -C tests/pkg_fatfs_vfs/ test/available test-on-ci-enabled 
BOARD=samr21-xpro make --no-print-directory -C tests/pkg_fatfs_vfs/ test/available test-on-ci-enabled 
/home/harter/work/git/RIOT/makefiles/murdock.inc.mk:47: recipe for target 'test-on-ci-enabled' failed
make: *** [test-on-ci-enabled] Error 1
```

with this diff

``` diff
diff --git a/tests/pkg_fatfs_vfs/Makefile b/tests/pkg_fatfs_vfs/Makefile
index edbf14bf4..d44376293 100644
--- a/tests/pkg_fatfs_vfs/Makefile
+++ b/tests/pkg_fatfs_vfs/Makefile
@@ -50,6 +50,8 @@ BOARD_WHITELIST := airfy-beacon arduino-due arduino-duemilanove \
 
 TEST_DEPS += image
 
+TEST_ON_CI_BLACKLIST += samr21-xpro
+
 include $(RIOTBASE)/Makefile.include
 
 image:
```
#### Kept behavior when `TEST_ON_CI_WHITELIST` is set

The previous behavior when `WHITELIST` is set did not change.

Enabled tests are still enabled:

```
# Only native enabled
BOARD=native make --no-print-directory -C tests/libfixmath_unittests/ test-on-ci-enabled
# all enabled
BOARD=samr21-xpro make --no-print-directory -C tests/bloom_bytes/ test-on-ci-enabled
BOARD=iotlab-m3 make --no-print-directory -C tests/bloom_bytes/ test-on-ci-enabled 
```

Not enabled tests are still not enabled

```
BOARD=samr21-xpro make --no-print-directory -C tests/libfixmath_unittests/ test-on-ci-enabled 
/home/harter/work/git/RIOT/makefiles/murdock.inc.mk:47: recipe for target 'test-on-ci-enabled' failed
make: *** [test-on-ci-enabled] Error 1
```

If a board does not have enough memory, it is still disabled

```
RIOT_CI_BUILD=1 BOARD=samr21-xpro make --no-print-directory -C tests/unittests/ test-on-ci-enabled 
CI-build: skipping link step
/home/harter/work/git/RIOT/makefiles/murdock.inc.mk:47: recipe for target 'test-on-ci-enabled' failed
make: *** [test-on-ci-enabled] Error 1
```

### Issues/PRs references

https://github.com/RIOT-OS/RIOT/issues/11128#issuecomment-470555097
Some tests always fail on some boards, even some supported for testing in CI https://github.com/RIOT-OS/Release-Specs/issues/98#issuecomment-460984572

You can view, comment on, or merge this pull request online at:

  https://github.com/RIOT-OS/RIOT/pull/11680

-- Commit Summary --

  * makefiles/murdock.inc.mk: introduce way of blacklisting tests in CI
  * makefiles/murdock.inc.mk: change policy to run tests by default
  * applications: remove 'TEST_ON_CI_WHITELIST += all'

-- File Changes --

    M makefiles/murdock.inc.mk (35)
    M tests/bench_msg_pingpong/Makefile (2)
    M tests/bench_mutex_pingpong/Makefile (2)
    M tests/bench_runtime_coreapis/Makefile (2)
    M tests/bench_sched_nop/Makefile (2)
    M tests/bench_sizeof_coretypes/Makefile (2)
    M tests/bench_thread_flags_pingpong/Makefile (2)
    M tests/bench_thread_yield_pingpong/Makefile (2)
    M tests/bitarithm_timings/Makefile (2)
    M tests/bloom_bytes/Makefile (2)
    M tests/buttons/Makefile (2)
    M tests/cb_mux/Makefile (2)
    M tests/cb_mux_bench/Makefile (2)
    M tests/cpp11_condition_variable/Makefile (2)
    M tests/cpp11_mutex/Makefile (2)
    M tests/cpp11_thread/Makefile (2)
    M tests/embunit/Makefile (2)
    M tests/event_wait_timeout/Makefile (2)
    M tests/events/Makefile (2)
    M tests/evtimer_msg/Makefile (2)
    M tests/evtimer_underflow/Makefile (2)
    M tests/float/Makefile (2)
    M tests/fmt_print/Makefile (2)
    M tests/gnrc_ipv6_nib/Makefile (2)
    M tests/gnrc_ipv6_nib_6ln/Makefile (2)
    M tests/gnrc_ndp/Makefile (2)
    M tests/gnrc_netif/Makefile (2)
    M tests/gnrc_sixlowpan/Makefile (2)
    M tests/gnrc_sixlowpan_frag/Makefile (2)
    M tests/gnrc_sock_ip/Makefile (2)
    M tests/gnrc_sock_udp/Makefile (2)
    M tests/irq/Makefile (2)
    M tests/isr_yield_higher/Makefile (2)
    M tests/l2util/Makefile (2)
    M tests/libc_newlib/Makefile (1)
    M tests/libfixmath/Makefile (2)
    M tests/lwip_sock_ip/Makefile (2)
    M tests/msg_avail/Makefile (2)
    M tests/msg_send_receive/Makefile (2)
    M tests/msg_try_receive/Makefile (2)
    M tests/mutex_order/Makefile (3)
    M tests/mutex_unlock_and_sleep/Makefile (2)
    M tests/netdev_test/Makefile (2)
    M tests/nhdp/Makefile (2)
    M tests/od/Makefile (2)
    M tests/periph_hwrng/Makefile (2)
    M tests/periph_timer/Makefile (2)
    M tests/pipe/Makefile (2)
    M tests/pkg_c25519/Makefile (2)
    M tests/pkg_cayenne-lpp/Makefile (2)
    M tests/pkg_cmsis-dsp/Makefile (1)
    M tests/pkg_cn-cbor/Makefile (2)
    M tests/pkg_hacl/Makefile (2)
    M tests/pkg_heatshrink/Makefile (2)
    M tests/pkg_jsmn/Makefile (2)
    M tests/pkg_libb2/Makefile (2)
    M tests/pkg_libcoap/Makefile (2)
    M tests/pkg_libhydrogen/Makefile (2)
    M tests/pkg_littlefs/Makefile (2)
    M tests/pkg_lora-serialization/Makefile (1)
    M tests/pkg_micro-ecc-with-hwrng/Makefile (2)
    M tests/pkg_micro-ecc/Makefile (2)
    M tests/pkg_minmea/Makefile (2)
    M tests/pkg_monocypher/Makefile (1)
    M tests/pkg_qdsa/Makefile (2)
    M tests/pkg_relic/Makefile (2)
    M tests/pkg_spiffs/Makefile (2)
    M tests/pkg_tiny-asn1/Makefile (2)
    M tests/pkg_tinycbor/Makefile (1)
    M tests/pkg_tinycrypt/Makefile (2)
    M tests/pkg_ubasic/Makefile (2)
    M tests/pkg_umorse/Makefile (2)
    M tests/posix_semaphore/Makefile (2)
    M tests/posix_time/Makefile (2)
    M tests/ps_schedstatistics/Makefile (2)
    M tests/pthread/Makefile (2)
    M tests/pthread_barrier/Makefile (2)
    M tests/pthread_cleanup/Makefile (2)
    M tests/pthread_condition_variable/Makefile (2)
    M tests/pthread_cooperation/Makefile (2)
    M tests/pthread_tls/Makefile (2)
    M tests/riotboot/Makefile (2)
    M tests/rmutex/Makefile (2)
    M tests/sched_testing/Makefile (2)
    M tests/shell/Makefile (2)
    M tests/ssp/Makefile (2)
    M tests/struct_tm_utility/Makefile (2)
    M tests/thread_basic/Makefile (2)
    M tests/thread_cooperation/Makefile (2)
    M tests/thread_exit/Makefile (2)
    M tests/thread_flags/Makefile (2)
    M tests/thread_flags_xtimer/Makefile (2)
    M tests/thread_flood/Makefile (2)
    M tests/thread_msg/Makefile (2)
    M tests/thread_msg_block_race/Makefile (2)
    M tests/thread_msg_block_w_queue/Makefile (2)
    M tests/thread_msg_block_wo_queue/Makefile (2)
    M tests/thread_race/Makefile (2)
    M tests/trace/Makefile (2)
    M tests/trickle/Makefile (2)
    M tests/unittests/Makefile (3)
    M tests/warn_conflict/Makefile (2)
    M tests/xtimer_hang/Makefile (2)
    M tests/xtimer_msg/Makefile (2)
    M tests/xtimer_msg_receive_timeout/Makefile (2)
    M tests/xtimer_periodic_wakeup/Makefile (2)
    M tests/xtimer_remove/Makefile (2)
    M tests/xtimer_reset/Makefile (2)
    M tests/xtimer_usleep_short/Makefile (2)

-- Patch Links --

https://github.com/RIOT-OS/RIOT/pull/11680.patch
https://github.com/RIOT-OS/RIOT/pull/11680.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/RIOT-OS/RIOT/pull/11680
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.riot-os.org/pipermail/notifications/attachments/20190612/8b345d39/attachment-0001.html>


More information about the notifications mailing list