[riot-notifications] [RIOT-OS/RIOT] tests: increase timeout for nrf51dk (#15880)

Alexandre Abadie notifications at github.com
Thu Jan 28 13:55:14 CET 2021


<!--
The RIOT community cares a lot about code quality.
Therefore, before describing what your contribution is about, we would like
you to make sure that your modifications are compliant with the RIOT
coding conventions, see https://github.com/RIOT-OS/RIOT/wiki/Coding-conventions.
-->

### Contribution description

This PR increases the timeout value of some of the test scripts: `tests/pkg_libcose`, `tests/pkg_wolfcrypt-ed25519-verify`, `tests/pkg_wolfssl` and `tests/suit_manifest`. They all succeed but require quite some time on nrf51dk which is quite slow.

These applications are failing in the [test-on-iotlab action](https://github.com/RIOT-OS/RIOT/runs/1759015809?check_suite_focus=true) because of a timeout. There are other tests failing in test-on-iot-lab but some should now be fixed and other are working when run manually., let's wait for the next run (next sunday).

I'm not sure if I should set the `CI: Ready for CI build` label, since there shouldn't be any build failures with this PR and I could run the tests locally without issues. I let the reviewer decide :)

<!--
Put here the description of your contribution:
- describe which part(s) of RIOT is (are) involved
- if it's a bug fix, describe the bug that it solves and how it is solved
- you can also give more information to reviewers about how to test your changes
-->


### Testing procedure

Verify that the test target is now working (this can be tested on one of nrf51dk available on iot-lab):

<details><summary>tests/pkg_libcose</summary>

```
$ make BOARD=nrf51dk -C tests/pkg_libcose flash test IOTLAB_NODE=auto-ssh
make: Entering directory '/work/riot/RIOT/tests/pkg_libcose'
Building application "tests_pkg_libcose" for "nrf51dk" with MCU "nrf51".

"make" -C /work/riot/RIOT/pkg/hacl
"make" -C /work/riot/RIOT/build/pkg/hacl -f /work/riot/RIOT/Makefile.base
"make" -C /work/riot/RIOT/pkg/libcose
"make" -C /work/riot/RIOT/build/pkg/libcose/src -f /work/riot/RIOT/Makefile.base MODULE=libcose
"make" -C /work/riot/RIOT/build/pkg/libcose/src/crypt -f /work/riot/RIOT/pkg/libcose/Makefile.libcose_crypt
"make" -C /work/riot/RIOT/pkg/nanocbor
"make" -C /work/riot/RIOT/build/pkg/nanocbor/src -f /work/riot/RIOT/Makefile.base MODULE=nanocbor
"make" -C /work/riot/RIOT/boards/nrf51dk
"make" -C /work/riot/RIOT/core
"make" -C /work/riot/RIOT/cpu/nrf51
"make" -C /work/riot/RIOT/cpu/cortexm_common
"make" -C /work/riot/RIOT/cpu/cortexm_common/periph
"make" -C /work/riot/RIOT/cpu/nrf51/periph
"make" -C /work/riot/RIOT/cpu/nrf5x_common
"make" -C /work/riot/RIOT/cpu/nrf5x_common/periph
"make" -C /work/riot/RIOT/drivers
"make" -C /work/riot/RIOT/drivers/periph_common
"make" -C /work/riot/RIOT/sys
"make" -C /work/riot/RIOT/sys/auto_init
"make" -C /work/riot/RIOT/sys/embunit
"make" -C /work/riot/RIOT/sys/isrpipe
"make" -C /work/riot/RIOT/sys/luid
"make" -C /work/riot/RIOT/sys/malloc_thread_safe
"make" -C /work/riot/RIOT/sys/memarray
"make" -C /work/riot/RIOT/sys/newlib_syscalls_default
"make" -C /work/riot/RIOT/sys/random
"make" -C /work/riot/RIOT/sys/random/tinymt32
"make" -C /work/riot/RIOT/sys/stdio_uart
"make" -C /work/riot/RIOT/sys/test_utils/interactive_sync
"make" -C /work/riot/RIOT/sys/tsrb
   text	   data	    bss	    dec	    hex	filename
  39976	    144	   9596	  49716	   c234	/work/riot/RIOT/tests/pkg_libcose/bin/nrf51dk/tests_pkg_libcose.elf
iotlab-node --jmespath='keys(@)[0]' --format='int'  --list saclay,nrf51dk,4 --flash /work/riot/RIOT/tests/pkg_libcose/bin/nrf51dk/tests_pkg_libcose.bin | grep 0
0
r
ssh -t abadie at saclay.iot-lab.info 'socat - tcp:nrf51dk-4.saclay.iot-lab.info:20000' 
r
+zkw'Help: Press s to start test, r to print it is ready
READY
READY
s
START
main(): This is RIOT! (Version: 2021.04-devel-346-g1d0dbb)
...
OK (3 tests)

make: Leaving directory '/work/riot/RIOT/tests/pkg_libcose'
```

</details>

<details><summary>tests/pkg_wolfcrypt-ed25519-verify</summary>

```
$ RIOT_CI_BUILD=1 make BOARD=nrf51dk -C tests/pkg_wolfcrypt-ed25519-verify/ test IOTLAB_NODE=auto-ssh
make: Entering directory '/work/riot/RIOT/tests/pkg_wolfcrypt-ed25519-verify'
r
ssh -t abadie at saclay.iot-lab.info 'socat - tcp:nrf51dk-4.saclay.iot-lab.info:20000' 
gHelp: Press s to start test, r to print it is ready
READY
s
START
main(): This is RIOT! (Version: buildtest)
You are running RIOT on a(n) nrf51dk board.
This board features a(n) nrf51 MCU.
Starting ed25519 test.
The signature is valid!

make: Leaving directory '/work/riot/RIOT/tests/pkg_wolfcrypt-ed25519-verify'

```

</details>

<details><summary>tests/pkg_wolfssl</summary>

```
$ RIOT_CI_BUILD=1 make BOARD=nrf51dk -C tests/pkg_wolfssl flash test IOTLAB_NODE=auto-ssh
make: Entering directory '/work/riot/RIOT/tests/pkg_wolfssl'
Building application "tests_pkg_wolfssl" for "nrf51dk" with MCU "nrf51".

"make" -C /work/riot/RIOT/build/pkg/wolfssl/wolfcrypt/src -f /work/riot/RIOT/pkg/wolfssl/Makefile.wolfcrypt
"make" -C /work/riot/RIOT/build/pkg/wolfssl/wolfcrypt/benchmark -f /work/riot/RIOT/pkg/wolfssl/Makefile.wolfcrypt-benchmark
"make" -C /work/riot/RIOT/build/pkg/wolfssl/wolfcrypt/test -f /work/riot/RIOT/pkg/wolfssl/Makefile.wolfcrypt-test
"make" -C /work/riot/RIOT/build/pkg/wolfssl/src -f /work/riot/RIOT/pkg/wolfssl/Makefile.wolfssl
   text	   data	    bss	    dec	    hex	filename
 115500	    832	   6660	 122992	  1e070	/work/riot/RIOT/tests/pkg_wolfssl/bin/nrf51dk/tests_pkg_wolfssl.elf
iotlab-node --jmespath='keys(@)[0]' --format='int'  --list saclay,nrf51dk,4 --flash /work/riot/RIOT/tests/pkg_wolfssl/bin/nrf51dk/tests_pkg_wolfssl.bin | grep 0
0
r
r
r
r
ssh -t abadie at saclay.iot-lab.info 'socat - tcp:nrf51dk-4.saclay.iot-lab.info:20000' 
r
zw'Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
Help: Press s to start test, r to print it is ready
READY
READY
READY
READY
READY
s
START
main(): This is RIOT! (Version: buildtest)
wolfSSL Crypto Test!
------------------------------------------------------------------------------
 wolfSSL version 4.5.0
------------------------------------------------------------------------------
error    test passed!
MEMORY   test passed!
base64   test passed!
asn      test passed!
RANDOM   test passed!
SHA-256  test passed!
SHA-384  test passed!
SHA-512  test passed!
Hash     test passed!
HC-128   test passed!
Chacha   test passed!
POLY1305 test passed!
ChaCha20-Poly1305 AEAD test passed!
AES      test passed!
AES192   test passed!
AES256   test passed!
ECC      test passed!
ECC buffer test passed!
CURVE25519 test passed!
ED25519  test passed!
logging  test passed!
mutex    test passed!
Test complete
wolfSSL Benchmark!
------------------------------------------------------------------------------
 wolfSSL version 4.5.0
------------------------------------------------------------------------------
wolfCrypt Benchmark (block bytes 1024, min 1.0 sec each)
RNG                 50 KB took 1.030 seconds,   48.557 KB/s
AES-128-CBC-enc    100 KB took 1.147 seconds,   87.170 KB/s
AES-128-CBC-dec    100 KB took 1.216 seconds,   82.237 KB/s
AES-192-CBC-enc    100 KB took 1.292 seconds,   77.424 KB/s
AES-192-CBC-dec     75 KB took 1.023 seconds,   73.289 KB/s
AES-256-CBC-enc     75 KB took 1.077 seconds,   69.635 KB/s
AES-256-CBC-dec     75 KB took 1.135 seconds,   66.098 KB/s
CHACHA             250 KB took 1.052 seconds,  237.601 KB/s
CHA-POLY           100 KB took 1.257 seconds,   79.581 KB/s
POLY1305           150 KB took 1.152 seconds,  130.229 KB/s
SHA-256            150 KB took 1.156 seconds,  129.770 KB/s
SHA-384             75 KB took 1.365 seconds,   54.965 KB/s
SHA-512             75 KB took 1.364 seconds,   54.966 KB/s
ECC      256 key gen         1 ops took 8.729 sec, avg 8729.142 ms, 0.115 ops/sec
ECDHE    256 agree           2 ops took 17.455 sec, avg 8727.657 ms, 0.115 ops/sec
ECDSA    256 sign            2 ops took 18.520 sec, avg 9259.859 ms, 0.108 ops/sec
ECDSA    256 verify          2 ops took 34.116 sec, avg 17058.201 ms, 0.059 ops/sec
CURVE  25519 key gen         1 ops took 5.757 sec, avg 5756.740 ms, 0.174 ops/sec
CURVE  25519 agree           2 ops took 11.509 sec, avg 5754.717 ms, 0.174 ops/sec
ED     25519 key gen         1 ops took 5.804 sec, avg 5804.147 ms, 0.172 ops/sec
ED     25519 sign            2 ops took 11.785 sec, avg 5892.404 ms, 0.170 ops/sec
ED     25519 verify          2 ops took 24.404 sec, avg 12201.779 ms, 0.082 ops/sec
Benchmark complete

make: Leaving directory '/work/riot/RIOT/tests/pkg_wolfssl'

```

</details>

<details><summary>tests/suit_manifest</summary>

```
$ RIOT_CI_BUILD=1 make BOARD=nrf51dk -C tests/suit_manifest test IOTLAB_NODE=auto-ssh
make: Entering directory '/work/riot/RIOT/tests/suit_manifest'
r
ssh -t abadie at saclay.iot-lab.info 'socat - tcp:nrf51dk-4.saclay.iot-lab.info:20000' 
Help: Press s to start test, r to print it is ready
READY
s
START
main(): This is RIOT! (Version: buildtest)
.Stored sequence number: 1

--- testing manifest 0
---- res=-6 (expected=-6)

--- testing manifest 1
suit: verifying manifest signature
suit: validated manifest version
)Retrieved sequence number: 1
Manifest seq_no: 1, highest available: 1
seq_nr <= running image
)---- res=-5 (expected=-5)

--- testing manifest 2
suit: verifying manifest signature
suit: validated manifest version
)Retrieved sequence number: 1
Manifest seq_no: 2, highest available: 1
suit: validated sequence number
)Formatted component name: .ram.0
---- res=-4 (expected=-4)

--- testing manifest 3
suit: verifying manifest signature
suit: validated manifest version
)Retrieved sequence number: 1
Manifest seq_no: 2, highest available: 1
suit: validated sequence number
)Formatted component name: .ram.0
validating vendor ID
Comparing 547d0d74-6d3a-5a92-9662-4881afd9407b to 547d0d74-6d3a-5a92-9662-4881afd9407b from manifest
validating vendor ID: OK
validating class id
Comparing d485483d-13fb-5f9f-b4e0-c42c9b4c2310 to d485483d-13fb-5f9f-b4e0-c42c9b4c2310 from manifest
validating class id: OK
SUIT policy check OK.
Formatted component name: .ram.0
Mock writing payload 0
Verifying image digest
Starting digest verification against image
Install correct payload
Verifying image digest
Starting digest verification against image
Install correct payload
---- res=0 (expected=0)

--- testing manifest 4
suit: verifying manifest signature
suit: validated manifest version
)Retrieved sequence number: 1
Manifest seq_no: 2, highest available: 1
suit: validated sequence number
)Formatted component name: .ram.0
Formatted component name: .ram.1
Formatted component name: .ram.0
Setting component index to 0
validating vendor ID
Comparing 547d0d74-6d3a-5a92-9662-4881afd9407b to 547d0d74-6d3a-5a92-9662-4881afd9407b from manifest
validating vendor ID: OK
validating class id
Comparing d485483d-13fb-5f9f-b4e0-c42c9b4c2310 to d485483d-13fb-5f9f-b4e0-c42c9b4c2310 from manifest
validating class id: OK
Formatted component name: .ram.1
Setting component index to 1
validating vendor ID
Comparing 547d0d74-6d3a-5a92-9662-4881afd9407b to 547d0d74-6d3a-5a92-9662-4881afd9407b from manifest
validating vendor ID: OK
validating class id
Comparing d485483d-13fb-5f9f-b4e0-c42c9b4c2310 to d485483d-13fb-5f9f-b4e0-c42c9b4c2310 from manifest
validating class id: OK
Formatted component name: .ram.0
Setting component index to 0
SUIT policy check OK.
Formatted component name: .ram.0
Mock writing payload 0
Verifying image digest
Starting digest verification against image
Install correct payload
Formatted component name: .ram.1
Setting component index to 1
SUIT policy check OK.
Formatted component name: .ram.1
Mock writing payload 1
Verifying image digest
Starting digest verification against image
Install correct payload
Formatted component name: .ram.0
Setting component index to 0
Verifying image digest
Starting digest verification against image
Install correct payload
Formatted component name: .ram.1
Setting component index to 1
Verifying image digest
Starting digest verification against image
Install correct payload
---- res=0 (expected=0)

OK (1 tests)

```

</details>

<!--
Details steps to test your contribution:
- which test/example to compile for which board and is there a 'test' command
- how to know that it was not working/available in master
- the expected success test output
-->


### Issues/PRs references

<!--
Examples: Fixes #1234. See also #5678. Depends on PR #9876.

Please use keywords (e.g., fixes, resolve) with the links to the issues you
resolved, this way they will be automatically closed when your pull request
is merged. See https://help.github.com/articles/closing-issues-using-keywords/.
-->

You can view, comment on, or merge this pull request online at:

  https://github.com/RIOT-OS/RIOT/pull/15880

-- Commit Summary --

  * tests/pkg_libcose: increase test timeout for nrf51dk
  * tests/pkg_wolfcrypt-ed25519-verify: increase timeout for nrf51dk
  * tests/pkg_wolfssl: increase timeout for nrf51dk
  * tests/suit_manifest: increase timeout for nrf51dk

-- File Changes --

    M tests/pkg_libcose/tests/01-run.py (4)
    M tests/pkg_wolfcrypt-ed25519-verify/tests/01-run.py (7)
    M tests/pkg_wolfssl/tests/01-run.py (6)
    M tests/suit_manifest/tests/01-run.py (3)

-- Patch Links --

https://github.com/RIOT-OS/RIOT/pull/15880.patch
https://github.com/RIOT-OS/RIOT/pull/15880.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/RIOT-OS/RIOT/pull/15880
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.riot-os.org/pipermail/notifications/attachments/20210128/9f543c30/attachment-0001.htm>


More information about the notifications mailing list