-----Original Message-----
From: Jan Kiszka [mailto:jan.kiszka@...]
Sent: 10 March 2020 00:23
To: Pavel Machek <pavel@...>; Bhola, Bikram <Bikram_Bhola@...>; Quirin Gylstorff <quirin.gylstorff@...>
Cc: cip-dev@...
Subject: Re: I need de0-nano testing for -rt release was Re: [cip-dev] 4.19.106-cip21-rt8 problems on de0-nano

On 09.03.20 11:21, Pavel Machek wrote:

Hi!

I pushed candidate for -cip-rt, but it seems to fail on de0-nano
board. Code under testing is at:

https://gitlab.com/cip-project/cip-kernel/linux-cip/tree/ci/pavel
/linux-cip

It is pipeline

https://gitlab.com/cip-project/cip-kernel/linux-cip/pipelines/1227
62401

I'll reuse the branch for more testing.

I managed to narrow the bad commit to the -rt tree, between:

OK 122904930 pick 69aa73357e6a rcu: Don't allow to change
rcu_normal_after_boot on RT pick 849ef8789077 pci/switchtec: fix
stream_open.cocci warnings pick ad8a5e8279c4 sched/core: Drop a
preempt_disable_rt() statement pick 966f066d96cb timers: Redo the
notification of canceling timers on -RT pick 0393fd5a4f9a Revert "futex: Ensure lock/unlock symetry versus pi_lock and hash bucket lock"
pick 84eb0b64a27a Revert "futex: Fix bug on when a requeued RT task times out"
pick fcc893280f4e Revert "rtmutex: Handle the various new futex race conditions"
pick 2eac93cf9d16 Revert "futex: workaround migrate_disable/enable in different context"
pick 9b8964629f4f futex: Make the futex_hash_bucket lock raw pick
cc1812bf198b futex: Delay deallocation of pi_state

pick f5e115c43100 mm/zswap: Do not disable preemption in
zswap_frontswap_store()

pick e0d0d09a08ad revert-aio
pick a0a40bfb4300 fs/aio: simple simple work pick 0fae581d8c5e
revert-thermal pick c0d95b4a8a1b thermal: Defer thermal wakups to
threads pick 700fbb4afb6e revert-block pick 4cda50ff12cf block:
blk-mq: move blk_queue_usage_counter_release() into process context
pick 9e982f55745b workqueue: rework pick c0db53dc3bf4 i2c: exynos5:
Remove IRQF_ONESHOT pick 1f160d170203 i2c: hix5hd2: Remove
IRQF_ONESHOT BAD 122882826 eae5a7cab722 sched/deadline: Ensure
inactive_timer runs in hardirq context

And something went seriously wrong after these tests. I submitted
same tree twice, and got different results.

First this -- de0-nano succeeds:

https://gitlab.com/cip-project/cip-kernel/linux-cip/pipelines/122904
930

Now this -- de0-nano fails (and ipc227e is unfinished for long time):

https://gitlab.com/cip-project/cip-kernel/linux-cip/pipelines/122959
477

I'll need some help here.

The logs read like the targets are not (always) coming up, e.g.
https://gitlab.com/cip-project/cip-kernel/linux-cip/-/jobs/457824214#
L377

Yes... I don't need that target, but I need de0-nano... and it did not
work last time I checked.

Bikram, could someone on your side check the board status in the Mentor lab? Thanks!

On a related note... it would be good to somehow show difference
between "kernel test failure" and "target failure".

If we see bootloader in the logs, and then test fails/timeouts =>
"kernel test failure", I need to solve it.

If we don't get messages from the bootloader => "target failure",
someone needs to check the power relays or something...

I'm not happy about the parsability of those LAVA logs either, but I have no idea if/how that can be improved best. Maybe Quirin has some idea based on his work with them.

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux