4.19.106-cip21-rt8 problems on de0-nano


Pavel Machek
 

Hi!

I pushed candidate for -cip-rt, but it seems to fail on de0-nano
board. Code under testing is at:

https://gitlab.com/cip-project/cip-kernel/linux-cip/tree/ci/pavel/linux-cip

So far testing fails, I did few steps to narrow it down.

? 0f2960c75dd68d339f0aff2935f51652b5625fbf 4.19.106-rt44
? 68541faf398c02bfa6a65ffcf4484e71b8e0306d v4.19.103-cip20.
ok 122663369 c7691f9c497d0f8393724875a8875b61daa9fc29 v4.19.100-rt41.
? b7f7a3d15716de6d78b6d4807e23fa46de106c2f v4.19.103-rt42.
ok 122667425 51184ac81b2de92545a3b9d330d4d53a176e0976 v4.19.104-rt43.
bad 122671461 f019fe97731d222ee59ac8e54a62753e9c69ef31 v4.19.106-rt44.
bad 122657523 881b771ab11616e2aab6e914d950467f7c8914bd v4.19.106-cip21.

So it seems failure was introduced between v4.19.104-rt43 and
v4.19.106-rt44. Unfortunately, I don't understand the logs enough to
know what the real failure is. If someone can help...

I'll proceed to narrow it down.

Best regards,
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html


Pavel Machek
 

On Tue 2020-03-03 10:15:56, Pavel Machek wrote:
Hi!

I pushed candidate for -cip-rt, but it seems to fail on de0-nano
board. Code under testing is at:

https://gitlab.com/cip-project/cip-kernel/linux-cip/tree/ci/pavel/linux-cip
It is pipeline

https://gitlab.com/cip-project/cip-kernel/linux-cip/pipelines/122762401

I'll reuse the branch for more testing.

Best regards,
Pavel
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany


Pavel Machek
 

Hi!

I pushed candidate for -cip-rt, but it seems to fail on de0-nano
board. Code under testing is at:

https://gitlab.com/cip-project/cip-kernel/linux-cip/tree/ci/pavel/linux-cip
It is pipeline

https://gitlab.com/cip-project/cip-kernel/linux-cip/pipelines/122762401

I'll reuse the branch for more testing.
I managed to narrow the bad commit to the -rt tree, between:

OK 122904930 pick 69aa73357e6a rcu: Don't allow to change rcu_normal_after_boot on RT
pick 849ef8789077 pci/switchtec: fix stream_open.cocci warnings
pick ad8a5e8279c4 sched/core: Drop a preempt_disable_rt() statement
pick 966f066d96cb timers: Redo the notification of canceling timers on -RT
pick 0393fd5a4f9a Revert "futex: Ensure lock/unlock symetry versus pi_lock and hash bucket lock"
pick 84eb0b64a27a Revert "futex: Fix bug on when a requeued RT task times out"
pick fcc893280f4e Revert "rtmutex: Handle the various new futex race conditions"
pick 2eac93cf9d16 Revert "futex: workaround migrate_disable/enable in different context"
pick 9b8964629f4f futex: Make the futex_hash_bucket lock raw
pick cc1812bf198b futex: Delay deallocation of pi_state
pick f5e115c43100 mm/zswap: Do not disable preemption in zswap_frontswap_store()
pick e0d0d09a08ad revert-aio
pick a0a40bfb4300 fs/aio: simple simple work
pick 0fae581d8c5e revert-thermal
pick c0d95b4a8a1b thermal: Defer thermal wakups to threads
pick 700fbb4afb6e revert-block
pick 4cda50ff12cf block: blk-mq: move blk_queue_usage_counter_release() into process context
pick 9e982f55745b workqueue: rework
pick c0db53dc3bf4 i2c: exynos5: Remove IRQF_ONESHOT
pick 1f160d170203 i2c: hix5hd2: Remove IRQF_ONESHOT
BAD 122882826 eae5a7cab722 sched/deadline: Ensure inactive_timer runs in hardirq context

Best regards,
Pavel
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany


Pavel Machek
 

Hi!

I pushed candidate for -cip-rt, but it seems to fail on de0-nano
board. Code under testing is at:

https://gitlab.com/cip-project/cip-kernel/linux-cip/tree/ci/pavel/linux-cip
It is pipeline

https://gitlab.com/cip-project/cip-kernel/linux-cip/pipelines/122762401

I'll reuse the branch for more testing.
I managed to narrow the bad commit to the -rt tree, between:

OK 122904930 pick 69aa73357e6a rcu: Don't allow to change rcu_normal_after_boot on RT
pick 849ef8789077 pci/switchtec: fix stream_open.cocci warnings
pick ad8a5e8279c4 sched/core: Drop a preempt_disable_rt() statement
pick 966f066d96cb timers: Redo the notification of canceling timers on -RT
pick 0393fd5a4f9a Revert "futex: Ensure lock/unlock symetry versus pi_lock and hash bucket lock"
pick 84eb0b64a27a Revert "futex: Fix bug on when a requeued RT task times out"
pick fcc893280f4e Revert "rtmutex: Handle the various new futex race conditions"
pick 2eac93cf9d16 Revert "futex: workaround migrate_disable/enable in different context"
pick 9b8964629f4f futex: Make the futex_hash_bucket lock raw
pick cc1812bf198b futex: Delay deallocation of pi_state
pick f5e115c43100 mm/zswap: Do not disable preemption in zswap_frontswap_store()
pick e0d0d09a08ad revert-aio
pick a0a40bfb4300 fs/aio: simple simple work
pick 0fae581d8c5e revert-thermal
pick c0d95b4a8a1b thermal: Defer thermal wakups to threads
pick 700fbb4afb6e revert-block
pick 4cda50ff12cf block: blk-mq: move blk_queue_usage_counter_release() into process context
pick 9e982f55745b workqueue: rework
pick c0db53dc3bf4 i2c: exynos5: Remove IRQF_ONESHOT
pick 1f160d170203 i2c: hix5hd2: Remove IRQF_ONESHOT
BAD 122882826 eae5a7cab722 sched/deadline: Ensure inactive_timer runs in hardirq context
And something went seriously wrong after these tests. I submitted same
tree twice, and got different results.

First this -- de0-nano succeeds:

https://gitlab.com/cip-project/cip-kernel/linux-cip/pipelines/122904930

Now this -- de0-nano fails (and ipc227e is unfinished for long time):

https://gitlab.com/cip-project/cip-kernel/linux-cip/pipelines/122959477

I'll need some help here.

Best regards,
Pavel
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany


Jan Kiszka
 

On 03.03.20 22:28, Pavel Machek wrote:
Hi!

I pushed candidate for -cip-rt, but it seems to fail on de0-nano
board. Code under testing is at:

https://gitlab.com/cip-project/cip-kernel/linux-cip/tree/ci/pavel/linux-cip
It is pipeline

https://gitlab.com/cip-project/cip-kernel/linux-cip/pipelines/122762401

I'll reuse the branch for more testing.
I managed to narrow the bad commit to the -rt tree, between:

OK 122904930 pick 69aa73357e6a rcu: Don't allow to change rcu_normal_after_boot on RT
pick 849ef8789077 pci/switchtec: fix stream_open.cocci warnings
pick ad8a5e8279c4 sched/core: Drop a preempt_disable_rt() statement
pick 966f066d96cb timers: Redo the notification of canceling timers on -RT
pick 0393fd5a4f9a Revert "futex: Ensure lock/unlock symetry versus pi_lock and hash bucket lock"
pick 84eb0b64a27a Revert "futex: Fix bug on when a requeued RT task times out"
pick fcc893280f4e Revert "rtmutex: Handle the various new futex race conditions"
pick 2eac93cf9d16 Revert "futex: workaround migrate_disable/enable in different context"
pick 9b8964629f4f futex: Make the futex_hash_bucket lock raw
pick cc1812bf198b futex: Delay deallocation of pi_state
pick f5e115c43100 mm/zswap: Do not disable preemption in zswap_frontswap_store()
pick e0d0d09a08ad revert-aio
pick a0a40bfb4300 fs/aio: simple simple work
pick 0fae581d8c5e revert-thermal
pick c0d95b4a8a1b thermal: Defer thermal wakups to threads
pick 700fbb4afb6e revert-block
pick 4cda50ff12cf block: blk-mq: move blk_queue_usage_counter_release() into process context
pick 9e982f55745b workqueue: rework
pick c0db53dc3bf4 i2c: exynos5: Remove IRQF_ONESHOT
pick 1f160d170203 i2c: hix5hd2: Remove IRQF_ONESHOT
BAD 122882826 eae5a7cab722 sched/deadline: Ensure inactive_timer runs in hardirq context
And something went seriously wrong after these tests. I submitted same
tree twice, and got different results.
First this -- de0-nano succeeds:
https://gitlab.com/cip-project/cip-kernel/linux-cip/pipelines/122904930
Now this -- de0-nano fails (and ipc227e is unfinished for long time):
https://gitlab.com/cip-project/cip-kernel/linux-cip/pipelines/122959477
I'll need some help here.
The logs read like the targets are not (always) coming up, e.g.
https://gitlab.com/cip-project/cip-kernel/linux-cip/-/jobs/457824214#L377

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux