cip/linux-4.19.y-cip baseline: 121 runs, 1 regressions (v4.19.217-cip62) #kernelci


kernelci.org bot <bot@...>
 

cip/linux-4.19.y-cip baseline: 121 runs, 1 regressions (v4.19.217-cip62)

Regressions Summary
-------------------

platform | arch | lab | compiler | defconfig | regressions
---------+------+---------------+----------+---------------------+------------
panda | arm | lab-collabora | gcc-10 | omap2plus_defconfig | 1

Details: https://kernelci.org/test/job/cip/branch/linux-4.19.y-cip/kernel/v4.19.217-cip62/plan/baseline/

Test: baseline
Tree: cip
Branch: linux-4.19.y-cip
Describe: v4.19.217-cip62
URL: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git
SHA: dc62e26e3be875a7324b85b8274c13a335e610dd


Test Regressions
----------------


platform | arch | lab | compiler | defconfig | regressions
---------+------+---------------+----------+---------------------+------------
panda | arm | lab-collabora | gcc-10 | omap2plus_defconfig | 1

Details: https://kernelci.org/test/plan/id/61a587a0ab3b0079bd18f6d7

Results: 5 PASS, 1 FAIL, 0 SKIP
Full config: omap2plus_defconfig
Compiler: gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
Plain log: https://storage.kernelci.org//cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-collabora/baseline-panda.txt
HTML log: https://storage.kernelci.org//cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-collabora/baseline-panda.html
Rootfs: http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05-6-g8983f3b738df/armel/baseline/rootfs.cpio.gz


* baseline.dmesg.emerg: https://kernelci.org/test/case/id/61a587a0ab3b0079bd18f6dd
new failure (last pass: v4.19.216-cip61)
2 lines

2021-11-30T02:08:20.120053 kern :emerg : BUG: spinlock bad magic on CPU#0, udevd/110
2021-11-30T02:08:20.129545 kern :emerg : lock: emif_lock+0x0/0xffffecfc [emif], .magic: dead4ead, .owner: <none>/-1, .owner_cpu: -1
2021-11-30T02:08:20.143995 <8>[ 21.198120] <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=emerg RESULT=fail UNITS=lines MEASUREMENT=2>


Pavel Machek
 

Hi!

platform | arch | lab | compiler | defconfig | regressions
---------+------+---------------+----------+---------------------+------------
panda | arm | lab-collabora | gcc-10 | omap2plus_defconfig | 1

Details: https://kernelci.org/test/plan/id/61a587a0ab3b0079bd18f6d7

Results: 5 PASS, 1 FAIL, 0 SKIP
Full config: omap2plus_defconfig
Compiler: gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
Plain log: https://storage.kernelci.org//cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-collabora/baseline-panda.txt
HTML log:
https://storage.kernelci.org//cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-collabora/baseline-panda.html
So this has enough information to tell me that we have a kernel
problem there, good:

02:08:17.792502 <6>[ 18.847320] omap-mailbox 4a0f4000.mailbox: omap
mailbox rev 0x400
02:08:17.800596 <0>[ 18.857543] BUG: spinlock bad magic on CPU#0,
udevd/110
02:08:17.808843 <6>[ 18.857788] emif 4c000000.emif: emif_probe:
device configured with addr = (ptrval) and IRQ26
02:08:17.818753 <0>[ 18.863311] lock: emif_lock+0x0/0xffffecfc
[emif], .magic: dead4ead, .owner: <none>/-1, .owner_cpu: -1
02:08:17.825277 <4>[ 18.877502] CPU: 0 PID: 110 Comm: udevd Not
tainted 4.19.217-cip62 #1
02:08:17.831655 <4>[ 18.888275] Hardware name: Generic OMAP4
(Flattened Device Tree)
02:08:17.839660 <4>[ 18.894195] [<c01123bc>] (unwind_backtrace)
from [<c010cc28>] (show_stack+0x10/0x14)
02:08:17.847570 <4>[ 18.900482] [<c010cc28>] (show_stack) from
[<c0949dd4>] (dump_stack+0xe0/0x114)
02:08:17.855211 <4>[ 18.905761] [<c0949dd4>] (dump_stack) from
[<c01a9168>] (do_raw_spin_lock+0xbc/0x124)

Unfortunately, not enough information to debug the problem, I'm
afraid. Questions to start debugging this would be:

a) is the problem deterministic?

b) is the problem present in v4.19.217 ?

c) what were the last -stable and -cip kernels that worked?

d) what device/module is causing the BUG?

Best regards,
Pavel
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany


Alice Ferrazzi
 

On Tue, Nov 30, 2021 at 5:41 PM Pavel Machek <pavel@...> wrote:

Hi!

platform | arch | lab | compiler | defconfig | regressions
---------+------+---------------+----------+---------------------+------------
panda | arm | lab-collabora | gcc-10 | omap2plus_defconfig | 1

Details: https://kernelci.org/test/plan/id/61a587a0ab3b0079bd18f6d7

Results: 5 PASS, 1 FAIL, 0 SKIP
Full config: omap2plus_defconfig
Compiler: gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
Plain log: https://storage.kernelci.org//cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-collabora/baseline-panda.txt
HTML log:
https://storage.kernelci.org//cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-collabora/baseline-panda.html
So this has enough information to tell me that we have a kernel
problem there, good:

02:08:17.792502 <6>[ 18.847320] omap-mailbox 4a0f4000.mailbox: omap
mailbox rev 0x400
02:08:17.800596 <0>[ 18.857543] BUG: spinlock bad magic on CPU#0,
udevd/110
02:08:17.808843 <6>[ 18.857788] emif 4c000000.emif: emif_probe:
device configured with addr = (ptrval) and IRQ26
02:08:17.818753 <0>[ 18.863311] lock: emif_lock+0x0/0xffffecfc
[emif], .magic: dead4ead, .owner: <none>/-1, .owner_cpu: -1
02:08:17.825277 <4>[ 18.877502] CPU: 0 PID: 110 Comm: udevd Not
tainted 4.19.217-cip62 #1
02:08:17.831655 <4>[ 18.888275] Hardware name: Generic OMAP4
(Flattened Device Tree)
02:08:17.839660 <4>[ 18.894195] [<c01123bc>] (unwind_backtrace)
from [<c010cc28>] (show_stack+0x10/0x14)
02:08:17.847570 <4>[ 18.900482] [<c010cc28>] (show_stack) from
[<c0949dd4>] (dump_stack+0xe0/0x114)
02:08:17.855211 <4>[ 18.905761] [<c0949dd4>] (dump_stack) from
[<c01a9168>] (do_raw_spin_lock+0xbc/0x124)

Unfortunately, not enough information to debug the problem, I'm
afraid. Questions to start debugging this would be:

a) is the problem deterministic?
as the testing board are actually different is unclear.
The board and results can be seen here:
https://linux.kernelci.org/test/plan/id/61a587a0ab3b0079bd18f6d7/

the output log is here:
https://storage.kernelci.org/cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-collabora/baseline-panda.html
working board example log is here (works for 4.19.216-cip61 but using
a board on baylibre that could be different from the board on
collabora):
https://storage.kernelci.org/cip/linux-4.19.y-cip/v4.19.216-cip61/arm/omap2plus_defconfig/gcc-10/lab-baylibre/baseline-panda.html
looks like there is also a passed test on lab-baylibre for the panda
board with the same kernel:
https://storage.kernelci.org/cip/linux-4.19.y-cip/v4.19.217-cip62/arm/omap2plus_defconfig/gcc-10/lab-baylibre/baseline-panda.html

The job json file for replicate the job on a lava lab with panda board
can be found here:
https://lava.collabora.co.uk/scheduler/job/5113437/definition


b) is the problem present in v4.19.217 ?
here is the diff with 4.19.217 stable
https://www.diffchecker.com/JjmXiP7b

c) what were the last -stable and -cip kernels that worked?
if you click on the link on the details or the link here under:
https://linux.kernelci.org/test/case/id/61a587a0ab3b0079bd18f6dd/
is wrote:
New regression, last pass: v4.19.216-cip61

This is the one that passed:
https://linux.kernelci.org/test/plan/id/618f6407f33d0dd1e633595d/


- About bisecting the kernel.
We are currently in the process of activating CIP to the list of
automated bisected kernel
for getting more information on this and future problems.
The following issue is the request to KernelCI for adding CIP to the
automated bisected kernel list.
https://github.com/kernelci/kernelci-core/issues/922



Thanks,
Alice


--
======================================
Cybertrust Japan Co.,Ltd.
Alice Ferrazzi
alice.ferrazzi@...
======================================