Jump to content

    

Причина Kernel panic

Привет, электроникс!

 

Как выяснить причину кернел паники? Собираю OpenWRT trunk под кастомное железо. Лог загрузки ниже

 

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.1.16 (vladimir@vladimir-Ubuntu) (gcc version 5.2.0 (OpenWrt GCC 5.2.0 r48506) ) #15 Wed Jan 27 15:16:58 MSK 2016
[    0.000000] CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=0005317f
[    0.000000] CPU: VIVT data cache, VIVT instruction cache
[    0.000000] Machine model: I2SE Duckbill
[    0.000000] Memory policy: Data cache writeback
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32512
[    0.000000] Kernel command line: console=ttyAPP4,115200 rootfstype=ubifs ubi.mtd=5 root=ubi0_0 rw mtdparts=gpmi-nand:10m(bootloader)ro,256k(environment),128k(fdt),5m(kernel),40m(backupfs)ro,-(rootfs)
[    0.000000] PID hash table entries: 512 (order: -1, 2048 bytes)
[    0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
[    0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.000000] Memory: 123308K/131072K available (4630K kernel code, 162K rwdata, 1308K rodata, 156K init, 213K bss, 7764K reserved, 0K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
[    0.000000]     vmalloc : 0xc8800000 - 0xff000000   ( 872 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xc8000000   ( 128 MB)
[    0.000000]     modules : 0xbf000000 - 0xc0000000   (  16 MB)
[    0.000000]       .text : 0xc0008000 - 0xc05d4d6c   (5940 kB)
[    0.000000]       .init : 0xc05d5000 - 0xc05fc000   ( 156 kB)
[    0.000000]       .data : 0xc05fc000 - 0xc0624864   ( 163 kB)
[    0.000000]        .bss : 0xc0624864 - 0xc0659e8c   ( 214 kB)
[    0.000000] NR_IRQS:16 nr_irqs:16 16
[    0.000000] clocksource mxs_timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[    0.000024] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
[    0.000491] Calibrating delay loop... 226.09 BogoMIPS (lpj=1130496)
[    0.080319] pid_max: default: 32768 minimum: 301
[    0.080640] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.080688] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.082011] CPU: Testing write buffer coherency: ok
[    0.082882] Setting up static identity map for 0x40008400 - 0x40008458
[    0.107601] clocksource jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.107981] pinctrl core: initialized pinctrl subsystem
[    0.113983] NET: Registered protocol family 16
[    0.115425] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.119282] cpuidle: using governor ladder
[    0.119368] cpuidle: using governor menu
[    0.152924] Serial: AMBA PL011 UART driver
[    0.153663] 80074000.serial: ttyAMA0 at MMIO 0x80074000 (irq = 216, base_baud = 0) is a PL011 rev2
[    0.241938] mxs-dma 80004000.dma-apbh: initialized
[    0.248140] mxs-dma 80024000.dma-apbx: initialized
[    0.253738] usbcore: registered new interface driver usbfs
[    0.254076] usbcore: registered new interface driver hub
[    0.254355] usbcore: registered new device driver usb
[    0.255273] pps_core: LinuxPPS API ver. 1 registered
[    0.255310] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.255484] PTP clock support registered
[    0.259425] Switched to clocksource mxs_timer
[    0.264020] NET: Registered protocol family 2
[    0.266008] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    0.266095] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    0.266151] TCP: Hash tables configured (established 1024 bind 1024)
[    0.266355] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.266431] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.266988] NET: Registered protocol family 1
[    0.292963] RPC: Registered named UNIX socket transport module.
[    0.293008] RPC: Registered udp transport module.
[    0.293027] RPC: Registered tcp transport module.
[    0.293044] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.296603] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.301332] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.302166] NFS: Registering the id_resolver key type
[    0.302299] Key type id_resolver registered
[    0.302332] Key type id_legacy registered
[    0.302429] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.302594] jffs2: version 2.2 (NAND) (SUMMARY) (ZLIB) (LZO) (LZMA) (RTIME) (RUBIN) (CMODE_PRIORITY) © 2001-2006 Red Hat, Inc.
[    0.310156] io scheduler noop registered
[    0.310232] io scheduler deadline registered (default)
[    0.311123] Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
[    0.312972] Serial: AMBA driver
[    0.314120] 8006c000.serial: ttyAPP1 at MMIO 0x8006c000 (irq = 214, base_baud = 1500000) is a 8006c000.serial
[    0.314787] mxs-auart 8006c000.serial: Found APPUART 3.1.0
[    0.315532] 80072000.serial: ttyAPP4 at MMIO 0x80072000 (irq = 215, base_baud = 1500000) is a 80072000.serial
[    0.771570] console [ttyAPP4] enabled
[    0.775877] mxs-auart 80072000.serial: Found APPUART 3.1.0
[    0.781992] dummy-irq: no IRQ given.  Use irq=N
[    0.801970] nand: device found, Manufacturer ID: 0xc2, Chip ID: 0xda
[    0.808364] nand: Macronix MX30LF2G18AC
[    0.812316] nand: 256 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 64
[    0.820100] Scanning device for bad blocks
[    0.824702] Bad eraseblock 1 at 0x000000020000
[    0.829412] Bad eraseblock 2 at 0x000000040000
[    0.834227] Bad eraseblock 3 at 0x000000060000
[    1.231228] 6 cmdlinepart partitions found on MTD device gpmi-nand
[    1.237446] Creating 6 MTD partitions on "gpmi-nand":
[    1.242635] 0x000000000000-0x000000a00000 : "bootloader"
[    1.250902] 0x000000a00000-0x000000a40000 : "environment"
[    1.258946] 0x000000a40000-0x000000a60000 : "fdt"
[    1.266415] 0x000000a60000-0x000000f60000 : "kernel"
[    1.274250] 0x000000f60000-0x000003760000 : "backupfs"
[    1.282507] 0x000003760000-0x000010000000 : "rootfs"
[    1.291556] mtd: device 5 (rootfs) set to be root filesystem
[    1.306009] mtdsplit: no squashfs found in "rootfs"
[    1.311115] gpmi-nand 8000c000.gpmi-nand: driver registered.
[    1.318964] libphy: Fixed MDIO Bus: probed
[    1.325230] 800f0000.ethernet supply phy not found, using dummy regulator
[    1.447819] libphy: fec_enet_mii_bus: probed
[    1.454670] 800f4000.ethernet supply phy not found, using dummy regulator
[    1.465341] PPP generic driver version 2.4.2
[    1.470328] NET: Registered protocol family 24
[    1.475166] usbcore: registered new interface driver catc
[    1.480968] usbcore: registered new interface driver kaweth
[    1.486606] pegasus: v0.9.3 (2013/04/25), Pegasus/Pegasus II USB Ethernet driver
[    1.494350] usbcore: registered new interface driver pegasus
[    1.500381] usbcore: registered new interface driver rtl8150
[    1.506302] usbcore: registered new interface driver r8152
[    1.512161] usbcore: registered new interface driver asix
[    1.517822] usbcore: registered new interface driver ax88179_178a
[    1.524301] usbcore: registered new interface driver cdc_ether
[    1.530509] usbcore: registered new interface driver cdc_eem
[    1.536468] usbcore: registered new interface driver dm9601
[    1.542425] usbcore: registered new interface driver sr9700
[    1.548266] usbcore: registered new interface driver CoreChips
[    1.554558] usbcore: registered new interface driver smsc75xx
[    1.560778] usbcore: registered new interface driver smsc95xx
[    1.566791] usbcore: registered new interface driver gl620a
[    1.572760] usbcore: registered new interface driver net1080
[    1.578712] usbcore: registered new interface driver plusb
[    1.584623] usbcore: registered new interface driver rndis_host
[    1.590941] usbcore: registered new interface driver MOSCHIP usb-ethernet driver
[    1.598661] usbcore: registered new interface driver sierra_net
[    1.605060] usbcore: registered new interface driver cdc_ncm
[    1.611109] usbcore: registered new interface driver huawei_cdc_ncm
[    1.617644] usbcore: registered new interface driver cdc_mbim
[    1.623985] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.630683] ehci-platform: EHCI generic platform driver
[    1.636750] usbcore: registered new interface driver cdc_acm
[    1.642555] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[    1.650969] usbcore: registered new interface driver cdc_wdm
[    1.657244] usbcore: registered new interface driver usbserial
[    1.663451] usbcore: registered new interface driver usbserial_generic
[    1.670341] usbserial: USB Serial support registered for generic
[    1.676634] usbcore: registered new interface driver qcserial
[    1.682745] usbserial: USB Serial support registered for Qualcomm USB modem
[    1.690115] usbcore: registered new interface driver sierra
[    1.695934] usbserial: USB Serial support registered for Sierra USB modem
[    1.703388] i2c /dev entries driver
[    1.710667] ledtrig-cpu: registered to indicate activity on CPUs
[    1.717515] usbcore: registered new interface driver usbhid
[    1.723240] usbhid: USB HID core driver
[    1.727970] Netfilter messages via NETLINK v0.30.
[    1.732931] nfnl_acct: registering with nfnetlink.
[    1.737943] nf_conntrack version 0.5.0 (1926 buckets, 7704 max)
[    1.744874] ctnetlink v0.93: registering with nfnetlink.
[    1.751057] nf_tables: © 2007-2009 Patrick McHardy <kaber@trash.net>
[    1.757696] nf_tables_compat: © 2012 Pablo Neira Ayuso <pablo@netfilter.org>
[    1.765467] ip_set: protocol 6
[    1.769994] ip_tables: © 2000-2006 Netfilter Core Team
[    1.775842] arp_tables: © 2002 David S. Miller
[    1.780958] NET: Registered protocol family 10
[    1.808709] ip6_tables: © 2000-2006 Netfilter Core Team
[    1.814899] NET: Registered protocol family 17
[    1.819665] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[    1.832512] Bridge firewalling registered
[    1.836588] Ebtables v2.0 registered
[    1.841047] 8021q: 802.1Q VLAN Support v1.8
[    1.845442] Key type dns_resolver registered
[    1.856744] ubi0: attaching mtd5
[    2.120314] random: nonblocking pool is initialized
[    4.000210] ubi0: scanning is finished
[    4.028174] ubi0: attached mtd5 (name "rootfs", size 200 MiB)
[    4.034088] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
[    4.041078] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[    4.047893] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
[    4.054977] ubi0: good PEBs: 1605, bad PEBs: 0, corrupted PEBs: 0
[    4.061183] ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128
[    4.068433] ubi0: max/mean erase counter: 2/1, WL threshold: 4096, image sequence number: 0
[    4.076882] ubi0: available PEBs: 0, total reserved PEBs: 1605, PEBs reserved for bad PEB handling: 40
[    4.086351] hctosys: unable to open rtc device (rtc0)
[    4.099082] ubi0: background thread "ubi_bgt0d" started, PID 417
[    4.113582] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 419
[    4.244236] UBIFS (ubi0:0): UBIFS: mounted UBI device 0, volume 0, name "rootfs"
[    4.251807] UBIFS (ubi0:0): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
[    4.261843] UBIFS (ubi0:0): FS size: 119357440 bytes (113 MiB, 940 LEBs), journal size 9023488 bytes (8 MiB, 72 LEBs)
[    4.272551] UBIFS (ubi0:0): reserved for root: 0 bytes (0 KiB)
[    4.278425] UBIFS (ubi0:0): media format: w4/r0 (latest is w4/r0), UUID 30876C20-3549-4DCA-94D4-86F7C04A28C9, small LPT model
[    4.293692] VFS: Mounted root (ubifs filesystem) on device 0:11.
[    4.300630] Freeing unused kernel memory: 156K (c05d5000 - c05fc000)
[    4.417652] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    4.417652]
[    4.426841] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    4.426841]

Share this post


Link to post
Share on other sites
[ 4.293692] VFS: Mounted root (ubifs filesystem) on device 0:11.

[ 4.300630] Freeing unused kernel memory: 156K (c05d5000 - c05fc000)

это обычно пишется перед тем как ядро запускает init-скрипт.

 

какой init-скрипт запускать - указывается в bootargs. Если не указано, то пробуются значения по умолчанию.

цитирую:

General non-device-specific boot arguments

'init=...'

This sets the initial command to be executed by the kernel.

If this is not set, or cannot be found, the kernel will try

/sbin/init, then /etc/init, then /bin/init, then /bin/sh and

panic if all of this fails.

видимо, у вас на rootfs таких файлов нет.

Share this post


Link to post
Share on other sites
какой init-скрипт запускать - указывается в bootargs. Если не указано, то пробуются значения по умолчанию.

 

видимо, у вас на rootfs таких файлов нет.

Спасибо за подсказку, но похоже, это не то. Все на месте. Пробовал и /sbin/init ставить, и /bin/sh - все равно кернел паник.

Edited by vgovseychuk

Share this post


Link to post
Share on other sites
Спасибо за подсказку, но похоже, это не то. Все на месте. Пробовал и /sbin/init ставить, и /bin/sh - все равно кернел паник.

Нет, скорее всего речь не про то. Как я понял, krux, имел специальный скрипт предварительной инициализации (preinit). Это по сути опциональная штука.

1. Покажите строку cmdline, которая передается ядру при старте.

2. Другими методами (например загрузка из RAM диска) убедитесь, что на разделе UBI у вас валидная файловая система (rootfs).

Share this post


Link to post
Share on other sites
...скрипт предварительной инициализации (preinit). Это по сути опциональная штука

1. Покажите строку cmdline, которая передается ядру при старте.

 

Если Вы про /etc/preinit , то вот он:

#!/bin/sh
# Copyright © 2006 OpenWrt.org
# Copyright © 2010 Vertical Communications

[ -z "$PREINIT" ] && exec /sbin/init

export PATH=/usr/sbin:/usr/bin:/sbin:/bin

pi_ifname=
pi_ip=192.168.1.1
pi_broadcast=192.168.1.255
pi_netmask=255.255.255.0

fs_failsafe_ifname=
fs_failsafe_ip=192.168.1.1
fs_failsafe_broadcast=192.168.1.255
fs_failsafe_netmask=255.255.255.0

fs_failsafe_wait_timeout=2

pi_suppress_stderr="y"
pi_init_suppress_stderr="y"
pi_init_path="/usr/sbin:/usr/bin:/sbin:/bin"
pi_init_cmd="/sbin/init"

. /lib/functions.sh
. /lib/functions/preinit.sh
. /lib/functions/system.sh

boot_hook_init preinit_essential
boot_hook_init preinit_main
boot_hook_init failsafe
boot_hook_init initramfs
boot_hook_init preinit_mount_root

for pi_source_file in /lib/preinit/*; do
. $pi_source_file
done

boot_run_hook preinit_essential

pi_mount_skip_next=false
pi_jffs2_mount_success=false
pi_failsafe_net_message=false

boot_run_hook preinit_main

 

Проблемы начались после того, как я обновил и пересобрал транк(было ядро 4.15, стало 4.16). Со старым образом rootfs.img работает, но это неудобно использовать ядро из одного билда, а фс от другого

Построчно сравнил логи рабочей и нерабочей системы, вот отличающиеся строки рабочей(разница в PID да незначительная объемах занятой оперативки):

[    0.000000] Memory: 123196K/131072K available (4730K kernel code, 172K rwdata, 1316K rodata, 152K init, 211K bss, 7876K reserved, 0K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]       .text : 0xc0008000 - 0xc05efd6c   (6048 kB)
[    0.000000]       .init : 0xc05f0000 - 0xc0616000   ( 152 kB)
[    0.000000]       .data : 0xc0616000 - 0xc064115c   ( 173 kB)
[    0.000000]        .bss : 0xc064115c - 0xc0675d8c   ( 212 kB)
[    0.302558] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    0.303289] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.

[    4.067113] ubi0: background thread "ubi_bgt0d" started, PID 419
[    4.081556] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 421
[    4.267277] Freeing unused kernel memory: 152K (c05f0000 - c0616000)
[    4.612754] init: Console is alive

Edited by vgovseychuk

Share this post


Link to post
Share on other sites

ядро же четко вам пишет

Attempted to kill init!

init - это программа которая должна запустить всё остальное, и при этом сама не должна завершаться, пока не потребуется перезагрузка/выключение.

если выполнение init закончится - будет kernel panic с таким же сообщением.

 

стоит начать с проверки типов файлов, и под ту ли платформу вы их пересобрали. Может бинарники сравнить.

но лучше в командной строке набрать

file /bin/sh /sbin/init

и проверить, вдруг он вам напишет

ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),

вместо вашего ARM-а

 

Share this post


Link to post
Share on other sites
стоит начать с проверки типов файлов, и под ту ли платформу вы их пересобрали. Может бинарники сравнить.

но лучше в командной строке набрать

file /bin/sh /sbin/init

и проверить, вдруг он вам напишет

ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),

вместо вашего ARM-а

 

v@v-Ubuntu:~/wrt2801/rootfs$ file ./bin/sh ./sbin/init
./bin/sh:    symbolic link to `busybox' 
./sbin/init: ELF 32-bit LSB  executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), corrupted section header size

Это Вам говорит о чем-нибудь?

Edited by vgovseychuk

Share this post


Link to post
Share on other sites

Проверьте версию компилятора. Может быть ядро собирали gnueabi-gcc, а все остальное gnueabihf-gcc. Или наоборот.

Share this post


Link to post
Share on other sites

Короче, проблема такая же, как у этих ребят: https://dev.openwrt.org/ticket/19896

Какой-то баг в транке. Откатился на десяток версий назад (на 48450) и все заработало.

Вывод: пользуйтесь стабильными релизами. Но мне принципиально нужно было 4-е ядро, поэтому такой вот воркэраунд.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this