[_lexa_ 0]

Всем добрый день!

Работаю со следующей системой: SoC - am3358, linux-rt-4.19.94-rt39-ga242ccf3f1 из SDK 06_03_00_106

Столкнулся со следующей проблемой - есть приложение, в котором работают, скажем, 8 потоков, 1 из которых RT, остальные неRT. В RT-потоке создается и запускается таймер с уведомлением о срабвтывании с помощью потока (timer_create(), timer_settime()), Также из RT-потока с периодом от 0,1 до 2 с производится обновление времени срабатывания этого таймера. Через какое-то время работы приложения процессор начинает сначала перегружаться, затем вовсе зависает. В то время, когда процессор перегружен, получилось снять такой лог:

Spoiler

[   67.012111] sched: RT throttling activated (curr: pid 654, comm My_app)
[   86.392105] rcu: INFO: rcu_preempt self-detected stall on CPU
[   86.392131] rcu:     0-...!: (2100 ticks this GP) idle=24e/1/0x40000002 softirq=0/0 fqs=0
[   86.392133] rcu:      (t=2100 jiffies g=8889 q=47)
[   86.392147] rcu: rcu_preempt kthread starved for 2100 jiffies! g8889 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
[   86.392150] rcu: RCU grace-period kthread stack dump:
[   86.392155] rcu_preempt     I    0    11      2 0x00000000
[   86.392165] Backtrace:
[   86.392202] [<c08d7c90>] (__schedule) from [<c08d8390>] (schedule+0x68/0x12c)
[   86.392214]  r10:c0d503e6 r9:c0d17f00 r8:c0d17ec0 r7:c0d05888 r6:cf073ed4 r5:ffffe000
[   86.392217]  r4:cf06c700
[   86.392230] [<c08d8328>] (schedule) from [<c08db1e4>] (schedule_timeout+0x16c/0x29c)
[   86.392235]  r5:c0d17f00 r4:ffffa45c
[   86.392256] [<c08db078>] (schedule_timeout) from [<c017eb88>] (rcu_gp_kthread+0x61c/0xce4)
[   86.392265]  r9:c0d17774 r8:00000001 r7:c0d17ec0 r6:cf072000 r5:c0d17634 r4:00000001
[   86.392277] [<c017e56c>] (rcu_gp_kthread) from [<c0147d58>] (kthread+0x158/0x160)
[   86.392281]  r7:cf072000
[   86.392291] [<c0147c00>] (kthread) from [<c01010f0>] (ret_from_fork+0x14/0x24)
[   86.392297] Exception stack(0xcf073fb0 to 0xcf073ff8)
[   86.392305] 3fa0:                                     00000000 00000000 00000000 00000000
[   86.392315] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   86.392323] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[   86.392333]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0147c00
[   86.392336]  r4:cf02ec00
[   86.392345] Task dump for CPU 0:
[   86.392348] My_app             R  running task        0   654    639 0x00000000
[   86.392356] Backtrace:
[   86.392368] [<c010c7f0>] (dump_backtrace) from [<c010cb60>] (show_stack+0x18/0x1c)
[   86.392376]  r7:c0d17634 r6:0000027f r5:c0d05888 r4:c900d500
[   86.392392] [<c010cb48>] (show_stack) from [<c0152aa4>] (sched_show_task.part.2+0xe4/0x100)
[   86.392401] [<c01529c0>] (sched_show_task.part.2) from [<c0152bb4>] (dump_cpu_task+0x38/0x3c)
[   86.392408]  r6:c0d05954 r5:00000000 r4:c0d17634
[   86.392420] [<c0152b7c>] (dump_cpu_task) from [<c0180948>] (rcu_dump_cpu_stacks+0x90/0xdc)
[   86.392431] [<c01808b8>] (rcu_dump_cpu_stacks) from [<c017fdb8>] (rcu_check_callbacks+0x640/0x930)
[   86.392440]  r10:c0d17ec0 r9:c0d05948 r8:00000000 r7:c0d17634 r6:c0d17a04 r5:c0d17c88
[   86.392445]  r4:c0d17634 r3:bbecd04e
[   86.392459] [<c017f778>] (rcu_check_callbacks) from [<c0185e70>] (update_process_times+0x3c/0x6c)
[   86.392469]  r10:200d0193 r9:00000002 r8:00000000 r7:00000014 r6:00000000 r5:c900d500
[   86.392472]  r4:ffffe000
[   86.392490] [<c0185e34>] (update_process_times) from [<c0196248>] (tick_sched_handle+0x5c/0x60)
[   86.392497]  r7:00000014 r6:1d3ed5e6 r5:ce7d5ea0 r4:c0d19a80
[   86.392507] [<c01961ec>] (tick_sched_handle) from [<c0196564>] (tick_sched_timer+0x50/0xac)
[   86.392518] [<c0196514>] (tick_sched_timer) from [<c0186a60>] (__hrtimer_run_queues.constprop.3+0xe4/0x230)
[   86.392526]  r7:ffffe000 r6:c0d19040 r5:c0d19a80 r4:c0d19090
[   86.392536] [<c018697c>] (__hrtimer_run_queues.constprop.3) from [<c01872e8>] (hrtimer_interrupt+0x120/0x334)
[   86.392546]  r10:c0d503d5 r9:ffffffff r8:7fffffff r7:00000003 r6:200d0193 r5:ffffe000
[   86.392549]  r4:c0d19040
[   86.392570] [<c01871c8>] (hrtimer_interrupt) from [<c0119df4>] (omap2_gp_timer_interrupt+0x30/0x38)
[   86.392579]  r10:c0d503d5 r9:cf007100 r8:00000010 r7:ce7d5df0 r6:00000000 r5:c0d05888
[   86.392582]  r4:c0d0a900
[   86.392597] [<c0119dc4>] (omap2_gp_timer_interrupt) from [<c016dc30>] (__handle_irq_event_percpu+0x68/0x14c)
[   86.392607] [<c016dbc8>] (__handle_irq_event_percpu) from [<c016dd68>] (handle_irq_event_percpu+0x54/0xa4)
[   86.392616]  r10:00000000 r9:ce7d4000 r8:cf008000 r7:cf007100 r6:00000000 r5:c0d05888
[   86.392619]  r4:c0179e64
[   86.392628] [<c016dd14>] (handle_irq_event_percpu) from [<c016de1c>] (handle_irq_event+0x64/0xa4)
[   86.392636]  r8:cf008000 r7:00000000 r6:00000001 r5:00000000 r4:cf007100
[   86.392649] [<c016ddb8>] (handle_irq_event) from [<c01717a0>] (handle_level_irq+0xd0/0x180)
[   86.392653]  r5:00000000 r4:cf007100
[   86.392662] [<c01716d0>] (handle_level_irq) from [<c016cce0>] (generic_handle_irq+0x2c/0x3c)
[   86.392667]  r5:00000000 r4:c0d4ea44
[   86.392675] [<c016ccb4>] (generic_handle_irq) from [<c016d564>] (__handle_domain_irq+0x5c/0xb0)
[   86.392694] [<c016d508>] (__handle_domain_irq) from [<c042d7ec>] (omap_intc_handle_irq+0x3c/0x94)
[   86.392703]  r9:ce7d4000 r8:c0190108 r7:ce7d5ed4 r6:ffffffff r5:600d0013 r4:c0d83210
[   86.392714] [<c042d7b0>] (omap_intc_handle_irq) from [<c0101a0c>] (__irq_svc+0x6c/0xbc)
[   86.392718] Exception stack(0xce7d5ea0 to 0xce7d5ee8)
[   86.392729] 5ea0: 00000000 ce7d5f2c c900d500 00000001 c0d05888 00000000 ce7d5f2c 00000000
[   86.392739] 5ec0: c0190108 c9445010 00000000 ce7d5efc ce7d5f00 ce7d5ef0 c018fabc c0179e64
[   86.392744] 5ee0: 600d0013 ffffffff
[   86.392749]  r5:600d0013 r4:c0179e64
[   86.392765] [<c0179e3c>] (__rcu_read_lock) from [<c018fabc>] (__lock_timer+0x20/0xe4)
[   86.392776] [<c018fa9c>] (__lock_timer) from [<c0190f00>] (sys_timer_settime+0x11c/0x1b8)
[   86.392783]  r7:00000000 r6:c090392c r5:c9445000 r4:c0d05888
[   86.392792] [<c0190de4>] (sys_timer_settime) from [<c0101000>] (ret_fast_syscall+0x0/0x5c)
[   86.392796] Exception stack(0xce7d5fa8 to 0xce7d5ff0)
[   86.392805] 5fa0:                   00000000 b4bfcc0c 00000000 00000000 b4bfc8bc 00000000
[   86.392815] 5fc0: 00000000 b4bfcc0c b4bfd460 00000102 befffbf6 00000000 befffc80 b4bfcf9c
[   86.392822] 5fe0: 0056ac14 b4bfc8ac 004360d3 b6f7ecd4
[   86.392832]  r10:00000102 r9:ce7d4000 r8:c0101204 r7:00000102 r6:b4bfd460 r5:b4bfcc0c
[   86.392835]  r4:00000000

Судя по RCU stall detector:

Spoiler

    The following problems can result in RCU CPU stall warnings:

    ... A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might happen to preempt a low-priority task in the middle of an RCU read-side critical section. This is especially damaging if that low-priority task is not permitted to run on any other CPU, in which case the next RCU grace period can never complete, which will eventually cause the system to run out of memory and hang.

    ... A CPU-bound real-time task in a CONFIG_PREEMPT_RT kernel that is running at a higher priority than the RCU softirq threads. This will prevent RCU callbacks from ever being invoked, and in a CONFIG_PREEMPT_RCU kernel will further prevent RCU grace periods from ever completing. Either way, the system will eventually run out of memory and hang.

такая ситуация возможна при использовании механизма RCU в RT-потоках. Я так понимаю, при вызове timer_settime() и при срабоатывании таймера выполняется синхронизаация чтения/записи RCU. Я предполагаю, что наложились события срабатывания таймера и обновление времени срабатывания из RT-потока, в которых задействованы механизмы RCU (что-то вроде - драйвер таймера находился в критической секции, RT-поток вытеснил этот процесс и завис на ожидении grace period).

Пробовал выичтывать время до срабатывания таймера и если оно меньше 1 мс - ожидать срабатывания, не помогло. Помогает - если RT-поток сделать не RT потоком

Вопрос - как избежать подобных зависаний, при том, что поток, вызывающий обновления таймера, должен быть реального времени?

Изменено 29 августа, 2023 пользователем _lexa_