Add a wakeup monitor to detect a lower-priority task waking up a
higher-priority task.
The rtapp/sleep monitor already detects this. However, that monitor
triggers an error in the context of the wakee task and user only gets
the stacktrace of that task. It is also extremely useful to get the
stacktrace of the waker task, which this monitor offers. In other
words, this monitor complements the rtapp/sleep monitor.
Signed-off-by: Nam Cao <[email protected]>
Reviewed-by: Gabriele Monaco <[email protected]>
Link: https://lore.kernel.org/r/ba5658fa13e49ada466b84a2c211f233037180b5.1781852967.git.namcao@linutronix.de
Signed-off-by: Gabriele Monaco <[email protected]>
The rtapp/sleep monitor's primary purpose is detecting common mistakes
with user-space real-time design. Monitoring real-time issues with
kernel threads is a bonus.
However, accomodating kernel threads complicates the monitor due to
the edge cases which is seen by the monitor as lower-priority task
waking higher-priority task:
- kthread_stop() wakes up the task in order to stop it.
- The rcu thread and migration thread can be woken by any task.
- The ktimerd thread is woken near the end of irq_exit_rcu(), where
the preempt counter is "broken" and falsely says this is task
context. This requires the monitor to use the hardirq_context flag
instead of the preempt counter.
Beside complicating the monitor, the final case also requires enabling
CONFIG_TRACE_IRQFLAGS (so that "hardirq_context" can be used). This
adds overhead to the kernel even when the monitor is not active. This
may be an obstacle to enabling this monitor in distros' kernels.
Furthermore, kernel threads usually are started before the monitor is
enabled. Consequently, the threads' states (i.o.w. the monitor's
atomic propositions for the threads) are not fully known to the
monitor. As a result, the kernel threads mostly cannot be monitored.
Overall, the downsides of accomodating kernel threads outweights the
benefits. Thus, exclude kernel threads to simplify the monitor.
Signed-off-by: Nam Cao <[email protected]>
Reviewed-by: Gabriele Monaco <[email protected]>
Link: https://lore.kernel.org/r/eec2ca5224bcdacc45b8e1eb2f0e68109e1cae7a.1781852967.git.namcao@linutronix.de
Signed-off-by: Gabriele Monaco <[email protected]>
The rtapp/sleep monitor detects real-time tasks which go to sleep in an
real-time-unsafe manner. If this happen, the monitor triggers a trace event
in the sched_wakeup tracepoint's handler.
However, the invoking context of that trace event is not the most
informative, because of the stack trace of that event is the wakeup's code
path which is not very helpful:
74.669317: rv:error_sleep: condvar[254]: violation detected
ltl_validate+0x345 ([kernel.kallsyms])
handle_sched_wakeup+0x34 ([kernel.kallsyms])
ttwu_do_activate+0xff ([kernel.kallsyms])
sched_ttwu_pending+0x104 ([kernel.kallsyms])
__flush_smp_call_function_queue+0x15b ([kernel.kallsyms])
__sysvec_call_function_single+0x18 ([kernel.kallsyms])
sysvec_call_function_single+0x66 ([kernel.kallsyms])
asm_sysvec_call_function_single+0x1a ([kernel.kallsyms])
pv_native_safe_halt+0xf ([kernel.kallsyms])
default_idle+0x9 ([kernel.kallsyms])
default_idle_call+0x33 ([kernel.kallsyms])
do_idle+0x234 ([kernel.kallsyms])
cpu_startup_entry+0x24 ([kernel.kallsyms])
start_secondary+0xf8 ([kernel.kallsyms])
common_startup_64+0x13e ([kernel.kallsyms])
What would be much more valuable is the stack trace of the task itself.
Instead of using the sched_wakeup tracepoint, use the sched_exit
tracepoint. This makes the event happen in the task's context, making
the stack trace far more informative for user:
rv:error_sleep: condvar[254]: violation detected
ltl_validate+0x345 ([kernel.kallsyms])
handle_sched_exit+0x39 ([kernel.kallsyms])
__schedule+0x80f ([kernel.kallsyms])
schedule+0x22 ([kernel.kallsyms])
futex_do_wait+0x33 ([kernel.kallsyms])
__futex_wait+0x8c ([kernel.kallsyms])
futex_wait+0x73 ([kernel.kallsyms])
do_futex+0xc6 ([kernel.kallsyms])
__x64_sys_futex+0x121 ([kernel.kallsyms])
do_syscall_64+0xf3 ([kernel.kallsyms])
entry_SYSCALL_64_after_hwframe+0x77 ([kernel.kallsyms])
__futex_abstimed_wait_common64+0xc6 (inlined)
__futex_abstimed_wait_common+0xc6 (/usr/lib/x86_64-linux-gnu/libc.so.6)
Signed-off-by: Nam Cao <[email protected]>
Reviewed-by: Gabriele Monaco <[email protected]>
Link: https://lore.kernel.org/r/d97b4b5c476e5792b6875ec9bbf8dc214f999516.1781852967.git.namcao@linutronix.de
Signed-off-by: Gabriele Monaco <[email protected]>
Add the deadline monitors collection to validate the deadline scheduler,
both for deadline tasks and servers.
The currently implemented monitors are:
* nomiss:
validate dl entities run to completion before their deadiline
Reviewed-by: Nam Cao <[email protected]>
Reviewed-by: Juri Lelli <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Gabriele Monaco <[email protected]>
The opid monitor validates that wakeup and need_resched events only
occur with interrupts and preemption disabled by following the
preemptirq tracepoints.
As reported in [1], those tracepoints might be inaccurate in some
situations (e.g. NMIs).
Since the monitor doesn't validate other ordering properties, remove the
dependency on preemptirq tracepoints and convert the monitor to a hybrid
automaton to validate the constraint during event handling.
This makes the monitor more robust by also removing the workaround for
interrupts missing the preemption tracepoints, which was working on
PREEMPT_RT only and allows the monitor to be built on kernels without
the preemptirqs tracepoints.
[1] - https://lore.kernel.org/lkml/[email protected]
Reviewed-by: Nam Cao <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Gabriele Monaco <[email protected]>
Add a monitor for checking that real-time tasks do not go to sleep in a
manner that may cause undesirable latency.
Also change
RV depends on TRACING
to
RV select TRACING
to avoid the following recursive dependency:
error: recursive dependency detected!
symbol TRACING is selected by PREEMPTIRQ_TRACEPOINTS
symbol PREEMPTIRQ_TRACEPOINTS depends on TRACE_IRQFLAGS
symbol TRACE_IRQFLAGS is selected by RV_MON_SLEEP
symbol RV_MON_SLEEP depends on RV
symbol RV depends on TRACING
Cc: John Ogness <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Mathieu Desnoyers <[email protected]>
Link: https://lore.kernel.org/75bc5bcc741d153aa279c95faf778dff35c5c8ad.1752088709.git.namcao@linutronix.de
Reviewed-by: Gabriele Monaco <[email protected]>
Signed-off-by: Nam Cao <[email protected]>
Signed-off-by: Steven Rostedt (Google) <[email protected]>
Add 3 per-cpu monitors as part of the sched model:
* scpd: schedule called with preemption disabled
Monitor to ensure schedule is called with preemption disabled
* snep: schedule does not enable preempt
Monitor to ensure schedule does not enable preempt
* sncid: schedule not called with interrupt disabled
Monitor to ensure schedule is not called with interrupt disabled
To: Ingo Molnar <[email protected]>
To: Peter Zijlstra <[email protected]>
Cc: Juri Lelli <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: John Kacur <[email protected]>
Cc: Clark Williams <[email protected]>
Link: https://lore.kernel.org/[email protected]
Signed-off-by: Gabriele Monaco <[email protected]>
Signed-off-by: Steven Rostedt (Google) <[email protected]>
Nam Cao