可观测性 kprobe trace
介绍
kprobe event与基于追踪点的事件有点类似，它是基于kprobe。它可以使用kprobe探测任何函数(除了NOKPROBE_SYMBOL标记的黑名单函数之外)。
同样它需要内核编译的时候配置才能启用该功能，配置项为CONFIG_KPROBE_EVENTS=y
与ftrace其它的event tracer一样通过debugfs文件系统与用户交互，
- 通过
/sys/kernel/tracing/kprobe_events 或 /sys/kernel/tracing/dynamic_events添加 Kprobes events。 - 通过
/sys/kernel/tracing/events/kprobes/EVENT/enable启用/停止它。 - 通过
/sys/kernel/debug/tracing/trace查看事件。
原理
工作原理参考 可观测性 kprobe
event 语法格式
p[:[GRP/][EVENT]] [MOD:]SYM[+offs]|MEMADDR [FETCHARGS] : Set a probe
r[MAXACTIVE][:[GRP/][EVENT]] [MOD:]SYM[+0] [FETCHARGS] : Set a return probe
p[:[GRP/][EVENT]] [MOD:]SYM[+0]%return [FETCHARGS] : Set a return probe
-:[GRP/][EVENT] : Clear a probe
GRP : Group name. If omitted, use "kprobes" for it.
EVENT : Event name. If omitted, the event name is generated
based on SYM+offs or MEMADDR.
MOD : Module name which has given SYM.
SYM[+offs] : Symbol+offset where the probe is inserted.
SYM%return : Return address of the symbol
MEMADDR : Address where the probe is inserted.
MAXACTIVE : Maximum number of instances of the specified function that
can be probed simultaneously, or 0 for the default value
as defined in Documentation/trace/kprobes.rst section 1.3.1.
FETCHARGS : Arguments. Each probe can have up to 128 args.
%REG : Fetch register REG
@ADDR : Fetch memory at ADDR (ADDR should be in kernel)
@SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbol)
$stackN : Fetch Nth entry of stack (N >= 0)
$stack : Fetch stack address.
$argN : Fetch the Nth function argument. (N >= 1) (\*1)
$retval : Fetch return value.(\*2)
$comm : Fetch current task comm.
+|-[u]OFFS(FETCHARG) : Fetch memory at FETCHARG +|- OFFS address.(\*3)(\*4)
\IMM : Store an immediate value to the argument.
NAME=FETCHARG : Set NAME as the argument name of FETCHARG.
FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types
(u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types
(x8/x16/x32/x64), "char", "string", "ustring", "symbol", "symstr"
and bitfield are supported.
(\*1) only for the probe on function entry (offs == 0).
(\*2) only for return probe.
(\*3) this is useful for fetching a field of data structures.
(\*4) "u" means user-space dereference. See :ref:`user_mem_access`.
FETCHARGS 指定 kprobe event 数据显示的格式
- s、u: 分别表示signed、unsigned；
- x：表示十六进制；
- 数字：s、u十进制，x十六进制，如果没有类型转换，则使用“x32”或“x64”取决于体系结构（例如，x86-32 使用 x32，x86-64 使用 x64）；
- 字符串：在内存中读取一个“null-terminated”的字符串；
- 对于$comm，默认类型是“string”；任何其他类型均无效。
清除event
如果有event处于enable状态无法清除，会报错误，-bash: /sys/kernel/debug/tracing/kprobe_events: Device or resource busy需要先disable event。
清除所有的event
echo > /sys/kernel/debug/tracing/kprobe_events清除某个evnet echo -:ping » kprobe_events
# echo 0 > events/kprobes/ping/enable
# cat kprobe_events
p:kprobes/ping __icmp_send
p:kprobes/icmp icmp_rcv
# echo -:ping >> kprobe_events
# cat kprobe_events
p:kprobes/icmp icmp_rcv
实验
监控 icmp_rcv event
# cd /sys/kernel/debug/tracing
echo 'p:icmp icmp_rcv' >> kprobe_events
# echo 1 > events/kprobes/icmp/enable
ping 127.0.0.1
查看事件
# cat trace
# tracer: nop
#
# entries-in-buffer/entries-written: 6/6 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
<...>-151418 [002] ..s1 170851.977121: icmp: (icmp_rcv+0x0/0x3e0)
<...>-151418 [002] ..s1 170851.977128: icmp: (icmp_rcv+0x0/0x3e0)
<...>-151418 [002] ..s1 170852.995083: icmp: (icmp_rcv+0x0/0x3e0)
<...>-151418 [002] ..s1 170852.995106: icmp: (icmp_rcv+0x0/0x3e0)
<...>-151418 [002] ..s1 170854.017821: icmp: (icmp_rcv+0x0/0x3e0)
<...>-151418 [002] ..s1 170854.017848: icmp: (icmp_rcv+0x0/0x3e0)
参考
https://docs.kernel.org/trace/kprobetrace.html
欢迎大家转发分享。未经授权，严禁任何复制、转载、摘编或以其它方式进行使用，转载须注明来自eBPFLAB并附上本文链接。如果有侵犯到您权益的地方，请及时联系我删除。