systemtap

Probe language

• The language describes an association of handler subroutines with probe points. 类似AWK的工作方式，绑定一些handle到probe points上面
• Probe points are abstract names given to identify a particular place in kernel/user code, or a particular event (timers, counters) that may occur at any time. （所谓探测点是用来识别内核或者是用户代码的一些位置，或者是某些事件的发生）
• Handlers are subroutines written in the script language, which are run whenever the probe points are hit.（而handler就是当探测点出发的时候执行的过程）
• The language resembles dtrace’s “D”, itself inspired by the old UNIX tool awk（语言名字叫做D引用dtrace,非常类似awk的语言）
  • These are simplified from C, lacking types, declarations, and most indirection 类似C但是没有类型和声明以及间接引用
  • but adding associative arrays and simplified string process-ing. 关联数组实现
  • The language includes some extensions to interoperate with the target software being instrumented, in order to refer to its data and program state. 增加扩展用来引用系统或者是应用程序特定的数据。

Elaboration

• Elaboration is a processing phase that analyzes the input script, and resolves any needed symbolic references to the kernel, user programs, or other any “tapsets”. （这个过程解析符号交叉引用，包括内核，用户程序以及tapset）
• Tapsets are libraries of script or C code used to extend the capability of a basic script（tapset类似library，有D实现也有C扩展）
• Elaboration is analogous to linking an object file with needed libraries, turn them into a self-contained executable.（类似与链接过程）
• References to kernel data such as function parameters, local and global variables, functions, source locations, all need to be resolved to actual run-time addresses.（对于引用kenel或者是可执行文件的符号都是在run之前就完成解析的）
  • This is most rigorously done by processing the DWARF debugging information emitted by the compiler, in the same way as an ordinary debugger would（通过处理DWARF debugging信息来完成的）
  • However, such debug data processing is transformed into an executable form ahead of time, so that during actual probe execution, no explicit decoding is necessary.（但是这些debug数据因为是静态的，所以可以在run之前完成）

Translation

• Once an entire set of probe functions is processed through the elaboration stage, they are translated to a quantity of C code.（生成C代码）
  • Each systemtap construct is expanded to a block of C that includes whatever locking and safety checks are necessary.
  • Control-flow constructs translate to include runaway-prevention logic.
  • Each variable shared amongst probes is mapped to an appropriate static declaration, and accesses are protected by locks.（全局变量生成static并且通过lock来保护）
  • Each group of local variables is placed into a synthetic call frame structure that keeps them off the tiny real kernel stacks.
• Probe handlers are wrapped by an interface function which uses whatever probe point registration API is appropriate. （所有的probe handler都被包装成为function然后注册到probe point回调，但是方式有所不同）
  • For location type probe points targeting the kernel, this generally uses kprobes.(如果是内核探测点的话，那么使用kprobe)
  • Where the target software is user-level, probe points would need to be inserted into specific processes’ executable segments, using a mechanism yet to be specified.（如果是用户程序探测点的话，那么需要修改进程内存）
• When complete, the generated C code is compiled, and linked with the runtime, into a stand-alone kernel module. For security reasons, the module may be cryptographically signed, so that it may be archived and later reused here, or on another computer without a compiler installed.(编译成为ko模块，并且签名做cache)

Execution

• To run the probes, the systemtap driver program simply loads the kernel module using insmod. 使用insmod安装ko模块
• The module will initialize itself, insert the probes, then sit back and let the probe handlers be triggered by the system to collect and pass data. It will eventually remove the probes at unload time.（模块初始化之后安装probe然后等待handle触发，在unload的时候会将probes全部移除）
• When a probe is hit, the associated handler routine takes over the processor, suspending the target software briefly. When all handlers for that probe point have been executed, the target program resumes.(probe hit之后handler会执行，等待所有的handler执行完成之后目标程序才开始运行，因此最好不用hold住handler)
• The probe run concludes when the user sends an interrupt to the driver, or when the probe script runs an exit primitive. (This primitive might simply send a SIGINT to the running user-level driver process.) 通过发起信号结束

A systemtap script file has the suffix “.stp”

A script file is a sequence of top-level constructs, of which there are three types: 下面这些元素组成

• probe definitions, 探针定义
• auxiliary function definitions, 辅助函数定义
• and global variable declarations. 全局变量
• These may occur in any order, and forward references are permitted. 可以向前引用

Multiple probe handlers may execute concurrently on a multiprocessor. Multiple probe definitions may end up referring to the same event or program location （多个probe handler可能会在多个CPU上同时执行，并且不同的probe def可能引用到program的相同位置，因此需要注意多线程问题）

A script may make references to an identifier defined elsewhere in library of script tapsets. Such a cross-reference causes the entire tapset file providing the definition to be merged into the elaborated script, as if it was simply concatenated. (如果引用其他script变量的话，那么elaboration阶段会将引用的script全部包含进来，简单地看就像是合并)

Fatal errors that occur during script execution cause a winddown of activity associated with the systemtap script, and an early abort. Running out of memory, dividing by zero, exceeding an operation count limit, calling too many nested functions, are just a few types of fatal errors（运行中如果出现问题的话会使得script提前中止）

Data collected from systemtap in the kernel must somehow be transmitted to userspace. This transport must have high performance and minimal performance impact on the monitored system. 在内核态收集的数据必须发送到用户态空间，这个传输过程必须满足高性能。