1 Event Tracing 2 3 Documentation written by Theodore Ts'o 4 Updated by Li Zefan and Tom Zanussi 5 61. Introduction 7=============== 8 9Tracepoints (see Documentation/trace/tracepoints.txt) can be used 10without creating custom kernel modules to register probe functions 11using the event tracing infrastructure. 12 13Not all tracepoints can be traced using the event tracing system; 14the kernel developer must provide code snippets which define how the 15tracing information is saved into the tracing buffer, and how the 16tracing information should be printed. 17 182. Using Event Tracing 19====================== 20 212.1 Via the 'set_event' interface 22--------------------------------- 23 24The events which are available for tracing can be found in the file 25/sys/kernel/debug/tracing/available_events. 26 27To enable a particular event, such as 'sched_wakeup', simply echo it 28to /sys/kernel/debug/tracing/set_event. For example: 29 30 # echo sched_wakeup >> /sys/kernel/debug/tracing/set_event 31 32[ Note: '>>' is necessary, otherwise it will firstly disable 33 all the events. ] 34 35To disable an event, echo the event name to the set_event file prefixed 36with an exclamation point: 37 38 # echo '!sched_wakeup' >> /sys/kernel/debug/tracing/set_event 39 40To disable all events, echo an empty line to the set_event file: 41 42 # echo > /sys/kernel/debug/tracing/set_event 43 44To enable all events, echo '*:*' or '*:' to the set_event file: 45 46 # echo *:* > /sys/kernel/debug/tracing/set_event 47 48The events are organized into subsystems, such as ext4, irq, sched, 49etc., and a full event name looks like this: <subsystem>:<event>. The 50subsystem name is optional, but it is displayed in the available_events 51file. All of the events in a subsystem can be specified via the syntax 52"<subsystem>:*"; for example, to enable all irq events, you can use the 53command: 54 55 # echo 'irq:*' > /sys/kernel/debug/tracing/set_event 56 572.2 Via the 'enable' toggle 58--------------------------- 59 60The events available are also listed in /sys/kernel/debug/tracing/events/ hierarchy 61of directories. 62 63To enable event 'sched_wakeup': 64 65 # echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable 66 67To disable it: 68 69 # echo 0 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable 70 71To enable all events in sched subsystem: 72 73 # echo 1 > /sys/kernel/debug/tracing/events/sched/enable 74 75To enable all events: 76 77 # echo 1 > /sys/kernel/debug/tracing/events/enable 78 79When reading one of these enable files, there are four results: 80 81 0 - all events this file affects are disabled 82 1 - all events this file affects are enabled 83 X - there is a mixture of events enabled and disabled 84 ? - this file does not affect any event 85 862.3 Boot option 87--------------- 88 89In order to facilitate early boot debugging, use boot option: 90 91 trace_event=[event-list] 92 93event-list is a comma separated list of events. See section 2.1 for event 94format. 95 963. Defining an event-enabled tracepoint 97======================================= 98 99See The example provided in samples/trace_events 100 1014. Event formats 102================ 103 104Each trace event has a 'format' file associated with it that contains 105a description of each field in a logged event. This information can 106be used to parse the binary trace stream, and is also the place to 107find the field names that can be used in event filters (see section 5). 108 109It also displays the format string that will be used to print the 110event in text mode, along with the event name and ID used for 111profiling. 112 113Every event has a set of 'common' fields associated with it; these are 114the fields prefixed with 'common_'. The other fields vary between 115events and correspond to the fields defined in the TRACE_EVENT 116definition for that event. 117 118Each field in the format has the form: 119 120 field:field-type field-name; offset:N; size:N; 121 122where offset is the offset of the field in the trace record and size 123is the size of the data item, in bytes. 124 125For example, here's the information displayed for the 'sched_wakeup' 126event: 127 128# cat /sys/kernel/debug/tracing/events/sched/sched_wakeup/format 129 130name: sched_wakeup 131ID: 60 132format: 133 field:unsigned short common_type; offset:0; size:2; 134 field:unsigned char common_flags; offset:2; size:1; 135 field:unsigned char common_preempt_count; offset:3; size:1; 136 field:int common_pid; offset:4; size:4; 137 field:int common_tgid; offset:8; size:4; 138 139 field:char comm[TASK_COMM_LEN]; offset:12; size:16; 140 field:pid_t pid; offset:28; size:4; 141 field:int prio; offset:32; size:4; 142 field:int success; offset:36; size:4; 143 field:int cpu; offset:40; size:4; 144 145print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid, 146 REC->prio, REC->success, REC->cpu 147 148This event contains 10 fields, the first 5 common and the remaining 5 149event-specific. All the fields for this event are numeric, except for 150'comm' which is a string, a distinction important for event filtering. 151 1525. Event filtering 153================== 154 155Trace events can be filtered in the kernel by associating boolean 156'filter expressions' with them. As soon as an event is logged into 157the trace buffer, its fields are checked against the filter expression 158associated with that event type. An event with field values that 159'match' the filter will appear in the trace output, and an event whose 160values don't match will be discarded. An event with no filter 161associated with it matches everything, and is the default when no 162filter has been set for an event. 163 1645.1 Expression syntax 165--------------------- 166 167A filter expression consists of one or more 'predicates' that can be 168combined using the logical operators '&&' and '||'. A predicate is 169simply a clause that compares the value of a field contained within a 170logged event with a constant value and returns either 0 or 1 depending 171on whether the field value matched (1) or didn't match (0): 172 173 field-name relational-operator value 174 175Parentheses can be used to provide arbitrary logical groupings and 176double-quotes can be used to prevent the shell from interpreting 177operators as shell metacharacters. 178 179The field-names available for use in filters can be found in the 180'format' files for trace events (see section 4). 181 182The relational-operators depend on the type of the field being tested: 183 184The operators available for numeric fields are: 185 186==, !=, <, <=, >, >= 187 188And for string fields they are: 189 190==, != 191 192Currently, only exact string matches are supported. 193 194Currently, the maximum number of predicates in a filter is 16. 195 1965.2 Setting filters 197------------------- 198 199A filter for an individual event is set by writing a filter expression 200to the 'filter' file for the given event. 201 202For example: 203 204# cd /sys/kernel/debug/tracing/events/sched/sched_wakeup 205# echo "common_preempt_count > 4" > filter 206 207A slightly more involved example: 208 209# cd /sys/kernel/debug/tracing/events/signal/signal_generate 210# echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter 211 212If there is an error in the expression, you'll get an 'Invalid 213argument' error when setting it, and the erroneous string along with 214an error message can be seen by looking at the filter e.g.: 215 216# cd /sys/kernel/debug/tracing/events/signal/signal_generate 217# echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter 218-bash: echo: write error: Invalid argument 219# cat filter 220((sig >= 10 && sig < 15) || dsig == 17) && comm != bash 221^ 222parse_error: Field not found 223 224Currently the caret ('^') for an error always appears at the beginning of 225the filter string; the error message should still be useful though 226even without more accurate position info. 227 2285.3 Clearing filters 229-------------------- 230 231To clear the filter for an event, write a '0' to the event's filter 232file. 233 234To clear the filters for all events in a subsystem, write a '0' to the 235subsystem's filter file. 236 2375.3 Subsystem filters 238--------------------- 239 240For convenience, filters for every event in a subsystem can be set or 241cleared as a group by writing a filter expression into the filter file 242at the root of the subsystem. Note however, that if a filter for any 243event within the subsystem lacks a field specified in the subsystem 244filter, or if the filter can't be applied for any other reason, the 245filter for that event will retain its previous setting. This can 246result in an unintended mixture of filters which could lead to 247confusing (to the user who might think different filters are in 248effect) trace output. Only filters that reference just the common 249fields can be guaranteed to propagate successfully to all events. 250 251Here are a few subsystem filter examples that also illustrate the 252above points: 253 254Clear the filters on all events in the sched subsystem: 255 256# cd /sys/kernel/debug/tracing/events/sched 257# echo 0 > filter 258# cat sched_switch/filter 259none 260# cat sched_wakeup/filter 261none 262 263Set a filter using only common fields for all events in the sched 264subsystem (all events end up with the same filter): 265 266# cd /sys/kernel/debug/tracing/events/sched 267# echo common_pid == 0 > filter 268# cat sched_switch/filter 269common_pid == 0 270# cat sched_wakeup/filter 271common_pid == 0 272 273Attempt to set a filter using a non-common field for all events in the 274sched subsystem (all events but those that have a prev_pid field retain 275their old filters): 276 277# cd /sys/kernel/debug/tracing/events/sched 278# echo prev_pid == 0 > filter 279# cat sched_switch/filter 280prev_pid == 0 281# cat sched_wakeup/filter 282common_pid == 0 283