1 2Introduction 3============ 4 5This document describes how to use the dynamic debug (ddebug) feature. 6 7Dynamic debug is designed to allow you to dynamically enable/disable kernel 8code to obtain additional kernel information. Currently, if 9CONFIG_DYNAMIC_DEBUG is set, then all pr_debug()/dev_debug() calls can be 10dynamically enabled per-callsite. 11 12Dynamic debug has even more useful features: 13 14 * Simple query language allows turning on and off debugging statements by 15 matching any combination of: 16 17 - source filename 18 - function name 19 - line number (including ranges of line numbers) 20 - module name 21 - format string 22 23 * Provides a debugfs control file: <debugfs>/dynamic_debug/control which can be 24 read to display the complete list of known debug statements, to help guide you 25 26Controlling dynamic debug Behaviour 27=============================== 28 29The behaviour of pr_debug()/dev_debug()s are controlled via writing to a 30control file in the 'debugfs' filesystem. Thus, you must first mount the debugfs 31filesystem, in order to make use of this feature. Subsequently, we refer to the 32control file as: <debugfs>/dynamic_debug/control. For example, if you want to 33enable printing from source file 'svcsock.c', line 1603 you simply do: 34 35nullarbor:~ # echo 'file svcsock.c line 1603 +p' > 36 <debugfs>/dynamic_debug/control 37 38If you make a mistake with the syntax, the write will fail thus: 39 40nullarbor:~ # echo 'file svcsock.c wtf 1 +p' > 41 <debugfs>/dynamic_debug/control 42-bash: echo: write error: Invalid argument 43 44Viewing Dynamic Debug Behaviour 45=========================== 46 47You can view the currently configured behaviour of all the debug statements 48via: 49 50nullarbor:~ # cat <debugfs>/dynamic_debug/control 51# filename:lineno [module]function flags format 52/usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:323 [svcxprt_rdma]svc_rdma_cleanup - "SVCRDMA Module Removed, deregister RPC RDMA transport\012" 53/usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:341 [svcxprt_rdma]svc_rdma_init - "\011max_inline : %d\012" 54/usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:340 [svcxprt_rdma]svc_rdma_init - "\011sq_depth : %d\012" 55/usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:338 [svcxprt_rdma]svc_rdma_init - "\011max_requests : %d\012" 56... 57 58 59You can also apply standard Unix text manipulation filters to this 60data, e.g. 61 62nullarbor:~ # grep -i rdma <debugfs>/dynamic_debug/control | wc -l 6362 64 65nullarbor:~ # grep -i tcp <debugfs>/dynamic_debug/control | wc -l 6642 67 68Note in particular that the third column shows the enabled behaviour 69flags for each debug statement callsite (see below for definitions of the 70flags). The default value, no extra behaviour enabled, is "-". So 71you can view all the debug statement callsites with any non-default flags: 72 73nullarbor:~ # awk '$3 != "-"' <debugfs>/dynamic_debug/control 74# filename:lineno [module]function flags format 75/usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svcsock.c:1603 [sunrpc]svc_send p "svc_process: st_sendto returned %d\012" 76 77 78Command Language Reference 79========================== 80 81At the lexical level, a command comprises a sequence of words separated 82by whitespace characters. Note that newlines are treated as word 83separators and do *not* end a command or allow multiple commands to 84be done together. So these are all equivalent: 85 86nullarbor:~ # echo -c 'file svcsock.c line 1603 +p' > 87 <debugfs>/dynamic_debug/control 88nullarbor:~ # echo -c ' file svcsock.c line 1603 +p ' > 89 <debugfs>/dynamic_debug/control 90nullarbor:~ # echo -c 'file svcsock.c\nline 1603 +p' > 91 <debugfs>/dynamic_debug/control 92nullarbor:~ # echo -n 'file svcsock.c line 1603 +p' > 93 <debugfs>/dynamic_debug/control 94 95Commands are bounded by a write() system call. If you want to do 96multiple commands you need to do a separate "echo" for each, like: 97 98nullarbor:~ # echo 'file svcsock.c line 1603 +p' > /proc/dprintk ;\ 99> echo 'file svcsock.c line 1563 +p' > /proc/dprintk 100 101or even like: 102 103nullarbor:~ # ( 104> echo 'file svcsock.c line 1603 +p' ;\ 105> echo 'file svcsock.c line 1563 +p' ;\ 106> ) > /proc/dprintk 107 108At the syntactical level, a command comprises a sequence of match 109specifications, followed by a flags change specification. 110 111command ::= match-spec* flags-spec 112 113The match-spec's are used to choose a subset of the known dprintk() 114callsites to which to apply the flags-spec. Think of them as a query 115with implicit ANDs between each pair. Note that an empty list of 116match-specs is possible, but is not very useful because it will not 117match any debug statement callsites. 118 119A match specification comprises a keyword, which controls the attribute 120of the callsite to be compared, and a value to compare against. Possible 121keywords are: 122 123match-spec ::= 'func' string | 124 'file' string | 125 'module' string | 126 'format' string | 127 'line' line-range 128 129line-range ::= lineno | 130 '-'lineno | 131 lineno'-' | 132 lineno'-'lineno 133// Note: line-range cannot contain space, e.g. 134// "1-30" is valid range but "1 - 30" is not. 135 136lineno ::= unsigned-int 137 138The meanings of each keyword are: 139 140func 141 The given string is compared against the function name 142 of each callsite. Example: 143 144 func svc_tcp_accept 145 146file 147 The given string is compared against either the full 148 pathname or the basename of the source file of each 149 callsite. Examples: 150 151 file svcsock.c 152 file /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svcsock.c 153 154module 155 The given string is compared against the module name 156 of each callsite. The module name is the string as 157 seen in "lsmod", i.e. without the directory or the .ko 158 suffix and with '-' changed to '_'. Examples: 159 160 module sunrpc 161 module nfsd 162 163format 164 The given string is searched for in the dynamic debug format 165 string. Note that the string does not need to match the 166 entire format, only some part. Whitespace and other 167 special characters can be escaped using C octal character 168 escape \ooo notation, e.g. the space character is \040. 169 Alternatively, the string can be enclosed in double quote 170 characters (") or single quote characters ('). 171 Examples: 172 173 format svcrdma: // many of the NFS/RDMA server dprintks 174 format readahead // some dprintks in the readahead cache 175 format nfsd:\040SETATTR // one way to match a format with whitespace 176 format "nfsd: SETATTR" // a neater way to match a format with whitespace 177 format 'nfsd: SETATTR' // yet another way to match a format with whitespace 178 179line 180 The given line number or range of line numbers is compared 181 against the line number of each dprintk() callsite. A single 182 line number matches the callsite line number exactly. A 183 range of line numbers matches any callsite between the first 184 and last line number inclusive. An empty first number means 185 the first line in the file, an empty line number means the 186 last number in the file. Examples: 187 188 line 1603 // exactly line 1603 189 line 1600-1605 // the six lines from line 1600 to line 1605 190 line -1605 // the 1605 lines from line 1 to line 1605 191 line 1600- // all lines from line 1600 to the end of the file 192 193The flags specification comprises a change operation followed 194by one or more flag characters. The change operation is one 195of the characters: 196 197- 198 remove the given flags 199 200+ 201 add the given flags 202 203= 204 set the flags to the given flags 205 206The flags are: 207 208p 209 Causes a printk() message to be emitted to dmesg 210 211Note the regexp ^[-+=][scp]+$ matches a flags specification. 212Note also that there is no convenient syntax to remove all 213the flags at once, you need to use "-psc". 214 215Examples 216======== 217 218// enable the message at line 1603 of file svcsock.c 219nullarbor:~ # echo -n 'file svcsock.c line 1603 +p' > 220 <debugfs>/dynamic_debug/control 221 222// enable all the messages in file svcsock.c 223nullarbor:~ # echo -n 'file svcsock.c +p' > 224 <debugfs>/dynamic_debug/control 225 226// enable all the messages in the NFS server module 227nullarbor:~ # echo -n 'module nfsd +p' > 228 <debugfs>/dynamic_debug/control 229 230// enable all 12 messages in the function svc_process() 231nullarbor:~ # echo -n 'func svc_process +p' > 232 <debugfs>/dynamic_debug/control 233 234// disable all 12 messages in the function svc_process() 235nullarbor:~ # echo -n 'func svc_process -p' > 236 <debugfs>/dynamic_debug/control 237 238// enable messages for NFS calls READ, READLINK, READDIR and READDIR+. 239nullarbor:~ # echo -n 'format "nfsd: READ" +p' > 240 <debugfs>/dynamic_debug/control 241