1<html><head><title>toybox source code walkthrough</title></head> 2<!--#include file="header.html" --> 3 4<p><h1><a name="style" /><a href="#style">Code style</a></h1></p> 5 6<p>The primary goal of toybox is _simple_ code. Keeping the code small is 7second, with speed and lots of features coming in somewhere after that. 8(For more on that, see the <a href=design.html>design</a> page.)</p> 9 10<p>A simple implementation usually takes up fewer lines of source code, 11meaning more code can fit on the screen at once, meaning the programmer can 12see more of it on the screen and thus keep more if in their head at once. 13This helps code auditing and thus reduces bugs. That said, sometimes being 14more explicit is preferable to being clever enough to outsmart yourself: 15don't be so terse your code is unreadable.</p> 16 17<p>Toybox has an actual coding style guide over on 18<a href=design.html#codestyle>the design page</a>, but in general we just 19want the code to be consistent.</p> 20 21<p><h1><a name="building" /><a href="#building">Building Toybox</a></h1></p> 22 23<p>Toybox is configured using the Kconfig language pioneered by the Linux 24kernel, and adopted by many other projects (uClibc, OpenEmbedded, etc). 25This generates a ".config" file containing the selected options, which 26controls which features are included when compiling toybox.</p> 27 28<p>Each configuration option has a default value. The defaults indicate the 29"maximum sane configuration", I.E. if the feature defaults to "n" then it 30either isn't complete or is a special-purpose option (such as debugging 31code) that isn't intended for general purpose use.</p> 32 33<p>For a more compact human-editable version .config files, you can use the 34<a href=http://landley.net/aboriginal/FAQ.html#dev_miniconfig>miniconfig</a> 35format.</p> 36 37<p>The standard build invocation is:</p> 38 39<ul> 40<li>make defconfig #(or menuconfig)</li> 41<li>make</li> 42<li>make install</li> 43</ul> 44 45<p>Type "make help" to see all available build options.</p> 46 47<p>The file "configure" contains a number of environment variable definitions 48which influence the build, such as specifying which compiler to use or where 49to install the resulting binaries. This file is included by the build, but 50accepts existing definitions of the environment variables, so it may be sourced 51or modified by the developer before building and the definitions exported 52to the environment will take precedence.</p> 53 54<p>(To clarify: ".config" lists the features selected by defconfig/menuconfig, 55I.E. "what to build", and "configure" describes the build and installation 56environment, I.E. "how to build it".)</p> 57 58<p>By default "make install" puts files in /usr/toybox. Adding this to the 59$PATH is up to you. The environment variable $PREFIX can change the 60install location, ala "PREFIX=/usr/local/bin make install".</p> 61 62<p>If you need an unstripped (debug) version of any of these binaries, 63look in generated/unstripped.</p> 64 65<p><h1><a name="running"><a href="#running">Running a command</a></h1></p> 66 67<h2>main</h2> 68 69<p>The toybox main() function is at the end of main.c at the top level. It has 70two possible codepaths, only one of which is configured into any given build 71of toybox.</p> 72 73<p>If CONFIG_SINGLE is selected, toybox is configured to contain only a single 74command, so most of the normal setup can be skipped. In this case the 75multiplexer isn't used, instead main() calls toy_singleinit() (also in main.c) 76to set up global state and parse command line arguments, calls the command's 77main function out of toy_list (in the CONFIG_SINGLE case the array has a single entry, no need to search), and if the function returns instead of exiting 78it flushes stdout (detecting error) and returns toys.exitval.</p> 79 80<p>When CONFIG_SINGLE is not selected, main() uses basename() to find the 81name it was run as, shifts its argument list one to the right so it lines up 82with where the multiplexer function expects it, and calls toybox_main(). This 83leverages the multiplexer command's infrastructure to find and run the 84appropriate command. (A command name starting with "toybox" will 85recursively call toybox_main(); you can go "./toybox toybox toybox toybox ls" 86if you want to...)</p> 87 88<h2>toybox_main</h2> 89 90<p>The toybox_main() function is also in main,c. It handles a possible 91--help option ("toybox --help ls"), prints the list of available commands if no 92arguments were provided to the multiplexer (or with full path names if any 93other option is provided before a command name, ala "toybox --list"). 94Otherwise it calls toy_exec() on its argument list.</p> 95 96<p>Note that the multiplexer is the first entry in toy_list (the rest of the 97list is sorted alphabetically to allow binary search), so toybox_main can 98cheat and just grab the first entry to quickly set up its context without 99searching. Since all command names go through the multiplexer at least once 100in the non-TOYBOX_SINGLE case, this avoids a redundant search of 101the list.</p> 102 103<p>The toy_exec() function is also in main.c. It performs toy_find() to 104perform a binary search on the toy_list array to look up the command's 105entry by name and saves it in the global variable which, calls toy_init() 106to parse command line arguments and set up global state (using which->options), 107and calls the appropriate command's main() function (which->toy_main). On 108return it flushes all pending ansi FILE * I/O, detects if stdout had an 109error, and then calls xexit() (which uses toys.exitval).</p> 110 111<p><h1><a name="infrastructure" /><a href="#infrastructure">Infrastructure</a></h1></p> 112 113<p>The toybox source code is in following directories:</p> 114<ul> 115<li>The <a href="#top">top level directory</a> contains the file main.c (were 116execution starts), the header file toys.h (included by every command), and 117other global infrastructure.</li> 118<li>The <a href="#lib">lib directory</a> contains common functions shared by 119multiple commands:</li> 120<ul> 121<li><a href="#lib_lib">lib/lib.c</a></li> 122<li><a href="#lib_xwrap">lib/xwrap.c</a></li> 123<li><a href="#lib_llist">lib/llist.c</a></li> 124<li><a href="#lib_args">lib/args.c</a></li> 125<li><a href="#lib_dirtree">lib/dirtree.c</a></li> 126</ul> 127<li>The <a href="#toys">toys directory</a> contains the C files implementating 128each command. Currently it contains five subdirectories categorizing the 129commands: posix, lsb, other, example, and pending.</li> 130<li>The <a href="#scripts">scripts directory</a> contains the build and 131test infrastructure.</li> 132<li>The <a href="#kconfig">kconfig directory</a> contains the configuration 133infrastructure implementing menuconfig (copied from the Linux kernel).</li> 134<li>The <a href="#generated">generated directory</a> contains intermediate 135files generated from other parts of the source code.</li> 136<li>The <a href="#tests">tests directory</a> contains the test suite. 137NOSPACE=1 to allow tests to pass with diff -b</li> 138</ul> 139 140<a name="adding" /> 141<p><h1><a href="#adding">Adding a new command</a></h1></p> 142<p>To add a new command to toybox, add a C file implementing that command to 143one of the subdirectories under the toys directory. No other files need to 144be modified; the build extracts all the information it needs (such as command 145line arguments) from specially formatted comments and macros in the C file. 146(See the description of the <a href="#generated">"generated" directory</a> 147for details.)</p> 148 149<p>Currently there are five subdirectories under "toys", one for commands 150defined by the POSIX standard, one for commands defined by the Linux Standard 151Base, an "other" directory for commands not covered by an obvious standard, 152a directory of example commands (templates to use when starting new commands), 153and a "pending" directory of commands that need further review/cleanup 154before moving to one of the other directories (run these at your own risk, 155cleanup patches welcome). 156These directories are just for developer convenience sorting the commands, 157the directories are otherwise functionally identical. To add a new category, 158create the appropriate directory with a README file in it whose first line 159is the description menuconfig should use for the directory.)</p> 160 161<p>An easy way to start a new command is copy the file "toys/example/hello.c" 162to the name of the new command, and modify this copy to implement the new 163command (more or less by turning every instance of "hello" into the 164name of your command, updating the command line arguments, globals, and 165help data, and then filling out its "main" function with code that does 166something interesting).</p> 167 168<p>You could also start with "toys/example/skeleton.c", which provides a lot 169more example code (showing several variants of command line option 170parsing, how to implement multiple commands in the same file, and so on). 171But usually it's just more stuff to delete.</p> 172 173<p>Here's a checklist of steps to turn hello.c into another command:</p> 174 175<ul> 176<li><p>First "cp toys/example/hello.c toys/other/yourcommand.c" and open 177the new file in your preferred text editor.</p> 178<ul><li><p>Note that the 179name of the new file is significant: it's the name of the new command you're 180adding to toybox. The build includes all *.c files under toys/*/ whose 181names are a case insensitive match for an enabled config symbol. So 182toys/posix/cat.c only gets included if you have "CAT=y" in ".config".</p></li> 183</ul></p></li> 184 185<li><p>Change the one line comment at the top of the file (currently 186"hello.c - A hello world program") to describe your new file.</p></li> 187 188<li><p>Change the copyright notice to your name, email, and the current 189year.</p></li> 190 191<li><p>Give a URL to the relevant standards document, where applicable. 192(Sample links to SUSv4 and LSB are provided, feel free to link to other 193documentation or standards as appropriate.)</p></li> 194 195<li><p>Update the USE_YOURCOMMAND(NEWTOY(yourcommand,"blah",0)) line. 196The NEWTOY macro fills out this command's <a href="#toy_list">toy_list</a> 197structure. The arguments to the NEWTOY macro are:</p> 198 199<ol> 200<li><p>the name used to run your command</p></li> 201<li><p>the command line argument <a href="#lib_args">option parsing string</a> (0 if none)</p></li> 202<li><p>a bitfield of TOYFLAG values 203(defined in toys.h) providing additional information such as where your 204command should be installed on a running system, whether to blank umask 205before running, whether or not the command must run as root (and thus should 206retain root access if installed SUID), and so on.</p></li> 207</ol> 208</li> 209 210<li><p>Change the kconfig data (from "config YOURCOMMAND" to the end of the 211comment block) to supply your command's configuration and help 212information. The uppper case config symbols are used by menuconfig, and are 213also what the CFG_ and USE_() macros are generated from (see [TODO]). The 214help information here is used by menuconfig, and also by the "help" command to 215describe your new command. (See [TODO] for details.) By convention, 216unfinished commands default to "n" and finished commands default to "y", 217so "make defconfig" selects all finished commands. (Note, "finished" means 218"ready to be used", not that it'll never change again.)<p> 219 220<p>Each help block should start with a "usage: yourcommand" line explaining 221any command line arguments added by this config option. The "help" command 222outputs this text, and scripts/config2help.c in the build infrastructure 223collates these usage lines for commands with multiple configuration 224options when producing generated/help.h.</p> 225</li> 226 227<li><p>Change the "#define FOR_hello" line to "#define FOR_yourcommand" right 228before the "#include <toys.h>". (This selects the appropriate FLAG_ macros and 229does a "#define TT this.yourcommand" so you can access the global variables 230out of the space-saving union of structures. If you aren't using any command 231flag bits and aren't defining a GLOBAL block, you can delete this line.)</p></li> 232 233<li><p>Update the GLOBALS() macro to contain your command's global 234variables. If your command has no global variables, delete this macro.</p> 235 236<p>Variables in the GLOBALS() block are are stored in a space saving 237<a href="#toy_union">union of structures</a> format, which may be accessed 238using the TT macro as if TT were a global structure (so TT.membername). 239If you specified two-character command line arguments in 240NEWTOY(), the first few global variables will be initialized by the automatic 241argument parsing logic, and the type and order of these variables must 242correspond to the arguments specified in NEWTOY(). 243(See <a href="#lib_args">lib/args.c</a> for details.)</p></li> 244 245<li><p>Rename hello_main() to yourcommand_main(). This is the main() function 246where execution of your command starts. Your command line options are 247already sorted into this.optflags, this.optargs, this.optc, and the GLOBALS() 248as appropriate by the time this function is called. (See 249<a href="#lib_args">get_optflags()</a> for details.)</p></li> 250 251<li><p>Switch on TOYBOX_DEBUG in menuconfig (toybox global settings menu) 252the first time you build and run your new command. If anything is wrong 253with your option string, that will give you error messages.</p> 254 255<p>Otherwise it'll just segfault without 256explanation when it falls off the end because it didn't find a matching 257end parantheses for a longopt, or you put a nonexistent option in a square 258bracket grouping... Since these kind of errors can only be caused by a 259developer, not by end users, we don't normally want runtime checks for 260them. Once you're happy with your option string, you can switch TOYBOX_DEBUG 261back off.</p></li> 262</ul> 263 264<a name="headers" /><h2><a href="#headers">Headers.</a></h2> 265 266<p>Commands generally don't have their own headers. If it's common code 267it can live in lib/, if it isn't put it in the command's .c file. (The line 268between implementing multiple commands in a C file via OLDTOY() to share 269infrastructure and moving that shared infrastructure to lib/ is a judgement 270call. Try to figure out which is simplest.)</p> 271 272<p>The top level toys.h should #include all the standard (posix) headers 273that any command uses. (Partly this is friendly to ccache and partly this 274makes the command implementations shorter.) Individual commands should only 275need to include nonstandard headers that might prevent that command from 276building in some context we'd care about (and thus requiring that command to 277be disabled to avoid a build break).</p> 278 279<p>Target-specific stuff (differences between compiler versions, libc versions, 280or operating systems) should be confined to lib/portability.h and 281lib/portability.c. (There's even some minimal compile-time environment probing 282that writes data to generated/portability.h, see scripts/genconfig.sh.)</p> 283 284<p>Only include linux/*.h headers from individual commands (not from other 285headers), and only if you really need to. Data that varies per architecture 286is a good reason to include a header. If you just need a couple constants 287that haven't changed since the 1990's, it's ok to #define them yourself or 288just use the constant inline with a comment explaining what it is. (A 289#define that's only used once isn't really helping.)</p> 290 291<p><a name="top" /><h1><a href="#top">Top level directory.</a></h1></p> 292 293<p>This directory contains global infrastructure.</p> 294 295<h3>toys.h</h3> 296<p>Each command #includes "toys.h" as part of its standard prolog. It 297may "#define FOR_commandname" before doing so to get some extra entries 298specific to this command.</p> 299 300<p>This file sucks in most of the commonly used standard #includes, so 301individual files can just #include "toys.h" and not have to worry about 302stdargs.h and so on. Individual commands still need to #include 303special-purpose headers that may not be present on all systems (and thus would 304prevent toybox from building that command on such a system with that command 305enabled). Examples include regex support, any "linux/" or "asm/" headers, mtab 306support (mntent.h and sys/mount.h), and so on.</p> 307 308<p>The toys.h header also defines structures for most of the global variables 309provided to each command by toybox_main(). These are described in 310detail in the description for main.c, where they are initialized.</p> 311 312<p>The global variables are grouped into structures (and a union) for space 313savings, to more easily track the amount of memory consumed by them, 314so that they may be automatically cleared/initialized as needed, and so 315that access to global variables is more easily distinguished from access to 316local variables.</p> 317 318<h3>main.c</h3> 319<p>Contains the main() function where execution starts, plus 320common infrastructure to initialize global variables and select which command 321to run. The "toybox" multiplexer command also lives here. (This is the 322only command defined outside of the toys directory.)</p> 323 324<p>Execution starts in main() which trims any path off of the first command 325name and calls toybox_main(), which calls toy_exec(), which calls toy_find() 326and toy_init() before calling the appropriate command's function from 327toy_list[] (via toys.which->toy_main()). 328If the command is "toybox", execution recurses into toybox_main(), otherwise 329the call goes to the appropriate commandname_main() from a C file in the toys 330directory.</p> 331 332<p>The following global variables are defined in main.c:</p> 333<ul> 334<a name="toy_list" /> 335<li><p><b>struct toy_list toy_list[]</b> - array describing all the 336commands currently configured into toybox. The first entry (toy_list[0]) is 337for the "toybox" multiplexer command, which runs all the other built-in commands 338without symlinks by using its first argument as the name of the command to 339run and the rest as that command's argument list (ala "./toybox echo hello"). 340The remaining entries are the commands in alphabetical order (for efficient 341binary search).</p> 342 343<p>This is a read-only array initialized at compile time by 344defining macros and #including generated/newtoys.h.</p> 345 346<p>Members of struct toy_list (defined in "toys.h") include:</p> 347<ul> 348<li><p>char *<b>name</b> - the name of this command.</p></li> 349<li><p>void (*<b>toy_main</b>)(void) - function pointer to run this 350command.</p></li> 351<li><p>char *<b>options</b> - command line option string (used by 352get_optflags() in lib/args.c to intialize toys.optflags, toys.optargs, and 353entries in the toy's GLOBALS struct). When this is NULL, no option 354parsing is done before calling toy_main().</p></li> 355<li><p>int <b>flags</b> - Behavior flags for this command. The following flags are currently understood:</p> 356 357<ul> 358<li><b>TOYFLAG_USR</b> - Install this command under /usr</li> 359<li><b>TOYFLAG_BIN</b> - Install this command under /bin</li> 360<li><b>TOYFLAG_SBIN</b> - Install this command under /sbin</li> 361<li><b>TOYFLAG_NOFORK</b> - This command can be used as a shell builtin.</li> 362<li><b>TOYFLAG_UMASK</b> - Call umask(0) before running this command.</li> 363<li><b>TOYFLAG_STAYROOT</b> - Don't drop permissions for this command if toybox is installed SUID root.</li> 364<li><b>TOYFLAG_NEEDROOT</b> - This command cannot function unless run with root access.</li> 365</ul> 366<br> 367 368<p>These flags are combined with | (or). For example, to install a command 369in /usr/bin, or together TOYFLAG_USR|TOYFLAG_BIN.</p> 370</ul> 371</li> 372 373<li><p><b>struct toy_context toys</b> - global structure containing information 374common to all commands, initializd by toy_init() and defined in "toys.h". 375Members of this structure include:</p> 376<ul> 377<li><p>struct toy_list *<b>which</b> - a pointer to this command's toy_list 378structure. Mostly used to grab the name of the running command 379(toys->which.name).</p> 380</li> 381<li><p>int <b>exitval</b> - Exit value of this command. Defaults to zero. The 382error_exit() functions will return 1 if this is zero, otherwise they'll 383return this value.</p></li> 384<li><p>char **<b>argv</b> - "raw" command line options, I.E. the original 385unmodified string array passed in to main(). Note that modifying this changes 386"ps" output, and is not recommended. This array is null terminated; a NULL 387entry indicates the end of the array.</p> 388<p>Most commands don't use this field, instead the use optargs, optflags, 389and the fields in the GLOBALS struct initialized by get_optflags().</p> 390</li> 391<li><p>unsigned <b>optflags</b> - Command line option flags, set by 392<a href="#lib_args">get_optflags()</a>. Indicates which of the command line options listed in 393toys->which.options occurred this time.</p> 394 395<p>The rightmost command line argument listed in toys->which.options sets bit 3961, the next one sets bit 2, and so on. This means the bits are set in the same 397order the binary digits would be listed if typed out as a string. For example, 398the option string "abcd" would parse the command line "-c" to set optflags to 2, 399"-a" would set optflags to 8, and "-bd" would set optflags to 6 (4|2).</p> 400 401<p>Only letters are relevant to optflags. In the string "a*b:c#d", d=1, c=2, 402b=4, a=8. Punctuation after a letter initializes global variables at the 403start of the GLOBALS() block (see <a href="#toy_union">union toy_union this</a> 404for details).</p> 405 406<p>The build infrastructure creates FLAG_ macros for each option letter, 407corresponding to the bit position, so you can check (toys.optflags & FLAG_x) 408to see if a flag was specified. (The correct set of FLAG_ macros is selected 409by defining FOR_mycommand before #including toys.h. The macros live in 410toys/globals.h which is generated by scripts/make.sh.)</p> 411 412<p>For more information on option parsing, see <a href="#lib_args">get_optflags()</a>.</p> 413 414</li> 415<li><p>char **<b>optargs</b> - Null terminated array of arguments left over 416after get_optflags() removed all the ones it understood. Note: optarg[0] is 417the first argument, not the command name. Use toys.which->name for the command 418name.</p></li> 419<li><p>int <b>optc</b> - Optarg count, equivalent to argc but for 420optargs[].<p></li> 421</ul> 422 423<a name="toy_union" /> 424<li><p><b>union toy_union this</b> - Union of structures containing each 425command's global variables.</p> 426 427<p>Global variables are useful: they reduce the overhead of passing extra 428command line arguments between functions, they conveniently start prezeroed to 429save initialization costs, and the command line argument parsing infrastructure 430can also initialize global variables with its results.</p> 431 432<p>But since each toybox process can only run one command at a time, allocating 433space for global variables belonging to other commands you aren't currently 434running would be wasteful.</p> 435 436<p>Toybox handles this by encapsulating each command's global variables in 437a structure, and declaring a union of those structures with a single global 438instance (called "this"). The GLOBALS() macro contains the global 439variables that should go in the current command's global structure. Each 440variable can then be accessed as "this.commandname.varname". 441If you #defined FOR_commandname before including toys.h, the macro TT is 442#defined to this.commandname so the variable can then be accessed as 443"TT.variable". See toys/hello.c for an example.</p> 444 445<p>A command that needs global variables should declare a structure to 446contain them all, and add that structure to this union. A command should never 447declare global variables outside of this, because such global variables would 448allocate memory when running other commands that don't use those global 449variables.</p> 450 451<p>The first few fields of this structure can be intialized by <a href="#lib_args">get_optargs()</a>, 452as specified by the options field off this command's toy_list entry. See 453the get_optargs() description in lib/args.c for details.</p> 454</li> 455 456<li><b>char toybuf[4096]</b> - a common scratch space buffer guaranteed 457to start zeroed, so commands don't need to allocate/initialize their own. 458Any command is free to use this, and it should never be directly referenced 459by functions in lib/ (although commands are free to pass toybuf in to a 460library function as an argument).</li> 461 462<li><b>char libbuf[4096]</b> - like toybuf, but for use by common code in 463lib/*.c. Commands should never directly reference libbuf, and library 464could should nnever directly reference toybuf.</li> 465</ul> 466 467<p>The following functions are defined in main.c:</p> 468<ul> 469<li><p>struct toy_list *<b>toy_find</b>(char *name) - Return the toy_list 470structure for this command name, or NULL if not found.</p></li> 471<li><p>void <b>toy_init</b>(struct toy_list *which, char *argv[]) - fill out 472the global toys structure, calling get_optargs() if necessary.</p></li> 473<li><p>void <b>toy_exec</b>(char *argv[]) - Run a built-in command with 474arguments.</p> 475<p>Calls toy_find() on argv[0] (which must be just a command name 476without path). Returns if it can't find this command, otherwise calls 477toy_init(), toys->which.toy_main(), and exit() instead of returning.</p> 478 479<p>Use the library function xexec() to fall back to external executables 480in $PATH if toy_exec() can't find a built-in command. Note that toy_exec() 481does not strip paths before searching for a command, so "./command" will 482never match an internal command.</li> 483 484<li><p>void <b>toybox_main</b>(void) - the main function for the multiplexer 485command (I.E. "toybox"). Given a command name as its first argument, calls 486toy_exec() on its arguments. With no arguments, it lists available commands. 487If the first argument starts with "-" it lists each command with its default 488install path prepended.</p></li> 489 490</ul> 491 492<h3>Config.in</h3> 493 494<p>Top level configuration file in a stylized variant of 495<a href=http://kernel.org/doc/Documentation/kbuild/kconfig-language.txt>kconfig</a> format. Includes generated/Config.in.</p> 496 497<p>These files are directly used by "make menuconfig" to select which commands 498to build into toybox (thus generating a .config file), and by 499scripts/config2help.py to create generated/help.h.</p> 500 501<a name="generated" /> 502<h1><a href="#generated">Temporary files:</a></h1> 503 504<p>There is one temporary file in the top level source directory:</p> 505<ul> 506<li><p><b>.config</b> - Configuration file generated by kconfig, indicating 507which commands (and options to commands) are currently enabled. Used 508to make generated/config.h and determine which toys/*/*.c files to build.</p> 509 510<p>You can create a human readable "miniconfig" version of this file using 511<a href=http://landley.net/aboriginal/new_platform.html#miniconfig>these 512instructions</a>.</p> 513</li> 514</ul> 515 516<p><h2>Directory generated/</h2></p> 517 518<p>The remaining temporary files live in the "generated/" directory, 519which is for files generated at build time from other source files.</p> 520 521<ul> 522<li><p><b>generated/Config.in</b> - Kconfig entries for each command, included 523from the top level Config.in. The help text here is used to generate 524help.h.</p> 525 526<p>Each command has a configuration entry with an upper case version of 527the command name. Options to commands start with the command 528name followed by an underscore and the option name. Global options are attached 529to the "toybox" command, and thus use the prefix "TOYBOX_". This organization 530is used by scripts/cfg2files to select which toys/*/*.c files to compile for a 531given .config.</p> 532</li> 533 534<li><p><b>generated/config.h</b> - list of CFG_SYMBOL and USE_SYMBOL() macros, 535generated from .config by a sed invocation in scripts/make.sh.</p> 536 537<p>CFG_SYMBOL is a comple time constant set to 1 for enabled symbols and 0 for 538disabled symbols. This allows the use of normal if() statements to remove 539code at compile time via the optimizer's dead code elimination (which removes 540from the binary any code that cannot be reached). This saves space without 541cluttering the code with #ifdefs or leading to configuration dependent build 542breaks. (See the 1992 Usenix paper 543<a href=http://doc.cat-v.org/henry_spencer/ifdef_considered_harmful.pdf>#ifdef 544Considered Harmful</a> for more information.)</p> 545 546<p>When you can't entirely avoid an #ifdef, the USE_SYMBOL(code) macro 547provides a less intrusive alternative, evaluating to the code in parentheses 548when the symbol is enabled, and nothing when the symbol is disabled. This 549is most commonly used around NEWTOY() declarations (so only the enabled 550commands show up in toy_list), and in option strings. This can also be used 551for things like varargs or structure members which can't always be 552eliminated by a simple test on CFG_SYMBOL. Remember, unlike CFG_SYMBOL 553this is really just a variant of #ifdef, and can still result in configuration 554dependent build breaks. Use with caution.</p> 555</li> 556 557<li><p><b>generated/flags.h</b> - FLAG_? macros indicating which command 558line options were seen. The option parsing in lib/args.c sets bits in 559toys.optflags, which can be tested by anding with the appropriate FLAG_ 560macro. (Bare longopts, which have no corresponding short option, will 561have the longopt name after FLAG_. All others use the single letter short 562option.)</p> 563 564<p>To get the appropriate macros for your command, #define FOR_commandname 565before #including toys.h. To switch macro sets (because you have an OLDTOY() 566with different options in the same .c file), #define CLEANUP_oldcommand 567and also #define FOR_newcommand, then #include "generated/flags.h" to switch. 568</p> 569</li> 570 571<li><p><b>generated/globals.h</b> - 572Declares structures to hold the contents of each command's GLOBALS(), 573and combines them into "global_union this". (Yes, the name was 574chosen to piss off C++ developers who think that C 575is merely a subset of C++, not a language in its own right.)</p> 576 577<p>The union reuses the same memory for each command's global struct: 578since only one command's globals are in use at any given time, collapsing 579them together saves space. The headers #define TT to the appropriate 580"this.commandname", so you can refer to the current command's global 581variables out of "this" as TT.variablename.</p> 582 583<p>The globals start zeroed, and the first few are filled out by the 584lib/args.c argument parsing code called from main.c.</p> 585</li> 586 587<li><p><b>toys/help.h</b> - Help strings for use by the "help" command and 588--help options. This file #defines a help_symbolname string for each 589symbolname, but only the symbolnames matching command names get used 590by show_help() in lib/help.c to display help for commands.</p> 591 592<p>This file is created by scripts/make.sh, which compiles scripts/config2help.c 593into the binary generated/config2help, and then runs it against the top 594level .config and Config.in files to extract the help text from each config 595entry and collate together dependent options.</p> 596 597<p>This file contains help text for all commands, regardless of current 598configuration, but only the ones currently enabled in the .config file 599wind up in the help_data[] array, and only the enabled dependent options 600have their help text added to the command they depend on.</p> 601</li> 602 603<li><p><b>generated/newtoys.h</b> - 604All the NEWTOY() and OLDTOY() macros from toys/*/*.c. The "toybox" multiplexer 605is the first entry, the rest are in alphabetical order. Each line should be 606inside an appropriate USE_ macro, so code that #includes this file only sees 607the currently enabled commands.</p> 608 609<p>By #definining NEWTOY() to various things before #including this file, 610it may be used to create function prototypes (in toys.h), initialize the 611help_data array (in lib/help.c), initialize the toy_list array (in main.c, 612the alphabetical order lets toy_find() do a binary search, the exception to 613the alphabetical order lets it use the multiplexer without searching), and so 614on. (It's even used to initialize the NEED_OPTIONS macro, which produces a 1 615or 0 for each command using command line option parsing, which is ORed together 616to allow compile-time dead code elimination to remove the whole of 617lib/args.c if nothing currently enabled is using it.)<p> 618 619<p>Each NEWTOY and OLDTOY macro contains the command name, command line 620option string (telling lib/args.c how to parse command line options for 621this command), recommended install location, and miscelaneous data such 622as whether this command should retain root permissions if installed suid.</p> 623</li> 624 625<li><p><b>toys/oldtoys.h</b> - Macros with the command line option parsing 626string for each NEWTOY. This allows an OLDTOY that's just an alias for an 627existing command to refer to the existing option string instead of 628having to repeat it.</p> 629</li> 630</ul> 631 632<a name="lib"> 633<h2>Directory lib/</h2> 634 635<p>TODO: document lots more here.</p> 636 637<p>lib: getmountlist(), error_msg/error_exit, xmalloc(), 638strlcpy(), xexec(), xopen()/xread(), xgetcwd(), xabspath(), find_in_path(), 639itoa().</p> 640 641 642 643<a name="lib_xwrap"><h3>lib/xwrap.c</h3> 644 645<p>Functions prefixed with the letter x call perror_exit() when they hit 646errors, to eliminate common error checking. This prints an error message 647and the strerror() string for the errno encountered.</p> 648 649<p>We replaced exit(), _exit(), and atexit() with xexit(), _xexit(), and 650sigatexit(). This gives _xexit() the option to siglongjmp(toys.rebound, 1) 651instead of exiting, lets xexit() report stdout flush failures to stderr 652and change the exit code to indicate error, lets our toys.exit function 653change happen for signal exit paths and lets us remove the functions 654after we've called them.</p> 655 656<p>You can intercept our exit by assigning a setjmp/longjmp buffer to 657toys.rebound (set it back to zero to restore the default behavior). 658If you do this, cleaning up resource leaks is your problem.</p> 659 660<ul> 661<li><b>void xstrncpy(char *dest, char *src, size_t size)</b></li> 662<li><p><b><p>void _xexit(void)</b></p> 663<p>Calls siglongjmp(toys.rebound, 1), or else _exit(toys.exitval). This 664lets you ignore errors with the NO_EXIT() macro wrapper, or intercept 665them with WOULD_EXIT().</p> 666<li><b><p>void xexit(void)</b></p> 667<p>Calls toys.xexit functions (if any) and flushes stdout/stderr (reporting 668failure to write to stdout both to stderr and in the exit code), then 669calls _xexit().</p> 670</li> 671<li><b>void *xmalloc(size_t size)</b></li> 672<li><b>void *xzalloc(size_t size)</b></li> 673<li><b>void *xrealloc(void *ptr, size_t size)</b></li> 674<li><b>char *xstrndup(char *s, size_t n)</b></li> 675<li><b>char *xstrdup(char *s)</b></li> 676<li><b>char *xmprintf(char *format, ...)</b></li> 677<li><b>void xprintf(char *format, ...)</b></li> 678<li><b>void xputs(char *s)</b></li> 679<li><b>void xputc(char c)</b></li> 680<li><b>void xflush(void)</b></li> 681<li><b>pid_t xfork(void)</b></li> 682<li><b>void xexec_optargs(int skip)</b></li> 683<li><b>void xexec(char **argv)</b></li> 684<li><b>pid_t xpopen(char **argv, int *pipes)</b></li> 685<li><b>int xpclose(pid_t pid, int *pipes)</b></li> 686<li><b>void xaccess(char *path, int flags)</b></li> 687<li><b>void xunlink(char *path)</b></li> 688<li><p><b>int xcreate(char *path, int flags, int mode)<br /> 689int xopen(char *path, int flags)</b></p> 690 691<p>The xopen() and xcreate() functions open an existing file (exiting if 692it's not there) and create a new file (exiting if it can't).</p> 693 694<p>They default to O_CLOEXEC so the filehandles aren't passed on to child 695processes. Feed in O_CLOEXEC to disable this.</p> 696</li> 697<li><p><b>void xclose(int fd)</b></p> 698 699<p>Because NFS is broken, and won't necessarily perform the requested 700operation (and report the error) until you close the file. Of course, this 701being NFS, it's not guaranteed to report the error there either, but it 702_can_.</p> 703 704<p>Nothing else ever reports an error on close, everywhere else it's just a 705VFS operation freeing some resources. NFS is _special_, in a way that 706other network filesystems like smbfs and v9fs aren't..</p> 707</li> 708<li><b>int xdup(int fd)</b></li> 709<li><p><b>size_t xread(int fd, void *buf, size_t len)</b></p> 710 711<p>Can return 0, but not -1.</p> 712</li> 713<li><p><b>void xreadall(int fd, void *buf, size_t len)</b></p> 714 715<p>Reads the entire len-sized buffer, retrying to complete short 716reads. Exits if it can't get enough data.</p></li> 717 718<li><p><b>void xwrite(int fd, void *buf, size_t len)</b></p> 719 720<p>Retries short writes, exits if can't write the entire buffer.</p></li> 721 722<li><b>off_t xlseek(int fd, off_t offset, int whence)</b></li> 723<li><b>char *xgetcwd(void)</b></li> 724<li><b>void xstat(char *path, struct stat *st)</b></li> 725<li><p><b>char *xabspath(char *path, int exact) </b></p> 726 727<p>After several years of 728<a href=http://landley.net/notes-2007.html#18-06-2007>wrestling</a> 729<a href=http://landley.net/notes-2008.html#19-01-2008>with</a> realpath(), 730I broke down and <a href=http://landley.net/notes-2012.html#20-11-2012>wrote 731my own</a> implementation that doesn't use the one in libc. As I explained: 732 733<blockquote><p>If the path ends with a broken link, 734readlink -f should show where the link points to, not where the broken link 735lives. (The point of readlink -f is "if I write here, where would it attempt 736to create a file".) The problem is, realpath() returns NULL for a path ending 737with a broken link, and I can't beat different behavior out of code locked 738away in libc.</p></blockquote> 739 740<p> 741</li> 742<li><b>void xchdir(char *path)</b></li> 743<li><b>void xchroot(char *path)</b></li> 744 745<li><p><b>struct passwd *xgetpwuid(uid_t uid)<br /> 746struct group *xgetgrgid(gid_t gid)<br /> 747struct passwd *xgetpwnam(char *name)</b></p> 748</li> 749 750<li><b>void xsetuser(struct passwd *pwd)</b></li> 751<li><b>char *xreadlink(char *name)</b></li> 752<li><b>char *xreadfile(char *name, char *buf, off_t len)</b></li> 753<li><b>int xioctl(int fd, int request, void *data)</b></li> 754<li><b>void xpidfile(char *name)</b></li> 755<li><b>void xsendfile(int in, int out)</b></li> 756<li><b>long xparsetime(char *arg, long units, long *fraction)</b></li> 757<li><b>void xregcomp(regex_t *preg, char *regex, int cflags)</b></li> 758</ul> 759 760<a name="lib_lib"><h3>lib/lib.c</h3> 761<p>Eight gazillion common functions, see lib/lib.h for the moment:</p> 762 763<h3>lib/portability.h</h3> 764 765<p>This file is automatically included from the top of toys.h, and smooths 766over differences between platforms (hardware targets, compilers, C libraries, 767operating systems, etc).</p> 768 769<p>This file provides SWAP macros (SWAP_BE16(x) and SWAP_LE32(x) and so on).</p> 770 771<p>A macro like SWAP_LE32(x) means "The value in x is stored as a little 772endian 32 bit value, so perform the translation to/from whatever the native 77332-bit format is". You do the swap once on the way in, and once on the way 774out. If your target is already little endian, the macro is a NOP.</p> 775 776<p>The SWAP macros come in BE and LE each with 16, 32, and 64 bit versions. 777In each case, the name of the macro refers to the _external_ representation, 778and converts to/from whatever your native representation happens to be (which 779can vary depending on what you're currently compiling for).</p> 780 781<a name="lib_llist"><h3>lib/llist.c</h3> 782 783<p>Some generic single and doubly linked list functions, which take 784advantage of a couple properties of C:</p> 785 786<ul> 787<li><p>Structure elements are laid out in memory in the order listed, and 788the first element has no padding. This means you can always treat (typecast) 789a pointer to a structure as a pointer to the first element of the structure, 790even if you don't know anything about the data following it.</p></li> 791 792<li><p>An array of length zero at the end of a structure adds no space 793to the sizeof() the structure, but if you calculate how much extra space 794you want when you malloc() the structure it will be available at the end. 795Since C has no bounds checking, this means each struct can have one variable 796length array.</p></li> 797</ul> 798 799<p>Toybox's list structures always have their <b>next</b> pointer as 800the first entry of each struct, and singly linked lists end with a NULL pointer. 801This allows generic code to traverse such lists without knowing anything 802else about the specific structs composing them: if your pointer isn't NULL 803typecast it to void ** and dereference once to get the next entry.</p> 804 805<p><b>lib/lib.h</b> defines three structure types:</p> 806<ul> 807<li><p><b>struct string_list</b> - stores a single string (<b>char str[0]</b>), 808memory for which is allocated as part of the node. (I.E. llist_traverse(list, 809free); can clean up after this type of list.)</p></li> 810 811<li><p><b>struct arg_list</b> - stores a pointer to a single string 812(<b>char *arg</b>) which is stored in a separate chunk of memory.</p></li> 813 814<li><p><b>struct double_list</b> - has a second pointer (<b>struct double_list 815*prev</b> along with a <b>char *data</b> for payload.</p></li> 816</ul> 817 818<b>List Functions</b> 819 820<ul> 821<li><p>void *<b>llist_pop</b>(void **list) - advances through a list ala 822<b>node = llist_pop(&list);</b> This doesn't modify the list contents, 823but does advance the pointer you feed it (which is why you pass the _address_ 824of that pointer, not the pointer itself).</p></li> 825 826<li><p>void <b>llist_traverse</b>(void *list, void (*using)(void *data)) - 827iterate through a list calling a function on each node.</p></li> 828 829<li><p>struct double_list *<b>dlist_add</b>(struct double_list **llist, char *data) 830- append an entry to a circular linked list. 831This function allocates a new struct double_list wrapper and returns the 832pointer to the new entry (which you can usually ignore since it's llist->prev, 833but if llist was NULL you need it). The argument is the ->data field for the 834new node.</p></li> 835<ul><li><p>void <b>dlist_add_nomalloc</b>(struct double_list **llist, 836struct double_list *new) - append existing struct double_list to 837list, does not allocate anything.</p></li></ul> 838</ul> 839 840<b>List code trivia questions:</b> 841 842<ul> 843<li><p><b>Why do arg_list and double_list contain a char * payload instead of 844a void *?</b> - Because you always have to typecast a void * to use it, and 845typecasting a char * does no harm. Since strings are the most common 846payload, and doing math on the pointer ala 847"(type *)(ptr+sizeof(thing)+sizeof(otherthing))" requires ptr to be char * 848anyway (at least according to the C standard), defaulting to char * saves 849a typecast.</p> 850</li> 851 852<li><p><b>Why do the names ->str, ->arg, and ->data differ?</b> - To force 853you to keep track of which one you're using, calling free(node->str) would 854be bad, and _failing_ to free(node->arg) leaks memory.</p></li> 855 856<li><p><b>Why does llist_pop() take a void * instead of void **?</b> - 857because the stupid compiler complains about "type punned pointers" when 858you typecast and dereference on the same line, 859due to insane FSF developers hardwiring limitations of their optimizer 860into gcc's warning system. Since C automatically typecasts any other 861pointer type to and from void *, the current code works fine. It's sad that it 862won't warn you if you forget the &, but the code crashes pretty quickly in 863that case.</p></li> 864 865<li><p><b>How do I assemble a singly-linked-list in order?</b> - use 866a double_list, dlist_add() your entries, and then call dlist_terminate(list) 867to break the circle when done (turning the last ->next and the first ->prev 868into NULLs).</p> 869</ul> 870 871<a name="lib_args"><h3>lib/args.c</h3> 872 873<p>Toybox's main.c automatically parses command line options before calling the 874command's main function. Option parsing starts in get_optflags(), which stores 875results in the global structures "toys" (optflags and optargs) and "this".</p> 876 877<p>The option parsing infrastructure stores a bitfield in toys.optflags to 878indicate which options the current command line contained, and defines FLAG 879macros code can use to check whether each argument's bit is set. Arguments 880attached to those options are saved into the command's global structure 881("this"). Any remaining command line arguments are collected together into 882the null-terminated array toys.optargs, with the length in toys.optc. (Note 883that toys.optargs does not contain the current command name at position zero, 884use "toys.which->name" for that.) The raw command line arguments get_optflags() 885parsed are retained unmodified in toys.argv[].</p> 886 887<p>Toybox's option parsing logic is controlled by an "optflags" string, using 888a format reminiscent of getopt's optargs but with several important differences. 889Toybox does not use the getopt() 890function out of the C library, get_optflags() is an independent implementation 891which doesn't permute the original arguments (and thus doesn't change how the 892command is displayed in ps and top), and has many features not present in 893libc optargs() (such as the ability to describe long options in the same string 894as normal options).</p> 895 896<p>Each command's NEWTOY() macro has an optflags string as its middle argument, 897which sets toy_list.options for that command to tell get_optflags() what 898command line arguments to look for, and what to do with them. 899If a command has no option 900definition string (I.E. the argument is NULL), option parsing is skipped 901for that command, which must look at the raw data in toys.argv to parse its 902own arguments. (If no currently enabled command uses option parsing, 903get_optflags() is optimized out of the resulting binary by the compiler's 904--gc-sections option.)</p> 905 906<p>You don't have to free the option strings, which point into the environment 907space (I.E. the string data is not copied). A TOYFLAG_NOFORK command 908that uses the linked list type "*" should free the list objects but not 909the data they point to, via "llist_free(TT.mylist, NULL);". (If it's not 910NOFORK, exit() will free all the malloced data anyway unless you want 911to implement a CONFIG_TOYBOX_FREE cleanup for it.)</p> 912 913<h4>Optflags format string</h4> 914 915<p>Note: the optflags option description string format is much more 916concisely described by a large comment at the top of lib/args.c.</p> 917 918<p>The general theory is that letters set optflags, and punctuation describes 919other actions the option parsing logic should take.</p> 920 921<p>For example, suppose the command line <b>command -b fruit -d walrus -a 42</b> 922is parsed using the optflags string "<b>a#b:c:d</b>". (I.E. 923toys.which->options="a#b:c:d" and argv = ["command", "-b", "fruit", "-d", 924"walrus", "-a", "42"]). When get_optflags() returns, the following data is 925available to command_main(): 926 927<ul> 928<li><p>In <b>struct toys</b>: 929<ul> 930<li>toys.optflags = 13; // FLAG_a = 8 | FLAG_b = 4 | FLAG_d = 1</li> 931<li>toys.optargs[0] = "walrus"; // leftover argument</li> 932<li>toys.optargs[1] = NULL; // end of list</li> 933<li>toys.optc = 1; // there was 1 leftover argument</li> 934<li>toys.argv[] = {"-b", "fruit", "-d", "walrus", "-a", "42"}; // The original command line arguments 935</ul> 936<p></li> 937 938<li><p>In <b>union this</b> (treated as <b>long this[]</b>): 939<ul> 940<li>this[0] = NULL; // -c didn't get an argument this time, so get_optflags() didn't change it and toys_init() zeroed "this" during setup.)</li> 941<li>this[1] = (long)"fruit"; // argument to -b</li> 942<li>this[2] = 42; // argument to -a</li> 943</ul> 944</p></li> 945</ul> 946 947<p>If the command's globals are:</p> 948 949<blockquote><pre> 950GLOBALS( 951 char *c; 952 char *b; 953 long a; 954) 955</pre></blockquote> 956 957<p>That would mean TT.c == NULL, TT.b == "fruit", and TT.a == 42. (Remember, 958each entry that receives an argument must be a long or pointer, to line up 959with the array position. Right to left in the optflags string corresponds to 960top to bottom in GLOBALS().</p> 961 962<p>Put globals not filled out by the option parsing logic at the end of the 963GLOBALS block. Common practice is to list the options one per line (to 964make the ordering explicit, first to last in globals corresponds to right 965to left in the option string), then leave a blank line before any non-option 966globals.</p> 967 968<p><b>long toys.optflags</b></p> 969 970<p>Each option in the optflags string corresponds to a bit position in 971toys.optflags, with the same value as a corresponding binary digit. The 972rightmost argument is (1<<0), the next to last is (1<<1) and so on. If 973the option isn't encountered while parsing argv[], its bit remains 0.</p> 974 975<p>Each option -x has a FLAG_x macro for the command letter. Bare --longopts 976with no corresponding short option have a FLAG_longopt macro for the long 977optionname. Commands enable these macros by #defining FOR_commandname before 978#including <toys.h>. When multiple commands are implemented in the same 979source file, you can switch flag contexts later in the file by 980#defining CLEANUP_oldcommand and #defining FOR_newcommand, then 981#including <generated/flags.h>.</p> 982 983<p>Options disabled in the current configuration (wrapped in 984a USE_BLAH() macro for a CONFIG_BLAH that's switched off) have their 985corresponding FLAG macro set to zero, so code checking them ala 986if (toys.optargs & FLAG_x) gets optimized out via dead code elimination. 987#defining FORCE_FLAGS when switching flag context disables this 988behavior: the flag is never zero even if the config is disabled. This 989allows code shared between multiple commands to use the same flag 990values, as long as the common flags match up right to left in both option 991strings.</p> 992 993<p>For example, 994the optflags string "abcd" would parse the command line argument "-c" to set 995optflags to 2, "-a" would set optflags to 8, "-bd" would set optflags to 9966 (I.E. 4|2), and "-a -c" would set optflags to 10 (2|8). To check if -c 997was encountered, code could test: if (toys.optflags & FLAG_c) printf("yup"); 998(See the toys/examples directory for more.)</p> 999 1000<p>Only letters are relevant to optflags, punctuation is skipped: in the
1001string "a*b:c#d", d=1, c=2, b=4, a=8. The punctuation after a letter 1002usually indicate that the option takes an argument.</p> 1003 1004<p>Since toys.optflags is an unsigned int, it only stores 32 bits. (Which is 1005the amount a long would have on 32-bit platforms anyway; 64 bit code on 100632 bit platforms is too expensive to require in common code used by almost 1007all commands.) Bit positions beyond the 1<<31 aren't recorded, but 1008parsing higher options can still set global variables.</p> 1009 1010<p><b>Automatically setting global variables from arguments (union this)</b></p> 1011 1012<p>The following punctuation characters may be appended to an optflags 1013argument letter, indicating the option takes an additional argument:</p> 1014 1015<ul> 1016<li><b>:</b> - plus a string argument, keep most recent if more than one.</li> 1017<li><b>*</b> - plus a string argument, appended to a linked list.</li> 1018<li><b>@</b> - plus an occurrence counter (stored in a long)</li> 1019<li><b>#</b> - plus a signed long argument. 1020<li><b>-</b> - plus a signed long argument defaulting to negative (start argument with + to force a positive value).</li> 1021<li><b>.</b> - plus a floating point argument (if CFG_TOYBOX_FLOAT).</li> 1022<ul>The following can be appended to a float or double: 1023<li><b><123</b> - error if argument is less than this</li> 1024<li><b>>123</b> - error if argument is greater than this</li> 1025<li><b>=123</b> - default value if argument not supplied</li> 1026</ul> 1027</ul> 1028 1029<p><b>GLOBALS</b></p> 1030 1031<p>Options which have an argument fill in the corresponding slot in the global 1032union "this" (see generated/globals.h), treating it as an array of longs 1033with the rightmost saved in this[0]. As described above, using "a*b:c#d", 1034"-c 42" would set this[0] = 42; and "-b 42" would set this[1] = "42"; each 1035slot is left NULL if the corresponding argument is not encountered.</p> 1036 1037<p>This behavior is useful because the LP64 standard ensures long and pointer 1038are the same size. C99 guarantees structure members will occur in memory 1039in the same order they're declared, and that padding won't be inserted between 1040consecutive variables of register size. Thus the first few entries can 1041be longs or pointers corresponding to the saved arguments.</p> 1042 1043<p>The main downside is that numeric arguments ("#" and "-" format) 1044are limited to +- 2 billion on 32 bit platforms (the "truncate -s 8G" 1045problem), because long is only 64 bits on 64 bit hosts, so the capabilities 1046of some tools differ when built in 32 bit vs 64 bit mode. Fixing this 1047kind of ugly and even embedded designs are slowly moving to 64 bits, 1048so our current plan is to document the problem and wait it out. (If 1049"x32 mode" and similar becomes popular enough, we may revisit this 1050decision.)</p> 1051 1052<p>See toys/example/*.c for longer examples of parsing options into the 1053GLOBALS block.</p> 1054 1055<p><b>char *toys.optargs[]</b></p> 1056 1057<p>Command line arguments in argv[] which are not consumed by option parsing 1058(I.E. not recognized either as -flags or arguments to -flags) will be copied 1059to toys.optargs[], with the length of that array in toys.optc. 1060(When toys.optc is 0, no unrecognized command line arguments remain.) 1061The order of entries is preserved, and as with argv[] this new array is also 1062terminated by a NULL entry.</p> 1063 1064<p>Option parsing can require a minimum or maximum number of optargs left 1065over, by adding "<1" (read "at least one") or ">9" ("at most nine") to the 1066start of the optflags string.</p> 1067 1068<p>The special argument "--" terminates option parsing, storing all remaining 1069arguments in optargs. The "--" itself is consumed.</p> 1070 1071<p><b>Other optflags control characters</b></p> 1072 1073<p>The following characters may occur at the start of each command's 1074optflags string, before any options that would set a bit in toys.optflags:</p> 1075 1076<ul> 1077<li><b>^</b> - stop at first nonoption argument (for nice, xargs...)</li> 1078<li><b>?</b> - allow unknown arguments (pass non-option arguments starting 1079with - through to optargs instead of erroring out).</li> 1080<li><b>&</b> - the first argument has imaginary dash (ala tar/ps. If given twice, all arguments have imaginary dash.)</li> 1081<li><b><</b> - must be followed by a decimal digit indicating at least this many leftover arguments are needed in optargs (default 0)</li> 1082<li><b>></b> - must be followed by a decimal digit indicating at most this many leftover arguments allowed (default MAX_INT)</li> 1083</ul> 1084 1085<p>The following characters may be appended to an option character, but do 1086not by themselves indicate an extra argument should be saved in this[]. 1087(Technically any character not recognized as a control character sets an 1088optflag, but letters are never control characters.)</p> 1089 1090<ul> 1091<li><b>^</b> - stop parsing options after encountering this option, everything else goes into optargs.</li> 1092<li><b>|</b> - this option is required. If more than one marked, only one is required.</li> 1093</ul> 1094 1095<p>The following may be appended to a float or double:</p> 1096 1097<ul> 1098<li><b><123</b> - error if argument is less than this</li> 1099<li><b>>123</b> - error if argument is greater than this</li> 1100<li><b>=123</b> - default value if argument not supplied</li> 1101</ul> 1102 1103<p>Option parsing only understands <>= after . when CFG_TOYBOX_FLOAT 1104is enabled. (Otherwise the code to determine where floating point constants 1105end drops out. When disabled, it can reserve a global data slot for the 1106argument so offsets won't change, but will never fill it out.) You can handle 1107this by using the USE_BLAH() macros with C string concatenation, ala:</p> 1108 1109<blockquote>"abc." USE_TOYBOX_FLOAT("<1.23>4.56=7.89") "def"</blockquote> 1110 1111<p><b>--longopts</b></p> 1112 1113<p>The optflags string can contain long options, which are enclosed in 1114parentheses. They may be appended to an existing option character, in 1115which case the --longopt is a synonym for that option, ala "a:(--fred)" 1116which understands "-a blah" or "--fred blah" as synonyms.</p> 1117 1118<p>Longopts may also appear before any other options in the optflags string, 1119in which case they have no corresponding short argument, but instead set 1120their own bit based on position. So for "(walrus)#(blah)xy:z", "command 1121--walrus 42" would set toys.optflags = 16 (-z = 1, -y = 2, -x = 4, --blah = 8) 1122and would assign this[1] = 42;</p> 1123 1124<p>A short option may have multiple longopt synonyms, "a(one)(two)", but 1125each "bare longopt" (ala "(one)(two)abc" before any option characters) 1126always sets its own bit (although you can group them with +X).</p> 1127 1128<p>Only bare longopts have a FLAG_ macro with the longopt name 1129(ala --fred would #define FLAG_fred). Other longopts use the short 1130option's FLAG macro to test the toys.optflags bit.</p> 1131 1132<p>Options with a semicolon ";" after their data type can only set their 1133corresponding GLOBALS() entry via "--longopt=value". For example, option 1134string "x(boing): y" would set TT.x if it saw "--boing=value", but would 1135treat "--boing value" as setting FLAG_x in toys.optargs, leaving TT.x NULL, 1136and keeping "value" in toys.optargs[]. (This lets "ls --color" and 1137"ls --color=auto" both work.)</p> 1138 1139<p><b>[groups]</b></p> 1140 1141<p>At the end of the option string, square bracket groups can define 1142relationships between existing options. (This only applies to short 1143options, bare --longopts can't participate.)</p> 1144 1145<p>The first character of the group defines the type, the remaining 1146characters are options it applies to:</p> 1147 1148<ul> 1149<li><b>-</b> - Exclusive, switch off all others in this group.</li> 1150<li><b>+</b> - Inclusive, switch on all others in this group.</li> 1151<li><b>!</b> - Error, fail if more than one defined.</li> 1152</ul> 1153 1154<p>So "abc[-abc]" means -ab = -b, -ba = -a, -abc = -c. "abc[+abc]" 1155means -ab=-abc, -c=-abc, and "abc[!abc] means -ab calls error_exit("no -b 1156with -a"). Note that [-] groups clear the GLOBALS option slot of 1157options they're switching back off, but [+] won't set options it didn't see 1158(just the optflags).</p> 1159 1160<p><b>whitespace</b></p> 1161 1162<p>Arguments may occur with or without a space (I.E. "-a 42" or "-a42"). 1163The command line argument "-abc" may be interepreted many different ways: 1164the optflags string "cba" sets toys.optflags = 7, "c:ba" sets toys.optflags=4 1165and saves "ba" as the argument to -c, and "cb:a" sets optflags to 6 and saves 1166"c" as the argument to -b.</p> 1167 1168<p>Note that & changes whitespace handling, so that the command line 1169"tar cvfCj outfile.tar.bz2 topdir filename" is parsed the same as 1170"tar filename -c -v -j -f outfile.tar.bz2 -C topdir". Note that "tar -cvfCj 1171one two three" would equal "tar -c -v -f Cj one two three". (This matches 1172historical usage.)</p> 1173 1174<p>Appending a space to the option in the option string ("a: b") makes it 1175require a space, I.E. "-ab" is interpreted as "-a" "-b". That way "kill -stop" 1176differs from "kill -s top".</p> 1177 1178<p>Appending ; to a longopt in the option string makes its argument optional, 1179and only settable with =, so in ls "(color):;" can accept "ls --color" and 1180"ls --color=auto" without complaining that the first has no argument.</p> 1181 1182<a name="lib_dirtree"><h3>lib/dirtree.c</h3> 1183 1184<p>The directory tree traversal code should be sufficiently generic 1185that commands never need to use readdir(), scandir(), or the fts.h family 1186of functions.</p> 1187 1188<p>These functions do not call chdir() or rely on PATH_MAX. Instead they 1189use openat() and friends, using one filehandle per directory level to 1190recurse into subdirectories. (I.E. they can descend 1000 directories deep 1191if setrlimit(RLIMIT_NOFILE) allows enough open filehandles, and the default 1192in /proc/self/limits is generally 1024.)</p> 1193 1194<p>There are two main ways to use dirtree: 1) assemble a tree of nodes 1195representing a snapshot of directory state and traverse them using the 1196->next and ->child pointers, or 2) traverse the tree calling a callback 1197function on each entry, and freeing its node afterwards. (You can also 1198combine the two, using the callback as a filter to determine which nodes 1199to keep.)</p> 1200 1201<p>The basic dirtree functions are:</p> 1202 1203<ul> 1204<li><p><b>struct dirtree *dirtree_read(char *path, int (*callback)(struct 1205dirtree node))</b> - recursively read files and directories, calling 1206callback() on each, and returning a tree of saved nodes (if any). 1207If path doesn't exist, returns DIRTREE_ABORTVAL. If callback is NULL, 1208returns a single node at that path.</p> 1209 1210<li><p><b>dirtree_notdotdot(struct dirtree *new)</b> - standard callback 1211which discards "." and ".." entries and returns DIRTREE_SAVE|DIRTREE_RECURSE 1212for everything else. Used directly, this assembles a snapshot tree of 1213the contents of this directory and its subdirectories 1214to be processed after dirtree_read() returns (by traversing the 1215struct dirtree's ->next and ->child pointers from the returned root node).</p> 1216 1217<li><p><b>dirtree_path(struct dirtree *node, int *plen)</b> - malloc() a 1218string containing the path from the root of this tree to this node. If 1219plen isn't NULL then *plen is how many extra bytes to malloc at the end 1220of string.</p></li> 1221 1222<li><p><b>dirtree_parentfd(struct dirtree *node)</b> - return fd of 1223directory containing this node, for use with openat() and such.</p></li> 1224</ul> 1225 1226<p>The <b>dirtree_read()</b> function is the standard way to start 1227directory traversal. It takes two arguments: a starting path for 1228the root of the tree, and a callback function. The callback() is called 1229on each directory entry, its argument is a fully populated 1230<b>struct dirtree *</b> (from lib/lib.h) describing the node, and its 1231return value tells the dirtree infrastructure what to do next.</p> 1232 1233<p>(There's also a three argument version, 1234<b>dirtree_flagread(char *path, int flags, int (*callback)(struct 1235dirtree node))</b>, which lets you apply flags like DIRTREE_SYMFOLLOW and 1236DIRTREE_SHUTUP to reading the top node, but this only affects the top node. 1237Child nodes use the flags returned by callback().</p> 1238 1239<p><b>struct dirtree</b></p> 1240 1241<p>Each struct dirtree node contains <b>char name[]</b> and <b>struct stat 1242st</b> entries describing a file, plus a <b>char *symlink</b> 1243which is NULL for non-symlinks.</p> 1244 1245<p>During a callback function, the <b>int dirfd</b> field of directory nodes 1246contains a directory file descriptor (for use with the openat() family of 1247functions). This isn't usually used directly, intstead call dirtree_parentfd() 1248on the callback's node argument. The <b>char again</b> field is 0 for the 1249first callback on a node, and 1 on the second callback (triggered by returning 1250DIRTREE_COMEAGAIN on a directory, made after all children have been processed). 1251</p> 1252 1253<p>Users of this code may put anything they like into the <b>long extra</b> 1254field. For example, "cp" and "mv" use this to store a dirfd for the destination 1255directory (and use DIRTREE_COMEAGAIN to get the second callback so they can 1256close(node->extra) to avoid running out of filehandles). 1257This field is not directly used by the dirtree code, and 1258thanks to LP64 it's large enough to store a typecast pointer to an 1259arbitrary struct.</p> 1260 1261<p>The return value of the callback combines flags (with boolean or) to tell 1262the traversal infrastructure how to behave:</p> 1263 1264<ul> 1265<li><p><b>DIRTREE_SAVE</b> - Save this node, assembling a tree. (Without 1266this the struct dirtree is freed after the callback returns. Filtering out 1267siblings is fine, but discarding a parent while keeping its child leaks 1268memory.)</p></li> 1269<li><p><b>DIRTREE_ABORT</b> - Do not examine any more entries in this 1270directory. (Does not propagate up tree: to abort entire traversal, 1271return DIRTREE_ABORT from parent callbacks too.)</p></li> 1272<li><p><b>DIRTREE_RECURSE</b> - Examine directory contents. Ignored for 1273non-directory entries. The remaining flags only take effect when 1274recursing into the children of a directory.</p></li> 1275<li><p><b>DIRTREE_COMEAGAIN</b> - Call the callback on this node a second time 1276after examining all directory contents, allowing depth-first traversal. 1277On the second call, dirtree->again is nonzero.</p></li> 1278<li><p><b>DIRTREE_SYMFOLLOW</b> - follow symlinks when populating children's 1279<b>struct stat st</b> (by feeding a nonzero value to the symfollow argument of 1280dirtree_add_node()), which means DIRTREE_RECURSE treats symlinks to 1281directories as directories. (Avoiding infinite recursion is the callback's 1282problem: the non-NULL dirtree->symlink can still distinguish between 1283them. The "find" command follows ->parent up the tree to the root node 1284each time, checking to make sure that stat's dev and inode pair don't 1285match any ancestors.)</p></li> 1286</ul> 1287 1288<p>Each struct dirtree contains three pointers (next, parent, and child) 1289to other struct dirtree.</p> 1290 1291<p>The <b>parent</b> pointer indicates the directory 1292containing this entry; even when not assembling a persistent tree of 1293nodes the parent entries remain live up to the root of the tree while 1294child nodes are active. At the top of the tree the parent pointer is 1295NULL, meaning the node's name[] is either an absolute path or relative 1296to cwd. The function dirtree_parentfd() gets the directory file descriptor 1297for use with openat() and friends, returning AT_FDCWD at the top of tree.</p> 1298 1299<p>The <b>child</b> pointer points to the first node of the list of contents of 1300this directory. If the directory contains no files, or the entry isn't 1301a directory, child is NULL.</p> 1302 1303<p>The <b>next</b> pointer indicates sibling nodes in the same directory as this 1304node, and since it's the first entry in the struct the llist.c traversal 1305mechanisms work to iterate over sibling nodes. Each dirtree node is a 1306single malloc() (even char *symlink points to memory at the end of the node), 1307so llist_free() works but its callback must descend into child nodes (freeing 1308a tree, not just a linked list), plus whatever the user stored in extra.</p> 1309 1310<p>The <b>dirtree_flagread</b>() function is a simple wrapper, calling <b>dirtree_add_node</b>() 1311to create a root node relative to the current directory, then calling 1312<b>dirtree_handle_callback</b>() on that node (which recurses as instructed by the callback 1313return flags). The flags argument primarily lets you 1314control whether or not to follow symlinks to the root node; symlinks 1315listed on the command line are often treated differently than symlinks 1316encountered during recursive directory traversal. 1317 1318<p>The ls command not only bypasses this wrapper, but never returns 1319<b>DIRTREE_RECURSE</b> from the callback, instead calling <b>dirtree_recurse</b>() manually 1320from elsewhere in the program. This gives ls -lR manual control 1321of traversal order, which is neither depth first nor breadth first but 1322instead a sort of FIFO order requried by the ls standard.</p> 1323 1324<a name="toys"> 1325<h1><a href="#toys">Directory toys/</a></h1> 1326 1327<p>This directory contains command implementations. Each command is a single 1328self-contained file. Adding a new command involves adding a single 1329file, and removing a command involves removing that file. Commands use 1330shared infrastructure from the lib/ and generated/ directories.</p> 1331 1332<p>Currently there are five subdirectories under "toys/" containing "posix" 1333commands described in POSIX-2008, "lsb" commands described in the Linux 1334Standard Base 4.1, "other" commands not described by either standard, 1335"pending" commands awaiting cleanup (which default to "n" in menuconfig 1336because they don't necessarily work right yet), and "example" code showing 1337how toybox infrastructure works and providing template/skeleton files to 1338start new commands.</p> 1339 1340<p>The only difference directory location makes is which menu the command 1341shows up in during "make menuconfig", the directories are otherwise identical. 1342Note that the commands exist within a single namespace at runtime, so you can't 1343have the same command in multiple subdirectories. (The build tries to fail 1344informatively when you do that.)</p> 1345 1346<p>There is one more sub-menus in "make menuconfig" containing global 1347configuration options for toybox. This menu is defined in the top level 1348Config.in.</p> 1349 1350<p>See <a href="#adding">adding a new command</a> for details on the 1351layout of a command file.</p> 1352 1353<a name="scripts"> 1354<h2>Directory scripts/</h2> 1355 1356<p>Build infrastructure. The makefile calls scripts/make.sh for "make" 1357and scripts/install.sh for "make install".</p> 1358 1359<p>There's also a test suite, "make test" calls make/test.sh, which runs all 1360the tests in make/test/*. You can run individual tests via 1361"scripts/test.sh command", or "TEST_HOST=1 scripts/test.sh command" to run 1362that test against the host implementation instead of the toybox one.</p> 1363 1364<h3>scripts/cfg2files.sh</h3> 1365 1366<p>Run .config through this filter to get a list of enabled commands, which 1367is turned into a list of files in toys via a sed invocation in the top level 1368Makefile. 1369</p> 1370 1371<h2>Directory kconfig/</h2> 1372 1373<p>Menuconfig infrastructure copied from the Linux kernel. See the 1374Linux kernel's Documentation/kbuild/kconfig-language.txt</p> 1375 1376<!-- todo 1377 1378Better OLDTOY and multiple command explanation. From Config.in: 1379 1380<p>A command with multiple names (or multiple similar commands implemented in 1381the same .c file) should have config symbols prefixed with the name of their 1382C file. I.E. config symbol prefixes are NEWTOY() names. If OLDTOY() names 1383have config symbols they must be options (symbols with an underscore and 1384suffix) to the NEWTOY() name. (See generated/toylist.h)</p> 1385--> 1386 1387<!--#include file="footer.html" --> 1388