toybox/www/code.html
<<
>>
Prefs
   1<html><head><title>toybox source code walkthrough</title></head>
   2<!--#include file="header.html" -->
   3
   4<p><h1><a name="style" /><a href="#style">Code style</a></h1></p>
   5
   6<p>The primary goal of toybox is _simple_ code. Keeping the code small is
   7second, with speed and lots of features coming in somewhere after that.
   8(For more on that, see the <a href=design.html>design</a> page.)</p>
   9
  10<p>A simple implementation usually takes up fewer lines of source code,
  11meaning more code can fit on the screen at once, meaning the programmer can
  12see more of it on the screen and thus keep more if in their head at once.
  13This helps code auditing and thus reduces bugs. That said, sometimes being
  14more explicit is preferable to being clever enough to outsmart yourself:
  15don't be so terse your code is unreadable.</p>
  16
  17<p>Toybox has an actual coding style guide over on
  18<a href=design.html#codestyle>the design page</a>, but in general we just
  19want the code to be consistent.</p>
  20
  21<p><h1><a name="building" /><a href="#building">Building Toybox</a></h1></p>
  22
  23<p>Toybox is configured using the
  24<a href=https://github.com/torvalds/linux/blob/v2.6.16/Documentation/kbuild/kconfig-language.txt>Kconfig language</a> pioneered by the Linux
  25kernel, and adopted by many other projects (buildroot, OpenEmbedded, etc).
  26This generates a ".config" file containing the selected options, which
  27controls which features are included when compiling toybox.</p>
  28
  29<p>Each configuration option has a default value. The defaults indicate the
  30"maximum sane configuration", I.E. if the feature defaults to "n" then it
  31either isn't complete or is a special-purpose option (such as debugging
  32code) that isn't intended for general purpose use.</p>
  33
  34<p>For a more compact human-editable version .config files, you can use the
  35<a href=http://landley.net/aboriginal/FAQ.html#dev_miniconfig>miniconfig</a>
  36format.</p>
  37
  38<p>The standard build invocation is:</p>
  39
  40<ul>
  41<li>make defconfig #(or menuconfig)</li>
  42<li>make</li>
  43<li>make install</li>
  44</ul>
  45
  46<p>Type "make help" to see all available build options.</p>
  47
  48<p>The file "configure" contains a number of environment variable definitions
  49which influence the build, such as specifying which compiler to use or where
  50to install the resulting binaries. This file is included by the build, but
  51accepts existing definitions of the environment variables, so it may be sourced
  52or modified by the developer before building and the definitions exported
  53to the environment will take precedence.</p>
  54
  55<p>(To clarify: ".config" lists the features selected by defconfig/menuconfig,
  56I.E. "what to build", and "configure" describes the build and installation
  57environment, I.E. "how to build it".)</p>
  58
  59<p>By default "make install" puts files in /usr/toybox. Adding this to the
  60$PATH is up to you. The environment variable $PREFIX can change the
  61install location, ala "PREFIX=/usr/local/bin make install".</p>
  62
  63<p>If you need an unstripped (debug) version of any of these binaries,
  64look in generated/unstripped.</p>
  65
  66<p><h1><a name="running"><a href="#running">Running a command</a></h1></p>
  67
  68<h2>main</h2>
  69
  70<p>The toybox main() function is at the end of main.c at the top level. It has
  71two possible codepaths, only one of which is configured into any given build
  72of toybox.</p>
  73
  74<p>If CONFIG_SINGLE is selected, toybox is configured to contain only a single
  75command, so most of the normal setup can be skipped. In this case the
  76multiplexer isn't used, instead main() calls toy_singleinit() (also in main.c)
  77to set up global state and parse command line arguments, calls the command's
  78main function out of toy_list (in the CONFIG_SINGLE case the array has a single entry, no need to search), and if the function returns instead of exiting
  79it flushes stdout (detecting error) and returns toys.exitval.</p>
  80
  81<p>When CONFIG_SINGLE is not selected, main() uses basename() to find the
  82name it was run as, shifts its argument list one to the right so it lines up
  83with where the multiplexer function expects it, and calls toybox_main(). This
  84leverages the multiplexer command's infrastructure to find and run the
  85appropriate command. (A command name starting with "toybox" will
  86recursively call toybox_main(); you can go "./toybox toybox toybox toybox ls"
  87if you want to...)</p>
  88
  89<h2>toybox_main</h2>
  90
  91<p>The toybox_main() function is also in main,c. It handles a possible
  92--help option ("toybox --help ls"), prints the list of available commands if no
  93arguments were provided to the multiplexer (or with full path names if any
  94other option is provided before a command name, ala "toybox --list").
  95Otherwise it calls toy_exec() on its argument list.</p>
  96
  97<p>Note that the multiplexer is the first entry in toy_list (the rest of the
  98list is sorted alphabetically to allow binary search), so toybox_main can
  99cheat and just grab the first entry to quickly set up its context without
 100searching. Since all command names go through the multiplexer at least once
 101in the non-TOYBOX_SINGLE case, this avoids a redundant search of
 102the list.</p>
 103
 104<p>The toy_exec() function is also in main.c. It performs toy_find() to
 105perform a binary search on the toy_list array to look up the command's
 106entry by name and saves it in the global variable which, calls toy_init()
 107to parse command line arguments and set up global state (using which->options),
 108and calls the appropriate command's main() function (which->toy_main). On
 109return it flushes all pending ansi FILE * I/O, detects if stdout had an
 110error, and then calls xexit() (which uses toys.exitval).</p>
 111
 112<p><h1><a name="infrastructure" /><a href="#infrastructure">Infrastructure</a></h1></p>
 113
 114<p>The toybox source code is in following directories:</p>
 115<ul>
 116<li>The <a href="#top">top level directory</a> contains the file main.c (were
 117execution starts), the header file toys.h (included by every command), and
 118other global infrastructure.</li>
 119<li>The <a href="#lib">lib directory</a> contains common functions shared by
 120multiple commands:</li>
 121<ul>
 122<li><a href="#lib_lib">lib/lib.c</a></li>
 123<li><a href="#lib_xwrap">lib/xwrap.c</a></li>
 124<li><a href="#lib_llist">lib/llist.c</a></li>
 125<li><a href="#lib_args">lib/args.c</a></li>
 126<li><a href="#lib_dirtree">lib/dirtree.c</a></li>
 127</ul>
 128<li>The <a href="#toys">toys directory</a> contains the C files implementating
 129each command. Currently it contains five subdirectories categorizing the
 130commands: posix, lsb, other, example, and pending.</li>
 131<li>The <a href="#scripts">scripts directory</a> contains the build and
 132test infrastructure.</li>
 133<li>The <a href="#kconfig">kconfig directory</a> contains the configuration
 134infrastructure implementing menuconfig (copied from the Linux kernel).</li>
 135<li>The <a href="#generated">generated directory</a> contains intermediate
 136files generated from other parts of the source code.</li>
 137<li>The <a href="#tests">tests directory</a> contains the test suite.
 138NOSPACE=1 to allow tests to pass with diff -b</li>
 139</ul>
 140
 141<a name="adding" />
 142<p><h1><a href="#adding">Adding a new command</a></h1></p>
 143<p>To add a new command to toybox, add a C file implementing that command to
 144one of the subdirectories under the toys directory.  No other files need to
 145be modified; the build extracts all the information it needs (such as command
 146line arguments) from specially formatted comments and macros in the C file.
 147(See the description of the <a href="#generated">"generated" directory</a>
 148for details.)</p>
 149
 150<p>Currently there are five subdirectories under "toys", one for commands
 151defined by the POSIX standard, one for commands defined by the Linux Standard
 152Base, an "other" directory for commands not covered by an obvious standard,
 153a directory of example commands (templates to use when starting new commands),
 154and a "pending" directory of commands that need further review/cleanup
 155before moving to one of the other directories (run these at your own risk,
 156cleanup patches welcome).
 157These directories are just for developer convenience sorting the commands,
 158the directories are otherwise functionally identical. To add a new category,
 159create the appropriate directory with a README file in it whose first line
 160is the description menuconfig should use for the directory.)</p>
 161
 162<p>An easy way to start a new command is copy the file "toys/example/hello.c"
 163to the name of the new command, and modify this copy to implement the new
 164command (more or less by turning every instance of "hello" into the
 165name of your command, updating the command line arguments, globals, and
 166help data, and then filling out its "main" function with code that does
 167something interesting).</p> 
 168
 169<p>You could also start with "toys/example/skeleton.c", which provides a lot
 170more example code (showing several variants of command line option
 171parsing, how to implement multiple commands in the same file, and so on).
 172But usually it's just more stuff to delete.</p>
 173
 174<p>Here's a checklist of steps to turn hello.c into another command:</p>
 175
 176<ul>
 177<li><p>First "cp toys/example/hello.c toys/other/yourcommand.c" and open
 178the new file in your preferred text editor.</p>
 179<ul><li><p>Note that the
 180name of the new file is significant: it's the name of the new command you're
 181adding to toybox. The build includes all *.c files under toys/*/ whose
 182names are a case insensitive match for an enabled config symbol. So
 183toys/posix/cat.c only gets included if you have "CAT=y" in ".config".</p></li>
 184</ul></p></li>
 185
 186<li><p>Change the one line comment at the top of the file (currently
 187"hello.c - A hello world program") to describe your new file.</p></li>
 188
 189<li><p>Change the copyright notice to your name, email, and the current
 190year.</p></li>
 191
 192<li><p>Give a URL to the relevant standards document, where applicable.
 193(Sample links to SUSv4, LSB, IETF RFC, and man7.org are provided, feel free to
 194link to other documentation or standards as appropriate.)</p></li>
 195
 196<li><p>Update the USE_YOURCOMMAND(NEWTOY(yourcommand,"blah",0)) line.
 197The NEWTOY macro fills out this command's <a href="#toy_list">toy_list</a>
 198structure.  The arguments to the NEWTOY macro are:</p>
 199
 200<ol>
 201<li><p>the name used to run your command</p></li>
 202<li><p>the command line argument <a href="#lib_args">option parsing string</a> (0 if none)</p></li>
 203<li><p>a bitfield of TOYFLAG values
 204(defined in toys.h) providing additional information such as where your
 205command should be installed on a running system, whether to blank umask
 206before running, whether or not the command must run as root (and thus should
 207retain root access if installed SUID), and so on.</p></li>
 208</ol>
 209</li>
 210
 211<li><p>Change the kconfig data (from "config YOURCOMMAND" to the end of the
 212comment block) to supply your command's configuration and help
 213information.  The uppper case config symbols are used by menuconfig, and are
 214also what the CFG_ and USE_() macros are generated from (see [TODO]).  The
 215help information here is used by menuconfig, and also by the "help" command to
 216describe your new command.  (See [TODO] for details.)  By convention,
 217unfinished commands default to "n" and finished commands default to "y",
 218so "make defconfig" selects all finished commands.  (Note, "finished" means
 219"ready to be used", not that it'll never change again.)<p>
 220
 221<p>Each help block should start with a "usage: yourcommand" line explaining
 222any command line arguments added by this config option.  The "help" command
 223outputs this text, and scripts/config2help.c in the build infrastructure
 224collates these usage lines for commands with multiple configuration
 225options when producing generated/help.h.</p>
 226</li>
 227
 228<li><p>Change the "#define FOR_hello" line to "#define FOR_yourcommand" right
 229before the "#include <toys.h>". (This selects the appropriate FLAG_ macros and
 230does a "#define TT this.yourcommand" so you can access the global variables
 231out of the space-saving union of structures. If you aren't using any command
 232flag bits and aren't defining a GLOBAL block, you can delete this line.)</p></li>
 233
 234<li><p>Update the GLOBALS() macro to contain your command's global
 235variables. If your command has no global variables, delete this macro.</p>
 236
 237<p>Variables in the GLOBALS() block are are stored in a space saving
 238<a href="#toy_union">union of structures</a> format, which may be accessed
 239using the TT macro as if TT were a global structure (so TT.membername).
 240If you specified two-character command line arguments in
 241NEWTOY(), the first few global variables will be initialized by the automatic
 242argument parsing logic, and the type and order of these variables must
 243correspond to the arguments specified in NEWTOY().
 244(See <a href="#lib_args">lib/args.c</a> for details.)</p>
 245
 246<blockquote><p>NOTE: the GLOBALS() block creates a "this.filename" entry
 247in generated/globals.h. If your toys/*/filename.c does not match the first
 248command name, you'll need to "#define TT this.filename" yourself before
 249#including toys.h if you want to use TT globals</p></blockquote>
 250</li>
 251
 252<li><p>Rename hello_main() to yourcommand_main().  This is the main() function
 253where execution of your command starts. Your command line options are
 254already sorted into this.optflags, this.optargs, this.optc, and the GLOBALS()
 255as appropriate by the time this function is called. (See
 256<a href="#lib_args">get_optflags()</a> for details.)</p></li>
 257
 258<li><p>Switch on TOYBOX_DEBUG in menuconfig (toybox global settings menu)
 259the first time you build and run your new command. If anything is wrong
 260with your option string, that will give you error messages.</p>
 261
 262<p>Otherwise it'll just segfault without
 263explanation when it falls off the end because it didn't find a matching
 264end parantheses for a longopt, or you put a nonexistent option in a square
 265bracket grouping... Since these kind of errors can only be caused by a
 266developer, not by end users, we don't normally want runtime checks for
 267them. Once you're happy with your option string, you can switch TOYBOX_DEBUG
 268back off.</p></li>
 269</ul>
 270
 271<a name="headers" /><h2><a href="#headers">Headers.</a></h2>
 272
 273<p>Commands are implemented as self-contained .c files, and generally don't
 274have their own .h files. If it's common code put it in lib/, and if it's
 275something like a local structure definition just put it in the command's .c
 276file. If it would only ever be #included from one place, inline it.
 277(The line between implementing multiple commands in a C file via OLDTOY()
 278to share infrastructure and moving that shared infrastructure to lib/ is a
 279judgement call. Try to figure out which is simplest.)</p>
 280
 281<p>The top level toys.h should #include all the standard (posix) headers
 282that any command uses. (Partly this is friendly to ccache and partly this
 283makes the command implementations shorter.) Individual commands should only
 284need to include nonstandard headers that might prevent that command from
 285building in some context we'd care about (and thus requiring that command to
 286be disabled to avoid a build break).</p>
 287
 288<p>Target-specific stuff (differences between compiler versions, libc versions,
 289or operating systems) should be confined to lib/portability.h and
 290lib/portability.c. (There's even some minimal compile-time environment probing
 291that writes data to generated/portability.h, see scripts/genconfig.sh.)</p>
 292
 293<p>Only include &lt;linux/*.h&gt; headers from individual commands (not from other
 294headers), and only if you really need to. Data that varies per architecture
 295is a good reason to include a header. If you just need a couple constants
 296that haven't changed since the 1990's, it's ok to #define them yourself or
 297just use the constant inline with a comment explaining what it is. (A
 298#define that's only used once isn't really helping.)</p>
 299
 300<p><a name="top" /><h1><a href="#top">Top level directory.</a></h1></p>
 301
 302<p>This directory contains global infrastructure.</p>
 303
 304<h3>toys.h</h3>
 305<p>Each command #includes "toys.h" as part of its standard prolog. It
 306may "#define FOR_commandname" before doing so to get some extra entries
 307specific to this command.</p>
 308
 309<p>This file sucks in most of the commonly used standard #includes, so
 310individual files can just #include "toys.h" and not have to worry about
 311stdargs.h and so on.  Individual commands still need to #include
 312special-purpose headers that may not be present on all systems (and thus would
 313prevent toybox from building that command on such a system with that command
 314enabled).  Examples include regex support, any "linux/" or "asm/" headers, mtab
 315support (mntent.h and sys/mount.h), and so on.</p>
 316
 317<p>The toys.h header also defines structures for most of the global variables
 318provided to each command by toybox_main().  These are described in
 319detail in the description for main.c, where they are initialized.</p>
 320
 321<p>The global variables are grouped into structures (and a union) for space
 322savings, to more easily track the amount of memory consumed by them,
 323so that they may be automatically cleared/initialized as needed, and so
 324that access to global variables is more easily distinguished from access to
 325local variables.</p>
 326
 327<h3>main.c</h3>
 328<p>Contains the main() function where execution starts, plus
 329common infrastructure to initialize global variables and select which command
 330to run.  The "toybox" multiplexer command also lives here.  (This is the
 331only command defined outside of the toys directory.)</p>
 332
 333<p>Execution starts in main() which trims any path off of the first command
 334name and calls toybox_main(), which calls toy_exec(), which calls toy_find()
 335and toy_init() before calling the appropriate command's function from
 336toy_list[] (via toys.which->toy_main()).
 337If the command is "toybox", execution recurses into toybox_main(), otherwise
 338the call goes to the appropriate commandname_main() from a C file in the toys
 339directory.</p>
 340
 341<p>The following global variables are defined in main.c:</p>
 342<ul>
 343<a name="toy_list" />
 344<li><p><b>struct toy_list toy_list[]</b> - array describing all the
 345commands currently configured into toybox.  The first entry (toy_list[0]) is
 346for the "toybox" multiplexer command, which runs all the other built-in commands
 347without symlinks by using its first argument as the name of the command to
 348run and the rest as that command's argument list (ala "./toybox echo hello").
 349The remaining entries are the commands in alphabetical order (for efficient
 350binary search).</p>
 351
 352<p>This is a read-only array initialized at compile time by
 353defining macros and #including generated/newtoys.h.</p>
 354
 355<p>Members of struct toy_list (defined in "toys.h") include:</p>
 356<ul>
 357<li><p>char *<b>name</b> - the name of this command.</p></li>
 358<li><p>void (*<b>toy_main</b>)(void) - function pointer to run this
 359command.</p></li>
 360<li><p>char *<b>options</b> - command line option string (used by
 361get_optflags() in lib/args.c to intialize toys.optflags, toys.optargs, and
 362entries in the toy's GLOBALS struct).  When this is NULL, no option
 363parsing is done before calling toy_main().</p></li>
 364<li><p>int <b>flags</b> - Behavior flags for this command.  The following flags are currently understood:</p>
 365
 366<ul>
 367<li><b>TOYFLAG_USR</b> - Install this command under /usr</li>
 368<li><b>TOYFLAG_BIN</b> - Install this command under /bin</li>
 369<li><b>TOYFLAG_SBIN</b> - Install this command under /sbin</li>
 370<li><b>TOYFLAG_NOFORK</b> - This command can be used as a shell builtin.</li>
 371<li><b>TOYFLAG_UMASK</b> - Call umask(0) before running this command.</li>
 372<li><b>TOYFLAG_STAYROOT</b> - Don't drop permissions for this command if toybox is installed SUID root.</li>
 373<li><b>TOYFLAG_NEEDROOT</b> - This command cannot function unless run with root access.</li>
 374</ul>
 375<br>
 376
 377<p>These flags are combined with | (or).  For example, to install a command
 378in /usr/bin, or together TOYFLAG_USR|TOYFLAG_BIN.</p>
 379</ul>
 380</li>
 381
 382<li><p><b>struct toy_context toys</b> - global structure containing information
 383common to all commands, initializd by toy_init() and defined in "toys.h".
 384Members of this structure include:</p>
 385<ul>
 386<li><p>struct toy_list *<b>which</b> - a pointer to this command's toy_list
 387structure.  Mostly used to grab the name of the running command
 388(toys->which.name).</p>
 389</li>
 390<li><p>int <b>exitval</b> - Exit value of this command.  Defaults to zero.  The
 391error_exit() functions will return 1 if this is zero, otherwise they'll
 392return this value.</p></li>
 393<li><p>char **<b>argv</b> - "raw" command line options, I.E. the original
 394unmodified string array passed in to main().  Note that modifying this changes
 395"ps" output, and is not recommended.  This array is null terminated; a NULL
 396entry indicates the end of the array.</p>
 397<p>Most commands don't use this field, instead the use optargs, optflags,
 398and the fields in the GLOBALS struct initialized by get_optflags().</p>
 399</li>
 400<li><p>unsigned <b>optflags</b> - Command line option flags, set by
 401<a href="#lib_args">get_optflags()</a>.  Indicates which of the command line options listed in
 402toys->which.options occurred this time.</p>
 403
 404<p>The rightmost command line argument listed in toys->which.options sets bit
 4051, the next one sets bit 2, and so on.  This means the bits are set in the same
 406order the binary digits would be listed if typed out as a string.  For example,
 407the option string "abcd" would parse the command line "-c" to set optflags to 2,
 408"-a" would set optflags to 8, and "-bd" would set optflags to 6 (4|2).</p>
 409
 410<p>Only letters are relevant to optflags.  In the string "a*b:c#d", d=1, c=2,
 411b=4, a=8.  Punctuation after a letter initializes global variables at the
 412start of the GLOBALS() block (see <a href="#toy_union">union toy_union this</a>
 413for details).</p>
 414
 415<p>The build infrastructure creates FLAG_ macros for each option letter,
 416corresponding to the bit position, so you can check (toys.optflags & FLAG_x)
 417to see if a flag was specified. (The correct set of FLAG_ macros is selected
 418by defining FOR_mycommand before #including toys.h. The macros live in
 419toys/globals.h which is generated by scripts/make.sh.)</p>
 420
 421<p>For more information on option parsing, see <a href="#lib_args">get_optflags()</a>.</p>
 422
 423</li>
 424<li><p>char **<b>optargs</b> - Null terminated array of arguments left over
 425after get_optflags() removed all the ones it understood.  Note: optarg[0] is
 426the first argument, not the command name.  Use toys.which->name for the command
 427name.</p></li>
 428<li><p>int <b>optc</b> - Optarg count, equivalent to argc but for
 429optargs[].<p></li>
 430</ul>
 431
 432<a name="toy_union" />
 433<li><p><b>union toy_union this</b> - Union of structures containing each
 434command's global variables.</p>
 435
 436<p>Global variables are useful: they reduce the overhead of passing extra
 437command line arguments between functions, they conveniently start prezeroed to
 438save initialization costs, and the command line argument parsing infrastructure
 439can also initialize global variables with its results.</p>
 440
 441<p>But since each toybox process can only run one command at a time, allocating
 442space for global variables belonging to other commands you aren't currently
 443running would be wasteful.</p>
 444
 445<p>Toybox handles this by encapsulating each command's global variables in
 446a structure, and declaring a union of those structures with a single global
 447instance (called "this").  The GLOBALS() macro contains the global
 448variables that should go in the current command's global structure.  Each
 449variable can then be accessed as "this.commandname.varname".
 450If you #defined FOR_commandname before including toys.h, the macro TT is
 451#defined to this.commandname so the variable can then be accessed as
 452"TT.variable".  See toys/hello.c for an example.</p>
 453
 454<p>A command that needs global variables should declare a structure to
 455contain them all, and add that structure to this union.  A command should never
 456declare global variables outside of this, because such global variables would
 457allocate memory when running other commands that don't use those global
 458variables.</p>
 459
 460<p>The first few fields of this structure can be intialized by <a href="#lib_args">get_optargs()</a>,
 461as specified by the options field off this command's toy_list entry.  See
 462the get_optargs() description in lib/args.c for details.</p>
 463</li>
 464
 465<li><b>char toybuf[4096]</b> - a common scratch space buffer guaranteed
 466to start zeroed, so commands don't need to allocate/initialize their own.
 467Any command is free to use this, and it should never be directly referenced
 468by functions in lib/ (although commands are free to pass toybuf in to a
 469library function as an argument).</li>
 470
 471<li><b>char libbuf[4096]</b> - like toybuf, but for use by common code in
 472lib/*.c. Commands should never directly reference libbuf, and library
 473could should nnever directly reference toybuf.</li>
 474</ul>
 475
 476<p>The following functions are defined in main.c:</p>
 477<ul>
 478<li><p>struct toy_list *<b>toy_find</b>(char *name) - Return the toy_list
 479structure for this command name, or NULL if not found.</p></li>
 480<li><p>void <b>toy_init</b>(struct toy_list *which, char *argv[]) - fill out
 481the global toys structure, calling get_optargs() if necessary.</p></li>
 482<li><p>void <b>toy_exec</b>(char *argv[]) - Run a built-in command with
 483arguments.</p>
 484<p>Calls toy_find() on argv[0] (which must be just a command name
 485without path).  Returns if it can't find this command, otherwise calls
 486toy_init(), toys->which.toy_main(), and exit() instead of returning.</p>
 487
 488<p>Use the library function xexec() to fall back to external executables
 489in $PATH if toy_exec() can't find a built-in command.  Note that toy_exec()
 490does not strip paths before searching for a command, so "./command" will
 491never match an internal command.</li>
 492
 493<li><p>void <b>toybox_main</b>(void) - the main function for the multiplexer
 494command (I.E. "toybox").  Given a command name as its first argument, calls
 495toy_exec() on its arguments.  With no arguments, it lists available commands.
 496If the first argument starts with "-" it lists each command with its default
 497install path prepended.</p></li>
 498
 499</ul>
 500
 501<h3>Config.in</h3>
 502
 503<p>Top level configuration file in a stylized variant of
 504<a href=http://kernel.org/doc/Documentation/kbuild/kconfig-language.txt>kconfig</a> format.  Includes generated/Config.in.</p>
 505
 506<p>These files are directly used by "make menuconfig" to select which commands
 507to build into toybox (thus generating a .config file), and by
 508scripts/config2help.py to create generated/help.h.</p>
 509
 510<a name="generated" />
 511<h1><a href="#generated">Temporary files:</a></h1>
 512
 513<p>There is one temporary file in the top level source directory:</p>
 514<ul>
 515<li><p><b>.config</b> - Configuration file generated by kconfig, indicating
 516which commands (and options to commands) are currently enabled.  Used
 517to make generated/config.h and determine which toys/*/*.c files to build.</p>
 518
 519<p>You can create a human readable "miniconfig" version of this file using
 520<a href=http://landley.net/aboriginal/new_platform.html#miniconfig>these
 521instructions</a>.</p>
 522</li>
 523</ul>
 524
 525<p><h2>Directory generated/</h2></p>
 526
 527<p>The remaining temporary files live in the "generated/" directory,
 528which is for files generated at build time from other source files.</p>
 529
 530<ul>
 531<li><p><b>generated/Config.in</b> - Kconfig entries for each command, included
 532from the top level Config.in. The help text here is used to generate
 533help.h.</p>
 534
 535<p>Each command has a configuration entry with an upper case version of
 536the command name. Options to commands start with the command
 537name followed by an underscore and the option name. Global options are attached
 538to the "toybox" command, and thus use the prefix "TOYBOX_".  This organization
 539is used by scripts/cfg2files to select which toys/*/*.c files to compile for a
 540given .config.</p>
 541</li>
 542
 543<li><p><b>generated/config.h</b> - list of CFG_SYMBOL and USE_SYMBOL() macros,
 544generated from .config by a sed invocation in scripts/make.sh.</p>
 545
 546<p>CFG_SYMBOL is a comple time constant set to 1 for enabled symbols and 0 for
 547disabled symbols. This allows the use of normal if() statements to remove
 548code at compile time via the optimizer's dead code elimination (which removes
 549from the binary any code that cannot be reached). This saves space without
 550cluttering the code with #ifdefs or leading to configuration dependent build
 551breaks. (See the 1992 Usenix paper
 552<a href=http://doc.cat-v.org/henry_spencer/ifdef_considered_harmful.pdf>#ifdef
 553Considered Harmful</a> for more information.)</p>
 554
 555<p>When you can't entirely avoid an #ifdef, the USE_SYMBOL(code) macro
 556provides a less intrusive alternative, evaluating to the code in parentheses
 557when the symbol is enabled, and nothing when the symbol is disabled. This
 558is most commonly used around NEWTOY() declarations (so only the enabled
 559commands show up in toy_list), and in option strings. This can also be used
 560for things like varargs or structure members which can't always be
 561eliminated by a simple test on CFG_SYMBOL. Remember, unlike CFG_SYMBOL
 562this is really just a variant of #ifdef, and can still result in configuration
 563dependent build breaks. Use with caution.</p>
 564</li>
 565
 566<li><p><b>generated/flags.h</b> - FLAG_? macros indicating which command
 567line options were seen. The option parsing in lib/args.c sets bits in
 568toys.optflags, which can be tested by anding with the appropriate FLAG_
 569macro. (Bare longopts, which have no corresponding short option, will
 570have the longopt name after FLAG_. All others use the single letter short
 571option.)</p>
 572
 573<p>To get the appropriate macros for your command, #define FOR_commandname
 574before #including toys.h. To switch macro sets (because you have an OLDTOY()
 575with different options in the same .c file), #define CLEANUP_oldcommand
 576and also #define FOR_newcommand, then #include "generated/flags.h" to switch.
 577</p>
 578</li>
 579
 580<li><p><b>generated/globals.h</b> -
 581Declares structures to hold the contents of each command's GLOBALS(),
 582and combines them into "global_union this". (Yes, the name was
 583chosen to piss off C++ developers who think that C
 584is merely a subset of C++, not a language in its own right.)</p>
 585
 586<p>The union reuses the same memory for each command's global struct:
 587since only one command's globals are in use at any given time, collapsing
 588them together saves space. The headers #define TT to the appropriate
 589"this.commandname", so you can refer to the current command's global
 590variables out of "this" as TT.variablename.</p>
 591
 592<p>The globals start zeroed, and the first few are filled out by the 
 593lib/args.c argument parsing code called from main.c.</p>
 594</li>
 595
 596<li><p><b>toys/help.h</b> - Help strings for use by the "help" command and
 597--help options. This file #defines a help_symbolname string for each
 598symbolname, but only the symbolnames matching command names get used
 599by show_help() in lib/help.c to display help for commands.</p>
 600
 601<p>This file is created by scripts/make.sh, which compiles scripts/config2help.c
 602into the binary generated/config2help, and then runs it against the top
 603level .config and Config.in files to extract the help text from each config
 604entry and collate together dependent options.</p>
 605
 606<p>This file contains help text for all commands, regardless of current
 607configuration, but only the ones currently enabled in the .config file
 608wind up in the help_data[] array, and only the enabled dependent options
 609have their help text added to the command they depend on.</p>
 610</li>
 611
 612<li><p><b>generated/newtoys.h</b> - 
 613All the NEWTOY() and OLDTOY() macros from toys/*/*.c. The "toybox" multiplexer
 614is the first entry, the rest are in alphabetical order. Each line should be
 615inside an appropriate USE_ macro, so code that #includes this file only sees
 616the currently enabled commands.</p>
 617
 618<p>By #definining NEWTOY() to various things before #including this file,
 619it may be used to create function prototypes (in toys.h), initialize the
 620help_data array (in lib/help.c),  initialize the toy_list array (in main.c,
 621the alphabetical order lets toy_find() do a binary search, the exception to
 622the alphabetical order lets it use the multiplexer without searching), and so
 623on.  (It's even used to initialize the NEED_OPTIONS macro, which produces a 1
 624or 0 for each command using command line option parsing, which is ORed together
 625to allow compile-time dead code elimination to remove the whole of
 626lib/args.c if nothing currently enabled is using it.)<p>
 627
 628<p>Each NEWTOY and OLDTOY macro contains the command name, command line
 629option string (telling lib/args.c how to parse command line options for
 630this command), recommended install location, and miscelaneous data such
 631as whether this command should retain root permissions if installed suid.</p>
 632</li>
 633
 634<li><p><b>toys/oldtoys.h</b> - Macros with the command line option parsing
 635string for each NEWTOY. This allows an OLDTOY that's just an alias for an
 636existing command to refer to the existing option string instead of
 637having to repeat it.</p>
 638</li>
 639</ul>
 640
 641<a name="lib">
 642<h2>Directory lib/</h2>
 643
 644<p>TODO: document lots more here.</p>
 645
 646<p>lib: getmountlist(), error_msg/error_exit, xmalloc(),
 647strlcpy(), xexec(), xopen()/xread(), xgetcwd(), xabspath(), find_in_path(),
 648itoa().</p>
 649
 650
 651
 652<a name="lib_xwrap"><h3>lib/xwrap.c</h3>
 653
 654<p>Functions prefixed with the letter x call perror_exit() when they hit
 655errors, to eliminate common error checking. This prints an error message
 656and the strerror() string for the errno encountered.</p>
 657
 658<p>We replaced exit(), _exit(), and atexit() with xexit(), _xexit(), and
 659sigatexit(). This gives _xexit() the option to siglongjmp(toys.rebound, 1)
 660instead of exiting, lets xexit() report stdout flush failures to stderr
 661and change the exit code to indicate error, lets our toys.exit function
 662change happen for signal exit paths and lets us remove the functions
 663after we've called them.</p>
 664
 665<p>You can intercept our exit by assigning a sigsetjmp/siglongjmp buffer to
 666toys.rebound (set it back to zero to restore the default behavior).
 667If you do this, cleaning up resource leaks is your problem.</p>
 668
 669<ul>
 670<li><b>void xstrncpy(char *dest, char *src, size_t size)</b></li>
 671<li><p><b><p>void _xexit(void)</b></p>
 672<p>Calls siglongjmp(toys.rebound, 1), or else _exit(toys.exitval). This
 673lets you ignore errors with the NO_EXIT() macro wrapper, or intercept
 674them with WOULD_EXIT().</p>
 675<li><b><p>void xexit(void)</b></p>
 676<p>Calls toys.xexit functions (if any) and flushes stdout/stderr (reporting
 677failure to write to stdout both to stderr and in the exit code), then
 678calls _xexit().</p>
 679</li>
 680<li><b>void *xmalloc(size_t size)</b></li>
 681<li><b>void *xzalloc(size_t size)</b></li>
 682<li><b>void *xrealloc(void *ptr, size_t size)</b></li>
 683<li><b>char *xstrndup(char *s, size_t n)</b></li>
 684<li><b>char *xstrdup(char *s)</b></li>
 685<li><b>char *xmprintf(char *format, ...)</b></li>
 686<li><b>void xprintf(char *format, ...)</b></li>
 687<li><b>void xputs(char *s)</b></li>
 688<li><b>void xputc(char c)</b></li>
 689<li><b>void xflush(void)</b></li>
 690<li><b>pid_t xfork(void)</b></li>
 691<li><b>void xexec_optargs(int skip)</b></li>
 692<li><b>void xexec(char **argv)</b></li>
 693<li><b>pid_t xpopen(char **argv, int *pipes)</b></li>
 694<li><b>int xpclose(pid_t pid, int *pipes)</b></li>
 695<li><b>void xaccess(char *path, int flags)</b></li>
 696<li><b>void xunlink(char *path)</b></li>
 697<li><p><b>int xcreate(char *path, int flags, int mode)<br />
 698int xopen(char *path, int flags)</b></p>
 699
 700<p>The xopen() and xcreate() functions open an existing file (exiting if
 701it's not there) and create a new file (exiting if it can't).</p>
 702
 703<p>They default to O_CLOEXEC so the filehandles aren't passed on to child
 704processes. Feed in O_CLOEXEC to disable this.</p>
 705</li>
 706<li><p><b>void xclose(int fd)</b></p>
 707
 708<p>Because NFS is broken, and won't necessarily perform the requested
 709operation (and report the error) until you close the file. Of course, this
 710being NFS, it's not guaranteed to report the error there either, but it
 711_can_.</p>
 712
 713<p>Nothing else ever reports an error on close, everywhere else it's just a
 714VFS operation freeing some resources. NFS is _special_, in a way that
 715other network filesystems like smbfs and v9fs aren't..</p>
 716</li>
 717<li><b>int xdup(int fd)</b></li>
 718<li><p><b>size_t xread(int fd, void *buf, size_t len)</b></p>
 719
 720<p>Can return 0, but not -1.</p>
 721</li>
 722<li><p><b>void xreadall(int fd, void *buf, size_t len)</b></p>
 723
 724<p>Reads the entire len-sized buffer, retrying to complete short
 725reads. Exits if it can't get enough data.</p></li>
 726
 727<li><p><b>void xwrite(int fd, void *buf, size_t len)</b></p>
 728
 729<p>Retries short writes, exits if can't write the entire buffer.</p></li>
 730
 731<li><b>off_t xlseek(int fd, off_t offset, int whence)</b></li>
 732<li><b>char *xgetcwd(void)</b></li>
 733<li><b>void xstat(char *path, struct stat *st)</b></li>
 734<li><p><b>char *xabspath(char *path, int exact) </b></p>
 735
 736<p>After several years of
 737<a href=http://landley.net/notes-2007.html#18-06-2007>wrestling</a>
 738<a href=http://landley.net/notes-2008.html#19-01-2008>with</a> realpath(), 
 739I broke down and <a href=http://landley.net/notes-2012.html#20-11-2012>wrote
 740my own</a> implementation that doesn't use the one in libc. As I explained:
 741
 742<blockquote><p>If the path ends with a broken link,
 743readlink -f should show where the link points to, not where the broken link
 744lives. (The point of readlink -f is "if I write here, where would it attempt
 745to create a file".) The problem is, realpath() returns NULL for a path ending
 746with a broken link, and I can't beat different behavior out of code locked
 747away in libc.</p></blockquote>
 748
 749<p>
 750</li>
 751<li><b>void xchdir(char *path)</b></li>
 752<li><b>void xchroot(char *path)</b></li>
 753
 754<li><p><b>struct passwd *xgetpwuid(uid_t uid)<br />
 755struct group *xgetgrgid(gid_t gid)<br />
 756struct passwd *xgetpwnam(char *name)</b></p>
 757</li>
 758
 759<li><b>void xsetuser(struct passwd *pwd)</b></li>
 760<li><b>char *xreadlink(char *name)</b></li>
 761<li><b>char *xreadfile(char *name, char *buf, off_t len)</b></li>
 762<li><b>int xioctl(int fd, int request, void *data)</b></li>
 763<li><b>void xpidfile(char *name)</b></li>
 764<li><b>void xsendfile(int in, int out)</b></li>
 765<li><b>long xparsetime(char *arg, long units, long *fraction)</b></li>
 766<li><b>void xregcomp(regex_t *preg, char *regex, int cflags)</b></li>
 767</ul>
 768
 769<a name="lib_lib"><h3>lib/lib.c</h3>
 770<p>Eight gazillion common functions, see lib/lib.h for the moment:</p>
 771
 772<h3>lib/portability.h</h3>
 773
 774<p>This file is automatically included from the top of toys.h, and smooths
 775over differences between platforms (hardware targets, compilers, C libraries,
 776operating systems, etc).</p>
 777
 778<p>This file provides SWAP macros (SWAP_BE16(x) and SWAP_LE32(x) and so on).</p>
 779
 780<p>A macro like SWAP_LE32(x) means "The value in x is stored as a little
 781endian 32 bit value, so perform the translation to/from whatever the native
 78232-bit format is".  You do the swap once on the way in, and once on the way
 783out. If your target is already little endian, the macro is a NOP.</p>
 784
 785<p>The SWAP macros come in BE and LE each with 16, 32, and 64 bit versions.
 786In each case, the name of the macro refers to the _external_ representation,
 787and converts to/from whatever your native representation happens to be (which
 788can vary depending on what you're currently compiling for).</p>
 789
 790<a name="lib_llist"><h3>lib/llist.c</h3>
 791
 792<p>Some generic single and doubly linked list functions, which take
 793advantage of a couple properties of C:</p>
 794
 795<ul>
 796<li><p>Structure elements are laid out in memory in the order listed, and
 797the first element has no padding. This means you can always treat (typecast)
 798a pointer to a structure as a pointer to the first element of the structure,
 799even if you don't know anything about the data following it.</p></li>
 800
 801<li><p>An array of length zero at the end of a structure adds no space
 802to the sizeof() the structure, but if you calculate how much extra space
 803you want when you malloc() the structure it will be available at the end.
 804Since C has no bounds checking, this means each struct can have one variable
 805length array.</p></li>
 806</ul>
 807
 808<p>Toybox's list structures always have their <b>next</b> pointer as
 809the first entry of each struct, and singly linked lists end with a NULL pointer.
 810This allows generic code to traverse such lists without knowing anything
 811else about the specific structs composing them: if your pointer isn't NULL
 812typecast it to void ** and dereference once to get the next entry.</p>
 813
 814<p><b>lib/lib.h</b> defines three structure types:</p>
 815<ul>
 816<li><p><b>struct string_list</b> - stores a single string (<b>char str[0]</b>),
 817memory for which is allocated as part of the node. (I.E. llist_traverse(list,
 818free); can clean up after this type of list.)</p></li>
 819
 820<li><p><b>struct arg_list</b> - stores a pointer to a single string
 821(<b>char *arg</b>) which is stored in a separate chunk of memory.</p></li>
 822
 823<li><p><b>struct double_list</b> - has a second pointer (<b>struct double_list
 824*prev</b> along with a <b>char *data</b> for payload.</p></li>
 825</ul>
 826
 827<b>List Functions</b>
 828
 829<ul>
 830<li><p>void *<b>llist_pop</b>(void **list) - advances through a list ala
 831<b>node = llist_pop(&list);</b>  This doesn't modify the list contents,
 832but does advance the pointer you feed it (which is why you pass the _address_
 833of that pointer, not the pointer itself).</p></li>
 834
 835<li><p>void <b>llist_traverse</b>(void *list, void (*using)(void *data)) -
 836iterate through a list calling a function on each node.</p></li>
 837
 838<li><p>struct double_list *<b>dlist_add</b>(struct double_list **llist, char *data)
 839- append an entry to a circular linked list.
 840This function allocates a new struct double_list wrapper and returns the
 841pointer to the new entry (which you can usually ignore since it's llist->prev,
 842but if llist was NULL you need it). The argument is the ->data field for the
 843new node.</p></li>
 844<ul><li><p>void <b>dlist_add_nomalloc</b>(struct double_list **llist,
 845struct double_list *new) - append existing struct double_list to
 846list, does not allocate anything.</p></li></ul>
 847</ul>
 848
 849<b>List code trivia questions:</b>
 850
 851<ul>
 852<li><p><b>Why do arg_list and double_list contain a char * payload instead of
 853a void *?</b> - Because you always have to typecast a void * to use it, and
 854typecasting a char * does no harm. Since strings are the most common
 855payload, and doing math on the pointer ala
 856"(type *)(ptr+sizeof(thing)+sizeof(otherthing))" requires ptr to be char *
 857anyway (at least according to the C standard), defaulting to char * saves
 858a typecast.</p>
 859</li>
 860
 861<li><p><b>Why do the names ->str, ->arg, and ->data differ?</b> - To force
 862you to keep track of which one you're using, calling free(node->str) would
 863be bad, and _failing_ to free(node->arg) leaks memory.</p></li>
 864
 865<li><p><b>Why does llist_pop() take a void * instead of void **?</b> -
 866because the stupid compiler complains about "type punned pointers" when
 867you typecast and dereference on the same line,
 868due to insane FSF developers hardwiring limitations of their optimizer
 869into gcc's warning system. Since C automatically typecasts any other
 870pointer type to and from void *, the current code works fine. It's sad that it
 871won't warn you if you forget the &, but the code crashes pretty quickly in
 872that case.</p></li>
 873
 874<li><p><b>How do I assemble a singly-linked-list in order?</b> - use
 875a double_list, dlist_add() your entries, and then call dlist_terminate(list)
 876to break the circle when done (turning the last ->next and the first ->prev
 877into NULLs).</p>
 878</ul>
 879
 880<a name="lib_args"><h3>lib/args.c</h3>
 881
 882<p>Toybox's main.c automatically parses command line options before calling the
 883command's main function. Option parsing starts in get_optflags(), which stores
 884results in the global structures "toys" (optflags and optargs) and "this".</p>
 885
 886<p>The option parsing infrastructure stores a bitfield in toys.optflags to
 887indicate which options the current command line contained, and defines FLAG
 888macros code can use to check whether each argument's bit is set. Arguments
 889attached to those options are saved into the command's global structure
 890("this"). Any remaining command line arguments are collected together into
 891the null-terminated array toys.optargs, with the length in toys.optc. (Note
 892that toys.optargs does not contain the current command name at position zero,
 893use "toys.which->name" for that.) The raw command line arguments get_optflags()
 894parsed are retained unmodified in toys.argv[].</p>
 895
 896<p>Toybox's option parsing logic is controlled by an "optflags" string, using
 897a format reminiscent of getopt's optargs but with several important differences.
 898Toybox does not use the getopt()
 899function out of the C library, get_optflags() is an independent implementation
 900which doesn't permute the original arguments (and thus doesn't change how the
 901command is displayed in ps and top), and has many features not present in
 902libc optargs() (such as the ability to describe long options in the same string
 903as normal options).</p>
 904
 905<p>Each command's NEWTOY() macro has an optflags string as its middle argument,
 906which sets toy_list.options for that command to tell get_optflags() what
 907command line arguments to look for, and what to do with them.
 908If a command has no option
 909definition string (I.E. the argument is NULL), option parsing is skipped
 910for that command, which must look at the raw data in toys.argv to parse its
 911own arguments. (If no currently enabled command uses option parsing,
 912get_optflags() is optimized out of the resulting binary by the compiler's
 913--gc-sections option.)</p>
 914
 915<p>You don't have to free the option strings, which point into the environment
 916space (I.E. the string data is not copied). A TOYFLAG_NOFORK command
 917that uses the linked list type "*" should free the list objects but not
 918the data they point to, via "llist_free(TT.mylist, NULL);". (If it's not
 919NOFORK, exit() will free all the malloced data anyway unless you want
 920to implement a CONFIG_TOYBOX_FREE cleanup for it.)</p>
 921
 922<h4>Optflags format string</h4>
 923
 924<p>Note: the optflags option description string format is much more
 925concisely described by a large comment at the top of lib/args.c.</p>
 926
 927<p>The general theory is that letters set optflags, and punctuation describes
 928other actions the option parsing logic should take.</p>
 929
 930<p>For example, suppose the command line <b>command -b fruit -d walrus -a 42</b>
 931is parsed using the optflags string "<b>a#b:c:d</b>".  (I.E.
 932toys.which->options="a#b:c:d" and argv = ["command", "-b", "fruit", "-d",
 933"walrus", "-a", "42"]).  When get_optflags() returns, the following data is
 934available to command_main():
 935
 936<ul>
 937<li><p>In <b>struct toys</b>:
 938<ul>
 939<li>toys.optflags = 13; // FLAG_a = 8 | FLAG_b = 4 | FLAG_d = 1</li>
 940<li>toys.optargs[0] = "walrus"; // leftover argument</li>
 941<li>toys.optargs[1] = NULL; // end of list</li>
 942<li>toys.optc = 1; // there was 1 leftover argument</li>
 943<li>toys.argv[] = {"-b", "fruit", "-d", "walrus", "-a", "42"}; // The original command line arguments
 944</ul>
 945<p></li>
 946
 947<li><p>In <b>union this</b> (treated as <b>long this[]</b>):
 948<ul>
 949<li>this[0] = NULL; // -c didn't get an argument this time, so get_optflags() didn't change it and toys_init() zeroed "this" during setup.)</li>
 950<li>this[1] = (long)"fruit"; // argument to -b</li>
 951<li>this[2] = 42; // argument to -a</li>
 952</ul>
 953</p></li>
 954</ul>
 955
 956<p>If the command's globals are:</p>
 957
 958<blockquote><pre>
 959GLOBALS(
 960        char *c;
 961        char *b;
 962        long a;
 963)
 964</pre></blockquote>
 965
 966<p>That would mean TT.c == NULL, TT.b == "fruit", and TT.a == 42.  (Remember,
 967each entry that receives an argument must be a long or pointer, to line up
 968with the array position.  Right to left in the optflags string corresponds to
 969top to bottom in GLOBALS().</p>
 970
 971<p>Put globals not filled out by the option parsing logic at the end of the
 972GLOBALS block. Common practice is to list the options one per line (to
 973make the ordering explicit, first to last in globals corresponds to right
 974to left in the option string), then leave a blank line before any non-option
 975globals.</p>
 976
 977<p><b>long toys.optflags</b></p>
 978
 979<p>Each option in the optflags string corresponds to a bit position in
 980toys.optflags, with the same value as a corresponding binary digit.  The
 981rightmost argument is (1<<0), the next to last is (1<<1) and so on.  If
 982the option isn't encountered while parsing argv[], its bit remains 0.</p>
 983
 984<p>Each option -x has a FLAG_x macro for the command letter. Bare --longopts
 985with no corresponding short option have a FLAG_longopt macro for the long
 986optionname. Commands enable these macros by #defining FOR_commandname before
 987#including <toys.h>. When multiple commands are implemented in the same
 988source file, you can switch flag contexts later in the file by
 989#defining CLEANUP_oldcommand and #defining FOR_newcommand, then
 990#including <generated/flags.h>.</p>
 991
 992<p>Options disabled in the current configuration (wrapped in
 993a USE_BLAH() macro for a CONFIG_BLAH that's switched off) have their
 994corresponding FLAG macro set to zero, so code checking them ala
 995if (toys.optargs & FLAG_x) gets optimized out via dead code elimination.
 996#defining FORCE_FLAGS when switching flag context disables this
 997behavior: the flag is never zero even if the config is disabled. This
 998allows code shared between multiple commands to use the same flag
 999values, as long as the common flags match up right to left in both option
1000strings.</p>
1001
1002<p>For example,
1003the optflags string "abcd" would parse the command line argument "-c" to set
1004optflags to 2, "-a" would set optflags to 8, "-bd" would set optflags to
10056 (I.E. 4|2), and "-a -c" would set optflags to 10 (2|8). To check if -c
1006was encountered, code could test: if (toys.optflags & FLAG_c) printf("yup");
1007(See the toys/examples directory for more.)</p>
1008
1009<p>Only letters are relevant to optflags, punctuation is skipped: in the
1010string "a*b:c#d", d=1, c=2, b=4, a=8. The punctuation after a letter
1011usually indicate that the option takes an argument.</p>
1012
1013<p>Since toys.optflags is an unsigned int, it only stores 32 bits. (Which is
1014the amount a long would have on 32-bit platforms anyway; 64 bit code on
101532 bit platforms is too expensive to require in common code used by almost
1016all commands.) Bit positions beyond the 1<<31 aren't recorded, but
1017parsing higher options can still set global variables.</p>
1018
1019<p><b>Automatically setting global variables from arguments (union this)</b></p>
1020
1021<p>The following punctuation characters may be appended to an optflags
1022argument letter, indicating the option takes an additional argument:</p>
1023
1024<ul>
1025<li><b>:</b> - plus a string argument, keep most recent if more than one.</li>
1026<li><b>*</b> - plus a string argument, appended to a linked list.</li>
1027<li><b>@</b> - plus an occurrence counter (stored in a long)</li>
1028<li><b>#</b> - plus a signed long argument.
1029<li><b>-</b> - plus a signed long argument defaulting to negative (start argument with + to force a positive value).</li>
1030<li><b>.</b> - plus a floating point argument (if CFG_TOYBOX_FLOAT).</li>
1031<ul>The following can be appended to a float or double:
1032<li><b>&lt;123</b> - error if argument is less than this</li>
1033<li><b>&gt;123</b> - error if argument is greater than this</li>
1034<li><b>=123</b> - default value if argument not supplied</li>
1035</ul>
1036</ul>
1037
1038<p><b>GLOBALS</b></p>
1039
1040<p>Options which have an argument fill in the corresponding slot in the global
1041union "this" (see generated/globals.h), treating it as an array of longs
1042with the rightmost saved in this[0].  As described above, using "a*b:c#d",
1043"-c 42" would set this[0] = 42; and "-b 42" would set this[1] = "42"; each
1044slot is left NULL if the corresponding argument is not encountered.</p>
1045
1046<p>This behavior is useful because the LP64 standard ensures long and pointer
1047are the same size. C99 guarantees structure members will occur in memory
1048in the same order they're declared, and that padding won't be inserted between
1049consecutive variables of register size.  Thus the first few entries can
1050be longs or pointers corresponding to the saved arguments.</p>
1051
1052<p>The main downside is that numeric arguments ("#" and "-" format)
1053are limited to +- 2 billion on 32 bit platforms (the "truncate -s 8G"
1054problem), because long is only 64 bits on 64 bit hosts, so the capabilities
1055of some tools differ when built in 32 bit vs 64 bit mode. Fixing this
1056kind of ugly and even embedded designs are slowly moving to 64 bits,
1057so our current plan is to document the problem and wait it out. (If
1058"x32 mode" and similar becomes popular enough, we may revisit this
1059decision.)</p>
1060
1061<p>See toys/example/*.c for longer examples of parsing options into the
1062GLOBALS block.</p>
1063
1064<p><b>char *toys.optargs[]</b></p>
1065
1066<p>Command line arguments in argv[] which are not consumed by option parsing
1067(I.E. not recognized either as -flags or arguments to -flags) will be copied
1068to toys.optargs[], with the length of that array in toys.optc.
1069(When toys.optc is 0, no unrecognized command line arguments remain.)
1070The order of entries is preserved, and as with argv[] this new array is also
1071terminated by a NULL entry.</p>
1072
1073<p>Option parsing can require a minimum or maximum number of optargs left
1074over, by adding "<1" (read "at least one") or ">9" ("at most nine") to the
1075start of the optflags string.</p>
1076
1077<p>The special argument "--" terminates option parsing, storing all remaining
1078arguments in optargs.  The "--" itself is consumed.</p>
1079
1080<p><b>Other optflags control characters</b></p>
1081
1082<p>The following characters may occur at the start of each command's
1083optflags string, before any options that would set a bit in toys.optflags:</p>
1084
1085<ul>
1086<li><b>^</b> - stop at first nonoption argument (for nice, xargs...)</li>
1087<li><b>?</b> - allow unknown arguments (pass non-option arguments starting
1088with - through to optargs instead of erroring out).</li>
1089<li><b>&amp;</b> - the first argument has imaginary dash (ala tar/ps.  If given twice, all arguments have imaginary dash.)</li>
1090<li><b>&lt;</b> - must be followed by a decimal digit indicating at least this many leftover arguments are needed in optargs (default 0)</li>
1091<li><b>&gt;</b> - must be followed by a decimal digit indicating at most this many leftover arguments allowed (default MAX_INT)</li>
1092</ul>
1093
1094<p>The following characters may be appended to an option character, but do
1095not by themselves indicate an extra argument should be saved in this[].
1096(Technically any character not recognized as a control character sets an
1097optflag, but letters are never control characters.)</p>
1098
1099<ul>
1100<li><b>^</b> - stop parsing options after encountering this option, everything else goes into optargs.</li>
1101<li><b>|</b> - this option is required.  If more than one marked, only one is required.</li>
1102</ul>
1103
1104<p>The following may be appended to a float or double:</p>
1105
1106<ul>
1107<li><b>&lt;123</b> - error if argument is less than this</li>
1108<li><b>&gt;123</b> - error if argument is greater than this</li>
1109<li><b>=123</b> - default value if argument not supplied</li>
1110</ul>
1111
1112<p>Option parsing only understands <>= after . when CFG_TOYBOX_FLOAT
1113is enabled. (Otherwise the code to determine where floating point constants
1114end drops out.  When disabled, it can reserve a global data slot for the
1115argument so offsets won't change, but will never fill it out.) You can handle
1116this by using the USE_BLAH() macros with C string concatenation, ala:</p>
1117
1118<blockquote>"abc." USE_TOYBOX_FLOAT("<1.23>4.56=7.89") "def"</blockquote>
1119
1120<p><b>--longopts</b></p>
1121
1122<p>The optflags string can contain long options, which are enclosed in
1123parentheses. They may be appended to an existing option character, in
1124which case the --longopt is a synonym for that option, ala "a:(--fred)"
1125which understands "-a blah" or "--fred blah" as synonyms.</p>
1126
1127<p>Longopts may also appear before any other options in the optflags string,
1128in which case they have no corresponding short argument, but instead set
1129their own bit based on position. So for "(walrus)#(blah)xy:z", "command
1130--walrus 42" would set toys.optflags = 16 (-z = 1, -y = 2, -x = 4, --blah = 8)
1131and would assign this[1] = 42;</p>
1132
1133<p>A short option may have multiple longopt synonyms, "a(one)(two)", but
1134each "bare longopt" (ala "(one)(two)abc" before any option characters)
1135always sets its own bit (although you can group them with +X).</p>
1136
1137<p>Only bare longopts have a FLAG_ macro with the longopt name
1138(ala --fred would #define FLAG_fred). Other longopts use the short
1139option's FLAG macro to test the toys.optflags bit.</p>
1140
1141<p>Options with a semicolon ";" after their data type can only set their
1142corresponding GLOBALS() entry via "--longopt=value". For example, option
1143string "x(boing): y" would set TT.x if it saw "--boing=value", but would
1144treat "--boing value" as setting FLAG_x in toys.optargs, leaving TT.x NULL,
1145and keeping "value" in toys.optargs[]. (This lets "ls --color" and
1146"ls --color=auto" both work.)</p>
1147
1148<p><b>[groups]</b></p>
1149
1150<p>At the end of the option string, square bracket groups can define
1151relationships between existing options. (This only applies to short
1152options, bare --longopts can't participate.)</p>
1153
1154<p>The first character of the group defines the type, the remaining
1155characters are options it applies to:</p>
1156
1157<ul>
1158<li><b>-</b> - Exclusive, switch off all others in this group.</li>
1159<li><b>+</b> - Inclusive, switch on all others in this group.</li>
1160<li><b>!</b> - Error, fail if more than one defined.</li>
1161</ul>
1162
1163<p>So "abc[-abc]" means -ab = -b, -ba = -a, -abc = -c. "abc[+abc]"
1164means -ab=-abc, -c=-abc, and "abc[!abc] means -ab calls error_exit("no -b
1165with -a"). Note that [-] groups clear the GLOBALS option slot of
1166options they're switching back off, but [+] won't set options it didn't see
1167(just the optflags).</p>
1168
1169<p><b>whitespace</b></p>
1170
1171<p>Arguments may occur with or without a space (I.E. "-a 42" or "-a42").
1172The command line argument "-abc" may be interepreted many different ways:
1173the optflags string "cba" sets toys.optflags = 7, "c:ba" sets toys.optflags=4
1174and saves "ba" as the argument to -c, and "cb:a" sets optflags to 6 and saves
1175"c" as the argument to -b.</p>
1176
1177<p>Note that &amp; changes whitespace handling, so that the command line
1178"tar cvfCj outfile.tar.bz2 topdir filename" is parsed the same as
1179"tar filename -c -v -j -f outfile.tar.bz2 -C topdir". Note that "tar -cvfCj
1180one two three" would equal "tar -c -v -f Cj one two three". (This matches
1181historical usage.)</p>
1182
1183<p>Appending a space to the option in the option string ("a: b") makes it
1184require a space, I.E. "-ab" is interpreted as "-a" "-b". That way "kill -stop"
1185differs from "kill -s top".</p>
1186
1187<p>Appending ; to a longopt in the option string makes its argument optional,
1188and only settable with =, so in ls "(color):;" can accept "ls --color" and
1189"ls --color=auto" without complaining that the first has no argument.</p>
1190
1191<a name="lib_dirtree"><h3>lib/dirtree.c</h3>
1192
1193<p>The directory tree traversal code should be sufficiently generic
1194that commands never need to use readdir(), scandir(), or the fts.h family
1195of functions.</p>
1196
1197<p>These functions do not call chdir() or rely on PATH_MAX. Instead they
1198use openat() and friends, using one filehandle per directory level to
1199recurse into subdirectories. (I.E. they can descend 1000 directories deep
1200if setrlimit(RLIMIT_NOFILE) allows enough open filehandles, and the default
1201in /proc/self/limits is generally 1024.)</p>
1202
1203<p>There are two main ways to use dirtree: 1) assemble a tree of nodes
1204representing a snapshot of directory state and traverse them using the
1205->next and ->child pointers, or 2) traverse the tree calling a callback
1206function on each entry, and freeing its node afterwards. (You can also
1207combine the two, using the callback as a filter to determine which nodes
1208to keep.)</p>
1209
1210<p>The basic dirtree functions are:</p>
1211
1212<ul>
1213<li><p><b>struct dirtree *dirtree_read(char *path, int (*callback)(struct
1214dirtree node))</b> - recursively read files and directories, calling
1215callback() on each, and returning a tree of saved nodes (if any).
1216If path doesn't exist, returns DIRTREE_ABORTVAL. If callback is NULL,
1217returns a single node at that path.</p>
1218
1219<li><p><b>dirtree_notdotdot(struct dirtree *new)</b> - standard callback
1220which discards "." and ".." entries and returns DIRTREE_SAVE|DIRTREE_RECURSE
1221for everything else. Used directly, this assembles a snapshot tree of
1222the contents of this directory and its subdirectories
1223to be processed after dirtree_read() returns (by traversing the
1224struct dirtree's ->next and ->child pointers from the returned root node).</p>
1225
1226<li><p><b>dirtree_path(struct dirtree *node, int *plen)</b> - malloc() a
1227string containing the path from the root of this tree to this node. If
1228plen isn't NULL then *plen is how many extra bytes to malloc at the end
1229of string.</p></li>
1230
1231<li><p><b>dirtree_parentfd(struct dirtree *node)</b> - return fd of
1232directory containing this node, for use with openat() and such.</p></li>
1233</ul>
1234
1235<p>The <b>dirtree_read()</b> function is the standard way to start
1236directory traversal. It takes two arguments: a starting path for
1237the root of the tree, and a callback function. The callback() is called
1238on each directory entry, its argument is a fully populated
1239<b>struct dirtree *</b> (from lib/lib.h) describing the node, and its
1240return value tells the dirtree infrastructure what to do next.</p>
1241
1242<p>(There's also a three argument version,
1243<b>dirtree_flagread(char *path, int flags, int (*callback)(struct
1244dirtree node))</b>, which lets you apply flags like DIRTREE_SYMFOLLOW and
1245DIRTREE_SHUTUP to reading the top node, but this only affects the top node.
1246Child nodes use the flags returned by callback().</p>
1247
1248<p><b>struct dirtree</b></p>
1249
1250<p>Each struct dirtree node contains <b>char name[]</b> and <b>struct stat
1251st</b> entries describing a file, plus a <b>char *symlink</b>
1252which is NULL for non-symlinks.</p>
1253
1254<p>During a callback function, the <b>int dirfd</b> field of directory nodes
1255contains a directory file descriptor (for use with the openat() family of
1256functions). This isn't usually used directly, intstead call dirtree_parentfd()
1257on the callback's node argument. The <b>char again</b> field is 0 for the
1258first callback on a node, and 1 on the second callback (triggered by returning
1259DIRTREE_COMEAGAIN on a directory, made after all children have been processed).
1260</p>
1261
1262<p>Users of this code may put anything they like into the <b>long extra</b>
1263field. For example, "cp" and "mv" use this to store a dirfd for the destination
1264directory (and use DIRTREE_COMEAGAIN to get the second callback so they can
1265close(node->extra) to avoid running out of filehandles).
1266This field is not directly used by the dirtree code, and
1267thanks to LP64 it's large enough to store a typecast pointer to an
1268arbitrary struct.</p>
1269
1270<p>The return value of the callback combines flags (with boolean or) to tell
1271the traversal infrastructure how to behave:</p>
1272
1273<ul>
1274<li><p><b>DIRTREE_SAVE</b> - Save this node, assembling a tree. (Without
1275this the struct dirtree is freed after the callback returns. Filtering out
1276siblings is fine, but discarding a parent while keeping its child leaks
1277memory.)</p></li>
1278<li><p><b>DIRTREE_ABORT</b> - Do not examine any more entries in this
1279directory. (Does not propagate up tree: to abort entire traversal,
1280return DIRTREE_ABORT from parent callbacks too.)</p></li>
1281<li><p><b>DIRTREE_RECURSE</b> - Examine directory contents. Ignored for
1282non-directory entries. The remaining flags only take effect when
1283recursing into the children of a directory.</p></li>
1284<li><p><b>DIRTREE_COMEAGAIN</b> - Call the callback on this node a second time
1285after examining all directory contents, allowing depth-first traversal.
1286On the second call, dirtree->again is nonzero.</p></li>
1287<li><p><b>DIRTREE_SYMFOLLOW</b> - follow symlinks when populating children's
1288<b>struct stat st</b> (by feeding a nonzero value to the symfollow argument of
1289dirtree_add_node()), which means DIRTREE_RECURSE treats symlinks to
1290directories as directories. (Avoiding infinite recursion is the callback's
1291problem: the non-NULL dirtree->symlink can still distinguish between
1292them. The "find" command follows ->parent up the tree to the root node
1293each time, checking to make sure that stat's dev and inode pair don't
1294match any ancestors.)</p></li>
1295</ul>
1296
1297<p>Each struct dirtree contains three pointers (next, parent, and child)
1298to other struct dirtree.</p>
1299
1300<p>The <b>parent</b> pointer indicates the directory
1301containing this entry; even when not assembling a persistent tree of
1302nodes the parent entries remain live up to the root of the tree while
1303child nodes are active. At the top of the tree the parent pointer is
1304NULL, meaning the node's name[] is either an absolute path or relative
1305to cwd. The function dirtree_parentfd() gets the directory file descriptor
1306for use with openat() and friends, returning AT_FDCWD at the top of tree.</p>
1307
1308<p>The <b>child</b> pointer points to the first node of the list of contents of
1309this directory. If the directory contains no files, or the entry isn't
1310a directory, child is NULL.</p>
1311
1312<p>The <b>next</b> pointer indicates sibling nodes in the same directory as this
1313node, and since it's the first entry in the struct the llist.c traversal
1314mechanisms work to iterate over sibling nodes. Each dirtree node is a
1315single malloc() (even char *symlink points to memory at the end of the node),
1316so llist_free() works but its callback must descend into child nodes (freeing
1317a tree, not just a linked list), plus whatever the user stored in extra.</p>
1318
1319<p>The <b>dirtree_flagread</b>() function is a simple wrapper, calling <b>dirtree_add_node</b>()
1320to create a root node relative to the current directory, then calling
1321<b>dirtree_handle_callback</b>() on that node (which recurses as instructed by the callback
1322return flags). The flags argument primarily lets you
1323control whether or not to follow symlinks to the root node; symlinks
1324listed on the command line are often treated differently than symlinks
1325encountered during recursive directory traversal.
1326
1327<p>The ls command not only bypasses this wrapper, but never returns
1328<b>DIRTREE_RECURSE</b> from the callback, instead calling <b>dirtree_recurse</b>() manually
1329from elsewhere in the program. This gives ls -lR manual control
1330of traversal order, which is neither depth first nor breadth first but
1331instead a sort of FIFO order requried by the ls standard.</p>
1332
1333<a name="toys">
1334<h1><a href="#toys">Directory toys/</a></h1>
1335
1336<p>This directory contains command implementations. Each command is a single
1337self-contained file. Adding a new command involves adding a single
1338file, and removing a command involves removing that file. Commands use
1339shared infrastructure from the lib/ and generated/ directories.</p>
1340
1341<p>Currently there are five subdirectories under "toys/" containing "posix"
1342commands described in POSIX-2008, "lsb" commands described in the Linux
1343Standard Base 4.1, "other" commands not described by either standard,
1344"pending" commands awaiting cleanup (which default to "n" in menuconfig
1345because they don't necessarily work right yet), and "example" code showing
1346how toybox infrastructure works and providing template/skeleton files to
1347start new commands.</p>
1348
1349<p>The only difference directory location makes is which menu the command
1350shows up in during "make menuconfig", the directories are otherwise identical.
1351Note that the commands exist within a single namespace at runtime, so you can't
1352have the same command in multiple subdirectories. (The build tries to fail
1353informatively when you do that.)</p>
1354
1355<p>There is one more sub-menus in "make menuconfig" containing global
1356configuration options for toybox. This menu is defined in the top level
1357Config.in.</p>
1358
1359<p>See <a href="#adding">adding a new command</a> for details on the
1360layout of a command file.</p>
1361
1362<a name="scripts">
1363<h2>Directory scripts/</h2>
1364
1365<p>Build infrastructure. The makefile calls scripts/make.sh for "make"
1366and scripts/install.sh for "make install".</p>
1367
1368<p>There's also a test suite, "make test" calls make/test.sh, which runs all
1369the tests in make/test/*. You can run individual tests via
1370"scripts/test.sh command", or "TEST_HOST=1 scripts/test.sh command" to run
1371that test against the host implementation instead of the toybox one.</p>
1372
1373<h3>scripts/cfg2files.sh</h3>
1374
1375<p>Run .config through this filter to get a list of enabled commands, which
1376is turned into a list of files in toys via a sed invocation in the top level
1377Makefile.
1378</p>
1379
1380<h2>Directory kconfig/</h2>
1381
1382<p>Menuconfig infrastructure copied from the Linux kernel a long time ago
1383(version 2.6.16).  See the
1384Linux kernel's Documentation/kbuild/kconfig-language.txt</p>
1385
1386<!-- todo
1387
1388Better OLDTOY and multiple command explanation. From Config.in:
1389
1390<p>A command with multiple names (or multiple similar commands implemented in
1391the same .c file) should have config symbols prefixed with the name of their
1392C file. I.E. config symbol prefixes are NEWTOY() names. If OLDTOY() names
1393have config symbols they must be options (symbols with an underscore and
1394suffix) to the NEWTOY() name. (See generated/toylist.h)</p>
1395-->
1396
1397<!--#include file="footer.html" -->
1398