1<html><head><title>The design of toybox</title></head>
   2<!--#include file="header.html" -->
   4<a name="goals"><b><h2><a href="#goals">Design goals</a></h2></b>
   6<p>Toybox should be simple, small, fast, and full featured. In that order.</p>
   8<p>When these goals need to be balanced off against each other, keeping the code
   9as simple as it can be to do what it does is the most important (and hardest)
  10goal. Then keeping it small is slightly more important than making it fast.
  11Features are the reason we write code in the first place but this has all
  12been implemented before so if we can't do a better job why bother?</p>
  14<p>It should be possible to get 80% of the way to each goal
  15before they really start to fight. Here they are in reverse order
  16of importance:</p>
  20<p>The hard part is deciding what NOT to include.
  21A project without boundaries will bloat itself
  22to death. One of the hardest but most important things a project must
  23do is draw a line and say "no, this is somebody else's problem, not
  24something we should do."</p>
  26<p>Some things are simply outside the scope of the project: even though
  27posix defines commands for compiling and linking, we're not going to include
  28a compiler or linker (and support for a potentially infinite number of hardware
  29targets). And until somebody comes up with a ~30k ssh implementation (with
  30a crypto algorithm that won't need replacing every 5 years), we're
  31going to point you at dropbear or bearssl.</p>
  33<p>The <a href=roadmap.html>roadmap</a> has the list of features we're
  34trying to implement, and the reasons why we decided to include those
  35features. After the 1.0 release some of that material may get moved here,
  36but for now it needs its own page.</p>
  38<p>There are potential features (such as a screen/tmux implementation)
  39that might be worth adding after 1.0, in part because they could share
  40infrastructure with things like "less" and "vi" so might be less work for
  41us to do than an external from-scratch implementation. But for now, major
  42new features outside posix, android's existing commands, and the needs of
  43development systems, are a distraction from the 1.0 release.</p>
  47<p>It's easy to say lots about optimizing for speed (which is why this section
  48is so long), but at the same time it's the optimization we care the least about.
  49The essence of speed is being as efficient as possible, which means doing as
  50little work as possible.  A design that's small and simple gets you 90% of the
  51way there, and most of the rest is either fine-tuning or more trouble than
  52it's worth (and often actually counterproductive).  Still, here's some
  55<p>First, understand the darn problem you're trying to solve.  You'd think
  56I wouldn't have to say this, but I do.  Trying to find a faster sorting
  57algorithm is no substitute for figuring out a way to skip the sorting step
  58entirely.  The fastest way to do anything is not to have to do it at all,
  59and _all_ optimization boils down to avoiding unnecessary work.</p>
  61<p>Speed is easy to measure; there are dozens of profiling tools for Linux
  62(although personally I find the "time" command a good starting place).
  63Don't waste too much time trying to optimize something you can't measure,
  64and there's no much point speeding up things you don't spend much time doing
  67<p>Understand the difference between throughput and latency.  Faster
  68processors improve throughput, but don't always do much for latency.
  69After 30 years of Moore's Law, most of the remaining problems are latency,
  70not throughput.  (There are of course a few exceptions, like data compression
  71code, encryption, rsync...)  Worry about throughput inside long-running
  72loops, and worry about latency everywhere else.  (And don't worry too much
  73about avoiding system calls or function calls or anything else in the name
  74of speed unless you are in the middle of a tight loop that's you've already
  75proven isn't running fast enough.)</p>
  77<p>"Locality of reference" is generally nice, in all sorts of contexts.
  78It's obvious that waiting for disk access is 1000x slower than doing stuff in
  79RAM (and making the disk seek is 10x slower than sequential reads/writes),
  80but it's just as true that a loop which stays in L1 cache is many times faster
  81than a loop that has to wait for a DRAM fetch on each iteration.  Don't worry
  82about whether "&" is faster than "%" until your executable loop stays in L1
  83cache and the data access is fetching cache lines intelligently.  (To
  84understand DRAM, L1, and L2 cache, read Hannibal's marvelous ram guide at Ars
  86<a href=>part one</a>,
  87<a href=>part two</a>,
  88<a href=>part three</a>,
  89plus this
  90<a href=>article on
  91cacheing</a>, and this one on
  92<a href=>bandwidth
  93and latency</a>.
  94And there's <a href=>more where that came from</a>.)
  95Running out of L1 cache can execute one instruction per clock cycle, going
  96to L2 cache costs a dozen or so clock cycles, and waiting for a worst case dram
  97fetch (round trip latency with a bank switch) can cost thousands of
  98clock cycles.  (Historically, this disparity has gotten worse with time,
  99just like the speed hit for swapping to disk.  These days, a _big_ L1 cache
 100is 128k and a big L2 cache is a couple of megabytes.  A cheap low-power
 101embedded processor may have 8k of L1 cache and no L2.)</p>
 103<p>Learn how <a href=>virtual memory and
 104memory managment units work</a>.  Don't touch
 105memory you don't have to.  Even just reading memory evicts stuff from L1 and L2
 106cache, which may have to be read back in later.  Writing memory can force the
 107operating system to break copy-on-write, which allocates more memory.  (The
 108memory returned by malloc() is only a virtual allocation, filled with lots of
 109copy-on-write mappings of the zero page.  Actual physical pages get allocated
 110when the copy-on-write gets broken by writing to the virtual page.  This
 111is why checking the return value of malloc() isn't very useful anymore, it
 112only detects running out of virtual memory, not physical memory.  Unless
 113you're using a <a href=>NOMMU system</a>, where all bets are off.)</p>
 115<p>Don't think that just because you don't have a swap file the system can't
 116start swap thrashing: any file backed page (ala mmap) can be evicted, and
 117there's a reason all running programs require an executable file (they're
 118mmaped, and can be flushed back to disk when memory is short).  And long
 119before that, disk cache gets reclaimed and has to be read back in.  When the
 120operating system really can't free up any more pages it triggers the out of
 121memory killer to free up pages by killing processes (the alternative is the
 122entire OS freezing solid).  Modern operating systems seldom run out of
 123memory gracefully.</p>
 125<p>Also, it's better to be simple than clever.  Many people think that mmap()
 126is faster than read() because it avoids a copy, but twiddling with the memory
 127management is itself slow, and can cause unnecessary CPU cache flushes.  And
 128if a read faults in dozens of pages sequentially, but your mmap iterates
 129backwards through a file (causing lots of seeks, each of which your program
 130blocks waiting for), the read can be many times faster.  On the other hand, the
 131mmap can sometimes use less memory, since the memory provided by mmap
 132comes from the page cache (allocated anyway), and it can be faster if you're
 133doing a lot of different updates to the same area.  The moral?  Measure, then
 134try to speed things up, and measure again to confirm it actually _did_ speed
 135things up rather than made them worse.  (And understanding what's really going
 136on underneath is a big help to making it happen faster.)</p>
 138<p>In general, being simple is better than being clever.  Optimization
 139strategies change with time.  For example, decades ago precalculating a table
 140of results (for things like isdigit() or cosine(int degrees)) was clearly
 141faster because processors were so slow.  Then processors got faster and grew
 142math coprocessors, and calculating the value each time became faster than
 143the table lookup (because the calculation fit in L1 cache but the lookup
 144had to go out to DRAM).  Then cache sizes got bigger (the Pentium M has
 1452 megabytes of L2 cache) and the table fit in cache, so the table became
 146fast again...  Predicting how changes in hardware will affect your algorithm
 147is difficult, and using ten year old optimization advice and produce
 148laughably bad results.  But being simple and efficient is always going to
 149give at least a reasonable result.</p>
 151<p>The famous quote from Ken Thompson, "When in doubt, use brute force",
 152applies to toybox.  Do the simple thing first, do as little of it as possible,
 153and make sure it's right.  You can always speed it up later.</p>
 156<p>Again, being simple gives you most of this. An algorithm that does less work
 157is generally smaller.  Understand the problem, treat size as a cost, and
 158get a good bang for the byte.</p>
 160<p>Understand the difference between binary size, heap size, and stack size.
 161Your binary is the executable file on disk, your heap is where malloc() memory
 162lives, and your stack is where local variables (and function call return
 163addresses) live.  Optimizing for binary size is generally good: executing
 164fewer instructions makes your program run faster (and fits more of it in
 165cache).  On embedded systems, binary size is especially precious because
 166flash is expensive (and its successor, MRAM, even more so).  Small stack size
 167is important for nommu systems because they have to preallocate their stack
 168and can't make it bigger via page fault.  And everybody likes a small heap.</p>
 170<p>Measure the right things.  Especially with modern optimizers, expecting
 171something to be smaller is no guarantee it will be after the compiler's done
 172with it.  Binary size isn't the most accurate indicator of the impact of a
 173given change, because lots of things get combined and rounded during
 174compilation and linking.  Matt Mackall's bloat-o-meter is a python script
 175which compares two versions of a program, and shows size changes in each
 176symbol (using the "nm" command behind the scenes).  To use this, run
 177"make baseline" to build a baseline version to compare against, and
 178then "make bloatometer" to compare that baseline version against the current
 181<p>Avoid special cases.  Whenever you see similar chunks of code in more than
 182one place, it might be possible to combine them and have the users call shared
 183code. (This is the most commonly cited trick, which doesn't make it easy. If
 184seeing two lines of code do the same thing makes you slightly uncomfortable,
 185you've got the right mindset.)</p>
 187<p>Some specific advice: Using a char in place of an int when doing math
 188produces significantly larger code on some platforms (notably arm),
 189because each time the compiler has to emit code to convert it to int, do the
 190math, and convert it back.  Bitfields have this problem on most platforms.
 191Because of this, using char to index a for() loop is probably not a net win,
 192although using char (or a bitfield) to store a value in a structure that's
 193repeated hundreds of times can be a good tradeoff of binary size for heap
 198<p>Complexity is a cost, just like code size or runtime speed. Treat it as
 199a cost, and spend your complexity budget wisely. (Sometimes this means you
 200can't afford a feature because it complicates the code too much to be
 201worth it.)</p>
 203<p>Simplicity has lots of benefits.  Simple code is easy to maintain, easy to
 204port to new processors, easy to audit for security holes, and easy to
 207<p>Simplicity itself can have subtle non-obvious aspects requiring a tradeoff
 208between one kind of simplicity and another: simple for the computer to
 209execute and simple for a human reader to understand aren't always the
 210same thing. A compact and clever algorithm that does very little work may
 211not be as easy to explain or understand as a larger more explicit version
 212requiring more code, memory, and CPU time. When balancing these, err on the
 213side of doing less work, but add comments describing how you
 214could be more explicit.</p>
 216<p>In general, comments are not a substitute for good code (or well chosen
 217variable or function names). Commenting "x += y;" with "/* add y to x */"
 218can actually detract from the program's readability. If you need to describe
 219what the code is doing (rather than _why_ it's doing it), that means the
 220code itself isn't very clear.</p>
 222<p>Environmental dependencies are another type of complexity, so needing other
 223packages to build or run is a big downside. For example, we don't use curses
 224when we can simply output ansi escape sequences and trust all terminal
 225programs written in the past 30 years to be able to support them. Regularly
 226testing that we work with C libraries which support static linking (musl does,
 227glibc doesn't) is another way to be self-contained with known boundaries:
 228it doesn't have to be the only way to build the project, but should be regularly
 229tested and supported.</p>
 231<p>Prioritizing simplicity tends to serve our other goals: simplifying code
 232generally reduces its size (both in terms of binary size and runtime memory
 233usage), and avoiding unnecessary work makes code run faster. Smaller code
 234also tends to run faster on modern hardware due to CPU cacheing: fitting your
 235code into L1 cache is great, and staying in L2 cache is still pretty good.</p>
 237<p>But a simple implementation is not always the smallest or fastest, and
 238balancing simplicity vs the other goals can be difficult. For example, the
 239atolx_range() function in lib/lib.c always uses the 64 bit "long long" type,
 240which produces larger and slower code on 32 bit platforms and
 241often assigned into smaller interger types. Although libc has parallel
 242implementations for different data sizes (atoi, atol, atoll) we chose a
 243common codepath which can cover all cases (every user goes through the
 244same codepath, with the maximum amount of testing and minimum and avoids
 245surprising variations in behavior).</p>
 247<p>On the other hand, the "tail" command has two codepaths, one for seekable
 248files and one for nonseekable files. Although the nonseekable case can handle
 249all inputs (and is required when input comes from a pipe or similar, so cannot
 250be removed), reading through multiple gigabytes of data to reach the end of
 251seekable files was both a common case and hugely penalized by a nonseekable
 252approach (half-minute wait vs instant results). This is one example
 253where performance did outweigh simplicity of implementation.</p>
 255<p><a href=>Joel
 256Spolsky argues against throwing code out and starting over</a>, and he has
 257good points: an existing debugged codebase contains a huge amount of baked
 258in knowledge about strange real-world use cases that the designers didn't
 259know about until users hit the bugs, and most of this knowledge is never
 260explicitly stated anywhere except in the source code.</p>
 262<p>That said, the Mythical Man-Month's "build one to throw away" advice points
 263out that until you've solved the problem you don't properly understand it, and
 264about the time you finish your first version is when you've finally figured
 265out what you _should_ have done.  (The corrolary is that if you build one
 266expecting to throw it away, you'll actually wind up throwing away two.  You
 267don't understand the problem until you _have_ solved it.)</p>
 269<p>Joel is talking about what closed source software can afford to do: Code
 270that works and has been paid for is a corporate asset not lightly abandoned.
 271Open source software can afford to re-implement code that works, over and
 272over from scratch, for incremental gains.  Before toybox, the unix command line
 273has already been reimplemented from scratch several times (the
 274original AT&amp;T Unix command line in assembly and then in C, the BSD
 275versions, Coherent was the first full from-scratch Unix clone in 1980,
 276Minix was another clone which Linux was inspired by and developed under,
 277the GNU tools were yet another rewrite intended for use in the stillborn
 278"Hurd" project, BusyBox was still another rewrite, and more versions
 279were written in Plan 9, uclinux, klibc, sash, sbase, s6, and of course
 280android toolbox...). But maybe toybox can do a better job. :)</p>
 282<p>As Antoine de St. Exupery (author of "The Little Prince" and an early
 283aircraft designer) said, "Perfection is achieved, not when there
 284is nothing left to add, but when there is nothing left to take away."
 285And Ken Thompson (creator of Unix) said "One of my most productive
 286days was throwing away 1000 lines of code." It's always possible to
 287come up with a better way to do it.</p>
 289<p>P.S. How could I resist linking to an article about
 290<a href=>why
 291programmers should strive to be lazy and dumb</a>?</p>
 293<a name="portability"><b><h2><a href="#portability">Portability issues</a></h2></b>
 296<p>Toybox should run on Android (all commands with musl-libc, as large a subset
 297as practical with bionic), and every other hardware platform Linux runs on.
 298Other posix/susv4 environments (perhaps MacOS X or newlib+libgloss) are vaguely
 299interesting but only if they're easy to support; I'm not going to spend much
 300effort on them.</p>
 302<p>I don't do windows.</p>
 304<p>We depend on C99 and posix-2008 libc features such as the openat() family of
 305functions. We also assume certain "modern" linux kernel behavior such
 306as large environment sizes (<a href=>linux commit b6a2fea39318</a>, went into 2.6.22
 307released <a href=faq.html#support_horizon>July 2007</a>). In theory this shouldn't prevent us from working on
 308older kernels or other implementations (ala BSD), but we don't police their
 309corner cases.</p>
 311<b><h3>32/64 bit</h3></b>
 312<p>Toybox should work on both 32 bit and 64 bit systems. 64 bit desktop
 313hardware went mainstream in 2005 and was essentially ubiquitous
 314by the end of the decade, but 32 bit hardware will continue to be important
 315in embedded devices for several more years.</p>
 317<p>Toybox relies on the fact that on any Unix-like platform, pointer and long
 318are always the same size (on both 32 and 64 bit). Pointer and int are _not_
 319the same size on 64 bit systems, but pointer and long are.
 320This is guaranteed by the LP64 memory model, a Unix standard (which Linux
 321and MacOS X both implement, and which modern 64 bit processors such as
 322x86-64 were <a href=>designed for</a>).  See
 323<a href=>the LP64 standard</a> and
 324<a href=>the LP64
 325rationale</a> for details.</p>
 327<p>Note that Windows doesn't work like this, and I don't care.
 328<a href=>The
 329insane legacy reasons why this is broken on Windows are explained here.</a></p>
 331<b><h3>Signedness of char</h3></b>
 332<p>On platforms like x86, variables of type char default to unsigned.  On
 333platforms like arm, char defaults to signed.  This difference can lead to
 334subtle portability bugs, and to avoid them we specify which one we want by
 335feeding the compiler -funsigned-char.</p>
 337<p>The reason to pick "unsigned" is that way char strings are 8-bit clean by
 338default, which makes UTF-8 support easier.</p>
 340<p><h3>Error messages and internationalization:</h3></p>
 342<p>Error messages are extremely terse not just to save bytes, but because we
 343don't use any sort of _("string") translation infrastructure. (We're not
 344translating the command names themselves, so we must expect a minimum amount of
 345english knowledge from our users, but let's keep it to a minimum.)</p>
 347<p>Thus "bad -A '%c'" is
 348preferable to "Unrecognized address base '%c'", because a non-english speaker
 349can see that -A was the problem (giving back the command line argument they
 350supplied). A user with a ~20 word english vocabulary is
 351more likely to know (or guess) "bad" than the longer message, and you can
 352use "bad" in place of "invalid", "inappropriate", "unrecognized"...
 353Similarly when atolx_range() complains about range constraints with
 354"4 < 17" or "12 > 5", it's intentional: those don't need to be translated.</p>
 356<p>The strerror() messages produced by perror_exit() and friends should be
 357localized by libc, and our error functions also prepend the command name
 358(which non-english speakers can presumably recognize already). Keep the
 359explanation in between to a minimum, and where possible feed back the values
 360they passed in to identify _what_ we couldn't process.
 361If you say perror_exit("setsockopt"), you've identified the action you
 362were trying to take, and the perror gives a translated error message (from libc)
 363explaining _why_ it couldn't do it, so you probably don't need to add english
 364words like "failed" or "couldn't assign".</p>
 366<p>All commands should be 8-bit clean, with explicit
 367<a href=>UTF-8</a> support where
 368necessary. Assume all input data might be utf8, and at least preserve
 369it and pass it through. (For this reason, our build is -funsigned-char on
 370all architectures; "char" is unsigned unless you stick "signed" in front
 371of it.)</p>
 373<p>Locale support isn't currently a goal; that's a presentation layer issue
 374(I.E. a GUI problem).</p>
 376<p>Someday we should probably have translated --help text, but that's a
 377post-1.0 issue.</p>
 379<p><h3>Shared Libraries</h3></p>
 381<p>Toybox's policy on shared libraries is that they should never be
 382required, but can optionally be used to improve performance.</p>
 384<p>Toybox should provide the command line utilities for
 385<a href=roadmap.html#dev_env>self-hosting development envirionments</a>,
 386and an easy way to set up "hermetic builds" (I.E. builds which provide
 387their own dependencies, isolating the build logic from host command version
 388skew with a simple known build environment). In both cases, external
 389dependencies defeat the purpose.</p>
 391<p>This means toybox should provide full functionality without relying
 392on any external dependencies (other than libc). But toybox may optionally use
 393libraries such as zlib and openssl to improve performance for things like
 394deflate and sha1sum, which lets the corresponding built-in implementations
 395be simple (and thus slow). But the built-in implementations need to exist and
 398<p>(This is why we use an external https wrapper program, because depending on
 399openssl or similar to be linked in would change the behavior of toybox.)</p>
 401<a name="codestyle" />
 402<h2>Coding style</h2>
 404<p>The real coding style holy wars are over things that don't matter
 405(whitespace, indentation, curly bracket placement...) and thus have no
 406obviously correct answer. As in academia, "the fighting is so vicious because
 407the stakes are so small". That said, being consistent makes the code readable,
 408so here's how to make toybox code look like other toybox code.</p>
 410<p>Toybox source uses two spaces per indentation level, and wraps at 80
 411columns. (Indentation of continuation lines is awkward no matter what
 412you do, sometimes two spaces looks better, sometimes indenting to the
 413contents of a parentheses looks better.)</p>
 415<p>I'm aware this indentation style creeps some people out, so here's
 416the sed invocation to convert groups of two leading spaces to tabs:</p>
 418sed -i ':loop;s/^\( *\)  /\1\t/;t loop' filename
 421<p>And here's the sed invocation to convert leading tabs to two spaces each:</p>
 423sed -i ':loop;s/^\( *\)\t/\1  /;t loop' filename
 426<p>There's a space after C flow control statements that look like functions, so
 427"if (blah)" instead of "if(blah)". (Note that sizeof is actually an
 428operator, so we don't give it a space for the same reason ++ doesn't get
 429one. Yeah, it doesn't need the parentheses either, but it gets them.
 430These rules are mostly to make the code look consistent, and thus easier
 431to read.) We also put a space around assignment operators (on both sides),
 432so "int x = 0;".</p>
 434<p>Blank lines (vertical whitespace) go between thoughts. "We were doing that,
 435now we're doing this." (Not a hard and fast rule about _where_ it goes,
 436but there should be some for the same reason writing has paragraph breaks.)</p>
 438<p>Variable declarations go at the start of blocks, with a blank line between
 439them and other code. Yes, c99 allows you to put them anywhere, but they're
 440harder to find if you do that. If there's a large enough distance between
 441the declaration and the code using it to make you uncomfortable, maybe the
 442function's too big, or is there an if statement or something you can
 443use as an excuse to start a new closer block?</p>
 445<p>An * binds to a variable name not a type name, so space it that way.
 446(In C "char *a, b;" and "char* a, b;" mean the same thing: "a" is a pointer
 447but "b" is not. Spacing it the second way is not how C works.)</p>
 449<p>If statments with a single line body go on the same line if the result
 450fits in 80 columns, on a second line if it doesn't. We usually only use
 451curly brackets if we need to, either because the body is multiple lines or
 452because we need to distinguish which if an else binds to. Curly brackets go
 453on the same line as the test/loop statement. The exception to both cases is
 454if the test part of an if statement is long enough to split into multiple
 455lines, then we put the curly bracket on its own line afterwards (so it doesn't
 456get lost in the multple line variably indented mess), and we put it there
 457even if it's only grouping one line (because the indentation level is not
 458providing clear information in that case).</p>
 464if (thingy) thingy;
 465else thingy;
 467if (thingy) {
 468  thingy;
 469  thingy;
 470} else thingy;
 472if (blah blah blah...
 473    && blah blah blah)
 475  thingy;
 479<p>Gotos are allowed for error handling, and for breaking out of
 480nested loops. In general, a goto should only jump forward (not back), and
 481should either jump to the end of an outer loop, or to error handling code
 482at the end of the function. Goto labels are never indented: they override the
 483block structure of the file. Putting them at the left edge makes them easy
 484to spot as overrides to the normal flow of control, which they are.</p>
 486<p>When there's a shorter way to say something, we tend to do that for
 487consistency. For example, we tend to say "*blah" instead of "blah[0]" unless
 488we're referring to more than one element of blah. Similarly, NULL is
 489really just 0 (and C will automatically typecast 0 to anything, except in
 490varargs), "if (function() != NULL)" is the same as "if (function())",
 491"x = (blah == NULL);" is "x = !blah;", and so on.</p>
 493<p>The goal is to be
 494concise, not cryptic: if you're worried about the code being hard to
 495understand, splitting it to multiple steps on multiple lines is
 496better than a NOP operation like "!= NULL". A common sign of trying too
 497hard is nesting ? : three levels deep, sometimes if/else and a temporary
 498variable is just plain easier to read. If you think you need a comment,
 499you may be right.</p>
 501<p>Comments are nice, but don't overdo it. Comments should explain _why_,
 502not how. If the code doesn't make the how part obvious, that's a problem with
 503the code. Sometimes choosing a better variable name is more revealing than a
 504comment. Comments on their own line are better than comments on the end of
 505lines, and they usually have a blank line before them. Most of toybox's
 506comments are c99 style // single line comments, even when there's more than
 507one of them. The /* multiline */ style is used at the start for the metadata,
 508but not so much in the code itself. They don't nest cleanly, are easy to leave
 509accidentally unterminated, need extra nonfunctional * to look right, and if
 510you need _that_ much explanation maybe what you really need is a URL citation
 511linking to a standards document? Long comments can fall out of sync with what
 512the code is doing. Comments do not get regression tested. There's no such
 513thing as self-documenting code (if nothing else, code with _no_ comments
 514is a bit unfriendly to new readers), but "chocolate sauce isn't the answer
 515to bad cooking" either. Don't use comments as a crutch to explain unclear
 516code if the code can be fixed.</p>
 518<!--#include file="footer.html" -->