1<html><head><title>toybox news</title> 2<!--#include file="header.html" --> 3 4<ul> 5<li><h2><a href="#capitalize">Do you capitalize toybox?</a></h2></li> 6<li><h2><a href="#why_toybox">Why toybox? (What was wrong with busybox?)</a></h2></li> 7<li><h2><a href="#support_horizon">Why a 7 year support horizon?</a></h2></li> 8<li><h2><a href="#releases">Why time based releases?</a></h2></li> 9<li><h2><a href="#code">Where do I start understanding the toybox source code?</a></h2></li> 10</ul> 11 12<a name="capitalize" /> 13<h2>Q: Do you capitalize toybox?</h2> 14 15<p>A: Only at the start of a sentence. The command name is all lower case so 16it seems silly to capitalize the project name, but not capitalizing the 17start of sentences is awkward, so... compromise. (It is _not_ "ToyBox".)</p> 18 19<a name="why_toybox" /> 20<h2>Q: "Why is there toybox? What was wrong with busybox?"</h2> 21 22<p>A: Toybox started back in 2006 when I 23<a href=https://lwn.net/Articles/202106/>handed off BusyBox maintainership</a> 24and <a href=http://landley.net/notes-2006.html#28-09-2006>started over from 25scratch</a> on a new codebase after a 26<a href=http://lists.busybox.net/pipermail/busybox/2006-September/058617.html>protracted licensing argument</a> took all the fun out of working on BusyBox.</p> 27 28<p>Toybox was just a personal project until it got 29<a href=http://landley.net/notes-2011.html#13-11-2011>relaunched 30in November 2011</a> with a new goal to 31<a href=http://landley.net/aboriginal/about.html#selfhost>make Android 32self-hosting</a>. This involved me relicensing my own 33code, which <a href=https://lwn.net/Articles/478308/>made people who had 34never used or participated in the project loudly angry</a>. The switch came 35after a lot of thinking <a href=http://landley.net/talks/ohio-2013.txt>about 36licenses</a> and <a href=http://landley.net/notes-2011.html#21-03-2011>the 37transition to smartphones</a>, which led to a 38<a href=http://landley.net/talks/celf-2013.txt>2013</a> 39<a href=https://www.youtube.com/watch?v=SGmtP5Lg_t0>talk</a> laying 40out a strategy to make Android self-hosting using toybox. This helped 41<a href=https://code.google.com/p/android/issues/detail?id=76861>bring 42it to Android's attention</a>, and they 43<a href=https://lwn.net/Articles/629362/>merged it</a> into Android M.</p> 44 45<p>The answer to the second question is "licensing". BusyBox predates Android 46by almost a decade but Android still doesn't ship with it because GPLv3 came 47out around the same time Android did and caused many people to throw 48out the GPLv2 baby with the GPLv3 bathwater. 49Android <a href=https://source.android.com/source/licenses.html>explicitly 50discourages</a> use of GPL and LGPL licenses in its products, and has gradually 51reimplemented historical GPL components such as its bluetooth stack under the 52Apache license. Similarly, Apple froze xcode at the last GPLv2 releases 53(GCC 4.2.1 with binutils 2.17) for over 5 years while it sponsored the 54development of new projects (clang/llvm/lld) to replace them, 55implemented its SMB server from scratch to replace samba, 56<a href=http://meta.ath0.com/2012/02/05/apples-great-gpl-purge/>and so 57on</a>. Toybox itself exists because somebody with in a legacy position 58just wouldn't shut up about GPLv3, otherwise I would probably 59still happily be maintaining BusyBox. (For more on how I wound 60up working on busybox in the first place, 61<a href=http://landley.net/aboriginal/history.html>see here</a>.)</p> 62 63<h2><a name="support_horizon">Q: Why a 7 year support horizon?</a></h2> 64 65<p>A: Our <a href=http://lists.busybox.net/pipermail/busybox/2006-September/058440.html>longstanding rule of thumb</a> is to try to run and build on 66hardware and distributions released up to 7 years ago, and feel ok dropping 67support for stuff older than that. (This is a little longer than Ubuntu's 68Long Term Support, but not by much.)</p> 69 70<p>If a kernel or libc feature is less than 7 years old, I try to have a 71build-time configure test for it and let the functionality cleanly drop out. 72I also keep old Ubuntu images around in VMs and perform the occasional 73defconfig build there to see what breaks. (I'm not perfect about this, 74but I accept bug reports.)</p> 75 76<p>My original theory was "4 to 5 18-month cycles of moore's law should cover 77the vast majority of the installed base of PC hardware", loosely based on some 78research I did <a href=http://www.catb.org/esr/halloween/halloween9.html#id2867629>back in 2003</a> 79and <a href=http://catb.org/esr/writings/world-domination/world-domination-201.html#id248066>updated in 2006</a> 80which said that low end systems were 2 iterations of moore's 81law below the high end systems, and that another 2-3 iterations should cover 82the useful lifetime of most systems no longer being sold but still in use and 83potentially being upgraded to new software releases.</p> 84 85<p>It turns out <a href=http://landley.net/notes-2011.html#26-06-2011>I missed 86industry changes</a> in the 1990's that stretched the gap 87from low end to high end from 2 cycles to 4 cycles, and _that_ analysis 88ignored the switch from PC to smartphone cutting off the R&D air supply of the 89laptop market. Meanwhile the Moore's Law s-curve started bending 90down in 2000 and these days is pretty flat because the drive for faster clock 91speeds <a href=http://www.anandtech.com/show/613>stumbled</a> 92then <a href=http://www.pcworld.com/article/118603/article.html>died</a>, and 93the subsequent drive to go wide maxed out around 4x SMP with ~2 megabyte 94caches for most applications. These days the switch from exponential to 95linear growth in hardware capabilities is 96<a href=https://www.cnet.com/news/end-of-moores-law-its-not-just-about-physics/>common</a> 97<a href=http://www.acm.org/articles/people-of-acm/2016/david-patterson>knowledge</a>.</p> 98 99<p>But the 7 year rule of thumb stuck around anyway: if a kernel or libc 100feature is less than 7 years old, I try to have a build-time configure test 101for it and let the functionality cleanly drop out. I also keep old Ubuntu 102images around in VMs and perform the occasional defconfig build there to 103see what breaks.</p> 104 105<h2><a name="releases" />Q: Why time based releases?</h2> 106<p>A: Toybox targets quarterly releases (a similar schedule to the Linux 107kernel) because Martin Michlmayr's 108<a href=http://www.youtube.com/watch?v=IKsQsxubuAA>talk</a> on the 109subject was convincing.</p> 110 111<p>Releases provide synchronization points where the developers certify 112"it worked for me". Each release is a known version with predictable behavior, 113and right or wrong at least everyone should be seeing 114similar results where you might be able to google an unexpected outcome. 115Releases focus end-user testing on specific versions 116where issues can be reproduced, diagnosed, and fixed. 117Releases also force the developers to do periodic tidying, packaging, 118documentation review, finish up partially implemented features languishing 119in their private trees, and give regular checkpoints to measure progress.</p> 120 121<p>Over time feature sets change, data formats change, control knobs change... 122For example toybox's switch from "ls -q" to "ls -b" as the default output 123format wasn't exactly a bug, it was a design improvement... but the 124difference is academic if the change breaks somebody's script. 125Releases give you the option to schedule upgrades later, and not to rock 126the boat just now: just use a known working release version.</p> 127 128<p>The counter-argument is that "continuous integration" 129can be made robust with sufficient automated testing. But like the 130<a href=http://www.shirky.com/weblog/2013/11/healthcare-gov-and-the-gulf-between-planning-and-reality/>waterfall method</a>, this places insufficent 131emphasis on end-user feedback and learning from real world experience. 132Developer testing is either testing that the code does what the developers 133expect given expected inputs running in an expected environment, or it's 134regression testing against bugs previously found in the field. No plan 135survives contact with the enemy, and technology always breaks once it 136leaves the lab and encounters real world data and use cases, not just 137at runtime but in different build environments.</p> 138 139<p>The best way to give new users a reasonable first experience is to point 140them at specific stable versions where development quiesced and 141extra testing occurred. There will still be teething troubles, but multiple 142people experiencing the _same_ teething troubles can potentially 143help each other out.</p> 144 145<p>As for why releases on a schedule are better than releases "when it's 146ready", watch the video.</p> 147 148<h2><a name="code" />Q: Where do I start understanding the source code?</h2> 149 150<p>A: Toybox is written in C. There are longer writeups of the 151<a href=design.html>design ideas</a> and a <a href=code.html>code walkthrough</a>, 152and the <a href=about.html>about page</a> summarizes what we're trying to 153accomplish, but here's a quick start:</p> 154 155<p>Toybox uses the standard three stage configure/make/install 156<a href=code.html#building>build</a>, in this case "<b>make defconfig; 157make; make install</b>". Type "<b>make help</b>" to 158see available make targets.</p> 159 160<p><b>The configure stage is copied from the Linux kernel</b> (in the "kconfig" 161directory), and saves your selections in the file ".config" at the top 162level. The "defconfig" target selects the 163maximum sane configuration (enabling all the commands and features that 164aren't unfinished, only intended as examples, debug code, etc) and is 165probably what you want. You can use "make menuconfig" to manually select 166specific commands to include, through an interactive menu (cursor up and 167down, enter to descend into a sub-menu, space to select an entry, ? to see 168an entry's help text, esc to exit). The menuconfig help text is the 169same as the command's --help output.</p> 170 171<p><b>The "make" stage creates a toybox binary</b> (which is stripped, look in 172generated/unstripped for the debug versions), and "install" adds a bunch of 173symlinks to toybox under the various command names. Toybox determines which 174command to run based on the filename, or you can use the "toybox" name in which case the first 175argument is the command to run (ala "toybox ls -l"). <b>You can also build 176individual commands as standalone executables</b>, ala "make sed cat ls".</p> 177 178<p><b>The main() function is in main.c at the top level</b>, 179along with setup plumbing and selecting which command to run this time. 180The function toybox_main() implements the "toybox" multiplexer command.</p> 181 182<p><b>The individual command implementations are under "toys"</b>, and are grouped 183into categories (mostly based on which standard they come from, posix, lsb, 184android...) The "pending" directory contains unfinished commands, and the 185"examples" directory contains examples. Commands in those two directories 186are _not_ selected by defconfig. (These days pending directory is mostly 187third party submissions that have not yet undergone proper code review.)</p> 188 189<p><b>Common infrastructure shared between commands is under "lib"</b>. Most 190commands call lib/args.c to parse their command line arguments before calling 191the command's own main() function, which uses the option string in 192the command's NEWTOY() macro. This is similar to the libc function getopt(), 193but more powerful, and is documented at the top of lib/args.c.</p> 194 195<p>Most of the actual <b>build/install infrastructure is shell scripts under 196"scripts"</b>. <b>These populate the "generated" directory</b> with headers 197created from other files, which are <a href=code.html#generated>described</a> 198in the code walkthrough. All the 199build's temporary files live under generated, including the .o files built 200from the .c files (in generated/obj). The "make clean" target deletes that 201directory. ("make distclean" also deletes your .config and deletes the 202kconfig binaries that process .config.)</p> 203 204<p>Each command's file contains all the information for that command, so 205<b>adding a command to toybox means adding a single file under "toys"</b>. 206Usually you <a href=code.html#adding>start a new command</a> by copying an 207existing command file to a new filename 208(toys/examples/hello.c, toys/examples/skeleton.c, toys/posix/cat.c, 209and toys/posix/true.c have all been used for this purpose) and then replacing 210all instances of its old name with the new name (which should match the 211new filename), and modifying the help text, argument string, and what the 212code does. You might have to "make distclean" before you new command 213shows up in defconfig or menuconfig.</p> 214 215<p><b>The toybox test suite lives in the "tests" directory</b>. From the top 216level you can "make tests" to test everything, or "make test_sed" test a 217single command's standalone version (which should behave identically) 218but that's why we test.</p> 219 220<!--#include file="footer.html" --> 221