1 Unicode support in busybox 2 3There are several scenarios where we need to handle unicode 4correctly. 5 6 Shell input 7 8We want to correctly handle input of unicode characters. 9There are several problems with it. Just handling input 10as sequence of bytes would break any editing. This was fixed 11and now lineedit operates on the array of wchar_t's. 12But we also need to handle the following problematic moments: 13 14* It is unreasonable to expect that output device supports 15 _any_ unicode chars. Perhaps we need to avoid printing 16 those chars which are not supported by output device. 17 Examples: chars which are not present in the font, 18 chars which are not assigned in unicode, 19 combining chars (especially trying to combine bad pairs: 20 a_chinese_symbol + "combining grave accent" = ??!) 21 22* We need to account for the fact that unicode chars have 23 different widths: 0 for combining chars, 1 for usual, 24 2 for ideograms (are there 3+ wide chars?). 25 26* Bidirectional handling. If user wants to echo a phrase 27 in Hebrew, he types: echo "srettel werbeH" 28 29 Editors (vi, ed) 30 31This case is a bit similar to "shell input", but unlike shell, 32editors may encounter many more unexpected unicode sequences 33(try to load a random binary file...), and they need to preserve 34them, unlike shell which can afford to drop bogus input. 35 36 more, less 37 38Need to correctly display any input file. Ideally, with 39ASCII/unicode/filtered_unicode option or keyboard switch. 40Note: need to handle tabs and backspaces specially 41(bksp is for manpage compat). 42 43 cut, fold, watch 44 45May need ability to cut unicode string to specified number of wchars 46and/or to specified screen width. Need to handle tabs specially. 47 48 sed, awk, grep 49 50Handle unicode-aware regexp match 51 52 ls (multi-column display) 53 54ls will fail to line up columnar output if it will not account 55for character widths (and maybe filter out some of them, see 56above). OTOH, non-columnar views (ls -1, ls -l, ls | car) 57should NOT filter out bad unicode (but need to filter out 58control chars (coreutils does that). Note that unlike more/less, 59tabs and backspaces need not special handling. 60 61 top, ps 62 63Need to perform filtering similar to ls. 64 65 Filename display (in error messages and elsewhere) 66 67Need to perform filtering similar to ls. 68 69 70TODO: write an email to Asmus Freytag (asmus@unicode.org), 71author of http://unicode.org/reports/tr11/ 72