blob: 9fc799646a3ad44a3fcf011d2662a377746c4463 [file] [log] [blame]
Denis Vlasenko4efeaee2007-03-15 19:52:42 +00001 Keeping data small
Denis Vlasenko75605782007-03-14 00:07:51 +00002
3When many applets are compiled into busybox, all rw data and
4bss for each applet are concatenated. Including those from libc,
Denis Vlasenkoe84aeb52007-03-20 11:08:39 +00005if static busybox is built. When busybox is started, _all_ this data
Denis Vlasenko75605782007-03-14 00:07:51 +00006is allocated, not just that one part for selected applet.
7
8What "allocated" exactly means, depends on arch.
Denis Vlasenkoe84aeb52007-03-20 11:08:39 +00009On NOMMU it's probably bites the most, actually using real
Denis Vlasenko75605782007-03-14 00:07:51 +000010RAM for rwdata and bss. On i386, bss is lazily allocated
11by COWed zero pages. Not sure about rwdata - also COW?
12
Denis Vlasenkoe84aeb52007-03-20 11:08:39 +000013In order to keep busybox NOMMU and small-mem systems friendly
Denis Vlasenko4efeaee2007-03-15 19:52:42 +000014we should avoid large global data in our applets, and should
15minimize usage of libc functions which implicitly use
Denis Vlasenkoe84aeb52007-03-20 11:08:39 +000016such structures.
Denis Vlasenko4efeaee2007-03-15 19:52:42 +000017
Denis Vlasenkoe84aeb52007-03-20 11:08:39 +000018Small experiment to measure "parasitic" bbox memory consumption:
19here we start 1000 "busybox sleep 10" in parallel.
20busybox binary is practically allyesconfig static one,
21built against uclibc. Run on x86-64 machine with 64-bit kernel:
Denis Vlasenko75605782007-03-14 00:07:51 +000022
Denis Vlasenkoe84aeb52007-03-20 11:08:39 +000023bash-3.2# nmeter '%t %c %m %p %[pn]'
2423:17:28 .......... 168M 0 147
2523:17:29 .......... 168M 0 147
2623:17:30 U......... 168M 1 147
2723:17:31 SU........ 181M 244 391
2823:17:32 SSSSUUU... 223M 757 1147
2923:17:33 UUU....... 223M 0 1147
3023:17:34 U......... 223M 1 1147
3123:17:35 .......... 223M 0 1147
3223:17:36 .......... 223M 0 1147
3323:17:37 S......... 223M 0 1147
3423:17:38 .......... 223M 1 1147
3523:17:39 .......... 223M 0 1147
3623:17:40 .......... 223M 0 1147
3723:17:41 .......... 210M 0 906
3823:17:42 .......... 168M 1 147
3923:17:43 .......... 168M 0 147
Denis Vlasenko75605782007-03-14 00:07:51 +000040
41This requires 55M of memory. Thus 1 trivial busybox applet
Denis Vlasenkoe84aeb52007-03-20 11:08:39 +000042takes 55k of memory on 64-bit x86 kernel.
43
44On 32-bit kernel we need ~26k per applet.
45
Denis Vlasenko5a654472007-06-10 17:11:59 +000046Script:
47
48i=1000; while test $i != 0; do
49 echo -n .
50 busybox sleep 30 &
51 i=$((i - 1))
52done
53echo
54wait
55
Denis Vlasenkoe84aeb52007-03-20 11:08:39 +000056(Data from NOMMU arches are sought. Provide 'size busybox' output too)
Denis Vlasenko75605782007-03-14 00:07:51 +000057
Denis Vlasenko75605782007-03-14 00:07:51 +000058
Denis Vlasenko4efeaee2007-03-15 19:52:42 +000059 Example 1
Denis Vlasenko75605782007-03-14 00:07:51 +000060
61One example how to reduce global data usage is in
Denys Vlasenko8a6a2f92012-03-06 16:27:48 +010062archival/libarchive/decompress_unzip.c:
Denis Vlasenko75605782007-03-14 00:07:51 +000063
64/* This is somewhat complex-looking arrangement, but it allows
65 * to place decompressor state either in bss or in
66 * malloc'ed space simply by changing #defines below.
67 * Sizes on i386:
68 * text data bss dec hex
69 * 5256 0 108 5364 14f4 - bss
70 * 4915 0 0 4915 1333 - malloc
71 */
72#define STATE_IN_BSS 0
73#define STATE_IN_MALLOC 1
74
Denis Vlasenko4efeaee2007-03-15 19:52:42 +000075(see the rest of the file to get the idea)
76
Denis Vlasenko75605782007-03-14 00:07:51 +000077This example completely eliminates globals in that module.
Denis Vlasenkoc14d39e2007-06-08 13:05:39 +000078Required memory is allocated in unpack_gz_stream() [its main module]
Denis Vlasenko972288e2007-03-15 00:57:01 +000079and then passed down to all subroutines which need to access 'globals'
Denis Vlasenko75605782007-03-14 00:07:51 +000080as a parameter.
81
Denis Vlasenko4efeaee2007-03-15 19:52:42 +000082
83 Example 2
Denis Vlasenko75605782007-03-14 00:07:51 +000084
85In case you don't want to pass this additional parameter everywhere,
86take a look at archival/gzip.c. Here all global data is replaced by
Denis Vlasenko972288e2007-03-15 00:57:01 +000087single global pointer (ptr_to_globals) to allocated storage.
Denis Vlasenko75605782007-03-14 00:07:51 +000088
89In order to not duplicate ptr_to_globals in every applet, you can
90reuse single common one. It is defined in libbb/messages.c
Denis Vlasenko4efeaee2007-03-15 19:52:42 +000091as struct globals *const ptr_to_globals, but the struct globals is
Denis Vlasenko972288e2007-03-15 00:57:01 +000092NOT defined in libbb.h. You first define your own struct:
Denis Vlasenko75605782007-03-14 00:07:51 +000093
Denis Vlasenko972288e2007-03-15 00:57:01 +000094struct globals { int a; char buf[1000]; };
Denis Vlasenko75605782007-03-14 00:07:51 +000095
96and then declare that ptr_to_globals is a pointer to it:
97
Denis Vlasenko75605782007-03-14 00:07:51 +000098#define G (*ptr_to_globals)
99
Denis Vlasenko4efeaee2007-03-15 19:52:42 +0000100ptr_to_globals is declared as constant pointer.
101This helps gcc understand that it won't change, resulting in noticeably
Denis Vlasenko574f2f42008-02-27 18:41:59 +0000102smaller code. In order to assign it, use SET_PTR_TO_GLOBALS macro:
Denis Vlasenko75605782007-03-14 00:07:51 +0000103
Denis Vlasenko574f2f42008-02-27 18:41:59 +0000104 SET_PTR_TO_GLOBALS(xzalloc(sizeof(G)));
Denis Vlasenko75605782007-03-14 00:07:51 +0000105
Denis Vlasenko4efeaee2007-03-15 19:52:42 +0000106Typically it is done in <applet>_main().
Denis Vlasenko972288e2007-03-15 00:57:01 +0000107
Denis Vlasenko4efeaee2007-03-15 19:52:42 +0000108Now you can reference "globals" by G.a, G.buf and so on, in any function.
109
110
111 bb_common_bufsiz1
112
113There is one big common buffer in bss - bb_common_bufsiz1. It is a much
114earlier mechanism to reduce bss usage. Each applet can use it for
115its needs. Library functions are prohibited from using it.
116
117'G.' trick can be done using bb_common_bufsiz1 instead of malloced buffer:
118
119#define G (*(struct globals*)&bb_common_bufsiz1)
120
Denis Vlasenkoe84aeb52007-03-20 11:08:39 +0000121Be careful, though, and use it only if globals fit into bb_common_bufsiz1.
122Since bb_common_bufsiz1 is BUFSIZ + 1 bytes long and BUFSIZ can change
123from one libc to another, you have to add compile-time check for it:
124
Denis Vlasenko17a15262007-03-26 20:48:46 +0000125if (sizeof(struct globals) > sizeof(bb_common_bufsiz1))
Denis Vlasenkoe84aeb52007-03-20 11:08:39 +0000126 BUG_<applet>_globals_too_big();
Denis Vlasenko4efeaee2007-03-15 19:52:42 +0000127
128
129 Drawbacks
130
131You have to initialize it by hand. xzalloc() can be helpful in clearing
132allocated storage to 0, but anything more must be done by hand.
133
134All global variables are prefixed by 'G.' now. If this makes code
135less readable, use #defines:
136
137#define dev_fd (G.dev_fd)
138#define sector (G.sector)
139
140
Denys Vlasenkoabb154b2010-06-02 13:28:17 +0200141 Finding non-shared duplicated strings
142
143strings busybox | sort | uniq -c | sort -nr
144
145
Denis Vlasenko3d101dd2007-03-19 16:04:11 +0000146 gcc's data alignment problem
147
148The following attribute added in vi.c:
149
150static int tabstop;
151static struct termios term_orig __attribute__ ((aligned (4)));
152static struct termios term_vi __attribute__ ((aligned (4)));
153
Denis Vlasenkoe84aeb52007-03-20 11:08:39 +0000154reduces bss size by 32 bytes, because gcc sometimes aligns structures to
Denis Vlasenko3d101dd2007-03-19 16:04:11 +0000155ridiculously large values. asm output diff for above example:
156
157 tabstop:
158 .zero 4
159 .section .bss.term_orig,"aw",@nobits
160- .align 32
161+ .align 4
162 .type term_orig, @object
163 .size term_orig, 60
164 term_orig:
165 .zero 60
166 .section .bss.term_vi,"aw",@nobits
167- .align 32
168+ .align 4
169 .type term_vi, @object
170 .size term_vi, 60
171
172gcc doesn't seem to have options for altering this behaviour.
Denis Vlasenkoe84aeb52007-03-20 11:08:39 +0000173
Denis Vlasenkof3630652007-03-20 15:53:11 +0000174gcc 3.4.3 and 4.1.1 tested:
175char c = 1;
Denis Vlasenkoe84aeb52007-03-20 11:08:39 +0000176// gcc aligns to 32 bytes if sizeof(struct) >= 32
Denis Vlasenkof3630652007-03-20 15:53:11 +0000177struct {
178 int a,b,c,d;
179 int i1,i2,i3;
180} s28 = { 1 }; // struct will be aligned to 4 bytes
181struct {
182 int a,b,c,d;
183 int i1,i2,i3,i4;
184} s32 = { 1 }; // struct will be aligned to 32 bytes
Denis Vlasenkoe84aeb52007-03-20 11:08:39 +0000185// same for arrays
186char vc31[31] = { 1 }; // unaligned
187char vc32[32] = { 1 }; // aligned to 32 bytes
Denis Vlasenkof3630652007-03-20 15:53:11 +0000188
Denis Vlasenkob8e72fd2007-03-21 10:07:01 +0000189-fpack-struct=1 reduces alignment of s28 to 1 (but probably
190will break layout of many libc structs) but s32 and vc32
191are still aligned to 32 bytes.
192
193I will try to cook up a patch to add a gcc option for disabling it.
194Meanwhile, this is where it can be disabled in gcc source:
195
196gcc/config/i386/i386.c
197int
198ix86_data_alignment (tree type, int align)
199{
200#if 0
201 if (AGGREGATE_TYPE_P (type)
202 && TYPE_SIZE (type)
203 && TREE_CODE (TYPE_SIZE (type)) == INTEGER_CST
204 && (TREE_INT_CST_LOW (TYPE_SIZE (type)) >= 256
205 || TREE_INT_CST_HIGH (TYPE_SIZE (type))) && align < 256)
206 return 256;
207#endif
208
209Result (non-static busybox built against glibc):
210
211# size /usr/srcdevel/bbox/fix/busybox.t0/busybox busybox
212 text data bss dec hex filename
213 634416 2736 23856 661008 a1610 busybox
214 632580 2672 22944 658196 a0b14 busybox_noalign
Denys Vlasenkoa7bb3c12009-10-08 12:28:08 +0200215
216
217
218 Keeping code small
219
Bartosz Golaszewski28a20942013-10-16 19:18:05 +0200220Use scripts/bloat-o-meter to check whether introduced changes
221didn't generate unnecessary bloat. This script needs unstripped binaries
222to generate a detailed report. To automate this, just use
223"make bloatcheck". It requires busybox_old binary to be present,
224use "make baseline" to generate it from unmodified source, or
225copy busybox_unstripped to busybox_old before modifying sources
226and rebuilding.
227
Denys Vlasenkoa7bb3c12009-10-08 12:28:08 +0200228Set CONFIG_EXTRA_CFLAGS="-fno-inline-functions-called-once",
229produce "make bloatcheck", see the biggest auto-inlined functions.
230Now, set CONFIG_EXTRA_CFLAGS back to "", but add NOINLINE
231to some of these functions. In 1.16.x timeframe, the results were
232(annotated "make bloatcheck" output):
233
234function old new delta
235expand_vars_to_list - 1712 +1712 win
236lzo1x_optimize - 1429 +1429 win
237arith_apply - 1326 +1326 win
238read_interfaces - 1163 +1163 loss, leave w/o NOINLINE
239logdir_open - 1148 +1148 win
240check_deps - 1148 +1148 loss
241rewrite - 1039 +1039 win
242run_pipe 358 1396 +1038 win
243write_status_file - 1029 +1029 almost the same, leave w/o NOINLINE
244dump_identity - 987 +987 win
245mainQSort3 - 921 +921 win
246parse_one_line - 916 +916 loss
247summarize - 897 +897 almost the same
248do_shm - 884 +884 win
249cpio_o - 863 +863 win
250subCommand - 841 +841 loss
251receive - 834 +834 loss
252
253855 bytes saved in total.
Denys Vlasenkoadf922e2009-10-08 14:35:37 +0200254
255scripts/mkdiff_obj_bloat may be useful to automate this process: run
256"scripts/mkdiff_obj_bloat NORMALLY_BUILT_TREE FORCED_NOINLINE_TREE"
257and select modules which shrank.