| .. _scale: |
| |
| Scale |
| ----- |
| |
| The only limiting factor on FIB scale is the amount of memory |
| allocated to each heap the FIB uses, and there are 4: |
| |
| * The IP4 heap |
| * The IP6 heap |
| * The main heap |
| * The stats heap |
| |
| |
| IP4 Heap |
| -------- |
| |
| The IPv4 heap is used to allocate the memory needed for the |
| data-structures within which the IPv4 prefixes are stored. Each |
| table, created by the user, i.e. with; |
| |
| .. code-block:: console |
| |
| $ ip table add 1 |
| |
| or the default table, comprises 2 *ip4_fib_t* objects. |
| The 'non-forwarding' *ip4_fib_t* contains all the entries in the table |
| and, the 'forwarding' contains the entries that are matched against in |
| the data-plane. The difference between the two sets are the entries |
| that should not be matched in the data-plane. |
| Each *ip4_fib_t* comprises an mtrie (for fast lookup in the data-plane) |
| and a hash table per-prefix length (for lookup in the control plane). |
| |
| To see the amount of memory consumed by the IPv4 tables use: |
| |
| .. code-block:: console |
| |
| vpp# sh ip fib mem |
| ipv4-VRF:0 mtrie:333056 hash:3523 |
| ipv4-VRF:1 mtrie:333056 hash:3523 |
| totals: mtrie:666112 hash:7046 all:673158 |
| |
| Mtrie Mheap Usage: total: 32.06M, used: 662.44K, free: 31.42M, trimmable: 31.09M |
| free chunks 3 free fastbin blks 0 |
| max total allocated 32.06M |
| no traced allocations |
| |
| this output shows two 'empty' (i.e. no added routes) tables. Each |
| mtrie uses about 150k of memory, so each table about 300k. the total |
| heap usage statistics for the IP4 heap are shown at the end. |
| |
| |
| Below the output having added 1M, 2M and 4M routes respectively: |
| |
| .. code-block:: console |
| |
| vpp# sh ip fib mem |
| ipv4-VRF:0 mtrie:335744 hash:4695 |
| totals: mtrie:335744 hash:4695 all:340439 |
| |
| Mtrie Mheap Usage: total: 1.00G, used: 335.20K, free: 1023.74M, trimmable: 1023.72M |
| free chunks 3 free fastbin blks 0 |
| max total allocated 1.00G |
| no traced allocations |
| |
| .. code-block:: console |
| |
| vpp# sh ip fib mem |
| ipv4-VRF:0 mtrie:5414720 hash:41177579 |
| totals: mtrie:5414720 hash:41177579 all:46592299 |
| |
| Mtrie Mheap Usage: total: 1.00G, used: 46.87M, free: 977.19M, trimmable: 955.93M |
| free chunks 61 free fastbin blks 0 |
| max total allocated 1.00G |
| no traced allocations |
| |
| .. code-block:: console |
| |
| vpp# sh ip fib mem |
| ipv4-VRF:0 mtrie:22452608 hash:168544508 |
| totals: mtrie:22452608 hash:168544508 all:190997116 |
| |
| Mtrie Mheap Usage: total: 1.00G, used: 198.37M, free: 825.69M, trimmable: 748.24M |
| free chunks 219 free fastbin blks 0 |
| max total allocated 1.00G |
| no traced allocations |
| |
| VPP was started with a 1G IP4 heap. |
| |
| IP6 Heap |
| -------- |
| |
| The IPv6 heap is used to allocate the memory needed for the |
| data-structure within which the IPv6 prefixes are stored. IPv6 also |
| has the concept of forwarding and non-forwarding entries, however for |
| IPv6 all the forwardind entries are stored in a single hash table |
| (same goes for the non-forwarding). The key to the hash table includes |
| the IPv6 table-id. |
| |
| To see the amount of memory consumed by the IPv4 tables use: |
| |
| .. code-block:: console |
| |
| vpp# sh ip6 fib mem |
| IPv6 Non-Forwarding Hash Table: |
| Hash table ip6 FIB non-fwding table |
| 7 active elements 7 active buckets |
| 1 free lists |
| 0 linear search buckets |
| arena: base 7f2fe28bf000, next 803c0 |
| used 525248 b (0 Mbytes) of 33554432 b (32 Mbytes) |
| |
| IPv6 Forwarding Hash Table: |
| Hash table ip6 FIB fwding table |
| 7 active elements 7 active buckets |
| 1 free lists |
| 0 linear search buckets |
| arena: base 7f2fe48bf000, next 803c0 |
| used 525248 b (0 Mbytes) of 33554432 b (32 Mbytes) |
| |
| as we scale to 128k IPv6 entries: |
| |
| .. code-block:: console |
| |
| vpp# sh ip6 fib mem |
| IPv6 Non-Forwarding Hash Table: |
| Hash table ip6 FIB non-fwding table |
| 131079 active elements 32773 active buckets |
| 2 free lists |
| [len 1] 2 free elts |
| 0 linear search buckets |
| arena: base 7fed7a514000, next 4805c0 |
| used 4720064 b (4 Mbytes) of 1073741824 b (1024 Mbytes) |
| |
| IPv6 Forwarding Hash Table: |
| Hash table ip6 FIB fwding table |
| 131079 active elements 32773 active buckets |
| 2 free lists |
| [len 1] 2 free elts |
| 0 linear search buckets |
| arena: base 7fedba514000, next 4805c0 |
| used 4720064 b (4 Mbytes) of 1073741824 b (1024 Mbytes) |
| |
| and 256k: |
| |
| .. code-block:: console |
| |
| vpp# sh ip6 fib mem |
| IPv6 Non-Forwarding Hash Table: |
| Hash table ip6 FIB non-fwding table |
| 262151 active elements 65536 active buckets |
| 2 free lists |
| [len 1] 6 free elts |
| 0 linear search buckets |
| arena: base 7fed7a514000, next 880840 |
| used 8915008 b (8 Mbytes) of 1073741824 b (1024 Mbytes) |
| |
| IPv6 Forwarding Hash Table: |
| Hash table ip6 FIB fwding table |
| 262151 active elements 65536 active buckets |
| 2 free lists |
| [len 1] 6 free elts |
| 0 linear search buckets |
| arena: base 7fedba514000, next 880840 |
| used 8915008 b (8 Mbytes) of 1073741824 b (1024 Mbytes) |
| |
| and 1M: |
| |
| .. code-block:: console |
| |
| vpp# sh ip6 fib mem |
| IPv6 Non-Forwarding Hash Table: |
| Hash table ip6 FIB non-fwding table |
| 1048583 active elements 65536 active buckets |
| 4 free lists |
| [len 1] 65533 free elts |
| [len 2] 65531 free elts |
| [len 4] 9 free elts |
| 0 linear search buckets |
| arena: base 7fed7a514000, next 3882740 |
| used 59254592 b (56 Mbytes) of 1073741824 b (1024 Mbytes) |
| |
| IPv6 Forwarding Hash Table: |
| Hash table ip6 FIB fwding table |
| 1048583 active elements 65536 active buckets |
| 4 free lists |
| [len 1] 65533 free elts |
| [len 2] 65531 free elts |
| [len 4] 9 free elts |
| 0 linear search buckets |
| arena: base 7fedba514000, next 3882740 |
| used 59254592 b (56 Mbytes) of 1073741824 b (1024 Mbytes) |
| |
| as can be seen from the output the IPv6 heap in this case was scaled |
| to 1GB and 1million prefixes has used 56MB of it. |
| |
| |
| Main Heap |
| --------- |
| |
| The main heap is used to allocate objects that represent the FIB |
| entries in the control and data plane (see :ref:`controlplane` and |
| :ref:`dataplane`) such as *fib_entry_t* and *load_balance_t*. These come |
| from the main heap because they are not protocol specific |
| (i.e. they are used to represent either IPv4, IPv6 or MPLS |
| entries). |
| |
| With 1M prefixes allocated the memory usage is: |
| |
| .. code-block:: console |
| |
| vpp# sh fib mem |
| FIB memory |
| Tables: |
| SAFI Number Bytes |
| IPv4 unicast 1 33619968 |
| IPv6 unicast 2 118502784 |
| MPLS 0 0 |
| IPv4 multicast 1 1175 |
| IPv6 multicast 1 525312 |
| Nodes: |
| Name Size in-use /allocated totals |
| Entry 72 1048589/ 1048589 75498408/75498408 |
| Entry Source 40 1048589/ 1048589 41943560/41943560 |
| Entry Path-Extensions 76 0 / 0 0/0 |
| multicast-Entry 192 6 / 6 1152/1152 |
| Path-list 40 18 / 18 720/720 |
| uRPF-list 16 14 / 14 224/224 |
| Path 72 22 / 22 1584/1584 |
| Node-list elements 20 1048602/ 1048602 20972040/20972040 |
| Node-list heads 8 24 / 24 192/192 |
| |
| and with 2M |
| |
| .. code-block:: console |
| |
| vpp# sh fib mem |
| FIB memory |
| Tables: |
| SAFI Number Bytes |
| IPv4 unicast 1 33619968 |
| IPv6 unicast 2 252743040 |
| MPLS 0 0 |
| IPv4 multicast 1 1175 |
| IPv6 multicast 1 525312 |
| Nodes: |
| Name Size in-use /allocated totals |
| Entry 72 2097165/ 2097165 150995880/150995880 |
| Entry Source 40 2097165/ 2097165 83886600/83886600 |
| Entry Path-Extensions 76 0 / 0 0/0 |
| multicast-Entry 192 6 / 6 1152/1152 |
| Path-list 40 18 / 19 720/760 |
| uRPF-list 16 18 / 18 288/288 |
| Path 72 22 / 23 1584/1656 |
| Node-list elements 20 2097178/ 2097178 41943560/41943560 |
| Node-list heads 8 24 / 24 192/192 |
| |
| However, the situation is not a simple as that. All of the 1M prefixes |
| added above were reachable via the same next-hop, so the path-list |
| (and path) they use is shared. As prefixes are added that use |
| different (sets of) next-hops, the number of path-lists and paths |
| requires will increase. |
| |
| |
| Stats Heap |
| ---------- |
| |
| VPP collects statistics for each route. For each route VPP collects |
| byte and packet counters for packets sent to the prefix (i.e. the |
| route was matched in the data-plane) and packets sent via the prefix (i.e. the |
| matching prefix is reachable through it - like a BGP peer). This |
| requires 4 counters per route in the stats segment. |
| |
| Below shows the size of the stats segment with 1M, 2M and 4M routes. |
| |
| .. code-block:: console |
| |
| total: 1023.99M, used: 127.89M, free: 896.10M, trimmable: 830.94M |
| total: 1023.99M, used: 234.14M, free: 789.85M, trimmable: 668.15M |
| total: 1023.99M, used: 456.83M, free: 567.17M, trimmable: 388.91M |
| |
| VPP was started with a 1G stats heap. |
| |