05cac30a92
and document scaling Change-Id: I65d8999e65616d77e525963c770d91e9b0d5e593 Signed-off-by: Neale Ranns <nranns@cisco.com>
284 lines
9.2 KiB
ReStructuredText
284 lines
9.2 KiB
ReStructuredText
.. _scale:
|
|
|
|
Scale
|
|
-----
|
|
|
|
The only limiting factor on FIB scale is the amount of memory
|
|
allocated to each heap the FIB uses, and there are 4:
|
|
|
|
* The IP4 heap
|
|
* The IP6 heap
|
|
* The main heap
|
|
* The stats heap
|
|
|
|
|
|
IP4 Heap
|
|
--------
|
|
|
|
The IPv4 heap is used to allocate the memory needed for the
|
|
data-structures within which the IPv4 prefixes are stored. Each
|
|
table, created by the user, i.e. with;
|
|
|
|
.. code-block:: console
|
|
|
|
$ ip table add 1
|
|
|
|
or the default table, comprises 2 *ip4_fib_t* objects.
|
|
The 'non-forwarding' *ip4_fib_t* contains all the entries in the table
|
|
and, the 'forwarding' contains the entries that are matched against in
|
|
the data-plane. The difference between the two sets are the entries
|
|
that should not be matched in the data-plane.
|
|
Each *ip4_fib_t* comprises an mtrie (for fast lookup in the data-plane)
|
|
and a hash table per-prefix length (for lookup in the control plane).
|
|
|
|
To see the amount of memory consumed by the IPv4 tables use:
|
|
|
|
.. code-block:: console
|
|
|
|
vpp# sh ip fib mem
|
|
ipv4-VRF:0 mtrie:333056 hash:3523
|
|
ipv4-VRF:1 mtrie:333056 hash:3523
|
|
totals: mtrie:666112 hash:7046 all:673158
|
|
|
|
Mtrie Mheap Usage: total: 32.06M, used: 662.44K, free: 31.42M, trimmable: 31.09M
|
|
free chunks 3 free fastbin blks 0
|
|
max total allocated 32.06M
|
|
no traced allocations
|
|
|
|
this output shows two 'empty' (i.e. no added routes) tables. Each
|
|
mtrie uses about 150k of memory, so each table about 300k. the total
|
|
heap usage statistics for the IP4 heap are shown at the end.
|
|
|
|
|
|
Below the output having added 1M, 2M and 4M routes respectively:
|
|
|
|
.. code-block:: console
|
|
|
|
vpp# sh ip fib mem
|
|
ipv4-VRF:0 mtrie:335744 hash:4695
|
|
totals: mtrie:335744 hash:4695 all:340439
|
|
|
|
Mtrie Mheap Usage: total: 1.00G, used: 335.20K, free: 1023.74M, trimmable: 1023.72M
|
|
free chunks 3 free fastbin blks 0
|
|
max total allocated 1.00G
|
|
no traced allocations
|
|
|
|
.. code-block:: console
|
|
|
|
vpp# sh ip fib mem
|
|
ipv4-VRF:0 mtrie:5414720 hash:41177579
|
|
totals: mtrie:5414720 hash:41177579 all:46592299
|
|
|
|
Mtrie Mheap Usage: total: 1.00G, used: 46.87M, free: 977.19M, trimmable: 955.93M
|
|
free chunks 61 free fastbin blks 0
|
|
max total allocated 1.00G
|
|
no traced allocations
|
|
|
|
.. code-block:: console
|
|
|
|
vpp# sh ip fib mem
|
|
ipv4-VRF:0 mtrie:22452608 hash:168544508
|
|
totals: mtrie:22452608 hash:168544508 all:190997116
|
|
|
|
Mtrie Mheap Usage: total: 1.00G, used: 198.37M, free: 825.69M, trimmable: 748.24M
|
|
free chunks 219 free fastbin blks 0
|
|
max total allocated 1.00G
|
|
no traced allocations
|
|
|
|
VPP was started with a 1G IP4 heap.
|
|
|
|
IP6 Heap
|
|
--------
|
|
|
|
The IPv6 heap is used to allocate the memory needed for the
|
|
data-structure within which the IPv6 prefixes are stored. IPv6 also
|
|
has the concept of forwarding and non-forwarding entries, however for
|
|
IPv6 all the forwardind entries are stored in a single hash table
|
|
(same goes for the non-forwarding). The key to the hash table includes
|
|
the IPv6 table-id.
|
|
|
|
To see the amount of memory consumed by the IPv4 tables use:
|
|
|
|
.. code-block:: console
|
|
|
|
vpp# sh ip6 fib mem
|
|
IPv6 Non-Forwarding Hash Table:
|
|
Hash table ip6 FIB non-fwding table
|
|
7 active elements 7 active buckets
|
|
1 free lists
|
|
0 linear search buckets
|
|
arena: base 7f2fe28bf000, next 803c0
|
|
used 525248 b (0 Mbytes) of 33554432 b (32 Mbytes)
|
|
|
|
IPv6 Forwarding Hash Table:
|
|
Hash table ip6 FIB fwding table
|
|
7 active elements 7 active buckets
|
|
1 free lists
|
|
0 linear search buckets
|
|
arena: base 7f2fe48bf000, next 803c0
|
|
used 525248 b (0 Mbytes) of 33554432 b (32 Mbytes)
|
|
|
|
as we scale to 128k IPv6 entries:
|
|
|
|
.. code-block:: console
|
|
|
|
vpp# sh ip6 fib mem
|
|
IPv6 Non-Forwarding Hash Table:
|
|
Hash table ip6 FIB non-fwding table
|
|
131079 active elements 32773 active buckets
|
|
2 free lists
|
|
[len 1] 2 free elts
|
|
0 linear search buckets
|
|
arena: base 7fed7a514000, next 4805c0
|
|
used 4720064 b (4 Mbytes) of 1073741824 b (1024 Mbytes)
|
|
|
|
IPv6 Forwarding Hash Table:
|
|
Hash table ip6 FIB fwding table
|
|
131079 active elements 32773 active buckets
|
|
2 free lists
|
|
[len 1] 2 free elts
|
|
0 linear search buckets
|
|
arena: base 7fedba514000, next 4805c0
|
|
used 4720064 b (4 Mbytes) of 1073741824 b (1024 Mbytes)
|
|
|
|
and 256k:
|
|
|
|
.. code-block:: console
|
|
|
|
vpp# sh ip6 fib mem
|
|
IPv6 Non-Forwarding Hash Table:
|
|
Hash table ip6 FIB non-fwding table
|
|
262151 active elements 65536 active buckets
|
|
2 free lists
|
|
[len 1] 6 free elts
|
|
0 linear search buckets
|
|
arena: base 7fed7a514000, next 880840
|
|
used 8915008 b (8 Mbytes) of 1073741824 b (1024 Mbytes)
|
|
|
|
IPv6 Forwarding Hash Table:
|
|
Hash table ip6 FIB fwding table
|
|
262151 active elements 65536 active buckets
|
|
2 free lists
|
|
[len 1] 6 free elts
|
|
0 linear search buckets
|
|
arena: base 7fedba514000, next 880840
|
|
used 8915008 b (8 Mbytes) of 1073741824 b (1024 Mbytes)
|
|
|
|
and 1M:
|
|
|
|
.. code-block:: console
|
|
|
|
vpp# sh ip6 fib mem
|
|
IPv6 Non-Forwarding Hash Table:
|
|
Hash table ip6 FIB non-fwding table
|
|
1048583 active elements 65536 active buckets
|
|
4 free lists
|
|
[len 1] 65533 free elts
|
|
[len 2] 65531 free elts
|
|
[len 4] 9 free elts
|
|
0 linear search buckets
|
|
arena: base 7fed7a514000, next 3882740
|
|
used 59254592 b (56 Mbytes) of 1073741824 b (1024 Mbytes)
|
|
|
|
IPv6 Forwarding Hash Table:
|
|
Hash table ip6 FIB fwding table
|
|
1048583 active elements 65536 active buckets
|
|
4 free lists
|
|
[len 1] 65533 free elts
|
|
[len 2] 65531 free elts
|
|
[len 4] 9 free elts
|
|
0 linear search buckets
|
|
arena: base 7fedba514000, next 3882740
|
|
used 59254592 b (56 Mbytes) of 1073741824 b (1024 Mbytes)
|
|
|
|
as can be seen from the output the IPv6 heap in this case was scaled
|
|
to 1GB and 1million prefixes has used 56MB of it.
|
|
|
|
|
|
Main Heap
|
|
---------
|
|
|
|
The main heap is used to allocate objects that represent the FIB
|
|
entries in the control and data plane (see :ref:`controlplane` and
|
|
:ref:`dataplane`) such as *fib_entry_t* and *load_balance_t*. These come
|
|
from the main heap because they are not protocol specific
|
|
(i.e. they are used to represent either IPv4, IPv6 or MPLS
|
|
entries).
|
|
|
|
With 1M prefixes allocated the memory usage is:
|
|
|
|
.. code-block:: console
|
|
|
|
vpp# sh fib mem
|
|
FIB memory
|
|
Tables:
|
|
SAFI Number Bytes
|
|
IPv4 unicast 1 33619968
|
|
IPv6 unicast 2 118502784
|
|
MPLS 0 0
|
|
IPv4 multicast 1 1175
|
|
IPv6 multicast 1 525312
|
|
Nodes:
|
|
Name Size in-use /allocated totals
|
|
Entry 72 1048589/ 1048589 75498408/75498408
|
|
Entry Source 40 1048589/ 1048589 41943560/41943560
|
|
Entry Path-Extensions 76 0 / 0 0/0
|
|
multicast-Entry 192 6 / 6 1152/1152
|
|
Path-list 40 18 / 18 720/720
|
|
uRPF-list 16 14 / 14 224/224
|
|
Path 72 22 / 22 1584/1584
|
|
Node-list elements 20 1048602/ 1048602 20972040/20972040
|
|
Node-list heads 8 24 / 24 192/192
|
|
|
|
and with 2M
|
|
|
|
.. code-block:: console
|
|
|
|
vpp# sh fib mem
|
|
FIB memory
|
|
Tables:
|
|
SAFI Number Bytes
|
|
IPv4 unicast 1 33619968
|
|
IPv6 unicast 2 252743040
|
|
MPLS 0 0
|
|
IPv4 multicast 1 1175
|
|
IPv6 multicast 1 525312
|
|
Nodes:
|
|
Name Size in-use /allocated totals
|
|
Entry 72 2097165/ 2097165 150995880/150995880
|
|
Entry Source 40 2097165/ 2097165 83886600/83886600
|
|
Entry Path-Extensions 76 0 / 0 0/0
|
|
multicast-Entry 192 6 / 6 1152/1152
|
|
Path-list 40 18 / 19 720/760
|
|
uRPF-list 16 18 / 18 288/288
|
|
Path 72 22 / 23 1584/1656
|
|
Node-list elements 20 2097178/ 2097178 41943560/41943560
|
|
Node-list heads 8 24 / 24 192/192
|
|
|
|
However, the situation is not a simple as that. All of the 1M prefixes
|
|
added above were reachable via the same next-hop, so the path-list
|
|
(and path) they use is shared. As prefixes are added that use
|
|
different (sets of) next-hops, the number of path-lists and paths
|
|
requires will increase.
|
|
|
|
|
|
Stats Heap
|
|
----------
|
|
|
|
VPP collects statistics for each route. For each route VPP collects
|
|
byte and packet counters for packets sent to the prefix (i.e. the
|
|
route was matched in the data-plane) and packets sent via the prefix (i.e. the
|
|
matching prefix is reachable through it - like a BGP peer). This
|
|
requires 4 counters per route in the stats segment.
|
|
|
|
Below shows the size of the stats segment with 1M, 2M and 4M routes.
|
|
|
|
.. code-block:: console
|
|
|
|
total: 1023.99M, used: 127.89M, free: 896.10M, trimmable: 830.94M
|
|
total: 1023.99M, used: 234.14M, free: 789.85M, trimmable: 668.15M
|
|
total: 1023.99M, used: 456.83M, free: 567.17M, trimmable: 388.91M
|
|
|
|
VPP was started with a 1G stats heap.
|
|
|