288 Commits

Author SHA1 Message Date
6b224de844 svm: fix overflow when no region is big enough
- if no region is big enough to accommodate need_n_bits, we can scan
   past the end of the bitmask
 - rewrite the search loop for clarity

Type: fix

Change-Id: I82637bc91eb7e7ae922785bc626fa224638cb4e4
Signed-off-by: Benoît Ganne <bganne@cisco.com>
2024-11-29 21:55:28 +00:00
afd05739d6 svm: mq use poll instead of SO_RCVTIMEO
setsockopt does not work on eventfds

Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I652a2b78160abe1bc15879fd8bc320ba4ef38e63
2024-11-05 05:31:36 +00:00
7f163b682a misc: replace index() with strchr()
Type: improvement
Change-Id: I471e514ebef0b4c1f86067115b2ebe5a5517c6fb
Signed-off-by: Matus Fabian <matfabia@cisco.com>
2024-07-22 17:07:02 +00:00
c3148b1be8 misc: remove GNU Indent directives
Type: refactor
Change-Id: I5235bf3e9aff58af6ba2c14e8c6529c4fc9ec86c
Signed-off-by: Damjan Marion <damarion@cisco.com>
2024-03-12 19:29:56 +00:00
eb5a08e91d svm: fix svm queue overwrite while adding 2 elements
Adding two elements to the full svm queue passes exact
bounds check, therefore tail gets overwritten w/o any
waiting. Fix it with requiring at lease two free slots.

Type: fix
Signed-off-by: Vladislav Grishenko <themiron@yandex-team.ru>
Change-Id: I704ee606c47b523952cb965f848339ae1988cb60
2024-03-03 18:47:41 +00:00
eceef16b5c svm: Use ftruncate to expand svm on FreeBSD
Linux doesn't support the Linux idiom of using lseek and a write to set
the size of a file, instead use ftruncate to accomplish the same effect.

This change is taken from the Nanoteq VPP port commit:
04a1b19b37

Type: improvement
Change-Id: Ie0b83e751b8b8f20b6814e5c9f760035747dfad9
Signed-off-by: Tom Jones <thj@freebsd.org>
2024-02-23 18:48:29 +00:00
9a97fb8ad7 svm: Only include sys/user on Linux
Type: improvement
Change-Id: Idb318eb386617013015c43153bf0bd92fa28b9b6
Signed-off-by: Tom Jones <thj@freebsd.org>
2024-02-18 16:08:21 +00:00
775002358f svm: Add FreeBSD specific signal handling path
FreeBSD's siginfo_t doesn't have the ptr value used on linux to encode a
token. Fow now recover the code from siginfo_t and leave the opaque
value as 0.

Type: improvement
Change-Id: Ic6ef96f018b326b99d68534a4488cfc49fdf16e1
Signed-off-by: Tom Jones <thj@freebsd.org>
2024-02-13 16:33:06 +00:00
caf9d4f4f8 svm: Include stdint on FreeBSD
On FreeBSD we need to include stdint.h to get __WORDSIZE.

Type: improvement
Change-Id: I784aa0856fd8298f33016b7ccedecfa3672d5429
Signed-off-by: Tom Jones <thj@freebsd.org>
2024-02-13 16:25:18 +00:00
0242d30fc7 session: async rx event notifications
Move from synchronous flushing of io and ctrl events from transports to
applications to an async model via a new session_input input node that
runs in interrupt mode. Events are coalesced per application worker.

On the one hand, this helps by minimizing message queue locking churn.
And on the other, it opens the possibility for further optimizations of
event message generation, obviates need for rx rescheduling rpcs and is
a first step towards a fully async data/io rx path.

Type: improvement

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: Id6bebcb65fc9feef8aa02ddf1af6d9ba6f6745ce
2023-08-09 18:45:26 +00:00
81e3243c1b svm: convert fifo want_deq_ntf ops to atomics
Type: improvement

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: Iba2e4de24ff0724e31859f5d2f4ffa3cfe2cf284
2023-06-08 19:23:51 +00:00
9f1ae220d0 session svm: fix mq cleanup
Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I8519bcd76a0ade2f24b62bf69f6a103379639cb7
2023-04-20 22:33:08 +00:00
6531cf5d1c vppinfra: fix AddressSanitizer
When checking for CLIB_SANITIZE_ADDR to enable specific behavior for
AddressSanitizer, we must have vppinfra/clib.h included as it is defined
there.

Type: fix

Change-Id: I9060c3c29c1289d28596c215a1d1709b2ea7c84e
Signed-off-by: Benoît Ganne <bganne@cisco.com>
2022-10-11 15:13:47 +00:00
c795b8836d vcl: add hugepage for vcl configure and svm
add hugepage for vcl configure and svm

Type: feature

Signed-off-by: Junfeng Wang <drenfong.wang@intel.com>
Change-Id: I6a8905e3fec23d840e629114b1e5a403d0a258ef
2022-09-15 23:15:26 +00:00
3ebe1efce1 svm: fix coverity 249207,249209
Zero-initialize the temporary struct.

Type: fix
Change-Id: I6f7a35ace6002aa75dc986c7c7eca614c9c5c3ed
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2022-08-17 15:47:23 +00:00
ffa7bac6f5 svm: check svm_msg_q_size_to_alloc success
svm_msg_q_size_to_alloc must return a valid base address, if it fails
pass up the error for handling

Type: fix

Change-Id: I408492f65f646862122acb9a187819b3bbf4f91c
Signed-off-by: Ofer Heifetz <oferh@marvell.com>
2022-06-29 20:21:27 +03:00
80d100cd59 session svm: fix mq producer wait on q and ring
Make sure producer drops lock when it waits for empty ring slot.

Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: Id77d54ee8c01bed20c9eaf5ad372ed4b1e9fa712
2022-04-21 16:48:15 +00:00
79934e855f vppinfra: refactor address sanitizer
Type: refactor
Change-Id: I5ca142ec1557d5b5c3806b43553ad9d3b5ea1112
Signed-off-by: Damjan Marion <damarion@cisco.com>
2022-04-05 14:46:15 +00:00
8bea589cfe vppinfra: make _vec_len() read-only
Use of _vec_len() to set vector length breaks address sanitizer.
Users should use vec_set_len(), vec_inc_len(), vec_dec_len () instead.

Type: improvement
Change-Id: I441ae948771eb21c23a61f3ff9163bdad74a2cb8
Signed-off-by: Damjan Marion <damarion@cisco.com>
2022-04-04 23:17:13 +00:00
5e06e4f311 svm: Close client shm fd
Type: fix

Signed-off-by: Emmanuel Scaria <emmanuelscaria11@gmail.com>
Change-Id: I41bb6e16735b77efa744b0368eed7dfa9583e960
2022-04-03 16:39:25 +00:00
299571aca3 vppinfra: vector allocator rework
- support of in-place growth of vectors (if there is available space next to
  existing alloc)
- drops the need for alloc_aligned_at_offset from memory allocator,
  which allows easier swap to different memory allocator and reduces
  malloc overhead
- rework of pool and vec macros to inline functions to improve debuggability
- fix alignment - in many cases macros were not using native alignment
  of the particular datatype. Explicitly setting alignment with XXX_aligned()
  versions of the macro is not needed anymore in > 99% of cases
- fix ASAN usage
- avoid use of vector of voids, this was root cause of several bugs
  found in vec_* and pool_* function where sizeof() was used on voids
  instead of real vector data type
- introduce minimal alignment which is currently 8 bytes, vectors will
  be always aligned at least to that value (underlay allocator actually always
  provide 16-byte aligned allocs)

Type: improvement
Change-Id: I20f4b081bb13bbf7bc0ace85cc4e301787f12fdf
Signed-off-by: Damjan Marion <damarion@cisco.com>
2022-03-30 18:27:13 +00:00
b85de19145 svm: update number of segments in svm_fifo_segments
In addition to returning the number of bytes also update the number of
segments to reflect the number used.

Type: improvement

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: Ia87dc2aa62cea38b18dfa83df94dc2abe29d5121
2022-01-19 15:58:20 +00:00
95bbbe479e svm: fix return for partial segment enqueue
Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I9953d9bf04e708ac8ea475127e3d2f606cc1c8d9
2022-01-19 15:58:20 +00:00
03e40e6230 svm: merge want deq and leq threshold notifications
Type: improvement

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I2775af35386a4e52ba82991c68bce6c56f13410f
2021-12-30 09:11:17 -08:00
fb079585cf svm: Add support for thresh dequeue notification
Add the ability to set a threshold on the fifo that will
be used to notify caller when de-queue from the fifo reached
a value that is less than or equal to the threshold.

Type: feature

Change-Id: I70ba1a05f783ce5247409e9beebe4e336b7c1eb5
Signed-off-by: Michal Kalderon <mkalderon@marvell.com>
Signed-off-by: Yuval Caduri <cyuval@marvell.com>
2021-12-29 17:54:41 +00:00
cdfe8abc61 session svm: track fs and seg manager index in fs
Simplifies allocation of fifos as fifo segment and segment manager
indices can be set at alloc time.

Type: improvement

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: Ibd357b3ff0279d8deefcdcb17010b4068007ccb7
2021-12-23 16:17:57 +00:00
a0024a66fe session: improve sh segment-manager cli
Type: improvement

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I5d669fcba609bcdb35103f57c45e0a270213d84a
2021-12-20 19:37:47 +00:00
2711ca710a session: improve fifo segment allocation
This patch ensures that fifo segment has at least the size that was
requested during allocation.

Type: improvement

Signed-off-by: Filip Tehlar <ftehlar@cisco.com>
Change-Id: Iea8a885ac290183e25e5c8f9163bba226c5efa15
2021-11-24 19:48:52 +00:00
d60b432b13 svm: remove unused fifo functions
Type: refactor

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I4525bc669d1c42c41dbc398cf239a093b6853298
2021-09-23 12:15:47 +00:00
afbb33aaf2 svm: make more fifo segment header fields private
Type: improvement

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: Idab0f455b52a38efd2cf3781497d8cc7f57bfa42
2021-08-11 14:41:32 +00:00
1ef38743fd svm: fix asan check failed @svm_map_region on arm
==283032==AddressSanitizer CHECK failed: compiler-rt/lib/asan/asan_mapping.h:366
          "((AddrIsInMem(p))) != (0)" (0x0, 0x0)
    #0 0x49c128 in __asan::AsanCheckFailed
    #1 0x4ae8dc in __sanitizer::CheckFailed
    #2 0x495dec in __asan::ShadowSegmentEndpoint::ShadowSegmentEndpoint
    #3 0x495e48 in __asan_unpoison_memory_region
    #4 0xfffff4e851f8 in svm_map_region /home/vpp/src/svm/svm.c:611:7
    #5 0xfffff4e86d9c in svm_region_init_internal /home/vpp/src/svm/svm.c:797:8
    #6 0xfffff4e87ce4 in svm_region_init_args /home/vpp/src/svm/svm.c:880:3
    #7 0xfffff7f30d30 in vlibmemory_init /home/vpp/src/vlibmemory/memory_api.c:974:3
    #8 0xfffff4fd5368 in vlib_main /home/vpp/src/vlib/main.c:1986:16

svm_global_region_base_va 0x200000000000 is not in the aarch64 mapping range,
leading check failure and vpp cannot start.

aarch64 asan mapping
|| `[0x201000000000, 0xffffffffffff]` || HighMem    ||
|| `[0x041200000000, 0x200fffffffff]` || HighShadow ||
|| `[0x001200000000, 0x0411ffffffff]` || ShadowGap  ||
|| `[0x001000000000, 0x0011ffffffff]` || LowShadow  ||
|| `[0x000000000000, 0x000fffffffff]` || LowMem     ||

x86 asan mapping
|| `[0x10007fff8000, 0x7fffffffffff]` || HighMem    ||
|| `[0x02008fff7000, 0x10007fff7fff]` || HighShadow ||
|| `[0x00008fff7000, 0x02008fff6fff]` || ShadowGap  ||
|| `[0x00007fff8000, 0x00008fff6fff]` || LowShadow  ||
|| `[0x000000000000, 0x00007fff7fff]` || LowMem     ||

Type: fix

Signed-off-by: Tianyu Li <tianyu.li@arm.com>
Change-Id: I55ddbdcd361d66d4cfaf6459b2fa20fd8b64af37
2021-06-24 14:40:58 +00:00
da78c5abc8 session: optimize ct fifo segment allocations
Allocate per app pair segments with space for more than one fifo.

Type: feature

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: Ib96fe12b899cb14ff20c0be607814011e2c3fc6a
2021-06-17 14:29:47 +00:00
8eb8d50ecf vcl: fix fifo sharing
Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: Iec1fe8315a057214901250d5fb06d1c1e33dda46
2021-06-16 23:23:50 +00:00
324d161963 svm: release mem order for fifo chunk list CAS
Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Change-Id: Ifad679f46abd6e9c18a3eaf7e55800a09f3791ab
2021-06-03 17:17:43 +00:00
5c520919e8 svm: fix inadequate atomic load for chunk mgr
- Fixes intermittent LDP cut thru iperf3 test
  failure on AARCH64

Type: fix

Change-Id: Id21a078d642e03d974bacacd1f4d0faa42fb6652
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-05-28 16:08:09 +00:00
e3c6a54995 tls svm: prealloc tcp fifo chunks before ssl write
Type: improvement

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I7c47b55ec6f0c83f2d13e0e737d0559a32f7c837
2021-05-20 10:03:57 +00:00
0d31b9cb9f svm: try chunk alloc only if enough space available
Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I3c52dd783601d50e981f75bd05a7b95c03bb5c22
2021-05-10 14:38:36 +00:00
00338e0912 svm: allow external chunk alloc on segment
Type: improvement

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I89f9caa8dd44dc640615a58fe7708f388fdd84e9
2021-04-20 22:04:49 +00:00
014dba38cb svm: lock-free fifo chunk list push and pop
This avoids chunk allocation/collection deadlocks if either of the sides
crashes.

Type: improvement

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I98619e6e035fa8688889ca34db2143c8898732df
2021-04-02 17:15:07 +00:00
8c517c8fb4 svm: cleanup mq private data on fifo segment cleanup
Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I65373f2cd5a33381da1e51ed7ab7a8b1358cef29
2021-03-30 13:17:42 +00:00
41d5f541d3 svm session vcl: per app rx message queues
Add option to use per app private segments for app to vpp message
queues, as opposed to exposing internal message queues segment.

When so configured, internal message queues are still polled by the
session queue node but external app message queues are handled by a new
input node (appsl-rx-mqs-input) that runs in interrupt state. Signaling
of the node, when mqs receive new messages, is done through eventfds
epolled by worker epoll input nodes.

Type: feature

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: Iffe8ce5a9944a56a14e6d0f492a850cb9e392d16
2021-03-29 20:20:03 +00:00
89c98a4bff svm: add producer wait function
Type: refactor

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I9488ad7e045c908b60b5821d9c48583f6d513c2f
2021-03-25 23:50:14 +00:00
abd8b36d19 svm: fix producer deadlock in svm_msg_q_wait()
1. When producer invokes svm_msg_q_wait() in svm_msg_q_lock_and_alloc_msg_w_ring(), queue mutex is held by itself.
2. Sometimes, svm msg queue is not full and ring is full, svm_msg_q_wait() do nothing with mutex held, consumer will blocking at svm_msg_q_send_signal().

Type: fix

Signed-off-by: nandfan <fanyufei521@outlook.com>
Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: Ib90b87ab76534cd42e9a4c3e11703e80d93ca678
2021-03-25 19:54:18 +00:00
f1af21c9d7 session svm: segment manager and fifo segment leaks
Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I4e00dd7f8ce1e56092dde9a073decae62d5475de
2021-02-28 01:34:40 +00:00
9c499e3a68 svm: fix shared hdr migration
Avoid changing the header on attach as it may be in use. Instead, as for
chunks, allocate header to be collected on detach.

Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: Ib316ecb5d61ae161032869b6f6a1863f1105a1d9
2021-02-26 15:51:00 +00:00
cbb5e824af vcl: cleanup fifos detached from segments
Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I412024731c1f561680736ad7bfabb99b595e3dff
2021-02-22 22:26:45 +00:00
44fadb0a60 svm: free shared fifo on detach
Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I639560ee3dd0a1d605ec2866dce5cdd13fda8201
2021-02-22 22:26:31 +00:00
255554fd7e svm: return chunks to slice on fifo detach
Ensure chunk alloc distribution is maintained on fifo detach.

Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: I5aa5524e06a703dc50e90da6d177663d2d997aa4
2021-02-20 19:10:59 +00:00
ee5cd4e470 session vppinfra: asan fixes
Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: Ie709d76438542783cbc8c6174b5e712ef18a6276
2021-02-19 08:16:06 +00:00
5c01dbc47c svm: fix active fifo ll on attach
Type: fix

Signed-off-by: Florin Coras <fcoras@cisco.com>
Change-Id: Idf44f8d54c97fc43da5d5760e5ce477af07e5fbf
2021-02-18 23:18:26 +00:00