TCP/TLS performance issue

[snarbles]

Any way to improve Event Per Second (EPS) using TCP/TLS? We seem to be throttled around 40k EPS which is less than our UDP which is about 80k EPS:

source_palo_tls:
type: syslog
address: 0.0.0.0:10001
mode: tcp
receive_buffer_bytes: 268435456
max_length: 409600
tls:
enabled: true
crt_file: /etc/vector/cert/logs.pem
key_file: /etc/vector/cert/logs.key
ca_file: /etc/vector/cert/ca_chain.pem
# if sending system presents a cert, set to true
verify_certificate: false
verify_hostname: false

We also set these settings in sysctl.d:
net.core.rmem_default = 268435456
net.core.wmem_default = 268435456
net.core.rmem_max = 268435456
net.core.wmem_max = 268435456
net.core.netdev_max_backlog = 2000

net.ipv4.tcp_rmem = 8192 2097152 268435456
net.ipv4.tcp_wmem = 4096 524288 268435456

[snarbles]

Currently we have this running and it looks like vector is having trouble keeping up with the Recv-Q from a high volume host when running "ss -t -a -n":
ESTAB 1560785438 0 10.X.X.X:10001 10.X.X.X:42499

[zhoufragon]

More comments on the issue.

ss -t -m -n|grep 175 -A1

ESTAB 1605607548 0 10.xxx.xxx.xxx:10001 10.xxx.xxx.xxx:37677
skmem:(r2147365632,rb2147483646,t0,tb2097152,f3328,w0,o0,bl0,d15066209)

This indicates that the received buffer has almost been fully used.

netstat -s

...
4454052 packets pruned from receive queue because of socket buffer overrun
399344 packets dropped from out-of-order queue because of socket buffer overrun
8206943950 packets collapsed in receive queue due to low socket buffer
truncated

Both outputs indicate that the vector's capability of handling data is far behind data received.

Notes that during these time, the CPU and memory and I/O are pretty good (util% < 5%)

which means that vector is hang on something, or waiting for something , it is not fully using OS resource.

[snarbles]

Performance begines to degrade after 22-35 TCP connections.

[snarbles]

Upgraded to .47 and performance decreased.