why my unoptimized erlang echo server is 3 times faster than tokio under load

[Geo-7]

I used basic TCP echo server with Tokio from github and I developed a simple clients for testing the echo server
with one client Tokio server and simple Erlang echo server behave the same but as soon as I increase the clients
tokio gets slower. what optimization I should do on tokio sample?
result for 255 clients each one sending 1000 msg(each 85 bytes) data simultaneously

Erlang result:

22.44s user 23.73s system 555% cpu 8.311 total

(cargo build --release)
Tokio result:

33.32s user 36.93s system 285% cpu 24.576 total

[[package]]
name = "tokio"
version = "1.12.0"

Linux 5.14.7-arch1-1 #1 SMP PREEMPT Wed, 22 Sep 2021 21:35:11 +0000 x86_64 GNU/Linux

rustc 1.55.0 (c8dfcfe04 2021-09-06)

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 43 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 7 2700X Eight-Core Processor
CPU family: 23

UPDATE : Sample codes.
compile erlang codes with erlc *.erl
for running single clinet:
time erl -noshell -s single_client start
single_client.erl:

-module(single_client).

-export([connect/2, start/0, start/4, send_data/4]).

connect(Host, Port) ->
    io:format("Starting connect~n"),
    case gen_tcp:connect(Host, Port, [binary, {packet, 2}, {keepalive, true}]) of
      {error, econnrefused} ->
          io:format("Server is not listening on ~s:~p try to reconnect in 5sec~n", [Host, Port]),
          timer:sleep(5000),
          connect(Host, Port);
      %connect(Host, Port);
      {error, closed} ->
          io:format("Server closed my connection ~s:~p try to reconnect in 5sec~n", [Host, Port]),
          timer:sleep(5000),

          connect(Host, Port);
      {error, econnreset} ->
          io:format("Server reset my connection ~s:~p try to reconnect in 5sec~n", [Host, Port]),
          timer:sleep(5000),
          connect(Host, Port);
      %connect(Host, Port);
      {ok, Socket} ->
          io:format("Connected to backend ~s:~p~n", [Host, Port]),
          Socket
    end.


send_data(Socket, 0, direct, _) ->
    io:format("Test Finished~n"),
    gen_tcp:close(Socket),
    halt();
send_data(Socket, 0, Pid, Terminal) ->
    io:format("Instace ~p Finished~n", [Terminal]),
    gen_tcp:close(Socket),
    Pid ! {finished, self(), Terminal};
send_data(Socket, No, Option, Terminal) ->
    DummyRequest = [ 32, 127, 138, 0, 0, 50, 0, 211, 12, 215, 57, 14, 19, 54, 34, 41,
                    16, 50, 53, 95, 33, 42, 42, 3, 18, 1, 12, 17, 23, 12, 12, 40, rand:uniform(255), 5,
                    18, 19, 24, 20, 224, 125, 56, 50, 40, 22, 12, 23, 38, 98, 35, 42, 81, 6,
                    54, 7, 4, 20, 112, 0, 207, 4, 5, 85, 40, 85, 0, 5, 177, 60, 42, 202, 10, 50,
                    49, 48, 61, 49, 20, 46, 48, 55, 59, 12, 52, 40, 50],
    case gen_tcp:send(Socket, DummyRequest) of
      ok ->
          %io:format("Packet send to server"),
          %io:format("P=~p",[Option]),
          {ok};
      {error, closed} ->
          io:format("Damn server is closed in sendig~n")
    end,
    receive
      {tcp, Socket, Bin} ->
          %io:format("Received Response from Backend=~p~n", [Bin]),
          send_data(Socket, No - 1, Option, Terminal)
    end.

start() ->
    S = connect("127.0.0.1", 9000),
    send_data(S, 200000, direct, 0).

start(Pid, N, Terminal, Port) ->
    io:format("I am client ~p my parent is ~p~n", [self(), Pid]),
    S = connect("127.0.0.1", Port),
    send_data(S, N, Pid, Terminal).

for running multiple clinet:
time erl -noshell -s multi_client start
multi_client.erl:
You can change number of concurrent client on this line
CIDs = lists:seq(1,255),
on my AMD Ryzen 8 Core CPU I set it to 255 with no problem but on my Mac M1 I should change it to 5 !

-module(multi_client).
-export([start/0,tester/4,collectResult/1]).

start() ->
	CIDs = lists:seq(1,255),
	Self = self(),
	io:format("I am supervisor ~p~n",[Self]),
	Testers = [spawn_link(fun()-> tester(Self,100000,CID,9000) end) || CID <- CIDs],
	[Tester ! {start} || Tester <- Testers],
	ok = collectResult(CIDs),
	halt().

tester(Pid,Number,CID,Port) ->
	receive
		{start} -> 
			io:format("I am tester ~p~n",[self()]),
			single_client:start(Pid,Number,CID,Port);
		Msg ->
			io:format("Unknown Msd in tester~w~n",[Msg]),
			tester(Pid,Number,CID,Port)

	end.

collectResult([]) -> ok;
collectResult(Names) ->
	io:format("CIDs are ~w~n",[Names]),
	receive
	{finished,Pid,Name} -> 
			io:format("Instance ~p Pid=~p is finished~n",[Name,Pid]),
			collectResult(Names -- [Name]);
	{'EXIT',Pid,normal} -> ok,collectResult(Names);
	Msg -> 
			io:format("what?~p~n",[Msg]),
			collectResult(Names)
	end.

for running erlnag echo server :
erl -noshell -s echo_server start
echo_server:

-module(echo_server).
-export([nano_server/2,par_accept/1,loop/1,start/0,start/1]).
-import(iso,[parse_trx/2]).
start() ->
	nano_server("127.0.0.1",9000),
	timer:sleep(infinity).
start(1) ->
	nano_server("127.0.0.1",9000).
nano_server(Ip,Port) ->
	{ok,Listen} =gen_tcp:listen(Port,[binary,{packet,2},{active,true},{reuseaddr,true}]),
	spawn(fun()->par_accept(Listen) end).
par_accept(Listen) -> 
	{ok,Socket} = gen_tcp:accept(Listen),
	spawn(fun()->par_accept(Listen) end),
	%ok = gen_tcp:controlling_process(Listen,Pid),
	loop(Socket).
loop(Socket) ->
	receive
	  {tcp,Socket,Bin} ->
			gen_tcp:send(Socket,Bin),
			loop(Socket);
	  {tcp_close,Socket} ->
			gen_tcp:close(Socket)
	end.

rust echo server :
I did not implement reading packet size so I changed buffer size to 87 (size of dummy msg)
change it to 1024 does not have any effect

use tokio::net::TcpListener;
use tokio::io::{AsyncReadExt, AsyncWriteExt};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let listener = TcpListener::bind("127.0.0.1:9000").await?;

    loop {
        let (mut socket, _) = listener.accept().await?;

        tokio::spawn(async move {
            let mut buf = [0; 87];

            // In a loop, read data from the socket and write the data back.
            loop {
                let n = match socket.read(&mut buf).await {
                    // socket closed
                    Ok(n) if n == 0 => return,
                    Ok(n) => n,
                    Err(e) => {
                        eprintln!("failed to read from socket; err = {:?}", e);
                        return;
                    }
                };

                // Write the data back
                if let Err(e) = socket.write_all(&buf[0..n]).await {
                    eprintln!("failed to write to socket; err = {:?}", e);
                    return;
                }
            }
        });
    }
}

[Noah-Kennedy]

Can you link the source code for both the erlang implementation and the client? This will make it easier to step through the differences and speed up the server.

The examples echo server is not the most well-optimized, as it is there to be a simple example to show how to do simple things with tokio.

Without the client, I can't easily tell what exactly is being done or measured here. Typically, for something like this, I would create a set of histograms showing the various distributions of latencies with varying amounts of simultaneous clients running.

[Geo-7]

Erlang echo server is not optimized, In optimized version I would use OTP and multiple supervisor on Listen socket and multiple gen_server.

[Geo-7]

@Noah-Kennedy Thanks, I added sample codes to the first POST.

[Noah-Kennedy]

Any particular reason that the erlang client is gobbling up my entire CPU? I'm not terribly familiar with erlang.

[Noah-Kennedy]

I'm unable to get significantly different results on my machine. Rust is operating at around 1070 Mibps, erlang server goes at around 950 Mibps. The client is also using my entire CPU btw. I have a Ryzen 3900X (12-core).

[Geo-7]

Thank you for testing I think if you increase number of packets in multi_client.erl for each tester you would notice that performance will drop in tokio in this line:
Testers = [spawn_link(fun()-> tester(Self,100000,CID,9000) end) || CID <- CIDs],
change it to higher number for example 200000.
I do not know why Erlang client use a lot of memory maybe I should create a tester in C.
Even if I achieve the same performance in simple rust echo server I think I'll have problems later.
for example erlang gen_tcp has a nice feature to read a packet size at the beginning of packet ({packet, PacketType})
I try to manually implement it in rust but my code was slow :(.
Or maybe in using SQLite, I've tested ETS with 1 million record in memory and did not notice any performance Issue.
Sometimes I used BEAM hot code reloading in production and OTP Supervisors can automatically restart crashed component
I'll lose these things and I should use Kubernetes for achieving same functionality.