ASP.NET Core Identity bad performance on Kubernetes

[GiuseppePatane]

I'm running load tests on an aspnet core 3.0 API. The project is basically the default template with aspnet core identity. So I wasn't expecting these bad performances on /api/Account/Login:

 public class LogInController : ControllerBase
    {
        private readonly SignInManager<ApplicationUser> _signInManager;
        public LogInController(SignInManager<ApplicationUser> signInManager)
        {
            _signInManager = signInManager;
        }
        [HttpPost]
        [AllowAnonymous]
        [Route("/api/Account/Login")]
        public async Task<IActionResult> Login([FromBody] LoginViewModel model)
        {
            var result = await _signInManager.PasswordSignInAsync(model.UserName, model.Password, model.RememberMe, false);
            if (!result.Succeeded)
            {
                return BadRequest(result);
            }
            return Ok();
        }
}

I'm running the API on :

• 1 Pod on kubernetes(GCP)
• PostgreeSQL on GCP (vCPU: 4 | Ram: 15 GB | Hard Disk SSD:15 GB)
• AspentUser count: 3.387.988

The load test result:

am I missing some kestrel or EF core configuration? how many concurrent client Aspnet core Identity can handle? I didn't find any benchmark about this

Thanks in advance

[SteveSandersonMS]

Do the numbers look substantially different if you have fewer users in the database? If the performance gradually gets worse for larger user counts (you mention you have > 3 million) then maybe there's a database configuration issue, for example missing indexes, or some single badly-performing query.

@HaoK Do these numbers make you suspect there's some kind of misconfiguration?

[GiuseppePatane]

thanks for the answer, I don't think it's a database problem, it's default DB of Aspnet core Identity that has indexes on the fields most used by queries.
the ef core migration script:

 	migrationBuilder.CreateTable(
                name: "AspNetUsers",
                columns: table => new
                {
                    Id = table.Column<string>(nullable: false),
                    UserName = table.Column<string>(maxLength: 256, nullable: true),
                    NormalizedUserName = table.Column<string>(maxLength: 256, nullable: true),
                    Email = table.Column<string>(maxLength: 256, nullable: true),
                    NormalizedEmail = table.Column<string>(maxLength: 256, nullable: true),
                    EmailConfirmed = table.Column<bool>(nullable: false),
                    PasswordHash = table.Column<string>(nullable: true),
                    SecurityStamp = table.Column<string>(nullable: true),
                    ConcurrencyStamp = table.Column<string>(nullable: true),
                    PhoneNumber = table.Column<string>(nullable: true),
                    PhoneNumberConfirmed = table.Column<bool>(nullable: false),
                    TwoFactorEnabled = table.Column<bool>(nullable: false),
                    LockoutEnd = table.Column<DateTimeOffset>(nullable: true),
                    LockoutEnabled = table.Column<bool>(nullable: false),
                    AccessFailedCount = table.Column<int>(nullable: false)
                },
                constraints: table =>
                {
                    table.PrimaryKey("PK_AspNetUsers", x => x.Id);
                });
 	migrationBuilder.CreateIndex(
                name: "EmailIndex",
                table: "AspNetUsers",
                column: "NormalizedEmail");

          migrationBuilder.CreateIndex(
                name: "UserNameIndex",
                table: "AspNetUsers",
                column: "NormalizedUserName",
                unique: true);

Looking at the Postgres machine logs, the query response times are under 500ms but the API gets to have response times even over 20 seconds.
Running a test with 300 concurrent clients with the strace command active there seems to be a mutex (or a lock of some kind) where processes wait for 83% of the execution time (19 seconds)

root 1 0 9 11:33 ? 00:00:30 dotnet AccountMicroService.dll

time	seconds	usecs/call	calls	errors syscall
83.09	19.102581	347	54909	7687 futex
10.13	2.329314	1188	1960	epoll_wait
2.17	0.498592	12	38859	sched_yield
1.88	0.431286	27	15957	write
1.16	0.267053	33381	8	3 restart_syscall
0.40	0.092198	59	1550	7 sendmsg
0.25	0.057737	21	2729	1152 recvmsg
0.20	0.045349	19	2331	read
0.16	0.036459	27	1336	mprotect
0.14	0.032481	17	1816	getpid
0.06	0.012840	27	473	setsockopt
0.06	0.012673	33	374	epoll_ctl
0.05	0.011986	46	260	16 accept4
0.05	0.010666	14	750	fcntl
0.04	0.010315	21	488	getsockname
0.04	0.010116	77	130	130 connect
0.03	0.006112	25	244	close
0.02	0.005557	47	116	1 shutdown
0.02	0.004015	30	130	getsockopt
0.02	0.003771	23	161	socket
0.01	0.003364	38	87	25 openat
0.01	0.001268	84	15	mmap
0.00	0.001072	14	74	getrusage
0.00	0.000382	5	70	flock
0.00	0.000372	31	12	5 stat
0.00	0.000355	4	73	fstat
0.00	0.000349	31	11	5 lstat
0.00	0.000319	4	65	lseek
0.00	0.000196	4	42	membarrier
0.00	0.000099	33	3	sched_setaffinity
0.00	0.000057	19	3	clone
0.00	0.000048	24	2	getdents64
0.00	0.000046	6	7	sigaltstack
0.00	0.000043	6	7	sched_getaffinity
0.00	0.000041	13	3	set_robust_list
0.00	0.000032	4	7	munmap
0.00	0.000026	13	2	fadvise64
0.00	0.000016	8	2	tgkill
0.00	0.000013	6	2	rt_sigreturn
0.00	0.000013	13	1	madvise
0.00	0.000003	1	2	poll
0.00	0.000002	0	3	gettid
0.00	0.000002	1	2	pipe2
0.00	0.000000	0	2	pread64
100.00	22.989219	125078	9031	Total

Could it be something related to the maximum number of threads or socket connections?
By scaling from 1 to 3 pods the response time returns to be under one second. But it seems strange to me that the starting template of aspnet identity can handle so few parallel requests