httpext: replace HTTP digest library

[Juneezee]

What?

This PR replaces our current HTTP digest library github.com/Soontao/goHttpDigestClient with github.com/icholy/digest.

Why?

The current HTTP Digest library,
github.com/Soontao/goHttpDigestClient, has not been updated since March 2017 and is known to be buggy.

github.com/icholy/digest is a newer library that provides an http.RoundTripper implementation, allowing digest challenges to be reused. This aligns well with our use case.

Checklist

• I have performed a self-review of my code.
• I have commented on my code, particularly in hard-to-understand areas.
• I have added tests for my changes.
• I have run linter and tests locally (make check) and all pass.

Checklist: Documentation (only for k6 maintainers and if relevant)

Please do not merge this PR until the following items are filled out.

• I have added the correct milestone and labels to the PR.
• I have updated the k6-documentation: grafana/k6-docs#PR-NUMBER
• I have updated the TypeScript definitions: grafana/k6-DefinitelyTyped#PR-NUMBER
• I have updated the release notes: link

Related PR(s)/Issue(s)

Closes #800.

[Juneezee]

I have performed a self-review of my changes.

/cc @na--, author of Issue #800

[Juneezee]

github.com/icholy/digest will handle the 401 response and cache the digest challenge for us.

https://github.com/icholy/digest/blob/v1.1.0/transport.go#L141-L188

[mstoykov]

Thank you for the PR @Juneezee 🙇

I think that we can drop the whole digest_transport.go file and not have one more transport wrapper.

I wonder if we can get some tests to see that we do not get 401 each time , as that is one of the consequences of the original issue that we want fixed.

[Juneezee]

I wonder if we can get some tests to see that we do not get 401 each time , as that is one of the consequences of the original issue that we want fixed.

@mstoykov I tried it, and it seems that the 401 Unauthorized case is unavoidable. The screenshot below shows the debugger attached to the TestRequest/Params/digest/auth/success test case, where we can see that the response status code is 401.

k6/js/modules/k6/http/request_test.go

Lines 956 to 957 in dafd7f8

	t.Run("digest", func(t *testing.T) {
	t.Run("success", func(t *testing.T) {

[kapsh]

Will it be possible to support custom digest header set by js Params.auth? I encountered service that sends challenge in non-standard x-something-authenticate and I'd like to use k6 to test it.

[mstoykov]

I can not at this point look into this more in depth, but I would argue swithcing the implementation wihtout making it make less 401 is not actually beneficial.

We have used the previous library for years and have had zero amounts of problems or issues reported AFAIK. To be honest I do not know if this is even used all that much. Swithcing the library at this point just adds potential for breakage and bugs.

I think the problem is that we do not save the transport between invocations, which will potentially make it possible to cache this.

[Juneezee]

wihtout making it make less 401

@mstoykov I wonder if that is possible? The whole "first request may return 401" is part of the specification for digest auth. Its purpose is to get the challenge from the server and use that in the second request.

From Postman docs (screenshot included too just in case the link becomes invalid in the future):

the client sends a first request to the API, and the server responds back with details. Response details include a number that can be used only once (a nonce), a realm value, and a 401 Unauthorized response. You then send back an encrypted array of data, including a username and password combined with the data received from the server in the first request.

From RFC 7616 - HTTP Digest Access Authentication, Section 3.6 Digest Operation:

the server MAY return a 401 response with a new nonce value in the WWW-Authenticate header field, causing the client to retry the request;

From RFC 7616 - HTTP Digest Access Authentication, Section 3.7 Security Protocol Negotiation:

When a server receives a request to access a resource, the server might challenge the client by responding with "401 Unauthorized" response and include one or more WWW-Authenticate header fields.

[mstoykov]

Sorry for the slow reply 🙇

It is possible - otherwise this would not be a usable case if for each request that needs to be authenticated you need to always go through the whole thing

From the third paragraph in https://datatracker.ietf.org/doc/html/rfc7616#section-3.6

The authentication session lasts until the client receives another
WWW-Authenticate challenge from any server in the protection space.
A client SHOULD remember the username, password, nonce, nonce count,
and opaque values associated with an authentication session to use to
construct the Authorization header field in future requests within
that protection space.

This is also what the whole digest transport you propose to use is doing - it is caching the header values and continues to give them until it gets requested again.

The problem is that we make new of those transport on each request. This means that we will now have to keep the transport for digest for the duration of the ... iteration or maybe for the duration of the life of the VU. I guess we have to take into account no vu connection reuse. Partly because while this maybe isn't a connection it means that you will reuse authentication between connections.

To be honest this looks like it might require some more options/parameters to be actually usefully configured.

But in all cases we will need to keep the digest transport in between invocaitons to http.get and co.

Hope this explains the problem better.