Improve read/write handling on "Invalid Credentials" error

[magreenblatt]

Access tokens might be invalidated while a build is running, resulting in "Invalid Credentials" errors for read and write operations. When a read operation fails with "Invalid Credentials" sccache still attempts to write that same object after local compilation. Additionally, sccache reports these failures as "Cache write errors" instead of as "Cache read errors".

I'd therefore like to propose the following improvements to sccache behavior in the case of "Invalid Credentials" read errors:

1. Attempt to refresh the access token (if a refresh attempt hasn't already been made)
2. If read still fails with "Invalid Credentials" then report the failure as "Cache read errors"
3. Do not attempt to write the same object after read fails with "Invalid Credentials"

Example debug logging for the read/write failure:

[0m[38;5;8m[[0m2025-06-28T17:36:18Z [0m[1m[31mERROR[0m opendal::services[0m[38;5;8m][0m service=gcs name=sccache-bucket path=7/9/9/799ca2d5ea0c4120cdb00e301153ff6b619a34c95d097fad836c15dd428b8acd: read failed Unexpected (permanent) at read => Invalid Credentials
    
    Context:
       uri: https://storage.googleapis.com/storage/v1/b/sccache-bucket/o/7%2F9%2F9%2F799ca2d5ea0c4120cdb00e301153ff6b619a34c95d097fad836c15dd428b8acd?alt=media
       response: Parts { status: 401, version: HTTP/1.1, headers: {"content-type": "text/html; charset=UTF-8", "x-guploader-uploadid": "...", "date": "Sat, 28 Jun 2025 17:36:18 GMT", "vary": "Origin", "vary": "X-Origin", "server": "UploadServer", "expires": "Sat, 28 Jun 2025 17:36:18 GMT", "cache-control": "private, max-age=0", "content-length": "19"} }
       service: gcs
       path: 7/9/9/799ca2d5ea0c4120cdb00e301153ff6b619a34c95d097fad836c15dd428b8acd
       range: 0-

[0m[38;5;8m[[0m2025-06-28T17:36:19Z [0m[1m[31mERROR[0m opendal::services[0m[38;5;8m][0m service=gcs name=sccache-bucket path=7/9/9/799ca2d5ea0c4120cdb00e301153ff6b619a34c95d097fad836c15dd428b8acd written=8290: Writer::close failed Unexpected (permanent) at Writer::close => GcsErrorResponse { error: GcsError { code: 401, message: "Invalid Credentials", errors: [GcsErrorDetail { domain: "global", location: "Authorization", location_type: "header", message: "Invalid Credentials", reason: "authError" }] } }
    
    Context:
       uri: https://storage.googleapis.com/upload/storage/v1/b/sccache-bucket/o?uploadType=media&name=7/9/9/799ca2d5ea0c4120cdb00e301153ff6b619a34c95d097fad836c15dd428b8acd
       response: Parts { status: 401, version: HTTP/1.1, headers: {"x-guploader-uploadid": "...", "date": "Sat, 28 Jun 2025 17:36:19 GMT", "vary": "Origin", "vary": "X-Origin", "content-type": "text/html; charset=UTF-8", "cache-control": "no-cache, no-store, max-age=0, must-revalidate", "expires": "Mon, 01 Jan 1990 00:00:00 GMT", "pragma": "no-cache", "content-length": "285", "server": "UploadServer"} }
       service: gcs
       path: 7/9/9/799ca2d5ea0c4120cdb00e301153ff6b619a34c95d097fad836c15dd428b8acd
       written: 8290

Current behavior was tested with sccache main branch.

[magreenblatt]

Attempt to refresh the access token (if a refresh attempt hasn't already been made)

All read/write requests will continue to fail until the token is refreshed. If we don't refresh the token immediately then we might instead switch to local compilation until the token is refreshed automatically (e.g. near expiration).

[magreenblatt]

This particular "Invalid Credentials" error was caused by gcs.rs code assuming 1 hour expiration for tokens. We should fix that as well.

[magreenblatt]

This particular "Invalid Credentials" error was caused by gcs.rs code assuming 1 hour expiration for tokens. We should fix that as well.

Fixed in #2416