az connectedk8s connect does not support kubeconfig that requires extensions #8788
Description
Describe the bug
Gardener exposes a kubeconfig like this:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: removed
extensions:
- extension:
gardenClusterIdentity: leafcloud-production
shootRef:
name: genai
namespace: garden-gnh8hcnimj
name: client.authentication.k8s.io/exec
server: removed
name: garden-gnh8hcnimj--genai-external
contexts:
- context:
cluster: garden-gnh8hcnimj--genai-external
namespace: default
user: garden-gnh8hcnimj--genai
name: garden-gnh8hcnimj--genai-external
current-context: garden-gnh8hcnimj--genai-external
kind: Config
preferences: {}
users:
- name: garden-gnh8hcnimj--genai
user:
exec:
apiVersion: client.authentication.k8s.io/v1
args:
- get-client-certificate
command: kubectl-gardenlogin
env: null
installHint: Follow the instructions on https://github.com/gardener/gardenlogin#installation
to install gardenlogin
interactiveMode: IfAvailable
provideClusterInfo: trueFor this, the kubectl-gardenlogin get-client-certificate command needs to be provided with the extension attributes in order to correctly get the token.
The current version of the kubernetes package does not support 1) the provideClusterInfo properties, and 2) the additional support for the extension properties.
The following fixes have already been implemented on the Kubernetes client side:
One more fix awaiting there to support the extension properties:
Upgrading the Kubernetes package to the latest (kubernetes==32.0.1) and manually patching the run command of the Exec Provider like this fixes the issue (in custom.py):
def run(self, previous_response=None):
import sys
import subprocess
from kubernetes.config.config_exception import ConfigException
is_interactive = hasattr(sys.stdout, 'isatty') and sys.stdout.isatty()
kubernetes_exec_info = {
'apiVersion': self.api_version,
'kind': 'ExecCredential',
'spec': {
'interactive': is_interactive
}
}
if previous_response:
kubernetes_exec_info['spec']['response'] = previous_response
if self.cluster:
kubernetes_exec_info['spec']['cluster'] = self.cluster.value
for extension in self.cluster.value["extensions"]:
if extension["name"] == "client.authentication.k8s.io/exec":
kubernetes_exec_info["spec"]["cluster"]["config"] = extension["extension"]
break
self.env['KUBERNETES_EXEC_INFO'] = json.dumps(kubernetes_exec_info)
process = subprocess.Popen(
self.args,
stdout=subprocess.PIPE,
stderr=sys.stderr if is_interactive else subprocess.PIPE,
stdin=sys.stdin if is_interactive else None,
cwd=self.cwd,
env=self.env,
universal_newlines=True,
shell=self.shell)
(stdout, stderr) = process.communicate()
exit_code = process.wait()
if exit_code != 0:
msg = 'exec: process returned %d' % exit_code
stderr = stderr.strip()
if stderr:
msg += '. %s' % stderr
raise ConfigException(msg)
try:
data = json.loads(stdout)
except ValueError as de:
raise ConfigException(
'exec: failed to decode process output: %s' % de)
for key in ('apiVersion', 'kind', 'status'):
if key not in data:
raise ConfigException(
'exec: malformed response. missing key \'%s\'' % key)
if data['apiVersion'] != self.api_version:
raise ConfigException(
'exec: plugin api version %s does not match %s' %
(data['apiVersion'], self.api_version))
return data['status']
from kubernetes.config.exec_provider import ExecProvider
ExecProvider.run = runApplying this fix will significantly help other users connecting their Gardener clusters (for instance, on leaf.cloud) as Arc-Enabled Kubernetes Clusters
Related command
az connectedk8s connect
Errors
'ERROR:root:exec: process returned 1. Error: name must be specified. Hint: update kubectl in case you are using a version older than v1.20.0
Usage:
gardenlogin get-client-certificate [flags
]
Flags:
...
Issue script & Debug output
Relevant debug output:
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /metadata/endpoints?api-version=2022-09-01 HTTP/1.1" 200 1795
root: exec: process returned 1. Error: failed to complete command options: ExecCredential does not contain cluster information
Usage:
gardenlogin get-client-certificate [flags]
Flags:
--access-level string Defines the access level of the credential returned by the plugin. Can be "auto", "admin", or "viewer".
"auto" - Attempts to obtain admin-level credentials. If unsuccessful, it defaults to viewer-level credentials.
"admin" - Returns a credential with cluster-admin privileges.
"viewer" - Returns a credential with read-only access to non-encrypted API resources. (default "auto")
--certificate-cache-dir string Directory of the certificate cache (default "/Users/pnlmw585/.kube/cache/gardenlogin")
--expiration-seconds int Validity duration of the requested credential (default 900)
--garden-cluster-identity string Cluster identity of the garden cluster
-h, --help help for get-client-certificate
--name string Name of the shoot cluster
--namespace string Namespace of the shoot cluster
Global Flags:
--add-dir-header If true, adds the file directory to the header of the log messages
--alsologtostderr log to standard error as well as files (no effect when -logtostderr=true)
--config string config file (default is ~/.garden/gardenlogin.yaml)
--log-backtrace-at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log-dir string If non-empty, write log files in this directory (no effect when -logtostderr=true)
--log-file string If non-empty, use this log file (no effect when -logtostderr=true)
--log-file-max-size uint Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--one-output If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)
--skip-headers If true, avoid header prefixes in the log messages
--skip-log-headers If true, avoid headers when opening log files (no effect when -logtostderr=true)
--stderrthreshold severity logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=false) (default 2)
-v, --v Level number for the log level verbosity
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
Error: failed to complete command options: ExecCredential does not contain cluster information
Step: 2025-05-24T06-38-31Z: Checking Connectivity to Cluster
kubernetes.client.rest: response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
Expected behavior
The azconnectedk8s using both the Cluster Information and the extension attributes
Environment Summary
az --version
azure-cli 2.71.0 *
core 2.71.0 *
telemetry 1.1.0
Extensions:
connectedk8s 1.10.7
Dependencies:
msal 1.31.2b1
azure-mgmt-resource 23.1.1
Python location '/opt/homebrew/Cellar/azure-cli/2.71.0/libexec/bin/python'
Config directory '/Users/mats/.azure'
Extensions directory '/Users/mats/.azure/cliextensions'
Python (Darwin) 3.12.9 (main, Feb 4 2025, 14:38:38) [Clang 16.0.0 (clang-1600.0.26.6)]
Legal docs and information: aka.ms/AzureCliLegal
You have 2 update(s) available. Consider updating your CLI installation with 'az upgrade'
Additional context
No response