-
Notifications
You must be signed in to change notification settings - Fork 50
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Hi! I was looking to set up a funcx endpoint, where the funcx-endpoint start X
command would be run by the flux broker, thus giving it access to an entire cluster in Kubernetes. This is what I was hoping to use for my config:
from funcx_endpoint.endpoint.utils.config import Config
from funcx_endpoint.executors import FluxExecutor
import os
# Retrieve settings from the environment
working_dir = os.environ.get('FUNCX_WORKDIR')
# The executor will be run under the broker, with access to all its resources
executor = FluxExecutor(working_dir=working_dir)
executor.launch_cmd="{flux} submit {python} {manager} {protocol} {hostname} {port}"
config = Config(executors=[executor])
# For now, visible_to must be a list of URNs for globus auth users or groups, e.g.:
# urn:globus:auth:identity:{user_uuid}
# urn:globus:groups:id:{group_uuid}
meta = {
"name": "default",
"description": "The Flux Operator FuncX Endpoint",
"organization": "Lawrence Livermore National Laboratory",
"department": "The Harry Potter Closet",
"public": False,
"visible_to": [],
}
But I was looking here: https://github.com/funcx-faas/funcX/blob/main/funcx_endpoint/funcx_endpoint/executors/__init__.py and I don't see it as an option. Is this something we could potentially add?
Note that it's already an executor in parsl (alongside the high throughput - why I am hoping we can add it here!)
To give some context, the endpoint would already be started in a flux instance with an allocated set of resources, so we would just need to receive a task and do flux submit
via the Python sdk.
Thank you!
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request