Skip to content

FluxExecutor exposed as endpoint executor? #1077

@vsoch

Description

@vsoch

Hi! I was looking to set up a funcx endpoint, where the funcx-endpoint start X command would be run by the flux broker, thus giving it access to an entire cluster in Kubernetes. This is what I was hoping to use for my config:

from funcx_endpoint.endpoint.utils.config import Config
from funcx_endpoint.executors import FluxExecutor
import os

# Retrieve settings from the environment
working_dir = os.environ.get('FUNCX_WORKDIR')

# The executor will be run under the broker, with access to all its resources
executor = FluxExecutor(working_dir=working_dir)
executor.launch_cmd="{flux} submit {python} {manager} {protocol} {hostname} {port}"
config = Config(executors=[executor])

# For now, visible_to must be a list of URNs for globus auth users or groups, e.g.:
# urn:globus:auth:identity:{user_uuid}
# urn:globus:groups:id:{group_uuid}
meta = {
    "name": "default",
    "description": "The Flux Operator FuncX Endpoint",
    "organization": "Lawrence Livermore National Laboratory",
    "department": "The Harry Potter Closet",
    "public": False,
    "visible_to": [],
}

But I was looking here: https://github.com/funcx-faas/funcX/blob/main/funcx_endpoint/funcx_endpoint/executors/__init__.py and I don't see it as an option. Is this something we could potentially add?

Note that it's already an executor in parsl (alongside the high throughput - why I am hoping we can add it here!)

https://github.com/Parsl/parsl/blob/7a235a4f5e285bc8deb7b57726f01cea001f4c89/parsl/executors/flux/executor.py#L1

To give some context, the endpoint would already be started in a flux instance with an allocated set of resources, so we would just need to receive a task and do flux submit via the Python sdk.

Thank you!

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions