Skip to content

Commit 6bb87ca

Browse files
jeremymanningclaude
andcommitted
Fix CloudProviderManager import issue by resolving module/package naming conflict
- Renamed cloud_providers.py to cloud_provider_manager.py to avoid conflict with cloud_providers/ package - Updated import in executor.py to use the renamed module - Fixed all 3 failing tests related to cloud provider auto-configuration - All 292 tests now pass with 100% success rate - No linting issues The issue was that Python's import system was importing the cloud_providers package (directory) instead of the cloud_providers.py module when both had the same name. The package takes precedence in Python's import resolution. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 09fa4e1 commit 6bb87ca

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

52 files changed

+1785
-304
lines changed

clustrix.egg-info/PKG-INFO

Lines changed: 195 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -99,11 +99,14 @@ Clustrix is a Python package that enables seamless distributed computing on clus
9999

100100
- **Simple Decorator Interface**: Just add `@cluster` to any function
101101
- **Interactive Jupyter Widget**: `%%clusterfy` magic command with GUI configuration manager
102-
- **Multiple Cluster Support**: SLURM, PBS, SGE, Kubernetes, and SSH
102+
- **Multiple Cluster Support**: SLURM, PBS, SGE, Kubernetes, SSH, and major cloud providers
103+
- **Cloud Provider Integration**: Native support for AWS (EC2/EKS), Google Cloud (GCE/GKE), Azure (VM/AKS), Lambda Cloud, and HuggingFace Spaces
103104
- **Automatic Dependency Management**: Captures and replicates your exact Python environment
104105
- **Native Cost Monitoring**: Built-in cost tracking for all major cloud providers
106+
- **Kubernetes Support**: Deploy to EKS, GKE, AKS, or any Kubernetes cluster
105107
- **Loop Parallelization**: Automatically distributes loops across cluster nodes
106108
- **Flexible Configuration**: Easy setup with config files, environment variables, or interactive widget
109+
- **Dynamic Instance Selection**: Auto-populated dropdowns for cloud instance types and regions
107110
- **Error Handling**: Comprehensive error reporting and job monitoring
108111

109112
## Quick Start
@@ -167,13 +170,88 @@ import clustrix # Auto-loads the magic command
167170
# - Save/load configurations to files
168171
```
169172

173+
#### Interactive Configuration Widget
174+
175+
The Clustrix widget provides a comprehensive GUI for managing cluster configurations directly in Jupyter notebooks. Here's what you'll see when you use the `%%clusterfy` magic command:
176+
177+
##### Default View
178+
When the widget first loads, it displays the "Local Single-core" configuration for quick testing:
179+
180+
![Default widget view showing Local Single-core configuration](https://github.com/user-attachments/assets/44baefd7-9dd7-452d-bc43-ef53136d13a4)
181+
182+
##### Configuration Dropdown
183+
The dropdown menu includes pre-built templates for various cluster types and cloud providers:
184+
185+
![Configuration dropdown showing available templates](https://github.com/user-attachments/assets/744e2eef-e03e-46bd-9bb1-e0ed2c11cf45)
186+
170187
The widget includes pre-built templates for:
171-
- **Local Development**: Run jobs on your local machine
172-
- **AWS GPU Instances**: p3.2xlarge, p3.8xlarge templates
173-
- **Google Cloud**: CPU and GPU instance configurations
174-
- **Azure**: Virtual machine templates with GPU support
175-
- **SLURM HPC**: University cluster configurations
176-
- **Kubernetes**: Container-based execution
188+
- **Local Development**:
189+
- Local Single-core: Run jobs on one CPU core
190+
- Local Multi-core: Utilize all available CPU cores
191+
- **HPC Clusters**:
192+
- SLURM: University and research cluster support
193+
- PBS/SGE: Traditional HPC schedulers
194+
- **Cloud Providers**:
195+
- **AWS**: EC2 instances and EKS Kubernetes clusters
196+
- **Google Cloud**: Compute Engine VMs and GKE clusters
197+
- **Azure**: Virtual Machines and AKS Kubernetes clusters
198+
- **Lambda Cloud**: GPU-optimized instances for ML/AI
199+
- **HuggingFace Spaces**: Deploy to HF Spaces infrastructure
200+
- **Kubernetes**: Native container orchestration support
201+
202+
##### Cluster Configuration Examples
203+
204+
###### SLURM Cluster Configuration
205+
For traditional HPC clusters, the widget provides all essential fields:
206+
207+
![SLURM configuration with basic settings](https://github.com/user-attachments/assets/994a803d-485e-4d14-bf93-2f0223066510)
208+
209+
The advanced settings accordion reveals additional options:
210+
211+
![SLURM advanced configuration options](https://github.com/user-attachments/assets/0930eee0-ee52-4d53-9165-81e907c7d962)
212+
213+
Advanced options include:
214+
- Module loads (e.g., `python/3.9`, `cuda/11.2`)
215+
- Environment variables
216+
- Pre-execution commands
217+
- Custom SSH key paths
218+
- Cost monitoring toggles
219+
220+
###### Cloud Provider Configuration
221+
222+
**Google Cloud Platform:**
223+
When selecting a cloud provider, only relevant fields are displayed:
224+
225+
![GCP VM configuration interface](https://github.com/user-attachments/assets/53acb782-65cd-4b65-93a0-98286144f223)
226+
227+
**Lambda Cloud GPU Instances:**
228+
The widget dynamically populates instance type dropdowns based on the selected provider:
229+
230+
![Lambda Cloud with GPU instance dropdown](https://github.com/user-attachments/assets/4bd0e176-d5f3-430e-86af-ddbfff827d77)
231+
232+
##### Key Widget Features
233+
234+
1. **Dynamic Field Visibility**: Only shows fields relevant to the selected cluster type
235+
2. **Provider-Specific Options**:
236+
- AWS: Region selection, instance types, EKS cluster options
237+
- Azure: Resource groups, VM sizes, AKS configuration
238+
- GCP: Projects, zones, machine types, GKE options
239+
- Lambda Cloud: GPU instance selection with live pricing
240+
3. **Input Validation**: Real-time validation for hostnames, IP addresses, and configuration values
241+
4. **Tooltips**: Hover over any field label to see detailed help text
242+
5. **Configuration Management**:
243+
- Save configurations to YAML/JSON files
244+
- Load existing configurations
245+
- Test configurations before applying
246+
- Add/delete custom configurations
247+
248+
##### Using the Widget
249+
250+
1. **Select a Configuration**: Choose from the dropdown or create a new one
251+
2. **Edit Settings**: Modify cluster connection details and resource requirements
252+
3. **Advanced Options**: Expand the accordion for environment setup and additional settings
253+
4. **Apply Configuration**: Click "Apply Configuration" to use these settings for subsequent `@cluster` decorated functions
254+
5. **Save for Later**: Use "Save Configuration" to persist settings to a file
177255

178256
### Configuration File
179257

@@ -282,6 +360,116 @@ clustrix.configure(cluster_type='kubernetes')
282360
clustrix.configure(cluster_type='ssh', cluster_host='server.example.com')
283361
```
284362

363+
### Cloud Provider Integration
364+
365+
Clustrix provides native integration with major cloud providers for both VM and Kubernetes deployments:
366+
367+
#### AWS Integration
368+
369+
```python
370+
# Configure AWS credentials and region
371+
clustrix.configure(
372+
cluster_type='aws',
373+
access_key_id='YOUR_ACCESS_KEY',
374+
secret_access_key='YOUR_SECRET_KEY',
375+
region='us-west-2'
376+
)
377+
378+
# Run on EC2 instance
379+
@cluster(provider='aws', instance_type='p3.2xlarge', cores=8, memory='61GB')
380+
def train_on_aws():
381+
# GPU-accelerated training on AWS
382+
pass
383+
384+
# Run on EKS Kubernetes cluster
385+
@cluster(provider='aws', cluster_type='kubernetes', cluster_name='my-eks-cluster')
386+
def distributed_training():
387+
# Runs on Amazon EKS
388+
pass
389+
```
390+
391+
#### Google Cloud Integration
392+
393+
```python
394+
# Configure GCP with service account
395+
clustrix.configure(
396+
cluster_type='gcp',
397+
project_id='your-project-id',
398+
service_account_key='path/to/service-account-key.json',
399+
region='us-central1'
400+
)
401+
402+
# Run on Compute Engine
403+
@cluster(provider='gcp', machine_type='n1-highmem-8', cores=8, memory='52GB')
404+
def analyze_data():
405+
# High-memory computation on GCP
406+
pass
407+
408+
# Run on GKE cluster
409+
@cluster(provider='gcp', cluster_type='kubernetes', cluster_name='my-gke-cluster')
410+
def kubernetes_job():
411+
# Runs on Google Kubernetes Engine
412+
pass
413+
```
414+
415+
#### Azure Integration
416+
417+
```python
418+
# Configure Azure with service principal
419+
clustrix.configure(
420+
cluster_type='azure',
421+
subscription_id='YOUR_SUBSCRIPTION_ID',
422+
client_id='YOUR_CLIENT_ID',
423+
client_secret='YOUR_CLIENT_SECRET',
424+
tenant_id='YOUR_TENANT_ID',
425+
region='eastus'
426+
)
427+
428+
# Run on Azure VM
429+
@cluster(provider='azure', vm_size='Standard_NC6', cores=6, memory='56GB')
430+
def gpu_workload():
431+
# GPU computation on Azure
432+
pass
433+
434+
# Run on AKS cluster
435+
@cluster(provider='azure', cluster_type='kubernetes', cluster_name='my-aks-cluster')
436+
def container_workload():
437+
# Runs on Azure Kubernetes Service
438+
pass
439+
```
440+
441+
#### Lambda Cloud Integration
442+
443+
```python
444+
# Configure Lambda Cloud for GPU workloads
445+
clustrix.configure(
446+
cluster_type='lambda_cloud',
447+
api_key='YOUR_LAMBDA_API_KEY'
448+
)
449+
450+
# Run on Lambda GPU instance
451+
@cluster(provider='lambda_cloud', instance_type='gpu_1x_a100', cores=30, memory='200GB')
452+
def train_large_model():
453+
# A100 GPU training on Lambda Cloud
454+
pass
455+
```
456+
457+
#### HuggingFace Spaces Integration
458+
459+
```python
460+
# Configure HuggingFace Spaces
461+
clustrix.configure(
462+
cluster_type='huggingface_spaces',
463+
token='YOUR_HF_TOKEN'
464+
)
465+
466+
# Deploy to HuggingFace Spaces
467+
@cluster(provider='huggingface_spaces', space_hardware='gpu-t4-medium')
468+
def inference_endpoint():
469+
# Runs on HuggingFace infrastructure
470+
pass
471+
```
472+
285473
## Command Line Interface
286474

287475
```bash

clustrix.egg-info/SOURCES.txt

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ LICENSE
22
README.md
33
setup.py
44
clustrix/__init__.py
5+
clustrix/async_executor_simple.py
56
clustrix/cli.py
67
clustrix/cloud_providers.py
78
clustrix/config.py
@@ -20,13 +21,21 @@ clustrix.egg-info/entry_points.txt
2021
clustrix.egg-info/not-zip-safe
2122
clustrix.egg-info/requires.txt
2223
clustrix.egg-info/top_level.txt
24+
clustrix/cloud_providers/__init__.py
25+
clustrix/cloud_providers/aws.py
26+
clustrix/cloud_providers/azure.py
27+
clustrix/cloud_providers/base.py
28+
clustrix/cloud_providers/gcp.py
29+
clustrix/cloud_providers/huggingface_spaces.py
30+
clustrix/cloud_providers/lambda_cloud.py
2331
clustrix/cost_providers/__init__.py
2432
clustrix/cost_providers/aws.py
2533
clustrix/cost_providers/azure.py
2634
clustrix/cost_providers/gcp.py
2735
clustrix/cost_providers/lambda_cloud.py
2836
tests/__init__.py
2937
tests/conftest.py
38+
tests/test_async_execution.py
3039
tests/test_cli.py
3140
tests/test_cloud_providers.py
3241
tests/test_config.py
File renamed without changes.

clustrix/cloud_providers/__init__.py

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -34,10 +34,4 @@
3434
except ImportError:
3535
pass
3636

37-
# Import CloudProviderManager from the old cloud_providers module
38-
try:
39-
from ..cloud_providers import CloudProviderManager # noqa: F401
40-
except ImportError:
41-
pass
42-
43-
__all__ = ["PROVIDERS", "CloudProviderManager"]
37+
__all__ = ["PROVIDERS"]

clustrix/executor.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,10 @@ def _setup_kubernetes(self):
6666
and self.config.cluster_type == "kubernetes"
6767
):
6868
try:
69-
from . import cloud_providers as cloud_providers_module
69+
# Import CloudProviderManager from the renamed module
70+
from .cloud_provider_manager import CloudProviderManager
7071

71-
cloud_manager = cloud_providers_module.CloudProviderManager(self.config)
72+
cloud_manager = CloudProviderManager(self.config)
7273
result = cloud_manager.auto_configure()
7374

7475
if result.get("auto_configured"):
2.75 KB
Binary file not shown.
1.88 KB
Binary file not shown.
7.32 KB
Binary file not shown.
49.7 KB
Binary file not shown.

docs/build/html/_modules/clustrix/config.html

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,7 @@ <h1>Source code for clustrix.config</h1><div class="highlight"><pre>
219219
<span class="n">cleanup_on_success</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span>
220220
<span class="n">prefer_local_parallel</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span>
221221
<span class="n">local_parallel_threshold</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">1000</span> <span class="c1"># Use local if iterations &lt; threshold</span>
222+
<span class="n">async_submit</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span> <span class="c1"># Use asynchronous job submission</span>
222223

223224
<span class="c1"># Advanced settings</span>
224225
<span class="n">environment_variables</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]]</span> <span class="o">=</span> <span class="kc">None</span>
@@ -395,7 +396,7 @@ <h1>Source code for clustrix.config</h1><div class="highlight"><pre>
395396
&copy; Copyright 2025, Contextual Dynamics Laboratory
396397
</div>
397398
<div class="text-center text-white-50 font-italic mt-3">
398-
<small>Last updated: Jun 27, 2025</small>
399+
<small>Last updated: Jun 28, 2025</small>
399400
<small>
400401
Created using
401402
<a class="text-white" href="https://github.com/wagtail/sphinx_wagtail_theme" rel="nofollow" target="_blank">

0 commit comments

Comments
 (0)