Skip to content

Commit 09fa4e1

Browse files
jeremymanningclaude
andcommitted
Update documentation with widget screenshots and cloud provider examples (Issue #48)
- Add comprehensive widget documentation section with all 6 screenshots - Document default view, configuration dropdown, and examples - Add SLURM cluster configuration screenshots (basic and advanced) - Add cloud provider screenshots (GCP VM and Lambda Cloud GPU) - Document key widget features and dynamic field visibility - Add detailed cloud provider integration examples for all providers - Include code examples for AWS (EC2/EKS), GCP (GCE/GKE), Azure (VM/AKS) - Add Lambda Cloud and HuggingFace Spaces integration examples - Update features list to highlight cloud provider support - Fix linting issues in test_notebook_magic.py (E712 comparisons) - Fix CloudProviderManager import issue in executor.py Screenshots included: 1. Default widget view (Local Single-core) 2. Configuration dropdown with all templates 3. SLURM basic configuration 4. SLURM advanced settings 5. GCP VM configuration 6. Lambda Cloud GPU instance selection All cloud providers documented with authentication and usage examples. Tests: 289/292 passing (3 cloud auto-config tests need mock adjustment) Linting: Full compliance achieved Documentation: Builds successfully with no errors 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 0271a12 commit 09fa4e1

File tree

3 files changed

+199
-14
lines changed

3 files changed

+199
-14
lines changed

README.md

Lines changed: 195 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,14 @@ Clustrix is a Python package that enables seamless distributed computing on clus
1212

1313
- **Simple Decorator Interface**: Just add `@cluster` to any function
1414
- **Interactive Jupyter Widget**: `%%clusterfy` magic command with GUI configuration manager
15-
- **Multiple Cluster Support**: SLURM, PBS, SGE, Kubernetes, and SSH
15+
- **Multiple Cluster Support**: SLURM, PBS, SGE, Kubernetes, SSH, and major cloud providers
16+
- **Cloud Provider Integration**: Native support for AWS (EC2/EKS), Google Cloud (GCE/GKE), Azure (VM/AKS), Lambda Cloud, and HuggingFace Spaces
1617
- **Automatic Dependency Management**: Captures and replicates your exact Python environment
1718
- **Native Cost Monitoring**: Built-in cost tracking for all major cloud providers
19+
- **Kubernetes Support**: Deploy to EKS, GKE, AKS, or any Kubernetes cluster
1820
- **Loop Parallelization**: Automatically distributes loops across cluster nodes
1921
- **Flexible Configuration**: Easy setup with config files, environment variables, or interactive widget
22+
- **Dynamic Instance Selection**: Auto-populated dropdowns for cloud instance types and regions
2023
- **Error Handling**: Comprehensive error reporting and job monitoring
2124

2225
## Quick Start
@@ -80,13 +83,88 @@ import clustrix # Auto-loads the magic command
8083
# - Save/load configurations to files
8184
```
8285

86+
#### Interactive Configuration Widget
87+
88+
The Clustrix widget provides a comprehensive GUI for managing cluster configurations directly in Jupyter notebooks. Here's what you'll see when you use the `%%clusterfy` magic command:
89+
90+
##### Default View
91+
When the widget first loads, it displays the "Local Single-core" configuration for quick testing:
92+
93+
![Default widget view showing Local Single-core configuration](https://github.com/user-attachments/assets/44baefd7-9dd7-452d-bc43-ef53136d13a4)
94+
95+
##### Configuration Dropdown
96+
The dropdown menu includes pre-built templates for various cluster types and cloud providers:
97+
98+
![Configuration dropdown showing available templates](https://github.com/user-attachments/assets/744e2eef-e03e-46bd-9bb1-e0ed2c11cf45)
99+
83100
The widget includes pre-built templates for:
84-
- **Local Development**: Run jobs on your local machine
85-
- **AWS GPU Instances**: p3.2xlarge, p3.8xlarge templates
86-
- **Google Cloud**: CPU and GPU instance configurations
87-
- **Azure**: Virtual machine templates with GPU support
88-
- **SLURM HPC**: University cluster configurations
89-
- **Kubernetes**: Container-based execution
101+
- **Local Development**:
102+
- Local Single-core: Run jobs on one CPU core
103+
- Local Multi-core: Utilize all available CPU cores
104+
- **HPC Clusters**:
105+
- SLURM: University and research cluster support
106+
- PBS/SGE: Traditional HPC schedulers
107+
- **Cloud Providers**:
108+
- **AWS**: EC2 instances and EKS Kubernetes clusters
109+
- **Google Cloud**: Compute Engine VMs and GKE clusters
110+
- **Azure**: Virtual Machines and AKS Kubernetes clusters
111+
- **Lambda Cloud**: GPU-optimized instances for ML/AI
112+
- **HuggingFace Spaces**: Deploy to HF Spaces infrastructure
113+
- **Kubernetes**: Native container orchestration support
114+
115+
##### Cluster Configuration Examples
116+
117+
###### SLURM Cluster Configuration
118+
For traditional HPC clusters, the widget provides all essential fields:
119+
120+
![SLURM configuration with basic settings](https://github.com/user-attachments/assets/994a803d-485e-4d14-bf93-2f0223066510)
121+
122+
The advanced settings accordion reveals additional options:
123+
124+
![SLURM advanced configuration options](https://github.com/user-attachments/assets/0930eee0-ee52-4d53-9165-81e907c7d962)
125+
126+
Advanced options include:
127+
- Module loads (e.g., `python/3.9`, `cuda/11.2`)
128+
- Environment variables
129+
- Pre-execution commands
130+
- Custom SSH key paths
131+
- Cost monitoring toggles
132+
133+
###### Cloud Provider Configuration
134+
135+
**Google Cloud Platform:**
136+
When selecting a cloud provider, only relevant fields are displayed:
137+
138+
![GCP VM configuration interface](https://github.com/user-attachments/assets/53acb782-65cd-4b65-93a0-98286144f223)
139+
140+
**Lambda Cloud GPU Instances:**
141+
The widget dynamically populates instance type dropdowns based on the selected provider:
142+
143+
![Lambda Cloud with GPU instance dropdown](https://github.com/user-attachments/assets/4bd0e176-d5f3-430e-86af-ddbfff827d77)
144+
145+
##### Key Widget Features
146+
147+
1. **Dynamic Field Visibility**: Only shows fields relevant to the selected cluster type
148+
2. **Provider-Specific Options**:
149+
- AWS: Region selection, instance types, EKS cluster options
150+
- Azure: Resource groups, VM sizes, AKS configuration
151+
- GCP: Projects, zones, machine types, GKE options
152+
- Lambda Cloud: GPU instance selection with live pricing
153+
3. **Input Validation**: Real-time validation for hostnames, IP addresses, and configuration values
154+
4. **Tooltips**: Hover over any field label to see detailed help text
155+
5. **Configuration Management**:
156+
- Save configurations to YAML/JSON files
157+
- Load existing configurations
158+
- Test configurations before applying
159+
- Add/delete custom configurations
160+
161+
##### Using the Widget
162+
163+
1. **Select a Configuration**: Choose from the dropdown or create a new one
164+
2. **Edit Settings**: Modify cluster connection details and resource requirements
165+
3. **Advanced Options**: Expand the accordion for environment setup and additional settings
166+
4. **Apply Configuration**: Click "Apply Configuration" to use these settings for subsequent `@cluster` decorated functions
167+
5. **Save for Later**: Use "Save Configuration" to persist settings to a file
90168

91169
### Configuration File
92170

@@ -195,6 +273,116 @@ clustrix.configure(cluster_type='kubernetes')
195273
clustrix.configure(cluster_type='ssh', cluster_host='server.example.com')
196274
```
197275

276+
### Cloud Provider Integration
277+
278+
Clustrix provides native integration with major cloud providers for both VM and Kubernetes deployments:
279+
280+
#### AWS Integration
281+
282+
```python
283+
# Configure AWS credentials and region
284+
clustrix.configure(
285+
cluster_type='aws',
286+
access_key_id='YOUR_ACCESS_KEY',
287+
secret_access_key='YOUR_SECRET_KEY',
288+
region='us-west-2'
289+
)
290+
291+
# Run on EC2 instance
292+
@cluster(provider='aws', instance_type='p3.2xlarge', cores=8, memory='61GB')
293+
def train_on_aws():
294+
# GPU-accelerated training on AWS
295+
pass
296+
297+
# Run on EKS Kubernetes cluster
298+
@cluster(provider='aws', cluster_type='kubernetes', cluster_name='my-eks-cluster')
299+
def distributed_training():
300+
# Runs on Amazon EKS
301+
pass
302+
```
303+
304+
#### Google Cloud Integration
305+
306+
```python
307+
# Configure GCP with service account
308+
clustrix.configure(
309+
cluster_type='gcp',
310+
project_id='your-project-id',
311+
service_account_key='path/to/service-account-key.json',
312+
region='us-central1'
313+
)
314+
315+
# Run on Compute Engine
316+
@cluster(provider='gcp', machine_type='n1-highmem-8', cores=8, memory='52GB')
317+
def analyze_data():
318+
# High-memory computation on GCP
319+
pass
320+
321+
# Run on GKE cluster
322+
@cluster(provider='gcp', cluster_type='kubernetes', cluster_name='my-gke-cluster')
323+
def kubernetes_job():
324+
# Runs on Google Kubernetes Engine
325+
pass
326+
```
327+
328+
#### Azure Integration
329+
330+
```python
331+
# Configure Azure with service principal
332+
clustrix.configure(
333+
cluster_type='azure',
334+
subscription_id='YOUR_SUBSCRIPTION_ID',
335+
client_id='YOUR_CLIENT_ID',
336+
client_secret='YOUR_CLIENT_SECRET',
337+
tenant_id='YOUR_TENANT_ID',
338+
region='eastus'
339+
)
340+
341+
# Run on Azure VM
342+
@cluster(provider='azure', vm_size='Standard_NC6', cores=6, memory='56GB')
343+
def gpu_workload():
344+
# GPU computation on Azure
345+
pass
346+
347+
# Run on AKS cluster
348+
@cluster(provider='azure', cluster_type='kubernetes', cluster_name='my-aks-cluster')
349+
def container_workload():
350+
# Runs on Azure Kubernetes Service
351+
pass
352+
```
353+
354+
#### Lambda Cloud Integration
355+
356+
```python
357+
# Configure Lambda Cloud for GPU workloads
358+
clustrix.configure(
359+
cluster_type='lambda_cloud',
360+
api_key='YOUR_LAMBDA_API_KEY'
361+
)
362+
363+
# Run on Lambda GPU instance
364+
@cluster(provider='lambda_cloud', instance_type='gpu_1x_a100', cores=30, memory='200GB')
365+
def train_large_model():
366+
# A100 GPU training on Lambda Cloud
367+
pass
368+
```
369+
370+
#### HuggingFace Spaces Integration
371+
372+
```python
373+
# Configure HuggingFace Spaces
374+
clustrix.configure(
375+
cluster_type='huggingface_spaces',
376+
token='YOUR_HF_TOKEN'
377+
)
378+
379+
# Deploy to HuggingFace Spaces
380+
@cluster(provider='huggingface_spaces', space_hardware='gpu-t4-medium')
381+
def inference_endpoint():
382+
# Runs on HuggingFace infrastructure
383+
pass
384+
```
385+
198386
## Command Line Interface
199387

200388
```bash

clustrix/executor.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,9 @@ def _setup_kubernetes(self):
6666
and self.config.cluster_type == "kubernetes"
6767
):
6868
try:
69-
from .cloud_providers import CloudProviderManager
69+
from . import cloud_providers as cloud_providers_module
7070

71-
cloud_manager = CloudProviderManager(self.config)
71+
cloud_manager = cloud_providers_module.CloudProviderManager(self.config)
7272
result = cloud_manager.auto_configure()
7373

7474
if result.get("auto_configured"):

tests/test_notebook_magic.py

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,6 @@
1717
load_config_from_file,
1818
validate_ip_address,
1919
validate_hostname,
20-
display_config_widget,
21-
auto_display_on_import,
22-
load_ipython_extension,
2320
)
2421

2522
# Import widget conditionally to avoid GitHub Actions import issues
@@ -325,7 +322,7 @@ def test_save_config_from_widgets(self, mock_ipython_environment):
325322
assert config["default_cores"] == 8
326323
assert config["default_memory"] == "32GB"
327324
assert config["package_manager"] == "conda"
328-
assert config["cost_monitoring"] == True
325+
assert config["cost_monitoring"] is True
329326

330327
def test_load_config_to_widgets(self, mock_ipython_environment):
331328
"""Test loading configuration into widgets."""
@@ -374,7 +371,7 @@ def test_load_config_to_widgets(self, mock_ipython_environment):
374371
assert widget.memory_field.value == "64GB"
375372
assert widget.k8s_namespace.value == "production"
376373
assert widget.package_manager.value == "uv"
377-
assert widget.cost_monitoring_checkbox.value == True
374+
assert widget.cost_monitoring_checkbox.value is True
378375

379376
def test_cluster_type_field_visibility(self, mock_ipython_environment):
380377
"""Test field visibility changes based on cluster type."""

0 commit comments

Comments
 (0)