@@ -12,11 +12,14 @@ Clustrix is a Python package that enables seamless distributed computing on clus
12
12
13
13
- ** Simple Decorator Interface** : Just add ` @cluster ` to any function
14
14
- ** Interactive Jupyter Widget** : ` %%clusterfy ` magic command with GUI configuration manager
15
- - ** Multiple Cluster Support** : SLURM, PBS, SGE, Kubernetes, and SSH
15
+ - ** Multiple Cluster Support** : SLURM, PBS, SGE, Kubernetes, SSH, and major cloud providers
16
+ - ** Cloud Provider Integration** : Native support for AWS (EC2/EKS), Google Cloud (GCE/GKE), Azure (VM/AKS), Lambda Cloud, and HuggingFace Spaces
16
17
- ** Automatic Dependency Management** : Captures and replicates your exact Python environment
17
18
- ** Native Cost Monitoring** : Built-in cost tracking for all major cloud providers
19
+ - ** Kubernetes Support** : Deploy to EKS, GKE, AKS, or any Kubernetes cluster
18
20
- ** Loop Parallelization** : Automatically distributes loops across cluster nodes
19
21
- ** Flexible Configuration** : Easy setup with config files, environment variables, or interactive widget
22
+ - ** Dynamic Instance Selection** : Auto-populated dropdowns for cloud instance types and regions
20
23
- ** Error Handling** : Comprehensive error reporting and job monitoring
21
24
22
25
## Quick Start
@@ -80,13 +83,88 @@ import clustrix # Auto-loads the magic command
80
83
# - Save/load configurations to files
81
84
```
82
85
86
+ #### Interactive Configuration Widget
87
+
88
+ The Clustrix widget provides a comprehensive GUI for managing cluster configurations directly in Jupyter notebooks. Here's what you'll see when you use the ` %%clusterfy ` magic command:
89
+
90
+ ##### Default View
91
+ When the widget first loads, it displays the "Local Single-core" configuration for quick testing:
92
+
93
+ ![ Default widget view showing Local Single-core configuration] ( https://github.com/user-attachments/assets/44baefd7-9dd7-452d-bc43-ef53136d13a4 )
94
+
95
+ ##### Configuration Dropdown
96
+ The dropdown menu includes pre-built templates for various cluster types and cloud providers:
97
+
98
+ ![ Configuration dropdown showing available templates] ( https://github.com/user-attachments/assets/744e2eef-e03e-46bd-9bb1-e0ed2c11cf45 )
99
+
83
100
The widget includes pre-built templates for:
84
- - ** Local Development** : Run jobs on your local machine
85
- - ** AWS GPU Instances** : p3.2xlarge, p3.8xlarge templates
86
- - ** Google Cloud** : CPU and GPU instance configurations
87
- - ** Azure** : Virtual machine templates with GPU support
88
- - ** SLURM HPC** : University cluster configurations
89
- - ** Kubernetes** : Container-based execution
101
+ - ** Local Development** :
102
+ - Local Single-core: Run jobs on one CPU core
103
+ - Local Multi-core: Utilize all available CPU cores
104
+ - ** HPC Clusters** :
105
+ - SLURM: University and research cluster support
106
+ - PBS/SGE: Traditional HPC schedulers
107
+ - ** Cloud Providers** :
108
+ - ** AWS** : EC2 instances and EKS Kubernetes clusters
109
+ - ** Google Cloud** : Compute Engine VMs and GKE clusters
110
+ - ** Azure** : Virtual Machines and AKS Kubernetes clusters
111
+ - ** Lambda Cloud** : GPU-optimized instances for ML/AI
112
+ - ** HuggingFace Spaces** : Deploy to HF Spaces infrastructure
113
+ - ** Kubernetes** : Native container orchestration support
114
+
115
+ ##### Cluster Configuration Examples
116
+
117
+ ###### SLURM Cluster Configuration
118
+ For traditional HPC clusters, the widget provides all essential fields:
119
+
120
+ ![ SLURM configuration with basic settings] ( https://github.com/user-attachments/assets/994a803d-485e-4d14-bf93-2f0223066510 )
121
+
122
+ The advanced settings accordion reveals additional options:
123
+
124
+ ![ SLURM advanced configuration options] ( https://github.com/user-attachments/assets/0930eee0-ee52-4d53-9165-81e907c7d962 )
125
+
126
+ Advanced options include:
127
+ - Module loads (e.g., ` python/3.9 ` , ` cuda/11.2 ` )
128
+ - Environment variables
129
+ - Pre-execution commands
130
+ - Custom SSH key paths
131
+ - Cost monitoring toggles
132
+
133
+ ###### Cloud Provider Configuration
134
+
135
+ ** Google Cloud Platform:**
136
+ When selecting a cloud provider, only relevant fields are displayed:
137
+
138
+ ![ GCP VM configuration interface] ( https://github.com/user-attachments/assets/53acb782-65cd-4b65-93a0-98286144f223 )
139
+
140
+ ** Lambda Cloud GPU Instances:**
141
+ The widget dynamically populates instance type dropdowns based on the selected provider:
142
+
143
+ ![ Lambda Cloud with GPU instance dropdown] ( https://github.com/user-attachments/assets/4bd0e176-d5f3-430e-86af-ddbfff827d77 )
144
+
145
+ ##### Key Widget Features
146
+
147
+ 1 . ** Dynamic Field Visibility** : Only shows fields relevant to the selected cluster type
148
+ 2 . ** Provider-Specific Options** :
149
+ - AWS: Region selection, instance types, EKS cluster options
150
+ - Azure: Resource groups, VM sizes, AKS configuration
151
+ - GCP: Projects, zones, machine types, GKE options
152
+ - Lambda Cloud: GPU instance selection with live pricing
153
+ 3 . ** Input Validation** : Real-time validation for hostnames, IP addresses, and configuration values
154
+ 4 . ** Tooltips** : Hover over any field label to see detailed help text
155
+ 5 . ** Configuration Management** :
156
+ - Save configurations to YAML/JSON files
157
+ - Load existing configurations
158
+ - Test configurations before applying
159
+ - Add/delete custom configurations
160
+
161
+ ##### Using the Widget
162
+
163
+ 1 . ** Select a Configuration** : Choose from the dropdown or create a new one
164
+ 2 . ** Edit Settings** : Modify cluster connection details and resource requirements
165
+ 3 . ** Advanced Options** : Expand the accordion for environment setup and additional settings
166
+ 4 . ** Apply Configuration** : Click "Apply Configuration" to use these settings for subsequent ` @cluster ` decorated functions
167
+ 5 . ** Save for Later** : Use "Save Configuration" to persist settings to a file
90
168
91
169
### Configuration File
92
170
@@ -195,6 +273,116 @@ clustrix.configure(cluster_type='kubernetes')
195
273
clustrix.configure(cluster_type = ' ssh' , cluster_host = ' server.example.com' )
196
274
```
197
275
276
+ ### Cloud Provider Integration
277
+
278
+ Clustrix provides native integration with major cloud providers for both VM and Kubernetes deployments:
279
+
280
+ #### AWS Integration
281
+
282
+ ``` python
283
+ # Configure AWS credentials and region
284
+ clustrix.configure(
285
+ cluster_type = ' aws' ,
286
+ access_key_id = ' YOUR_ACCESS_KEY' ,
287
+ secret_access_key = ' YOUR_SECRET_KEY' ,
288
+ region = ' us-west-2'
289
+ )
290
+
291
+ # Run on EC2 instance
292
+ @cluster (provider = ' aws' , instance_type = ' p3.2xlarge' , cores = 8 , memory = ' 61GB' )
293
+ def train_on_aws ():
294
+ # GPU-accelerated training on AWS
295
+ pass
296
+
297
+ # Run on EKS Kubernetes cluster
298
+ @cluster (provider = ' aws' , cluster_type = ' kubernetes' , cluster_name = ' my-eks-cluster' )
299
+ def distributed_training ():
300
+ # Runs on Amazon EKS
301
+ pass
302
+ ```
303
+
304
+ #### Google Cloud Integration
305
+
306
+ ``` python
307
+ # Configure GCP with service account
308
+ clustrix.configure(
309
+ cluster_type = ' gcp' ,
310
+ project_id = ' your-project-id' ,
311
+ service_account_key = ' path/to/service-account-key.json' ,
312
+ region = ' us-central1'
313
+ )
314
+
315
+ # Run on Compute Engine
316
+ @cluster (provider = ' gcp' , machine_type = ' n1-highmem-8' , cores = 8 , memory = ' 52GB' )
317
+ def analyze_data ():
318
+ # High-memory computation on GCP
319
+ pass
320
+
321
+ # Run on GKE cluster
322
+ @cluster (provider = ' gcp' , cluster_type = ' kubernetes' , cluster_name = ' my-gke-cluster' )
323
+ def kubernetes_job ():
324
+ # Runs on Google Kubernetes Engine
325
+ pass
326
+ ```
327
+
328
+ #### Azure Integration
329
+
330
+ ``` python
331
+ # Configure Azure with service principal
332
+ clustrix.configure(
333
+ cluster_type = ' azure' ,
334
+ subscription_id = ' YOUR_SUBSCRIPTION_ID' ,
335
+ client_id = ' YOUR_CLIENT_ID' ,
336
+ client_secret = ' YOUR_CLIENT_SECRET' ,
337
+ tenant_id = ' YOUR_TENANT_ID' ,
338
+ region = ' eastus'
339
+ )
340
+
341
+ # Run on Azure VM
342
+ @cluster (provider = ' azure' , vm_size = ' Standard_NC6' , cores = 6 , memory = ' 56GB' )
343
+ def gpu_workload ():
344
+ # GPU computation on Azure
345
+ pass
346
+
347
+ # Run on AKS cluster
348
+ @cluster (provider = ' azure' , cluster_type = ' kubernetes' , cluster_name = ' my-aks-cluster' )
349
+ def container_workload ():
350
+ # Runs on Azure Kubernetes Service
351
+ pass
352
+ ```
353
+
354
+ #### Lambda Cloud Integration
355
+
356
+ ``` python
357
+ # Configure Lambda Cloud for GPU workloads
358
+ clustrix.configure(
359
+ cluster_type = ' lambda_cloud' ,
360
+ api_key = ' YOUR_LAMBDA_API_KEY'
361
+ )
362
+
363
+ # Run on Lambda GPU instance
364
+ @cluster (provider = ' lambda_cloud' , instance_type = ' gpu_1x_a100' , cores = 30 , memory = ' 200GB' )
365
+ def train_large_model ():
366
+ # A100 GPU training on Lambda Cloud
367
+ pass
368
+ ```
369
+
370
+ #### HuggingFace Spaces Integration
371
+
372
+ ``` python
373
+ # Configure HuggingFace Spaces
374
+ clustrix.configure(
375
+ cluster_type = ' huggingface_spaces' ,
376
+ token = ' YOUR_HF_TOKEN'
377
+ )
378
+
379
+ # Deploy to HuggingFace Spaces
380
+ @cluster (provider = ' huggingface_spaces' , space_hardware = ' gpu-t4-medium' )
381
+ def inference_endpoint ():
382
+ # Runs on HuggingFace infrastructure
383
+ pass
384
+ ```
385
+
198
386
## Command Line Interface
199
387
200
388
``` bash
0 commit comments