Complete difFUBAR server integration

[stevenweaver]

Overview

Complete the server-side integration for difFUBAR analysis. The Julia backend is fully functional but needs to be properly integrated with the datamonkey-js-server infrastructure.

Tasks

1. Server Route Registration

Add difFUBAR routes to server.js:

// In server.js
difFubar = require("./app/difFubar/difFubar.js"),

r.route("difFubar", {
  spawn: function(stream, params) {
    new difFubar.difFubar(socket, stream, params.job);
  },
  progress: function(stream, params) {
    helpers.jobProgress(params.job, stream);
  },
  isrunning: function(stream, params) {
    helpers.isJobRunning(params.job, stream);
  }
});

2. Configuration Parameters

Add to config.json:

{
  "difFubar_walltime": "24:00:00",
  "difFubar_procs": 8,
  "difFubar_nodes": 1,
  "difFubar_memory": "32GB",
  "julia_path": "/usr/local/bin/julia",
  "julia_project": "/path/to/difFUBAR/julia_env"
}

3. Result Processing Pipeline

Implement custom result processing in difFubar.js:

difFubar.prototype.onComplete = function() {
  // Parse difFUBAR JSON output
  const results = JSON.parse(fs.readFileSync(this.results_fn));
  
  // Validate results format
  if (\!results.data || \!results.data.sites) {
    throw new Error("Invalid difFUBAR results format");
  }
  
  // Process differential selection sites
  const detectedSites = results.data.sites.filter(site => 
    Math.max(site["P(ω1 > ω2)"], site["P(ω2 > ω1)"], 
             site["P(ω1 > 1)"], site["P(ω2 > 1)"]) > this.pos_threshold
  );
  
  // Store processed results
  this.results = {
    detected_sites: detectedSites,
    summary: results.summary,
    parameters: results.input
  };
};

4. Julia Environment Setup

• Document Julia installation requirements
• Add Julia environment variables to server startup
• Ensure proper permissions for Julia execution

Acceptance Criteria

• Server can spawn difFUBAR jobs via API/web interface
• Job status tracking works correctly
• Results are properly parsed and stored
• Error handling captures Julia-specific failures
• Configuration supports Julia paths and resources

Dependencies

• Julia 1.10+ installed on compute nodes
• CodonMolecularEvolution.jl package environment
• Existing PR: Enhanced difFUBAR backend with robust NEXUS parsing and debugging #329

Priority

HIGH - Core functionality cannot work without this integration

[stevenweaver]

This is unnecessary as datamonkey-js will take care of this:

3. Result Processing Pipeline

[stevenweaver]

✅ Item (1) Complete: Server Route Integration

I've successfully implemented the difFUBAR server route integration as requested in this issue.

Changes Made:

1. Server Route Registration (server.js)

• Added difFubar = require("./app/difFubar/difFubar.js") import
• Registered difFubar route with all standard handlers:
  • spawn - Starts new difFUBAR jobs via API/web interface
  • check - Checks job status and handles job monitoring
  • resubscribe - Reconnects to existing jobs for progress tracking
  • cancel - Cancels running jobs

2. Code Organization Improvements

• Separated Julia analysis code into standalone difFubar_analysis.jl file
• Refactored difFubar.sh to use external Julia script with proper environment variable passing
• Enhanced error handling and output verification

Integration Status:

✅ difFUBAR can now be spawned via API/web interface
✅ Job status tracking implemented using existing infrastructure
✅ Follows established datamonkey-js-server patterns (same as busted, meme, slac, etc.)
✅ Integrates with Redis-based job management system

The route registration follows the same proven pattern used by all other analyses in the server, ensuring consistency and reliability.

Commit: a390882

[stevenweaver]

✅ Item (2) Complete: Configuration Options Verification

I've verified and fixed the usage of configuration parameters in config.json. Here's the detailed analysis:

Configuration Options Status:

Previously Missing:

• ❌ difFubar_nodes - Was hardcoded as "nodes=1"
• ❌ difFubar_memory - Not included in resource requirements
• ❌ julia_path - Not passed to shell script
• ❌ julia_project - Not passed to shell script

Already Working:

• ✅ difFubar_walltime - Used in qsub walltime parameter
• ✅ difFubar_procs - Used in qsub processor count

Changes Made:

1. Updated difFubar.js:

// Before: hardcoded nodes and missing memory
"-l walltime=" + config.difFubar_walltime + ",nodes=1:ppn=" + config.difFubar_procs,

// After: using all config options  
"-l walltime=" + config.difFubar_walltime + 
",nodes=" + config.difFubar_nodes + ":ppn=" + config.difFubar_procs + 
",mem=" + config.difFubar_memory,

2. Updated environment variable passing:

// Added Julia configuration to environment variables
",julia_path=" + config.julia_path +
",julia_project=" + config.julia_project

3. Updated difFubar.sh:

# Use configured Julia path and project environment
JULIA_CMD="${julia_path:-julia}"
JULIA_PROJECT_PATH="${julia_project:-$SCRIPT_DIR/../../.julia_env}"

# Verify configured Julia path works
if \! command -v "$JULIA_CMD" &> /dev/null; then
    echo "ERROR: Julia not found at $JULIA_CMD" > $sfn
    exit 1
fi

# Use configured paths
"$JULIA_CMD" --project="$JULIA_PROJECT_PATH" -t auto "$SCRIPT_DIR/difFubar_analysis.jl"

Current Configuration Usage:

✅ All 6 configuration options now properly utilized:

• difFubar_walltime → PBS/SLURM walltime limit
• difFubar_procs → Number of processors requested
• difFubar_nodes → Number of compute nodes requested
• difFubar_memory → Memory allocation requirement
• julia_path → Path to Julia executable
• julia_project → Julia project environment directory

Commit: f049338