MyoWeb: Online viewer for MyoSuite

[vikashplus]

The generality of Human motor behavior is rooted in the complexity of its physiology. At one end, it has challenged researchers to understand and faithfully replicate the human motions; on the other end, even the exploration of the existing datasets and results has been challenging, given the complexity of prior frameworks.

MyoSuite provides an ecosystem that significantly lowers these barriers

• Access is as simple as pip install myosuite
• model exploration is as simple as an online webdemo

Opening a call to explore visualization options (onscreen, offscreen, web, GPU accelerated, etc) to further accelerate progress by lowering entry barriers for model, dataset, and policy exploration.

[Vittorio-Caggiano]

cc: @ttktjmt

[ttktjmt]

I've been working on creating a browser-based visualization solution for MyoSuite to help developers and researchers visualize the performance of reinforcement learning models in an accessible way.

Purpose:

• Enable developers and researchers to broadly share their RL model performance for MyoSuite on the web, allowing others to view and interact with the model without requiring technical setup or deep scientific understanding
• Create an accessible visualization platform similar to mujoco_playground's live demo or brax's visualizer, but specifically tailored for MyoSuite environments

Completed:

• Investigated different architecture options for browser-based visualization:
  1. Client-only
     • Client: Three.js (rendering) + MuJoCo WASM (physics) + TensorFlow.js (RL inference)
     • Server: None
  2. RL API
     • Client: Three.js (rendering) + MuJoCo WASM (physics)
     • ↓observation ↑action
     • Server: API (RL inference)
  3. Server-simulation
     • Client: Three.js (rendering)
     • ↓user input ↑simulation result
     • Server: MJX (physics) + Python RL library (RL inference)
• Extended the myosuite_demo project with the bimanual model
  • Added RL control options to the UI
  • Hosted the app on GitHub Pages: ttktjmt.github.io/myosuite_demo
• Created an offline simulation visualization of an MJX elbow model using Brax's renderer; brax.io.html.render ttktjmt.github.io/myosuite
  • Found that tendon visualization is not supported in Brax's renderer and raised an issue: Tendon/Muscle Rendering in Brax Visualizer google/brax#581
• Adapted the Brax's Flask visualizer app and tested it with musculoskeletal models

Current Development Plans:

• Fix bugs in myosuite_demo
  • Memory overflow bug occurs when user reloads models multiple times
• Add RL capabilities to myosuite_demo
  • Load RL Model (inference only)
  • Re-implement the get_obs() function in JavaScript
  • Control the model with action output from a model
  • Add RL control option UI
  • Add the target elbow angle control option
• Implement tendon visualization support in brax
• Integrate the web app myosuite_demo into the myosuite repo