Add RaspberryPi Tutorials to deploy & infer llama model #15109

psiddh · 2025-10-14T15:12:41Z

Summary

[PLEASE REMOVE] See CONTRIBUTING.md's Pull Requests for ExecuTorch PR guidelines.

[PLEASE REMOVE] If this PR closes an issue, please add a Fixes #<issue-id> line.

[PLEASE REMOVE] If this PR introduces a fix or feature that should be the upcoming release notes, please add a "Release notes: " label. For a list of available release notes labels, check out CONTRIBUTING.md's Pull Requests.

Test plan

[PLEASE REMOVE] How did you test this PR? Please write down any manual commands you used and note down tests that you have written if applicable.

pytorch-bot · 2025-10-14T15:12:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15109

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 10 New Failures

As of commit 5062655 with merge base fca0f38 ():

NEW FAILURES - The following jobs have failed:

pull / test-qnn-wheel-packages-linux (3.10) / linux-job (gh)
RuntimeError: Command docker exec -t 47c48fff87dd6e69da5121efe43c7aacba0883dca08014fef5440848dca6498a /exec failed with exit code 1
pull / test-qnn-wheel-packages-linux (3.11) / linux-job (gh)
RuntimeError: Command docker exec -t 5d0f11be1bf2127ce1b36c825a086e0b568dd44bc41efe5abb25228590547165 /exec failed with exit code 1
pull / test-qnn-wheel-packages-linux (3.12) / linux-job (gh)
RuntimeError: Command docker exec -t addf01c94e07c6d57e63f0ee89bc16a894acf67f75d8b1c963ef7730aa18b37d /exec failed with exit code 1
pull / test-samsung-models-linux / linux-job (gh)
RuntimeError: Command docker exec -t 72aabdb62b7f08ee4b26c8940208ba00e9cb0e1cda9ad1706bfbb7a5aafbfb22 /exec failed with exit code 1
Test CUDA Builds / export-voxtral-cuda-artifact / linux-job (gh)
RuntimeError: Command docker exec -t 41c9adb8c0e7a94fdf1d31928fe8183dfcb97945587f13aee168fafbca2166e2 /exec failed with exit code 2
Test CUDA Builds / test-models-cuda (add_mul) / linux-job (gh)
RuntimeError: Command docker exec -t 6143eb3bf7f006bac2fd26b22230d3445e397eab6b320f46fd281e6ad3959721 /exec failed with exit code 1
Test CUDA Builds / test-models-cuda (add) / linux-job (gh)
RuntimeError: Command docker exec -t 9a967c866b17ea5ed9d2c83005388386114ce1567c451cb809c34cb3eed49089 /exec failed with exit code 1
Test CUDA Builds / test-models-cuda (conv1d) / linux-job (gh)
RuntimeError: Command docker exec -t aa73771ca07ff22b9bf835c82cec835037a2d9132a8dbe76784334a3c2fdd438 /exec failed with exit code 1
Test CUDA Builds / test-models-cuda (linear) / linux-job (gh)
RuntimeError: Command docker exec -t 69b5054f1cf17f3c9e4b5f53c9c796b9cc2eacf2c118d510eab0fdbccc32b9ef /exec failed with exit code 1
Test CUDA Builds / test-models-cuda (resnet18) / linux-job (gh)
RuntimeError: Command docker exec -t 0f6f2634d3008367daa649ee73331aef30945a99af3639b136f96684388d0363 /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2025-10-14T15:13:21Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

digantdesai · 2025-10-14T15:15:20Z

docs/source/raspberry_pi_llama_tutorial.md

+
+- **Python 3.10-3.12** (ExecuTorch requirement)
+- **conda** or **venv** for environment management
+- **CMake 3.29.6+** for cross-compilation


Suggested change

- **CMake 3.29.6+** for cross-compilation

- **CMake 3.29.6+**

digantdesai · 2025-10-14T15:15:37Z

docs/source/raspberry_pi_llama_tutorial.md

+
+### Target Device Requirements
+
+**Supported Devices**: **Raspberry Pi 4** and **Raspberry Pi 5** with **64-bit OS**


why do we restrict to 64-bit target os?

We haven't tested on other hardware yet, It should work mostly

digantdesai · 2025-10-14T15:15:54Z

docs/source/raspberry_pi_llama_tutorial.md

+
+- **Minimum 4GB RAM** (8GB recommended for larger models)
+- **8GB+ storage** for model files and binaries
+- **64-bit Raspberry Pi OS** (Bullseye or newer)


same here, and also why is this under memory reqs?

This is needed for llama I assume

Since we haven't done exhaustive per evals yet, better to leave out specific details out of this section for now..
Let me know what you both think about this, something as follows:

RAM & Storage Varies by model size and optimization level

digantdesai · 2025-10-14T15:16:45Z

docs/source/raspberry_pi_llama_tutorial.md

+python3 --version  # Should be 3.10-3.12
+
+# Check required tools
+which cmake git md5sum


s/which/hash?

digantdesai · 2025-10-14T15:17:36Z

docs/source/raspberry_pi_llama_tutorial.md

+mkdir ~/executorch-rpi && cd ~/executorch-rpi
+
+# Clone ExecuTorch repository
+git clone -b release/1.0 https://github.com/pytorch/executorch.git
+cd executorch


do all conditional.. i.e. a && b && c && d

digantdesai · 2025-10-14T15:18:02Z

docs/source/raspberry_pi_llama_tutorial.md

+cd executorch
+```
+
+### Create Conda Environment


why are we repeating ET setup here? Just point to the main one?

Just below this, I left the link to the main one

digantdesai · 2025-10-14T15:18:13Z

docs/source/raspberry_pi_llama_tutorial.md

+pip install --upgrade pip
+```
+
+Refer to → {doc}`getting-started` for more details.


digantdesai · 2025-10-14T15:18:52Z

docs/source/raspberry_pi_llama_tutorial.md

+
+## Cross-Compilation Toolchain Setup
+
+Run the following automated cross compile script on your Linux host machine:


this reads off..

digantdesai · 2025-10-14T15:19:33Z

docs/source/raspberry_pi_llama_tutorial.md

+
+## Model Preparation and Export
+
+### Download Llama Models


again can we point people to use standard llama tutorial to create an XNNPACK pte as opposed to repeating steps here?

digantdesai · 2025-10-14T15:21:28Z

docs/source/raspberry_pi_llama_tutorial.md

+sudo apt update
+
+# Install newer GLIBC packages
+sudo apt-get -t sid install libc6 libstdc++6


how do we know that will be compatible with our toolchain?

digantdesai · 2025-10-14T15:22:10Z

docs/source/raspberry_pi_llama_tutorial.md

+./llama_main --model_path ./llama3_2.pte --tokenizer_path ./tokenizer.model --seq_len 128 --prompt "What is the meaning of life?"
+```
+
+Happy Inferencing!


what happened to your script? why are we writing these steps again manually, may be I missed something..

There are two parts to this tutorials.

Linux cross compilation on host: This is scripted with this PR

All the Steps on RPI, involves moving required files from host machine onto RPI, and then ensuring basic environment is ok, then potentially trouble shoot if needed The tutorials gives step-by-step guidance on RPI (2nd step)

digantdesai

Thanks Sid. I left some comments..

mergennachin · 2025-10-14T15:54:14Z

docs/source/embedded-section.md


 - {doc}`tutorial-arm-ethos-u` — Export a simple PyTorch model for the ExecuTorch Ethos-U backend
-
+- {doc}`raspberry_pi_llama_tutorial` — Deploy a LLaMA model on a Raspberry Pi with the ExecuTorch Ethos-U backend


There's no ethos-U backend on Raspberry Pi

Also, this tutorial is not quite embedded I suppose it's running on Cortex-A based Raspberry Pi

So, I recommend also having another link in the https://docs-preview.pytorch.org/pytorch/executorch/15109/desktop-section.html

but also keep it here

There's no ethos-U backend on Raspberry Pi

Fixes

mergennachin · 2025-10-14T15:57:29Z

docs/source/raspberry_pi_llama_tutorial.md

+# Run the Raspberry Pi setup script for Pi 5
+examples/raspberry_pi/setup.sh pi5
+


@psiddh

we should protect in our CI as a fast follow-on. please create an issue

mergennachin · 2025-10-14T16:00:24Z

docs/source/raspberry_pi_llama_tutorial.md

+
+### Host Machine Requirements
+
+**Operating System**: Linux x86_64 (Ubuntu 20.04+ or CentOS Stream 9+)


Can the host machine be MacOS if you're cross compiling?

We do not support MacOs , unfortunately the required arm toolchain for RPi on host Mac is not supported

cccclai · 2025-10-14T16:51:03Z

There are many good comments from Digant and Mergen, overall is reasonable

psiddh requested a review from mergennachin as a code owner October 14, 2025 15:12

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 14, 2025

psiddh requested review from GregoryComer, cccclai, digantdesai and larryliu0820 October 14, 2025 15:13

digantdesai reviewed Oct 14, 2025

View reviewed changes

mergennachin reviewed Oct 14, 2025

View reviewed changes

psiddh mentioned this pull request Oct 14, 2025

Add Tutorials for CortexM / Cortex A (Rpi & Pico) #14410

Open

psiddh mentioned this pull request Oct 14, 2025

Create a CI job that automate RPI cross compilation on Linux Host #15124

Open

Add RaspberryPi Tutorials to deploy & infer llama model

5062655

psiddh force-pushed the rpi_tut branch from 3742017 to 5062655 Compare October 14, 2025 21:46

mergennachin approved these changes Oct 15, 2025

View reviewed changes

	- CMake 3.29.6+ for cross-compilation
	- CMake 3.29.6+


		### Target Device Requirements

		Supported Devices: Raspberry Pi 4 and Raspberry Pi 5 with 64-bit OS


		## Cross-Compilation Toolchain Setup

		Run the following automated cross compile script on your Linux host machine:


		- {doc}`tutorial-arm-ethos-u` — Export a simple PyTorch model for the ExecuTorch Ethos-U backend

		- {doc}`raspberry_pi_llama_tutorial` — Deploy a LLaMA model on a Raspberry Pi with the ExecuTorch Ethos-U backend

		# Run the Raspberry Pi setup script for Pi 5
		examples/raspberry_pi/setup.sh pi5


		### Host Machine Requirements

		Operating System: Linux x86_64 (Ubuntu 20.04+ or CentOS Stream 9+)

Add RaspberryPi Tutorials to deploy & infer llama model #15109

Are you sure you want to change the base?

Add RaspberryPi Tutorials to deploy & infer llama model #15109

Conversation

psiddh commented Oct 14, 2025

Summary

Test plan

Uh oh!

pytorch-bot bot commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15109

❌ 10 New Failures

Uh oh!

github-actions bot commented Oct 14, 2025

This PR needs a release notes: label

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cccclai commented Oct 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pytorch-bot bot commented Oct 14, 2025 •

edited

Loading

This PR needs a `release notes:` label