Skip to content

openSBI with sail-riscv #886

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Arielfoever opened this issue Apr 25, 2025 · 10 comments
Open

openSBI with sail-riscv #886

Arielfoever opened this issue Apr 25, 2025 · 10 comments

Comments

@Arielfoever
Copy link
Contributor

I am trying to replace the old os-boot with opensbi (with default payload). I'd be very happy if someone can add it to agenda and help me to debug it.

There are two problems.

DYN problem

When the ELF is DYN (Position-Independent Executable file) , it can not be loaded by sail-riscv due to load_elf

sail-riscv/c_emulator/riscv_sim.cpp

Line 346 in 5872942

load_elf(f, &is32bit, &entry);
like this. This is the hard-coded in opensbi. (See also riscv-software-src/opensbi#369).

[ariel@archlinux opensbi]$ readelf -h build/platform/generic/firmware/fw_payload.elf
ELF 头：
  Magic：  7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  类别:                              ELF64
  数据:                              2 补码，小端序 (little endian)
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI 版本:                          0
  类型:                              DYN (Position-Independent Executable file)
  系统架构:                          RISC-V
  版本:                              0x1
  入口点地址：              0x0
  程序头起点：              64 (bytes into file)
  Start of section headers:          1614712 (bytes into file)
  标志：             0x1, RVC, soft-float ABI
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         6
  Size of section headers:           64 (bytes)
  Number of section headers:         26
  Section header string table index: 25

[ariel@archlinux sail-riscv]$ build/c_emulator/riscv_sim_rv64d ~/sail/opensbi/build/platform/generic/firmware/fw_payload.elf
Running file /home/ariel/sail/opensbi/build/platform/generic/firmware/fw_payload.elf.
Invalid ELF type or machine for class (64-bit)

loops

If you remove all Pies in opensbi you can get a normal elf and put your work into sail-riscv. You will get these above.

trapping from M to M to handle fetch-access-fault
handling exc#0x01 at priv M with tval 0x0000000000000000
CSR mstatus <- 0x0000000A00001800
within_phys_mem: 0x0000000000000000 not within phys-mem:
  plat_rom_base: 0x0000000000001000
  plat_rom_size: 0x0000000000001000
  plat_ram_base: 0x0000000080000000
  plat_ram_size: 0x0000000080000000
trapping from M to M to handle fetch-access-fault
handling exc#0x01 at priv M with tval 0x0000000000000000
CSR mstatus <- 0x0000000A00001800
within_phys_mem: 0x0000000000000000 not within phys-mem:
  plat_rom_base: 0x0000000000001000
  plat_rom_size: 0x0000000000001000
  plat_ram_base: 0x0000000080000000
  plat_ram_size: 0x0000000080000000
trapping from M to M to handle fetch-access-fault
handling exc#0x01 at priv M with tval 0x0000000000000000
CSR mstatus <- 0x0000000A00001800
within_phys_mem: 0x0000000000000000 not within phys-mem:
  plat_rom_base: 0x0000000000001000
  plat_rom_size: 0x0000000000001000
  plat_ram_base: 0x0000000080000000
  plat_ram_size: 0x0000000080000000
trapping from M to M to handle fetch-access-fault
handling exc#0x01 at priv M with tval 0x0000000000000000
CSR mstatus <- 0x0000000A00001800
within_phys_mem: 0x0000000000000000 not within phys-mem:
  plat_rom_base: 0x0000000000001000
  plat_rom_size: 0x0000000000001000
  plat_ram_base: 0x0000000080000000
  plat_ram_size: 0x0000000080000000
trapping from M to M to handle fetch-access-fault
handling exc#0x01 at priv M with tval 0x0000000000000000
CSR mstatus <- 0x0000000A00001800
within_phys_mem: 0x0000000000000000 not within phys-mem:
  plat_rom_base: 0x0000000000001000
  plat_rom_size: 0x0000000000001000
  plat_ram_base: 0x0000000080000000
  plat_ram_size: 0x0000000080000000
trapping from M to M to handle fetch-access-fault
handling exc#0x01 at priv M with tval 0x0000000000000000
CSR mstatus <- 0x0000000A00001800
within_phys_mem: 0x0000000000000000 not within phys-mem:
  plat_rom_base: 0x0000000000001000
  plat_rom_size: 0x0000000000001000
  plat_ram_base: 0x0000000080000000
  plat_ram_size: 0x0000000080000000
trapping from M to M to handle fetch-access-fault
handling exc#0x01 at priv M with tval 0x0000000000000000
CSR mstatus <- 0x0000000A00001800
within_phys_mem: 0x0000000000000000 not within phys-mem:
  plat_rom_base: 0x0000000000001000
  plat_rom_size: 0x0000000000001000
  plat_ram_base: 0x0000000080000000
  plat_ram_size: 0x0000000080000000
trapping from M to M to handle fetch-access-fault
handling exc#0x01 at priv M with tval 0x0000000000000000
CSR mstatus <- 0x0000000A00001800
within_phys_mem: 0x0000000000000000 not within phys-mem:
  plat_rom_base: 0x0000000000001000
  plat_rom_size: 0x0000000000001000
  plat_ram_base: 0x0000000080000000
  plat_ram_size: 0x0000000080000000
trapping from M to M to handle fetch-access-fault
handling exc#0x01 at priv M with tval 0x0000000000000000
CSR mstatus <- 0x0000000A00001800
within_phys_mem: 0x0000000000000000 not within phys-mem:
  plat_rom_base: 0x0000000000001000
  plat_rom_size: 0x0000000000001000
  plat_ram_base: 0x0000000080000000
  plat_ram_size: 0x0000000080000000
trapping from M to M to handle fetch-access-fault
handling exc#0x01 at priv M with tval 0x0000000000000000
CSR mstatus <- 0x0000000A00001800
within_phys_mem: 0x0000000000000000 not within phys-mem:
  plat_rom_base: 0x0000000000001000
  plat_rom_size: 0x0000000000001000
  plat_ram_base: 0x0000000080000000
  plat_ram_size: 0x0000000080000000
trapping from M to M to handle fetch-access-fault
handling exc#0x01 at priv M with tval 0x0000000000000000
CSR mstatus <- 0x0000000A00001800
within_phys_mem: 0x0000000000000000 not within phys-mem:
  plat_rom_base: 0x0000000000001000
  plat_rom_size: 0x0000000000001000
  plat_ram_base: 0x0000000080000000
  plat_ram_size: 0x0000000080000000
trapping from M to M to handle fetch-access-fault
handling exc#0x01 at priv M with tval 0x0000000000000000
CSR mstatus <- 0x0000000A00001800
within_phys_mem: 0x0000000000000000 not within phys-mem:
  plat_rom_base: 0x0000000000001000
  plat_rom_size: 0x0000000000001000
  plat_ram_base: 0x0000000080000000
  plat_ram_size: 0x0000000080000000
trapping from M to M to handle fetch-access-fault
handling exc#0x01 at priv M with tval 0x0000000000000000
CSR mstatus <- 0x0000000A00001800
within_phys_mem: 0x0000000000000000 not within phys-mem:
  plat_rom_base: 0x0000000000001000
  plat_rom_size: 0x0000000000001000
  plat_ram_base: 0x0000000080000000
  plat_ram_size: 0x0000000080000000
trapping from M to M to handle fetch-access-fault
@Arielfoever
Copy link
Contributor Author

HTIF may be connected with #346

This issue is for #580.

@Arielfoever
Copy link
Contributor Author

The third question is -l option is failed when you are in question 2.

@Timmmm
Copy link
Collaborator

Timmmm commented Apr 25, 2025

DYN

I think that indicates your executable is dynamically linked... It doesn't really have anything to do with position independent code (although it is true that dynamically linked code is almost always position independent).

When I build OpenSBI it gives me a static executable:

❯ readelf -h build/opensbi-1.5.1/build/platform/generic/firmware/fw_payload.elf
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           RISC-V
  Version:                           0x1
  Entry point address:               0x80000000
  Start of program headers:          64 (bytes into file)
  Start of section headers:          23476352 (bytes into file)
  Flags:                             0x1, RVC, soft-float ABI
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         7
  Size of section headers:           64 (bytes)
  Number of section headers:         30
  Section header string table index: 29

I couldn't actually get it to give me a DYN executable. Can you share your build commands?

Loops

This is fairly obvious. Your entry point is 0, but we don't define any memory to exist there. You can change where the RAM is defined but I would recommend just relocating OpenSBI to 0x80000000 (as shown above) because that's also what SPIKE is happy with.

Here's the command I use, note FW_TEXT_START

make -C build/opensbi-$(OPENSBI_VERSION) FW_TEXT_START=0x80000000 FW_PAYLOAD=y FW_PAYLOAD_PATH=../$(LINUX_VERSION)/arch/riscv/boot/Image CROSS_COMPILE=riscv64-linux- PLATFORM=generic

I have a Makefile that builds something that works. I'll clean it up a bit and share it.

@Timmmm
Copy link
Collaborator

Timmmm commented Apr 25, 2025

Ok here's the Makefile I use. It doesn't really have all the dependencies set up right so it's fragile, but if you run

make download_gcc
make download_linux
make download_opensbi
make linux
make opensbi_linux.elf
make sail.dtb

You'll get a boot image and then you can try

make spike

or

make sail

to run it.

It is possible to embed the DTB in the image by adding FW_FDT_PATH=../../sail.dtb to the OpenSBI make command, but I can't remember how well that works. IIRC maybe Spike doesn't like it or something.

If you want to turn the Makefile into something that actually has all the incremental build stuff set up properly that would be amazing!

Also note that there's no userspace so when it eventually gets to running init it dies. I didn't get as far as learning how that works and tbh I was only interested in performance - 250 million instructions is plenty for that!

.PHONY: clean download_gcc download_linux download_opensbi linux opensbi spike sail sail_profile pydrofoil qemu

# These versions are probably a bit old because I did this last year.
RISCV_COMPILER := riscv64-lp64d--glibc--stable-2024.05-1
LINUX_VERSION := linux-6.11.1
OPENSBI_VERSION := 1.5.1

download_gcc:
	mkdir -p build
	curl -L https://toolchains.bootlin.com/downloads/releases/toolchains/riscv64-lp64d/tarballs/$(RISCV_COMPILER).tar.xz -o build/gcc.tar.xz
	tar -C build -xf build/gcc.tar.xz

download_linux:
	mkdir -p build
	curl -L https://cdn.kernel.org/pub/linux/kernel/v6.x/$(LINUX_VERSION).tar.xz -o build/linux.tar.xz
	tar -C build -xf build/linux.tar.xz

download_opensbi:
	mkdir -p build
	curl -L https://github.com/riscv-software-src/opensbi/archive/refs/tags/v$(OPENSBI_VERSION).tar.gz -o build/opensbi.tar.xz
	tar -C build -xf build/opensbi.tar.xz

CROSS_COMPILE := riscv64-linux-
export PATH := $(shell pwd)/build/$(RISCV_COMPILER)/bin:$(PATH)

elf := build/opensbi-$(OPENSBI_VERSION)/build/platform/generic/firmware/fw_payload.elf

# Number of instructions to run.
instructions := 250000000

linux:
	make -C build/$(LINUX_VERSION) ARCH=riscv CROSS_COMPILE=$(CROSS_COMPILE) defconfig
	make -C build/$(LINUX_VERSION) -j6 CONFIG_HVC_RISCV_SBI=y ARCH=riscv CROSS_COMPILE=$(CROSS_COMPILE) Image vmlinux

# FW_TEXT_START is 0 by default which doesn't leave space for the emulator bootloader.
# 0x80000000 is the default start of Spike's memory.
opensbi_linux.elf:
	make -C build/opensbi-$(OPENSBI_VERSION) FW_TEXT_START=0x80000000 FW_PAYLOAD=y FW_PAYLOAD_PATH=../$(LINUX_VERSION)/arch/riscv/boot/Image CROSS_COMPILE=riscv64-linux- PLATFORM=generic
	cp $(elf) $@

%.dtb: %.dts
	dtc $< -o $@

# Run on Spike.
spike:
	spike --instructions=$(instructions) $(elf)

# Path to Sail emulator.
SAIL := riscv_sim_rv64d

# Run on Sail.
sail:
	$(SAIL) --no-trace -p -l $(instructions) --device-tree-blob sail.dtb opensbi_linux.elf

# For profiling performance.
sail_profile:
	CPUPROFILE=prof.out LD_PRELOAD=/usr/local/lib64/libtcmalloc_and_profiler.so $(SAIL) --no-trace -p -l $(instructions) --device-tree-blob sail.dtb opensbi_linux.elf
	pprof -http : prof.out

# Path to Pydrofoil Sail emulator.
PYDROFOIL := pydrofoil

# Run on Pydrofoil.
pydrofoil:
	$(PYDROFOIL) -l $(instructions) --device-tree-blob sail.dtb opensbi_linux.elf

# QEMU attempt. Not sure I ever got this working.
qemu:
	qemu-system-riscv32 \
	    -machine spike \
	    -cpu rv64 \
	    -nographic \
	    -kernel build/$(LINUX_VERSION)/arch/riscv/boot/Image.gz \
	    -dtb sail.dtb
#	    -d exec -D qemu.log \
#	    -d in_asm -D qemu.log \
#	    -device loader,file="$(elf)",cpu-num=0 \

# Clean OpenSBI... Haven't got incremental builds working quite right yet.
clean:
	rm -f opensbi_linux.elf
	rm -rf build/opensbi-$(OPENSBI_VERSION)/build

Here is sail.dts:

/dts-v1/;

/ {
  #address-cells = <2>;
  #size-cells = <2>;
  compatible = "ucbbar,spike-bare-dev";
  model = "ucbbar,spike-bare";
  chosen {
    bootargs = "console=hvc0 earlycon=sbi";
  };
  cpus {
    #address-cells = <1>;
    #size-cells = <0>;
    timebase-frequency = <10000000>;
    CPU0: cpu@0 {
      device_type = "cpu";
      reg = <0>;
      status = "okay";
      compatible = "riscv";
      riscv,isa = "rv64imafd";
      mmu-type = "riscv,sv39";
      riscv,pmpregions = <16>;
      riscv,pmpgranularity = <4>;
      clock-frequency = <1000000000>;
      CPU0_intc: interrupt-controller {
        #address-cells = <2>;
        #interrupt-cells = <1>;
        interrupt-controller;
        compatible = "riscv,cpu-intc";
      };
    };
  };
  memory@80000000 {
    device_type = "memory";
    reg = <0x0 0x80000000 0x0 0x80000000>;
  };
  soc {
    #address-cells = <2>;
    #size-cells = <2>;
    compatible = "ucbbar,spike-bare-soc", "simple-bus";
    ranges;
    clint@2000000 {
      compatible = "riscv,clint0";
      interrupts-extended = <&CPU0_intc 3 &CPU0_intc 7 >;
      reg = <0x0 0x2000000 0x0 0xc0000>;
    };
  };
  htif {
    compatible = "ucb,htif0";
  };
};

@jordancarlin
Copy link
Collaborator

It would be great to get this cleaned up and then either add it to or maybe even replace what’s in the os-boot directory.

@jrtc27
Copy link
Collaborator

jrtc27 commented Apr 25, 2025

No, ET_DYN is for PIEs. Well, really it's for shared libraries, but that got repurposed to also be for executables. PDEs use ET_EXEC.

I don't know why you're not getting a PIE. OpenSBI these days requires it.

@arichardson
Copy link
Collaborator

Passing the dtb to qemu is unlikely to work since QEMU does not interpret that at all and just places devices where they have been programmed. It generates its own device tree blob so should not be needed.

Getting a PIE binary to work here shouldn't be too difficult we just need a flag where to load it or default to start of RAM.

@Arielfoever
Copy link
Contributor Author

DYN

I think that indicates your executable is dynamically linked... It doesn't really have anything to do with position independent code (although it is true that dynamically linked code is almost always position independent).

When I build OpenSBI it gives me a static executable:

❯ readelf -h build/opensbi-1.5.1/build/platform/generic/firmware/fw_payload.elf
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           RISC-V
  Version:                           0x1
  Entry point address:               0x80000000
  Start of program headers:          64 (bytes into file)
  Start of section headers:          23476352 (bytes into file)
  Flags:                             0x1, RVC, soft-float ABI
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         7
  Size of section headers:           64 (bytes)
  Number of section headers:         30
  Section header string table index: 29

I couldn't actually get it to give me a DYN executable. Can you share your build commands?

Loops

This is fairly obvious. Your entry point is 0, but we don't define any memory to exist there. You can change where the RAM is defined but I would recommend just relocating OpenSBI to 0x80000000 (as shown above) because that's also what SPIKE is happy with.

Here's the command I use, note FW_TEXT_START

make -C build/opensbi-$(OPENSBI_VERSION) FW_TEXT_START=0x80000000 FW_PAYLOAD=y FW_PAYLOAD_PATH=../$(LINUX_VERSION)/arch/riscv/boot/Image CROSS_COMPILE=riscv64-linux- PLATFORM=generic

I have a Makefile that builds something that works. I'll clean it up a bit and share it.

for dyn, try newer version. The pie is added in newer ones. riscv-software-src/opensbi@99aabc6

@Arielfoever
Copy link
Contributor Author

Arielfoever commented Apr 26, 2025 •
edited
Loading

I have done all work with your help. Thank you very much.

I will put my work to os-boot. It has linux 6.14.4 and opensbi 1.6.

[ariel@archlinux bin]$ cat /tmp/console.log

OpenSBI v1.6
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name               : ucbbar,spike-bare
Platform Features           : medeleg
Platform HART Count         : 1
Platform IPI Device         : aclint-mswi
Platform Timer Device       : aclint-mtimer @ 10000000Hz
Platform Console Device     : htif
Platform HSM Device         : ---
Platform PMU Device         : ---
Platform Reboot Device      : htif
Platform Shutdown Device    : htif
Platform Suspend Device     : ---
Platform CPPC Device        : ---
Firmware Base               : 0x80000000
Firmware Size               : 317 KB
Firmware RW Offset          : 0x40000
Firmware RW Size            : 61 KB
Firmware Heap Offset        : 0x46000
Firmware Heap Size          : 37 KB (total), 2 KB (reserved), 11 KB (used), 23 KB (free)
Firmware Scratch Size       : 4096 B (total), 392 B (used), 3704 B (free)
Runtime SBI Version         : 2.0
Standard SBI Extensions     : time,rfnc,ipi,base,hsm,srst,pmu,dbcn,legacy
Experimental SBI Extensions : fwft,sse

Domain0 Name                : root
Domain0 Boot HART           : 0
Domain0 HARTs               : 0*
Domain0 Region00            : 0x0000000000000000-0x0000000000000fff M: (I,R,W) S/U: (R,W)
Domain0 Region01            : 0x0000000080040000-0x000000008004ffff M: (R,W) S/U: ()
Domain0 Region02            : 0x0000000002080000-0x00000000020bffff M: (I,R,W) S/U: ()
Domain0 Region03            : 0x0000000080000000-0x000000008003ffff M: (R,X) S/U: ()
Domain0 Region04            : 0x0000000002000000-0x000000000207ffff M: (I,R,W) S/U: ()
Domain0 Region05            : 0x0000000000000000-0xffffffffffffffff M: () S/U: (R,W,X)
Domain0 Next Address        : 0x0000000080200000
Domain0 Next Arg1           : 0x0000000082200000
Domain0 Next Mode           : S-mode
Domain0 SysReset            : yes
Domain0 SysSuspend          : yes

Boot HART ID                : 0
Boot HART Domain            : root
Boot HART Priv Version      : v1.12
Boot HART Base ISA          : rv64imafdcbv
Boot HART ISA Extensions    : sscofpmf,zicntr,zihpm,smcntrpmf,sdtrig
Boot HART PMP Count         : 16
Boot HART PMP Granularity   : 2 bits
Boot HART PMP Address Bits  : 54
Boot HART MHPM Info         : 29 (0xfffffff8)
Boot HART Debug Triggers    : 0 triggers
Boot HART MIDELEG           : 0x0000000000002222
Boot HART MEDELEG           : 0x000000000004b109
[    0.000000] Linux version 6.14.4 (ariel@archlinux) (riscv64-unknown-linux-gnu-gcc (g04696df09) 14.2.0, GNU ld (GNU Binutils) 2.44) #1 SMP Sat Apr 26 14:42:54 CST 2025
[    0.000000] Machine model: ucbbar,spike-bare
[    0.000000] SBI specification v2.0 detected
[    0.000000] SBI implementation ID=0x1 Version=0x10006
[    0.000000] SBI TIME extension detected
[    0.000000] SBI IPI extension detected
[    0.000000] SBI RFENCE extension detected
[    0.000000] SBI SRST extension detected
[    0.000000] SBI DBCN extension detected
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: legacy bootconsole [sbi0] enabled
[    0.000000] efi: UEFI not found.

@Timmmm
Copy link
Collaborator

Timmmm commented Apr 28, 2025

No, ET_DYN is for PIEs. Well, really it's for shared libraries, but that got repurposed to also be for executables. PDEs use ET_EXEC.

I stand corrected. This stuff seems to be really badly documented and barely understood by anyone, e.g. see this thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Timmmm @jrtc27 @arichardson @jordancarlin @Arielfoever