Notes on how to audit a maximized flashed firmware image

[tlaurion]

This is raw notes. This will get edited multiple times prior of having a base to create a wiki page.

The quick way, no-brainer, is to reflash the same downloaded/compiled firmware and keeping settings, as already documented at:
https://osresearch.net/Updating#reflashing-the-same-firmware

Unless flashrom itself was compromised (meaning that the payload is tampered), reflashing the same firmware image using the menu options to flash new firmware while keeping settings will reflash IFD+ME+GBE+coreboot+payload+current CBFS files. Since coreboot measures its bootblock + ramstage+romstage+payload, and then Heads measures added CBFS content, TOTP/HOTP measurements should be the same, and TOTP/HOTP unsealing of secrets should match.

But that doesn't answer the introspection need of some users to actually validate that the firmware components are actually in the expected state. And those notes, even better, if translated into code, should answer those needs.

---

Taking an internal backup of the firmware to be inspected externally

From Heads Recovery console, one can easily obtain actual BOARD_NAME and FW_VER that includes the bits we want to differenciate properly the actual board and flashed Heads commit ID in case we want to go back.

From Heads recovery shell, with a USB thumb drive ready to receive backup, we mount usb drive in read+write and we take a backup with proper naming scheme:

mount-usb rw
flashrom -p internal -r /media/$CONFIG_BOARD-$(echo $FW_VER|awk -F ' ' '{print $2}'_backup.rom
umount /media

Extracting content from that firmware image for validation

Unfortunately, coreboot-utils are precompiled for a limited number of distributions today. I'll take for granted here that debian12 is installed under QubesOS (see unman's repo https://qubes.3isec.org/Templates_4.1/)
Then a quick sudo apt install coreboot-utils binwalk will make the tools available with/without sudo.

Each Heads board's build comes with a hashes.txt file. It is either dowloadable from CircleCI artifacts, or can be seen, and copy pasted from a build's output given on screen.
Following download instructions will lead you to the hashes step of any CircleCI build: https://osresearch.net/Downloading

Now.
After having mounted our USB thumb drive under a debian-12 based qube with tools installed, from command line, having passed the USB thumb drive to our newly booted qube:

sudo mount /dev/sda1 /media
mkdir -p /tmp/inspection
cp /media/x230-hotp-maximized-Heads-v0.2.0-1296-g139ecb8_backup.rom /tmp/inspection
cd /tmp/inspection

And then we work on our copied to memory image.

We can extract partitions not managed from coreboot actual versions measurements:

user@heads-backup-extraction:/tmp/inspection$ ifdtool -x x230-hotp-maximized-Heads-v0.2.0-1296-g139ecb8_backup.rom
File x230-hotp-maximized-Heads-v0.2.0-1296-g139ecb8_backup.rom is 12582912 bytes
  Flash Region 0 (Flash Descriptor): 00000000 - 00000fff 
  Flash Region 1 (BIOS): 0001b000 - 00bfffff 
  Flash Region 2 (Intel ME): 00003000 - 0001afff 
  Flash Region 3 (GbE): 00001000 - 00002fff 
  Flash Region 4 (Platform Data): 00fff000 - 00000fff (unused)
user@heads-backup-extraction:/tmp/inspection$ ls -al
total 24592
drwxr-xr-x  3 user user     4096 Nov 25 10:56 .
drwxrwxrwt 12 root root     4096 Nov 25 11:22 ..
-rw-r--r--  1 user user     4096 Nov 25 12:05 flashregion_0_flashdescriptor.bin
-rw-r--r--  1 user user 12472320 Nov 25 12:05 flashregion_1_bios.bin
-rw-r--r--  1 user user    98304 Nov 25 12:05 flashregion_2_intel_me.bin
-rw-r--r--  1 user user     8192 Nov 25 12:05 flashregion_3_gbe.bin
-rw-r--r--  1 user user 12582912 Nov 25 10:14 x230-hotp-maximized-Heads-v0.2.0-1296-g139ecb8_backup.rom

Here one could compare ME against what is downloaded and extracted from Heads scripts, GBE and IFD configs against what is stored under Heads tree.

But most questions are related to the firmware integrity itself, Heads itself:

user@heads-backup-extraction:/tmp/inspection$  cbfstool x230-hotp-maximized-Heads-v0.2.0-1296-g139ecb8_backup.rom print
FMAP REGION: COREBOOT
Name                           Offset     Type           Size   Comp
cbfs master header             0x0        cbfs header        32 none
fallback/romstage              0x80       (unknown)       85100 none
cpu_microcode_blob.bin         0x14d80    microcode       26624 none
fallback/ramstage              0x1b600    (unknown)       97676 none
config                         0x33400    raw               834 none
revision                       0x33780    raw               691 none
fallback/dsdt.aml              0x33a80    raw             14615 none
vbt.bin                        0x37400    raw              1433 LZMA (4281 decompressed)
cmos_layout.bin                0x37a00    cmos_layout      1884 none
fallback/postcar               0x381c0    (unknown)       25816 none
fallback/payload               0x3e700    simple elf    7320519 none
heads/initrd/.gnupg/pubring.kbx 0x739b00   raw             11549 none
heads/initrd/.gnupg/trustdb.gpg 0x73c880   raw              1320 none
heads/initrd/etc/config.user   0x73ce00   raw                22 none
(empty)                        0x73ce80   null          4337432 none
bootblock                      0xb5fdc0   bootblock       65536 none

Here again, a lot of stuff there.

Get build hashes for current rom

x230-hotp-maximized-Heads-v0.2.0-1296-g139ecb8_backup.rom is commit 139ecb8
We go over linuxboot/heads@139ecb8
We click grewn mark, follow the rabbit to the build for x230-hot-maximized at https://app.circleci.com/jobs/github/osresearch/heads/5448, click the "Output hashes" if this is an old build without artifacts, and copy the output of the hashes there to a local hashes.txt file we make sure to have available for our situation. We can directly save the output from CircleCI's Download Icon, which in our case leads to https://circleci.com/api/v1.1/project/github/osresearch/heads/5448/output/104/0?file=true&allocation-id=63751e76e6fb893f12bf3473-0-build%2F97AF006

From now on I consider we have that file saved into hashes.txt to be used locally.

Extract content from coreboot's payload

user@heads-backup-extraction:/tmp/inspection$ mkdir -p payload_content
user@heads-backup-extraction:/tmp/inspection$ cbfstool x230-hotp-maximized-Heads-v0.2.0-1296-g139ecb8_backup.rom extract -n fallback/payload -f payload_content/payload -m x86

We now have coreboot's payload.

user@heads-backup-extraction:/tmp/inspection/payload_content$ ls -al
total 7180
drwxr-xr-x 2 user user    4096 Nov 25 12:22 .
drwxr-xr-x 3 user user    4096 Nov 25 10:56 ..
-rw-r--r-- 1 user user   11525 Nov 25 10:58 hashes.txt
-rw-r--r-- 1 user user 7328543 Nov 25 12:14 payload

We extract the content with binwalk

user@heads-backup-extraction:/tmp/inspection/payload_content$ binwalk --extract payload 

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             ELF, 32-bit LSB executable, Intel 80386, version 1 (SYSV)
13013         0x32D5          xz compressed data
2072933       0x1FA165        MySQL MISAM compressed data file Version 8
2910340       0x2C6884        bix header, header size: 64 bytes, header CRC: 0xE802F8, created: 2106-02-07 01:21:09, image size: 186 bytes, Data Address: 0x1000000, Entry Point: 0x83BB9000, data CRC: 0x50F, compression type: none, image name: ""
2923392       0x2C9B80        xz compressed data
3035935       0x2E531F        xz compressed data

user@heads-backup-extraction:/tmp/inspection/payload_content$ ls -al
total 7184
drwxr-xr-x 3 user user    4096 Nov 25 12:22 .
drwxr-xr-x 3 user user    4096 Nov 25 10:56 ..
-rw-r--r-- 1 user user   11525 Nov 25 10:58 hashes.txt
-rw-r--r-- 1 user user 7328543 Nov 25 12:14 payload
drwxr-xr-x 2 user user    4096 Nov 25 12:22 _payload.extracted

user@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ ls | while read file; do sha256sum $file; done | while read line; do hash=$(echo $line | awk -F " " '{print $1}'); file=$(echo $line | awk -F " " '{print $2}'); grep $hash ../hashes.txt && echo "we have match for $hash in $file"; done
b3387d6b04c6246198638a46a2fec5bb7623a99a97189a26c5e019524aef8992  build/x86/x230-hotp-maximized/initrd.cpio.xz
we have match for b3387d6b04c6246198638a46a2fec5bb7623a99a97189a26c5e019524aef8992 in 2E531F.xz

So we have a matching initrd.cpio.xz into 2E531F.xz
We could dig more into that, and will (after all, everything hashes.txt is detailing why we have a match for the payload).

hashes.txt can be used under recovery shell to verify hashes of the initrd binaries

The paths undr hashes.txt are relative. If we have hashes.txt under USB thumb drive, from Heads recovery shell:
cd /
mount-usb
sha256sum -c /media/hashes.txt

Will give OK reports for all files that were found, including libraries and binaries and scripts user by heads, as seen under

Check bzImage

Unfortunately, this requires CircleCI bzImage since mathing can only be made against decompressed binary.

After our binwalk --extract command:

user@heads-backup-extraction:/tmp/inspection/payload_content$ cd _payload.extracted/
user@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ ls
2C9B80.xz  2E531F  2E531F.xz  32D5  32D5.xz
user@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ wget -O extract-vmlinux https://raw.githubusercontent.com/torvalds/linux/master/scripts/extract-vmlinux
user@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ chmod +x ./extract-vmlinux 
user@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ ls -al
total 47316
drwxr-xr-x 2 user user     4096 Nov 25 12:34 .
drwxr-xr-x 3 user user     4096 Nov 25 12:22 ..
-rw-r--r-- 1 user user  4405151 Nov 25 12:22 2C9B80.xz
-rw-r--r-- 1 user user 12882944 Nov 25 12:22 2E531F
-rw-r--r-- 1 user user  4292608 Nov 25 12:22 2E531F.xz
-rw-r--r-- 1 user user 19528104 Nov 25 12:22 32D5
-rw-r--r-- 1 user user  7315530 Nov 25 12:22 32D5.xz
-rwxr-xr-x 1 user user     1695 Nov 25 12:34 extract-vmlinux

user@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ ./extract-vmlinux 32D5 > vmlinux_local

Download bzImage from CircleCI artifact

user@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ ./extract-vmlinux bzImage > vmlinux_remote
user@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ diff vmlinux_local vmlinux_remote
user@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$

[newbieAtGithub]

hi @tlaurion

I would advise into asking specific questions in the issue I created as raw notes for integrity validation, which hopefully will result into an additional wiki page to answer all those questions.

okay, since we want to create additional wiki page,
then i will make conclusion here, from #109 #110 #111 #112 #113 #114

All commits are signed, before being pushed into repository,
then being reviewed, before merging, & tested by regression testing,
This is how to distrust the infrastructures & ensure integrity,
across network, repository, Circle CI, local compile machine, & flashing tool.

2 ways, it verifies authenticity & integrity, prior compiling:

• commit signed, before being pushed into repository,
• git status will notify, if Heads' code base, related to specific commit, is modified.
  Name of generated ROM, will indicate, whether code base is clean or not.

If Heads' code base, related to specific commit, is modified, prior compiling,
either Circle CI compiling, or Local compiling, then it will produce ROM,
& "-dirty" will be appended to the ROM name, to notify user that the ROM build is not clean,
otherwise, if ROM name contains no "-dirty", then we can assume it is clean.

Commit ID will be appended to the ROM name,
as an information, that the ROM was built, by using what commit.

hashes,txt containing hashes of built files & ROM,
will be generated after built,
& can be used to verify integrity, for 3 conditions:

• after local compiling
• after Circle CI compiling & download
• after flashing & backup (external / internal flashing & external / internal backup)

hashes.txt generated, after built, will only contain hashes for built files & ROM,
it will not contain any hashes for any files before built.

Currently, Heads has reproducibility issues, which mean,
using different machine, to build / compile Heads,
i.e. using Circle CI or Debian laptop or docker,
will produce ROM with different hash / checksum.

hashes.txt will show individual hashes, of everything packed in the firmware, and also the firmware image itself.
If final hash of final ROM image is same, it means all parts inside are good.

Compressed under initrd.cpio.xz:

• heads.cpio contains all the scripts and configs under initrd dir of github
• tools.cpio contains all the compiled binaries and libraries
• modules.cpio contains all kernel modules that were packed for a board
  Coreboot does its stitching and includes the kernel and initrd as its payload.

hashes.txt under ./blobs/ section, are different from hashes.txt, generated during compiling:

• hashes.txt inside ./blobs/ contains hashes for blobs (me, gbe, ifd) , i.e. xx20, hashes.txt
• hashes.txt generated during compiling, contains hashes of files & ROM, that are produced by compiling.
• hashes.txt generated during compiling, will contain no hashes for blobs (ME, GBE, IFD).

Blobs (ME, GBE, IFD) will have different hashes, they are unique for each laptop:

• me.bin has different version, therefore has different hash
• gbe.bin has different MAC address, therefore has different hash
• ifd.bin contains information from motherboard, there has different hash

By using blobs (ME, GBE, IFD) from Heads repository,
then we will have same blobs & its hashes, & also same MAC address, as other Heads' user.
But using your own blobs, will have unique MAC address & motherboard info, etc.

Flashed Heads ROM, will contain MRC cache, GPG public key ring, & config.user override.
Therefore, while backup-ed, will have different hash checksum,
so need to extract / unpack 1st, then validate extracted component with hashes.txt, of same commit.

Re-flashing / Updating Heads externally, provide more integrity, than internally re-flashing / updating,
but for most cases, is not necessary. Internal flashing rely on internal flashrom tool,
in case of distrusting the internal flashrom tool, then can do external re-flashing.
Re-flashing externally will invalidate measurements, since the ROM won't contain the same GPG key ring.
The risk of flashrom being compromised is low, but possible.

Internal flashing, rely on internal flashrom tool.
sha256sum -c is a subprogram of busybox, to check digest integrity.

After re-flashing, check integrity internally, by using sha256sum of internal busybox,
and take backup internally / externally, & unpack / extract for external re-check.

[tlaurion]

From top post

After our binwalk --extract command:\n\nuser@heads-backup-extraction:/tmp/inspection/payload_content$ cd _payload.extracted/\nuser@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ ls\n2C9B80.xz 2E531F 2E531F.xz 32D5 32D5.xz\nuser@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ wget -O extract-vmlinux https://raw.githubusercontent.com/torvalds/linux/master/scripts/extract-vmlinux\nuser@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ chmod +x ./extract-vmlinux \nuser@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ ls -al\ntotal 47316\ndrwxr-xr-x 2 user user 4096 Nov 25 12:34 .\ndrwxr-xr-x 3 user user 4096 Nov 25 12:22 ..\n-rw-r--r-- 1 user user 4405151 Nov 25 12:22 2C9B80.xz\n-rw-r--r-- 1 user user 12882944 Nov 25 12:22 2E531F\n-rw-r--r-- 1 user user 4292608 Nov 25 12:22 2E531F.xz\n-rw-r--r-- 1 user user 19528104 Nov 25 12:22 32D5\n-rw-r--r-- 1 user user 7315530 Nov 25 12:22 32D5.xz\n-rwxr-xr-x 1 user user 1695 Nov 25 12:34 extract-vmlinux\n\nuser@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ ./extract-vmlinux 32D5 > vmlinux_local\n

This is coming up again and again and again: how to (reasonably) trust USB Security Dongles's flashing green light (reverse HOTP, Heads compatible dongle) / TOTP matching scanned Qr code at TOTP sealing / that TPM Disk Unlock Key passphrase isn't typed/captured in/from compromised environment

Note that here, cbmem -L under recovery shell will output coreboot measurements (event log) for each measured boot part which encompasses bootblock up to payload.
The technical coreboot doc is here https://doc.coreboot.org/security/vboot/measured_boot.html

Coreboot creates an event log that can be audited through cbmem log and specifically through cbmem -L. But that is produced/reported by the same authentic/potentially compromised stages which could be crafted to report fake measurements (theoretically, not practically proven as of now, but still...)

And again, payload being measured and sealed, by Heads, into TOTP/HOTP secret, takes care of the "meta measures" sealed from PCR 0-4 into a secret (TOTP) that can be unlocked only if the same sealed/unsealed PCR measurements are as intended (matching event log). But how to trust that the coreboot phases are not lying on recalculated reported values? We cannot reasonably trust bootblock measurement from hypothetical possible attacks.

Then Heads extends those even log measurements with runtime information, including what is seen here as "user stuff" early at boot : key material, config user etc, where going to recovery shell also modifying tpm PCRs and preventing unsealing of secrets.

Same for TPM DUK, which add up to what is measured and sealed. Including loaded on demand kernel modules and LUKS header.
All documented at https://osresearch.net/Keys/#tpm-pcrs

To see those in action, one would have to activate debug/tracing in recent ROM images from config settings menu and save in rom, reseal, signing and follow onscreen instructions to reseal TPM DUK if that feature is activated when setting a default boot configuration.

But instructions on top post permits to validate things only manually. Could we automate another layer of verification that would require other layers to be compromised as well for an attack to succeed? Yes, we could.

Heads could add an additional check.
bootblock could be extracted (cbfs tool from flashtool) and hashed in ram(sha256sum from busybox) and be part of detached signatures (gpg tool from gnupg) under /boot as additional crosscheck, since the paranoia going on, untrusting bootblock not faking measurements for itself and next stages is still present... And bootblock being our root of trust in lack of an hardware root of trust.

Alternative

• set bootblock in read only in SPI flash, but bootblock changes across coreboot version and is not adapted to Heads workflow. See https://docs.dasharo.com/variants/asus_kgpe_d16/spi-wp/

So we could extract bootblock (cbfs being compiled from modules/flashtools not coreboot) and hash it in memory (modules/busybox sha256sum) and have those hashes as part of the detached signed digest signed and verified by gpg (modules/gnupg2).

We would add an additional delay to verify bootblock prior of trusting Heads and coreboot measured boot alone.

Recap.
Measured boot from coreboot starts by bootblock enabling tpm. And then having bootblock measuring itself and reporting its own measurement to extend TPM PCR 2.
And then bootblock measures next stage, extends PCR 2 again, and so on with other stages up to payload(heads kernel+initrd) .
So compromising bootblock would definitely changes its checksum. And not trusting bootblock measurement through coreboot measured boot alone could be an improvement, considering that not only coreboot would need to be compromised but also Heads.

Going back in previous notes of this thread, not only cbfs would need to be tampered with in cbfs, but payload as well, to compromise also busybox, flashtools and gnupg2 modules which hashes would change (output at build time not yet verified on boot path but could also be done!)

Thoughts?

[tlaurion]

Should be considered under #62

[marmarek]

What exactly is the threat model here to defend against? Somebody modifying bootblock to lie about its measurement (and also measurements of next stage)? Modifying how? From recovery shell, flashrom from within the OS, external programmer?

Saving bootblock separately, to be dumped into /boot and signed makes such attack only slightly more complicated - instead of just saving "proper" hash, attacker would need to keep also the original bootblock, or at least parts that were changed. But since the attacker would modify coreboot payload anyway (after all, that's where DUK could be captured, not in the bootblock itself), faking also bootblock verification there would be quite simple (for example, modify cbfs to dump saved copy instead of the actual bootblock). Without hardware root of trust, it's hard to prevent such attack completely. One could detect it with external programmer by dumping flash content and comparing, but it isn't practical for everyday use.

One thing that could raise the bar for such attack is to write-protect bootblock in SPI flash. If done properly, it would force the attacker to use external programmer, which takes significantly more time (depending on hardware, sometimes just few screws, sometimes disassembly of most of the laptop). Case opened can be detected, for example with glitter on screws. Then updates would be more complicated, but that could be solved with either/and:

• forcing TPM reset before granting write access(*) to that SPI flash area
• have extra authentication (via the USB dongle, like signing some message?) before granting write access to that SPI flash area
• adding signature check on update, done by the old firmware

The last one is IMO desirable anyway, but it's a lot of work too. And there still needs to be some other update method, for example when switching between different build origins (signed with different keys).

(*) I write above "grant write access", but depending on technical details it could also mean "not protecting the area" under some conditions (as in: may require reboot into special mode, instead of reversing protection).

[krystian-hebel]

The technical coreboot doc is here https://doc.coreboot.org/security/vboot/measured_boot.html

This still could use some rework, I didn't do a good job reviewing it earlier. For example:

PCR-2 - Hash of Root of Trust for Measurement which includes all stages, data and blobs.

There is always only one entity that is the Root of Trust (although there may be more than one root/chain). It is implicitly (usually, unfortunately) trusted. It can be in hardware (Intel Boot Guard, AMD Hardware Validated Boot or whatever they're called today) or in software (initial part of firmware, e.g. coreboot bootblock). It is generally believed that hardware root of trust is safer, however in practice "hardware" RTM is usually just a software running on another chip, hopefully protected at least by RTV. "All stages, data and blobs" cannot be all part of root, otherwise you wouldn't have to measure anything, since by definition you trust the root.

theoretically, not practically proven as of now, but still...

Don't tempt me 🙂

And again, payload being measured and sealed, by Heads, into TOTP/HOTP secret, takes care of the "meta measures" sealed from PCR 0-4 into a secret (TOTP) that can be unlocked only if the same sealed/unsealed PCR measurements are as intended (matching event log).

This seems a bit backwards. Event log is only informational, it can't be trusted unless PCR values are correct (assuming that the chain of trust wasn't broken at any point since the respective root was obtained). One improvement I can think of is to somehow convey the expected PCR values (through a different channel than the binary) and compare them with actual values. They are long, checking them manually would be prone to errors, but it is still hard to get similar, but not identical hash so maybe this would work in practice. Perhaps a mobile validator app could scan the QR code and compare that to make it easier for users, but I don't know if all important PCR values would fit on it, even in binary form.

But how to trust that the coreboot phases are not lying on recalculated reported values? We cannot reasonably trust bootblock measurement from hypothetical possible attacks.

This is why you have to implicitly trust the root. There are ways to make it more trustworthy, e.g. by physically making bootblock read-only with WP pin or by using hardware RoT. In any case, you still trust that those mechanisms actually work as advertised, that they can resist attacks, that the underlying math isn't broken. If the root is broken, whole chain is. Adding more links to that chain won't improve its security, it can only make it worse.

Heads could add an additional check.

If Heads is started by compromised earlier stages, how can you trust it to be the proper Heads, without this check tampered with?

[tlaurion]

@marmarek @krystian-hebel : Decided to go with WiP/Poc code to show what is attempted to be done here at linuxboot/heads#1568 (comment)

[tlaurion]

@marmarek @krystian-hebel : PoC and output, leading base to measured boot introspection and paving the way to forward sealing at linuxboot/heads#1568 (comment) and next comment

[tlaurion]

Flashkeeper project accepted by NLnet : https://nlnet.nl/project/Flashkeeper/

Finishing MoU, will keep you all posted with details, and will present at QuebesOS mini-summit 2024.