Explain git commit signed commits and limitations (was Requesting Heads implement a signed release the way coreboot does: not planned)

[loftlifter31]

Is your feature request related to a problem? Please describe.
Github commits don't support user verifiable cryptographic signatures. A Heads user downloading the latest commit is trusting solely in TLS for download security. TLS as built on the current CA system is fatally flawed. CAs have been found to improperly issue certificates for domains they are not authorized for, issue certificates to untrustworthy servers, and to have lost control of their signing keys, as well as have multiple security failures. Certificate pinning can resolve some but not all of the issues underlying the CA dependent TLS security model.

Due to the broken nature of the TLS security model, a user downloading heads from a github commit is open to MITM attack where a malicious 3rd party can intercept the connection and impersonate github.com using a fraudulent certificate delivering a tampered version of the source code to introduce a backdoor or other vulnerability. The user has no way to verify the integrity of the download.

Describe the solution you'd like
Follow coreboot's example and issue releases with user verifiable pgp signature.

Describe alternatives you've considered
Could be as simple as once every couple of years upload the latest stable release of heads with a detached signature. Or as often as you like. Quarterly maybe. You could add a script to CircleCI to automatically sign builds using a private key local to that machine for nightly builds and for more major releases sign with the key of one of the principal developers. This adds an extra layer of security which would improve the overall security without much overhead or effort.

Additional context
Add any other context or screenshots about the feature request here.

[tlaurion]

You could add a script to CircleCI to automatically sign builds using a private key local to that machine for nightly builds and for more major releases sign with the key of one of the principal developers.

Unfortunately not possible unless CircleCI has a private key to sign such trusted thing which wouldn't make it more trustable, key/secret leaks happen everyday and CI also made news leaking secrets.

Heads is rolling release based on reproducible build ideology. If one single line of code built was different, the rom produced would be different. What is missing today is the opposite of what you propose here @loftlifter31, based on the "distrusting infrastructures" principle: We miss a CircleCI orb that would ouput hashes of each produced ROM, just like embedded currently under zip files from CircleCI artififacts, that if unmatches would make the infrastructure distrusted (and what we currently wait for), where hashes.txt in build dir says where binary failed to be reproducible to help figure what went wrong.

Since #1661, every build for each PR/master made is reproducible, from local/CircleCI.

We could release code as tarballs as releases, but we couldn't release roms, since they contain blobs which we aren't permitted to redistribute. We can only redistribute blobs download scripts, which downloads and extract blobs and build reproducible roms. That was a 4 years ago and accomplished under #703. If we were to distribute roms as release, we might get the project shutdown without notice. This is legal grey area I won't play and worked really hard to get away of that redistribution issue doing exactly what is done today: a rolling release with CI producing ROMs, available for 30 days. And #1661 finally fixed the issue of being able to reproduce the same rom hashes for a commit forever, with the problem left of having mirrors that won't become inaccessible over time (seperate issue) to download tarballs which hashes are verified prior of decompression anyway, which satisfies also your requirements if I understand your concern well.

Quaterly signed releases: Who would sign them? coreboot releases are TBH, kinda random in terms of stability historically and picked/post-poned with now releases not being versioned but date based. I would sign code tarballs? Above my pay grade as well unfortunately. :(

Current model is downstream doing releases (Nitrokey/novacustom/Purism/others) choosing an upstream commit they tested well enough to decide that release meets their requirements in term of support, and challenges upstream in PR prior of those PR being merged, and test the CircleCI/local builds themselves and have their own release cycles.

That will be subject of my talk https://cfp.3mdeb.com/qubes-os-summit-2024/talk/HL9MSV/ sunday Sept 22 2024, and I will take this into account into the discussions. That will be filmed and will be available live and after the fact, if you want to be there and be part of the discussions as well.

Also see #1793 OP, which will also be covered in that talk/discussion to find the best way forward.

---

The alternative desired solution: If someone has experience and time to implement CircleCI orb to comment on PR upon successful builds and output direct links to zip and external roms URLS and final hashes of roms: that would be awesome.

I will tag this with Help wanted @loftlifter31 without renaming the issue until I feel you come to a common understanding here.

Side note: Libreboot borrowed Heads logic of blobs download script+extraction and changed their policies to minimal blobs policy; but they do not provide full roms, leaving it as an exercice to do by the end user; producing half-backed roms, left to be inected with blobs by the end user; and they do not try to fill the reproducible builds requirements AFAIK. Same for Skulls, same for every FOSS firmware projects I know, and same for coreboot not providing roms either. Signed tarballs? Why if we have reproducible builds and users can raise issues if there is a reproducible build issue? Not reproducible? You are targeted/Ci was targeted.

---

If what you want, still, is signed releases of tarballs, then that would "release of sourcecode could be done under a orb too, but I would have to download them all, and detach signed them myself and then add detached signatures myself: after the fact. There is no way I would have a CI have a private key of some sort, and CI history shows secrets having leaked in the past. This is not a question of if but when, again, and I would not go in that direction with public infrastructures we use (and kind of abuse their free tier with Heads).

And no, I checked: there is no way of interrupting a CircleCI build and login through SSH unless the build fails to pass scdaemon + socket back using local GPG keyring+ smartcard to safely do this. I can only log in through ssh into a build when builds fail, and those builds will never be marked as successful. If CircleCI was permitting otherwise, that would be a security risk for build tampering as well.

---

TLDR: Heads is rolling release backed with reproducible builds principles. If a user arrive to a different hash then what was built by CircleCI for same commit: that is a good reason to open an issue, today and since #1661.

If retrying to build a old commit, the end user is left to check which reproducible docker image was used under CircleCI config file for that build. If the user wants to recreate the docker image reproducibly, he has to go back in time and check from which commit that docker image was build, since there is still uninvestigated strings embedded in the docker image that are related to the commit that was used to build that docker image reproducibly for the commit id.

Otherwise, latest docker image will always permit to build master and PRs reproducibly, where CircleCI PINs to versioned docker images, not latest, for reproducible builds reasons.

---

@loftlifter31 you can close this issue and open another one if that satisfies your curiosity on the reasons why linuxboot/heads won't provide releases, even less sign them, anytime soon.

Meanwhile: Thanks for challengin ideas/code and your contributions, well appreciated.

[loftlifter31]

@tlaurion Let's leave aside producing full roms with proprietary code blobs as that was not my request. I'm not sure if I'm understanding you correctly, but it sounds like you have 2 reasons: 1) You have a reproducible build system which you think is superior to signed releases. 2) Signing releases is "above your pay grade".

Responding to point 1, let me see if I understand this correctly. The idea is that a user will build locally using master and compare hashes for the build artifacts to hashes for the latest CircleCI build. If both hashes match, there was no tampering with the source code download. Is that right?

But if we assume a MITM attacker, why wouldn't the attacker just modify the CircleCI hashes in transit to match whatever the tainted code they injected will build as? Signed releases avoid this issue because the key used to sign the realease can be signed by other trusted parties in the web of trust. Also the signing key would only be downloaded once, possibly months or years in advance of the download of the signed code. An MITM attacker is highly unlikely to maintain persistent MITM access for months or years.

Regarding point 2, I'm afraid I don't understand this. "Above my pay grade" is military slang for division of responsibilities. An O-3 can sign off on things an O-2 can't. I'm not clear on how this relates to our current discussion. Is there someone "above" you in a heirarchy who would be permitted to sign code where you can't?

[tlaurion]

@tlaurion Let's leave aside producing full roms with proprietary code blobs as that was not my request. I'm not sure if I'm understanding you correctly, but it sounds like you have 2 reasons: 1) You have a reproducible build system which you think is superior to signed releases. 2) Signing releases is "above your pay grade".

Responding to point 1, let me see if I understand this correctly. The idea is that a user will build locally using master and compare hashes for the build artifacts to hashes for the latest CircleCI build. If both hashes match, there was no tampering with the source code download. Is that right?

Yes.

But if we assume a MITM attacker, why wouldn't the attacker just modify the CircleCI hashes in transit to match whatever the tainted code they injected will build as? Signed releases avoid this issue because the key used to sign the realease can be signed by other trusted parties in the web of trust. Also the signing key would only be downloaded once, possibly months or years in advance of the download of the signed code. An MITM attacker is highly unlikely to maintain persistent MITM access for months or years.

I don't understand your concern with git. Signed commits + TLS is enough.
Please do git log --show-signature from command line after having cloned repo.

Signed release is a way of distribution, and coreboot takes point in time and create a tarball, yes. But git is far better at giving origin of changes and better integrity+authenticity of changes through git that is, not tarballs downloads. Signed tarballs permits to make sure that the release was signed, yes, but nothing more.

Regarding point 2, I'm afraid I don't understand this. "Above my pay grade" is military slang for division of responsibilities. An O-3 can sign off on things an O-2 can't. I'm not clear on how this relates to our current discussion. Is there someone "above" you in a heirarchy who would be permitted to sign code where you can't?

No. That simply means that I invested a lot of time and resource into making builds reproducibles and following best practices for code signing and PR reviews, where tarballs would encourage users to use that instead of git. I want users to use git, use that reproducible docker image that CircleCI uses (see circleci config to see versioned image) and verify that buikds are reproducibles.

Signed tarballs would be inferior, yes. I do not plan to do this. If there is anything still unclear on the whys, please quote and reply. When I say above my pay grade, its because we don't want CI to have private keys (that could leak). I do not intend to provision private keys, and doing releases would just be point in time. I plan to do branches, though, and feature freezes.

More details in QubesOS mini-summit discussion at https://youtu.be/mAb_kHrF6SQ?list=PLuISieMwVBpL5S7kPUHKenoFj_YJ8Y0_d

[loftlifter31]

Are you serious? The git log shows me there are more than 24 code signers on this project and not all of them even use the same keyserver to host thier keys. I spent a couple hours today hunting down and downloading as many keys as I could find and still couldn't get them all. And you expect users to review the entire git log for every commit to make sure the signature is valid? That is a huge investment in time. Users might as well audit every line of code personally while they are at it. And then we find little gems like this:

commit d6ef65c
gpg: Signature made Thu 26 Sep 2024 07:59:42 AM CDT
gpg: using RSA key 8735540225E98BDBC82491B41E9C3CA91AE25114
gpg: issuer "jonathon.hall@puri.sm"
gpg: Good signature from "Jonathon Hall jonathon.hall@puri.sm" [expired]
gpg: Note: This key has expired!
Primary key fingerprint: 8735 5402 25E9 8BDB C824 91B4 1E9C 3CA9 1AE2 5114
Author: Jonathon Hall jonathon.hall@puri.sm
Date: Thu Sep 26 08:59:01 2024 -0400

bin/fetch_source_archive.sh: Add storage.puri.st mirror

storage.puri.st is an alternate host name for storage.puri.sm, in case
there is another issue with the .sm name registration.

Signed-off-by: Jonathon Hall <jonathon.hall@puri.sm>

Seems several commits have been signed with an expired key. Should the code be trusted? A coreboot user can download the latest tarball and verify 1 signature for code integrity up to present date. A heads user has do hours of research and downloading to accomplish the same task. If you are so elitist you don't care about making code verification easy for users, that is your choice, but you are also creating an attack vector for users who can't be bothered to jump through all of these hoops required to verify code and simply trust what they are presented.

[tlaurion]

Are you serious? The git log shows me there are more than 24 code signers on this project and not all of them even use the same keyserver to host thier keys. I spent a couple hours today hunting down and downloading as many keys as I could find and still couldn't get them all. And you expect users to review the entire git log for every commit to make sure the signature is valid? That is a huge investment in time. Users might as well audit every line of code personally while they are at it. And then we find little gems like this:

commit d6ef65c
gpg: Signature made Thu 26 Sep 2024 07:59:42 AM CDT
gpg: using RSA key 8735540225E98BDBC82491B41E9C3CA91AE25114
gpg: issuer "jonathon.hall@puri.sm"
gpg: Good signature from "Jonathon Hall jonathon.hall@puri.sm" [expired]
gpg: Note: This key has expired!
Primary key fingerprint: 8735 5402 25E9 8BDB C824 91B4 1E9C 3CA9 1AE2 5114
Author: Jonathon Hall jonathon.hall@puri.sm
Date: Thu Sep 26 08:59:01 2024 -0400

bin/fetch_source_archive.sh: Add storage.puri.st mirror

storage.puri.st is an alternate host name for storage.puri.sm, in case
there is another issue with the .sm name registration.

Signed-off-by: Jonathon Hall <jonathon.hall@puri.sm>

Seems several commits have been signed with an expired key. Should the code be trusted? A coreboot user can download the latest tarball and verify 1 signature for code integrity up to present date. A heads user has do hours of research and downloading to accomplish the same task. If you are so elitist you don't care about making code verification easy for users, that is your choice, but you are also creating an attack vector for users who can't be bothered to jump through all of these hoops required to verify code and simply trust what they are presented.

@JonathonHall-Purism seems like you have to extend your public key expiration date and upload it back to github.

@loftlifter31 I don't know how to handle the rudeness in your tone.

Heads depends on modules/* which enforces tarballs checksum validation with pinned versions and known good hashes.

Otherwise, scripts in codebase under initrd/* all were part of github pull requests, merged by 3 maintainers in the past. @osresearch @tlaurion and @JonathonHall-Purism.

If I was to detach sign a branch, turned into a version (data tag Ala coreboot), and that being signed with my key, would that make you more confortable? Why? You say circle of trust, but would you sign it? After which verification? Why would you trust it more then git today where all PRs were scrutinized prior of merging? I'm not sure how to handle this issue or explain better the reasoning here.

I merge pull request after verification. @JonathonHall-Purism approves my PRs prior of merging/let me merge it. There were moments where Bugfixes needed to be merged fast because regressions were introduced (see bug tag) which still goes through PR, but as maintainer, I chose to merge prior of review because bug was intruduced prior of weekend and bug could result in a brick.

I understand your worries, but I'm not sure I understand why what you propose would change anything about them.

See hashes.txt of Circleci artifacts. Follow reproductivity notes. It's actually the consensual best approach to address your concern: you are sure that all the build process produced the same final hash of rom, where anything changing one byte of either cpio included in initrd (heads.cpio: scripts, modules.cpio: on demand loaded kernel modules, tools.cpio: modules binaries/libraries compiled -> stitches in initrd +kernel stitched in coreboot payload +coreboot) is reproducible. To my opinion, it cannot get better than that because of reproducible builds guarantees.

@JonathonHall-Purism maybe my wording sounds elitist but that is not my intention. If you have more reassuring words or agree that versioned code tarballs + detached signatures would be better (how) then please chime in. This is not dictatorship, just my opinion. As always, I welcome other opinions but my time is limited explaining past choices.

Yes there is collaborators to this project, we are lucky. Their code was proposed through PR, those were signed commits otherwise I created superseeding PRs co-authoring their commits with my gpg key after reviewing the code. I'm not sure I understand your criticism. Looking at coreboot codebase will show some commits that aren't signed either, but that doesn't matter because code review is what matters. This is also git, but you seem to trust this. You would be correct based on "more eyes" premises and you would be right. Same applies to linux kernel. This is a supply chain problem. Only thing that we can do here is what is done under modules/*, and sign commits for authenticity /integrity validation. Yes keys expires. That a reality of gpg when public keys have expiration dates. That doesn't mean the key wasn't expired when it was signed though: just that the key expired since signed.

[JonathonHall-Purism]

commit d6ef65c
gpg: Signature made Thu 26 Sep 2024 07:59:42 AM CDT
gpg: using RSA key 8735540225E98BDBC82491B41E9C3CA91AE25114
gpg: issuer "jonathon.hall@puri.sm"
gpg: Good signature from "Jonathon Hall jonathon.hall@puri.sm" [expired]
gpg: Note: This key has expired!
Primary key fingerprint: 8735 5402 25E9 8BDB C824 91B4 1E9C 3CA9 1AE2 5114
Author: Jonathon Hall jonathon.hall@puri.sm
Date: Thu Sep 26 08:59:01 2024 -0400

I extended this key back in June and uploaded it both to GitHub and keys.puri.sm. Are you still getting the expired key somewhere?

[tlaurion]

commit d6ef65c
gpg: Signature made Thu 26 Sep 2024 07:59:42 AM CDT
gpg: using RSA key 8735540225E98BDBC82491B41E9C3CA91AE25114
gpg: issuer "jonathon.hall@puri.sm"
gpg: Good signature from "Jonathon Hall jonathon.hall@puri.sm" [expired]
gpg: Note: This key has expired!
Primary key fingerprint: 8735 5402 25E9 8BDB C824 91B4 1E9C 3CA9 1AE2 5114
Author: Jonathon Hall jonathon.hall@puri.sm
Date: Thu Sep 26 08:59:01 2024 -0400

I extended this key back in June and uploaded it both to GitHub and keys.puri.sm. Are you still getting the expired key somewhere?

I cannot replicate ffrom GUI (expected). This means @loftlifter31 got it from somewhere where it was not updated redoing

I don't understand your concern with git. Signed commits + TLS is enough.
Please do git log --show-signature from command line after having cloned repo.

[tlaurion]

@JonathonHall-Purism Maybe we could public key in repo for others to use git log --show-signature without friction? And some gpg instructions in a README.md under that directory?

@loftlifter31 would that satisfy your requirement?

[JonathonHall-Purism]

@JonathonHall-Purism seems like you have to extend your public key expiration date and upload it back to github.

I did this back in June. If the old key is still up somewhere, I'd be happy to re-upload (asked above).

@JonathonHall-Purism maybe my wording sounds elitist but that is not my intention. If you have more reassuring words or agree that versioned code tarballs + detached signatures would be better (how) then please chime in. This is not dictatorship, just my opinion. As always, I welcome other opinions but my time is limited explaining past choices.

Signing a commit signs the entire content of the repo at that point in time, not just that change (that's how Git works). If you trust the signer of the last commit (which you'd have to do to trust a signed tarball), you don't have to check all the signatures.

For signing a tarball to have more value, the key(s) used would have to be handled in some way that makes it more trustworthy than either @tlaurion's or my own individual keys. E.g. in some way that requires both us to take action to sign a release, so one of us cannot create a signature on our own. (I'm not going to go down the rabbit hole of how this can be done, there are ways.)

Heads also does not currently have releases. Downstream distributions do (e.g. PureBoot - I run thorough tests for each release to ensure it is functional on all Purism devices, but Heads master will always be ahead as a result.)

While it is certainly possible to address each of those issues, I don't feel that either @tlaurion or I have spare bandwidth available to spend on this implementation at this time for free. If anyone is offering to construct this solution, such that the two of us would approve, generate keys, create releases following some as-yet-undefined criteria, and start signing, I'd be supportive. Or if someone is offering the funding for one or both of our organizations to execute this, I'd be happy to discuss a funded effort.

[JonathonHall-Purism]

@JonathonHall-Purism Maybe we could public key in repo for others to use git log --show-signature without friction? And some gpg instructions in a README.md under that directory?

You'd have a chicken-egg problem. How do you know the key in the repository is my real key? I guess because it's part of a signed commit or tarball. How do you check the commit/tarball signature? I guess with the key in the repository. How do you know that's my real key? [....etc., until heat death of the universe]

Obtaining the key from another source (GitHub, keys.puri.sm, or Purism's GitLab) builds confidence that it is my real key because an imposter would have to control one of those accounts to change it (or all of them).

Of course, those three links I just posted are all still anchored in TLS. But you can distribute that trust by checking all three of them, comparing the certs used, etc. If anybody would like to invite me to a key-signing party, let me know what kind of pizza you will have 😉

[tlaurion]

@JonathonHall-Purism : Well, this led to a small practical experiment, which shows some keys not being easily found publicly. Opened a draft PR for others to see results, showing keys that should be on gpg public servers but aren't and some that were that now aren't.

See #1804

---

@loftlifter31 :

ROMs contain blobs because those are needed to be flashed internally/externally AND tested by the community owning the boards, code reviewed in PR that produces those roms.

There is no release because Heads is the upstream Open Source project, and hosting those blobs (in releases) could have this project shutdown.

Downstream forks do what they feel best to do for their businesses and customers to thrive, and coreboot/forks are the ones dealing with blobs redistribution issues/whatnot, not Heads, which is user of those coreboot forks providing the blobs.

Heads is upstream and tries to stay neutral/dodge legal grey areas as much as possible providing code that will download/extract/generate blobs where needed, and tries to fix bugs as fast as possible with resources available so downstream can deploy in their systems following their release cycles, where downstream also reviews changes between releases and extensively test their releases, based on Heads upstream commit + their rebranding and their own coreboot fork changes.

If you can propose something better, feel free to open a PR proving your point. We will happily review, comment and discuss there. This is why I pointed you to the conferences I just did. You are not the first one asking this. But this is the same answer. This won't happen for before mentioned reasons.

---

@loftlifter31 feel free to close this issue/ comment/tag me wherever needed.

Once again, I didn't intend to shock before.

I'm here to collaborate for the the whole ecosystem benefit, following Pareto principle as much as I can, even more today since downstram releases are multiplying.

What you proposed is high effort for low gains.

I didn't want to sound rude. I restate that what we have today under Heads is way better compared to so many other open source firmware projects. It can be built locally, local scripts changes tested in qemu prior of doing a PR, built from your own Circleci instance if you follow your fork from there, where Ci uses reproducible tagged version docker images and produce the same rom image everywhere if you do so. Anywhere, as long as instructions to do so are read, followed and reproduced. As far as I know, no other project does that nor produce fully working rom images. Brick? Flash last used version externally. It cannot get simpler in my opinion.

Feel free to contribute any improvement you see beneficial to this project and the whole ecosystem.

I'm extensively replying here to you because of your interest for your galp5 system76 platform and the benefit that this could have if we collaborated better. As said in that draft PR, where I tried to show you in code where things should be addressed, the more contributions upstream the better for the whole ecosystem.

A lot of people ask for X or Y. But when asked back to do Z for X or Y to happen, people disappear and work in their own silo. This is not how open source project thrives.

Maintainers then feel they lost their precious time, even if they showed white paws and fixed things along the way, even if X or Y is not delivered yet... because Z never happened.