Unable to merge bottle json

[wathenj]

Hello,

I have a project that uses homebrew to package our python application, and we've been getting an error for a while now that I'm trying to resolve. When we brew bottle --json our project, make changes (or not) to the json, then attempt to --merge the json back to the formula, we get the following error:

Error: undefined method `path' for nil:NilClass
Did you mean?  paths
/opt/homebrew/Library/Homebrew/formula_versions.rb:25:in `initialize'
/opt/homebrew/Library/Homebrew/dev-cmd/bottle.rb:614:in `new'
/opt/homebrew/Library/Homebrew/dev-cmd/bottle.rb:614:in `block in merge'
/opt/homebrew/Library/Homebrew/dev-cmd/bottle.rb:602:in `each'
/opt/homebrew/Library/Homebrew/dev-cmd/bottle.rb:602:in `merge'
/opt/homebrew/Library/Homebrew/dev-cmd/bottle.rb:100:in `bottle'
/opt/homebrew/Library/Homebrew/brew.rb:95:in `<main>'

Checking out and playing with formula_versions.rb, it seems like tap isn't being initialized, therefore @repository = formula.tap.path generates that error.

I don't believe we're missing anything in our formula, but I'm otherwise not sure what might resolve this. Any ideas?

We don't use formula revisions so I'm considering commenting out lines 614-616 in dev-cmd/bottle.rb to skip those checks any time we need to package as a workaround.

FormulaVersions.new(formula).formula_at_revision("origin/HEAD") do |upstream_f|
    old_pkg_version = upstream_f.pkg_version
end

Thanks

[Bo98]

I suspect this might happen if you're passing a file path of the formula to brew bottle rather than some/tap/formula. Other than that, I'm not sure.

Do you have more information on how you are using brew bottle and how I might be able to reproduce the issue you are having?

[wathenj]

Sure,

The relevant subset of commands we're running are as follows:

brew uninstall --force <name>
brew install ./Formula/<name>.rb --build-bottle --formula
brew bottle ./Formula/<name>.rb --force-core-tap --no-rebuild --json
brew bottle ./<name>--<version>.arm64_monterey.bottle.json --merge --write --no-commit

I was previously building on an old mojave Mac, and I'm still getting the same error here so the platform shouldn't be relevant.
Everything is fine until the --merge. Not exactly helpful, but this process used to work until maybe March of last year. Not 100% on that.

The formula follows the following structure:

class .... < Formula
  include Language::Python::Virtualenv

  desc "...."
  homepage "...."
  url "...."
  sha256 "...."
  head "....git", branch: "develop"

  depends_on "rust" => :build
  depends_on "python@3.9"

  # --- Begin Python Dependencies ---

  resource "....." do
    url "....."
    sha256 "....."
  end

  ...

  # --- End Python Dependencies ---

  def install
    venv = virtualenv_create(libexec, Formula["python@3.9"].opt_bin/"python3.9")
    libexec.install("pip.conf")
    venv.pip_install resources
    venv.pip_install [buildpath]
    bin.install libexec/"bin/..."
    man1.install libexec/"share/man/man1/...1.gz"
    man5.install libexec/"share/man/man5/....5.gz"
    bash_completion.install libexec/"share/bash-completion/completions/...."
    zsh_completion.install libexec/"share/zsh/vendor-completions/_...."
  end

  def caveats
    <<~EOS
      ...
    EOS
  end

  test do
    system bin/"...", "-v"
  end
end

[wathenj]

Based on your suggestion I tried using the package name instead of the formula file path and that definitely runs and updates the tap formula file, but we're looking for the functionality of updating the local dir formula file with an updated bottle do stanza.

[Bo98]

If your local file is located at /opt/homebrew/Library/Taps/.../homebrew-.../Formula/<name>.rb then it should work. It does not need to be available remotely on the tap repository so you don't need to worry about pushing anything until you are ready. That directory is ultimately just a regular git clone of your repository.

If your working directory cannot be in that location, symlinking the tap directory might work.

If neither of these are appropriate for you, let me know why and I can see what else can be done.

Documentation states the --force-core-tap flag is to Build a bottle even if formula is not in homebrew/core or any installed taps. Is that wording inaccurate or did you mean using it says the formula is NOT from homebrew-core?

It's accurate in that it allows that scenario to work, but it doesn't detail how it achieves that: you will see the tap in the JSON etc is recorded as "Homebrew/core" even if that may not be ultimately true. I'm thinking we could replace that with something else that allows you to specify the tap, but I would need to know a use case and why the existing setups won't fit, particularly given we've not had this request from other tap maintainers (and you're the first to report --force-core-tap as broken).

[wathenj]

I moved the generated .rb to the Taps path and everything bottled correctly as far as I can tell! Thanks!
Unfortunately, I tried installing from the generated bottle and got this. I'll have to investigate that myself tomorrow, but I appreciate the help so far!

brew install -v ./dist/<name>-<version>.arm64_monterey.bottle.tar.gz
==> Pouring <name>-<version>.arm64_monterey.bottle.tar.gz
tar --extract --no-same-owner --file /Users/.../dist/<name>-<version>.arm64_monterey.bottle.tar.gz --directory /private/tmp/d20221016-79814-137ygfj
cp -pR /private/tmp/d20221016-79814-137ygfj/<name>/. /opt/homebrew/Cellar/<name>
chmod -Rf +w /private/tmp/d20221016-79814-137ygfj
Error: new ID must be a String
/opt/homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/ruby-macho-3.0.0/lib/macho/macho_file.rb:318:in `change_dylib_id'
/opt/homebrew/Library/Homebrew/os/mac/mach.rb:73:in `change_dylib_id'
/opt/homebrew/Library/Homebrew/os/mac/keg.rb:10:in `change_dylib_id'
/opt/homebrew/Library/Homebrew/extend/os/mac/keg_relocate.rb:26:in `block (2 levels) in relocate_dynamic_linkage'
/opt/homebrew/Library/Homebrew/extend/pathname.rb:341:in `ensure_writable'
/opt/homebrew/Library/Homebrew/extend/os/mac/keg_relocate.rb:23:in `block in relocate_dynamic_linkage'
/opt/homebrew/Library/Homebrew/extend/os/mac/keg_relocate.rb:22:in `each'
/opt/homebrew/Library/Homebrew/extend/os/mac/keg_relocate.rb:22:in `relocate_dynamic_linkage'
/opt/homebrew/Library/Homebrew/keg_relocate.rb:132:in `replace_placeholders_with_locations'
/opt/homebrew/Library/Homebrew/formula_installer.rb:1220:in `pour'
/opt/homebrew/Library/Homebrew/formula_installer.rb:419:in `install'
/opt/homebrew/Library/Homebrew/upgrade.rb:203:in `install_formula'
/opt/homebrew/Library/Homebrew/install.rb:350:in `install_formula'
/opt/homebrew/Library/Homebrew/install.rb:340:in `block in install_formulae'
/opt/homebrew/Library/Homebrew/install.rb:339:in `each'
/opt/homebrew/Library/Homebrew/install.rb:339:in `install_formulae'
/opt/homebrew/Library/Homebrew/cmd/install.rb:231:in `install'
/opt/homebrew/Library/Homebrew/brew.rb:95:in `<main>'

[wathenj]

Just a quick update that I moved the formula.rb to the correct Tap path and tried bottling using brew bottle <name> which gave me that error. I also just tried out brew bottle <the full path to the tap formula.rb> and installing the bottle that was created seems to work without generating that error. I'll need to run through this whole process later on from scratch, but I think everything is working as needed for now!

[carlocab]

Note that passing full paths to formula files is a long-deprecated API. It might work for now, but it can break at any point without notice. (It's also already broken for some things.)

[wathenj]

Alright, thanks for the info. I'll try to find more documentation about proper bottling etc since we're clearly not following correct practices. Are there any good references or discussions I could look over?