Skip to content

Use ptrdiff_t sized offsets for gvars_offsets to allow large sysimages #57366

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

gbaraldi
Copy link
Member

Hopefully this fixes JuliaLang/PackageCompiler.jl#1019. This does mean a very slight size increase (each offset is twice as large but shouldn't make too much difference (it's an extra 32bit per gvar))

@gbaraldi gbaraldi requested a review from vtjnash February 11, 2025 21:31
@gbaraldi gbaraldi added backport 1.11 Change should be backported to release-1.11 backport 1.12 Change should be backported to release-1.12 labels Feb 11, 2025
@oscardssmith oscardssmith added the bugfix This change fixes an existing bug label Feb 11, 2025
@topolarity
Copy link
Member

Can you test that this is working before merging?

@test success(`$(Base.julia_cmd()) -J $(dir)/sys.so -e 'Base.scrub_repl_backtrace(nothing); exit()'`)


if !Sys.iswindows() #Windows doesn't support large images
Copy link
Member

@giordano giordano Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this work on 32-bit systems? Edit: ah, you skip the test below? Isn't it simpler to just exclude 32-bit systems together with Windows here?

This was referenced Feb 14, 2025
@gbaraldi gbaraldi added the merge me PR is reviewed. Merge when all tests are passing label Feb 19, 2025
@giordano
Copy link
Member

x86_64 Darwin tests are systematically failing on this PR despite multiple retries.

@giordano
Copy link
Member

This keeps failing: https://buildkite.com/julialang/julia-master/builds/45061#01952ab6-a051-466e-aa0a-49ff9c521962/758-1238

      From worker 6:	lld: error: /private/var/tmp/agent-tempdirs/default-macmini-x64-4.0/tmp/jl_jFNgE1/sys.o.a(text#0.o):(symbol _japi1_reverse_273487+0x47e): relocation SIGNED is out of range: 2402433694 is not in [-2147483648, 2147483647]; references _jl_globalYY.127339
      From worker 6:	lld: error: /private/var/tmp/agent-tempdirs/default-macmini-x64-4.0/tmp/jl_jFNgE1/sys.o.a(text#0.o):(symbol _japi1_reverse_273487+0x40b): relocation SIGNED is out of range: 2402361361 is not in [-2147483648, 2147483647]; references _SUM.CoreDOT.GenericMemoryRefYY.43295
      From worker 6:	lld: error: /private/var/tmp/agent-tempdirs/default-macmini-x64-4.0/tmp/jl_jFNgE1/sys.o.a(text#0.o):(symbol _japi1_reverse_273487+0x399): relocation SIGNED is out of range: 2402375627 is not in [-2147483648, 2147483647]; references _jl_globalYY.214098
      From worker 6:	lld: error: /private/var/tmp/agent-tempdirs/default-macmini-x64-4.0/tmp/jl_jFNgE1/sys.o.a(text#0.o):(symbol _japi1_reverse_273487+0x38c): relocation SIGNED is out of range: 2402376008 is not in [-2147483648, 2147483647]; references _SUM.CoreDOT.TupleYY.50182
      From worker 6:	lld: error: /private/var/tmp/agent-tempdirs/default-macmini-x64-4.0/tmp/jl_jFNgE1/sys.o.a(text#0.o):(symbol _japi1_reverse_273487+0x385): relocation SIGNED is out of range: 2402318159 is not in [-2147483648, 2147483647]; references _jl_globalYY.180619
      From worker 6:	lld: error: /private/var/tmp/agent-tempdirs/default-macmini-x64-4.0/tmp/jl_jFNgE1/sys.o.a(text#0.o):(symbol _japi1_reverse_273487+0x34e): relocation SIGNED is out of range: 2402390694 is not in [-2147483648, 2147483647]; references _jl_globalYY.50155
      From worker 6:	lld: error: /private/var/tmp/agent-tempdirs/default-macmini-x64-4.0/tmp/jl_jFNgE1/sys.o.a(text#0.o):(symbol _japi1_reverse_273487+0x347): relocation SIGNED is out of range: 2402312757 is not in [-2147483648, 2147483647]; references _jl_globalYY.43543
      From worker 6:	lld: error: /private/var/tmp/agent-tempdirs/default-macmini-x64-4.0/tmp/jl_jFNgE1/sys.o.a(text#0.o):(symbol _japi1_reverse_273487+0x227): relocation SIGNED is out of range: 2402445277 is not in [-2147483648, 2147483647]; references _jl_globalYY.73842
      From worker 6:	lld: error: /private/var/tmp/agent-tempdirs/default-macmini-x64-4.0/tmp/jl_jFNgE1/sys.o.a(text#0.o):(symbol _japi1_reverse_273487+0x194): relocation SIGNED is out of range: 2402405512 is not in [-2147483648, 2147483647]; references _SUM.CoreDOT.ArrayYY.45751
      From worker 6:	lld: error: /private/var/tmp/agent-tempdirs/default-macmini-x64-4.0/tmp/jl_jFNgE1/sys.o.a(text#0.o):(symbol _japi1_reverse_273487+0x188): relocation SIGNED is out of range: 2402333196 is not in [-2147483648, 2147483647]; references _jl_globalYY.44727
      From worker 6:	lld: error: /private/var/tmp/agent-tempdirs/default-macmini-x64-4.0/tmp/jl_jFNgE1/sys.o.a(text#0.o):(symbol _japi1_reverse_273487+0x161): relocation SIGNED is out of range: 2402419971 is not in [-2147483648, 2147483647]; references _SUM.CoreDOT.TupleYY.62458
      From worker 6:	lld: error: /private/var/tmp/agent-tempdirs/default-macmini-x64-4.0/tmp/jl_jFNgE1/sys.o.a(text#0.o):(symbol _japi1_reverse_273487+0x14f): relocation SIGNED is out of range: 2402463005 is not in [-2147483648, 2147483647]; references _jl_small_typeof
      From worker 6:	lld: error: /private/var/tmp/agent-tempdirs/default-macmini-x64-4.0/tmp/jl_jFNgE1/sys.o.a(text#0.o):(symbol _japi1_reverse_273487+0x119): relocation SIGNED is out of range: 2402305235 is not in [-2147483648, 2147483647]; references _jl_globalYY.42860
[...]

@KristofferC KristofferC mentioned this pull request Mar 11, 2025
71 tasks
Copy link
Member

@vtjnash vtjnash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are relying on the .ldata section existing to be allowed to use large images like this, but that is an ELF-only feature. Since we use the Small or Medium CodeModels, individual object file relocations will only be able to reference a shared object smaller than about 2GB

@gbaraldi
Copy link
Member Author

I guess we need to detect large images and switch to the large code model. (Maybe printing an info that the image is very large and will have lower performance. I think aarch64 is fine here because there the code model allows for 4GB relocations

@JeffBezanson JeffBezanson removed the merge me PR is reviewed. Merge when all tests are passing label Mar 20, 2025
This was referenced Mar 24, 2025
@KristofferC KristofferC mentioned this pull request Apr 4, 2025
51 tasks
@KristofferC KristofferC mentioned this pull request May 9, 2025
58 tasks
@KristofferC KristofferC mentioned this pull request Jun 6, 2025
60 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 1.11 Change should be backported to release-1.11 backport 1.12 Change should be backported to release-1.12 bugfix This change fixes an existing bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2GB limit to sysimage size?
6 participants