Try to reduce memory usage during sysimg build#50237
Conversation
|
Cross-compiling from 64bit to 32bit on Linux currently OOMs for me on master. |
|
Note that we can't merge this until the windows builders are back online, there is still an unfixed win32 bug somewhere. |
|
Embedding test failed due to: |
vchuravy
force-pushed
the
pc/32bit-memory
branch
from
a7d2065 to
2c7daf4
Compare
pchintalapudi
force-pushed
the
pc/32bit-memory
branch
from
2c7daf4 to
9e5aba6
Compare
|
@vchuravy could you confirm the memory benefits on the new implementation? |
|
This PR fails to build for me, after VIRT memory usage crosses 4GB. So I guess not. |
|
That's odd, for me I'm not seeing it cross 3.7 virtual/3.3 reserved on a 64 bit build (default options, no multiversioning) with 1 thread. |
Resolved offline, peak memory usage varies fairly significantly between runs, but this PR does seem to reduce memory usage overall. |
| sysimgM.setDataLayout(DL); | ||
| sysimgM.setStackProtectorGuard(StackProtectorGuard); | ||
| sysimgM.setOverrideStackAlignment(OverrideStackAlignment); | ||
| Constant *data = ConstantDataArray::get(Context, |
There was a problem hiding this comment.
I think you also now could move the call to jl_create_system_image here, and avoid needing all of that extra memory (for z) until after LLVM is completely done and after we've freed most of the llvm::Module objects.
There was a problem hiding this comment.
I suspect that change is more complicated, since we call jl_create_native as part of jl_create_system_image, and then I think we do a few more transformations on the module while generating z.
There was a problem hiding this comment.
Yeah, but in that case the change might make more sense too, since the current organization sounds complicated also
4 participants
At least in the past, the most memory hungry step was emitting the system image to the object file. This is likely due to there being 3 or more copies of the sysimage live at that time, specifically:
This PR tries to reduce the number of live copies to 2, by incrementally copying to the next stage of code generation and then explicitly freeing the previous stage's resources.