Update v24.5.0#151
Conversation
…x host compilation
…ttings trust domain evaluation
I've encountered an architecture mismatch problem when cross-compiling between an ARM host and x86_64 target, and vice versa. I could confirm, however, that the project compiles successfully for the x64-simulator target on a Mac with an Intel chip and Xcode 16. What I'm still unsure about is whether the Github Actions setup will work in this case - new macos runners run on ARM machines by default and their Intel versions are supposedly available only in paid plans, so I left it on macos-13 with Xcode 15. Ideally, this problem is solved on the project configuration level, not the environment, but my use case doesn't require an Intel simulator and I image that the importance of that target is gradually decreasing, so I decided to go the easy way here. |
@staltz what do you usually test the updates with? I ran a small subset of tests I found suitable for mobile platforms in the |
|
and yes I do use nodejs's mobile node-gyp to rebuild the full error on android is: |
|
Not that i can still what your problem is, but to build native modules against libnode i had to write a script (via zx) that had code like this. Although some of this has been around since i used npm and nodejs-mobile 16.x and got modified to work with pnpm async function getAndroidApiLevelFromLibNode(path: string) {
const output = await $`file ${path}`;
const fileOutput = output.toString();
const match = fileOutput.match(/for Android (\d+)/);
if (!match) throw new Error(`Could not determine Android version from path: ${path} with ${fileOutput}`);
return parseInt(match[1]);
}
async function setupAndroid(options: AndroidBuildOptions) {
const { ndkPath, libNodePath } = options;
const toolchainBinPath = path.join(ndkPath, '/toolchains/llvm/prebuilt/linux-x86_64/bin');
const toolchainTargetArch = 'aarch64';
const apiLevel = await getAndroidApiLevelFromLibNode(path.join(libNodePath, 'bin/arm64-v8a/libnode.so'));
// compiler env vars
process.env.PATH = `${process.env.PATH}:${toolchainBinPath}`;
process.env.CC = path.join(toolchainBinPath, `${toolchainTargetArch}-linux-android${apiLevel}-clang`);
process.env.CXX = path.join(toolchainBinPath, `${toolchainTargetArch}-linux-android${apiLevel}-clang++`);
process.env.LINK = path.join(toolchainBinPath, `${toolchainTargetArch}-linux-android${apiLevel}-clang++`);
process.env.AR = path.join(toolchainBinPath, 'llvm-ar');
// env var used in node's openssl config
process.env.ANDROID_NDK_ROOT = ndkPath;
// TODO: See if this is a real variable or not. We've been using it for a long time, but I don't know where it goes or where it's from
process.env.node_config_format = 'make-android';
// using this as env variables since pnpm doesn't seem to want to pass them along.
process.env.npm_config_build_from_source = 'true';
process.env.npm_config_arch = 'arm64';
process.env.npm_config_platform = 'android';
process.env.npm_config_nodedir = libNodePath;
process.env.npm_config_android_ndk_path = ndkPath;
} |
|
yes this is also what i do as you can see in the link i sent in my previous message, and it worked for nodejs mobile on node18 but not node24 |
|
I don't know anything about this bit |
|
the problem isn't caused by the module, it's caused by these patches, which is why I'm commenting here, not on stack overflow, because as I said on the nodejs18 base version it works fine, but on node24 the symbols for NAPI don't seem to be exposed, so this is for @jsamol not you jade |
| os.environ['CC_host'] = os.popen('command -v clang').read().strip() | ||
| os.environ['CXX_host'] = os.popen('command -v clang++').read().strip() |
There was a problem hiding this comment.
I wonder, what's the purpose of this patch? following the github workflow, this will no longer use the installed gcc, won't it?
There was a problem hiding this comment.
It might be an oversight on my side, I went through a couple of iterations where I was trying different build systems and configurations, looking for a combination that would require the least adjustments. I remember that at one point I had issues with gcc and then switched to clang. I'll update the workflow accordingly.
|
@ThaUnknown I'm not that familiar with building native addons, could you tell me exactly how I can reproduce the issue? Also I can't promise it'll be fixed quickly, I don't know when I'll have time to look into this properly. |
|
@jsamol as I said I provided the reproduction steps above, you can see the script in here: https://github.com/hayase-app/capacitor/blob/main/public/nodejs/setup-deps.sh, the code to reproduce the issue is legitimately just https://github.com/hayase-app/capacitor/blob/main/public/nodejs/setup-deps.sh#L46-L48 these lines specifically arent needed, this was just me trying things to try to fix this issue |
|
@jsamol I have a question, since you worked on this branch for a bit now: I have a general impression that the development model of this repository is currently less than ideal. The way it worked is that nodejs upstream is merged in every now and then, with its own commits interleaved here and there weaving in patches in a combined history of commits. Now this PR squashes all of that history into a single commit, losing upstream history and building a stack on top of that. I found both of these approaches difficult to browse and figure out what actually changed vs upstream and why. As is, the mega-squash first commit can basically contain arbitrary code, which is a big problem from a supply chain integrity perspective. I imagine a better model could be to maintain a patch stack on top of upstream, and rebase that stack to do an update. That way the actually required patches could be cleanly maintained and documented (via commit messages), kept up to date with current requirements, and can be discovered and reasoned about by a new developer. A drawback would be that it doesn't cleanly lend itself to a pull request workflow, but my feeling is that it's already a daunting task to review PRs here as is. Does this sound plausible to you? Do you have a sense of how difficult it might be to get there, and whether that might be worth it? I apologize for hijacking this thread a bit, I wanted to hear from your experience and gauge how I might invest some resources to keep this going in a sustainable way :) |
@Valodim that's been briefly brought up before (see discussion in #3), in case it provides any helpful context. No strong opinions on merits but in case it helps inform any decisions. As someone who works on a non-trivial project that heavily relies on this, great to see interest and discussion around maintenance! Appreciate the efforts that people have made so far :) |
|
@Valodim :I think a more important thing would be make a bigger fuss upstream to get these patches merged there so there's even less here to worry about in the first place. Some PRs already exist, but they are unmerged. We could do better here if we could find someone upstream who is willing to help us. After that, the amount of additional patches could stay small and be much easier to manage. |
|
problem is these patches are broken, NAPI and fetch doesn't work correctly |
|
If anyone is interested, I started some work transforming this branch into a patchstack on top of upstream: https://github.com/heylogin/nodejs-mobile/commits/v25.5.0-mobile There is still some more information to backport from the old commits, splitting and reordering and general cleanup to do, but it already looks much better and cleaner like this imo |
@Valodim yes, I had very similar thoughts. The current update model is far from ideal, the whole process was really cumbersome to me, mostly due to, as you noticed, the lack of clear history. I know that with the update commit I overwrote some patches that had been made in previous updates and noticed it only later when I was debugging compilation errors. I think that the model you propose is definitely an upgrade, I'd expect most of the changes to be very short configuration adjustments, which should remain readable even in a form of a git patch. Also it looks like Termux also went with that approach. Migrating this PR, or just the project in fact, to a patched based model might be quite tricky, though. With no clear history it's easy to leave a crucial fix unnoticed in this jungle of files, I can tell you already that my squashed update commit already contains some previous patches that made their way there when I had to resolve conflicts. Also, since I see that you've also started a contribution, you might find my latest comment here important.
@ThaUnknown what do you mean fetch doesn't work correctly? I haven't observed any issues on Android. It doesn't work on iOS, yes, but it's nothing new (#71). |
|
@jsamol what u said, doesnt work on IOS, but not the fault of these patches, but NAPI aka native bindings don't work with these patches as I've mentioned previously, I REALLY want this fixed as it's important for any form of high performance operations, which is important on devices as starved for CPU as mobile devices, so if I can help any further tell me, that said I already provided reproduction steps, compilation steps and stack traces, so idk what else I can do, I tried fixing it but to no luck |
|
@ThaUnknown I understand your frustration, but as I previously mentioned I haven't got any time to debug this issue. I'm not a maintainer of this project and I opened this PR simply because I'd already done the work and thought that this would be a huge step towards having the project up to date. Once the future of this project is clearer, I might re-prioritize to see this PR merged (or reorganized to something else), this would include fixing the issue you reported. But since effectively there's no maintainer atm, I can't dedicate it more time. |
Please also consider that everyone here is ultimately contributing for their own interests. To some degree those interests align, most of us probably want 16kb alignment for android. But what is vitally important for you isn't necessarily for everyone else, and it's unlikely that someone will free up their schedule to address your problem just because you put more and more urgency in your posts. If you need this fixed, it's up to you to figure out what the problem is and contribute a patch to get it working. |
|
@jsamol I worked on it a bit longer and I think I'm pretty happy with my current approach. It looks like a reasonable amount of effort to maintain and rebase on new versions, and is okay to read. The biggest drawback is that some of the "why" got lost along the way, but that was mostly already the case it just wasn't as obvious. The actual diff to your branch compared to mine is extremely small when ignoring whitespace changes: Point by point:
I'll think about the maintenance thing, really not sure I want to allocate resources for it. Too bad that Andre doesn't have time anymore, not too surprising at this point though. Oh well, priorities shift. |
|
Anyone remember how to generate the "Versions" list that's part of release notes? |
Yes: |
|
Hi @jsamol, Thank you so much for your hard work on this PR! I really appreciate the effort to update Node.js to v24.5.0 for mobile platforms. Unfortunately, I don’t have the proper build environment set up to compile the binaries myself. Would it be possible for you to provide a pre-built release so that others like me can test and use this update more easily? Thanks again for your contribution! |
Updates the upstream-sync procedure to maintain mobile-specific changes as a stack of atomic, individually-attributable commits rebased on top of an upstream release tag. The squash-merge format-patch workflow has been the basis of every nodejs-mobile release to date and reflects significant sustained work by maintainers; the patch-stack model is intended to make the next major upgrade more incremental, by scoping conflicts to single patches and making each mobile change reviewable on its own. The model follows the approach demonstrated by Valodim on PR nodejs-mobile#151 and implemented on heylogin/nodejs-mobile@icu. Adds doc_mobile/UPGRADING.md as the canonical rebase procedure; rewrites doc_mobile/CONTRIBUTING.md to document the branch layout and per-commit discipline. Refs: nodejs-mobile#134, nodejs-mobile#150, nodejs-mobile#151
Inventories the known issues from PRs nodejs-mobile#134, nodejs-mobile#150, nodejs-mobile#151 and the heylogin patch stack. Each blocker has a verification command, likely cause, and a proposed fix path so contributors can pick one up without re-deriving the diagnosis. Two blockers from earlier PRs are already resolved by patches inherited from heylogin/icu (host-toolchain split, NDK CRC32 crash); marked as such under "Resolved".
Adds a canonical description of the patch-stack maintenance model on which mobile/v24 is based. The mobile-specific changes live as discrete, atomic commits on top of an upstream nodejs/node release tag, so each patch is reviewable and individually rebaseable when upstream moves. The squash-merge format-patch workflow has been the basis of every nodejs-mobile release to date and reflects significant sustained work by maintainers; the patch-stack model is intended to make the next major upgrade more incremental, by scoping conflicts to single patches and making each mobile change reviewable on its own. The model follows the approach demonstrated by Valodim on nodejs-mobile#151 and implemented on heylogin/nodejs-mobile@icu. doc_mobile/MAINTENANCE_MODEL.md is the strategy overview; doc_mobile/UPGRADING.md is the canonical step-by-step rebase procedure. The legacy doc_mobile/CONTRIBUTING.md (carrying the deprecated format-patch instructions) lives on the legacy main branch and will land on mobile/v24 via a later layer-A patch import; a follow-up commit will wire the two together. Refs: nodejs-mobile#134, nodejs-mobile#150, nodejs-mobile#151
Inventories the known issues from PRs nodejs-mobile#134, nodejs-mobile#150, nodejs-mobile#151 and the heylogin patch stack. Each blocker has a verification command, likely cause, and a proposed fix path so contributors can pick one up without re-deriving the diagnosis. Two blockers from earlier PRs are already resolved by patches that will be inherited from heylogin/icu when layer-B lands (host-toolchain split, NDK CRC32 crash); marked as such under "Resolved". Refs: nodejs-mobile#134, nodejs-mobile#150, nodejs-mobile#151
mobile/v24 is maintained as a patch stack on top of an upstream nodejs/node release tag (recorded in doc_mobile/upstream-base.txt). MAINTENANCE_MODEL.md describes the model and branch layout; UPGRADING.md is the canonical rebase procedure for moving to a newer upstream version. Refs: nodejs-mobile#134, nodejs-mobile#150, nodejs-mobile#151
Closes #152
tools/mobile-testprojects but these are outdated to begin with, and even after bringing their configuration up to date most of the tests fail even on themainbranch, so I'm not sure if this is the correct way to test