Skip to content

Conversation

@chrisgavin
Copy link
Contributor

Currently we call CodeQL without specifying --ram. This means the amount of memory it uses is whatever Java happens to decide. In the case of the Actions workers that seems to be 1732MB. The actions workers have a ton more memory than this though, and since Actions steps are generally run serially we can use almost all of it if we want to.

This change adds automatic detection of how much memory to use based on the total system memory minus 256MB for other processes that might be running. I've also added an input that allows overriding this, but I don't think most people will need it so I've left it out of the documentation for now.

Merge / deployment checklist

  • Run test builds as necessary. Can be on this repository or elsewhere as needed in order to test the change - please include links to tests in other repos!
    • CodeQL using init/analyze actions
    • 3rd party tool using upload action (N/A)
  • Confirm this change is backwards compatible with existing workflows.
  • Confirm the readme has been updated if necessary. (N/A)

@chrisgavin chrisgavin requested a review from a team May 14, 2020 17:05
Copy link
Contributor

@robertbrignull robertbrignull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

How can we tell if it's working? There should be projects that don't build now but do with this change, but I'm unsure how to find one to test with. Queries should theoretically be a bit faster too, but I'm not sure if it'll be a noticeable amount.

const memoryToUseString = core.getInput("ram");
if (memoryToUseString) {
memoryToUseMegaBytes = Number(memoryToUseString);
if (Number.isNaN(memoryToUseMegaBytes)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably worth adding a check that it's greater than 0 too.

Suggested change
if (Number.isNaN(memoryToUseMegaBytes)) {
if (Number.isNaN(memoryToUseMegaBytes) || memoryToUseMegaBytes <= 0) {

@chrisgavin
Copy link
Contributor Author

chrisgavin commented May 18, 2020

The way I tested it was just to run ps aux while the queries are running and see how much memory was being passed to the evaluator. It should be 50% of the amount you actually want as the other 50% is used for off-heap allocations.

Before this change you get:

/opt/hostedtoolcache/CodeQL/1.0.0/x64/codeql/tools/linux64/java/bin/java -Xmx866M -Djava.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib -cp /opt/hostedtoolcache/CodeQL/1.0.0/x64/codeql/tools/codeql.jar com.semmle.cli2.CodeQL execute queries -J-Xmx866M --off-heap-ram=866 --verbosity=progress --logdir=/home/runner/work/******/codeql_databases/javascript/log --ml-model-path=/home/runner/work/******/****** --native-library-path=/home/runner/work/******/****** --warnings=show --no-rerun --output=/home/runner/work/******/codeql_databases/javascript/results -- /home/runner/work/******/codeql_databases/javascript/db-javascript javascript-code-scanning.qls

And afterwards it is:

/opt/hostedtoolcache/CodeQL/1.0.0/x64/codeql/tools/linux64/java/bin/java -Xmx3336M -Djava.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib -cp /opt/hostedtoolcache/CodeQL/1.0.0/x64/codeql/tools/codeql.jar com.semmle.cli2.CodeQL execute queries -J-Xmx3336M --off-heap-ram=3335 --verbosity=progress --logdir=/home/runner/work/******/codeql_databases/javascript/log --ml-model-path=/home/runner/work/******/****** --native-library-path=/home/runner/work/******/****** --warnings=show --no-rerun --output=/home/runner/work/******/codeql_databases/javascript/results -- /home/runner/work/******/codeql_databases/javascript/db-javascript javascript-code-scanning.qls

There should also be projects that were too big to analyze that now work, but I also don't know any examples off the top of my head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants