Skip to content

🐞 The git_repo_scanner triggers an abuse detection mechanism at GitHub for large organisations  #325

Closed

Description

@glermaidt

Describe the bug

I am having an issue with a git-repo-scanner + gitleaks working in concert to scan my company's org, which has over 2k repos. The scan is ending around 900 repos with the following error:

INFO:git_repo_scanner: 901 - crp-legacy-ui-dhtmlWins
INFO:git_repo_scanner: Github API Exception: 403 -> You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.

This is causing CrashLoopBackoff events with the pod. Ultimately the job fails with error.

I understand what the error is telling me. My main question is: What is the largest number of repos does this scanner support? I would like to scan my entire org in one run, but I have not been successful, with the number of repos and the GH API throttle limits. The --ignore-repo is a start, but seems not too effective given my large list of repos.

Other questions:
Is there a way to throttle api calls?
Is there a recommended number of batches I should run with?
Would there be a way to pass an argument to break up the scans?

To Reproduce

Steps to reproduce the behavior:

  1. Install via helm: see second screenshot below.
  2. Create a git-repo-scanner yaml file.
  3. Run the git-repo-scanner job on a github org with over 2000 repos and with cascadingrules triggering gitleaks.
  4. Tail the created job's log, for example: k logs job/scan-gh-repo-scan-org-coretech-wdbns -c git-repo-scanner -f
  5. See 403 error in the first screenshot below.

Expected behavior

System (please complete the following information):

  • secureCodeBox 2.4.0
  • OS: Linux
  • Kubernetes Version [command: kubectl version]

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-21T20:23:45Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

  • Docker Version [command: docker -v]

Docker version 20.10.2, build 2291f61

  • Browser [e.g. chrome, safari, firefox,...]
    N/A

Screenshots / Logs

image

image

Additional context

Activity

  1. rfelber commented on Mar 12, 2021

    @rfelber
    Member

    Hi @glermaidt

    thx for your great issue documentation 👍🏻

    We are also running this combination in our company against GitHub and GitLab with thousands of repos. For now we didn't reach the throttling limits. But it's good to know this can be a problem 🤔

    Have you configured a GitHub accesstoken (this will raise the throttling limits)? You will find an example configuration in our documentation https://docs.securecodebox.io/docs/scanners/git-repo-scanner or in the issue #320 which is related a bit to yours.

  2. glermaidt commented on Mar 12, 2021

    @glermaidt
    Author

    Hi. Thanks for the response. Yes, I am using an access token, which is why this is so odd. I shall take a look at your doc and issue reference. Let me know if I can provide any additional details.

  3. J12934 commented on Mar 12, 2021

    @J12934
    Member

    I guess we haven't encountered this so far as we have most of our repos in an self hosted gitlab, which doesn't have this kind of abuse detection build in.

    We probably need to add some form of rate limiting into the repo scanner.

  4. glermaidt commented on Mar 12, 2021

    @glermaidt
    Author

    Here is some additional info i found

    Not sure if this means much:

    Form this doc: https://docs.github.com/en/rest/guides/best-practices-for-integrators#dealing-with-abuse-rate-limits

    "

    • Make authenticated requests, or use your application's client ID and secret. Unauthenticated requests are subject to more aggressive abuse rate limiting.
    • Make requests for a single user or client ID serially. Do not make requests for a single user or client ID concurrently.
    • If you're making a large number of POST, PATCH, PUT, or DELETE requests for a single user or client ID, wait at least one second between each request.
    • When you have been limited, use the Retry-After response header to slow down. The value of the Retry-After header will always be an integer, representing the number of seconds you should wait before making requests again. For example, Retry-After: 30 means you should wait 30 seconds before sending more requests.
    • Requests that create content which triggers notifications, such as issues, comments and pull requests, may be further limited and will not include a Retry-After header in the response. Please create this content at a reasonable pace to avoid further limiting."

    Hope that helps some...

    Best Regards,
    George

  5. rfelber commented on Mar 13, 2021

    @rfelber
    Member

    Found the GitHub rate limits for different scenarios and how to test them:

    • For unauthenticated requests, the rate limit allows for up to 60 requests per hour. Unauthenticated requests are associated with the originating IP address, and not the user making requests.
    • When using the built-in GITHUB_TOKEN in GitHub Actions, the rate limit is 1,000 requests per hour per repository.
    • This means that all OAuth applications authorized by a user share the same quota of 5,000 requests per hour when they authenticate with different tokens owned by the same user.
    • For organizations that belong to a GitHub Enterprise Cloud account, this limit is 15,000 requests per hour per repository.

    The returned HTTP headers of any API request show your current rate limit status:

    $ curl -I https://api.github.com/users/octocat
    > HTTP/1.1 200 OK
    > Date: Mon, 01 Jul 2013 17:27:06 GMT
    > Status: 200 OK
    > X-RateLimit-Limit: 60
    > X-RateLimit-Remaining: 56
    > X-RateLimit-Reset: 1372700873

    Source: https://docs.github.com/en/rest/overview/resources-in-the-rest-api#rate-limiting

  6. rfelber commented on Mar 13, 2021

    @rfelber
    Member

    GitHub Rate Limits

    Some helpful hints found for the Python GitHub Client we using in our implementation

    :

    GitLab Rate Limits

    "python-gitlab obeys the rate limit of the GitLab server by default. On receiving a 429 response (Too Many Requests), python-gitlab sleeps for the amount of time in the Retry-After header that GitLab sends back. If GitLab does not return a response with the Retry-After header, python-gitlab will perform an exponential backoff."

    Source: https://python-gitlab.readthedocs.io/en/stable/api-usage.html#rate-limits

  7. glermaidt commented on Mar 13, 2021

    @glermaidt
    Author

    I would be happy to test any proposed fixes as I can reproduce the issue pretty consistently. :)

  8. glermaidt commented on Mar 13, 2021

    @glermaidt
    Author

    BTW: I built a custom version of git-repo-scanner docker image, with a modified git-repo-scanner.py program, with a time.sleep (1) call at the end. I then configured the helm chart with that image.

    image

    This allowed me to run through my entire org's list of repos (2178 as of today) and successfully avoid the error reported above. The obvious consequence is that it took much longer to process all repos through the cascading calls to gitleak however, IMO, that is a minor one.

    So it seems to support my belief that there are two dimensions to this issue to consider:

    1. The speed of calls being made to Github API, which could possibly trigger the 403 -> You have triggered an abuse detection mechanism error.
    2. The maximum number of calls that can be made in a period of time (in my case 5000/hour), which could trigger the 403 -> API rate limit exceeded for user ID 0000000. error.

    My presumption is that a combination of the two are needed to avoid both types of 403s that can occur when scanning many hundreds or thousands of repos.

    I'm not advocating that the time.sleep(1) is the answer, hence i did not submit a PR. It was merely a test. - Perhaps a configurable -- delay <N seconds could be considered as an optionn

    Again, anything I can do to assist, please do let me know.
    George

  9. added
    pythonIssues based on python implementations
    on Mar 20, 2021
  10. added this to the v2.6.0 milestone on Mar 20, 2021
  11. changed the title [-]INFO:git_repo_scanner: Github API Exception: 403 -> You have triggered an abuse detection mechanism. [/-] [+]🐞 The git_repo_scanner triggers an abuse detection mechanism at GitHub for large organisations [/+] on Mar 20, 2021
  12. self-assigned this
    on Mar 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugBugspythonIssues based on python implementations

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions