The CloudQuery GitHub plugin extracts your GitHub API and loads it into any supported CloudQuery destination
Publisher
cloudquery
Latest version
v15.1.2
Type
Source
Platforms
Date Published
Loading plugin documentation
CloudQuery's use of cookies
We use tracking cookies to understand how you use the product and help us improve it. Your consent is required before we can enable these cookies. You can opt out via the link in the footer.
private_key as plaintext #| to pass the multi-line private key as plaintext.- org: cloudquery
private_key: |
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA3eVv6PCn9P8zO+EP8K7pLMfxcA2uVrSZ2f+H3GgYIavDxWtO
vM9tE3jAA8mOjZpdLaG5yy4QfV1LQ3R7kO49JCB6VbClwN2lNvd8Iw49JCBDid7D
...
-----END RSA PRIVATE KEY-----
app_id: your_app_id
private_key as environment variable #private_key as a string from environment variables, you will need to replace all the new lines in your PEM file with \n otherwise the new line and indent will prevent CloudQuery from reading the variable correctly.- org: cloudquery
private_key: "${GITHUB_PRI_KEY}"
app_id: your_app_id
...
GITHUB_PRI_KEY="-----BEGIN RSA PRIVATE KEY-----\nMIIEpQIBAAKCAQEA3eVv6PCn9P8zO+EP8K7pLMfxcA2uVrSZ2f+H3GgYIavDxWtO\n...vM9tE3jAA8mOjZpdLaG5yy4QfV1LQ3R7kO49JCB6VbClwN2lNvd8Iw==\n-----END RSA PRIVATE KEY-----"
.yml file in your CloudQuery configuration directory.cloudquery/cloudquery repository:kind: source
spec:
# Source spec section
name: github
path: cloudquery/github
registry: cloudquery
version: "v15.1.2"
tables: ["github_issues"]
destinations: ["postgresql"]
# Learn more about the configuration options at https://cql.ink/github_source
spec:
access_token: "${GITHUB_PERSONAL_ACCESS_TOKEN}" # Personal Access Token, required if not using App Authentication.
# # App Authentication (one per org):
# app_auth:
# - org: cloudquery
# private_key: <PRIVATE_KEY> # Private key as a string
# private_key_path: <PATH_TO_PRIVATE_KEY> # Path to private key file
# app_id: <YOUR_APP_ID> # App ID, required for App Authentication.
# installation_id: <ORG_INSTALLATION_ID> # Installation ID for this org
# # List of organizations to sync from. You must specify either orgs or repos in the configuration.
# orgs: []
# # List of repositories to sync from. The format is `owner/repo` (e.g. `cloudquery/cloudquery`). You must specify either `orgs` or `repos` in the configuration.
# repos: ["cloudquery/cloudquery"]
# # List of GitHub Enterprise Cloud slugs to sync enterprise-level tables from.
# # The authenticated user must be an enterprise owner or billing manager.
# enterprises: ["my-enterprise"]
# # GitHub Enterprise
# # In order to enable GHE you have to provide two urls, the base url of the server and the upload url.
# # Quote from GitHub's client:
# # If the base URL does not have the suffix "/api/v3/", it will be added automatically. If the upload URL does not have the suffix "/api/uploads", it will be added automatically.
# # Another important thing is that by default, the GitHub Enterprise URL format should be http(s)://[hostname]/api/v3/ or you will always receive the 406 status code. The upload URL format should be http(s)://[hostname]/api/uploads/"
# # If you are not configuring against an enterprise server please omit the enterprise configuration bellow
# enterprise:
# base_url: "http(s)://[your-ghe-hostname]/api/v3/"
# upload_url: "http(s)://[your-ghe-hostname]/api/uploads/"
# # Optional parameters
# concurrency: 1500 # Optional. The best effort maximum number of Go routines to use. Lower this number to reduce memory usage or to avoid hitting GitHub API rate limits. Default 1500.
# discovery_concurrency: 1 # Optional. Number of concurrent requests to GitHub API during discovery phase. Default 1.
# include_archived_repos: false # Optional. Include archived repositories in the sync. Default false.
# local_cache_path: "" # Optional. Path to a local directory that will hold the cache. If set, the plugin will cache the GitHub API responses in this directory. Defaults to an empty string (no cache).
# table_options:
# github_workflow_runs:
# created_since: "" # e.g. "7 days ago", defaults to all workflow runs
# github_issues:
# state: "" # e.g. "open, all, closed", defaults to `all`
orgs or repos in the configuration. If a repository is specified in both orgs and repos, it will be extracted only once, and other repositories from that organization will be ignored.private_key or private_key_path in the configuration, but not both.GITHUB_ACCESS_TOKEN:spec:
access_token: ${GITHUB_ACCESS_TOKEN}
repos ([]string, optional. Default: empty)
List of repositories to sync from. The format is owner/repo (e.g. cloudquery/cloudquery). You must specify either orgs or repos in the configuration.orgs ([]string, optional. Default: empty):
List of organizations to sync from. You must specify either orgs or repos in the configuration.enterprises ([]string, optional. Default: empty):
List of GitHub Enterprise Cloud slugs to sync enterprise-level tables from (e.g. github_enterprise_copilot_metrics, github_enterprise_billing_advanced_security). The authenticated user must be an enterprise owner or billing manager.concurrency (integer, optional, default: 1500)
A best effort maximum number of Go routines to use. Lower this number to reduce memory usage or to avoid hitting GitHub API rate limits.discovery_concurrency (integer, optional, default: 1)orgs, to be used later on during the sync process.
By default the plugin discovers repositories one organization at a time. You can increase discovery_concurrency to discover multiple organizations in parallel, or use a negative value to discover all organizations in parallel.
Please note that it's possible to hit GitHub API rate limits when using a high value for discovery_concurrency.scheduler (string, optional, default: dfs)
The scheduler to use when determining the priority of resources to sync.
Supported values are dfs (depth-first search), round-robin, shuffle and shuffle-queue.include_archived_repos (bool) (default: false)include_archived_repos to true.local_cache_path (string, optional, default: empty)
Path to a local directory that will hold the cache. If set, the plugin will cache the GitHub API responses in this directory. Defaults to an empty string (no cache).
By using a cache, the plugin can use conditional requests when appropriate, and help avoid hitting GitHub API rate limits.table_options (Table Options spec) (optional)github_workflow_runscreated_since (string in natural date format) (optional)7 days ago, last month (see more here)github_issuesstate (string) (optional)open, closed, all. Defaults to all.labels ([]string) (optional)since (string in natural date format) (optional)7 days ago, last month (see more here)assignee (string) (optional)creator (string) (optional)mentioned (string) (optional)github_repositoriesupdated_after (Time) (optional)github_repository_commitsuntil (Time) (optional)github_copilot_org_metricssince (Time) (optional)github_organization_dependabot_alertsupdated_after (Time) (optional)github_repository_dependabot_alertsupdated_after (Time) (optional)github_code_scanning_alertsupdated_after (Time) (optional)github_secret_scanning_alertsupdated_after (Time) (optional)github_pull_requestsupdated_after (Time) (optional)github_pull_request_closing_issues_referencesupdated_after (Time) (optional)github_organization_members_with_rolepage_size (integer, optional, default: 50)2025-01-01T12:00:00+00:00.
Relative timestamps can take this format:nowx seconds [ago|from now]x minutes [ago|from now]x hours [ago|from now]x days [ago|from now]kind: source
spec:
name: gihtub
path: cloudquery/github
registry: cloudquery
version: "v15.1.2-fips"
...