Skip to content

Migrate Alpine importer to advisory V2#2111

Open
ziadhany wants to merge 7 commits intoaboutcode-org:mainfrom
ziadhany:alpine-migrate
Open

Migrate Alpine importer to advisory V2#2111
ziadhany wants to merge 7 commits intoaboutcode-org:mainfrom
ziadhany:alpine-migrate

Conversation

@ziadhany
Copy link
Collaborator

@ziadhany ziadhany commented Jan 8, 2026 •
edited
Loading

@ziadhany
Copy link
Collaborator Author

ziadhany commented Jan 8, 2026 •
edited
Loading 

INFO 2026-01-26 19:15:30.575619 UTC Pipeline [AlpineLinuxImporterPipeline] starting
INFO 2026-01-26 19:15:30.575748 UTC Step [collect_and_store_advisories] starting
Importing data using alpine_linux_importer_v2
INFO 2026-01-26 22:39:08.084020 UTC Successfully collected 108,252 advisories
INFO 2026-01-26 22:39:08.084139 UTC Step [collect_and_store_advisories] completed in 12218 seconds (3.4 hours)
INFO 2026-01-26 22:39:08.084171 UTC Pipeline completed in 12218 seconds (3.4 hours)


from vulnerabilities.models import AdvisoryV2
from django.db.models import Count
duplicates = (
    AdvisoryV2.objects
    .values('avid')
    .annotate(count=Count('id'))
    .filter(count__gt=1)
)
len(duplicates)
Out[2]: 0
AdvisoryV2.objects.count()
Out[3]: 108252

@ziadhany
Copy link
Collaborator Author

ziadhany commented Jan 15, 2026 •
edited
Loading

@TG1999 @pombredanne I have a question about Alpine migration. We are fetching one URL and processing the data without grouping by CVE.

The problem is that each URL reports a package version along with its fixed CVEs. How can we obtain a unique identifier for this importer? Is it a good idea to restructure the data and create a large mapping, using the CVE as the unique identifier?

Proposed structure:
CVE: [purl_1, purl_2, ...]

Example:
Package: aom

Sources:
https://secdb.alpinelinux.org/v3.22/main.json -> CVEs: "CVE-2021-30473", "CVE-2021-30474", "CVE-2021-30475"
https://secdb.alpinelinux.org/v3.21/main.json -> CVEs: "CVE-2021-30473", "CVE-2021-30474", "CVE-2021-30475"

ziadhany
ziadhany commented Jan 26, 2026
View reviewed changes
vulnerabilities/pipelines/v2_importers/alpine_linux_importer.py Outdated
)

for cve in aliases:
advisory_id = f"{pkg_infos['name']}/{qualifiers['distroversion']}/{cve}"
Copy link
Collaborator Author

@ziadhany ziadhany Jan 26, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ex:

apache2/v3.20/2.4.26-r0/CVE-2017-7668

@ziadhany
Copy link
Collaborator Author

ziadhany commented Jan 28, 2026 •
edited
Loading

The logs in debug mode:

alpine.zip

@ziadhany ziadhany requested a review from keshav-space January 28, 2026 13:50
keshav-space
keshav-space requested changes Jan 29, 2026
View reviewed changes
Copy link
Member

@keshav-space keshav-space left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ziadhany, see comments below.

@ziadhany
Migrate Alpine importer to advisory V2
c12ed5f 
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Fix typo Alpine Linux importer pipeline inherits from VulnerableCodeB…
b935cf9 
…aseImporterPipelineV2

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Update Alpine to avoid fetching the same URL links multiple times
b7aff19 
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Update alpine linux so for every vulnerability id AdvisoryData
e9bd3cd 
Fix duplication on advisory_id

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Update Alpine pipeline to use aboutcode-mirror-alpine-secdb
0bb7b03 
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@keshav-space keshav-space keshav-space requested changes

@TG1999 TG1999 Awaiting requested review from TG1999

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ziadhany @keshav-space