Skip to content

Avoid download archive if there is no updates #3365

@kalidor

Description

@kalidor

Is your feature request related to a problem? Please describe.
We use to download the archive all.zip (https://osv-vulnerabilities.storage.googleapis.com/all.zip), unzip it, and work with data.
Some of our tools can download it few times, but avoiding it will be better for the bandwith.
The tools are not working together, but having a way to detect if an archive has been updated.

Describe the solution you'd like
Adding a simple JSON file with the list of files and associated hash (hash of the content of the file, not the archive), will let us detect if something changed, compare to the previous download.
Something like:

{
  "filename1.json": "<computed hash1>",
  "filename2.json": "<computed hash2>",
  ... and so on
}

This could also be applied to each ecosystem archive.

Describe alternatives you've considered
None. However, i read the other ticket about this (#1993) which has been considered as completed.

Additional context
This could also be useful on your side to not update the modified field date if the content of the OSV entry doesn't change at all.

I read the contribution are welcome, that's nice, i didn't find a way to potentially test code before making a PR...

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestworkerWorker-related infrastructure

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions