This pull request has merge conflicts that must be resolved before it can be
merged. @bjhargrave please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

This pull request has merge conflicts that must be resolved before it can be
merged. @bjhargrave please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Right now taxonomy diff does 5 layers of checking. Those 5 layers will be handled by this now.

@nathan-weinberg can you describe the 5 layers here?

This came up in our team weekly meeting today, when it was suggested taxonomy diff should check the yaml schema version and let the user know if supported or not. Right now we flag at sdg generate and that's a bit late in the process.

The parsing/validation code will check the knowledge version for v3 and report an error for v1/v2. There is a test case in this PR for this.

https://github.com/instructlab/instructlab/pull/1962/files#diff-f2fb20ddc8bf7f3b80b1a99bfe9fd3e563c5675f9faf7f914ac7f74e276d15f1R348-R368

The parsing/validation code will handle the linting and json schema validation reporting errors if found. I don't know what the 5 layers are.

5 layers are:

1. file extension check (is it .yaml)
2. content check (is the file empty)
3. YAML lint check (is the file valid yaml)
4. Schema check (does the file follow the taxonomy schema)
5. Knowledge check (does the file have everything needed for knowledge if applicable)

This PR should defintely be a 0.19.0 priority IMO - we want to align the schema checking as much as possible while minimizing code duplication (there's some code for this in the SDG library as well)