Skip to content

Failure To Handle External References #603

Closed

Description

References to local files or external files outside of the file being parsed fail to parse correctly. The failure results in the properties being parsed to just be completely ignored.

$ref: path.to.my.yaml

Fails with diagnostic error:
The reference string 'path.to.my.yaml' has invalid format.

This appears to be caused by OpenApiV3VersionService.ConvertToOpenApiReference(..) expecting ReferenceType? == null for external references. However, OpenApiV3Deserializer is blindly passing in ReferenceType.Schema.

Links:
OpenApiV3VersionService
OpenApiV3Deserializer

Is this a known issue?

Activity

  1. Cooksauce commented on May 27, 2021

    @Cooksauce
    Author

    To clarify the use case here. It doesn't matter to me whether OpenAPI.NET can resolve the reference or not, I'm just trying to use it to parse a set of OpenAPI documents. This issue causes me a problem because it throws away the properties & the reference... which should be able to at least be parsed into the OpenAPI C# objects. It seems like this issue would also block any progress on the 'Workspace' feature or resolving of external references.

    For example:

    title: My Object
    type: object
    properties:
      code:
        type: integer
      something:
        $ref: 'path.to.my.yaml'
    

    If you parse this with:

    var reader = new OpenApiStreamReader(new OpenApiReaderSettings() {
          ReferenceResolution = ReferenceResolutionSetting.DoNotResolveReferences
    });
    var modelSchema = reader.ReadFragment<OpenApiSchema>(stream, OpenApiSpecVersion.OpenApi3_0, out var diagnostic);
    

    modelSchema.Properties is empty.

    I would expect modelSchema.Properties["something"].Reference.ExternalResource == "path.to.my.yaml"

  2. darrelmiller commented on May 30, 2021

    @darrelmiller
    Member

    Mitchell Cookson (@Cooksauce) I would expect the example you show above to work as you expect. It should also work as you expect if you online resolve internal references. I'll take a look and see if I can repro your issue.

    The external references should resolve completely if you provide an absolute path. I need to work on resolving relative paths.

  3. V0v1kkk commented on May 30, 2021

    @V0v1kkk

    Hello Mitchell Cookson (@Cooksauce) , Darrel (@darrelmiller).
    I have been looked at new Workspaces feature too.

    I have found root of the problem.
    Responsibility to resolve external resource is delegating to IStreamLoader interface.
    In the default implementation (DefaultStreamLoader) used Uri for resolve references.
    In this case format of correct reference is: file://path.to.my.yaml, but it's wrong for OpenAPI.

    And it's not that I expect too.
    I have written replacement for DefaultStreamLoader:

    public class ResourceLoaderFixed : IStreamLoader
    {
        private readonly string _mainFilePath;
        private readonly HttpClient _httpClient = new HttpClient();
    
        public ResourceLoaderFixed(string mainFilePath)
        {
            _mainFilePath = mainFilePath;
        }
    
        public Stream Load(Uri uri)
        {
            return LoadAsync(uri).GetAwaiter().GetResult();
        }
    
        public async Task<Stream> LoadAsync(Uri uri)
        {
            try
            {
                var uriScheme = uri.Scheme;
            
                switch (uriScheme)
                {
                    case "file":
                        return File.OpenRead(uri.AbsolutePath);
                    case "http":
                    case "https":
                        return await _httpClient.GetStreamAsync(uri);
                    default:
                        throw new ArgumentException("Unsupported scheme");
                }
            }
            catch (InvalidOperationException) // it seems that it's path to local file without 'file://' prefix
            {
                string originalAbsolutePath = uri.OriginalString;
                string compositeAbsolutePath = Path.Combine(_mainFilePath, uri.OriginalString);
                
                string finalFilePath;
                if (File.Exists(originalAbsolutePath))
                {
                    finalFilePath = originalAbsolutePath;
                }
                else if (File.Exists(compositeAbsolutePath))
                {
                    finalFilePath = compositeAbsolutePath;
                }
                else
                {
                    throw new Exception($"Can't to resolve reference to {originalAbsolutePath}");
                }
    
                return File.OpenRead(finalFilePath);
            }
            
        }
    }

    And use it as follows:

    var specificationPath = @"C:\Git_Root\Robocop\Veeam.AC.Web.REST\Specification\models\rest_v3.yaml";
    var specificationFolder = Path.GetDirectoryName(specificationPath);
    
    // Create a reader that will resolve all references
    var reader = new OpenApiStreamReader(new OpenApiReaderSettings()
    {
        ReferenceResolution = ReferenceResolutionSetting.ResolveAllReferences,
        CustomExternalLoader = new ResourceLoaderFixed(specificationFolder)
    });
    
    ReadResult result;
    using (var stream = File.OpenRead(specificationPath))
    {
        result = await reader.ReadAsync(stream);
    }

    I expected that Micosoft.OpenAPI will resolve all JSON references and inline all files into the one document.
    But now the feature works wired.
    I found the great tutorial by specification splitting and I can do nothing of that.

    In the current implementation you can reference only structural elements of OpenAPI. For example: schema, response, etc.
    And furthermore, you need to add a header to each of secondary files:

    openapi: 3.0.0
    info:
      title: Components for the main app
      version: 1.0.0
    paths: {}
    components:

    And another bug

    1. Open the primary specification with a reference to a schema in another file.
    2. Save the primary specification.

    The reference will be replaced to the reference to the primary file, but the schema doesn't copy to primary file.

  4. darrelmiller commented on May 30, 2021

    @darrelmiller
    Member

    Thanks for the additional info, this will be very helpful.

    Supporting of referencing external fragments will be supported. However, you can only use $ref to replace referenceable components, not just any arbitrary chunk of JSON. If you want to reference arbitrary chunks of JSON you will need to run the file through a pre-processor.

    Inlining is not supposed to happen automatically. There is a setting in the writer if you want to inline on write.

    ReferenceInline = inline == true ? ReferenceInlineSetting.InlineLocalReferences : ReferenceInlineSetting.DoNotInlineReferences

  5. V0v1kkk commented on May 31, 2021

    @V0v1kkk

    Thanks Darrel (@darrelmiller) !

    But ReferenceInlineSetting there are only three values: DoNotInlineReferences, InlineLocalReferences and InlineAllReferences.
    In my case I want to inline only remote references. 😒

    Well, to be quite honest, I just want to split a huge specification (~34k lines in yaml) to smaller parts.
    And merge the parts together on preprocessing without losing any information (names of schemes for example).

    The bug that I mentioned above can be even helpful.
    If I copy all parts from secondary OpenApiDocuments to primary, references will be fix automatically. 😆

  6. Cooksauce commented on May 31, 2021

    @Cooksauce
    Author

    Mitchell Cookson (@Cooksauce) I would expect the example you show above to work as you expect. It should also work as you expect if you online resolve internal references. I'll take a look and see if I can repro your issue.

    Darrel (@darrelmiller) Thanks for taking a look. To be clear, I'm not concerned even whether or not it can resolve the reference, I just need it to be able to read the $ref string without throwing an error which prevents other properties from being parsed.

    Additionally, it's worth noting that $ref: path.to.my.yaml#/ does work, while $ref: path.to.my.yaml does not. IMO, the #/ part should not be required in order for parsing to work (since they mean the same thing essentially). You can see why that happens here. I have have the reader settings set to ReferenceResolutionSetting.DoNotResolveReferences. I've tried both with the latest stable nuget release and the latest preview.

  7. peteraritchie commented on Jun 16, 2021

    @peteraritchie
    Contributor

    I think this is related:

        var text = new OpenApiV3VersionService()
            .ConvertToOpenApiReference(
                "externalSchema.json#/externalPathSegment1/externalPathSegment2/externalPathSegment3", null)
                .ReferenceV3

    text ends up being "externalSchema.json#//externalPathSegment1/externalPathSegment2/externalPathSegment3"

    Note the two slashes // after the fragment indicator (#).

  8. peteraritchie commented on Jun 16, 2021

    @peteraritchie
    Contributor
    new OpenApiV3VersionService()
            .ConvertToOpenApiReference(
                "externalSchema.json#/externalPathSegment1/externalPathSegment2/externalPathSegment3", null)
                .ReferenceV3

    Interestingly, if I use realistic path segments (i.e:)

    new OpenApiV3VersionService() .ConvertToOpenApiReference( "externalSchema.json#/components/schemas/Pet", null) .ReferenceV3

    ...I get "common.yaml#/Pet"

    But serializing that reference ends up with "$ref: common.yaml#/Pet" (via new OpenApiV3VersionService() .ConvertToOpenApiReference( "common.yaml#/components/schemas/Pet", null) .SerializeAsYaml(OpenApiSpecVersion.OpenApi3_0)

    That is, processing "externalSchema.json#/components/schemas/Pet" when serialized back to a file becomes "$ref: common.yaml#/Pet"


    this is with vNext

  9. added this to the 1.3 milestone on Jun 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions