{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": null, "outputs": [], "source": [ "#r \"nuget: FSharp.Data,8.1.8\"\n", "\n", "Formatter.SetPreferredMimeTypesFor(typeof\u003cobj\u003e, \"text/plain\")\n", "Formatter.Register(fun (x: obj) (writer: TextWriter) -\u003e fprintfn writer \"%120A\" x)\n", "#endif\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "[![Binder](../img/badge-binder.svg)](https://mybinder.org/v2/gh/fsprojects/FSharp.Data/gh-pages?filepath=library/JsonProvider.ipynb)\u0026emsp;\n", "[![Script](../img/badge-script.svg)](https://fsprojects.github.io/FSharp.Data//library/JsonProvider.fsx)\u0026emsp;\n", "[![Notebook](../img/badge-notebook.svg)](https://fsprojects.github.io/FSharp.Data//library/JsonProvider.ipynb)\n", "\n", "# JSON Type Provider\n", "\n", "This article demonstrates how to use the JSON Type Provider to access JSON files\n", "in a statically typed way. We first look at how the structure is inferred and then\n", "demonstrate the provider by parsing data from WorldBank and Twitter.\n", "\n", "The JSON Type Provider provides statically typed access to JSON documents.\n", "It takes a sample document as an input (or a document containing a JSON array of samples).\n", "The generated type can then be used to read files with the same structure.\n", "\n", "If the loaded file does not match the structure of the sample, a runtime error may occur\n", "(but only when explicitly accessing an element incompatible with the original sample — e.g. if it is no longer present).\n", "\n", "## Introducing the provider\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": null, "outputs": [], "source": [ "\u003cdiv class=\"container-fluid\" style=\"margin:15px 0px 15px 0px;\"\u003e\n", " \u003cdiv class=\"row-fluid\"\u003e\n", " \u003cdiv class=\"span1\"\u003e\u003c/div\u003e\n", " \u003cdiv class=\"span10\" id=\"anim-holder\"\u003e\n", " \u003ca id=\"lnk\" href=\"../images/json.gif\"\u003e\u003cimg id=\"anim\" src=\"../images/json.gif\" /\u003e\u003c/a\u003e\n", " \u003c/div\u003e\n", " \u003cdiv class=\"span1\"\u003e\u003c/div\u003e\n", " \u003c/div\u003e\n", "\u003c/div\u003e\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "To use this type provider, reference the `FSharp.Data` NuGet package. Open the namespace:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 2, "outputs": [], "source": [ "open FSharp.Data\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "### Inferring types from the sample\n", "\n", "The `JsonProvider\u003c...\u003e` takes one static parameter of type `string`. The parameter can\n", "be *either* a sample string *or* a sample file (relative to the current folder or online\n", "accessible via `http` or `https`). It is not likely that this could lead to ambiguities.\n", "\n", "The following sample passes a small JSON string to the provider:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 3, "outputs": [ { "data": { "text/plain": ["type Simple = JsonProvider\u003c...\u003e", "", "val simple: JsonProvider\u003c...\u003e.Root = {", "", " \"name\": \"Tomas\",", "", " \"age\": 4", "", "}", "", "val it: string = \"Tomas\""] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" }], "source": [ "type Simple = JsonProvider\u003c\"\"\" { \"name\":\"John\", \"age\":94 } \"\"\"\u003e\n", "let simple = Simple.Parse(\"\"\" { \"name\":\"Tomas\", \"age\":4 } \"\"\")\n", "simple.Age\n", "simple.Name\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "You can see that the generated type has two properties - `Age` of type `int` and `Name` of\n", "type `string`. The provider successfully infers the types from the sample and exposes the\n", "fields as properties (with PascalCase name to follow standard naming conventions).\n", "\n", "### Inferring numeric types\n", "\n", "In the previous case, the sample document simply contained an integer and so the provider\n", "inferred the type `int`. Sometimes, the types in the sample document (or a list of samples)\n", "may not match exactly. For example, a list may mix integers and floats:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 4, "outputs": [ { "data": { "text/plain": ["type Numbers = JsonProvider\u003c...\u003e", "", "val nums: decimal array = [|1.2M; 45.1M; 98.2M; 5M|]", "", "val total: decimal = 149.5M"] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" }], "source": [ "type Numbers = JsonProvider\u003c\"\"\" [1, 2, 3, 3.14] \"\"\"\u003e\n", "let nums = Numbers.Parse(\"\"\" [1.2, 45.1, 98.2, 5] \"\"\")\n", "let total = nums |\u003e Seq.sum\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "When the sample is a collection, the type provider generates a type that can be used to store\n", "all values in the sample. In this case, the resulting type is `decimal`, because one\n", "of the values is not an integer. In general, the provider supports (and prefers them\n", "in this order): `int`, `int64`, `decimal` and `float`.\n", "\n", "Other primitive types cannot be combined into a single type. For example, if the list contains\n", "numbers *and* strings. In this case, the provider generates two methods that can be used\n", "to get values that match one of the types:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 5, "outputs": [ { "data": { "text/plain": ["type Mixed = JsonProvider\u003c...\u003e", "", "val mixed: JsonProvider\u003c...\u003e.Root = [", "", " 4,", "", " 5,", "", " \"hello\",", "", " \"world\"", "", "]", "", "val it: string = \"4, 5, hello, world\""] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" }], "source": [ "type Mixed = JsonProvider\u003c\"\"\" [1, 2, \"hello\", \"world\"] \"\"\"\u003e\n", "let mixed = Mixed.Parse(\"\"\" [4, 5, \"hello\", \"world\" ] \"\"\")\n", "\n", "mixed.Numbers |\u003e Seq.sum\n", "mixed.Strings |\u003e String.concat \", \"\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the `Mixed` type has properties `Numbers` and `Strings` that\n", "return only `int` and `string` values from the collection. This means that we get\n", "type-safe access to the values, but not in the original order (if order matters, then\n", "you can use the `mixed.JsonValue` property to get the underlying `JsonValue` and\n", "process it dynamically as described in [the documentation for `JsonValue`](JsonValue.html).\n", "\n", "### Inferring date types\n", "\n", "String values in JSON that look like dates are inferred as `DateTime` or `DateTimeOffset`.\n", "On .NET 6 and later, when you set `PreferDateOnly = true`, strings that represent a date without a time component (e.g. `\"2023-01-15\"`)\n", "are inferred as `DateOnly`, and time-only strings are inferred as `TimeOnly`. By default (`PreferDateOnly = false`),\n", "all dates are inferred as `DateTime` for backward compatibility.\n", "\n", "Set `PreferDateTimeOffset = true` to infer all date-time values (that would otherwise be `DateTime`) as `DateTimeOffset`.\n", "Values that already contain an explicit timezone offset (e.g. `\"2023-06-15T10:30:00+05:30\"`) are always inferred as\n", "`DateTimeOffset` regardless of this flag.\n", "\n", "### Inferring record types\n", "\n", "Now let\u0027s look at a sample JSON document that contains a list of records. The\n", "following example uses two records - one with `name` and `age` and the second with just\n", "`name`. If a property is missing, then the provider infers it as optional.\n", "\n", "If we want to just use the same text used for the schema at runtime, we can use the `GetSamples` method:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 6, "outputs": [ { "data": { "text/plain": ["John (94)", "", "Tomas ", "", "type People = JsonProvider\u003c...\u003e", "", "val it: unit = ()"] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" }], "source": [ "type People =\n", " JsonProvider\u003c\"\"\"\n", " [ { \"name\":\"John\", \"age\":94 },\n", " { \"name\":\"Tomas\" } ] \"\"\"\u003e\n", "\n", "for item in People.GetSamples() do\n", " printf \"%s \" item.Name\n", " item.Age |\u003e Option.iter (printf \"(%d)\")\n", " printfn \"\"\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "The inferred type for `items` is a collection of (anonymous) JSON entities - each entity\n", "has properties `Name` and `Age`. As `Age` is unavailable for all records in the sample\n", "data set, it is inferred as `option\u003cint\u003e`. The above sample uses `Option.iter` to print\n", "the value only when it is available.\n", "\n", "In the previous case, the values of individual properties had common types - `string`\n", "for the `Name` property and numeric type for `Age`. However, what if the property of\n", "a record can have multiple different types? In that case, the type provider behaves\n", "as follows:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 7, "outputs": [ { "data": { "text/plain": ["Numeric: 94", "", "Text: Tomas", "", "type Values = JsonProvider\u003c...\u003e", "", "val it: unit = ()"] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }], "source": [ "type Values = JsonProvider\u003c\"\"\" [{\"value\":94 }, {\"value\":\"Tomas\" }] \"\"\"\u003e\n", "\n", "for item in Values.GetSamples() do\n", " match item.Value.Number, item.Value.String with\n", " | Some num, _ -\u003e printfn \"Numeric: %d\" num\n", " | _, Some str -\u003e printfn \"Text: %s\" str\n", " | _ -\u003e printfn \"Some other value!\"\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "Here, the `Value` property is either a number or a string, The type provider generates\n", "a type that has an optional property for each possible option, so we can use\n", "simple pattern matching on `option\u003cint\u003e` and `option\u003cstring\u003e` values to distinguish\n", "between the two options. This is similar to the handling of heterogeneous arrays.\n", "\n", "Note that we have a `GetSamples` method because the sample is a JSON list. If it was a JSON\n", "object, we would have a `GetSample` method instead.\n", "\n", "#### More complex object type on the root level\n", "\n", "If you want the root type to be an object type, not an array, but\n", "you need more samples at the root level, you can use the `SampleIsList` parameter.\n", "Applied to the previous example, this would be:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 8, "outputs": [ { "data": { "text/plain": ["type People2 = JsonProvider\u003c...\u003e", "", "val person: JsonProvider\u003c...\u003e.Root = {", "", " \"name\": \"Gustavo\"", "", "}"] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" }], "source": [ "type People2 =\n", " JsonProvider\u003c\n", " \"\"\"\n", " [ { \"name\":\"John\", \"age\":94 },\n", " { \"name\":\"Tomas\" } ] \"\"\",\n", " SampleIsList=true\n", " \u003e\n", "\n", "let person = People2.Parse(\"\"\"{ \"name\":\"Gustavo\" }\"\"\")\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "Note that starting with version 4.2.9 of this package, JSON comments are supported\n", "(Comments are either single-line and start with `//` or multi-line when wrapped in `/*` and `*/`).\n", "This is not a standard feature of JSON, but it can be really convenient,\n", "e.g. to annotate each sample when using multiple ones.\n", "\n", "## Type inference hints / inline schemas\n", "\n", "Starting with version 4.2.10 of this package, it\u0027s possible to enable basic type annotations\n", "directly in the sample used by the provider, to complete or to override type inference.\n", "(Only basic types are supported. See the reference documentation of the provider for the full list)\n", "\n", "This feature is disabled by default and has to be explicitly enabled with the `InferenceMode`\n", "static parameter.\n", "\n", "Let\u0027s consider an example where this can be useful:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 9, "outputs": [ { "data": { "text/plain": ["type AmbiguousEntity = JsonProvider\u003c...\u003e", "", "val code: float = 123.0", "", "val length: decimal = 42M"] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" }], "source": [ "type AmbiguousEntity =\n", " JsonProvider\u003c\n", " Sample=\"\"\"\n", " { \"code\":\"000\", \"length\":\"0\" }\n", " { \"code\":\"123\", \"length\":\"42\" }\n", " { \"code\":\"4E5\", \"length\":\"1.83\" }\n", " \"\"\",\n", " SampleIsList=true\n", " \u003e\n", "\n", "let code = (AmbiguousEntity.GetSamples()[1]).Code\n", "let length = (AmbiguousEntity.GetSamples()[1]).Length\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "In the previous example, `Code` is inferred as a `float`,\n", "even though it looks more like it should be a `string`.\n", "(`4E5` is interpreted as an exponential float notation instead of a string)\n", "\n", "Now, let\u0027s enable inline schemas:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 10, "outputs": [ { "data": { "text/plain": ["type AmbiguousEntity2 = JsonProvider\u003c...\u003e", "", "val code2: string = \"123\"", "", "val length2: JsonProvider\u003c...\u003e.DecimalOrString = \"42\""] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" }], "source": [ "open FSharp.Data.Runtime.StructuralInference\n", "\n", "type AmbiguousEntity2 =\n", " JsonProvider\u003c\n", " Sample=\"\"\"\n", " { \"code\":\"typeof\u003cstring\u003e\", \"length\":\"typeof\u003c float\u003cmetre\u003e \u003e\" }\n", " { \"code\":\"123\", \"length\":\"42\" }\n", " { \"code\":\"4E5\", \"length\":\"1.83\" }\n", " \"\"\",\n", " SampleIsList=true,\n", " InferenceMode=InferenceMode.ValuesAndInlineSchemasOverrides\n", " \u003e\n", "\n", "let code2 = (AmbiguousEntity2.GetSamples().[1]).Code\n", "let length2 = (AmbiguousEntity2.GetSamples().[1]).Length\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "With the `ValuesAndInlineSchemasOverrides` inference mode, the `typeof\u003cstring\u003e` inline schema\n", "takes priority over the type inferred from other values.\n", "`Code` is now a `string`, as we wanted it to be!\n", "\n", "Note that an alternative to obtain the same result would have been to replace all the `Code` values\n", "in the samples with unambiguous string values. (But this can be very cumbersome, especially with big samples)\n", "\n", "If we had used the `ValuesAndInlineSchemasHints` inference mode instead, our inline schema\n", "would have had the same precedence as the types inferred from other values, and `Code`\n", "would have been inferred as a choice between either a number or a string,\n", "exactly as if we had added another sample with an unambiguous string value for `Code`.\n", "\n", "You can use either angle brackets `\u003c\u003e` or curly brackets `{}` when defining inline schemas.\n", "\n", "### Units of measure\n", "\n", "Inline schemas also enable support for units of measure.\n", "\n", "In the previous example, the `Length` property is now inferred as a `float`\n", "with the `metre` unit of measure (from the default SI units).\n", "\n", "Warning: units of measures are discarded when merged with types without a unit or with a different unit.\n", "As mentioned previously, with the `ValuesAndInlineSchemasHints` inference mode,\n", "inline schemas types are merged with other inferred types with the same precedence.\n", "Since values-inferred types never have units, inline-schemas-inferred types will lose their\n", "unit if the sample contains other values...\n", "\n", "## Loading WorldBank data\n", "\n", "Now, let\u0027s use the type provider to process some real data. We use a data set returned by\n", "[the WorldBank](https://data.worldbank.org), which has (roughly) the following structure:\n", "\n", " [lang=js]\n", " [ { \"page\": 1, \"pages\": 1, \"total\": 53 },\n", " [ { \"indicator\": {\"value\": \"Central government debt, total (% of GDP)\"},\n", " \"country\": {\"id\":\"CZ\",\"value\":\"Czech Republic\"},\n", " \"value\":null,\"decimal\":\"1\",\"date\":\"2000\"},\n", " { \"indicator\": {\"value\": \"Central government debt, total (% of GDP)\"},\n", " \"country\": {\"id\":\"CZ\",\"value\":\"Czech Republic\"},\n", " \"value\":\"16.6567773464055\",\"decimal\":\"1\",\"date\":\"2010\"} ] ]\n", "\n", "The response to a request contains an array of two items. The first item is a record\n", "with general information about the response (page, total pages, etc.) and the second item\n", "is another array which contains the actual data points. For every data point, we get\n", "some information and the actual `value`. Note that the `value` is passed as a string\n", "(for some unknown reason). It is wrapped in quotes, so the provider infers its type as\n", "`string` (and we need to convert it manually).\n", "\n", "The following sample generates type based on the [`data/WorldBank.json`](../data/WorldBank.json)\n", "file and loads it:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 11, "outputs": [], "source": [ "[\u003cLiteral\u003e]\n", "let ResolutionFolder = __SOURCE_DIRECTORY__\n", "\n", "type WorldBank = JsonProvider\u003c\"../data/WorldBank.json\", ResolutionFolder=ResolutionFolder\u003e\n", "let doc = WorldBank.GetSample()\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "Note that we can also load the data directly from the web both in the `Load` method and in\n", "the type provider sample parameter, and there\u0027s an asynchronous `AsyncLoad` method available too:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 12, "outputs": [ { "data": { "text/plain": ["val wbReq: string =", "", " \"https://api.worldbank.org/country/cz/indicator/GC.DOD.TOTL.GD\"+[15 chars]", "", "val docAsync: Async\u003cJsonProvider\u003c...\u003e.Root\u003e"] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" }], "source": [ "let wbReq =\n", " \"https://api.worldbank.org/country/cz/indicator/\"\n", " + \"GC.DOD.TOTL.GD.ZS?format=json\"\n", "\n", "let docAsync = WorldBank.AsyncLoad(wbReq)\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "The `doc` is an array of heterogeneous types, so the provider generates a type\n", "that can be used to get the record and the array, respectively. Note that the\n", "provider infers that there is only one record and one array. We can print the data set as follows:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 13, "outputs": [ { "data": { "text/plain": ["Showing page 1 of 1. Total records 53", "", "2010: 35.142297", "", "2009: 31.034880", "", "2008: 25.475164", "", "2007: 24.193320", "", "2006: 23.708055", "", "2005: 22.033462", "", "2004: 20.108379", "", "2003: 18.267725", "", "2002: 15.425565", "", "2001: 14.874434", "", "2000: 13.218869", "", "1999: 11.356696", "", "1998: 10.178780", "", "1997: 10.153566", "", "1996: 10.520301", "", "1995: 12.707834", "", "1994: 14.781808", "", "1993: 16.656777", "", "val info: JsonProvider\u003c...\u003e.Record2 =", "", " {", "", " \"page\": 1,", "", " \"pages\": 1,", "", " \"per_page\": \"1000\",", "", " \"total\": 53", "", "}", "", "val it: unit = ()"] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" }], "source": [ "// Print general information\n", "let info = doc.Record\n", "printfn \"Showing page %d of %d. Total records %d\" info.Page info.Pages info.Total\n", "\n", "// Print all data points\n", "for record in doc.Array do\n", " record.Value |\u003e Option.iter (fun value -\u003e printfn \"%d: %f\" record.Date value)\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "When printing the data points, some of the values might be missing (in the input, the value\n", "is `null` instead of a valid number). This is another example of a heterogeneous type -\n", "the type is either `Number` or some other type (representing `null` value). This means\n", "that `record.Value` has a `Number` property (when the value is a number) and we can use\n", "it to print the result only when the data point is available.\n", "\n", "## Parsing Twitter stream\n", "\n", "We now look at how to parse tweets returned by the [Twitter API](http://dev.twitter.com/).\n", "Tweets are quite heterogeneous, so we infer the structure from a *list* of inputs rather than from\n", "just a single input. To do that, we use the file [`data/TwitterStream.json`](../data/TwitterStream.json)\n", "(containing a list of tweets) and pass an optional parameter `SampleIsList=true` which tells the\n", "provider that the sample is actually a *list of samples*:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 14, "outputs": [ { "data": { "text/plain": [""] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" }], "source": [ "type Tweet = JsonProvider\u003c\"../data/TwitterStream.json\", SampleIsList=true, ResolutionFolder=ResolutionFolder\u003e\n", "\n", "let text = (*[omit:(omitted)]*)\n", "\n", "let tweet = Tweet.Parse(text)\n", "\n", "printfn \"%s (retweeted %d times)\\n:%s\" tweet.User.Value.Name tweet.RetweetCount.Value tweet.Text.Value\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "After creating the `Tweet` type, we parse a single sample tweet and print some details about the\n", "tweet. As you can see, the `tweet.User` property has been inferred as optional, and so are\n", "`RetweetCount` and `Text`. The reason is that `TwitterStream.json` contains not only tweet objects\n", "but also other event types (such as `delete` events) with a completely different schema. When the\n", "type provider merges multiple heterogeneous sample objects, any field that does not appear in all\n", "samples is inferred as optional. We unsafely get the values using the `Value` property since we\n", "know our input is a tweet.\n", "\n", "## Getting and creating GitHub issues\n", "\n", "In this example we will now also create JSON in addition to consuming it.\n", "Let\u0027s start by listing the 5 most recently updated open issues in the FSharp.Data repository.\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 15, "outputs": [ { "data": { "text/plain": ["#879 Bug when call request from Http module", "", "#867 XmlProvider in 2.2.5 on F# 4 project causes multiple FSharp.Core assembly references", "", "#877 Header being considered as data row in HTMLProvider", "", "#878 Replace GitHub JsonProvider example with something else in ConsoleTests because of rate limit", "", "#873 Fix HtmlInference inferListType when passing an empty seq", "", "type GitHub = JsonProvider\u003c...\u003e", "", "val topRecentlyUpdatedIssues: JsonProvider\u003c...\u003e.Root seq", "", "val it: unit = ()"] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" }], "source": [ "// GitHub.json downloaded from\n", "// https://api.github.com/repos/fsharp/FSharp.Data/issues\n", "// to prevent rate limit when generating these docs\n", "type GitHub = JsonProvider\u003c\"../data/GitHub.json\", ResolutionFolder=ResolutionFolder\u003e\n", "\n", "let topRecentlyUpdatedIssues =\n", " GitHub.GetSamples()\n", " |\u003e Seq.filter (fun issue -\u003e issue.State = \"open\")\n", " |\u003e Seq.sortBy (fun issue -\u003e System.DateTimeOffset.Now - issue.UpdatedAt)\n", " |\u003e Seq.truncate 5\n", "\n", "for issue in topRecentlyUpdatedIssues do\n", " printfn \"#%d %s\" issue.Number issue.Title\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "And now let\u0027s create a new issue. We look into the documentation at [http://developer.github.com/v3/issues/#create-an-issue](http://developer.github.com/v3/issues/#create-an-issue) and we see that\n", "we need to post a JSON value similar to this:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 16, "outputs": [], "source": [ "[\u003cLiteral\u003e]\n", "let issueSample =\n", " \"\"\"\n", "{\n", " \"title\": \"Found a bug\",\n", " \"body\": \"I\u0027m having a problem with this.\",\n", " \"assignee\": \"octocat\",\n", " \"milestone\": 1,\n", " \"labels\": [\n", " \"Label1\",\n", " \"Label2\"\n", " ]\n", "}\n", "\"\"\"\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "This JSON is different from what we got for each issue in the previous API call, so we\u0027ll define a new type based on this sample,\n", "create an instance, and send a POST request:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": null, "outputs": [], "source": [ "type GitHubIssue = JsonProvider\u003cissueSample, RootName=\"issue\"\u003e\n", "\n", "let newIssue =\n", " GitHubIssue.Issue(\n", " \"Test issue\",\n", " \"This is a test issue created in FSharp.Data documentation\",\n", " assignee = \"\",\n", " labels = [||],\n", " milestone = 0\n", " )\n", "\n", "newIssue.JsonValue.Request \"https://api.github.com/repos/fsharp/FSharp.Data/issues\"\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": null, "outputs": [], "source": [ "\u003ca name=\"jsonlib\"\u003e\u003c/a\u003e\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "## Using JSON provider in a library\n", "\n", "You can use the types created by JSON type provider in a public API of a library that you are building,\n", "but there is one important thing to keep in mind - when the user references your library, the type\n", "provider will be loaded, and the types will be generated at that time (the JSON provider is not\n", "currently a *generative* type provider). This means that the type provider will need to be able to\n", "access the sample JSON. This works fine when the sample is specified inline, but it won\u0027t work when\n", "the sample is specified as a local file (unless you distribute the samples with your library).\n", "\n", "For this reason, the JSON provider lets you specify samples as embedded resources using the\n", "static parameter `EmbeddedResource`. When this parameter is set, the type provider at design time\n", "reads the sample from the local path, but at runtime (when the library is loaded by a consumer)\n", "it reads the sample from the embedded resource in the compiled assembly.\n", "\n", "### Step-by-step guide\n", "\n", "**Step 1**: Mark your sample file as an embedded resource in the `.fsproj` file:\n", "\n", "```xml\n", "\u003cItemGroup\u003e\n", " \u003cEmbeddedResource Include=\"data/worldbank.json\" /\u003e\n", "\u003c/ItemGroup\u003e\n", "```\n", "\n", "**Step 2**: Use the `EmbeddedResource` static parameter. The value must be\n", "`\"AssemblyName, AssemblyName.dotted.path.to.file.json\"` where:\n", "- `AssemblyName` is the name of your library assembly (without `.dll`)\n", "- The file path uses **dots** (not slashes) as separators, prefixed with `AssemblyName`\n", "\n", "So for a file `data/worldbank.json` in a library `MyLib`, the value is:\n", "\n" ] } , { "cell_type": "code", "metadata": { "dotnet_interactive": { "language": "fsharp" }, "polyglot_notebook": { "kernelName": "fsharp" } }, "execution_count": 17, "outputs": [], "source": [ "type WB =\n", " JsonProvider\u003c\n", " \"../data/WorldBank.json\",\n", " EmbeddedResource=\"MyLib, MyLib.data.worldbank.json\",\n", " ResolutionFolder=ResolutionFolder\n", " \u003e\n" ] } , { "cell_type": "markdown", "metadata": {}, "source": [ "You still need to specify the local path, but this is only used when compiling `MyLib.dll`.\n", "When a user of your library references `MyLib.dll` later, the JSON Type Provider will be able\n", "to load `MyLib.dll` and locate the sample `worldbank.json` as a resource of the library. When\n", "this succeeds, it does not attempt to find the local file and so your library can be used\n", "without providing a local copy of the sample JSON files.\n", "\n", "\u003e **Common pitfall**: If you get a cryptic error where the type provider interprets the file path\n", "as the CSV/JSON content itself (resulting in a single-column type named after the path), you\n", "have likely forgotten to add the `EmbeddedResource` parameter, or the assembly name or resource\n", "path in the parameter value is incorrect.\n", "\u003e \n", "\n", "\u003e To verify the embedded resource name, you can inspect the compiled `.dll` using a tool such as\n", "`ildasm`, `dotnet-ildasm`, or ILSpy and look at the `.mresource` entries.\n", "\u003e \n", "\n", "## Related articles\n", "\n", "* [JSON Parser](JsonValue.html) - provides more information about\n", "working with JSON values dynamically.\n", "\n", "* API Reference: [JsonProvider](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-jsonprovider.html)\n", "\n", "* API Reference: [JsonValue](https://fsprojects.github.io/FSharp.Data/reference/fsharp-data-jsonvalue.html)\n", "\n" ] } ], "metadata": { "kernelspec": { "display_name": ".NET (F#)", "language": "F#", "name": ".net-fsharp" }, "language_info": { "file_extension": ".fs", "mimetype": "text/x-fsharp", "name": "polyglot-notebook", "pygments_lexer": "fsharp" }, "polyglot_notebook": { "kernelInfo": { "defaultKernelName": "fsharp", "items": [ { "aliases": [], "languageName": "fsharp", "name": "fsharp" } ] } } }, "nbformat": 4, "nbformat_minor": 2 }