title: APIs, time-series, and weather Data

```{r, echo=FALSE, message=FALSE, results='hide', purl=FALSE}

<iframe src="https://en.wikipedia.org/wiki/Application_programming_interface" width=100% height=400px></iframe>

* Browse to page, `File->Save As`, repeat.

* Deal with ugly html stuff...

```{r, message=F}

```{r, results='asis'}

* Government Demographic data (census, etc.)

* Government Environmental data

* Google, Twitter, etc.

* Get the data you want, when you want it

* Automatic updates - just re-run the request

* Dependent on real-time access

* APIs, permissions, etc. can change and break code

1. Provide R with a URL to request information

2. The API sends you back a response

* JavaScript Object Notation (JSON)

* Extensible Markup Language (XML).

The URL you will request information from.

* Data.gov API: https://www.data.gov/developers/apis

* Github API: https://api.github.com

* National Elevation Dataset digital elevation models (1 and 1/3 arc-second; USGS)

* National Hydrography Dataset (USGS)

* Soil Survey Geographic (SSURGO) database

* International Tree Ring Data Bank.

* Global Historical Climatology Network (GHCN)

* Others

<iframe src="https://www.ncdc.noaa.gov/cdo-web/webservices/v2" width=100% height=400px></iframe>

[National Climatic Data Center application programming interface (API)]( http://www.ncdc.noaa.gov/cdo-web/webservices/v2).

Handles downloading data directly from NOAA APIv2.

* `buoy_*` NOAA Buoy data from the National Buoy Data Center

* `ghcnd_*` GHCND daily data from NOAA

* `isd_*` ISD/ISH data from NOAA

* `homr_*` Historical Observing Metadata Repository

* `ncdc_*` NOAA National Climatic Data Center (NCDC)

* `seaice` Sea ice

* `storm_` Storms (IBTrACS)

* `swdi` Severe Weather Data Inventory (SWDI)

* `tornadoes` From the NOAA Storm Prediction Center

<iframe src="https://www.ncdc.noaa.gov/ghcn-daily-description" width=100% height=400px></iframe>

```{r,results='hide',message=FALSE, warning=F}

Download the GHCN station inventory with `ghcnd_stations()` and convert to `sf` object. The download can take a minute or two.

* **PRCP** Precipitation (tenths of mm)

* **SNOW** Snowfall (mm)

* **SNWD** Snow depth (mm)

* **TMAX** Maximum temperature

* **TMIN** Minimum temperature

* **ACMC** Average cloudiness midnight to midnight from 30-second ceilometer

* **AWND** Average daily wind speed

* **FMTM** Time of fastest mile or fastest 1-minute wind

* **MDSF** Multiday snowfall total

First, get a global country polygon

```{r, warning=F, message=F}

```{r, echo=F}

`ghcnd()` will download a `.dly` text file for a particular station. But how to choose?

Geocodes a location (find latitude and longitude) using either (1) the Data Science Toolkit (http://www.datasciencetoolkit.org/about) or (2) Google Maps.

```{r, message=F}

However, you have to be careful:

```{r, message=F, warning=F}

But this is pretty safe for well known and well-defined places.

```{r, message=F, warning=F}

Now use that location to spatially filter stations.

```{r, message=F, warning=F}

With the station ID, we can download daily data from NOAA.

See [CDO Daily Description](http://www1.ncdc.noaa.gov/pub/data/cdo/documentation/GHCND_documentation.pdf) and raw [GHCND metadata](http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/readme.txt) for more details.

Measurement Flag/Attribute

* **Blank** no measurement information applicable

* **B** precipitation total formed from two twelve-hour totals

* **H** represents highest or lowest hourly temperature (TMAX or TMIN) or average of hourly values (TAVG)

* **K** converted from knots

* ...

See [CDO Description](http://www1.ncdc.noaa.gov/pub/data/cdo/documentation/GHCND_documentation.pdf)

* **Blank** did not fail any quality assurance check

* **D** failed duplicate check

* **G** failed gap check

* **I** failed internal consistency check

* **K** failed streak/frequent-value check

* **N** failed naught check

* **O** failed climatological outlier check

* **S** failed spatial consistency check

* **T** failed temporal consistency check

* **W** temperature too warm for snow

* ...

See [CDO Description](http://www1.ncdc.noaa.gov/pub/data/cdo/documentation/GHCND_documentation.pdf)

Indicates the source of the data...

Summarize the QC flags. How many of which type are there? Should we be more conservative?

* **G** failed gap check

* **I** failed internal consistency check

* **S** failed spatial consistency check

Plot temperatures

```{r, warning=F}

Limit to a few years and plot the daily range and average temperatures.

```{r, fig.height=6}

Infrastructure for Regular and Irregular Time Series (Z's Ordered Observations)

* `rollmean()`: Rolling mean

* `rollsum()`: Rolling sum

* `rollapply()`: Custom functions

Use rollmean to calculate a rolling 60-day average.

* `align` whether the index of the result should be left- or right-aligned or centered

```{r, warning=F}

Most timeseries functions use the time series class (`ts`)

Values are highly correlated!

* autocorrelation $x$ vs. $x_{t-1}$ (lag=1)

* partial autocorrelation. $x$ vs. $x_{n}$ _after_ controlling for correlations $\in t-1:n$

How to convert years into 'decades'?

Now we can make a 'decade' grouping variable.

Calculate seasonal and decadal mean temperatures.

How to assess change? Simple differences?

But be careful, there were lots of missing data in the beginning of the record

```{r, warning=F}

Plot 10-year means (excluding years without complete data):

```{r, warning=F}

Let's compare 2017 with all the previous years in the dataset. First add 'day of year' to the data to facilitate showing all years on the same plot.

```{r, warning=F}

Then plot all years (in grey) and add 2017 in red.

```{r, warning=F}

Then 'zoom' into just the past few months and add 2017 in red.

```{r, warning=F, message=F}

So there was an unusually warm spell in late September.

```{r, warning=F,fig.height=12}

```{r, warning=F}

```{r, warning=F}

See [Time Series Analysis Task View](https://cran.r-project.org/web/views/TimeSeries.html) for summary of available packages/models.

* Moving average (MA) models

* autoregressive (AR) models

* autoregressive moving average (ARMA) models

* frequency analysis

* Many, many more...

Indices representing extreme aspects of climate derived from daily data:

<img src="../08_assets/climdex.png" alt="alt text" width="50%">

Climate Change Research Centre (CCRC) at University of New South Wales (UNSW) ([climdex.org](http://www.climdex.org)).

For example:

* **FD** Number of frost days: Annual count of days when TN (daily minimum temperature) < 0C.

* **SU** Number of summer days: Annual count of days when TX (daily maximum temperature) > 25C.

* **ID** Number of icing days: Annual count of days when TX (daily maximum temperature) < 0C.

* **TR** Number of tropical nights: Annual count of days when TN (daily minimum temperature) > 20C.

* **GSL** Growing season length: Annual (1st Jan to 31st Dec in Northern Hemisphere (NH), 1st July to 30th June in Southern Hemisphere (SH)) count between first span of at least 6 days with daily mean temperature TG>5C and first span after July 1st (Jan 1st in SH) of 6 days with TG<5C.

* **TXx** Monthly maximum value of daily maximum temperature

* **TN10p** Percentage of days when TN < 10th percentile

* **Rx5day** Monthly maximum consecutive 5-day precipitation

* **SDII** Simple pricipitation intensity index

[ClimDex](http://www.climdex.org/indices.html)