title: Learning pandas by Exploring COVID-19 Data

slug: learn-pandas-basic-commands-explore-covid-19-data

meta: Use the pandas data analysis tool to explore the free COVID-19 data set provided by the European Centre for Disease Prevention and Control.

category: post

date: 2020-03-28

modified: 2020-03-28

newsletter: False

headerimage: /img/200328-covid-19-pandas/header.png

headeralt: pandas logo. Copyright the PyData Foundation.

The

[European Centre for Disease Prevention and Control](https://www.ecdc.europa.eu/en)

provides

[daily-updated worldwide COVID-19 data](https://www.ecdc.europa.eu/en/publications-data/download-todays-data-geographic-distribution-covid-19-cases-worldwide)

that is easy to download in JSON, CSV or XML formats. In this tutorial,

we will use the [pandas](/pandas.html) data analysis tool on the

comma-separated values (CSV) data to learn some of the basic pandas

commands and explore what is contained within the data set.

Make sure you have Python 3 installed. As of right now,

[Python 3.8.2](https://www.python.org/downloads/) is the latest.

During this tutorial we're also going to use

[pandas](https://pandas.pydata.org/).

Install it now into a new virtual environment with the following

commands:

python -m venv covidpandas

source covidpandas/bin/activate

pip install pandas

We're now ready to get the COVID-19 data and start analyzing it with

pandas.

Go to the

[download today’s data on the geographic distribution of COVID-19 cases worldwide](https://www.ecdc.europa.eu/en/publications-data/download-todays-data-geographic-distribution-covid-19-cases-worldwide)

page in your web browser. It should look something like the following

screenshot.

<img src="/img/200328-covid-19-pandas/covid-19-data-download.png" width="100%" class="shot rnd outl" alt="Download the CSV version of the COVID-19 data.">

There should be a link to download the

data in CSV format, but the organization has changed the page layout

several times in the past few weeks, which makes it difficult to find

formats other than Excel (XLSX). If you have trouble obtaining the

CSV version, just download

[this one from GitHub](https://raw.githubusercontent.com/fullstackpython/blog-code-examples/master/pandas-covid-19/covid-19-cases-march-28-2020.csv)

which is pegged to a copy downloaded on March 28th, 2020.

We have the data in a CSV now we need to import it into a pandas

DataFrame.

Start by running the Python REPL:

python

>>>

The REPL is ready to go, now we need to import pandas so we can read

the data we downloaded.

from pandas import read_csv

df = read_csv("covid-19-cases-march-28-2020.csv")

Don't worry if you get an error like

`UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe7...`.

Run this command instead which explicitly sets the file encoding

so pandas can properly read the CSV.

df = read_csv("covid-19-cases-march-28-2020.csv", encoding="ISO-8859-1")

We now have our data loaded into a

[pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html)

and can start running code to poke and prod and what's inside the

data set.

Let's first take a peek at what a sample of the data looks like. I

typically run the `head` and `tail` functions when I open something

up to find out what are contained in the first five and last five rows.

df.head()

You should see six lines of output: one as the columns header and the

first five rows of data from the CSV:

The `tail` function looks at the last five rows in a DataFrame.

`tail` output will look something like this:

Note that you can also pass an integer into `head` or `tail` like

`df.head(10)` to get the first or last **n** number of rows.

It looks like based on the `tail` function we have around 7320 rows of

data (since the first row is 0 indexed). We can confirm how much

data is in each column with the `count` function.

`count`'s output will look like:

What if we want to look at one of those columns and find, for example,

the highest value of cases?

In this data set we get 18695 as the output. What about looking at

standard statistical measures across all columns? That's where the

`describe` function comes in handy.

`describe` presents standard statistical measures such as min, max,

median and mean for everything in your data set. In this case we

receive as output:

How about a quick view into whether or not columns' data are correlated

with each other? The `corr` function is what we need.

For our data set, `corr` outputs:

Not surprisingly, we see 1.000000 correlation between a column and itself.

We'd have to worry if we didn't see that result! For other columns it may

not make sense to look at their correlation. This is where you need to

think about the data. There is often correlation between completely unrelated

columns just because the data is structured a certain way.

If you are a developer like me without a rigorous background in statistics

(Stats 200 in college was a **long** time ago), you may need to brush up

on your stats knowledge before you are able to say whether something in the

data matters or not.

Let's keep going exploring the data. We can select columns and determine how

many unique items are held within it. For example, how many unique countries

and territories are listed?

In this case the result should be 196.

Those functions are fine for basic querying to learn what's in the

data set, but how do we ask real questions by stringing together some

commands?

We now know there are 7320 rows in this set since we used the `count`

function above. Each row represents a single day within a country. Now

to ask a question. How many days across these countries were there 10

or more cases reported?

Let's create a new dataframe named df2 with the rows that only have

10 or more cases reported on that day, then count the number of rows

within it.

That should give us the value 1531. There have been 1531 instances

of 10 or more COVID-19 cases reported on a single day, across the

196 countries or terrorities listed. But the 1531 is hard to explain

to people. We should pick out a single country and show how many times

10 or more cases were reported on one day. How about a smaller

country like Vietnam that is not being reported on as much as China,

the United States or Italy?

This will give us the full output of data by column:

We can also use the `count` function here to confirm there have been

five days in which 10 or more new cases have been reported in Vietnam

so far:

We get the output of 5 for the columns. Unfortunately, when you look at

the full data it appears these rows are all very recent and the virus

is just beginning to spread more widely there. Let's hope they along

with every other country is able to turn the tide, flatten the curve

and keep more people from getting sick as we continue onwards.

That's a good spot to leave off, but we covered a lot of pandas ground

in this tutorial!

We just imported and took a look at what's in the European Centre

for Disease Prevention and Control's COVID-19 data set using

[pandas](/pandas.html). That was a quick tour of some basic pandas

commands and I strongly recommend you peruse the

[DataFrame documentation list](https://pandas.pydata.org/pandas-docs/stable/reference/frame.html)

to learn about all of the other handy functions that this tool

provides to developers.

You can also get an idea of what to code next in your Python project by

reading the

[Full Stack Python table of contents page](/table-of-contents.html).

Questions? Contact me via Twitter

or [@mattmakai](https://twitter.com/mattmakai). I'm also on GitHub with

the username [mattmakai](https://github.com/mattmakai).

Something wrong with this post? Fork

[this page's source on GitHub](https://github.com/mattmakai/fullstackpython.com/blob/master/content/posts/200328-explore-covid-pandas.markdown)

and submit a pull request.