Skip to content

Commit 4175134

Browse files
authored
First pass of documentation in mdBook (apache#364)
* First pass of documentation in mdBook * Address code review comments * Add missing ASF headers
1 parent 31a86ee commit 4175134

11 files changed

Lines changed: 474 additions & 1 deletion

File tree

‎.gitignore‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,4 +22,6 @@ venv
2222
apache-rat-*.jar
2323
*rat.txt
2424
.env
25-
CHANGELOG.md.bak
25+
CHANGELOG.md.bak
26+
27+
docs/mdbook/book

‎docs/mdbook/README.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
Unless required by applicable law or agreed to in writing,
11+
software distributed under the License is distributed on an
12+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
13+
KIND, either express or implied. See the License for the
14+
specific language governing permissions and limitations
15+
under the License.
16+
-->
17+
# DataFusion Book
18+
19+
This folder builds a DataFusion user guide using [mdBook](https://github.com/rust-lang/mdBook).
20+
21+
## Build and run book locally
22+
23+
Build the latest files with `mdbook build`.
24+
25+
Open the book locally by running `open book/index.html`.
26+
27+
## Install mdBook
28+
29+
Download the `mdbook` binary.
30+
31+
Then manually open it, so you have permissions to run it on your Mac.
32+
33+
Add it to your path with a command like this so you can easily run the commands: `mv ~/Downloads/mdbook /Users/matthew.powers/.local/bin`.

‎docs/mdbook/book.toml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
[book]
19+
authors = ["Apache Arrow <dev@arrow.apache.org>"]
20+
language = "en"
21+
multilingual = false
22+
src = "src"
23+
title = "DataFusion Book"

‎docs/mdbook/src/SUMMARY.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
Unless required by applicable law or agreed to in writing,
11+
software distributed under the License is distributed on an
12+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
13+
KIND, either express or implied. See the License for the
14+
specific language governing permissions and limitations
15+
under the License.
16+
-->
17+
# Summary
18+
19+
- [Index](./index.md)
20+
- [Installation](./installation.md)
21+
- [Quickstart](./quickstart.md)
22+
- [Usage](./usage/index.md)
23+
- [Create a table](./usage/create-table.md)
24+
- [Query a table](./usage/query-table.md)
84.1 KB
Loading

‎docs/mdbook/src/index.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
Unless required by applicable law or agreed to in writing,
11+
software distributed under the License is distributed on an
12+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
13+
KIND, either express or implied. See the License for the
14+
specific language governing permissions and limitations
15+
under the License.
16+
-->
17+
# DataFusion Book
18+
19+
DataFusion is a blazing fast query engine that lets you run data analyses quickly and reliably.
20+
21+
DataFusion is written in Rust, but also exposes Python and SQL bindings, so you can easily query data in your langauge of choice. You don't need to know any Rust to be a happy and productive user of DataFusion.
22+
23+
DataFusion lets you run queries faster than pandas. Let's compare query runtimes for a 5GB CSV file with 100 million rows of data.
24+
25+
Take a look at a few rows of the data:
26+
27+
```
28+
+-------+-------+--------------+-----+-----+-------+----+----+-----------+
29+
| id1 | id2 | id3 | id4 | id5 | id6 | v1 | v2 | v3 |
30+
+-------+-------+--------------+-----+-----+-------+----+----+-----------+
31+
| id016 | id016 | id0000042202 | 15 | 24 | 5971 | 5 | 11 | 37.211254 |
32+
| id039 | id045 | id0000029558 | 40 | 49 | 39457 | 5 | 4 | 48.951141 |
33+
| id047 | id023 | id0000071286 | 68 | 20 | 74463 | 2 | 14 | 60.469241 |
34+
+-------+-------+--------------+-----+-----+-------+----+----+-----------+
35+
```
36+
37+
Suppose you'd like to run the following query: `SELECT id1, sum(v1) AS v1 from the_table GROUP BY id1`.
38+
39+
If you use pandas, then this query will take 43.6 seconds to execute.
40+
41+
It only takes DataFusion 9.8 seconds to execute the same query.
42+
43+
DataFusion is easy to use, powerful, and fast. Let's learn more!

‎docs/mdbook/src/installation.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
Unless required by applicable law or agreed to in writing,
11+
software distributed under the License is distributed on an
12+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
13+
KIND, either express or implied. See the License for the
14+
specific language governing permissions and limitations
15+
under the License.
16+
-->
17+
# Installation
18+
19+
DataFusion is easy to install, just like any other Python library.
20+
21+
## Using pip
22+
23+
``` bash
24+
pip install datafusion
25+
```
26+
27+
## Conda & JupyterLab setup
28+
29+
This section explains how to install DataFusion in a conda environment with other libraries that allow for a nice Jupyter workflow. This setup is completely optional. These steps are only needed if you'd like to run DataFusion in a Jupyter notebook and have an interface like this:
30+
31+
![DataFusion in Jupyter](https://github.com/MrPowers/datafusion-book/raw/main/src/images/datafusion-jupyterlab.png)
32+
33+
Create a conda environment with DataFusion, Jupyter, and other useful dependencies in the `datafusion-env.yml` file:
34+
35+
```
36+
name: datafusion-env
37+
channels:
38+
- conda-forge
39+
- defaults
40+
dependencies:
41+
- python=3.9
42+
- ipykernel
43+
- nb_conda
44+
- jupyterlab
45+
- jupyterlab_code_formatter
46+
- isort
47+
- black
48+
- pip
49+
- pip:
50+
- datafusion
51+
52+
```
53+
54+
Create the environment with `conda env create -f datafusion-env.yml`.
55+
56+
Activate the environment with `conda activate datafusion-env`.
57+
58+
Run `jupyter lab` or open the [JupyterLab Desktop application](https://github.com/jupyterlab/jupyterlab-desktop) to start running DataFusion in a Jupyter notebook.
59+
60+
## Examples
61+
62+
See the [DataFusion Python Examples](https://github.com/apache/arrow-datafusion-python/tree/main/examples) for a variety of Python scripts that show DataFusion in action!

‎docs/mdbook/src/quickstart.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
Unless required by applicable law or agreed to in writing,
11+
software distributed under the License is distributed on an
12+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
13+
KIND, either express or implied. See the License for the
14+
specific language governing permissions and limitations
15+
under the License.
16+
-->
17+
# DataFusion Quickstart
18+
19+
You can easily query a DataFusion table with the Python API or with pure SQL.
20+
21+
Let's create a small DataFrame and then run some queries with both APIs.
22+
23+
Start by creating a DataFrame with four rows of data and two columns: `a` and `b`.
24+
25+
```python
26+
from datafusion import SessionContext
27+
28+
ctx = SessionContext()
29+
30+
df = ctx.from_pydict({"a": [1, 2, 3, 1], "b": [4, 5, 6, 7]}, name="my_table")
31+
```
32+
33+
Let's append a column to this DataFrame that adds columns `a` and `b` with the SQL API.
34+
35+
```
36+
ctx.sql("select a, b, a + b as sum_a_b from my_table")
37+
38+
+---+---+---------+
39+
| a | b | sum_a_b |
40+
+---+---+---------+
41+
| 1 | 4 | 5 |
42+
| 2 | 5 | 7 |
43+
| 3 | 6 | 9 |
44+
| 1 | 7 | 8 |
45+
+---+---+---------+
46+
```
47+
48+
DataFusion makes it easy to run SQL queries on DataFrames.
49+
50+
Now let's run the same query with the DataFusion Python API:
51+
52+
```python
53+
from datafusion import col
54+
55+
df.select(
56+
col("a"),
57+
col("b"),
58+
col("a") + col("b"),
59+
)
60+
```
61+
62+
We get the same result as before:
63+
64+
```
65+
+---+---+-------------------------+
66+
| a | b | my_table.a + my_table.b |
67+
+---+---+-------------------------+
68+
| 1 | 4 | 5 |
69+
| 2 | 5 | 7 |
70+
| 3 | 6 | 9 |
71+
| 1 | 7 | 8 |
72+
+---+---+-------------------------+
73+
```
74+
75+
DataFusion also allows you to query data with a well-designed Python interface.
76+
77+
Python users have two great ways to query DataFusion tables.
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
Unless required by applicable law or agreed to in writing,
11+
software distributed under the License is distributed on an
12+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
13+
KIND, either express or implied. See the License for the
14+
specific language governing permissions and limitations
15+
under the License.
16+
-->
17+
# DataFusion Create Table
18+
19+
It's easy to create DataFusion tables from a variety of data sources.
20+
21+
## Create Table from Python Dictionary
22+
23+
Here's how to create a DataFusion table from a Python dictionary:
24+
25+
```python
26+
from datafusion import SessionContext
27+
28+
ctx = SessionContext()
29+
30+
df = ctx.from_pydict({"a": [1, 2, 3, 1], "b": [4, 5, 6, 7]}, name="my_table")
31+
```
32+
33+
Supplying the `name` parameter is optional. You only need to name the table if you'd like to query it with the SQL API.
34+
35+
You can also create a DataFrame without a name that can be queried with the Python API:
36+
37+
```python
38+
from datafusion import SessionContext
39+
40+
ctx = SessionContext()
41+
42+
df = ctx.from_pydict({"a": [1, 2, 3, 1], "b": [4, 5, 6, 7]})
43+
```
44+
45+
## Create Table from CSV
46+
47+
You can read a CSV into a DataFusion DataFrame. Here's how to read the `G1_1e8_1e2_0_0.csv` file into a table named `csv_1e8`:
48+
49+
```python
50+
ctx.register_csv("csv_1e8", "G1_1e8_1e2_0_0.csv")
51+
```
52+
53+
## Create Table from Parquet
54+
55+
You can read a Parquet file into a DataFusion DataFrame. Here's how to read the `yellow_tripdata_2021-01.parquet` file into a table named `taxi`.
56+
57+
```python
58+
ctx.register_parquet("taxi", "yellow_tripdata_2021-01.parquet")
59+
```

‎docs/mdbook/src/usage/index.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
Unless required by applicable law or agreed to in writing,
11+
software distributed under the License is distributed on an
12+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
13+
KIND, either express or implied. See the License for the
14+
specific language governing permissions and limitations
15+
under the License.
16+
-->
17+
# Usage
18+
19+
This section shows how to create DataFusion DataFrames from a variety of data sources like CSV files and Parquet files.
20+
21+
You'll learn more about the SQL statements that are supported by DataFusion.
22+
23+
You'll also learn about the DataFusion's Python API for querying data.
24+
25+
The documentation will wrap up with a variety of real-world data processing tasks that are well suited for DataFusion. The lightning-fast speed and reliable execution makes DataFusion the best technology for a variety of data processing tasks.

0 commit comments

Comments
 (0)