Skip to content

Commit 35f8a34

Browse files
xhochywesm
xhochy
authored and
wesm
committed
ARROW-3834: [Doc] Merge C++ and Python documentation
Author: Uwe L. Korn <uwelk@xhochy.com> Author: Korn, Uwe <Uwe.Korn@blue-yonder.com> Author: Wes McKinney <wesm+git@apache.org> Closes apache#2856 from xhochy/doc-merge and squashes the following commits: 5b687ff <Wes McKinney> Add simple README for the format/ directory 071d16a <Uwe L. Korn> Move format specifications back to /format/ 337088b <Uwe L. Korn> Review comments fbe99c9 <Uwe L. Korn> Add more C++ docs 78a5eaf <Uwe L. Korn> Fix Python docs build 0b4dd33 <Uwe L. Korn> Rename doc to docs 918e762 <Uwe L. Korn> Convert format docs to reST 7aeff65 <Uwe L. Korn> Add doc generation to docker-compose 185cba8 <Uwe L. Korn> Add pre-commit check for RAT 671d244 <Uwe L. Korn> Fix references to format documents bdd824c <Uwe L. Korn> Move doc to top-level 985d428 <Uwe L. Korn> Move Sphinx doc to top-level directory f7d5e92 <Uwe L. Korn> Build C++ API docs 7850db8 <Uwe L. Korn> Add breathe as a requirement d4cf542 <Uwe L. Korn> Fix linter issues fd75660 <Korn, Uwe> Fix Sphinx build for sphinx>=1.8 9be6fbe <Korn, Uwe> Merge C++ and Python documentation
1 parent 187b98e commit 35f8a34

55 files changed

Lines changed: 1543 additions & 1202 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.dockerignore‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
.git
1919
docker_cache
20+
docs/_build
2021

2122
# IDE
2223
.vscode
@@ -49,7 +50,6 @@ python/dist
4950
python/*.egg-info
5051
python/*.egg
5152
python/*.pyc
52-
python/doc/_build
5353
__pycache__/
5454
*/__pycache__/
5555
*/*/__pycache__/

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,9 @@
1515
# specific language governing permissions and limitations
1616
# under the License.
1717

18+
apache-rat-*.jar
19+
arrow-src.tar
20+
1821
# Compiled source
1922
*.a
2023
*.dll
@@ -34,7 +37,9 @@ MANIFEST
3437
*.iml
3538

3639
cpp/.idea/
40+
cpp/apidoc/xml/
3741
python/.eggs/
42+
python/doc/
3843
.vscode
3944
.idea/
4045
.pytest_cache/

‎.pre-commit-config.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,14 @@
2121
# To run all hooks on all files use `pre-commit run -a`
2222

2323
repos:
24+
- repo: local
25+
hooks:
26+
- id: rat
27+
name: rat
28+
language: system
29+
entry: bash -c "git archive HEAD --prefix=apache-arrow/ --output=arrow-src.tar && ./dev/release/run-rat.sh arrow-src.tar"
30+
always_run: true
31+
pass_filenames: false
2432
- repo: git://github.com/pre-commit/pre-commit-hooks
2533
sha: v1.2.3
2634
hooks:

‎ci/conda_env_python.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,5 +21,6 @@ numpy
2121
pandas
2222
pytest
2323
python
24+
rsync
2425
setuptools
2526
setuptools_scm

‎ci/conda_env_sphinx.yml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
# Requirements for building the documentation
19+
breathe
20+
doxygen
21+
ipython
22+
sphinx
23+
sphinx_rtd_theme

‎ci/docker_build_sphinx.sh‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one or more
4+
# contributor license agreements. See the NOTICE file distributed with
5+
# this work for additional information regarding copyright ownership.
6+
# The ASF licenses this file to You under the Apache License, Version 2.0
7+
# (the "License"); you may not use this file except in compliance with
8+
# the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
#
18+
19+
set -ex
20+
21+
pushd /arrow/cpp/apidoc
22+
doxygen
23+
popd
24+
25+
pushd /arrow/python
26+
python setup.py build_sphinx -s ../docs/source --build-dir ../docs/_build
27+
popd
28+
29+
mkdir -p /arrow/site/asf-site/docs/latest
30+
rsync -r /arrow/docs/_build/html/ /arrow/site/asf-site/docs/latest/

‎ci/travis_script_python.sh‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -61,11 +61,7 @@ conda install -y -q pip \
6161

6262
if [ "$ARROW_TRAVIS_PYTHON_DOCS" == "1" ] && [ "$PYTHON_VERSION" == "3.6" ]; then
6363
# Install documentation dependencies
64-
conda install -y -q \
65-
ipython \
66-
numpydoc \
67-
sphinx=1.7.9 \
68-
sphinx_rtd_theme
64+
conda install -y -c conda-forge --file ci/conda_env_sphinx.yml
6965
fi
7066

7167
# ARROW-2093: PyTorch increases the size of our conda dependency stack
@@ -190,7 +186,10 @@ if [ "$ARROW_TRAVIS_COVERAGE" == "1" ]; then
190186
fi
191187

192188
if [ "$ARROW_TRAVIS_PYTHON_DOCS" == "1" ] && [ "$PYTHON_VERSION" == "3.6" ]; then
193-
cd doc
189+
pushd ../cpp/apidoc
190+
doxygen
191+
popd
192+
cd ../docs
194193
sphinx-build -q -b html -d _build/doctrees -W source _build/html
195194
fi
196195

‎cpp/apidoc/Doxyfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1919,7 +1919,7 @@ MAN_LINKS = NO
19191919
# captures the structure of the code including all documentation.
19201920
# The default value is: NO.
19211921

1922-
GENERATE_XML = NO
1922+
GENERATE_XML = YES
19231923

19241924
# The XML_OUTPUT tag is used to specify where the XML pages will be put. If a
19251925
# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of

‎cpp/apidoc/index.md‎

Lines changed: 0 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -41,60 +41,3 @@ Table of Contents
4141
* [Convert a vector of row-wise data into an Arrow table](tutorials/row_wise_conversion.md)
4242
* [Using the Plasma In-Memory Object Store](tutorials/plasma.md)
4343
* [Use Plasma to Access Tensors from C++ in Python](tutorials/tensor_to_py.md)
44-
45-
Getting Started
46-
---------------
47-
48-
The most basic structure in Arrow is an `arrow::Array`. It holds a sequence
49-
of values with known length all having the same type. It consists of the data
50-
itself and an additional bitmap that indicates if the corresponding entry of
51-
array is a null-value. Note that for array with zero null entries, we can omit
52-
this bitmap.
53-
54-
As Arrow objects are immutable, there are classes provided that should help you
55-
build these objects. To build an array of `int64_t` elements, we can use the
56-
`arrow::Int64Builder`. In the following example, we build an array of the range
57-
1 to 8 where the element that should hold the number 4 is nulled.
58-
59-
Int64Builder builder;
60-
builder.Append(1);
61-
builder.Append(2);
62-
builder.Append(3);
63-
builder.AppendNull();
64-
builder.Append(5);
65-
builder.Append(6);
66-
builder.Append(7);
67-
builder.Append(8);
68-
69-
std::shared_ptr<Array> array;
70-
builder.Finish(&array);
71-
72-
The resulting Array (which can be casted to `arrow::Int64Array` if you want
73-
to access its values) then consists of two `arrow::Buffer`. The first one is
74-
the null bitmap holding a single byte with the bits `0|0|0|0|1|0|0|0`.
75-
As we use [least-significant bit (LSB) numbering](https://en.wikipedia.org/wiki/Bit_numbering)
76-
this indicates that the fourth entry in the array is null. The second
77-
buffer is simply an `int64_t` array containing all the above values.
78-
As the fourth entry is null, the value at that position in the buffer is
79-
undefined.
80-
81-
// Cast the Array to its actual type to access its data
82-
std::shared_ptr<Int64Array> int64_array = std::static_pointer_cast<Int64Array>(array);
83-
84-
// Get the pointer to the null bitmap.
85-
const uint8_t* null_bitmap = int64_array->null_bitmap_data();
86-
87-
// Get the pointer to the actual data
88-
const int64_t* data = int64_array->raw_values();
89-
90-
In the above example, we have yet skipped explaining two things in the code.
91-
On constructing the builder, we have passed `arrow::int64()` to it. This is
92-
the type information with which the resulting array will be annotated. In
93-
this simple form, it is solely a `std::shared_ptr<arrow::Int64Type>`
94-
instantiation.
95-
96-
Furthermore, we have passed `arrow::default_memory_pool()` to the constructor.
97-
This `arrow::MemoryPool` is used for the allocations of heap memory. Besides
98-
tracking the amount of memory allocated, the allocator also ensures that the
99-
allocated memory regions are 64-byte aligned (as required by the Arrow
100-
specification).

‎cpp/src/arrow/array.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -397,6 +397,7 @@ class ARROW_EXPORT PrimitiveArray : public FlatArray {
397397
const uint8_t* raw_values_;
398398
};
399399

400+
/// Concrete Array class for numeric data.
400401
template <typename TYPE>
401402
class ARROW_EXPORT NumericArray : public PrimitiveArray {
402403
public:

0 commit comments

Comments
 (0)