ARROW-7277: [Java] [Doc] Add discussion about vector lifecycle

liyafan82 · emkornfield · commit b178e152ea08 · 2020-01-10T20:34:21.000-08:00
As discussed in https://issues.apache.org/jira/browse/ARROW-7254?focusedCommentId=16983284&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16983284, we need a discussion about the lifecycle of a vector. Each vector has a lifecycle, and different operations should be performed in particular phases of the lifecycle. If we violate this, some unexpected results may be produced. This may cause some confusion for Arrow users. So we want to add a new section to the prose document, to make it clear and explicit. Closes apache#5969 from liyafan82/fly_1203_doc and squashes the following commits: 48be293 <liyafan82> Resolve comments 3ad95de <liyafan82> Fix styles for other Java documents 030d9da <liyafan82> Add discussion about vector lifecycle Authored-by: liyafan82 <fan_li_ya@foxmail.com> Signed-off-by: Micah Kornfield <emkornfield@gmail.com>
diff --git a/docs/source/java/ipc.rst b/docs/source/java/ipc.rst
@@ -30,7 +30,9 @@ Arrow defines two types of binary formats for serializing record batches:
 
 Writing and Reading Streaming Format
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-First, let's populate a :class:`VectorSchemaRoot` with a small batch of records::
+First, let's populate a :class:`VectorSchemaRoot` with a small batch of records
+
+.. code-block:: Java
 
     BitVector bitVector = new BitVector("boolean", allocator);
     VarCharVector varCharVector = new VarCharVector("varchar", allocator);
@@ -52,7 +54,9 @@ Now, we can begin writing a stream containing some number of these batches. For
     ArrowStreamWriter writer = new ArrowStreamWriter(root, /*DictionaryProvider=*/null, Channels.newChannel(out));
 
 
-Here we used an in-memory stream, but this could have been a socket or some other IO stream. Then we can do::
+Here we used an in-memory stream, but this could have been a socket or some other IO stream. Then we can do
+
+.. code-block:: Java
 
     writer.start();
     // write the first batch
@@ -78,7 +82,9 @@ could overwrite previous ones.
 
 Now the :class:`ByteArrayOutputStream` contains the complete stream which contains 5 record batches.
 We can read such a stream with :class:`ArrowStreamReader`, note that :class:`VectorSchemaRoot` within
-reader will be loaded with new values on every call to :class:`loadNextBatch()`::
+reader will be loaded with new values on every call to :class:`loadNextBatch()`
+
+.. code-block:: Java
 
     try (ArrowStreamReader reader = new ArrowStreamReader(new ByteArrayInputStream(out.toByteArray()), allocator)) {
       Schema schema = reader.getVectorSchemaRoot().getSchema();
@@ -91,7 +97,9 @@ reader will be loaded with new values on every call to :class:`loadNextBatch()`:
 
     }
 
-Here we also give a simple example with dictionary encoded vectors::
+Here we also give a simple example with dictionary encoded vectors
+
+.. code-block:: Java
 
     DictionaryProvider.MapDictionaryProvider provider = new DictionaryProvider.MapDictionaryProvider();
     // create dictionary and provider
@@ -147,7 +155,9 @@ Here we also give a simple example with dictionary encoded vectors::
 
 Writing and Reading Random Access Files
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The :class:`ArrowFileWriter` has the same API as :class:`ArrowStreamWriter`::
+The :class:`ArrowFileWriter` has the same API as :class:`ArrowStreamWriter`
+
+.. code-block:: Java
 
     ByteArrayOutputStream out = new ByteArrayOutputStream();
     ArrowFileWriter writer = new ArrowFileWriter(root, null, Channels.newChannel(out));
@@ -163,7 +173,9 @@ The :class:`ArrowFileWriter` has the same API as :class:`ArrowStreamWriter`::
 
 The difference between :class:`ArrowFileReader` and :class:`ArrowStreamReader` is that the input source
 must have a ``seek`` method for random access. Because we have access to the entire payload, we know the
-number of record batches in the file, and can read any at random::
+number of record batches in the file, and can read any at random
+
+.. code-block:: Java
 
     try (ArrowFileReader reader = new ArrowFileReader(
         new ByteArrayReadableSeekableByteChannel(out.toByteArray()), allocator)) {
diff --git a/docs/source/java/vector.rst b/docs/source/java/vector.rst
@@ -31,7 +31,92 @@ Table with non-intuitive names (BigInt = 64 bit integer, etc).
 
 It is important that vector is allocated before attempting to read or write,
 :class:`ValueVector` "should" strive to guarantee this order of operation:
-allocate > mutate > set valuecount > access > clear (or allocate to start the process over)
+create > allocate > mutate > set value count > access > clear (or allocate to start the process over).
+We will go through a concrete example to demonstrate each operation in the next section.
+
+Vector Life Cycle
+====================
+As discussed above, each vector goes through several steps in its life cycle,
+and each step is triggered by a vector operation. In particular, we have the following vector operations:
+
+1. **Vector creation**: we create a new vector object by, for example, the vector constructor.
+The following code creates a new ``IntVector`` by the constructor:
+
+.. code-block:: Java
+
+    RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+    ...
+    IntVector vector = new IntVector("int vector", allocator);
+
+By now, a vector object is created. However, no underlying memory has been allocated, so we need the
+following step.
+
+2. **Vector allocation**: in this step, we allocate memory for the vector. For most vectors, we
+have two options: 1) if we know the maximum vector capacity, we can specify it by calling the
+``allocateNew(int)`` method; 2) otherwise, we should call the ``allocateNew()`` method, and  a default
+capacity will be allocated for it. For our running example, we assume that the vector capacity never
+exceeds 10:
+
+.. code-block:: Java
+
+    vector.allocateNew(10);
+
+3. **Vector mutation**: now we can populate the vector with values we desire. For all vectors, we can populate
+vector values through vector writers (An example will be given in the next section). For primitive types,
+we can also mutate the vector by the set methods. There are two classes of set methods: 1) if we can
+be sure the vector has enough capacity, we can call the ``set(index, value)`` method. 2) if we are not sure
+about the vector capacity, we should call the ``setSafe(index, value)`` method, which will automatically
+take care of vector reallocation, if the capacity is not sufficient. For our running example, we know the
+vector has enough capacity, so we can call
+
+.. code-block:: Java
+
+    vector.set(/*index*/5, /*value*/25);
+
+4. **Set value count**: for this step, we set the value count of the vector by calling the
+``setValueCount(int)`` method:
+
+.. code-block:: Java
+
+    vector.setValueCount(10);
+
+After this step, the vector enters an immutable state. In other words, we should no longer mutate it.
+(Unless we reuse the vector by allocating it again. This will be discussed shortly.)
+
+5. **Vector access**: it is time to access vector values. Similarly, we have two options to access values:
+1) get methods and 2) vector reader. Vector reader works for all types of vectors, while get methods are
+only available for primitive vectors. A concrete example for vector reader will be given in the next section.
+Below is an example of vector access by get method:
+
+.. code-block:: Java
+
+    int value = vector.get(5);  // value == 25
+
+6. **Vector clear**: when we are done with the vector, we should clear it to release its memory. This is done by
+calling the ``close()`` method:
+
+.. code-block:: Java
+
+    vector.close();
+
+Some points to note about the steps above:
+
+* The steps are not necessarily performed in a linear sequence. Instead, they can be in a loop. For example,
+  when a vector enters the access step, we can also go back to the vector mutation step, and then set value
+  count, access vector, and so on.
+
+* We should try to make sure the above steps are carried out in order. Otherwise, the vector
+  may be in an undefined state, and some unexpected behavior may occur. However, this restriction
+  is not strict. That means it is possible that we violates the order above, but still get
+  correct results.
+
+* When mutating vector values through set methods, we should prefer ``set(index, value)`` methods to
+  ``setSafe(index, value)`` methods whenever possible, to avoid unnecessary performance overhead of handling
+  vector capacity.
+
+* All vectors implement the ``AutoCloseable`` interface. So they must be closed explicitly when they are
+  no longer used, to avoid resource leak. To make sure of this, it is recommended to place vector related operations
+  into a try-with-resources block.
 
 Building ValueVector
 ====================
@@ -41,50 +126,58 @@ Note that the current implementation doesn't enforce the rule that Arrow objects
 set/setSafe APIs and concrete subclasses of FieldWriter for populating values.
 
 For example, the code below shows how to build a :class:`BigIntVector`, in this case, we build a
-vector of the range 0 to 7 where the element that should hold the fourth value is nulled::
-
-   BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE);
-
-   BigIntVector vector = new BigIntVector("vector", allocator);
-   vector.allocateNew(8);
-   vector.set(0, 1);
-   vector.set(1, 2);
-   vector.set(2, 3);
-   vector.setNull(3);
-   vector.set(4, 5);
-   vector.set(5, 6);
-   vector.set(6, 7);
-   vector.set(7, 8);
-   vector.setValueCount(8); // this will finalizes the vector by convention.
+vector of the range 0 to 7 where the element that should hold the fourth value is nulled
+
+.. code-block:: Java
+
+    try (BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+      BigIntVector vector = new BigIntVector("vector", allocator)) {
+      vector.allocateNew(8);
+      vector.set(0, 1);
+      vector.set(1, 2);
+      vector.set(2, 3);
+      vector.setNull(3);
+      vector.set(4, 5);
+      vector.set(5, 6);
+      vector.set(6, 7);
+      vector.set(7, 8);
+      vector.setValueCount(8); // this will finalizes the vector by convention.
+      ...
+    }
 
 The :class:`BigIntVector` holds two ArrowBufs. The first buffer holds the null bitmap, which consists
 here of a single byte with the bits 1|1|1|1|0|1|1|1 (the bit is 1 if the value is non-null).
 The second buffer contains all the above values. As the fourth entry is null, the value at that position
 in the buffer is undefined. Note compared with set API, setSafe API would check value capacity before setting
 values and reallocate buffers if necessary.
 
-Here is how to build a vector using writer::
-
-   BigIntVector vector = new BigIntVector("vector", allocator);
-   BigIntWriter writer = new BigIntWriterImpl(vector);
-   writer.setPosition(0);
-   writer.writeBigInt(1);
-   writer.setPosition(1);
-   writer.writeBigInt(2);
-   writer.setPosition(2);
-   writer.writeBigInt(3);
-   // writer.setPosition(3) is not called which means the forth value is null.
-   writer.setPosition(4);
-   writer.writeBigInt(5);
-   writer.setPosition(5);
-   writer.writeBigInt(6);
-   writer.setPosition(6);
-   writer.writeBigInt(7);
-   writer.setPosition(7);
-   writer.writeBigInt(8);
+Here is how to build a vector using writer
+
+.. code-block:: Java
+
+    try (BigIntVector vector = new BigIntVector("vector", allocator);
+      BigIntWriter writer = new BigIntWriterImpl(vector)) {
+      writer.setPosition(0);
+      writer.writeBigInt(1);
+      writer.setPosition(1);
+      writer.writeBigInt(2);
+      writer.setPosition(2);
+      writer.writeBigInt(3);
+      // writer.setPosition(3) is not called which means the forth value is null.
+      writer.setPosition(4);
+      writer.writeBigInt(5);
+      writer.setPosition(5);
+      writer.writeBigInt(6);
+      writer.setPosition(6);
+      writer.writeBigInt(7);
+      writer.setPosition(7);
+      writer.writeBigInt(8);
+    }
 
 There are get API and concrete subclasses of :class:`FieldReader` for accessing vector values, what needs
-to be declared is that writer/reader is not as efficient as direct access::
+to be declared is that writer/reader is not as efficient as direct access
+
+.. code-block:: Java
 
     // access via get API
     for (int i = 0; i < vector.getValueCount(); i++) {
@@ -106,7 +199,9 @@ to be declared is that writer/reader is not as efficient as direct access::
 Slicing
 =======
 Similar with C++ implementation, it is possible to make zero-copy slices of vectors to obtain a vector
-referring to some logical sub-sequence of the data through :class:`TransferPair`::
+referring to some logical sub-sequence of the data through :class:`TransferPair`
+
+.. code-block:: Java
 
     IntVector vector = new IntVector("intVector", allocator);
     for (int i = 0; i < 10; i++) {
diff --git a/docs/source/java/vector_schema_root.rst b/docs/source/java/vector_schema_root.rst
@@ -30,7 +30,9 @@ of batches rather than creating a new :class:`VectorSchemaRoot` instance each ti
 may have no data (say it was transferred downstream or not yet populated).
 
 
-Here is the example of building a :class:`VectorSchemaRoot`::
+Here is the example of building a :class:`VectorSchemaRoot`
+
+.. code-block:: Java
 
     BitVector bitVector = new BitVector("boolean", allocator);
     VarCharVector varCharVector = new VarCharVector("varchar", allocator);
@@ -49,7 +51,9 @@ Here is the example of building a :class:`VectorSchemaRoot`::
 
 The vectors within a :class:`VectorSchemaRoot` could be loaded/unloaded via :class:`VectorLoader` and :class:`VectorUnloader`.
 :class:`VectorLoader` and :class:`VectorUnloader` handles converting between :class:`VectorSchemaRoot` and :class:`ArrowRecordBatch`(
-representation of a RecordBatch :doc:`IPC <../format/IPC.rst>` message). Examples as below::
+representation of a RecordBatch :doc:`IPC <../format/IPC.rst>` message). Examples as below
+
+.. code-block:: Java
 
     // create a VectorSchemaRoot root1 and convert its data into recordBatch
     VectorSchemaRoot root1 = new VectorSchemaRoot(fields, vectors);
@@ -61,7 +65,9 @@ representation of a RecordBatch :doc:`IPC <../format/IPC.rst>` message). Example
     VectorLoader loader = new VectorLoader(root2);
     loader.load(recordBatch);
 
-A new :class:`VectorSchemaRoot` could be sliced from an existing instance with zero-copy::
+A new :class:`VectorSchemaRoot` could be sliced from an existing instance with zero-copy
+
+.. code-block:: Java
 
     // 0 indicates start index (inclusive) and 5 indicated length (exclusive).
     VectorSchemaRoot newRoot = vectorSchemaRoot.slice(0, 5);
