Skip to content

Commit b3cd616

Browse files
jacques-nwesm
jacques-n
authored and
wesm
committed
ARROW-249: [JAVA] Flight GRPC Implementation
- A grpc defined protocol (flight.proto) - A Java implementation of the GRPC-based FlightService framework - An Example Java implementation of a FlightService that provides an in-memory store for Flight streams - A short demo script to show how to use the FlightService from Java and Python Author: Jacques Nadeau <jacques@apache.org> Closes apache#2102 from jacques-n/flight and squashes the following commits: 135a2d7 <Jacques Nadeau> Update NOTICE for additional source code included. 01bdc07 <Jacques Nadeau> Update for new checkstyle rules 2d00e1f <Jacques Nadeau> Fix checkstyle warnings c5a065a <Jacques Nadeau> Update code for change in FlightService GRPC package. 723914d <Jacques Nadeau> Ensure full shutdown of flight server. 12924a2 <Jacques Nadeau> Add license headers 1e5733f <Jacques Nadeau> Attempt to correct memory leakage on Linux. Enable assertions in flight temporarily to see where memory is leaking. fc9a490 <Jacques Nadeau> Remove use of Guava from arrow-vector and arrow-memory modules so change in version doesn't impact downstream consumers. 03babac <Jacques Nadeau> Rebase and address new checkstyle rules. 0a12b35 <Jacques Nadeau> More Updates d42d58f <Jacques Nadeau> backpressure explore efd6409 <Jacques Nadeau> Update action to support streaming response. Add initial proto for handshake. faf567f <Jacques Nadeau> Initial perf test server 7f1f2d1 <Jacques Nadeau> More cleanup and comments on proto. b7e45f8 <Jacques Nadeau> More comments and some cleanups based on comments d5be1fb <Jacques Nadeau> ARROW-249 Flight GRPC Implementation
1 parent ea8940a commit b3cd616

97 files changed

Lines changed: 6929 additions & 282 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
*.so.*
2525
*.dylib
2626
.build_cache_dir
27+
dependency-reduced-pom.xml
2728
MANIFEST
2829

2930
# Generated Visual Studio files

‎NOTICE.txt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,14 @@ This product includes software from the Ibis project (Apache 2.0)
4141
* Copyright (c) 2015 Cloudera, Inc.
4242
* https://github.com/cloudera/ibis
4343

44+
This product includes software from Dremio (Apache 2.0)
45+
* Copyright (C) 2017-2018 Dremio Corporation
46+
* https://github.com/dremio/dremio-oss
47+
48+
This product includes software from Google Guava (Apache 2.0)
49+
* Copyright (C) 2007 The Guava Authors
50+
* https://github.com/google/guava
51+
4452
The web site includes files generated by Jekyll.
4553

4654
--------------------------------------------------------------------------------

‎java/flight/README.md‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# Arrow Flight Java Package
21+
22+
Exposing Apache Arrow data on the wire.
23+
24+
[Protocol Description Slides](https://www.slideshare.net/JacquesNadeau5/apache-arrow-flight-overview)
25+
26+
[GRPC Protocol Definition](https://github.com/jacques-n/arrow/blob/flight/java/flight/src/main/protobuf/flight.proto)
27+
28+
## Example usage
29+
30+
* Compile the java tree:
31+
32+
```
33+
cd java
34+
mvn clean install -DskipTests
35+
```
36+
37+
* Go Into the Flight tree
38+
39+
```
40+
cd flight
41+
```
42+
43+
44+
* Start the ExampleFlightServer (supports get/put of streams and listing these streams)
45+
46+
```
47+
mvn exec:exec
48+
```
49+
50+
* In new terminal, run the TestExampleServer to populate the server with example data
51+
52+
```
53+
cd arrow/java/flight
54+
mvn surefire:test -DdisableServer=true -Dtest=TestExampleServer
55+
```
56+
57+
## Python Example Usage
58+
59+
* Compile example python headers
60+
61+
```
62+
mkdir target/generated-python
63+
pip install grpcio-tools # or conda install grpcio
64+
python -m grpc_tools.protoc -I./src/main/protobuf/ --python_out=./target/generated-python --grpc_python_out=./target/generated-python src/main/protobuf/flight.proto
65+
```
66+
67+
* Connect to the Flight Service
68+
69+
```
70+
cd target/generated-python
71+
python
72+
```
73+
74+
75+
```
76+
import grpc
77+
import flight_pb2
78+
import flight_pb2_grpc as flightrpc
79+
channel = grpc.insecure_channel('localhost:12233')
80+
service = flightrpc.FlightServiceStub(channel)
81+
```
82+
83+
* List the Flight from Python
84+
85+
```
86+
for f in service.ListFlights(flight_pb2.Criteria()): f
87+
```
88+
89+
* Try to Drop
90+
91+
```
92+
action = flight_pb2.Action()
93+
action.type="drop"
94+
service.DoAction(action)
95+
```

‎java/flight/pom.xml‎

Lines changed: 235 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,235 @@
1+
<?xml version="1.0"?>
2+
<!-- Copyright (C) 2017-2018 Dremio Corporation Licensed under the Apache
3+
License, Version 2.0 (the "License"); you may not use this file except in
4+
compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
5+
Unless required by applicable law or agreed to in writing, software distributed
6+
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
7+
OR CONDITIONS OF ANY KIND, either express or implied. See the License for
8+
the specific language governing permissions and limitations under the License. -->
9+
<project xmlns="http://maven.apache.org/POM/4.0.0"
10+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
11+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
12+
<modelVersion>4.0.0</modelVersion>
13+
<parent>
14+
<groupId>org.apache.arrow</groupId>
15+
<artifactId>arrow-java-root</artifactId>
16+
<version>0.11.0-SNAPSHOT</version>
17+
</parent>
18+
19+
<artifactId>arrow-flight</artifactId>
20+
<name>Arrow Flight</name>
21+
<packaging>jar</packaging>
22+
23+
<properties>
24+
<dep.grpc.version>1.14.0</dep.grpc.version>
25+
<dep.protobuf.version>3.5.1</dep.protobuf.version>
26+
<forkCount>1</forkCount>
27+
</properties>
28+
29+
<dependencies>
30+
<dependency>
31+
<groupId>com.vlkan</groupId>
32+
<artifactId>flatbuffers</artifactId>
33+
</dependency>
34+
<dependency>
35+
<groupId>org.apache.arrow</groupId>
36+
<artifactId>arrow-vector</artifactId>
37+
<version>${project.version}</version>
38+
</dependency>
39+
<dependency>
40+
<groupId>org.apache.arrow</groupId>
41+
<artifactId>arrow-memory</artifactId>
42+
<version>${project.version}</version>
43+
</dependency>
44+
<dependency>
45+
<groupId>org.apache.arrow</groupId>
46+
<artifactId>arrow-format</artifactId>
47+
<version>${project.version}</version>
48+
</dependency>
49+
<dependency>
50+
<groupId>io.grpc</groupId>
51+
<artifactId>grpc-netty</artifactId>
52+
<version>${dep.grpc.version}</version>
53+
<scope>provided</scope>
54+
</dependency>
55+
<dependency>
56+
<groupId>io.grpc</groupId>
57+
<artifactId>grpc-netty</artifactId>
58+
<version>${dep.grpc.version}</version>
59+
<scope>provided</scope>
60+
</dependency>
61+
<dependency>
62+
<groupId>io.grpc</groupId>
63+
<artifactId>grpc-core</artifactId>
64+
<version>${dep.grpc.version}</version>
65+
<scope>provided</scope>
66+
</dependency>
67+
<dependency>
68+
<groupId>io.grpc</groupId>
69+
<artifactId>grpc-protobuf</artifactId>
70+
<version>${dep.grpc.version}</version>
71+
<scope>provided</scope>
72+
</dependency>
73+
<dependency>
74+
<groupId>io.netty</groupId>
75+
<artifactId>netty-tcnative-boringssl-static</artifactId>
76+
<version>2.0.12.Final</version>
77+
</dependency>
78+
<dependency>
79+
<groupId>io.netty</groupId>
80+
<artifactId>netty-buffer</artifactId>
81+
</dependency>
82+
<dependency>
83+
<groupId>com.google.guava</groupId>
84+
<artifactId>guava</artifactId>
85+
</dependency>
86+
<dependency>
87+
<groupId>io.grpc</groupId>
88+
<artifactId>grpc-stub</artifactId>
89+
<version>${dep.grpc.version}</version>
90+
<scope>provided</scope>
91+
</dependency>
92+
<dependency>
93+
<groupId>com.google.protobuf</groupId>
94+
<artifactId>protobuf-java</artifactId>
95+
<version>3.5.1</version>
96+
</dependency>
97+
98+
<dependency>
99+
<groupId>com.fasterxml.jackson.core</groupId>
100+
<artifactId>jackson-core</artifactId>
101+
</dependency>
102+
<dependency>
103+
<groupId>com.fasterxml.jackson.core</groupId>
104+
<artifactId>jackson-annotations</artifactId>
105+
</dependency>
106+
<dependency>
107+
<groupId>com.fasterxml.jackson.core</groupId>
108+
<artifactId>jackson-databind</artifactId>
109+
</dependency>
110+
<dependency>
111+
<groupId>org.slf4j</groupId>
112+
<artifactId>slf4j-api</artifactId>
113+
</dependency>
114+
</dependencies>
115+
<build>
116+
<extensions>
117+
<extension>
118+
<groupId>kr.motd.maven</groupId>
119+
<artifactId>os-maven-plugin</artifactId>
120+
<version>1.5.0.Final</version>
121+
</extension>
122+
</extensions>
123+
<plugins>
124+
<plugin>
125+
<artifactId>maven-surefire-plugin</artifactId>
126+
<configuration>
127+
<enableAssertions>false</enableAssertions>
128+
</configuration>
129+
</plugin>
130+
<plugin>
131+
<groupId>org.apache.maven.plugins</groupId>
132+
<artifactId>maven-shade-plugin</artifactId>
133+
<version>3.1.1</version>
134+
<executions>
135+
<execution>
136+
<phase>package</phase>
137+
<goals>
138+
<goal>shade</goal>
139+
</goals>
140+
<configuration>
141+
<shadedArtifactAttached>true</shadedArtifactAttached>
142+
<shadedClassifierName>shaded</shadedClassifierName>
143+
<artifactSet>
144+
<includes>
145+
<include>io.grpc:*</include>
146+
<include>com.google.protobuf:*</include>
147+
</includes>
148+
</artifactSet>
149+
<relocations>
150+
<relocation>
151+
<pattern>com.google.protobuf</pattern>
152+
<shadedPattern>arrow.flight.com.google.protobuf</shadedPattern>
153+
</relocation>
154+
</relocations>
155+
<transformers>
156+
<transformer
157+
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
158+
</transformers>
159+
</configuration>
160+
</execution>
161+
</executions>
162+
</plugin>
163+
<plugin>
164+
<groupId>org.xolstice.maven.plugins</groupId>
165+
<artifactId>protobuf-maven-plugin</artifactId>
166+
<version>0.5.0</version>
167+
<configuration>
168+
<protocArtifact>com.google.protobuf:protoc:${dep.protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
169+
<clearOutputDirectory>false</clearOutputDirectory>
170+
<pluginId>grpc-java</pluginId>
171+
<pluginArtifact>io.grpc:protoc-gen-grpc-java:${dep.grpc.version}:exe:${os.detected.classifier}</pluginArtifact>
172+
</configuration>
173+
<executions>
174+
<execution>
175+
<id>src</id>
176+
<configuration>
177+
<protoSourceRoot>${basedir}/../../format/</protoSourceRoot>
178+
<outputDirectory>${project.build.directory}/generated-sources/protobuf</outputDirectory>
179+
</configuration>
180+
<goals>
181+
<goal>compile</goal>
182+
<goal>compile-custom</goal>
183+
</goals>
184+
</execution>
185+
<execution>
186+
<id>test</id>
187+
<configuration>
188+
<protoSourceRoot>${basedir}/src/test/protobuf</protoSourceRoot>
189+
<outputDirectory>${project.build.directory}/generated-test-sources//protobuf</outputDirectory>
190+
</configuration>
191+
<goals>
192+
<goal>compile</goal>
193+
<goal>compile-custom</goal>
194+
</goals>
195+
</execution>
196+
</executions>
197+
</plugin>
198+
<plugin>
199+
<groupId>org.codehaus.mojo</groupId>
200+
<artifactId>exec-maven-plugin</artifactId>
201+
<version>1.6.0</version>
202+
<configuration>
203+
<executable>java</executable>
204+
<classpathScope>test</classpathScope>
205+
<arguments>
206+
<argument>-classpath</argument>
207+
<classpath/>
208+
<argument>-Xms64m</argument>
209+
<argument>-Xmx64m</argument>
210+
<argument>-XX:MaxDirectMemorySize=4g</argument>
211+
<argument>org.apache.arrow.flight.example.ExampleFlightServer</argument>
212+
</arguments>
213+
</configuration>
214+
</plugin>
215+
<plugin>
216+
<groupId>org.apache.maven.plugins</groupId>
217+
<artifactId>maven-dependency-plugin</artifactId>
218+
<executions>
219+
<execution>
220+
<id>analyze</id>
221+
<phase>verify</phase>
222+
<goals>
223+
<goal>analyze-only</goal>
224+
</goals>
225+
<configuration>
226+
<ignoredDependencies combine.self="append">
227+
<ignoredDependency>io.netty:netty-tcnative-boringssl-static:*</ignoredDependency>
228+
</ignoredDependencies>
229+
</configuration>
230+
</execution>
231+
</executions>
232+
</plugin>
233+
</plugins>
234+
</build>
235+
</project>

0 commit comments

Comments
 (0)