-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy pathintroduction.html
More file actions
265 lines (237 loc) · 16.2 KB
/
introduction.html
File metadata and controls
265 lines (237 loc) · 16.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="./">
<head>
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Introduction — Feldera Python SDK 0.275.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" />
<link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=19f00094" />
<!--[if lt IE 9]>
<script src="_static/js/html5shiv.min.js"></script>
<![endif]-->
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=057883d4"></script>
<script src="_static/doctools.js?v=9a2dae69"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Pandas Compatibility" href="pandas.html" />
<link rel="prev" title="Feldera Python SDK Documentation" href="index.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="index.html" class="icon icon-home">
Feldera Python SDK
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul class="current">
<li class="toctree-l1 current"><a class="current reference internal" href="#">Introduction</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#installation">Installation</a></li>
<li class="toctree-l2"><a class="reference internal" href="#key-concepts">Key Concepts</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="pandas.html">Pandas Compatibility</a></li>
<li class="toctree-l1"><a class="reference internal" href="examples.html">Examples</a></li>
</ul>
<p class="caption" role="heading"><span class="caption-text">API Reference:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="modules.html">feldera</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">Feldera Python SDK</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item active">Introduction</li>
<li class="wy-breadcrumbs-aside">
<a href="_sources/introduction.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="introduction">
<h1>Introduction<a class="headerlink" href="#introduction" title="Link to this heading"></a></h1>
<p>The Feldera Python SDK is meant to provide an easy and convenient way of
interacting with Feldera.</p>
<p>Please submit any feature request / bug reports to:
<a class="reference external" href="https://github.com/feldera/feldera">https://github.com/feldera/feldera</a></p>
<section id="installation">
<h2>Installation<a class="headerlink" href="#installation" title="Link to this heading"></a></h2>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>pip<span class="w"> </span>install<span class="w"> </span>feldera
</pre></div>
</div>
<p>Installing from Github:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>pip<span class="w"> </span>install<span class="w"> </span>git+https://github.com/feldera/feldera#subdirectory<span class="o">=</span>python
</pre></div>
</div>
<p>Similarly, to install from a specific branch:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>git+https://github.com/feldera/feldera@<span class="o">{</span>BRANCH_NAME<span class="o">}</span><span class="c1">#subdirectory=python</span>
</pre></div>
</div>
<p>Replace <code class="docutils literal notranslate"><span class="pre">{BRANCH_NAME}</span></code> with the name of the branch you want to install from.</p>
<p>Installing from Local Directory:</p>
<p>If you have cloned the Feldera repo, you can install the python SDK as follows:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># the Feldera Python SDK is present inside the python/ directory</span>
pip<span class="w"> </span>install<span class="w"> </span>python/
</pre></div>
</div>
</section>
<section id="key-concepts">
<h2>Key Concepts<a class="headerlink" href="#key-concepts" title="Link to this heading"></a></h2>
<ul>
<li><dl>
<dt><a class="reference internal" href="feldera.rest.html#feldera.rest.feldera_client.FelderaClient" title="feldera.rest.feldera_client.FelderaClient"><code class="xref py py-class docutils literal notranslate"><span class="pre">FelderaClient</span></code></a></dt><dd><ul>
<li><p>This is the actual HTTP client used to make requests to your Feldera
instance.</p></li>
<li><p>creating an instance of <a class="reference internal" href="feldera.rest.html#feldera.rest.feldera_client.FelderaClient" title="feldera.rest.feldera_client.FelderaClient"><code class="xref py py-class docutils literal notranslate"><span class="pre">FelderaClient</span></code></a> is usually the first thing you
will do while working with Feldera.</p></li>
<li><p>Example:</p>
<blockquote>
<div><div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">feldera</span><span class="w"> </span><span class="kn">import</span> <span class="n">FelderaClient</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">FelderaClient</span><span class="p">(</span><span class="s2">"https://try.feldera.com"</span><span class="p">,</span> <span class="n">api_key</span><span class="o">=</span><span class="s2">"YOUR_API_KEY"</span><span class="p">)</span>
</pre></div>
</div>
<ul class="simple">
<li><p>The API key may not be required if you are running Feldera locally.</p></li>
</ul>
</div></blockquote>
</li>
</ul>
</dd>
</dl>
</li>
<li><dl>
<dt><a class="reference internal" href="feldera.html#feldera.pipeline_builder.PipelineBuilder" title="feldera.pipeline_builder.PipelineBuilder"><code class="xref py py-class docutils literal notranslate"><span class="pre">PipelineBuilder</span></code></a></dt><dd><p>Builder class for constructing new Feldera pipelines. Configure the builder with pipeline name, SQL code, and other optional attributes and call one of two methods to create the pipeline, possibly overwriting an existing pipeline with the same name:</p>
<blockquote>
<div><ul class="simple">
<li><p><a class="reference internal" href="feldera.html#feldera.pipeline_builder.PipelineBuilder.create" title="feldera.pipeline_builder.PipelineBuilder.create"><code class="xref py py-meth docutils literal notranslate"><span class="pre">PipelineBuilder.create()</span></code></a></p></li>
<li><p><a class="reference internal" href="feldera.html#feldera.pipeline_builder.PipelineBuilder.create_or_replace" title="feldera.pipeline_builder.PipelineBuilder.create_or_replace"><code class="xref py py-meth docutils literal notranslate"><span class="pre">PipelineBuilder.create_or_replace()</span></code></a></p></li>
<li><p><code class="xref py py-meth docutils literal notranslate"><span class="pre">PipelineBuilder.get()</span></code></p></li>
</ul>
</div></blockquote>
<ul>
<li><p>Example:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">feldera</span><span class="w"> </span><span class="kn">import</span> <span class="n">PipelineBuilder</span>
<span class="n">pipeline</span> <span class="o">=</span> <span class="n">PipelineBuilder</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">"example"</span><span class="p">,</span> <span class="n">sql</span><span class="o">=</span><span class="n">sql</span><span class="p">)</span><span class="o">.</span><span class="n">create</span><span class="p">()</span>
</pre></div>
</div>
</li>
</ul>
</dd>
</dl>
</li>
<li><dl>
<dt><a class="reference internal" href="feldera.html#feldera.pipeline.Pipeline.start" title="feldera.pipeline.Pipeline.start"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Pipeline.start()</span></code></a></dt><dd><ul>
<li><p>Starts the Feldera Pipeline and keeps it running indefinitely.</p></li>
<li><p>Example:</p>
<blockquote>
<div><div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">pipeline</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</pre></div>
</div>
<ul class="simple">
<li><p>This tells Feldera to go ahead and start processing the data.</p></li>
</ul>
</div></blockquote>
</li>
</ul>
</dd>
</dl>
</li>
<li><dl>
<dt><a class="reference internal" href="feldera.html#feldera.pipeline.Pipeline.wait_for_completion" title="feldera.pipeline.Pipeline.wait_for_completion"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Pipeline.wait_for_completion()</span></code></a></dt><dd><ul>
<li><p>Blocks this pipeline until completion, i.e., until the end-of-file (EOI)
has been reached for all input sources.</p></li>
<li><p>Takes a parameter <code class="docutils literal notranslate"><span class="pre">force_stop</span></code>, when set stops the pipeline down after completion.</p></li>
<li><p>Example:</p>
<blockquote>
<div><div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">feldera</span><span class="w"> </span><span class="kn">import</span> <span class="n">FelderaClient</span><span class="p">,</span> <span class="n">PipelineBuilder</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">pandas</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">pd</span>
<span class="n">tbl_name</span> <span class="o">=</span> <span class="s2">"user_data"</span>
<span class="n">view_name</span> <span class="o">=</span> <span class="s2">"select_view"</span>
<span class="n">sql</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"""</span>
<span class="s2"> -- Declare input tables</span>
<span class="s2"> CREATE TABLE </span><span class="si">{</span><span class="n">tbl_name</span><span class="si">}</span><span class="s2"> (name STRING);</span>
<span class="s2"> -- Create Views based on your queries</span>
<span class="s2"> CREATE VIEW </span><span class="si">{</span><span class="n">view_name</span><span class="si">}</span><span class="s2"> AS SELECT * FROM </span><span class="si">{</span><span class="n">tbl_name</span><span class="si">}</span><span class="s2">;</span>
<span class="s2">"""</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">FelderaClient</span><span class="p">(</span><span class="s2">"https://try.feldera.com"</span><span class="p">,</span> <span class="n">api_key</span><span class="o">=</span><span class="s2">"YOUR_API_KEY"</span><span class="p">)</span>
<span class="n">pipeline</span> <span class="o">=</span> <span class="n">PipelineBuilder</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">"example"</span><span class="p">,</span> <span class="n">sql</span><span class="o">=</span><span class="n">sql</span><span class="p">)</span><span class="o">.</span><span class="n">create</span><span class="p">()</span>
<span class="c1"># start the pipeline</span>
<span class="n">pipeline</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="c1"># read input data</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_csv</span><span class="p">(</span><span class="s2">"data.csv"</span><span class="p">)</span>
<span class="n">pipeline</span><span class="o">.</span><span class="n">input_pandas</span><span class="p">(</span><span class="n">tbl_name</span><span class="p">,</span> <span class="n">df</span><span class="p">)</span>
<span class="c1"># wait for the pipeline to complete</span>
<span class="n">pipeline</span><span class="o">.</span><span class="n">wait_for_completion</span><span class="p">(</span><span class="n">force_stop</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</pre></div>
</div>
<ul class="simple">
<li><p>Write the SQL query that creates a table and a view.
You can also create other views on top of existing views.</p></li>
<li><p>Create a <a class="reference internal" href="feldera.html#feldera.pipeline_builder.PipelineBuilder" title="feldera.pipeline_builder.PipelineBuilder"><code class="xref py py-class docutils literal notranslate"><span class="pre">PipelineBuilder</span></code></a> and build the pipeline.</p></li>
<li><p>Call <a class="reference internal" href="feldera.html#feldera.pipeline.Pipeline.start" title="feldera.pipeline.Pipeline.start"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Pipeline.start()</span></code></a> to start the pipeline.</p></li>
<li><p>Pass a pandas DataFrame as input to the table.</p></li>
<li><p>Finally, wait for the the pipeline to complete.</p></li>
</ul>
</div></blockquote>
</li>
</ul>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>If the data source is streaming, this will block forever.
In such cases, use <a class="reference internal" href="feldera.html#feldera.pipeline.Pipeline.wait_for_idle" title="feldera.pipeline.Pipeline.wait_for_idle"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Pipeline.wait_for_idle()</span></code></a> instead.</p>
</div>
</dd>
</dl>
</li>
</ul>
<p>Checkout the <a class="reference internal" href="examples.html"><span class="doc">Examples</span></a>.</p>
</section>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="index.html" class="btn btn-neutral float-left" title="Feldera Python SDK Documentation" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="pandas.html" class="btn btn-neutral float-right" title="Pandas Compatibility" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 2024, Feldera.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>