docs.feldera.com/python/_modules/feldera/output_handler.html at main · feldera/docs.feldera.com

History

203 lines (164 loc) · 14 KB

Raw

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

<!DOCTYPE html>

<head>

<title>feldera.output_handler — Feldera Python SDK 0.275.0 documentation</title>

<!--[if lt IE 9]>

<![endif]-->

</head>

Feldera Python SDK

</a>

</form>

</div>

</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">

Contents:

<ul>

<li class="toctree-l1"><a class="reference internal" href="../../introduction.html">Introduction</a></li>

<li class="toctree-l1"><a class="reference internal" href="../../pandas.html">Pandas Compatibility</a></li>

<li class="toctree-l1"><a class="reference internal" href="../../examples.html">Examples</a></li>

</ul>

API Reference:

<ul>

<li class="toctree-l1"><a class="reference internal" href="../../modules.html">feldera</a></li>

</ul>

</div>

</nav>

<a href="../../index.html">Feldera Python SDK</a>

</nav>

<li class="breadcrumb-item"><a href="../index.html">Module code</a></li>

<li class="breadcrumb-item active">feldera.output_handler</li>

</li>

</ul>

<hr/>

</div>

<h1>Source code for feldera.output_handler</h1><div class="highlight"><pre>

import pandas as pd

from typing import Optional

from threading import Event

from feldera import FelderaClient

from feldera._callback_runner import CallbackRunner

class OutputHandler:

def __init__(

self,

client: FelderaClient,

pipeline_name: str,

view_name: str,

):

"""

Initializes the output handler, but doesn't start it.

To start the output handler, call the `.OutputHandler.start` method.

"""

self.client: FelderaClient = client

self.pipeline_name: str = pipeline_name

self.view_name: str = view_name

self.buffer: list[pd.DataFrame] = []

self.exception: Optional[BaseException] = None

self.event = Event()

# the callback that is passed to the `CallbackRunner`

def callback(df: pd.DataFrame, _: int):

if not df.empty:

self.buffer.append(df)

def exception_callback(exception: BaseException):

self.exception = exception

# sets up the callback runner

self.handler = CallbackRunner(

self.client,

self.pipeline_name,

self.view_name,

callback,

exception_callback,

self.event,

)</div>

def start(self):

"""

Starts the output handler in a separate thread

"""

self.handler.start()

_ = self.event.wait()</div>

def to_pandas(self, clear_buffer: bool = True):

"""

Returns the output of the pipeline as a pandas DataFrame

:param clear_buffer: Whether to clear the buffer after getting the output.

"""

if self.exception is not None:

raise self.exception

if len(self.buffer) == 0:

return pd.DataFrame()

res = pd.concat(self.buffer, ignore_index=True)

if clear_buffer:

self.buffer.clear()

return res</div>

def to_dict(self, clear_buffer: bool = True):

"""

Returns the output of the pipeline as a list of python dictionaries

:param clear_buffer: Whether to clear the buffer after getting the output.

"""

return self.to_pandas(clear_buffer).to_dict(orient="records")</div>

</div>

</pre></div>

</div>

<hr/>

</div>

Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a

<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>

provided by <a href="https://readthedocs.org">Read the Docs</a>.

</footer>

</div>

</section>

</div>

jQuery(function () {

SphinxRtdTheme.Navigation.enable(true);

});

</script>

</body>

</html>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FilesExpand file tree

output_handler.html

Latest commit

History

output_handler.html

File metadata and controls