Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
256 commits
Select commit Hold shift + click to select a range
cd2963d
added basic pipelines.py file
daniel-sanche Jan 23, 2025
3cbc2d5
added yaml test file
daniel-sanche Jan 24, 2025
0c2241d
wrote basic parser for pipeline yaml
daniel-sanche Jan 24, 2025
9c8982c
encoded extra java system tests
daniel-sanche Jan 24, 2025
863bd1d
reconstruct pipeline expr objects
daniel-sanche Jan 24, 2025
7cd2c11
got yaml to run
daniel-sanche Jan 25, 2025
c328934
add data loading code
daniel-sanche Jan 25, 2025
de0a862
updated protos
daniel-sanche Feb 27, 2025
45c83aa
added pyyaml to system test dependencies
daniel-sanche Feb 27, 2025
fc57f04
added Expr methods
daniel-sanche Jan 28, 2025
0224f94
trying to improve accumulator api
daniel-sanche Jan 28, 2025
cdee937
improved aggregate/accumulators
daniel-sanche Jan 28, 2025
5d35c54
fixed naming in yaml
daniel-sanche Jan 28, 2025
11373e3
create objects in parsing code
daniel-sanche Jan 28, 2025
d2b4153
use order enum
daniel-sanche Jan 28, 2025
bfdfba3
standardize how I deal with map stages
daniel-sanche Jan 28, 2025
afa823a
fix broke super.__init__ calls
daniel-sanche Jan 28, 2025
3348127
added repr for custom classes
daniel-sanche Jan 28, 2025
e3c995d
added repr to pipeline
daniel-sanche Jan 28, 2025
2d604e1
fixed vector formatting
daniel-sanche Jan 28, 2025
3bed807
only treat capitalized strings as possible exprs
daniel-sanche Jan 28, 2025
9b9eaa8
Where uses positional args in yaml
daniel-sanche Jan 28, 2025
e35accf
support union stage
daniel-sanche Jan 28, 2025
72301a1
fixed testReplace
daniel-sanche Jan 28, 2025
07d25dc
fixed sample
daniel-sanche Jan 28, 2025
38fb7fa
iterating on sample options
daniel-sanche Jan 28, 2025
0c3852f
fixed import
daniel-sanche Jan 28, 2025
1d95e54
removing kwargs
daniel-sanche Jan 29, 2025
a3daa07
removing keargs
daniel-sanche Jan 29, 2025
8e45b68
use only positional arguments in yaml
daniel-sanche Jan 29, 2025
6b2a605
added _to_pb to stages
daniel-sanche Feb 27, 2025
6d2aa90
added _pb to pipeline expressions
daniel-sanche Feb 27, 2025
e51d9dc
added unimplemented stage._to_pbs
daniel-sanche Feb 27, 2025
8f6dea5
fixed proto formatting
daniel-sanche Feb 27, 2025
654e5f7
got protos to build with no errors
daniel-sanche Feb 28, 2025
69d4b76
added headers
daniel-sanche Feb 28, 2025
6d966a8
fixed some types
daniel-sanche Feb 28, 2025
33fee52
fixed expression typing
daniel-sanche Feb 28, 2025
cb8539d
added abstract to expr
daniel-sanche Feb 28, 2025
380dce9
fixed types in pipeline_stages
daniel-sanche Feb 28, 2025
831886f
align typing
daniel-sanche Mar 1, 2025
79df205
added stubs for execute_pipeline
daniel-sanche Mar 1, 2025
e760c44
fixed union stage proto
daniel-sanche Mar 25, 2025
ea1e2ba
propagate client in test parsers
daniel-sanche Mar 25, 2025
b6b082d
enable mypy for pipeline code
daniel-sanche Mar 25, 2025
658964c
added query.pipeline
daniel-sanche Mar 25, 2025
7e12b74
got queries to execute
daniel-sanche Mar 25, 2025
e463418
fixed some encodings
daniel-sanche Mar 25, 2025
4f527c1
broke pipelines into separate async/sync/base files
daniel-sanche Mar 27, 2025
a323e5b
added docstrings to expressions
daniel-sanche Mar 27, 2025
7e70022
improved ordering
daniel-sanche Mar 27, 2025
98887a0
added docstrings to pipeline classes
daniel-sanche Mar 27, 2025
549c590
added docstrings
daniel-sanche Mar 27, 2025
376015c
fixed sample_options
daniel-sanche Mar 27, 2025
e3de1b8
catch expected errors in test
daniel-sanche Mar 27, 2025
f725bc7
added alias to unnest
daniel-sanche Mar 28, 2025
128ab1c
fixed replace
daniel-sanche Mar 28, 2025
64a8bda
added pipeline to client
daniel-sanche Mar 28, 2025
ec06080
improved query.pipeline logic
daniel-sanche Mar 28, 2025
0703532
removed imports
daniel-sanche Mar 28, 2025
246d1c8
pass stages through
daniel-sanche Mar 28, 2025
b72d44a
fixed collection docstrings
daniel-sanche Mar 28, 2025
810ccd2
fixed collection setup
daniel-sanche Mar 28, 2025
78b5833
fixed some collection errors
daniel-sanche Mar 29, 2025
36e0228
implemented FitlerCondition._from_pb
daniel-sanche Mar 29, 2025
43f4cf4
renamed function
daniel-sanche Mar 29, 2025
51e9b25
added tests for query.pipeline
daniel-sanche Mar 29, 2025
35cb0bb
added verify pipeline to system tests
daniel-sanche Mar 31, 2025
6283d1a
fixed bug in filter conversion
daniel-sanche Mar 31, 2025
080bf42
return pipeline copy
daniel-sanche Mar 31, 2025
6cd5c63
updated results format
daniel-sanche Mar 31, 2025
7143247
added proto assertions to e2e tests
daniel-sanche Mar 31, 2025
05c4f23
added tests for FieldFilter._from_filter_pb
daniel-sanche Apr 1, 2025
7512a1a
fixed repr
daniel-sanche Apr 1, 2025
901c0e1
fixed typing issues
daniel-sanche Apr 1, 2025
6a5f72e
fixed client fixture
daniel-sanche Apr 1, 2025
072669f
fixed mapget in e2e yaml
daniel-sanche Apr 1, 2025
05a7498
fixed docstring
daniel-sanche Apr 1, 2025
9f8d4c8
fixed append bug
daniel-sanche Apr 1, 2025
863a09d
compare result data in tests
daniel-sanche Apr 1, 2025
e9f11c6
fixed incorrect test case ordering
daniel-sanche Apr 1, 2025
d2c33f2
yield document snapshots
daniel-sanche Apr 2, 2025
db1ecaa
added async test
daniel-sanche Apr 2, 2025
b6d74da
cleaning up e2e yaml
daniel-sanche Apr 2, 2025
70efea7
fixed scope in tests
daniel-sanche Apr 3, 2025
b36afa8
added _client to base pipeline
daniel-sanche Apr 3, 2025
7a26f7e
fixing test yaml
daniel-sanche Apr 3, 2025
fee6c90
renamed exceute_async to execute
daniel-sanche Apr 3, 2025
da18289
broke up pipeline tests into separate functions
daniel-sanche Apr 3, 2025
fc80062
improved faulty test yaml
daniel-sanche Apr 30, 2025
c049e21
remvoved unready stages and expressions
daniel-sanche May 2, 2025
41b91d4
fixed regex_match test
daniel-sanche May 2, 2025
7f69229
ran blacken
daniel-sanche May 2, 2025
0b4c294
turn Stage into an ABC
daniel-sanche May 2, 2025
255b698
removed extra stages and expressions
daniel-sanche May 2, 2025
bd9c2c4
stripped down to stubs
daniel-sanche May 2, 2025
4314141
Revert "stripped down to stubs"
daniel-sanche May 3, 2025
c38a925
Revert "removed extra stages and expressions"
daniel-sanche May 3, 2025
9129306
Revert "remvoved unready stages and expressions"
daniel-sanche May 3, 2025
d8dc10f
added PipelineStages
daniel-sanche May 6, 2025
1f2390a
removed collection.pipeline
daniel-sanche May 6, 2025
af371d7
fixed collection order
daniel-sanche May 6, 2025
a0f8e5d
Merge branch 'pipeline_queries_2_query_parity' into pipeline_queries_…
daniel-sanche May 6, 2025
ba39b81
fixed collection order in docstrings
daniel-sanche May 6, 2025
c0ebc00
Merge branch 'pipeline_queries_3_stable_stages' into pipeline_queries…
daniel-sanche May 6, 2025
ca38160
fixed docstrings
daniel-sanche May 6, 2025
39f261a
fixed docstrings
daniel-sanche May 6, 2025
48bf9f7
added pipeline_result
daniel-sanche May 6, 2025
b57424d
chore: updated gapic layer for execute_query
daniel-sanche May 8, 2025
8fde414
updated gapics
daniel-sanche May 8, 2025
9cac154
Merge branch 'pipeline_queries_approved' into pipeline_queries_1_stubs
daniel-sanche May 8, 2025
93003c0
removed unneeded code
daniel-sanche May 8, 2025
9a9cc3d
Merge branch 'pipeline_queries_1_stubs' into pipeline_queries_2_query…
daniel-sanche May 8, 2025
2750bd5
ran black
daniel-sanche May 8, 2025
d4dcac4
use pipeline source in query
daniel-sanche May 8, 2025
e8be99f
Merge branch 'pipeline_queries_2_query_parity' into pipeline_queries_…
daniel-sanche May 8, 2025
77dca01
added extra pipeline sources
daniel-sanche May 8, 2025
0051d57
Merge branch 'pipeline_queries_3_stable_stages' into pipeline_queries…
daniel-sanche May 8, 2025
79d016a
fixed lint
daniel-sanche May 8, 2025
907a551
added client tests
daniel-sanche May 8, 2025
8e20c11
added pipeline tests
daniel-sanche May 9, 2025
98c7ea5
added tests for execute
daniel-sanche May 9, 2025
af4fb20
broke out shared logic into base_pipeline
daniel-sanche May 9, 2025
cd38fc2
added pipeline stages tests
daniel-sanche May 12, 2025
0ac319e
removed unneeded stages
daniel-sanche May 12, 2025
7981a34
added tests for pipeline expressions
daniel-sanche May 12, 2025
62b6510
added pipeline_result tests
daniel-sanche May 12, 2025
035c6e6
added tests for pipeline source
daniel-sanche May 12, 2025
5dd1246
added transaction to execute call
daniel-sanche May 12, 2025
6434023
ran blacken
daniel-sanche May 12, 2025
a8beea4
fixed lint
daniel-sanche May 12, 2025
2d286bb
fixed mypy
daniel-sanche May 12, 2025
a1e8e77
Merge branch 'pipeline_queries_1_stubs' into pipeline_queries_2_query…
daniel-sanche May 12, 2025
8c00357
fixed lint
daniel-sanche May 13, 2025
09d45cb
removed e2e tests from this PR
daniel-sanche May 13, 2025
9cdb8c9
Merge branch 'pipeline_queries_2_query_parity' into pipeline_queries_…
daniel-sanche May 13, 2025
9a783d7
fixed lint
daniel-sanche May 13, 2025
6d29008
Merge branch 'pipeline_queries_3_stable_stages' into pipeline_queries…
daniel-sanche May 13, 2025
60a770c
fixed lint
daniel-sanche May 13, 2025
d2babd2
Merge branch 'pipeline_queries_approved' into pipeline_queries_1_stubs
daniel-sanche May 13, 2025
b46bdc1
fixed test issues
daniel-sanche May 13, 2025
28fd42d
added tests
daniel-sanche May 13, 2025
a9368b3
added stages unit tests
daniel-sanche May 13, 2025
2775da2
improve FilterCondition repr
daniel-sanche May 14, 2025
c4cd995
added tests for filter conditions
daniel-sanche May 14, 2025
63b83b8
added tests for expressions
daniel-sanche May 15, 2025
f22f11e
support tuples in pipelin_source.collection
daniel-sanche May 15, 2025
c8cfcee
added tests
daniel-sanche May 15, 2025
5beee36
ran blacken
daniel-sanche May 15, 2025
dc5b5ac
Merge branch 'pipeline_queries_1_stubs' into pipeline_queries_2_query…
daniel-sanche May 15, 2025
22b558c
Merge branch 'pipeline_queries_approved' into pipeline_queries_1_stubs
daniel-sanche Jun 9, 2025
0ff25c1
Merge branch 'pipeline_queries_1_stubs' into pipeline_queries_2_query…
daniel-sanche Jun 9, 2025
5a8eda3
Squashed commit of the following:
daniel-sanche Jun 9, 2025
3a16610
Merge branch 'pipeline_queries_2_query_parity' into pipeline_queries_…
daniel-sanche Jun 9, 2025
be6dbf7
added unit tests for new sources
daniel-sanche Jun 9, 2025
628f66e
added tests for stages
daniel-sanche Jun 9, 2025
4594a6c
ran black
daniel-sanche Jun 9, 2025
74a5d22
ran black
daniel-sanche Jun 9, 2025
e201b6f
moved SampleOptions into stages file
daniel-sanche Jun 9, 2025
386585b
updated infix test
daniel-sanche Jun 10, 2025
864bb66
added tests for ExprWithAlias
daniel-sanche Jun 10, 2025
40612ac
restructured expression tests
daniel-sanche Jun 10, 2025
aabcf33
added tests for expressions
daniel-sanche Jun 10, 2025
bf7adad
improved verify_pipeline check
daniel-sanche Jun 10, 2025
b2546f7
use TEST_DATABASES list
daniel-sanche Jun 10, 2025
38f00a7
ran black
daniel-sanche Jun 10, 2025
82d5da1
disallow mixed positional and kwargs in Aggregate
daniel-sanche Jun 11, 2025
244a9f2
improving aggregate
daniel-sanche Jun 12, 2025
c62e448
add pipeline() method to collections
daniel-sanche Jun 12, 2025
bf0dca2
fixed repr test
daniel-sanche Jun 12, 2025
3cd826b
fixed lint
daniel-sanche Jun 12, 2025
c8f9ca8
Squashed commit of the following:
daniel-sanche Jun 12, 2025
5cd1670
support str in add_field
daniel-sanche Jun 12, 2025
f019c16
updated test_pipelines
daniel-sanche Jun 12, 2025
ae00f1d
added equality checks
daniel-sanche Jun 12, 2025
36fd97b
fixed lint
daniel-sanche Jun 12, 2025
32eea9a
fixed lint
daniel-sanche Jun 12, 2025
5e771ed
Merge branch 'pipeline_queries_2_query_parity' into pipeline_queries_…
daniel-sanche Jun 12, 2025
b406003
removed str option from add_fields
daniel-sanche Jun 12, 2025
44c7533
Merge branch 'pipeline_queries_3_stable_stages' into pipeline_queries…
daniel-sanche Jun 12, 2025
699fc05
added unit tests
daniel-sanche Jun 12, 2025
a1c3537
fixed lint
daniel-sanche Jun 12, 2025
e6ad9d2
broke out TestFunctions from TestFilterConditions
daniel-sanche Jun 13, 2025
dc3564c
Merge branch 'pipeline_queries_3_stable_stages' into pipeline_queries…
daniel-sanche Jun 13, 2025
3de11fa
Squashed commit of the following:
daniel-sanche Jun 23, 2025
50f7d7b
Merge branch 'pipeline_queries_approved' into pipeline_queries_3_stab…
daniel-sanche Jun 23, 2025
3f675b7
fixed broken tests
daniel-sanche Jun 23, 2025
b81c637
fixed coverage
daniel-sanche Jun 24, 2025
1f0595a
fixed system test
daniel-sanche Jun 24, 2025
4552918
removed query -> pipeline conversions
daniel-sanche Jun 24, 2025
d512467
added back query -> pipeline conversion
daniel-sanche Jun 24, 2025
1ff2053
simplified event loop
daniel-sanche Jun 24, 2025
f732f01
cleaned up test file
daniel-sanche Jun 24, 2025
a4feed5
fix flaky test
daniel-sanche Jun 25, 2025
9f8b1c2
Merge branch 'pipeline_queries_3_5_query_conversion' into pipeline_qu…
daniel-sanche Jun 25, 2025
235dce7
run tests against enterprise db
daniel-sanche Jun 26, 2025
7ef0313
added _to_value helper
daniel-sanche Jul 1, 2025
2b47bfd
changed comment
daniel-sanche Jul 1, 2025
7064c62
sequence instead of list
daniel-sanche Jul 1, 2025
510ee03
fixed unneeded lines
daniel-sanche Jul 1, 2025
8344b15
ficed bug in StrConcat
daniel-sanche Jul 11, 2025
a90b1c3
added static methods to Function class
daniel-sanche Jul 11, 2025
6809e7c
fixed lint
daniel-sanche Jul 11, 2025
f813e69
Merge branch 'pipeline_queries_3_stable_stages' into pipeline_queries…
daniel-sanche Jul 11, 2025
046b803
Merge branch 'pipeline_queries_3_5_query_conversion' into pipeline_qu…
daniel-sanche Jul 11, 2025
061472e
fixed mypy
daniel-sanche Jul 11, 2025
057db44
Merge branch 'pipeline_queries_3_stable_stages' into pipeline_queries…
daniel-sanche Jul 11, 2025
8404765
Merge branch 'pipeline_queries_3_5_query_conversion' into pipeline_qu…
daniel-sanche Jul 11, 2025
407aa85
renamed classes
daniel-sanche Jul 12, 2025
8ed4d1b
Merge branch 'pipeline_queries_approved' into pipeline_queries_3_5_qu…
daniel-sanche Jul 16, 2025
8233f47
renamed expressions
daniel-sanche Oct 16, 2025
60924fe
added new math expressions
daniel-sanche Oct 16, 2025
e9dc12e
Merge branch 'pipeline_queries_approved' into pipeline_queries_3_5_qu…
daniel-sanche Oct 17, 2025
39af626
Merge branch 'pipeline_queries_3_5_query_conversion' into pipeline_qu…
daniel-sanche Oct 17, 2025
bbc8915
Merge branch 'pipeline_queries_4_all' into pipeline_queries_5_cleanup
daniel-sanche Oct 17, 2025
3217b00
fixed tests
daniel-sanche Oct 17, 2025
e1a2f15
fixed lint
daniel-sanche Oct 17, 2025
4de3908
added test for AliasedAggregate
daniel-sanche Oct 17, 2025
5fd0ba4
removed unused expressions
daniel-sanche Oct 17, 2025
5a3eb14
improved array functions
daniel-sanche Oct 17, 2025
3a35035
removed duplicate code
daniel-sanche Oct 18, 2025
32c4e4b
added map expressions
daniel-sanche Oct 18, 2025
d5e854c
renamed if to conditional
daniel-sanche Oct 18, 2025
781fa18
add root collection to pipeline tests
daniel-sanche Oct 20, 2025
50a2ed0
renamed expressions as needed
daniel-sanche Oct 20, 2025
16bd0d6
added conditional
daniel-sanche Oct 20, 2025
de273f4
removed custom classes
daniel-sanche Oct 20, 2025
fb4c2dc
fixed tests
daniel-sanche Oct 21, 2025
c2cc317
ran blacken
daniel-sanche Oct 21, 2025
fc798bd
made ListOfExprs internal
daniel-sanche Oct 21, 2025
938ab7d
cleaned up tests
daniel-sanche Oct 21, 2025
902e07a
added better docstrings
daniel-sanche Oct 21, 2025
ad2a37c
removed unneeded conditional static method
daniel-sanche Oct 21, 2025
bf67e0f
simplified code by removing extra classes
daniel-sanche Oct 21, 2025
8204337
fixed lint
daniel-sanche Oct 21, 2025
37510dc
added tests
daniel-sanche Oct 21, 2025
3193906
added tests
daniel-sanche Oct 21, 2025
b33fb25
fixed lint
daniel-sanche Oct 21, 2025
64fe3b0
Merge branch 'pipeline_queries_fixed' into pipeline_queries_3_5_query…
daniel-sanche Oct 21, 2025
581bac6
fixed references
daniel-sanche Oct 21, 2025
15af2a0
added empty count
daniel-sanche Oct 21, 2025
4260244
fixed paths
daniel-sanche Oct 21, 2025
c2c339a
updated tests
daniel-sanche Oct 21, 2025
899b6e0
added is_null; fixed stage
daniel-sanche Oct 21, 2025
1dfb191
added async verify pipeline tests
daniel-sanche Oct 21, 2025
2195bba
updated names, etc from base branch
daniel-sanche Oct 21, 2025
94d2bc6
added async verify_pipeline tests
daniel-sanche Oct 21, 2025
e63965b
Merge branch 'pipeline_queries_approved' into pipeline_queries_3_5_qu…
daniel-sanche Oct 22, 2025
06a6147
fixed docstrings
daniel-sanche Oct 23, 2025
0ea73f9
Update google/cloud/firestore_v1/base_aggregation.py
daniel-sanche Oct 23, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion google/cloud/firestore_v1/_pipeline_stages.py
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ def __init__(self, collection_id: str):
self.collection_id = collection_id

def _pb_args(self):
return [Value(string_value=self.collection_id)]
return [Value(reference_value=""), Value(string_value=self.collection_id)]


class Database(Stage):
Expand Down
57 changes: 56 additions & 1 deletion google/cloud/firestore_v1/base_aggregation.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,10 @@
from __future__ import annotations

import abc
import itertools

from abc import ABC
from typing import TYPE_CHECKING, Any, Coroutine, List, Optional, Tuple, Union
from typing import TYPE_CHECKING, Any, Coroutine, List, Optional, Tuple, Union, Iterable

from google.api_core import gapic_v1
from google.api_core import retry as retries
Expand All @@ -33,6 +34,10 @@
from google.cloud.firestore_v1.types import (
StructuredAggregationQuery,
)
from google.cloud.firestore_v1.pipeline_expressions import AggregateFunction
from google.cloud.firestore_v1.pipeline_expressions import Count
from google.cloud.firestore_v1.pipeline_expressions import AliasedExpr
from google.cloud.firestore_v1.pipeline_expressions import Field

# Types needed only for Type Hints
if TYPE_CHECKING: # pragma: NO COVER
Expand Down Expand Up @@ -66,6 +71,9 @@ def __init__(self, alias: str, value: float, read_time=None):
def __repr__(self):
return f"<Aggregation alias={self.alias}, value={self.value}, readtime={self.read_time}>"

def _to_dict(self):
return {self.alias: self.value}


class BaseAggregation(ABC):
def __init__(self, alias: str | None = None):
Expand All @@ -75,6 +83,27 @@ def __init__(self, alias: str | None = None):
def _to_protobuf(self):
"""Convert this instance to the protobuf representation"""

@abc.abstractmethod
def _to_pipeline_expr(
self, autoindexer: Iterable[int]
) -> AliasedExpr[AggregateFunction]:
"""
Convert this instance to a pipeline expression for use with pipeline.aggregate()

Args:
autoindexer: If an alias isn't supplied, one should be created with the format "field_n"
The autoindexer is an iterable that provides the `n` value to use for each expression
"""

def _pipeline_alias(self, autoindexer):
"""
Helper to build the alias for the pipeline expression
"""
if self.alias is not None:
return self.alias
else:
return f"field_{next(autoindexer)}"


class CountAggregation(BaseAggregation):
def __init__(self, alias: str | None = None):
Expand All @@ -88,6 +117,9 @@ def _to_protobuf(self):
aggregation_pb.count = StructuredAggregationQuery.Aggregation.Count()
return aggregation_pb

def _to_pipeline_expr(self, autoindexer: Iterable[int]):
return Count().as_(self._pipeline_alias(autoindexer))


class SumAggregation(BaseAggregation):
def __init__(self, field_ref: str | FieldPath, alias: str | None = None):
Expand All @@ -107,6 +139,9 @@ def _to_protobuf(self):
aggregation_pb.sum.field.field_path = self.field_ref
return aggregation_pb

def _to_pipeline_expr(self, autoindexer: Iterable[int]):
return Field.of(self.field_ref).sum().as_(self._pipeline_alias(autoindexer))


class AvgAggregation(BaseAggregation):
def __init__(self, field_ref: str | FieldPath, alias: str | None = None):
Expand All @@ -126,6 +161,9 @@ def _to_protobuf(self):
aggregation_pb.avg.field.field_path = self.field_ref
return aggregation_pb

def _to_pipeline_expr(self, autoindexer: Iterable[int]):
return Field.of(self.field_ref).average().as_(self._pipeline_alias(autoindexer))


def _query_response_to_result(
response_pb,
Expand Down Expand Up @@ -317,3 +355,20 @@ def stream(
StreamGenerator[List[AggregationResult]] | AsyncStreamGenerator[List[AggregationResult]]:
A generator of the query results.
"""

def pipeline(self):
"""
Convert this query into a Pipeline

Queries containing a `cursor` or `limit_to_last` are not currently supported

Raises:
- ValueError: raised if Query wasn't created with an associated client
- NotImplementedError: raised if the query contains a `cursor` or `limit_to_last`
Returns:
a Pipeline representing the query
"""
# use autoindexer to keep track of which field number to use for un-aliased fields
autoindexer = itertools.count(start=1)
exprs = [a._to_pipeline_expr(autoindexer) for a in self._aggregations]
return self._nested_query.pipeline().aggregate(*exprs)
13 changes: 13 additions & 0 deletions google/cloud/firestore_v1/base_collection.py
Original file line number Diff line number Diff line change
Expand Up @@ -602,6 +602,19 @@ def find_nearest(
distance_threshold=distance_threshold,
)

def pipeline(self):
"""
Convert this query into a Pipeline

Queries containing a `cursor` or `limit_to_last` are not currently supported

Raises:
- NotImplementedError: raised if the query contains a `cursor` or `limit_to_last`
Returns:
a Pipeline representing the query
"""
return self._query().pipeline()


def _auto_id() -> str:
"""Generate a "random" automatically generated ID.
Expand Down
69 changes: 69 additions & 0 deletions google/cloud/firestore_v1/base_query.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@
query,
)
from google.cloud.firestore_v1.vector import Vector
from google.cloud.firestore_v1 import pipeline_expressions

if TYPE_CHECKING: # pragma: NO COVER
from google.cloud.firestore_v1.async_stream_generator import AsyncStreamGenerator
Expand Down Expand Up @@ -1128,6 +1129,74 @@ def recursive(self: QueryType) -> QueryType:

return copied

def pipeline(self):
"""
Convert this query into a Pipeline

Queries containing a `cursor` or `limit_to_last` are not currently supported

Raises:
- ValueError: raised if Query wasn't created with an associated client
- NotImplementedError: raised if the query contains a `cursor` or `limit_to_last`
Returns:
a Pipeline representing the query
"""
if not self._client:
raise ValueError("Query does not have an associated client")
if self._all_descendants:
ppl = self._client.pipeline().collection_group(self._parent.id)
else:
ppl = self._client.pipeline().collection(self._parent._path)

# Filters
for filter_ in self._field_filters:
ppl = ppl.where(
pipeline_expressions.BooleanExpr._from_query_filter_pb(
filter_, self._client
)
)

# Projections
if self._projection and self._projection.fields:
ppl = ppl.select(*[field.field_path for field in self._projection.fields])

# Orders
orders = self._normalize_orders()
if orders:
exists = []
orderings = []
for order in orders:
field = pipeline_expressions.Field.of(order.field.field_path)
exists.append(field.exists())
direction = (
"ascending"
if order.direction == StructuredQuery.Direction.ASCENDING
else "descending"
)
orderings.append(pipeline_expressions.Ordering(field, direction))

# Add exists filters to match Query's implicit orderby semantics.
if len(exists) == 1:
ppl = ppl.where(exists[0])
else:
ppl = ppl.where(pipeline_expressions.And(*exists))

# Add sort orderings
ppl = ppl.sort(*orderings)

# Cursors, Limit and Offset
if self._start_at or self._end_at or self._limit_to_last:
raise NotImplementedError(
"Query to Pipeline conversion: cursors and limit_to_last is not supported yet."
)
else: # Limit & Offset without cursors
if self._offset:
ppl = ppl.offset(self._offset)
if self._limit:
ppl = ppl.limit(self._limit)

return ppl

def _comparator(self, doc1, doc2) -> int:
_orders = self._orders

Expand Down
48 changes: 41 additions & 7 deletions google/cloud/firestore_v1/pipeline_expressions.py
Original file line number Diff line number Diff line change
Expand Up @@ -587,6 +587,18 @@ def is_nan(self) -> "BooleanExpr":
"""
return BooleanExpr("is_nan", [self])

@expose_as_static
def is_null(self) -> "BooleanExpr":
"""Creates an expression that checks if this expression evaluates to 'Null'.

Example:
>>> Field.of("value").is_null()

Returns:
A new `Expr` representing the 'isNull' check.
"""
return BooleanExpr("is_null", [self])

@expose_as_static
def exists(self) -> "BooleanExpr":
"""Creates an expression that checks if a field exists in the document.
Expand Down Expand Up @@ -627,6 +639,7 @@ def average(self) -> "Expr":
"""
return AggregateFunction("average", [self])

@expose_as_static
def count(self) -> "Expr":
"""Creates an aggregation that counts the number of stage inputs with valid evaluations of the
expression or field.
Expand Down Expand Up @@ -1312,9 +1325,9 @@ def _from_query_filter_pb(filter_pb, client):
elif filter_pb.op == Query_pb.UnaryFilter.Operator.IS_NOT_NAN:
return And(field.exists(), Not(field.is_nan()))
elif filter_pb.op == Query_pb.UnaryFilter.Operator.IS_NULL:
return And(field.exists(), field.equal(None))
return And(field.exists(), field.is_null())
elif filter_pb.op == Query_pb.UnaryFilter.Operator.IS_NOT_NULL:
return And(field.exists(), Not(field.equal(None)))
return And(field.exists(), Not(field.is_null()))
else:
raise TypeError(f"Unexpected UnaryFilter operator type: {filter_pb.op}")
elif isinstance(filter_pb, Query_pb.FieldFilter):
Expand Down Expand Up @@ -1361,7 +1374,7 @@ class And(BooleanExpr):
Example:
>>> # Check if the 'age' field is greater than 18 AND the 'city' field is "London" AND
>>> # the 'status' field is "active"
>>> Expr.And(Field.of("age").greater_than(18), Field.of("city").equal("London"), Field.of("status").equal("active"))
>>> And(Field.of("age").greater_than(18), Field.of("city").equal("London"), Field.of("status").equal("active"))

Args:
*conditions: The filter conditions to 'AND' together.
Expand All @@ -1377,7 +1390,7 @@ class Not(BooleanExpr):

Example:
>>> # Find documents where the 'completed' field is NOT true
>>> Expr.Not(Field.of("completed").equal(True))
>>> Not(Field.of("completed").equal(True))

Args:
condition: The filter condition to negate.
Expand All @@ -1394,7 +1407,7 @@ class Or(BooleanExpr):
Example:
>>> # Check if the 'age' field is greater than 18 OR the 'city' field is "London" OR
>>> # the 'status' field is "active"
>>> Expr.Or(Field.of("age").greater_than(18), Field.of("city").equal("London"), Field.of("status").equal("active"))
>>> Or(Field.of("age").greater_than(18), Field.of("city").equal("London"), Field.of("status").equal("active"))

Args:
*conditions: The filter conditions to 'OR' together.
Expand All @@ -1411,7 +1424,7 @@ class Xor(BooleanExpr):
Example:
>>> # Check if only one of the conditions is true: 'age' greater than 18, 'city' is "London",
>>> # or 'status' is "active".
>>> Expr.Xor(Field.of("age").greater_than(18), Field.of("city").equal("London"), Field.of("status").equal("active"))
>>> Xor(Field.of("age").greater_than(18), Field.of("city").equal("London"), Field.of("status").equal("active"))

Args:
*conditions: The filter conditions to 'XOR' together.
Expand All @@ -1428,7 +1441,7 @@ class Conditional(BooleanExpr):

Example:
>>> # If 'age' is greater than 18, return "Adult"; otherwise, return "Minor".
>>> Expr.conditional(Field.of("age").greater_than(18), Constant.of("Adult"), Constant.of("Minor"));
>>> Conditional(Field.of("age").greater_than(18), Constant.of("Adult"), Constant.of("Minor"));

Args:
condition: The condition to evaluate.
Expand All @@ -1440,3 +1453,24 @@ def __init__(self, condition: BooleanExpr, then_expr: Expr, else_expr: Expr):
super().__init__(
"conditional", [condition, then_expr, else_expr], use_infix_repr=False
)

class Count(AggregateFunction):
"""
Represents an aggregation that counts the number of stage inputs with valid evaluations of the
expression or field.

Example:
>>> # Count the total number of products
>>> Field.of("productId").count().as_("totalProducts")
>>> Count(Field.of("productId"))
>>> Count().as_("count")

Args:
expression: The expression or field to count. If None, counts all stage inputs.
"""

def __init__(self, expression: Expr | None = None):
expression_list = [expression] if expression else []
super().__init__(
"count", expression_list, use_infix_repr=bool(expression_list)
)
1 change: 1 addition & 0 deletions tests/system/test__helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,4 @@
# run all tests against default database, and a named database
# TODO: add enterprise mode when GA (RunQuery not currently supported)
TEST_DATABASES = [None, FIRESTORE_OTHER_DB]
TEST_DATABASES_W_ENTERPRISE = TEST_DATABASES + [FIRESTORE_ENTERPRISE_DB]
Loading