Conversation
Codecov Report
@@ Coverage Diff @@
## main #312 +/- ##
==========================================
- Coverage 86.29% 84.28% -2.02%
==========================================
Files 134 134
Lines 6312 6306 -6
==========================================
- Hits 5447 5315 -132
- Misses 865 991 +126
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
| new_da = DocumentArray(self.map(*args, **kwargs)) | ||
| self.clear() | ||
| self.extend(new_da) | ||
| for doc in self.map(*args, **kwargs): |
There was a problem hiding this comment.
use batch_map would be efficient for some backends?
There was a problem hiding this comment.
hmm, interesting thought, that could be true. especially when the afterward da[d.id] = d is IO bounded https://docarray.jina.ai/fundamentals/documentarray/parallelization/#use-map-batch-to-overlap-cpu-and-network-time
@alaeddine-13 you think?
There was a problem hiding this comment.
then it's better to use it for apply_batch not apply.
I have applied the same logic for apply_batch, can you check ?
alaeddine-13
force-pushed
the
perf-apply-optimize-memory
branch
from
d254861 to
e3b4849
Compare
3 participants
Goals:
applymemory efficient and suitable for storage backendscloses: Modify apply in order to use less memory #311