Repository navigation
Feature Request: SuperCompress context compression for real-time AI pipelines #254
Description
Feature: Compress LLM Context in Pathway Pipelines
Problem: Pathway processes real-time data streams through LLM calls. Each window of streaming data sent to the LLM consumes tokens — compressing it before processing would reduce costs for streaming AI workloads.
Proposed Solution: Provide a SuperCompress integration example for Pathway's LLM processing.
from supercompress import Compressor
import pathway as pw
compressor = Compressor()
# In a Pathway LLM processing pipeline
def compress_and_process(data, query):
compressed = compressor.compress(data, query)
return llm_process(compressed)Metrics: ~65% compression | 100% oracle recall | ~60ms CPU | pip install supercompress
Activity
Shiv-61 commentedon Oct 17, 2026
Hi! I'd be interested in working on this, but before writing any code I'd like to check a few things with the maintainers, in line with CONTRIBUTING.md.
supercompressan optional, lazily imported dependency, with no change to core dependencies. Is that acceptable?CompressorAPI looks like (the snippet in the description looks like pseudocode), and the license, supported Python versions and maintenance status of the package. It would also help to see how the ~65% compression / 100% recall figures were measured and on what data, since I'd want to reproduce them before describing them in any docs.If a maintainer gives the go-ahead and a preferred shape, I'll open a small, self-contained PR with tests and a CHANGELOG entry, plus a short account of how it was built. Happy to drop it if this isn't a direction you'd like to take.
Thanks!