FEAT: Streaming support in fetchone, fetchmany, fetchall for varbinarymax data type.#232
Merged
gargsaumya merged 9 commits intomainfrom Sep 17, 2025
Merged
Conversation
bewithgaurav
requested changes
Sep 11, 2025
Collaborator
bewithgaurav
left a comment
bewithgaurav
left a comment
There was a problem hiding this comment.
avoiding repeated comments from other PRs, just added some minor changes here
I believe this PR will merge at last since other changes will merge here
bewithgaurav
previously approved these changes
Sep 15, 2025
github-actions
bot
added
pr-size: medium
gargsaumya
changed the base branch from
saumya/streaming-varbinarymax
to
main
gargsaumya
dismissed
bewithgaurav’s stale review
The base branch was changed.
github-actions
bot
added
pr-size: medium
sumitmsft
reviewed
Sep 15, 2025
sumitmsft
reviewed
Sep 15, 2025
sumitmsft
reviewed
Sep 15, 2025
sumitmsft
requested changes
Sep 15, 2025
Contributor
sumitmsft
left a comment
sumitmsft
left a comment
There was a problem hiding this comment.
Minor fixes are required.
gargsaumya
force-pushed
the
saumya/streaming-varbinarymax-fetch
branch
from
4e5cfe8 to
2ea9862
Compare
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds streaming support for VARBINARY(MAX) data types in the fetchone, fetchmany, and fetchall operations, significantly improving the driver's ability to handle large binary data. The changes enable proper insertion and retrieval of binary data larger than 8000 bytes, which previously had limitations.
- Implemented chunked streaming retrieval for large binary columns (LOBs) using a new
FetchLobColumnDatahelper function - Enhanced parameter binding to support Data-At-Execution (DAE) for large binary parameters
- Updated fetch operations to detect LOB columns and use appropriate streaming vs buffered approaches
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| mssql_python/pybind/ddbc_bindings.cpp | Core implementation of LOB streaming support, DAE parameter binding, and enhanced fetch operations |
| mssql_python/cursor.py | Updated parameter mapping to use VARBINARY(MAX) with DAE for large binary data |
| tests/test_004_cursor.py | Replaced limited test with comprehensive VARBINARY(MAX) tests covering various data sizes and fetch methods |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
gargsaumya
force-pushed
the
saumya/streaming-varbinarymax-fetch
branch
from
2ea9862 to
85a4487
Compare
github-actions
bot
added
pr-size: medium
…com/microsoft/mssql-python into saumya/streaming-varbinarymax-fetch
gargsaumya
force-pushed
the
saumya/streaming-varbinarymax-fetch
branch
from
7cee625 to
e50de23
Compare
github-actions
bot
added
pr-size: medium
github-actions
bot
added
pr-size: medium
bewithgaurav
approved these changes
Sep 16, 2025
sumitmsft
approved these changes
Sep 16, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work Item / Issue Reference
Summary
This pull request significantly improves support for streaming and fetching large binary (VARBINARY(MAX)) and large text columns in the MSSQL Python driver. The main changes include robust chunked retrieval of large objects (LOBs), correct handling of edge cases (such as empty or null values), and enhanced test coverage for these scenarios.
LOB Streaming and Fetching Enhancements:
FetchLobColumnDatato efficiently stream and assemble large binary/text columns (LOBs) from the database, handling chunking, null/empty values, and correct type conversion for both binary and (wide/narrow) string columns.SQLGetData_wrap,FetchBatchData,FetchMany_wrap, andFetchAll_wrapto detect LOB columns and use the new streaming path for fetching them, including proper fallback to row-by-row fetching when LOBs are present. [1] [2] [3] [4] [5] [6]Testing Improvements:
test_varbinarymax_insert_fetch) that verifies insertion and retrieval of empty, small, and large VARBINARY(MAX) values (including edge cases around the 8000-byte threshold) usingfetchone,fetchall, andfetchmany.These changes ensure that the driver can reliably handle large binary and text columns in all fetch scenarios, improving correctness and robustness for users working with LOB data.