UN-2954: Display average confidence of highlighted lines in Prompt Studio#1646
Conversation
Summary by CodeRabbit
WalkthroughConfidence and coordinate handling were adjusted across three frontend components: DocumentManager now formats varied confidence shapes explicitly; PdfViewer strips and ignores confidence when filtering zero-coordinate highlights; PromptCard extracts confidences from nested coordinate arrays, averages them, and uses that averaged value when selecting highlights. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant PromptCard
participant PdfViewer
participant DocumentManager
Note over User,PromptCard: User selects a highlight
User ->> PromptCard: selectHighlight(highlightData)
activate PromptCard
PromptCard ->> PromptCard: flatten coords (slice first 4)
PromptCard ->> PromptCard: extract any 5th-element confidences
PromptCard ->> PromptCard: compute average confidence or fallback
PromptCard ->> PdfViewer: send cleaned coords (no confidence)
deactivate PromptCard
activate PdfViewer
PdfViewer ->> PdfViewer: removeZerosAndDeleteIfAllZero (checks first 4 coords)
PdfViewer ->> DocumentManager: updated highlight payload (coords only)
deactivate PdfViewer
activate DocumentManager
DocumentManager ->> DocumentManager: format confidence display (inline function)
DocumentManager -->> User: render highlight + confidence
deactivate DocumentManager
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes
Pre-merge checks and finishing touches✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…udio - Extract confidence from 5th element of highlight coordinate arrays - Calculate and display average confidence across multiple lines - Strip 5th element before rendering PDF highlights - Support both new format (number) and old format (nested arrays) for confidence display
Deepak-Kesavan
force-pushed
the
UN-2954-display-average-confidence-of-highlighted-line-in-prompt-studio
branch
from
964b647 to
3a62040
Compare
- Fix quote style consistency (single to double quotes) - Fix code formatting and line breaks - Address ESLint/Prettier warnings
Deepak-Kesavan
requested review from
jagadeeswaran-zipstack and
vishnuszipstack
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
frontend/src/components/custom-tools/pdf-viewer/PdfViewer.jsx (1)
40-42: Consider eliminating redundant processing.The
removeZerosAndDeleteIfAllZerofunction is called twice with the same input (highlightData): once at line 40 to createprocessHighlightData, and again at line 80 within the effect. This redundant processing can be avoided by reusing the already-computedprocessHighlightDatain the effect.Consider this optimization:
+ const processHighlightData = useMemo( + () => (highlightData ? removeZerosAndDeleteIfAllZero(highlightData) : []), + [highlightData] + ); - const processHighlightData = highlightData - ? removeZerosAndDeleteIfAllZero(highlightData) - : []; // ... rest of code ... useEffect(() => { - const cleanedHighlightData = removeZerosAndDeleteIfAllZero(highlightData); + const cleanedHighlightData = processHighlightData; if (cleanedHighlightData && cleanedHighlightData.length > 0) { // ... rest of effect logic ... } - }, [highlightData, jumpToPage, currentHighlightIndex]); + }, [processHighlightData, jumpToPage, currentHighlightIndex]);Note: The comment at line 96 suggests the dependency was intentionally changed from
processedHighlightDatatohighlightData. If there was a specific reason for this change (e.g., preventing infinite loops), please disregard this suggestion.Also applies to: 79-96
frontend/src/components/custom-tools/prompt-card/PromptCard.jsx (1)
216-251: Consider validating confidence value range.The extraction logic correctly handles nested structures and computes averages. However, confidence scores are typically in the range [0, 1], and the function doesn't validate this assumption.
Apply this diff to add range validation:
if (Array.isArray(item)) { // Check if this is a coordinate array with 5 elements - if (item.length >= 5 && typeof item[4] === "number") { - confidenceValues.push(item[4]); + if ( + item.length >= 5 && + typeof item[4] === "number" && + item[4] >= 0 && + item[4] <= 1 + ) { + confidenceValues.push(item[4]); } else {frontend/src/components/custom-tools/document-manager/DocumentManager.jsx (1)
377-395: Normalize legacy confidence formatting to two decimals.New numeric confidences get
toFixed(2), but the legacy array branches return the raw value or"1", so older highlights can render with arbitrary precision (or no trailing zeros). Convert those branches throughNumber(...).toFixed(2)—and return"1.00"for the empty-array sentinel—to keep the display consistent with the stated two-decimal requirement.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to Reviews > Disable Cache setting
Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting
📒 Files selected for processing (3)
frontend/src/components/custom-tools/document-manager/DocumentManager.jsx(1 hunks)frontend/src/components/custom-tools/pdf-viewer/PdfViewer.jsx(1 hunks)frontend/src/components/custom-tools/prompt-card/PromptCard.jsx(3 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
🔇 Additional comments (3)
frontend/src/components/custom-tools/pdf-viewer/PdfViewer.jsx (1)
24-38: LGTM! Confidence stripping logic is correct.The implementation correctly strips the 5th element (confidence) from coordinate arrays while maintaining backward compatibility with 4-element arrays. The two-step approach (filter based on first 4 coordinates, then map to return only first 4 elements) ensures highlights render correctly with PDF viewer expectations.
frontend/src/components/custom-tools/prompt-card/PromptCard.jsx (2)
267-277: LGTM: Confidence prioritization logic is well-implemented.The fallback chain correctly prioritizes extracted confidence from coordinate arrays over API-provided confidence data, and the nullish coalescing operator (
??) ensures proper backward compatibility.
172-178: Verify coordinate array structure expectations and add defensive validation if needed.The coordinate stripping logic is correct—it safely handles arrays with fewer than 5 elements by passing them through unchanged. However, there is no validation ensuring coordinates have at least 4 elements
[page, x, y, width]before reaching rendering.The code at line 175 (
coords.length >= 5 ? coords.slice(0, 4) : coords) will pass through arrays with 1-3 elements, which could cause rendering issues downstream. While PdfViewer has a fallback[[0, 0, 0, 0]]for empty data, it doesn't guard against undersized coordinate arrays.Verification needed:
- Confirm that coordinate arrays from highlight data always have 4+ elements before
addCoordsToFlattenedis called- If source data can be incomplete, add a length check:
coordsOnly.length >= 4 ? coordsOnly : nullor similar validation
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
platform-service/pyproject.toml (1)
18-18: Optional: Consider standardizing extras order across multi-cloud dependencies.The extras are specified as
[gcs, azure, aws]here, while workers/pyproject.toml uses[aws, gcs, azure]. While functionally equivalent, standardizing the order across files would improve consistency and reduce noise in future diffs.prompt-service/pyproject.toml (1)
18-18: Inconsistent version pinning between unstract-sdk and unstract-sdk1.
unstract-sdkis pinned to~=0.79.0(line 18), butunstract-sdk1(line 22) lacks a version constraint. This asymmetry may lead to unpredictable builds if unstract-sdk1 receives incompatible updates. Consider pinning both consistently.Also applies to: 22-22
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to Reviews > Disable Cache setting
Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting
⛔ Files ignored due to path filters (5)
backend/uv.lockis excluded by!**/*.lockplatform-service/uv.lockis excluded by!**/*.lockprompt-service/uv.lockis excluded by!**/*.lockunstract/filesystem/uv.lockis excluded by!**/*.lockuv.lockis excluded by!**/*.lock
📒 Files selected for processing (11)
backend/pyproject.toml(1 hunks)frontend/src/components/custom-tools/prompt-card/PromptCard.jsx(3 hunks)platform-service/pyproject.toml(1 hunks)prompt-service/pyproject.toml(1 hunks)pyproject.toml(1 hunks)tools/classifier/requirements.txt(1 hunks)tools/structure/requirements.txt(1 hunks)tools/text_extractor/requirements.txt(1 hunks)unstract/filesystem/pyproject.toml(1 hunks)unstract/tool-registry/pyproject.toml(1 hunks)workers/pyproject.toml(1 hunks)
✅ Files skipped from review due to trivial changes (1)
- backend/pyproject.toml
🚧 Files skipped from review as they are similar to previous changes (1)
- frontend/src/components/custom-tools/prompt-card/PromptCard.jsx
🔇 Additional comments (11)
tools/structure/requirements.txt (2)
6-6: Frontend changes absent from review scope.The PR objectives detail changes to PromptCard.jsx, PdfViewer.jsx, and DocumentManager.jsx, but the files under review contain only backend Python dependency updates. Please confirm whether the frontend JSX files are included in this PR or scheduled for a separate review.
6-6: Dependency bump is consistent and reasonable.The version bump from ~=0.78.0 to ~=0.79.0 uses an appropriate compatible-release specifier. This specifier allows patch updates (0.79.x) while preventing unexpected minor version changes, which aligns with good dependency management practices.
tools/classifier/requirements.txt (1)
7-7: Consistent dependency update across tool packages.The bump to unstract-sdk[aws]~=0.79.0 mirrors the same change in tools/structure/requirements.txt, indicating a coordinated dependency update across multiple tool components.
pyproject.toml (1)
44-44: Development dependency bump is appropriate.The unstract-sdk update in the hook-check-django-migrations group (a dev/CI tooling dependency group) uses a conservative compatible-release specifier, which is suitable for development dependencies.
tools/text_extractor/requirements.txt (1)
7-7: Consistent tool package dependency update.The update to unstract-sdk[aws]~=0.79.0 aligns with the same version bump applied across tools/structure/requirements.txt and tools/classifier/requirements.txt.
unstract/tool-registry/pyproject.toml (1)
14-14: Runtime dependency bump is consistent.The update to unstract-sdk~=0.79.0 in the tool-registry package's runtime dependencies follows the same coordinated version bump pattern.
workers/pyproject.toml (1)
27-27: Core workers dependency update with multi-cloud extras.The bump to unstract-sdk[aws,gcs,azure]~=0.79.0 appropriately includes multi-cloud provider extras, consistent with the workers package's broader cloud support needs compared to individual tool packages.
platform-service/pyproject.toml (1)
18-18: Platform service dependency update with multi-cloud support.The update to unstract-sdk[gcs, azure, aws]~=0.79.0 appropriately reflects the platform-service's need for multi-cloud provider support, consistent with the workers package scope.
unstract/filesystem/pyproject.toml (2)
9-9: Filesystem package dependency update.The update to unstract-sdk~=0.79.0 is appropriate for the filesystem package, which requires the SDK without cloud-specific extras.
6-6: Verified: unstract-sdk 0.79.0 is available on PyPI and ready for adoption.unstract-sdk 0.79.0 exists on PyPI with appropriate cloud provider extras (aws, azure, gcs). All 8 files consistently use the safe version specifier ~=0.79.0, which permits patch-level updates while blocking breaking changes. Confirm the frontend JSX files referenced in PR objectives were reviewed separately if out of scope for these dependency updates.
prompt-service/pyproject.toml (1)
18-18: Verify that unstract-sdk 0.79.0 provides the necessary confidence extraction capabilities.The PR objectives describe confidence extraction features but primarily reference llmwhisperer-client 2.5.0 (for sdk1), not unstract-sdk 0.79.0. Confirm that this version bump delivers the confidence handling needed for the feature, or clarify if the llmwhisperer-client bump should be reflected here instead.
…ghted-line-in-prompt-studio
|
Test ResultsSummary
Runner Tests - Full Report
SDK1 Tests - Full Report
|
vishnuszipstack
deleted the
UN-2954-display-average-confidence-of-highlighted-line-in-prompt-studio
branch
3 participants
What
Frontend implementation to display average confidence scores of highlighted lines in Prompt Studio.
Why
To provide users with visibility into the confidence scores of extracted text, helping them assess the reliability of LLMWhisperer's OCR output.
How
PromptCard.jsx:
extractConfidenceFromHighlightData()function to recursively extract confidence values from the 5th element of coordinate arraysaddCoordsToFlattened()to strip the 5th element (confidence) before renderinghandleSelectHighlight()to prioritize extracted confidence over API-provided confidence dataPdfViewer.jsx:
removeZerosAndDeleteIfAllZero()to strip the 5th element before PDF renderingDocumentManager.jsx:
sdk1/pyproject.toml:
llmwhisperer-clientto2.5.0to support confidence extractionCan this PR break any existing features. If yes, please list possible items. If no, please explain why.
No, this PR is backward compatible:
confidenceDatafrom API if 5th element doesn't existDatabase Migrations
Env Config
Relevant Docs
[page, x, y, width, confidence]Related Issues or PRs
Dependencies Versions
llmwhisperer-client==2.5.0(bumped in sdk1/pyproject.toml)Notes on Testing
Screenshots
N/A
Checklist
I have read and understood the Contribution Guidelines.