Skip to content

fix: silent ignore of 4xx errors from reporting#5253

Merged
itsmihir merged 2 commits into
masterfrom
feature/obs-711-fix-silent-ignore-of-400-errors-in-reporting-service
Nov 6, 2024
Merged

fix: silent ignore of 4xx errors from reporting#5253
itsmihir merged 2 commits into
masterfrom
feature/obs-711-fix-silent-ignore-of-400-errors-in-reporting-service

Conversation

@itsmihir

Copy link
Copy Markdown
Contributor

Description

We are currently ignoring 4XX errors from reporting service (for example due to payload too large or invalid sourceDefinitionId value). This result in reports being silently dropped without generating any alert.

By treating 4XX as failure, we ensure that reports are retired, allowing us to detect and receive alerts for high 4XX rates or report backlogs.

Linear Ticket

https://linear.app/rudderstack/issue/OBS-711/fix-silent-ignore-of-400-errors-in-reporting-service

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@codecov

codecov Bot commented Oct 30, 2024

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.23%. Comparing base (2aae3ec) to head (9879825).
Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5253      +/-   ##
==========================================
+ Coverage   73.20%   73.23%   +0.02%     
==========================================
  Files         424      424              
  Lines       59967    59964       -3     
==========================================
+ Hits        43898    43912      +14     
+ Misses      13606    13595      -11     
+ Partials     2463     2457       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@cisse21

cisse21 commented Nov 4, 2024

Copy link
Copy Markdown
Member

Wouldn't it cause noise if we have a developer doing local testing with app.dev and prod reporting? Wouldn't it be better to classify the errors on reporting side and then add retries over here?

@itsmihir

itsmihir commented Nov 4, 2024

Copy link
Copy Markdown
Contributor Author

Wouldn't it cause noise if we have a developer doing local testing with app.dev and prod reporting? Wouldn't it be better to classify the errors on reporting side and then add retries over here?

We already have some error classification in place for the reporting service, and we retry these errors as needed. However, consider a recent issue we encountered: if the source definition ID is incorrect, the reporting service responds with a 400 error. As a result, those reports are completely lost.

@itsmihir itsmihir merged commit 9f2b7e7 into master Nov 6, 2024
@itsmihir itsmihir deleted the feature/obs-711-fix-silent-ignore-of-400-errors-in-reporting-service branch November 6, 2024 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants