Skip to content

feat: fast no-sync write support#142

Merged
jansimonb merged 14 commits intomainfrom
feat/no-sync-write
Jun 6, 2025
Merged

feat: fast no-sync write support#142
jansimonb merged 14 commits intomainfrom
feat/no-sync-write

Conversation

@jansimonb
Copy link
Contributor

@jansimonb jansimonb commented Jun 3, 2025

Proposed Changes

Support fast writes without waiting for WAL persistence:

  • New write option (WriteOptions.no_sync) added: True value means faster write but without the confirmation that
    the data was persisted. Default value: False.
  • Supported by self-managed InfluxDB 3 Core and Enterprise servers only!
  • Also configurable via environment variable (INFLUX_WRITE_NO_SYNC).
  • Long precision string values added from v3 HTTP API: "nanosecond", "microsecond", "millisecond", "second" (in addition to the existing "ns", "us", "ms", "s").

Checklist

  • CHANGELOG.md updated
  • Rebased/mergeable
  • A test has been added if appropriate
  • Tests pass
  • Commit messages are conventional
  • Sign CLA (if not already signed)

@codecov
Copy link

codecov bot commented Jun 3, 2025

Codecov Report

Attention: Patch coverage is 96.05263% with 3 lines in your changes missing coverage. Please review.

Project coverage is 65.44%. Comparing base (f8a629b) to head (f53032e).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...db_client_3/write_client/service/signin_service.py 0.00% 1 Missing ⚠️
...b_client_3/write_client/service/signout_service.py 0.00% 1 Missing ⚠️
...xdb_client_3/write_client/service/write_service.py 96.42% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #142      +/-   ##
==========================================
+ Coverage   63.43%   65.44%   +2.00%     
==========================================
  Files          34       35       +1     
  Lines        2210     2266      +56     
==========================================
+ Hits         1402     1483      +81     
+ Misses        808      783      -25     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jansimonb jansimonb marked this pull request as ready for review June 4, 2025 12:21
Copy link
Contributor

@karel-rehor karel-rehor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

started to review. But the global declaration of DEFAULT_WRITE_NO_SYNC seems to be in an odd place.

Copy link
Contributor

@karel-rehor karel-rehor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking good, but I can't help but feel that there is perhaps some unnecessary duplicated lines of code in write_client/service/write_service.py

Copy link
Contributor

@karel-rehor karel-rehor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK looks good to me. Tests run locally. Changes all make sense. 🚴 🏁

@jansimonb jansimonb merged commit 7b9e802 into main Jun 6, 2025
14 checks passed
@jansimonb jansimonb deleted the feat/no-sync-write branch June 6, 2025 12:23
@dburton-influxdata
Copy link

Is it possibly to get a syntax example on how we need to configure the no_sync within our Python code?

I am trying the following and getting some errors:
image

@dburton-influxdata
Copy link

Maybe this hasn't been added to a new release yet? The Repo and my installed version is reporting 0.13.0. which looks like it was released 3 weeks ago and this was just approved last week.

@jansimonb
Copy link
Contributor Author

Maybe this hasn't been added to a new release yet? The Repo and my installed version is reporting 0.13.0. which looks like it was released 3 weeks ago and this was just approved last week.

@dburton-influxdata , you can always check the CHANGELOG.md to see what features comes into which release. This change goes into the upcoming 0.14.0 version which was not released yet. Do you need this feature? Should we speed up the release process?

@dburton-influxdata
Copy link

@jansimonb yes, I am doing some v1 to V3 data migration development and testing. I believe I was able to patch my installed version with your fix last night. However our test run using the no_sync=True didn't show any time savings. I am wanting to test with the gzip feature possibly today as well.

Did you test the no_sync=True option to verify any write performance gains with V3?

@jansimonb
Copy link
Contributor Author

jansimonb commented Jun 16, 2025

Did you test the no_sync=True option to verify any write performance gains with V3?

Yes, I tested it against a local docker-installed influxdb3 server. It is much faster with no_sync=true:

With no_sync=false:

write 0 in 0.5645s
write 1 in 1.0031s
write 2 in 0.9984s
write 3 in 1.0062s
write 4 in 0.9919s

With no_sync=true:

write 0 in 0.0061s
write 1 in 0.0012s
write 2 in 0.0007s
write 3 in 0.0009s
write 4 in 0.0007s

One option is to set the no_sync flag is via constructor:

    with InfluxDBClient3(token=token, host=host, database=db,
                         write_client_options=write_client_options(
                             write_options=WriteOptions(
                                 write_type=WriteType.synchronous,
                                 write_precision=WritePrecision.US,
                                 no_sync=True
                             )
                         )) as _client:

or you can also set it via env:

    os.environ["INFLUX_HOST"] = host
    os.environ["INFLUX_TOKEN"] = token
    os.environ["INFLUX_DATABASE"] = db
    os.environ["INFLUX_WRITE_NO_SYNC"] = "true"

    with InfluxDBClient3.from_env() as _client:

But please note that this change is still not released yet.

@dburton-influxdata
Copy link

Jan, Thank you for your response and the examples. I believe I go tit to work for our initial testing and I look forward to the updated release.

@dburton-influxdata
Copy link

Question: How big was the data set that you used for testing?

@jansimonb
Copy link
Contributor Author

Question: How big was the data set that you used for testing?

I tested just with writing a single record. For a single record the saved time approximately equals to the wal-flush-interval set on server. For larger dataset this can be different based the overall WAL configuration.

@karel-rehor karel-rehor added this to the 0.14.0 milestone Jun 18, 2025
@jansimonb
Copy link
Contributor Author

@dburton-influxdata , fyi, the feature was released.

@dburton-influxdata
Copy link

@jansimonb Did this release include the should_gzip feature also get released?

Either way, do you know or can you find out if we can gzip the query response (payload)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants