Skip to content

fix: 103 batch write example#112

Merged
karel-rehor merged 9 commits intomainfrom
fix/103-batch-write-example
Sep 23, 2024
Merged

fix: 103 batch write example#112
karel-rehor merged 9 commits intomainfrom
fix/103-batch-write-example

Conversation

@karel-rehor
Copy link
Contributor

@karel-rehor karel-rehor commented Sep 17, 2024

Closes #103

Proposed Changes

  • updates the example batching_example.py to show making calls after the context manager for the write has closed.
  • adds integration tests for callbacks and for verifying close of client after writes.

Checklist

  • CHANGELOG.md updated
  • Rebased/mergeable
  • A test has been added if appropriate
  • Tests pass
  • Commit messages are conventional
  • Sign CLA (if not already signed)

@codecov
Copy link

codecov bot commented Sep 17, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.52%. Comparing base (e48744e) to head (232b4d3).
Report is 13 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #112      +/-   ##
==========================================
+ Coverage   55.12%   59.52%   +4.39%     
==========================================
  Files          33       33              
  Lines        2048     2048              
==========================================
+ Hits         1129     1219      +90     
+ Misses        919      829      -90     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@karel-rehor karel-rehor marked this pull request as draft September 17, 2024 12:01
@karel-rehor karel-rehor changed the title Fix/103 batch write example fix: 103 batch write example Sep 17, 2024
Copy link
Member

@bednar bednar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@karel-rehor karel-rehor marked this pull request as ready for review September 19, 2024 15:38
Copy link
Contributor

@jstirnaman jstirnaman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Unrelated (I think), but tests/test_write_file.py is failing:

> pytest
========================================================================================================================================= test session starts ==========================================================================================================================================
platform darwin -- Python 3.11.9, pytest-7.4.1, pluggy-1.3.0
rootdir: /Users/ja/Documents/github/influxdb3-python
configfile: pytest.ini
plugins: cov-2.12.1, pytest_codeblocks-0.16.1, anyio-4.3.0, dotenv-0.5.2
collected 62 items                                                                                                                                                                                                                                                                                     

tests/test_api_client.py .......                                                                                                                                                                                                                                                                 [ 11%]
tests/test_dataframe_serializer.py ........................                                                                                                                                                                                                                                      [ 50%]
tests/test_date_helper.py ...                                                                                                                                                                                                                                                                    [ 54%]
tests/test_deep_merge.py .......                                                                                                                                                                                                                                                                 [ 66%]
tests/test_influxdb_client_3.py ...                                                                                                                                                                                                                                                              [ 70%]
tests/test_influxdb_client_3_integration.py ssssss                                                                                                                                                                                                                                               [ 80%]
tests/test_merge_options.py ....                                                                                                                                                                                                                                                                 [ 87%]
tests/test_point.py ..                                                                                                                                                                                                                                                                           [ 90%]
tests/test_polars.py ..                                                                                                                                                                                                                                                                          [ 93%]
tests/test_query.py ...                                                                                                                                                                                                                                                                          [ 98%]
tests/test_write_file.py F                                                                                                                                                                                                                                                                       [100%]

=============================================================================================================================================== FAILURES ===============================================================================================================================================
__________________________________________________________________________________________________________________________________ TestWriteFile.test_write_file_csv ___________________________________________________________________________________________________________________________________

self = <tests.test_write_file.TestWriteFile testMethod=test_write_file_csv>

    def test_write_file_csv(self):
    
        mock_write = Mock(spec=WriteApi)
        self.client._write_api.write = mock_write.write
    
        self.client.write_file(file='tests/data/iot.csv', timestamp_column='time', measurement_name="iot-devices",
                               tag_columns=["building"], write_precision='s')
    
        expected_df = pd.DataFrame({
            "name": ['iot-devices', 'iot-devices', 'iot-devices'],
            "building": ['5a', '5a', '5a'],
            "temperature": [72.3, 72.1, 72.2],
            "time": pd.to_datetime(["2022-10-01T12:01:00Z", "2022-10-02T12:01:00Z", "2022-10-03T12:01:00Z"])
            .astype('datetime64[s, UTC]'),
        })
        expected = {
            'bucket': 'my_db',
            'record': expected_df,
            'data_frame_measurement_name': 'iot-devices',
            'data_frame_tag_columns': ['building'],
            'data_frame_timestamp_column': 'time',
            'write_precision': 's'
        }
    
        _, actual = mock_write.write.call_args
        assert mock_write.write.call_count == 1
>       assert expected_df.equals(actual['record'])
E       assert False
E        +  where False = <bound method NDFrame.equals of           name building  temperature                      time\n0  iot-devices       5a...devices       5a         72.1 2022-10-02 12:01:00+00:00\n2  iot-devices       5a         72.2 2022-10-03 12:01:00+00:00>(          name building  temperature                      time\n0  iot-devices       5a         72.3 2022-10-01 12:01:0...-devices       5a         72.1 2022-10-02 12:01:00+00:00\n2  iot-devices       5a         72.2 2022-10-03 12:01:00+00:00)
E        +    where <bound method NDFrame.equals of           name building  temperature                      time\n0  iot-devices       5a...devices       5a         72.1 2022-10-02 12:01:00+00:00\n2  iot-devices       5a         72.2 2022-10-03 12:01:00+00:00> =           name building  temperature                      time\n0  iot-devices       5a         72.3 2022-10-01 12:01:0...-devices       5a         72.1 2022-10-02 12:01:00+00:00\n2  iot-devices       5a         72.2 2022-10-03 12:01:00+00:00.equals

tests/test_write_file.py:52: AssertionError
======================================================================================================================================= short test summary info ========================================================================================================================================
FAILED tests/test_write_file.py::TestWriteFile::test_write_file_csv - assert False
================================================================================

batch_size = 10

def success(conf, data):
nonlocal write_success, write_count
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️
I didn't know about nonlocal

print(f"Elapsed time ms: {int(callback.elapsed() / 1_000_000)}")


if __name__ == "__main__":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

@karel-rehor karel-rehor merged commit 0303b3f into main Sep 23, 2024
@karel-rehor karel-rehor deleted the fix/103-batch-write-example branch September 23, 2024 15:01
@bednar bednar added this to the 0.10.0 milestone Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batch write example doesn't wait for callbacks and batch client doesn't support async/await

3 participants