-
Notifications
You must be signed in to change notification settings - Fork 12
Description
Not sure if this is a known issue or something that is easily identifiable, but I've noticed that there seems to be a bit of a performance downgrade using versions 0.3.0 or 0.3.1 compared to 0.2.3 when I'm retrieving data from an Excel file on a network path that I'm connected to via a VPN connection. I'm not on the most recent version of Python (environment attributes noted momentarily), so I don't know if the more recent versions of python-calamine are intended to be more compatible with more recent versions of Python.
The general conclusion of my testing is that this is only an issue when retrieving over the network that I'm connected to via a VPN connection and not on a local drive. The performance might be slightly better for 0.2.3 over a local drive compared to 0.3.0/.1 but I think that difference is negligible. I have not had the opportunity to test when I'm directly connected to the network as opposed to VPN. I would expect performance to improve in that scenario, and that the VPN itself is the ultimate performance bottleneck. However, I still do think it's worth noting the apparent downgrade from 0.2.3 to 0.3.x - there is something there I think.
Also, very important to mention, I'm running the program in a loop, and the initial run is much slower in comparison to subsequent runs in the loop for each calamine version. I'm guessing this is due to some sort of caching/"warm-up" to the network path.
Here are some baseline attributes of my environment:
- OS: Windows 10 Version 22H2 OS Build 19045.5247
- Machine: HP EliteBook x360 830 G8 Notebook PC
- Processor: 11th Gen Intel Core i5-1145G7 @2.60GHz 2611 Mhz
- RAM: 16 GB
- Disk: SSD
- VPN Provider: Cisco AnyConnect
- Python Version: 3.10.2
- Virtual Environment: venv (separate venvs for each version of python-calamine)
- File Type: xlsx
- File Size: 44,756 KB
- Data: 186,139 rows * 24 columns on a single sheet (there are a total of 31 sheets on the file, but I’m only ever reading from one of them)
I do not know if this is reproduceable because I do not know if it’s possible to mimic the exact attributes of my environment (specifically the VPN connection/network setup. I also can’t provide the exact .xlsx file due to data sensitivity, but maybe it’s possible to create one of a similar size at least), and if it is possible, I could not instruct anyone as to how. But here’s what I did:
-
Install
python-calamineversion 0.2.3, 0.3.0, or 0.3.1 in a clean virtual environment. -
Run the following code (I’m sure there is a more elegant and/or efficient way to do this testing, but this is what I did )for each setup venv containing the different python-calamine version in various scenarios. I used the following scenarios:
a. Retrieve the file data from the network path over the VPN in a simultaneous run (i.e. running each venv’s program at the same time)
b. Retrieve the file data from the network path over the VPN in a non-simultaneous (sequential) run, waiting for each program to finish prior to re-running
c. Retrieve the file data from the same file but on a local path in a non-simultaneous run
from python_calamine import CalamineWorkbook
import time
from typing import Any
def measure_time(func):
def wrapper(*args, **kwargs):
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
elapsed_time = end_time - start_time
print(f'{func.__name__} took {elapsed_time:.2f} sec')
return result
return wrapper
@measure_time
def read_excel_with_calamine(file: str, sheet_name: str) -> list[list[Any]]:
return CalamineWorkbook.from_path(file).get_sheet_by_name(sheet_name).to_python()
def loop_calamine():
for _ in range(1, 11):
read_excel_with_calamine("\\path\\to\\file.xlsx",
"tab_name")- I ran each scenario 3 times. However, while I'm not providing this data, note that when I did the simultaneous run retrieving from the network path, version 0.2.3 "lapped" the filed, as I was able to run it fully several times prior to 0.3.0/.1 completing once. Here are the raw results; I'm also attaching a simple spreadsheet that contains a more summarized breakdown.
Please let me know if there's any additional information I can/should provide or any further testing you might recommend.
0.2.3 network simultaneous test 1:
read_excel_with_calamine took 30.17 sec
read_excel_with_calamine took 4.62 sec
read_excel_with_calamine took 4.56 sec
read_excel_with_calamine took 4.83 sec
read_excel_with_calamine took 4.60 sec
read_excel_with_calamine took 5.77 sec
read_excel_with_calamine took 6.55 sec
read_excel_with_calamine took 6.14 sec
read_excel_with_calamine took 6.88 sec
read_excel_with_calamine took 6.31 sec
0.2.3 network simultaneous test 2:
read_excel_with_calamine took 4.77 sec
read_excel_with_calamine took 4.70 sec
read_excel_with_calamine took 5.37 sec
read_excel_with_calamine took 7.65 sec
read_excel_with_calamine took 9.37 sec
read_excel_with_calamine took 8.86 sec
read_excel_with_calamine took 8.25 sec
read_excel_with_calamine took 6.40 sec
read_excel_with_calamine took 9.41 sec
read_excel_with_calamine took 9.33 sec
0.2.3 network simultaneous test 3:
read_excel_with_calamine took 5.17 sec
read_excel_with_calamine took 7.41 sec
read_excel_with_calamine took 12.83 sec
read_excel_with_calamine took 11.12 sec
read_excel_with_calamine took 10.37 sec
read_excel_with_calamine took 9.78 sec
read_excel_with_calamine took 10.80 sec
read_excel_with_calamine took 10.22 sec
read_excel_with_calamine took 10.54 sec
read_excel_with_calamine took 8.73 sec
0.3.0 network simultaneous test 1:
read_excel_with_calamine took 239.91 sec
read_excel_with_calamine took 13.96 sec
read_excel_with_calamine took 12.48 sec
read_excel_with_calamine took 10.96 sec
read_excel_with_calamine took 11.33 sec
read_excel_with_calamine took 11.26 sec
read_excel_with_calamine took 11.15 sec
read_excel_with_calamine took 9.94 sec
read_excel_with_calamine took 9.53 sec
read_excel_with_calamine took 7.56 sec
0.3.0 network simultaneous test 2:
read_excel_with_calamine took 375.45 sec
read_excel_with_calamine took 8.94 sec
read_excel_with_calamine took 8.75 sec
read_excel_with_calamine took 9.54 sec
read_excel_with_calamine took 9.75 sec
read_excel_with_calamine took 8.46 sec
read_excel_with_calamine took 8.09 sec
read_excel_with_calamine took 8.21 sec
read_excel_with_calamine took 7.56 sec
read_excel_with_calamine took 6.38 sec
0.3.0 network simultaneous test 3:
read_excel_with_calamine took 398.54 sec
read_excel_with_calamine took 5.52 sec
read_excel_with_calamine took 6.29 sec
read_excel_with_calamine took 6.84 sec
read_excel_with_calamine took 6.82 sec
read_excel_with_calamine took 6.53 sec
read_excel_with_calamine took 6.52 sec
read_excel_with_calamine took 6.33 sec
read_excel_with_calamine took 7.01 sec
read_excel_with_calamine took 7.68 sec
0.3.1 network simultaneous test 1:
read_excel_with_calamine took 221.10 sec
read_excel_with_calamine took 13.65 sec
read_excel_with_calamine took 12.32 sec
read_excel_with_calamine took 10.99 sec
read_excel_with_calamine took 10.86 sec
read_excel_with_calamine took 11.54 sec
read_excel_with_calamine took 11.22 sec
read_excel_with_calamine took 9.87 sec
read_excel_with_calamine took 9.51 sec
read_excel_with_calamine took 7.71 sec
0.3.1 network simultaneous test 2:
read_excel_with_calamine took 378.07 sec
read_excel_with_calamine took 8.68 sec
read_excel_with_calamine took 8.54 sec
read_excel_with_calamine took 9.43 sec
read_excel_with_calamine took 9.70 sec
read_excel_with_calamine took 8.32 sec
read_excel_with_calamine took 8.27 sec
read_excel_with_calamine took 8.18 sec
read_excel_with_calamine took 7.88 sec
read_excel_with_calamine took 6.39 sec
0.3.1 network simultanoues test 3:
read_excel_with_calamine took 396.81 sec
read_excel_with_calamine took 5.51 sec
read_excel_with_calamine took 6.18 sec
read_excel_with_calamine took 6.84 sec
read_excel_with_calamine took 6.76 sec
read_excel_with_calamine took 6.51 sec
read_excel_with_calamine took 6.49 sec
read_excel_with_calamine took 6.30 sec
read_excel_with_calamine took 7.01 sec
read_excel_with_calamine took 7.63 sec
0.2.3 network non-simultaneous test 1:
read_excel_with_calamine took 32.26 sec
read_excel_with_calamine took 4.69 sec
read_excel_with_calamine took 4.86 sec
read_excel_with_calamine took 4.98 sec
read_excel_with_calamine took 5.57 sec
read_excel_with_calamine took 7.01 sec
read_excel_with_calamine took 8.38 sec
read_excel_with_calamine took 6.54 sec
read_excel_with_calamine took 8.28 sec
read_excel_with_calamine took 9.96 sec
0.2.3 network non-simultaneous test 2:
read_excel_with_calamine took 31.38 sec
read_excel_with_calamine took 5.49 sec
read_excel_with_calamine took 5.87 sec
read_excel_with_calamine took 5.29 sec
read_excel_with_calamine took 5.44 sec
read_excel_with_calamine took 5.03 sec
read_excel_with_calamine took 5.76 sec
read_excel_with_calamine took 6.36 sec
read_excel_with_calamine took 5.75 sec
read_excel_with_calamine took 5.95 sec
0.2.3 network non-simultaneous test 3:
read_excel_with_calamine took 31.58 sec
read_excel_with_calamine took 6.28 sec
read_excel_with_calamine took 5.63 sec
read_excel_with_calamine took 5.22 sec
read_excel_with_calamine took 5.01 sec
read_excel_with_calamine took 4.90 sec
read_excel_with_calamine took 4.92 sec
read_excel_with_calamine took 5.45 sec
read_excel_with_calamine took 5.06 sec
read_excel_with_calamine took 5.05 sec
0.3.0 network non-simultaneous test 1:
read_excel_with_calamine took 402.47 sec
read_excel_with_calamine took 6.04 sec
read_excel_with_calamine took 5.49 sec
read_excel_with_calamine took 8.78 sec
read_excel_with_calamine took 8.19 sec
read_excel_with_calamine took 7.95 sec
read_excel_with_calamine took 8.23 sec
read_excel_with_calamine took 8.32 sec
read_excel_with_calamine took 7.62 sec
read_excel_with_calamine took 7.29 sec
0.3.0 network non-simultaneous test 2:
read_excel_with_calamine took 403.66 sec
read_excel_with_calamine took 6.08 sec
read_excel_with_calamine took 5.93 sec
read_excel_with_calamine took 6.46 sec
read_excel_with_calamine took 6.19 sec
read_excel_with_calamine took 6.38 sec
read_excel_with_calamine took 6.23 sec
read_excel_with_calamine took 6.40 sec
read_excel_with_calamine took 6.18 sec
read_excel_with_calamine took 6.00 sec
0.3.0 network non-simultaneous test 3:
read_excel_with_calamine took 406.92 sec
read_excel_with_calamine took 5.90 sec
read_excel_with_calamine took 5.99 sec
read_excel_with_calamine took 6.10 sec
read_excel_with_calamine took 6.05 sec
read_excel_with_calamine took 8.24 sec
read_excel_with_calamine took 8.98 sec
read_excel_with_calamine took 7.19 sec
read_excel_with_calamine took 9.27 sec
read_excel_with_calamine took 10.16 sec
0.3.1 network non-simultaneous test 1:
read_excel_with_calamine took 403.22 sec
read_excel_with_calamine took 6.09 sec
read_excel_with_calamine took 6.03 sec
read_excel_with_calamine took 5.72 sec
read_excel_with_calamine took 5.95 sec
read_excel_with_calamine took 6.13 sec
read_excel_with_calamine took 6.02 sec
read_excel_with_calamine took 7.32 sec
read_excel_with_calamine took 7.04 sec
read_excel_with_calamine took 6.95 sec
0.3.1 network non-simultaneous test 2:
read_excel_with_calamine took 403.54 sec
read_excel_with_calamine took 12.19 sec
read_excel_with_calamine took 13.20 sec
read_excel_with_calamine took 12.02 sec
read_excel_with_calamine took 8.83 sec
read_excel_with_calamine took 8.60 sec
read_excel_with_calamine took 8.94 sec
read_excel_with_calamine took 8.17 sec
read_excel_with_calamine took 9.08 sec
read_excel_with_calamine took 9.22 sec
0.3.1 network non-simultaneous test 3:
read_excel_with_calamine took 396.97 sec
read_excel_with_calamine took 6.53 sec
read_excel_with_calamine took 6.85 sec
read_excel_with_calamine took 6.10 sec
read_excel_with_calamine took 6.95 sec
read_excel_with_calamine took 8.24 sec
read_excel_with_calamine took 8.93 sec
read_excel_with_calamine took 8.72 sec
read_excel_with_calamine took 8.78 sec
read_excel_with_calamine took 8.96 sec
0.2.3 local non-simultaneous test 1:
read_excel_with_calamine took 4.26 sec
read_excel_with_calamine took 4.27 sec
read_excel_with_calamine took 4.37 sec
read_excel_with_calamine took 5.70 sec
read_excel_with_calamine took 5.97 sec
read_excel_with_calamine took 6.11 sec
read_excel_with_calamine took 5.52 sec
read_excel_with_calamine took 6.82 sec
read_excel_with_calamine took 6.47 sec
read_excel_with_calamine took 5.90 sec
0.2.3 local non-simultaneous test 2:
read_excel_with_calamine took 4.65 sec
read_excel_with_calamine took 4.41 sec
read_excel_with_calamine took 4.52 sec
read_excel_with_calamine took 4.45 sec
read_excel_with_calamine took 4.32 sec
read_excel_with_calamine took 4.79 sec
read_excel_with_calamine took 4.63 sec
read_excel_with_calamine took 4.61 sec
read_excel_with_calamine took 4.66 sec
read_excel_with_calamine took 5.20 sec
0.2.3 local non-simultaneous test 3:
read_excel_with_calamine took 4.51 sec
read_excel_with_calamine took 4.94 sec
read_excel_with_calamine took 5.69 sec
read_excel_with_calamine took 5.97 sec
read_excel_with_calamine took 6.09 sec
read_excel_with_calamine took 4.59 sec
read_excel_with_calamine took 5.09 sec
read_excel_with_calamine took 6.12 sec
read_excel_with_calamine took 5.18 sec
read_excel_with_calamine took 4.71 sec
0.3.0 local non-simultaneous test 1:
read_excel_with_calamine took 7.70 sec
read_excel_with_calamine took 6.43 sec
read_excel_with_calamine took 5.50 sec
read_excel_with_calamine took 5.33 sec
read_excel_with_calamine took 5.35 sec
read_excel_with_calamine took 4.93 sec
read_excel_with_calamine took 4.98 sec
read_excel_with_calamine took 5.12 sec
read_excel_with_calamine took 4.88 sec
read_excel_with_calamine took 4.89 sec
0.3.0 local non-simultaneous test 2:
read_excel_with_calamine took 4.68 sec
read_excel_with_calamine took 4.69 sec
read_excel_with_calamine took 4.63 sec
read_excel_with_calamine took 4.64 sec
read_excel_with_calamine took 4.71 sec
read_excel_with_calamine took 4.67 sec
read_excel_with_calamine took 4.73 sec
read_excel_with_calamine took 5.00 sec
read_excel_with_calamine took 5.52 sec
read_excel_with_calamine took 5.23 sec
0.3.0 local non-simultaneous test 3:
read_excel_with_calamine took 4.69 sec
read_excel_with_calamine took 4.62 sec
read_excel_with_calamine took 5.01 sec
read_excel_with_calamine took 4.74 sec
read_excel_with_calamine took 4.89 sec
read_excel_with_calamine took 4.77 sec
read_excel_with_calamine took 4.91 sec
read_excel_with_calamine took 4.94 sec
read_excel_with_calamine took 4.94 sec
read_excel_with_calamine took 5.22 sec
0.3.1 local non-simultaneous test 1:
read_excel_with_calamine took 4.81 sec
read_excel_with_calamine took 4.87 sec
read_excel_with_calamine took 5.59 sec
read_excel_with_calamine took 5.05 sec
read_excel_with_calamine took 5.28 sec
read_excel_with_calamine took 5.02 sec
read_excel_with_calamine took 5.37 sec
read_excel_with_calamine took 5.06 sec
read_excel_with_calamine took 4.85 sec
read_excel_with_calamine took 4.93 sec
0.3.1 local non-simultaneous test 2:
read_excel_with_calamine took 6.02 sec
read_excel_with_calamine took 6.02 sec
read_excel_with_calamine took 6.03 sec
read_excel_with_calamine took 6.21 sec
read_excel_with