Skip to content

chore: Bump the default split page concurrency#122

Merged
awalker4 merged 1 commit intomainfrom
austin/concurrency-default
Jun 28, 2024
Merged

chore: Bump the default split page concurrency#122
awalker4 merged 1 commit intomainfrom
austin/concurrency-default

Conversation

@awalker4
Copy link
Copy Markdown
Collaborator

Verified that this shows a speedup by doing a local pip install and running the following snippet before and after the change:

from unstructured_client import UnstructuredClient
from unstructured_client.models import shared

s = UnstructuredClient(
    server_url=SERVER_URL,
    api_key_auth=API_KEY,
    )

filename = "../_sample_docs/layout-parser-paper.pdf"

with open(filename, "rb") as f:
    # Note that this currently only supports a single file
    files=shared.Files(
        content=f.read(),
        file_name=filename,
	)

req = shared.PartitionParameters(
    files=files,
    strategy="hi_res",
)

start_time = time.time()
resp = s.general.partition(req)
end_time = time.time()
print(f"Elapsed time: {end_time - start_time} seconds")

Verified that this shows a speedup by doing a local pip install and running the following snippet
before and after the change:

```
from unstructured_client import UnstructuredClient
from unstructured_client.models import shared

s = UnstructuredClient(
    server_url=SERVER_URL,
    api_key_auth=API_KEY,
    )

filename = "../_sample_docs/layout-parser-paper.pdf"

with open(filename, "rb") as f:
    # Note that this currently only supports a single file
    files=shared.Files(
        content=f.read(),
        file_name=filename,
	)

req = shared.PartitionParameters(
    files=files,
    strategy="hi_res",
)

start_time = time.time()
resp = s.general.partition(req)
end_time = time.time()
print(f"Elapsed time: {end_time - start_time} seconds")
```
@awalker4 awalker4 requested review from Coniferish and badGarnet June 27, 2024 19:18
Copy link
Copy Markdown
Contributor

@Coniferish Coniferish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM once tests pass

@awalker4
Copy link
Copy Markdown
Collaborator Author

Ah, the tests should pass once this is merged. The integration tests are pulling the latest unstructured-api image and testing requests against it, so we got blocked by the recent docx "None is not a valid mimetype` bug.

@awalker4 awalker4 enabled auto-merge (squash) June 28, 2024 13:54
@awalker4 awalker4 merged commit 1dd7794 into main Jun 28, 2024
@awalker4 awalker4 deleted the austin/concurrency-default branch June 28, 2024 14:21
awalker4 added a commit to Unstructured-IO/unstructured-js-client that referenced this pull request Jun 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants