Skip to content

Improve Airtable import #99

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
SkywalkerSpace merged 17 commits into from
Oct 10, 2024
Merged

Improve Airtable import #99

SkywalkerSpace merged 17 commits into from
Oct 10, 2024

Conversation

@simonhammes
Copy link
Collaborator

@simonhammes simonhammes commented Sep 23, 2024 •
edited
Loading

Improvements

  • Improved log messages
  • Airtable schema is retrieved using Airtable's meta API instead of trying to decude field types from row values
    • Note: This needs the schema.bases:read scope
  • Allows excluding specific column types (e.g. files) from data sync
  • Allows excluding specific columns from data sync using a list of tuples (table_name, column_name)
  • Creates a separate table in the SeaTable base to track columns that need to be manually migrated (e.g. link-formula)

I also commented out two sleep() statements since Airtable's API seems to be quite slow and I never hit a rate limit. Using a library like requests-ratelimiter might be a more robust option to a) prevent rate limits and b) prevent the migration process from being unnecessarily slow.

@SkywalkerSpace
Copy link
Contributor

SkywalkerSpace commented Sep 29, 2024

Hi, I tested the script and it cannot import data from Airtable.
The Airtable Metadata API is no longer maintained Metadata API documentation and the USER_API_KEY used in the Metadata API cannot be applied after 2024.1. USER_API_KEY and airtable_api_key are not universal.

@simonhammes
Copy link
Collaborator Author

simonhammes commented Oct 2, 2024 •
edited
Loading

Hi, I tested the script and it cannot import data from Airtable.

Hi,
did you receive an error message?
Are you sure that the token includes the schema.bases:read scope (as I wrote in the PR description)?

The Airtable Metadata API is no longer maintained Metadata API documentation and the USER_API_KEY used in the Metadata API cannot be applied after 2024.1. USER_API_KEY and airtable_api_key are not universal.

Where does it say that the API is deprecated? It only says that the documentation has moved and is now available at https://airtable.com/developers/web/api/introduction (or more specifically, https://airtable.com/developers/web/api/get-base-schema).

@simonhammes
Copy link
Collaborator Author

simonhammes commented Oct 2, 2024

I also added another commit (9f585ac) which ensures that table names are properly encoded before being sent to Airtable's API.

This fixes a recent issue by one of our customers who tried to migrate a base that contained table names with slashes.

@SkywalkerSpace
Copy link
Contributor

SkywalkerSpace commented Oct 10, 2024

Sorry, it was my mistake. It can import data after adding schema.bases:read permission to access token.

@SkywalkerSpace SkywalkerSpace merged commit ab2c683 into master Oct 10, 2024
@simonhammes
Copy link
Collaborator Author

simonhammes commented Oct 10, 2024

@SkywalkerSpace Thank you! Can you create a new release on PyPI?

@simonhammes simonhammes deleted the airtable-import-improvements branch October 10, 2024 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@simonhammes @SkywalkerSpace