GitHub - kaixinol/twitter_user_tweet_crawler: A Python crawler tool that can automatically simulate browser operations to crawl all users' tweet content and save all static resources (videos, pictures) locally without calling the Twitter API. Use the selenium library.

⚠️Warning: This project uses browser-simulated clicks to retrieve resources instead of using the GraphQL API, which may result in lower efficiency.

Warning

Do not leak your cookie.json, this will lead to your Twitter account being stolen.

This tool can automatically simulate browser operations to crawl all users' tweets and save all static resources (videos, pictures) locally without calling the Twitter API.
At the same time, sqlite3 is used to save the crawled data as an index file for easy query.

Install Python3.10+
Install Poetry
Install Chrome 119.0+
Run the command poetry install in the directory with pyproject.toml
Configure config.yaml
Change the display language of your Twitter account to English (More>Settings and Support>Account information>Languages>Display language)

poetry run python3 -m twitter_user_tweet_crawler

Name		Name	Last commit message	Last commit date
Latest commit History 45 Commits
.github/workflows		.github/workflows
tests		tests
twitter_user_tweet_crawler		twitter_user_tweet_crawler
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
README_zh_CN.md		README_zh_CN.md
__init__.py		__init__.py
config.yaml		config.yaml
poetry.lock		poetry.lock
pyproject.toml		pyproject.toml
test.py		test.py