Skip to content

refactor: migrate from Tinybird to ClickHouse#38

Draft
pauchiner wants to merge 47 commits into
mainfrom
fix/migrate-tinybird-to-clickhouse
Draft

refactor: migrate from Tinybird to ClickHouse#38
pauchiner wants to merge 47 commits into
mainfrom
fix/migrate-tinybird-to-clickhouse

Conversation

@pauchiner
Copy link
Copy Markdown
Contributor

@pauchiner pauchiner commented Feb 4, 2025 •
edited
Loading

This PR successfully migrates the analytics services from Tinybird to ClickHouse

Reasons

  • Self-Hosting: with clickhouse we can finally self host Formizee with just one button.
  • Pricing: Tinybird updates it's pricing plans in April, making analytics more expensive.

Major Changes

  • internal/metrics and internal/analytics merged.
  • Added the analytics service to the docker compose.
  • http metrics are removed in favor of one source of true.
  • api, vault, dashboard and web now uses clickhouse.
  • internal/cache and internal/keys also use clickhouse.

TO-DO

  • Compile the dev docker-compose without errors.
  • Reconciliate responses on dashboard queries.
  • Update tools/local and development experience.
  • Update documentation.
  • Deploy ClickHouse on production
  • Migrate data from tinybird to ClickHouse

@pauchiner
feat(internal/analytics): add the schema migrations for testing
0bcba54
@pauchiner
chore(internal/analytics): update the tsconfig.json to add paths
4174c34
@pauchiner
feat(internal/analytics): add the Dockerfile for testing purposes
5ed7a57
@pauchiner
fix(internal/analytics): update deps and create the base ch client
5551d1d
@pauchiner
fix(internal/analytics): update the imports and the exports, add the …
6f1a44b 
…testutils
@pauchiner
feat(internal/analytics): add the main analytics pipes
9fc320c
@pauchiner
test(internal/analytics): update the test description
4acaafa
@pauchiner
test(internal/analytics): add tests for billing
fbca79a
@pauchiner
test(internal/analytics): add tests for submissions
e5e41ed
@pauchiner
fix(internal/analytics): update a typo
ed494f4
@pauchiner
chore(internal/analytics): add a vitest.config.ts
a971e2f
@pauchiner
fix(internal/analytics): update the schema comment
b399ae3
@pauchiner
test(internal/analytics): add tests for requests
9db2c69
@pauchiner
test(internal/analytics): add tests for stadistics
f7ac1cd
@pauchiner
test(internal/analytics): add tests for telemetry
c367097
@pauchiner
feat(internal/analytics): add the metrics for cache and database latency
ea6a7bd
@pauchiner
fix(internal/analytics): update cache metrics schema
6b1726b
@pauchiner
fix(internal/cache): migrate cache package to the new analytics
b7b0a7a
@pauchiner
fix(internal/cache): migrate the keys to the new analytics service
e012246
@pauchiner
fix(apps/dashboard): migrate metrics to the new analytics service
c3fb4f8
@pauchiner
fix(internal/analytics): update biome-ignore comments
f0b9013
@pauchiner
fix(internal/analytics): remove the relative paths btw
5210cf2
@pauchiner
fix(apps/dashboard): make some adjustments for the analytics service
2bbf8f9
@pauchiner
fix(internal/analytics): add vault metrics and the time property
4a54470
@pauchiner
refactor(internal/analytics): update the request schema
2feae51
@pauchiner
fix(apps/api): migrate metrics to the new analytics service
eda0a39
@pauchiner
fix(apps/vault): migrate metrics to the new analytics service
9a3c638
@pauchiner
fix(internal): remove metrics package
5151eaf
@pauchiner
chore: update dependencies
e4ba942
@pauchiner
fix(apps/web): migrate to the new analytics service
cc96bf2
@pauchiner pauchiner added the refactor Improvement to the infrastructure label Feb 4, 2025
@pauchiner pauchiner self-assigned this Feb 4, 2025
@vercel
Copy link
Copy Markdown

vercel Bot commented Feb 4, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
webpage ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 20, 2025 11:19am
1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
dashboard ⬜️ Skipped (Inspect) Mar 20, 2025 11:19am

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 4, 2025 •
edited
Loading

Thank you for following the naming conventions for pull request titles! 🙏

@vercel vercel Bot temporarily deployed to Preview – dashboard February 9, 2025 19:05 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@pauchiner pauchiner

Labels

enhancement New feature or request refactor Improvement to the infrastructure

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@pauchiner