Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
7121db7
ES to postgres : readme and talk service
rattling Jul 12, 2025
c641921
ES to Postgres : db changes
rattling Jul 12, 2025
3b62727
ES to Postgre: engine changes
rattling Jul 12, 2025
3af1111
ES to Postgres: routers updates etc
rattling Jul 12, 2025
3a852f1
ES to Postgres: packages, scripts, tests
rattling Jul 12, 2025
6bf4199
move db tests from sqlite to postgres to test authentic search behaviour
rattling Jul 13, 2025
f6be274
clean architectural seperation with dependency injection
rattling Jul 13, 2025
7250af0
move to package rather than relative imports
rattling Jul 13, 2025
19f11c9
taxonomy phase 1 - API infra
rattling Jul 13, 2025
88aef50
taxonomy phase 2 - analytics and bulk ops
rattling Jul 13, 2025
cd285ba
testing and debug taxonomy backend
rattling Jul 13, 2025
5a52b9b
taxonomy - frontend draft
rattling Jul 13, 2025
5f07bac
add dockerfiles for front/back
rattling Jul 20, 2025
d0675cf
incremental data load for new/updated talks
rattling Jul 20, 2025
4b188b9
data delta fixes and architecture md
rattling Jul 20, 2025
9e54cd7
update clean architecture - phase 1 done
rattling Jul 20, 2025
b3c03b8
clean arch migration - phase 12 complete
rattling Jul 20, 2025
9149d61
update arch doc and add explore scripts
rattling Jul 20, 2025
5240c06
add backend as a 1st part package through pip (not pipenv)
rattling Jul 20, 2025
5116a62
added more clean arch exploration scripts
rattling Jul 20, 2025
2db4a50
updated tests to new arch
rattling Jul 20, 2025
c5e522f
DX: Pipenv test task, Makefile targets, pytest.ini, README quickstart…
rattling Aug 9, 2025
98e0aa7
Backend: FastAPI app, routers, services, SQLAlchemy models/repo, Alem…
rattling Aug 9, 2025
b461878
Tests: API, repo, and tagging tests; exclude archived tests from disc…
rattling Aug 9, 2025
7dd2335
Archive: move previous CLEAN_ARCH implementation and assets under arc…
rattling Aug 9, 2025
58c0228
Ingest: simplify engine to be domain-free; update data_pipeline for n…
rattling Aug 9, 2025
57d9cd5
Cleanup: remove legacy CLEAN_ARCH files, docker-compose, scripts, and…
rattling Aug 9, 2025
dddadcd
Docs: add HANDOFF.md and ARCHITECTURE.md; link from README
rattling Sep 14, 2025
940b0b3
DX: standardize make test to use pipenv pytest (no -q)
rattling Sep 14, 2025
cb4028e
Cleanup: remove obsolete README_NEW.md
rattling Sep 14, 2025
71c7a95
Repo cleanup: remove stray untracked test and legacy files; keep arch…
rattling Sep 14, 2025
703adfa
Revert "Repo cleanup: remove stray untracked test and legacy files; k…
rattling Sep 14, 2025
892d884
Merge remote-tracking branch 'origin/main' into simplify-arch
rattling Sep 14, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
backend/venv/
frontend/node_modules/
frontend/.env.development
app.db

# Byte-compiled / optimized / DLL files
__pycache__/
Expand Down Expand Up @@ -164,3 +165,6 @@ cython_debug/
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/

# Local Developer Stuff
jc_notes.txt
7 changes: 6 additions & 1 deletion .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
{
"python.defaultInterpreterPath": "/home/jcurry/.local/share/virtualenvs/python_ireland_talk_database-Lo83MkTA/bin/python",
"python.terminal.activateEnvironment": true,
"python.terminal.activateEnvInCurrentTerminal": true
"python.terminal.activateEnvInCurrentTerminal": true,
"python.testing.pytestArgs": [
"tests"
],
"python.testing.unittestEnabled": false,
"python.testing.pytestEnabled": true
}
20 changes: 20 additions & 0 deletions .vscode/tasks.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"version": "2.0.0",
"tasks": [
{
"type": "shell",
"label": "pytest -q",
"command": "pipenv",
"group": "build",
"isBackground": false,
"args": [
"run",
"pytest",
"-q"
],
"problemMatcher": [
"$python"
]
}
]
}
113 changes: 113 additions & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# Architecture Overview

This project uses a pragmatic layered design: thin HTTP routers (FastAPI), small application services, and a SQLAlchemy repository. Storage defaults to SQLite, with optional PostgreSQL for better full‑text search.

## System at a glance

```mermaid
flowchart TD
A[Frontend] -->|HTTP| B[FastAPI API]
B --> C[Services]
C --> D[SQLAlchemy Repo]
D --> E[(Database)]
E --> F[SQLite]
E --> G[PostgreSQL]
H[Ingestion] --> C
I[Sessionize + Meetup] --> H
J[Analytics] --> D
K[Logging] -.-> B
```

## Layers

- API (FastAPI): request validation, routing, response shape. See `backend/api/routers/*`.
- Services: light orchestration, auto‑tagging, resolving taxonomy name→ids. See `backend/app/service.py`.
- Repository: SQLAlchemy data access, search, analytics, ingest status. See `backend/db/repo.py`.
- Models: ORM entities and association table. See `backend/db/models.py`.

## Storage model

```mermaid
erDiagram
TALKS {
string id PK
string talk_type
string title
text description
json speaker_names
string source_url
json type_specific_data
text search_vector
json auto_tags
datetime created_at
datetime updated_at
}
TAXONOMIES {
int id PK
string name
text description
string created_by
boolean is_system
datetime created_at
}
TAXONOMY_VALUES {
int id PK
int taxonomy_id FK
string value
text description
string color
datetime created_at
}
TALK_TAXONOMY_VALUES {
string talk_id FK
int taxonomy_value_id FK
}
SYNC_STATUS {
int id PK
string source_type
datetime last_sync_time
datetime last_successful_sync
int sync_count
int error_count
text last_error
datetime created_at
datetime updated_at
}
TALKS ||--o{ TALK_TAXONOMY_VALUES : uses
TAXONOMY_VALUES ||--o{ TALK_TAXONOMY_VALUES : tags
TAXONOMIES ||--o{ TAXONOMY_VALUES : has
```

Notes

- Talks to TaxonomyValues is many‑to‑many via `talk_taxonomy_values`.
- A uniqueness constraint exists for `(source_type, source_id)` to upsert from providers.

## Search behavior

- When using PostgreSQL, search leverages FTS: `to_tsvector` + `plainto_tsquery` over the `search_vector` column.
- On SQLite, search falls back to case‑insensitive LIKE across `search_vector`, `title`, and `description`.
- The repo rebuilds `search_vector` from key fields on create/update. Ranking on PG can be added with `ts_rank` later.

## Ingestion

- Providers: Sessionize (HTML scrape) and Meetup (GraphQL/HTTP). They produce talk payloads that the service saves via the repo.
- Full ingest runs for both providers; incremental sync currently reuses full. `sync_status` tracks per‑source runs/error state.
- Network can be disabled with `INGEST_DISABLE_NETWORK` to make CI deterministic.

## Technologies

- Python 3.11, FastAPI, Uvicorn
- SQLAlchemy 2.x, Alembic
- SQLite (default) or PostgreSQL (recommended for FTS)
- Vite/React frontend (scaffolded)
- Requests/BeautifulSoup (ingestion)

## Environments

- Default: SQLite `app.db` (no setup). For Postgres, set `DATABASE_URL` and run `make migrate`.
- Logging: `LOG_LEVEL` (INFO default) with X‑Request‑ID propagation in responses.

## Testing

- Tests run on SQLite via Pipenv/pytest. Legacy tests under `archive/` are excluded. See `pytest.ini`.
138 changes: 138 additions & 0 deletions HANDOFF.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# Project Handoff: Python Ireland Talk Database

Concise status and next steps for new maintainers. See README.md for quickstart; this doc summarizes what’s working and where to pick up.

---

## TL;DR – Current State

- Backend API (FastAPI) is stable and tested.
- Talks: list/search (+advanced), get by id, create, list talk types.
- Taxonomies & tagging: CRUD for taxonomies/values; add/replace/remove tags on talks; get a talk’s tags.
- Analytics: taxonomy usage and popular tags.
- Ingestion: Sessionize and Meetup full ingest; incremental sync currently reuses full; SyncStatus tracked per source.
- Logging: structured request logs with X-Request-ID.
- Data & search
- SQLite by default (app.db). Postgres supported via DATABASE_URL.
- Search uses Postgres FTS when on PG; falls back to LIKE on SQLite.
- Alembic migrations in place and PG-safe (GIN index applied only on PG).
- Tests
- Test suite is green locally (SQLite). Legacy/archived tests excluded from discovery.
- Frontend
- Vite/React scaffold exists but not wired to new API yet. Temporary alias routes for tagging kept to ease FE migration.

---

## How to run

- Prereqs: Python 3.11+, Pipenv; Node 20+ for FE; Postgres optional.
- Dev API: `make dev` (http://localhost:8000, docs at /docs)
- Tests: `make test`
- Migrate (if using Postgres): set DATABASE_URL then `make migrate`
- Ingest sample data (API running): `make ingest`
- Frontend (not wired yet): `make frontend`

Environment:

- DATABASE_URL (optional, for Postgres)
- LOG_LEVEL (INFO default)
- INGEST_DISABLE_NETWORK (set to disable external calls during CI/local)

---

## What’s solid vs. pending

Solid

- Core CRUD/search APIs for talks, taxonomies, tags.
- Ingestion job runs end-to-end for Sessionize/Meetup; errors recorded in SyncStatus without failing the job.
- Repo layer supports advanced search by taxonomy (ANY/ALL) and Postgres FTS.

Pending/Partial

- Frontend pages and wiring to the new API.
- Search relevance on PG (no ranking/weights yet).
- Ingestion “delta” logic (sync currently equals full).
- Optional auth/admin flows for managing taxonomies.

---

## Suggested next steps (prioritized)

1. Frontend integration

- Point FE at http://localhost:8000/api/v1; wire talks list/detail/search.
- Hook tagging UI to tag endpoints; then remove temporary alias routes.

2. API contract tooling

- Generate TS types/client from OpenAPI (e.g., openapi-typescript or Orval) and add a Makefile target.

3. Search & data quality

- Add PG ranking (ts_rank + weights); keep SQLite LIKE fallback.
- Improve dedupe/merge across sources; consider stricter uniqueness where safe.

4. Ingestion polish

- Track last-seen IDs/timestamps for incremental sync per source.
- Make provider config explicit; document tokens if needed.

5. CI

- Add GH Actions: pipenv install + tests, optional OpenAPI client generation, and migration smoke test.

---

## Ownership notes

- Active branch: `simplify-arch` (consider merging to default once FE is wired).
- Logging is configured; X-Request-ID is included on responses.
- SQLite used for local/tests; Postgres recommended in prod for FTS performance.

---

## Appendix A – Endpoint map (brief)

Base: `/api/v1`

Talks

- GET `/talks` – list/search (q, talk_type, talk_types, limit, offset)
- GET `/talks/search` – same as list with q
- GET `/talks/search/advanced` – q, talk*types, taxonomy_value_ids, taxonomy*<Name>=Value, match=any|all
- GET `/talks/types` – list distinct talk types
- GET `/talks/{id}` – talk detail
- POST `/talks` – create talk

Taxonomies & tagging (prefix `/talks`)

- GET `/taxonomies` | POST `/taxonomies` | PUT `/taxonomies/{taxonomy_id}` | DELETE `/taxonomies/{taxonomy_id}`
- POST `/taxonomies/{taxonomy_id}/values` | PUT `/taxonomy-values/{value_id}` | DELETE `/taxonomy-values/{value_id}`
- GET `/{talk_id}/tags` | POST `/{talk_id}/tags/add` | PUT `/{talk_id}/tags` | DELETE `/{talk_id}/tags/{value_id}`
- Temporary aliases exist under `/talks/{talk_id}/tags*` (remove after FE updates).

Analytics

- GET `/talks/analytics/taxonomy-usage`
- GET `/talks/analytics/taxonomies/{taxonomy_id}/usage`
- GET `/talks/analytics/popular-tags?limit=`

Ingestion

- POST `/ingest/full` | POST `/ingest/sync` | GET `/ingest/status`

---

## Appendix B – Operations notes

- Migrations: Alembic configured for SQLite by default; PG-safe with conditional GIN index on `talks.search_vector`.
- Search: PG uses `to_tsvector`/`plainto_tsquery`; SQLite falls back to case-insensitive LIKE.
- Sync status: each source has a single row with counters, last error, and timestamps.

---

## Appendix C – Testing

- Run: `make test` (uses Pipenv)
- Tests run on SQLite; legacy archived tests are excluded via `pytest.ini`.
22 changes: 22 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
.PHONY: dev test migrate ingest frontend

# Start the FastAPI backend (http://localhost:8000)
dev:
@echo "Starting API on http://localhost:8000"
pipenv run uvicorn backend.main:app --reload --port 8000

# Run test suite
test:
pipenv run pytest

# Apply database migrations (uses DATABASE_URL if set)
migrate:
pipenv run alembic upgrade head

# Trigger a full ingest via API (backend must be running)
ingest:
curl -s -X POST http://localhost:8000/api/v1/ingest/full | python -m json.tool

# Start the frontend dev server (http://localhost:5173)
frontend:
cd frontend && npm install && npm run dev
2 changes: 1 addition & 1 deletion Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ pydantic-settings = "*"
passlib = {extras = ["bcrypt"], version = "*"}
python-jose = {extras = ["cryptography"], version = "*"}
python-multipart = "*"
elasticsearch = "==7.17.9"
beautifulsoup4 = "*"

[dev-packages]
Expand All @@ -25,6 +24,7 @@ pytest-asyncio = "*"
black = "*"
isort = "*"
flake8 = "*"
ipykernel = "*"

[requires]
python_version = "3.11"
Loading