This WebScraper API receives a list of seed urls from the client and a desired depth to traverse. This request is then turned into a pending job and written to a PSQL database where it awaits processing by a worker. When a client creates a job, a job id is returned, with this job id a client can view there jobs current status and when completed the results of the job - the pages discovered.

• There can be at maximum 50 seed urls
• There can be at maximum a depth of 10
• Must be http(s) schemes.

1. docker compose -f docker-compose.yml -f docker-compose.dev.yml build --no-cache
2. docker compose -f docker-compose.yml -f docker-compose.dev.yml up
3. Open the WebScraperWorker project and follow local running instructions (if not already done)

python3 -m pytest

1. aws ecr get-login-password --region us-west-1 | docker login --username AWS --password-stdin 657083456388.dkr.ecr.us-west-1.amazonaws.com
2. docker build -t webscraper-api .
3. docker tag webscraper-api:latest 657083456388.dkr.ecr.us-west-1.amazonaws.com/webscraper/api:latest
4. docker push 657083456388.dkr.ecr.us-west-1.amazonaws.com/webscraper/api:latest

docker exec -it job-db -U jobs -d jobs

Upon first setup ensure api is running locally then run alembic upgrade head

If the schema for the DB is changed ensure that the following commands are ran:

• alembic revision --autogenerate -m "<what changed>"
• alembic upgrade head

Ensure API container is running and run the following: docker exec -t job-db pg_dump --schema-only --no-owner --no-privileges -U jobs jobs > schema.sql

Place output in WebScraperWorker at src/test/resources/schema.sql

Updated the requirements.txt by using the following command pip freeze > requirements.txt

python generate_openapi.py