🌾 AgriPulse - Agriculture Crop Yield & Weather Data Engineering System

AgriPulse is an end-to-end Data Engineering project that demonstrates how to ingest, clean, and analyze agricultural crop yield and weather data using Python, SQL, and Airflow.

This project is designed to showcase real-world data engineering skills from data ingestion to final SQL analytics and warehousing.

To understand the relationship between crop yield and weather data (rainfall, temperature) across different regions using a data pipeline.

Raw Data (CSV) ➡️ Python (Ingestion & Cleaning)
          ➡️ SQL Database ➡️ SQL Analytics ➡️ Output CSV

• Python (Pandas)
• SQL (MySQL/PostgreSQL)
• Apache Airflow (for DAG Simulation)
• Git & GitHub

AgriPulse-DataEngineering/
│
├── README.md
├── data/
│   ├── raw/
│   │   ├── crop_production.csv
│   │   └── weather_data.csv
│   └── processed/
│       ├── clean_crop_data.csv
│       └── clean_weather_data.csv
│
├── scripts/
│   ├── data_ingestion.py
│   ├── data_cleaning.py
│   ├── data_transformation.py
│   └── database_load.py
│
├── sql/
│   ├── create_tables.sql
│   ├── insert_data.sql
│   └── analytics_queries.sql
│
├── airflow_dags/
│   └── agri_data_pipeline_dag.py
│
└── output/
    └── final_aggregated_data.csv

1. Place raw CSVs in data/raw/
2. Run scripts in order from scripts/
3. Load cleaned data into SQL using sql/create_tables.sql
4. Run analytics queries from sql/analytics_queries.sql

SELECT district, AVG(rainfall), AVG(yield)
FROM crops
JOIN weather USING (district, year)
GROUP BY district;

• Built an end-to-end data pipeline for crop vs weather analysis using Python, SQL, and Airflow
• Implemented data cleaning and transformation scripts
• Designed database schema and analytical queries to derive agricultural insights