Skip to content

LFound/concurrent-worker-queue

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Concurrent Worker Queue (Go)

A Go concurrency project demonstrating bounded queues, worker pools, backpressure, and load-dependent acceptance behaviour.

This project implements a concurrent job processing system using goroutines and channels, then explores how submission rate affects system saturation and rejection behaviour.


Overview

Many real-world backend systems must process incoming work concurrently while preventing overload.

This project models:

  • Bounded job queues
  • Concurrent worker pools
  • Backpressure under load
  • Rejection when capacity is exceeded
  • Load-driven behavioural changes

It demonstrates how concurrency primitives in Go can be used to build safe, scalable worker systems.


System Design

Core Components

  • Bounded Queue

    • Implemented using buffered channels
    • Rejects new work when full
  • Worker Pool

    • Multiple goroutines process jobs concurrently
    • Simulated processing delay
  • Metrics

    • Thread-safe counters using mutexes
    • Tracks accepted, rejected, and processed jobs

Experiments

The project runs controlled experiments varying job submission rate while keeping:

  • Queue size constant
  • Worker count constant
  • Processing time constant

This reveals the system’s saturation threshold.


Example Findings

Submission Delay Behaviour
Slow All jobs accepted, workers keep up
Medium Still stable, minimal queue pressure
Fast Queue fills, job rejection begins

This demonstrates how bounded systems behave under increasing pressure.


Example Plots

Accepted Jobs vs Submission Delay

Accepted Jobs

Rejected Jobs vs Submission Delay

Rejected Jobs


Project Structure

concurrent-worker-queue/
├── cmd/
│ └── main.go
├── internal/
│ ├── queue.go
│ ├── worker.go
│ └── metrics.go
├── scripts/
│ └── plot_results.py
├── results/
├── go.mod
└── README.md

Quick Start

Run the experiment

go run ./cmd

Results will be saved to:

results/experiment_results.csv

Generate plots

python scripts/plot_results.py

Plots will be saved to the results/ folder.


Purpose

This project is part of a systems engineering portfolio exploring:

  • Concurrent system design
  • Backpressure and overload protection
  • Capacity limits under load
  • Practical Go concurrency patterns

It is designed as a learning and experimental framework rather than a production service.


License

MIT

About

Go concurrency project demonstrating bounded queues, worker pools, backpressure and load-dependent acceptance behaviour.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors