Traditional Mode (Linux Kernel Networking)
┌─────────────────┐
│   Application    │
├─────────────────┤
│ syscall (context switch)
├─────────────────┤
│  Kernel Network  │ ← All applications share one stack
│     Stack        │
├─────────────────┤
│    NIC Driver    │
└─────────────────┘

F-Stack Mode (User-Space Networking)
┌─────────────────────────────────────────┐
│ Process1 (core0)  Process2 (core1)  Process3 (core2) │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐
│  │  App       │  │  App       │  │  App       │
│  │ FreeBSD    │  │ FreeBSD    │  │ FreeBSD    │
│  │  Stack     │  │  Stack     │  │  Stack     │
│  │  (local)   │  │  (local)   │  │  (local)   │
│  └────────────┘  └────────────┘  └────────────┘
│        ↓               ↓               ↓
│    DPDK Polling Loop  (no interrupts, no syscalls)
└─────────────────────────────────────────┘
         ↓
    ┌─────────────┐
    │  NIC Hardware│ (RSS hardware classification)
    └─────────────┘

/data/workspace/f-stack/
│
├── lib/                                    # F-Stack core library (~21K lines of C code)
│   ├── ff_dpdk_if.c          (2907 lines) # ⭐ Most critical: DPDK/NIC driver
│   ├── ff_glue.c             (1467 lines) # Kernel emulation layer
│   ├── ff_config.c           (1694 lines) # Configuration parsing
│   ├── ff_syscall_wrapper.c  (2265 lines) # Linux↔FreeBSD adaptation (+ R9 kqueue coexist + IPV6_V6ONLY + R10 readv/writev/ioctl/dup/dup2 kernel-fd routing)
│   ├── ff_init.c             (69 lines)   # Initialization coordination
│   ├── ff_epoll.c            (289 lines)  # Epoll compat (unified F-Stack+kernel)
│   ├── ff_host_interface.c   (617 lines)  # Host OS iface + FF_KERNEL_COEXIST bridges (32 ff_host_*)
│   ├── ff_dpdk_kni.c                      # Virtual NIC support
│   ├── ff_*.h                             # API and data structure definitions
│   └── Makefile              (765 lines)  # Build system
│
├── freebsd/                                # FreeBSD 15.0 kernel port (upgraded from 13.0 in 2025-2026)
│   ├── sys/
│   │   ├── netinet/          # IPv4 protocol stack (TCP/UDP/IP/ICMP)
│   │   ├── netinet6/         # IPv6 protocol stack
│   │   ├── net/              # Generic network interfaces
│   │   ├── kern/             # Kernel services (malloc/mutex/synch)
│   │   └── vm/               # Virtual memory management (mbuf)
│   ├── amd64/                # x86 architecture code
│   └── contrib/ck/           # ConcurrencyKit atomic operations
│
├── dpdk/                                   # DPDK 24.11.6 LTS dependency (submodule)
│   ├── lib/
│   │   ├── eal/              # Environment Abstraction Layer
│   │   ├── ethdev/           # NIC generic interface
│   │   ├── mempool/          # Memory pool
│   │   └── ring/             # Lock-free queue
│   └── drivers/
│       └── net/              # Vendor NIC drivers
│
├── mk/                                     # Build system
│   ├── kern.pre.mk                        # FreeBSD build rules
│   ├── kern.mk
│   └── compiler.mk           # Compiler configuration
│
├── app/                                    # Application integration examples
│   ├── nginx-1.28.0/         # Nginx integration
│   └── redis-6.2.6/          # Redis integration
│
├── example/                                # Development examples
│   ├── main.c                # kqueue mode (recommended)
│   └── main_epoll.c          # epoll mode
│
├── tools/                                  # Operations tools
│   ├── top/                  # CPU statistics
│   ├── sysctl/               # Parameter management
│   ├── ifconfig/             # NIC configuration
│   ├── route/                # Route management
│   ├── netstat/              # Network statistics
│   ├── arp/                  # ARP table management
│   ├── ipfw/                 # Firewall management
│   ├── knictl/               # KNI control
│   ├── traffic/              # Traffic statistics
│   ├── ndp/                  # IPv6 Neighbor Discovery
│   ├── ngctl/                # Netgraph control
│   └── compat/ff_ipc.*       # IPC communication library
│
├── adapter/                                # Network adapters
│   ├── micro_thread/             # Micro-thread interface for stateful applications using F-Stack
│   └── syscall/                  # Builds libff_syscall.so + fstack instance binary; intercepts
│                                  # Linux syscalls via LD_PRELOAD and forwards them to the fstack
│                                  # instance through Hugepage shared memory (sem path or
│                                  # FF_USE_RING_IPC lock-free ring path). See adapter/syscall/README.md
├── doc/                                    # Original English documentation
├── docs/                                   # Three-layer architecture knowledge base docs
├── config.ini                # Default configuration file
└── start.sh                  # Multi-process startup script

Application Layer
  ↓
┌─────────────────────────────────────────────┐
│  FF API Layer (ff_api.h)                     │
│  - socket/bind/listen/accept/connect        │
│  - read/write/send/recv                     │
│  - kqueue/kevent/epoll/select               │
└──────────────┬──────────────────────────────┘
               ↓
        ┌──────┴──────┐
        ↓             ↓
   FreeBSD Stack  Epoll Compat Layer
   TCP/UDP/IP     (ff_epoll.c)
   │              │
   └──────┬───────┘
          ↓
   ┌─────────────────────────────┐
   │  Glue Layer (ff_glue.c)     │
   │  - Lock/condition var emulation │
   │  - Memory management         │
   │  - Timer/soft interrupt       │
   └──────────────┬──────────────┘
                  ↓
      ┌───────────┴───────────┐
      ↓                       ↓
   Config System      Linux Compat Layer
  (ff_config.c)  (ff_syscall_wrapper.c)
      │                       │
      └───────────┬───────────┘
                  ↓
        ┌──────────────────────┐
        │  DPDK Library        │
        │  - EAL               │
        │  - Mempool/Ring      │
        │  - Ethdev            │
        └──────────────┬───────┘
                       ↓
        ┌──────────────────────┐
        │  PMD Driver + NIC HW │
        └──────────────────────┘

┌──────────────────────────────────────────────────────┐
│                    Application Layer                   │
│              (Nginx/Redis/Custom)                     │
└──────────────────────┬───────────────────────────────┘
                       │ FF API (80 export symbols)
┌──────────────────────▼───────────────────────────────┐
│              F-Stack Library (libfstack.a)            │
│  ├─ Socket API       (ff_socket/bind/listen/...)    │
│  ├─ I/O API          (ff_read/write/send/recv/...)  │
│  ├─ Event API        (ff_kqueue/ff_epoll/...)       │
│  └─ Route API        (ff_route_ctl/...)             │
└──────────┬─────────────────────────────────┬────────┘
           │                                  │
    ┌──────▼──────────┐          ┌───────────▼──────┐
    │ FreeBSD TCP/IP  │          │ Epoll Compat     │
    │ Protocol Stack  │          │ Layer            │
    │ ├─ TCP/UDP      │          │ (ff_epoll.c)     │
    │ ├─ IPv4/IPv6    │          │ Linux→kqueue     │
    │ ├─ ICMP/IGMP    │          │ conversion       │
    │ ├─ ARP          │          └──────────────────┘
    │ └─ Routing      │
    └────────┬────────┘
             │
    ┌────────▼──────────────────────────────────────┐
    │        Glue Layer (ff_glue.c 1467 lines)      │
    │ Kernel API User-Space Emulation                │
    │ ├─ Mutex/RWLock    (pthread_mutex_t)         │
    │ ├─ CondVar         (pthread_cond_t)          │
    │ ├─ malloc/free     (rte_malloc)              │
    │ ├─ callout/timer   (rte_timer)               │
    │ ├─ taskqueue       (soft interrupt emulation) │
    │ └─ Global vars     (ticks/vm_cnt/...)        │
    └────────┬──────────────────────────────────────┘
             │
    ┌────────▼──────────────────────────────────────┐
    │         DPDK Library (libdpdk.a)               │
    │                                                │
    │  EAL (Environment Abstraction Layer)           │
    │  ├─ Hugepage memory allocation                 │
    │  ├─ NUMA affinity                              │
    │  ├─ CPU core isolation and binding             │
    │  └─ Multi-process support (Primary/Secondary)  │
    │                                                │
    │  Mempool (High-efficiency memory pool)         │
    │  ├─ mbuf pre-allocation                        │
    │  ├─ Lock-free allocation/deallocation          │
    │  └─ Memory pool warm-up (avoid runtime alloc)  │
    │                                                │
    │  Ethdev (NIC generic interface)                │
    │  ├─ rte_eth_rx_burst()                        │
    │  ├─ rte_eth_tx_burst()                        │
    │  ├─ RSS configuration                          │
    │  └─ Hardware offload settings                  │
    │                                                │
    │  Ring (Lock-free queue)                        │
    │  ├─ Inter-process IPC message passing          │
    │  └─ Multi-process communication                │
    └────────┬──────────────────────────────────────┘
             │
    ┌────────▼──────────────────────────────────────┐
    │  PMD Driver (Poll Mode Driver)                │
    │  ├─ igb_uio kernel module (VF passthrough)    │
    │  ├─ vfio-pci (more secure)                    │
    │  └─ Vendor drivers (Intel i40e/ixgbe/ice)     │
    └────────┬──────────────────────────────────────┘
             │
    ┌────────▼──────────────────────────────────────┐
    │      NIC Hardware (Network Interface Card)     │
    │                                                │
    │  Hardware feature support:                     │
    │  ├─ RSS (Receive Side Scaling) - 5-tuple hash to RX queues │
    │  ├─ TSO (TCP Segmentation Offload)            │
    │  ├─ LSO (Large Send Offload)                  │
    │  ├─ RX/TX Checksum Offload                    │
    │  ├─ LRO (Large Receive Offload)               │
    │  └─ VLAN offload                              │
    │                                                │
    │  Communication method:                         │
    │  ├─ Interrupt-free polling (PMD)               │
    │  ├─ DMA direct memory access                   │
    │  ├─ Zero-Copy mbuf passing                     │
    │  └─ Optional: interrupt-driven + coalesce      │
    └────────────────────────────────────────────────┘

NIC Hardware
  ↓ [RSS processor computes hash based on 5-tuple (SIP,DIP,Sport,Dport,Proto)]
RX Queue Set (corresponding to different CPU cores)
  ↓ [Each core polls its own RX queue]
Poll Mode Driver
  ↓ rte_eth_rx_burst(port, queue, &mbufs, burst_size)
DPDK mbuf Buffer (Zero-Copy pointer)
  ↓
process_packets() in ff_dpdk_if.c
  ├─ Extract L2/L3/L4 headers
  ├─ Protocol filtering (ARP/IPv4/IPv6/Multicast)
  ├─ Optional: packet dispatch callback
  └─ Call FreeBSD protocol stack entry
  ↓
FreeBSD Network Stack
  ├─ eth_input()          [Ethernet processing]
  ├─ ip_input()           [IP layer]
  ├─ tcp_input()/udp_input() [L4 layer]
  └─ sorecvX()            [Socket receive buffer]
  ↓
Application retrieves data via ff_read()/ff_recv()/ff_recvfrom()/ff_recvmsg()

Application calls ff_write()/ff_send()/ff_sendto()/ff_sendmsg()
  ↓
FreeBSD TCP/UDP Protocol Stack
  ├─ tcp_output()      [TCP segmentation/sequencing]
  ├─ ip_output()       [IP addressing]
  └─ if_output()       [NIC output]
  ↓
if_start() in ff_glue.c
  ├─ Get mbuf from pending send queue
  ├─ Fill L2/L3/L4 headers
  ├─ Configure hardware offload options (e.g., TSO/Checksum)
  └─ Call send_single_packet()
  ↓
DPDK TX Queue
  ├─ rte_eth_tx_burst() [batch send]
  └─ Periodic drain (avoid packet delay)
  ↓
PMD Driver
  ↓ [DMA to NIC hardware]
NIC Hardware
  ├─ Execute TSO (if needed)
  ├─ Checksum computation
  └─ Ethernet transmission

Question: Why not multi-thread?
━━━━━━━━━━━━━━━━━━━━━━━
1. FreeBSD protocol stack is not thread-safe
   - Protocol stack designed for single-thread execution
   - Numerous global variables and static data structures
   - Heavy locking needed → performance degradation

2. High context switching overhead
   - CPU cache pollution
   - TLB flush
   - Far inferior to single-thread polling

3. Shared memory complexity
   - Socket data structures shared across threads
   - Race conditions difficult to debug

Answer: Multi-process architecture
━━━━━━━━━━━━━━━━━━━━━━━
Core features:
✓ Each process has an independent FreeBSD protocol stack instance
✓ Single-thread polling, no contention, no locking
✓ Connection affinity via DPDK RSS classification
✓ Zero shared state between processes (except mempool/NIC)
✓ One process crash does not affect other processes

Machine Resources:
├─ CPU: 8 cores
├─ NIC: 10GbE
└─ Memory: 16GB huge pages

F-Stack Deployment:
┌────────────────────────────────────────────────────┐
│  Primary Process - CPU 0                            │
│  ├─ DPDK EAL initialization                        │
│  ├─ Create shared hugepage/mempool                 │
│  ├─ Start all secondary processes                  │
│  └─ Listen for IPC messages from tools             │
│                                                     │
│  Secondary 1 - CPU 1         Secondary 2 - CPU 2  │
│  ┌──────────────────┐         ┌──────────────────┐
│  │ FreeBSD Stack     │         │ FreeBSD Stack     │
│  │ Instance          │         │ Instance          │
│  │ RX/TX Queue Map   │         │ RX/TX Queue Map   │
│  │ Independent Poll  │  ←RSS→  │ Independent Poll  │
│  └──────────────────┘         └──────────────────┘
│
│  Secondary 3 - CPU 3          ...Secondary N - CPU N
│  ...same structure              ...same structure
│
└────────────────────────────────────────────────────┘
         ↑                          ↑
         │ Shared                   │ Shared
    ┌────┴────────────────────────┴───┐
    │ DPDK Shared Resources            │
    │ ├─ Mempool (lock-free access)   │
    │ ├─ Ring (IPC message passing)   │
    │ ├─ Virtual NIC KNI (optional)   │
    │ └─ RSS Classification Table     │
    └────────┬────────────────────────┘
             │
    ┌────────▼────────────────────────┐
    │ NIC Hardware (single 10GbE)     │
    │ ├─ RX Queue 0 → Process 1      │
    │ ├─ RX Queue 1 → Process 2      │
    │ ├─ RX Queue 2 → Process 3      │
    │ └─ ...                          │
    └─────────────────────────────────┘

NIC Hardware RSS Computation:
┌─────────────────────────────────┐
│ Received packet header           │
├─────────────────────────────────┤
│ hash(SIP, DIP, Sport, Dport) % N│  ← 5-tuple hash
├─────────────────────────────────┤
│ Result = RX queue number (0..N-1)│
└─────────────────────────────────┘
         ↓
   ┌─────────────────────┐
   │ RX Queue Q           │
   │ ↓ (process bound to core) │
   │ Process P handles    │
   └─────────────────────┘

Effect:
✓ TCP connection {192.168.1.1:8000 ↔ 10.0.0.1:80}
✓ All packets (including ACK/DATA) go to queue 2
✓ Process 2 handles this connection independently
✓ No cross-core synchronization needed
✓ Completely avoids TCP reordering and cache pollution

bash start.sh  (startup script)
  ├─ Calculate number of enabled cores in lcore_mask
  ├─ Set proc_id environment variable = 0
  ├─ Set proc_type = primary
  └─ Start primary process
       ↓
       ff_init(argc, argv)
       ├─ ff_load_config()              # Load config.ini
       ├─ ff_dpdk_init()                # DPDK EAL initialization
       │  ├─ rte_eal_init()             # Initialize EAL
       │  ├─ rte_mempool_create()       # Create mbuf memory pool
       │  ├─ rte_eth_dev_configure()    # Configure NIC
       │  ├─ rte_eth_rx_queue_setup()   # Create RX queues
       │  ├─ rte_eth_tx_queue_setup()   # Create TX queues
       │  └─ rte_eth_dev_start()        # Start NIC
       │
       ├─ ff_freebsd_init()             # Initialize FreeBSD protocol stack
       │  ├─ ff_glue_init()             # Initialize glue layer
       │  ├─ init_network_stack()       # Initialize network stack
       │  └─ init_route_table()         # Initialize routing table
       │
       ├─ ff_dpdk_if_up()               # Bring up NIC
       └─ ff_run(loop_func, arg)        # Enter polling loop
            ↓
            [Primary process executes loop_func callback while handling IPC]
            
       # Secondary process startup (parallelized by startup script)
       proc_id = 1, proc_type = secondary
       ff_init(argc, argv)              # Same initialization
       └─ ff_run(loop_func, arg)        # Enter polling loop

Inter-Process Communication (IPC):
    ff_ipc_init()                       # Connect to primary process EAL
    ff_ipc_send(msg)                    # Send control message
    ff_ipc_recv(msg, type)              # Receive response
    
    [RPC-style communication via DPDK Ring]

Option A: Port FreeBSD Protocol Stack (F-Stack's choice)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Pros:
✓ Mature code (20+ years of optimization) with good clarity
✓ Feature-complete (TCP/UDP/ICMP/IGMP/IPv6)
✓ High RFC compliance
✓ Already has BBR/RACK/DCTCP algorithms
✓ Short development cycle (6-12 months)

Cons:
✗ Large migration effort (glue layer needed)
✗ Complex debugging (kernel API emulation)
✗ Maintenance cost (tracking upstream changes)

Option B: Custom Network Stack
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Pros:
✓ Highly customizable
✓ No external dependencies
✓ Autonomous maintenance

Cons:
✗ 2-3 year development cycle
✗ Many functional defects (non-RFC-compliant)
✗ Unknown performance (lack of validation)
✗ High protocol upgrade costs
✗ No modern algorithm support (BBR/RACK)

Option C: Use Linux Kernel Stack (Kernel Bypass)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✗ Does not meet requirements (core goal is to bypass kernel)

Porting to user-space pros:
✓ Fast iteration, early support for new features
✓ Generally slightly higher performance than FreeBSD

Porting to user-space cons:
✗ More complex code, less clear than FreeBSD
✗ Fast iteration means high effort to track community updates

Application Layer
  ↓ Business data
F-Stack (User-space)
  ├─ Main business: Handle HTTP/DNS, etc.
  └─ Side path: Special addresses, management traffic
       ↓ (optional)
    Virtual NIC veth0 (kernel)
       ↓ Executable in kernel
    ├─ tc (traffic control)
    ├─ iptables (firewall)
    ├─ Monitoring tools
    └─ Integration with other systems (tunnels, etc.)

Traditional approach:
NIC → kernel buffer → application buffer (2 copies)

F-Stack approach:
NIC → DPDK mbuf → FreeBSD mbuf (0 copies, pointer passing only) → application
# [Note] Zero-copy from mbuf to application via socket interface is not yet supported;
# the separate zero-copy API ff_zc_mbuf_read() is planned for future implementation

Single processing:
for i in 1..1M {
    process_packet(pkt[i])
}

Batch processing:
nb_rx = rte_eth_rx_burst(..., 32)  // Receive 32 at once
for i in 0..nb_rx-1 {
    process_packet(pkt[i])
}

Effect: Reduces function call overhead, improves CPU cache hit rate

At DPDK EAL startup:
  ├─ Reserve CPU cores for each lcore
  ├─ Disable CPU frequency scaling
  ├─ Disable CPU migration
  └─ Optimization: Ensure TLB, L1/L2/L3 cache hits

Result: Avoids cache pollution, reduces context switches

4KB small pages:
  │ Virtual address space │  TLB (64 entries) │ Physical address │
  └───────────────────────┘                    └──────────────────┘
  1M pages → Frequent TLB misses

2MB huge pages:
  │  Virtual address space  │  TLB (64 entries)  │  Physical address  │
  └─────────────────────────┘                     └────────────────────┘
  512 pages → Very few TLB misses

Effect: Reduces performance degradation from TLB misses