SQLMapper provides stream processing capabilities for handling large SQL files efficiently. This feature allows you to process SQL dumps in parallel, improving performance for large datasets.
The stream processing feature:
- Processes SQL statements in parallel
- Uses worker pools for efficient resource utilization
- Supports all major database types
- Provides real-time callback for processed objects
- Handles large SQL files without loading entire content into memory
- Includes comprehensive monitoring and logging system
parser := sqlmapper.NewStreamParser(sqlmapper.MySQL)
err := parser.ParseStream(sqlContent, func(obj interface{}) error {
switch v := obj.(type) {
case sqlmapper.Table:
// Handle table
case sqlmapper.View:
// Handle view
case sqlmapper.Function:
// Handle function
}
return nil
})parser := sqlmapper.NewStreamParser(sqlmapper.MySQL)
// Process with 4 workers
err := parser.ParseStreamParallel(sqlContent, 4, func(obj interface{}) error {
switch v := obj.(type) {
case sqlmapper.Table:
fmt.Printf("Processing table: %s\n", v.Name)
case sqlmapper.View:
fmt.Printf("Processing view: %s\n", v.Name)
}
return nil
})The number of workers can be configured based on your system's capabilities:
// For CPU-bound tasks
workers := runtime.NumCPU()
// For I/O-bound tasks
workers := runtime.NumCPU() * 2The stream processor can handle various SQL objects:
- Tables
- Views
- Functions
- Procedures
- Triggers
- Indexes
- Sequences
Each object is passed to the callback function as it's processed.
Errors during stream processing are handled gracefully:
err := parser.ParseStreamParallel(sqlContent, 4, func(obj interface{}) error {
if err := processObject(obj); err != nil {
return fmt.Errorf("failed to process object: %v", err)
}
return nil
})
if err != nil {
log.Printf("Stream processing error: %v", err)
}-
Worker Pool Size
- Start with number of CPU cores
- Adjust based on performance monitoring
- Consider memory constraints
-
Error Handling
- Always check for errors in callback
- Log errors appropriately
- Consider implementing retry logic
-
Memory Management
- Process objects as they come
- Avoid storing large amounts of data
- Use channels for communication
-
Performance Optimization
- Monitor CPU and memory usage
- Adjust worker count based on system load
- Consider batch processing for small objects
var processed int64
err := parser.ParseStreamParallel(sqlContent, 4, func(obj interface{}) error {
atomic.AddInt64(&processed, 1)
fmt.Printf("\rProcessed objects: %d", processed)
return nil
})err := parser.ParseStreamParallel(sqlContent, 4, func(obj interface{}) error {
switch v := obj.(type) {
case sqlmapper.Table:
return processTable(v)
case sqlmapper.View:
return processView(v)
case sqlmapper.Function:
return processFunction(v)
default:
return fmt.Errorf("unsupported object type")
}
})err := parser.ParseStreamParallel(sqlContent, 4, func(obj interface{}) error {
for attempts := 0; attempts < 3; attempts++ {
if err := processObject(obj); err != nil {
if attempts == 2 {
return err
}
continue
}
break
}
return nil
})- Maximum file size depends on available system memory
- Some complex SQL statements might require sequential processing
- Performance depends on system resources and SQL complexity
- Worker pool size affects memory usage
- Large SQL files might require batching
- Consider network I/O for database operations
- Monitor system resources during processing