Module 4: Refactoring and Solution Implementation

Implementing Memory Leak Fixes and Validation

⏱️ Duration: 2 hours
🎯 Learning Objectives:
- Implement the prioritized solutions identified in Module 3
- Understand the coco=false configuration pattern for immediate leak resolution
- Understand proper resource lifecycle management patterns
- Validate that fixes are correctly applied and effective
- Set up monitoring and alerting for ongoing protection

🚀 Implementation Strategy Overview

The Three-Phase Implementation Approach

Based on your analysis from Module 3, we'll implement solutions in priority order:

1. Phase 1: Emergency Response (5 minutes) - Configuration change for immediate relief
2. Phase 2: Validation and Monitoring (30 minutes) - Verify fixes and establish monitoring
3. Phase 3: Long-term Improvements (1 hour) - Enhanced patterns and prevention

💡 Learning Insight

The beauty of the coco=false pattern is that it demonstrates the complete solution already implemented in NoCocoController. This allows you to see the "after" state immediately while learning the specific patterns that prevent memory leaks!

⚡ Phase 1: Emergency Response - Configuration Change

🎯 Practical Exercise 1: Immediate Memory Leak Resolution

Let's implement the highest-priority solution with immediate impact:

Step 1: Verify Current Configuration

cd ./cursor-rules-java/examples/spring-boot-memory-leak-demo

# Check current configuration
echo "=== Current Configuration Status ==="
curl -s http://localhost:8080/actuator/env/coco | grep -E '"value"|"name"'

# Verify which controller is active by testing endpoints
echo "=== Testing Current Controller ==="
RESPONSE=$(curl -s -X POST http://localhost:8080/api/v1/objects/create)
echo "Object creation response: $RESPONSE"

# Check current memory usage
echo "=== Current Memory Usage ==="
curl -s http://localhost:8080/actuator/metrics/jvm.memory.used | grep -E '"name"|"value"'

Step 2: Apply the Configuration Change

Method 1: Runtime Configuration Override (Immediate)

# Stop the current application
# Press Ctrl+C in the terminal running the application

# Restart with coco=false
echo "Starting application with memory leak fixes enabled..."
./run-java-process-for-profiling.sh --profile vt
# This uses the application-vt.properties profile which has coco=false

Method 2: Direct Configuration File Edit (Permanent)

# Edit the main configuration file
cp src/main/resources/application.properties src/main/resources/application.properties.backup

# Change the configuration
sed -i.bak 's/coco=true/coco=false/' src/main/resources/application.properties

# Verify the change
echo "=== Configuration Change Applied ==="
grep coco src/main/resources/application.properties

# Restart application
./run-java-process-for-profiling.sh

Step 3: Immediate Validation

# Wait for application startup (30 seconds)
sleep 30

# Verify the configuration change took effect
echo "=== Post-Change Configuration Verification ==="
curl -s http://localhost:8080/actuator/env/coco

# Test that fixed controller is now active
echo "=== Testing Fixed Controller ==="
for i in {1..5}; do
    RESPONSE=$(curl -s -X POST http://localhost:8080/api/v1/objects/create)
    echo "Test $i: $RESPONSE"
done

# Verify bounds checking is working
echo "=== Testing Bounds Protection ==="
# The fixed controller has MAX_OBJECTS=10000, but we can test the pattern
curl -s -X POST http://localhost:8080/api/v1/objects/create | grep -E "created|limit"

🔍 Understanding the Fix: Code Comparison

Let's examine exactly what changed:

# View the problematic implementation
echo "=== PROBLEMATIC IMPLEMENTATION (CocoController) ==="
grep -A 10 -B 2 "createObject" src/main/java/info/jab/ms/CocoController.java

echo ""
echo "=== FIXED IMPLEMENTATION (NoCocoController) ==="
grep -A 15 -B 2 "createObject" src/main/java/info/jab/ms/NoCocoController.java

Key Differences Analysis:

Collection Bounds:
- CocoController (Problematic): objects.add(new MyPojo(...))
- NoCocoController (Fixed): if (objects.size() >= MAX_OBJECTS) bounds check
Error Handling:
- CocoController (Problematic): No bounds validation
- NoCocoController (Fixed): Returns 400 Bad Request when limit reached
Thread Management:
- CocoController (Problematic): Executors.newFixedThreadPool(5) per request
- NoCocoController (Fixed): Shared ExecutorService with lifecycle
Resource Cleanup:
- CocoController (Problematic): No cleanup logic
- NoCocoController (Fixed): @PreDestroy method with proper shutdown
Thread Safety:
- CocoController (Problematic): Basic ArrayList
- NoCocoController (Fixed): Collections.synchronizedList

💡 Learning Reinforcement

Notice how the fixed implementation follows enterprise patterns: bounded collections, shared resources, lifecycle management, and graceful error handling. These aren't just memory leak fixes - they're production-ready design patterns!

🔍 Phase 2: Validation and Monitoring

🎯 Practical Exercise 2: Comprehensive Fix Validation

Step 1: Functional Validation

# Create comprehensive validation script
cat > validate-fixes.sh << 'EOF'
#!/bin/bash
echo "=== MEMORY LEAK FIX VALIDATION ==="
echo "Timestamp: $(date)"
echo ""

# Test 1: Configuration verification
echo "1. Configuration Status:"
CONFIG_RESPONSE=$(curl -s http://localhost:8080/actuator/env/coco)
echo "$CONFIG_RESPONSE" | grep -E '"value"|"name"' | head -2

# Test 2: Bounds checking validation
echo ""
echo "2. Bounds Checking Test:"
for i in {1..10}; do
    RESPONSE=$(curl -s -X POST http://localhost:8080/api/v1/objects/create)
    echo "  Request $i: $RESPONSE"
done

# Test 3: Thread creation test
echo ""
echo "3. Thread Management Test:"
for i in {1..5}; do
    THREAD_RESPONSE=$(curl -s -X POST http://localhost:8080/api/v1/threads/create)
    echo "  Thread test $i: $THREAD_RESPONSE"
done

# Test 4: Memory usage baseline
echo ""
echo "4. Memory Usage Check:"
MEMORY_USED=$(curl -s http://localhost:8080/actuator/metrics/jvm.memory.used)
echo "$MEMORY_USED" | grep -E '"value"' | head -1

# Test 5: Application health
echo ""
echo "5. Application Health:"
HEALTH=$(curl -s http://localhost:8080/actuator/health)
echo "$HEALTH"

echo ""
echo "=== VALIDATION COMPLETE ==="
EOF
chmod +x validate-fixes.sh

# Run validation
./validate-fixes.sh

Step 2: Load Testing Validation

# Run sustained load test to verify stability
echo "=== SUSTAINED LOAD TEST ==="
echo "Running 10-minute stability test..."

# Start JMeter load test
./run-jmeter.sh -t 600 -c 3 -e both &
JMETER_PID=$!

# Monitor memory usage during load test
echo "Monitoring memory usage every 30 seconds..."
for i in {1..20}; do
    TIMESTAMP=$(date '+%H:%M:%S')
    MEMORY=$(curl -s http://localhost:8080/actuator/metrics/jvm.memory.used | grep -o '"value":[0-9]*' | grep -o '[0-9]*')
    echo "$TIMESTAMP: Memory used: $MEMORY bytes"
    sleep 30
done

# Wait for JMeter to complete
wait $JMETER_PID
echo "Load test completed successfully"

Step 3: Comparative Memory Analysis

# Generate post-fix profiling data for comparison
echo "=== GENERATING POST-FIX PROFILING DATA ==="

cd profiler/scripts
./profile-java-process.sh
# Select: 2. Memory Allocation Profiling (30s)
# This will generate new flamegraphs with the fixes applied

# Generate load during profiling
cd ../..
for i in {1..50}; do
    curl -s -X POST http://localhost:8080/api/v1/objects/create > /dev/null
    curl -s -X POST http://localhost:8080/api/v1/threads/create > /dev/null
    sleep 1
done

echo "Post-fix profiling data generated"

🛠️ Phase 3: Understanding the Implementation Patterns

Resource Lifecycle Management Deep Dive

Let's examine the specific patterns that prevent memory leaks:

🎯 Practical Exercise 3: Code Pattern Analysis

# Extract and analyze the key patterns
echo "=== RESOURCE LIFECYCLE PATTERNS ==="

# Pattern 1: Bounded Collections
echo "1. Bounded Collection Pattern:"
grep -A 5 -B 2 "MAX_OBJECTS" src/main/java/info/jab/ms/NoCocoController.java

echo ""
echo "2. Thread Pool Management Pattern:"
grep -A 10 -B 2 "sharedExecutorService" src/main/java/info/jab/ms/NoCocoController.java

echo ""
echo "3. Resource Cleanup Pattern:"
grep -A 10 -B 2 "@PreDestroy" src/main/java/info/jab/ms/NoCocoController.java

Understanding Each Pattern

1. Bounded Collections Pattern:

private static final int MAX_OBJECTS = 10000;
private final List<MyPojo> objects = Collections.synchronizedList(new ArrayList<>());

@PostMapping("/objects/create")
public ResponseEntity<String> createObject() {
    if (objects.size() >= MAX_OBJECTS) {
        return ResponseEntity.badRequest()
            .body("Maximum objects limit reached: " + MAX_OBJECTS);
    }
    objects.add(new MyPojo(UUID.randomUUID().toString(),
                          "A".repeat(1000),
                          System.currentTimeMillis()));
    return ResponseEntity.ok("Object created: " + objects.size());
}

Key Benefits:
- ✅ Memory Protection: Prevents unbounded growth
- ✅ Graceful Degradation: Returns meaningful error instead of crashing
- ✅ Thread Safety: Uses synchronized wrapper for concurrent access
- ✅ Monitoring: Provides current count for observability

2. Shared Thread Pool Pattern:

private final ExecutorService sharedExecutorService =
    Executors.newFixedThreadPool(10, new CustomizableThreadFactory("shared-pool-"));

@PostMapping("/threads/create")
public ResponseEntity<String> createThread() {
    sharedExecutorService.submit(() -> {
        try {
            Thread.sleep(1000);
            log.info("Task executed in thread: {}", Thread.currentThread().getName());
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    });
    return ResponseEntity.ok("Task submitted to shared thread pool");
}

Key Benefits:
- ✅ Resource Efficiency: Single shared pool vs. multiple pools
- ✅ Lifecycle Management: Proper initialization and cleanup
- ✅ Named Threads: CustomizableThreadFactory for debugging
- ✅ Interruption Handling: Proper interrupt signal management

3. Resource Cleanup Pattern:

@PreDestroy
public void cleanup() throws InterruptedException {
    log.info("Shutting down shared executor service...");
    sharedExecutorService.shutdown();

    if (!sharedExecutorService.awaitTermination(30, TimeUnit.SECONDS)) {
        log.warn("Executor did not terminate gracefully, forcing shutdown...");
        sharedExecutorService.shutdownNow();

        if (!sharedExecutorService.awaitTermination(10, TimeUnit.SECONDS)) {
            log.error("Executor did not terminate after forced shutdown");
        }
    }
    log.info("Shared executor service shutdown complete");
}

Key Benefits:
- ✅ Graceful Shutdown: Allows running tasks to complete
- ✅ Timeout Protection: Forces shutdown if graceful fails
- ✅ Logging: Provides visibility into shutdown process
- ✅ Resource Guarantee: Ensures threads are properly cleaned up

💡 Learning Reinforcement

These patterns represent enterprise-grade resource management. The @PreDestroy pattern is especially important - it's Spring's way of ensuring cleanup happens during application shutdown, preventing resource leaks even during deployment cycles!

📊 Monitoring and Alerting Setup

🎯 Practical Exercise 4: Implementing Memory Monitoring

Step 1: JVM Monitoring Configuration

# Add comprehensive JVM monitoring flags
cat > monitoring-flags.txt << 'EOF'
# Memory Monitoring JVM Flags
-XX:+PrintGC
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationStoppedTime
-Xloggc:gc-monitoring.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=5
-XX:GCLogFileSize=10M

# Memory Dump on OutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=./memory-dumps/

# JFR Continuous Profiling
-XX:+FlightRecorder
-XX:StartFlightRecording=duration=1h,filename=continuous-monitoring.jfr,settings=profile
EOF

echo "Monitoring configuration created. Add these flags to your startup script."

Step 2: Application Metrics Monitoring

# Create memory monitoring script
cat > monitor-memory.sh << 'EOF'
#!/bin/bash
MONITOR_DURATION=${1:-300}  # Default 5 minutes
INTERVAL=${2:-10}           # Default 10 seconds

echo "=== MEMORY MONITORING ==="
echo "Duration: $MONITOR_DURATION seconds"
echo "Interval: $INTERVAL seconds"
echo "Timestamp,HeapUsed,HeapMax,NonHeapUsed,ObjectCount" > memory-monitor.csv

START_TIME=$(date +%s)
END_TIME=$((START_TIME + MONITOR_DURATION))

while [ $(date +%s) -lt $END_TIME ]; do
    TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')

    # Get heap memory usage
    HEAP_USED=$(curl -s http://localhost:8080/actuator/metrics/jvm.memory.used?tag=area:heap | grep -o '"value":[0-9]*' | grep -o '[0-9]*')
    HEAP_MAX=$(curl -s http://localhost:8080/actuator/metrics/jvm.memory.max?tag=area:heap | grep -o '"value":[0-9]*' | grep -o '[0-9]*')

    # Get non-heap memory usage
    NONHEAP_USED=$(curl -s http://localhost:8080/actuator/metrics/jvm.memory.used?tag=area:nonheap | grep -o '"value":[0-9]*' | grep -o '[0-9]*')

    # Get object count from our controller
    OBJECT_RESPONSE=$(curl -s -X POST http://localhost:8080/api/v1/objects/create)
    OBJECT_COUNT=$(echo "$OBJECT_RESPONSE" | grep -o '[0-9]*$' || echo "0")

    # Log to CSV
    echo "$TIMESTAMP,$HEAP_USED,$HEAP_MAX,$NONHEAP_USED,$OBJECT_COUNT" >> memory-monitor.csv

    # Display current status
    HEAP_PCT=$((HEAP_USED * 100 / HEAP_MAX))
    echo "[$TIMESTAMP] Heap: ${HEAP_PCT}% ($HEAP_USED/$HEAP_MAX bytes), Objects: $OBJECT_COUNT"

    sleep $INTERVAL
done

echo "Monitoring complete. Data saved to memory-monitor.csv"
EOF
chmod +x monitor-memory.sh

# Run monitoring during load test
echo "Starting memory monitoring with load test..."
./monitor-memory.sh 300 15 &  # 5 minutes, 15-second intervals
MONITOR_PID=$!

# Generate load during monitoring
for i in {1..100}; do
    curl -s -X POST http://localhost:8080/api/v1/objects/create > /dev/null
    sleep 3
done

wait $MONITOR_PID
echo "Memory monitoring completed"

Step 3: Alerting Thresholds

# Create alerting configuration
cat > memory-alerts.sh << 'EOF'
#!/bin/bash
# Memory alerting script

HEAP_THRESHOLD=80  # Alert when heap usage > 80%
OBJECT_THRESHOLD=9000  # Alert when objects > 9000

check_memory_usage() {
    HEAP_USED=$(curl -s http://localhost:8080/actuator/metrics/jvm.memory.used?tag=area:heap | grep -o '"value":[0-9]*' | grep -o '[0-9]*')
    HEAP_MAX=$(curl -s http://localhost:8080/actuator/metrics/jvm.memory.max?tag=area:heap | grep -o '"value":[0-9]*' | grep -o '[0-9]*')

    if [ -n "$HEAP_USED" ] && [ -n "$HEAP_MAX" ]; then
        HEAP_PCT=$((HEAP_USED * 100 / HEAP_MAX))

        if [ $HEAP_PCT -gt $HEAP_THRESHOLD ]; then
            echo "ALERT: Heap usage at ${HEAP_PCT}% (threshold: ${HEAP_THRESHOLD}%)"
            return 1
        fi
    fi

    return 0
}

check_object_count() {
    OBJECT_RESPONSE=$(curl -s -X POST http://localhost:8080/api/v1/objects/create)
    OBJECT_COUNT=$(echo "$OBJECT_RESPONSE" | grep -o '[0-9]*$' || echo "0")

    if [ "$OBJECT_COUNT" -gt $OBJECT_THRESHOLD ]; then
        echo "ALERT: Object count at $OBJECT_COUNT (threshold: $OBJECT_THRESHOLD)"
        return 1
    fi

    return 0
}

# Run checks
echo "=== MEMORY ALERT CHECK ==="
ALERTS=0

if ! check_memory_usage; then
    ALERTS=$((ALERTS + 1))
fi

if ! check_object_count; then
    ALERTS=$((ALERTS + 1))
fi

if [ $ALERTS -eq 0 ]; then
    echo "✅ All memory metrics within normal ranges"
else
    echo "⚠️  $ALERTS alert(s) triggered"
fi

exit $ALERTS
EOF
chmod +x memory-alerts.sh

# Test alerting
./memory-alerts.sh

🎯 Module 4 Assessment

Implementation Validation Checklist

✅ Emergency Response Completed:
- [ ] Successfully changed configuration to coco=false
- [ ] Verified NoCocoController is now active
- [ ] Confirmed bounds checking is working (MAX_OBJECTS limit)
- [ ] Validated thread pool management is operational
- [ ] Application remains stable and responsive

✅ Code Pattern Understanding:
- [ ] Identified bounded collection implementation
- [ ] Understood shared thread pool lifecycle management
- [ ] Recognized @PreDestroy cleanup pattern
- [ ] Analyzed thread safety improvements
- [ ] Documented key differences between implementations

✅ Monitoring and Validation:
- [ ] Generated post-fix profiling data
- [ ] Completed sustained load testing (10+ minutes)
- [ ] Implemented memory usage monitoring
- [ ] Created alerting thresholds and scripts
- [ ] Documented baseline metrics for comparison

✅ Production Readiness:
- [ ] Validated fixes under realistic load conditions
- [ ] Established ongoing monitoring capabilities
- [ ] Created alerting for early problem detection
- [ ] Documented implementation for team knowledge transfer

🎯 Advanced Challenge: Custom Resource Management Pattern

Challenge: Implement your own resource management pattern following the principles learned

Scenario: Create a custom cache implementation that prevents memory leaks

Requirements:

// Implement this interface following NoCocoController patterns
public class SafeCache<K, V> {
    private static final int MAX_CACHE_SIZE = 1000;
    private final Map<K, V> cache = /* your implementation */;

    // TODO: Implement bounded put method
    public V put(K key, V value) {
        // Your bounds-checking implementation
    }

    // TODO: Implement cleanup method
    @PreDestroy
    public void cleanup() {
        // Your resource cleanup implementation
    }
}

Success Criteria:
- [ ] Implements bounds checking to prevent unbounded growth
- [ ] Provides graceful handling when limits are reached
- [ ] Includes proper cleanup lifecycle management
- [ ] Thread-safe for concurrent access
- [ ] Includes monitoring/observability features

🚀 Transition to Module 5

Fantastic implementation work! You've successfully:
- ✅ Applied the immediate configuration fix to eliminate memory leaks
- ✅ Understood and analyzed the resource management patterns
- ✅ Validated fixes through comprehensive testing
- ✅ Implemented monitoring and alerting infrastructure
- ✅ Demonstrated production-ready resource lifecycle management

What's Next?

In Module 5: Validation and Comparison, we'll focus on:
- Using @164-java-profiling-compare to rigorously validate improvements
- Generating before/after profiling comparisons
- Creating quantitative evidence of memory leak resolution
- Documenting the complete success story with measurable results

💡 Key Takeaway

"The best fixes are not just functional - they're observable, maintainable, and educational. By understanding the patterns in NoCocoController, you've learned enterprise-grade resource management that applies far beyond this demo!"

Ready to prove your fixes work with rigorous before/after analysis? Let's complete the journey in Module 5! 📊