Extend Memory-Mapped Arrays to Additional State Representations

[csmangum]

While issue #341 successfully implemented memory-mapped arrays for resource grids, the broader goal of using memmap for all large state representations remains incomplete. This issue focuses on extending the memmap system to additional high-memory state structures.

Background

The current memmap implementation only covers resource grids, but there are several other large state structures that would benefit from on-disk memory mapping:

1. Environmental Grids: OBSTACLES, TERRAIN_COST, VISIBILITY masks (dense grid storage)
2. Agent-centric Global Layers: World representations from each agent's perspective
3. Large Perception/Observation Grids: For agents with large observation windows
4. Temporal Channel State: DAMAGE_HEAT, TRAILS, ALLY_SIGNAL persistence layers

Goals

• Extend memmap support to environmental grids (OBSTACLES, TERRAIN_COST, VISIBILITY)
• Implement agent-centric global layer memmap storage
• Add memmap support for large temporal channel persistence
• Maintain compatibility with existing tensor operations
• Ensure multiprocess safety for concurrent access

Technical Requirements

• Update ResourceManager pattern to a more generic MemmapManager or extend existing class
• Support different data types and shapes for different grid types
• Handle sparse vs dense representations appropriately
• Maintain performance benchmarks showing no significant slowdown
• Cross-platform compatibility (current validation is Linux-only)

Acceptance Criteria

• Environmental grids (OBSTACLES, TERRAIN_COST, VISIBILITY) use memmap when enabled
• Agent observations can use memmap-backed global layers
• Performance tests show ≤ 1.25x baseline latency for memmap operations
• Compatible with existing torch tensor operations
• Memory usage scales with grid size, not loaded into RAM
• Validation script covers all new memmap structures

Implementation Steps

1. Design generic memmap interface for different grid types
2. Implement environmental grid memmap (OBSTACLES, TERRAIN_COST, VISIBILITY)
3. Add agent-centric global layer memmap support
4. Extend temporal channel persistence to use memmap
5. Update validation script with comprehensive testing
6. Add configuration options for different memmap targets

Dependencies

• Completion of issue Add Memory-Mapped Arrays for State Representations #341 (resource grid memmap)
• Performance benchmarking infrastructure
• Cross-platform memory measurement utilities

Priority: Medium-High

Estimate: 2-3 weeks

Labels: performance, memory, optimization, scalability