Add configurable session-segmented payload directory for agent access (#569)

## Implementation Plan: Shared Payload Directory for Large Payloads

- [x] Add configuration support for payload directory
  - [x] Add `PayloadDir` field to `GatewayConfig` struct
  - [x] Add support in TOML config (`payload_dir`)
  - [x] Add support in JSON/stdin config (`payloadDir`)
  - [x] Add command-line flag `--payload-dir`
  - [x] Add environment variable `MCP_GATEWAY_PAYLOAD_DIR`
  - [x] Set default to `/tmp/jq-payloads`
- [x] Update jqschema middleware to use session-based payload
directories
- [x] Modify `savePayload` to accept session ID and base payload
directory
- [x] Create session-specific subdirectories (e.g.,
`/tmp/jq-payloads/{sessionID}/`)
  - [x] Update `WrapToolHandler` to pass session ID from context
- [x] Use restrictive permissions (0600 for files, 0700 for directories)
- [x] Update middleware tests
  - [x] Fix test signatures to match new WrapToolHandler parameters
  - [x] Update test assertions for new directory structure
- [x] Add comprehensive tests for new functionality
  - [x] Config parsing tests for payload_dir (TOML)
  - [x] Config parsing tests for default payload_dir
  - [x] Test session directory creation
  - [x] Test payloadDir validation with absolute path requirement
- [x] Update documentation
  - [x] Update AGENTS.md with new configuration option
  - [x] Document environment variable `MCP_GATEWAY_PAYLOAD_DIR`
  - [x] Document command-line flag `--payload-dir`
  - [x] Document large payload handling mechanism
- [x] Security improvements
  - [x] Use restrictive file permissions (0600) for payload files
- [x] Use restrictive directory permissions (0700) for payload
directories
- [x] Session directory creation
- [x] Gateway checks if session subdirectory exists on session creation
  - [x] Creates session directory if it doesn't exist
  - [x] Added test for session directory creation
- [x] Schema validation
  - [x] Added validation for payloadDir field requiring absolute paths
  - [x] Validation matches schema pattern: ^(/|[A-Za-z]:\\)
  - [x] Added comprehensive tests for Unix and Windows paths
  - [x] Validation conforms to official MCP Gateway schema
- [x] Fix linting issues
  - [x] Run gofmt on all modified files
- [x] Run full test suite and verify changes

## Summary

All implementation tasks are complete. The gateway now:
1. Supports configurable payload directories with session-based
segmentation
2. Automatically creates session subdirectories when sessions are
initialized
3. Validates payloadDir as an absolute path per the gateway schema spec
4. Allows agents to mount their own session subdirectory and access
large payloads as regular files
5. All code is properly formatted and passes linting checks
6. payloadDir validation strictly conforms to the official MCP Gateway
schema specification

<!-- START COPILOT CODING AGENT TIPS -->
---

💡 You can make Copilot smarter by setting up custom instructions,
customizing its development environment and configuring Model Context
Protocol (MCP) servers. Learn more [Copilot coding agent
tips](https://gh.io/copilot-coding-agent-tips) in the docs.

Author: lpcox

AGENTS.md

@@ -17,6 +17,7 @@ Quick reference for AI agents working with MCP Gateway (Go-based MCP proxy serve
 **Agent-Finished**: `make agent-finished` (run format, build, lint, and all tests - ALWAYS run before completion)  
 **Run**: `./awmg --config config.toml`
 **Run with Custom Log Directory**: `./awmg --config config.toml --log-dir /path/to/logs`
+**Run with Custom Payload Directory**: `./awmg --config config.toml --payload-dir /path/to/payloads`
 
 ## Project Structure
 
@@ -47,6 +48,11 @@ Quick reference for AI agents working with MCP Gateway (Go-based MCP proxy serve
 
 **TOML** (`config.toml`):
 ```toml
+[gateway]
+port = 3000
+api_key = "your-api-key"
+payload_dir = "/tmp/jq-payloads"  # Optional: directory for large payload storage
+
 [servers.github]
 command = "docker"
 args = ["run", "--rm", "-e", "GITHUB_PERSONAL_ACCESS_TOKEN", "-i", "ghcr.io/github/github-mcp-server:latest"]
@@ -357,13 +363,21 @@ DEBUG_COLORS=0 DEBUG=* ./awmg --config config.toml
 - `DEBUG` - Enable debug logging (e.g., `DEBUG=*`, `DEBUG=server:*,launcher:*`)
 - `DEBUG_COLORS` - Control colored output (0 to disable, auto-disabled when piping)
 - `MCP_GATEWAY_LOG_DIR` - Log file directory (sets default for `--log-dir` flag, default: `/tmp/gh-aw/mcp-logs`)
+- `MCP_GATEWAY_PAYLOAD_DIR` - Large payload storage directory (sets default for `--payload-dir` flag, default: `/tmp/jq-payloads`)
 
 **File Logging:**
 - Operational logs are always written to `mcp-gateway.log` in the configured log directory
 - Default log directory: `/tmp/gh-aw/mcp-logs` (configurable via `--log-dir` flag or `MCP_GATEWAY_LOG_DIR` env var)
 - Falls back to stdout if log directory cannot be created
 - Logs include: startup, client interactions, backend operations, auth events, errors
 
+**Large Payload Handling:**
+- Large tool response payloads are stored in the configured payload directory
+- Default payload directory: `/tmp/jq-payloads` (configurable via `--payload-dir` flag, `MCP_GATEWAY_PAYLOAD_DIR` env var, or `payload_dir` in config)
+- Payloads are organized by session ID: `{payload_dir}/{sessionID}/{queryID}/payload.json`
+- This allows agents to mount their session-specific subdirectory to access full payloads
+- The jq middleware returns: preview (first 500 chars), schema, payloadPath, queryID, originalSize, truncated flag
+
 ## Error Debugging
 
 **Enhanced Error Context**: Command failures include:

internal/cmd/root.go

@@ -34,6 +34,7 @@ const (
 	defaultEnvFile          = ""
 	defaultEnableDIFC       = false
 	defaultLogDir           = "/tmp/gh-aw/mcp-logs"
+	defaultPayloadDir       = "/tmp/jq-payloads"
 	defaultSequentialLaunch = false
 )
 
@@ -46,6 +47,7 @@ var (
 	envFile          string
 	enableDIFC       bool
 	logDir           string
+	payloadDir       string
 	validateEnv      bool
 	sequentialLaunch bool
 	verbosity        int // Verbosity level: 0 (default), 1 (-v info), 2 (-vv debug), 3 (-vvv trace)
@@ -76,6 +78,7 @@ func init() {
 	rootCmd.Flags().StringVar(&envFile, "env", defaultEnvFile, "Path to .env file to load environment variables")
 	rootCmd.Flags().BoolVar(&enableDIFC, "enable-difc", defaultEnableDIFC, "Enable DIFC enforcement and session requirement (requires sys___init call before tool access)")
 	rootCmd.Flags().StringVar(&logDir, "log-dir", getDefaultLogDir(), "Directory for log files (falls back to stdout if directory cannot be created)")
+	rootCmd.Flags().StringVar(&payloadDir, "payload-dir", getDefaultPayloadDir(), "Directory for storing large payload files (segmented by session ID)")
 	rootCmd.Flags().BoolVar(&validateEnv, "validate-env", false, "Validate execution environment (Docker, env vars) before starting")
 	rootCmd.Flags().BoolVar(&sequentialLaunch, "sequential-launch", defaultSequentialLaunch, "Launch MCP servers sequentially during startup (parallel launch is default)")
 	rootCmd.Flags().CountVarP(&verbosity, "verbose", "v", "Increase verbosity level (use -v for info, -vv for debug, -vvv for trace)")
@@ -99,6 +102,15 @@ func getDefaultLogDir() string {
 	return defaultLogDir
 }
 
+// getDefaultPayloadDir returns the default payload directory, checking MCP_GATEWAY_PAYLOAD_DIR
+// environment variable first, then falling back to the hardcoded default
+func getDefaultPayloadDir() string {
+	if envPayloadDir := os.Getenv("MCP_GATEWAY_PAYLOAD_DIR"); envPayloadDir != "" {
+		return envPayloadDir
+	}
+	return defaultPayloadDir
+}
+
 const (
 	// Debug log patterns for different verbosity levels
 	debugMainPackages = "cmd:*,server:*,launcher:*"
@@ -117,6 +129,11 @@ func registerFlagCompletions(cmd *cobra.Command) {
 		return nil, cobra.ShellCompDirectiveFilterDirs
 	})
 
+	// Custom completion for --payload-dir flag (complete with directories)
+	cmd.RegisterFlagCompletionFunc("payload-dir", func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
+		return nil, cobra.ShellCompDirectiveFilterDirs
+	})
+
 	// Custom completion for --env flag (complete with .env files)
 	cmd.RegisterFlagCompletionFunc("env", func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
 		return []string{"env"}, cobra.ShellCompDirectiveFilterFileExt
@@ -242,6 +259,7 @@ func run(cmd *cobra.Command, args []string) error {
 	// Apply command-line flags to config
 	cfg.EnableDIFC = enableDIFC
 	cfg.SequentialLaunch = sequentialLaunch
+
 	if enableDIFC {
 		log.Println("DIFC enforcement and session requirement enabled")
 	} else {

internal/config/config.go

@@ -20,6 +20,8 @@ const (
 	DefaultStartupTimeout = 60
 	// DefaultToolTimeout is the default timeout for tool execution (seconds)
 	DefaultToolTimeout = 120
+	// DefaultPayloadDir is the default directory for storing large payload files
+	DefaultPayloadDir = "/tmp/jq-payloads"
 )
 
 // Config represents the MCPG configuration
@@ -37,6 +39,7 @@ type GatewayConfig struct {
 	Domain         string `toml:"domain"`
 	StartupTimeout int    `toml:"startup_timeout"` // Seconds
 	ToolTimeout    int    `toml:"tool_timeout"`    // Seconds
+	PayloadDir     string `toml:"payload_dir"`     // Directory for storing large payload files
 }
 
 // ServerConfig represents a single MCP server configuration
@@ -82,6 +85,7 @@ type StdinGatewayConfig struct {
 	Domain         string `json:"domain,omitempty"`
 	StartupTimeout *int   `json:"startupTimeout,omitempty"` // Seconds to wait for backend startup
 	ToolTimeout    *int   `json:"toolTimeout,omitempty"`    // Seconds to wait for tool execution
+	PayloadDir     string `json:"payloadDir,omitempty"`     // Directory for storing large payload files
 }
 
 // LoadFromFile loads configuration from a TOML file
@@ -115,6 +119,9 @@ func LoadFromFile(path string) (*Config, error) {
 		if cfg.Gateway.Port == 0 {
 			cfg.Gateway.Port = DefaultPort
 		}
+		if cfg.Gateway.PayloadDir == "" {
+			cfg.Gateway.PayloadDir = DefaultPayloadDir
+		}
 	}
 
 	logConfig.Printf("Successfully loaded %d servers from TOML file", len(cfg.Servers))
@@ -185,6 +192,7 @@ func LoadFromStdin() (*Config, error) {
 			Domain:         stdinCfg.Gateway.Domain,
 			StartupTimeout: DefaultStartupTimeout,
 			ToolTimeout:    DefaultToolTimeout,
+			PayloadDir:     DefaultPayloadDir,
 		}
 		if stdinCfg.Gateway.Port != nil {
 			cfg.Gateway.Port = *stdinCfg.Gateway.Port
@@ -195,6 +203,9 @@ func LoadFromStdin() (*Config, error) {
 		if stdinCfg.Gateway.ToolTimeout != nil {
 			cfg.Gateway.ToolTimeout = *stdinCfg.Gateway.ToolTimeout
 		}
+		if stdinCfg.Gateway.PayloadDir != "" {
+			cfg.Gateway.PayloadDir = stdinCfg.Gateway.PayloadDir
+		}
 	}
 
 	for name, server := range stdinCfg.MCPServers {

internal/config/config_test.go

@@ -435,6 +435,39 @@ func TestLoadFromStdin_GatewayWithAllFields(t *testing.T) {
 	assert.Equal(t, toolTimeout, *stdinCfg.Gateway.ToolTimeout, "Expected gateway toolTimeout")
 }
 
+func TestLoadFromStdin_GatewayWithoutPayloadDir(t *testing.T) {
+	jsonConfig := `{
+		"mcpServers": {
+			"test": {
+				"type": "stdio",
+				"container": "test/server:latest",
+				"entrypointArgs": ["server.js"]
+			}
+		},
+		"gateway": {
+			"port": 8080,
+			"apiKey": "test-key-123",
+			"domain": "localhost"
+		}
+	}`
+
+	r, w, _ := os.Pipe()
+	oldStdin := os.Stdin
+	os.Stdin = r
+	go func() {
+		w.Write([]byte(jsonConfig))
+		w.Close()
+	}()
+
+	cfg, err := LoadFromStdin()
+	os.Stdin = oldStdin
+
+	require.NoError(t, err, "LoadFromStdin() failed")
+	require.NotNil(t, cfg, "Config should not be nil")
+	require.NotNil(t, cfg.Gateway, "Gateway config should not be nil")
+	assert.Equal(t, DefaultPayloadDir, cfg.Gateway.PayloadDir, "Expected default payload directory when not specified")
+}
+
 func TestLoadFromStdin_ServerWithURL(t *testing.T) {
 	jsonConfig := `{
 		"mcpServers": {
@@ -917,6 +950,59 @@ args = ["run", "--rm", "-i", "test/container:latest"]
 	assert.Equal(t, 60, cfg.Gateway.ToolTimeout)
 }
 
+func TestLoadFromFile_WithGatewayPayloadDir(t *testing.T) {
+	tmpDir := t.TempDir()
+	tmpFile := filepath.Join(tmpDir, "config.toml")
+
+	tomlContent := `
+[gateway]
+port = 8080
+api_key = "test-key-123"
+domain = "localhost"
+payload_dir = "/custom/payload/path"
+
+[servers.test]
+command = "docker"
+args = ["run", "--rm", "-i", "test/container:latest"]
+`
+
+	err := os.WriteFile(tmpFile, []byte(tomlContent), 0644)
+	require.NoError(t, err, "Failed to write temp TOML file")
+
+	cfg, err := LoadFromFile(tmpFile)
+	require.NoError(t, err, "LoadFromFile() failed")
+	require.NotNil(t, cfg, "LoadFromFile() returned nil config")
+	require.NotNil(t, cfg.Gateway, "Gateway config should not be nil")
+
+	assert.Equal(t, "/custom/payload/path", cfg.Gateway.PayloadDir, "Expected custom payload directory")
+}
+
+func TestLoadFromFile_WithoutGatewayPayloadDir(t *testing.T) {
+	tmpDir := t.TempDir()
+	tmpFile := filepath.Join(tmpDir, "config.toml")
+
+	tomlContent := `
+[gateway]
+port = 8080
+api_key = "test-key-123"
+domain = "localhost"
+
+[servers.test]
+command = "docker"
+args = ["run", "--rm", "-i", "test/container:latest"]
+`
+
+	err := os.WriteFile(tmpFile, []byte(tomlContent), 0644)
+	require.NoError(t, err, "Failed to write temp TOML file")
+
+	cfg, err := LoadFromFile(tmpFile)
+	require.NoError(t, err, "LoadFromFile() failed")
+	require.NotNil(t, cfg, "LoadFromFile() returned nil config")
+	require.NotNil(t, cfg.Gateway, "Gateway config should not be nil")
+
+	assert.Equal(t, DefaultPayloadDir, cfg.Gateway.PayloadDir, "Expected default payload directory when not specified")
+}
+
 func TestLoadFromFile_InvalidTOMLWithLineNumber(t *testing.T) {
 	tmpDir := t.TempDir()
 	tmpFile := filepath.Join(tmpDir, "config.toml")

internal/config/rules/rules.go

@@ -187,3 +187,52 @@ func MountFormat(mount, jsonPath string, index int) *ValidationError {
 
 	return nil
 }
+
+// NonEmptyString validates that a string field is not empty (minLength: 1)
+// Returns nil if valid, *ValidationError if invalid
+func NonEmptyString(value, fieldName, jsonPath string) *ValidationError {
+	if value == "" {
+		return &ValidationError{
+			Field:      fieldName,
+			Message:    fmt.Sprintf("%s cannot be empty", fieldName),
+			JSONPath:   jsonPath,
+			Suggestion: fmt.Sprintf("Provide a non-empty value for %s", fieldName),
+		}
+	}
+	return nil
+}
+
+// AbsolutePath validates that a directory path is an absolute path
+// Per MCP Gateway schema: Unix paths start with '/', Windows paths start with a drive letter followed by ':\'
+// Pattern: ^(/|[A-Za-z]:\\)
+// Returns nil if valid, *ValidationError if invalid
+func AbsolutePath(value, fieldName, jsonPath string) *ValidationError {
+	if value == "" {
+		return &ValidationError{
+			Field:      fieldName,
+			Message:    fmt.Sprintf("%s cannot be empty", fieldName),
+			JSONPath:   jsonPath,
+			Suggestion: fmt.Sprintf("Provide an absolute path for %s", fieldName),
+		}
+	}
+
+	// Check for Unix absolute path (starts with /)
+	if strings.HasPrefix(value, "/") {
+		return nil
+	}
+
+	// Check for Windows absolute path (drive letter followed by :\)
+	// Pattern: [A-Za-z]:\\
+	if len(value) >= 3 &&
+		((value[0] >= 'A' && value[0] <= 'Z') || (value[0] >= 'a' && value[0] <= 'z')) &&
+		value[1] == ':' && value[2] == '\\' {
+		return nil
+	}
+
+	return &ValidationError{
+		Field:      fieldName,
+		Message:    fmt.Sprintf("%s must be an absolute path, got '%s'", fieldName, value),
+		JSONPath:   jsonPath,
+		Suggestion: "Use an absolute path: Unix paths start with '/' (e.g., '/tmp/payloads'), Windows paths start with a drive letter (e.g., 'C:\\payloads')",
+	}
+}