[INS-283] Support following symlinks in filesystem source #4742
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
| } | ||
| // If symlink resolves to directory scan that directory | ||
| ctx.Logger().V(5).Info("Resolved symlink is a directory", "path", resolvedPath) | ||
| err = s.scanDir(ctx, resolvedPath, chunksChan) |
There was a problem hiding this comment.
Recursive scanDir creates unbounded nested worker pools
Medium Severity
When a symlink to a directory is found during WalkDir, scanDir is called recursively at line 268 from within the walk callback. Each recursive scanDir creates its own errgroup.Group with SetLimit(s.concurrency). The outer worker pool may still have goroutines running (spawned for files found before the symlink), while the inner pool also spawns up to s.concurrency goroutines. With nested symlinked directories, the effective concurrency becomes depth * s.concurrency, bypassing the user-configured limit and potentially exhausting system resources.
Additional Locations (1)
| if s.maxSymlinkDepth > 0 { | ||
| s.followSymlinks = true | ||
| } | ||
| s.visitedSymlinkPaths = make(map[string]struct{}) |
There was a problem hiding this comment.
Visited symlink state leaks across independent ChunkUnit calls
Medium Severity
visitedSymlinkPaths is initialized once in Init and never cleared. markVisitedPath is called unconditionally at the start of every scanDir, including for non-symlink directories. When using the SourceUnitEnumChunker interface, where ChunkUnit is called per enumerated unit, the accumulated state from one unit's scan causes subsequent units to skip symlinks pointing to already-visited directories. This makes scan results depend on unit processing order and can silently miss content.
Additional Locations (1)
1 participant
Description:
This PR re-enables symlink following in filesystem traversal and introduces configurable controls for symlink behavior, including:
Symlink resolution uses Go’s
os.Readlink, which resolves a single symlink per call. If the number of resolutions exceeds the configured max depth,resolveSymLinkreturns an error.Additionally, directory scanning is now robust against symlink loops (e.g.,
A/linkToB -> BandB/linkToA -> A). We maintain a map (visitedSymlinkPaths) of already-resolved directory symlinks to prevent infinite recursion while keeping traversal predictable and stableChecklist:
make test-community)?make lintthis requires golangci-lint)?