Skip to content

Failed to upload large file #168

@fuzzbuster

Description

@fuzzbuster

Prerequisite

  • I have read the README
  • I have searched the opened & closed issues
  • I have searched the WIKI and its FAQ page

Environment Data

  • Merlin Version: 2.1.4
  • Operating System: All on Linux

Expected Behavior

Uploading large files via merlin-cli(version 1.15) should complete successfully without errors, and merlin-cli should remain running/responsive after the upload process.

Actual Behavior

When attempting to upload a large file (generated via dd), the upload fails midway, and merlin-cli crashes abruptly without providing a clear error message. The process exits completely, requiring a restart of merlin-cli to continue operations.

Steps to Reproduce Behavior

  1. Generate a large test file using dd:
    dd if=/dev/zero of=large_test_file.bin bs=1G count=1 (adjust count to increase file size, e.g., count=2 for 2GB)
  2. Start the Merlin server:
    ./merlin-server
  3. Launch merlin-cli and establish a connection to the server:
    ./merlin-cli ....
  4. In merlin-cli, navigate to the agent context and initiate the file upload:
    upload large_test_file.bin /remote/path/
  5. Observe that the upload progresses partially but then fails, and merlin-cli crashes/quits unexpectedly.

Misc Information

  • The issue is consistently reproducible with files larger than ~500MB (threshold may vary by environment).
  • No obvious error logs are generated in merlin-server.log or the terminal where merlin-cli was running before the crash.
  • Smaller files (e.g., 1MB) upload successfully without issues.
  • System resources (RAM, disk space) are sufficient during the upload attempt.
  • Upon reviewing the code, I noticed that chunked transfer is not implemented in the .proto definition for file transfers. I hypothesize that implementing chunked transfer (splitting files into smaller chunks for transmission) could resolve this large file upload failure and cli crash issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions