pilot-chunk-transfer

Large file transfer with automatic chunking, resume capability, and per-chunk integrity verification. Breaks large files into manageable chunks, tracks transfer progress, and enables resuming interrupted transfers.

Commands

Send file with chunking

FILE="/path/to/large-file.iso"
DEST="1:0001.AAAA.BBBB"
CHUNK_SIZE=$((1024 * 1024))  # 1MB

FILENAME=$(basename "$FILE")
FILESIZE=$(stat -f %z "$FILE" 2>/dev/null || stat -c %s "$FILE")
TOTAL_CHUNKS=$(( (FILESIZE + CHUNK_SIZE - 1) / CHUNK_SIZE ))

# Send metadata
pilotctl --json send-message "$DEST" \
  --data "{\"type\":\"chunk_transfer_start\",\"filename\":\"$FILENAME\",\"size\":$FILESIZE,\"total_chunks\":$TOTAL_CHUNKS}"

# Send chunks
for ((i=0; i<TOTAL_CHUNKS; i++)); do
  dd if="$FILE" bs="$CHUNK_SIZE" skip="$i" count=1 2>/dev/null > "/tmp/chunk_$i.dat"
  CHUNK_HASH=$(md5sum "/tmp/chunk_$i.dat" | cut -d' ' -f1)

  pilotctl --json send-message "$DEST" \
    --data "{\"type\":\"chunk_metadata\",\"chunk_id\":$i,\"hash\":\"$CHUNK_HASH\"}"

  pilotctl --json send-file "$DEST" "/tmp/chunk_$i.dat"
  rm "/tmp/chunk_$i.dat"
done

# Send completion
pilotctl --json send-message "$DEST" \
  --data "{\"type\":\"chunk_transfer_complete\",\"filename\":\"$FILENAME\"}"

Receive and reassemble chunks

RECV_DIR="$HOME/.pilot/chunk-recv"
mkdir -p "$RECV_DIR"

INBOX=$(pilotctl --json inbox)
echo "$INBOX" | jq -c '.messages[]' | while read -r msg; do
  TYPE=$(echo "$msg" | jq -r '.type')

  case "$TYPE" in
    chunk_transfer_start)
      FILENAME=$(echo "$msg" | jq -r '.filename')
      mkdir -p "$RECV_DIR/$FILENAME.chunks"
      ;;

    chunk_metadata)
      CHUNK_ID=$(echo "$msg" | jq -r '.chunk_id')
      EXPECTED_HASH=$(echo "$msg" | jq -r '.hash')
      # Verify chunk hash after receiving file
      ;;

    chunk_transfer_complete)
      FILENAME=$(echo "$msg" | jq -r '.filename')
      cat "$RECV_DIR/$FILENAME.chunks"/chunk_* > "$RECV_DIR/$FILENAME"
      rm -rf "$RECV_DIR/$FILENAME.chunks"
      echo "Reassembled: $RECV_DIR/$FILENAME"
      ;;
  esac
done

Resume interrupted transfer

STATE_FILE="/tmp/transfer_state.json"

if [ -f "$STATE_FILE" ]; then
  START_CHUNK=$(jq -r ".\"$FILENAME\".last_chunk // 0" "$STATE_FILE")
else
  START_CHUNK=0
fi

# Continue from START_CHUNK and update state after each chunk

Workflow Example

# Send large file in chunks with resume capability
./pilot-chunk-transfer.sh send /path/to/large.iso 1:0001.AAAA.BBBB

# Receive in background
./pilot-chunk-transfer.sh receive

Dependencies

Requires pilot-protocol skill with running daemon, jq for JSON parsing, dd for chunk extraction, md5sum for integrity verification, and bc for progress calculations.