File Splitter - Universal File Splitting Tool

Split large files into smaller, manageable chunks while preserving semantic structure.

Quick Start

python <skill_dir>/scripts/split_files.py --input <input_folder> --output <output_folder> [options]

Parameters

Examples

# Default 500KB split
python split_files.py --input "./corpus" --output "./corpus/chunks"

# Custom 200KB chunks
python split_files.py --input "./notes" --output "./notes/chunks" --max-size 204800 --min-size 153600

# JSON files only
python split_files.py --input "./data" --output "./data/out" --formats json

# Preview mode
python split_files.py --input "./data" --output "./data/out" --dry-run

Splitting Rules

JSON Files

• Splits at JSON array element boundaries
• Each chunk is a valid JSON array [...]
• Automatically extracts list values if top-level is an object
• Never cuts individual records in half

Markdown Files

• Splits at heading boundaries (# through ######)
• Each chunk maintains complete heading structure
• Never cuts content within a heading section

TXT Files

• Prefers splitting at paragraph boundaries (empty lines)
• Falls back to line-by-line splitting if no paragraphs exist
• Never cuts within a paragraph

Output Naming Convention

Format: {source_filename_without_extension}{9-digit_sequence_number}{extension}

Examples:

• dataset000000001.json
• dataset000000002.json
• notes000000001.md

Safety Features

1. Source File Preservation: Read-only access to source files; never deletes or modifies originals
2. Duplicate Detection: Automatically skips files that already have N-digit sequence suffixes to avoid re-splitting
3. Small File Skip: Files ≤ max-size are automatically skipped (no need to split)
4. Sequential Processing: Processes files one at a time to ensure stability
5. Data Validation: Compares total size/record count before and after splitting; reports verification results
6. UTF-8 Encoding: Forces UTF-8 for all read/write operations to avoid encoding issues on Windows

Notes

• Console may display garbled Chinese characters on Windows, but functionality is unaffected
• If a single data block/paragraph exceeds max-size, it becomes its own chunk (integrity takes priority over size limits)
• Output folder is automatically created if it doesn't exist
• License: MIT-0