Master Linux File Splitting and Merging with split and cat

Due to network transfer limits, large files often need to be split on Linux, transferred in smaller pieces, and then merged back.

File Splitting – split

The split command makes dividing large files easy.

Syntax

-a  # specify suffix length (default 2: aa, ab...)
-d  # use numeric suffixes
-l  # split by number of lines (default 1000)
-b  # binary split with size units (k/m)
-C  # split by size while preserving line integrity

split [-a] [-d] [-l <lines>] [-b <bytes>] [-C <bytes>] [input_file] [output_prefix]

Examples

# split by lines
$ split -l 300000 users.sql /data/users_

# numeric suffixes
$ split -d -l 300000 users.sql /data/users_

# split by byte size
$ split -d -b 100m users.sql /data/users_

Help

# help
$ split --help
Usage: split [OPTION]... [FILE [PREFIX]]
Output pieces of FILE to PREFIXaa, PREFIXab, ...
Default size is 1000 lines, default PREFIX is 'x'.

-a, --suffix-length=N   generate suffixes of length N (default 2)
--additional-suffix=SUFFIX  append an additional suffix to file names
-b, --bytes=SIZE        put SIZE bytes per output file
-C, --line-bytes=SIZE  put at most SIZE bytes of records per output file
-d, --numeric-suffixes[=FROM]  use numeric suffixes starting at 0
-e, --elide-empty-files  do not generate empty output files
--filter=COMMAND        write to shell COMMAND; file name is $FILE
-l, --lines=NUMBER     put NUMBER lines per output file
-n, --number=CHUNKS    generate CHUNKS output files
-t, --separator=SEP    use SEP instead of newline as record separator
-u, --unbuffered       copy input to output without buffering
--verbose              print diagnostic before each output file is opened
--help                 display this help and exit
--version              output version information and exit

SIZE may be an integer with optional unit (e.g., 10K = 10*1024). Units: K,M,G,T,P,E,Z,Y (powers of 1024) or KB,MB,... (powers of 1000).

File Merging – cat

The cat command easily concatenates multiple split files back into a single file.

Syntax

-n  # show line numbers
-e  # display $ at end of each line
-t  # show TAB characters (^I)
cat [-n] [-e] [-t] [output_file]

Example

# merge files
$ cat /data/users_* > users.sql

Help

# help
$ cat --help
Usage: cat [OPTION]... [FILE]...
Concatenate FILE(s) to standard output.

-A, --show-all          equivalent to -vET
-b, --number-nonblank   number nonempty output lines, overrides -n
-e, --show-ends          display $ at end of each line
-n, --number            number all output lines
-s, --squeeze-blank     suppress repeated empty output lines
-t, --show-tabs          display TAB characters as ^I
-u                       (ignored)
-v, --show-nonprinting   use ^ and M- notation
--help                  display this help and exit
--version               output version information and exit