join and split

The join and split commands solve different file-processing problems. join combines related records from two sorted text inputs, while split divides one input into a sequence of smaller files.

Joining Two Files by Their First Field

By default, join compares the first blank-separated field in exactly two input files. Consider these already sorted files:

people.txt:

1 John
2 Jane
3 Mary

surnames.txt:

1 Doe
2 Doe
3 Sue

Join records whose key fields are equal:

$ join people.txt surnames.txt
1 John Doe
2 Jane Doe
3 Mary Sue

The output contains the shared key once, then the remaining fields from the first and second files. join processes two files at a time; it does not accept three ordinary file operands as a three-way relational join.

CheckWith no field options, which records does join first.txt second.txt combine?

Lines whose first blank-separated fields are equal. Lines that occupy the same physical line number. Every line from the first file with every line from the second.

Sorting the Join Keys

Each input must be ordered by its join field using compatible comparison rules. For default field 1, prepare copies with sort -k 1,1:

$ LC_ALL=C sort -k 1,1 people-raw.txt > people.txt
$ LC_ALL=C sort -k 1,1 surnames-raw.txt > surnames.txt
$ LC_ALL=C join people.txt surnames.txt

Using the same locale for sorting and joining keeps collation rules consistent. Do not redirect a sort back to its own input pathname, because the shell would truncate that file first.

CheckWhat preparation does join normally require for reliable matching?

Both files must contain exactly the same number of physical lines. Both files must use filenames that sort next to each other alphabetically. Both files must be sorted by their respective join fields with compatible ordering.

Selecting Different Join Fields

Use -1 FIELD for the first file's key and -2 FIELD for the second file's key. Suppose the first input contains:

John 1
Jane 2
Mary 3

The second contains:

1 Doe
2 Doe
3 Sue

After sorting the first file by field 2 and the second by field 1, run:

$ join -1 2 -2 1 people.txt surnames.txt
1 John Doe
2 Jane Doe
3 Mary Sue

Use -t CHARACTER when a single nonblank character, such as :, separates fields. Options such as -a 1 or -a 2 can include unpaired lines from one input; the default output contains matched keys only.

CheckWhich options join field 2 of the first file to field 1 of the second?

-1 1 -2 2 -1 2 -2 1 -f 2 -d 1

Splitting by Line Count

split writes consecutive portions of one input to separate output files. It is not the inverse of a key-based join operation.

$ split large.txt

The default GNU behavior writes up to 1000 lines per output file and uses the prefix x, producing names such as xaa, xab, and xac.

Use -l NUMBER to choose a line count and add a final operand to choose the output prefix:

$ split -l 500 large.txt part-

This produces part-aa, part-ab, and so on, with at most 500 lines in each piece.

CheckWhich command splits large.txt into pieces of at most 500 lines named with the prefix part-?

split -b 500 large.txt part- split -l 500 large.txt part- join -l 500 large.txt part-

Splitting by Size

Use -b SIZE to divide input by byte size. GNU suffixes such as K, M, and G represent powers of 1024 in this context:

$ split -b 10M archive.bin chunk-

This requests pieces of 10 mebibytes except for a potentially smaller final piece. split does not create an archive manifest or reassembly metadata; preserve the ordering of suffixes and concatenate the pieces in order when reconstruction is appropriate.

CheckWhich command splits archive.bin into pieces of 10 MiB using the prefix chunk-?

split -l 10M archive.bin chunk- join -b 10M archive.bin chunk- split -b 10M archive.bin chunk-

To practice keyed joins and structured data processing, try these hands-on labs:

1. Linux join Command: File Joining - This lab provides a direct, hands-on introduction to the join command, allowing you to practice merging lines from two sorted text files based on a common field, just as discussed in the lesson.
2. Processing Employees Data - Apply your knowledge of join and other powerful Linux command-line utilities like awk to combine and process data from multiple sources, simulating a real-world data analysis scenario.

Summary

You can now combine sorted records or divide one input into ordered pieces.

1. Join exactly two files by equal key fields.
2. Sort both inputs consistently by their join keys.
3. Select nondefault key fields with -1 and -2.
4. Split by line count with -l.
5. Split by byte size with -b and a clear prefix.