How to Merge multiple files, removing duplicate values

0 min read 105 words

If you have multiple files and you want to remove duplicate values from, while creating a single file.

File1 (one.txt):

123
123
234
345

File2 (two.txt):

123
678
567
890

As you can see, the two files contain some duplicates. You can easily remove these using the following command:

sort -u one.txt two.txt

This will output:

123
234
345
567
678
890

You can as easily pipe this into a third file as follows:

sort -u one.txt two.txt > three.txt

This will take the first two files (you can add as many as you like), and pipe the output into a file called three.txt.

Simple!

Tags:
Andrew
Andrew

Andrew is a visionary software engineer and DevOps expert with a proven track record of delivering cutting-edge solutions that drive innovation at Ataiva.com. As a leader on numerous high-profile projects, Andrew brings his exceptional technical expertise and collaborative leadership skills to the table, fostering a culture of agility and excellence within the team. With a passion for architecting scalable systems, automating workflows, and empowering teams, Andrew is a sought-after authority in the field of software development and DevOps.

Tags

Recent Posts