How to merge JSON files in a directory using shell scripting

In this blog I’ll share basic steps to merge multiple JSON files residing in a directory to a single JSON file of similar structure.

Pre-requisite – We will be using jq libraries make sure it is installed to your linux system.

Use Case : We have 1000 JSON files in a directory of same structure. Reading each JSON record from file and sending data to target was time consuming. The target system did support bulk update but each JSON file were having 2-3 records only.

This was time consuming and total process would take around 3-4 hours to complete.

Solution: Since , bulk upsert is supported by the target system , we wrote a script which will merge the JSON files in a batch of 20 . For eg. 20 JSON files to 1 JSON file . The batch count is a configurable value and can be updated as needed. The merge process for all 1000 files would take only 1-10 seconds and processing of the merged files will get completed within 10 minutes.

Snippet used to merge the files –

list=$(find $targetDir -type f -name "*.JSON" | split -d -l 20 - $date"_jsonFiles")
		for file in $date"_jsonFiles"*;
		do
			jq -s '[.[][]]' $(cat $file) > $targetDir/$date_"merged"${file##jsonFiles}.json;
		done
		cd $targetDir
		rm *.JSON

In the above code $targetDir – > is the variable which stores directory path.
We are splitting the files in batch of 20 and placing into a new file which holds the name of each 20 files.
Inside loop we are opening each file and merging them to one file with help of jq library.
The new merged file can then be read and sent to target system.

Reference – https://www.baeldung.com/linux/jq-command-json