Using jq for JSON Log Extraction and Real‑Time Monitoring on the Command Line

JSON is a ubiquitous data format, often used for transmitting logs such as click and PV events; in many projects developers need to extract only the fields that have changed.

Instead of writing complex regular expressions, the author recommends the jq tool, which can format and query JSON directly from the shell.

Installation

Download the compressed package from the official repository:

https://github.com/stedolan/jq/releases/download/jq-1.5/jq-1.5.tar.gz

Copy it to a local directory, then compile and install:

./configure --disable-maintainer-mode && make && sudo make install

After a successful installation, you can run jq commands as shown below.

Basic field extraction

To extract the price field from a test log file:

Cat test.txt | awk -F '"open_ad_v1:stats:" ' '/10.x.x.x/{system("echo "$2" | jq .price");}'

This prints the price value for the specified IP address.

Using jq pipelines for multiple fields

When you need to check several fields (e.g., price and pid ) simultaneously, you can pipe jq output:

redis-cli -h 10.x.x.x -p 17801 monitor | awk -F '"open_ad_v1:stats:" ' '/10.x.x.x/{system("echo "$2" | jq \". | {price: .price, pid: .pid}\"");}'

Similarly, for PV logs you can extract rf and ip fields:

redis-cli -h 10.x.x.x -p 17802 monitor | awk -F '"open_ad_v1:stats:" ' '/10.10.10.10/{system("echo "$2" | jq .data[] | jq \". | {rf: .rf, ip: .ip}\"");}'

Processing complex combinelog files

Combinelog entries contain nested JSON arrays. To extract bidprice and cprice from each ad:

tailf /dev/shm/combineLog | awk -F 'DjShouZhuPv\t' '/10.x.x.x/{system("echo '\''"$2"'\'' | jq .ads[][] | jq \".| {bidprice: .bidprice, cprice: .cprice}\"")}'

To also include top‑level fields like ip and pvid :

tailf /dev/shm/combineLog | awk -F 'DjShouZhuPv\t' '/10.x.x.x/{system("echo '\''"$2"'\'' | jq \".| {ip, pvid, bidprice: .ads[][].bidprice, cprice: .ads[][].cprice}\"")}'

The results are displayed as screenshots (included in the original article) showing the extracted values.

Further resources

For a complete reference, visit the official jq manual at https://stedolan.github.io/jq/manual/v1.5 and try the online interpreter at https://jqplay.org/jq for interactive debugging.