centos下对文件某些特定字符串分组统计出现次数

假如现有数据：

 

{ "@timestamp": "2018-10-13T21:55:58+08:00","remote_addr": "100.120.34.3","referer": "-","request": "GET /api/gourd/activeupload?idfa=58237FA9-A1B3-4202-B5F3-9536983119E5&appid=1410137206&source=rehulu HTTP/1.1","status": 200,"request_time": 0.076,"cookie":"-","host":"cms.369wan.com","bytes": 48,"agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0","proxy_x_forwarded": "139.129.97.187,100.120.34.3" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.076 }
{ "@timestamp": "2018-10-13T21:56:06+08:00","remote_addr": "100.120.34.101","request": "GET /api/gourd/activeupload?idfa=E9D7F87A-9042-46B4-82E8-E5F64B74466B&appid=1410137206&source=rehulu HTTP/1.1",100.120.34.101" "upstr_addr": "127.0.0.1:9000","ups_resp_time": 0.076 }
{ "@timestamp": "2018-10-13T21:56:08+08:00","remote_addr": "100.120.34.29","request": "GET /api/gourd/activeupload?idfa=D5B924F3-7D25-4B52-BAE9-3270B08EA32D&appid=1410137206&source=rehulu HTTP/1.1","request_time": 0.075,100.120.34.29" "upstr_addr": "127.0.0.1:9000","ups_resp_time": 0.075 }
{ "@timestamp": "2018-10-13T21:56:10+08:00","remote_addr": "100.120.34.75","request": "GET /api/gourd/activeupload?idfa=D166459D-E823-4847-9094-6F4BF90625B2&appid=1410137206&source=rehulu HTTP/1.1","request_time": 0.078,100.120.34.75" "upstr_addr": "127.0.0.1:9000","ups_resp_time": 0.078 }
{ "@timestamp": "2018-10-13T21:56:18+08:00","remote_addr": "100.120.34.39","request": "GET /api/gourd/activeupload?idfa=08C65C3B-EED2-4A65-B0C1-67FC7FB78E18&appid=1410137206&source=rehulu HTTP/1.1","request_time": 0.082,100.120.34.39" "upstr_addr": "127.0.0.1:9000","ups_resp_time": 0.082 }
{ "@timestamp": "2018-10-13T21:56:31+08:00","remote_addr": "100.120.34.68","request_time": 0.079,100.120.34.68" "upstr_addr": "127.0.0.1:9000","ups_resp_time": 0.079 }

临时保存为tmp.log

awk -F ‘idfa=‘ ‘{print $2}‘ tmp.log 

 

出现如下结果：

58237FA9-A1B3-4202-B5F3-9536983119E5&appid=1410137206&source=rehulu HTTP/1.1","ups_resp_time": 0.076 }
E9D7F87A-9042-46B4-82E8-E5F64B74466B&appid=1410137206&source=rehulu HTTP/1.1","ups_resp_time": 0.076 }
D5B924F3-7D25-4B52-BAE9-3270B08EA32D&appid=1410137206&source=rehulu HTTP/1.1","ups_resp_time": 0.075 }

执行 

awk -F ‘idfa=‘ ‘{print $2}‘ tmp.log | awk -F ‘&source=‘ ‘{print $1}‘ 

出现如下结果：
58237FA9-A1B3-4202-B5F3-9536983119E5&appid=1410137206
E9D7F87A-9042-46B4-82E8-E5F64B74466B&appid=1410137206
D5B924F3-7D25-4B52-BAE9-3270B08EA32D&appid=1410137206

执行 

awk -F ‘idfa=‘ ‘{print $2}‘ tmp.log | awk -F ‘&source=‘ ‘{print $1}‘  | sort  (这一步将结果相同的放在一起)

出现如下结果：
58237FA9-A1B3-4202-B5F3-9536983119E5&appid=1410137206
E9D7F87A-9042-46B4-82E8-E5F64B74466B&appid=1410137206
D5B924F3-7D25-4B52-BAE9-3270B08EA32D&appid=1410137206

执行

awk -F ‘idfa=‘ ‘{print $2}‘ cms_\(2\).log | awk -F ‘&source=‘ ‘{print $1}‘ |sort| uniq -c出现最终结果(次数 和 各字符串)：1 58237FA9-A1B3-4202-B5F3-9536983119E5&appid=14101372061 E9D7F87A-9042-46B4-82E8-E5F64B74466B&appid=14101372061 D5B924F3-7D25-4B52-BAE9-3270B08EA32D&appid=1410137206