How To Filter Lines By Column In Python

I need to filter some lines of a .csv file: 2017/06/07 10:42:35,THREAT,url,192.168.1.100,52.25.xxx.xxx,Rule-VWIRE-03,13423523,,web-browsing,80,tcp,block-url 2017/06/07 10:43:35,THR

Solution 1:

You use str.find method, which returns index if found and -1 otherwise. In your case - if, for example, THREAT in line - it will return some non-zero number, but then you compare that number with string, which is obviously returns False. Also, you can union those if statements.

So, taking into account the above - your if statements should be:

if col[1] == "THREAT"or col[3] in ["192.168.1.100", "192.168.1.101"]:
    f.write(line)

In addition - i don't understand, why you use raw_input on each iteration and never use again that value?

I suggest you use this little optimized code:

import csv  # not used in provide snippet, could be deleted

file_log = open("file.log", 'r')  # better to use absoulete path 
filtered_log = open("column", 'w')  # same as previousfor line in file:  # no need to read entire file, just iterate over it line by line directly
    col = line.split(',')
    if col and (col[1] == "THREAT"or col[3] in ["192.168.1.100", "192.168.1. 101"]):
        filtered_log.write(line)

file_log.close()
filtered_log.close()

Solution 2:

Python's csv module provides a reader object which can be used to iterate over a .csv file lines.

In each line, you can extract column by it's index and apply some comparation logic before printing the line.

This implementation will filter the file as needed:

import csv

ip_list = ['192.168.1.100', '192.168.1.101']
withopen('file.csv', 'r') as csvfile:
    reader = csv.reader(csvfile)
    for line in reader:
        if (line[1]=="THREAT") and (line[3] in ip_list):
            print(','.join(line))

As you can see, this implementation stores the ips in a list for comparing them using the python's in operator.