Dynamically Generate Json File Keys And Write On S3

I am generating json file using python script, but the problem after for loop it is only picking up last updated value. Below is the code. 1 read watermark file: watermark_file = c

Solution 1:

The cause of the erroneous behavior of your code is that watermark_json = {} is inside the for n in range(len(contents)): loop. It should be located before the for loop.

And the code should be little bit further changed to get the output you want.

You can try the following code:

watermark_file = config_dict["watermark_file"] + "watermark.json"
current_date, flag = read_watermark_file(config_dict.get("out_bucket"), watermark_file)
contents = list_s3_files(opt={'Bucket': config_dict['inp_bucket'], 'Prefix': config_dict['inp_location']})
print("contents :", contents)
watermark_json = {'loop': {}}  # <- This line is changedfor n inrange(len(contents)):
    loop = {}
    zipped_fileName = contents[n].split("/")[-1]
    therapeutic_area = re.match("(.*?)_(.*)", zipped_fileName)[1]
    indication = re.match("(.*?)_(.*?)_(.*)", zipped_fileName)[2]
    print("value of n:", n)
    loop['item_' + str(n)] = {"therapeutic_area": therapeutic_area,
                              "indication": indication,
                              "s3_path": config_dict["inp_location"] + therapeutic_area + "/" + indication + "/"}
    print("loop :", loop)
    watermark_json['loop'].update(loop)  # <- This line is changedprint("watermark_json :", watermark_json)
# update water mark fileprint("watermark_file :", watermark_file)
watermark_json['date_dir'] = datetime.datetime.now().strftime("%Y/%m/%d/%H") + "/"
watermark_json['processed_flag'] = Falseprint("final watermark file ", watermark_json)
# refresh watermark file
write_to_s3(config_dict['out_bucket'], watermark_file, watermark_json, config_dict)