Breaking News: Grepper is joining You.com. Read the official announcement!

Python remove duplicate lines from a text file

Add Answer

Arif Muxtarov answered on September 19, 2020 Popularity 8/10 Helpfulness 3/10

answer Python remove duplicate lines from a text file

related txt file duplicate line remover python

related python delete duplicate lines in file

Python remove duplicate lines from a text file

Comment

import hashlib
def main():
    input_file = "in.txt"
    output_file = "out.txt"
    
    completed_hash = set()
    output_file = open(output_file, "w")
    
    for line in open(input_file,"r"):
        hashValue = hashlib.md5(line.strip().encode('utf-8')).hexdigest()
        
        if hashValue not in completed_hash:
            output_file.write(line)
            completed_hash.add(hashValue)
            
    output_file.close()


if __name__ == "__main__":
    main()

Popularity 8/10 Helpfulness 3/10 Language python

Source: www.codevscolor.com

Tags: file lines python text

Link to this answer
Share Copy Link

Contributed on Mar 16 2022

Arif Muxtarov

0 Answers Avg Quality 2/10

Closely Related Answers

txt file duplicate line remover python

Comment

lines_seen = set() # holds lines already seen

with open("file.txt", "r+") as f:
    d = f.readlines()
    f.seek(0)
    for i in d:
        if i not in lines_seen:
            f.write(i)
            lines_seen.add(i)
    f.truncate()

Popularity 9/10 Helpfulness 8/10 Language python

Source: stackoverflow.com

Tags: file

Link to this answer
Share Copy Link

Contributed on Sep 19 2020

Annoying Aardvark

0 Answers Avg Quality 2/10

python delete duplicate lines in file

Comment

with open("file.txt", "r") as txt_file:
  new_data = list(set(txt_file))
  return new_data

Popularity 9/10 Helpfulness 6/10 Language python

Source: Grepper

Tags: file

Link to this answer
Share Copy Link

Contributed on Nov 30 2020

Supermavster

0 Answers Avg Quality 2/10

Continue with Google

By continuing, I agree that I have read and agree to Greppers's Terms of Service and Privacy Policy.

Python remove duplicate lines from a text file

Contents

More Related Answers

Python remove duplicate lines from a text file

Closely Related Answers

txt file duplicate line remover python

python delete duplicate lines in file

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.