zoukankan      html  css  js  c++  java
  • How to remove duplicate lines in a large text file?

    How would you remove duplicate lines from a file that is  much too large to fit in memory? The duplicate lines are not necessarily adjacent, and say the file is 10 times bigger than RAM.

    A better solution is to use HashSet to store each line of input.txt. As set ignores duplicate values, so while storing a line, check if it already present in hashset. Write it to output.txt only if not present in hashset.

    Java:

    // Efficient Java program to remove 
    // duplicates from input.txt and  
    // save output to output.txt 
      
    import java.io.*; 
    import java.util.HashSet; 
      
    public class FileOperation 
    { 
        public static void main(String[] args) throws IOException  
        { 
            // PrintWriter object for output.txt 
            PrintWriter pw = new PrintWriter("output.txt"); 
              
            // BufferedReader object for input.txt 
            BufferedReader br = new BufferedReader(new FileReader("input.txt")); 
              
            String line = br.readLine(); 
              
            // set store unique values 
            HashSet<String> hs = new HashSet<String>(); 
              
            // loop for each line of input.txt 
            while(line != null) 
            { 
                // write only if not 
                // present in hashset 
                if(hs.add(line)) 
                    pw.println(line); 
                  
                line = br.readLine(); 
                  
            } 
              
            pw.flush(); 
              
            // closing resources 
            br.close(); 
            pw.close(); 
              
            System.out.println("File operation performed successfully"); 
        } 
    } 
    

      

  • 相关阅读:
    redis总结
    java程序启动脚本
    mysql生成百万测试数据脚本
    java分布式锁的实现及在springboot中的应用
    mysql使用总结
    一个java实现代码(服务)编排的思路(未完)
    sentinel自定义统一限流降级处理
    shell学习
    oracle查看被锁的事务进程sql
    Sql查询两个时间段有重叠的记录
  • 原文地址:https://www.cnblogs.com/lightwindy/p/9650718.html
Copyright © 2011-2022 走看看