JAVA文本去重查重判断文件重复工具程序源代码 查重是指对文本、论文、作业等进行重复率检测，以防止学术不端和抄袭 查重主要是通过

package javatest; import java.io.File; import java.io.FileInputStream; import java.io.FileNotFoundException; import java.io.FileOutputStream; import java.io.IOException; import java.io.InputStream; import java.text.DateFormat; import java.text.SimpleDateFormat; import java.util.ArrayList; import java.util.Calendar; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Properties; import javax.swing.JLabel; import javax.swing.JProgressBar; import org.apache.log4j.Logger; import org.apache.log4j.PropertyConfigurator; public class Progress extends Thread { private Logger logger = Logger.getLogger(JProgressBarTest.class);// 获取日志记录器，这个记录器将负责控制日志信息 private JProgressBar progressBar;// 进度条 private JLabel lbl_progress;// 进度文字 public int current;// 当前进行的文件索引 public int totalCount;// 文件总数 private String sandBox_dir;// 沙箱目录 private String target_dir;// 目标目录 public Progress(JProgressBar progressBar, JLabel lbl_progress, String target_dir, String sandBox_dir) { this.progressBar = progressBar; this.lbl_progress = lbl_progress; this.target_dir = target_dir; this.sandBox_dir = sandBox_dir; initLog(); } /** * 初始化日志文件 */ private void initLog() { File directory = new File("DoSameFile");// 设定为当前文件夹 String path = directory.getAbsolutePath(); Properties prop = new Properties(); Calendar instance = Calendar.getInstance(); DateFormat format = new SimpleDateFormat("yyyyMMddHHmmss"); try { prop.load(this.getClass().getResourceAsStream("/log4j.properties")); // 加载log4j.properties prop.setProperty("log4j.appender.D.File", "c:\\DoSameFile" + format.format(instance.getTime()) + ".log"); // 设置日志文件的输出路径 PropertyConfigurator.configure(prop); // 加载配置项 } catch (Exception e) { logger.info("", e); logger.info( "log4j.appender.D.File===" + path + File.separatorChar + prop.getProperty("log4j.appender.D.File")); } } /** * 获取待处理文件总数 * * @param dir * @return */ private int getFilesCount(String dir) { int count = 0; File file = new File(dir); File[] childrenFiles = file.listFiles(); for (File childFile : childrenFiles) if (childFile.isDirectory()) { count += getFilesCount(childFile.getAbsolutePath()); } else count++; return count; } /** * 新建沙箱目录 */ private void createSandBox() { File file = new File(sandBox_dir); if (!file.exists()) { file.mkdirs(); } } /** * 获取沙箱文件名称 * * @param name * @return */ private String initFileName(String name) { String ret = ""; int i = 0; int index = name.indexOf("."); String suffix = name.substring(index); String fileName = name.substring(0, index); String newName = fileName; while (true) { ret = sandBox_dir + File.separatorChar + newName + suffix; File file = new File(ret); if (!file.exists()) break; newName = fileName + ("(" + i++ + ")"); } return ret; } /** * 是否相同 * * @param file * @param f * @return */ private boolean isSame(File file, File f) { String fileStr = new String(getByte(file)); String fStr = new String(getByte(f)); return fileStr.equals(fStr); } private byte[] getByte(File file) { // 得到文件长度 long length = file.length(); if (length > 100 * 1024 * 1024) length = 100 * 1024 * 1024; byte[] b = new byte[(int) length]; InputStream in = null; try { in = new FileInputStream(file); try { in.read(b); } catch (IOException e) { e.printStackTrace(); } } catch (FileNotFoundException e) { e.printStackTrace(); return null; } finally { if (null != in) try { in.close(); } catch (IOException e) { e.printStackTrace(); } } return b; } /** * 复制文件 * * @param oldPath * @param newPath */ private void copyFile(String oldPath, String newPath) { InputStream inStream = null; FileOutputStream fs = null; try { int byteread = 0; File oldfile = new File(oldPath); if (oldfile.exists()) { // 文件存在时 inStream = new FileInputStream(oldPath); // 读入原文件 fs = new FileOutputStream(newPath); byte[] buffer = new byte[1444]; while ((byteread = inStream.read(buffer)) != -1) fs.write(buffer, 0, byteread); } } catch (Exception e) { System.out.println("复制单个文件操作出错"); e.printStackTrace(); } finally { if (inStream != null) try { inStream.close(); } catch (IOException e) { e.printStackTrace(); } if (fs != null) try { fs.close(); } catch (IOException e) { e.printStackTrace(); } } } /** * 判断转移重复文件 * * @param fileMap * @param file */ private void removeSameFile(Map<Long, List<String>> fileMap, File file) { if (file.isDirectory()) { File[] childrenFiles = file.listFiles(); for (File childFile : childrenFiles) { removeSameFile(fileMap, childFile); } } else { logger.info("准备处理文件" + file.getAbsolutePath()); Long fileLength = file.length(); List<String> fileList = fileMap.get(fileLength); if (fileList == null) fileList = new ArrayList<String>(); boolean same = false; for (String filePath : fileList) { File f = new File(filePath); same = isSame(file, f); if (same && needSandBox()) { logger.info(file.getAbsolutePath() + "与" + f.getAbsolutePath() + "相同"); String newpath = initFileName(file.getName()); logger.info("复制文件：" + file.getAbsolutePath() + ">>" + newpath); copyFile(file.getAbsolutePath(), newpath); logger.info("删除文件：" + file.getAbsolutePath()); file.delete(); break; } } if (!same) fileList.add(file.getAbsolutePath()); fileMap.put(fileLength, fileList); current++; progressBar.setValue(100 * current / totalCount); // 进度值 lbl_progress.setText("进度信息：" + current + "/" + totalCount); } } public void run() { logger.info("准备处理：" + target_dir); if (needSandBox()) logger.info("沙箱目录为：" + sandBox_dir); else logger.info("未设置沙箱目录"); totalCount = getFilesCount(target_dir); Map<Long, List<String>> fileMap = new HashMap<Long, List<String>>(); File file = new File(target_dir); if (needSandBox()) createSandBox(); removeSameFile(fileMap, file); progressBar.setValue(100 * current / totalCount); // 进度值 lbl_progress.setText("进度信息：" + current + "/" + totalCount); logger.info("处理完毕！"); // Set<Entry<Long, List<String>>> entrySet = fileMap.entrySet(); // for (Entry<Long, List<String>> en : entrySet) { // List<String> value = en.getValue(); // for (String str : value) // System.out.println(str); // } progressBar.setIndeterminate(false); // 采用确定的进度条 progressBar.setString("升级完成."); // 提示信息 } private boolean needSandBox() { return sandBox_dir != null && !sandBox_dir.equals(""); } }