zoukankan      html  css  js  c++  java
  • 断点续传 (HTTP) 归纳

    由于最近项目中要上传较大的文件,基于公司原有的底层框架的局限性,对于大文件的传输都束手无策,基于文件传输的安全性,考虑用断点续传(HTTP)以及FTP上传两种方式实现下面归纳下HTTP续传和FTP上传[FTP上传后续附上]

    实现断点续传 (HTTP)

    断点续传的原理:

    其实断点续传的原理很简单,就是在 Http 的请求上和一般的下载有所不同而已。 
    打个比方,浏览器请求服务器上的一个文时,所发出的请求如下: 
    假设服务器域名为 wwww.sjtu.edu.cn,文件名为 down.zip。 
    GET /down.zip HTTP/1.1 
    Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms- 
    excel, application/msword, application/vnd.ms-powerpoint, */* 
    Accept-Language: zh-cn 
    Accept-Encoding: gzip, deflate 
    User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) 
    Connection: Keep-Alive

    服务器收到请求后,按要求寻找请求的文件,提取文件的信息,然后返回给浏览器,返回信息如下:

    200 
    Content-Length=106786028 
    Accept-Ranges=bytes 
    Date=Mon, 30 Apr 2001 12:56:11 GMT 
    ETag=W/"02ca57e173c11:95b" 
    Content-Type=application/octet-stream 
    Server=Microsoft-IIS/5.0 
    Last-Modified=Mon, 30 Apr 2001 12:56:11 GMT

    所谓断点续传,也就是要从文件已经下载的地方开始继续下载。所以在客户端浏览器传给 Web 服务器的时候要多加一条信息 -- 从哪里开始。 
    下面是用自己编的一个"浏览器"来传递请求信息给 Web 服务器,要求从 2000070 字节开始。 
    GET /down.zip HTTP/1.0 
    User-Agent: NetFox 
    RANGE: bytes=2000070- 
    Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2

    仔细看一下就会发现多了一行 RANGE: bytes=2000070- 
    这一行的意思就是告诉服务器 down.zip 这个文件从 2000070 字节开始传,前面的字节不用传了。 
    服务器收到这个请求以后,返回的信息如下: 
    206 
    Content-Length=106786028 
    Content-Range=bytes 2000070-106786027/106786028 
    Date=Mon, 30 Apr 2001 12:55:20 GMT 
    ETag=W/"02ca57e173c11:95b" 
    Content-Type=application/octet-stream 
    Server=Microsoft-IIS/5.0 
    Last-Modified=Mon, 30 Apr 2001 12:55:20 GMT

    和前面服务器返回的信息比较一下,就会发现增加了一行: 
    Content-Range=bytes 2000070-106786027/106786028 
    返回的代码也改为 206 了,而不再是 200 了。

    知道了以上原理,就可以进行断点续传的编程了。

    实现断点续传的关键几点:

    1. (1) 用什么方法实现提交 RANGE: bytes=2000070-。 
      当然用最原始的 Socket 是肯定能完成的,不过那样太费事了,其实 Java 的 net 包中提供了这种功能。代码如下: 

      URL url = new URL("http://www.sjtu.edu.cn/down.zip"); 
      HttpURLConnection httpConnection = (HttpURLConnection)url.openConnection(); 

      // 设置 User-Agent 
      httpConnection.setRequestProperty("User-Agent","NetFox"); 
      // 设置断点续传的开始位置 
      httpConnection.setRequestProperty("RANGE","bytes=2000070"); 
      // 获得输入流 
      InputStream input = httpConnection.getInputStream(); 

      从输入流中取出的字节流就是 down.zip 文件从 2000070 开始的字节流。 大家看,其实断点续传用 Java 实现起来还是很简单的吧。 接下来要做的事就是怎么保存获得的流到文件中去了。

    2. 保存文件采用的方法。 
      我采用的是 IO 包中的 RandAccessFile 类。 
      操作相当简单,假设从 2000070 处开始保存文件,代码如下: 
      RandomAccess oSavedFile = new RandomAccessFile("down.zip","rw"); 
      long nPos = 2000070; 
      // 定位文件指针到 nPos 位置 
      oSavedFile.seek(nPos); 
      byte[] b = new byte[1024]; 
      int nRead; 
      // 从输入流中读入字节流,然后写到文件中 
      while((nRead=input.read(b,0,1024)) > 0) 

      oSavedFile.write(b,0,nRead); 
      }

    怎么样,也很简单吧。 接下来要做的就是整合成一个完整的程序了。包括一系列的线程控制等等。

    断点续传内核的实现:

    主要用了 6 个类,包括一个测试类。 

    SiteFileFetch.java 负责整个文件的抓取,控制内部线程 (FileSplitterFetch 类 )。 
    FileSplitterFetch.java 负责部分文件的抓取。 
    FileAccess.java 负责文件的存储。 
    SiteInfoBean.java 要抓取的文件的信息,如文件保存的目录,名字,抓取文件的 URL 等。 
    Utility.java 工具类,放一些简单的方法。

    SiteFileFetch.java 
      1 /* 
      2  /*
      3  * SiteFileFetch.java 
      4  */ 
      5  package NetFox; 
      6  import java.io.*; 
      7  import java.net.*; 
      8  public class SiteFileFetch extends Thread { 
      9  SiteInfoBean siteInfoBean = null; // 文件信息 Bean 
     10  long[] nStartPos; // 开始位置
     11  long[] nEndPos; // 结束位置
     12  FileSplitterFetch[] fileSplitterFetch; // 子线程对象
     13  long nFileLength; // 文件长度
     14  boolean bFirst = true; // 是否第一次取文件
     15  boolean bStop = false; // 停止标志
     16  File tmpFile; // 文件下载的临时信息
     17  DataOutputStream output; // 输出到文件的输出流
     18  public SiteFileFetch(SiteInfoBean bean) throws IOException 
     19  { 
     20  siteInfoBean = bean; 
     21  //tmpFile = File.createTempFile ("zhong","1111",new File(bean.getSFilePath())); 
     22  tmpFile = new File(bean.getSFilePath()+File.separator + bean.getSFileName()+".info");
     23  if(tmpFile.exists ()) 
     24  { 
     25  bFirst = false; 
     26  read_nPos(); 
     27  } 
     28  else 
     29  { 
     30  nStartPos = new long[bean.getNSplitter()]; 
     31  nEndPos = new long[bean.getNSplitter()]; 
     32  } 
     33  } 
     34  public void run() 
     35  { 
     36  // 获得文件长度
     37  // 分割文件
     38  // 实例 FileSplitterFetch 
     39  // 启动 FileSplitterFetch 线程
     40  // 等待子线程返回
     41  try{ 
     42  if(bFirst) 
     43  { 
     44  nFileLength = getFileSize(); 
     45  if(nFileLength == -1) 
     46  { 
     47  System.err.println("File Length is not known!"); 
     48  } 
     49  else if(nFileLength == -2) 
     50  { 
     51  System.err.println("File is not access!"); 
     52  } 
     53  else 
     54  { 
     55  for(int i=0;i<nStartPos.length;i++) 
     56  { 
     57  nStartPos[i] = (long)(i*(nFileLength/nStartPos.length)); 
     58  } 
     59  for(int i=0;i<nEndPos.length-1;i++) 
     60  { 
     61  nEndPos[i] = nStartPos[i+1]; 
     62  } 
     63  nEndPos[nEndPos.length-1] = nFileLength; 
     64  } 
     65  } 
     66  // 启动子线程
     67  fileSplitterFetch = new FileSplitterFetch[nStartPos.length]; 
     68  for(int i=0;i<nStartPos.length;i++) 
     69  { 
     70  fileSplitterFetch[i] = new FileSplitterFetch(siteInfoBean.getSSiteURL(), 
     71  siteInfoBean.getSFilePath() + File.separator + siteInfoBean.getSFileName(), 
     72  nStartPos[i],nEndPos[i],i); 
     73  Utility.log("Thread " + i + " , nStartPos = " + nStartPos[i] + ", nEndPos = " 
     74  + nEndPos[i]); 
     75  fileSplitterFetch[i].start(); 
     76  } 
     77  // fileSplitterFetch[nPos.length-1] = new FileSplitterFetch(siteInfoBean.getSSiteURL(),
     78  siteInfoBean.getSFilePath() + File.separator 
     79  + siteInfoBean.getSFileName(),nPos[nPos.length-1],nFileLength,nPos.length-1); 
     80  // Utility.log("Thread " +(nPos.length-1) + ",nStartPos = "+nPos[nPos.length-1]+",
     81  nEndPos = " + nFileLength); 
     82  // fileSplitterFetch[nPos.length-1].start(); 
     83  // 等待子线程结束
     84  //int count = 0; 
     85  // 是否结束 while 循环
     86  boolean breakWhile = false; 
     87  while(!bStop) 
     88  { 
     89  write_nPos(); 
     90  Utility.sleep(500); 
     91  breakWhile = true; 
     92  for(int i=0;i<nStartPos.length;i++) 
     93  { 
     94  if(!fileSplitterFetch[i].bDownOver) 
     95  { 
     96  breakWhile = false; 
     97  break; 
     98  } 
     99  } 
    100  if(breakWhile) 
    101  break; 
    102  //count++; 
    103  //if(count>4) 
    104  // siteStop(); 
    105  } 
    106  System.err.println("文件下载结束!"); 
    107  } 
    108  catch(Exception e){e.printStackTrace ();} 
    109  } 
    110  // 获得文件长度
    111  public long getFileSize() 
    112  { 
    113  int nFileLength = -1; 
    114  try{ 
    115  URL url = new URL(siteInfoBean.getSSiteURL()); 
    116  HttpURLConnection httpConnection = (HttpURLConnection)url.openConnection ();
    117  httpConnection.setRequestProperty("User-Agent","NetFox"); 
    118  int responseCode=httpConnection.getResponseCode(); 
    119  if(responseCode>=400) 
    120  { 
    121  processErrorCode(responseCode); 
    122  return -2; //-2 represent access is error 
    123  } 
    124  String sHeader; 
    125  for(int i=1;;i++) 
    126  { 
    127  //DataInputStream in = new DataInputStream(httpConnection.getInputStream ()); 
    128  //Utility.log(in.readLine()); 
    129  sHeader=httpConnection.getHeaderFieldKey(i); 
    130  if(sHeader!=null) 
    131  { 
    132  if(sHeader.equals("Content-Length")) 
    133  { 
    134  nFileLength = Integer.parseInt(httpConnection.getHeaderField(sHeader)); 
    135  break; 
    136  } 
    137  } 
    138  else 
    139  break; 
    140  } 
    141  } 
    142  catch(IOException e){e.printStackTrace ();} 
    143  catch(Exception e){e.printStackTrace ();} 
    144  Utility.log(nFileLength); 
    145  return nFileLength; 
    146  } 
    147  // 保存下载信息(文件指针位置)
    148  private void write_nPos() 
    149  { 
    150  try{ 
    151  output = new DataOutputStream(new FileOutputStream(tmpFile)); 
    152  output.writeInt(nStartPos.length); 
    153  for(int i=0;i<nStartPos.length;i++) 
    154  { 
    155  // output.writeLong(nPos[i]); 
    156  output.writeLong(fileSplitterFetch[i].nStartPos); 
    157  output.writeLong(fileSplitterFetch[i].nEndPos); 
    158  } 
    159  output.close(); 
    160  } 
    161  catch(IOException e){e.printStackTrace ();} 
    162  catch(Exception e){e.printStackTrace ();} 
    163  } 
    164  // 读取保存的下载信息(文件指针位置)
    165  private void read_nPos() 
    166  { 
    167  try{ 
    168  DataInputStream input = new DataInputStream(new FileInputStream(tmpFile)); 
    169  int nCount = input.readInt(); 
    170  nStartPos = new long[nCount]; 
    171  nEndPos = new long[nCount]; 
    172  for(int i=0;i<nStartPos.length;i++) 
    173  { 
    174  nStartPos[i] = input.readLong(); 
    175  nEndPos[i] = input.readLong(); 
    176  } 
    177  input.close(); 
    178  } 
    179  catch(IOException e){e.printStackTrace ();} 
    180  catch(Exception e){e.printStackTrace ();} 
    181  } 
    182  private void processErrorCode(int nErrorCode) 
    183  { 
    184  System.err.println("Error Code : " + nErrorCode); 
    185  } 
    186  // 停止文件下载
    187  public void siteStop() 
    188  { 
    189  bStop = true; 
    190  for(int i=0;i<nStartPos.length;i++) 
    191  fileSplitterFetch[i].splitterStop(); 
    192  } 
    193  }
    FileSplitterFetch.java
      1 /* 
      2  **FileSplitterFetch.java 
      3  */ 
      4  package NetFox; 
      5  import java.io.*; 
      6  import java.net.*; 
      7  public class FileSplitterFetch extends Thread { 
      8  String sURL; //File URL 
      9  long nStartPos; //File Snippet Start Position 
     10  long nEndPos; //File Snippet End Position 
     11  int nThreadID; //Thread's ID 
     12  boolean bDownOver = false; //Downing is over 
     13  boolean bStop = false; //Stop identical 
     14  FileAccessI fileAccessI = null; //File Access interface 
     15  public FileSplitterFetch(String sURL,String sName,long nStart,long nEnd,int id)
     16  throws IOException 
     17  { 
     18  this.sURL = sURL; 
     19  this.nStartPos = nStart; 
     20  this.nEndPos = nEnd; 
     21  nThreadID = id; 
     22  fileAccessI = new FileAccessI(sName,nStartPos); 
     23  } 
     24  public void run() 
     25  { 
     26  while(nStartPos < nEndPos && !bStop) 
     27  { 
     28  try{ 
     29  URL url = new URL(sURL); 
     30  HttpURLConnection httpConnection = (HttpURLConnection)url.openConnection (); 
     31  httpConnection.setRequestProperty("User-Agent","NetFox"); 
     32  String sProperty = "bytes="+nStartPos+"-"; 
     33  httpConnection.setRequestProperty("RANGE",sProperty); 
     34  Utility.log(sProperty); 
     35  InputStream input = httpConnection.getInputStream(); 
     36  //logResponseHead(httpConnection); 
     37  byte[] b = new byte[1024]; 
     38  int nRead; 
     39  while((nRead=input.read(b,0,1024)) > 0 && nStartPos < nEndPos 
     40  && !bStop) 
     41  { 
     42  nStartPos += fileAccessI.write(b,0,nRead); 
     43  //if(nThreadID == 1) 
     44  // Utility.log("nStartPos = " + nStartPos + ", nEndPos = " + nEndPos); 
     45  } 
     46  Utility.log("Thread " + nThreadID + " is over!"); 
     47  bDownOver = true; 
     48  //nPos = fileAccessI.write (b,0,nRead); 
     49  } 
     50  catch(Exception e){e.printStackTrace ();} 
     51  } 
     52  } 
     53  // 打印回应的头信息
     54  public void logResponseHead(HttpURLConnection con) 
     55  { 
     56  for(int i=1;;i++) 
     57  { 
     58  String header=con.getHeaderFieldKey(i); 
     59  if(header!=null) 
     60  //responseHeaders.put(header,httpConnection.getHeaderField(header)); 
     61  Utility.log(header+" : "+con.getHeaderField(header)); 
     62  else 
     63  break; 
     64  } 
     65  } 
     66  public void splitterStop() 
     67  { 
     68  bStop = true; 
     69  } 
     70  } 
     71  
     72  /* 
     73  **FileAccess.java 
     74  */ 
     75  package NetFox; 
     76  import java.io.*; 
     77  public class FileAccessI implements Serializable{ 
     78  RandomAccessFile oSavedFile; 
     79  long nPos; 
     80  public FileAccessI() throws IOException 
     81  { 
     82  this("",0); 
     83  } 
     84  public FileAccessI(String sName,long nPos) throws IOException 
     85  { 
     86  oSavedFile = new RandomAccessFile(sName,"rw"); 
     87  this.nPos = nPos; 
     88  oSavedFile.seek(nPos); 
     89  } 
     90  public synchronized int write(byte[] b,int nStart,int nLen) 
     91  { 
     92  int n = -1; 
     93  try{ 
     94  oSavedFile.write(b,nStart,nLen); 
     95  n = nLen; 
     96  } 
     97  catch(IOException e) 
     98  { 
     99  e.printStackTrace (); 
    100  } 
    101  return n; 
    102  } 
    103  } 
    104  
    105  /* 
    106  **SiteInfoBean.java 
    107  */ 
    108  package NetFox; 
    109  public class SiteInfoBean { 
    110  private String sSiteURL; //Site's URL 
    111  private String sFilePath; //Saved File's Path 
    112  private String sFileName; //Saved File's Name 
    113  private int nSplitter; //Count of Splited Downloading File 
    114  public SiteInfoBean() 
    115  { 
    116  //default value of nSplitter is 5 
    117  this("","","",5); 
    118  } 
    119  public SiteInfoBean(String sURL,String sPath,String sName,int nSpiltter)
    120  { 
    121  sSiteURL= sURL; 
    122  sFilePath = sPath; 
    123  sFileName = sName; 
    124  this.nSplitter = nSpiltter; 
    125  } 
    126  public String getSSiteURL() 
    127  { 
    128  return sSiteURL; 
    129  } 
    130  public void setSSiteURL(String value) 
    131  { 
    132  sSiteURL = value; 
    133  } 
    134  public String getSFilePath() 
    135  { 
    136  return sFilePath; 
    137  } 
    138  public void setSFilePath(String value) 
    139  { 
    140  sFilePath = value; 
    141  } 
    142  public String getSFileName() 
    143  { 
    144  return sFileName; 
    145  } 
    146  public void setSFileName(String value) 
    147  { 
    148  sFileName = value; 
    149  } 
    150  public int getNSplitter() 
    151  { 
    152  return nSplitter; 
    153  } 
    154  public void setNSplitter(int nCount) 
    155  { 
    156  nSplitter = nCount; 
    157  } 
    158  } 
    159  
    160  /* 
    161  **Utility.java 
    162  */ 
    163  package NetFox; 
    164  public class Utility { 
    165  public Utility() 
    166  { 
    167  } 
    168  public static void sleep(int nSecond) 
    169  { 
    170  try{ 
    171  Thread.sleep(nSecond); 
    172  } 
    173  catch(Exception e) 
    174  { 
    175  e.printStackTrace (); 
    176  } 
    177  } 
    178  public static void log(String sMsg) 
    179  { 
    180  System.err.println(sMsg); 
    181  } 
    182  public static void log(int sMsg) 
    183  { 
    184  System.err.println(sMsg); 
    185  } 
    186  } 
    187  
    188  /* 
    189  **TestMethod.java 
    190  */ 
    191  package NetFox; 
    192  public class TestMethod { 
    193  public TestMethod() 
    194  { ///xx/weblogic60b2_win.exe 
    195  try{ 
    196  SiteInfoBean bean = new SiteInfoBean("http://localhost/xx/weblogic60b2_win.exe",
    197      "L:\temp","weblogic60b2_win.exe",5); 
    198  //SiteInfoBean bean = new SiteInfoBean("http://localhost:8080/down.zip","L:\temp",
    199      "weblogic60b2_win.exe",5); 
    200  SiteFileFetch fileFetch = new SiteFileFetch(bean); 
    201  fileFetch.start(); 
    202  } 
    203  catch(Exception e){e.printStackTrace ();} 
    204  } 
    205  public static void main(String[] args) 
    206  { 
    207  new TestMethod(); 
    208  } 
    209  }
     
  • 相关阅读:
    保证测试通过的ip正则,antdIP/IP段的校验方法,antd的textArea中可以输入多个以换行分隔的ip/IP段,并自动检测出错行的原因
    TP5接口出错只能返回500
    UDP服务只能本机访问问题
    有出现了找半天的小BUG
    PHP本地安装redis扩展
    MYSQL数据库和es数据库同步
    QQ互联应用申请失败
    axios跨域问题解决
    elastic和kibana安装心得
    自增运算符理解
  • 原文地址:https://www.cnblogs.com/visec479/p/3906517.html
Copyright © 2011-2022 走看看