Java中用內(nèi)存映射處理大文件
在處理大文件時,如果利用普通的FileInputStream 或者FileOutputStream 抑或RandomAccessFile 來進(jìn)行頻繁的讀寫操作,都將導(dǎo)致進(jìn)程因頻繁讀寫外存而降低速度.如下為一個對比實驗。
- package test;
- import java.io.BufferedInputStream;
- import java.io.FileInputStream;
- import java.io.FileNotFoundException;
- import java.io.IOException;
- import java.io.RandomAccessFile;
- import java.nio.MappedByteBuffer;
- import java.nio.channels.FileChannel;
- public class Test {
- public static void main(String[] args) {
- try {
- FileInputStream fis=new FileInputStream("/home/tobacco/test/res.txt");
- int sum=0;
- int n;
- long t1=System.currentTimeMillis();
- try {
- while((n=fis.read())>=0){
- sum+=n;
- }
- } catch (IOException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- }
- long t=System.currentTimeMillis()-t1;
- System.out.println("sum:"+sum+" time:"+t);
- } catch (FileNotFoundException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- }
- try {
- FileInputStream fis=new FileInputStream("/home/tobacco/test/res.txt");
- BufferedInputStream bis=new BufferedInputStream(fis);
- int sum=0;
- int n;
- long t1=System.currentTimeMillis();
- try {
- while((n=bis.read())>=0){
- sum+=n;
- }
- } catch (IOException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- }
- long t=System.currentTimeMillis()-t1;
- System.out.println("sum:"+sum+" time:"+t);
- } catch (FileNotFoundException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- }
- MappedByteBuffer buffer=null;
- try {
- buffer=new RandomAccessFile("/home/tobacco/test/res.txt","rw").getChannel().map(FileChannel.MapMode.READ_WRITE, 0, 1253244);
- int sum=0;
- int n;
- long t1=System.currentTimeMillis();
- for(int i=0;i<1253244;i++){
- n=0x000000ff&buffer.get(i);
- sum+=n;
- }
- long t=System.currentTimeMillis()-t1;
- System.out.println("sum:"+sum+" time:"+t);
- } catch (FileNotFoundException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- } catch (IOException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- }
- }
- }
測試文件為一個大小為1253244字節(jié)的文件。測試結(jié)果:
- sum:220152087 time:1464
- sum:220152087 time:72
- sum:220152087 time:25
說明讀數(shù)據(jù)無誤。刪去其中的數(shù)據(jù)處理部分。
- package test;
- import java.io.BufferedInputStream;
- import java.io.FileInputStream;
- import java.io.FileNotFoundException;
- import java.io.IOException;
- import java.io.RandomAccessFile;
- import java.nio.MappedByteBuffer;
- import java.nio.channels.FileChannel;
- public class Test {
- public static void main(String[] args) {
- try {
- FileInputStream fis=new FileInputStream("/home/tobacco/test/res.txt");
- int sum=0;
- int n;
- long t1=System.currentTimeMillis();
- try {
- while((n=fis.read())>=0){
- //sum+=n;
- }
- } catch (IOException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- }
- long t=System.currentTimeMillis()-t1;
- System.out.println("sum:"+sum+" time:"+t);
- } catch (FileNotFoundException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- }
- try {
- FileInputStream fis=new FileInputStream("/home/tobacco/test/res.txt");
- BufferedInputStream bis=new BufferedInputStream(fis);
- int sum=0;
- int n;
- long t1=System.currentTimeMillis();
- try {
- while((n=bis.read())>=0){
- //sum+=n;
- }
- } catch (IOException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- }
- long t=System.currentTimeMillis()-t1;
- System.out.println("sum:"+sum+" time:"+t);
- } catch (FileNotFoundException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- }
- MappedByteBuffer buffer=null;
- try {
- buffer=new RandomAccessFile("/home/tobacco/test/res.txt","rw").getChannel().map(FileChannel.MapMode.READ_WRITE, 0, 1253244);
- int sum=0;
- int n;
- long t1=System.currentTimeMillis();
- for(int i=0;i<1253244;i++){
- //n=0x000000ff&buffer.get(i);
- //sum+=n;
- }
- long t=System.currentTimeMillis()-t1;
- System.out.println("sum:"+sum+" time:"+t);
- } catch (FileNotFoundException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- } catch (IOException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- }
- }
- }
測試結(jié)果:
- sum:0 time:1458
- sum:0 time:67
- sum:0 time:8
由此可見,將文件部分或者全部映射到內(nèi)存后進(jìn)行讀寫,速度將提高很多。
這是因為內(nèi)存映射文件首先將外存上的文件映射到內(nèi)存中的一塊連續(xù)區(qū)域,被當(dāng)成一個字節(jié)數(shù)組進(jìn)行處理,讀寫操作直接對內(nèi)存進(jìn)行操作,而后再將內(nèi)存區(qū)域重新映射到外存文件,這就節(jié)省了中間頻繁的對外存進(jìn)行讀寫的時間,大大降低了讀寫時間。
原文鏈接:http://blog.csdn.net/tobacco5648/article/details/7679105