-----Original Message-----
From: Maurik Holtrop [mailto:[log in to unmask]]
Sent: Thursday, July 11, 2013 12:34 PM
To: Graf, Norman A.; McCormick, Jeremy I.
Subject: Fwd: [Clas12_software] java file size test and jevio
Hello Norman and Jeremy,
I think that so far we've always kept the data files < 2GB. Here is another reason to keep doing so.
Just so we all know.
Cheers,
Maurik
Begin forwarded message:
From: David Heddle <[log in to unmask]>
Subject: [Clas12_software] java file size test and jevio
Date: July 11, 2013 1:28:33 PM EDT
To: [log in to unmask], Carl Timmer <[log in to unmask]>
The problem with files > 2GB in jevio is nasty. Java can "inherently" handle files greater than 2GB if they are read traditionally. It is the memory mapping that causes the problem. In jevio there is, for example, the call:
mappedByteBuffer = inputChannel.map(FileChannel.MapMode.READ_ONLY, 0L, sz);
This is creating the buffer, in memory, of a region of a file. The 0L argument is the offset (i.e., in this case the map begins at the beginning of the file) and the second argument is the length, which in jevio is essentially the size of the file.
The java documentation gives the weird bad news. The third argument (sz, in this case) is a long, but it is restricted to INTEGER.MAXINT (which is 2 GB). That's right: it is a long (64 bit) value that will throw an exception if it is bigger than the max for a 32 bit int.
To summarize: jevio assumes it can map the entire file to memory--but this will fail even if you have a 64 bit machine and sufficient (i.e. more than 2GB) of memory.
There is a request for a change at Oracle, but it doesn't seen to be high priority. They say the only workaround is to work your way through big files:
offset = 0L
while (!done) {
long sz = Math.min(remainingSize, INTEGER.MAXINT);
mappedByteBuffer = inputChannel.map(FileChannel.MapMode.READ_ONLY, offset, sz);
{do stuff}
offset += sz;
mappedByteBuffer = inputChannel.map(FileChannel.MapMode.READ_ONLY, 0L, sz);
}
I suspect that this would be a very nontrivial change to jevio withs lots of debugging. But I think we need to request it.
Note that we basically have to use memory mapped io--it is about 10x faster than traditional disk access IO.
--
David P. Heddle, Ph.D.
Associate Professor of Physics
Christopher Newport University
Newport News, VA 23606
757.594.8434 (CNU)
757.269.5719 (Jefferson Lab)
_______________________________________________
Clas12_software mailing list
[log in to unmask]
https://mailman.jlab.org/mailman/listinfo/clas12_software
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the HPS-SOFTWARE list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=HPS-SOFTWARE&A=1
