On 32-bit the memory mapping fails well before 2 GB. On 64-bit it still
fails at 2 GB. So the CODA folks understand this correctly.
On Thu, 11 Jul 2013, Maurik Holtrop wrote:
> Hi Sho,
>
> Is it actually resolved on 64-bit Java? In that case it is not an issue, and CODA & Dave Heddle should be informed of this.
>
> Best,
> Maurik
> On Jul 11, 2013, at 3:40 PM, Sho Uemura <[log in to unmask]> wrote:
>
>> This is something we found ourselves last year (and I should have told the CODA people, but I forgot). Running on 64-bit Java resolves this issue and for this reason we use 64-bit Java in production.
>>
>> On Thu, 11 Jul 2013, McCormick, Jeremy I. wrote:
>>
>>>
>>>
>>> -----Original Message-----
>>> From: Maurik Holtrop [mailto:[log in to unmask]]
>>> Sent: Thursday, July 11, 2013 12:34 PM
>>> To: Graf, Norman A.; McCormick, Jeremy I.
>>> Subject: Fwd: [Clas12_software] java file size test and jevio
>>>
>>> Hello Norman and Jeremy,
>>>
>>> I think that so far we've always kept the data files < 2GB. Here is another reason to keep doing so.
>>> Just so we all know.
>>>
>>> Cheers,
>>> Maurik
>>>
>>>
>>> Begin forwarded message:
>>>
>>>
>>> From: David Heddle <[log in to unmask]>
>>>
>>> Subject: [Clas12_software] java file size test and jevio
>>>
>>> Date: July 11, 2013 1:28:33 PM EDT
>>>
>>> To: [log in to unmask], Carl Timmer <[log in to unmask]>
>>>
>>>
>>> The problem with files > 2GB in jevio is nasty. Java can "inherently" handle files greater than 2GB if they are read traditionally. It is the memory mapping that causes the problem. In jevio there is, for example, the call:
>>>
>>>
>>> mappedByteBuffer = inputChannel.map(FileChannel.MapMode.READ_ONLY, 0L, sz);
>>>
>>>
>>> This is creating the buffer, in memory, of a region of a file. The 0L argument is the offset (i.e., in this case the map begins at the beginning of the file) and the second argument is the length, which in jevio is essentially the size of the file.
>>>
>>> The java documentation gives the weird bad news. The third argument (sz, in this case) is a long, but it is restricted to INTEGER.MAXINT (which is 2 GB). That's right: it is a long (64 bit) value that will throw an exception if it is bigger than the max for a 32 bit int.
>>>
>>> To summarize: jevio assumes it can map the entire file to memory--but this will fail even if you have a 64 bit machine and sufficient (i.e. more than 2GB) of memory.
>>>
>>> There is a request for a change at Oracle, but it doesn't seen to be high priority. They say the only workaround is to work your way through big files:
>>>
>>> offset = 0L
>>> while (!done) {
>>> long sz = Math.min(remainingSize, INTEGER.MAXINT);
>>> mappedByteBuffer = inputChannel.map(FileChannel.MapMode.READ_ONLY, offset, sz);
>>>
>>> {do stuff}
>>> offset += sz;
>>> mappedByteBuffer = inputChannel.map(FileChannel.MapMode.READ_ONLY, 0L, sz);
>>>
>>> }
>>>
>>> I suspect that this would be a very nontrivial change to jevio withs lots of debugging. But I think we need to request it.
>>>
>>> Note that we basically have to use memory mapped io--it is about 10x faster than traditional disk access IO.
>>>
>>>
>>> --
>>>
>>> David P. Heddle, Ph.D.
>>> Associate Professor of Physics
>>> Christopher Newport University
>>> Newport News, VA 23606
>>>
>>> 757.594.8434 (CNU)
>>> 757.269.5719 (Jefferson Lab)
>>> _______________________________________________
>>> Clas12_software mailing list
>>> [log in to unmask]
>>> https://mailman.jlab.org/mailman/listinfo/clas12_software
>>>
>>>
>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the HPS-SOFTWARE list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=HPS-SOFTWARE&A=1
>>>
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the HPS-SOFTWARE list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=HPS-SOFTWARE&A=1
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the HPS-SOFTWARE list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=HPS-SOFTWARE&A=1
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the HPS-SOFTWARE list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=HPS-SOFTWARE&A=1
