Oups sorry!
you're perfectly right David, thank you.
env set to => XRDCLHTTP_AVOIDRANAGE=1
But the problem is still there.
So now using curl, no problem. \o/
Using cvmfs, it seems that whatever the file, I got a file size of 25165824.
Looking at xcache logs:
At the beginning I have the first range (from 0 to 25165823):
210511 16:54:15 164541 sysXrdHttp: Parsing first line: GET //http://fiona.uvalight.net:8000/gwdata/O1/strain.16k/hdf.v1/H1/1136656384/H-H1_LOSC_16_V1-1137221632-4096.hdf5 HTTP/1.1
210511 16:54:15 164541 sysXrdHttp: rc:37 got hdr line: Host: our_xcache_server:1094
210511 16:54:15 164541 sysXrdHttp: rc:25 got hdr line: Range: bytes=0-25165823
210511 16:54:15 164541 sysXrdHttp: rc:13 got hdr line: Accept: */*
210511 16:54:15 164541 sysXrdHttp: rc:24 got hdr line: Connection: Keep-Alive
210511 16:54:15 164541 sysXrdHttp: rc:30 got hdr line: User-Agent: cvmfs Fuse 2.8.1
210511 16:54:15 164541 sysXrdHttp: rc:105 got hdr line: cvmfs-info: Part%20of%20/gwdata/O1/strain.16k/hdf.v1/H1/1136656384/H-H1_LOSC_16_V1-1137221632-4096.hdf5
Then download starts
.
.
.
Then I've got the next range (from 25165824-50331647)
210511 16:54:17 164540 sysXrdHttp: Parsing first line: GET //http://fiona.uvalight.net:8000/gwdata/O1/strain.16k/hdf.v1/H1/1136656384/H-H1_LOSC_16_V1-1137221632-4096.hdf5 HTTP/1.1
210511 16:54:17 164540 sysXrdHttp: rc:37 got hdr line: Host: our_xcache_server:1094
210511 16:54:17 164540 sysXrdHttp: rc:32 got hdr line: Range: bytes=25165824-50331647
210511 16:54:17 164540 sysXrdHttp: rc:13 got hdr line: Accept: */*
210511 16:54:17 164540 sysXrdHttp: rc:24 got hdr line: Connection: Keep-Alive
210511 16:54:17 164540 sysXrdHttp: rc:30 got hdr line: User-Agent: cvmfs Fuse 2.8.1
210511 16:54:17 164540 sysXrdHttp: rc:105 got hdr line: cvmfs-info: Part%20of%20/gwdata/O1/strain.16k/hdf.v1/H1/1136656384/H-H1_LOSC_16_V1-1137221632-4096.hdf5
Then download
.
.
.
But the next range is in fact the same than just before, so we got again (from 25165824-50331647):
210511 16:54:17 164541 sysXrdHttp: Parsing first line: GET //http://fiona.uvalight.net:8000/gwdata/O1/strain.16k/hdf.v1/H1/1136656384/H-H1_LOSC_16_V1-1137221632-4096.hdf5 HTTP/1.1
210511 16:54:17 164541 sysXrdHttp: rc:37 got hdr line: Host: our_xcache_server:1094
210511 16:54:17 164541 sysXrdHttp: rc:32 got hdr line: Range: bytes=25165824-50331647
210511 16:54:17 164541 sysXrdHttp: rc:13 got hdr line: Accept: */*
210511 16:54:17 164541 sysXrdHttp: rc:24 got hdr line: Connection: Keep-Alive
210511 16:54:17 164541 sysXrdHttp: rc:30 got hdr line: User-Agent: cvmfs Fuse 2.8.1
210511 16:54:17 164541 sysXrdHttp: rc:105 got hdr line: cvmfs-info: Part%20of%20/gwdata/O1/strain.16k/hdf.v1/H1/1136656384/H-H1_LOSC_16_V1-1137221632-4096.hdf5
210511 16:54:17 164541 sysXrdHttp: rc:18 got hdr line: Pragma: no-cache
210511 16:54:17 164541 sysXrdHttp: rc:25 got hdr line: Cache-Control: no-cache
and it seems to loop like this 7 times in total!
In the end there is no error but it seems to stop because of all these "retry"?
Finally,
in cvmfs repo, the catalog shows a file of size 517357705
after cvmfs copy, the file has a size of 25165824 (the first downloaded range I think)
in xcache data, the file has a size of 83886080 (OMHO must be totally corrupted)
regards,
Bertrand Rigaud
Centre de Calcul de l'IN2P3 - CNRS
21 avenue Pierre de Coubertin
69627 Villeurbanne CEDEX
Tél : 04.78.93.08.80
----- Mail original -----
De: "David Smith" <[log in to unmask]>
À: "Bertrand Rigaud" <[log in to unmask]>
Cc: "Wei Yang" <[log in to unmask]>, "Matevz Tadel" <[log in to unmask]>, "xrootd-l" <[log in to unmask]>
Envoyé: Mardi 11 Mai 2021 15:36:49
Objet: Re: http ingest using xcache fails
> On 11 May 2021, at 15:03, Bertrand RIGAUD <[log in to unmask]> wrote:
>
> Yes this one is even more important as it is parsed from url!
>
> I'm not sure if I understand well, when using curl + ?xrddclhttp_avoidrange everything works like a charm, whereas trying cp from cvmfs repo, the problem is still here.
> I added the env var to my service:
>
> [Service]
> Environment=HTTP_FILE_PLUG_IN_AVOIDRANGE_ENV=XRDCLHTTP_AVOIDRANGE
Hi Bertrand,
Perhaps you could try setting the environment in this way:
Environment=XRDCLHTTP_AVOIDRANAGE=1
Yours,
David
--
-------------------------------------------------------------------------
David Smith e-mail: [log in to unmask]
-------------------------------------------------------------------------
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
|