Hi Justas,
OK, let me make sure I have the arrangement correct. The host
sense-origin-0101.ultralight.org is either pulling data from host "x" (I
have no idea what that host is) or host "x" is pushing data to
sense-origin-0101.ultralight.org, right?
So, in pull mode it's a remote read to "x" that is shown below while in
push mode it's a local read on "x" that is shown, right?
If so, then remote reads come in as 1 MB reads to the Ofs plugin while
local reads come in as 64K reads to the same Ofs.
Is this via http or via xroot? If it's via http can we see the http trace
for this?
Andy
On Thu, 16 Feb 2023, Justas Balcas wrote:
> After a few more tests, gfal-copy --copy-mode push vs pull shows this
> issue. See below. So the question remains why in the case of push mode - it
> uses 65536, while in pull 1mb...
>
> In terms of performance, it is a huge decrease (2 to 4 factors).
>
>
> gfal-copy push mode
>
> 230217 01:08:34 647 jbalcas ofs_read: 65536@1073283072
> fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:34 647 jbalcas ofs_read: 65536@1073348608
> fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:34 647 jbalcas ofs_read: 65536@1073414144
> fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:34 647 jbalcas ofs_read: 65536@1073479680
> fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:34 647 jbalcas ofs_read: 65536@1073545216
> fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:34 647 jbalcas ofs_read: 65536@1073610752
> fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:34 647 jbalcas ofs_read: 65536@1073676288
> fn=/store/user/jbalcas/testSourceFile1
>
> gfal-copy pull mode:
>
> 230217 01:08:51 096 unknown.3:[log in to unmask]
> ofs_read: 1048576@0 fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:51 096 unknown.3:[log in to unmask]
> ofs_read: 1048576@1048576 fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:51 096 unknown.3:[log in to unmask]
> ofs_read: 1048576@2097152 fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:51 096 unknown.3:[log in to unmask]
> ofs_read: 1048576@3145728 fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:51 096 unknown.3:[log in to unmask]
> ofs_read: 1048576@4194304 fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:51 096 unknown.3:[log in to unmask]
> ofs_read: 1048576@5242880 fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:51 096 unknown.3:[log in to unmask]
> ofs_read: 1048576@6291456 fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:51 096 unknown.3:[log in to unmask]
> ofs_read: 1048576@7340032 fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:51 096 unknown.3:[log in to unmask]
> ofs_read: 1048576@8388608 fn=/store/user/jbalcas/testSourceFile1
> 230217 01:08:51 096 unknown.3:[log in to unmask]
> ofs_read: 1048576@9437184 fn=/store/user/jbalcas/testSourceFile1
>
>
> On Fri, 3 Feb 2023 at 21:03, Justas Balcas <[log in to unmask]> wrote:
>
>> Hello Andy,
>>
>> Both servers are local and both have:
>> xrootd.x86_64 1:5.4.2-1.osg35up.el8
>> @osg-upcoming
>> xrootd-client.x86_64 1:5.4.2-1.osg35up.el8
>> @osg-upcoming
>> xrootd-client-libs.x86_64 1:5.4.2-1.osg35up.el8
>> @osg-upcoming
>> xrootd-lcmaps.x86_64 1.7.8-3.osgup.el8
>> @osg-upcoming
>> xrootd-libs.x86_64 1:5.4.2-1.osg35up.el8
>> @osg-upcoming
>> xrootd-multiuser.x86_64 2.0.3-1.osg35up.el8
>> @osg-upcoming
>> xrootd-scitokens.x86_64 1:5.4.2-1.osg35up.el8
>> @osg-upcoming
>> xrootd-selinux.noarch 1:5.4.2-1.osg35up.el8
>> @osg-upcoming
>> xrootd-server.x86_64 1:5.4.2-1.osg35up.el8
>> @osg-upcoming
>> xrootd-server-libs.x86_64 1:5.4.2-1.osg35up.el8
>> @osg-upcoming
>>
>> Full xrootd config below (just in case):
>> all.export /tmp stage
>> frm.xfr.copycmd /bin/cp /dev/null $PFN
>> all.adminpath /var/spool/xrootd
>> all.pidpath /var/run/xrootd
>>
>> # ====================================================
>> # CUSTOM CONFIGS
>> # ====================================================
>> set xrdr = $XRD_REDIR
>> set port = $XRD_PORT
>> set mngport = $XRD_MNG_PORT
>> set localroot = $LOCAL_ROOT
>> set sitename = $SITENAME
>> set xrdcert = $XRD_CERT
>> set xrdkey = $XRD_KEY
>> # ====================================================
>> # XrootD Security
>> # ---------------------------------------
>> xrootd.seclib /usr/lib64/libXrdSec.so
>> sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates
>> -cert:$(xrdcert) -key:$(xrdkey) -crl:1 -authzfun:libXrdLcmaps.so
>> -authzto:900 -authzfunparms:lcmapscfg=/etc/xrootd/lcmaps.cfg -gmapopt:10
>> -gmapto:0
>> acc.authdb /etc/xrootd/auth_file
>> ofs.authorize
>> macaroons.secretkey /etc/xrootd/macaroon-secret
>> ofs.authlib ++ libXrdMacaroons.so
>> ofs.authlib ++ libXrdAccSciTokens.so
>>
>> all.sitename $(sitename)
>> xrd.port $(port)
>> all.manager $(xrdr):$(mngport)
>>
>> all.role server
>> all.manager meta $(xrdr):$(mngport)
>>
>> ofs.osslib ++ libXrdMultiuser.so
>> # Enable the checksum wrapper
>> ofs.ckslib * libXrdMultiuser.so
>> # Control of checksum
>> xrootd.chksum max 10 adler32
>> multiuser.checksumonwrite on
>> multiuser.umask 0002
>>
>> all.export /
>> cms.allow host *
>>
>> # Enable xrootd debugging
>> xrootd.trace emsg login stall redirect
>> cms.trace defer files forward redirect
>>
>> oss.localroot $(localroot)
>> # Enable https over XrootD
>> if exec xrootd
>> xrd.protocol http:$(port) /usr/lib64/libXrdHttp.so
>> http.cadir /etc/grid-security/certificates
>> http.cert $(xrdcert)
>> http.key $(xrdkey)
>> http.secxtractor /usr/lib64/libXrdLcmaps.so
>> http.secretkey <I am secret :P>
>> # Enable third-party-copy
>> http.exthandler xrdtpc libXrdHttpTPC.so
>> # Pass the bearer token to the Xrootd authorization framework.
>> http.header2cgi Authorization authz
>> http.listingdeny yes
>> http.desthttps yes
>> http.selfhttps2http no
>> http.staticpreload http://static/robots.txt /etc/xrootd/robots.txt
>> http.exthandler xrdmacaroons libXrdMacaroons.so
>> fi
>>
>> On Thu, 2 Feb 2023 at 07:46, Andrew Hanushevsky <[log in to unmask]>
>> wrote:
>>
>>> Hi Justas,
>>>
>>> You must be using an old release. The 64K reads come from async I/O
>>> processing. We turn that off in 5.3.0 for regular filesystem (it stays on
>>> for proxy servers). So, the read size winds for an xrootd server
>>> (assuming
>>> you are reading from xrootd) is determined by the source server relative
>>> to it's async settings. It's configurable but no something you can
>>> control -- the source site controls it. Should they upgrade to the latest
>>> release it will start using large blocks. The same applies to ghe
>>> destination server. So, what version is the source server?
>>>
>>> Andy
>>>
>>> On Tue, 31 Jan 2023, Justas Balcas wrote:
>>>
>>>> Hi, Does anyone know why xrootd 3rd party copy HTTPS implementation uses
>>>> 65536 for ofs_read, but if I read it to local dir - it uses 1048576?
>>> Can it
>>>> be configurable per site (or made to be configurable) - or are there any
>>>> implementation constraints?
>>>> Thanks!
>>>>
>>>> a) gfal-copy https://<endpoint>/filename <localfilename> on xrootd
>>> logs I
>>>> see:230126 15:09:13 024 jbalcas.8:31@login-1 ofs_read: 1048576@0
>>>> fn=/store/user/jbalcas/testSourceFile0230126 15:09:13 024
>>>> jbalcas.8:31@login-1 ofs_read: 1048576@1048576
>>>> fn=/store/user/jbalcas/testSourceFile0
>>>> b) gfal-copy (third party copy) https://<endpoint>/filename
>>>> https://<endpoint1>/filename:230126
>>>> 15:19:44 024 jbalcas ofs_read: 65536@0
>>>> fn=/store/user/jbalcas/testSourceFile0230126 15:19:44 024 jbalcas
>>> ofs_read:
>>>> 65536@65536 fn=/store/user/jbalcas/testSourceFile0230126 15:19:44 024
>>>> jbalcas ofs_read: 65536@131072
>>> fn=/store/user/jbalcas/testSourceFile0230126
>>>> 15:19:44 024 jbalcas ofs_read: 65536@196608
>>>> fn=/store/user/jbalcas/testSourceFile0
>>>>
>>>> --
>>>> ---------------------------------------
>>>> Justas Balcas
>>>> Caltech CMS Group
>>>> CIT Downs-Lauritsen 239
>>>>
>>>> ########################################################################
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the XROOTD-L list, click the following link:
>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>>>
>>>
>>
>>
>> --
>> ---------------------------------------
>> Justas Balcas
>> Caltech CMS Group
>> CIT Downs-Lauritsen 239
>>
>>
>
> --
> ---------------------------------------
> Justas Balcas
> Caltech CMS Group
> CIT Downs-Lauritsen 239
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
|