 |
 |
After a few more tests, gfal-copy --copy-mode push vs pull shows this issue. See below. So the question remains why in the case of push mode - it uses 65536, while in pull 1mb... In terms of performance, it is a huge decrease (2 to 4 factors). gfal-copy push mode 230217 01:08:34 647 jbalcas ofs_read: 65536@1073283072 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:34 647 jbalcas ofs_read: 65536@1073348608 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:34 647 jbalcas ofs_read: 65536@1073414144 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:34 647 jbalcas ofs_read: 65536@1073479680 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:34 647 jbalcas ofs_read: 65536@1073545216 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:34 647 jbalcas ofs_read: 65536@1073610752 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:34 647 jbalcas ofs_read: 65536@1073676288 fn=/store/user/jbalcas/testSourceFile1 gfal-copy pull mode: 230217 01:08:51 096 unknown.3:[log in to unmask] ofs_read: 1048576@0 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:51 096 unknown.3:[log in to unmask] ofs_read: 1048576@1048576 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:51 096 unknown.3:[log in to unmask] ofs_read: 1048576@2097152 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:51 096 unknown.3:[log in to unmask] ofs_read: 1048576@3145728 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:51 096 unknown.3:[log in to unmask] ofs_read: 1048576@4194304 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:51 096 unknown.3:[log in to unmask] ofs_read: 1048576@5242880 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:51 096 unknown.3:[log in to unmask] ofs_read: 1048576@6291456 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:51 096 unknown.3:[log in to unmask] ofs_read: 1048576@7340032 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:51 096 unknown.3:[log in to unmask] ofs_read: 1048576@8388608 fn=/store/user/jbalcas/testSourceFile1 230217 01:08:51 096 unknown.3:[log in to unmask] ofs_read: 1048576@9437184 fn=/store/user/jbalcas/testSourceFile1 On Fri, 3 Feb 2023 at 21:03, Justas Balcas <[log in to unmask]> wrote: > Hello Andy, > > Both servers are local and both have: > xrootd.x86_64 1:5.4.2-1.osg35up.el8 > @osg-upcoming > xrootd-client.x86_64 1:5.4.2-1.osg35up.el8 > @osg-upcoming > xrootd-client-libs.x86_64 1:5.4.2-1.osg35up.el8 > @osg-upcoming > xrootd-lcmaps.x86_64 1.7.8-3.osgup.el8 > @osg-upcoming > xrootd-libs.x86_64 1:5.4.2-1.osg35up.el8 > @osg-upcoming > xrootd-multiuser.x86_64 2.0.3-1.osg35up.el8 > @osg-upcoming > xrootd-scitokens.x86_64 1:5.4.2-1.osg35up.el8 > @osg-upcoming > xrootd-selinux.noarch 1:5.4.2-1.osg35up.el8 > @osg-upcoming > xrootd-server.x86_64 1:5.4.2-1.osg35up.el8 > @osg-upcoming > xrootd-server-libs.x86_64 1:5.4.2-1.osg35up.el8 > @osg-upcoming > > Full xrootd config below (just in case): > all.export /tmp stage > frm.xfr.copycmd /bin/cp /dev/null $PFN > all.adminpath /var/spool/xrootd > all.pidpath /var/run/xrootd > > # ==================================================== > # CUSTOM CONFIGS > # ==================================================== > set xrdr = $XRD_REDIR > set port = $XRD_PORT > set mngport = $XRD_MNG_PORT > set localroot = $LOCAL_ROOT > set sitename = $SITENAME > set xrdcert = $XRD_CERT > set xrdkey = $XRD_KEY > # ==================================================== > # XrootD Security > # --------------------------------------- > xrootd.seclib /usr/lib64/libXrdSec.so > sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates > -cert:$(xrdcert) -key:$(xrdkey) -crl:1 -authzfun:libXrdLcmaps.so > -authzto:900 -authzfunparms:lcmapscfg=/etc/xrootd/lcmaps.cfg -gmapopt:10 > -gmapto:0 > acc.authdb /etc/xrootd/auth_file > ofs.authorize > macaroons.secretkey /etc/xrootd/macaroon-secret > ofs.authlib ++ libXrdMacaroons.so > ofs.authlib ++ libXrdAccSciTokens.so > > all.sitename $(sitename) > xrd.port $(port) > all.manager $(xrdr):$(mngport) > > all.role server > all.manager meta $(xrdr):$(mngport) > > ofs.osslib ++ libXrdMultiuser.so > # Enable the checksum wrapper > ofs.ckslib * libXrdMultiuser.so > # Control of checksum > xrootd.chksum max 10 adler32 > multiuser.checksumonwrite on > multiuser.umask 0002 > > all.export / > cms.allow host * > > # Enable xrootd debugging > xrootd.trace emsg login stall redirect > cms.trace defer files forward redirect > > oss.localroot $(localroot) > # Enable https over XrootD > if exec xrootd > xrd.protocol http:$(port) /usr/lib64/libXrdHttp.so > http.cadir /etc/grid-security/certificates > http.cert $(xrdcert) > http.key $(xrdkey) > http.secxtractor /usr/lib64/libXrdLcmaps.so > http.secretkey <I am secret :P> > # Enable third-party-copy > http.exthandler xrdtpc libXrdHttpTPC.so > # Pass the bearer token to the Xrootd authorization framework. > http.header2cgi Authorization authz > http.listingdeny yes > http.desthttps yes > http.selfhttps2http no > http.staticpreload http://static/robots.txt /etc/xrootd/robots.txt > http.exthandler xrdmacaroons libXrdMacaroons.so > fi > > On Thu, 2 Feb 2023 at 07:46, Andrew Hanushevsky <[log in to unmask]> > wrote: > >> Hi Justas, >> >> You must be using an old release. The 64K reads come from async I/O >> processing. We turn that off in 5.3.0 for regular filesystem (it stays on >> for proxy servers). So, the read size winds for an xrootd server >> (assuming >> you are reading from xrootd) is determined by the source server relative >> to it's async settings. It's configurable but no something you can >> control -- the source site controls it. Should they upgrade to the latest >> release it will start using large blocks. The same applies to ghe >> destination server. So, what version is the source server? >> >> Andy >> >> On Tue, 31 Jan 2023, Justas Balcas wrote: >> >> > Hi, Does anyone know why xrootd 3rd party copy HTTPS implementation uses >> > 65536 for ofs_read, but if I read it to local dir - it uses 1048576? >> Can it >> > be configurable per site (or made to be configurable) - or are there any >> > implementation constraints? >> > Thanks! >> > >> > a) gfal-copy https://<endpoint>/filename <localfilename> on xrootd >> logs I >> > see:230126 15:09:13 024 jbalcas.8:31@login-1 ofs_read: 1048576@0 >> > fn=/store/user/jbalcas/testSourceFile0230126 15:09:13 024 >> > jbalcas.8:31@login-1 ofs_read: 1048576@1048576 >> > fn=/store/user/jbalcas/testSourceFile0 >> > b) gfal-copy (third party copy) https://<endpoint>/filename >> > https://<endpoint1>/filename:230126 >> > 15:19:44 024 jbalcas ofs_read: 65536@0 >> > fn=/store/user/jbalcas/testSourceFile0230126 15:19:44 024 jbalcas >> ofs_read: >> > 65536@65536 fn=/store/user/jbalcas/testSourceFile0230126 15:19:44 024 >> > jbalcas ofs_read: 65536@131072 >> fn=/store/user/jbalcas/testSourceFile0230126 >> > 15:19:44 024 jbalcas ofs_read: 65536@196608 >> > fn=/store/user/jbalcas/testSourceFile0 >> > >> > -- >> > --------------------------------------- >> > Justas Balcas >> > Caltech CMS Group >> > CIT Downs-Lauritsen 239 >> > >> > ######################################################################## >> > Use REPLY-ALL to reply to list >> > >> > To unsubscribe from the XROOTD-L list, click the following link: >> > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >> > >> > > > -- > --------------------------------------- > Justas Balcas > Caltech CMS Group > CIT Downs-Lauritsen 239 > > -- --------------------------------------- Justas Balcas Caltech CMS Group CIT Downs-Lauritsen 239 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1