 |
 |
Hi Andrew, Good morning. Thanks again for your help. All the client and server hosts have both IPv4 and IPv6 addresses. When querying the cmsxrootd-site1.fnal.gov redirector directly. I get: bash-4.2$ date; /usr/bin/xrdfs cmsxrootd-site1.fnal.gov:1094 locate -d /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root Mon Sep 18 18:09:30 CEST 2023 cmsxrootd-site1.fnal.gov:1093 Server Read bash-4.2$ date; /usr/bin/xrdfs cmsxrootd-site1.fnal.gov:1094 locate -d -m /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root Mon Sep 18 18:09:40 CEST 2023 cmsxrootd-site1.fnal.gov:1093 Server Read Both queries with/without "-m" option return hostname. ******** Also, for your info. Our cmsxrootd-site1.fnal.gov redirector at Port 1094 is basically a manager for both cmsd and xrootd with two redirectors registered with it: one at Port 1093 and other at Port 1092 (their roles are: all.role server). If I query them directly, I get: bash-4.2$ date; /usr/bin/xrdfs cmsxrootd-site1.fnal.gov:1093 locate -d /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root Mon Sep 18 18:09:57 CEST 2023 [::131.225.205.75]:1093 Server ReadWrite bash-4.2$ date; /usr/bin/xrdfs cmsxrootd-site1.fnal.gov:1093 locate -d -m /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root Mon Sep 18 18:10:07 CEST 2023 cmsxrootd-site1.fnal.gov:1093 Server ReadWrite The first one returns IP address without "-m" optioin and 2nd one returns hostname. --------------------------------------------------------------------------------------- Then, cmsxrootd-site1.fnal.gov manager registers with cmsxrootd.fnal.gov (An alias for xrootd.unl.edu and cmsxrootd2.fnal.gov): all.manager cmsxrootd-site1.fnal.gov:1213 all.manager meta all cmsxrootd.fnal.gov+:1213 bash-4.2$ host cmsxrootd.fnal.gov cmsxrootd.fnal.gov has address 131.225.184.17 (cmsxrootd2.fnal.gov) cmsxrootd.fnal.gov has address 129.93.239.131(xrootd.unl.edu) cmsxrootd.fnal.gov has IPv6 address 2600:900:6:1101:5054:ff:fe00:70cb(xrootd.unl.edu) cmsxrootd.fnal.gov has IPv6 address 2620:6a:0:187::184:17(cmsxrootd2.fnal.gov bash-4.2$ date; /usr/bin/xrdfs cmsxrootd2.fnal.gov:1094 locate -d -m /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root Mon Sep 18 18:35:26 CEST 2023 cmsxrootd-site2.fnal.gov:1093 Server Read cmsxrootd-site1.fnal.gov:1093 Server Read cmsxrootd-site3.fnal.gov:1093 Server Read bash-4.2$ date; /usr/bin/xrdfs cmsxrootd2.fnal.gov:1094 locate -d /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root Mon Sep 18 18:35:36 CEST 2023 [ERROR] Server responded with an error: [3011] No valid location found The weirdest part is: xrootd.unl.edu returns the same result with and without "-m": bash-4.2$ date; /usr/bin/xrdfs xrootd.unl.edu:1094 locate -d /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root Mon Sep 18 18:40:25 CEST 2023 cmsxrootd-site1.fnal.gov:1093 Server Read cmsxrootd-site2.fnal.gov:1093 Server Read cmsxrootd-site3.fnal.gov:1093 Server Read bash-4.2$ date; /usr/bin/xrdfs xrootd.unl.edu:1094 locate -d -m /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root Mon Sep 18 18:40:35 CEST 2023 cmsxrootd-site2.fnal.gov:1093 Server Read cmsxrootd-site1.fnal.gov:1093 Server Read cmsxrootd-site3.fnal.gov:1093 Server Read ------------------------------------------------------------------ This happens even after I put xrootd.unl.edu XRootD configuration file on cmsxrootd2.fnal.gov. Could you please advise what could be the cause on this? Or, if there is an option at server side that could force "-m" behavior? Thanks again for your help. Regards, Yujun ________________________________ From: Andrew Hanushevsky <[log in to unmask]> Sent: Friday, September 15, 2023 11:53 PM To: Yujun Wu <[log in to unmask]> Subject: Re: Help with a xrdfs query puzzle with "xrdfs locate -d" One more thing try doing a locate against cmsxrootd-site1.fnal.gov:1094 which in theory has the file. On Sat, 16 Sep 2023, Yujun Wu wrote: > Hi Andrew, > > Thank you for your help. Currently, CMS XRootD redirectors only allow access with voms proxy from /cms group. > > It seems all the files I tried have this issue. The following are two examples: > > > [enstore@fndca2b ~]$ date; xrdfs cmsxrootd2.fnal.gov:1094 locate -d -m /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root > > Fri Sep 15 22:41:40 CDT 2023 > > cmsxrootd-site3.fnal.gov:1093 Server Read > > cmsxrootd-site1.fnal.gov:1093 Server Read > > cmsxrootd-site2.fnal.gov:1093 Server Read > > [enstore@fndca2b ~]$ date; xrdfs cmsxrootd2.fnal.gov:1094 locate -d /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root > > Fri Sep 15 22:41:56 CDT 2023 > > [ERROR] Server responded with an error: [3011] No valid location found > > > ------------- > > [enstore@fndca2b ~]$ date; xrdfs cmsxrootd2.fnal.gov:1094 locate -d -m /store/test/xrootd/redirector/test-file > > Fri Sep 15 22:42:34 CDT 2023 > > cmsxrootd-site3.fnal.gov:1092 Server Read > > cmsxrootd-site1.fnal.gov:1096 Server Read > > [enstore@fndca2b ~]$ date; xrdfs cmsxrootd2.fnal.gov:1094 locate -d /store/test/xrootd/redirector/test-file > > Fri Sep 15 22:42:45 CDT 2023 > > [ERROR] Server responded with an error: [3011] No valid location found > > ------------- > For the 2nd file, the log files showed: > ###cmsxrootd2### > > [root@cmsxrootd2 tmp]# tail -n 100000 cmsd.log|grep test-file > > [root@cmsxrootd2 tmp]# tail -n 100000 xrootd.log|grep test-file > > 230915 22:42:34 17939 acc_Audit: enstore.233531:159@fndca2b grant host *@fndca2b.fnal.gov stat /store/test/xrootd/redirector/test-file > > 230915 22:42:34 17913 cms_Decode: cmsxrootd2 delays enstore.233531:159@fndca2b 1 /store/test/xrootd/redirector/test-file > > 230915 22:42:35 17939 acc_Audit: enstore.233531:159@fndca2b grant host *@fndca2b.fnal.gov stat /store/test/xrootd/redirector/test-file > > 230915 22:42:35 17913 cms_Decode: cmsxrootd2 sent enstore.233531:159@fndca2b 'Mrcmsxrootd-site1.fnal.gov:1094 Mrcmsxrootd-site3.fnal.gov:1094' /store/test/xrootd/redirector/test-file > > 230915 22:42:45 17869 acc_Audit: enstore.233544:463@fndca2b grant host *@fndca2b.fnal.gov stat /store/test/xrootd/redirector/test-file > > 230915 22:42:45 17913 cms_Decode: cmsxrootd2 sent enstore.233544:463@fndca2b 'Mr[2620:6a:0:8420::f8]:1094 Mr[2620:6a:0:8421::243]:1094' /store/test/xrootd/redirector/test-file > > > ###cmsxrootd-site1### > > > > [root@cmsxrootd-site1 site]# tail -n 1000000 cmsd.log|grep test-file > > 230915 22:26:02 1955 manager.0:[log in to unmask] cms_do_State: /store/test/xrootd/T2_BR_SPRACE/store/user/jmsilva/1MiB-subscription-test-file.raw > > 230915 22:42:34 1953 manager.0:27@cmsxrootd2 cms_do_State: /store/test/xrootd/redirector/test-file > > 230915 22:42:34 13496 server.1580:30@cmsxrootd-site1:1096 cms_do_Have: /store/test/xrootd/redirector/test-file > > 230915 22:42:36 20735 enstore.233531:3840@fndca2b cms_do_Locate: /store/test/xrootd/redirector/test-file > > [root@cmsxrootd-site1 site]# tail -n 1000000 xrootd.log|grep test-file > > 230915 22:42:36 31818 acc_Audit: enstore.233531:3840@fndca2b grant gsi [log in to unmask] stat /store/test/xrootd/redirector/test-file > > 230915 22:42:36 1944 cms_Decode: cmsxrootd-site1 sent enstore.233531:3840@fndca2b 'Srcmsxrootd-site1.fnal.gov:1096' /store/test/xrootd/redirector/test-file > > ###cmsxrootd-site3### > > [root@cmsxrootd-site3 site]# tail -n 1000000 cmsd.log|grep test-file > > 230915 22:26:02 33739 manager.0:[log in to unmask] cms_do_State: /store/test/xrootd/T2_BR_SPRACE/store/user/jmsilva/1MiB-subscription-test-file.raw > > 230915 22:42:34 33738 manager.0:31@cmsxrootd2 cms_do_State: /store/test/xrootd/redirector/test-file > > 230915 22:42:34 34190 server.33696:33@cmsxrootd-site3:1092 cms_do_Have: /store/test/xrootd/redirector/test-file > > 230915 22:42:36 34194 enstore.233531:49@fndca2b cms_do_Locate: /store/test/xrootd/redirector/test-file > > [root@cmsxrootd-site3 site]# tail -n 1000000 xrootd.log|grep test-file > > 230915 22:42:36 33982 acc_Audit: enstore.233531:49@fndca2b grant gsi [log in to unmask] stat /store/test/xrootd/redirector/test-file > > 230915 22:42:36 33731 cms_Decode: cmsxrootd-site3 sent enstore.233531:49@fndca2b 'Srcmsxrootd-site3.fnal.gov:1092' /store/test/xrootd/redirector/test-file > > ------------ > > So, the one without "-m" option didn't showed up in the cmxrootd-site1/3 redirectors. Could it be possible this was due to IP addresses were sent instead of host names when not using "-m" option? > > Thanks again for your help on this. > > > Regards, > Yujun > ________________________________ > From: Andrew Hanushevsky <[log in to unmask]> > Sent: Friday, September 15, 2023 7:12 PM > To: Yujun Wu <[log in to unmask]> > Subject: Re: Help with a xrdfs query puzzle with "xrdfs locate -d" > > Well, that's strange because when I try it I always get > > Server responded with an error: [3011] No valid location found > > regardless of whether -m is there or not. Could you provide another file > that should be present? > > On Fri, 15 Sep 2023, Yujun Wu wrote: > >> Good afternoon. >> >> Could XRootD developers please help with a xrdfs puzzle? >> >> Bockjoo first noticed. When we query our XRootD regional redirector cmsxrootd2.fnal.gov, we were able to get locate results using "-m" option, but no results without "-m" option: >> >> ### xrdfs locate -d with "-m" option >> >> >> bash-4.2$ date; xrdfs cmsxrootd2.fnal.gov:1094 locate -d -m /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >> >> Fri Sep 15 21:22:52 CEST 2023 >> >> cmsxrootd-site3.fnal.gov:1093 Server Read >> >> cmsxrootd-site2.fnal.gov:1093 Server Read >> >> cmsxrootd-site1.fnal.gov:1093 Server Read >> >> cmsio5.rc.ufl.edu:1094 Server ReadWrite >> >> cmsio7.rc.ufl.edu:1094 Server ReadWrite >> >> cmsio9.rc.ufl.edu:1094 Server ReadWrite >> >> cmsio4.rc.ufl.edu:1094 Server ReadWrite >> >> cmsio3.rc.ufl.edu:1094 Server ReadWrite >> >> cmsio8.rc.ufl.edu:1094 Server ReadWrite >> >> cmsio6.rc.ufl.edu:1094 Server ReadWrite >> >> >> ------------------- >> >> ### xrdfs locate -d without "-m" option >> >> >> bash-4.2$ date; xrdfs cmsxrootd2.fnal.gov:1094 locate -d /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >> >> Fri Sep 15 21:27:13 CEST 2023 >> >> [ERROR] Server responded with an error: [3011] No valid location found >> >> >> -------------------- >> >> On the server side, the first one showed: >> >> 230915 14:22:53 18227 acc_Audit: yujun.12664:[log in to unmask] grant host *@lxplus734.cern.ch stat /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >> >> 230915 14:22:53 17913 cms_Receive: cmsxrootd2 125 bytes on 727534553 >> >> 230915 14:22:53 17913 cms_Decode: cmsxrootd2 sent yujun.12664:[log in to unmask] 'Mwcmsio2.rc.ufl.edu:1094 Mrcmsxrootd-site1.fnal.gov:1094 Mrcmsxrootd-site3.fnal.gov:1094 Mrcmsxrootd-site2.fnal.gov:1094' /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >> >> 230915 14:22:53 17913 cms_Receive: cmsxrootd2 32 bytes on 727535577 >> >> >> >> While the 2nd one without "-m" option showed: >> >> >> 230915 14:27:13 27864 acc_Audit: yujun.9149:[log in to unmask] grant host *@lxplus734.cern.ch stat /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >> >> 230915 14:27:13 17913 cms_Receive: cmsxrootd2 135 bytes on 743171027 >> >> 230915 14:27:13 17913 cms_Decode: cmsxrootd2 sent yujun.9149:[log in to unmask] 'Mw[2620:104:1f:1000:faf2:1eff:fed9:dd80]:1094 Mr[2620:6a:0:8420::f8]:1094 Mr[2620:6a:0:8421::243]:1094 Mr[2620:6a:0:8420::f9]:1094' /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >> >> 230915 14:27:13 17913 cms_Receive: cmsxrootd2 4 bytes on 743172051 >> >> >> =================== >> >> >> As you can see, the successful one sent multiple hostnames while the failed one sent multiple IP addresses: >> >> 230915 14:27:13 27864 acc_Audit: yujun.9149:[log in to unmask] grant host *@lxplus734.cern.ch stat /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >> >> 230915 14:27:13 17913 cms_Receive: cmsxrootd2 135 bytes on 743171027 >> >> 230915 14:27:13 17913 cms_Decode: cmsxrootd2 sent yujun.9149:[log in to unmask] 'Mw[2620:104:1f:1000:faf2:1eff:fed9:dd80]:1094 Mr[2620:6a:0:8420::f8]:1094 Mr[2620:6a:0:8421::243]:1094 Mr[2620:6a:0:8420::f9]:1094' /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >> >> 230915 14:27:13 17913 cms_Receive: cmsxrootd2 4 bytes on 743172051 >> >> >> ------------------------------------ >> >> >> In contrast, another USCMS regional redirector xrootd.unl.edu could return results with and without "-m" option: >> >> >> bash-4.2$ date; xrdfs xrootd.unl.edu:1094 locate -d /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >> >> Fri Sep 15 21:38:17 CEST 2023 >> >> cmsxrootd-site3.fnal.gov:1093 Server Read >> >> cmsio5.rc.ufl.edu:1094 Server ReadWrite >> >> cmsio7.rc.ufl.edu:1094 Server ReadWrite >> >> cmsio9.rc.ufl.edu:1094 Server ReadWrite >> >> cmsio4.rc.ufl.edu:1094 Server ReadWrite >> >> cmsio3.rc.ufl.edu:1094 Server ReadWrite >> >> cmsio8.rc.ufl.edu:1094 Server ReadWrite >> >> cmsio6.rc.ufl.edu:1094 Server ReadWrite >> >> cmsxrootd-site1.fnal.gov:1093 Server Read >> >> cmsxrootd-site2.fnal.gov:1093 Server Read >> >> >> I was told their log file showed hostnames were sent even without "-m" option. >> >> >> >> For cmsxrootd2.fnal.gov XRootD redirector, I added >> >> >> xrd.network dnr >> >> >> option in our server side xrootd.cfg but it seems IP addresses still were sent. Could you please tell how I could force hostnames instead of IP addresses to be sent >> >> when running "xrdfs cmsxrootd2.fnal.gov:1094 locate -d" command? >> >> >> Thanks in advance for any advice on this. >> >> >> >> Regards, >> >> Yujun >> >> >> >> >> >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIBAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=bFk3iAgykb1xfdm5362Q9w&m=u4ZeuNN02gPONHd_hstIHONjzndQrVW3s6axSEhW_-qx4IqYlkKlkgW5N3y0N0HY&s=HTkD3kSVa8SHsVmhFbg5SevVFe2CM4HmftLiX3yPCbg&e= >> > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1