 |
 |
Hi Yujun, On 9/19/23 07:16, Yujun Wu wrote: > Hi Matevz and Bockjoo, > > Thanks for your help with the investigation. Those cmsxrootd-site[1-3].fnal.gov > are connected with our dCache and EOS XROOTD doors. Is fnal eos still the 4.12 based one? Maybe there was a bug with -m support there. And maybe the same is true for the dCache doors? I remember 4.12 has some serious trouble parsing certs with DNs containing ',' ... and maybe some other characters (took me a long while t ofigure out why UCSD xcache wasn't able to auth to fnal eos :) ). I'd try to figure out the lowest level where the -m/-d problem appears. Is it on cmsxrootd-site[123].fnal.gov:1094 redirs? What versions are those? But if there is a problem with the underlying servers (dCache & EOS doors) this should really be fixed there. XRD_LOGLEVEL=Info xrdfs cmsxrootd2.fnal.gov:1094 locate -d /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root You can also try XRD_LOGLEVEL=Debug (and see other XrdCl env vars, esp. XRD_TLSDBGLVL and timeouts, as descibed in man xrdcp). Have you tried doing ipv4 vs. 6 test yet? XRD_NETWORKSTACK=... eg: matevz@black src> XRD_NOTLSOK=1 XRD_NETWORKSTACK=IPv4 XRD_LOGLEVEL=Info xrdfs cmsxrootd2.fnal.gov:1094 locate -d /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root [2023-09-19 16:50:17.116271 -0700][Info ][Utility ] Env: Importing from shell XRD_NOTLSOK=1 as notlsok [2023-09-19 16:50:17.116359 -0700][Info ][Utility ] Env: Importing from shell XRD_NETWORKSTACK=IPv4 as networkstack [::131.225.188.39]:1093 Server Read cmsxrootd-site3.fnal.gov:1093 Server Read cmsxrootd-site2.fnal.gov:1093 Server Read cmsxrootd-site1.fnal.gov:1093 Server Read cmsio5.rc.ufl.edu:1094 Server ReadWrite cmsio7.rc.ufl.edu:1094 Server ReadWrite cmsio9.rc.ufl.edu:1094 Server ReadWrite cmsio4.rc.ufl.edu:1094 Server ReadWrite cmsio3.rc.ufl.edu:1094 Server ReadWrite cmsio8.rc.ufl.edu:1094 Server ReadWrite cmsio6.rc.ufl.edu:1094 Server ReadWrite Cheers, Matevz > It looks like the following work fine: > > > 1. Without both "-d" and "-m" option: > > bash-4.2$ time xrdfs cmsxrootd2.fnal.gov:1094 locate > /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root > > [2620:6a:0:8420::f8]:1094 Manager Read > > [2620:6a:0:8421::243]:1094 Manager Read > > [2620:6a:0:8420::f9]:1094 Manager Read > > > real0m0.440s > > user0m0.006s > > sys0m0.007s > > > 2.Only with "-m" option: > > bash-4.2$ time xrdfs cmsxrootd2.fnal.gov:1094 locate -m > /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root > > cmsxrootd-site1.fnal.gov:1094 Manager Read > > cmsxrootd-site3.fnal.gov:1094 Manager Read > > cmsxrootd-site2.fnal.gov:1094 Manager Read > > > real0m0.442s > > user0m0.005s > > sys0m0.008s > > > > 3. With both "-d" and "-m" option: > > bash-4.2$ time xrdfs cmsxrootd2.fnal.gov:1094 locate -d -m > /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root > > cmsxrootd-site3.fnal.gov:1093 Server Read > > cmsxrootd-site2.fnal.gov:1093 Server Read > > cmsxrootd-site1.fnal.gov:1093 Server Read > > > real1m16.579s > > user0m0.199s > > sys0m0.010s > > > (Try again) > > bash-4.2$time xrdfs cmsxrootd2.fnal.gov:1094 locate -d -m > /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root > > cmsxrootd-site3.fnal.gov:1093 Server Read > > cmsxrootd-site1.fnal.gov:1093 Server Read > > cmsxrootd-site2.fnal.gov:1093 Server Read > > > real0m1.859s > > user0m0.182s > > sys0m0.013s > > > ------------- > > It does NOT work with only "-d" option: > > bash-4.2$ time xrdfs cmsxrootd2.fnal.gov:1094 locate -d > /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root > > [ERROR] Server responded with an error: [3011] No valid location found > > > > real0m0.763s > > user0m0.009s > > sys0m0.007s > > > ------------ > > It seems "-d" is a pretty expensive operation. It goes deep to all levels of > XRootD redirectors and only stop after reaching all data servers. As you can > see, the first time my query with "-d" and "-m" options took more than over 76 > seconds to do the requests and the 2^nd time the query returned after 1.86 > seconds as the results were still in cache. > > I have a test setup. It seems "-d" option could work if I reduce one layer of > authentication at cmsxrootd-site level, but not sure if this will violate the > CMS metadata access policy. I will put this in our test server and see how it goes. > > Anyway, it will be good to have xrdfs locate with ONLY "-d" option work. On the > other hand, I just looked at your monitoring table. Could you please tell if it > is possible to get the info without "-d" option? Also, personally I would use > "-m" option to make sure to get the hostnames instead of IP addresses. > > > Regards, > Yujun > > > > > 2. > > > > > > -------------------------------------------------------------------------------- > *From:* Bockjoo Kim <[log in to unmask]> > *Sent:* Tuesday, September 19, 2023 3:25 AM > *To:* Matevz Tadel <[log in to unmask]>; Yujun Wu > <[log in to unmask]>; Andrew Hanushevsky > <[log in to unmask]> > *Cc:* xrootd-l <[log in to unmask]>; Bockjoo Kim <[log in to unmask]> > *Subject:* Re: Help with a xrdfs query puzzle with "xrdfs locate -d" > > 1093 port is served by the dCache door. > > Because file locations became imbalanced according to the top level > redirector: > > https://bockjoo.web.cern.ch/aaaRedirectorMonitoring.txt-20230912 > <https://urldefense.com/v3/__https://bockjoo.web.cern.ch/aaaRedirectorMonitoring.txt-20230912__;!!Mih3wA!EOUiA0ZCGIAwKRfijYGjhakArUROAghVShw4MbKscFcBfdJpEJpfLTicmkRIHa6jlaFe6iiDiCh0jYH8w9Rt$> > > where one can see the jobs at American sites are redirected to the EU > regional redirectors > > instead of the US regional redirector. > > xrdfs should work with or without '-d' option, no? > > Thanks, > > Bockjoo > > On 9/19/23 01:58, Matevz Tadel wrote: >> xrdmapc cmsxrootd2.fnal.gov:1094 >> 1 Man cmsxrootd-site2.fnal.gov:1094 >> Srv cmsxrootd-site2.fnal.gov:1093 >> 1 Man cmsxrootd-site3.fnal.gov:1094 >> Srv cmsxrootd-site3.fnal.gov:1093 >> Srv cmsxrootd-site3.fnal.gov:1092 >> 1 Man cmsxrootd-site1.fnal.gov:1094 >> Srv cmsxrootd-site1.fnal.gov:1096 >> Srv cmsxrootd-site1.fnal.gov:1093 >> .... >> >> Are those servers dCache doors? >> >> Things also seem to work better without the -d flag. Why are you using -d? >> >> Matevz >> >> On 9/18/23 09:46, Yujun Wu wrote: >>> Hi Andrew, >>> >>> Good morning. >>> >>> Thanks again for your help. All the client and server hosts have both IPv4 and >>> IPv6 addresses. When querying the cmsxrootd-site1.fnal.gov redirector directly. >>> I get: >>> >>> bash-4.2$ date; /usr/bin/xrdfs cmsxrootd-site1.fnal.gov:1094 locate -d >>> /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root >>> Mon Sep 18 18:09:30 CEST 2023 >>> cmsxrootd-site1.fnal.gov:1093 Server Read >>> bash-4.2$ date; /usr/bin/xrdfs cmsxrootd-site1.fnal.gov:1094 locate -d -m >>> /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root >>> Mon Sep 18 18:09:40 CEST 2023 >>> cmsxrootd-site1.fnal.gov:1093 Server Read >>> Both queries with/without "-m" option return hostname. >>> >>> ******** >>> Also, for your info. Our cmsxrootd-site1.fnal.gov redirector at Port 1094 is >>> basically a manager for both cmsd and xrootd with two redirectors registered >>> with it: >>> one at Port 1093 and other at Port 1092 (their roles are: all.role server). If I >>> query them directly, I get: >>> >>> bash-4.2$ date; /usr/bin/xrdfs cmsxrootd-site1.fnal.gov:1093 locate -d >>> /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root >>> Mon Sep 18 18:09:57 CEST 2023 >>> [::131.225.205.75]:1093 Server ReadWrite >>> >>> bash-4.2$ date; /usr/bin/xrdfs cmsxrootd-site1.fnal.gov:1093 locate -d -m >>> /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root >>> Mon Sep 18 18:10:07 CEST 2023 >>> cmsxrootd-site1.fnal.gov:1093 Server ReadWrite >>> The first one returns IP address without "-m" optioin and 2^nd one returns >>> hostname. >>> --------------------------------------------------------------------------------------- >>> Then, cmsxrootd-site1.fnal.gov manager registers with cmsxrootd.fnal.gov (An >>> alias for xrootd.unl.edu and cmsxrootd2.fnal.gov): >>> all.manager cmsxrootd-site1.fnal.gov:1213 >>> all.manager meta all cmsxrootd.fnal.gov+:1213 >>> >>> bash-4.2$ host cmsxrootd.fnal.gov >>> cmsxrootd.fnal.gov has address 131.225.184.17 (cmsxrootd2.fnal.gov) >>> cmsxrootd.fnal.gov has address 129.93.239.131(xrootd.unl.edu) >>> cmsxrootd.fnal.gov has IPv6 address >>> 2600:900:6:1101:5054:ff:fe00:70cb(xrootd.unl.edu) >>> cmsxrootd.fnal.gov has IPv6 address 2620:6a:0:187::184:17(cmsxrootd2.fnal.gov >>> >>> >>> bash-4.2$ date; /usr/bin/xrdfs cmsxrootd2.fnal.gov:1094 locate -d -m >>> /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root >>> Mon Sep 18 18:35:26 CEST 2023 >>> cmsxrootd-site2.fnal.gov:1093 Server Read >>> cmsxrootd-site1.fnal.gov:1093 Server Read >>> cmsxrootd-site3.fnal.gov:1093 Server Read >>> bash-4.2$ date; /usr/bin/xrdfs cmsxrootd2.fnal.gov:1094 locate -d >>> /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root >>> Mon Sep 18 18:35:36 CEST 2023 >>> [ERROR] Server responded with an error: [3011] No valid location found >>> >>> The weirdest part is: xrootd.unl.edu returns the same result with and without "-m": >>> bash-4.2$ date; /usr/bin/xrdfs xrootd.unl.edu:1094 locate -d >>> /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root >>> Mon Sep 18 18:40:25 CEST 2023 >>> cmsxrootd-site1.fnal.gov:1093 Server Read >>> cmsxrootd-site2.fnal.gov:1093 Server Read >>> cmsxrootd-site3.fnal.gov:1093 Server Read >>> bash-4.2$ date; /usr/bin/xrdfs xrootd.unl.edu:1094 locate -d -m >>> /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root >>> Mon Sep 18 18:40:35 CEST 2023 >>> cmsxrootd-site2.fnal.gov:1093 Server Read >>> cmsxrootd-site1.fnal.gov:1093 Server Read >>> cmsxrootd-site3.fnal.gov:1093 Server Read >>> ------------------------------------------------------------------ >>> >>> This happens even after I put xrootd.unl.edu XRootD configuration file on >>> cmsxrootd2.fnal.gov. >>> >>> Could you please advise what could be the cause on this? Or, if there is an >>> option at server side that could force "-m" behavior? Thanks again for your help. >>> >>> >>> Regards, >>> Yujun >>> >>> >>> -------------------------------------------------------------------------------- >>> *From:* Andrew Hanushevsky <[log in to unmask]> >>> *Sent:* Friday, September 15, 2023 11:53 PM >>> *To:* Yujun Wu <[log in to unmask]> >>> *Subject:* Re: Help with a xrdfs query puzzle with "xrdfs locate -d" >>> >>> One more thing try doing a locate against cmsxrootd-site1.fnal.gov:1094 >>> which in theory has the file. >>> >>> On Sat, 16 Sep 2023, Yujun Wu wrote: >>> >>>> Hi Andrew, >>>> >>>> Thank you for your help. Currently, CMS XRootD redirectors only allow access with voms proxy from /cms group. >>>> >>>> It seems all the files I tried have this issue. The following are two examples: >>>> >>>> >>>> [enstore@fndca2b ~]$ date; xrdfs cmsxrootd2.fnal.gov:1094 locate -d -m /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root >>>> >>>> Fri Sep 15 22:41:40 CDT 2023 >>>> >>>> cmsxrootd-site3.fnal.gov:1093 Server Read >>>> >>>> cmsxrootd-site1.fnal.gov:1093 Server Read >>>> >>>> cmsxrootd-site2.fnal.gov:1093 Server Read >>>> >>>> [enstore@fndca2b ~]$ date; xrdfs cmsxrootd2.fnal.gov:1094 locate -d /store/test/xrootd/T1_US_FNAL/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root >>>> >>>> Fri Sep 15 22:41:56 CDT 2023 >>>> >>>> [ERROR] Server responded with an error: [3011] No valid location found >>>> >>>> >>>> ------------- >>>> >>>> [enstore@fndca2b ~]$ date; xrdfs cmsxrootd2.fnal.gov:1094 locate -d -m /store/test/xrootd/redirector/test-file >>>> >>>> Fri Sep 15 22:42:34 CDT 2023 >>>> >>>> cmsxrootd-site3.fnal.gov:1092 Server Read >>>> >>>> cmsxrootd-site1.fnal.gov:1096 Server Read >>>> >>>> [enstore@fndca2b ~]$ date; xrdfs cmsxrootd2.fnal.gov:1094 locate -d /store/test/xrootd/redirector/test-file >>>> >>>> Fri Sep 15 22:42:45 CDT 2023 >>>> >>>> [ERROR] Server responded with an error: [3011] No valid location found >>>> >>>> ------------- >>>> For the 2nd file, the log files showed: >>>> ###cmsxrootd2### >>>> >>>> [root@cmsxrootd2 tmp]# tail -n 100000 cmsd.log|grep test-file >>>> >>>> [root@cmsxrootd2 tmp]# tail -n 100000 xrootd.log|grep test-file >>>> >>>> 230915 22:42:34 17939 acc_Audit: enstore.233531:159@fndca2b grant host *@fndca2b.fnal.gov stat /store/test/xrootd/redirector/test-file >>>> >>>> 230915 22:42:34 17913 cms_Decode: cmsxrootd2 delays enstore.233531:159@fndca2b 1 /store/test/xrootd/redirector/test-file >>>> >>>> 230915 22:42:35 17939 acc_Audit: enstore.233531:159@fndca2b grant host *@fndca2b.fnal.gov stat /store/test/xrootd/redirector/test-file >>>> >>>> 230915 22:42:35 17913 cms_Decode: cmsxrootd2 sent enstore.233531:159@fndca2b 'Mrcmsxrootd-site1.fnal.gov:1094 Mrcmsxrootd-site3.fnal.gov:1094' /store/test/xrootd/redirector/test-file >>>> >>>> 230915 22:42:45 17869 acc_Audit: enstore.233544:463@fndca2b grant host *@fndca2b.fnal.gov stat /store/test/xrootd/redirector/test-file >>>> >>>> 230915 22:42:45 17913 cms_Decode: cmsxrootd2 sent enstore.233544:463@fndca2b 'Mr[2620:6a:0:8420::f8]:1094 Mr[2620:6a:0:8421::243]:1094' /store/test/xrootd/redirector/test-file >>>> >>>> >>>> ###cmsxrootd-site1### >>>> >>>> >>>> >>>> [root@cmsxrootd-site1 site]# tail -n 1000000 cmsd.log|grep test-file >>>> >>>> 230915 22:26:02 1955 manager.0:[log in to unmask] cms_do_State: /store/test/xrootd/T2_BR_SPRACE/store/user/jmsilva/1MiB-subscription-test-file.raw >>>> >>>> 230915 22:42:34 1953 manager.0:27@cmsxrootd2 cms_do_State: /store/test/xrootd/redirector/test-file >>>> >>>> 230915 22:42:34 13496 server.1580:30@cmsxrootd-site1:1096 cms_do_Have: /store/test/xrootd/redirector/test-file >>>> >>>> 230915 22:42:36 20735 enstore.233531:3840@fndca2b cms_do_Locate: /store/test/xrootd/redirector/test-file >>>> >>>> [root@cmsxrootd-site1 site]# tail -n 1000000 xrootd.log|grep test-file >>>> >>>> 230915 22:42:36 31818 acc_Audit: enstore.233531:3840@fndca2b grant gsi [log in to unmask] stat /store/test/xrootd/redirector/test-file >>>> >>>> 230915 22:42:36 1944 cms_Decode: cmsxrootd-site1 sent enstore.233531:3840@fndca2b 'Srcmsxrootd-site1.fnal.gov:1096' /store/test/xrootd/redirector/test-file >>>> >>>> ###cmsxrootd-site3### >>>> >>>> [root@cmsxrootd-site3 site]# tail -n 1000000 cmsd.log|grep test-file >>>> >>>> 230915 22:26:02 33739 manager.0:[log in to unmask] cms_do_State: /store/test/xrootd/T2_BR_SPRACE/store/user/jmsilva/1MiB-subscription-test-file.raw >>>> >>>> 230915 22:42:34 33738 manager.0:31@cmsxrootd2 cms_do_State: /store/test/xrootd/redirector/test-file >>>> >>>> 230915 22:42:34 34190 server.33696:33@cmsxrootd-site3:1092 cms_do_Have: /store/test/xrootd/redirector/test-file >>>> >>>> 230915 22:42:36 34194 enstore.233531:49@fndca2b cms_do_Locate: /store/test/xrootd/redirector/test-file >>>> >>>> [root@cmsxrootd-site3 site]# tail -n 1000000 xrootd.log|grep test-file >>>> >>>> 230915 22:42:36 33982 acc_Audit: enstore.233531:49@fndca2b grant gsi [log in to unmask] stat /store/test/xrootd/redirector/test-file >>>> >>>> 230915 22:42:36 33731 cms_Decode: cmsxrootd-site3 sent enstore.233531:49@fndca2b 'Srcmsxrootd-site3.fnal.gov:1092' /store/test/xrootd/redirector/test-file >>>> >>>> ------------ >>>> >>>> So, the one without "-m" option didn't showed up in the cmxrootd-site1/3 redirectors. Could it be possible this was due to IP addresses were sent instead of host names when not using "-m" option? >>>> >>>> Thanks again for your help on this. >>>> >>>> >>>> Regards, >>>> Yujun >>>> ________________________________ >>>> From: Andrew Hanushevsky <[log in to unmask]> >>>> Sent: Friday, September 15, 2023 7:12 PM >>>> To: Yujun Wu <[log in to unmask]> >>>> Subject: Re: Help with a xrdfs query puzzle with "xrdfs locate -d" >>>> >>>> Well, that's strange because when I try it I always get >>>> >>>> Server responded with an error: [3011] No valid location found >>>> >>>> regardless of whether -m is there or not. Could you provide another file >>>> that should be present? >>>> >>>> On Fri, 15 Sep 2023, Yujun Wu wrote: >>>> >>>>> Good afternoon. >>>>> >>>>> Could XRootD developers please help with a xrdfs puzzle? >>>>> >>>>> Bockjoo first noticed. When we query our XRootD regional redirector cmsxrootd2.fnal.gov, we were able to get locate results using "-m" option, but no results without "-m" option: >>>>> >>>>> ### xrdfs locate -d with "-m" option >>>>> >>>>> >>>>> bash-4.2$ date; xrdfs cmsxrootd2.fnal.gov:1094 locate -d -m /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >>>>> >>>>> Fri Sep 15 21:22:52 CEST 2023 >>>>> >>>>> cmsxrootd-site3.fnal.gov:1093 Server Read >>>>> >>>>> cmsxrootd-site2.fnal.gov:1093 Server Read >>>>> >>>>> cmsxrootd-site1.fnal.gov:1093 Server Read >>>>> >>>>> cmsio5.rc.ufl.edu:1094 Server ReadWrite >>>>> >>>>> cmsio7.rc.ufl.edu:1094 Server ReadWrite >>>>> >>>>> cmsio9.rc.ufl.edu:1094 Server ReadWrite >>>>> >>>>> cmsio4.rc.ufl.edu:1094 Server ReadWrite >>>>> >>>>> cmsio3.rc.ufl.edu:1094 Server ReadWrite >>>>> >>>>> cmsio8.rc.ufl.edu:1094 Server ReadWrite >>>>> >>>>> cmsio6.rc.ufl.edu:1094 Server ReadWrite >>>>> >>>>> >>>>> ------------------- >>>>> >>>>> ### xrdfs locate -d without "-m" option >>>>> >>>>> >>>>> bash-4.2$ date; xrdfs cmsxrootd2.fnal.gov:1094 locate -d /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >>>>> >>>>> Fri Sep 15 21:27:13 CEST 2023 >>>>> >>>>> [ERROR] Server responded with an error: [3011] No valid location found >>>>> >>>>> >>>>> -------------------- >>>>> >>>>> On the server side, the first one showed: >>>>> >>>>> 230915 14:22:53 18227 acc_Audit: yujun.12664:[log in to unmask] grant host *@lxplus734.cern.ch stat /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >>>>> >>>>> 230915 14:22:53 17913 cms_Receive: cmsxrootd2 125 bytes on 727534553 >>>>> >>>>> 230915 14:22:53 17913 cms_Decode: cmsxrootd2 sent yujun.12664:[log in to unmask] 'Mwcmsio2.rc.ufl.edu:1094 Mrcmsxrootd-site1.fnal.gov:1094 Mrcmsxrootd-site3.fnal.gov:1094 Mrcmsxrootd-site2.fnal.gov:1094' /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >>>>> >>>>> 230915 14:22:53 17913 cms_Receive: cmsxrootd2 32 bytes on 727535577 >>>>> >>>>> >>>>> >>>>> While the 2nd one without "-m" option showed: >>>>> >>>>> >>>>> 230915 14:27:13 27864 acc_Audit: yujun.9149:[log in to unmask] grant host *@lxplus734.cern.ch stat /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >>>>> >>>>> 230915 14:27:13 17913 cms_Receive: cmsxrootd2 135 bytes on 743171027 >>>>> >>>>> 230915 14:27:13 17913 cms_Decode: cmsxrootd2 sent yujun.9149:[log in to unmask] 'Mw[2620:104:1f:1000:faf2:1eff:fed9:dd80]:1094 Mr[2620:6a:0:8420::f8]:1094 Mr[2620:6a:0:8421::243]:1094 Mr[2620:6a:0:8420::f9]:1094' /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >>>>> >>>>> 230915 14:27:13 17913 cms_Receive: cmsxrootd2 4 bytes on 743172051 >>>>> >>>>> >>>>> =================== >>>>> >>>>> >>>>> As you can see, the successful one sent multiple hostnames while the failed one sent multiple IP addresses: >>>>> >>>>> 230915 14:27:13 27864 acc_Audit: yujun.9149:[log in to unmask] grant host *@lxplus734.cern.ch stat /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >>>>> >>>>> 230915 14:27:13 17913 cms_Receive: cmsxrootd2 135 bytes on 743171027 >>>>> >>>>> 230915 14:27:13 17913 cms_Decode: cmsxrootd2 sent yujun.9149:[log in to unmask] 'Mw[2620:104:1f:1000:faf2:1eff:fed9:dd80]:1094 Mr[2620:6a:0:8420::f8]:1094 Mr[2620:6a:0:8421::243]:1094 Mr[2620:6a:0:8420::f9]:1094' /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >>>>> >>>>> 230915 14:27:13 17913 cms_Receive: cmsxrootd2 4 bytes on 743172051 >>>>> >>>>> >>>>> ------------------------------------ >>>>> >>>>> >>>>> In contrast, another USCMS regional redirector xrootd.unl.edu could return results with and without "-m" option: >>>>> >>>>> >>>>> bash-4.2$ date; xrdfs xrootd.unl.edu:1094 locate -d /store/mc/RunIIFall17FSPrePremix/Neutrino_E-10_gun/GEN-SIM-DIGI-RAW/PUFor2016_94X_mcRun2_asymptotic_v3-v1/2810000/7AB76D28-6C1D-EE11-B331-FA163EEB94EB.root >>>>> >>>>> Fri Sep 15 21:38:17 CEST 2023 >>>>> >>>>> cmsxrootd-site3.fnal.gov:1093 Server Read >>>>> >>>>> cmsio5.rc.ufl.edu:1094 Server ReadWrite >>>>> >>>>> cmsio7.rc.ufl.edu:1094 Server ReadWrite >>>>> >>>>> cmsio9.rc.ufl.edu:1094 Server ReadWrite >>>>> >>>>> cmsio4.rc.ufl.edu:1094 Server ReadWrite >>>>> >>>>> cmsio3.rc.ufl.edu:1094 Server ReadWrite >>>>> >>>>> cmsio8.rc.ufl.edu:1094 Server ReadWrite >>>>> >>>>> cmsio6.rc.ufl.edu:1094 Server ReadWrite >>>>> >>>>> cmsxrootd-site1.fnal.gov:1093 Server Read >>>>> >>>>> cmsxrootd-site2.fnal.gov:1093 Server Read >>>>> >>>>> >>>>> I was told their log file showed hostnames were sent even without "-m" option. >>>>> >>>>> >>>>> >>>>> For cmsxrootd2.fnal.gov XRootD redirector, I added >>>>> >>>>> >>>>> xrd.network dnr >>>>> >>>>> >>>>> option in our server side xrootd.cfg but it seems IP addresses still were sent. Could you please tell how I could force hostnames instead of IP addresses to be sent >>>>> >>>>> when running "xrdfs cmsxrootd2.fnal.gov:1094 locate -d" command? >>>>> >>>>> >>>>> Thanks in advance for any advice on this. >>>>> >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Yujun >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> ######################################################################## >>>>> Use REPLY-ALL to reply to list >>>>> >>>>> To unsubscribe from the XROOTD-L list, click the following link: >>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIBAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=bFk3iAgykb1xfdm5362Q9w&m=u4ZeuNN02gPONHd_hstIHONjzndQrVW3s6axSEhW_-qx4IqYlkKlkgW5N3y0N0HY&s=HTkD3kSVa8SHsVmhFbg5SevVFe2CM4HmftLiX3yPCbg&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIBAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=bFk3iAgykb1xfdm5362Q9w&m=u4ZeuNN02gPONHd_hstIHONjzndQrVW3s6axSEhW_-qx4IqYlkKlkgW5N3y0N0HY&s=HTkD3kSVa8SHsVmhFbg5SevVFe2CM4HmftLiX3yPCbg&e=> <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIBAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=bFk3iAgykb1xfdm5362Q9w&m=u4ZeuNN02gPONHd_hstIHONjzndQrVW3s6axSEhW_-qx4IqYlkKlkgW5N3y0N0HY&s=HTkD3kSVa8SHsVmhFbg5SevVFe2CM4HmftLiX3yPCbg&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIBAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=bFk3iAgykb1xfdm5362Q9w&m=u4ZeuNN02gPONHd_hstIHONjzndQrVW3s6axSEhW_-qx4IqYlkKlkgW5N3y0N0HY&s=HTkD3kSVa8SHsVmhFbg5SevVFe2CM4HmftLiX3yPCbg&e=>> >>>>> >>> -------------------------------------------------------------------------------- >>> >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=bFk3iAgykb1xfdm5362Q9w&m=8DWcaSpkMu8Mk-iLymsl2CLja72X84_QpbAai4GwqH1aR8ZMjmsvEg59qsQ3wOb7&s=KEre7NbD46BvBgfpo7nEWYdPDGVfF9G1Y3VE2IN6YYs&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=bFk3iAgykb1xfdm5362Q9w&m=8DWcaSpkMu8Mk-iLymsl2CLja72X84_QpbAai4GwqH1aR8ZMjmsvEg59qsQ3wOb7&s=KEre7NbD46BvBgfpo7nEWYdPDGVfF9G1Y3VE2IN6YYs&e=> >>> <https://urldefense.com/v3/__https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1__;!!Mih3wA!FSW9Tek3rz3nD0Ff-wL6cRUEPKE83ZXADszSIAkXo3sLVs_pQ8ziG7k59rdA2THK2UmsqQnv0uiBTZggmZOs-EfTLl0w3tZWBnohgSwj68vI8L5j$ <https://urldefense.com/v3/__https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1__;!!Mih3wA!FSW9Tek3rz3nD0Ff-wL6cRUEPKE83ZXADszSIAkXo3sLVs_pQ8ziG7k59rdA2THK2UmsqQnv0uiBTZggmZOs-EfTLl0w3tZWBnohgSwj68vI8L5j$>> >>> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=bFk3iAgykb1xfdm5362Q9w&m=8DWcaSpkMu8Mk-iLymsl2CLja72X84_QpbAai4GwqH1aR8ZMjmsvEg59qsQ3wOb7&s=KEre7NbD46BvBgfpo7nEWYdPDGVfF9G1Y3VE2IN6YYs&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=bFk3iAgykb1xfdm5362Q9w&m=8DWcaSpkMu8Mk-iLymsl2CLja72X84_QpbAai4GwqH1aR8ZMjmsvEg59qsQ3wOb7&s=KEre7NbD46BvBgfpo7nEWYdPDGVfF9G1Y3VE2IN6YYs&e=> ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1