Hi Andy,
On 9/2/16 2:27 PM, Andrew Hanushevsky wrote:
> Hi Marian,
>
> I see that the cluster configuration for cmsxrootd.fnal.gov is:
>
> 0: [::ffff:129.93.239.131] xrootd.unl.edu v4.3.0
> 1: [::ffff:131.225.204.174] cmsxrootd1.fnal.gov v4.3.0-rc3
> 2: [2600:900:6:1101:5054:ff:fe00:70cb] xrootd.unl.edu v4.3.0
which goofer you used get this info? Let me learn something here, please.
This is actually problem now with global redirectors, too. They have v6
active interface and everyone using CMSSW now against global redirecotr
cms-xrd-global.cern.ch hit this. We're not sure whether this is some
coincidence or not with lxplus, but it's very weird because global
redirector is on dual-stack for long time and noone noticed any issues
until couple days ago when we put in EU regional level of redirection
dual-stack host (xrootd-cms-redir-01.cr.cnaf.infn.it). Today we've taken
it out, but problem for users is still there for this particular
"dual-stack: false" reason.
> Any particular reason why the fnal one is not on a supported version of
> xrootd? In any case, I will look to see if we addressed this issue in
> 4.4.0.
not in OSG repo yet, just created OSG SW JIRA ticket update package to
4.4.0. FNAL is still in -rc3, yeah, I need to ping them. Let me know if
you find this fixed in 4.4.0, that'll be great.
> In the mean time, could you do:
>
> xrdfs cmsxrootd.fnal.gov:1094 locate -h <the_problem_path>
>
> and return the result to me? I can't do it as it appears you have turned
> on access control for the locate command.
looks like I'm blocked, too (with my proxy):
$ xrdfs cmsxrootd.fnal.gov:1094 locate -h
/store/mc/RunIISpring16reHLT80/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/AODSIM/PUSpring16RAWAODSIM_reHLT_80X_mcRun2_asymptotic_v14-v1/40000/003A70D5-E55C-E611-9B14-0025905C53A8.root
[ERROR] Server responded with an error: [3010] Unable to locate
/store/mc/RunIISpring16reHLT80/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/AODSIM/PUSpring16RAWAODSIM_reHLT_80X_mcRun2_asymptotic_v14-v1/40000/003A70D5-E55C-E611-9B14-0025905C53A8.root;
permission denied
Where is this control configurable, on which level and what knobs, please?
Thanks,
Marian
> Andy
>
> B.T.W Version 4.4.0 is much better than 4.3.0 so you should consider
> upgrading.
>
> On Fri, 2 Sep 2016, Marian Zvada wrote:
>
>> Hello!
>>
>> We're experiencing issues with older clients vs redirector IPv6
> enabled where client host is dual-stack. Relevant piece of behavior is
> from the log as follows:
>>
>> `[2016-09-02 19:40:10.529888 +0200][Debug ][XRootDTransport ]
> [cmsxrootd.fnal.gov:1094 #0.0] Sending out kXR_login request, username:
> zvada, cgi: ?xrd.cc=ch&xrd.tz=1&xrd.appname=xrdcp&xrd.info=, dual-stack:
> false`
>> ...
>> ...
>> ```
>> [2016-09-02 19:41:00.221320 +0200][Debug ][File ]
> [0x2000000@root://cmsxrootd.fnal.gov:1094//store/mc/RunIISpring16reHLT80/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/AODSIM/PUSpring16RAWAODSIM_reHLT_80X_mcRun2_asymptotic_v14-v1/40000/003A70D5-E55C-E611-9B14-0025905C53A8.root]
> Error while opening at cms-xrd-transit.cern.ch:1094: [ERROR] Server
> responded with an error: [3011] No servers are available to read the file.
>> [0B/0B][100%][==================================================][0B/s]
>> Run: [ERROR] Server responded with an error: [3011] No servers are
> available to read the file.
>> $ xrdcp -version
>> v4.0.4
>> ```
>> Forcing XRD_NETWORKSTACK=IPv4 solve the issue on client side, though:
>> ```
>> $ XRD_NETWORKSTACK=IPv4 xrdcp -f
> root://cmsxrootd.fnal.gov//store/mc/RunIISpring16reHLT80/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/AODSIM/PUSpring16RAWAODSIM_reHLT_80X_mcRun2_asymptotic_v14-v1/40000/003A70D5-E55C-E611-9B14-0025905C53A8.root
> /dev/null
>> [80MB/3.495GB][ 2%][=> ][8.889MB/s]
>> $ xrdcp -version
>> v4.0.4
>> ```
>> Suggestion is provide on the server side a configuration parameter to
> force dual-stack to true - we'd assume that IPv4 is always present.
> Clients >=v4.3.x handle dual-stack situation well, but for users running
> anything below that would be preferable be cautions and act on the
> server side with config tweak if possible.
>>
>> Thanks,
>> Marian
>>
>> --
>> You are receiving this because you are subscribed to this thread.
>> Reply to this email directly or view it on GitHub:
>> https://github.com/xrootd/xrootd/issues/411
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/xrootd/xrootd/issues/411#issuecomment-244466567>, or
> mute the thread
> <https://github.com/notifications/unsubscribe-auth/AFfkqjI-5tU0e3_SDAUAZowJ6QnHFKy1ks5qmHiFgaJpZM4Jz8WB>.
>
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/xrootd/xrootd","title":"xrootd/xrootd","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/xrootd/xrootd"}},"updates":{"snippets":[{"icon":"PERSON","message":"@zvada in #411: Hi Andy,\n\nOn 9/2/16 2:27 PM, Andrew Hanushevsky wrote:\n\u003e Hi Marian,\n\u003e\n\u003e I see that the cluster configuration for cmsxrootd.fnal.gov is:\n\u003e\n\u003e 0: [::ffff:129.93.239.131] xrootd.unl.edu v4.3.0\n\u003e 1: [::ffff:131.225.204.174] cmsxrootd1.fnal.gov v4.3.0-rc3\n\u003e 2: [2600:900:6:1101:5054:ff:fe00:70cb] xrootd.unl.edu v4.3.0\n\nwhich goofer you used get this info? Let me learn something here, please.\n\nThis is actually problem now with global redirectors, too. They have v6 \nactive interface and everyone using CMSSW now against global redirecotr \ncms-xrd-global.cern.ch hit this. We're not sure whether this is some \ncoincidence or not with lxplus, but it's very weird because global \nredirector is on dual-stack for long time and noone noticed any issues \nuntil couple days ago when we put in EU regional level of redirection \ndual-stack host (xrootd-cms-redir-01.cr.cnaf.infn.it). Today we've taken \nit out, but problem for users is still there for this particular \n\"dual-stack: false\" reason.\n\n\u003e Any particular reason why the fnal one is not on a supported version of\n\u003e xrootd? In any case, I will look to see if we addressed this issue in\n\u003e 4.4.0.\n\nnot in OSG repo yet, just created OSG SW JIRA ticket update package to \n4.4.0. FNAL is still in -rc3, yeah, I need to ping them. Let me know if \nyou find this fixed in 4.4.0, that'll be great.\n\n\u003e In the mean time, could you do:\n\u003e\n\u003e xrdfs cmsxrootd.fnal.gov:1094 locate -h \u003cthe_problem_path\u003e\n\u003e\n\u003e and return the result to me? I can't do it as it appears you have turned\n\u003e on access control for the locate command.\n\nlooks like I'm blocked, too (with my proxy):\n$ xrdfs cmsxrootd.fnal.gov:1094 locate -h \n/store/mc/RunIISpring16reHLT80/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/AODSIM/PUSpring16RAWAODSIM_reHLT_80X_mcRun2_asymptotic_v14-v1/40000/003A70D5-E55C-E611-9B14-0025905C53A8.root \n\n[ERROR] Server responded with an error: [3010] Unable to locate \n/store/mc/RunIISpring16reHLT80/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/AODSIM/PUSpring16RAWAODSIM_reHLT_80X_mcRun2_asymptotic_v14-v1/40000/003A70D5-E55C-E611-9B14-0025905C53A8.root; \npermission denied\n\nWhere is this control configurable, on which level and what knobs, please?\n\nThanks,\nMarian\n\n\u003e Andy\n\u003e\n\u003e B.T.W Version 4.4.0 is much better than 4.3.0 so you should consider\n\u003e upgrading.\n\u003e\n\u003e On Fri, 2 Sep 2016, Marian Zvada wrote:\n\u003e\n\u003e\u003e Hello!\n\u003e\u003e\n\u003e\u003e We're experiencing issues with older clients vs redirector IPv6\n\u003e enabled where client host is dual-stack. Relevant piece of behavior is\n\u003e from the log as follows:\n\u003e\u003e\n\u003e\u003e `[2016-09-02 19:40:10.529888 +0200][Debug ][XRootDTransport ]\n\u003e [cmsxrootd.fnal.gov:1094 #0.0] Sending out kXR_login request, username:\n\u003e zvada, cgi: ?xrd.cc=ch\u0026xrd.tz=1\u0026xrd.appname=xrdcp\u0026xrd.info=, dual-stack:\n\u003e false`\n\u003e\u003e ...\n\u003e\u003e ...\n\u003e\u003e ```\n\u003e\u003e [2016-09-02 19:41:00.221320 +0200][Debug ][File ]\n\u003e [0x2000000@root://cmsxrootd.fnal.gov:1094//store/mc/RunIISpring16reHLT80/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/AODSIM/PUSpring16RAWAODSIM_reHLT_80X_mcRun2_asymptotic_v14-v1/40000/003A70D5-E55C-E611-9B14-0025905C53A8.root]\n\u003e Error while opening at cms-xrd-transit.cern.ch:1094: [ERROR] Server\n\u003e responded with an error: [3011] No servers are available to read the file.\n\u003e\u003e [0B/0B][100%][==================================================][0B/s]\n\u003e\u003e Run: [ERROR] Server responded with an error: [3011] No servers are\n\u003e available to read the file.\n\u003e\u003e $ xrdcp -version\n\u003e\u003e v4.0.4\n\u003e\u003e ```\n\u003e\u003e Forcing XRD_NETWORKSTACK=IPv4 solve the issue on client side, though:\n\u003e\u003e ```\n\u003e\u003e $ XRD_NETWORKSTACK=IPv4 xrdcp -f\n\u003e root://cmsxrootd.fnal.gov//store/mc/RunIISpring16reHLT80/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/AODSIM/PUSpring16RAWAODSIM_reHLT_80X_mcRun2_asymptotic_v14-v1/40000/003A70D5-E55C-E611-9B14-0025905C53A8.root\n\u003e /dev/null\n\u003e\u003e [80MB/3.495GB][ 2%][=\u003e ][8.889MB/s]\n\u003e\u003e $ xrdcp -version\n\u003e\u003e v4.0.4\n\u003e\u003e ```\n\u003e\u003e Suggestion is provide on the server side a configuration parameter to\n\u003e force dual-stack to true - we'd assume that IPv4 is always present.\n\u003e Clients \u003e=v4.3.x handle dual-stack situation well, but for users running\n\u003e anything below that would be preferable be cautions and act on the\n\u003e server side with config tweak if possible.\n\u003e\u003e\n\u003e\u003e Thanks,\n\u003e\u003e Marian\n\u003e\u003e\n\u003e\u003e --\n\u003e\u003e You are receiving this because you are subscribed to this thread.\n\u003e\u003e Reply to this email directly or view it on GitHub:\n\u003e\u003e https://github.com/xrootd/xrootd/issues/411\n\u003e\n\u003e —\n\u003e You are receiving this because you authored the thread.\n\u003e Reply to this email directly, view it on GitHub\n\u003e \u003chttps://github.com/xrootd/xrootd/issues/411#issuecomment-244466567\u003e, or\n\u003e mute the thread\n\u003e \u003chttps://github.com/notifications/unsubscribe-auth/AFfkqjI-5tU0e3_SDAUAZowJ6QnHFKy1ks5qmHiFgaJpZM4Jz8WB\u003e.\n\u003e\n"}],"action":{"name":"View Issue","url":"https://github.com/xrootd/xrootd/issues/411#issuecomment-244491450"}}}
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
