Print

Print

Hi, Andrew.

 

Thank you for your kind information.

 

When I was in version 2.16 of dCache, I was providing services in a similar way.

 

CMS AAA can be served without an XRootD local redirector which an XRootD server can connect directly to the XRootD Global redirector.

 

Before, I set the dCache NFS v4.1 was previously mounted to "/pnfs" and then the directory was Global Redirection via the XRootD Server.

 

The problem is the recent incident.

 

I recently upgraded my dCache to v5.2 and suddenly received a large number of XRootD Global Redirection requests.

(I guess that a reason is that the grid jobs using Custom Dataset which was created by a local user requested the data from around the world.)

 

Although we cannot keep track of the number of connections at the time,

 

we noticed a problem with the NFSv4.1 service in dCache when there were approximately 2,000 to 3,000 connections.

 

The symptom is that the dCache NFS directory was hang (/pnfs).

 

When the problem occurred, there was no other solution than a reboot (temporary action was possible with the lazy umount but was eventually frozen).

 

It is unclear if the cause was a problem of the NFS door in dCache. 

 

 

However, after I set up "pss" on the XRootD server, it appears to be responding to quite large connection requests.

 

I have recently confirmed that it works with 5,000 connections.

 

Of course, SAM3 test has problems due to too many connections, such as 139 errors.

 

However, recovery is automatic or at the least, there have been no problems requiring a reboot as before.

 

I personally believe that the method through directory mount will cause problems in the kernel if there are problems due to many connection requests, which will inevitably result in rebooting. 

 

The ultimate solution I think is to do the perfect XRootD Redirection in an XRootD service without fuse or nfs mounts with dCache.

 

For now, "XRootD Server with pss option" is the best configuration for the environment.

 

 

The information you said about a plug-in that performs the cmsd for dCache is a little interesting.

 

Could you give more detailed information about the plugin?

 

Regards,

 


--------------------------------------------------------------------------------------------------
Geonmo Ryu / 류건모

Korea Institute of Science and Technology Information (KISTI)
Global Science Experimental Data Hub Center (GSDC)
245 Daehak-ro, Yuseong-gu, Daejeon, 305-806, Republic of Korea
Tel :  +82-42-869-1639, +82-10-4337-9423
Mail : [log in to unmask] / [log in to unmask]
-------------------------------------------------------------------------------------------------- 

 

-----------------------원본 메세지-----------------------
보낸사람: "Andrew Hanushevsky "<[log in to unmask]>
받는사람: "Geonmo Ryu" <[log in to unmask]>,[log in to unmask]
보낸시간: 2019-09-18 10:45:14 GMT +0900 (ROK)
제목: Re: About XRootD's redirect feature

 

 

Hi Geonmo,
 
Now I see what you want to do. There is actually a way to do this though not everyone is totally happy with it. This is how to do it:
 
1) Setup a regular server type xrootd/cmsd pair on machine A.
2) In the xrootd config put in a static redirect for “/” to the dCache head node.
3) Mount the dCache name space as an NFS file system on machine A.
4) Configure the xrootd/cmsd as if it were exporting the ,mounted NFS file system.
5) Configure the cmsd indicating the manager is the AAA redirector.
 
There are several variations of this but this is likely the simplest one. You will never serve data through the xrootd, you are merely using the xrootd/cmsd to export the dCache namespace to the CMS AAA redirector. That one will redirect clients to your xrootd but it will merely redirect the client to the dCache instance.
 
The is also a way to get rid of the xrootd server if you would like by using the “cms.altds” directive. You may want to consider that as well (you will need to adjust the cms.delay servers so that the cmsd connects without and subscribers. Also, the cmsd will need to run on the same host as the dCache door if you go with this option.
 
You can avoid mounting the dCache name space as well. However, you would need to write a plug-in for the cmsd to do that. It’s pretty simple but no one has asked for one. If you choose that route I can point you to the right place and, well, you can contribute the plug-in back.
 
Andy
 
 
From: [log in to unmask]">"Geonmo Ryu"
Sent: Tuesday, September 17, 2019 5:55 PM
Subject: RE: Re: About XRootD's redirect feature
 

Hello, Andrew.

 

We are trying to support CMS AAA service for CMS research.

 

As far as I know, the XRootD Door in dCache performs as XRootD redirector ("all.role manager").

 

In order to use this XRootD door to service CMS AAA, it must be redirected from that door to global redirector,

 

but dCache does not appear to offer that feature.

(We have found a plug-in with similar functions, but we are hesitant to apply it because it is developed for ATLAS' FAX services not CMS AAA.)

 

As you said, we are currently using the proxy function of XRootD to provide CMS AAA.

 

However, if we use the "pss" option to provide the service, I afraid that the service network performance will be limited to the network bandwidth of one proxy server.

 

If the "redirect" option was fully working like as my idea, I expect that a global request can use all networks bandwidth like as a local job.

 

Of course, since our external network bandwidth is shared 10Gbps and the server has provided 10Gbps, there is no loss.

 

However, after the network equipment has been upgraded, I think that the setting must be changed.

 

 

I wondered if I could provide a better environment.

 

If it was not possible and the "redirect" option was not prepared to perform the function, we will keep the existing settings.

 

Regards, 

 

--------------------------------------------------------------------------------------------------
Geonmo Ryu / 류건모

Korea Institute of Science and Technology Information (KISTI)
Global Science Experimental Data Hub Center (GSDC)
245 Daehak-ro, Yuseong-gu, Daejeon, 305-806, Republic of Korea
Tel :  +82-42-869-1639, +82-10-4337-9423
Mail : [log in to unmask] / [log in to unmask]
--------------------------------------------------------------------------------------------------

 

-----------------------원본 메세지-----------------------
보낸사람: "Andrew Hanushevsky "<[log in to unmask]>
받는사람: "Geonmo Ryu" <[log in to unmask]>,[log in to unmask]
보낸시간: 2019-09-18 05:25:06 GMT +0900 (ROK)
제목: Re: About XRootD's redirect feature

 

 

Hi Ryu,
 
I’m not sure what you wish to accomplish here. Normally, people fronting dCache use an xrootd proxy server to get around firewall issues. Here there seems to be no firewall so I would have thought people would simply directly go to dCache. Why an xrootd in between? Since I am not a dCache expert (though I can get you in contact with one) I am not sure what restrictions there may be when bypassing the dCache head node and simply going to one of the doors. It would appear that there are some restrictions and likely there is missing CGI information for certain operations. That said, I’m surprised that this works at all.
 
Shall I get you in contact with someone who knows more about dCache?
 
Andy
 
From: [log in to unmask]">"Geonmo Ryu"
Sent: Tuesday, September 17, 2019 2:26 AM
Subject: About XRootD's redirect feature
 

Hello, XRootD experts.

 

I have a question about "xrootd.redirect" feature of xrootd.

 

I'm trying to connect the xrootd server (not redirector) to a dCache's xrootd door as a backend.

 

There are many examples, so I'm going to try to follow them, but it doesn't work well.

 

The settings for the xrootd server I set are as follows.

xrd.port 1097

all.export /

all.role server

xrootd.redirect cms-t2-se01.sdfarm.kr:1094 all /

ofs.forward cms-t2-se01.sdfarm.kr:1094 all

 

cms.dfs lookup central

cms.trace all

Symptoms that followed these settings are as follows:

1. Directory navigation works well. The directory structure of dCache xrootd door on the 1094 port is displayed well.

2. Meta commands such as "stat" is work well for files.

3. Attempting to open or copy a file (such as root command or xrdcp) results in an Internal Error.

[root@cms-t2-se01 xrootd]# xrdfs cms-t2-se01.sdfarm.kr:1097 ls /pnfs/sdfarm.kr/data/cms/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000

 

/pnfs/sdfarm.kr/data/cms/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root

/pnfs/sdfarm.kr/data/cms/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root

/pnfs/sdfarm.kr/data/cms/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/CE860B10-5D76-E711-BCA8-FA163EAA761A.root

 

[root@cms-t2-se01 xrootd]# xrdfs cms-t2-se01.sdfarm.kr:1097 stat /pnfs/sdfarm.kr/data/cms/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root

Path:   /pnfs/sdfarm.kr/data/cms/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root

Id:     0

Size:   240461986

MTime:  2017-10-07 00:23:34

Flags:  48 (IsReadable|IsWritable)

 

(base) [geonmo@ui10 ~]$ root -l  root://cms-t2-se01.sdfarm.kr:1097///pnfs/sdfarm.kr/data/cms/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root

root [0]

Attaching file root://cms-t2-se01.sdfarm.kr:1097//pnfs/sdfarm.kr/data/cms/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root as _file0...

Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3011] Unable to open /pnfs/sdfarm.kr/data/cms/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root; no such file or directory

(TFile *) nullptr

 

xrootd log:

[root@cms-t2-se01 xrootd]# tail -n 3 /var/log/xrootd/redir/xrootd.log

190917 18:24:03 22127 XrootdXeq: geonmo.1763304:22@[2001:320:15:124:d6ae:52ff:fe8c:6610] disc 0:00:01

190917 18:24:08 25086 XrootdXeq: geonmo.1763339:21@[2001:320:15:124:d6ae:52ff:fe8c:6610] pub IP64 login

190917 18:24:08 25086 ofs_open: geonmo.1763339:21@[2001:320:15:124:d6ae:52ff:fe8c:6610] Unable to open /pnfs/sdfarm.kr/data/cms/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root; no such file or directory

 

How can I use it as a complete redirect?

 

Regards,

 

 

 

--------------------------------------------------------------------------------------------------
Geonmo Ryu / 류건모

Korea Institute of Science and Technology Information (KISTI)
Global Science Experimental Data Hub Center (GSDC)
245 Daehak-ro, Yuseong-gu, Daejeon, 305-806, Republic of Korea
Tel :  +82-42-869-1639, +82-10-4337-9423
Mail : [log in to unmask] / [log in to unmask]
--------------------------------------------------------------------------------------------------

 

Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1

 

Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1

 

 


Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1