It's fixed on the API level, so everyone who uses the FileSystem class
benefits. It laverages the kXR_dstat flag
(http://xrootd.org/doc/dev45/XRdv310.pdf) when possible, and submits the
requests in parallel when not.
Lukasz
On Mon, Apr 10, 2017, at 16:36, Yang, Wei wrote:
> Hi Lukasz,
>
> thanks for the clarification. you mean it is fixed in xrdfs?
>
> regards,
> --
> Wei Yang | [log in to unmask] | 650-926-3338
>
> ________________________________________
> From: Lukasz Janyst <[log in to unmask]>
> Sent: Monday, April 10, 2017 1:54 AM
> To: Yang, Wei; Stephan Zimmer
> Cc: xrootd-l
> Subject: Re: python bindings raise error, while standard xrdfs is working
> as expected
>
> Hi Wei,
>
> Actually, the scaling issues you're referring to have been fixed. In the
> cases where the server supports bulk stat, the client will use it. In
> the cases where it does not, the client will send as many stats in
> parallel as it can to compensate for the latency.
>
> https://github.com/xrootd/xrootd/blob/master/src/XrdCl/XrdClFileSystem.cc#L1021
>
> Lukasz
>
> On Sun, Apr 9, 2017, at 22:14, Yang, Wei wrote:
> > Hi Stephan,
> >
> >
> > I think both suffer from the scaling issues. I know that xrdfs can work.
> > If you just read the dirents without stat(), the scaling issue is
> > minimized. With small number of dirents in a directory, it may still be
> > OK.
> >
> >
> > It looks to me that the python binding can't do what you wanted (Maybe it
> > is just a plain implementation over the xrootd posix IO). In that sense,
> > it is not a choice for you.
> >
> >
> > Looking at the Xrootd's usage history, I don't think dir listing is the
> > mainstream usage in xrootd - Babar or LHC experiments always have their
> > own database to keep track of file locations, so they don't do dir
> > listing.
> >
> >
> > regards,
> >
> > --
> > Wei Yang | [log in to unmask] | 650-926-3338
> > ________________________________
> > From: Stephan Zimmer <[log in to unmask]>
> > Sent: Sunday, April 9, 2017 11:24 AM
> > To: Yang, Wei
> > Cc: Stephan Zimmer; xrootd-l
> > Subject: Re: python bindings raise error, while standard xrdfs is working
> > as expected
> >
> > Hi Wei,
> > Thanks for this explanation. Does this mean you recommend using xrdfs
> > over the python bindings? Or do both suffer from the same scaling
> > problems?
> > Thanks,
> > Stephan
> >
> > Sent from BlueMail<http://www.bluemail.me/r?b=9479>
> > On Apr 9, 2017, at 20:13, "Yang, Wei"
> > <[log in to unmask]<mailto:[log in to unmask]>> wrote:
> >
> > xrdfs understands the topology of a xrootd cluster, and therefore piece
> > together the dirents from all data servers. I guess the xrootd python
> > binding doesn't have this function.
> >
> > regardless, doing this from a remote client (be that xrdfs or others) is
> > not scalable. If a dir has many items, the latency over WAN will kill the
> > performance if you do an equivalent of "ls -l" (readdir() + stat())
> >
> > --
> > Wei Yang | [log in to unmask] | 650-926-3338
> >
> > ________________________________
> >
> > From: [log in to unmask] <[log in to unmask]> on behalf
> > of Stephan Zimmer <[log in to unmask]>
> > Sent: Sunday, April 9, 2017 3:15 AM
> > To: xrootd-l
> > Subject: python bindings raise error, while standard xrdfs is working as
> > expected
> >
> > Hi XrootD list,
> > i have setup some automatic agents which use the python bindings for
> > xrootd (version: 4.5.0).
> >
> > In python I call:
> >
> > from XRootD import client
> > xc = client.FileSystem("root://xrootd-dampe.cloud.ba.infn.it:1094")
> > xc.dirlist("/MC/simu")
> > (<status: 1, code: 400, ok: False, errno: 3011, error: True, message:
> > '[ERROR] Server responded with an error: [3011] Unable to open directory
> > /MC/simu; no such file or directory\n', fatal: False, shellcode: 54>,
> > None)
> >
> > the server appears to be there though:
> > xc.ping()
> > (<status: 0, code: 0, ok: True, errno: 0, error: False, message:
> > '[SUCCESS] ', fatal: False, shellcode: 0>, None)
> >
> > and when i do the same with xrdfs, i get the directory listing correctly
> > returned.
> >
> > [zimmers@login1 log]$ xrdfs xrootd-dampe.cloud.ba.infn.it:1094 ls
> > /MC/simu
> > /MC/simu/allMuon-v5r4p0_1GeV_100GeV
> > /MC/simu/allElectron_v5r3p0_1GeV_100GeV_Orbit_57388
> > /MC/simu/flatProton-v5r3p1_500GeV_5TeV
> > /MC/simu/allProton-v5r3p1_100GeV_10TeV-p3
> > ...
> >
> > so what am I perhaps doing wrongly here? Or could this be due to some
> > xrootd server configuration part?
> > Thanks!
> > -Stephan
> >
> >
> >
> >
> > ########################################################################
> > Use REPLY-ALL to reply to list
> >
> > To unsubscribe from the XROOTD-L list, click the following link:
> > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
> >
> > ########################################################################
> > Use REPLY-ALL to reply to list
> >
> > To unsubscribe from the XROOTD-L list, click the following link:
> > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
