Hi Wei,
Actually, the scaling issues you're referring to have been fixed. In the
cases where the server supports bulk stat, the client will use it. In
the cases where it does not, the client will send as many stats in
parallel as it can to compensate for the latency.
https://github.com/xrootd/xrootd/blob/master/src/XrdCl/XrdClFileSystem.cc#L1021
Lukasz
On Sun, Apr 9, 2017, at 22:14, Yang, Wei wrote:
> Hi Stephan,
>
>
> I think both suffer from the scaling issues. I know that xrdfs can work.
> If you just read the dirents without stat(), the scaling issue is
> minimized. With small number of dirents in a directory, it may still be
> OK.
>
>
> It looks to me that the python binding can't do what you wanted (Maybe it
> is just a plain implementation over the xrootd posix IO). In that sense,
> it is not a choice for you.
>
>
> Looking at the Xrootd's usage history, I don't think dir listing is the
> mainstream usage in xrootd - Babar or LHC experiments always have their
> own database to keep track of file locations, so they don't do dir
> listing.
>
>
> regards,
>
> --
> Wei Yang | [log in to unmask] | 650-926-3338
> ________________________________
> From: Stephan Zimmer <[log in to unmask]>
> Sent: Sunday, April 9, 2017 11:24 AM
> To: Yang, Wei
> Cc: Stephan Zimmer; xrootd-l
> Subject: Re: python bindings raise error, while standard xrdfs is working
> as expected
>
> Hi Wei,
> Thanks for this explanation. Does this mean you recommend using xrdfs
> over the python bindings? Or do both suffer from the same scaling
> problems?
> Thanks,
> Stephan
>
> Sent from BlueMail<http://www.bluemail.me/r?b=9479>
> On Apr 9, 2017, at 20:13, "Yang, Wei"
> <[log in to unmask]<mailto:[log in to unmask]>> wrote:
>
> xrdfs understands the topology of a xrootd cluster, and therefore piece
> together the dirents from all data servers. I guess the xrootd python
> binding doesn't have this function.
>
> regardless, doing this from a remote client (be that xrdfs or others) is
> not scalable. If a dir has many items, the latency over WAN will kill the
> performance if you do an equivalent of "ls -l" (readdir() + stat())
>
> --
> Wei Yang | [log in to unmask] | 650-926-3338
>
> ________________________________
>
> From: [log in to unmask] <[log in to unmask]> on behalf
> of Stephan Zimmer <[log in to unmask]>
> Sent: Sunday, April 9, 2017 3:15 AM
> To: xrootd-l
> Subject: python bindings raise error, while standard xrdfs is working as
> expected
>
> Hi XrootD list,
> i have setup some automatic agents which use the python bindings for
> xrootd (version: 4.5.0).
>
> In python I call:
>
> from XRootD import client
> xc = client.FileSystem("root://xrootd-dampe.cloud.ba.infn.it:1094")
> xc.dirlist("/MC/simu")
> (<status: 1, code: 400, ok: False, errno: 3011, error: True, message:
> '[ERROR] Server responded with an error: [3011] Unable to open directory
> /MC/simu; no such file or directory\n', fatal: False, shellcode: 54>,
> None)
>
> the server appears to be there though:
> xc.ping()
> (<status: 0, code: 0, ok: True, errno: 0, error: False, message:
> '[SUCCESS] ', fatal: False, shellcode: 0>, None)
>
> and when i do the same with xrdfs, i get the directory listing correctly
> returned.
>
> [zimmers@login1 log]$ xrdfs xrootd-dampe.cloud.ba.infn.it:1094 ls
> /MC/simu
> /MC/simu/allMuon-v5r4p0_1GeV_100GeV
> /MC/simu/allElectron_v5r3p0_1GeV_100GeV_Orbit_57388
> /MC/simu/flatProton-v5r3p1_500GeV_5TeV
> /MC/simu/allProton-v5r3p1_100GeV_10TeV-p3
> ...
>
> so what am I perhaps doing wrongly here? Or could this be due to some
> xrootd server configuration part?
> Thanks!
> -Stephan
>
>
>
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
