On 10/16/2013 08:58 PM, Andrew Hanushevsky wrote: > Hi Adrian, Hi! > OK, looks like this is a plugin problem. The oss plugin is being > initialized twice. Probably by libXrdxFtsOfs.so. Who wrote that? hmm, might be some ALICE thing .. i will ask > The cms part doesn't have this problem and I assume that thie figures it > displays: > > 131012 10:07:18 13529 Meter: Found 3 filesystem(s); 39TB total (100% > util); 18GB free (6GB max) > > are correct. This is one for the person who developed that plugin. yes, this are the correct numbers! Thanks a lot for pinpointing the problem!!! Adrian > > Andy > > On Wed, 16 Oct 2013, Adrian Sevcenco wrote: > >> Hi! Any news on this? Sorry for thread bumping but i have an storage >> server waiting to be put online and spurious test error with "no space >> left on disk" and i don't know what to do... >> >> Can i help the debugging process somehow? >> >> Thank you! >> Adrian >> >> >> On 10/12/2013 10:20 AM, Adrian Sevcenco wrote: >>> On 10/12/2013 12:44 AM, Andrew Hanushevsky wrote: >>>> Hi Adrian, >>> Hi! >>> >>>> Very odd. You must have a log somewhere that includes xrootd >>>> initialization. I'd like to see that to find out why it thinks the >>>> config is not what appears to be in the config file. Could you send me >>>> the snippet that just includes the lines from start-up to >>>> "initialization completed". >>> below the outputs with cmslog and xrdlog (preview : cms see corect size >>> in xrd:: storage system initialization i have double inclusion of the >>> storage filesystems) >>> >>> Thanks!!! >>> Adrian >>> >>> in cmslog : >>> Copr. 2004-2012 Stanford University, xrd version v3.3.2 >>> ++++++ cmsd [log in to unmask] initialization started. >>> Config using configuration file >>> /home/aliprod/xrdserver/etc/xrootd/server/xrootd.cf >>> =====> xrd.protocol xrootd * >>> =====> all.adminpath /home/aliprod/xrdserver/admin >>> Config warning: ignoring unknown xrd directive 'pidpath'. >>> =====> xrd.pidpath >>> =====> xrd.port 1094 >>> =====> xrd.sched mint 32 maxt 2048 avlt 512 idle 780 >>> =====> xrd.network buffsz 0 nodnr >>> Config maximum number of connections restricted to 65000 >>> Copr. 2007 Stanford University/SLAC cmsd. >>> ++++++ [log in to unmask] phase 1 initialization started. >>> =====> all.export / nolock r/w nocheck norcreate >>> =====> all.role server >>> =====> all.manager rd.spacescience.ro 3122 >>> =====> all.adminpath /home/aliprod/xrdserver/admin >>> =====> cms.pidpath /home/aliprod/xrdserver/admin >>> =====> cms.sched cpu 10 io 10 space 80 >>> =====> oss.defaults nomig nodread nocheck norcreate nolock >>> =====> oss.namelib >>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so >>> /home/aliprod/data >>> =====> oss.localroot /storage01/xrdnamespace >>> =====> cms.space min 1g 1g >>> The following paths are available to the redirector: >>> w / >>> >>> ------ [log in to unmask] phase 1 server initialization >>> completed. >>> ++++++ [log in to unmask] phase 2 server initialization >>> started. >>> 131012 10:07:18 13529 Configure2 Global System Identification: anon-s >>> 3122rd.spacescience.ro >>> Plugin loaded unversioned XrdOucgetName2Name from namelib >>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so >>> ++++++ XrdAggregatingN2N initializing. Local lfn prefix >>> '/home/aliprod/data'. >>> ++++++ XrdAggregatingN2N initializing. Remote root is null >>> ++++++ Storage system initialization started. >>> =====> all.export / nolock r/w nocheck norcreate >>> =====> oss.defaults nomig nodread nocheck norcreate nolock >>> =====> oss.alloc 512M 2 0 >>> =====> oss.fdlimit * max >>> =====> oss.space public /storage01/xrddata >>> =====> oss.space public /storage02/xrddata >>> =====> oss.space public /storage03/xrddata >>> =====> oss.namelib >>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so >>> /home/aliprod/data >>> =====> oss.localroot /storage01/xrdnamespace >>> Plugin loaded unversioned XrdOucgetName2Name from namelib >>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so >>> ++++++ XrdAggregatingN2N initializing. Local lfn prefix >>> '/home/aliprod/data'. >>> ++++++ XrdAggregatingN2N initializing. Remote root is null >>> ++++++ Configuring standalone mode . . . >>> 131012 10:07:18 13529 oss_AioInit: started AIO read signal thread; >>> tid=3135912256 >>> 131012 10:07:18 13529 oss_AioInit: started AIO write signal thread; >>> tid=3136964928 >>> XrdAggregatingN2N processing. buff='/' >>> XrdAggregatingN2N processing. buff='/home/aliprod/data/' >>> XrdAggregatingN2N processing. buff='/' >>> XrdAggregatingN2N processing. buff='/home/aliprod/data/' >>> Config effective /home/aliprod/xrdserver/etc/xrootd/server/xrootd.cf oss >>> configuration: >>> oss.alloc 536870912 2 0 >>> oss.cachescan 600 >>> oss.fdlimit 32500 65000 >>> oss.maxsize 0 >>> oss.namelib >>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so >>> oss.localroot /storage01/xrdnamespace >>> oss.trace fff >>> oss.xfr 1 deny 10800 keep 1200 >>> oss.memfile off max 2071848960 >>> oss.space public /storage01/xrddata >>> oss.space public /storage02/xrddata >>> oss.space public /storage03/xrddata >>> oss.defaults r/w nocheck nodread nomig norcreate nopurge >>> nostage xattr >>> oss.path / r/w nocheck nodread nomig norcreate nopurge nostage >>> xattr >>> ------ Storage system initialization completed. >>> 131012 10:07:18 13529 Start Srv=0 dfs=0 lcl=0 Pre=1 dmLife=0 0 >>> 131012 10:07:18 13529 Start Lim=0 0 fix=0 Qmax=1 >>> 131012 10:07:18 13529 calcSpace New fs info; maxfree=6473MB >>> utilized=100% >>> 131012 10:07:18 13529 Meter: Found 3 filesystem(s); 39TB total (100% >>> util); 18GB free (6GB max) >>> ------ [log in to unmask] phase 2 server initialization >>> completed. >>> >>> >>> ################################################################## >>> >>> xrdlog: >>> >>> >>> 131012 10:07:18 13496 Scalla is starting. . . >>> Copr. 2004-2012 Stanford University, xrd version v3.3.2 >>> >>> ++++++ xrootd [log in to unmask] initialization started. >>> Config using configuration file >>> /home/aliprod/xrdserver/etc/xrootd/server/xrootd.cf >>> =====> xrd.protocol xrootd * >>> =====> all.adminpath /home/aliprod/xrdserver/admin >>> Config warning: ignoring unknown xrd directive 'pidpath'. >>> =====> xrd.pidpath >>> =====> xrd.port 1094 >>> =====> xrd.sched mint 32 maxt 2048 avlt 512 idle 780 >>> =====> xrd.network buffsz 0 nodnr >>> Config maximum number of connections restricted to 65000 >>> Copr. 2012 Stanford University, xrootd protocol 2.9.7 version v3.3.2 >>> ++++++ xrootd protocol initialization started. >>> 131012 10:07:18 13496 XrootdConfig: non-absolute export path - server >>> =====> all.export / nolock >>> =====> xrootd.async off >>> =====> xrootd.fslib /home/aliprod/xrdserver/lib64/libXrdxFtsOfs.so >>> =====> xrootd.seclib /home/aliprod/xrdserver/lib64/libXrdSec.so >>> Config exporting / >>> Plugin loaded >>> >>> >>> ++++++ Authentication system initialization started. >>> sec_PM: Loading unix protocol object from >>> /home/aliprod/xrdserver/lib64/libXrdSecunix.so >>> Plugin loaded >>> =====> sec.protocol /home/aliprod/xrdserver/lib64 unix >>> Config 1 authentication directives processed in >>> /home/aliprod/xrdserver/etc/xrootd/server/xrootd.cf >>> 131012 10:07:18 13496 sec_ProtBind_Complete: Default sectoken built: >>> '&P=unix' >>> ------ Authentication system initialization completed. >>> Plugin loaded unversioned XrdSfsGetFileSystem from fslib >>> /home/aliprod/xrdserver/lib64/libXrdxFtsOfs.so >>> ++++++ (c) 2012 CERN/IT-DSS v 2.0 >>> >>> >>> ++++++ Storage system initialization started. >>> =====> all.export / nolock r/w nocheck norcreate >>> =====> oss.defaults nomig nodread nocheck norcreate nolock >>> =====> oss.alloc 512M 2 0 >>> =====> oss.fdlimit * max >>> =====> oss.space public /storage01/xrddata >>> =====> oss.space public /storage02/xrddata >>> =====> oss.space public /storage03/xrddata >>> =====> oss.namelib >>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so >>> /home/aliprod/data >>> =====> oss.localroot /storage01/xrdnamespace >>> Plugin loaded unversioned XrdOucgetName2Name from namelib >>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so >>> ++++++ XrdAggregatingN2N initializing. Local lfn prefix >>> '/home/aliprod/data'. >>> ++++++ XrdAggregatingN2N initializing. Remote root is null >>> 131012 10:07:18 13496 oss_AioInit: started AIO read signal thread; >>> tid=52988224 >>> 131012 10:07:18 13496 oss_AioInit: started AIO write signal thread; >>> tid=54040896 >>> XrdAggregatingN2N processing. buff='/' >>> XrdAggregatingN2N processing. buff='/home/aliprod/data/' >>> XrdAggregatingN2N processing. buff='/' >>> XrdAggregatingN2N processing. buff='/home/aliprod/data/' >>> Config effective /home/aliprod/xrdserver/etc/xrootd/server/xrootd.cf oss >>> configuration: >>> oss.alloc 536870912 2 0 >>> oss.cachescan 600 >>> oss.fdlimit 32500 65000 >>> oss.maxsize 0 >>> oss.namelib >>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so >>> oss.localroot /storage01/xrdnamespace >>> oss.trace fff >>> oss.xfr 1 deny 10800 keep 1200 >>> oss.memfile off max 2071848960 >>> oss.space public /storage01/xrddata >>> oss.space public /storage02/xrddata >>> oss.space public /storage03/xrddata >>> oss.defaults r/w nocheck nodread nomig norcreate nopurge >>> nostage xattr >>> oss.path / r/w nocheck nodread nomig norcreate nopurge nostage >>> xattr >>> ------ Storage system initialization completed. >>> >>> >>> =====> ftsofs.thirdparty: yes >>> =====> ftsofs.thirdparty.slots: 20 >>> =====> ftsofs.thirdparty.rate: 50 Mb/s >>> =====> ftsofs.thirdparty.statedirectory: /home/aliprod/xrdserver/admin >>> ++++++ File system initialization started. >>> =====> all.role server >>> =====> ofs.trace open >>> =====> ofs.authlib /home/aliprod/xrdserver/lib64/libXrdAliceTokenAcc.so >>> =====> ofs.authorize >>> Plugin loaded unversioned XrdAccAuthorizeObject from authlib >>> /home/aliprod/xrdserver/lib64/libXrdAliceTokenAcc.so >>> ++++++ (c) 2008 CERN/IT-DM-SMD AliceTokenAcc (Alice Token Access >>> Authorization) v 1.0 >>> =====> XrdAliceTokenAcc: No Authorizationfile set via environment >>> variable 'TTOKENAUTHZ_AUTHORIZATIONFILE' >>> =====> XrdAliceTokenAcc: No Authorizationfile like >>> '/etc/grid-security/xrootd/TkAuthz.Authorization' found >>> =====> XrdAliceTokenAcc: No Authorizationfile like >>> '/home/aliprod/.globus/xrootd/TkAuthz.Authorization' found >>> =====> XrdAliceTokenAcc: Using Authorizationfile >>> '/home/aliprod/.authz/xrootd/TkAuthz.Authorization'! >>> ------ AliceTokenAcc initialization completed >>> >>> >>> ++++++ Storage system initialization started. >>> =====> all.export / nolock r/w nocheck norcreate >>> =====> oss.defaults nomig nodread nocheck norcreate nolock >>> =====> oss.alloc 512M 2 0 >>> =====> oss.fdlimit * max >>> =====> oss.space public /storage01/xrddata >>> =====> oss.space public /storage02/xrddata >>> =====> oss.space public /storage03/xrddata >>> =====> oss.namelib >>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so >>> /home/aliprod/data >>> =====> oss.localroot /storage01/xrdnamespace >>> Plugin loaded unversioned XrdOucgetName2Name from namelib >>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so >>> >>> ++++++ XrdAggregatingN2N initializing. Local lfn prefix >>> '/home/aliprod/data'. >>> ++++++ XrdAggregatingN2N initializing. Remote root is null >>> 131012 10:07:18 13496 oss_AioInit: started AIO read signal thread; >>> tid=63666496 >>> 131012 10:07:18 13496 oss_AioInit: started AIO write signal thread; >>> tid=64719168 >>> XrdAggregatingN2N processing. buff='/' >>> XrdAggregatingN2N processing. buff='/home/aliprod/data/' >>> XrdAggregatingN2N processing. buff='/' >>> XrdAggregatingN2N processing. buff='/home/aliprod/data/' >>> Config effective /home/aliprod/xrdserver/etc/xrootd/server/xrootd.cf oss >>> configuration: >>> oss.alloc 536870912 2 0 >>> oss.cachescan 600 >>> oss.fdlimit 32500 65000 >>> oss.maxsize 0 >>> oss.namelib >>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so >>> oss.localroot /storage01/xrdnamespace >>> oss.trace fff >>> oss.xfr 1 deny 10800 keep 1200 >>> oss.memfile off max 2071848960 >>> oss.space public /storage01/xrddata >>> oss.space public /storage02/xrddata >>> oss.space public /storage03/xrddata >>> oss.space public /storage01/xrddata >>> oss.space public /storage02/xrddata >>> oss.space public /storage03/xrddata >>> oss.defaults r/w nocheck nodread nomig norcreate nopurge >>> nostage xattr >>> oss.path / r/w nocheck nodread nomig norcreate nopurge nostage >>> xattr >>> ------ Storage system initialization completed. >>> >>> >>> ++++++ Configuring server role. . . >>> =====> all.manager rd.spacescience.ro 3122 >>> =====> all.adminpath /home/aliprod/xrdserver/admin >>> 131012 10:07:18 13496 Configure Global System Identification: anon-s >>> 3122rd.spacescience.ro >>> Config effective /home/aliprod/xrdserver/etc/xrootd/server/xrootd.cf ofs >>> configuration: >>> ofs.role server >>> ofs.authorize >>> ofs.authlib /home/aliprod/xrdserver/lib64/libXrdAliceTokenAcc.so >>> ofs.maxdelay 60 >>> ofs.persist manual hold 600 logdir >>> /home/aliprod/xrdserver/admin/.ofs/posc.log >>> ofs.trace 4 >>> ------ File system server initialization completed. >>> 131012 10:07:18 13496 none ftsofs_configure: Configured >>> Config warning: asynchronous I/O has been disabled! >>> Config warning: 'xrootd.prepare logdir' not specified; prepare tracking >>> disabled. >>> 131012 10:07:18 13523 cms_Open: Unable to connect socket to >>> /home/aliprod/xrdserver/admin/.olb/olbd.admin; connection refused >>> ------ xrootd protocol initialization completed. >>> ------ xrootd [log in to unmask]:1094 initialization >>> completed. >>> >>> >>> >>> >>> >>> >>> >>>> >>>> Andy >>>> >>>> On Fri, 11 Oct 2013, Adrian Sevcenco wrote: >>>> >>>>> On 10/11/2013 12:57 AM, Andrew Hanushevsky wrote: >>>>>> Hi Adrian, >>>>> Hi! >>>>> >>>>>> Looks to me that the double space problem comes from the fact that >>>>>> /storage01 has been included twice in the configuration. >>>>>> Additionally, >>>>> in configuration i have nothing twice .. >>>>> >>>>>> it would appear that /storage02 and /storage03 have not been >>>>>> included in >>>>>> the configuration. Could you send me your config file? I suspect that >>>>>> this is the problem here. >>>>> all config files attached .. xrootd.xrootd.cf is (re) created based on >>>>> system.cnf file >>>>> >>>>> Thanks for helping me with this! >>>>> Adrian >>>>> >>>>>> >>>>>> Andy >>>>>> >>>>>> -----Original Message----- From: Adrian Sevcenco >>>>>> Sent: Thursday, October 10, 2013 12:04 AM >>>>>> To: Andrew Hanushevsky >>>>>> Cc: [log in to unmask] >>>>>> Subject: Re: xrootd :: server 3.3.2 bug :: double size reporting >>>>>> >>>>>> On 10/10/2013 08:52 AM, Adrian Sevcenco wrote: >>>>>>> On 10/10/2013 03:26 AM, Andrew Hanushevsky wrote: >>>>>>>> Hi Adrian, >>>>>>> Hi! >>>>>>> >>>>>>>> Could you try to use xrdfs (the new client based xrd replacement) >>>>>>>> to see >>>>>>>> what you get there? >>>>>>> it reports double space also: >>>>>>> aliprod@storage03: ~ $ xrdfs localhost statvfs / >>>>>>> Path: / >>>>>>> Nodes with RW space: 1 >>>>>>> Size of RW space (MB): 7978 >>>>>> this is not so informative it seems .. it reports only ~8 Gb space! >>>>>> >>>>>> aliprod@storage03: ~ $ xrdfs localhost query space / >>>>>> oss.cgroup=public&oss.space=86636386320384&oss.free=11850958060&oss.maxf=2089872388&oss.used=10885940771&oss.quota=-1 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> aliprod@storage03: ~ $ xrdfs localhost query stats total >>>>>> <statistics tod="1381388316" ver="v3.3.2" >>>>>> src="storage03.spacescience.ro:1094" tos="1381231940" pgm="xrootd" >>>>>> ins="anon" pid="12987" site=""><stats >>>>>> id="info"><host>storage03.spacescience.ro</host><port>1094</port><name>anon</name></stats><stats >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> id="buff"><reqs>105006</reqs><mem>137090048</mem><buffs>180</buffs><adj>0</adj></stats><stats >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> id="link"><num>10</num><maxn>73</maxn><tot>28885</tot><in>18096047509</in><out>1510999236517</out><ctime>1995692</ctime><tmo>45893</tmo><stall>0</stall><sfps>0</sfps></stats><stats >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> id="poll"><att>10</att><en>49466</en><ev>45888</ev><int>0</int></stats><stats >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> id="proc"><usr><s>40</s><u>428853</u></usr><sys><s>2509</s><u>239537</u></sys></stats><stats >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> id="xrootd"><num>28884</num><ops><open>40772</open><rf>0</rf><rd>19695905</rd><pr>0</pr><rv>931313</rv><rs>10051282</rs><wr>4169</wr><sync>0</sync><getf>0</getf><putf>0</putf><misc>86049</misc></ops><aio><num>0</num><max>0</max><rej>0</rej></aio><err>201</err><rdr>0</rdr><dly>0</dly><lgn><num>28882</num><af>0</af><au>28880</au><ua>0</ua></lgn></stats><stats >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> id="ofs"><role>server</role><opr>3</opr><opw>0</opw><opp>0</opp><ups>0</ups><han>3</han><rdr>0</rdr><bxq>0</bxq><rep>0</rep><err>0</err><dly>0</dly><sok>0</sok><ser>0</ser><tpc><grnt>0</grnt><deny>0</deny><err>0</err><exp>0</exp></tpc></stats><stats >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> id="oss" v="2"><paths>2<stats >>>>>> id="0"><lp>"/"</lp><rp>"/storage01/xrdnamespace/home/aliprod/data/"</rp><tot>13959964628</tot><free>7213124</free><ino>1772814336</ino><ifr>1769450005</ifr></stats><stats >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> id="1"><lp>"/"</lp><rp>"/storage01/xrdnamespace/home/aliprod/data/"</rp><tot>13959964628</tot><free>7213124</free><ino>1772814336</ino><ifr>1769450005</ifr></stats></paths><space>2<stats >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> id="0"><name>public</name><tot>84605846016</tot><free>2010157</free><maxf>443841</maxf><fsn>6</fsn><usg>10693215</usg></stats></space></stats><stats >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> id="sched"><jobs>78587</jobs><inq>0</inq><maxinq>2</maxinq><threads>28</threads><idle>27</idle><tcr>28</tcr><tde>0</tde><tlimr>0</tlimr></stats><stats >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> id="sgen"><as>1</as><et>9</et><toe>1381388316</toe></stats></statistics> >>>>>> >>>>>> >>>>>> >>>>>> the problem is more clear in this output.. >>>>>> >>>>>> Thanks! >>>>>> Adrian >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Utilization of RW space (%): c >>>>>>> Nodes with staging space: 0 >>>>>>> Size of staging space (MB): 0 >>>>>>> Utilization of staging space (%): >>>>>>> >>>>>>> >>>>>>> Thanks! >>>>>>> Adrian >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Andy >>>>>>>> >>>>>>>> -----Original Message----- From: Adrian Sevcenco >>>>>>>> Sent: Tuesday, October 08, 2013 4:58 AM >>>>>>>> To: [log in to unmask] >>>>>>>> Subject: xrootd :: server 3.3.2 bug :: double size reporting >>>>>>>> >>>>>>>> Hi! I have a nagging problem with the reporting of size in xrootd: >>>>>>>> >>>>>>>> aliprod@storage03: ~ $ echo exit | ~/xrdserver/bin/xrd localhost >>>>>>>> queryspace / - >>>>>>>> Disk space approximations (MB): >>>>>>>> Total : 82622896 >>>>>>>> Free : 39192 >>>>>>>> Used : 0 >>>>>>>> Largest chunk : 7190 >>>>>>>> >>>>>>>> aliprod@storage03: ~ $ df -B M | grep storage >>>>>>>> /dev/sdc1 13632778M 13626433M 6346M 100% /storage01 >>>>>>>> /dev/sdc2 13632778M 13625588M 7191M 100% /storage02 >>>>>>>> /dev/sdc3 14045893M 14039832M 6061M 100% /storage03 >>>>>>>> >>>>>>>> aliprod@storage03: ~ $ df -BM | grep storage | awk 'BEGIN {total=0} >>>>>>>> {gsub("M",""); total+= $2;} END {print total}' >>>>>>>> 41311449 >>>>>>>> >>>>>>>> given that the problem is the queryspace result, i imagine that >>>>>>>> is a >>>>>>>> problem internal to xrd. ( it seems that the 3.2.6 version is >>>>>>>> ok.)(this >>>>>>>> are ALICE packaged versions) >>>>>>>> >>>>>>>> Any idea about the problem and how can i investigate more? >>>>>>>> Thanks a lot! >>>>>>>> Adrian >>>>>>>> >>>>>>>> >>>>>>>> ######################################################################## >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Use REPLY-ALL to reply to list >>>>>>>> >>>>>>>> To unsubscribe from the XROOTD-L list, click the following link: >>>>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >>>>>>> >>>>>>> >>>>>>> >>>>>>> ######################################################################## >>>>>>> >>>>>>> >>>>>>> >>>>>>> Use REPLY-ALL to reply to list >>>>>>> >>>>>>> To unsubscribe from the XROOTD-L list, click the following link: >>>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>> >>> >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >> >> >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >> > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1