Hi Adrian,
OK, looks like this is a plugin problem. The oss plugin is being
initialized twice. Probably by libXrdxFtsOfs.so. Who wrote that?
The cms part doesn't have this problem and I assume that thie figures it
displays:
131012 10:07:18 13529 Meter: Found 3 filesystem(s); 39TB total (100%
util); 18GB free (6GB max)
are correct. This is one for the person who developed that plugin.
Andy
On Wed, 16 Oct 2013, Adrian Sevcenco wrote:
> Hi! Any news on this? Sorry for thread bumping but i have an storage server
> waiting to be put online and spurious test error with "no space left on disk"
> and i don't know what to do...
>
> Can i help the debugging process somehow?
>
> Thank you!
> Adrian
>
>
> On 10/12/2013 10:20 AM, Adrian Sevcenco wrote:
>> On 10/12/2013 12:44 AM, Andrew Hanushevsky wrote:
>>> Hi Adrian,
>> Hi!
>>
>>> Very odd. You must have a log somewhere that includes xrootd
>>> initialization. I'd like to see that to find out why it thinks the
>>> config is not what appears to be in the config file. Could you send me
>>> the snippet that just includes the lines from start-up to
>>> "initialization completed".
>> below the outputs with cmslog and xrdlog (preview : cms see corect size
>> in xrd:: storage system initialization i have double inclusion of the
>> storage filesystems)
>>
>> Thanks!!!
>> Adrian
>>
>> in cmslog :
>> Copr. 2004-2012 Stanford University, xrd version v3.3.2
>> ++++++ cmsd [log in to unmask] initialization started.
>> Config using configuration file
>> /home/aliprod/xrdserver/etc/xrootd/server/xrootd.cf
>> =====> xrd.protocol xrootd *
>> =====> all.adminpath /home/aliprod/xrdserver/admin
>> Config warning: ignoring unknown xrd directive 'pidpath'.
>> =====> xrd.pidpath
>> =====> xrd.port 1094
>> =====> xrd.sched mint 32 maxt 2048 avlt 512 idle 780
>> =====> xrd.network buffsz 0 nodnr
>> Config maximum number of connections restricted to 65000
>> Copr. 2007 Stanford University/SLAC cmsd.
>> ++++++ [log in to unmask] phase 1 initialization started.
>> =====> all.export / nolock r/w nocheck norcreate
>> =====> all.role server
>> =====> all.manager rd.spacescience.ro 3122
>> =====> all.adminpath /home/aliprod/xrdserver/admin
>> =====> cms.pidpath /home/aliprod/xrdserver/admin
>> =====> cms.sched cpu 10 io 10 space 80
>> =====> oss.defaults nomig nodread nocheck norcreate nolock
>> =====> oss.namelib
>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so
>> /home/aliprod/data
>> =====> oss.localroot /storage01/xrdnamespace
>> =====> cms.space min 1g 1g
>> The following paths are available to the redirector:
>> w /
>>
>> ------ [log in to unmask] phase 1 server initialization
>> completed.
>> ++++++ [log in to unmask] phase 2 server initialization
>> started.
>> 131012 10:07:18 13529 Configure2 Global System Identification: anon-s
>> 3122rd.spacescience.ro
>> Plugin loaded unversioned XrdOucgetName2Name from namelib
>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so
>> ++++++ XrdAggregatingN2N initializing. Local lfn prefix
>> '/home/aliprod/data'.
>> ++++++ XrdAggregatingN2N initializing. Remote root is null
>> ++++++ Storage system initialization started.
>> =====> all.export / nolock r/w nocheck norcreate
>> =====> oss.defaults nomig nodread nocheck norcreate nolock
>> =====> oss.alloc 512M 2 0
>> =====> oss.fdlimit * max
>> =====> oss.space public /storage01/xrddata
>> =====> oss.space public /storage02/xrddata
>> =====> oss.space public /storage03/xrddata
>> =====> oss.namelib
>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so
>> /home/aliprod/data
>> =====> oss.localroot /storage01/xrdnamespace
>> Plugin loaded unversioned XrdOucgetName2Name from namelib
>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so
>> ++++++ XrdAggregatingN2N initializing. Local lfn prefix
>> '/home/aliprod/data'.
>> ++++++ XrdAggregatingN2N initializing. Remote root is null
>> ++++++ Configuring standalone mode . . .
>> 131012 10:07:18 13529 oss_AioInit: started AIO read signal thread;
>> tid=3135912256
>> 131012 10:07:18 13529 oss_AioInit: started AIO write signal thread;
>> tid=3136964928
>> XrdAggregatingN2N processing. buff='/'
>> XrdAggregatingN2N processing. buff='/home/aliprod/data/'
>> XrdAggregatingN2N processing. buff='/'
>> XrdAggregatingN2N processing. buff='/home/aliprod/data/'
>> Config effective /home/aliprod/xrdserver/etc/xrootd/server/xrootd.cf oss
>> configuration:
>> oss.alloc 536870912 2 0
>> oss.cachescan 600
>> oss.fdlimit 32500 65000
>> oss.maxsize 0
>> oss.namelib
>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so
>> oss.localroot /storage01/xrdnamespace
>> oss.trace fff
>> oss.xfr 1 deny 10800 keep 1200
>> oss.memfile off max 2071848960
>> oss.space public /storage01/xrddata
>> oss.space public /storage02/xrddata
>> oss.space public /storage03/xrddata
>> oss.defaults r/w nocheck nodread nomig norcreate nopurge
>> nostage xattr
>> oss.path / r/w nocheck nodread nomig norcreate nopurge nostage
>> xattr
>> ------ Storage system initialization completed.
>> 131012 10:07:18 13529 Start Srv=0 dfs=0 lcl=0 Pre=1 dmLife=0 0
>> 131012 10:07:18 13529 Start Lim=0 0 fix=0 Qmax=1
>> 131012 10:07:18 13529 calcSpace New fs info; maxfree=6473MB utilized=100%
>> 131012 10:07:18 13529 Meter: Found 3 filesystem(s); 39TB total (100%
>> util); 18GB free (6GB max)
>> ------ [log in to unmask] phase 2 server initialization
>> completed.
>>
>>
>> ##################################################################
>>
>> xrdlog:
>>
>>
>> 131012 10:07:18 13496 Scalla is starting. . .
>> Copr. 2004-2012 Stanford University, xrd version v3.3.2
>>
>> ++++++ xrootd [log in to unmask] initialization started.
>> Config using configuration file
>> /home/aliprod/xrdserver/etc/xrootd/server/xrootd.cf
>> =====> xrd.protocol xrootd *
>> =====> all.adminpath /home/aliprod/xrdserver/admin
>> Config warning: ignoring unknown xrd directive 'pidpath'.
>> =====> xrd.pidpath
>> =====> xrd.port 1094
>> =====> xrd.sched mint 32 maxt 2048 avlt 512 idle 780
>> =====> xrd.network buffsz 0 nodnr
>> Config maximum number of connections restricted to 65000
>> Copr. 2012 Stanford University, xrootd protocol 2.9.7 version v3.3.2
>> ++++++ xrootd protocol initialization started.
>> 131012 10:07:18 13496 XrootdConfig: non-absolute export path - server
>> =====> all.export / nolock
>> =====> xrootd.async off
>> =====> xrootd.fslib /home/aliprod/xrdserver/lib64/libXrdxFtsOfs.so
>> =====> xrootd.seclib /home/aliprod/xrdserver/lib64/libXrdSec.so
>> Config exporting /
>> Plugin loaded
>>
>>
>> ++++++ Authentication system initialization started.
>> sec_PM: Loading unix protocol object from
>> /home/aliprod/xrdserver/lib64/libXrdSecunix.so
>> Plugin loaded
>> =====> sec.protocol /home/aliprod/xrdserver/lib64 unix
>> Config 1 authentication directives processed in
>> /home/aliprod/xrdserver/etc/xrootd/server/xrootd.cf
>> 131012 10:07:18 13496 sec_ProtBind_Complete: Default sectoken built:
>> '&P=unix'
>> ------ Authentication system initialization completed.
>> Plugin loaded unversioned XrdSfsGetFileSystem from fslib
>> /home/aliprod/xrdserver/lib64/libXrdxFtsOfs.so
>> ++++++ (c) 2012 CERN/IT-DSS v 2.0
>>
>>
>> ++++++ Storage system initialization started.
>> =====> all.export / nolock r/w nocheck norcreate
>> =====> oss.defaults nomig nodread nocheck norcreate nolock
>> =====> oss.alloc 512M 2 0
>> =====> oss.fdlimit * max
>> =====> oss.space public /storage01/xrddata
>> =====> oss.space public /storage02/xrddata
>> =====> oss.space public /storage03/xrddata
>> =====> oss.namelib
>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so
>> /home/aliprod/data
>> =====> oss.localroot /storage01/xrdnamespace
>> Plugin loaded unversioned XrdOucgetName2Name from namelib
>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so
>> ++++++ XrdAggregatingN2N initializing. Local lfn prefix
>> '/home/aliprod/data'.
>> ++++++ XrdAggregatingN2N initializing. Remote root is null
>> 131012 10:07:18 13496 oss_AioInit: started AIO read signal thread;
>> tid=52988224
>> 131012 10:07:18 13496 oss_AioInit: started AIO write signal thread;
>> tid=54040896
>> XrdAggregatingN2N processing. buff='/'
>> XrdAggregatingN2N processing. buff='/home/aliprod/data/'
>> XrdAggregatingN2N processing. buff='/'
>> XrdAggregatingN2N processing. buff='/home/aliprod/data/'
>> Config effective /home/aliprod/xrdserver/etc/xrootd/server/xrootd.cf oss
>> configuration:
>> oss.alloc 536870912 2 0
>> oss.cachescan 600
>> oss.fdlimit 32500 65000
>> oss.maxsize 0
>> oss.namelib
>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so
>> oss.localroot /storage01/xrdnamespace
>> oss.trace fff
>> oss.xfr 1 deny 10800 keep 1200
>> oss.memfile off max 2071848960
>> oss.space public /storage01/xrddata
>> oss.space public /storage02/xrddata
>> oss.space public /storage03/xrddata
>> oss.defaults r/w nocheck nodread nomig norcreate nopurge
>> nostage xattr
>> oss.path / r/w nocheck nodread nomig norcreate nopurge nostage
>> xattr
>> ------ Storage system initialization completed.
>>
>>
>> =====> ftsofs.thirdparty: yes
>> =====> ftsofs.thirdparty.slots: 20
>> =====> ftsofs.thirdparty.rate: 50 Mb/s
>> =====> ftsofs.thirdparty.statedirectory: /home/aliprod/xrdserver/admin
>> ++++++ File system initialization started.
>> =====> all.role server
>> =====> ofs.trace open
>> =====> ofs.authlib /home/aliprod/xrdserver/lib64/libXrdAliceTokenAcc.so
>> =====> ofs.authorize
>> Plugin loaded unversioned XrdAccAuthorizeObject from authlib
>> /home/aliprod/xrdserver/lib64/libXrdAliceTokenAcc.so
>> ++++++ (c) 2008 CERN/IT-DM-SMD AliceTokenAcc (Alice Token Access
>> Authorization) v 1.0
>> =====> XrdAliceTokenAcc: No Authorizationfile set via environment
>> variable 'TTOKENAUTHZ_AUTHORIZATIONFILE'
>> =====> XrdAliceTokenAcc: No Authorizationfile like
>> '/etc/grid-security/xrootd/TkAuthz.Authorization' found
>> =====> XrdAliceTokenAcc: No Authorizationfile like
>> '/home/aliprod/.globus/xrootd/TkAuthz.Authorization' found
>> =====> XrdAliceTokenAcc: Using Authorizationfile
>> '/home/aliprod/.authz/xrootd/TkAuthz.Authorization'!
>> ------ AliceTokenAcc initialization completed
>>
>>
>> ++++++ Storage system initialization started.
>> =====> all.export / nolock r/w nocheck norcreate
>> =====> oss.defaults nomig nodread nocheck norcreate nolock
>> =====> oss.alloc 512M 2 0
>> =====> oss.fdlimit * max
>> =====> oss.space public /storage01/xrddata
>> =====> oss.space public /storage02/xrddata
>> =====> oss.space public /storage03/xrddata
>> =====> oss.namelib
>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so
>> /home/aliprod/data
>> =====> oss.localroot /storage01/xrdnamespace
>> Plugin loaded unversioned XrdOucgetName2Name from namelib
>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so
>>
>> ++++++ XrdAggregatingN2N initializing. Local lfn prefix
>> '/home/aliprod/data'.
>> ++++++ XrdAggregatingN2N initializing. Remote root is null
>> 131012 10:07:18 13496 oss_AioInit: started AIO read signal thread;
>> tid=63666496
>> 131012 10:07:18 13496 oss_AioInit: started AIO write signal thread;
>> tid=64719168
>> XrdAggregatingN2N processing. buff='/'
>> XrdAggregatingN2N processing. buff='/home/aliprod/data/'
>> XrdAggregatingN2N processing. buff='/'
>> XrdAggregatingN2N processing. buff='/home/aliprod/data/'
>> Config effective /home/aliprod/xrdserver/etc/xrootd/server/xrootd.cf oss
>> configuration:
>> oss.alloc 536870912 2 0
>> oss.cachescan 600
>> oss.fdlimit 32500 65000
>> oss.maxsize 0
>> oss.namelib
>> /home/aliprod/xrdserver/lib64/libXrdAggregatingName2Name.so
>> oss.localroot /storage01/xrdnamespace
>> oss.trace fff
>> oss.xfr 1 deny 10800 keep 1200
>> oss.memfile off max 2071848960
>> oss.space public /storage01/xrddata
>> oss.space public /storage02/xrddata
>> oss.space public /storage03/xrddata
>> oss.space public /storage01/xrddata
>> oss.space public /storage02/xrddata
>> oss.space public /storage03/xrddata
>> oss.defaults r/w nocheck nodread nomig norcreate nopurge
>> nostage xattr
>> oss.path / r/w nocheck nodread nomig norcreate nopurge nostage
>> xattr
>> ------ Storage system initialization completed.
>>
>>
>> ++++++ Configuring server role. . .
>> =====> all.manager rd.spacescience.ro 3122
>> =====> all.adminpath /home/aliprod/xrdserver/admin
>> 131012 10:07:18 13496 Configure Global System Identification: anon-s
>> 3122rd.spacescience.ro
>> Config effective /home/aliprod/xrdserver/etc/xrootd/server/xrootd.cf ofs
>> configuration:
>> ofs.role server
>> ofs.authorize
>> ofs.authlib /home/aliprod/xrdserver/lib64/libXrdAliceTokenAcc.so
>> ofs.maxdelay 60
>> ofs.persist manual hold 600 logdir
>> /home/aliprod/xrdserver/admin/.ofs/posc.log
>> ofs.trace 4
>> ------ File system server initialization completed.
>> 131012 10:07:18 13496 none ftsofs_configure: Configured
>> Config warning: asynchronous I/O has been disabled!
>> Config warning: 'xrootd.prepare logdir' not specified; prepare tracking
>> disabled.
>> 131012 10:07:18 13523 cms_Open: Unable to connect socket to
>> /home/aliprod/xrdserver/admin/.olb/olbd.admin; connection refused
>> ------ xrootd protocol initialization completed.
>> ------ xrootd [log in to unmask]:1094 initialization completed.
>>
>>
>>
>>
>>
>>
>>
>>>
>>> Andy
>>>
>>> On Fri, 11 Oct 2013, Adrian Sevcenco wrote:
>>>
>>>> On 10/11/2013 12:57 AM, Andrew Hanushevsky wrote:
>>>>> Hi Adrian,
>>>> Hi!
>>>>
>>>>> Looks to me that the double space problem comes from the fact that
>>>>> /storage01 has been included twice in the configuration. Additionally,
>>>> in configuration i have nothing twice ..
>>>>
>>>>> it would appear that /storage02 and /storage03 have not been
>>>>> included in
>>>>> the configuration. Could you send me your config file? I suspect that
>>>>> this is the problem here.
>>>> all config files attached .. xrootd.xrootd.cf is (re) created based on
>>>> system.cnf file
>>>>
>>>> Thanks for helping me with this!
>>>> Adrian
>>>>
>>>>>
>>>>> Andy
>>>>>
>>>>> -----Original Message----- From: Adrian Sevcenco
>>>>> Sent: Thursday, October 10, 2013 12:04 AM
>>>>> To: Andrew Hanushevsky
>>>>> Cc: [log in to unmask]
>>>>> Subject: Re: xrootd :: server 3.3.2 bug :: double size reporting
>>>>>
>>>>> On 10/10/2013 08:52 AM, Adrian Sevcenco wrote:
>>>>>> On 10/10/2013 03:26 AM, Andrew Hanushevsky wrote:
>>>>>>> Hi Adrian,
>>>>>> Hi!
>>>>>>
>>>>>>> Could you try to use xrdfs (the new client based xrd replacement)
>>>>>>> to see
>>>>>>> what you get there?
>>>>>> it reports double space also:
>>>>>> aliprod@storage03: ~ $ xrdfs localhost statvfs /
>>>>>> Path: /
>>>>>> Nodes with RW space: 1
>>>>>> Size of RW space (MB): 7978
>>>>> this is not so informative it seems .. it reports only ~8 Gb space!
>>>>>
>>>>> aliprod@storage03: ~ $ xrdfs localhost query space /
>>>>> oss.cgroup=public&oss.space=86636386320384&oss.free=11850958060&oss.maxf=2089872388&oss.used=10885940771&oss.quota=-1
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> aliprod@storage03: ~ $ xrdfs localhost query stats total
>>>>> <statistics tod="1381388316" ver="v3.3.2"
>>>>> src="storage03.spacescience.ro:1094" tos="1381231940" pgm="xrootd"
>>>>> ins="anon" pid="12987" site=""><stats
>>>>> id="info"><host>storage03.spacescience.ro</host><port>1094</port><name>anon</name></stats><stats
>>>>>
>>>>>
>>>>>
>>>>> id="buff"><reqs>105006</reqs><mem>137090048</mem><buffs>180</buffs><adj>0</adj></stats><stats
>>>>>
>>>>>
>>>>>
>>>>> id="link"><num>10</num><maxn>73</maxn><tot>28885</tot><in>18096047509</in><out>1510999236517</out><ctime>1995692</ctime><tmo>45893</tmo><stall>0</stall><sfps>0</sfps></stats><stats
>>>>>
>>>>>
>>>>>
>>>>> id="poll"><att>10</att><en>49466</en><ev>45888</ev><int>0</int></stats><stats
>>>>>
>>>>>
>>>>>
>>>>> id="proc"><usr><s>40</s><u>428853</u></usr><sys><s>2509</s><u>239537</u></sys></stats><stats
>>>>>
>>>>>
>>>>>
>>>>> id="xrootd"><num>28884</num><ops><open>40772</open><rf>0</rf><rd>19695905</rd><pr>0</pr><rv>931313</rv><rs>10051282</rs><wr>4169</wr><sync>0</sync><getf>0</getf><putf>0</putf><misc>86049</misc></ops><aio><num>0</num><max>0</max><rej>0</rej></aio><err>201</err><rdr>0</rdr><dly>0</dly><lgn><num>28882</num><af>0</af><au>28880</au><ua>0</ua></lgn></stats><stats
>>>>>
>>>>>
>>>>>
>>>>> id="ofs"><role>server</role><opr>3</opr><opw>0</opw><opp>0</opp><ups>0</ups><han>3</han><rdr>0</rdr><bxq>0</bxq><rep>0</rep><err>0</err><dly>0</dly><sok>0</sok><ser>0</ser><tpc><grnt>0</grnt><deny>0</deny><err>0</err><exp>0</exp></tpc></stats><stats
>>>>>
>>>>>
>>>>>
>>>>> id="oss" v="2"><paths>2<stats
>>>>> id="0"><lp>"/"</lp><rp>"/storage01/xrdnamespace/home/aliprod/data/"</rp><tot>13959964628</tot><free>7213124</free><ino>1772814336</ino><ifr>1769450005</ifr></stats><stats
>>>>>
>>>>>
>>>>>
>>>>> id="1"><lp>"/"</lp><rp>"/storage01/xrdnamespace/home/aliprod/data/"</rp><tot>13959964628</tot><free>7213124</free><ino>1772814336</ino><ifr>1769450005</ifr></stats></paths><space>2<stats
>>>>>
>>>>>
>>>>>
>>>>> id="0"><name>public</name><tot>84605846016</tot><free>2010157</free><maxf>443841</maxf><fsn>6</fsn><usg>10693215</usg></stats></space></stats><stats
>>>>>
>>>>>
>>>>>
>>>>> id="sched"><jobs>78587</jobs><inq>0</inq><maxinq>2</maxinq><threads>28</threads><idle>27</idle><tcr>28</tcr><tde>0</tde><tlimr>0</tlimr></stats><stats
>>>>>
>>>>>
>>>>>
>>>>> id="sgen"><as>1</as><et>9</et><toe>1381388316</toe></stats></statistics>
>>>>>
>>>>>
>>>>> the problem is more clear in this output..
>>>>>
>>>>> Thanks!
>>>>> Adrian
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Utilization of RW space (%): c
>>>>>> Nodes with staging space: 0
>>>>>> Size of staging space (MB): 0
>>>>>> Utilization of staging space (%):
>>>>>>
>>>>>>
>>>>>> Thanks!
>>>>>> Adrian
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Andy
>>>>>>>
>>>>>>> -----Original Message----- From: Adrian Sevcenco
>>>>>>> Sent: Tuesday, October 08, 2013 4:58 AM
>>>>>>> To: [log in to unmask]
>>>>>>> Subject: xrootd :: server 3.3.2 bug :: double size reporting
>>>>>>>
>>>>>>> Hi! I have a nagging problem with the reporting of size in xrootd:
>>>>>>>
>>>>>>> aliprod@storage03: ~ $ echo exit | ~/xrdserver/bin/xrd localhost
>>>>>>> queryspace / -
>>>>>>> Disk space approximations (MB):
>>>>>>> Total : 82622896
>>>>>>> Free : 39192
>>>>>>> Used : 0
>>>>>>> Largest chunk : 7190
>>>>>>>
>>>>>>> aliprod@storage03: ~ $ df -B M | grep storage
>>>>>>> /dev/sdc1 13632778M 13626433M 6346M 100% /storage01
>>>>>>> /dev/sdc2 13632778M 13625588M 7191M 100% /storage02
>>>>>>> /dev/sdc3 14045893M 14039832M 6061M 100% /storage03
>>>>>>>
>>>>>>> aliprod@storage03: ~ $ df -BM | grep storage | awk 'BEGIN {total=0}
>>>>>>> {gsub("M",""); total+= $2;} END {print total}'
>>>>>>> 41311449
>>>>>>>
>>>>>>> given that the problem is the queryspace result, i imagine that is a
>>>>>>> problem internal to xrd. ( it seems that the 3.2.6 version is
>>>>>>> ok.)(this
>>>>>>> are ALICE packaged versions)
>>>>>>>
>>>>>>> Any idea about the problem and how can i investigate more?
>>>>>>> Thanks a lot!
>>>>>>> Adrian
>>>>>>>
>>>>>>>
>>>>>>> ########################################################################
>>>>>>>
>>>>>>>
>>>>>>> Use REPLY-ALL to reply to list
>>>>>>>
>>>>>>> To unsubscribe from the XROOTD-L list, click the following link:
>>>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>>>>>
>>>>>>
>>>>>>
>>>>>> ########################################################################
>>>>>>
>>>>>>
>>>>>> Use REPLY-ALL to reply to list
>>>>>>
>>>>>> To unsubscribe from the XROOTD-L list, click the following link:
>>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>
>>
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>
>
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
