Hi James,
Is there a reason why they can't mount it? Is it LAPP or CC?
I would recommend that you don't use the software as an input but you
download it explicitely from the job if you cannot find it in CVMFS.
And/or the tarball should be copied to the French site storage closest
to their nodes.
The tarball on our storage was being accessed by 1500 processes
concurrently on the same machine earlier today and I had already to
replicate 3 times the file to try to spread the load on others. I'm
surprised you didn't have time outs.
cheers
alessandra
On 07/06/2019 14:59, PERRY James wrote:
> Hi Alessandra,
>
> We are mostly using CVMFS, but one of the compute nodes in France
> doesn't mount our CVMFS repository so we need the tarball for that one.
> Unfortunately because I can't predict when I submit a job whether it
> will go to that node or not, all the jobs have the tarball listed as an
> input file. I tried uploading copies to other storage elements as well
> when I first put it on the grid, but at the time only Manchester was
> working for me. I'm happy to discuss other solutions to this if it's
> causing problems.
>
> Thanks,
> James
>
>
> On 07/06/2019 14:52, Alessandra Forti wrote:
>> Hi James,
>>
>> can you let me know how you do software distribution? It seems you have
>> 1 single tarball on the Manchester storage that is creating a large
>> amount of connections.
>>
>> They might be among the causes of the current load we are experiencing.
>> Manchester isn't running anything at the moment, so either those are ill
>> closed connections (could be) or the tar ball you have on the manchester
>> storage is the only source access by WNs at other sites in the UK.
>>
>> We always said that until the software was in development and LSST run
>> smaller scale the storage was fine, but it wouldn't work if too many
>> jobs tried to access the same file on one storage. Have you thought
>> about using cvmfs or at the very least replicate the tarball at other
>> sites?
>>
>> thanks
>>
>> cheers
>> alessandra
>>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> James Perry Room 2.41, Bayes Centre
> Software Architect The University of Edinburgh
> EPCC 47 Potterrow
> Tel: +44 131 650 5173 Edinburgh, EH8 9BT
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
--
Respect is a rational process. \\//
For Ur-Fascism, disagreement is treason. (U. Eco)
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the LSST-DESC-GRID list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=LSST-DESC-GRID&A=1
|
