Hi Dominique,
It's the ImSim code. I think someone recommended not to clone it from
GitHub as this can result in GitHub blacklisting the worker nodes if
they do this too many times, so I went with the tarball instead. We
could try it using GitHub directly if you think it would be safe.
Cheers,
James
On 07/06/2019 15:49, Dominique Boutigny wrote:
> Hi Alessandra and James,
>
> I add Fabio in the loop.
> I don't think that there is any problem to copy the tar ball at CC-IN2P3.
> By the way what is this tar ball ? Is that the instance catalog or the
> imsim code ? If it is imsim, I thought that we decided to download it
> from github and to build it locally as it is very fast to do so.
>
> Cheers,
>
> Dominique
>
> On 07/06/2019 16:30, Alessandra Forti wrote:
>> We upgraded the system and changed the storage system configuration so
>> there might be other factors at play, but this was the first thing
>> that jumped out and until we reduce it we cannot know if other things
>> are affecting the responsiveness of the storage.
>>
>> Said that 1500 processes trying to access 1 file on 1 machine is not
>> healthy.
>>
>> cheers
>> alessandra
>>
>> On 07/06/2019 15:25, PERRY James wrote:
>>> Hi Alessandra,
>>>
>>> The site is CC. They didn't seem to want to mount the cvmfs repository
>>> but maybe we could convince them to.
>>>
>>> I can download the file explicitly instead when required. Sorry, I
>>> hadn't realised that this would put such a load on the system.
>>>
>>> Thanks,
>>> James
>>>
>>>
>>> On 07/06/2019 15:16, Alessandra Forti wrote:
>>>> Hi James,
>>>>
>>>> Is there a reason why they can't mount it? Is it LAPP or CC?
>>>>
>>>> I would recommend that you don't use the software as an input but you
>>>> download it explicitely from the job if you cannot find it in CVMFS.
>>>> And/or the tarball should be copied to the French site storage closest
>>>> to their nodes.
>>>>
>>>> The tarball on our storage was being accessed by 1500 processes
>>>> concurrently on the same machine earlier today and I had already to
>>>> replicate 3 times the file to try to spread the load on others. I'm
>>>> surprised you didn't have time outs.
>>>>
>>>> cheers
>>>> alessandra
>>>>
>>>> On 07/06/2019 14:59, PERRY James wrote:
>>>>> Hi Alessandra,
>>>>>
>>>>> We are mostly using CVMFS, but one of the compute nodes in France
>>>>> doesn't mount our CVMFS repository so we need the tarball for that
>>>>> one.
>>>>> Unfortunately because I can't predict when I submit a job whether it
>>>>> will go to that node or not, all the jobs have the tarball listed
>>>>> as an
>>>>> input file. I tried uploading copies to other storage elements as well
>>>>> when I first put it on the grid, but at the time only Manchester was
>>>>> working for me. I'm happy to discuss other solutions to this if it's
>>>>> causing problems.
>>>>>
>>>>> Thanks,
>>>>> James
>>>>>
>>>>>
>>>>> On 07/06/2019 14:52, Alessandra Forti wrote:
>>>>>> Hi James,
>>>>>>
>>>>>> can you let me know how you do software distribution? It seems you
>>>>>> have
>>>>>> 1 single tarball on the Manchester storage that is creating a large
>>>>>> amount of connections.
>>>>>>
>>>>>> They might be among the causes of the current load we are
>>>>>> experiencing.
>>>>>> Manchester isn't running anything at the moment, so either those
>>>>>> are ill
>>>>>> closed connections (could be) or the tar ball you have on the
>>>>>> manchester
>>>>>> storage is the only source access by WNs at other sites in the UK.
>>>>>>
>>>>>> We always said that until the software was in development and LSST
>>>>>> run
>>>>>> smaller scale the storage was fine, but it wouldn't work if too many
>>>>>> jobs tried to access the same file on one storage. Have you thought
>>>>>> about using cvmfs or at the very least replicate the tarball at other
>>>>>> sites?
>>>>>>
>>>>>> thanks
>>>>>>
>>>>>> cheers
>>>>>> alessandra
>>>>>>
>>>>> --
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>> James Perry Room 2.41, Bayes Centre
>>>>> Software Architect The University of Edinburgh
>>>>> EPCC 47 Potterrow
>>>>> Tel: +44 131 650 5173 Edinburgh, EH8 9BT
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>> The University of Edinburgh is a charitable body, registered in
>>>>> Scotland, with registration number SC005336.
>>
>
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
James Perry Room 2.41, Bayes Centre
Software Architect The University of Edinburgh
EPCC 47 Potterrow
Tel: +44 131 650 5173 Edinburgh, EH8 9BT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the LSST-DESC-GRID list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=LSST-DESC-GRID&A=1
|
