 |
 |
On 2020-07-08 14:19, Andrew Hanushevsky wrote: > What release? Git head? This was from Nikolai's image, 5-rc4. \m > On Wed, 8 Jul 2020, Matevz Tadel wrote: > >> Hi Andy, >> >> On 2020-07-08 13:49, Andrew Hanushevsky wrote: >>> Hi Matevz, >>> >>> Well, what kind of authentication? Clearly, the kind we use doesn't cause this >>> problem. It could be just a random core smash but if it's random we should be >>> various effects not just a crash in this particular code path, right? >> >> xcache without any security config, everything works smooth. >> >> xcache with sec.protocol /usr/lib64 gsi --- trouble: >> >> 200708 13:38:44 240995 XrootdXeq: matevz.241046:31@uaf-7 pub IPv4 login as >> d0ba0e6c.0 >> 200708 13:38:44 240995 Posix_P2L: file >> /eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root >> pfn2lfn /eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root >> [2020-07-08 13:38:44.739012 -0700][Error ][AsyncSock ] >> [[log in to unmask]:1094.0] Unable to connect: network is unreachable >> [2020-07-08 13:38:44.739092 -0700][Error ][PostMaster ] >> [[log in to unmask]:1094] elapsed = 0, pConnectionWindow = 120 seconds. >> [2020-07-08 13:38:45.637583 -0700][Error ][XRootDTransport ] >> [[log in to unmask]:1094.0] Authentication with gsi failed: >> [2020-07-08 13:38:45.974332 -0700][Error ][AsyncSock ] >> [[log in to unmask]:1095.0] Unable to connect: network is unreachable >> [2020-07-08 13:38:45.974400 -0700][Error ][PostMaster ] >> [[log in to unmask]:1095] elapsed = 0, pConnectionWindow = 120 seconds. >> 200708 13:38:46 240995 XrdPfc_Manager: info Cache::Attach() >> root:[log in to unmask] >> >> 200708 13:38:46 240995 XrdPfc_Manager: debug Cache::GetFile >> eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root, io >> 0xe30f50 >> 200708 13:38:46 240995 XrdPfc_IO: debug IOEntireFile::initCachedStat get stat >> from client res = 0, size = 2272072 >> root:[log in to unmask] >> >> 200708 13:38:46 240995 XrdPfc_File: debug Creating new file info, data size = >> 2272072 num blocks = 3 >> eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root >> 200708 13:38:46 240995 XrdPfc_Manager: debug Cache::inc_ref_cnt >> eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root, cnt at >> exit = 1 >> 200708 13:38:46 240995 XrdPfc_File: debug File::AddIO() io = 0xe30f50 >> eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root >> 200708 13:38:46 240995 XrdPfc_Manager: debug Cache::Attach() >> root:[log in to unmask] >> location: [log in to unmask]:1095 >> [2020-07-08 13:38:47.022428 -0700][Error ][AsyncSock ] >> [[log in to unmask]:1095.0] Socket error encountered: [ERROR] Invalid >> arguments >> [2020-07-08 13:38:47.022506 -0700][Error ][XRootD ] >> [[log in to unmask]:1095] Unable to get the response to request >> kXR_read (handle: 0x00000000, offset: 0, size: 1048576) >> [2020-07-08 13:38:47.022625 -0700][Error ][File ] >> [0xf0b040@root:[log in to unmask]:1094//eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root?xrdcl.requuid=6730e10b-8b40-43bf-9d0a-75da982939e8] >> Fatal file state error. Message kXR_read (handle: 0x00000000, offset: 0, size: >> 1048576) returned with [ERROR] Invalid arguments >> 200708 13:38:47 241052 XrdPfc_File: error File::ProcessBlockResponse block >> 0xff3440 0 error=-22 >> eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root >> 200708 13:38:47 240995 XrdPfc_File: error File::Read() io 0xe30f50, block 0 >> finished with error 22 invalid argument >> eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root >> src/tcmalloc.cc:284] Attempt to free invalid pointer 0x313262003543620a >> >> Note that while we get all these network errors at the start, cache still got >> the stat info from the server (knows the size of the file). >> >> I must admit I never test xcache with auth on :( I'll try it out now, well, >> after lunch :) >> >> Matevz >> >>> Andy >>> >>> >>> On Wed, 8 Jul 2020, Matevz Tadel wrote: >>> >>>> Yay, that was a journey ... but I can reproduce it now! >>>> >>>> It is super strange this happens with xcache with authentication on only ... >>>> this really should have no effect. I first tried without it and it worked and >>>> then something rang a bell that you said so in the email :). >>>> >>>> Andy, does this ring any bells for you? It looks like interaction between >>>> server / client usage of X509 stuffe. >>>> >>>> Anyway, I'm digging on on the xcache side ... >>>> >>>> Cheers, >>>> Matevz >>>> >>>> >>>> >>>> On 2020-07-08 07:41, Nikolai Hartmann wrote: >>>>> Hi Matevz, >>>>> >>>>> I might have something like a "minimal failing example". Unfortunately >>>>> the problem only appears when authentication is required, so the example >>>>> will only work on a machine that has a valid host certificate and the >>>>> corresponding directory has to be bind-mounted into the container. >>>>> >>>>> I uploaded my container image here: >>>>> >>>>> https://urldefense.com/v3/__https://cloud.physik.lmu.de/index.php/s/RFC6Q89FBxxNMXF__;!!Mih3wA!S4S4O0y7f1Z5oNAgkr2EZ2J5683bZ5LRbG55GbcoHhyJTwOzaS2lABcIifddJxDGMy-N$ >>>>> >>>>> >>>>> >>>>> and made a directory structure (tar archive attached) to bind mount into >>>>> the container (and containing the minimal failing xcache config and a >>>>> script for starting gdb inside the container) >>>>> >>>>> To reproduce, extract the archive, enter the directory and run (as >>>>> non-root user) >>>>> >>>>> singularity run -B $(pwd)/data:/data -B $(pwd)/config:/etc/xrootd:ro -B >>>>> <hostkey-dir>:/etc/grid-security:ro <singularity-image> >>>>> >>>>> where <hostkey-dir> is a directory that contains >>>>> >>>>> hostkey.pem >>>>> hostcert.pem >>>>> vomsdir (will become X509_VOMS_DIR) >>>>> certificates (will become X509_CERT_DIR) >>>>> >>>>> and <singularity-image> is the path to the singularity image. >>>>> >>>>> That should run xrootd and the log should appear in >>>>> data/xrd/var/log/xrootd.log >>>>> >>>>> I used this example to produce the failure: >>>>> >>>>> xrdcp -f >>>>> root://lcg-lrz-xcache0.grid.lrz.de:1094//root://eospublic.cern.ch//eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root >>>>> >>>>> >>>>> /dev/null >>>>> >>>>> The simplest way to run gdb seemed to directly start xrootd with gdb. >>>>> This can be done with the script run_xcache_debug.sh in the attached >>>>> archive. Instead of the command above just use >>>>> >>>>> singularity exec -B $(pwd)/data:/data -B $(pwd)/config:/etc/xrootd:ro -B >>>>> <hostkey-dir>:/etc/grid-security:ro <singularity-image> >>>>> ./run_xcache_debug.sh >>>>> >>>>> Note: Before restarting, best delete the content of the data directory >>>>> since the bug also did not seem to occur when the file was already >>>>> cached (e.g after testing without authentication) >>>>> >>>>> Sorry for the overly complicated reproducing steps, but since it only >>>>> happened when i authentication was enabled i didn't know how to do it >>>>> simpler. I hope it helps. >>>>> >>>>> Thanks, >>>>> Nikolai >>>>> >>>>> On 7/7/20 8:42 PM, Matevz Tadel wrote: >>>>>> Thanks Nikolai, I shall continue my investigation :) >>>>>> >>>>>> Matevz >>>>>> >>>>>> On 2020-07-06 23:59, Nikolai Hartmann wrote: >>>>>>> Hi Matevz, >>>>>>> >>>>>>> Thanks a lot for looking into this. >>>>>>> >>>>>>> - The crash seems to happen always when i make a request >>>>>>> - Currently prefetching is disabled >>>>>>> - Yes, i think it is direct proxy mode >>>>>>> - stack trace is attached >>>>>>> >>>>>>> A similar setup seems to work for Ilija without issues with the xcaches >>>>>>> using slate - i tried to mimic that setup closely. Running xrootd from >>>>>>> this container image: >>>>>>> >>>>>>> https://urldefense.com/v3/__https://gitlab.physik.uni-muenchen.de/Nikolai.Hartmann/xcache-singularity-lrz/-/blob/51d2da52829eb6d8ea377539884f337208141aca/xcache.singularity.def__;!!Mih3wA!SJibOzmy2P3rdD8Ut7m7gYp_bah2pQX2dR2V9U6xiTq9PoQtfjb_MHHDljpOV0aWvVYj$ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> using this config >>>>>>> >>>>>>> https://urldefense.com/v3/__https://gitlab.physik.uni-muenchen.de/Nikolai.Hartmann/xcache-singularity-lrz/-/blob/51d2da52829eb6d8ea377539884f337208141aca/etc/xrootd/xcache.cfg__;!!Mih3wA!SJibOzmy2P3rdD8Ut7m7gYp_bah2pQX2dR2V9U6xiTq9PoQtfjb_MHHDljpOVzHQF5CU$ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Cheers, >>>>>>> Nikolai >>>>>>> >>>>>>> On 7/7/20 1:38 AM, Matevz Tadel wrote: >>>>>>>> Hi Nikolai, >>>>>>>> >>>>>>>> I tried to reproduce it with current master in nearly all ways, >>>>>>>> with/without prefetching and with direct/forwarding mode. Also, with std >>>>>>>> malloc and tcmalloc. No luck :( >>>>>>>> >>>>>>>> Backtrace or core would help a lot at this point. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Matevz >>>>>>>> >>>>>>>> On 2020-07-03 00:54, Nikolai Hartmann wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I'm trying to upgrade to xrootd5 rc4 for our xcache server to >>>>>>>>> mitigate a >>>>>>>>> problem with dCache. >>>>>>>>> >>>>>>>>> Now when i try to read a file through xcache it crashes with >>>>>>>>> "Attempt to >>>>>>>>> free invalid pointer". I attached the corresponding part of the log. >>>>>>>>> Any ideas? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Nikolai >>>>>>>>> >>>>>>>>> ######################################################################## >>>>>>>>> >>>>>>>>> Use REPLY-ALL to reply to list >>>>>>>>> >>>>>>>>> To unsubscribe from the XROOTD-L list, click the following link: >>>>>>>>> https://urldefense.com/v3/__https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1__;!!Mih3wA!Xzk53aW-mEg2pavzme9Hd49MPmno8frpbkh2YetRsquNyAt5jiVsDB91pTNUHA$ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>> >>>> ######################################################################## >>>> Use REPLY-ALL to reply to list >>>> >>>> To unsubscribe from the XROOTD-L list, click the following link: >>>> https://urldefense.com/v3/__https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1__;!!Mih3wA!XFGFw19U2eq-rA0gnMt46KV3Nmc-QzeRzIK6fXXO8cvBFHyGZUlCmc9OXbuf2OOR3Nx7$ >>>> >>>> >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://urldefense.com/v3/__https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1__;!!Mih3wA!UUgXa89ev5J0vPpdGvvHbICVnr6QNXvQ2IcZ9n1-1EWyCIX2l3I_RnFsdnvmvvfjFolt$ >> > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://urldefense.com/v3/__https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1__;!!Mih3wA!UUgXa89ev5J0vPpdGvvHbICVnr6QNXvQ2IcZ9n1-1EWyCIX2l3I_RnFsdnvmvvfjFolt$ > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1