 |
 |
What release? Git head? On Wed, 8 Jul 2020, Matevz Tadel wrote: > Hi Andy, > > On 2020-07-08 13:49, Andrew Hanushevsky wrote: >> Hi Matevz, >> >> Well, what kind of authentication? Clearly, the kind we use doesn't cause this >> problem. It could be just a random core smash but if it's random we should be >> various effects not just a crash in this particular code path, right? > > xcache without any security config, everything works smooth. > > xcache with sec.protocol /usr/lib64 gsi --- trouble: > > 200708 13:38:44 240995 XrootdXeq: matevz.241046:31@uaf-7 pub IPv4 login as d0ba0e6c.0 > 200708 13:38:44 240995 Posix_P2L: file /eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root pfn2lfn /eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root > [2020-07-08 13:38:44.739012 -0700][Error ][AsyncSock ] [[log in to unmask]:1094.0] Unable to connect: network is unreachable > [2020-07-08 13:38:44.739092 -0700][Error ][PostMaster ] [[log in to unmask]:1094] elapsed = 0, pConnectionWindow = 120 seconds. > [2020-07-08 13:38:45.637583 -0700][Error ][XRootDTransport ] [[log in to unmask]:1094.0] Authentication with gsi failed: > [2020-07-08 13:38:45.974332 -0700][Error ][AsyncSock ] [[log in to unmask]:1095.0] Unable to connect: network is unreachable > [2020-07-08 13:38:45.974400 -0700][Error ][PostMaster ] [[log in to unmask]:1095] elapsed = 0, pConnectionWindow = 120 seconds. > 200708 13:38:46 240995 XrdPfc_Manager: info Cache::Attach() root:[log in to unmask] > 200708 13:38:46 240995 XrdPfc_Manager: debug Cache::GetFile eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root, io 0xe30f50 > 200708 13:38:46 240995 XrdPfc_IO: debug IOEntireFile::initCachedStat get stat from client res = 0, size = 2272072 root:[log in to unmask] > 200708 13:38:46 240995 XrdPfc_File: debug Creating new file info, data size = 2272072 num blocks = 3 eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root > 200708 13:38:46 240995 XrdPfc_Manager: debug Cache::inc_ref_cnt eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root, cnt at exit = 1 > 200708 13:38:46 240995 XrdPfc_File: debug File::AddIO() io = 0xe30f50 eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root > 200708 13:38:46 240995 XrdPfc_Manager: debug Cache::Attach() root:[log in to unmask] location: [log in to unmask]:1095 > [2020-07-08 13:38:47.022428 -0700][Error ][AsyncSock ] [[log in to unmask]:1095.0] Socket error encountered: [ERROR] Invalid arguments > [2020-07-08 13:38:47.022506 -0700][Error ][XRootD ] [[log in to unmask]:1095] Unable to get the response to request kXR_read (handle: 0x00000000, offset: 0, size: 1048576) > [2020-07-08 13:38:47.022625 -0700][Error ][File ] [0xf0b040@root:[log in to unmask]:1094//eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root?xrdcl.requuid=6730e10b-8b40-43bf-9d0a-75da982939e8] Fatal file state error. Message kXR_read (handle: 0x00000000, offset: 0, size: 1048576) returned with [ERROR] Invalid arguments > 200708 13:38:47 241052 XrdPfc_File: error File::ProcessBlockResponse block 0xff3440 0 error=-22 eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root > 200708 13:38:47 240995 XrdPfc_File: error File::Read() io 0xe30f50, block 0 finished with error 22 invalid argument eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root > src/tcmalloc.cc:284] Attempt to free invalid pointer 0x313262003543620a > > Note that while we get all these network errors at the start, cache still got the stat info from the server (knows the size of the file). > > I must admit I never test xcache with auth on :( I'll try it out now, well, after lunch :) > > Matevz > >> Andy >> >> >> On Wed, 8 Jul 2020, Matevz Tadel wrote: >> >>> Yay, that was a journey ... but I can reproduce it now! >>> >>> It is super strange this happens with xcache with authentication on only ... >>> this really should have no effect. I first tried without it and it worked and >>> then something rang a bell that you said so in the email :). >>> >>> Andy, does this ring any bells for you? It looks like interaction between >>> server / client usage of X509 stuffe. >>> >>> Anyway, I'm digging on on the xcache side ... >>> >>> Cheers, >>> Matevz >>> >>> >>> >>> On 2020-07-08 07:41, Nikolai Hartmann wrote: >>>> Hi Matevz, >>>> >>>> I might have something like a "minimal failing example". Unfortunately >>>> the problem only appears when authentication is required, so the example >>>> will only work on a machine that has a valid host certificate and the >>>> corresponding directory has to be bind-mounted into the container. >>>> >>>> I uploaded my container image here: >>>> >>>> https://urldefense.com/v3/__https://cloud.physik.lmu.de/index.php/s/RFC6Q89FBxxNMXF__;!!Mih3wA!S4S4O0y7f1Z5oNAgkr2EZ2J5683bZ5LRbG55GbcoHhyJTwOzaS2lABcIifddJxDGMy-N$ >>>> >>>> >>>> and made a directory structure (tar archive attached) to bind mount into >>>> the container (and containing the minimal failing xcache config and a >>>> script for starting gdb inside the container) >>>> >>>> To reproduce, extract the archive, enter the directory and run (as >>>> non-root user) >>>> >>>> singularity run -B $(pwd)/data:/data -B $(pwd)/config:/etc/xrootd:ro -B >>>> <hostkey-dir>:/etc/grid-security:ro <singularity-image> >>>> >>>> where <hostkey-dir> is a directory that contains >>>> >>>> hostkey.pem >>>> hostcert.pem >>>> vomsdir (will become X509_VOMS_DIR) >>>> certificates (will become X509_CERT_DIR) >>>> >>>> and <singularity-image> is the path to the singularity image. >>>> >>>> That should run xrootd and the log should appear in >>>> data/xrd/var/log/xrootd.log >>>> >>>> I used this example to produce the failure: >>>> >>>> xrdcp -f >>>> root://lcg-lrz-xcache0.grid.lrz.de:1094//root://eospublic.cern.ch//eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root >>>> >>>> /dev/null >>>> >>>> The simplest way to run gdb seemed to directly start xrootd with gdb. >>>> This can be done with the script run_xcache_debug.sh in the attached >>>> archive. Instead of the command above just use >>>> >>>> singularity exec -B $(pwd)/data:/data -B $(pwd)/config:/etc/xrootd:ro -B >>>> <hostkey-dir>:/etc/grid-security:ro <singularity-image> >>>> ./run_xcache_debug.sh >>>> >>>> Note: Before restarting, best delete the content of the data directory >>>> since the bug also did not seem to occur when the file was already >>>> cached (e.g after testing without authentication) >>>> >>>> Sorry for the overly complicated reproducing steps, but since it only >>>> happened when i authentication was enabled i didn't know how to do it >>>> simpler. I hope it helps. >>>> >>>> Thanks, >>>> Nikolai >>>> >>>> On 7/7/20 8:42 PM, Matevz Tadel wrote: >>>>> Thanks Nikolai, I shall continue my investigation :) >>>>> >>>>> Matevz >>>>> >>>>> On 2020-07-06 23:59, Nikolai Hartmann wrote: >>>>>> Hi Matevz, >>>>>> >>>>>> Thanks a lot for looking into this. >>>>>> >>>>>> - The crash seems to happen always when i make a request >>>>>> - Currently prefetching is disabled >>>>>> - Yes, i think it is direct proxy mode >>>>>> - stack trace is attached >>>>>> >>>>>> A similar setup seems to work for Ilija without issues with the xcaches >>>>>> using slate - i tried to mimic that setup closely. Running xrootd from >>>>>> this container image: >>>>>> >>>>>> https://urldefense.com/v3/__https://gitlab.physik.uni-muenchen.de/Nikolai.Hartmann/xcache-singularity-lrz/-/blob/51d2da52829eb6d8ea377539884f337208141aca/xcache.singularity.def__;!!Mih3wA!SJibOzmy2P3rdD8Ut7m7gYp_bah2pQX2dR2V9U6xiTq9PoQtfjb_MHHDljpOV0aWvVYj$ >>>>>> >>>>>> >>>>>> >>>>>> using this config >>>>>> >>>>>> https://urldefense.com/v3/__https://gitlab.physik.uni-muenchen.de/Nikolai.Hartmann/xcache-singularity-lrz/-/blob/51d2da52829eb6d8ea377539884f337208141aca/etc/xrootd/xcache.cfg__;!!Mih3wA!SJibOzmy2P3rdD8Ut7m7gYp_bah2pQX2dR2V9U6xiTq9PoQtfjb_MHHDljpOVzHQF5CU$ >>>>>> >>>>>> >>>>>> >>>>>> Cheers, >>>>>> Nikolai >>>>>> >>>>>> On 7/7/20 1:38 AM, Matevz Tadel wrote: >>>>>>> Hi Nikolai, >>>>>>> >>>>>>> I tried to reproduce it with current master in nearly all ways, >>>>>>> with/without prefetching and with direct/forwarding mode. Also, with std >>>>>>> malloc and tcmalloc. No luck :( >>>>>>> >>>>>>> Backtrace or core would help a lot at this point. >>>>>>> >>>>>>> Cheers, >>>>>>> Matevz >>>>>>> >>>>>>> On 2020-07-03 00:54, Nikolai Hartmann wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> I'm trying to upgrade to xrootd5 rc4 for our xcache server to >>>>>>>> mitigate a >>>>>>>> problem with dCache. >>>>>>>> >>>>>>>> Now when i try to read a file through xcache it crashes with >>>>>>>> "Attempt to >>>>>>>> free invalid pointer". I attached the corresponding part of the log. >>>>>>>> Any ideas? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Nikolai >>>>>>>> >>>>>>>> ######################################################################## >>>>>>>> >>>>>>>> Use REPLY-ALL to reply to list >>>>>>>> >>>>>>>> To unsubscribe from the XROOTD-L list, click the following link: >>>>>>>> https://urldefense.com/v3/__https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1__;!!Mih3wA!Xzk53aW-mEg2pavzme9Hd49MPmno8frpbkh2YetRsquNyAt5jiVsDB91pTNUHA$ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://urldefense.com/v3/__https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1__;!!Mih3wA!XFGFw19U2eq-rA0gnMt46KV3Nmc-QzeRzIK6fXXO8cvBFHyGZUlCmc9OXbuf2OOR3Nx7$ >>> > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1