 |
 |
Hi Andy, Am 24.09.20 um 09:10 schrieb Andrew Hanushevsky: > Hi Oliver, > > So, if the reason is that "It seems /usr/lib64/libXrdVoms-5.so links against /lib64/libvomsapi.so.1 from voms 2.0.14, which matches the statement above" then the question is why is that so? The plugin should link to whatever is available on that machine and if it happens to be an old version, well, that's what will happen. Andrea has confirmed by now that the current breakage is regarded as a bug in the upstream VOMS servers, since it was not planned to drop VOMS 2 client support (yet?). The version I use is the pre-built one from the XRootD repositories. However, I would suspect that even if VOMS 3.x is installed on the system on which XRootD is compiled, it might not be picked up — VOMS 3 is a rewrite in Java as far as I know, so likely it needs changes in the CMake scripts and may even need adapting the code. After Andrea's statement, I think migration there is not a pressing issue (if at all), though. Cheers, Oliver > > Andy > > On Wed, 23 Sep 2020, Oliver Freyermuth wrote: > >> Dear colleagues, >> >> a few minutes after sending out this mail, it randomly started working again for me (not every try, but ~50 % of requests are now ok). >> >> I tracked this further, and found an update on the VOMS upgrade notice: >> https://cern.service-now.com/service-portal?id=outage&n=OTG0059138 >> >> Trying to force one of the two servers (upgraded and non-upgraded): >> >> 1) Force using VOMS 3.8.0 server: >> chmod a-r /etc/grid-security/vomsdir/atlas/voms2.cern.ch.lsc >> => All requests with VOMS proxies against our XRootDs consistenly fail, and we always get "AC issuer key unreadable or unverifiable." >> >> 2) Force using not-yet-upgraded server: >> chmod a-r /etc/grid-security/vomsdir/atlas/lcg-voms2.cern.ch.lsc >> => All requests with VOMS proxies against our XRootDs consistenly work. >> >> So combining this with the OTG message, quoting here: >> | During the upgrade it was noticed that the old legacy clients, voms-proxy-init2, fail to verify the proxy that has been generated. >> | Until this issue and its impact has been understood, the upgrade has been put on hold. Currently lcg-voms2.cern.ch has been upgraded and is showing the issue, >> | voms2.cern.ch still has the old version. >> >> => XRootD seems to behave like an "old legacy client" according to this statement. >> >> For reference, we have these related packages installed from EPEL 7 / XRootD upstream: >> voms-clients-java-3.3.0-3.el7.noarch >> xrootd-voms-5.0.2-1.el7.x86_64 >> wlcg-voms-dteam-1.0.0-1.el7.noarch >> voms-2.0.14-1.el7.x86_64 >> wlcg-voms-atlas-1.0.0-1.el7.noarch >> wlcg-voms-ops-1.0.0-1.el7.noarch >> voms-api-java-3.3.0-1.el7.noarch >> >> It seems /usr/lib64/libXrdVoms-5.so links against /lib64/libvomsapi.so.1 from voms 2.0.14, which matches the statement above, >> so any site using XRootD with VOMS bindings (and likely also other storage stacks) is affected from this issue. >> >> We'll probably force one of the servers on our infrastructure for now. Since I am unsure what the "correct" solution is, I left in all lists ? >> to make it explicit, it is unclear to me if XrdVoms needs to be ported to use the voms-api-java package, or the new VOMS server version should be made compatible. >> At least I hope my summary clarifies the impact (at least to some degree). >> >> Cheers, >> Oliver >> >> Am 23.09.20 um 16:25 schrieb Oliver Freyermuth: >>> Dear colleagues, >>> >>> I'm not sure if others have seen it yet, but likely since the VOMS upgrades today, it seems all XRootD sites (both production and testing sites) have been broken depending on which VOMS proxies are used: >>> >>> $ voms-proxy-init3 -voms atlas -bits 2048 -old >>> $ xrdfs se1.oscer.ou.edu ls /xrd/atlasdatadisk/ >/dev/null >>> => works >>> >>> $ voms-proxy-init3 -voms atlas -bits 2048 -rfc >>> $ xrdfs se1.oscer.ou.edu ls /xrd/atlasdatadisk/ >/dev/null >>> [ERROR] Server responded with an error: [3010] Unable to locate /xrd/atlasdatadisk/; permission denied >>> >>> I can also reproduce with our Bonn Tier3s production storage: >>> $ xrdfs xrootd.physik.uni-bonn.de ls /cephfs/grid/atlas/user/scratch >>> [ERROR] Server responded with an error: [3010] Unable to locate /cephfs/grid/atlas/user/scratch; permission denied >>> >>> Checking the XRootD server logs, I find: >>> --------------------------------------------- >>> 200923 16:18:39 3004 XrdVomsFun: proxy: /C=DE/O=GermanGrid/OU=UniBonn/CN=Oliver Freyermuth/CN=1295746099 >>> 200923 16:18:39 3004 XrdVomsFun: adding cert: /C=DE/O=GermanGrid/OU=UniBonn/CN=Oliver Freyermuth >>> 200923 16:18:39 3004 XrdVomsFun: retrieval FAILED: AC issuer key unreadable or unverifiable. >>> --------------------------------------------- >>> >>> I'm sorry for adding so many lists, but I think this is important for the DOMA-TPC group (since many participants are affected), the VOMS admins (who may know if there's still something amiss?) >>> and the XRootD developers (who may be able to guess what is broken). >>> >>> We find this error going back in our logs starting from ~12:30 today. The EGI WebDAV probe finally started failing about an hour ago (likely when it refreshed it's proxy?). >>> >>> If there's any better place to contact about this issue, please let me know ? since nothing changed locally and many sites are affected, I presume it's an effect of the VOMS upgrade. >>> >>> Cheers, >>> Oliver >>> >> >> -- >> Oliver Freyermuth >> Universität Bonn >> Physikalisches Institut, Raum 1.047 >> Nußallee 12 >> 53115 Bonn >> -- >> Tel.: +49 228 73 2367 >> Fax: +49 228 73 7869 >> -- >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >> > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.047 Nußallee 12 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 -- ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1