 |
 |
Hi Oliver, So, if the reason is that "It seems /usr/lib64/libXrdVoms-5.so links against /lib64/libvomsapi.so.1 from voms 2.0.14, which matches the statement above" then the question is why is that so? The plugin should link to whatever is available on that machine and if it happens to be an old version, well, that's what will happen. Andy On Wed, 23 Sep 2020, Oliver Freyermuth wrote: > Dear colleagues, > > a few minutes after sending out this mail, it randomly started working again > for me (not every try, but ~50 % of requests are now ok). > > I tracked this further, and found an update on the VOMS upgrade notice: > https://cern.service-now.com/service-portal?id=outage&n=OTG0059138 > > Trying to force one of the two servers (upgraded and non-upgraded): > > 1) Force using VOMS 3.8.0 server: > chmod a-r /etc/grid-security/vomsdir/atlas/voms2.cern.ch.lsc > => All requests with VOMS proxies against our XRootDs consistenly fail, > and we always get "AC issuer key unreadable or unverifiable." > > 2) Force using not-yet-upgraded server: > chmod a-r /etc/grid-security/vomsdir/atlas/lcg-voms2.cern.ch.lsc > => All requests with VOMS proxies against our XRootDs consistenly work. > > So combining this with the OTG message, quoting here: > | During the upgrade it was noticed that the old legacy clients, > voms-proxy-init2, fail to verify the proxy that has been generated. > | Until this issue and its impact has been understood, the upgrade has been > put on hold. Currently lcg-voms2.cern.ch has been upgraded and is showing the > issue, > | voms2.cern.ch still has the old version. > > => XRootD seems to behave like an "old legacy client" according to this > statement. > > For reference, we have these related packages installed from EPEL 7 / XRootD > upstream: > voms-clients-java-3.3.0-3.el7.noarch > xrootd-voms-5.0.2-1.el7.x86_64 > wlcg-voms-dteam-1.0.0-1.el7.noarch > voms-2.0.14-1.el7.x86_64 > wlcg-voms-atlas-1.0.0-1.el7.noarch > wlcg-voms-ops-1.0.0-1.el7.noarch > voms-api-java-3.3.0-1.el7.noarch > > It seems /usr/lib64/libXrdVoms-5.so links against /lib64/libvomsapi.so.1 from > voms 2.0.14, which matches the statement above, > so any site using XRootD with VOMS bindings (and likely also other storage > stacks) is affected from this issue. > > We'll probably force one of the servers on our infrastructure for now. Since > I am unsure what the "correct" solution is, I left in all lists ? > to make it explicit, it is unclear to me if XrdVoms needs to be ported to use > the voms-api-java package, or the new VOMS server version should be made > compatible. > At least I hope my summary clarifies the impact (at least to some degree). > > Cheers, > Oliver > > Am 23.09.20 um 16:25 schrieb Oliver Freyermuth: >> Dear colleagues, >> >> I'm not sure if others have seen it yet, but likely since the VOMS upgrades >> today, it seems all XRootD sites (both production and testing sites) have >> been broken depending on which VOMS proxies are used: >> >> $ voms-proxy-init3 -voms atlas -bits 2048 -old >> $ xrdfs se1.oscer.ou.edu ls /xrd/atlasdatadisk/ >/dev/null >> => works >> >> $ voms-proxy-init3 -voms atlas -bits 2048 -rfc >> $ xrdfs se1.oscer.ou.edu ls /xrd/atlasdatadisk/ >/dev/null >> [ERROR] Server responded with an error: [3010] Unable to locate >> /xrd/atlasdatadisk/; permission denied >> >> I can also reproduce with our Bonn Tier3s production storage: >> $ xrdfs xrootd.physik.uni-bonn.de ls /cephfs/grid/atlas/user/scratch >> [ERROR] Server responded with an error: [3010] Unable to locate >> /cephfs/grid/atlas/user/scratch; permission denied >> >> Checking the XRootD server logs, I find: >> --------------------------------------------- >> 200923 16:18:39 3004 XrdVomsFun: proxy: >> /C=DE/O=GermanGrid/OU=UniBonn/CN=Oliver Freyermuth/CN=1295746099 >> 200923 16:18:39 3004 XrdVomsFun: adding cert: >> /C=DE/O=GermanGrid/OU=UniBonn/CN=Oliver Freyermuth >> 200923 16:18:39 3004 XrdVomsFun: retrieval FAILED: AC issuer key >> unreadable or unverifiable. >> --------------------------------------------- >> >> I'm sorry for adding so many lists, but I think this is important for the >> DOMA-TPC group (since many participants are affected), the VOMS admins (who >> may know if there's still something amiss?) >> and the XRootD developers (who may be able to guess what is broken). >> >> We find this error going back in our logs starting from ~12:30 today. The >> EGI WebDAV probe finally started failing about an hour ago (likely when it >> refreshed it's proxy?). >> >> If there's any better place to contact about this issue, please let me know >> ? since nothing changed locally and many sites are affected, I presume it's >> an effect of the VOMS upgrade. >> >> Cheers, >> Oliver >> > > -- > Oliver Freyermuth > Universität Bonn > Physikalisches Institut, Raum 1.047 > Nußallee 12 > 53115 Bonn > -- > Tel.: +49 228 73 2367 > Fax: +49 228 73 7869 > -- > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1