Well, I don't quite understand the problem here. It certainly doesn't look
any more complicated than what I would expect to find in a regular Linux
distribution teaming with even more cross-party dependencies. Is it that
people are indiscriminately pulling new version of xrootd or that third
party providers are trying to release stuff that depends on things not in
EPEL? If either is true, then this is *not* a technical problem. This is a
people problem and, as such, impossible to fix with a technical solution.
So, the only technical part that I do see here is:
a) Binaries that link to versioned shared libraries, and
b) Plugins that may link to other versioned shared libraries.
Is there a conflict between the two? It doesn't seem like it. The RPM
doesn't (I think) remove old version of shared libraries. So, anyone
depending on the old versions can continue. The only issue, as I see it, is
in the area of plugins. Of course, as long as the ABI is compatible with the
plug-in it should work. If it isn't, then in the current version the plugin
is likely to crash. Just like the old days of Mozzilla.
Now, that said, in 3.3.0, xrootd will have internal version consistency
checking (I'm a couple of days from checking it in). If a plug-in (actually
any plug-in associated cross-dependency) is not compatible with the plugin
loader, the plugin will not be loaded and a message will be issued showing
the two version numbers that are incompatible. As this is currently
voluntary (except for authentication plugins) it will work like the old
system until plugin writers get up to speed by including two lines in their
plugin to record the version number.
Will this solve the problem? Not fully, it merely moves it to a different,
but less crash-prone, space.
So, how can we solve it? Well, for the DPM case below, they simply cannot
expect to distribute things that depend on things that are not in EPEL. It's
totally illogical for them to do otherwise. Practically, they must provide
alternate sources for yum to load dependencies and install those relative to
the DPM package, not in the default location which would clobber everything
else. Of course, in this case, the DPM folks need the new features on xrootd
as opposed to some benign dependency. As long as xrootd sites and DPM sites
are mutually exclusive then there is no problem. It's probably the case and
if this is not, then this problem is no different than any other software
package that requires two different versions to be run.
As for people pulling new versions without thought, we should just tell them
to stop. This is pure mismanagement of a site, plain and simple. As an
aside, the way we have solved this problem for some experiments is by
actually versioning the installation so we can have multiple version
available and making one the default one that can't change without admin
approval.
Finally, to answer Lukasz's proposals:
#1 absolutely not, it's not a responsible thing to do.
#2 No, as you point out, it will always be incomplete. This is why we have
release notes. People should read them before upgrading. However, we should
make it very plain as to any incompatibilities. Which, btw, are relatively
rare.
#3 Pulling out of EPEL is, both, a technical and a political issue. So, we
should get the political side of the house to weigh in. I see advantages
being in EPEL and I don't see why we can't if we can just convince people to
not have unrealistic expectations on what service EPEL actually provides.
Any more thoughts?
Andy
-----Original Message-----
From: Lukasz Janyst
Sent: Tuesday, June 05, 2012 9:18 AM
To: [log in to unmask]
Subject: Fwd: Re: xrootd 3.2 in epel
FYI. This is really a hard one... Please let me know if you have any
thoughts
on all this.
Lukasz
---------- Forwarded Message ----------
Subject: Re: xrootd 3.2 in epel
Date: Tuesday, June 05, 2012, 06:07:32 PM
From: Lukasz Janyst <[log in to unmask]>
To: Ricardo Rocha <[log in to unmask]>
CC: Mattias Ellert <[log in to unmask]>, Wahid Bhimji
<[log in to unmask]>, Simone Campana <[log in to unmask]>, Sam
Skipsey <[log in to unmask]>, Cristina Aiftimiei
<[log in to unmask]>
Hello everyone,
thanks for these emails, they help me understand where we are and where
the
problem really is, and it's a hard one.
The concern of Mattias (and the Fedora guidelines writers) is that the
third
party code depending on the xrootd libraries will break when the not ABI-
compatible binaries are pushed. This is a valid concern. However, the only
thing that should depend on the xrootd libraries is the plugins that are
loaded by the xrootd server daemons and the clients. Since we don't version
the daemon executables, they won't be able to load the plugins depending on
old versions of the libraries. So having the compat packages solves the
problem only for the clients, for the servers it only pushes it from
build/installation time to runtime.
IMO the only viable solutions are:
1) Not updating to new non-bugfix releases of XRootD in EPEL repos meant for
released Fedora/RHEL - apparently not acceptable.
2) Always consult all the packages depending on XRootD when updating
(painful,
long, and probably very inaccurate)
3) Pull xrootd out from EPEL and have it managed externally or have a way to
safely override EPEL provided XRootD with externally provided XRootD, so
that
the sites who don't care about every possible third party package that
depends
on EPEL still get what they want via external yum.
Mattias, I would be very much interested in your thoughts about this.
Cheers,
Lukasz
On Tuesday, June 05, 2012 04:38:00 PM Ricardo Rocha wrote:
> I cc Lukasz as he was mentioning he might work on providing such a
> compat package.
>
> I still think that getting in contact with every user - we know them
> all - would be the simpler solution.
>
> On Mon, Jun 4, 2012 at 7:55 PM, Ricardo Rocha <[log in to unmask]>
> wrote:
> > Hi Mattias.
> >
> > Here's our issue... the dpm-xrootd component requires xrootd 3.2 or
> > above.
> >
> > I do not know the details of the incompatibility being introduced, i
> > just know that we need this version. So for us 3.2 not getting in EPEL
> > means dpm-xrootd will not be getting in EPEL either.
> >
> > Which means we'll keep distributing this component in the EMI and UMD
> > repositories, and need xrootd 3.2 in these externals.
> >
> > I'll go and discuss with the xrootd guys in the meantime... but
> > honestly i think the easiest and only solution is still to coordinate
> > the introduction of this new version with all the dependencies - and
> > try not to have this happening again in the future.
> >
> > Thanks again,
> > Ricardo
> >
> > On Mon, Jun 4, 2012 at 7:43 PM, Mattias Ellert
> >
> > <[log in to unmask]> wrote:
> >> ons 2012-05-30 klockan 13:22 +0100 skrev Wahid Bhimji:
> >>> Hi Mattias,
> >>>
> >>> We are keen to test (and use) the new DPM xrootd server but
> >>> apparently
> >>> this requires xrootd 3.2 in epel .
> >>>
> >>> I notice there are some builds but don't see the rpm in epel-6 (or
> >>> testing). Any idea when this might be available?
> >>>
> >>> Many Thanks
> >>>
> >>> Wahid
> >>>
> >>> -----------------------------
> >>> Dr Wahid Bhimji
> >>
> >> Hi!
> >>
> >> There are many things to consider here - in particular the following:
> >>
> >> Updating xrootd in EPEL would mean a backward incompatible update with
> >> soname bumps. The EPEL update guidelines says the following about
> >> updates that changes the ABI of a package:
> >>
> >> http://fedoraproject.org/wiki/EPEL/GuidelinesAndPolicies
> >>
> >> "this update should be avoided if possible at all. If there really is
> >> no
> >> other way out to fix a serious bug then it rare cases it might be
> >> acceptable to build the new version for the testing branch and mention
> >> the update and the needed adjustments in the release notes for the
> >> next
> >> update. An additional compat- packages with the old libs is necessary
> >> if
> >> the ABI changed."
> >>
> >> Making an xrootd-compat with the old version that works correctly
> >> would
> >> require a lot of work, since xrootd uses plugins loaded from the
> >> default
> >> library path, and the non-versioned library names for these would
> >> conflict between the main xrootd package and the compat package in
> >> that
> >> case, unless a lot of plugin handling in xrootd was rewritten.
> >>
> >> So if the package would be updated to version 3.2 it would have to be
> >> without adding a compat package, which would violate the guidelines.
> >> If
> >> only the packages in Fedora and EPEL were concerned, the coordination
> >> would not be unmanageable -- not that many packages in Fedora depend
> >> on
> >> xrootd. But this will potentially affect a lot of third party software
> >> repositories and privately compiled software, that would become
> >> unusable
> >> when the update is made. This would most likely have severe impact on
> >> sites running EMI and UMD software and could potentially cause havoc
> >> for
> >> a lot of production systems.
> >>
> >> These are the reasons I have not made updates that changes the sonames
> >> in released versions of Fedora and EPEL, and why F15, EPEL 5 and EPEL
> >> 6
> >> has stayed at version 3.0.5, F16 and F17 has stayed at version 3.1.1
> >> and
> >> only F18 has been updated to version 3.2.1.
> >>
> >> If you rely on new functionality in the xrootd libraries that were
> >> introduced after version 3.0.5, and want to use this in EPEL 5 and 6,
> >> another possibility would be to backport this new functionality to the
> >> 3.0.5 version, making sure that the existing ABI is preserved. I know
> >> that xrootd upstream only maintains their latest released version, so
> >> getting help from them in this effort is probably not possible. How
> >> much
> >> work this is, or if it is even possible I don't know. I don't know if
> >> this is a small thing or if the changes you need are tightly coupled
> >> to
> >> the changes in the ABI which would make backporting impossible.
> >>
> >> Making the update to version 3.2 in EPEL is of course not entirely
> >> ruled
> >> out - guidelines are not laws after all. But any such suggestion must
> >> be
> >> extremely well motivated, and in my mind only a non-backportable
> >> security issue would really fly. If such an update would be attempted
> >> it
> >> must be thoroughly coordinated with all known users of the xrootd
> >> libraries - the EMI consortia, UMD release management, OSG software
> >> developers and who knows who else. As you can see this is not
> >> something
> >> that can be done easily.
> >>
> >> Mattias
-----------------------------------------
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
|
