Hi Andy and Lukasz,

I think we should try with Andy's proposal. I expect to meet Matthias in Sweden on Thu next week. Lets try to meet before I leave on Wed.

Cheers, Dirk


On 07.06.2012, at 07:26, "Andrew Hanushevsky" <[log in to unmask]> wrote:

> Hi Lukasz,
> 
> Well, a few things are going one here.
> 
> 1) it's pretty hard to easily support multiple version if they delete the previous set of libraries. I can see their reluctance as it solves one set of problems but merely introduces another set. There are several work arounds for that but none of them that they would like. The way we do it (and many others for different reasons) is simply installing the package in a versioned directory with symlinks to the "production" version. Then people can pick and choose which version they would like to run if not the production one. Some people like that others hate it. However, I don't see a clear solution other than that because as things stand now, to run a different version of package A means you have to install a full release of all other packages that A was bundled with (I suppose RedHat has the same problem). Not a pretty situation.
> 
> So, if EPEL wants to be a RedHat-like distributor then what's in EPEL is the "sanctioned" release and if someone wants something different then they are on their own to install that particular package from an external source and come up with a way to deploy it. Not nice; but it fulfills EPEL's mission.
> 
> 2) As for ABI compatability, I think it's a stretch to say it's impossible to provide ABI compatability. We actually have been much better at this than most software distributers. I can count on one hand where people had to recompile a plugin between releases. Of course, that depends on what they define as ABI compatability. I think we should only go as far as plugins. We could do that but not when they delete previous versions of the libraries. Supplying new features is completely separate from providing ABI compatability to plugins and both can co-exist, depending on how it's managed. Doing it via compat libraries is pretty painful, so I can't say I'm convinced that's a good way to go.
> 
> So, what's the bottom line? Basically, I'd say that EPEL's mission conflicts with what other software providors want to accomplish. That's not to say that EPEL's mission is bad. Hey, people love RH simply because they know exactly what they are getting. It's just that it doesn't fit a fast moving environment. It's not supposed to.
> 
> The only real solution it seems is for EPEL to loosen up a bit and we become more strict. My proposal would be:
> 
> a) We create a shared library of utilities commonly used by plug-in writers. It's pretty easy to see what they would be. This library is managed just as we manage plugin interfaces, as follows:
> 
> o All patch releases (i.e. x.y.<patch> maintain ABI compatability and do
>  not introduce any new features.
> o We are allowed to introduce new features in a minor release (i.e.,
>  x.<minor>.<patch>) as long as they keep ABI compatability.
> o Major releases may require a recompilation and would only be included in
>  the next EPEL release.
> 
> b) We make no gaurantees on interfaces in our private shared libraries.
>   If a plugin writer wants to use something there, they are on their own.
>   Hence, they would be versioned differently than the common utilities
>   library (i.e. its version would stay the same across patch and minor
>   releases).
> c) We provide a distribution that people can use to over-ride what's in
>   EPEL if they choose to go that route.
> 
> Why don't we get together with the relevant parties while I am at CERN next week to sort this out, if possible.
> 
> Andy
> 
> On Wed, 6 Jun 2012, Lukasz Janyst wrote:
> 
>> Hi Andy,
>> 
>>  I believe it's a problem of release cycles. For EPEL the repository they
>> provide for, say, RHEL5 is released, third-parties have built upon it, and
>> it's considered stable, so that everyone who depends on it can be sure that
>> nothing will ever be broken. This will make it very hard to convince them to
>> accept anything that is not ABI-compatible. As you could see from the email of
>> Mattias, he had no problem pushing new things to Rawhide, which is their
>> development branch.
>> 
>>  Now, for us the situation is quite different. RHEL5 is out primary
>> development platform and it's in our interest to have the latest and the
>> greatest things distributed there, but at the same time it's close to
>> impossible to ensure the binary compatibility.
>> 
>>  If it comes to RPMs, they do delete the old versions of the libraries, but
>> this is not that much of a problem because, so called, compat packages can be
>> built. It's somewhat painful, but doable. As both I and Mattias mention, the
>> problem arises when we consider the plug-ins. In my email I have (silently)
>> assumed that we have a way of versioning the plug-ins. This, however, doesn't
>> help much in this case because we will still (gracefully) fail, since new
>> daemons won't be able to load the old third-party plug-ins, hence this only
>> moves the problem from one space to another. If this is acceptable, then we
>> indeed have no real problem, just quite some work on my part. I am afraid this
>> is not really acceptable for EPEL though, and this is why we may need an
>> independent distribution channel.
>> 
>> Cheers,
>>  Lukasz
>> 
>> On Tuesday, June 05, 2012 02:36:58 PM Andrew Hanushevsky wrote:
>>> Well, I don't quite understand the problem here. It certainly doesn't look
>>> any more complicated than what I would expect to find in a regular Linux
>>> distribution teaming with even more cross-party dependencies. Is it that
>>> people are indiscriminately pulling new version of xrootd or that third
>>> party providers are trying to release stuff that depends on things not in
>>> EPEL? If either is true, then this is *not* a technical problem. This is a
>>> people problem and, as such, impossible to fix with a technical solution.
>>> 
>>> So, the only technical part that I do see here is:
>>> a) Binaries that link to versioned shared libraries, and
>>> b) Plugins that may link to other versioned shared libraries.
>>> 
>>> Is there a conflict between the two? It doesn't seem like it. The RPM
>>> doesn't (I think) remove old version of shared libraries. So, anyone
>>> depending on the old versions can continue. The only issue, as I see it, is
>>> in the area of plugins. Of course, as long as the ABI is compatible with the
>>> plug-in it should work. If it isn't, then in the current version the plugin
>>> is likely to crash. Just like the old days of Mozzilla.
>>> 
>>> Now, that said, in 3.3.0, xrootd will have internal version consistency
>>> checking (I'm a couple of days from checking it in). If a plug-in (actually
>>> any plug-in associated cross-dependency) is not compatible with the plugin
>>> loader, the plugin will not be loaded and a message will be issued showing
>>> the two version numbers that are incompatible. As this is currently
>>> voluntary (except for authentication plugins) it will work like the old
>>> system until plugin writers get up to speed by including two lines in their
>>> plugin to record the version number.
>>> 
>>> Will this solve the problem? Not fully, it merely moves it to a different,
>>> but less crash-prone, space.
>>> 
>>> So, how can we solve it? Well, for the DPM case below, they simply cannot
>>> expect to distribute things that depend on things that are not in EPEL. It's
>>> totally illogical for them to do otherwise.  Practically, they must provide
>>> alternate sources for yum to load dependencies and install those relative
>>> to the DPM package, not in the default location which would clobber
>>> everything else. Of course, in this case, the DPM folks need the new
>>> features on xrootd as opposed to some benign dependency. As long as xrootd
>>> sites and DPM sites are mutually exclusive then there is no problem. It's
>>> probably the case and if this is not, then this problem is no different
>>> than any other software package that requires two different versions to be
>>> run.
>>> 
>>> As for people pulling new versions without thought, we should just tell them
>>> to stop. This is pure mismanagement of a site, plain and simple. As an
>>> aside, the way we have solved this problem for some experiments is by
>>> actually versioning the installation so we can have multiple version
>>> available and making one the default one that can't change without admin
>>> approval.
>>> 
>>> Finally, to answer Lukasz's proposals:
>>> #1 absolutely not, it's not a responsible thing to do.
>>> #2 No, as you point out, it will always be incomplete. This is why we have
>>> release notes. People should read them before upgrading. However, we should
>>> make it very plain as to any incompatibilities. Which, btw, are relatively
>>> rare.
>>> #3 Pulling out of EPEL is, both, a technical and a political issue. So, we
>>> should get the political side of the house to weigh in. I see advantages
>>> being in EPEL and I don't see why we can't if we can just convince people to
>>> not have unrealistic expectations on what service EPEL actually provides.
>>> 
>>> Any more thoughts?
>>> 
>>> Andy
>>> 
>>> -----Original Message-----
>>> From: Lukasz Janyst
>>> Sent: Tuesday, June 05, 2012 9:18 AM
>>> To: [log in to unmask]
>>> Subject: Fwd: Re: xrootd 3.2 in epel
>>> 
>>> FYI. This is really a hard one... Please let me know if you have any
>>> thoughts
>>> on all this.
>>> 
>>> Lukasz
>>> 
>>> ----------  Forwarded Message  ----------
>>> 
>>> Subject: Re: xrootd 3.2 in epel
>>> Date: Tuesday, June 05, 2012, 06:07:32 PM
>>> From: Lukasz Janyst <[log in to unmask]>
>>> To: Ricardo Rocha <[log in to unmask]>
>>> CC: Mattias Ellert <[log in to unmask]>, Wahid Bhimji
>>> <[log in to unmask]>, Simone Campana <[log in to unmask]>, Sam
>>> Skipsey <[log in to unmask]>, Cristina Aiftimiei
>>> <[log in to unmask]>
>>> 
>>> Hello everyone,
>>> 
>>>   thanks for these emails, they help me understand where we are and where
>>> the
>>> problem really is, and it's a hard one.
>>> 
>>>  The concern of Mattias (and the Fedora guidelines writers) is that the
>>> third
>>> party code depending on the xrootd libraries will break when the not ABI-
>>> compatible binaries are pushed. This is a valid concern. However, the only
>>> thing that should depend on the xrootd libraries is the plugins that are
>>> loaded by the xrootd server daemons and the clients. Since we don't version
>>> the daemon executables, they won't be able to load the plugins depending on
>>> old versions of the libraries. So having the compat packages solves the
>>> problem only for the clients, for the servers it only pushes it from
>>> build/installation time to runtime.
>>> 
>>>   IMO the only viable solutions are:
>>> 
>>> 1) Not updating to new non-bugfix releases of XRootD in EPEL repos meant for
>>> released Fedora/RHEL - apparently not acceptable.
>>> 2) Always consult all the packages depending on XRootD when updating
>>> (painful,
>>> long, and probably very inaccurate)
>>> 3) Pull xrootd out from EPEL and have it managed externally or have a way to
>>> safely override EPEL provided XRootD with externally provided XRootD, so
>>> that
>>> the sites who don't care about every possible third party package that
>>> depends
>>> on EPEL still get what they want via external yum.
>>> 
>>>   Mattias, I would be very much interested in your thoughts about this.
>>> 
>>> Cheers,
>>>   Lukasz
>>> 
>>> On Tuesday, June 05, 2012 04:38:00 PM Ricardo Rocha wrote:
>>>> I cc Lukasz as he was mentioning he might work on providing such a
>>>> compat package.
>>>> 
>>>> I still think that getting in contact with every user - we know them
>>>> all - would be the simpler solution.
>>>> 
>>>> On Mon, Jun 4, 2012 at 7:55 PM, Ricardo Rocha <[log in to unmask]>
>>>> 
>>>> wrote:
>>>>> Hi Mattias.
>>>>> 
>>>>> Here's our issue... the dpm-xrootd component requires xrootd 3.2 or
>>>>> above.
>>>>> 
>>>>> I do not know the details of the incompatibility being introduced, i
>>>>> just know that we need this version. So for us 3.2 not getting in
>>>>> EPEL
>>>>> means dpm-xrootd will not be getting in EPEL either.
>>>>> 
>>>>> Which means we'll keep distributing this component in the EMI and
>>>>> UMD
>>>>> repositories, and need xrootd 3.2 in these externals.
>>>>> 
>>>>> I'll go and discuss with the xrootd guys in the meantime... but
>>>>> honestly i think the easiest and only solution is still to
>>>>> coordinate
>>>>> the introduction of this new version with all the dependencies - and
>>>>> try not to have this happening again in the future.
>>>>> 
>>>>> Thanks again,
>>>>> Ricardo
>>>>> 
>>>>> On Mon, Jun 4, 2012 at 7:43 PM, Mattias Ellert
>>>>> 
>>>>> <[log in to unmask]> wrote:
>>>>>> ons 2012-05-30 klockan 13:22 +0100 skrev Wahid Bhimji:
>>>>>>> Hi Mattias,
>>>>>>> 
>>>>>>> We are keen to test (and use) the new DPM xrootd server but
>>>>>>> apparently
>>>>>>> this requires xrootd 3.2 in epel .
>>>>>>> 
>>>>>>> I notice there are some builds but don't see the rpm in epel-6
>>>>>>> (or
>>>>>>> testing). Any idea when this might be available?
>>>>>>> 
>>>>>>> Many Thanks
>>>>>>> 
>>>>>>> Wahid
>>>>>>> 
>>>>>>> -----------------------------
>>>>>>> Dr Wahid Bhimji
>>>>>> 
>>>>>> Hi!
>>>>>> 
>>>>>> There are many things to consider here - in particular the
>>>>>> following:
>>>>>> 
>>>>>> Updating xrootd in EPEL would mean a backward incompatible update
>>>>>> with
>>>>>> soname bumps. The EPEL update guidelines says the following about
>>>>>> updates that changes the ABI of a package:
>>>>>> 
>>>>>> http://fedoraproject.org/wiki/EPEL/GuidelinesAndPolicies
>>>>>> 
>>>>>> "this update should be avoided if possible at all. If there really
>>>>>> is
>>>>>> no
>>>>>> other way out to fix a serious bug then it rare cases it might be
>>>>>> acceptable to build the new version for the testing branch and
>>>>>> mention
>>>>>> the update and the needed adjustments in the release notes for the
>>>>>> next
>>>>>> update. An additional compat- packages with the old libs is
>>>>>> necessary
>>>>>> if
>>>>>> the ABI changed."
>>>>>> 
>>>>>> Making an xrootd-compat with the old version that works correctly
>>>>>> would
>>>>>> require a lot of work, since xrootd uses plugins loaded from the
>>>>>> default
>>>>>> library path, and the non-versioned library names for these would
>>>>>> conflict between the main xrootd package and the compat package in
>>>>>> that
>>>>>> case, unless a lot of plugin handling in xrootd was rewritten.
>>>>>> 
>>>>>> So if the package would be updated to version 3.2 it would have to
>>>>>> be
>>>>>> without adding a compat package, which would violate the
>>>>>> guidelines.
>>>>>> If
>>>>>> only the packages in Fedora and EPEL were concerned, the
>>>>>> coordination
>>>>>> would not be unmanageable -- not that many packages in Fedora
>>>>>> depend
>>>>>> on
>>>>>> xrootd. But this will potentially affect a lot of third party
>>>>>> software
>>>>>> repositories and privately compiled software, that would become
>>>>>> unusable
>>>>>> when the update is made. This would most likely have severe impact
>>>>>> on
>>>>>> sites running EMI and UMD software and could potentially cause
>>>>>> havoc
>>>>>> for
>>>>>> a lot of production systems.
>>>>>> 
>>>>>> These are the reasons I have not made updates that changes the
>>>>>> sonames
>>>>>> in released versions of Fedora and EPEL, and why F15, EPEL 5 and
>>>>>> EPEL
>>>>>> 6
>>>>>> has stayed at version 3.0.5, F16 and F17 has stayed at version
>>>>>> 3.1.1
>>>>>> and
>>>>>> only F18 has been updated to version 3.2.1.
>>>>>> 
>>>>>> If you rely on new functionality in the xrootd libraries that were
>>>>>> introduced after version 3.0.5, and want to use this in EPEL 5 and
>>>>>> 6,
>>>>>> another possibility would be to backport this new functionality to
>>>>>> the
>>>>>> 3.0.5 version, making sure that the existing ABI is preserved. I
>>>>>> know
>>>>>> that xrootd upstream only maintains their latest released version,
>>>>>> so
>>>>>> getting help from them in this effort is probably not possible.
>>>>>> How
>>>>>> much
>>>>>> work this is, or if it is even possible I don't know. I don't know
>>>>>> if
>>>>>> this is a small thing or if the changes you need are tightly
>>>>>> coupled
>>>>>> to
>>>>>> the changes in the ABI which would make backporting impossible.
>>>>>> 
>>>>>> Making the update to version 3.2 in EPEL is of course not entirely
>>>>>> ruled
>>>>>> out - guidelines are not laws after all. But any such suggestion
>>>>>> must
>>>>>> be
>>>>>> extremely well motivated, and in my mind only a non-backportable
>>>>>> security issue would really fly. If such an update would be
>>>>>> attempted
>>>>>> it
>>>>>> must be thoroughly coordinated with all known users of the xrootd
>>>>>> libraries - the EMI consortia, UMD release management, OSG
>>>>>> software
>>>>>> developers and who knows who else. As you can see this is not
>>>>>> something
>>>>>> that can be done easily.
>>>>>> 
>>>>>>       Mattias
>>> 
>>> -----------------------------------------
>>> 
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>> 
>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>> 
> 
> ########################################################################
> Use REPLY-ALL to reply to list
> 
> To unsubscribe from the XROOTD-DEV list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1