Hi Fabrizio, Andreas,
I fully agree with Fabrizio's wishlist. And it would be great if you
two can sit together to "once and for all" fix this. Once done,
decommission asap the classic configure and go forward from there.
Cheers, Fons.
Fabrizio Furano wrote:
> Ok Andreas, thanks.
> So we can commit something general which can help in setting up a simple
> system in a simple way. I realized that this is a problem which is worse
> than I thought. Better late than never, but maybe I should have realized
> this before. At least 4 naive script sets to do the same thing seems a
> bit too much to me just to continue believing that everything is fine...
>
> Fabrizio
>
>
> Il giorno 14-dic-07, alle ore 00:18, Andreas Joachim Peters ha scritto:
>
>> Fabrizio,
>> I pass by tomorrow and we can produce what you need in half an hour.
>> You will see, you can live perfectly with the autotools make as it is
>> - we need just one tiny
>>
>> modification and we can produce two init.d scripts with copy and paste
>> from the CAF or Castor ones.
>> We should also fix LD_LIBRARY_PATH in StartXRD etc. for Mac
>> (DY_LD_LIBRARY_PATH) and should cover the library location on 64bit
>> machines (lib64).
>>
>> Cheers Andreas.
>>
>> -----Original Message-----
>> From: Fabrizio Furano [mailto:[log in to unmask]]
>> Sent: Thu 12/13/2007 7:34 PM
>> To: Fons Rademakers
>> Cc: Derek Feichtinger; [log in to unmask]; Wilko Kroger;
>> [log in to unmask]; Andreas Joachim Peters
>> Subject: Re: Autotools install
>>
>> Hi Fons,
>>
>> I agree with you. I don't want to throw away the autotools build, but I
>> don't want to throw away the classic build, since it's the one that up to
>> now works without having to mess up too much, and it's more easily
>> adaptable.
>>
>> In general, what should be achieved is that after the 'make' the
>> software
>> works locally, and a 'make install' installs it system-wide, like in the
>> ROOT case btw.
>>
>> My point as a developer is that the autotools build fails the
>> validation
>> now, so the bridges we should burn are the ones which are built on top of
>> the current status of it. Just too obscure and complicated to start
>> even a
>> simple system. An indication of this should be that afaik the only
>> customer
>> of it is Alice, due to the decision of choosing the placement of the
>> various pieces with a very fine granularity, and the classic build cannot
>> do that, because it lacks a 'make install', but for the rest is perfect.
>>
>> Only doing that IMO we can fix the autotools or classic build without
>> making it more complicated, no matter if in Alien or xrdcastor or Proof
>> there can be a lot of naive scripts to fix (or purge/simplify)
>> accordingly.
>>
>> From my perspective there are not so many things to fix, and an init.d
>> startup script to add. My wishlist is as follows:
>>
>> - right after make:
>> - the libs should be in a unique visible directory lib
>> - the executables should be in a unique visible directory bin
>> - etc, utils and similar should stay where they are now
>> ... otherwise the developers will never consider seriously to adopt it.
>>
>> - after make install:
>> - a possibly unique init.d script is there
>> - if I start it, both xrootd and cmsd start in a very basic fashion,
>> serving data from /tmp and trying to subscribe to a fake manager
>> - the init.d script invokes the classic StartXRD/OLB scripts
>> - the utils package should appear somewhere as it is in the
>> pre-install
>> tree, i.e. without spreading its content. A nice place could be in
>> ...etc/xrootd/utils
>>
>> I'd like to hear the opinion from the guys who are involved, however.
>> Again, my point now is that I am puzzled about what should I do to set
>> up a
>> cluster. Writing my own (new?) scripts over the current autotools, the
>> classic make, or devoting this time to enhance the current make system
>> instead of thinking about the Nth workaround?
>>
>> Fabrizio
>>
>>
>>
>>
>> Fons Rademakers wrote:
>> > THe main problem is that the old system has been kept alive. You
>> need to
>> > burn bridges to go forward. In ROOT we use several other OS packages
>> > based on autotools (freetype, pcre, libAfterImage). No problems to use
>> > the from our build system. It is up to the xrootd developers to
>> validate
>> > the new auotools system and throw out the old one. We will be happy to
>> > adapt to it.
>> >
>> > Cheers Fons.
>> >
>> >
>> > Derek Feichtinger wrote:
>> >> Ciao, Fabrizio
>> >>
>> >> On Thursday 13 December 2007, Fabrizio Furano wrote:
>> >>> Hi Derek,
>> >>>
>> >>> I appreciate your responsiveness, but I don't want to steer into
>> >>> philosophical talks. Maybe ZX spectrum was better than C=64, but I
>> don't
>> >>> care now.
>> >>
>> >> Well, I did not want to sound philosophical, but I tried to state the
>> >> real advantages and disadvantages as I see them. Xrootd has made an
>> >> own standard of its build and deployment structure, and a large number
>> >> of users have adapted to this. The autotools build follows a very
>> >> widely spread standard in unix software engineering, and this was also
>> >> why I was asked to implement it (which caused me and Gerri a lot of
>> >> work at that time). I still think that it is worth make a reasonable
>> >> effort to try to comply with standards, but naturally in the crazy
>> >> fast changing world of software development, reasonable is defined
>> >> very differently by the users.
>> >>
>> >> As I said. I can introduce and adapt the configure easily, if this is
>> >> needed and if you can suggest to me exactly what you want. On the
>> >> other hand, one could rethink the decision to support autotools.
>> >> However, if xrootd gets a wider range of users, as it certainly
>> >> deserves, it may be an advantage to keep the autobuild, especially
>> >> since the main work has already been done.
>> >>
>> >> Cheers,
>> >> Derek
>> >>
>> >>
>> >>
>> >>> The point is that there are tenths of sites (with many different
>> >>> architectures, and not only babarians) using the plain classic
>> build and
>> >>> the rpms that Wilko provides.
>> >>> So, from my perspective, I just see that if I try to use the
>> autotools
>> >>> build:
>> >>>
>> >>> - as a developer i don't feel comfortable with it. For instance, I
>> don't
>> >>> want to choose if to look at the libraries in a hidden directory or
>> >>> being
>> >>> forced to install everything in /usr/, or having to look at the
>> >>> executables
>> >>> in N different places.
>> >>> - if I want to setup a cluster, or even a single server, I have no
>> >>> clue at
>> >>> all about how to start it, since the bundled scripts (used by a
>> lot of
>> >>> people) do not work.
>> >>> - I don't want to mess up things and write shell scripts on top of
>> >>> something just to start my own executables (btw this is what Alien
>> >>> seems to
>> >>> do, workarounds over workarounds). It should just work out of the
>> >>> box, like
>> >>> in the classic case.
>> >>>
>> >>>
>> >>> Given this, which I think is pretty pragmatic, I'd like to be even
>> >>> more
>> >>> pragmatic. Since this morning I am fighting to install a manager
>> with a
>> >>> trivial config file, with no success. This makes me quite upset. I am
>> >>> not
>> >>> able to deal with my own stuff.
>> >>> I switched back to configure.classic and it works. So, I have no
>> >>> idea of
>> >>> what's good or bad, but please, if you have a clean list of
>> >>> hints/fixes to
>> >>> do, my wish is to work together and put the 'alternate' build system
>> >>> in a
>> >>> state where it is usable and it does not need workarounds.
>> >>> For how it is now, if I have to setup a cluster somewhere, I will
>> >>> simply
>> >>> ignore it, but that's not what I'd prefer.
>> >>>
>> >>> I hope that I don't make you upset with this. Let's make this thing
>> >>> work
>> >>> together if you want.
>> >>>
>> >>>
>> >>> Fabrizio
>> >>>
>> >>> Derek Feichtinger wrote:
>> >>>> Hi, Fabrizio
>> >>>>
>> >>>> I'm sorry that you have so much fuzz with this.
>> >>>>
>> >>>> Some comments follow:
>> >>>>> So, if I keep the default prefix /usr/local, I expect to have the
>> >>>>> config files in /usr/local/etc, and the libs in /usr/local/lib, but
>> >>>>> what I see now is that "make install" puts that stuff into
>> /usr/local/
>> >>>>> etc/xrootd and similar for lib.
>> >>>> The choice for $PREFIX/etc/xrootd instead of just $PREFIX/etc was
>> taken
>> >>>> because an administrator does not want /etc cluttered with a whole
>> >>>> number
>> >>>> of files from a single package, so it is nicer to have it in a
>> separate
>> >>>> directory named for the service.
>> >>>>
>> >>>> StartOLB
>> >>>> StartOLB.cf.example
>> >>>> StartXRD
>> >>>> StartXRD.cf.example
>> >>>> StopOLB
>> >>>> StopXRD
>> >>>> XrdOlbMonPerf
>> >>>> xrootd.cf.example
>> >>>>
>> >>>> I can easily build in an option, if this is a problem (hardcoded
>> >>>> locations?). But usually, one wants to have typical init scripts
>> which
>> >>>> fit into a system's service startup and shutdown structure.
>> >>>>
>> >>>> For the libraries you looked wrongly, for they are in $PREFIX/lib
>> >>>> and not
>> >>>> in a separate folder. This would break the standard convention for
>> >>>> libraries. But the include files are also segregated into
>> >>>> $PREFIX/include/xrootd/ for better structuring.
>> >>>>
>> >>>>> This is not compatible with the standard StartXRD scripts, which
>> >>>>> everybody use (except Alice afaik). What do you think about
>> this? Are
>> >>>>> you aware of any workaround for that?
>> >>>> While I was working for ALICE, we always used custom scripts. But
>> since
>> >>>> xrootd needs a minimum of switches and mainly relies on the config
>> >>>> files,
>> >>>> this never seemed a drawback to me. It involved just a few lines of
>> >>>> shell
>> >>>> code.
>> >>>>
>> >>>>> Btw in the meantime, I will
>> >>>>> install my machines like I always did, i.e. with the plain
>> >>>>> configure.classic, but that is not what the Alice guys are used to,
>> >>>>> even if it's much simpler by now imho.
>> >>>> Well, configure.classic works well and is faster than the autotools
>> >>>> build. But autotools is still _the_ standard in the build system
>> and
>> >>>> portability world. New systems like cmake are popping up, but these
>> >>>> things are really difficult and painful to develop, and they take
>> long
>> >>>> until a big community adopts them.
>> >>>>
>> >>>> Also, if you look at a xrootd Makefile.am and then at the
>> corresponding
>> >>>> classic GNUMakefile, the Makefile.am is structured much simpler.
>> >>>> It is trivial to build RPMs and other packages from an autotools
>> build,
>> >>>> since it correctly observes the DESTDIR setting (packaging
>> directory),
>> >>>> and the libraries correctly contain the right -rpath, so that no
>> >>>> LD_LIBRARY_PATH needs to be set.
>> >>>> Autotools configure is slower, because it makes real compilation
>> tests
>> >>>> for the system's features. The generated configure script shows
>> all the
>> >>>> standard behavior expected by users and offers a wide range of user
>> >>>> options. Also, you can build multiple architectures from the same
>> >>>> sources, e.g. by having them on a shared filesystem.
>> >>>>
>> >>>> Configure.classic is a separate and well working system, but it has
>> >>>> completely non-standard behavior and if I want to deploy software
>> in a
>> >>>> standard way, I have to do extra work.
>> >>>>
>> >>>> The technology used by autotools to generate the configure (m4,
>> >>>> etc.) is
>> >>>> too old and inconvenient, and some things are clearly too complex.
>> >>>> However, since so much work has been done before, you usually don't
>> >>>> have
>> >>>> to deal with these kind of issues, and you more or less just use the
>> >>>> ready made macros. Don't forget that a large part of the complexity
>> >>>> comes
>> >>>> for the innate problems that portability implies - years of
>> operations
>> >>>> systems development and complex and subtle differences across their
>> >>>> versions.
>> >>>>
>> >>>> Sorry again for your losing time because of these issues, but it
>> really
>> >>>> was hard for me in the last few weeks to pay close attention to
>> >>>> this, and
>> >>>> keeping a separate build is not always easy.
>> >>>>
>> >>>> Cheers,
>> >>>> Derek
>> >>>>
>> >>>> On Thursday 13 December 2007, Fabrizio Furano wrote:
>> >>>>> Hi Derek,
>> >>>>>
>> >>>>> well, I am sorry to bother you so much, but I am not able to
>> get out
>> >>>>> from this maze.
>> >>>>>
>> >>>>> The problem I find is that when I make install, it puts the
>> things
>> >>>>> in a way which looks incompatible with the normal start/stop
>> scripts,
>> >>>>> which hence do not work. This may be one reason for the bloody
>> mess of
>> >>>>> alternative install/start/stop scripts that I see around.
>> >>>>>
>> >>>>> So, if I keep the default prefix /usr/local, I expect to have the
>> >>>>> config files in /usr/local/etc, and the libs in /usr/local/lib, but
>> >>>>> what I see now is that "make install" puts that stuff into
>> /usr/local/
>> >>>>> etc/xrootd and similar for lib.
>> >>>>>
>> >>>>> This is not compatible with the standard StartXRD scripts, which
>> >>>>> everybody use (except Alice afaik). What do you think about
>> this? Are
>> >>>>> you aware of any workaround for that? Btw in the meantime, I will
>> >>>>> install my machines like I always did, i.e. with the plain
>> >>>>> configure.classic, but that is not what the Alice guys are used to,
>> >>>>> even if it's much simpler by now imho.
>> >>>>>
>> >>>>> Fabrizio Furano
>> >>>>> [log in to unmask]
>> >>
>> >>
>> >>
>> >
>>
>
> Fabrizio Furano
> [log in to unmask]
>
>
>
--
Org: CERN, European Laboratory for Particle Physics.
Mail: 1211 Geneve 23, Switzerland
E-Mail: [log in to unmask] Phone: +41 22 7679248
WWW: http://fons.rademakers.org Fax: +41 22 7669640
