Hi Frossie,
In order to build Docker images relying on latest qserv deps (i.e.
updated during monthly sprint) I'd like to cron:
[lsstsw@lsst-dev [master] ~]$ publish -t qserv-dev -b b1765 qserv_distrib
eupspkg.create: package contents created for 'qserv_distrib-1.0.0+441',
sources will be fetched via 'package'.
Adding tag 'qserv-dev' at the distribution server.
So that I coud do in my Dockerfile:
RUN bash -c ". /qserv/stack/loadLSST.bash && eups distrib install
qserv_distrib -t qserv-dev --onlydepend"
In order to update qserv deps before building a qserv branch in order to
produce a Docker container.
Would it be possible?
Thanks!
On 11/05/2015 09:06 AM, Fritz Mueller wrote:
> Hi Frossie,
>
> I think the qserv monthlies are closer in spirit to the whole-stack
> bi-annuals than they are to the whole-stack weeklies? These are in
> some sense "external" releases for the qserv-only audience (such as it
> is), and we would like to have them published on the eups distrib
> server with easy-to-tell-someone names (i.e. not containing git SHAs).
>
> Operationally, these are the checkpoints at which we would have new
> folks interested in exploring qserv get on the bus. They are also the
> points at which we bootstrap new devs, in terms of all our package
> dependencies (the qserv team as yet uses the eups-centric dev model,
> and not the lsstsw one, generally.) And this is the stride with which
> we currently update our externally visible published documentation,
> summarize our git log into a more user-centric changelog, etc. This
> has worked well for us because it matches our sprint schedule as well.
>
> I don't think we (qserv) currently have anything like the whole-stack
> weeklies -- the monthlies provide a rapid-enough stride for users, and
> devs operate with latest-monthly base + manually upgraded
> distrib-published dependency packages (when necessary) on top, plus
> latest qserv sources on top. Devs generally resync their stack for
> dependencies after each monthly release. Since we have so few
> dependencies, we usually have only one or two, if any, dependent
> packages that ever need to be manually updated between the monthly
> releases (e.g. xrootd, db, log, or boost).
>
> Hope that provides some insight into how we view/use/do these.
> Suggestions for how to improve/streamline are of course always welcomed!
>
> thanks,
> --FritzM.
>
>
> On 11/04/2015 10:07 PM, Becla, Jacek wrote:
>> Frossie,
>>
>> We typically care about two most recent monthly releases, no reason
>> to keep very old ones, if we could keep the last few that’d be enough.
>>
>> —
>>
>> I believe we all kind of like real tags instead of master+g{ref} for
>> the few
>> packages we care most about, like qserv, qserv_testdata, xrootd, db,
>> log,
>> dax_*. Could we possibly tag just these?
>>
>> ---
>>
>> Yes, we do switch between releases using eups, like:
>>
>> eups distrib install —tag 2015_09 qserv_distrib
>>
>> I think that is important to us!
>>
>> ---
>>
>> Jenkins testing does not help us for final validation because we want
>> to run integration tests, compilation is not enough. Hopefully that will
>> get better with container-based technology.
>>
>> —
>>
>> Same tags for both (dax_ and qserv_) would be fine.
>>
>> Thanks
>> Jacek
>>
>>
>>
>>> On Nov 4, 2015, at 8:57 PM, Frossie Economou <[log in to unmask]> wrote:
>>>
>>> Okay so for those who care about the release process:
>>>
>>> These are the points where the documented QServ process diverges
>>> with where I am going with the stack release process for engineering
>>> releases (dailies/weeklies/monthlies). Now this does NOT mean the
>>> QServ process has to change, but obviously the more similar they are
>>> the easier it is, and the more serendipitous improvements will
>>> happen. So I’m listing them here so that you folks can tell me what
>>> is what for a reason, or for no particular signing.
>>>
>>> If someone can get me an answer to #4 I can do the release « new
>>> style » otherwise I’ll just follow the script as is for this one.
>>>
>>> 1- Cryptographic signing. Jacek covered that on hitchat, it doesn’t
>>> seem like a hard requirement. My process uses the Github API to make
>>> the tags, so at least there is some protection against spoofing. But
>>> since there’s no real intent for auth, looks like I can that this
>>> the same as the stack weeklies etc.
>>>
>>> 2- The qserv-install.sh - whatever this is meant to catch, we should
>>> roll it in as a Jenkins test or something - by the time a release is
>>> getting tagged, we should have faith it is a good one.
>>>
>>> 3- This is the big one. So for the weeklies I follow a different
>>> process from the « public » 6-monthly releases. The public releases
>>> do get a build from the tag ref and so do end up with an eups
>>> package version like v11.0. For the weeklies I don’t do that; I
>>> apply *git* tags but the eups package version remains master+g{ref}
>>> or whatever. There are two reasons for this: (a) it leaves me the
>>> freedom to delete a git tag to avoid excess tag noise on the repos
>>> (e.g. who cares about the weekly from a year ago?) and (b) it is
>>> part of weaning the release process away from eups. Now the git tag
>>> exists, so one can ALWAYS submit this as a ref to the CI system and
>>> get a build reproducing that particular release; you just can’t do
>>> it solely from within eups. So if you guys really do switch between
>>> monthlies using eups you should tell me to treat these as official
>>> releases. If you’re happy as long as you have a gitref tagged
>>> qserv_2015_10 because you can always build from that, I can treat
>>> your monthlies like my weeklies, which simplifies things.
>>>
>>> 4- Git tags applied: I have a heuristic based on Github team names -
>>> e.g., repos belonging to Data Management team in the LSST org get
>>> the stack tags. That way, if someone adds a repo I don’t have to
>>> care - the tagger will magically find it. So right now there is a
>>> Database team that has both qserv and dax_ repos in it. The question
>>> is, do you want the same git tag for both (in which case do you want
>>> qserv.2015.10 or db.2015.20) or do you want different tags for dax,
>>> in which case we should make a new team and stuff the dax repos in it?
>>>
>>> 5- Docs. So you guys have a sphinx site. Fab, that’ll make it super
>>> easy to drop it into the new doc system that hosts sphinx on
>>> readthedocs - you can see a small example here
>>> http://sqr-001.lsst.codes/en/master/ - the idea is to continuously
>>> deploy the docs so the release step should be very lightweight and
>>> doable via API calls. (My holy grail is to do a release with no
>>> local checkout involved).
>>>
>>> --
>>> Frossie Economou
>>> Science Quality and Reliability Engineering (SQuaRE)
>>> Large Synoptic Survey Telescope
>>>
>>> ########################################################################
>>>
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the QSERV-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the QSERV-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the QSERV-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|
