Hi All,



Since it seems like me & Jan Erik are getting involved in this
BRECO-business, I have some questions/comments.


> The situation
> -------------
>
>   o Currently, analysis-20  is being built and close  to release.  The
>     executable is expected to be about  as fast as an OBJY exe ("a bit
>     slower"). It should be capable of running on OBJY micro.
>
>   o At the  end of next week, a substantial amount  of skimmed data is
>     expected to become available (as  of now it's only 60/fb). By then
>     it might be something like 130/fb. Maybe/hopefully.
>
>   o Clare needs (or would like to use) more than just 80/fb for her
>     thesis.
>
>   o The same is more or less true for the b2ulnu analysis (and Ed's
>     thesis)
>
>   o The  projection for skimmed  MC CM2 availability is  slipping.  At
>     the Wednesday physics meeting we heard "June".
>
>
> The idea
> --------
>
>   o Produce new ntuples with minimal (=no) changes with respect to the
>     old "big" ntuples. Except: Just produce ROOT files. This allows to
>     use them for analysis immediately  and is probably the only way to
>     use RUN3/4 data for public results before the end of this year.
>
>     This is explicitly NOT a CM2 analysis, just an attempt to get more
>     data for the short time scale.

Do you plan to
  * run on "AllEvents", and re-do the full B reconstruction, or
  * run over the skim and use the persistent BRECO candidates

>   o Run on RUN4 CM2 data as more  becomes available
>
>   o Run on  MC as soon as possible. We should  test whether running on
>     SP5 OBJY is viable.

Even though that may be technically possible, it may give you the wrong answer :
Not that "CM2-converted" data is not really a conversion, it is a re-processing
starting with the release-12 mini, but using a lot of new reconstruction
algorithms. Especially for the muon-id, the extrapolation of tracks through
the IFR is very different in CM2-converted data (or MC). So

                  "SP5-Objy != CM2-converted SP5"


> The plan
> --------
>
>   o Ed will provide a set of tags to build IslBrecoilUserApp, based on
>     analysis-20. He will do basic validation, i.e. that it is running.
>     This also includes tcl (steering) files for CM2 and OBJY running.
>
>   o Urs will do a bit  more of validation, looking at all variables in
>     the "h1" tree.
>
>   o Royal  Holloway will  organize the production.  I think  this will
>     involve the following:
>
>       - backup the  current HBOOK ntuples of  Henning/Oliver to mstore
>         and/or RAL or somewhere else.
>
>       - Create tcl files  for skimmed data. In the  following I detail
>         what issues  must be considered in a  low-tech approach (based
>         on  "run",  the   run-script  with  built-in  bookkeeping  and
>         optimized  queue saturation :-)  Other possibilities  exist of
>         course,  I  just  don't   know  them.  Whoever  organizes  the
>         production is free to choose whatever works!
>
>         +  The  naming  scheme  for  the  "basename"  should  be  well
>           designed.  In the last production we had a bit of a mess and
>           it made life difficult.  .  A possible solution is something
>           like the following:
>
>              genbch-run1-.....
>              genbnu-run1-.....
>              genccb-run1-.....
>              genuds-run1-.....
>
>              cktbch-run1-.....
>              cktbnu-run1-.....
>              cktb2u-run1-.....
>
>              b2unre-run1-.....
>              b2ures-run1-.....
>              b2umix-run1-.....

What has been the strategy so far ? One run per file ?

In CM2, there are usually many runs per collection, and so far I did not manage
to squeeze out tcl files which process single runs (or a self-defined run range)
from a given collection. Does anybody know how to do this ?

Also, it seems that in it's current state, BbkDataSetTcl delivers somewhat
unordered tcl-files, in the sense that subsequent "input add" lines do
not contain collections which are subsequent in terms of run numbers.

>
>           NOTE: The  total amount of files will  likely exceed 100000.
>           (We had something like 30k for the previous production.)
>
>           NOTE: We used to have  something like 2k events per file. We
>           have  to think  whether we  want to  merge the  rootfiles to
>           reflect the merged CM2 files. Another possibility is to have
>           in the filename (in the  ..... part above) the start and end
>           events in the  merged CM2 files (see next  item if this NOTE
>           is not clear).
>
>         + The  size of  the tcl  files needs to  be optimized  for the
>           queue length. (kanga?)

You should contact the experts about that. In our PID-tuple production, the
only reasonnable queue was "bfobjy", which we were told was "illegal". However,
kanga and xlong had too few machines assigned at that time. Maybe it has changed
in the meanwhile ???


>           NOTE: I think this could mean that we cannot run one job per
>           CM2 merged skim file. This needs to be studied!!!

Very likely. For PID, it turned out that some skims were too large to be handled
within one job, even for bfobjy. I complained about that in the Bookkeeping
HN, and got the usual answer : "Yes, you are right, but once everything is
in place, the bookeeping will handle all this automatically and you do
no have to care".

>
>         + The tool of choice is probably "BbkDatasetTcl".
>
>         + I think that the tcl  files should be in a logical directory
>           structure to avoid too many files per directory
>
>              $BASE/tcl/SemiExclBreco-2004a/data
>              $BASE/tcl/SemiExclBreco-2004a/data/run1
>              $BASE/tcl/SemiExclBreco-2004a/data/run2
>              $BASE/tcl/SemiExclBreco-2004a/data/run3
>              $BASE/tcl/SemiExclBreco-2004a/data/run4
>
>              $BASE/tcl/SemiExclBreco-2004a/mc
>              $BASE/tcl/SemiExclBreco-2004a/mc/run1
>              $BASE/tcl/SemiExclBreco-2004a/mc/run1/bch
>              $BASE/tcl/SemiExclBreco-2004a/mc/run1/bnu
>              $BASE/tcl/SemiExclBreco-2004a/mc/run1/ccb
>              $BASE/tcl/SemiExclBreco-2004a/mc/run1/uds
>              $BASE/tcl/SemiExclBreco-2004a/mc/run1/sig
>              $BASE/tcl/SemiExclBreco-2004a/mc/run1/ckt
>
>       - The output root files should  be stored in a way that reflects
>         this structure:
>
>              $BASE/output/SemiExclBreco-2004a/data
>              $BASE/output/SemiExclBreco-2004a/data/run1
>              $BASE/output/SemiExclBreco-2004a/data/run2
>
>              $BASE/output/SemiExclBreco-2004a/mc/run1
>              $BASE/output/SemiExclBreco-2004a/mc/run1/bch
>
>
>       - A few notes on the directories:
>
>          + It does not  really matter  what names  we choose,  but it
>            should be  something that  is consistent and  extensible to
>            new productions, which could end up in, e.g.
>
>              $BASE/SemiExclBreco-2004b/
>
>          + We  should avoid too  many subdirectories, but  should make
>            sure  that not  too many  files  end up  in one  directory.
>            (Note: In the old production,  80/fb data and 240/fb MC, we
>            had 11000 gen B+ files in total.)
>
>          + Not all directories need to be physically below $BASE, they
>            could be symbolic links to a different disk.  But we should
>            see all from one base location.
>
>
>       - Of course, the logfiles should be stored similarly:
>
>              $BASE/log/SemiExclBreco-2004a/data
>              $BASE/log/SemiExclBreco-2004a/mc
>
>         and the corresponding subdirectories. If we use "run" this is
>         essential.
>
>       - The jobs will be run by a bunch of people, organized (and
>         tabulated) by someone. "Volunteers" so far are
>
>              Clare
>              Ed
>              Henning
>              Rolf
>              Urs
>              Oliver ('s account, at least)
>              Other GradStudents
>
>         Given this  amount of manpower, we might  actually get through
>         the  unskimmed SP5  OBJY  (700/fb!) on  a relatively(?)  short
>         timescale.

One more comment about "unskimmed" data. If you have collections which are
the output of a "Release12 -> CM2" conversion, you should be aware that
the tag-part of the data has NOT been converted. So raw "converted CM2"
data
still contains the tag-bits as they were in release-12. No re-computation
of the tag bits is done during conversion. This is why we are supposed to
use the skims. In the skims (including the "AllEvents" - Skim), the tag
bits are correct.

>   o Diskspace might  be sufficient once we delete  the old HBOOK files
>     (from the  previous production)  and then ask  for some  more when
>     it's critical and we have enough momentum.


BTW, how are people using these ntuples ? Are analyses run directly from these
ntuples, or does each analysis have its own set of "reduced ntuples" which
are extracted from these "event-store" ntuples ?

 Cheers,
             Thorsten