Hi All,



Since it seems like me & Jan Erik are getting involved in this
BRECO-business, I have some questions/comments.


> The situation
> -------------
>
> o Currently, analysis-20 is being built and close to release. The
> executable is expected to be about as fast as an OBJY exe ("a bit
> slower"). It should be capable of running on OBJY micro.
>
> o At the end of next week, a substantial amount of skimmed data is
> expected to become available (as of now it's only 60/fb). By then
> it might be something like 130/fb. Maybe/hopefully.
>
> o Clare needs (or would like to use) more than just 80/fb for her
> thesis.
>
> o The same is more or less true for the b2ulnu analysis (and Ed's
> thesis)
>
> o The projection for skimmed MC CM2 availability is slipping. At
> the Wednesday physics meeting we heard "June".
>
>
> The idea
> --------
>
> o Produce new ntuples with minimal (=no) changes with respect to the
> old "big" ntuples. Except: Just produce ROOT files. This allows to
> use them for analysis immediately and is probably the only way to
> use RUN3/4 data for public results before the end of this year.
>
> This is explicitly NOT a CM2 analysis, just an attempt to get more
> data for the short time scale.

Do you plan to
  * run on "AllEvents", and re-do the full B reconstruction, or
  * run over the skim and use the persistent BRECO candidates

> o Run on RUN4 CM2 data as more becomes available
>
> o Run on MC as soon as possible. We should test whether running on
> SP5 OBJY is viable.

Even though that may be technically possible, it may give you the wrong answer :
Not that "CM2-converted" data is not really a conversion, it is a re-processing
starting with the release-12 mini, but using a lot of new reconstruction
algorithms. Especially for the muon-id, the extrapolation of tracks through
the IFR is very different in CM2-converted data (or MC). So

                  "SP5-Objy != CM2-converted SP5"


> The plan
> --------
>
> o Ed will provide a set of tags to build IslBrecoilUserApp, based on
> analysis-20. He will do basic validation, i.e. that it is running.
> This also includes tcl (steering) files for CM2 and OBJY running.
>
> o Urs will do a bit more of validation, looking at all variables in
> the "h1" tree.
>
> o Royal Holloway will organize the production. I think this will
> involve the following:
>
> - backup the current HBOOK ntuples of Henning/Oliver to mstore
> and/or RAL or somewhere else.
>
> - Create tcl files for skimmed data. In the following I detail
> what issues must be considered in a low-tech approach (based
> on "run", the run-script with built-in bookkeeping and
> optimized queue saturation :-) Other possibilities exist of
> course, I just don't know them. Whoever organizes the
> production is free to choose whatever works!
>
> + The naming scheme for the "basename" should be well
> designed. In the last production we had a bit of a mess and
> it made life difficult. . A possible solution is something
> like the following:
>
> genbch-run1-.....
> genbnu-run1-.....
> genccb-run1-.....
> genuds-run1-.....
>
> cktbch-run1-.....
> cktbnu-run1-.....
> cktb2u-run1-.....
>
> b2unre-run1-.....
> b2ures-run1-.....
> b2umix-run1-.....

What has been the strategy so far ? One run per file ?

In CM2, there are usually many runs per collection, and so far I did not manage
to squeeze out tcl files which process single runs (or a self-defined run range)
from a given collection. Does anybody know how to do this ?

Also, it seems that in it's current state, BbkDataSetTcl delivers somewhat
unordered tcl-files, in the sense that subsequent "input add" lines do
not contain collections which are subsequent in terms of run numbers.

>
> NOTE: The total amount of files will likely exceed 100000.
> (We had something like 30k for the previous production.)
>
> NOTE: We used to have something like 2k events per file. We
> have to think whether we want to merge the rootfiles to
> reflect the merged CM2 files. Another possibility is to have
> in the filename (in the ..... part above) the start and end
> events in the merged CM2 files (see next item if this NOTE
> is not clear).
>
> + The size of the tcl files needs to be optimized for the
> queue length. (kanga?)

You should contact the experts about that. In our PID-tuple production, the
only reasonnable queue was "bfobjy", which we were told was "illegal". However,
kanga and xlong had too few machines assigned at that time. Maybe it has changed
in the meanwhile ???


> NOTE: I think this could mean that we cannot run one job per
> CM2 merged skim file. This needs to be studied!!!

Very likely. For PID, it turned out that some skims were too large to be handled
within one job, even for bfobjy. I complained about that in the Bookkeeping
HN, and got the usual answer : "Yes, you are right, but once everything is
in place, the bookeeping will handle all this automatically and you do
no have to care".

>
> + The tool of choice is probably "BbkDatasetTcl".
>
> + I think that the tcl files should be in a logical directory
> structure to avoid too many files per directory
>
> $BASE/tcl/SemiExclBreco-2004a/data
> $BASE/tcl/SemiExclBreco-2004a/data/run1
> $BASE/tcl/SemiExclBreco-2004a/data/run2
> $BASE/tcl/SemiExclBreco-2004a/data/run3
> $BASE/tcl/SemiExclBreco-2004a/data/run4
>
> $BASE/tcl/SemiExclBreco-2004a/mc
> $BASE/tcl/SemiExclBreco-2004a/mc/run1
> $BASE/tcl/SemiExclBreco-2004a/mc/run1/bch
> $BASE/tcl/SemiExclBreco-2004a/mc/run1/bnu
> $BASE/tcl/SemiExclBreco-2004a/mc/run1/ccb
> $BASE/tcl/SemiExclBreco-2004a/mc/run1/uds
> $BASE/tcl/SemiExclBreco-2004a/mc/run1/sig
> $BASE/tcl/SemiExclBreco-2004a/mc/run1/ckt
>
> - The output root files should be stored in a way that reflects
> this structure:
>
> $BASE/output/SemiExclBreco-2004a/data
> $BASE/output/SemiExclBreco-2004a/data/run1
> $BASE/output/SemiExclBreco-2004a/data/run2
>
> $BASE/output/SemiExclBreco-2004a/mc/run1
> $BASE/output/SemiExclBreco-2004a/mc/run1/bch
>
>
> - A few notes on the directories:
>
> + It does not really matter what names we choose, but it
> should be something that is consistent and extensible to
> new productions, which could end up in, e.g.
>
> $BASE/SemiExclBreco-2004b/
>
> + We should avoid too many subdirectories, but should make
> sure that not too many files end up in one directory.
> (Note: In the old production, 80/fb data and 240/fb MC, we
> had 11000 gen B+ files in total.)
>
> + Not all directories need to be physically below $BASE, they
> could be symbolic links to a different disk. But we should
> see all from one base location.
>
>
> - Of course, the logfiles should be stored similarly:
>
> $BASE/log/SemiExclBreco-2004a/data
> $BASE/log/SemiExclBreco-2004a/mc
>
> and the corresponding subdirectories. If we use "run" this is
> essential.
>
> - The jobs will be run by a bunch of people, organized (and
> tabulated) by someone. "Volunteers" so far are
>
> Clare
> Ed
> Henning
> Rolf
> Urs
> Oliver ('s account, at least)
> Other GradStudents
>
> Given this amount of manpower, we might actually get through
> the unskimmed SP5 OBJY (700/fb!) on a relatively(?) short
> timescale.

One more comment about "unskimmed" data. If you have collections which are
the output of a "Release12 -> CM2" conversion, you should be aware that
the tag-part of the data has NOT been converted. So raw "converted CM2"
data
still contains the tag-bits as they were in release-12. No re-computation
of the tag bits is done during conversion. This is why we are supposed to
use the skims. In the skims (including the "AllEvents" - Skim), the tag
bits are correct.

> o Diskspace might be sufficient once we delete the old HBOOK files
> (from the previous production) and then ask for some more when
> it's critical and we have enough momentum.


BTW, how are people using these ntuples ? Are analyses run directly from these
ntuples, or does each analysis have its own set of "reduced ntuples" which
are extracted from these "event-store" ntuples ?

 Cheers,
             Thorsten