URL:
<http://savannah.cern.ch/support/?136834>
Summary: [REQ] Feature request: dumping access log
Project: XROOTD
Submitted by: davidlt
Submitted on: 2013-04-02 16:30
Category: None
Priority: 5 - Normal
Severity: 3 - Normal
Status: None
Privacy: Public
Assigned to: None
Originator Email:
Open/Closed: Open
Discussion Lock: Any
Operating System: None
_______________________________________________________
Details:
xrootd can be used as a library, or binaries (e.g., xrdcp) could be used to
copy/access files.
In CMS we had permanent datasets deletion campaign and I needed to figure out
every single file used from EOS in CMSSW Integration Build (IB). That include
<200 tests running, which can have hardcoded files. In addition to that a
number of unit tests and other kind of tests. Some tests script could use
xrdcp, some could use CMSSW, which wraps around xrootd.
I would like to have a feature implemented in xrootd, that allows from
multiple instances (it could be xrdcp or xrootd library) dump information
about accessed files to a single log file.
After reading standards I see that concurrent writes to a file are safe if
written chunk size is less than PIPE_BUF, which is mostly 4K on modern
systems.
Below you will find a very simply and not-production ready patch
demonstrating it.
In environment you would define:
export CMSSW_LOG_ACCESS=1
export CMSSW_LOG_ACCESS_FILE=/build/davidlt/access.log
Once these are defined, every time XrdClient is created it will write a path
to a file. I assume XrdClient is the place to put the code fragment, as CMSSW
uses it in wrapper and I assume xrdcp also use it.
A patch would need to be cleaned up and secured. A single environment
variable could (should) be used. Situations needs to be handled correctly
(what if I cannot write path, should xrootd fail? Or how it should report the
issue?). Also situations were path is longer than PIPE_BUF, maybe two writes,
but that is already a bit unsafe. No guarantees to be atomic.
An additional environment variable could be added to allow to pick a format,
e.g., status of opening file (failed, opened successfully).
I would like to use the following feature to constantly monitor accessed
files and their status (site, deprecated/valid, ownership, etc). in
integration builds.
Thanks,
david
--- a/src/XrdClient/XrdClient.cc
+++ b/src/XrdClient/XrdClient.cc
@@ -28,6 +28,7 @@ const char *XrdClientCVSID = "$Id$";
#include "XrdClient/XrdClientReadAhead.hh"
#include "XrdClient/XrdClientCallback.hh"
+#include <stdlib.h>
#include <stdio.h>
#ifndef WIN32
#include <unistd.h>
@@ -37,6 +38,7 @@ const char *XrdClientCVSID = "$Id$";
#include <sys/stat.h>
#include <fcntl.h>
#include <signal.h>
+#include <string.h>
XrdSysSemWait XrdClient::fConcOpenSem(DFLT_MAXCONCURRENTOPENS);
@@ -76,6 +78,30 @@ XrdClient::XrdClient(const char *url,
// Pick-up the latest setting of the debug level
DebugSetLevel(EnvGetLong(NAME_DEBUG));
+
+ // Dump access log
+ if (getenv("CMSSW_LOG_ACCESS") != NULL) {
+ const char * accessLogFile = getenv("CMSSW_LOG_ACCESS_FILE");
+ int accessLogFD;
+ if ((accessLogFD = open(accessLogFile, O_WRONLY | O_APPEND | O_CREAT,
S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH)) < 0) {
+ perror("open() failed");
+ }
+ if (accessLogFD > 0) {
+ // Keep one char for '\n' symbol
+ size_t logLineLength = std::char_traits<char>::length(url) + 1;
+ char * logLine = new char[logLineLength];
+ memcpy(logLine, url, logLineLength - 1);
+ logLine[logLineLength - 1] = '\n';
+ if (logLineLength < PIPE_BUF) {
+ int tt = write(accessLogFD, logLine, logLineLength);
+ fsync(accessLogFD);
+ } else {
+ write(accessLogFD, "Not safe\n", 8);
+ }
+ delete logLine;
+ close(accessLogFD);
+ }
+ }
if (!ConnectionManager)
Info(XrdClientDebug::kUSERDEBUG,
Access log output example:
root://eoscms//eos/cms/store/relval/CMSSW_6_2_0_pre2-START61_V11_g496p1/RelValTTbar/GEN-SIM-DIGI-RAW-HLTDEBUG/v1/00000/082C0F0A-137A-E211-8D1C-003048CFABFC.root?svcClass=default
root://eoscms//eos/cms/store/relval/CMSSW_6_2_0_pre2-START61_V11_g496p1/RelValMinBias/GEN-SIM/v2/00000/9E4FF022-177C-E211-BAD6-003048D37580.root?svcClass=default
root://eoscms//eos/cms/store/relval/CMSSW_6_2_0_pre2-START61_V11_g496p1/RelValMinBias/GEN-SIM/v2/00000/9E4FF022-177C-E211-BAD6-003048D37580.root?svcClass=default
root://eoscms//eos/cms/store/relval/CMSSW_6_1_0_pre6-START61_V5_FastSim/RelValSingleMuPt10/GEN-SIM-DIGI-RECO/v1/00000/5C6BFF9A-9033-E211-8BA2-003048FFD736.root?svcClass=default
root://eoscms//eos/cms/store/relval/CMSSW_6_2_0_pre2-START61_V11_g496p1/RelValMinBias/GEN-SIM/v2/00000/9E4FF022-177C-E211-BAD6-003048D37580.root?svcClass=default
root://eoscms//eos/cms/store/relval/CMSSW_6_2_0_pre2-START61_V11_g496p1/RelValMinBias/GEN-SIM/v2/00000/9E4FF022-177C-E211-BAD6-003048D37580.root?svcClass=default
root://eoscms//eos/cms/store/relval/CMSSW_6_1_0_pre6-START61_V5_FastSim/RelValTTbar/GEN-SIM-DIGI-RECO/v1/00000/00C0B960-8633-E211-AE7F-0018F3D0960E.root?svcClass=default
_______________________________________________________
Reply to this item at:
<http://savannah.cern.ch/support/?136834>
_______________________________________________
Message sent via/by LCG Savannah
http://savannah.cern.ch/
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
