See <https://st-ci.cern.ch/job/xrootd-rhel/architecture=i386,platform=fc-22/280/changes> Changes: [abh] [Posix] Stage 1: Implement async cache interface. [abh] Add Stat() and Update() to the Cache2 objects. [mtadel] Replace Prefetch class with File class to do async reads. [mtadel] Remove ReadInBlocks(). [mtadel] Compile success. [mtadel] Implement write queue. [mtadel] Fixes in destruction. [mtadel] Change variable names. [mtadel] Cleanup.e [mtadel] Global limit of RAM resources. [mtadel] Use same lock for RAM and disk download status. Add Prefetch function(). [mtadel] Add bit array for prefetch status. [mtadel] Add prefetch option in configuration. [mtadel] Implement prefetch score. [mtadel] Implement prefetch. [mtadel] Cache prefetch score. [mtadel] Remove oboslete member. [mtadel] Move base IO class to separate file. [mtadel] Move base IO class to separate file. [mtadel] Add missing initialization. [mtadel] Add dump log messages. [mtadel] Don't increase ref count to prefetching blocks. [mtadel] Add missing initialization. [mtadel] Remove prefetching block from map in ProcessBlockResponse if it does not [mtadel] Removed failed prefetched block in ProcessRequest(). [mtadel] return immediately if read from disk failes in Read(). [mtadel] Write statistics in info file. [mtadel] Save score as float. [mtadel] Print info message about prefetch score at the end of destructor. [mtadel] Reserver ram resources in prefetching. [mtadel] Fix class name in log messages. [mtadel] Check write queue size. [mtadel] Change log messages. [mtadel] Fix spelling in log. [mtadel] Add log message in Cache::HaveFreeWritingSlots. [mtadel] Changes in log messages. [mtadel] Unregister file from cache prefetch when it is complete. [mtadel] Temporary change -- Randomize prefetch file list. [mtadel] Keep failed blocks in map. [mtadel] Fix valgrind error in constructor. [mtadel] Handle case where File::Read() does not wait blocks to be processed. [mtadel] Handle case where File::Read() does not wait blocks to be processed. [mtadel] Add util function to release block. [mtadel] Report client error codes in log. [mtadel] Add path info in File::InitiateClose log. [mtadel] Add asserts. [mtadel] Check free writing slots before requesting ram block in File::Read(). [mtadel] Increase severity of message if bytes were missed. [mtadel] Fix double read size from direct reads. [mtadel] Fix error in de-registering blocks from write queue at the distruction [mtadel] Fix evaluation of prefetch score. [mtadel] Handle failed async requests. [mtadel] Fix error from the previous commit. [mtadel] Set errno in RequestBlock() and RequestBlocksDirect() if failed. [mtadel] Wait for signal in Prefetch() function when there is no open file [mtadel] in Prefetch wait 5ms if ram limit has been reached. [mtadel] Optimize random file access in GetNextFileToPrefetch(). [mtadel] Optimize detection of file available for prefetching. [mtadel] DeRegister/Register files in case when they are ready for prefetching. [mtadel] First implementation of File::ReadV(). [mtadel] Fix compilation errror. [mtadel] Import vread code from async-vread-baq branch. [mtadel] Add debug print. [mtadel] Parse prefetch entry correctly. [mtadel] Fix bug in VReadProcessBlocks -- removing vector while iterating. [mtadel] Fix iteration in VReadFromDisk. [mtadel] Fix offset in disk reads [mtadel] Review open issues, make it compile. [mtadel] Whitespace changes. [mtadel] pfc.prefetch now takes int argument: max number of blocks to prefetch [amraktadel] Merge Factory class into Cache class. This makes OucCache::Create() [amraktadel] Add changed files missing in the previous commit c00a774. [abh] Implement async Posix I/O and deferred file open. [amraktadel] Moving to XrdOucCache2. [amraktadel] Fix problem with XrdOucCache2 instantiation in XrdPssSys::getCache(). [amraktadel] Set version info relative to XrdOucGetCache2. [amraktadel] Add asynchronous read. [amraktadel] Fix white space created in the previous commit. [abh] Remove unused variable. [abh] Corrections for partially implemented deferred open. [abh] More fixes to defered open. [abh] Implement Fstat() at the cache level. [amraktadel] Pass input source as a pointer instead of reference. [amraktadel] Put file size in the info file. [amraktadel] Implement Fstat. [amraktadel] Implement virtual ouc-cache Prepare(). [amraktadel] Use file size in info file. [amraktadel] Clear block map repetitively in InitiateClose(). [amraktadel] Fix restoration of file size (bug from the previous commit). [amraktadel] Cache local stat. [amraktadel] InitateClose(): Don't remove this blocks from global write queue. [amraktadel] Add missing initialization. [amraktadel] Use cached values of file path and file size instead of access them [amraktadel] Add missing initialization. [amraktadel] Initiate buffer for prefetch score statistics. [abh] Corectly spell guard! [amraktadel] Sync disk every 100 new blocks. [amraktadel] Remove obosolete parameter cachedir. [amraktadel] Reject incompatible versions. [amraktadel] Remove obosolete function argument. [amraktadel] Remove obosolete function argument. [amraktadel] Aligh code. [amraktadel] Merge configuration parameters prefetch and prefetch_max_blocks. [amraktadel] Define diskusage as an absolute value (GB or TB). This is merge from [amraktadel] Set max RAM usage in absolute units. [amraktadel] Fix compilation error. [amraktadel] Rename member for prefetch list. [amraktadel] Rename write queue member. [amraktadel] Rename parameters for prefetch and available ram. [amraktadel] Handle case where new file is attached while a CacheIO object with the [amraktadel] Fix error in Info initialization. [amraktadel] Update. [amraktadel] Use 'prefetch' string to name a parameter for prefetch level. [amraktadel] Remove obsolete paramter. [amraktadel] Run prefetch thread after parsing configuration. [amraktadel] Make RAM usage a required configuration parameter. [amraktadel] Fix possible crash when scheduler calls disk sync at the time of [amraktadel] Fix logical error. [mtadel] Make sure info about blocks since the last sync is also written out. [amraktadel] Improve printout of configuration status. [amraktadel] Fix bug in selecting file to prefetch. [amraktadel] Add check if reading local stat from cinfo failed. [amraktadel] Implement virtual Cache::Stat(). [amraktadel] Properly set disk read and write offsets in hdfs healing mode (files [amraktadel] In hdfs healing mode forward stat query to origin. [amraktadel] In hdfs healing mode forward stat query to origin. [amraktadel] Include file offset (hdfs healing mode). [amraktadel] FileBlockMode: recycle File object during delayed destruction. [amraktadel] Introduce trace macros. [amraktadel] Parse trace level. [amraktadel] Replace XrdCl::Log with XrdOucTrace [amraktadel] Remove abosolete argument in Info constructor [amraktadel] Pass blocksize to File in newBlockFile(). [amraktadel] Document trace option. [amraktadel] Protect from invalid reads. [amraktadel] Default debug level is warning. [amraktadel] Print severity with message. [amraktadel] Add log message in Read() when 0 bytes read. [amraktadel] Add getter for file size. [amraktadel] In Open() assert info file exists and its size is nonzero. [amraktadel] Init stat when File is recycled. [amraktadel] Improve readability of relinquish file process. [amraktadel] Rename File::initateClose() to ioActive() [amraktadel] Rename Info::CheckComplete() to Info::UpdateDownloadCompleteStaus(). [amraktadel] Remove m_stopped varaible. Add stopped prefetch state. [amraktadel] Remove Mark/UnMark prefech functions. [amraktadel] Delete disk syncer in destructor. [amraktadel] Set prefetch state when file is recycled by some other IO. [amraktadel] Complete lock/unlock in Open (change relate to previous commit) [amraktadel] Propagate input change from IO to File. This problem solves warning [amraktadel] protect against invalid reads: check IO has relinquished File. [amraktadel] Remove debug print. [amraktadel] Add XrdFileCacheIO.cc source [amraktadel] In destructor properly check if Info struct is created. [amraktadel] Check XrdOss::Create() return value in File::Open(). [amraktadel] Replace assert with an error message. [amraktadel] Add error message in initCachedStat, if open of info file fails [amraktadel] Add warning with errno if can't open info file. [abh] Add Location() method to CacheIO2 to get file location. [amraktadel] Fix in Detach for hdfs mode. [amraktadel] Fix compilation error from the previous commit [amraktadel] Fix index offset i Sync(). This applies only to hdfs healing mode. [amraktadel] Change in severity of read messages. [mtadel] Add location printout into Attach() and Update(). [mtadel] Remove unnecessary cast. [mtadel] Improve printing of Location() in Info trace. [amraktadel] Remove obsolete method. [amraktadel] Remove trace mesage in Detach becuse similar message is printed already [amraktadel] Store access statistics in bytes instead of number of blocks [mtadel] Bugfix in hdfs healing mode. [amraktadel] Implement defered open for hdfs healing mode. [mtadel] Properly handle open and read failures of cinfo files. [mtadel] Beautify block status printout. [mtadel] Include filename in traces on most frequent paths. [mtadel] Make sure file-block mode also works, needs further fix. [mtadel] Whitespace. [abh] [Posix] Stage 1: Implement async cache interface. [abh] Add Stat() and Update() to the Cache2 objects. [abh] Replace Prefetch class with File class to do async reads. [abh] Remove ReadInBlocks(). [abh] Compile success. [abh] Implement write queue. [abh] Fixes in destruction. [abh] Change variable names. [abh] Cleanup.e [abh] Global limit of RAM resources. [abh] Use same lock for RAM and disk download status. Add Prefetch function(). [abh] Add bit array for prefetch status. [abh] Add prefetch option in configuration. [abh] Implement prefetch score. [abh] Implement prefetch. [abh] Cache prefetch score. [abh] Remove oboslete member. [abh] Move base IO class to separate file. [abh] Move base IO class to separate file. [abh] Add missing initialization. [abh] Add dump log messages. [abh] Don't increase ref count to prefetching blocks. [abh] Add missing initialization. [abh] Remove prefetching block from map in ProcessBlockResponse if it does not [abh] Removed failed prefetched block in ProcessRequest(). [abh] return immediately if read from disk failes in Read(). [abh] Write statistics in info file. [abh] Save score as float. [abh] Print info message about prefetch score at the end of destructor. [abh] Reserver ram resources in prefetching. [abh] Fix class name in log messages. [abh] Check write queue size. [abh] Change log messages. [abh] Fix spelling in log. [abh] Add log message in Cache::HaveFreeWritingSlots. [abh] Changes in log messages. [abh] Unregister file from cache prefetch when it is complete. [abh] Temporary change -- Randomize prefetch file list. [abh] Keep failed blocks in map. [abh] Fix valgrind error in constructor. [abh] Handle case where File::Read() does not wait blocks to be processed. [abh] Handle case where File::Read() does not wait blocks to be processed. [abh] Add util function to release block. [abh] Report client error codes in log. [abh] Add path info in File::InitiateClose log. [abh] Add asserts. [abh] Check free writing slots before requesting ram block in File::Read(). [abh] Increase severity of message if bytes were missed. [abh] Fix double read size from direct reads. [abh] Fix error in de-registering blocks from write queue at the distruction [abh] Fix evaluation of prefetch score. [abh] Handle failed async requests. [abh] Fix error from the previous commit. [abh] Set errno in RequestBlock() and RequestBlocksDirect() if failed. [abh] Wait for signal in Prefetch() function when there is no open file [abh] in Prefetch wait 5ms if ram limit has been reached. [abh] Optimize random file access in GetNextFileToPrefetch(). [abh] Optimize detection of file available for prefetching. [abh] DeRegister/Register files in case when they are ready for prefetching. [abh] First implementation of File::ReadV(). [abh] Fix compilation errror. [abh] Import vread code from async-vread-baq branch. [abh] Add debug print. [abh] Parse prefetch entry correctly. [abh] Fix bug in VReadProcessBlocks -- removing vector while iterating. [abh] Fix iteration in VReadFromDisk. [abh] Fix offset in disk reads [abh] Review open issues, make it compile. [abh] Whitespace changes. [abh] pfc.prefetch now takes int argument: max number of blocks to prefetch [abh] Merge Factory class into Cache class. This makes OucCache::Create() [abh] Add changed files missing in the previous commit c00a774. [abh] Implement async Posix I/O and deferred file open. [abh] Moving to XrdOucCache2. [abh] Fix problem with XrdOucCache2 instantiation in XrdPssSys::getCache(). [abh] Set version info relative to XrdOucGetCache2. [abh] Add asynchronous read. [abh] Fix white space created in the previous commit. [abh] Remove unused variable. [abh] Corrections for partially implemented deferred open. [abh] More fixes to defered open. [abh] Implement Fstat() at the cache level. [abh] Pass input source as a pointer instead of reference. [abh] Put file size in the info file. [abh] Implement Fstat. [abh] Implement virtual ouc-cache Prepare(). [abh] Use file size in info file. [abh] Clear block map repetitively in InitiateClose(). [abh] Fix restoration of file size (bug from the previous commit). [abh] Cache local stat. [abh] InitateClose(): Don't remove this blocks from global write queue. [abh] Add missing initialization. [abh] Use cached values of file path and file size instead of access them [abh] Add missing initialization. [abh] Initiate buffer for prefetch score statistics. [abh] Corectly spell guard! [abh] Sync disk every 100 new blocks. [abh] Remove obosolete parameter cachedir. [abh] Reject incompatible versions. [abh] Remove obosolete function argument. [abh] Remove obosolete function argument. [abh] Aligh code. [abh] Merge configuration parameters prefetch and prefetch_max_blocks. [abh] Define diskusage as an absolute value (GB or TB). This is merge from [abh] Set max RAM usage in absolute units. [abh] Fix compilation error. [abh] Rename member for prefetch list. [abh] Rename write queue member. [abh] Rename parameters for prefetch and available ram. [abh] Handle case where new file is attached while a CacheIO object with the [abh] Fix error in Info initialization. [abh] Update. [abh] Use 'prefetch' string to name a parameter for prefetch level. [abh] Remove obsolete paramter. [abh] Run prefetch thread after parsing configuration. [abh] Make RAM usage a required configuration parameter. [abh] Fix possible crash when scheduler calls disk sync at the time of [abh] Fix logical error. [abh] Make sure info about blocks since the last sync is also written out. [abh] Improve printout of configuration status. [abh] Fix bug in selecting file to prefetch. [abh] Add check if reading local stat from cinfo failed. [abh] Implement virtual Cache::Stat(). [abh] Properly set disk read and write offsets in hdfs healing mode (files [abh] In hdfs healing mode forward stat query to origin. [abh] In hdfs healing mode forward stat query to origin. [abh] Include file offset (hdfs healing mode). [abh] FileBlockMode: recycle File object during delayed destruction. [abh] Introduce trace macros. [abh] Parse trace level. [abh] Replace XrdCl::Log with XrdOucTrace [abh] Remove abosolete argument in Info constructor [abh] Pass blocksize to File in newBlockFile(). [abh] Document trace option. [abh] Protect from invalid reads. [abh] Default debug level is warning. [abh] Print severity with message. [abh] Add log message in Read() when 0 bytes read. [abh] Add getter for file size. [abh] In Open() assert info file exists and its size is nonzero. [abh] Init stat when File is recycled. [abh] Improve readability of relinquish file process. [abh] Rename File::initateClose() to ioActive() [abh] Rename Info::CheckComplete() to Info::UpdateDownloadCompleteStaus(). [abh] Remove m_stopped varaible. Add stopped prefetch state. [abh] Remove Mark/UnMark prefech functions. [abh] Delete disk syncer in destructor. [abh] Set prefetch state when file is recycled by some other IO. [abh] Complete lock/unlock in Open (change relate to previous commit) [abh] Propagate input change from IO to File. This problem solves warning [abh] protect against invalid reads: check IO has relinquished File. [abh] Remove debug print. [abh] Add XrdFileCacheIO.cc source [abh] In destructor properly check if Info struct is created. [abh] Check XrdOss::Create() return value in File::Open(). [abh] Replace assert with an error message. [abh] Add error message in initCachedStat, if open of info file fails [abh] Add warning with errno if can't open info file. [abh] Add Location() method to CacheIO2 to get file location. [abh] Fix in Detach for hdfs mode. [abh] Fix compilation error from the previous commit [abh] Fix index offset i Sync(). This applies only to hdfs healing mode. [abh] Change in severity of read messages. [abh] Add location printout into Attach() and Update(). [abh] Remove unnecessary cast. [abh] Improve printing of Location() in Info trace. [abh] Remove obsolete method. [abh] Remove trace mesage in Detach becuse similar message is printed already [abh] Store access statistics in bytes instead of number of blocks [abh] Bugfix in hdfs healing mode. [abh] Implement defered open for hdfs healing mode. [amraktadel] Print access statistics in top *cinfo class. [amraktadel] Include access statistics status in return value of Read() function. [amraktadel] Always open info file in RW mode. [amraktadel] move code to print access stat to separate function. [amraktadel] Don't init download status buffer when bufferSize is not initated. This [amraktadel] Write access count in info file, skip writing downloaded bits [amraktadel] Fix return value of GetSizeInBytes() when buffer size is not [amraktadel] Negate info version to mark info file is top level file in hdfs healing. [mtadel] IO::Update() call location on m_io, not on myself. [amraktadel] Fix memory leak in Info file in case read is not valid. [amraktadel] Print access statistics for top level info file in file block mode. [amraktadel] Fix unloc of unlocked mutex in destructor. Write info header in [amraktadel] Another correction of file sync during destruction. [amraktadel] Mark status in sync when claim non-ioActive state to prevent scheduling [mtadel] Small fix for hdfs fallback mode. [mtadel] Rename FStat -> Fstat in .cc file, too :) [mtadel] Purge data and cinfo file when cinfo file can not be correctly read. [mtadel] Handle spurious condition wakeups in two cases. [mtadel] Do not call async read with File::downloadCond locked. [mtadel] Advice oss about expected file size via oss.asize. [amraktadel] rename varaible m_stateCond to m_prefetchStateCond [amraktadel] Rename m_buff_fetched-> m_buff_written,and rename m_buff_write_called - [amraktadel] Use m_downloadCond for sync status. [amraktadel] Fix bug in pStatus, pFileState error. [amraktadel] Remove m_prefetchStateCond [amraktadel] Protect against spurious wakeups in SyncResponseHandler. [amraktadel] Fix locking of prfetch score. [mtadel] Use result as error code in response handlers, not errno. [amraktadel] Move members that are written/read in a physical info file to a separate [amraktadel] Include cksum in info file. [amraktadel] Have verbose print for version 0. [amraktadel] Return with an error whrn checksum does not match. [mtadel] Make oss spaces used for data and metadata files configurable. [mtadel] Set errno to be positive (stored as negative in respnse handlers). Add [amraktadel] Reuse cksum calculator. [amraktadel] Store active files in a map instead of vector. Don't need to keep IO <-> [amraktadel] Check if file is active before deleteing it. [amraktadel] Remove file from active map when ioActive returns false. [amraktadel] Make element remove in Cache::Detach() more clear [amraktadel] Move sync opearations from destructor to ioActive. [amraktadel] Add creation time and attach time in the access statistics. Limit number [amraktadel] V1 and V2 compatibility. [amraktadel] Beautify code with Uncrustify. [amraktadel] Remove obsolete method [amraktadel] Fixes in prefetch statistics. [amraktadel] Review AMT comments [amraktadel] Indent to 3 spaces. [amraktadel] member alignment. [amraktadel] Code formatting. [amraktadel] Review MT comments. [alja] Fix compilation warning on SL6 [alja] Remove debug printout. [alja] Fixes V1 and V2 compatibility. [alja] Allow multiple paths. [alja] Allow multiple paths. [amraktadel] Write cinfo at close time. [amraktadel] Additional checks for prefetch statistics. [amraktadel] Change max number of accesses from 3 to 20. [amraktadel] Improve decription of entries. [amraktadel] Revert "Protect against spurious wakeups in SyncResponseHandler." ------------------------------------------ Started by upstream project "xrootd-rhel" build number 280 originally caused by: Started by an SCM change Building remotely on st-buildnode04 (rhel-node) in workspace <https://st-ci.cern.ch/job/xrootd-rhel/architecture=i386,platform=fc-22/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url git://github.com/xrootd/xrootd.git # timeout=10 Fetching upstream changes from git://github.com/xrootd/xrootd.git > git --version # timeout=10 > git -c core.askpass=true fetch --tags --progress git://github.com/xrootd/xrootd.git +refs/heads/*:refs/remotes/origin/* Checking out Revision 93c42be1c09ee54777ecc9c5f627ed2e3266f8a0 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 93c42be1c09ee54777ecc9c5f627ed2e3266f8a0 > git rev-list 4245eef1a2130fcce84c2d4ba0b4a36b3102c741 # timeout=10 Set build name. New build name is '#280: origin/master' [fc-22] $ /bin/sh -xe /tmp/hudson6768777126886651572.sh + git checkout origin/master HEAD is now at 93c42be... Merge pull request #429 from alja/pfc-V2-bare + git clean -dffx Removing build/ Removing packaging/dss-ci-mock/ Removing packaging/xrootd-20161110.7bfa284-1.el7.cern.src.rpm + cd packaging/ + ./makesrpm.sh [i] Working on: ../ [i] Storing the output to: . [i] Working with version: v20170111-93c42be [i] RPM compliant version: 20170111.93c42be-1 [i] Working in: /tmp/xrootd.srpm.pyXNnSiLax [i] Creating the source RPM... [i] Done. + git clone ssh:[log in to unmask]:7999/dss/dss-ci-mock.git Cloning into 'dss-ci-mock'... ECDSA host key for IP address '188.184.65.204' not in list of known hosts. ++ basename origin/master + mock --yum --init --uniqueext=xrootd-master -r dss-ci-mock/xrootd-templates/fc-22-i386.cfg --rebuild xrootd-20170111.93c42be-1.el7.cern.src.rpm --resultdir ../build/ '--define=_with_tests 1' -D 'dist .el2' INFO: mock.py version 1.2.17 starting (python version = 2.7.5)... Start: init plugins INFO: tmpfs initialized INFO: selinux enabled Finish: init plugins Start: run INFO: Start(xrootd-20170111.93c42be-1.el7.cern.src.rpm) Config(fc-22-i386) Start: clean chroot Finish: clean chroot Start: chroot init INFO: mounting tmpfs at /var/lib/mock/fc-22-i386-xrootd-master/root. INFO: calling preinit hooks INFO: enabled root cache INFO: dss-ci-mock/xrootd-templates/fc-22-i386.cfg newer than root cache; cache will be rebuilt INFO: enabled yum cache Start: cleaning yum metadata Finish: cleaning yum metadata Mock Version: 1.2.17 INFO: Mock Version: 1.2.17 Start: yum install Finish: yum install Start: creating cache Finish: creating cache Finish: chroot init INFO: Installed packages: Start: build phase for xrootd-20170111.93c42be-1.el7.cern.src.rpm Start: build setup for xrootd-20170111.93c42be-1.el7.cern.src.rpm Finish: build setup for xrootd-20170111.93c42be-1.el7.cern.src.rpm Start: rpmbuild xrootd-20170111.93c42be-1.el7.cern.src.rpm ERROR: Exception(xrootd-20170111.93c42be-1.el7.cern.src.rpm) Config(fc-22-i386) 4 minutes 16 seconds INFO: Results and/or logs in: ../build/ INFO: Cleaning up build root ('cleanup_on_failure=True') Start: clean chroot INFO: unmounting tmpfs. Finish: clean chroot ERROR: Command failed. See logs for output. # bash --login -c /usr/bin/rpmbuild -bb --target i686 --nodeps /builddir/build/SPECS/xrootd.spec Build step 'Execute shell' marked build as failure Set build name. New build name is '#280: origin/master' Variable with name 'BUILD_DISPLAY_NAME' already exists, current value: '#280: origin/master', new value: '#280: origin/master' Archiving artifacts ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1