We are using Rucio download client to
transfer job outputs (it is build on top of gfal2 python
interface).
https://vokac.web.cern.ch/vokac/tmp/dpmpool22.xrootd.log
https://vokac.web.cern.ch/vokac/tmp/dpmpool24.xrootd.log
this is the full job output unfortunately with minimal gfal2
logging output.
https://vokac.web.cern.ch/vokac/tmp/grid.2258136.4.out-from-2001:718:401:6e03::1:21-to-dpmpool22.txt
https://vokac.web.cern.ch/vokac/tmp/grid.2255921.7.out-from-2001:718:401:6e03::1:32-to-dpmpool24.txt
It looks like according gfal2 transfer was OK
2019-07-17 15:09:17,782 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_xrootd.so loaded with success
2019-07-17 15:09:17,782 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_rfio.so loaded with success
2019-07-17 15:09:17,783 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_gridftp.so loaded with success
2019-07-17 15:09:17,783 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_http.so loaded with success
2019-07-17 15:09:17,783 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_lfc.so loaded with success
2019-07-17 15:09:17,783 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_file.so loaded with success
2019-07-17 15:09:17,784 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_sftp.so loaded with success
2019-07-17 15:09:17,784 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_dcap.so loaded with success
2019-07-17 15:09:17,784 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_srm.so loaded with success
2019-07-17 15:09:17,793 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_xrootd.so loaded with success
2019-07-17 15:09:17,793 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_rfio.so loaded with success
2019-07-17 15:09:17,793 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_gridftp.so loaded with success
2019-07-17 15:09:17,794 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_http.so loaded with success
2019-07-17 15:09:17,794 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_lfc.so loaded with success
2019-07-17 15:09:17,794 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_file.so loaded with success
2019-07-17 15:09:17,794 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_sftp.so loaded with success
2019-07-17 15:09:17,794 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_dcap.so loaded with success
2019-07-17 15:09:17,794 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_srm.so loaded with success
2019-07-17 15:09:17,795 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_xrootd.so loaded with success
2019-07-17 15:09:17,795 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_rfio.so loaded with success
2019-07-17 15:09:17,795 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_gridftp.so loaded with success
2019-07-17 15:09:17,795 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_http.so loaded with success
2019-07-17 15:09:17,796 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_lfc.so loaded with success
2019-07-17 15:09:17,796 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_file.so loaded with success
2019-07-17 15:09:17,796 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_sftp.so loaded with success
2019-07-17 15:09:17,796 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_dcap.so loaded with success
2019-07-17 15:09:17,796 | INFO | copytool_out | gfal2 | connect | [gfal_module_load] plugin /usr/lib64/gfal2-plugins//libgfal_plugin_srm.so loaded with success
2019-07-17 15:09:17,810 | INFO | copytool_out | gfal2 | __gfal2_copy | Event triggered: BOTH GFAL2:CORE:COPY LIST:ENTER
2019-07-17 15:09:17,810 | INFO | copytool_out | gfal2 | __gfal2_copy | Event triggered: BOTH GFAL2:CORE:COPY LIST:ITEM file:///scratch/condor/dir_24031/atlas_HudmF2he/PanDA_Pilot-4419868006/log.18636890._001059.job.log.tgz.1 => root://golias100.farm.particle.cz:1094//dpm/farm.particle.cz/home/atlas/atlasdatadisk/rucio/mc15_13TeV/26/98/log.18636890._001059.job.log.tgz.1.rucio.upload
2019-07-17 15:09:17,810 | INFO | copytool_out | gfal2 | __gfal2_copy | Event triggered: BOTH GFAL2:CORE:COPY LIST:EXIT
2019-07-17 15:09:17,811 | INFO | Dummy-2 | gfal2 | (unknown function) | Event triggered: BOTH xroot TRANSFER:ENTER file://localhost///scratch/condor/dir_24031/atlas_HudmF2he/PanDA_Pilot-4419868006/log.18636890._001059.job.log.tgz.1?xrd.gsiusrpxy=/scratch/condor/dir_24031/job.9XONDm6yx7unnoBGSqYX2MjnABFKDmABFKDmSyhTDmEBFKDmqAGN3n.proxy&xrd.wantprot=gsi,unix => root://golias100.farm.particle.cz:1094///dpm/farm.particle.cz/home/atlas/atlasdatadisk/rucio/mc15_13TeV/26/98/log.18636890._001059.job.log.tgz.1.rucio.upload?xrd.gsiusrpxy=/scratch/condor/dir_24031/job.9XONDm6yx7unnoBGSqYX2MjnABFKDmABFKDmSyhTDmEBFKDmqAGN3n.proxy&x
2019-07-17 15:09:17,811 | INFO | Dummy-2 | gfal2 | (unknown function) | Event triggered: BOTH xroot TRANSFER:TYPE streamed
2019-07-17 15:11:23,914 | INFO | Dummy-2 | gfal2 | (unknown function) | Event triggered: BOTH xroot TRANSFER:EXIT Job finished, [SUCCESS]
but probably rucio tries to verify checksum (gfal-sum via python
bindings) and it fails
2019-07-17 15:15:23,774 | INFO | copytool_out | pilot.copytool.rucio | copy_out | stderr = An unknown exception occurred.
Details: Checksum not validated
because DPM is unable to calculate checksum for 128TB file within
timeout.
Petr
On 7/19/19 5:07 AM, Andrew Hanushevsky wrote: