LISTSERV 16.5 - XROOTD-DEV Archives

The issue here is with Davix. Using xrdcp in the way you do requires that
Davix first write out the data into a local file before transferring the
local file to EOS. As you see, your local file system runs our of space to
hold a copy of a 4GB file (this file, by default, is written to /tmp which
simply does nor have enough space). After that, everything goes downhill.
The SEGV is clearly an issue but with atleast a stack trace with statement
numbers there is not much we can do.

The solution here, since you are transferring an actual file into EOS, is
to use a tool like curl or some other http based copy program.

Andy

On Thu, 2 Feb 2023, Jaroslav Guenther wrote:

>
> Reproduction of the problem transferring over http to EOS storage.
> Preparation checks:
> ```
> ***@***.*** ~]# ll /var/tmp/4G.file
> -rw-r--r--. 1 root root 4294967296 Jan 25 16:53 /var/tmp/4G.file
>
>
> ***@***.*** ~]# rpm -ql xrdcl-http
> /etc/xrootd/client.plugins.d/xrdcl-http-plugin.conf
> /usr/lib64/libXrdClHttp-5.so
> ***@***.*** ~]# rpm -qa xrdcl-http
> xrdcl-http-5.5.1-1.el7.x86_64
>
> ***@***.*** ~]# which xrdcp
> /usr/bin/xrdcp
> ***@***.*** ~]# xrdcp --version
> v5.5.1
>
> ***@***.*** ~]# ll /tmp/x509up_u12354
> -rw-------. 1 root root 6325 Feb 2 14:12 /tmp/x509up_u12354
>
> ***@***.*** ~]# export X509_USER_PROXY=/tmp/x509up_u12354
>
> ***@***.*** (mgm:master mq:slave) ~]$ eos ls -l /eos/ams02/opstest/guenther/4G.file
> Unable to stat /eos/ams02/opstest/guenther/4G.file; No such file or directory (errc=2) (No such file or directory)
> ```
>
> The execution of the transfer command:
> ```
> ***@***.*** ~]# time XrdSecPROTOCOL=gsi,unix xrdcp -f --debug 2 --allow-http /var/tmp/4G.file "https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.ruid=12354&eos.rgid=2763"
> [2023-02-02 14:13:03.709247 +0100][Debug ][Utility ] Env: overriding entry: tlsnodata=0 with 1
> [2023-02-02 14:13:03.709485 +0100][Debug ][Utility ] CopyProcess: 2 jobs to prepare
> [2023-02-02 14:13:03.709556 +0100][Debug ][Utility ] Creating a classic copy job, from file://localhost/var/tmp/4G.file to https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.rgid=2763&eos.ruid=12354
> [2023-02-02 14:13:03.709579 +0100][Debug ][Utility ] Monitor library name not set. No monitoring
> [2023-02-02 14:13:03.709652 +0100][Debug ][Utility ] Opening file://localhost/var/tmp/4G.file for reading
> [2023-02-02 14:13:03.709741 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] Sending an open command
> [2023-02-02 14:13:03.709839 +0100][Debug ][Utility ] Env: trying to get a non-existent string entry: pollerpreference
> [2023-02-02 14:13:03.709849 +0100][Debug ][Poller ] Available pollers: built-in
> [2023-02-02 14:13:03.709854 +0100][Debug ][Poller ] Attempting to create a poller according to preference: built-in
> [2023-02-02 14:13:03.709859 +0100][Debug ][Poller ] Creating poller: built-in
> [2023-02-02 14:13:03.709870 +0100][Debug ][Poller ] Creating and starting the built-in poller...
> [2023-02-02 14:13:03.710119 +0100][Debug ][Poller ] Using 1 poller threads
> [2023-02-02 14:13:03.710136 +0100][Debug ][TaskMgr ] Starting the task manager...
> [2023-02-02 14:13:03.710175 +0100][Debug ][TaskMgr ] Task manager started
> [2023-02-02 14:13:03.710181 +0100][Debug ][JobMgr ] Starting the job manager...
> [2023-02-02 14:13:03.710255 +0100][Debug ][JobMgr ] Job manager started, 3 workers
> [2023-02-02 14:13:03.710264 +0100][Debug ][TaskMgr ] Registering task: "FileTimer task" to be run at: [2023-02-02 14:13:03 +0100]
> [2023-02-02 14:13:03.710375 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] Open has returned with status [SUCCESS]
> [2023-02-02 14:13:03.710400 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] successfully opened at localhost, handle: 0x7, session id: 1
> [2023-02-02 14:13:03.710483 +0100][Debug ][Utility ] Opening https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.rgid=2763&eos.ruid=12354&oss.asize=4294967296 for writing
> [2023-02-02 14:13:03.710536 +0100][Debug ][XrdClHttp ] HttpFilePlugin constructed.
> [2023-02-02 14:13:03.747783 +0100][Debug ][XrdClHttp ] Open: URL: https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.rgid=2763&eos.ruid=12354&oss.asize=4294967296, XRootD flags: 34, POSIX flags: 578
> [2023-02-02 14:13:03.766476 +0100][Debug ][XrdClHttp ] Opened: https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.rgid=2763&eos.ruid=12354&oss.asize=4294967296
> [2023-02-02 14:13:03.766583 +0100][Error ][Utility ] The given URL is empty
> [2023-02-02 14:13:03.766661 +0100][Error ][PostMaster ] Unable to get transport handler for protocol
> [2023-02-02 14:13:03.784673 +0100][Debug ][XrdClHttp ] Stat-ed URL: https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.rgid=2763&eos.ruid=12354&oss.asize=4294967296
> [2023-02-02 14:13:03.784786 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] Sending a read command for handle 0x7 to localhost
> [2023-02-02 14:13:03.784917 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] Sending a read command for handle 0x7 to localhost
> [2023-02-02 14:13:03.784994 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] Sending a read command for handle 0x7 to localhost
> [2023-02-02 14:13:03.785017 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] Sending a read command for handle 0x7 to localhost
> [2023-02-02 14:13:03.811201 +0100][Debug ][XrdClHttp ] Wrote 8388608 bytes, at offset 0, to URL: https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.rgid=2763&eos.ruid=12354&oss.asize=4294967296
> [8MB/4GB][ 0%][> ][8MB/s] [2023-02-02 14:13:03.811375 +0100][Debug ][File ] ***@***.***://localhost/var/tmp/4G.file?xrdcl.requuid=8f7a6bcb-e1bf-4700-ba40-c281f63f5648] Sending a read command for handle 0x7 to localhost
> [2023-02-02 14:13:03.836593 +0100][Error ][XrdClHttp ] Could not write URL: https://eosams.cern.ch:443//eos/ams02/opstest/guenther/4G.file?eos.rgid=2763&eos.ruid=12354&oss.asize=4294967296, error: [ERROR] Internal error: not a directory: Impossible to write to fdNo space left on device
> [2023-02-02 14:13:03.838100 +0100][Debug ][XrdClHttp ] Closing davix fd: 20923904
> Segmentation fault
>
> real 0m0.317s
> user 0m0.078s
> sys 0m0.139s
> ```
>
> Result is an incomplete file present on the storage:
>
> ```
> ***@***.*** (mgm:master mq:slave) ~]$ eos ls -l /eos/ams02/opstest/guenther/4G.file
> -rw-r--r-- 2 guenther it 8454144 Feb 2 14:17 4G.file
> ``
>
> I do have space in that path on EOS:
> ```
> ***@***.*** (mgm:master mq:slave) ~]$ eos quota ls /eos/ams02/opstest/guenther/
>
> ??> Quota Node: /eos/ams02/opstest/guenther/
> ???????????????????????????????????????????????????????????????????????????????????????????????????????????????
> ?user ?used bytes?logi bytes?used files?aval bytes?aval logib?aval files? filled[%]?vol-status?ino-status?
> ???????????????????????????????????????????????????????????????????????????????????????????????????????????????
> guenther 33.82 MB 16.91 MB 4 2.00 TB 1.00 TB 0 0.00 % ok ignored
> ```
>
> --
> Reply to this email directly or view it on GitHub:
> https://github.com/xrootd/xrootd/issues/1891
> You are receiving this because you are subscribed to this thread.
>
> Message ID: ***@***.***>

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.

[ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/xrootd/xrootd/issues/1891#issuecomment-1413975273", "url": "https://github.com/xrootd/xrootd/issues/1891#issuecomment-1413975273", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1