Print

---

(Copying the email into an issue to hopefully make it easier to track. @abh3 @bbockelm @ffurano )

We are seeing unusual behavior from XRootD's HTTP interface. I'm suspecting that HTTP implementation is calling xrootd functions, then unintentionally getting behavior only intended for the xrootd protocol. I'm finding both unexpected output as well as a leak in XrdLink objects and file descriptors.

The samples below were run with xrootd-4.7.0-1.osg33.el7.x86_64 and a minimal configuration:

xrd.port 1094
all.role server
all.export /tmp
xrd.protocol http:8000 libXrdHttp.so

For a curl web request of a short file (12 bytes of "Hello world\n"), XrdHttp appears to return the content twice. From tcpdump (or sending the request with telnet):

> GET /tmp/hello.txt HTTP/1.1
> Host: xrootd-itb.unl.edu:8000
> User-Agent: curl/7.47.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 12
<
< Hello world
< Hello world

The connection remains open as expected with HTTP/1.1, but trying to submit a second request triggers the server to close the connection.

For a longer file (60kB, 10000 lines), the content is not sent twice, but 12 nulls and what looks like a kXR_stat response (inode, size, flags, and mtime fields) are appended to the reply:

> GET /tmp/test.10000L HTTP/1.1
> Host: xrootd-itb.unl.edu:8000
> User-Agent: curl/7.47.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 60000
<
< 00001
< 00002
< 00003
[snip]
< 09998
< 09999
< 10000
< ............193273593089 60000 16 1506616974.

Again, the connection remains open, but trying to submit a second request triggers the server to close the connection. Note that for the larger file, sendfile() is used to deliver the data. The change in code path might account for the different behavior.

Oddly enough, enabling request logging with "http.trace request" changes the server behavior for the larger 60kB file (but not the 12 byte one). The replies look fine (no additional kXR_stat data) and multiple requests can be submitted in the same HTTP connection. Is there a timing issue exposed by the logging overhead?

It also appears that XrdHttp sometimes holds onto XrdLink objects, particularly if the connection is not cleanly closed. This issue also appears to be timing related, with the problem showing up most easily by testing from localhost or a client on the same network. I first noticed the XrdLink counts (from xrdfs query stats l) and open file descriptors steadily increasing for the xrootd process. It turns out that Nagios check_http is particularly good at triggering the issue.

check_http --IP-address=127.0.0.1 --port=8000 --url=/tmp/hello.txt --no-body

The following python program should replicate the behavior of Nagios check_http. It does an HTTP request then immediately closes the connection. When requesting a small (~100 byte) file, it causes an XrdLink object to be left open roughly half the time.

    import socket

    for i in range(1, 50):
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.settimeout(1)

        s.connect(('127.0.0.1', 8000))

        s.send("GET /tmp/hello.txt HTTP/1.1\r\n")
        s.send("Host: localhost:8000\r\n")
        s.send("\r\n")

        resp = s.recv(8192)
        print len(resp)

        s.close()

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1