Hi Paul and Wei,

This is a standard problem with multi-homed servers. Wei is correct, a data 
server subscribes to a redirector using whatever it's default IP address is 
that is compatible with the target host. That also means the redirector will 
use that IP address when redirecting a client to a data server, whether or 
not that client can actually reach that IP address. For instance, if the 
client has access to the public network but not the private network but the 
server used the private network to get to the redirector, the client will 
generally not be able to get to the data server. I say generally because 
actual xrootd clients determine the routing at runtime but xrootdFS does 
not. That's why xrdcp works but a mounted filesystem does not.

If you want to provide dual access that you would need to have two 
redirectors. One only lives on the public network and another only lives on 
the private network. Data servers accessible via both would use their public 
addresses to subscribe to the public redirector and private addresses to 
subscribe to the private redirector. Normally, the choice of out-going IP 
address happens automatically for simple networks. For more complicated ones 
you would need to establish static routing to make sure. So, if you only 
want to use the private network then use the redirector on the private 
network (this also implies that if you *only* want to use the private 
network then you don't need the public redirector at all).

To get a better feel for this, you can look in the cmsd log to see what IP 
address is actually being used when a data server connects to a redirector. 
I suspect that the data server is mapping the redirector hostname to the 
public IP address which forces it to use it's own public IP address when 
contacting the redirector. If so, try specifying the redirectors private IP 
address (as opposed to hostname) in the config file. Other sites that I know 
of have actually setup two DNS entries for the redirector and data servers, 
one associated with the public address and one with the private address. 
Then they use whichever one needed to implicitly map out the required 
routing.

Andy

-----Original Message----- 
From: Yang, Wei
Sent: Tuesday, April 26, 2011 10:55 PM
To: Paul T. Keener
Cc: xrootd-l
Subject: Re: XrootdFS

Hi Paul, Andy,

It is OK to point xrootdfs to a single data server. XrootdFS gets the public 
IP because that is what xrootd on the data server returns. As a xrootd 
client, XrootdFS it has no way to know the network configuration at the 
server side. You can verify this by doing

xrd private_IP locateall /scratch

I guess it will return public_IP:0. In general, xrootd server on the data 
server node return the IP address by checking the hostname against DNS 
server.

Andy, I seem to remember we have got several cases like this that a host has 
two NICs on private and public network. Is there any work around?

regards,
Wei Yang  |  [log in to unmask]  |  650-926-3338(O)




On Apr 26, 2011, at 5:46 PM, Paul T. Keener wrote:

> I am having trouble with XrootdFS. In particular, I am trying to force
> all xrootd traffic over my private network.  I can xrdcp using
> root://foo.at3f//scratch fine on a machine different from foo.  However,
> if I try to use xrootdfs to mount this filesystem, and set
> XROOTDFS_RDRURL to "root://foo.at3f:1094//scratch" (or even replace
> "foo.at3f" with the IP address), it will mount the filesystem, but if I
> try to access it, I get syslogd messages:
>
> WARNING: stat(root://<Public IP>:0//scratch/) failed (connection timeout)
>
> Looking at the network traffic, it is clear that these xrootd requests
> are going over the public network interfaces.  The timeout is then
> caused by the firewall.
>
> How do I force all communication between xrootdfs and the xrootd service
> to go over the private network?  Moreover, how is xrootdfs getting the
> public IP address of the server?
>
> Thanks.
>
> Paul T. Keener
> Department of Physics and Astronomy
> University of Pennsylvania