 |
 |
Hi Marian, I don't immediately see the problem other than perhaps a misconfiguration of the client machine where it really want to resolve the host to an IPv6 address. Note that there is only one IPv4 address for the problematic host 0: [::ffff:193.205.76.72] stormgf1.pi.infn.it So, forcing IPv4 would be the obvious solution and would indicate that there is a problem with the way resolution works on the client host. On the other hand, this may be an issue with the redirector. The addresses assigned to the redirector make no sense: allipmap xrootd-cms.infn.it 0: [::ffff:134.158.132.31] llrxrd-redir.in2p3.fr 1: [::ffff:131.154.216.11] xrootd-cms-redir-01.cr.cnaf.infn.it 2: [::ffff:193.205.76.83] xrootd-redic.pi.infn.it 3: [::ffff:90.147.66.75] xrootd.ba.infn.it 4: [2001:760:4205:216::11] xrootd-cms-redir-01.cr.cnaf.infn.it What's going on here? Andy On Fri, 26 Aug 2016, Marian Zvada wrote: > Hi Andy et al, > > at PIC we're having case where client is not working as expected. We have dual-stack host tds219.pic.es. Network interfaces information about the host is here [*]. IPv4 private is NATed while IPv6 is public (AAAA record exist) and v6 address live on fping6. > > We do following test from tsd219.pic.es: > `$ xrdcp -d 2 -f root://xrootd-cms.infn.it//store/test/xrootd/CMSSAM//store/mc/SAM/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0013/CE4D66EB-5AAE-E111-96D6-003048D37524.root /dev/null > ` > and seeing error: > ``` > [2016-08-25 17:15:57.476506 +0200][Error ][Utility ] Unable to resolve stormgf1.pi.infn.it:1094: Name or service not known > [2016-08-25 17:15:57.476526 +0200][Error ][PostMaster ] [stormgf1.pi.infn.it:1094 #0] Unable to resolve IP address for the host > [2016-08-25 17:15:57.476535 +0200][Debug ][XRootD ] [stormgf1.pi.infn.it:1094] Handling error while processing kXR_open (file: /store/test/xrootd/CMSSAM//store/mc/SAM/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0013/CE4D66EB-5AAE-E111-96D6-003048D37524.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [FATAL] Invalid address. > ... > ... > ``` > This is xrootd version of the client: > ``` > [root@tds219 ~]# xrdcp --version > v4.3.0 > ``` > However, things work when we force XRD_NETWORKSTACK=IPv4 while on other dual-stack node (v4/v6 enabled in public domain hcc-marian.unl.edu) it works just fine without specifying stack version. > > Also, we can resolve the remote host address from tds219.pic.es localy [**]. > > We're stumped at this point why xrdcp doesn't resolve it properly then and copy prematurely fails due that? Can you provide any insight what might be happening here? Would be combination of IPv4 NAT address and AAAA records with v6 enabled on the host any sort of problem? More to the confusion, Fernando tried from the node which has both interfaces enabled in public (like my private VM hcc-marian.unl.edu) and result is again negative [***]. > > Thanks, > Marian > > [*] > ``` > [flopez@tds219 ~]$ ip addr list > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > inet 127.0.0.1/8 scope host lo > inet6 ::1/128 scope host > valid_lft forever preferred_lft forever > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000 > link/ether a0:36:9f:83:64:1c brd ff:ff:ff:ff:ff:ff > inet6 2001:67c:1148:301::219/64 scope global > valid_lft forever preferred_lft forever > inet6 fe80::a236:9fff:fe83:641c/64 scope link > valid_lft forever preferred_lft forever > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000 > link/ether 00:1e:67:db:e2:6e brd ff:ff:ff:ff:ff:ff > inet 192.168.100.21/22 brd 192.168.103.255 scope global eth1 > inet6 fe80::21e:67ff:fedb:e26e/64 scope link > valid_lft forever preferred_lft forever > 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 > link/ether 00:1e:67:db:e2:6f brd ff:ff:ff:ff:ff:ff > 5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN > link/ether c2:83:6e:6e:29:d3 brd ff:ff:ff:ff:ff:ff > inet 172.17.42.1/16 scope global docker0 > inet6 fe80::c083:6eff:fe6e:29d3/64 scope link > valid_lft forever preferred_lft forever > ``` > > [**] > ``` > [flopez@tds219 ~]$ dig stormgf1.pi.infn.it > > ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.7 <<>> stormgf1.pi.infn.it > ;; global options: +cmd > ;; Got answer: > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12337 > ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3 > > ;; QUESTION SECTION: > ;stormgf1.pi.infn.it.INA > > ;; ANSWER SECTION: > stormgf1.pi.infn.it.825INA193.205.76.72 > > ;; AUTHORITY SECTION: > pi.infn.it.60758INNSserver2.infn.it. > pi.infn.it.60758INNSns1.pi.infn.it. > pi.infn.it.60758INNSdspi01.pi.infn.it. > > ;; ADDITIONAL SECTION: > server2.infn.it.36433INA131.154.1.3 > ns1.pi.infn.it.61008INA212.189.152.2 > dspi01.pi.infn.it.61008INA192.84.133.15 > > ;; Query time: 1 msec > ;; SERVER: 193.109.175.5#53(193.109.175.5) > ;; WHEN: Thu Aug 25 17:24:35 2016 > ;; MSG SIZE rcvd: 162 > ``` > > [***] > ``` > [zvada@hcc-marian ~]$ host psb01.pic.es > psb01.pic.es has address 193.109.172.187 > psb01.pic.es has IPv6 address 2001:67c:1148:202::2 > [zvada@hcc-marian ~]$ host 193.109.172.187 > 187.172.109.193.in-addr.arpa domain name pointer psb01.pic.es. > [zvada@hcc-marian ~]$ host 2001:67c:1148:202::2 > 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.2.0.8.4.1.1.c.7.6.0.1.0.0.2.ip6.arpa domain name pointer psb01.pic.es. > ``` > And here Fernando's test shows: > ``` > ... > ... > [2016-08-25 17:38:43.672734 +0200][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: xrootd-cms-redir-01.cr.cnaf.infn.it:1194" to be run at: [2016-08-25 17:38:58 +0200] > [2016-08-25 17:38:43.672800 +0200][Debug ][PostMaster ] [xrootd-cms-redir-01.cr.cnaf.infn.it:1194] Found 1 address(es): [2001:760:4205:216::11]:1194 > [2016-08-25 17:38:43.672814 +0200][Debug ][AsyncSock ] [xrootd-cms-redir-01.cr.cnaf.infn.it:1194 #0.0] Attempting connection to [2001:760:4205:216::11]:1194 > [2016-08-25 17:38:43.672836 +0200][Debug ][Poller ] Adding socket 0x680014b0 to the poller > [2016-08-25 17:38:43.714938 +0200][Debug ][AsyncSock ] [xrootd-cms-redir-01.cr.cnaf.infn.it:1194 #0.0] Async connection call returned > [2016-08-25 17:38:43.714958 +0200][Debug ][XRootDTransport ] [xrootd-cms-redir-01.cr.cnaf.infn.it:1194 #0.0] Sending out the initial hand shake + kXR_protocol > [2016-08-25 17:38:43.757092 +0200][Debug ][XRootDTransport ] [xrootd-cms-redir-01.cr.cnaf.infn.it:1194 #0.0] Got the server hand shake response (type: manager [], protocol version 300) > [2016-08-25 17:38:43.757111 +0200][Debug ][XRootDTransport ] [xrootd-cms-redir-01.cr.cnaf.infn.it:1194 #0.0] kXR_protocol successful (type: manager [], protocol version 300) > [2016-08-25 17:38:43.757192 +0200][Debug ][XRootDTransport ] [xrootd-cms-redir-01.cr.cnaf.infn.it:1194 #0.0] Sending out kXR_login request, username: root, cgi: ?xrd.cc=es&xrd.tz=1&xrd.appname=xrdcp&xrd.info=&xrd.hostname=psb01.pic.es, dual-stack: true, private IPv4: false, private IPv6: false > [2016-08-25 17:38:43.799350 +0200][Debug ][XRootDTransport ] [xrootd-cms-redir-01.cr.cnaf.infn.it:1194 #0.0] Logged in, session: 27ad000030620000a10000002aad0000 > [2016-08-25 17:38:43.799366 +0200][Debug ][PostMaster ] [xrootd-cms-redir-01.cr.cnaf.infn.it:1194 #0] Stream 0 connected. > [2016-08-25 17:38:43.841746 +0200][Debug ][PostMaster ] Creating new channel to: stormgf1.pi.infn.it:1094 1 stream(s) > [2016-08-25 17:38:43.841766 +0200][Debug ][PostMaster ] [stormgf1.pi.infn.it:1094 #0] Stream parameters: Network Stack: IPv6, Connection Window: 120, ConnectionRetry: 5, Stream Error Widnow: 1800 > [2016-08-25 17:38:43.841825 +0200][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: stormgf1.pi.infn.it:1094" to be run at: [2016-08-25 17:38:58 +0200] > [2016-08-25 17:38:43.841877 +0200][Error ][Utility ] Unable to resolve stormgf1.pi.infn.it:1094: Name or service not known > [2016-08-25 17:38:43.841886 +0200][Error ][PostMaster ] [stormgf1.pi.infn.it:1094 #0] Unable to resolve IP address for the host > [2016-08-25 17:38:43.841894 +0200][Debug ][XRootD ] [stormgf1.pi.infn.it:1094] Handling error while processing kXR_open (file: /store/test/xrootd/CMSSAM//store/mc/SAM/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0013/CE4D66EB-5AAE-E111-96D6-003048D37524.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [FATAL] Invalid address. > [2016-08-25 17:38:43.927558 +0200][Debug ][PostMaster ] Creating new channel to: stormgf2.pi.infn.it:1094 1 stream(s) > [2016-08-25 17:38:43.927579 +0200][Debug ][PostMaster ] [stormgf2.pi.infn.it:1094 #0] Stream parameters: Network Stack: IPv6, Connection Window: 120, ConnectionRetry: 5, Stream Error Widnow: 1800 > [2016-08-25 17:38:43.927640 +0200][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: stormgf2.pi.infn.it:1094" to be run at: [2016-08-25 17:38:58 +0200] > [2016-08-25 17:38:43.927698 +0200][Error ][Utility ] Unable to resolve stormgf2.pi.infn.it:1094: Name or service not known > [2016-08-25 17:38:43.927707 +0200][Error ][PostMaster ] [stormgf2.pi.infn.it:1094 #0] Unable to resolve IP address for the host > [2016-08-25 17:38:43.927714 +0200][Debug ][XRootD ] [stormgf2.pi.infn.it:1094] Handling error while processing kXR_open (file: /store/test/xrootd/CMSSAM//store/mc/SAM/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0013/CE4D66EB-5AAE-E111-96D6-003048D37524.root?tried=stormgf1.pi.infn.it, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [FATAL] Invalid address. > [2016-08-25 17:38:44.057286 +0200][Debug ][TaskMgr ] Registering task: "WaitTask for: 0x0x808560" to be run at: [2016-08-25 17:39:14 +0200] > ... > ... > ``` > Information about eth devices: > ``` > [root@psb01 tmp]# ip addr list > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > inet 127.0.0.1/8 scope host lo > inet6 ::1/128 scope host > valid_lft forever preferred_lft forever > 2: p1p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 10000 > link/ether a0:36:9f:15:a6:70 brd ff:ff:ff:ff:ff:ff > 3: p1p2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 10000 > link/ether 90:b1:1c:44:bc:9b brd ff:ff:ff:ff:ff:ff > 4: em1: <BROADCAST,MULTICAST,SLAVE,UP> mtu 9000 qdisc mq master bond0 state UNKNOWN qlen 10000 > link/ether 90:b1:1c:44:bc:9b brd ff:ff:ff:ff:ff:ff > 5: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 10000 > link/ether 90:b1:1c:44:bc:9c brd ff:ff:ff:ff:ff:ff > 6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP qlen 10000 > link/ether 90:b1:1c:44:bc:9b brd ff:ff:ff:ff:ff:ff > inet 193.109.172.187/27 brd 193.109.172.191 scope global bond0 > inet6 2001:67c:1148:202::2/64 scope global > valid_lft forever preferred_lft forever > inet6 fe80::92b1:1cff:fe44:bc9b/64 scope link > valid_lft forever preferred_lft forever > > ``` > > -- > You are receiving this because you are subscribed to this thread. > Reply to this email directly or view it on GitHub: > https://github.com/xrootd/xrootd/issues/407 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/xrootd/xrootd/issues/407#issuecomment-243578859 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1