Hi Brian,
Could you produce similar stats for how many state requests each server got?
All that should be in the log. As for the 10 second variations that could
happen if the clients are being told to hold for one or two periods or the
not-available time of 10 seconds. That should be in the log as well.
Andy
-----Original Message-----
From: Brian Bockelman
Sent: Thursday, March 03, 2011 5:38 PM
To: Andrew Hanushevsky
Cc: xrootd-dev
Subject: Re: Delay for SelPrep
On Mar 3, 2011, at 5:52 PM, Andrew Hanushevsky wrote:
> Hi Brian,
>
> It sort of works that way right now. As long as clients are asking for
> files we keep extending the deadline (sort of well, if this many clients
> are asking we better not fail mentality). So, it might be possible.
>
I'd put this pretty high on the priority list for us: there's something
funny happening here, and we definitely can't pass a moderate stress test.
I was looking through the logs for a similar occurrence, and there's
definitely a pattern here: every 10 seconds, the number of "do_Have"
responses processed increases by 2 orders of magnitude.
It could be xrootd, it could be the remote system - either way, there should
be some tolerance for late responses under load.
Ideas? It's like we are hitting some timer somewhere.
Brian
[root@xrootd-itb ~]# grep '18:3' /var/log/xrootd/cmsd.log | grep do_Have |
awk '{print $2;}' | sort | uniq -c
12 12:18:33
96 18:31:11
516 18:31:12
110 18:31:13
1 18:31:17
1 18:31:21
48 18:31:22
46 18:31:23
12 18:31:24
112 18:31:32
129 18:31:33
22 18:31:34
9 18:31:35
7 18:31:36
9 18:31:37
15 18:31:41
521 18:31:42
59 18:31:43
3 18:31:44
4 18:31:46
1 18:31:47
7 18:31:51
949 18:31:52
477 18:31:53
39 18:31:54
22 18:31:55
10 18:31:56
10 18:31:57
6 18:31:58
5 18:31:59
22 18:32:01
142 18:32:02
32 18:32:03
1 18:32:06
4 18:32:07
3 18:32:08
3 18:32:11
271 18:32:12
141 18:32:13
10 18:32:14
5 18:32:15
15 18:32:17
14 18:32:21
441 18:32:22
138 18:32:23
25 18:32:24
4 18:32:25
11 18:32:26
5 18:32:28
4 18:32:29
6 18:32:31
733 18:32:32
402 18:32:33
21 18:32:34
9 18:32:35
3 18:32:36
17 18:32:37
1 18:32:38
4 18:32:39
11 18:32:41
150 18:32:42
70 18:32:43
6 18:32:44
2 18:32:45
4 18:32:46
1 18:32:48
1 18:32:51
165 18:32:52
245 18:32:53
48 18:32:54
23 18:32:55
30 18:32:56
29 18:32:57
22 18:32:58
2 18:32:59
7 18:33:00
6 18:33:01
296 18:33:02
214 18:33:03
51 18:33:04
37 18:33:05
42 18:33:06
23 18:33:07
14 18:33:08
1 18:33:09
20 18:33:10
9 18:33:11
564 18:33:12
376 18:33:13
56 18:33:14
56 18:33:15
5 18:33:16
6 18:33:17
5 18:33:18
3 18:33:20
1 18:33:21
104 18:33:22
87 18:33:23
146 18:33:24
14 18:33:25
84 18:33:26
9 18:33:27
4 18:33:28
12 18:33:29
2 18:33:30
49 18:33:31
296 18:33:32
366 18:33:33
131 18:33:34
32 18:33:35
7 18:33:36
4 18:33:37
13 18:33:38
55 18:33:39
3 18:33:40
1 18:33:41
322 18:33:42
776 18:33:43
40 18:33:44
1 18:33:48
7 18:33:49
4 18:33:50
3 18:34:00
12 18:34:07
12 18:35:56
12 18:36:00
12 18:36:21
12 18:36:28
12 18:36:41
12 18:38:09
12 19:18:39
