On Mar 3, 2011, at 5:52 PM, Andrew Hanushevsky wrote:
> Hi Brian,
>
> It sort of works that way right now. As long as clients are asking for files we keep extending the deadline (sort of well, if this many clients are asking we better not fail mentality). So, it might be possible.
>
I'd put this pretty high on the priority list for us: there's something funny happening here, and we definitely can't pass a moderate stress test.
I was looking through the logs for a similar occurrence, and there's definitely a pattern here: every 10 seconds, the number of "do_Have" responses processed increases by 2 orders of magnitude.
It could be xrootd, it could be the remote system - either way, there should be some tolerance for late responses under load.
Ideas? It's like we are hitting some timer somewhere.
Brian
[root@xrootd-itb ~]# grep '18:3' /var/log/xrootd/cmsd.log | grep do_Have | awk '{print $2;}' | sort | uniq -c
12 12:18:33
96 18:31:11
516 18:31:12
110 18:31:13
1 18:31:17
1 18:31:21
48 18:31:22
46 18:31:23
12 18:31:24
112 18:31:32
129 18:31:33
22 18:31:34
9 18:31:35
7 18:31:36
9 18:31:37
15 18:31:41
521 18:31:42
59 18:31:43
3 18:31:44
4 18:31:46
1 18:31:47
7 18:31:51
949 18:31:52
477 18:31:53
39 18:31:54
22 18:31:55
10 18:31:56
10 18:31:57
6 18:31:58
5 18:31:59
22 18:32:01
142 18:32:02
32 18:32:03
1 18:32:06
4 18:32:07
3 18:32:08
3 18:32:11
271 18:32:12
141 18:32:13
10 18:32:14
5 18:32:15
15 18:32:17
14 18:32:21
441 18:32:22
138 18:32:23
25 18:32:24
4 18:32:25
11 18:32:26
5 18:32:28
4 18:32:29
6 18:32:31
733 18:32:32
402 18:32:33
21 18:32:34
9 18:32:35
3 18:32:36
17 18:32:37
1 18:32:38
4 18:32:39
11 18:32:41
150 18:32:42
70 18:32:43
6 18:32:44
2 18:32:45
4 18:32:46
1 18:32:48
1 18:32:51
165 18:32:52
245 18:32:53
48 18:32:54
23 18:32:55
30 18:32:56
29 18:32:57
22 18:32:58
2 18:32:59
7 18:33:00
6 18:33:01
296 18:33:02
214 18:33:03
51 18:33:04
37 18:33:05
42 18:33:06
23 18:33:07
14 18:33:08
1 18:33:09
20 18:33:10
9 18:33:11
564 18:33:12
376 18:33:13
56 18:33:14
56 18:33:15
5 18:33:16
6 18:33:17
5 18:33:18
3 18:33:20
1 18:33:21
104 18:33:22
87 18:33:23
146 18:33:24
14 18:33:25
84 18:33:26
9 18:33:27
4 18:33:28
12 18:33:29
2 18:33:30
49 18:33:31
296 18:33:32
366 18:33:33
131 18:33:34
32 18:33:35
7 18:33:36
4 18:33:37
13 18:33:38
55 18:33:39
3 18:33:40
1 18:33:41
322 18:33:42
776 18:33:43
40 18:33:44
1 18:33:48
7 18:33:49
4 18:33:50
3 18:34:00
12 18:34:07
12 18:35:56
12 18:36:00
12 18:36:21
12 18:36:28
12 18:36:41
12 18:38:09
12 19:18:39
