-
Notifications
You must be signed in to change notification settings - Fork 3
Expand file tree
/
Copy pathindex.html
More file actions
1993 lines (828 loc) · 78.4 KB
/
Copy pathindex.html
File metadata and controls
1993 lines (828 loc) · 78.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!doctype html>
<html class="theme-next mist use-motion" lang="zh-Hans">
<head>
<meta charset="UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1"/>
<meta http-equiv="Cache-Control" content="no-transform" />
<meta http-equiv="Cache-Control" content="no-siteapp" />
<link href="/lib/fancybox/source/jquery.fancybox.css?v=2.1.5" rel="stylesheet" type="text/css" />
<link href="//fonts.googleapis.com/css?family=Lato:300,300italic,400,400italic,700,700italic&subset=latin,latin-ext" rel="stylesheet" type="text/css">
<link href="/lib/font-awesome/css/font-awesome.min.css?v=4.6.2" rel="stylesheet" type="text/css" />
<link href="/css/main.css?v=5.1.1" rel="stylesheet" type="text/css" />
<meta name="keywords" content="Hexo, NexT" />
<link rel="alternate" href="/atom.xml" title="System Notes" type="application/atom+xml" />
<link rel="shortcut icon" type="image/x-icon" href="/favicon.ico?v=5.1.1" />
<meta name="description" content="Diary Notebook">
<meta property="og:type" content="website">
<meta property="og:title" content="System Notes">
<meta property="og:url" content="http://yoursite.com/index.html">
<meta property="og:site_name" content="System Notes">
<meta property="og:description" content="Diary Notebook">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="System Notes">
<meta name="twitter:description" content="Diary Notebook">
<script type="text/javascript" id="hexo.configurations">
var NexT = window.NexT || {};
var CONFIG = {
root: '/',
scheme: 'Mist',
sidebar: {"position":"left","display":"post","offset":12,"offset_float":0,"b2t":false,"scrollpercent":false},
fancybox: true,
motion: true,
duoshuo: {
userId: '0',
author: '博主'
},
algolia: {
applicationID: '',
apiKey: '',
indexName: '',
hits: {"per_page":10},
labels: {"input_placeholder":"Search for Posts","hits_empty":"We didn't find any results for the search: ${query}","hits_stats":"${hits} results found in ${time} ms"}
}
};
</script>
<link rel="canonical" href="http://yoursite.com/"/>
<title>System Notes</title>
</head>
<body itemscope itemtype="http://schema.org/WebPage" lang="zh-Hans">
<script type="text/javascript">
var _hmt = _hmt || [];
(function() {
var hm = document.createElement("script");
hm.src = "https://hm.baidu.com/hm.js?6958d5097bd406d16931b3fa137c6a9f";
var s = document.getElementsByTagName("script")[0];
s.parentNode.insertBefore(hm, s);
})();
</script>
<div class="container sidebar-position-left
page-home
">
<div class="headband"></div>
<header id="header" class="header" itemscope itemtype="http://schema.org/WPHeader">
<div class="header-inner"><div class="site-brand-wrapper">
<div class="site-meta ">
<div class="custom-logo-site-title">
<a href="/" class="brand" rel="start">
<span class="logo-line-before"><i></i></span>
<span class="site-title">System Notes</span>
<span class="logo-line-after"><i></i></span>
</a>
</div>
<p class="site-subtitle"></p>
</div>
<div class="site-nav-toggle">
<button>
<span class="btn-bar"></span>
<span class="btn-bar"></span>
<span class="btn-bar"></span>
</button>
</div>
</div>
<nav class="site-nav">
<ul id="menu" class="menu">
<li class="menu-item menu-item-home">
<a href="/" rel="section">
<i class="menu-item-icon fa fa-fw fa-home"></i> <br />
首页
</a>
</li>
<li class="menu-item menu-item-categories">
<a href="/categories" rel="section">
<i class="menu-item-icon fa fa-fw fa-th"></i> <br />
分类
</a>
</li>
<li class="menu-item menu-item-about">
<a href="/about" rel="section">
<i class="menu-item-icon fa fa-fw fa-user"></i> <br />
关于
</a>
</li>
<li class="menu-item menu-item-archives">
<a href="/archives" rel="section">
<i class="menu-item-icon fa fa-fw fa-archive"></i> <br />
归档
</a>
</li>
<li class="menu-item menu-item-tags">
<a href="/tags" rel="section">
<i class="menu-item-icon fa fa-fw fa-tags"></i> <br />
标签
</a>
</li>
</ul>
</nav>
</div>
</header>
<main id="main" class="main">
<div class="main-inner">
<div class="content-wrap">
<div id="content" class="content">
<section id="posts" class="posts-expand">
<article class="post post-type-normal " itemscope itemtype="http://schema.org/Article">
<link itemprop="mainEntityOfPage" href="http://yoursite.com/2017/04/21/three-iscsitargets-io-flow/">
<span hidden itemprop="author" itemscope itemtype="http://schema.org/Person">
<meta itemprop="name" content="Wu Dong">
<meta itemprop="description" content="">
<meta itemprop="image" content="/images/avatar.png">
</span>
<span hidden itemprop="publisher" itemscope itemtype="http://schema.org/Organization">
<meta itemprop="name" content="System Notes">
</span>
<header class="post-header">
<h1 class="post-title" itemprop="name headline">
<a class="post-title-link" href="/2017/04/21/three-iscsitargets-io-flow/" itemprop="url">图解三种iscsi target软件的IO框架</a></h1>
<div class="post-meta">
<span class="post-time">
<span class="post-meta-item-icon">
<i class="fa fa-calendar-o"></i>
</span>
<span class="post-meta-item-text">发表于</span>
<time title="创建于" itemprop="dateCreated datePublished" datetime="2017-04-21T19:45:22+08:00">
2017-04-21
</time>
</span>
<span class="post-category" >
<span class="post-meta-divider">|</span>
<span class="post-meta-item-icon">
<i class="fa fa-folder-o"></i>
</span>
<span class="post-meta-item-text">分类于</span>
<span itemprop="about" itemscope itemtype="http://schema.org/Thing">
<a href="/categories/storage/" itemprop="url" rel="index">
<span itemprop="name">storage</span>
</a>
</span>
，
<span itemprop="about" itemscope itemtype="http://schema.org/Thing">
<a href="/categories/storage/iscsi/" itemprop="url" rel="index">
<span itemprop="name">iscsi</span>
</a>
</span>
</span>
</div>
</header>
<div class="post-body" itemprop="articleBody">
<p><img src="/2017/04/21/three-iscsitargets-io-flow/iet-io-flow.png" alt="three-iscsitargets-io-flow"></p>
<p><img src="/2017/04/21/three-iscsitargets-io-flow/tgt-rdwr-io-flow.png" alt="three-iscsitargets-io-flow"></p>
<p><img src="/2017/04/21/three-iscsitargets-io-flow/tgt-async-io-flow.png" alt="three-iscsitargets-io-flow"></p>
<p><img src="/2017/04/21/three-iscsitargets-io-flow/lio-io-flow.png" alt="three-iscsitargets-io-flow"></p>
<h2 id="三种I-O模型对比"><a href="#三种I-O模型对比" class="headerlink" title="三种I/O模型对比"></a>三种I/O模型对比</h2><p>上面3种target的I/O处理流程实际上就是对应常见的3种I/O模型。</p>
<ul>
<li><strong>1个连接对应1个网络线程+n个I/O线程</strong>：这种模型首先需要1个线程来处理所有的connect请求，然后连接建立的时候才能去创建该连接的网络线程及I/O线程（在IET的实现中，用户态的ietd担任这个角色）。这种模型的好处是每个连接有自己的网络线程，各个连接之间不干扰，并且该连接上的I/O处理有专门的I/O线程负责，网络线程只用做好网络收发及协议解析相关的操作，各司其职，并且可以利用多cpu的资源，充分发挥cpu的利用率；缺点就是当连接比较多的时候很耗资源；</li>
<li><strong>epoll+多个I/O线程</strong>：epoll是现在比较流行的处理模型，一个epoll的处理能力大概10万qps，如果在epoll线程里只是处理网络包及协议的解析之类的，那么epoll的能力足够了，不过如果在里面有涉及到I/O的操作，就有可能处理时间比较长导致其他连接上的请求无法响应，因此将I/O处理单独出来比较好。epoll的有个缺点就是只能利用单个cpu的资源，如果需要更高的处理能力，单epoll的方式会成为瓶颈，一个比较好的扩展方式是多epoll+多个I/O线程，将不同连接分配到不同的epoll线程去处理（比如采用hash的方式分配），然后I/O仍然由另外的I/O线程来处理；</li>
<li><strong>1个连接对应2个线程（receive和send）</strong>：这个是比较过时的做法，好处就是比较简单，明显的缺点就是连接多了会导致线程资源占用过多给系统带来瓶颈。</li>
</ul>
</div>
<div>
</div>
<div>
</div>
<div>
</div>
<footer class="post-footer">
<div class="post-eof"></div>
</footer>
</article>
<article class="post post-type-normal " itemscope itemtype="http://schema.org/Article">
<link itemprop="mainEntityOfPage" href="http://yoursite.com/2017/03/15/iscsiadm-affect-reconnect/">
<span hidden itemprop="author" itemscope itemtype="http://schema.org/Person">
<meta itemprop="name" content="Wu Dong">
<meta itemprop="description" content="">
<meta itemprop="image" content="/images/avatar.png">
</span>
<span hidden itemprop="publisher" itemscope itemtype="http://schema.org/Organization">
<meta itemprop="name" content="System Notes">
</span>
<header class="post-header">
<h1 class="post-title" itemprop="name headline">
<a class="post-title-link" href="/2017/03/15/iscsiadm-affect-reconnect/" itemprop="url">频繁执行iscsiadm导致iscsid不进行target的重连</a></h1>
<div class="post-meta">
<span class="post-time">
<span class="post-meta-item-icon">
<i class="fa fa-calendar-o"></i>
</span>
<span class="post-meta-item-text">发表于</span>
<time title="创建于" itemprop="dateCreated datePublished" datetime="2017-03-15T16:29:59+08:00">
2017-03-15
</time>
</span>
<span class="post-category" >
<span class="post-meta-divider">|</span>
<span class="post-meta-item-icon">
<i class="fa fa-folder-o"></i>
</span>
<span class="post-meta-item-text">分类于</span>
<span itemprop="about" itemscope itemtype="http://schema.org/Thing">
<a href="/categories/iscsi/" itemprop="url" rel="index">
<span itemprop="name">iscsi</span>
</a>
</span>
</span>
</div>
</header>
<div class="post-body" itemprop="articleBody">
<p>open-iscsi实现了iscsi的自动重连，当target所在机器重启或者target的网络异常时，open-iscsi都会一直尝试connect，一旦当target重建出来后，open-iscsi就能够重新login成功。但是测试时发现，当频繁执行iscsiadm的命令时，会导致open-iscsi不去进行重连了。</p>
<h2 id="观察到的现象"><a href="#观察到的现象" class="headerlink" title="观察到的现象"></a>观察到的现象</h2><p>当target异常时，strace跟踪iscsid进程会发现一直尝试重连，不过一直失败。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ sudo strace -e connect -p 28022 -t</span><br><span class="line">Process 28022 attached - interrupt to quit</span><br><span class="line">15:19:31 connect(10, {sa_family=AF_INET, sin_port=htons(3260), sin_addr=inet_addr("10.180.0.30")}, 128) = -1 EINPROGRESS (Operation now in progress)</span><br><span class="line">15:19:34 connect(10, {sa_family=AF_INET, sin_port=htons(3260), sin_addr=inet_addr("10.180.0.30")}, 128) = -1 EINPROGRESS (Operation now in progress)</span><br><span class="line">15:19:37 connect(10, {sa_family=AF_INET, sin_port=htons(3260), sin_addr=inet_addr("10.180.0.30")}, 128) = -1 EINPROGRESS (Operation now in progress)</span><br><span class="line">15:19:40 connect(10, {sa_family=AF_INET, sin_port=htons(3260), sin_addr=inet_addr("10.180.0.30")}, 128) = -1 EINPROGRESS (Operation now in progress)</span><br></pre></td></tr></table></figure></p>
<p>模拟频繁执行iscsiadm的命令<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ for i in {1..10000};do sudo iscsiadm -m session -P3 2&> /dev/null;done</span><br></pre></td></tr></table></figure></p>
<p>这个时候去strace跟踪会发现没有去重连target的信息，反而是一些奇怪的记录。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">17:24:12 connect(9, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)</span><br><span class="line">17:24:13 connect(9, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)</span><br><span class="line">17:24:14 connect(9, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)</span><br><span class="line">17:24:14 connect(9, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)</span><br><span class="line">17:24:15 connect(9, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)</span><br></pre></td></tr></table></figure></p>
<!--noindex-->
<div class="post-button text-center">
<a class="btn" href="/2017/03/15/iscsiadm-affect-reconnect/#more" rel="contents">
阅读全文 »
</a>
</div>
<!--/noindex-->
</div>
<div>
</div>
<div>
</div>
<div>
</div>
<footer class="post-footer">
<div class="post-eof"></div>
</footer>
</article>
<article class="post post-type-normal " itemscope itemtype="http://schema.org/Article">
<link itemprop="mainEntityOfPage" href="http://yoursite.com/2017/03/15/error-disk-cause-iohang2/">
<span hidden itemprop="author" itemscope itemtype="http://schema.org/Person">
<meta itemprop="name" content="Wu Dong">
<meta itemprop="description" content="">
<meta itemprop="image" content="/images/avatar.png">
</span>
<span hidden itemprop="publisher" itemscope itemtype="http://schema.org/Organization">
<meta itemprop="name" content="System Notes">
</span>
<header class="post-header">
<h1 class="post-title" itemprop="name headline">
<a class="post-title-link" href="/2017/03/15/error-disk-cause-iohang2/" itemprop="url">坏盘导致IO hang问题分析（续）</a></h1>
<div class="post-meta">
<span class="post-time">
<span class="post-meta-item-icon">
<i class="fa fa-calendar-o"></i>
</span>
<span class="post-meta-item-text">发表于</span>
<time title="创建于" itemprop="dateCreated datePublished" datetime="2017-03-15T16:21:22+08:00">
2017-03-15
</time>
</span>
<span class="post-category" >
<span class="post-meta-divider">|</span>
<span class="post-meta-item-icon">
<i class="fa fa-folder-o"></i>
</span>
<span class="post-meta-item-text">分类于</span>
<span itemprop="about" itemscope itemtype="http://schema.org/Thing">
<a href="/categories/storage/" itemprop="url" rel="index">
<span itemprop="name">storage</span>
</a>
</span>
</span>
</div>
</header>
<div class="post-body" itemprop="articleBody">
<p>经过前面的分析测试，使用fio直接测试裸盘（direct=1）的时候，将后端设备suspend住后，fio的IO hang的时间跟前面计算的一致；但是当在虚拟机里对这块盘做了文件系统，然后fio测试文件系统（direct=1）的时候，将后端设备suspend住后，fio的IO hang的时间多了20~30s。</p>
<h2 id="初步分析"><a href="#初步分析" class="headerlink" title="初步分析"></a>初步分析</h2><p>一开始怀疑是由于虚拟机这层的作用，因此还调整过KVM使用的盘的缓存策略（none、writethrough、directsync），同样的测试，仍然是IO hang比预期的多20~30s。</p>
<p>打开iscsi session debug开关，开启scsi error和timeout的日志，然后分析这样测试过程中的日志信息。下面截取部分日志，发现有3个时间点：16:45:21、16:45:31、16:45:41比较奇怪，没有连续的日志，都有10s的空白期。而且从16:45:41到16:45:51后，就是”Device offlined - not ready after error recovery”，而之前从裸盘上测试时是没有这几次10s的空白期，而且是I/O error，而不是device offlined后再I/O error。<br></p>
<!--noindex-->
<div class="post-button text-center">
<a class="btn" href="/2017/03/15/error-disk-cause-iohang2/#more" rel="contents">
阅读全文 »
</a>
</div>
<!--/noindex-->
</div>
<div>
</div>
<div>
</div>
<div>
</div>
<footer class="post-footer">
<div class="post-eof"></div>
</footer>
</article>
<article class="post post-type-normal " itemscope itemtype="http://schema.org/Article">
<link itemprop="mainEntityOfPage" href="http://yoursite.com/2017/03/15/error-disk-cause-iohang/">
<span hidden itemprop="author" itemscope itemtype="http://schema.org/Person">
<meta itemprop="name" content="Wu Dong">
<meta itemprop="description" content="">
<meta itemprop="image" content="/images/avatar.png">
</span>
<span hidden itemprop="publisher" itemscope itemtype="http://schema.org/Organization">
<meta itemprop="name" content="System Notes">
</span>
<header class="post-header">
<h1 class="post-title" itemprop="name headline">
<a class="post-title-link" href="/2017/03/15/error-disk-cause-iohang/" itemprop="url">坏盘导致IO hang问题分析</a></h1>
<div class="post-meta">
<span class="post-time">
<span class="post-meta-item-icon">
<i class="fa fa-calendar-o"></i>
</span>
<span class="post-meta-item-text">发表于</span>
<time title="创建于" itemprop="dateCreated datePublished" datetime="2017-03-15T16:09:40+08:00">
2017-03-15
</time>
</span>
<span class="post-category" >
<span class="post-meta-divider">|</span>
<span class="post-meta-item-icon">
<i class="fa fa-folder-o"></i>
</span>
<span class="post-meta-item-text">分类于</span>
<span itemprop="about" itemscope itemtype="http://schema.org/Thing">
<a href="/categories/storage/" itemprop="url" rel="index">
<span itemprop="name">storage</span>
</a>
</span>
</span>
</div>
</header>
<div class="post-body" itemprop="articleBody">
<p>近期线上系统出现坏盘导致IO卡住的问题，有两种情况：</p>
<p>1）一种是出现坏盘过程中raid卡的行为有所异常，在这台机器上的执行raid卡的命令megacli都卡住，观察到这台机器上的物理盘的io都时不时异常繁忙（io不大，但是svctm甚至达到几千ms），从而导致我们的块存储的卷IO hang住，表现就是用户的卷io util 100%较长时间；</p>
<p>2）另外一种情况是坏盘后，将坏盘从raid卡中剔除（做的单盘raid0，默认WB缓存策略，盘坏后需要从raid卡里删除逻辑卷），这台机器上的物理盘都io卡住一会，并且megacli命令也卡住。从而也导致部分用户的卷io util 100%较长时间；</p>
<p>情况1）通过收集日志反馈给厂家确认是由于线上机器的raid卡的固件版本比较低，需要升级raid卡固件；<br>情况2）因为坏盘后，raid卡就进入保护模式，物理盘的缓存策略都从writeback变成writethrough，然后剔除坏盘时，就会将raid卡缓存里的数据进行回刷到磁盘上，这个过程中是会有短时间的物理盘io卡住；</p>
<p>上面两种情况，物理盘的io卡住都在1分钟内，可是从用户反馈来看用户卷的io卡了好几分钟，比物理盘卡的时间长不少。另外，我们这个线上环境的块存储采用的2副本，有些卷的其中一个副本落在坏盘的机器上，卡了这么久为什么没有副本降级。<br></p>
<!--noindex-->
<div class="post-button text-center">
<a class="btn" href="/2017/03/15/error-disk-cause-iohang/#more" rel="contents">
阅读全文 »
</a>
</div>
<!--/noindex-->
</div>
<div>
</div>
<div>
</div>
<div>
</div>
<footer class="post-footer">
<div class="post-eof"></div>
</footer>
</article>
<article class="post post-type-normal " itemscope itemtype="http://schema.org/Article">
<link itemprop="mainEntityOfPage" href="http://yoursite.com/2016/08/29/ceph-io-sequence/">
<span hidden itemprop="author" itemscope itemtype="http://schema.org/Person">
<meta itemprop="name" content="Wu Dong">
<meta itemprop="description" content="">
<meta itemprop="image" content="/images/avatar.png">
</span>
<span hidden itemprop="publisher" itemscope itemtype="http://schema.org/Organization">
<meta itemprop="name" content="System Notes">
</span>
<header class="post-header">
<h1 class="post-title" itemprop="name headline">
<a class="post-title-link" href="/2016/08/29/ceph-io-sequence/" itemprop="url">ceph中对象读写的顺序性及并发性保证</a></h1>
<div class="post-meta">
<span class="post-time">
<span class="post-meta-item-icon">
<i class="fa fa-calendar-o"></i>
</span>
<span class="post-meta-item-text">发表于</span>
<time title="创建于" itemprop="dateCreated datePublished" datetime="2016-08-29T19:54:23+08:00">
2016-08-29
</time>
</span>
<span class="post-category" >
<span class="post-meta-divider">|</span>
<span class="post-meta-item-icon">
<i class="fa fa-folder-o"></i>
</span>
<span class="post-meta-item-text">分类于</span>
<span itemprop="about" itemscope itemtype="http://schema.org/Thing">
<a href="/categories/ceph/" itemprop="url" rel="index">
<span itemprop="name">ceph</span>
</a>
</span>
</span>
</div>
</header>
<div class="post-body" itemprop="articleBody">
<p>分布式系统中经常需要考虑对象（或者记录、文件、数据块等）的读写的顺序以及并发访问问题。通常来说，如果两个对象没有共享的资源，就可以进行并发的访问，如果有共享的部分，就需要对这部分资源进行加锁；而对于同一个对象的并发读写（尤其是并发写更新时），就需要注意顺序性以及并发访问的控制，以免数据错乱。本文主要对ceph中对象读写的顺序及并发性保证机制进行介绍。</p>
<h1 id="PG的概念"><a href="#PG的概念" class="headerlink" title="PG的概念"></a>PG的概念</h1><p>ceph中引入PG（Placement Group）的概念，这是一个逻辑上的组，通过crush算法映射到一组OSD上，每个PG里包含完整的副本关系，比如3个数据副本分布到一个PG里的3个不同的OSD上。下面引用ceph论文中的一张图就可以比较直观的理解了。将文件按照指定的对象大小分割成多个对象，每个对象根据hash映射到某个pg，然后再根据crush算法映射到后端的某几个osd上。<br><img src="/2016/08/29/ceph-io-sequence/pg.png" alt="ceph-io-sequence"></p>
<h1 id="不同对象的并发控制"><a href="#不同对象的并发控制" class="headerlink" title="不同对象的并发控制"></a>不同对象的并发控制</h1><p>不同的对象有可能落到同一个pg里，ceph实现里，在OSD的处理线程中就会给PG加锁，一直到queue_transactions里把事务放到journal的队列里（以filestore为例）才释放PG的锁。从这里可以看出，对于同一个PG里的不同对象，是通过PG锁来进行并发的控制，好在这个过程中没有涉及到对象的I/O，不会太影响效率；对于不同PG的对象，就可以直接进行并发访问。<br></p>
<!--noindex-->
<div class="post-button text-center">
<a class="btn" href="/2016/08/29/ceph-io-sequence/#more" rel="contents">
阅读全文 »
</a>
</div>
<!--/noindex-->
</div>
<div>
</div>
<div>
</div>
<div>
</div>
<footer class="post-footer">
<div class="post-eof"></div>
</footer>
</article>