[DRBD-user] Resync Incredibly Slow

[Eric Robinson]

I've been using drbd for a decade, but this is my first experience with drbd 9.0.14. The resync is incredibly slow. Normally, resync takes a few seconds to a few minutes, but this is taking hours, advancing just a few hundredths of a percent every few seconds...


ha01_mysql role:Secondary
disk:Inconsistent
001db01b role:Primary
replication:SyncTarget peer-disk:UpToDate done:93.84

ha02_mysql role:Primary
disk:UpToDate
001db01b role:Secondary
replication:SyncSource peer-disk:Inconsistent done:84.49


Also, the numbers shown by drbdadm status are way different than what is shown under /sys...


[***@001db01b 0]# cat /sys/kernel/debug/drbd/resources/ha02_mysql/connections/001db01a/0/proc_drbd
1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
ns:8 nr:175023740 dw:657162492 dr:615264242 al:2325 bm:0 lo:0 pe:[0;0] ua:0 ap:[0;0] ep:1 wo:2 oos:307063536
[======>.............] sync'ed: 36.4% (299864/470788)M
finish: 1:00:41 speed: 84,300 (91,452 -- 80,028) want: 614,400 K/sec
11% sector pos: 373912576/3221127096
resync: used:0/61 hits:696233 misses:2854 starving:0 locked:0 changed:1427
act_log: used:0/1237 hits:842881 misses:5965929 starving:0 locked:0 changed:118214
blocked on activity log: 0
[***@001db01b 0]# cat /sys/kernel/debug/drbd/resources/ha01_mysql/connections/001db01a/0/proc_drbd
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:370199208 nr:580 dw:1039815524 dr:197034909 al:50060 bm:0 lo:0 pe:[0;0] ua:0 ap:[0;0] ep:1 wo:2 oos:105671448
[=>..................] sync'ed: 10.0% (103192/114580)M
finish: 3:50:34 speed: 7,632 (2,804 -- 4,988) K/sec
0% sector pos: 0/3221127096
resync: used:0/61 hits:188990 misses:364 starving:0 locked:0 changed:186
act_log: used:0/1237 hits:173070247 misses:3163320 starving:0 locked:0 changed:130073
blocked on activity log: 0


[sig]

[Adrian Pascalau]

I’ve been using drbd for a decade, but this is my first experience with drbd 9.0.14. The resync is incredibly slow. Normally, resync takes a few seconds to a few minutes, but this is taking hours, advancing just a few hundredths of a percent every few seconds…

Hi Eric,

I am wondering, are you running drbd in physical hosts, or vitual machines?

I am in a worse situation with the same version of drbd, installed in
two up to date Centos 7 virtual machines running in an OpenStack pike
environment, with the same version of drbd installed from the elrepo
repository:

[***@drbd1 ~]# rpm -qa | grep kmod-drbd
kmod-drbd90-9.0.14-1.el7_5.elrepo.x86_64

I configured this environment last week Thursday, and believe it or
not, as of today sync is at 0.1%:

[***@drbd1 ~]# cat
/sys/kernel/debug/drbd/resources/drbd0/connections/drbd2.novalocal/0/proc_drbd
0: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r-----
ns:6363972 nr:0 dw:0 dr:112432036 al:0 bm:0 lo:0 pe:[0;4] ua:52
ap:[0;0] ep:1 wo:1 oos:5242556
[>....................] sync'ed: 0.1% (5116/5116)M
finish: 127:25:23 speed: 0 (0 -- 0) K/sec
0% sector pos: 0/10485368
resync: used:1/61 hits:62 misses:2 starving:0 locked:0 changed:1
act_log: used:0/1237 hits:0 misses:0 starving:0 locked:0 changed:0
blocked on activity log: 0

Whatever I did, did not help, so finally I replicated the environment
in my laptop, in a VMware Workstation 11 environment, using the same
version of drbd, and in this case I have no issues.

This is my first experience with drbd, so I do not know how to
troubleshoot. Now I am thinking to look into older 8.x version, that
might be working better.

Regards,
Adrian

[Papastavros Vaggelis]

Hi guys ,

Probably you need to adjust some of the parameters like

max-epoch-size ;
max-buffers ;
sndbuf-size ;
of the general drbd configuration file

Please read also the following link
https://serverfault.com/questions/740311/drbd-terrible-sync-performance-on-10gige

BR
Sent from my iPad

Post by Adrian Pascalau

I’ve been using drbd for a decade, but this is my first experience with drbd 9.0.14. The resync is incredibly slow. Normally, resync takes a few seconds to a few minutes, but this is taking hours, advancing just a few hundredths of a percent every few seconds


Hi Eric,
I am wondering, are you running drbd in physical hosts, or vitual machines?
I am in a worse situation with the same version of drbd, installed in
two up to date Centos 7 virtual machines running in an OpenStack pike
environment, with the same version of drbd installed from the elrepo
kmod-drbd90-9.0.14-1.el7_5.elrepo.x86_64
I configured this environment last week Thursday, and believe it or
/sys/kernel/debug/drbd/resources/drbd0/connections/drbd2.novalocal/0/proc_drbd
0: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r-----
ns:6363972 nr:0 dw:0 dr:112432036 al:0 bm:0 lo:0 pe:[0;4] ua:52
ap:[0;0] ep:1 wo:1 oos:5242556
[>....................] sync'ed: 0.1% (5116/5116)M
finish: 127:25:23 speed: 0 (0 -- 0) K/sec
0% sector pos: 0/10485368
resync: used:1/61 hits:62 misses:2 starving:0 locked:0 changed:1
act_log: used:0/1237 hits:0 misses:0 starving:0 locked:0 changed:0
blocked on activity log: 0
Whatever I did, did not help, so finally I replicated the environment
in my laptop, in a VMware Workstation 11 environment, using the same
version of drbd, and in this case I have no issues.
This is my first experience with drbd, so I do not know how to
troubleshoot. Now I am thinking to look into older 8.x version, that
might be working better.
Regards,
Adrian
_______________________________________________
drbd-user mailing list
http://lists.linbit.com/mailman/listinfo/drbd-user

...

[Eric Robinson]

Post by Adrian Pascalau

I’ve been using drbd for a decade, but this is my first experience
with drbd 9.0.14. The resync is incredibly slow. Normally, resync
takes a few seconds to a few minutes, but this is taking hours,
advancing just a few hundredths of a percent every few seconds…

Hi Eric,
I am wondering, are you running drbd in physical hosts, or vitual machines?
I am in a worse situation with the same version of drbd, installed in two up to
date Centos 7 virtual machines running in an OpenStack pike environment, with
the same version of drbd installed from the elrepo

...

We have Centos 7.5 servers running in Microsoft Azure.

Post by Adrian Pascalau
kmod-drbd90-9.0.14-1.el7_5.elrepo.x86_64
I configured this environment last week Thursday, and believe it or not, as of
/sys/kernel/debug/drbd/resources/drbd0/connections/drbd2.novalocal/0/proc_
drbd
0: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r-----
ns:6363972 nr:0 dw:0 dr:112432036 al:0 bm:0 lo:0 pe:[0;4] ua:52 ap:[0;0]
ep:1 wo:1 oos:5242556
[>....................] sync'ed: 0.1% (5116/5116)M
finish: 127:25:23 speed: 0 (0 -- 0) K/sec
0% sector pos: 0/10485368
resync: used:1/61 hits:62 misses:2 starving:0 locked:0 changed:1
act_log: used:0/1237 hits:0 misses:0 starving:0 locked:0 changed:0
blocked on activity log: 0
Whatever I did, did not help, so finally I replicated the environment in my
laptop, in a VMware Workstation 11 environment, using the same version of
drbd, and in this case I have no issues.

...

Hmm. I have absolutely no idea what to say about that. If I did, my own issue would be fixed. 😉

Post by Adrian Pascalau
This is my first experience with drbd, so I do not know how to troubleshoot.
Now I am thinking to look into older 8.x version, that might be working better.
Regards,
Adrian