[DRBD-user] drbdadm status blocked:lower

Post by Garrido, Cristina
Hello,
I have two drbd devices configured on my cluster. On both nodes the status shows "blocked:lower" although everything seems to be fine. We have conducted IO tests on the physical devices and on the drbd devices with good results. Do you know why this message is shown and how to debug it?
xxxx:/dev/mapper # drbdsetup status --verbose --statistics
ASCS node-id:1 role:Primary suspended:no
write-ordering:flush
volume:0 minor:0 disk:UpToDate
size:10452636 read:3247 written:8185665 al-writes:53 bm-writes:0 upper-pending:0 lower-pending:0 al-suspended:no blocked:lower

"blocked:lower" means that the in-kernel API for querying block
device info congestion reported "congestion" for the backing device.
Why it did that, and whether that was actually the case, and what
that actually means is very much dependend on that backing device,
and how it "felt" at the time of that status output.

--
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker

DRBD® and LINBIT® are registered trademarks of LINBIT
__
please don't Cc me, but send to list -- I'm subscribed

VictorSanchez2

2018-10-11 12:06:11 UTC

Thanks Lars,

Do you know how DRBD asks kernel about congestion information? Which is
the system call it makes?

We want to know why is marking it as "blocked:lower", because we are
making heavy performance test and seems that there is no problem at disk
or network level. We think that DRBD/kernel is not getting the correct
information from the system.

Best regards,

Victor

Lars Ellenberg

2018-10-18 19:51:32 UTC

Thanks Lars,
Do you know how DRBD asks kernel about congestion information? Which is the
system call it makes?

DRBD is part of the kernel. No system call involved.
We call bdi_congested() which is a wrapper around wb_congested(),
both defined in linux/include/backing-dev.h

Post by VictorSanchez2
We want to know why is marking it as "blocked:lower",

just ignore that wording. don't panic just because it says "blocked"...

Post by VictorSanchez2
because we are making heavy performance test and seems that there is
no problem at disk or network level.

"congestion" does not mean "no progress".
Just that you reached some kind of, well, congestion, and likely, that,
if you where to even increase the "IO load", you'd probably just make
the latency tail longer, and not improve throughput or IOPS anymore.

so you throw "heavy" IO against the IO stack. as a result, you drive
the IO stack into "congestion". and if you ask it for some status,
it reports that back.

no surprise there.

Post by VictorSanchez2
We think that DRBD/kernel is not getting the correct information from
the system.

afaics, blk_set_congested() is called when a queue has more than
"nr_congestion_on" requests "in flight", and it is cleared once that
drops below "nr_congestion_off" again. both hysteresis watermarks are
set in relation to the queue "nr_requests", which again is a tunable.

VictorSanchez2

2018-10-19 08:18:12 UTC

Thanks Lars,
Do you know how DRBD asks kernel about congestion information? Which is the
system call it makes?

DRBD is part of the kernel. No system call involved.
We call bdi_congested() which is a wrapper around wb_congested(),
both defined in linux/include/backing-dev.h

Post by VictorSanchez2
We want to know why is marking it as "blocked:lower",

just ignore that wording. don't panic just because it says "blocked"...

Post by VictorSanchez2
because we are making heavy performance test and seems that there is
no problem at disk or network level.

"congestion" does not mean "no progress".
Just that you reached some kind of, well, congestion, and likely, that,
if you where to even increase the "IO load", you'd probably just make
the latency tail longer, and not improve throughput or IOPS anymore.
so you throw "heavy" IO against the IO stack. as a result, you drive
the IO stack into "congestion". and if you ask it for some status,
it reports that back.
no surprise there.

Post by VictorSanchez2
We think that DRBD/kernel is not getting the correct information from
the system.

Thanks Lars,

how we can tune nr_requests? By default is at 128, and we can't increase it:

# cat /sys/block/drbd1/queue/nr_requests
128
echo 129 > /sys/block/drbd1/queue/nr_requests
-bash: echo: write error: Invalid argument
# uname -r
4.4.140-94.42-default

in any case, I think that increase the nr_requests will not solve the
problem. I'm worried about how congestion is taking place, if not
writings are being executing in the device at the moment. How I can see
the current queue size, or request in flight?

Looking information for these functions, I have seen that there are
reported bugs, that I don't know if they are related:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=linux-4.18.y&qt=grep&q=wb_congested&showmsg=1

kernel in these systems is 4.4.140-94.42-default

Lars Ellenberg

2018-10-24 18:06:51 UTC

Thanks Lars,
Do you know how DRBD asks kernel about congestion information? Which is the
system call it makes?

DRBD is part of the kernel. No system call involved.
We call bdi_congested() which is a wrapper around wb_congested(),
both defined in linux/include/backing-dev.h

Post by VictorSanchez2
We want to know why is marking it as "blocked:lower",

just ignore that wording. don't panic just because it says "blocked"...

Post by VictorSanchez2
because we are making heavy performance test and seems that there is
no problem at disk or network level.

"congestion" does not mean "no progress".
Just that you reached some kind of, well, congestion, and likely, that,
if you where to even increase the "IO load", you'd probably just make
the latency tail longer, and not improve throughput or IOPS anymore.
so you throw "heavy" IO against the IO stack. as a result, you drive
the IO stack into "congestion". and if you ask it for some status,
it reports that back.
no surprise there.

Post by VictorSanchez2
We think that DRBD/kernel is not getting the correct information from
the system.

Thanks Lars,

It's not about DRBD, it's about the storage backend.

Post by VictorSanchez2
# cat /sys/block/drbd1/queue/nr_requests
128
echo 129 > /sys/block/drbd1/queue/nr_requests
-bash: echo: write error: Invalid argument

sure. DRBD is a "virtual" device, which does not even have a queue.
nr_requests for DRBD has no actual meaning.

Post by VictorSanchez2
in any case, I think that increase the nr_requests will not solve the
problem.

Well, do you have any indication that there actually is a "problem"?

If your only "problem" is the string "blocked:lower"
in the drbdsetup status output, may I suggest to just ignore that?

Roland Kammerer

2018-10-25 08:05:07 UTC