[DRBD-user] DRBD 9 Primary-Secondary, Pacemaker, and STONITH

[Bryan K. Walton]

I have a two-node DRBD 9 resource configured in Primary-Secondary mode
with automatic failover configured with Pacemaker.

I know that I need to configure STONITH in Pacemaker and then set DRBD's
fencing to "resource-and-stonith".

The nodes are Supermicro servers with IPMI. I'm planning to use IPMI
for my (first) fencing level.

Where I'm confused is regarding whether I must have a second fencing
level beyond IPMI? Or will DRBD's fencing configuration, combined with
IPMI be good enough?

Looking at:
http://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/pdf/Clusters_from_Scratch/Pacemaker-1.1-Clusters_from_Scratch-en-US.pdf

It reads:

"A common mistake people make when choosing a STONITH device
is to use a remote power switch (such as many on-board IPMI controllers)
that shares power with the node it controls. If the power fails in such
a case, the cluster cannot be sure whether the node is really offline,
or active and suffering from a network fault, so the cluster will stop
all resources to avoid a possible split-brain situation."

I don't understand this. If the power fails to a node, then won't the
node, by definition be down (since there is no power going to the node)?
So, how then could there be a split brain when one node has no power?

Is the above quote stating that if Pacemaker can't confirm that one
node has been STONITHed, that it won't allow the remaining node to work,
either?

Thanks!
Bryan Walton

[Igor Cicimov]

On Wed, Nov 14, 2018 at 8:31 AM Bryan K. Walton

Post by Bryan K. Walton
I have a two-node DRBD 9 resource configured in Primary-Secondary mode
with automatic failover configured with Pacemaker.
I know that I need to configure STONITH in Pacemaker and then set DRBD's
fencing to "resource-and-stonith".
The nodes are Supermicro servers with IPMI. I'm planning to use IPMI
for my (first) fencing level.
Where I'm confused is regarding whether I must have a second fencing
level beyond IPMI? Or will DRBD's fencing configuration, combined with
IPMI be good enough?
http://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/pdf/Clusters_from_Scratch/Pacemaker-1.1-Clusters_from_Scratch-en-US.pdf
"A common mistake people make when choosing a STONITH device
is to use a remote power switch (such as many on-board IPMI controllers)
that shares power with the node it controls. If the power fails in such
a case, the cluster cannot be sure whether the node is really offline,
or active and suffering from a network fault, so the cluster will stop
all resources to avoid a possible split-brain situation."
I don't understand this. If the power fails to a node, then won't the
node, by definition be down (since there is no power going to the node)?
So, how then could there be a split brain when one node has no power?

And how is the other node suppose to know that it's peer crashed due
to power failure? From it's stand point the node disappeared and it
has to attempt to STONITH. Now, if the STONITH device on the other
side gets it's power via the same supply as the server then STONITH is
not possible because the stonith device itself will be powered down
too. You see the problem now?

Post by Bryan K. Walton
Is the above quote stating that if Pacemaker can't confirm that one
node has been STONITHed, that it won't allow the remaining node to work,
either?
Thanks!
Bryan Walton
_______________________________________________
drbd-user mailing list
http://lists.linbit.com/mailman/listinfo/drbd-user

[Robert Altnoeder]

Post by Igor Cicimov
On Wed, Nov 14, 2018 at 8:31 AM Bryan K. Walton

Post by Bryan K. Walton
I don't understand this. If the power fails to a node, then won't the
node, by definition be down (since there is no power going to the node)?
So, how then could there be a split brain when one node has no power?

And how is the other node suppose to know that it's peer crashed due
to power failure? From it's stand point the node disappeared and it
has to attempt to STONITH. Now, if the STONITH device on the other
side gets it's power via the same supply as the server then STONITH is
not possible because the stonith device itself will be powered down
too. You see the problem now?

That is exactly the problem.

Post by Igor Cicimov

Post by Bryan K. Walton
Is the above quote stating that if Pacemaker can't confirm that one
node has been STONITHed, that it won't allow the remaining node to work,
either?

At least in the default configuration, if fencing fails, the cluster
freezes. The cluster does not stop anything that is already running on
some other node, but it will also not start any services that are not
running (e.g. those that were running on the node that is probably down).
This situation continues until either a fencing retry succeeds, or until
an operator manually confirms to the cluster that the node the cluster
was trying to fence is down.

br,
Robert

[Bryan K. Walton]

Post by Robert Altnoeder

Post by Bryan K. Walton
Is the above quote stating that if Pacemaker can't confirm that one
node has been STONITHed, that it won't allow the remaining node to work,
either?

At least in the default configuration, if fencing fails, the cluster
freezes. The cluster does not stop anything that is already running on
some other node, but it will also not start any services that are not
running (e.g. those that were running on the node that is probably down).
This situation continues until either a fencing retry succeeds, or until
an operator manually confirms to the cluster that the node the cluster
was trying to fence is down.

This makes a lot of sense, to me now. Thanks!

-Bryan

[Bryan K. Walton]

Post by Igor Cicimov

Post by Bryan K. Walton
I don't understand this. If the power fails to a node, then won't the
node, by definition be down (since there is no power going to the node)?
So, how then could there be a split brain when one node has no power?

And how is the other node suppose to know that it's peer crashed due
to power failure? From it's stand point the node disappeared and it
has to attempt to STONITH. Now, if the STONITH device on the other
side gets it's power via the same supply as the server then STONITH is
not possible because the stonith device itself will be powered down
too. You see the problem now?

Thank you for this clarification, Igor.

-Bryan