Home > MongoDB, NoSQL > MongoDB Learning Notes – Simulate automate failover

MongoDB Learning Notes – Simulate automate failover

To simulate automate failover I kill the primary node with Ctrl+C or use command db.shutdownServer(). When the primary killed, the secondary become primary because a majority of the original nodes (the arbiter and the secondary) are still able ping each other.

myapp:PRIMARY> db.shutdownServer()
shutdown command only works with the admin database; try 'use admin'
myapp:PRIMARY> use admin
switched to db admin
myapp:PRIMARY> db.shutdownServer()

And in secondary node log:

2015-02-22T13:58:33.558+0700 [rsHealthPoll] replSet member bluething-PC:20000 is now in state DOWN
2015-02-22T13:58:34.391+0700 [conn501] replSet info voting yea for bluething-PC:20001 (1)
2015-02-22T13:58:36.565+0700 [rsHealthPoll] warning: Failed to connect to 192.168.19.1:20000, reason
: errno:10061 No connection could be made because the target machine actively refused it.
2015-02-22T13:58:36.565+0700 [rsHealthPoll] replset info bluething-PC:20000 heartbeat failed, retrying
2015-02-22T13:58:36.632+0700 [rsHealthPoll] replSet member bluething-PC:20001 is now in state PRIMARY

Connect to secondary and check status

myapp:SECONDARY> rs.status()
{
 "set" : "myapp",
 "date" : ISODate("2015-02-22T07:00:28Z"),
 "myState" : 1,
 "members" : [
 {
 "_id" : 0,
 "name" : "bluething-PC:20000",
 "health" : 0,
 "state" : 8,
 "stateStr" : "(not reachable/healthy)",
 "uptime" : 0,
 "optime" : Timestamp(1424581294, 1),
 "optimeDate" : ISODate("2015-02-22T05:01:34Z"),
 "lastHeartbeat" : ISODate("2015-02-22T07:00:20Z"),
 "lastHeartbeatRecv" : ISODate("2015-02-22T06:58:31Z"),
 "pingMs" : 0
 },
 {
 "_id" : 1,
 "name" : "bluething-PC:20001",
 "health" : 1,
 "state" : 1,
 "stateStr" : "PRIMARY",
 "uptime" : 9518,
 "optime" : Timestamp(1424581294, 1),
 "optimeDate" : ISODate("2015-02-22T05:01:34Z"),
 "electionTime" : Timestamp(1424588314, 1),
 "electionDate" : ISODate("2015-02-22T06:58:34Z"),
 "self" : true
 },
 {
 "_id" : 2,
 "name" : "bluething-PC:20002",
 "health" : 1,
 "state" : 7,
 "stateStr" : "ARBITER",
 "uptime" : 7553,
 "lastHeartbeat" : ISODate("2015-02-22T07:00:27Z"),
 "lastHeartbeatRecv" : ISODate("2015-02-22T07:00:26Z"),
 "pingMs" : 0
 }
 ],
 "ok" : 1
}
myapp:PRIMARY>

Now the replica set contains only two nodes. Because the arbiter has no data, the application will continue to function as long as it communicates with primary node only.
Now bring the old primary on to became the new secondary.

C:\MongoDB\mongodb-win32-x86_64-2008plus-2.6.6\bin>mongod --replSet myapp --dbpath /data/node1 --port 20000

Connect to the new secondary and check for status

> rs.status()
2015-02-22T14:04:06.263+0700 trying reconnect to localhost:20000 (127.0.0.1) failed
2015-02-22T14:04:06.264+0700 reconnect localhost:20000 (127.0.0.1) ok
{
 "set" : "myapp",
 "date" : ISODate("2015-02-22T07:04:06Z"),
 "myState" : 2,
 "syncingTo" : "bluething-PC:20001",
 "members" : [
 {
 "_id" : 0,
 "name" : "bluething-PC:20000",
 "health" : 1,
 "state" : 2,
 "stateStr" : "SECONDARY",
 "uptime" : 30,
 "optime" : Timestamp(1424581294, 1),
 "optimeDate" : ISODate("2015-02-22T05:01:34Z"),
 "infoMessage" : "syncing to: bluething-PC:20001",
 "self" : true
 },
 {
 "_id" : 1,
 "name" : "bluething-PC:20001",
 "health" : 1,
 "state" : 1,
 "stateStr" : "PRIMARY",
 "uptime" : 30,
 "optime" : Timestamp(1424581294, 1),
 "optimeDate" : ISODate("2015-02-22T05:01:34Z"),
 "lastHeartbeat" : ISODate("2015-02-22T07:04:04Z"),
 "lastHeartbeatRecv" : ISODate("2015-02-22T07:04:04Z"),
 "pingMs" : 0,
 "electionTime" : Timestamp(1424588314, 1),
 "electionDate" : ISODate("2015-02-22T06:58:34Z")
 },
 {
 "_id" : 2,
 "name" : "bluething-PC:20002",
 "health" : 1,
 "state" : 7,
 "stateStr" : "ARBITER",
 "uptime" : 30,
 "lastHeartbeat" : ISODate("2015-02-22T07:04:04Z"),
 "lastHeartbeatRecv" : ISODate("2015-02-22T07:04:05Z"),
 "pingMs" : 1
 }
 ],
 "ok" : 1
}

 

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: