MongoDB .local Bengaluru 2019: Becoming an Ops Manager Backup Superhero!
1 like338 views
The document provides detailed guidance on diagnosing and troubleshooting issues related to the MongoDB Ops Manager backup process. It outlines common problems such as slow backups and connectivity issues, along with step-by-step instructions on how to address them. Additionally, it includes examples of error logs and metrics for monitoring the backup process effectively.
![[2019/05/05 21:23:35.245] [agent.error]
[components/agent.go:manageReplicaSets:202] Error starting syncs. RsId:
myReplicaSetError getting a valid session. Err: ip-172-31-9-252.ap-southeast-
2.compute.internal:27020 error dialing: Failed to make a direct connection. Address:
ip-172-31-9-252.ap-southeast-2.compute.internal:27020 Err: no reachable servers
…
error dialing primary: Error dialing to replica set. RsId: myReplicaSet Hosts: [ip-172-
31-9-252.ap-southeast-2.compute.internal:27020] Err: no reachable servers
Connectivity issue to source](https://image.slidesharecdn.com/becoming-a-backup-superhero-publish-190911193615/85/MongoDB-local-Bengaluru-2019-Becoming-an-Ops-Manager-Backup-Superhero-32-320.jpg)
![[2019/05/19 04:13:15.191]
[agent.sync.myReplicaSet.5ce0d7a80bb131032173e503.error]
[components/sync.go:Run:246] Error sending sync slices.
Error syncing a collection. Namespace: `database.collection` Err:
operation was interrupted
Connectivity issue to source](https://image.slidesharecdn.com/becoming-a-backup-superhero-publish-190911193615/85/MongoDB-local-Bengaluru-2019-Becoming-an-Ops-Manager-Backup-Superhero-38-320.jpg)
![[2019/05/13 12:18:08.993]
Pushing sync slice. Sync ID: ObjectIdHex("5cd958960bb131032172269e")
Namespace: test.foo Slice #356
[2019/05/13 12:18:27.993]
Finished pushing sync slice. Sync ID:
ObjectIdHex("5cd958960bb131032172269e") Namespace: test.foo Slice #356
Request Time 18589ms
Slow transfer to Sync Store](https://image.slidesharecdn.com/becoming-a-backup-superhero-publish-190911193615/85/MongoDB-local-Bengaluru-2019-Becoming-an-Ops-Manager-Backup-Superhero-40-320.jpg)
![[2019/05/02 12:37:21.357]
[agent.sync.production.5cd4d1bb0bb1317ecc9d0a17.info]
[components/mothership.go:PushSyncSlice:320] Total Slice #1003 - server
syncStore is full. 11th attempt. Will resend this slice again soon.
Issues writing to Sync Store](https://image.slidesharecdn.com/becoming-a-backup-superhero-publish-190911193615/85/MongoDB-local-Bengaluru-2019-Becoming-an-Ops-Manager-Backup-Superhero-41-320.jpg)
![2018-02-10T20:22:06.780-0600 [class com.xgen.svc.brs.svc.AssignmentThread
=>
Space used: 2,252,138,352,640 bytes,
Space free: 324,684,738,560 bytes
2018-02-10T20:22:06.782-0600 [class com.xgen.svc.brs.svc.AssignmentThread
=> backup.assignment.5ccbd7d20bb1317ecc8256ee.rs.production] ERROR
backup.assignment.5ccbd7d20bb1317ecc8256ee.rs.production
[assignmentFailed:527]
Assignment failed: No Daemon found with suitable conditions.
Filesize: 402.7 GB Oplog Churn: 0.0 GB/hr RequireSSD: false, rsId=production,
groupId=5ccbd7d20bb1317ecc8256ee
No suitable daemon](https://image.slidesharecdn.com/becoming-a-backup-superhero-publish-190911193615/85/MongoDB-local-Bengaluru-2019-Becoming-an-Ops-Manager-Backup-Superhero-45-320.jpg)
![[2019/05/13 12:20:12.513] [agent.oplog.myReplicaSet.debug]
[components/agent.go:func1:359] Successfully finished pushing oplog slice. {ts:
1557740046:1} -> {ts: 1557740106:1} Num slices: 1 Num docs: 5. Request
Time 11ms
[2019/05/13 13:12:39.277] [agent.oplog.myReplicaSet.debug]
[components/agent.go:func1:359] Successfully finished pushing oplog slice. {ts:
1557752912:192} -> {ts: 1557752912:3292} Num slices: 1 Num docs: 3100.
Request Time 1004ms
Backup agent too slow](https://image.slidesharecdn.com/becoming-a-backup-superhero-publish-190911193615/85/MongoDB-local-Bengaluru-2019-Becoming-an-Ops-Manager-Backup-Superhero-55-320.jpg)
![2019-05-18T11:38:29.170-0500 [Daemon #1: class
com.xgen.svc.brs.job.ApplyOpsJob] DEBUG backup.jobs.
5cd958960bb131032172269e.production [OplogSliceCompiler.java.work:270]
- OplogSlice: Range: 1557217382:2778 -> 1557217385:6790; NumDocs:
22930
2019-05-18T11:38:38.817-0500 [Daemon #1: class
com.xgen.svc.brs.job.ApplyOpsJob] DEBUG backup.jobs.
5cd958960bb131032172269e.production [OplogSliceCompiler.java.work:421]
- Oplogs to apply: 22930; Skipped before: 0; Skipped from overlap: 0, Skipped
after: 0
Daemon can’t keep up](https://image.slidesharecdn.com/becoming-a-backup-superhero-publish-190911193615/85/MongoDB-local-Bengaluru-2019-Becoming-an-Ops-Manager-Backup-Superhero-61-320.jpg)
![2019-05-18T13:00:54.809-0500 [Daemon #1: class
com.xgen.svc.brs.job.SnapshotJob] INFO backup.jobs.
5cd958960bb131032172269e.production
[SnapshotJob.java.initiateSnapshot:123] - Starting a snapshot job.
2019-05-18T15:01:07.072-0500 [Daemon #1: class
com.xgen.svc.brs.job.SnapshotJob =>
5cd958960bb131032172269e/production ] DEBUG
com.xgen.svc.brs.grid.Daemon [Daemon.java.iterate:138] - Job: class
com.xgen.svc.brs.job.SnapshotJob finished. JobResult: OK.
Snapshot is taking too long to
complete](https://image.slidesharecdn.com/becoming-a-backup-superhero-publish-190911193615/85/MongoDB-local-Bengaluru-2019-Becoming-an-Ops-Manager-Backup-Superhero-62-320.jpg)
![[2019/05/10 06:39:51.135] [agent.oplog.myReplicaSet.warn]
[components/oplog.go:TailOplog:253] Bad match. Expected: {ts:
1557469781:1 h: -786638763670375692, t: 1} Received: {ts: 1557470350:1 h:
-8704482524044120259, t: -1}
…
[2019/05/10 06:40:51.152] [agent.commonPoints.myReplicaSet.warn]
[components/rollback.go:Run:127] Failed to find a common point.
Lost oplog tail](https://image.slidesharecdn.com/becoming-a-backup-superhero-publish-190911193615/85/MongoDB-local-Bengaluru-2019-Becoming-an-Ops-Manager-Backup-Superhero-69-320.jpg)
![In the daemon logs:
Error applying ops. Requesting resync.
In the HEAD DB logs:
2019-05-21T03:13:53.140+0000 [conn3] replication update of non-mod failed:
{ ts: Timestamp 1558162386103|12, h: 361936184013300100, v: 2, op: "u",
ns: ”database.collection", o2: {actual update here}
Error in applyOps](https://image.slidesharecdn.com/becoming-a-backup-superhero-publish-190911193615/85/MongoDB-local-Bengaluru-2019-Becoming-an-Ops-Manager-Backup-Superhero-70-320.jpg)
![In the daemon logs:
2019-05-24T00:01:17.522+0000 [Daemon #3: class
com.xgen.svc.brs.job.ApplyOpsJob =>
5ccbd7d20bb1317ecc8256ee/myReplicaSet] DEBUG
backup.jobs.5ccbd7d20bb1317ecc8256ee.myReplicaSet
[ReplicaSetJob.java.startMongo:127] - MongodManager - Requested Version:
4.0.9, Matching Version: 4.0.9, Matching Path: /opt/mongodb/mms/mongodb-
releases/mongodb-linux-x86_64-amazon-4.0.9/bin/mongod, HEAD Path:
/backup/5ccbd7d20bb1317ecc8256ee/myReplicaSet/head/
HEAD db logs](https://image.slidesharecdn.com/becoming-a-backup-superhero-publish-190911193615/85/MongoDB-local-Bengaluru-2019-Becoming-an-Ops-Manager-Backup-Superhero-71-320.jpg)
MongoDB .local Bengaluru 2019: Becoming an Ops Manager Backup Superhero!
- 1. Dinesh Chander Becoming an Ops Manager Backup Superhero! dineshchandernsit
- 2. Dinesh Chander Team Lead, Technical Services, MongoDB
- 3. About Me
- 4. Ops Manager - Capabilities • On-prem software – comes with Enterprise Advanced • Monitoring MongoDB deployments • Backup • Management of MongoDB • Upgrades • Index builds • Cluster administration
- 5. Ops Manager - Backup
- 6. Ops Manager - Backup
- 7. Ops Manager - Backup - Uh oh!
- 8. Ops Manager - Backup - Uh oh!
- 9. Ops Manager - Backup - Uh oh!
- 10. Ops Manager - Backup - Uh oh!
- 11. My backups are stuck My backups are too slow I don’t know what’s happening What’s wrong?
- 13. Technical Services can save the day!
- 14. Agenda • How Ops Manager Backup works • How to diagnose some common errors
- 15. How OM Backup works
- 16. How OM Backup works 1. Initial sync 2. Oplog apply 3. Snapshot
- 17. Initial sync Source Backup agent HTTP service Sync store Oplog store Backup daemon HEAD DB Sync slices Oplog slices Ops Manager Source HEAD DB
- 18. Oplog apply Source Backup agent Oplog store Backup daemon Oplog slices HTTP service HEAD DB
- 19. Snapshot Oplog store Backup daemon Oplog slices HEAD DB files Break up files into blocks MongoDB Blockstore Filesystem store S3HEAD DB
- 22. Insufficient oplog size Source Backup agent HTTP service Backup daemon HEAD DB Sync slices Oplog slices Ops Manager Sync store Oplog store
- 24. Insufficient oplog size - how to diagnose ❑ Look at metrics for Oplog Window ❑ Wait for the oplog window to increase naturally ❑ Increase size of oplog
- 27. Starting… (forever) Source Backup agent HTTP service Backup daemon Sync slices Oplog slices HEAD DBSync store Oplog store
- 29. Starting… (forever) – alerts
- 32. [2019/05/05 21:23:35.245] [agent.error] [components/agent.go:manageReplicaSets:202] Error starting syncs. RsId: myReplicaSetError getting a valid session. Err: ip-172-31-9-252.ap-southeast- 2.compute.internal:27020 error dialing: Failed to make a direct connection. Address: ip-172-31-9-252.ap-southeast-2.compute.internal:27020 Err: no reachable servers … error dialing primary: Error dialing to replica set. RsId: myReplicaSet Hosts: [ip-172- 31-9-252.ap-southeast-2.compute.internal:27020] Err: no reachable servers Connectivity issue to source
- 33. Sending collection information. Sync ID: ObjectIdHex("5cd4d1bb0bb1317ecc9d0a17") Namespace: admin.system.version Sending collection information. Sync ID: ObjectIdHex("5cd4d1bb0bb1317ecc9d0a17") Namespace: foo.bar Sending numStreamingNamespaces information. Sync ID: ObjectIdHex("5cd4d1bb0bb1317ecc9d0a17") number: 2 Finished namespace collectionInfo. Large number of collections
- 34. Starting… (forever) – how to diagnose ❑ Check backup agent is functional ❑ Connectivity issues to source – check backup agent logs ❑ Still gathering collection info – do you have a lot of collections?
- 37. Transferring… (forever) Source Backup agent HTTP service Backup daemon Sync slices Oplog slices HEAD DBSync store Oplog store
- 38. [2019/05/19 04:13:15.191] [agent.sync.myReplicaSet.5ce0d7a80bb131032173e503.error] [components/sync.go:Run:246] Error sending sync slices. Error syncing a collection. Namespace: `database.collection` Err: operation was interrupted Connectivity issue to source
- 39. Transferring… (forever) Source Backup agent HTTP service Sync store Oplog store Backup daemon Sync slices Oplog slices HEAD DB
- 40. [2019/05/13 12:18:08.993] Pushing sync slice. Sync ID: ObjectIdHex("5cd958960bb131032172269e") Namespace: test.foo Slice #356 [2019/05/13 12:18:27.993] Finished pushing sync slice. Sync ID: ObjectIdHex("5cd958960bb131032172269e") Namespace: test.foo Slice #356 Request Time 18589ms Slow transfer to Sync Store
- 41. [2019/05/02 12:37:21.357] [agent.sync.production.5cd4d1bb0bb1317ecc9d0a17.info] [components/mothership.go:PushSyncSlice:320] Total Slice #1003 - server syncStore is full. 11th attempt. Will resend this slice again soon. Issues writing to Sync Store
- 43. Transferring… (forever) Source Backup agent HTTP service Backup daemon Sync slices Oplog slices HEAD DBSync store Oplog store
- 45. 2018-02-10T20:22:06.780-0600 [class com.xgen.svc.brs.svc.AssignmentThread => Space used: 2,252,138,352,640 bytes, Space free: 324,684,738,560 bytes 2018-02-10T20:22:06.782-0600 [class com.xgen.svc.brs.svc.AssignmentThread => backup.assignment.5ccbd7d20bb1317ecc8256ee.rs.production] ERROR backup.assignment.5ccbd7d20bb1317ecc8256ee.rs.production [assignmentFailed:527] Assignment failed: No Daemon found with suitable conditions. Filesize: 402.7 GB Oplog Churn: 0.0 GB/hr RequireSSD: false, rsId=production, groupId=5ccbd7d20bb1317ecc8256ee No suitable daemon
- 46. Missing binaries Failed to decompress mongodb-linux-x86_64-rhel70-4.0.9.tgz to /opt/mongodb/mms/mongodb-releases java.io.IOException: No space left on device Could not find appropriate mongod in /opt/mongodb/mongodb-releases/, versions available to MMS: 3.4.3, 3.4.4, 3.4.2, 3.4.9. Expecting version 3.6.12 or greater, module preference: enterprisePreferred
- 47. Other issues on the daemon • Missing prerequisites for MongoDB Enterprise • Problems with the HEAD database • Corruption • Many collections (which slows mongod startup) • Resource contention
- 48. ❑ Check backup agent logs ❑ Problems connecting to source? ❑ Problems sending to sync store? ❑ Is the job assigned to a daemon? ❑ Check backup daemon logs ❑ Is the daemon working on the job? Starting… (forever) – how to diagnose
- 49. Oplog behind
- 50. Oplog behind
- 51. Oplog behind
- 52. Oplog behind Source Backup agent Oplog store Backup daemon Oplog slices HTTP service HEAD DB
- 53. Agent down
- 54. Agent down
- 55. [2019/05/13 12:20:12.513] [agent.oplog.myReplicaSet.debug] [components/agent.go:func1:359] Successfully finished pushing oplog slice. {ts: 1557740046:1} -> {ts: 1557740106:1} Num slices: 1 Num docs: 5. Request Time 11ms [2019/05/13 13:12:39.277] [agent.oplog.myReplicaSet.debug] [components/agent.go:func1:359] Successfully finished pushing oplog slice. {ts: 1557752912:192} -> {ts: 1557752912:3292} Num slices: 1 Num docs: 3100. Request Time 1004ms Backup agent too slow
- 56. Oplog behind – how to diagnose ❑ Check agents page ❑ Check backup agent logs ❑ Check network to oplog store ❑ Check health of oplog store
- 57. Snapshot behind
- 58. Snapshot behind
- 59. Snapshot behind
- 60. Snapshot behind Oplog store Backup daemon Oplog slices HEAD DB files Break up files into blocks MongoDB Blockstore Filesystem store S3HEAD DB
- 61. 2019-05-18T11:38:29.170-0500 [Daemon #1: class com.xgen.svc.brs.job.ApplyOpsJob] DEBUG backup.jobs. 5cd958960bb131032172269e.production [OplogSliceCompiler.java.work:270] - OplogSlice: Range: 1557217382:2778 -> 1557217385:6790; NumDocs: 22930 2019-05-18T11:38:38.817-0500 [Daemon #1: class com.xgen.svc.brs.job.ApplyOpsJob] DEBUG backup.jobs. 5cd958960bb131032172269e.production [OplogSliceCompiler.java.work:421] - Oplogs to apply: 22930; Skipped before: 0; Skipped from overlap: 0, Skipped after: 0 Daemon can’t keep up
- 62. 2019-05-18T13:00:54.809-0500 [Daemon #1: class com.xgen.svc.brs.job.SnapshotJob] INFO backup.jobs. 5cd958960bb131032172269e.production [SnapshotJob.java.initiateSnapshot:123] - Starting a snapshot job. 2019-05-18T15:01:07.072-0500 [Daemon #1: class com.xgen.svc.brs.job.SnapshotJob => 5cd958960bb131032172269e/production ] DEBUG com.xgen.svc.brs.grid.Daemon [Daemon.java.iterate:138] - Job: class com.xgen.svc.brs.job.SnapshotJob finished. JobResult: OK. Snapshot is taking too long to complete
- 63. • Change in data • Many updates - less deduplication (if applicable) • Many inserts - more data to save • Network issue • Storage speed issue Snapshot is taking too long to complete
- 64. Snapshot behind – how to diagnose ❑ Did the daemon start the snapshot? ❑ Check the logs to see why it’s falling behind ❑ Is the snapshot taking a long time? ❑ Storage slowness ❑ Network slowness
- 65. Needs resync
- 66. Needs resync
- 67. Needs resync
- 68. Needs resync Source Backup agent HTTP service Oplog store Backup daemon Oplog slices HEAD DB
- 69. [2019/05/10 06:39:51.135] [agent.oplog.myReplicaSet.warn] [components/oplog.go:TailOplog:253] Bad match. Expected: {ts: 1557469781:1 h: -786638763670375692, t: 1} Received: {ts: 1557470350:1 h: -8704482524044120259, t: -1} … [2019/05/10 06:40:51.152] [agent.commonPoints.myReplicaSet.warn] [components/rollback.go:Run:127] Failed to find a common point. Lost oplog tail
- 70. In the daemon logs: Error applying ops. Requesting resync. In the HEAD DB logs: 2019-05-21T03:13:53.140+0000 [conn3] replication update of non-mod failed: { ts: Timestamp 1558162386103|12, h: 361936184013300100, v: 2, op: "u", ns: ”database.collection", o2: {actual update here} Error in applyOps
- 71. In the daemon logs: 2019-05-24T00:01:17.522+0000 [Daemon #3: class com.xgen.svc.brs.job.ApplyOpsJob => 5ccbd7d20bb1317ecc8256ee/myReplicaSet] DEBUG backup.jobs.5ccbd7d20bb1317ecc8256ee.myReplicaSet [ReplicaSetJob.java.startMongo:127] - MongodManager - Requested Version: 4.0.9, Matching Version: 4.0.9, Matching Path: /opt/mongodb/mms/mongodb- releases/mongodb-linux-x86_64-amazon-4.0.9/bin/mongod, HEAD Path: /backup/5ccbd7d20bb1317ecc8256ee/myReplicaSet/head/ HEAD db logs
- 72. Needs resync – how to diagnose ❑ Resync the backup to get it running again ❑ Backup agent initiated ❑ Why did it fail? Slowness? Rollback? ❑ Backup daemon initiated ❑ What operation did it fail on?
- 73. Summary
- 74. Investigation – where in the process? ❑ Where in the process am I stuck? ❑ Initial sync ❑ Oplog apply ❑ Snapshot
- 75. Investigation – which component? ❑ Which components are involved? ❑ Source replica set ❑ Backup agent ❑ Sync store / oplog store ❑ Backup daemon ❑ HEAD DB ❑ Snapshot storage
- 76. Investigation – what to look at? ❑ Backup agent logs ❑ Backup daemon logs ❑ HEAD DB logs ❑ Monitoring metrics ❑ Storage ❑ Network ❑ General utilisation
- 77. Investigation – I’m still stuck! • Open a support ticket and we’ll help you out! • Make sure you include the relevant data • Logs • Backup daemon • Backup agent • Screenshots • The diagnostic archive
- 81. Dinesh Chander - Team Lead, Technical Services Thank You! Thank You