Solr consistency and recovery internals
Download as PPTX, PDF4 likes2,244 views
The document presents an overview of Solr's consistency and recovery mechanisms, detailing the processes of leader election and recovery actions in Solr clusters. It addresses both routine and non-routine recovery scenarios, including server crashes and network failures. Additionally, it discusses leader-initiated recovery and the auto-add replica feature, highlighting their benefits and limitations.
Solr consistency and recovery internals
- 1. 1© Cloudera, Inc. All rights reserved. Solr consistency and recovery internals Mano Kovacs | July 13, 2017
- 2. 2© Cloudera, Inc. All rights reserved. Intro • Mano Kovacs • Cloudera Search engineer • Working on “Why is my Solr cluster down?” mysteries. • 15 yrs of dev, high-performant web services, IoT platform • Amature slideshow enthusiast
- 3. 3© Cloudera, Inc. All rights reserved. Agenda • Consistency basics (leaders/follower) • Leader election • When to recover • General recovery (peersync, replication) • Recovery in detail • Leader-Initiated Recovery • Auto Add Replica
- 4. 4© Cloudera, Inc. All rights reserved. Basics • Shards in collection • One leader per shard • Leader gets writes • Replicates
- 5. 5© Cloudera, Inc. All rights reserved. Leader Election • Zookeeper Leader election recipe • Sequential, ephemeral nodes for each replica • The order dictates the leader candidates • First in order becomes leader candidate • Replicas watch the previous candidate to get notified • If leader fails, next in line will be the candidate • Leader candidates follow leader preparation process
- 6. 6© Cloudera, Inc. All rights reserved. Leader Election - leader candidate • On restart: waits all replicas to participate (default 3 mins) • Sync changes from other replicas • Verify last state ACTIVE if not startup • If all were DOWN, shard hangs (SOLR-7065) • Verify there was no error reported (LIR… tbd)
- 7. 7© Cloudera, Inc. All rights reserved. What causes Recovery? • Routine Events •Add or Move Replica - not having the data •Restart (upgrade/tuning) - might missed updates • Not Routine Events •Server crash •Leader •Replica •Network failure (Lose ZK Connection) •Replica partitioned: can access ZK, but not the leader
- 8. 8© Cloudera, Inc. All rights reserved. Recovery (from 30k fts.) • Replaying unfinished updates from tlog • Check if we are synced • If no, “How much am I behind?” • If N (def=100) docs or less • Retrieving delta • Else • Replication: pulling full index • Go ACTIVE
- 9. 9© Cloudera, Inc. All rights reserved. Recovery (from 1000 fts.) • Buffering new updates • So we won’t get behind over and over again • Waiting leader to notice us • Otherwise we don’t get updates • Replay buffered updates • Hopefully replay catches up with incoming updates
- 10. 10© Cloudera, Inc. All rights reserved. Recovery (from 100 fts.) • Updates are versioned • Timestamp+counter • PeerSync: last N updates by version • Index has fingerprint (hash of doc versions) • If there is other updates missing, fingerprint will fail • Consistency safety net if others fail
- 11. 11© Cloudera, Inc. All rights reserved. Leader-Initiated Recovery • Partitioning Leader from Replica, but not ZK • Leader will send recovery requests to replica (with retries) • If Replica went down, it will do normal recovery process anyway • If replica is partitioned and up, it will still serve stale reads :(
- 12. 12© Cloudera, Inc. All rights reserved. LIR problems - SOLR-9555 • Race condition between LIR and standard Recovery • Mike Drob’s patch is almost done • Solves problem with partitioned replicas too with ZK watches
- 13. 13© Cloudera, Inc. All rights reserved. AutoAddReplica • Using shared file system (e.g. HDFS) • Provides durability • Instances share index folders • Move cores to live nodes on failure • Use same index folder • Pros • Durability with rep factor 1 • Handle perm. node loss • Cons • Still no HA and read scalability if using single replica • Lots of fix from Mark Miller lately
- 14. 14© Cloudera, Inc. All rights reserved. Summary • Details about SolrCloud cluster • Help to improve! • PlantUML is cool to document
- 15. 15© Cloudera, Inc. All rights reserved. Thank you E: [email protected] T: @manokovacs