From MySQL to MongoDB at Wordnik (Tony Tam)
- 1. MongoSF 4/30/2010From MySQL to MongoDBMigrating a Live ApplicationTony Tam
- 2. What is WordnikProject to track language like GPS for EnglishDictionary is a road block to the languageRoughly 200 new words created dailyLanguage is not staticCapture information about all wordsMeaning is often undefined in traditional senseMachines can determine meaning through analysisNeeds LOTS of data
- 3. Why should You careEvery Developer can use a Robust Language API!Wordnik migrated to MongoDB> 5 Billion documents> 1.2 TBZero application downtimeLearn from our Experience
- 4. WordnikNot just a website!But we have oneLaunched Wordnik entirely on MySQLHit road bumps with insert speed ~4B rows on MyISAMtablesTables locked for 10’s of seconds during insertsBut we need more data!Created elaborate update schemes to work around itLost lots of sleep babysitting servers while researching LT solution
- 5. Wordnik + MongoDBWhat are our storage needs?Database vs. Application LogicNo PK/FK constraintsNo Stored ProceduresConsistency?Lots of R&DTried most all noSQL solutions
- 6. Migrating Storage EnginesMany parts to this effortSetup & AdministrationSoftware DesignOptimizationMany types of data at WordnikCorpusStructured HierarchicalDataUser DataMigrated #1 & #2
- 7. Server InfrastructureWordnik is Heavily Read-onlyMaster / Slave deploymentLooking at replica pairsMongoDB loves system resourcesWordnik runs dedicated boxes to avoid other apps being sent to disk (aka time-out)Memory + Disk = Happy MongoMany X the disk space of MySQLEasy pill to swallow until…
- 8. Server InfrastructurePhysical Hardware2 x 4 core CPU, 32gb RAM, FC SANHad bad luck on VMs(you might not)Disk speed => performance
- 9. Software DesignTwo distinct use cases for MongoDBIdentical structure, different storage engineSame underlying objects, same storage fidelity (largelykey/value)Hierarchical data structureSame underlying objects, document-oriented storage
- 10. Software DesignCreate BasicDBObjects from POJOs and used collection methodsBasicDBObjectdbo = new BasicDBObject("sentence",s.getSentence()) .append("rating",s.getRating()).append(...);ID Generation to manage unique _ID valuesAnalogous to MySQL AutoIncrement behaviorCompatible with MySQL Ids (more later)dbo.append("_ID", getId());collection.save(dbo);Implemented all CRUD methods in DAOSwappable between MongoDB and MySQL at runtime
- 11. Software DesignKey-Value storage use caseEasy as implementing new DAOsSentenceHandlerh = new MongoDBSentenceHandler();Save methods construct BasicDBObject and call save() on collectionImplement same interfaceSame methods against DAO between MySQL and MongoDB versionsData Abstraction 101
- 12. Software DesignWhat about bulk inserts?FAF Queued approachAdd objects to queue, return to callerEvery X seconds, process queueAll objects from same collection are appended to a single List<DBObject>Call collection.insert(…) before 2M charactersReduces network overheadVery fast inserts
- 13. Software DesignHierarchical Data done more elegantlyWordnik Dictionary ModelJava POJOs already had JAXB annotationsPart of public REST apiUsed Mysql12+ tables13 DAOs2500 lines of code50 requests/second uncachedMemcache needed to maintain reasonable speed
- 15. Software DesignMongoDB’s Document Storage let us…Turn the Objects into JSON via Jackson Mapper (fasterxml.com)Call saveSupport all fetch types, enhanced filters1000 requests / secondNo explicit cachingNo less scary code
- 16. Software DesignSaving a complex objectString rawJSON = getMapper().writeValueAsString(veryComplexObject);collection.save(newBasicDBOBject(getId(),JSON.parse(rawJSON));Fetching complex objectBasicDBObjectdbo = cursor.next();ComplexObjectobj = getMapper().readValue(dbo.toString(), ComplexObject.class);No joins, 20x faster
- 17. Migrating DataMigrating => existing data logicUse logic to select DAOs appropriatelyRead from old, write with newGreat system test for MongoDBSentenceHandlermysqlSh = new MySQLSentenceHandler();SentenceHandlermongoSh = new MongoDbSentenceHandler();while(hasMoreData){mongoSh.asyncWrite(mysqlSh.next()); ...}
- 18. Migrating DataWordnik moved 5 billion rows from MySQLSustained 100,000 inserts/secondMigration tool was CPU boundID generation logic, among otherWordnik reads MongoDB fastRead + create java objects @ 250k/second (!)
- 19. Going live to ProductionChoose your use case carefully if migrating incrementallyScary no matter whatTest your perf monitoring system first!Use your DAOs from migrationTurn on MongoDB on one server, monitor, tune (rollback, repeat)Full switch over when comfortable
- 20. Going live to ProductionReally?SentenceHandlerh = null;if(useMongoDb){h = new MongoDbSentenceHandler();}else{h = new MySQLDbSentenceHandler();}return h.find(...);
- 21. Optimizing PerformanceHome-grown connection poolingMaster onlyConnectionManager.getReadWriteConnection()Slave onlyConnectionManager.getReadOnlyConnection()Round-robin all servers, bias on slavesConnectionManager.getConnection()
- 22. Optimizing PerformanceCachingHad complex logic to handle cache invalidationOut-of-process caches are not freeMongoDB loves your RAMLet it do your LRU cache (it will anyway)HardwareDo not skimp on your disk or RAMIndexesSchema-less designEven if no values in any document, needs to read document schema to check
- 23. Optimizing PerformanceDisk spaceSchemaless => schema per document (row)Choose your mappings wisely({veryLongAttributeName:true}) => more disk space than ({vlan:true})
- 24. Optimizing PerformanceA Typical Day at the Office for MongoDBAPI call rate: 47.7 calls/sec
- 25. Other TipsData TypesUse caution when changingDBObjectobj = cur.next();long id = (Long) obj.get(“IWasAnIntOnce”)Attribute namesDon’t change w/o migrating existing data!WTFDMDG????
- 26. What’s next?GridFSStore audio files on diskRequires clustered file system for shared accessCapped Collections (rolling out this week)UGC from MySQL => MongoDBBeg/Bribe 10gen for some Features
- 27. Questions?