SQLBits VI - Improving database performance by removing the database
0 likes391 views
The document discusses the challenges faced by database professionals, including rising demands for performance, costs, and scalability, while emphasizing the need for adaptation in database management practices. It critiques the limitations of SQL in handling modern data complexities and suggests that alternative approaches and technologies may provide better solutions. Ultimately, it calls for database experts to reclaim their influence and embrace change in the management of data.
SQLBits VI - Improving database performance by removing the database
- 1. Improving Database Performance by Removing the DatabaseSimon Munro
- 2. This is not about NoSQLIt is about why NoSQL won’t dieFor NoSQL, watch the video from SQLBits Goes West
- 3. Data GrowthEverything ElseIsn’t it a great time to be a data dude?
- 4. Information is powerInformation is on the increaseWe administer the informationWe have the power!Viva DBA, Viva!
- 5. So why are we so stressed?
- 6. We need to handle increasing demandsData VolumesTransactional VolumesUser VolumesTime to MarketResponsiveness
- 7. With better levels of serviceAvailabilityReliabilitySecurityFunctionalityPerformance(Yeah right… and with the same fake smiles)
- 8. Rising CostsSpecialisedHardwareLicensing CostsVendor Lock-inOperational Costs
- 9. It all takes more effortDesign EffortOperational EffortCollaboration EffortProcurementSetupMigrationDecommissioning
- 10. But we have to lower costsRecession ImpactFocus on EfficiencyStaff cutsMarginal business casesInfrequent Upgrades
- 11. What can change?Networking Topologies?Hardware Infrastructure?Storage?Database Platforms?Application Architectures?Demands?Requirements?
- 12. Do we want it to change?It seems to workIt is familiarWe have the skillsRisk is lowWe have the infrastructureDoes it really work?
- 14. Misused Designs
- 15. SQL as a Good DesignDe-facto data storage mechanism(Fairly) well understood patternsDatabase structuresQueryingClose to classic relational modelOptimised embedded languages
- 16. SQL as Misused DesignAPI is far to open‘Post relational’ datatypesSuboptimal for some domains
- 17. Works for meWhy?SQL is simply fantastic!That is what is was designed to doIt’s always been that wayHave you looked at alternatives?Have you checked your assumptions?
- 18. SQL Patterns have become data management patternsBackupQuerySecurityACIDData modelsApplication developmentTHIS is how we work with data!
- 19. ScalabilityIt is hardand expensiveand unfamiliarand those are just the people problems!
- 20. Scalability and NoSQLLack of SQL scalability out is the popular NoSQL argumentActually, most NoSQL databases do not scale out eitherjamesgolick.comCouchDBRedisTokyo cabinetMemcacheDBBerkeley DBCassandraRiakVoldemort
- 21. Scalability is more than an engineering problemOperational processesMaintenanceSkillsPartnershipsLegal
- 23. The core flaw in SQL oriented designMSDNhttp://technet.microsoft.com/en-us/library/cc966500.aspx“The log records associated with the transaction (and all previous log records) must be written to stable storage. The transaction is not considered committed until the log records are correctly flushed to stable media. (Log is hardened.)”
- 24. Stack of TurtlesProcessorMemoryDisk ControllerDisk
- 25. More turtles!ProcessorMemorySan ControllerNetworkSystem ManagementSwitchFirewallSAN ControllerBackupSANEncryptionFibre NetworkRAID ControllerVirtualisationDiskIs our performance bottleneck based on this?Monitoring
- 26. Storage Gets BetterTape is deadDisk is tapeFlash is diskSSDs to the rescue!Memory is cheapWhat if we kept everything in memory?
- 27. Database Ivory Towers That Piss Me Off
- 28. All data should be in the databaseThe CultureQ: How long do you need tokeep this data?A: 5 yearsOriginal atomic records need tobe stored for future analysisKeeping it in SQL is best sothat it can be queried
- 29. All data should be in the databaseThe RealityThe business value of datais misunderstoodThe cost of data retention is hiddenUsers cannot query dataA lot of data is not in thedatabase anyway
- 30. Single Version of the Truth(One fact, one place, once)The CultureMaster Data ManagementNormalisationOnly editable in this databaseRemoval of duplicatesIntegration is tedious
- 31. Single Version of the Truth(One fact, one place, once)The RealityA complete mythData always lands up everywhereSystem integration duplicates dataMDM effort is highThere are a lot of spreadsheetsData is inherently temporal
- 32. All processing should be done in the batch runThe CultureSingle databaseSQL Database as a source for all dataAggregation load is to high for transactional systemsIt is optimal and fair for all departments
- 33. All processing should be done in the batch runThe RealityBatches always tend toward usingup all available timeBatch code is the worstBatch failures are a source ofPanic, risk and stressOperational costs for batchruns are high
- 34. QueryabilityThe CultureAll fields are queryableUser demand... apparentlyThis is what the relational database is for
- 35. QueryabilityThe RealityPerformance issues limitsearches to key fieldsOther structures are created tomake querying easierRows, columns and tables are an abstraction anyway
- 36. Required by AuditorsThe Culture“We need an audit trail”“Audit requires that we…”
- 37. Required by AuditorsThe RealityReal life auditors are seldom making requestsWhat are auditors doing specifying IT architectures anyway?Auditors are change averse
- 38. Required by RegulationsThe Culture“We can’t do that because of regulations”“We do it like this for regulatory reasons”
- 39. Required by RegulationsThe RealityRegulations are difficult to understandRegulations are full of legalese and contradictionsMost regulatory requirements are based on myths
- 40. ConsistencyThe CultureEvery database operation needsto be within a transactional contextWe have to ensure that in theevent of a failure that data is correctData has to be in a consistent state before it can be usedAll clients executing a simultaneous query should get the same result
- 41. ConsistencyThe RealityConsistency is impossible in aDistributed environmentThe Internet is a distributedenvironmentGiven the choice, business would probably spend theirmoney elsewhereEven the most consistent datamay not reflect reality
- 43. In memory dataHosted ServiceService InterfaceMessage BusClientsCan we just store data in memory?
- 44. What are the patterns?We already have themMainframes, Swift, Reuters, TradingKnowledge is hidden or rustyDiverted to SOA for a whileBeing revived by the cloudAlready being used withoutDBA knowing
- 45. The Influence of the CloudSomeone else can deal with the stack of turtlesCannot make assumptionsabout availability of a specific processNo control over underlying hardware Applications are failure aware
- 46. SQL Removal TechniquesChange your approach to dealing with dataChange your applicationarchitecturesChange how business treats dataWork with other disciplines (e.g. developers and compliance)
- 47. Changing Approach to DataCacheRead-only data storesSpecialised Data StoresMain Memory DatabasesPre-emptive archiving
- 48. Changing Approach to Application ArchitecturesMessage OrientationEventual ConsistencyStore and process data as Close to source as possibleDesign for service degradationApology based computing
- 49. An Example : Changing Batch JobsThe Scenario:Users require access to Summaries of a large transactional databaseThe transactional databaseis under load so summariesare run in the overnight batchjob
- 50. The ResultReporting DatabaseTransaction DatabaseOccasional RefreshWorkflowEngineReliable MessagingMessage BusTransaction Source
- 51. Resistance to ChangeIncumbent InvestmentRisk of FailureFear of FailureJobs ProtectionEgos
- 52. Vendor InterestsQ3 2010New softwarelicencesUS$1.7BUpdates and product support US$3.2B2009 Annual ReportRevenue US$ 58BClient (Windows) US$ 14.7BServer and Tools US$ 14.1B
- 53. It’s Not EasyHigh Engineering CostLack of Design PatternsLack of ExperienceRisk of Poor ImplementationSponsor SupportNo Big Vendor
- 54. Change ThisFocus hereKeep thisWho Cares?
- 55. The Data API is changingData demands are explodingWe are wasting effort, money and sanitySQL doesn’t do everythingAlternatives can make a big difference to solutionsGet involved in the debateClose the gap to ‘them’Advance the state of the artBe less restrictiveEmbrace changeUnderstand the business needsIts time for database professionals to take back the database and stop people pissing all over our domain
- 56. What kind of trusted advisor are you?
- 57. Fill in feedbackVisit the sponsors pleaseLook for the video uploadsSlides (and trump cards) on simonmunro.com and SQLBits soon