Per Berglund

When studying database internals at university, I always thought the most important part of building a production query planner is getting the cost-based optimizations right. At TUM, there's actually a whole course that focuses on cost-based join reordering. It's based on Guido Moerkotte's great book "Building Query Compilers". Here are the slides: https://lnkd.in/dcMt56M3 🎓 In that light, rule-based optimizations always seemed like the more basic part of query planning. In the past years at Firebolt my perspective on that has changed a lot. Especially with machine-generated SQL, we often see tons of redundancy that a (very) good rule-based optimizer should eliminate before cost-based strategies even kick in. Andrés wrote a great blog post about how we tackle this at Firebolt through redundant join elimination: https://lnkd.in/dGPU9_dM This is a great piece of writing if you want to learn more about how industry query planners work.

554

17 kommentarer

Accidentally discovered a new way to work with GitHub Copilot today: write new tests based off test coverage reports. Copilot (or I) run this command to generate a report of all the uncovered lines in the current diff: pytest --cov --cov-report=xml && \ diff-cover coverage.xml --html-report coverage_report.html && \ open coverage_report.html Then Copilot reviews the report, summarizes the uncovered situations, and proposes new test cases to add, or if the code is truly unreachable, suggests deleting it. Just another reason why you should add diff-cover (or similar tool) to your projects!

115

4 kommentarer

This is quality work. So glad to see compiler team digging in and getting a richer picture than “build fast” or “build slow”. I definitely think we need to focus on the perception of rust as having slow builds (and the reality it’s based on), but to do that, we need to be clear on what workflows and use patterns we are targeting. The fix for fast CI turnaround is not the same as the fix for fast iteration on a broken test, and iterating on a test is not the same as debugging a type error. https://lnkd.in/eSKkNgXb

98

Dbt + Fivetran = Open wallet, closed source? Let me be clear, dbt Core is an excellent tool. It has had a major positive impact on the data industry, creating the discipline of Analytics Engineering and helping thousands of teams improve their work. The community around it is a great one. However, the recent acquisition by Fivetran marks a significant shift. For me, it signals the end of the "Modern Data Stack" era as we have known it. This is likely not a product story, it's a financial one, a consolidation aimed at creating value for its shareholders, not necessarily for the open-source community that built its foundation. The signs are quite clear: 1. The most telling signal is that Fivetran acquired and was (according to rumors) already using SQLMesh internally before this deal. This suggests the acquisition was not primarily about technology, but about acquiring dbt's large user base. 2. The new marketing term "open data infrastructure" seems to refer to open formats like Iceberg, not open-source tools. This strategy allows compatibility at the data layer while creating a more closed, proprietary tooling platform. 3. The official statement is revealing: "dbt Core will remain open source...and continue to be developed with and for the community." The word "with" is interesting. It could imply a shift where the community is expected to take on more development responsibility, while the company focuses on its commercial products. For those of us managing self-hosted dbt Core platforms, this creates a clear strategic risk. We are now tied to a vendor whose commercial model can introduce significant cost uncertainty, and the future of the open-source tool is uncertain. But this situation also presents an opportunity. The potential for the community to take a larger role in dbt Core's development could be a positive thing. For years, we have been a strong user community. Now, we have the chance to become a more active developer community. This requires us to think more strategically about building for portability, exploring true alternatives, and shifting our identity beyond a single tool. dbt Core was a catalyst for our industry. The real value, however, is in the community's talent and expertise. The period of VC-funded, open-core growth is changing. Now it's time for a more resilient, practitioner-driven approach to our data stacks. I'm curious to hear how others see it. How is your team de-risking your data transformation layer for the future? #dbt #Fivetran #AnalyticsEngineering

127

12 kommentarer

Choosing a framework to measure software development can be a difficult and confusing step, but it doesn’t have to be. We've just published an article that aims to guide you in selecting the right framework for your team for right now. Looking forward to your feedback and hearing how your team and organization are approaching this topic! https://lnkd.in/eUNyBvdr #DORA #GBGB #metrics

38