Sergej Novik

When data agents fail, they often fail silently - giving confident-sounding answers that are wrong, and it can be hard to figure out what caused the failure. "Building and Evaluating Data Agents" is a new short course created with Snowflake and taught by Anupam Datta and Josh Reini. This course teaches you to build data agents with comprehensive evaluation built in. Skills you'll gain: - Build reliable LLM data agents using the Goal-Plan-Action framework and runtime evaluations that catch failures mid-execution - Use OpenTelemetry tracing and evaluation infrastructure to diagnose exactly where agents fail and systematically improve performance - Orchestrate multi-step workflows across web search, SQL, and document retrieval in LangGraph-based agents The result: visibility into every step of your agent's reasoning, so if something breaks, you have a systematic approach to fix it. Sign up to get started: https://lnkd.in/gVwKuCCQ

6.638

223 Kommentare

Steps here👇the game has completely changed since 2022 - it’s not just leetcode anymore. Here’s what Viktoria Semaan shared that actually worked: ✅ Actually USE the platform (yes, she had to learn Databricks + Spark from scratch) ✅ Nail the coding assignment without ChatGPT crutches ✅ Come with CLEAR impact stories for leadership rounds ✅ Do your homework on every interviewer (LinkedIn stalking is real prep 😅) The technical bar is higher, but the human connection piece is everything. When you can tie your stories to their experience and show genuine interest in their work - that’s when you stand out. 2025 interviews hit different. Preparation is everything 💪 What’s the biggest interview challenge you’re facing right now? Drop it in the comments 👇 #techjobs #interviewtips

187

5 Kommentare

✨ Big news: the 2nd Edition of Data Modeling with Snowflake is officially out today! It's hard to believe that only two years have passed since the first edition, yet so much has changed. New Snowflake features, such as Dynamic and Iceberg tables, have transformed the way data is shared and processed. Additionally, the introduction of Cortex and the semantic layer allows AI to execute queries using natural language, adding even more capabilities. As AI continues to evolve, it's evident that data modeling is playing an increasingly important role in creating scalable and cost-effective data architectures. The rising costs of inefficiency highlight this relevance. However, despite its importance, data modeling remains far from ubiquitous. One major reason data modeling initiatives often fail is not due to technical issues, but rather personal ones: the engineering team often struggles to effectively communicate the business value to stakeholders. My favorite chapter in the second edition addresses this by breaking down business metrics and the ROI of data modeling. It provides data teams with a playbook for engaging stakeholders, persuading the budget committee, and fostering business alignment. Since publishing the first edition, I've heard directly from readers who have used this book to transform their careers and convince colleagues that data modeling is a worthwhile endeavor. I am confident that the second edition will build on that success and provide readers with even more tools to establish a solid foundation for creating efficient and scalable data solutions. A huge thank you to Joe Reis for writing the introduction to this edition and for promoting data modeling within the data community through his ongoing work. If you aren't already following him, be sure to do so, and keep an eye out for his highly anticipated data modeling publication. Also, stay tuned for a release webinar and AMA session in the coming week. I'll be taking questions and giving away a copy of the book as well. Get your copy on Amazon: https://a.co/d/iVak9qF or direct from the publisher: https://lnkd.in/dEjk-Xf3

170

19 Kommentare

What I love about #ApacheIceberg is how many different workloads folks are running on the same copy of data. Graph queries are a really cool example. Instead of copying data into specialized graph databases, PuppyGraph lets you run graph traversals directly on your existing Iceberg tables. Pretty cool stuff. But here's the thing - to get truly great performance, you need to ensure your Iceberg tables are well maintained and optimized. That's where Ryft comes in. We make sure your Iceberg tables are automatically maintained and optimized, no matter what engine you are using. It was terrific to work with Zhenni Wu and Weimo Liu, to create this blog post on how you can use Ryft with PuppyGraph to run graph queries on a fully optimized Iceberg lake! https://lnkd.in/giwnwMuc #ApacheIceberg #GraphQueries

99

2 Kommentare

Death by a thousand defaults on #Databricks. Databricks clusters: latest ML runtime + auto-termination set to 3 days. Dashboards: refresh every few minutes. Alerts: happily running queries on your most expensive cluster. Online tables: syncing more often than anyone needs. Not obvious. Sometimes hidden. Always expensive.

136

10 Kommentare

5 Good Books on Snowflake Best book on Snowflake: https://lnkd.in/epCefdtx Title: Snowflake - The Definitive Guide: Architecting, Designing, and Deploying on the Snowflake Data Cloud Author: Joyce Kay Avila Snowflake data modelling book: https://lnkd.in/ewq3_M9J Title: Data Modeling with Snowflake: A practical guide to accelerating Snowflake development using universal data modeling techniques Author: Serge Gershkovich Snowflake data engineering book: https://lnkd.in/eD33wkhP Title: Snowflake Data Engineering Author: Maja Ferle Snowflake Security: https://lnkd.in/eCChXjiM Title: Snowflake Security: Securing Your Snowflake Data Cloud Author: Ben Herzberg and Yoav Cohen Snowflake Cost and Performance Tuning: https://lnkd.in/e2CEtfni Title: Tuning the Snowflake Data Cloud: Optimizing Your Data Platform to Minimize Cost and Maximize Performance Author: Andrew Carruthers I've read the first two and they are absolute treasure for me. Very useful in doing my job. The other 3 I had a look at their content and it look useful and well written. The last one (Andrew Carruthers) wrote another Snowflake book on marketplace if you are interested. Happy reading, keep learning! My Linkedin articles: https://lnkd.in/eRTNN6GP My blog: https://lnkd.in/eCGzJJCB #Snowflake #DataWarehouse #Data #Book

130

9 Kommentare

It's a controversial opinion but I'm going to go ahead and say it anyway I think dbt Labs was the best thing that ever happened to data engineering Why? Because FOREVER data engineering has been some sort of weird pseudo science living within IT, hidden away from the business, shielded behind firewalls, databases, and a whole lot of politics. While you could say the move to cloud, Snowflake , ETL to ELT made it all possible, what dbt did was blast open the doors for anyone to start doing data engineering Sure - -writing a little bit of SQL and having it run in order didn't seem like much, but it's a HUGE part of data engineering today and it made the entire area WAY more accessible to more people And those people are critical -- I have long talked aboutt he Silo trap - how central data teams create SILOS that mean people with BUSINESS needs have to triage and go to a central team for insights This is fine but not at scale. At scale it creates hugely debilitating and slow workflows that kill projects before they're even getting started That's why we've designed Orchestra not just to be the best orchestration tool but also one that seamlessly fits with dbt. Analysts use dbt Cloud to write and manage dbt. This helps them self serve away from dashboards, and to the transformation layer. But what about self-serving the rest? What about running end to end flows? What about controlling for DQ? What about the infrastructure itself?? Today, that still requires an experienced platform team running an Airflow cluster full-time. WIth Orchestra we want to truly democratise the ability for anyone to run production-grade data pipelines and AI Workflows. I want people to be dangerous. I want everyone to remember their first dbt run You can read more about the combination of dbt Cloud and Orchestra in the link below #dbt #orchestra #dataengineering #analyticsengineering

212

21 Kommentare

Apache Iceberg is attracting a lot of attention in the analytic community, and I encounter an increasing number companies with production apps on it. Iceberg V3 is a big step forward in capabilities, especially around data types. VARIANT and nanosecond timestamps sound obscure to the uninitiated, but they are critical in real-time applications. Here's a nice summary from Google. https://lnkd.in/gVtmsZuq #apache #iceberg #realtime #analytics #opensource

36