Rubén Canosa Pérez

Death by a thousand defaults on #Databricks. Databricks clusters: latest ML runtime + auto-termination set to 3 days. Dashboards: refresh every few minutes. Alerts: happily running queries on your most expensive cluster. Online tables: syncing more often than anyone needs. Not obvious. Sometimes hidden. Always expensive.

136

10 comentarios

Delta Lake 4.0 now supports Variant data type designed for working with semi-structured data like JSON with evolving schemas! Key capabilities include: 🔹 𝗦𝗰𝗵𝗲𝗺𝗮-𝗼𝗻-𝗿𝗲𝗮𝗱: No need to define a fixed schema in advance. Structured data can be extracted at query time. 🔹 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝘀𝘁𝗼𝗿𝗮𝗴𝗲: Dynamic payloads are stored using high-performance encoding for better storage efficiency. 🔹 𝗦𝗵𝗿𝗲𝗱𝗱𝗲𝗱 𝗩𝗮𝗿𝗶𝗮𝗻𝘁𝘀 (𝗣𝗿𝗲𝘃𝗶𝗲𝘄): Allows extraction of specific columns from variant data and collection of statistics to speed up queries and file skipping. This feature was developed collaboratively with contributions from Apache Parquet, Apache Arrow, Apache Iceberg, #DeltaLake, Apache Spark, and others, supporting integration across the broader data ecosystem. 🔗 For more details, see the full release notes: https://lnkd.in/edemXCS5 #opensource #oss #linuxfoundation #variantdata

223

6 comentarios

Databricks' has two fairly new account offerings: The "Free Edition" & "Trial Accounts". Both are good for learning how to use Databricks, but they have different intended audiences. In this brief video, Greg Kroleski from Databricks' "Money Team" explains some of the key differences between these two offerings in a very practical way. Link: https://lnkd.in/eERbd7Ni

39

1 comentario

OpenAI has announced a $100M partnership with Databricks. What this really means is that OpenAI’s most advanced models will now be natively available inside the Databricks ecosystem, across both the Data Intelligence Platform and Agent Bricks. With 20,000+ Databricks customers already working with secure enterprise data on the platform, this move feels like a natural step forward. The impact is clear: enterprises won’t need to wrestle with complex integrations or data movement anymore. They’ll be able to experiment, deploy, and scale AI agents right where their data already lives, while still keeping performance and governance intact. It’s a big milestone in bringing frontier AI closer to real-world business use cases ✅ 🚨Last Sunday we created 4 End-To-End industry grade projects on Databricks in my NEWLY UPGRADED "Complete Data Engineering With Azure (5.0) - Basic To Advance" BootCAMP 🌟 Admission Open Now 🎯 Join From Here - https://lnkd.in/gsdQus7U ➡️ Dedicated Placement Assistance & Doubt Support ➡️ Call/WhatsApp us at (+91) 9893181542 Cheers - Grow Data Skills ❤️ #dataengineering #databricks #openai

131

2 comentarios

How do we evaluate Agentic AI Systems? We have seen tremendous progress in the tools to evaluate and monitor Agentic AI systems, mlflow3.0 from Databricks being the one of the most comprehensive. We now need rigorous methods for taking Agentic AI systems into production. This is Part-1 of my exploration. https://lnkd.in/gzmwvTN7 #databricks #AI #AgenticAI

52

2 comentarios

🧪 If You Cannot Measure It, It Doesn’t Exist! That’s why we built Feldera’s new latency measurement framework. It gives you precise visibility into: ✅ End-to-end pipeline latency ✅ Internal processing latency of our incremental query engine With this new API, you can see how Feldera reduces your time-to-insight from hours to seconds in your Grafana dashboard. 👉 Dive into the details: https://lnkd.in/gw8HRVYm

47

3 comentarios