Giulio Scotti

Apache Parquet Vs New File Formats. There’s no denying the impact Apache Parquet has had on the analytics ecosystem. - Columnar layout - Strong compression & encoding - The de facto format in data lakes and warehouses for over a decade But there are also a lot of new things happening with our analytical workloads. ❌ Workloads are no longer just batch analytics - they involve AI pipelines, low-latency use cases, and hardware-accelerated compute ❌ Hardware is not only CPU but also GPU, from x86 to ARM/RISC-V, with wide SIMD instruction sets becoming the norm ❌ And performance bottlenecks are seen in decompression speed, memory pressure and lack of vectorized execution paths As a result, we’re seeing a wave of innovation around next-gen file formats. Efforts like: BtrBlocks – introduces cascaded compression with lightweight codecs Nimble (Meta) – optimized for fast scan & inference LanceDB – tailored for vector search and ML use cases And the past week I read about “Fastlanes” Here are some highlights from their VLDB paper ✅ Expression Encoding: Flexible chains of lightweight codecs (FFOR, DELTA, DICT, FSST) that outperform heavyweight compression like Zstd ✅ Multi-Column Compression: Exploits inter-column correlations (e.g. equality, one-to-one mappings) to go beyond traditional columnar compression ✅ Segmented Layout: Decompresses in small vectors (1024 values) rather than rowgroups, reducing memory pressure and improving cache efficiency ✅ Compressed Execution Support: Returns compressed vectors to engines like DuckDB/Velox for SIMD/GPU-friendly query execution Here are some numbers I picked up on Performance: - 800× faster random access than Parquet+Zstd - 41% better compression than Parquet+Snappy - 2% better compression than Parquet+Zstd (without using heavyweight codecs) - Decoding accelerates with AVX512 (up to 40% faster) I think we are going to see more of fit-for-purpose formats in the future as we tackle specific use cases around AI workloads and to target performance. Paper link in comments. #dataengineering #softwareengineering

377

13 commenti

PostgreSQL 18 is almost out. Two new game changing features: 🐘 Asynchronous I/O AIO 🐘 UUID7 AIO Introduction of a native asynchronous I/O (AIO) subsystem. This new architecture is designed to significantly increase I/O throughput and reduce latency by allowing the database to perform other work while waiting for data to be read from storage. The AIO feature allows PostgreSQL to issue read requests without blocking, which is especially beneficial for systems with slower, network-attached storage common in cloud environments. The initial implementation specifically targets read-heavy operations, including: ✔️ Sequential scans ✔️ Bitmap heap scans ✔️ Vacuums For databases running these types of workloads, this could translate into performance gains of 2-3 times. However, it's crucial to note that this first release focuses on reads, so write-intensive OLTP applications will not see a direct benefit just yet. UUID7 PostgreSQL 18 now includes native support for UUIDv7, a modern standard for generating globally unique and sortable identifiers. This is a game-changing feature for developers building distributed systems who need efficient, index-friendly primary keys. Traditional random UUIDs (like UUIDv4) are excellent for ensuring uniqueness but are terrible for database performance when used as primary keys. Because they are completely random, new entries are inserted at random locations within a B-tree index. This leads to poor data locality and frequent page splits, a costly operation that fragments the index and slows down writes and reads. UUIDv7 solves this problem by embedding a timestamp at the beginning of the identifier. This makes the UUIDs sequential and naturally ordered by creation time. When new UUIDv7 keys are inserted, they are always added to the end of the index, which is far more efficient. This approach dramatically improves caching and minimizes page splits, resulting in significantly better performance. #PostgreSQL #PostgreSQL18 #Database #DBA #DatabaseAdministrator #DevOps #SystemAdministrator #CloudComputing #DataEngineering #DatabaseManagement #ITInfrastructure #CloudDatabase #DatabasePerformance #postgres #aio #uuid7

362

2 commenti

Don’t get confused while learning PySpark. Understand these core concepts step by step to become truly proficient in PySpark. 𝗕𝗮𝘀𝗶𝗰𝘀 𝗼𝗳 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - PySpark Architecture - SparkContext and SparkSession - RDDs (Resilient Distributed Datasets) - DataFrames - Transformations and Actions - Lazy Evaluation 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗗𝗮𝘁𝗮𝗙𝗿𝗮𝗺𝗲𝘀: - Creating DataFrames - Reading Data from CSV, JSON, Parquet - DataFrame Operations - Filtering, Selecting, and Aggregating Data - Joins and Merging DataFrames - Working with Null Values 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗖𝗼𝗹𝘂𝗺𝗻 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: - Defining and Using UDFs (User Defined Functions) - Column Operations (Select, Rename, Drop) - Handling Complex Data Types (Array, Map) - Working with Dates and Timestamps 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗦𝗵𝘂𝗳𝗳𝗹𝗲 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: - Understanding Partitions - Repartitioning and Coalescing - Managing Shuffle Operations - Optimizing Partition Sizes for Performance 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝗮𝗻𝗱 𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗶𝗻𝗴 𝗗𝗮𝘁𝗮: - When to Cache or Persist - Memory vs Disk Caching - Checking Storage Levels 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗪𝗶𝘁𝗵 𝗦𝗤𝗟: - Spark SQL Introduction - Creating Temp Views - Running SQL Queries - Optimizing SQL Queries with Catalyst Optimizer - Working with Hive Tables in PySpark 𝗪𝗼𝗿𝗸𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗗𝗮𝘁𝗮 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Data Cleaning and Preparation - Handling Missing Values - Data Normalization and Transformation - Working with Categorical Data 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗧𝗼𝗽𝗶𝗰𝘀 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Broadcasting Variables - Accumulators - PySpark Window Functions - PySpark with Machine Learning (MLlib) - Working with Streaming Data (Spark Streaming) 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗧𝘂𝗻𝗶𝗻𝗴 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Understanding Job, Stage, and Task - Tungsten Execution Engine - Memory Management and Garbage Collection - Tuning Parallelism - Using Spark UI for Performance Monitoring Once you master these PySpark fundamentals, you won’t just write code — you’ll design efficient, scalable, and production-ready data pipelines that harness the full potential of Spark.

270

33 commenti

Just in case you want to explain the difference between ETL (extract-transform-load) and ELT (extract-load-transform) approaches in data engineering to your 5 y.o. child, I have something for you. It’s actually easy: either you get something, store it, and make sense out of it later; or you get something, make some sense out of it, and store only the results. What is better? Depends on the use case and a few parameters like size of your storage space and complexity of the processing needed. UPD: for more information about ETL and ELT please check the episode of my video-blog, which is dedicated to this topic: https://lnkd.in/eP52bjFc #data #leadership #etl #elt #apples

700

65 commenti

We finally have 𝘂𝘀𝗲𝗿-𝗱𝗲𝗳𝗶𝗻𝗲𝗱 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝘀 (𝗨𝗗𝗙) in DAX! Don't make mistakes; this is not a user data function in Fabric. This is a new DAX feature, the most important one in the last five years. First article and video to introduce how UDFs in DAX work.

80

1 commento

🚀 𝗗𝗲𝗺𝘆𝘀𝘁𝗶𝗳𝘆𝗶𝗻𝗴 𝗟𝗮𝗸𝗲𝗵𝗼𝘂𝘀𝗲 𝗙𝗲𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝗶𝗻 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 Curious about how 𝗾𝘂𝗲𝗿𝘆 𝗳𝗲𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻 and 𝗰𝗮𝘁𝗮𝗹𝗼𝗴 𝗳𝗲𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻 work in Databricks? 🤔 I’ve put together a 𝗾𝘂𝗶𝗰𝗸 𝗴𝘂𝗶𝗱𝗲 that explains: 🔹 What Lakehouse Federation is 🔹 Key differences between Query vs Catalog Federation 🔹 Practical use cases for both 🔹 Simple setup steps to get started Whether you’re a 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿, 𝗮𝗻𝗮𝗹𝘆𝘀𝘁, 𝗼𝗿 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁, this PDF will help you understand how to run queries seamlessly across multiple data sources without moving all your data into one place. 📥 Download & Read the PDF: [Attached] 💡 𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 for more insights on Databricks, Azure Data, Data Engineering & AI. #Databricks #Lakehouse #DataEngineering #AzureData #BigData #CloudComputing #AI #MachineLearning #DataArchitecture #TechLearning #MicrosoftFabric #LearningOnLinkedIn

65

1 commento

Using DeGirum PySDK and running into “unable to connect to server” or “cloud model zoo doesn’t exists” errors? I put together a practical troubleshooting guide that starts at degirum.connect() and walks through the common failure points—env setup, model zoo config and access token issues. Check it out : https://lnkd.in/dm5JUDnm

29

2 commenti

🧊  I’m excited to announce the official launch of Ryft Adaptive Optimization for Apache Iceberg! “All optimized Iceberg tables are alike; each unoptimized table is unoptimized in its own way” Every table is unique, with different workloads, write patterns, data characteristics and query usage patterns. This is what makes managing Iceberg tables so manual and complex. You find yourself in an endless cycle of monitoring and manual tuning - adjusting compaction schedules, debugging failed maintenance jobs, tweaking compression settings, optimizing file sizes, indexing columns, and reordering data. With Ryft Adaptive Optimization, engineers don’t need to think about when, how and with what parameters to run maintenance and optimize Iceberg tables. It’s all automatic. Ryft's engine learns the unique characteristics of each table and automatically adapts optimization strategies accordingly. This results in up to 5× faster queries, ~7× higher compaction efficiency, up to 10× storage reduction, and mostly - zero manual management. Originally, we set out to build the simplest solution for managing Apache Iceberg Tables. Today, we’re moving one step closer towards making data management simple, and I can’t wait to share with you what else we have in the works over the next few months! One last thing - I hate benchmarks. Never trust a vendor posting a benchmark. That's why we decided to share real world results from what we saw with our customers. For more details, read the full blog post: https://lnkd.in/gTXmiWPu

189

28 commenti

#DataWarehouse, #DataLake, and #DataMesh ..... What’s the Difference? If you are working in the data space, you have probably come across these terms. While they often get used interchangeably, each serves a very different purpose. A Data Warehouse is a centralized system that stores clean, structured, and processed data from different sources. It is designed for reporting, business intelligence, and analytics. Think of it as a well-organized library where everything is neatly labeled and easy to search. In contrast, a Data Lake is built to handle large volumes of raw data in any format structured, semi-structured, or unstructured. It is widely used in big data and machine learning environments where flexibility is more important than immediate usability. Imagine it as a giant reservoir where all data flows in, but it needs further processing before it becomes useful. Then comes the Data Mesh, which takes a different approach. Rather than relying on a single, centralized data team, Data Mesh promotes a decentralized architecture. Each business domain like marketing, sales, or HR owns and manages its own data as a product, while following shared standards and governance. This model scales well in large organizations and encourages accountability and faster data delivery. In short, a data warehouse gives you reliable reports, a data lake gives you flexibility with raw data, and a data mesh gives you scalability through distributed ownership. Each has its place in a modern data architecture and often, organizations use a combination of all three to meet different needs. Which one are you using or moving towards in your organization? Avinash S. #DataEngineering #DataArchitecture #DataWarehouse #DataLake #DataMesh #Analytics #CloudDataPlatform #ModernDataStack

67

4 commenti

⚡ Spark – Creating Custom UDFs and Using Pandas UDFs for Scalability 🐼🚀 Need to run custom Python logic on your Spark DataFrame? Spark lets you define UDFs (User Defined Functions)—and for speed, you can use Pandas UDFs to process data in vectorized batches. ⚡ 💡 How? See code snippet below 🔗 Why Pandas UDFs? - Process data in vectorized batches, avoiding Python overhead - Works great for numeric-heavy transformations - Keeps your logic scalable while still writing in Python 💡 Pro Tip: Use Pandas UDFs over standard UDFs whenever possible—they can give a massive performance booston large datasets. 🔹 Follow me for daily DataFrame manipulation tips and other great data engineering content!

19

📣 𝗥𝗲𝗿𝗮𝗻𝗸𝗶𝗻𝗴 𝗶𝗻 𝗠𝗼𝘀𝗮𝗶𝗰 𝗔𝗜 𝗩𝗲𝗰𝘁𝗼𝗿 𝗦𝗲𝗮𝗿𝗰𝗵 𝗳𝗼𝗿 𝗙𝗮𝘀𝘁𝗲𝗿, 𝗦𝗺𝗮𝗿𝘁𝗲𝗿 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗶𝗻 𝗥𝗔𝗚 𝗔𝗴𝗲𝗻𝘁𝘀 Reranking is coming to Databricks Mosaic AI Vector Search! Improve RAG agent retrieval accuracy by 15% on enterprise benchmarks with a single parameter change. This novel, parallelized architecture achieves latencies as low as 1.5 seconds over 50 documents. Key features:  • 𝗢𝗻𝗲-𝗟𝗶𝗻𝗲 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻: Add a single parameter to your Vector Search queries for instant improvements.  • 𝗠𝘂𝗹𝘁𝗶-𝗖𝗼𝗹𝘂𝗺𝗻 𝗥𝗲𝗿𝗮𝗻𝗸𝗶𝗻𝗴: Leverage metadata (summaries, categories, etc.) for richer context and higher relevance.  • 𝗟𝗮𝗻𝗴𝗖𝗵𝗮𝗶𝗻 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻: Seamlessly integrates with VectorSearchRetrieverTool, requiring no code changes.  • 𝗧𝗿𝗮𝗻𝘀𝗽𝗮𝗿𝗲𝗻𝘁 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲: Reranker latency included in query debug info for end-to-end performance analysis. This feature addresses the critical issue of surfacing relevant context in RAG pipelines, where vector databases alone often fall short. Stop wasting time building and maintaining custom reranking systems. Try it out today and boost your agent quality! https://lnkd.in/eN4U6mdV

19

Claude, ChatGPT, Snowflake Intelligence, Cortex Analyst, Databricks Genie, Zenlytic, TextQL. Think about any AI agent you connect to your data. They all get questions from business users like: “What was the average number of daily active users in the last quarter?” These AI agent connects to Euno’s MCP server to retrieve metadata. That’s when Euno gives these agents the missing context they need to understand your data. ** Just a reminder that Euno is a metadata platform so we only access your metadata ** Euno runs a semantic search and looks for dashboards that use DAU (which is semantically equivalent to Daily Active Users). It filters for certified dashboards using the trust signals you’ve already set in Euno. In this example Euno finds two candidates: 1. DAU used in the User Engagement dashboards 2. DAU_NEW used in the Product Statistics dashboard Both dashboards are certified. But Euno also shows how often each has been used in the past 30 days, along with the owner or creator. With that information, the business user can confidently choose the right one and the AI agent can deliver the correct answer. The same applies to tools like Cursor. Let’s say you’re an engineer and ask: “Can you find me the Tableau dashboard for Customer Success?” Cursor will reach out to Euno’s AI assistant to get that BI context because Cursor is focused on dbt development and doesn’t have access to that layer. You can even go further and ask which dbt models power that dashboard and how those models are calculated. Euno feeds Cursor with that real-time context as you build. It’s not just for business users. It’s for developers too.

25

5 commenti

KQL (Kusto Query Language) is a powerful language used to query large datasets in #Azure Data Explorer and other #Microsoft services like Log Analytics and Application Insights. It’s designed for fast, read-only data exploration, using intuitive operators like where for filtering, summarize for aggregation, and project for selecting columns. Explore the document below for a quick start guide to #KQL in Microsoft Fabric. For more tips on #Microsoftfabric connect with me and follow CloudLitics Hub #OneLake #DeltaLake #Warehouse #PowerBI #FabricTips #technology

18