Skip to content

About

An interim home for data-related & python-based utility functionality

Resources

Stars

0 stars

Watchers

9 watching

Forks

Repository files navigation

Corvus.Python

This provides a library of Python utility functions and classes, generally in the data and analytics space. Many components have been designed to help streamline local development of cloud-based solutions.

Sub-modules

spark_utils

Component Name Object Type Description Import syntax
get_spark_utils Function Returns the notebook utilities for the current environment: mssparkutils on Synapse, native notebookutils on Fabric, and a config-driven mirror locally. See On Fabric for how Fabric differs. Note: Config file required for local development - see section below. from corvus_python.spark_utils import get_spark_utils

get_spark_utils()

Supported operations

The currently supported operations of the mssparkutils API are as follows:

  • credentials
    • getSecretWithLS(linkedService, secret)
    • getToken(audience)
  • env
    • getWorkspaceName()
Configuration

This function requires a configuration file to be present in the repo, and for the file to follow a particular structure. Namely, the config file has been designed to largely mimic the API interface of the mssparkutils API.

There is a top-level property for each (supported) sub-module of mssparkutils. Second-level properties follow the names of the functions associated with each sub-module. Within these second-level properties, the config structure depends on the implementation of the mirrored function found in the corresponding class in the package. E.g. the structure of credentials.getSecretWithLS() can be inferred from the LocalCredentialUtils class.

Below shows the current, complete specification of the config file for the supported operations (NOTE: not all operations require configuration):

{
    "credentials": {
        "getSecretWithLS": {
            "<linked_service_name>": {
                "<key_vault_secret_name>": {
                    "type": "static",
                    "value": "<key_vault_secret_value>"
                }
            }
        },
        "getToken": {
            "tenantId": "<tenant_id (optional)>"
        }
    },
    "env": {
        "getWorkspaceName": "<workspace_name>"
    }
}

By default, a file in the root of the current working directory with file name local-spark-utils-config.json will be automatically discovered. If the file resides in a different location, and/or has a different file name, then the absolute path must be specified when calling get_spark_utils().

On Fabric

On Fabric, get_spark_utils() returns Fabric's native notebookutils, which works in both Spark and Python notebooks. It is not a drop-in replacement for Synapse's mssparkutils:

  • No linked services. credentials.getSecretWithLS does not exist; use credentials.getSecret(vault_name, secret_name).
  • Tokens need full resource scopes. credentials.getToken("Synapse") is rejected; pass https://dev.azuresynapse.net/.default instead (see SYNAPSE_AUDIENCE_SCOPES).

Fabric-only APIs are available on the same object — for example, variable libraries via variableLibrary.get("$(/**/<library>/<variable>)").

Use get_platform() where behaviour needs to differ, and FabricTokenCredential to use the Azure SDKs from a Fabric notebook.

platform

Component Name Object Type Description Import syntax
get_platform Function Returns 'fabric', 'synapse' or 'local'. Requires no Spark session, so it works in Fabric Python notebooks as well as Spark ones. Has no side effects. from corvus_python.platform import get_platform

fabric

Component Name Object Type Description Import syntax
FabricTokenCredential Class azure-identity compatible credential backed by notebookutils, for using the Azure SDKs inside a Fabric notebook. from corvus_python.fabric import FabricTokenCredential
configure_tls_trust_store Function Points Rust-based TLS clients at a CA bundle they can parse. No-op off Fabric. See TLS on Fabric. from corvus_python.fabric import configure_tls_trust_store

Fabric is detected via notebookutils.runtime.context["productType"] (see platform), which works in both Spark and Python notebooks. Do not detect Fabric using MMLSPARK_PLATFORM_INFO or AZURE_SERVICE — Fabric Spark sessions set both to their Synapse values.

FabricTokenCredential lets the Azure SDKs authenticate inside a Fabric notebook as the notebook's executing identity, using tokens from notebookutils.credentials.getToken. It passes the requested scope straight through, because Fabric rejects short keyword audiences such as synapse but accepts full resource scopes.

from azure.keyvault.secrets import SecretClient
from corvus_python.fabric import FabricTokenCredential

client = SecretClient("https://my-key-vault.vault.azure.net/", FabricTokenCredential())

Composition is left to the consumer: corvus cannot know what a project calls its variable library, which vault it uses, or how it maps secrets to configuration.

TLS on Fabric

Fabric sets SSL_CERT_FILE to /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt, an OpenSSL extended trust bundle made of BEGIN TRUSTED CERTIFICATE blocks. OpenSSL-based clients — requests and the Azure SDKs — read it without trouble. rustls, used by object_store and therefore by deltalake and polars' Delta reader, silently skips those blocks, loads no roots at all, and fails every handshake with invalid peer certificate: UnknownIssuer.

configure_tls_trust_store() repoints SSL_CERT_FILE at a plain PEM bundle when the current one can't be parsed, and returns the path it settled on (or None). The Azure Data Lake and Fabric Lakehouse storage configuration classes call it on construction, so reading data through them needs no extra setup. If you read storage by some other route, call it yourself before the first object_store request. object_store reads SSL_CERT_FILE each time it builds an HTTP client, so a client built before the call keeps the unreadable bundle and fails.

pyspark.utilities

⚠️ Note: This module requires the 'pyspark' extra to be installed: corvus-python[pyspark]

Includes utility functions when working with PySpark to build data processing solutions. Primary API interfaces:

Component Name Object Type Description Import syntax
get_or_create_spark_session Function Gets or creates a Spark Session, depending on the environment. Supports Synapse or a Local Spark Session configuration. from corvus_python.pyspark.utilities import get_or_create_spark_session
null_safe_join Function Joins two Spark DataFrames incorporating null-safe equality. from corvus_python.pyspark.utilities import null_safe_join

pyspark.synapse

⚠️ Note: This module requires the 'pyspark' extra to be installed: corvus-python[pyspark]

Component Name Object Type Description Import syntax
sync_synapse_tables_to_local_spark Function Reads tables from a Synapse SQL Serverless endpoint and clones to a local Hive metastore. Useful for local development, to avoid continuously sending data over the wire. from corvus_python.pyspark.synapse import sync_synapse_tables_to_local_spark
ObjectSyncDetails Class Dataclass representing a database and corresponding tables to be synced using the sync_synapse_tables_to_local_spark function. from corvus_python.pyspark.synapse import ObjectSyncDetails

sync_synapse_tables_to_local_spark()

Here is an example code snippet to utilize this function:

from corvus_python.pyspark.synapse import sync_synapse_tables_to_local_spark, ObjectSyncDetails

sync_synapse_tables_to_local_spark(
    workspace_name='my_workspace_name',
    object_sync_details=[
        ObjectSyncDetails(
            database_name='database_1',
            tables=['table_1', 'table_2']
        ),
        ObjectSyncDetails(
            database_name='database_2',
            tables=['table_1', 'table_2']
        )
    ],
    # overwrite = True,  # Uncomment if local clones already exist and you wish to overwrite.
    # spark = spark,     # Uncomment if you wish to provide your own Spark Session (assumed stored within "spark" variable).
)

synapse

Includes utility functions when working with Synapse Analytics. Primary API interfaces:

Component Name Object Type Description Import syntax
SynapseUtilities Class A utility class for interacting with Azure Synapse Analytics. from corvus_python.synapse import SynapseUtilities

SynapseUtilities gives every HTTP request a timeout. See HTTP timeouts.


auth

Includes utility functions when working with authentication libraries within Python. Primary API interfaces:

Component Name Object Type Description Import syntax
get_az_cli_token Function Gets an Entra ID token from the Azure CLI for a specified resource (/audience) and tenant. Useful for local development. from corvus_python.auth import get_az_cli_token

sharepoint

Includes utility functions when working with SharePoint REST API. Primary API interfaces:

Component Name Object Type Description Import syntax
SharePointUtilities Class A utility class for interacting with SharePoint REST API. from corvus_python.sharepoint import SharePointUtilities

HTTP timeouts

requests waits forever for a response unless it's given a timeout, so a connection that stalls without closing would block the calling thread indefinitely. Every HTTP request made by SharePointUtilities, SharePointFileStorage and SynapseUtilities therefore has one, from corvus_python.http_timeouts:

Constant Value (seconds, (connect, read)) Used for
DEFAULT_HTTP_TIMEOUT (10, 60) All requests except uploads.
DEFAULT_UPLOAD_HTTP_TIMEOUT (10, 300) File uploads: SharePointUtilities.save_file and SharePointFileStorage.write_file.

The read timeout is the longest wait between bytes from the server, not a limit on the whole request. To override them, pass the keyword-only timeout argument, either a number of seconds or a (connect, read) tuple: to each SharePointUtilities method, or to the SynapseUtilities and SharePointFileStorage constructors (SharePointFileStorage also takes upload_timeout).

When a request times out, requests.Timeout (a subclass of requests.ConnectionError) is raised. It is not caught or retried; that's for callers to decide.

import requests
from corvus_python.sharepoint import SharePointUtilities

try:
    image = SharePointUtilities.retrieve_image_as_base64(drive_id, file_path, token, timeout=(5, 30))
except requests.Timeout:
    ...  # retry, skip or fail

email

Includes utility classes and models for sending emails using Azure Communication Services (ACS). Primary API interfaces:

Component Name Object Type Description Import syntax
AcsEmailService Class A service class for sending emails through Azure Communication Services. from corvus_python.email import AcsEmailService
EmailContent Class Dataclass representing email content including subject, plain text, and HTML. from corvus_python.email import EmailContent
EmailRecipients Class Dataclass representing email recipients (to, cc, bcc). from corvus_python.email import EmailRecipients
EmailRecipient Class Dataclass representing a single email recipient with address and display name. from corvus_python.email import EmailRecipient
EmailAttachment Class Dataclass representing an email attachment with name, content type, and base64-encoded content. from corvus_python.email import EmailAttachment
EmailError Class Exception class for email-related errors. from corvus_python.email import EmailError

Usage Example

from corvus_python.email import (
    AcsEmailService, 
    EmailContent, 
    EmailRecipients, 
    EmailRecipient, 
    EmailAttachment
)

# Initialize the email service
email_service = AcsEmailService(
    acs_connection_string="your_acs_connection_string",
    from_email="sender@yourdomain.com",
    email_sending_disabled=False  # Set to True for testing/development
)

# Create email content
content = EmailContent(
    subject="Welcome to Our Service",
    plain_text="Welcome! Thank you for joining our service.",
    html="<h1>Welcome!</h1><p>Thank you for joining our service.</p>"
)

# Define recipients
recipients = EmailRecipients(
    to=[
        EmailRecipient("user1@example.com", "User One"),
        EmailRecipient("user2@example.com", "User Two")
    ],
    cc=[EmailRecipient("manager@example.com", "Manager")],
    bcc=[EmailRecipient("admin@example.com", "Admin")]
)

# Optional: Add attachments
attachments = [
    EmailAttachment(
        name="welcome_guide.pdf",
        content_type="application/pdf",
        content_in_base64="base64_encoded_content_here"
    )
]

# Send the email
try:
    email_service.send_email(content, recipients, attachments)
    print("Email sent successfully!")
except EmailError as e:
    print(f"Failed to send email: {e}")

Configuration Notes:

  • Requires an Azure Communication Services resource with email capability configured
  • The from_email must use a configured MailFrom address from your ACS resource
  • Set email_sending_disabled=True during development to prevent actual emails from being sent
  • Attachments must be base64-encoded before adding to the EmailAttachment object

monitoring

Includes helpers for tracing with OpenTelemetry.

Function Description
start_as_current_span_with_method_name Decorator that runs a function in a new span, named after the function.
all_methods_start_new_current_span_with_method_name Class decorator that does the same for every method.
add_attributes_to_span, add_attributes_to_current_span Add key-value pairs to a span as attributes.
add_kwargs_to_span, add_kwargs_to_current_span Add the named keyword arguments to a span as attributes.
in_current_context Wraps a function to run in the calling thread's context, so work on a thread pool stays in the caller's trace.

A thread pool's worker threads start with an empty context, so a span started on one, such as an instrumented HTTP request's, begins a trace of its own. Wrap the function you hand the pool in in_current_context, on the thread whose span it should be part of:

from concurrent.futures import ThreadPoolExecutor
from corvus_python.monitoring import in_current_context

with tracer.start_as_current_span("download_images"), ThreadPoolExecutor() as pool:
    images = list(pool.map(in_current_context(download_image), urls))

storage

Provides storage configuration abstractions for data lake operations, with implementations for local, Azure Data Lake Gen2 and Microsoft Fabric OneLake storage.

Component Name Object Type Description Import syntax
DataLakeLayer Enum Enumeration of data lake layers: BRONZE, SILVER, GOLD. from corvus_python.storage import DataLakeLayer
StorageConfiguration Class (abstract) Base class for storage configurations. Provides a get_full_path() method and storage_options dict. from corvus_python.storage import StorageConfiguration
LocalFileSystemStorageConfiguration Class Storage configuration backed by the local file system. Useful for local development. from corvus_python.storage import LocalFileSystemStorageConfiguration
AzureDataLakeFileSystemPerLayerConfiguration Class ADLS Gen2 configuration where each data lake layer maps to a separate file system (bronze, silver, gold). from corvus_python.storage import AzureDataLakeFileSystemPerLayerConfiguration
AzureDataLakeSingleFileSystemConfiguration Class ADLS Gen2 configuration using a single file system with top-level folders for each layer. from corvus_python.storage import AzureDataLakeSingleFileSystemConfiguration
FabricLakehouseTablesConfiguration Class Fabric OneLake configuration where each data lake layer maps to a separate Lakehouse (bronze, silver, gold by default), in one workspace or a workspace per layer. Paths are rooted in the Lakehouse's managed Tables/ area, for Delta tables. from corvus_python.storage import FabricLakehouseTablesConfiguration
FabricLakehouseFilesConfiguration Class As above, but rooted in the Lakehouse's unmanaged Files/ area, for raw files such as CSV, JSON or Excel. from corvus_python.storage import FabricLakehouseFilesConfiguration
StorageConfigurationFileStorage Class FileStorage rooted at one layer of any StorageConfiguration, for byte-level access (read, write, find the latest file matching a prefix, list subfolders). Uses obstore, so the configuration's storage_options work unchanged. from corvus_python.storage.storage_configuration_file_storage import StorageConfigurationFileStorage

Usage Example

from corvus_python.storage import (
    DataLakeLayer,
    LocalFileSystemStorageConfiguration,
    AzureDataLakeFileSystemPerLayerConfiguration,
    AzureDataLakeSingleFileSystemConfiguration,
    FabricLakehouseTablesConfiguration,
    FabricLakehouseFilesConfiguration,
)
from corvus_python.storage.storage_configuration_file_storage import StorageConfigurationFileStorage

# Local filesystem (for development)
local_config = LocalFileSystemStorageConfiguration(base_path="./data")
path = local_config.get_full_path(DataLakeLayer.BRONZE, "my_database/my_table")
# -> ./data/bronze/my_database/my_table

# Azure Data Lake - separate file system per layer
adls_per_layer = AzureDataLakeFileSystemPerLayerConfiguration(
    storage_account_name="mystorageaccount",
    storage_options={"account_key": "..."},
)
path = adls_per_layer.get_full_path(DataLakeLayer.SILVER, "my_database/my_table")
# -> abfss://silver@mystorageaccount.dfs.core.windows.net/my_database/my_table

# Azure Data Lake - single file system
adls_single = AzureDataLakeSingleFileSystemConfiguration(
    storage_account_name="mystorageaccount",
    file_system_name="datalake",
)
path = adls_single.get_full_path(DataLakeLayer.GOLD, "my_database/my_table")
# -> abfss://datalake@mystorageaccount.dfs.core.windows.net/gold/my_database/my_table

# Fabric OneLake - separate Lakehouse per layer, with a class per Lakehouse area
tables = FabricLakehouseTablesConfiguration(
    workspace_name="MyWorkspace",
    lakehouse_names={DataLakeLayer.GOLD: "lh_gold"},  # optional; defaults to bronze/silver/gold
    workspace_names={DataLakeLayer.GOLD: "MyReportingWorkspace"},  # optional; defaults to workspace_name
)
path = tables.get_full_path(DataLakeLayer.SILVER, "my_schema/my_table")
# -> abfss://MyWorkspace@onelake.dfs.fabric.microsoft.com/silver.Lakehouse/Tables/my_schema/my_table

files = FabricLakehouseFilesConfiguration(workspace_name="MyWorkspace")
path = files.get_full_path(DataLakeLayer.BRONZE, "raw/orders.csv")
# -> abfss://MyWorkspace@onelake.dfs.fabric.microsoft.com/bronze.Lakehouse/Files/raw/orders.csv

# Byte-level access to one layer of any configuration
bronze_files = StorageConfigurationFileStorage(files, DataLakeLayer.BRONZE)
latest = bronze_files.get_latest_matching_file_bytes("raw/orders_")

Fabric only discovers managed tables at Tables/<table> or, in schema-enabled Lakehouses, Tables/<schema>/<table>. When using PolarsDeltaTableRepository with FabricLakehouseTablesConfiguration, pass an empty base_path so the database name becomes the schema.

Workspaces and Lakehouses must be given by name, not ID. OneLake rejects paths that mix IDs and names with 400 Bad Request, so the Fabric configurations raise ValueError if a value looks like an ID.

On the Fabric Spark runtime, OneLake traffic goes through a proxy that presents a self-signed CA certificate. Polars, deltalake and obstore use rustls, which rejects it with invalid peer certificate: CaUsedAsEndEntity. OpenSSL-based clients, the Fabric Python runtime and direct ADLS access are not affected. To read OneLake with Polars from a Spark notebook, pass allow_invalid_certificates=True. This disables TLS certificate validation for OneLake requests, and only takes effect on the Fabric Spark runtime; elsewhere it is ignored and an info message is logged. It removes protection against interception, so enable it only where it is needed.


repositories

Provides repository classes for reading and writing structured data across various storage backends, built on Polars. All repositories accept a StorageConfiguration to abstract over local and cloud storage.

Supporting Data Classes

Component Name Object Type Description Import syntax
DatabaseDefinition Dataclass Defines a logical database by name and a list of TableDefinition instances. from corvus_python.repositories import DatabaseDefinition
TableDefinition Dataclass Defines a table by name, Pandera schema, optional title, and optional db_schema (the SQL schema name, e.g. dbo, used when creating SQL views over the table in Synapse or Fabric). from corvus_python.repositories import TableDefinition

PolarsDeltaTableRepository

Manages Delta Lake tables within a specified data lake layer. Handles schema validation using Pandera and integrates with OpenTelemetry for tracing.

from corvus_python.repositories import PolarsDeltaTableRepository, DatabaseDefinition, TableDefinition
from corvus_python.storage import LocalFileSystemStorageConfiguration, DataLakeLayer
Method Description
read_data(table_name) Reads a Delta table into a LazyFrame. Returns None if the table is empty.
overwrite_table(table_name, data, overwrite_schema=False) Overwrites the table after eagerly validating the full dataset against its Pandera schema. Takes a DataFrame or a LazyFrame, which it collects first. Writes the columns the schema declares not nullable as non-nullable.
overwrite_table_lazy(table_name, data, overwrite_schema=False) Overwrites the table using streaming execution, for data too large to collect. Validates columns and types only, not values: duplicate unique keys and failed checks are written, and a null in a non-nullable column is only stopped by the table's own constraint. With overwrite_schema, every column becomes nullable, removing those constraints.
overwrite_table_with_condition(table_name, data, predicate, overwrite_schema=False) Overwrites only rows matching the given predicate (e.g. for partition-level updates).
append_to_table(table_name, data) Appends data to an existing Delta table.

PolarsCsvDataRepository

Reads CSV files from a hive-partitioned path (snapshot_time=<timestamp>/<name>.csv).

from corvus_python.repositories import PolarsCsvDataRepository
Method Description
load_csv(object_name, snapshot_timestamp, include_file_paths=None) Loads a CSV file into a DataFrame. Strips .csv suffix from object_name if present.

PolarsExcelDataRepository

Reads Excel workbooks from a hive-partitioned path, returning all sheets as a dict[str, DataFrame].

The workbook is read through obstore, so it works with local, Azure Data Lake and Fabric OneLake configurations, and authenticates with the configuration's storage_options in the same way as the CSV, NDJSON and Delta repositories.

from corvus_python.repositories import PolarsExcelDataRepository
Method Description
load_excel(snapshot_timestamp, workbook_name, relative_path=None) Loads all sheets from an .xlsx workbook into a dict keyed by sheet name.

PolarsNdJsonDataRepository

Reads Newline Delimited JSON (NDJSON) files from a partitioned path (<load_type>/snapshot_time=<timestamp>/<name>.json).

from corvus_python.repositories import PolarsNdJsonDataRepository
Method Description
load_ndjson(object_name, load_type, snapshot_timestamp, include_file_paths=None, schema_overrides=None, schema=None) Loads an NDJSON file into a DataFrame. Supports schema overrides and full schema specification.

PolarsAzureTableRepository

Queries Azure Table Storage, returning results as Polars DataFrames. Authenticates using DefaultAzureCredential.

from corvus_python.repositories import PolarsAzureTableRepository
Method Description
query(table_name, query_filter, parameters, schema=None) Queries an Azure Table with an OData filter and named parameters.
get_entities_partition_key_starts_with(table_name, partition_key_prefix, schema=None) Retrieves all entities whose PartitionKey starts with the given prefix.

sql

Includes utility functions for working with SQL databases via pyodbc, with AAD token-based authentication. Provides helpers for connecting to Synapse serverless SQL and Fabric SQL Analytics endpoints, managing views over Delta Lake tables, and executing DDL statements.

⚠️ Note: This module requires ODBC Driver 18 for SQL Server or later.

Component Name Object Type Description Import syntax
get_pyodbc_connection Function Opens a pyodbc connection to a Synapse serverless SQL or Fabric SQL Analytics endpoint using AAD tokens. from corvus_python.sql import get_pyodbc_connection
get_pyodbc_connection_with_token Function Opens a pyodbc connection using a pre-acquired AAD token. Useful when running inside Synapse notebooks. from corvus_python.sql import get_pyodbc_connection_with_token
execute_ddl Function Executes a DDL statement (e.g. CREATE VIEW) using a pyodbc connection. from corvus_python.sql import execute_ddl
create_or_alter_view_over_delta_table Function Creates or alters a SQL view over a Delta Lake table, with support for inferred or explicit column types. from corvus_python.sql import create_or_alter_view_over_delta_table
drop_views_in_schema Function Drops all views in a given schema, except any named in exclude (compared case-insensitively). from corvus_python.sql import drop_views_in_schema
SelectColumn Class Dataclass representing a column to select, with an optional display title. from corvus_python.sql import SelectColumn
WithColumn Class Dataclass representing a column with an explicit type for use in OPENROWSET WITH clauses. from corvus_python.sql import WithColumn

sql.synapse

Convenience wrappers specific to Synapse serverless SQL endpoints.

Component Name Object Type Description Import syntax
get_synapse_sql_pyodbc_connection Function Opens a pyodbc connection to a Synapse serverless SQL endpoint using AAD tokens. Builds the server URL from the workspace name. from corvus_python.sql.synapse import get_synapse_sql_pyodbc_connection
get_synapse_sql_pyodbc_connection_with_token Function Opens a pyodbc connection to a Synapse serverless SQL endpoint using a pre-acquired AAD token. Ideal for Synapse notebooks. from corvus_python.sql.synapse import get_synapse_sql_pyodbc_connection_with_token
create_database_if_not_exists Function Creates a database if it doesn't already exist. Requires a connection to the master database. from corvus_python.sql.synapse import create_database_if_not_exists

sql.fabric

Convenience wrappers specific to Fabric SQL Analytics endpoints.

Component Name Object Type Description Import syntax
get_fabric_sql_pyodbc_connection Function Opens a pyodbc connection to a Fabric SQL Analytics endpoint using AAD tokens. from corvus_python.sql.fabric import get_fabric_sql_pyodbc_connection

sql.repositories

Helpers for creating SQL views over the Delta tables described by TableDefinitions. Each view is named after the table's title (or its name) and selects the schema's columns, aliased to their titles.

Component Name Object Type Description Import syntax
sync_sql_views_from_tables Function Drops the views in a schema that no table produces, then creates or alters a view for each table. Use this rather than drop_views_in_schema followed by create_sql_views_from_tables. from corvus_python.sql.repositories import sync_sql_views_from_tables
create_sql_views_from_tables Function Creates or alters a view for each table. from corvus_python.sql.repositories import create_sql_views_from_tables
SqlViewCreationError Exception Raised, inside an ExceptionGroup, for each view that failed to be created when creating views in parallel. Has schema_name and view_name attributes. from corvus_python.sql.repositories import SqlViewCreationError

Both functions take an optional max_workers (default 1: views are created one at a time on conn, and the first error is raised). Above 1, views are created concurrently, with each worker using its own connection from the required connection_factory, since pyodbc connections can't be shared across threads. Every view is attempted, and the failures are raised together in an ExceptionGroup. Stale views are always dropped sequentially on conn.

from corvus_python.sql.repositories import sync_sql_views_from_tables
from corvus_python.sql.synapse import get_synapse_sql_pyodbc_connection_with_token

token = mssparkutils.credentials.getToken("DW")


def connect():
    # Reuses the one token across every worker's connection.
    return get_synapse_sql_pyodbc_connection_with_token("myworkspace", "my_database", token)


conn = connect()
try:
    sync_sql_views_from_tables(
        conn,
        schema_name="dbo",
        tables=TABLES,
        build_delta_table_path_func=lambda table_name: f"abfss://silver@myaccount.dfs.core.windows.net/{table_name}",
        max_workers=8,
        connection_factory=connect,
    )
finally:
    conn.close()

Usage Example

from corvus_python.sql import (
    get_pyodbc_connection,
    create_or_alter_view_over_delta_table,
    drop_views_in_schema,
    SelectColumn,
    WithColumn,
)

# Connect using DefaultAzureCredential
conn = get_pyodbc_connection(
    server="myworkspace-ondemand.sql.azuresynapse.net",
    database="my_database",
    use_managed_identity=False,
)

# Create a view with inferred types
create_or_alter_view_over_delta_table(
    conn=conn,
    schema_name="dbo",
    view_name="my_view",
    delta_table_path="abfss://container@storageaccount.dfs.core.windows.net/path/to/table",
    infer_types=True,
    select_columns=[
        SelectColumn(name="id"),
        SelectColumn(name="full_name", title="Name"),
    ],
)

# Create a view with explicit types
create_or_alter_view_over_delta_table(
    conn=conn,
    schema_name="dbo",
    view_name="my_typed_view",
    delta_table_path="abfss://container@storageaccount.dfs.core.windows.net/path/to/table",
    infer_types=False,
    with_columns=[
        WithColumn(name="id", type="INT"),
        WithColumn(name="full_name", type="VARCHAR(200)", title="Name"),
    ],
)

# Drop all views in a schema
drop_views_in_schema(conn, schema_name="dbo")

Or use the Synapse/Fabric-specific helpers:

from corvus_python.sql.synapse import get_synapse_sql_pyodbc_connection

conn = get_synapse_sql_pyodbc_connection(
    workspace_name="myworkspace",
    database="my_database",
    use_managed_identity=False,
)

Usage from a Synapse Notebook

When running inside a Synapse notebook, you can use mssparkutils to acquire a token and pass it directly:

from corvus_python.sql.synapse import get_synapse_sql_pyodbc_connection_with_token

token = mssparkutils.credentials.getToken("DW")

conn = get_synapse_sql_pyodbc_connection_with_token(
    workspace_name="myworkspace",
    database="my_database",
    token=token,
)

About

An interim home for data-related & python-based utility functionality

Resources

Stars

0 stars

Watchers

9 watching

Forks

Releases

Packages

Used by

Contributors

Languages