This provides a library of Python utility functions and classes, generally in the data and analytics space. Many components have been designed to help streamline local development of cloud-based solutions.
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
get_spark_utils |
Function | Returns the notebook utilities for the current environment: mssparkutils on Synapse, native notebookutils on Fabric, and a config-driven mirror locally. See On Fabric for how Fabric differs. Note: Config file required for local development - see section below. | from corvus_python.spark_utils import get_spark_utils |
The currently supported operations of the mssparkutils API are as follows:
credentialsgetSecretWithLS(linkedService, secret)getToken(audience)
envgetWorkspaceName()
This function requires a configuration file to be present in the repo, and for the file to follow a particular structure. Namely, the config file has been designed to largely mimic the API interface of the mssparkutils API.
There is a top-level property for each (supported) sub-module of mssparkutils. Second-level properties follow the names of the functions associated with each sub-module. Within these second-level properties, the config structure depends on the implementation of the mirrored function found in the corresponding class in the package. E.g. the structure of credentials.getSecretWithLS() can be inferred from the LocalCredentialUtils class.
Below shows the current, complete specification of the config file for the supported operations (NOTE: not all operations require configuration):
{
"credentials": {
"getSecretWithLS": {
"<linked_service_name>": {
"<key_vault_secret_name>": {
"type": "static",
"value": "<key_vault_secret_value>"
}
}
},
"getToken": {
"tenantId": "<tenant_id (optional)>"
}
},
"env": {
"getWorkspaceName": "<workspace_name>"
}
}
By default, a file in the root of the current working directory with file name local-spark-utils-config.json will be automatically discovered. If the file resides in a different location, and/or has a different file name, then the absolute path must be specified when calling get_spark_utils().
On Fabric, get_spark_utils() returns Fabric's native notebookutils, which works in both Spark and Python notebooks. It is not a drop-in replacement for Synapse's mssparkutils:
- No linked services.
credentials.getSecretWithLSdoes not exist; usecredentials.getSecret(vault_name, secret_name). - Tokens need full resource scopes.
credentials.getToken("Synapse")is rejected; passhttps://dev.azuresynapse.net/.defaultinstead (seeSYNAPSE_AUDIENCE_SCOPES).
Fabric-only APIs are available on the same object — for example, variable libraries via variableLibrary.get("$(/**/<library>/<variable>)").
Use get_platform() where behaviour needs to differ, and FabricTokenCredential to use the Azure SDKs from a Fabric notebook.
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
get_platform |
Function | Returns 'fabric', 'synapse' or 'local'. Requires no Spark session, so it works in Fabric Python notebooks as well as Spark ones. Has no side effects. |
from corvus_python.platform import get_platform |
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
FabricTokenCredential |
Class | azure-identity compatible credential backed by notebookutils, for using the Azure SDKs inside a Fabric notebook. |
from corvus_python.fabric import FabricTokenCredential |
configure_tls_trust_store |
Function | Points Rust-based TLS clients at a CA bundle they can parse. No-op off Fabric. See TLS on Fabric. | from corvus_python.fabric import configure_tls_trust_store |
Fabric is detected via notebookutils.runtime.context["productType"] (see platform), which works in both Spark and Python notebooks. Do not detect Fabric using MMLSPARK_PLATFORM_INFO or AZURE_SERVICE — Fabric Spark sessions set both to their Synapse values.
FabricTokenCredential lets the Azure SDKs authenticate inside a Fabric notebook as the notebook's executing identity, using tokens from notebookutils.credentials.getToken. It passes the requested scope straight through, because Fabric rejects short keyword audiences such as synapse but accepts full resource scopes.
from azure.keyvault.secrets import SecretClient
from corvus_python.fabric import FabricTokenCredential
client = SecretClient("https://my-key-vault.vault.azure.net/", FabricTokenCredential())Composition is left to the consumer: corvus cannot know what a project calls its variable library, which vault it uses, or how it maps secrets to configuration.
Fabric sets SSL_CERT_FILE to /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt, an OpenSSL extended trust bundle made of BEGIN TRUSTED CERTIFICATE blocks. OpenSSL-based clients — requests and the Azure SDKs — read it without trouble. rustls, used by object_store and therefore by deltalake and polars' Delta reader, silently skips those blocks, loads no roots at all, and fails every handshake with invalid peer certificate: UnknownIssuer.
configure_tls_trust_store() repoints SSL_CERT_FILE at a plain PEM bundle when the current one can't be parsed, and returns the path it settled on (or None). The Azure Data Lake and Fabric Lakehouse storage configuration classes call it on construction, so reading data through them needs no extra setup. If you read storage by some other route, call it yourself before the first object_store request. object_store reads SSL_CERT_FILE each time it builds an HTTP client, so a client built before the call keeps the unreadable bundle and fails.
corvus-python[pyspark]
Includes utility functions when working with PySpark to build data processing solutions. Primary API interfaces:
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
get_or_create_spark_session |
Function | Gets or creates a Spark Session, depending on the environment. Supports Synapse or a Local Spark Session configuration. | from corvus_python.pyspark.utilities import get_or_create_spark_session |
null_safe_join |
Function | Joins two Spark DataFrames incorporating null-safe equality. | from corvus_python.pyspark.utilities import null_safe_join |
corvus-python[pyspark]
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
sync_synapse_tables_to_local_spark |
Function | Reads tables from a Synapse SQL Serverless endpoint and clones to a local Hive metastore. Useful for local development, to avoid continuously sending data over the wire. | from corvus_python.pyspark.synapse import sync_synapse_tables_to_local_spark |
ObjectSyncDetails |
Class | Dataclass representing a database and corresponding tables to be synced using the sync_synapse_tables_to_local_spark function. |
from corvus_python.pyspark.synapse import ObjectSyncDetails |
Here is an example code snippet to utilize this function:
from corvus_python.pyspark.synapse import sync_synapse_tables_to_local_spark, ObjectSyncDetails
sync_synapse_tables_to_local_spark(
workspace_name='my_workspace_name',
object_sync_details=[
ObjectSyncDetails(
database_name='database_1',
tables=['table_1', 'table_2']
),
ObjectSyncDetails(
database_name='database_2',
tables=['table_1', 'table_2']
)
],
# overwrite = True, # Uncomment if local clones already exist and you wish to overwrite.
# spark = spark, # Uncomment if you wish to provide your own Spark Session (assumed stored within "spark" variable).
)Includes utility functions when working with Synapse Analytics. Primary API interfaces:
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
SynapseUtilities |
Class | A utility class for interacting with Azure Synapse Analytics. | from corvus_python.synapse import SynapseUtilities |
SynapseUtilities gives every HTTP request a timeout. See HTTP timeouts.
Includes utility functions when working with authentication libraries within Python. Primary API interfaces:
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
get_az_cli_token |
Function | Gets an Entra ID token from the Azure CLI for a specified resource (/audience) and tenant. Useful for local development. | from corvus_python.auth import get_az_cli_token |
Includes utility functions when working with SharePoint REST API. Primary API interfaces:
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
SharePointUtilities |
Class | A utility class for interacting with SharePoint REST API. | from corvus_python.sharepoint import SharePointUtilities |
requests waits forever for a response unless it's given a timeout, so a connection that stalls without closing would block the calling thread indefinitely. Every HTTP request made by SharePointUtilities, SharePointFileStorage and SynapseUtilities therefore has one, from corvus_python.http_timeouts:
| Constant | Value (seconds, (connect, read)) |
Used for |
|---|---|---|
DEFAULT_HTTP_TIMEOUT |
(10, 60) |
All requests except uploads. |
DEFAULT_UPLOAD_HTTP_TIMEOUT |
(10, 300) |
File uploads: SharePointUtilities.save_file and SharePointFileStorage.write_file. |
The read timeout is the longest wait between bytes from the server, not a limit on the whole request. To override them, pass the keyword-only timeout argument, either a number of seconds or a (connect, read) tuple: to each SharePointUtilities method, or to the SynapseUtilities and SharePointFileStorage constructors (SharePointFileStorage also takes upload_timeout).
When a request times out, requests.Timeout (a subclass of requests.ConnectionError) is raised. It is not caught or retried; that's for callers to decide.
import requests
from corvus_python.sharepoint import SharePointUtilities
try:
image = SharePointUtilities.retrieve_image_as_base64(drive_id, file_path, token, timeout=(5, 30))
except requests.Timeout:
... # retry, skip or failIncludes utility classes and models for sending emails using Azure Communication Services (ACS). Primary API interfaces:
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
AcsEmailService |
Class | A service class for sending emails through Azure Communication Services. | from corvus_python.email import AcsEmailService |
EmailContent |
Class | Dataclass representing email content including subject, plain text, and HTML. | from corvus_python.email import EmailContent |
EmailRecipients |
Class | Dataclass representing email recipients (to, cc, bcc). | from corvus_python.email import EmailRecipients |
EmailRecipient |
Class | Dataclass representing a single email recipient with address and display name. | from corvus_python.email import EmailRecipient |
EmailAttachment |
Class | Dataclass representing an email attachment with name, content type, and base64-encoded content. | from corvus_python.email import EmailAttachment |
EmailError |
Class | Exception class for email-related errors. | from corvus_python.email import EmailError |
from corvus_python.email import (
AcsEmailService,
EmailContent,
EmailRecipients,
EmailRecipient,
EmailAttachment
)
# Initialize the email service
email_service = AcsEmailService(
acs_connection_string="your_acs_connection_string",
from_email="sender@yourdomain.com",
email_sending_disabled=False # Set to True for testing/development
)
# Create email content
content = EmailContent(
subject="Welcome to Our Service",
plain_text="Welcome! Thank you for joining our service.",
html="<h1>Welcome!</h1><p>Thank you for joining our service.</p>"
)
# Define recipients
recipients = EmailRecipients(
to=[
EmailRecipient("user1@example.com", "User One"),
EmailRecipient("user2@example.com", "User Two")
],
cc=[EmailRecipient("manager@example.com", "Manager")],
bcc=[EmailRecipient("admin@example.com", "Admin")]
)
# Optional: Add attachments
attachments = [
EmailAttachment(
name="welcome_guide.pdf",
content_type="application/pdf",
content_in_base64="base64_encoded_content_here"
)
]
# Send the email
try:
email_service.send_email(content, recipients, attachments)
print("Email sent successfully!")
except EmailError as e:
print(f"Failed to send email: {e}")Configuration Notes:
- Requires an Azure Communication Services resource with email capability configured
- The
from_emailmust use a configured MailFrom address from your ACS resource - Set
email_sending_disabled=Trueduring development to prevent actual emails from being sent - Attachments must be base64-encoded before adding to the
EmailAttachmentobject
Includes helpers for tracing with OpenTelemetry.
| Function | Description |
|---|---|
start_as_current_span_with_method_name |
Decorator that runs a function in a new span, named after the function. |
all_methods_start_new_current_span_with_method_name |
Class decorator that does the same for every method. |
add_attributes_to_span, add_attributes_to_current_span |
Add key-value pairs to a span as attributes. |
add_kwargs_to_span, add_kwargs_to_current_span |
Add the named keyword arguments to a span as attributes. |
in_current_context |
Wraps a function to run in the calling thread's context, so work on a thread pool stays in the caller's trace. |
A thread pool's worker threads start with an empty context, so a span started on one, such as an instrumented HTTP
request's, begins a trace of its own. Wrap the function you hand the pool in in_current_context, on the thread
whose span it should be part of:
from concurrent.futures import ThreadPoolExecutor
from corvus_python.monitoring import in_current_context
with tracer.start_as_current_span("download_images"), ThreadPoolExecutor() as pool:
images = list(pool.map(in_current_context(download_image), urls))Provides storage configuration abstractions for data lake operations, with implementations for local, Azure Data Lake Gen2 and Microsoft Fabric OneLake storage.
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
DataLakeLayer |
Enum | Enumeration of data lake layers: BRONZE, SILVER, GOLD. |
from corvus_python.storage import DataLakeLayer |
StorageConfiguration |
Class (abstract) | Base class for storage configurations. Provides a get_full_path() method and storage_options dict. |
from corvus_python.storage import StorageConfiguration |
LocalFileSystemStorageConfiguration |
Class | Storage configuration backed by the local file system. Useful for local development. | from corvus_python.storage import LocalFileSystemStorageConfiguration |
AzureDataLakeFileSystemPerLayerConfiguration |
Class | ADLS Gen2 configuration where each data lake layer maps to a separate file system (bronze, silver, gold). |
from corvus_python.storage import AzureDataLakeFileSystemPerLayerConfiguration |
AzureDataLakeSingleFileSystemConfiguration |
Class | ADLS Gen2 configuration using a single file system with top-level folders for each layer. | from corvus_python.storage import AzureDataLakeSingleFileSystemConfiguration |
FabricLakehouseTablesConfiguration |
Class | Fabric OneLake configuration where each data lake layer maps to a separate Lakehouse (bronze, silver, gold by default), in one workspace or a workspace per layer. Paths are rooted in the Lakehouse's managed Tables/ area, for Delta tables. |
from corvus_python.storage import FabricLakehouseTablesConfiguration |
FabricLakehouseFilesConfiguration |
Class | As above, but rooted in the Lakehouse's unmanaged Files/ area, for raw files such as CSV, JSON or Excel. |
from corvus_python.storage import FabricLakehouseFilesConfiguration |
StorageConfigurationFileStorage |
Class | FileStorage rooted at one layer of any StorageConfiguration, for byte-level access (read, write, find the latest file matching a prefix, list subfolders). Uses obstore, so the configuration's storage_options work unchanged. |
from corvus_python.storage.storage_configuration_file_storage import StorageConfigurationFileStorage |
from corvus_python.storage import (
DataLakeLayer,
LocalFileSystemStorageConfiguration,
AzureDataLakeFileSystemPerLayerConfiguration,
AzureDataLakeSingleFileSystemConfiguration,
FabricLakehouseTablesConfiguration,
FabricLakehouseFilesConfiguration,
)
from corvus_python.storage.storage_configuration_file_storage import StorageConfigurationFileStorage
# Local filesystem (for development)
local_config = LocalFileSystemStorageConfiguration(base_path="./data")
path = local_config.get_full_path(DataLakeLayer.BRONZE, "my_database/my_table")
# -> ./data/bronze/my_database/my_table
# Azure Data Lake - separate file system per layer
adls_per_layer = AzureDataLakeFileSystemPerLayerConfiguration(
storage_account_name="mystorageaccount",
storage_options={"account_key": "..."},
)
path = adls_per_layer.get_full_path(DataLakeLayer.SILVER, "my_database/my_table")
# -> abfss://silver@mystorageaccount.dfs.core.windows.net/my_database/my_table
# Azure Data Lake - single file system
adls_single = AzureDataLakeSingleFileSystemConfiguration(
storage_account_name="mystorageaccount",
file_system_name="datalake",
)
path = adls_single.get_full_path(DataLakeLayer.GOLD, "my_database/my_table")
# -> abfss://datalake@mystorageaccount.dfs.core.windows.net/gold/my_database/my_table
# Fabric OneLake - separate Lakehouse per layer, with a class per Lakehouse area
tables = FabricLakehouseTablesConfiguration(
workspace_name="MyWorkspace",
lakehouse_names={DataLakeLayer.GOLD: "lh_gold"}, # optional; defaults to bronze/silver/gold
workspace_names={DataLakeLayer.GOLD: "MyReportingWorkspace"}, # optional; defaults to workspace_name
)
path = tables.get_full_path(DataLakeLayer.SILVER, "my_schema/my_table")
# -> abfss://MyWorkspace@onelake.dfs.fabric.microsoft.com/silver.Lakehouse/Tables/my_schema/my_table
files = FabricLakehouseFilesConfiguration(workspace_name="MyWorkspace")
path = files.get_full_path(DataLakeLayer.BRONZE, "raw/orders.csv")
# -> abfss://MyWorkspace@onelake.dfs.fabric.microsoft.com/bronze.Lakehouse/Files/raw/orders.csv
# Byte-level access to one layer of any configuration
bronze_files = StorageConfigurationFileStorage(files, DataLakeLayer.BRONZE)
latest = bronze_files.get_latest_matching_file_bytes("raw/orders_")Fabric only discovers managed tables at Tables/<table> or, in schema-enabled Lakehouses, Tables/<schema>/<table>. When using PolarsDeltaTableRepository with FabricLakehouseTablesConfiguration, pass an empty base_path so the database name becomes the schema.
Workspaces and Lakehouses must be given by name, not ID. OneLake rejects paths that mix IDs and names with 400 Bad Request, so the Fabric configurations raise ValueError if a value looks like an ID.
On the Fabric Spark runtime, OneLake traffic goes through a proxy that presents a self-signed CA certificate. Polars, deltalake and obstore use rustls, which rejects it with invalid peer certificate: CaUsedAsEndEntity. OpenSSL-based clients, the Fabric Python runtime and direct ADLS access are not affected. To read OneLake with Polars from a Spark notebook, pass allow_invalid_certificates=True. This disables TLS certificate validation for OneLake requests, and only takes effect on the Fabric Spark runtime; elsewhere it is ignored and an info message is logged. It removes protection against interception, so enable it only where it is needed.
Provides repository classes for reading and writing structured data across various storage backends, built on Polars. All repositories accept a StorageConfiguration to abstract over local and cloud storage.
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
DatabaseDefinition |
Dataclass | Defines a logical database by name and a list of TableDefinition instances. |
from corvus_python.repositories import DatabaseDefinition |
TableDefinition |
Dataclass | Defines a table by name, Pandera schema, optional title, and optional db_schema (the SQL schema name, e.g. dbo, used when creating SQL views over the table in Synapse or Fabric). |
from corvus_python.repositories import TableDefinition |
Manages Delta Lake tables within a specified data lake layer. Handles schema validation using Pandera and integrates with OpenTelemetry for tracing.
from corvus_python.repositories import PolarsDeltaTableRepository, DatabaseDefinition, TableDefinition
from corvus_python.storage import LocalFileSystemStorageConfiguration, DataLakeLayer| Method | Description |
|---|---|
read_data(table_name) |
Reads a Delta table into a LazyFrame. Returns None if the table is empty. |
overwrite_table(table_name, data, overwrite_schema=False) |
Overwrites the table after eagerly validating the full dataset against its Pandera schema. Takes a DataFrame or a LazyFrame, which it collects first. Writes the columns the schema declares not nullable as non-nullable. |
overwrite_table_lazy(table_name, data, overwrite_schema=False) |
Overwrites the table using streaming execution, for data too large to collect. Validates columns and types only, not values: duplicate unique keys and failed checks are written, and a null in a non-nullable column is only stopped by the table's own constraint. With overwrite_schema, every column becomes nullable, removing those constraints. |
overwrite_table_with_condition(table_name, data, predicate, overwrite_schema=False) |
Overwrites only rows matching the given predicate (e.g. for partition-level updates). |
append_to_table(table_name, data) |
Appends data to an existing Delta table. |
Reads CSV files from a hive-partitioned path (snapshot_time=<timestamp>/<name>.csv).
from corvus_python.repositories import PolarsCsvDataRepository| Method | Description |
|---|---|
load_csv(object_name, snapshot_timestamp, include_file_paths=None) |
Loads a CSV file into a DataFrame. Strips .csv suffix from object_name if present. |
Reads Excel workbooks from a hive-partitioned path, returning all sheets as a dict[str, DataFrame].
The workbook is read through obstore, so it works with local, Azure Data Lake and Fabric OneLake configurations, and authenticates with the configuration's storage_options in the same way as the CSV, NDJSON and Delta repositories.
from corvus_python.repositories import PolarsExcelDataRepository| Method | Description |
|---|---|
load_excel(snapshot_timestamp, workbook_name, relative_path=None) |
Loads all sheets from an .xlsx workbook into a dict keyed by sheet name. |
Reads Newline Delimited JSON (NDJSON) files from a partitioned path (<load_type>/snapshot_time=<timestamp>/<name>.json).
from corvus_python.repositories import PolarsNdJsonDataRepository| Method | Description |
|---|---|
load_ndjson(object_name, load_type, snapshot_timestamp, include_file_paths=None, schema_overrides=None, schema=None) |
Loads an NDJSON file into a DataFrame. Supports schema overrides and full schema specification. |
Queries Azure Table Storage, returning results as Polars DataFrames. Authenticates using DefaultAzureCredential.
from corvus_python.repositories import PolarsAzureTableRepository| Method | Description |
|---|---|
query(table_name, query_filter, parameters, schema=None) |
Queries an Azure Table with an OData filter and named parameters. |
get_entities_partition_key_starts_with(table_name, partition_key_prefix, schema=None) |
Retrieves all entities whose PartitionKey starts with the given prefix. |
Includes utility functions for working with SQL databases via pyodbc, with AAD token-based authentication. Provides helpers for connecting to Synapse serverless SQL and Fabric SQL Analytics endpoints, managing views over Delta Lake tables, and executing DDL statements.
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
get_pyodbc_connection |
Function | Opens a pyodbc connection to a Synapse serverless SQL or Fabric SQL Analytics endpoint using AAD tokens. | from corvus_python.sql import get_pyodbc_connection |
get_pyodbc_connection_with_token |
Function | Opens a pyodbc connection using a pre-acquired AAD token. Useful when running inside Synapse notebooks. | from corvus_python.sql import get_pyodbc_connection_with_token |
execute_ddl |
Function | Executes a DDL statement (e.g. CREATE VIEW) using a pyodbc connection. | from corvus_python.sql import execute_ddl |
create_or_alter_view_over_delta_table |
Function | Creates or alters a SQL view over a Delta Lake table, with support for inferred or explicit column types. | from corvus_python.sql import create_or_alter_view_over_delta_table |
drop_views_in_schema |
Function | Drops all views in a given schema, except any named in exclude (compared case-insensitively). |
from corvus_python.sql import drop_views_in_schema |
SelectColumn |
Class | Dataclass representing a column to select, with an optional display title. | from corvus_python.sql import SelectColumn |
WithColumn |
Class | Dataclass representing a column with an explicit type for use in OPENROWSET WITH clauses. | from corvus_python.sql import WithColumn |
Convenience wrappers specific to Synapse serverless SQL endpoints.
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
get_synapse_sql_pyodbc_connection |
Function | Opens a pyodbc connection to a Synapse serverless SQL endpoint using AAD tokens. Builds the server URL from the workspace name. | from corvus_python.sql.synapse import get_synapse_sql_pyodbc_connection |
get_synapse_sql_pyodbc_connection_with_token |
Function | Opens a pyodbc connection to a Synapse serverless SQL endpoint using a pre-acquired AAD token. Ideal for Synapse notebooks. | from corvus_python.sql.synapse import get_synapse_sql_pyodbc_connection_with_token |
create_database_if_not_exists |
Function | Creates a database if it doesn't already exist. Requires a connection to the master database. | from corvus_python.sql.synapse import create_database_if_not_exists |
Convenience wrappers specific to Fabric SQL Analytics endpoints.
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
get_fabric_sql_pyodbc_connection |
Function | Opens a pyodbc connection to a Fabric SQL Analytics endpoint using AAD tokens. | from corvus_python.sql.fabric import get_fabric_sql_pyodbc_connection |
Helpers for creating SQL views over the Delta tables described by TableDefinitions. Each view is named after the table's title (or its name) and selects the schema's columns, aliased to their titles.
| Component Name | Object Type | Description | Import syntax |
|---|---|---|---|
sync_sql_views_from_tables |
Function | Drops the views in a schema that no table produces, then creates or alters a view for each table. Use this rather than drop_views_in_schema followed by create_sql_views_from_tables. |
from corvus_python.sql.repositories import sync_sql_views_from_tables |
create_sql_views_from_tables |
Function | Creates or alters a view for each table. | from corvus_python.sql.repositories import create_sql_views_from_tables |
SqlViewCreationError |
Exception | Raised, inside an ExceptionGroup, for each view that failed to be created when creating views in parallel. Has schema_name and view_name attributes. |
from corvus_python.sql.repositories import SqlViewCreationError |
Both functions take an optional max_workers (default 1: views are created one at a time on conn, and the first error is raised). Above 1, views are created concurrently, with each worker using its own connection from the required connection_factory, since pyodbc connections can't be shared across threads. Every view is attempted, and the failures are raised together in an ExceptionGroup. Stale views are always dropped sequentially on conn.
from corvus_python.sql.repositories import sync_sql_views_from_tables
from corvus_python.sql.synapse import get_synapse_sql_pyodbc_connection_with_token
token = mssparkutils.credentials.getToken("DW")
def connect():
# Reuses the one token across every worker's connection.
return get_synapse_sql_pyodbc_connection_with_token("myworkspace", "my_database", token)
conn = connect()
try:
sync_sql_views_from_tables(
conn,
schema_name="dbo",
tables=TABLES,
build_delta_table_path_func=lambda table_name: f"abfss://silver@myaccount.dfs.core.windows.net/{table_name}",
max_workers=8,
connection_factory=connect,
)
finally:
conn.close()from corvus_python.sql import (
get_pyodbc_connection,
create_or_alter_view_over_delta_table,
drop_views_in_schema,
SelectColumn,
WithColumn,
)
# Connect using DefaultAzureCredential
conn = get_pyodbc_connection(
server="myworkspace-ondemand.sql.azuresynapse.net",
database="my_database",
use_managed_identity=False,
)
# Create a view with inferred types
create_or_alter_view_over_delta_table(
conn=conn,
schema_name="dbo",
view_name="my_view",
delta_table_path="abfss://container@storageaccount.dfs.core.windows.net/path/to/table",
infer_types=True,
select_columns=[
SelectColumn(name="id"),
SelectColumn(name="full_name", title="Name"),
],
)
# Create a view with explicit types
create_or_alter_view_over_delta_table(
conn=conn,
schema_name="dbo",
view_name="my_typed_view",
delta_table_path="abfss://container@storageaccount.dfs.core.windows.net/path/to/table",
infer_types=False,
with_columns=[
WithColumn(name="id", type="INT"),
WithColumn(name="full_name", type="VARCHAR(200)", title="Name"),
],
)
# Drop all views in a schema
drop_views_in_schema(conn, schema_name="dbo")Or use the Synapse/Fabric-specific helpers:
from corvus_python.sql.synapse import get_synapse_sql_pyodbc_connection
conn = get_synapse_sql_pyodbc_connection(
workspace_name="myworkspace",
database="my_database",
use_managed_identity=False,
)When running inside a Synapse notebook, you can use mssparkutils to acquire a token and pass it directly:
from corvus_python.sql.synapse import get_synapse_sql_pyodbc_connection_with_token
token = mssparkutils.credentials.getToken("DW")
conn = get_synapse_sql_pyodbc_connection_with_token(
workspace_name="myworkspace",
database="my_database",
token=token,
)