v4.2.1-rc1
The release entry describes preparation for the Spark release v4.2.1-rc1.
Read more
What changed
Preparation is underway for Spark release v4.2.1-rc1.
11 articles · Showing 1–10
CrowdStrike Falcon Event Stream connector (Beta)
The CrowdStrike Falcon Event Stream connector is available in Beta in Lakeflow Connect. It ingests Falcon Event Stream events from CrowdStrike Falcon into Databricks. Authentication uses OAuth 2.0 client credentials for a CrowdStrike API client with Event streams read permission. Workspace administrators can control access through the Previews page.
Read more
What changed
Lakeflow Connect adds a Beta connector for ingesting CrowdStrike Falcon Event Stream events into Databricks. It authenticates with OAuth 2.0 client credentials for an API client with Event streams read permission, and access can be controlled through the Previews page.
Why it matters
The connector provides a way to ingest CrowdStrike Falcon event stream data into Databricks over a connection authenticated with OAuth 2.0.
Practical impact
Organizations seeking to ingest and analyze CrowdStrike Falcon events in Databricks can use the update to evaluate Lakeflow Connect.
Workday Activity Logging connector (Beta)
The Workday Activity Logging connector is available in Beta in Lakeflow Connect. It incrementally ingests Workday user activity log events into Databricks and authenticates with an OAuth refresh token. Workspace administrators can control access through the Previews page.
Read more
What changed
Lakeflow Connect adds a Beta connector for incremental ingestion of Workday user activity log events. It uses an OAuth refresh token for authentication, and workspace administrators can control access through the Previews page.
Why it matters
The connector provides an integration for bringing Workday user activity log events into Databricks and building workflows around those logs.
Practical impact
The feature is relevant to administrators who want to ingest Workday logs incrementally into Databricks.
Thinking Machine Labs Inkling is scheduled for retirement
Thinking Machine Labs Inkling is scheduled to retire on October 30, 2026. The release recommends GLM 5.3 and Kimi K3 as alternatives and points to the documentation on retired and deprecated models for migration guidance.
Read more
What changed
Thinking Machine Labs Inkling is scheduled for retirement on October 30, 2026. GLM 5.3 and Kimi K3 are recommended alternatives in the release.
Why it matters
Users of the retiring model need to plan a migration.
Practical impact
The notice is relevant to Inkling users preparing to assess alternative models and migrate before retirement.
Akamai WAF connector (Beta)
The Akamai WAF connector is available in Beta in Lakeflow Connect. It ingests Akamai Web Application Firewall security events into Databricks and authenticates with Akamai EdgeGrid API client credentials. Workspace administrators can control access through the Previews page.
Read more
What changed
Lakeflow Connect adds a Beta Akamai WAF connector for security event ingestion. It uses Akamai EdgeGrid API client credentials for authentication, and workspace administrators can manage access through the Previews page.
Why it matters
The connector expands the ingestion paths for bringing Akamai WAF security events into Databricks.
Practical impact
The feature is relevant to organizations that want to ingest Akamai WAF security events into Databricks.
Microsoft 365 Unified Audit Logs connector is available in Beta
A managed Microsoft 365 Unified Audit Logs connector is available in Beta in Databricks Lakeflow Connect. It ingests unified audit events from Microsoft Entra ID, Exchange, SharePoint, and other Microsoft 365 workloads into Databricks. The connector supports incremental ingestion with OAuth machine-to-machine authentication.
Read more
What changed
Lakeflow Connect adds a Beta connector for unified audit events from Microsoft 365 workloads including Microsoft Entra ID, Exchange, and SharePoint. It supports incremental ingestion using OAuth machine-to-machine authentication.
Why it matters
The connector provides managed ingestion of unified audit events from Microsoft 365 workloads and Microsoft Entra ID, with OAuth machine-to-machine authentication and incremental processing.
Practical impact
The update is relevant to readers who want to incrementally ingest audit events from Microsoft 365 workloads and Microsoft Entra ID into Databricks.
Okta System Logs connector is available in Beta
A managed Okta System Logs connector is available in Beta in Databricks Lakeflow Connect. It incrementally ingests audit and security events from Okta system logs into Databricks. The connector authenticates with an SSWS API token and supports Declarative Automation Bundles and API-based pipeline authoring.
Read more
What changed
Lakeflow Connect adds a Beta Okta System Logs connector supporting SSWS API token authentication, incremental ingestion of audit and security events, Declarative Automation Bundles, and API-based pipeline authoring.
Why it matters
The connector provides a managed way to bring audit and security events from Okta system logs into Databricks.
Practical impact
The update is relevant to readers building incremental ingestion pipelines for Okta audit and security events in Databricks Lakeflow Connect.
Tags for dashboards, notebooks, Databricks apps, and Genie Agents are now generally available
Tags for dashboards, notebooks, Databricks apps, and Genie Agents are now generally available. Users can apply both governed and ungoverned key-value tags to organize and categorize workspace assets. The release points to the documentation for applying tags to Unity Catalog securable objects.
Read more
What changed
Applying governed and ungoverned key-value tags to dashboards, notebooks, Databricks apps, and Genie Agents is now generally available.
Why it matters
Key-value tags provide a way to organize and categorize different types of workspace assets.
Practical impact
The update is relevant to readers considering how to organize dashboards, notebooks, Databricks apps, and Genie Agents.
How Delivery Hero rebuilt real-time ad measurement with Apache Flink
Delivery Hero moved its ad measurement platform from hourly batch processing to streaming with Amazon Managed Service for Apache Flink. In this company's implementation, the average time from event emission to recording fell from 61 minutes to 1.2 seconds, and monthly operating costs fell by approximately 57%. The pipeline deduplicates events, enriches data asynchronously, links ad interactions with orders across multiple days, and improves missing event information such as session IDs. Aggregated results are written to the reporting database every five minutes.
Read more
What changed
The previous architecture used synchronous API calls for enrichment and hourly batch aggregation. The new Flink pipeline expands and decrypts events, deduplicates them, enriches them asynchronously, and performs attribution and aggregation. Events arrive through Kinesis Data Streams, reference data comes from DynamoDB, and S3 stores events and checkpoints. Delivery Hero used EventBridge Pipes to move data from SQS to Kinesis in the MVP and Fargate in production.
Why it matters
Processing-time aggregation caused discrepancies when events arrived late or out of order, while hourly batches delayed reporting. Other challenges included scalability limits from synchronous APIs, database load for state management, complex reprocessing, and missing event information. Addressing these issues supported accurate ad billing and faster ad delivery decisions at Delivery Hero.
Practical impact
This case is relevant to ad-tech platforms that need budget pacing and ad measurement within seconds, and to engineers and architects managing state, deduplication, and attribution in large streaming pipelines.
Sep 24, 2026: DCM Projects capability updates
This release updates the availability of several DCM Projects capabilities. Statements including CREATE OR ALTER PIPE and CREATE OR ALTER STREAM, inherited grants, and container-level MANAGE GRANTS are now generally available. Preview additions include importing project assets and declaratively managing Code Bundles and Streamlit apps.
Read more
What changed
CREATE OR ALTER PIPE, CREATE OR ALTER STREAM, DEFINE PIPE, DEFINE SHARE, DEFINE STREAM, inherited grants, and container-level MANAGE GRANTS are now generally available. Preview capabilities include project assets that import source files declared in the project manifest, DEFINE CODE BUNDLE for declarative management of Code Bundles and their source files, and DEFINE STREAMLIT for declarative management of Streamlit apps, infrastructure, and access control.
Why it matters
The update changes availability stages and expands the set of generally available and preview capabilities in DCM Projects.
Practical impact
Readers managing pipelines, streams, Code Bundles, or Streamlit apps with DCM Projects can use this update to check which capabilities are generally available and which remain in preview.