Teradata to Databricks

Architecture

Teradata BTEQ in. Databricks out.

Deterministic parsers read the estate and emit native Databricks code — not BTEQ replayed through a Teradata-compatible shim.

Teradata BTEQ → MigryX parser → PySpark + Delta + Workflows

Teradata BTEQ

BTEQ scripts.bteq / SQL

FastLoadBulk ingest

MultiLoad / TPumpUpsert paths

QUALIFY + PIWindow + index

MigryX Parser

Deterministic parseAI where it helps

Row-level parityBefore cutover

PySpark emitSet-based, not loops

Workflow emitJob DAGs

Databricks

PySpark notebooksSet-based, not loops

Delta LakeACID + time travel

WorkflowsReplaces scheduler

Unity CatalogColumn lineage

Repos + DABsCI/CD promote

MLflowWhere models live

MigryX AI handles the logic parsers cannot resolve alone, and every change it makes goes through the same parity checks. It runs on a model you approve, air-gapped if your estate requires it.

Why Databricks

Teradata wasn't built for open lakehouse economics

BTEQ scripts become PySpark notebooks

BTEQ's .LOGON, .IF/.THEN, .GOTO control flow and embedded SQL are parsed and converted to idiomatic PySpark notebooks. .LABEL and .QUIT logic becomes structured Python error handling. No more proprietary CLI scripting.

QUALIFY and window functions translate cleanly

Teradata's QUALIFY clause with ROW_NUMBER(), RANK(), and DENSE_RANK() is converted to PySpark Window functions with a .filter() step. The analytical logic is preserved exactly—no manual rewrite needed.

FastLoad and MultiLoad become Auto Loader and COPY INTO

Teradata's proprietary bulk-loading utilities—FastLoad, MultiLoad, FastExport, TPump—are replaced by Databricks Auto Loader for streaming ingestion and COPY INTO for batch loads. No Teradata channel drivers required.

Parser output

BTEQ with QUALIFY and MERGE to PySpark on Delta

A BTEQ script using QUALIFY ROW_NUMBER() for deduplication and MERGE INTO for upserts—converted to PySpark Window functions and Delta Lake merge operations.

Teradata BTEQ

.LOGON tdserver/dbc,dbc;

COLLECT STATISTICS ON customer_orders
  COLUMN (customer_id);

CREATE VOLATILE TABLE latest_orders AS (
  SELECT customer_id, order_date, amount,
         order_status
  FROM customer_orders
  QUALIFY ROW_NUMBER() OVER (
    PARTITION BY customer_id
    ORDER BY order_date DESC
  ) = 1
) WITH DATA ON COMMIT PRESERVE ROWS;

MERGE INTO customer_summary tgt
USING latest_orders src
ON tgt.customer_id = src.customer_id
WHEN MATCHED THEN UPDATE SET
  last_order_date = src.order_date,
  last_amount     = src.amount,
  status          = src.order_status
WHEN NOT MATCHED THEN INSERT VALUES (
  src.customer_id, src.order_date,
  src.amount, src.order_status
);

.LOGOFF;
.QUIT;

→

MigryX
converts

PySpark on Databricks

# BTEQ → PySpark on Databricks
from pyspark.sql import functions as F
from pyspark.sql.window import Window
from delta.tables import DeltaTable

spark.sql("ANALYZE TABLE customer_orders
  COMPUTE STATISTICS FOR COLUMNS customer_id")

w = Window.partitionBy("customer_id") \
          .orderBy(F.col("order_date").desc())

latest_orders = (
    spark.read.table("customer_orders")
    .withColumn("_rn", F.row_number().over(w))
    .filter(F.col("_rn") == 1)
    .drop("_rn")
)

tgt = DeltaTable.forName(spark, "customer_summary")
tgt.alias("tgt").merge(
    latest_orders.alias("src"),
    "tgt.customer_id = src.customer_id"
).whenMatchedUpdate(set={
    "last_order_date": "src.order_date",
    "last_amount":     "src.amount",
    "status":          "src.order_status"
}).whenNotMatchedInsertAll().execute()

BTEQ control flow removed. QUALIFY ROW_NUMBER() becomes Window + filter. COLLECT STATISTICS becomes ANALYZE TABLE. MERGE INTO becomes Delta Lake merge. Volatile table becomes DataFrame.

Coverage