DataStage to Databricks

Why Databricks

DataStage's parallel engine has a natural successor

Parallel partitions map to Spark executors

DataStage parallel jobs already think in partitions, hash keys, and sort stages. PySpark on Databricks distributes the same logic across Spark executors with no manual repartitioning. What ran on one parallel engine now runs across a cluster.

Transformer logic becomes readable Python

Stage variables, derivations, and nested IF/THEN logic in Transformer stages become withColumn() calls with clear naming. No more reverse-engineering a Transformer expression buried inside a .dsx file.

Sequences become Databricks Workflows

DataStage Director sequences chain jobs with conditional logic, waits, and error handlers. Databricks Workflows handle the same DAG patterns with native retry, parameterized runs, and failure alerting built in.

Parser output

Transformer stage variables to PySpark columns

A Transformer stage with stage variables and derived outputs — converted to PySpark DataFrame operations on Databricks. Partition and sort logic preserved.

DataStage Transformer

-- Parallel Job: Customer_Risk_Score
-- Source: DB2_ACCOUNTS (DB2 connector)
-- Transformer stage:
--   sv_age = DateDiff(CurrentDate(),
--            lnk_acct.OPEN_DATE, "yy")
--   sv_balance = lnk_acct.BALANCE
--   out.RISK = If sv_age < 2 And
--     sv_balance > 50000 Then "HIGH"
--     Else If sv_age < 5 Then "MEDIUM"
--     Else "LOW"
-- Lookup: REGION_REF (reference link)
-- Target: RISK_SCORED (dataset)

→

MigryX
converts

PySpark on Databricks

# Transformer → PySpark on Databricks
from pyspark.sql import functions as F

df = spark.read.table("db2_accounts")
ref = spark.read.table("region_ref")

# Stage variables as columns
df = df.withColumn("sv_age",
    F.floor(F.datediff(
        F.current_date(), F.col("open_date")
    ) / 365))

df = df.join(F.broadcast(ref),
    df.region_id == ref.region_id, "left")

df = df.withColumn("risk",
    F.when((F.col("sv_age") < 2) &
           (F.col("balance") > 50000), "HIGH")
     .when(F.col("sv_age") < 5, "MEDIUM")
     .otherwise("LOW"))

df.write.format("delta").mode("overwrite") \
    .saveAsTable("risk_scored")

Stage variables become named columns. Lookup stages become broadcast joins. Derivations become withColumn() chains. Output writes to Delta Lake with ACID guarantees.

Coverage