DataStage to BigQuery

Why BigQuery

GCP-first estates don't need a DataStage hop

Serverless warehouse, zero infrastructure

DataStage parallel engines require server sizing, patching, and capacity planning. BigQuery runs SQL at warehouse scale with no infrastructure to manage. Pay per query, not per server.

BigQuery SQL for most jobs, Dataproc for the rest

Simple Transformer derivations become BigQuery SQL views and scheduled queries. Complex stage-variable logic with nested conditionals becomes PySpark on Dataproc, writing results back to BigQuery tables.

Cloud Composer replaces Director sequences

DataStage Director sequences manage job dependencies and error handlers. Cloud Composer (managed Airflow) handles the same DAG orchestration with built-in retry, alerting, and cross-service integration across GCP.

Parser output

Transformer derivation to BigQuery SQL

A Transformer stage with aggregation and a Lookup reference link — converted to a BigQuery SQL query with JOIN and GROUP BY.

DataStage Transformer

-- Parallel Job: Region_Revenue
-- Source: SALES_EXTRACT (connector)
-- Lookup: REGION_REF (reference link)
-- Transformer stage:
--   out.REGION = ref.REGION_NAME
--   out.REVENUE = lnk.QTY * lnk.PRICE
-- Aggregator stage:
--   GroupBy: REGION
--   Sum: REVENUE → TOTAL_REVENUE
--   Count → ORDER_COUNT
-- Target: REGION_SUMMARY (dataset)

→

MigryX
converts

BigQuery SQL

-- Transformer + Aggregator → BigQuery SQL
CREATE OR REPLACE TABLE
  `project.dataset.region_summary` AS
SELECT
    r.region_name AS region,
    SUM(s.qty * s.price) AS total_revenue,
    COUNT(*) AS order_count
FROM `project.dataset.sales_extract` s
LEFT JOIN `project.dataset.region_ref` r
    ON s.region_id = r.region_id
GROUP BY r.region_name;

Transformer derivations and Aggregator stages collapse into a single SQL query. Lookup reference links become JOINs. When logic requires Python, MigryX outputs PySpark on Dataproc instead.

Coverage