AI
Proof
Pricing
Book a demo
Scan your code free
AI
Proof
Pricing
Book a demo
Scan your code free
Hard sources — not covered by free tools
Also parsed — certify what free tools miss
Runtimes
After migration
Parser-driven modernization of dfPower Studio and Data Management Studio jobs — standardization schemes, parsing, match rules, and validation logic. Full lineage, automated conversion, validated parity.
DataFlux data job parsed into interactive lineage graph
Standardization and match logic sit in scheme and Quality Knowledge Base definitions, not in code you can diff or review. When a customer name is transformed in production, no version-controlled artifact explains why.
dfPower and Data Management Studio jobs execute away from the warehouse, so data is copied out to be cleansed and copied back. Every cleansing step becomes network movement and another copy to secure.
DataFlux expertise is scarce and shrinking, and the tooling predates current data platforms. Estates that still depend on it usually cannot safely change a match rule, so the rules ossify.
A match rule generating match codes at a chosen sensitivity, then clustering records into survivorship groups. This is the part customers assume cannot be moved, because the behaviour comes from the Quality Knowledge Base rather than from the job definition.
-- Data job: CUSTOMER_DEDUPE.ddf
Reader : CUSTOMER_RAW
Standardization : NAME scheme NAME_UPPER
ADDRESS scheme ADDR_STD
locale ENUSA
Parsing : NAME → tokens
(GIVEN_NAME, MIDDLE, FAMILY_NAME)
Match codes : NAME sensitivity 85
ADDRESS sensitivity 75
Clustering : cluster on (MATCH_NAME, MATCH_ADDR)
output CLUSTER_ID
Surviving record : rule = most complete, newest
output SURVIVOR_FLAG
Writer : CUSTOMER_MASTER
-- Standardize, match, cluster, survive
WITH standardized AS (
SELECT customer_id, updated_at,
UPPER(TRIM(name)) AS name_std,
-- ADDR_STD scheme rules, made explicit
REGEXP_REPLACE(UPPER(TRIM(address)),
'\\bSTREET\\b', 'ST') AS addr_std
FROM customer_raw
),
match_keys AS (
SELECT s.*,
-- match code at the configured sensitivity
SOUNDEX(name_std) AS match_name,
LEFT(REGEXP_REPLACE(addr_std,
'[^A-Z0-9]', ''), 12) AS match_addr
FROM standardized s
),
clustered AS (
SELECT m.*,
DENSE_RANK() OVER (ORDER BY match_name,
match_addr) AS cluster_id,
ROW_NUMBER() OVER (
PARTITION BY match_name, match_addr
ORDER BY updated_at DESC) AS survivor_rank
FROM match_keys m
)
SELECT *, (survivor_rank = 1) AS survivor_flag
FROM clustered;
Each scheme becomes a readable transformation, match codes become explicit key expressions at the configured sensitivity, and clustering and survivorship become window functions. The rules are now reviewable in version control instead of hidden in scheme files.
Every DataFlux operation in your jobs maps to a defined target equivalent, recorded in the lineage report.
| DataFlux Operation | Target Equivalent | Notes |
|---|---|---|
| Data job (.ddf) | Transformation pipeline | Node graph and column flow preserved |
| Process job (.djf) | Orchestrated job with branching | Control flow and conditions mapped |
| Reader / Writer nodes | Table or file read and write | Connections mapped to target connectors |
| Standardization scheme | Explicit transformation rules | Scheme entries become readable logic |
| Parsing definition | Token extraction into columns | Parsed tokens become named columns |
| Match codes and sensitivity | Deterministic key expressions | Sensitivity level carried into key design |
| Clustering | Grouped cluster identifiers | Cluster IDs generated deterministically |
| Surviving record rules | Ranked survivorship | Rule order becomes an explicit ordering |
| Validation and casing | Constraint checks and expressions | Failures routed to an exception table |
| Identification analysis | Classification expressions | Locale-specific behaviour made explicit |
| Expression nodes | Derived columns | DataFlux expression logic translated |
| Quality Knowledge Base locale | Documented rule set | Locale-dependent behaviour reported for review |
MigryX Data Matching compares DataFlux job output against the new pipeline output, row by row and column by column, which matters most for cluster assignment and survivorship, where differences must be understood rather than tolerated.
See how Data Matching works →The job definitions, meaning data jobs and process jobs, together with the scheme files and match definitions they reference. The referenced schemes matter as much as the jobs, because that is where the transformation behaviour is actually defined.
Match logic is re-expressed as deterministic, reviewable key generation and clustering at the sensitivity the job configured. It is not a bit-for-bit reimplementation of the Quality Knowledge Base, so cluster assignment is compared explicitly during validation and any differences are surfaced for a decision.
Locale-specific rules are reported alongside the converted logic rather than assumed equivalent. Name and address handling varies by locale, so this is treated as something to review with the data stewards who own those rules.
Usually, yes. Running standardization and matching where the data already lives removes the copy out and back to a separate quality tier, and it makes the rules visible to everyone querying the data instead of only to DataFlux users.