Jun
8
2008

New product: Data Quality Manager from Australia

Microsoft consultant David Lean from Australia sent us information about a new data quality product from down under named Data Quality Manager. There's a wealth of information about the product on the site. It uses, and integrates with, Integration Services. Here's some information about their SSIS integration:

SSIS integration

The core components of the products are encapsulated in stored procedures and functions, developed in T-SQL for superior performance. These functions are exposed as objects on the SSIS work flow canvas or made available as components on the DQM job design palette. So,all the pre-built functions, cleansing & match modules can be invoked by SSIS or be an integral part of the SSIS data flow.

This screen shot may not show up well online, but in the Toolbox, you see the large number of custom components available to you in the "DQM Profile & Investigate" and the "DQM Cleanse & Match" groups:

David Lean provided the following additional information and comments:

Their product is very cool, uses nearly every component of SQL Server & is SQL 2008 ready.

 

I know there are at least 3 other Data Cleansing apps that are closely aligned with SQL, that also build SSIS packages. So why these guys?

 

  • Not only do they do Syntactic / Rules Based cleansing ie:-
    • Valid phone numbers are (999) 999999
    • USA Addresses are usually “Number, Street, Suburb” whereas some European are “Street, Number, Suburb”
  • They also provide local reference data so they can do Semantic based Cleansing. Ie:-
    • Phone Numbers with Area code (425) are unlikely to be in Texas. OR
    • Firstname = Mary is unlikely to be a Title = “Mr”
    • Uses Spatial to find the actual location of an address to ensure it makes sense & can add geocode the Long/Lat if desired.
  • They also let you validate based on probability of your other data, ie: Fuzzy Matching, Grouping & other Data Mining.

They provide templates to help get the project started & SQL Reports / Analytics to see how it is going.

You can use their UI tool to create packages. Or more Technical people can use BIDS to add Data Cleansing Tasks into an existing  SSIS package.

 

It is worth checking out

------------

-Doug