This page summarises the Compliance Rules currently shipping with MettleCI. Each Compliance Rule file has a .grm extension, referring to the Gremlin language in which it is written. The three-letter sub-extension before .grm refers to the type of DataStage asset to which the rule can be applied:
|
Compliance Rule Extension |
Asset Type |
|---|---|
|
|
Parallel job |
|
|
Sequence job |
|
|
Server job |
|
|
Parallel shared container |
|
|
Server shared container |
You can read more about the MettleCI compliance commands here.
The current set of sample Compliance Rules are detailed below:
|
Compliance Rule |
Parallel Job |
Server Job |
Job Sequence |
Description |
|---|---|---|---|---|
|
Yes |
Yes |
- |
Identified job designs with adjacent Transformer stages. |
|
|
Yes |
- |
- |
Identifies Parallel Aggregator Stages not preceded by a ‘Check’ Sort Stage. |
|
|
Yes |
Yes |
Yes |
Identifies where sensitive information may potentially be present in DataStage Job and Sequence Annotations |
|
|
Yes |
Yes |
- |
Identifies the use of deprecated stages in Parallel Jobs and suggests an alternative |
|
|
- |
Yes |
- |
Identifies stage columns with names using characters not supported by the Parallel Engine. |
|
|
Yes |
Yes |
- |
Identifies stages that are not yet supported by IBM Cloud Pak for Data |
|
|
Yes |
Yes |
- |
This rule detects the use of Custom stages in Parallel Jobs |
|
|
Yes |
- |
- |
Identifies Data Sets not using the 'Same' partitioning method |
|
|
Yes |
Yes |
- |
Ensures a Database Connector uses Auto Generated SQL |
|
|
Yes |
Yes |
Yes |
Identifies Connectors Stages with a configured Database Row Limit |
|
|
Yes |
Yes |
- |
Identifies where database table references are fully-qualified |
|
|
Yes |
Yes |
- |
Lists the stages that are not yet supported by IBM Data Flow Designer |
|
|
Yes |
Yes |
Yes |
Identify whether the a ‘Job Description’ annotation contains instances of particular arbitrary text. This example rule looks for dates. |
|
|
Yes |
Yes |
- |
Identify Db2 Stages using bulk load with Non Recovery Load set to No |
|
|
Yes |
Yes |
- |
Ensures that all Database Connectors and Stages must use valid variants |
|
|
Yes |
- |
- |
Identifies row limits in debug stages (Peek, Sample, Tail) |
|
|
Yes |
Yes |
Yes |
Default Stage Names for Parallel Jobs, Server Jobs, and Job Sequences |
|
|
Yes |
Yes |
- |
Identify deprecated stages and suggests the recommended alternative |
|
|
Yes |
Yes |
- |
Identifies calls to deprecated functions in the Transformer Stage code generated by DataStage |
|
|
Yes |
Yes |
- |
Verifies that a file is only referenced once in a DataStage job (Sequential File, Complex Flat File and DataSet) |
|
|
Yes |
Yes |
- |
Detect duplicate Stage names in a job |
|
|
Yes |
Yes |
- |
Ensures that all Connectors and Stages must use encrypted Passwords |
|
|
Yes |
Yes |
- |
Ensures that all File Stages must use variables for determining paths |
|
|
Yes |
Yes |
- |
Identifies row limits in file-based stages (Sequential File, Complex Flat File) |
|
|
Yes |
Yes |
- |
Ensures that all Database Connectors and Stages must use variables for location and credentials |
|
|
Yes |
Yes |
- |
Ensures that all File Stages must use variables for determining paths |
|
|
Yes |
Yes |
- |
Here |
|
|
Yes |
Yes |
- |
Here |
|
|
Yes |
- |
- |
Detects the stages in Parallel Jobs that are not currently support by DataStage Saas |
|
|
Yes |
- |
- |
Identifies the use of the Java Integration Stage which is not supported by DataStage SaaS. |
|
|
- |
- |
Yes |
Identify Sequence Job activities with references to non-existent parameters. |
|
|
- |
- |
Yes |
Sequence Job with Hardcoded Parameter Values |
|
|
Yes |
Yes |
- |
Identifies Jobs with Job Control Routines defined |
|
|
Yes |
Yes |
Yes |
Identifies Jobs or Sequences where the MettleCI Unit Testing parameter is not present |
|
|
Yes |
Yes |
Yes |
Checks the job name against a blocklist of known bad names (e.g. CopyOf*) |
|
|
Yes |
Yes |
Yes |
Identify Job parameters within a default value |
|
|
Yes |
Yes |
Yes |
Identifies where naming standards for Job Parameters and Parameter Sets are breached |
|
|
Yes |
Yes |
Yes |
Identifies Job parameters which are not members of a parameter sets |
|
|
Yes |
- |
- |
Identifies Job parameters which are not used within the job. |
|
|
- |
- |
Yes |
Verifies that a job sequence automatically handles failed activities |
|
|
- |
- |
Yes |
Identifies Job Sequences that are not restartable |
|
|
Yes |
- |
- |
Identifies job using an ODBC connector |
|
|
Yes |
- |
- |
Recommends considering replacing a Transformer Stage using a Surrogate Key with a Surrogate Key Generator stage. |
|
|
Yes |
- |
- |
Identifies Join stages where the join key does not match the partitioning key |
|
|
Yes |
- |
- |
Identifies link sorts |
|
|
Yes |
Yes |
- |
Identify Stages configured to Log column values on first row error |
|
|
Yes |
- |
- |
Identifies Lookup Stages with Lookup set to Fail |
|
|
Yes |
Yes |
- |
Verifies that there is only one data flow in a DataStage job |
|
|
Yes |
- |
- |
Identify Oracle Stages not configured to use partitioned reads |
|
|
Yes |
- |
- |
Detect an 'Abort after rows' greater than 0 setting in a Parallel Transformer. |
|
|
Yes |
- |
- |
Identify the use of a Rnd() function in Parallel Transformers. |
|
|
Yes |
Yes |
- |
Identify unencrypted Job parameters used for storing passwords |
|
|
Yes |
Yes |
- |
Verifies that a job does not contain one or more prohibited stages |
|
|
Yes |
- |
- |
Checks that range lookups are correctly configured |
|
|
Yes |
Yes |
- |
Identifies redundant sorts within a job |
|
|
Yes |
- |
Identifies Row Splitter Stage with output format delimited but no delimiter specified |
||
|
Yes |
- |
- |
Detect the use of Schema Files |
|
|
Yes |
Yes |
- |
Identifies Connector Stages using customer SQL employing a ‘SELECT *’ clause |
|
|
- |
- |
Yes |
Identifies where a Job Activity Execution Action is not Reset if required, then run |
|
|
Yes |
- |
- |
Avoid Reading from Sequential Files Using the Same Partitioning Method |
|
|
Yes |
- |
- |
Identifies Sequential Files with a Reject Mode not set to fail |
|
|
Yes |
Yes |
- |
Identifies potentially redundant sorting (a sort stage or link sort) situated immediately after a Join stage |
|
|
Yes |
Yes |
- |
Identify the use of hardcoded SQL in Database Connectors or Enterprise Database Stages. |
|
|
Yes |
Yes |
Yes |
Stage Naming Standards for Parallel and Server Jobs and Job Sequences |
|
|
Yes |
Yes |
- |
This rule detects the use of stages in Parallel Jobs that don't support automatic lineage |
|
|
Yes |
- |
- |
Identifies the use of system time in the job |
|
|
Yes |
Yes |
- |
Identifies whether a Job has too many Stages |
|
|
Yes |
Yes |
- |
Identifies Transformer Stage with an unreferenced Stage Variable (including Loop Variables) |
|
|
Yes |
Yes |
- |
Identifies unique sorts |
|
|
Yes |
Yes |
- |
Identifies where a Db2 Sequence is used to provide a surrogate key value. |