Skip to main content

YouTube
LinkedIn
www

Main

MettleCI Documentation

Introduction

Welcome to MettleCI v2.0

Why MettleCI?

MettleCI Functional Overview
Use Cases

MettleCI Installation Roadmap
Definition of a Unit Test
Development Model Options & Recommendations

Installation and Configuration

Installation Topologies

A Summary of MettleCI Components
Standalone DataStage on Windows - Dedicated Agent Host
Standalone DataStage on Windows - Consolidated Agent Host
Standalone DataStage on Unix - Dedicated Agent Host
Standalone DataStage on Different Versions - Concurrent use for upgrades
Virtual Desktop Example Topology
The Mettle Agent Host
Using Multiple MettleCI Agents
MettleCI Components for Upgrades to DataStage to v11.7.X
MettleCI for DataStage Failover Configurations

MettleCI For DevOps - Technical Prerequisites

MettleCI - Infrastructure Requirements
MettleCI - User Accounts
MettleCI - Firewall Rules & Security
MettleCI - Software
MettleCI - Component Connections

Pre-installation checklist
MettleCI Installation Walkthrough Videos
Accessing the MettleCI Software Distribution
Prerequisite Java Installation

Installing MettleCI Workbench

MettleCI Workbench Installation Process

Installing or Upgrading Workbench on Unix
Installing or Upgrading Workbench on Windows

MettleCI Workbench Post-installation Setup

MettleCI Workbench Setup Wizard
MettleCI Workbench Initial Login
MettleCI Workbench Configure First Project
Integrating MettleCI Workbench and DataStage Designer on Windows

Customizing your MettleCI Workbench Installation

MettleCI Workbench Configuration Files
Configuring MettleCI Workbench to use a Custom Port Number
Configuring MettleCI Workbench to use HTTPS
Configuring MettleCI Workbench to use HTTPS with an existing certificate
Configuring MettleCI Workbench to use Custom App and Data Directories
Configuring MettleCI Workbench to communicate with Git over HTTPS
Configuring MettleCI Workbench to access systems via an HTTP/S proxy
Configuring MettleCI Workbench to increase session timeout
Configuring MettleCI Workbench to use 'secrets' to improve security of HTTPS connections

Installing or Upgrading the MettleCI Unit Test Harness

How the Parallel Unit Test Harness Integrates with DataStage
Installing the Unix Parallel Unit Test Harness
Installing the Windows Parallel Unit Test Harness
Installing the Server Unit Test Harness
Configuring Jobs for Unit Testing

Installing MettleCI Command Line Interface

Installing MettleCI CLI on Windows
Installing MettleCI CLI on Unix
Installing or Upgrading Individual MettleCI CLI Commands

Getting Started with MettleCI Workbench

Registering a DataStage Project with MettleCI
Add or modify a work item service integration in Workbench

Deploying MettleCI repository templates to your Git platform
Creating a Local Compliance Rule Git Repository

Compliance Testing

Compliance Testing from Workbench
Compliance Rule Tags
Making CI dependent on specific Compliance Rules

Compliance Rules Reference

Adjacent Transformers
Aggregator Not Preceded by ‘Check’ Sort
Asset using Java Integration Stage
Audit Annotation
CCMigrateTool Stages
Column Name Contains Unsupported Characters
Job Using Transformer Surrogate Key
Custom Stages
Data Sets not using 'Same' partitioning method
Database Connector does not Auto-Generate SQL
DataStage Flow Designer Stages
Database Row Limit
Date Format in Annotation
Database tables references are fully qualified
DataStage SaaS Unsupported Stages
DB2 with No Non Recovery Load
DBConnector Supported Variants
Debug Row Limit
Default Naming
Deprecated Stages
Deprecated Transformer Functions
Duplicate File References
Duplicate Stage Names
Encrypted DB Passwords
File Reference Missing Required Parameter
File Row Limit
Hardcoded DB Credentials
Hardcoded File Paths
CP4D Unsupported Stages
Job Activity References Deleted Child Parameter
Job Activity with Hardcoded Parameter Values
Job Control Routines are present
Job Does Not Contain DM Unit Test Parameter
Job Naming
Job Parameter Missing Default Value
Job Parameter Naming
Job Parameter Not in a Parameter Set
Job Parameter Not Used in a Job
Job Sequence Handles Activities That Fail
Job Sequence Is Restartable
Job Using Custom Function
Job Using ODBC Connector
Jobs With Before and After Routine
Join Partition vs Join Key
Link Sort
Log Column Values
Lookup Failure
One Dataflow
Oracle Connector not using Partition Read
Password Param Type not Encrypted
Prohibited Stages
Range Lookup
Redundant Sort
Row Splitter with No Delimiter
SELECT * used in custom SQL
Sequence Reset If Required
Sequential File Read Using Same Partitioning
SQL in DB Connectors
Schema Files
Sequential File With Reject Mode Not Set To Fail
Sort Post Join Stage
Stage Naming
Stages Not Supporting Lineage
System Time Dependency
Too Many Stages
Transformer Uses 'Abort after rows'
Transformer Uses 'Rnd()' Function
Transformer With Unreferenced Stage Variable
Unique Sort
Using DB2 Sequence to Generate Surrogate Key

Developing Custom Compliance Rules

Compliance Rule Graphs
Compliance Rule Development
Example Compliance Rule - Checking for Adjacent Transformers
Example Compliance Rule - Job Sequence is Restartable
Example Compliance Rule - Asset Naming Standards
Accessing Columns and Derivations using Compliance Rules
Avoiding a NullPointerException when processing Compliance Rules

Automated Unit Testing

Creating a Unit Test

Unit Test Specification Format
Intercepting Existing Test Data
Managing your Unit Test data volumes
Manually Editing Unit Test Data
Capturing a Baseline Test Result
Row Count Unit Testing

Executing a Unit Test
Unit Testing Job Sequences

Data Fabrication

Using MettleCI data fabrication
MettleCI’s data fabrication tools

Creating Custom Data Fabricators

Creating a custom test data generator bundle
Test your custom generator bundle at the command line
Upload a custom data fabrication bundle to MettleCI Workbench
Use your custom data generators in test data specifications

Unit Testing Patterns

Unit Testing Stages with Rejects
Unit Testing Stored Procedure Stages
Unit Testing Surrogate Key Generator Stages
Unit Testing Sparse Lookup Stages
Unit Testing Jobs with current date calculations
Omitting selected columns from Unit Testing
High Volume Unit Tests
Creating and Running Multiple Unit Tests for a Single Job
Unit Tests featuring Local and Shared Containers
Unit Testing Slowly Changing Dimension Stages

Command Line Interface

CLI Plugins

Command Reference

Compliance Namespace

Compliance Console Command 🔒
Compliance List-Tags Command
Compliance Query Command
Compliance Test Command

DataStage Namespace

DataStage Capture Command
DataStage Connector Migration Command
DataStage Cleanup-Projects Command
DataStage Compile Command
DataStage Create-Project Command
DataStage Delete-Project Command
DataStage Deploy Command
DataStage Execute Command

DSParams Namespace

DSParams Delete Command
DSParams Diff Command
DSParams Merge Command

Fabrication Namespace

Fabrication List Command
Fabrication Test Command

ISX Namespace

ISX Cat Command
ISX Cut Command
ISX Export Command
ISX Import Command
ISX Message-Handlers Command
ISX Set-Params Command

Properties Namespace

Properties Config Command

Remote Namespace

Remote Download Command
Remote Execute Command
Remote Upload Command

UnitTest Namespace

UnitTest Generate Command
UnitTest Install-Server-Test-Harness Command
UnitTest Migrate Command
UnitTest Test Command
UnitTest Uninstall-Server-Test-Harness Command

Workbench Namespace

Workbench Set-Branch Command

Git Support

Git and MettleCI
Trunk Based Development
Working with DataStage and Git Branches
Organising your Git repositories
Creating and Preparing a Git Repository for DataStage Assets
DataStage Git Repository Structure
Committing DataStage Assets to Git

Signed Commits

Why Sign Commits?
Configuring Signed Commits
Platforms Supporting Signed Commits

Addressing Production defects using Hotfixes

Handling Production Defects
Developing a hotfix
Configuring a Hotfix Trigger

MettleCI Integrations

Atlassian

Atlassian Jira

Jira and MettleCI Workbench via OAuth explained
Configuring Atlassian Jira issue lookup with MettleCI Workbench

Atlassian Bamboo

Atlassian Bamboo MettleCI Plugins Installation
Atlassian Bamboo MettleCI License Activation
Install and configure an Atlassian Bamboo Agent
Defining Atlassian Bamboo YAML Specs

Bamboo Tasks

Bamboo Properties Configuration Task
Bamboo Build Commit Log Task
Bamboo Compile DataStage Project Task
Bamboo Compliance Test Task
Bamboo Connector Migration Task
Bamboo DataStage Admin Task
Bamboo DataStage Message Handlers Task
Bamboo Execute DataStage Job Task
Bamboo Export DataStage Asset Task
Bamboo Import DataStage Asset Task
Bamboo Incremental Deployment Task
Bamboo Run DataStage Unit Tests Task
Bamboo SFTP Download Task
Bamboo SFTP Upload Task

Deploying Message Handlers with Bamboo (Deprecated)
Deploying Message Handlers with Bamboo

Atlassian Bitbucket

Configuring Authentication between Workbench and Atlassian Bitbucket
Installing the MettleCI Job Visualisation Plugin for Atlassian Bitbucket
Adding public keys to Bitbucket
Bitbucket Data Center vs. Bitbucket Cloud

Azure

Preparing your Azure environment

Azure Git Repositories

Configure MettleCI Workbench Authentication to Azure Git Repo
Azure Linked to GitHub Repositories

Integrating Azure DevOps Work Item Lookup with MettleCI Workbench

Azure Pipelines

Azure DevOps Deployment Topology
Azure DevOps Setup Roadmap
Install and configure an Azure DevOps self-hosted agent
Deploying an Azure DevOps CI/CD Pipeline
Understanding your Azure DevOps CI/CD Pipeline
Protecting sensitive information used in Azure DevOps Pipelines
Integrate Azure DevOps with Microsoft Teams
Configuring Manual Approval Steps in Azure DevOps

Automating project creation with the Azure API
Re-deploy a prior Azure DevOps release

GitHub

GitHub Authentication

Adding public keys to GitHub
Configuring MettleCI Authentication to GitHub

GitHub Pipelines

Install and Configure a GitHub Runner
JUnit Test Results in GitHub

GitHub Issue Lookup with MettleCI Workbench
Automating project creation with the GitHub API

GitLab

Setting up a GitLab Project/Repository
Configuring MettleCI Workbench SSH Authentication to GitLab
GitLab Issue Lookup with MettleCI Workbench
Automating project creation with the GitLab API

Jenkins

Introducing Jenkins Pipelines
Jenkins DevOps Setup Roadmap
Jenkins DevOps Deployment Topologies

Reusable Pipeline Templates in Jenkins (deprecated)

Jenkins Shared Library - mci_ccmt
Jenkins Shared Library - mci_compliance
Jenkins Shared Library - mci_deploy
Jenkins Shared Library - mci_unittest

Jenkins Build Agents

Install and Configure a Jenkins Agent
Jenkins Agent Assignment

Deploying a Jenkins CI/CD Pipeline
Understanding your Jenkins CI/CD Pipeline
Pipeline Parallelism in Jenkins
Jenkins Environment Variables
Protecting Sensitive Information used in Jenkins Pipelines
Configuring Manual Approval Steps in Jenkins
Jenkins Pipeline Parameters and Waiting for manual inputs
Using MettleCI Test Results in Jenkins

Micro Focus ALM

ServiceNow

Integrating ServiceNow Work Item Lookup with MettleCI Workbench

Building CI/CD Pipelines

MettleCI Components for CI and CD
Repeatable DataStage Project Deployments
Generic MettleCI Pipeline Description
Pipeline Trigger Strategies
Enabling File System Deployment
DataStage Feature Toggles
Deploying DataStage Binaries

MettleCI Sample Pipelines

MettleCI Example Pipeline for DevOps

Parameter Management for Automated Deployment
How the DSParams file is managed in a DevOps pipeline
How the DSParams file is managed in an Upgrade pipeline
How JUnit test results are managed in a build pipeline

Troubleshooting Index

A Workbench update doesn't appear to have installed
Azure pipeline error "You need the Git 'GenericContribute' permission"
Bamboo Agent produces chcp or -107374 errors
Bamboo 'Unable to open file' errors on Windows
Build Pipeline SFTP operations fail due to DataStage Engine name
Can't upload a license file to the MettleCI Setup Wizard
Command shell Error: Unable to access jarfile
Command timed out waiting to complete
Configuring Unit Testing for the Grid Toolkit
Configuring Workbench for HTTPS Git authentication causes "could not connect to server" at workbench user login
Enabling debug logging for MettleCI CLI
Enabling extended debug logging for MettleCI Workbench
Enabling extended logging in Atlassian Bamboo
Erroneous "failed to execute" reported when running unit tests
Error "can't create transaction lock" when installing MettleCI Workbench on Unix
Error running 'mettleci compliance query' command
Error while CI pipeline is 'Inspecting DataStage assets for last change'
Filename too Long error during Git operations
Failing to Read Assets from the DataStage Repository
Gitlab produces Let's Encrypt error when configuring HTTPS access
Handling Hanging Deployments
How do we remove trailing CTRL+M characters from our files in Git?
Jenkins Pipeline produces error 'No such DSL method findFiles'
Jira admin session timing out when setting up an App Link with Workbench
Jira displays error about Workbench 'not registered as a Consumer' when registering application
Jira issues aren't showing relevant Git commits made via MettleCI
JWT rejected due to invalid signature
Logging on to Workbench produces 'Failed to Authenticate with ASBNode' error
MettleCI CLI / Workbench: Job import/export-related functions fail after Information Server patched
MettleCI CLI produces error 'Incorrectly typed data found for annotation element'
MettleCI CLI produces error 'It was not possible to connect to the revocation server or a definitive response could not be obtained. [12057]'
MettleCI CLI produces error of the form 'Cannot run program "XXX"'
MettleCI CLI produces error 'The SSL certificate must have previously been accepted in order to connect'
MettleCI CLI produces 'Failed to connect to host' error
MettleCI CLI produces SLF4J errors
MettleCI CLI throws error 'java.lang.OutOfMemoryError: Java heap space'
MettleCI CLI throws an 'UnsupportedClassVersionError'
MettleCI CLI warns "'ds.client.path' not configured, assuming DataStage client is available on system PATH"
MettleCI command 'workbench set-branch' command produces a credentials error
MettleCI component throws 'UnknownAssetTypeException' error
MettleCI Unit Test execution throws 'java.lang.OutOfMemoryError: Java heap space' exception
No permission to install the DataStage Designer MettleCI menu items
Nothing happens when I run a Server Job in Unit Testing mode
Problems supplying passwords to the MettleCI Command Line
QualityStage Rulesets Ignore Incremental Compilation
Receiving a 'not authorized' message when trying to connect to Git repository.
Some MettleCI Command Line options are being ignored on Windows
Running Unit Test Interception produces an error 'Can not deserialize'
SSLHandshakeException in Workbench after configuring issue lookup
Shared Container image previews not available in Atlassian Bitbucket
The MettleCI Bamboo Command Plugin fails with NoSuchMethodError
The `mettleci datastage create-project` command fails
The MettleCI Bamboo Plugin Fails When Compiling a Job
The MettleCI Bamboo Command Plugin fails with ClassNotFoundException
The MettleCI Service doesn't automatically start after a reboot
The Problem With Message Handlers
The Workbench is unreachable, or the Workbench service won't start
Unexpected DataStage error occurred - Unhandled exception Type=Segmentation error
Unit Test fails due to exceeded memory threshold
Unit Test reports 'Invalid Decimal values are not supported by MCI Unit Testing'
Updating DataStage stops the MettleCI Parallel Unit Test Harness Working
Updating DataStage stops the MettleCI Server Unit Test Harness Working
What versions of TLS does MettleCI Workbench support?
Work item search error
Workbench login error 'The URL is incorrect or is not trusted'
Workbench / DataStage TLS not working on Red Hat Enterprise Linux 8 (RHEL-8) or later
Workbench exception 'The following certificate could not be verified'
Workbench login fails with 'No appropriate protocol' error
Workbench memory errors on AIX
Workbench operations return ‘Could not find specified assets’
Workbench throws 'OutOfMemoryError'
Workbench commit failed with java.lang.ArrayIndexOutOfBoundsException
Workbench throwing java.net.BindException: Address already in use exception
Workbench produces 'Failed to initialize DATASTAGE_ASB authentication' error on startup

Frequently Asked Questions

Am I entitled to use MettleCI via my IBM relationship?
Are there dependencies on software versions between DataStage and MettleCI?
Can I use my existing version of Java Virtual Machine?
Can MettleCI integrate with Active Directory?
Can MettleCI manage Job parameter files in version control?
Can MettleCI work with Talend?
Can MettleCI Workbench integrate with the DataStage Flow Designer?
Can we edit compliance rules or add our own?
Can we fabricate data for one data source that references a column in another data source?
Can MettleCI perform comparison tests between the data in two versions of a database?
Can MettleCI Unit Testing handle complex data sources?
Can MettleCI use a more recent version of Java than Java 8?
Can MettleCI prevent the promotion of CopyOf* jobs?
Can the MettleCI installation processes be automated/scripted?
Can we migrate from one Git system to Another (e.g. Bitbucket to GitHub)?
Can we subscribe to a subset of MettleCI functionality at a lower cost?
Can we unit test a sequence of Jobs?
Can we intercept and test full volume data?
Can you describe some typical MettleCI customers?
Can you provide MettleCI references in our industry, territory, language?
Do any MettleCI components require internet access?
Do Data Migrators offer installation support or consulting?
Do IBM recognize your solution?
Do Unit Testing data files support testing of all DataStage stage types?
Do we have to enable Git's LFS (Large Files Support) setting to support ISX files?
Do we need to modify our jobs to be compatible with MettleCI?
Does MettleCI only handle DataStage testing and deployment, or does it also handle IGC and IA?
Does MettleCI require Atlassian Jira?
Does MettleCI require the installation of any client tools?
Does MettleCI run on Windows or Linux?
Does Mettle CI support Git branching/merging functionality?
Does MettleCI support QualityStage jobs?
Does MettleCI use ActiveX, Java applets, Flash Player, or Silverlight?
Does MettleCI use the IBM Connector Migration Tool?
Does MettleCI work with Citrix Virtual Desktop?
Does MettleCI work on different operating systems and topologies?
Does MettleC work with Windows 2012?
Does MettleCI work with Jenkins?
Does the MettleCI Unit Test job parameter need to be removed from a job before promoting it to a Production environment?
How are our existing test data inputs and outputs captured?
How do I authenticate to MettleCI?
How do I determine the client ports being used by my InfoSphere installation?
How do we change which Browser is used for MettleCI Workbench?
How do we encrypt our DataStage credentials in the MettleCI Workbench configuration file?
How do we delete DataStage assets from a Git repository using MettleCI?
How do we disable deployment to a specific environment in Bamboo?
How do we re-run the Workbench Setup Wizard?
How do we safely update DataStage where the MettleCI Unit Test Harness has been Installed?
How do we segregate duty and ensure deployment to PROD is only available to admins?
🔒How do we size our Git and other ALM tools for integration with MettleCI?
How does MettleCI integrate with other tools?
How does MettleCI support non-DataStage assets such as database scripts, shell scripts, or parameter value files?
How does MettleCI Workbench integrate with Git?
How does MettleCI's Git functionality compare to that of Information Server v11.7?
How does MettleCI work with HP ALM?
How does MettleCI work with schema management tools like Liquibase?
How does the integration/configuration with BitBucket, Jira, and Bamboo work?
How does the MettleCI Workbench Service work in Unix?
How does Unit Testing work with RCP and/or schema files?
How to configure default credential mapping when user registry sharing is enabled
How is MettleCI deployed to our network? Which ports and permissions are required?
How strictly is MettleCI bound to the version of Jira we're using?
Is it possible to have projects on different servers ?
Is it possible to modify MettleCI's Git connection timeout?
Is it possible to rollback deployments?
Is it possible to run two different DataStage versions simultaneously?
Can data from the production environment be used for unit testing?
Is it possible to test an entire functional module end-to-end, without running each job’s individual unit test?
Is MettleCI compatible with my CI/CD technology?
What database does MettleCI use to store its metadata?
Should a Git client be installed on the MettleCI Agent Host?
Is the unit testing function dependent upon the design of our jobs? Are there any restrictions?
Is our current DataStage development environment mature enough to adopt MettleCI?
MettleCI CLI and the 'project-cache' directory
What data does MettleCI access and/or store?
What does a yellow coloured Stage mean in Jenkins?
What does the Workbench 'Integrated with DataStage Client' button do?
What file permissions are unit testing files created with?
What happens if we don’t renew our MettleCI subscription?
What happens when my MettleCI licence expires?
What is the format of the MettleCI logs?
What is the relationship between Data Migrators and MettleCI?
What is the MettleCI Licensing Model?
What is the MettleCI support model?
What JDK distributions is MettleCI compatible with?
What software frameworks/versions does MettleCI require? (Java, .NET, etc.)
When does my MettleCI license begin and end?
Where does the name 'MettleCI' come from?
Which credentials do I use to log in to the MettleCI Workbench?
Which Linux distribution and versions does MettleCI support?
Which user account does Workbench use to commit changes in Git?
Which versions of Atlassian software do you support?
Why do I have to install another version of Java?
Why does MettleCI manage assets as ISX files?
Why is MettleCI's Git Integration Special?
Will installing Java v1.8 break my DataStage environment?
Why is the Workbench application installed on the DataStage Engine tier?
Will MettleCI have access to our company's private data?
Will MettleCI maintain compatibility with future versions of DataStage?
Would the integrity of DataStage jobs be maintained if we later removed MettleCI from our servers ?
Where is MettleCI installed?
What third party/open source code is incorporated into MettleCI?
Why does the MettleCI Agent Host require Microsoft Windows?
Why don't Before/After job routine failures cause a Unit Test failure?
Why should some pipelines use 'ignore-test-failure' when testing fatal Compliance rules?
Can we skip steps in the CI process? (e.g. Compliance Testing)
What skills are required to use MettleCI on daily basis?
Which DataStage versions does MettleCI support?
Which Bamboo license do I need if I want to use it with MettleCI?
MettleCI Workbench Commit fails when Bitbucket Jira issue commit checker is enabled
What are the MettleCI Infrastructure Requirements?
What mandatory/optional software is required to use MettleCI?
How do I migrate from one Git provider to Another?

Reference

Ant File Patterns

Atlassian Bamboo Interface

Bamboo DataStage Capability

Bamboo Administration

Compliance Plans
Check-In Plans

Notifications

Manual Build User Recipient

Creating SSH keys manually
Finding a Host's MAC Address
Force Refreshing your Browser
JUnit Test Results
MettleCI Demo Video
MettleCI Open Source Reference
MettleCI Product Security Advisory - Log4j Vulnerable To Remote Code Execution - CVE-2021-44228

MettleCI Utilities

MettleCI DataStage Transmuters

Transmuter Transform Functions
Transmuter Recipes

Git Commit in Bulk Using MettleCI Command Shell and Git
Selective Promotion Process

Release
SSH Configuration
Starting, Stopping the Monitoring MettleCI Workbench Service
Unit Test Data Table - Keyboard Navigation
Workbench Installation - System Administrator Notes

Release History

MettleCI IBM OEM Release 2.0
MettleCI IBM OEM Release 1.3 Service Pack 1
MettleCI IBM OEM Release 1.3
MettleCI IBM OEM Release 1.2
MettleCI IBM OEM Release 1.1
MettleCI IBM OEM Release 1.0
MettleCI IBM OEM Release 1.0 Fix Pack 1

Entitlement, Procurement, and Licensing

Support Policy
Third Party Code in Data Migrators Products
Procurement Support Documents
Purchasing MettleCI
Company Information
Reseller FAQ

YouTube
LinkedIn
www

MettleCI Documentation

Introduction
Installation and Configuration
Compliance Testing
Automated Unit Testing
Data Fabrication
Unit Testing Patterns
Command Line Interface
- CLI Plugins
- Command Reference
Git Support
MettleCI Integrations
- Atlassian
- Azure
- GitHub
- GitLab
- Jenkins
- 2 more pages
Building CI/CD Pipelines
Troubleshooting Index
Frequently Asked Questions
Reference
Release History
Entitlement, Procurement, and Licensing

YouTube
LinkedIn
www

Copyright Data Migrators Pty Ltd
Powered by Scroll Sites & Atlassian Confluence