Skip to main content
YouTube
LinkedIn
www
Main navigation
Close navigation
Main
MettleCI Documentation
Introduction
Welcome to MettleCI v2.0
Why MettleCI?
MettleCI Functional Overview
Use Cases
MettleCI Installation Roadmap
Definition of a Unit Test
Development Model Options & Recommendations
Installation and Configuration
Installation Topologies
A Summary of MettleCI Components
Standalone DataStage on Windows - Dedicated Agent Host
Standalone DataStage on Windows - Consolidated Agent Host
Standalone DataStage on Unix - Dedicated Agent Host
Standalone DataStage on Different Versions - Concurrent use for upgrades
Virtual Desktop Example Topology
The Mettle Agent Host
Using Multiple MettleCI Agents
MettleCI Components for Upgrades to DataStage to v11.7.X
MettleCI for DataStage Failover Configurations
MettleCI For DevOps - Technical Prerequisites
MettleCI - Infrastructure Requirements
MettleCI - User Accounts
MettleCI - Firewall Rules & Security
MettleCI - Software
MettleCI - Component Connections
Pre-installation checklist
MettleCI Installation Walkthrough Videos
Accessing the MettleCI Software Distribution
Prerequisite Java Installation
Installing MettleCI Workbench
MettleCI Workbench Installation Process
Installing or Upgrading Workbench on Unix
Installing or Upgrading Workbench on Windows
MettleCI Workbench Post-installation Setup
MettleCI Workbench Setup Wizard
MettleCI Workbench Initial Login
MettleCI Workbench Configure First Project
Integrating MettleCI Workbench and DataStage Designer on Windows
Customizing your MettleCI Workbench Installation
MettleCI Workbench Configuration Files
Configuring MettleCI Workbench to use a Custom Port Number
Configuring MettleCI Workbench to use HTTPS
Configuring MettleCI Workbench to use HTTPS with an existing certificate
Configuring MettleCI Workbench to use Custom App and Data Directories
Configuring MettleCI Workbench to communicate with Git over HTTPS
Configuring MettleCI Workbench to access systems via an HTTP/S proxy
Configuring MettleCI Workbench to increase session timeout
Configuring MettleCI Workbench to use 'secrets' to improve security of HTTPS connections
Installing or Upgrading the MettleCI Unit Test Harness
How the Parallel Unit Test Harness Integrates with DataStage
Installing the Unix Parallel Unit Test Harness
Installing the Windows Parallel Unit Test Harness
Installing the Server Unit Test Harness
Configuring Jobs for Unit Testing
Installing MettleCI Command Line Interface
Installing MettleCI CLI on Windows
Installing MettleCI CLI on Unix
Installing or Upgrading Individual MettleCI CLI Commands
Getting Started with MettleCI Workbench
Registering a DataStage Project with MettleCI
Add or modify a work item service integration in Workbench
Deploying MettleCI repository templates to your Git platform
Creating a Local Compliance Rule Git Repository
Compliance Testing
Compliance Testing from Workbench
Compliance Rule Tags
Making CI dependent on specific Compliance Rules
Compliance Rules Reference
Adjacent Transformers
Aggregator Not Preceded by ‘Check’ Sort
Asset using Java Integration Stage
Audit Annotation
CCMigrateTool Stages
Column Name Contains Unsupported Characters
Job Using Transformer Surrogate Key
Custom Stages
Data Sets not using 'Same' partitioning method
Database Connector does not Auto-Generate SQL
DataStage Flow Designer Stages
Database Row Limit
Date Format in Annotation
Database tables references are fully qualified
DataStage SaaS Unsupported Stages
DB2 with No Non Recovery Load
DBConnector Supported Variants
Debug Row Limit
Default Naming
Deprecated Stages
Deprecated Transformer Functions
Duplicate File References
Duplicate Stage Names
Encrypted DB Passwords
File Reference Missing Required Parameter
File Row Limit
Hardcoded DB Credentials
Hardcoded File Paths
CP4D Unsupported Stages
Job Activity References Deleted Child Parameter
Job Activity with Hardcoded Parameter Values
Job Control Routines are present
Job Does Not Contain DM Unit Test Parameter
Job Naming
Job Parameter Missing Default Value
Job Parameter Naming
Job Parameter Not in a Parameter Set
Job Parameter Not Used in a Job
Job Sequence Handles Activities That Fail
Job Sequence Is Restartable
Job Using Custom Function
Job Using ODBC Connector
Jobs With Before and After Routine
Join Partition vs Join Key
Link Sort
Log Column Values
Lookup Failure
One Dataflow
Oracle Connector not using Partition Read
Password Param Type not Encrypted
Prohibited Stages
Range Lookup
Redundant Sort
Row Splitter with No Delimiter
SELECT * used in custom SQL
Sequence Reset If Required
Sequential File Read Using Same Partitioning
SQL in DB Connectors
Schema Files
Sequential File With Reject Mode Not Set To Fail
Sort Post Join Stage
Stage Naming
Stages Not Supporting Lineage
System Time Dependency
Too Many Stages
Transformer Uses 'Abort after rows'
Transformer Uses 'Rnd()' Function
Transformer With Unreferenced Stage Variable
Unique Sort
Using DB2 Sequence to Generate Surrogate Key
Developing Custom Compliance Rules
Compliance Rule Graphs
Compliance Rule Development
Example Compliance Rule - Checking for Adjacent Transformers
Example Compliance Rule - Job Sequence is Restartable
Example Compliance Rule - Asset Naming Standards
Accessing Columns and Derivations using Compliance Rules
Avoiding a NullPointerException when processing Compliance Rules
Automated Unit Testing
Creating a Unit Test
Unit Test Specification Format
Intercepting Existing Test Data
Managing your Unit Test data volumes
Manually Editing Unit Test Data
Capturing a Baseline Test Result
Row Count Unit Testing
Executing a Unit Test
Unit Testing Job Sequences
Data Fabrication
Using MettleCI data fabrication
MettleCI’s data fabrication tools
Creating Custom Data Fabricators
Creating a custom test data generator bundle
Test your custom generator bundle at the command line
Upload a custom data fabrication bundle to MettleCI Workbench
Use your custom data generators in test data specifications
Unit Testing Patterns
Unit Testing Stages with Rejects
Unit Testing Stored Procedure Stages
Unit Testing Surrogate Key Generator Stages
Unit Testing Sparse Lookup Stages
Unit Testing Jobs with current date calculations
Omitting selected columns from Unit Testing
High Volume Unit Tests
Creating and Running Multiple Unit Tests for a Single Job
Unit Tests featuring Local and Shared Containers
Unit Testing Slowly Changing Dimension Stages
Command Line Interface
CLI Plugins
Command Reference
Compliance Namespace
Compliance Console Command 🔒
Compliance List-Tags Command
Compliance Query Command
Compliance Test Command
DataStage Namespace
DataStage Capture Command
DataStage Connector Migration Command
DataStage Cleanup-Projects Command
DataStage Compile Command
DataStage Create-Project Command
DataStage Delete-Project Command
DataStage Deploy Command
DataStage Execute Command
DSParams Namespace
DSParams Delete Command
DSParams Diff Command
DSParams Merge Command
Fabrication Namespace
Fabrication List Command
Fabrication Test Command
ISX Namespace
ISX Cat Command
ISX Cut Command
ISX Export Command
ISX Import Command
ISX Message-Handlers Command
ISX Set-Params Command
Properties Namespace
Properties Config Command
Remote Namespace
Remote Download Command
Remote Execute Command
Remote Upload Command
UnitTest Namespace
UnitTest Generate Command
UnitTest Install-Server-Test-Harness Command
UnitTest Migrate Command
UnitTest Test Command
UnitTest Uninstall-Server-Test-Harness Command
Workbench Namespace
Workbench Set-Branch Command
Git Support
Git and MettleCI
Trunk Based Development
Working with DataStage and Git Branches
Organising your Git repositories
Creating and Preparing a Git Repository for DataStage Assets
DataStage Git Repository Structure
Committing DataStage Assets to Git
Signed Commits
Why Sign Commits?
Configuring Signed Commits
Platforms Supporting Signed Commits
Addressing Production defects using Hotfixes
Handling Production Defects
Developing a hotfix
Configuring a Hotfix Trigger
MettleCI Integrations
Atlassian
Atlassian Jira
Jira and MettleCI Workbench via OAuth explained
Configuring Atlassian Jira issue lookup with MettleCI Workbench
Atlassian Bamboo
Atlassian Bamboo MettleCI Plugins Installation
Atlassian Bamboo MettleCI License Activation
Install and configure an Atlassian Bamboo Agent
Defining Atlassian Bamboo YAML Specs
Bamboo Tasks
Bamboo Properties Configuration Task
Bamboo Build Commit Log Task
Bamboo Compile DataStage Project Task
Bamboo Compliance Test Task
Bamboo Connector Migration Task
Bamboo DataStage Admin Task
Bamboo DataStage Message Handlers Task
Bamboo Execute DataStage Job Task
Bamboo Export DataStage Asset Task
Bamboo Import DataStage Asset Task
Bamboo Incremental Deployment Task
Bamboo Run DataStage Unit Tests Task
Bamboo SFTP Download Task
Bamboo SFTP Upload Task
Deploying Message Handlers with Bamboo (Deprecated)
Deploying Message Handlers with Bamboo
Atlassian Bitbucket
Configuring Authentication between Workbench and Atlassian Bitbucket
Installing the MettleCI Job Visualisation Plugin for Atlassian Bitbucket
Adding public keys to Bitbucket
Bitbucket Data Center vs. Bitbucket Cloud
Azure
Preparing your Azure environment
Azure Git Repositories
Configure MettleCI Workbench Authentication to Azure Git Repo
Azure Linked to GitHub Repositories
Integrating Azure DevOps Work Item Lookup with MettleCI Workbench
Azure Pipelines
Azure DevOps Deployment Topology
Azure DevOps Setup Roadmap
Install and configure an Azure DevOps self-hosted agent
Deploying an Azure DevOps CI/CD Pipeline
Understanding your Azure DevOps CI/CD Pipeline
Protecting sensitive information used in Azure DevOps Pipelines
Integrate Azure DevOps with Microsoft Teams
Configuring Manual Approval Steps in Azure DevOps
Automating project creation with the Azure API
Re-deploy a prior Azure DevOps release
GitHub
GitHub Authentication
Adding public keys to GitHub
Configuring MettleCI Authentication to GitHub
GitHub Pipelines
Install and Configure a GitHub Runner
JUnit Test Results in GitHub
GitHub Issue Lookup with MettleCI Workbench
Automating project creation with the GitHub API
GitLab
Setting up a GitLab Project/Repository
Configuring MettleCI Workbench SSH Authentication to GitLab
GitLab Issue Lookup with MettleCI Workbench
Automating project creation with the GitLab API
Jenkins
Introducing Jenkins Pipelines
Jenkins DevOps Setup Roadmap
Jenkins DevOps Deployment Topologies
Reusable Pipeline Templates in Jenkins (deprecated)
Jenkins Shared Library - mci_ccmt
Jenkins Shared Library - mci_compliance
Jenkins Shared Library - mci_deploy
Jenkins Shared Library - mci_unittest
Jenkins Build Agents
Install and Configure a Jenkins Agent
Jenkins Agent Assignment
Deploying a Jenkins CI/CD Pipeline
Understanding your Jenkins CI/CD Pipeline
Pipeline Parallelism in Jenkins
Jenkins Environment Variables
Protecting Sensitive Information used in Jenkins Pipelines
Configuring Manual Approval Steps in Jenkins
Jenkins Pipeline Parameters and Waiting for manual inputs
Using MettleCI Test Results in Jenkins
Micro Focus ALM
ServiceNow
Integrating ServiceNow Work Item Lookup with MettleCI Workbench
Building CI/CD Pipelines
MettleCI Components for CI and CD
Repeatable DataStage Project Deployments
Generic MettleCI Pipeline Description
Pipeline Trigger Strategies
Enabling File System Deployment
DataStage Feature Toggles
Deploying DataStage Binaries
MettleCI Sample Pipelines
MettleCI Example Pipeline for DevOps
Parameter Management for Automated Deployment
How the DSParams file is managed in a DevOps pipeline
How the DSParams file is managed in an Upgrade pipeline
How JUnit test results are managed in a build pipeline
Troubleshooting Index
A Workbench update doesn't appear to have installed
Azure pipeline error "You need the Git 'GenericContribute' permission"
Bamboo Agent produces chcp or -107374 errors
Bamboo 'Unable to open file' errors on Windows
Build Pipeline SFTP operations fail due to DataStage Engine name
Can't upload a license file to the MettleCI Setup Wizard
Command shell Error: Unable to access jarfile
Command timed out waiting to complete
Configuring Unit Testing for the Grid Toolkit
Configuring Workbench for HTTPS Git authentication causes "could not connect to server" at workbench user login
Enabling debug logging for MettleCI CLI
Enabling extended debug logging for MettleCI Workbench
Enabling extended logging in Atlassian Bamboo
Erroneous "failed to execute" reported when running unit tests
Error "can't create transaction lock" when installing MettleCI Workbench on Unix
Error running 'mettleci compliance query' command
Error while CI pipeline is 'Inspecting DataStage assets for last change'
Filename too Long error during Git operations
Failing to Read Assets from the DataStage Repository
Gitlab produces Let's Encrypt error when configuring HTTPS access
Handling Hanging Deployments
How do we remove trailing CTRL+M characters from our files in Git?
Jenkins Pipeline produces error 'No such DSL method findFiles'
Jira admin session timing out when setting up an App Link with Workbench
Jira displays error about Workbench 'not registered as a Consumer' when registering application
Jira issues aren't showing relevant Git commits made via MettleCI
JWT rejected due to invalid signature
Logging on to Workbench produces 'Failed to Authenticate with ASBNode' error
MettleCI CLI / Workbench: Job import/export-related functions fail after Information Server patched
MettleCI CLI produces error 'Incorrectly typed data found for annotation element'
MettleCI CLI produces error 'It was not possible to connect to the revocation server or a definitive response could not be obtained. [12057]'
MettleCI CLI produces error of the form 'Cannot run program "XXX"'
MettleCI CLI produces error 'The SSL certificate must have previously been accepted in order to connect'
MettleCI CLI produces 'Failed to connect to host' error
MettleCI CLI produces SLF4J errors
MettleCI CLI throws error 'java.lang.OutOfMemoryError: Java heap space'
MettleCI CLI throws an 'UnsupportedClassVersionError'
MettleCI CLI warns "'ds.client.path' not configured, assuming DataStage client is available on system PATH"
MettleCI command 'workbench set-branch' command produces a credentials error
MettleCI component throws 'UnknownAssetTypeException' error
MettleCI Unit Test execution throws 'java.lang.OutOfMemoryError: Java heap space' exception
No permission to install the DataStage Designer MettleCI menu items
Nothing happens when I run a Server Job in Unit Testing mode
Problems supplying passwords to the MettleCI Command Line
QualityStage Rulesets Ignore Incremental Compilation
Receiving a 'not authorized' message when trying to connect to Git repository.
Some MettleCI Command Line options are being ignored on Windows
Running Unit Test Interception produces an error 'Can not deserialize'
SSLHandshakeException in Workbench after configuring issue lookup
Shared Container image previews not available in Atlassian Bitbucket
The MettleCI Bamboo Command Plugin fails with NoSuchMethodError
The `mettleci datastage create-project` command fails
The MettleCI Bamboo Plugin Fails When Compiling a Job
The MettleCI Bamboo Command Plugin fails with ClassNotFoundException
The MettleCI Service doesn't automatically start after a reboot
The Problem With Message Handlers
The Workbench is unreachable, or the Workbench service won't start
Unexpected DataStage error occurred - Unhandled exception Type=Segmentation error
Unit Test fails due to exceeded memory threshold
Unit Test reports 'Invalid Decimal values are not supported by MCI Unit Testing'
Updating DataStage stops the MettleCI Parallel Unit Test Harness Working
Updating DataStage stops the MettleCI Server Unit Test Harness Working
What versions of TLS does MettleCI Workbench support?
Work item search error
Workbench login error 'The URL is incorrect or is not trusted'
Workbench / DataStage TLS not working on Red Hat Enterprise Linux 8 (RHEL-8) or later
Workbench exception 'The following certificate could not be verified'
Workbench login fails with 'No appropriate protocol' error
Workbench memory errors on AIX
Workbench operations return ‘Could not find specified assets’
Workbench throws 'OutOfMemoryError'
Workbench commit failed with java.lang.ArrayIndexOutOfBoundsException
Workbench throwing java.net.BindException: Address already in use exception
Workbench produces 'Failed to initialize DATASTAGE_ASB authentication' error on startup
Frequently Asked Questions
Am I entitled to use MettleCI via my IBM relationship?
Are there dependencies on software versions between DataStage and MettleCI?
Can I use my existing version of Java Virtual Machine?
Can MettleCI integrate with Active Directory?
Can MettleCI manage Job parameter files in version control?
Can MettleCI work with Talend?
Can MettleCI Workbench integrate with the DataStage Flow Designer?
Can we edit compliance rules or add our own?
Can we fabricate data for one data source that references a column in another data source?
Can MettleCI perform comparison tests between the data in two versions of a database?
Can MettleCI Unit Testing handle complex data sources?
Can MettleCI use a more recent version of Java than Java 8?
Can MettleCI prevent the promotion of CopyOf* jobs?
Can the MettleCI installation processes be automated/scripted?
Can we migrate from one Git system to Another (e.g. Bitbucket to GitHub)?
Can we subscribe to a subset of MettleCI functionality at a lower cost?
Can we unit test a sequence of Jobs?
Can we intercept and test full volume data?
Can you describe some typical MettleCI customers?
Can you provide MettleCI references in our industry, territory, language?
Do any MettleCI components require internet access?
Do Data Migrators offer installation support or consulting?
Do IBM recognize your solution?
Do Unit Testing data files support testing of all DataStage stage types?
Do we have to enable Git's LFS (Large Files Support) setting to support ISX files?
Do we need to modify our jobs to be compatible with MettleCI?
Does MettleCI only handle DataStage testing and deployment, or does it also handle IGC and IA?
Does MettleCI require Atlassian Jira?
Does MettleCI require the installation of any client tools?
Does MettleCI run on Windows or Linux?
Does Mettle CI support Git branching/merging functionality?
Does MettleCI support QualityStage jobs?
Does MettleCI use ActiveX, Java applets, Flash Player, or Silverlight?
Does MettleCI use the IBM Connector Migration Tool?
Does MettleCI work with Citrix Virtual Desktop?
Does MettleCI work on different operating systems and topologies?
Does MettleC work with Windows 2012?
Does MettleCI work with Jenkins?
Does the MettleCI Unit Test job parameter need to be removed from a job before promoting it to a Production environment?
How are our existing test data inputs and outputs captured?
How do I authenticate to MettleCI?
How do I determine the client ports being used by my InfoSphere installation?
How do we change which Browser is used for MettleCI Workbench?
How do we encrypt our DataStage credentials in the MettleCI Workbench configuration file?
How do we delete DataStage assets from a Git repository using MettleCI?
How do we disable deployment to a specific environment in Bamboo?
How do we re-run the Workbench Setup Wizard?
How do we safely update DataStage where the MettleCI Unit Test Harness has been Installed?
How do we segregate duty and ensure deployment to PROD is only available to admins?
🔒How do we size our Git and other ALM tools for integration with MettleCI?
How does MettleCI integrate with other tools?
How does MettleCI support non-DataStage assets such as database scripts, shell scripts, or parameter value files?
How does MettleCI Workbench integrate with Git?
How does MettleCI's Git functionality compare to that of Information Server v11.7?
How does MettleCI work with HP ALM?
How does MettleCI work with schema management tools like Liquibase?
How does the integration/configuration with BitBucket, Jira, and Bamboo work?
How does the MettleCI Workbench Service work in Unix?
How does Unit Testing work with RCP and/or schema files?
How to configure default credential mapping when user registry sharing is enabled
How is MettleCI deployed to our network? Which ports and permissions are required?
How strictly is MettleCI bound to the version of Jira we're using?
Is it possible to have projects on different servers ?
Is it possible to modify MettleCI's Git connection timeout?
Is it possible to rollback deployments?
Is it possible to run two different DataStage versions simultaneously?
Can data from the production environment be used for unit testing?
Is it possible to test an entire functional module end-to-end, without running each job’s individual unit test?
Is MettleCI compatible with my CI/CD technology?
What database does MettleCI use to store its metadata?
Should a Git client be installed on the MettleCI Agent Host?
Is the unit testing function dependent upon the design of our jobs? Are there any restrictions?
Is our current DataStage development environment mature enough to adopt MettleCI?
MettleCI CLI and the 'project-cache' directory
What data does MettleCI access and/or store?
What does a yellow coloured Stage mean in Jenkins?
What does the Workbench 'Integrated with DataStage Client' button do?
What file permissions are unit testing files created with?
What happens if we don’t renew our MettleCI subscription?
What happens when my MettleCI licence expires?
What is the format of the MettleCI logs?
What is the relationship between Data Migrators and MettleCI?
What is the MettleCI Licensing Model?
What is the MettleCI support model?
What JDK distributions is MettleCI compatible with?
What software frameworks/versions does MettleCI require? (Java, .NET, etc.)
When does my MettleCI license begin and end?
Where does the name 'MettleCI' come from?
Which credentials do I use to log in to the MettleCI Workbench?
Which Linux distribution and versions does MettleCI support?
Which user account does Workbench use to commit changes in Git?
Which versions of Atlassian software do you support?
Why do I have to install another version of Java?
Why does MettleCI manage assets as ISX files?
Why is MettleCI's Git Integration Special?
Will installing Java v1.8 break my DataStage environment?
Why is the Workbench application installed on the DataStage Engine tier?
Will MettleCI have access to our company's private data?
Will MettleCI maintain compatibility with future versions of DataStage?
Would the integrity of DataStage jobs be maintained if we later removed MettleCI from our servers ?
Where is MettleCI installed?
What third party/open source code is incorporated into MettleCI?
Why does the MettleCI Agent Host require Microsoft Windows?
Why don't Before/After job routine failures cause a Unit Test failure?
Why should some pipelines use 'ignore-test-failure' when testing fatal Compliance rules?
Can we skip steps in the CI process? (e.g. Compliance Testing)
What skills are required to use MettleCI on daily basis?
Which DataStage versions does MettleCI support?
Which Bamboo license do I need if I want to use it with MettleCI?
MettleCI Workbench Commit fails when Bitbucket Jira issue commit checker is enabled
What are the MettleCI Infrastructure Requirements?
What mandatory/optional software is required to use MettleCI?
How do I migrate from one Git provider to Another?
Reference
Ant File Patterns
Atlassian Bamboo Interface
Bamboo DataStage Capability
Bamboo Administration
Compliance Plans
Check-In Plans
Notifications
Manual Build User Recipient
Creating SSH keys manually
Finding a Host's MAC Address
Force Refreshing your Browser
JUnit Test Results
MettleCI Demo Video
MettleCI Open Source Reference
MettleCI Product Security Advisory - Log4j Vulnerable To Remote Code Execution - CVE-2021-44228
MettleCI Utilities
MettleCI DataStage Transmuters
Transmuter Transform Functions
Transmuter Recipes
Git Commit in Bulk Using MettleCI Command Shell and Git
Selective Promotion Process
Release
SSH Configuration
Starting, Stopping the Monitoring MettleCI Workbench Service
Unit Test Data Table - Keyboard Navigation
Workbench Installation - System Administrator Notes
Release History
MettleCI IBM OEM Release 2.0
MettleCI IBM OEM Release 1.3 Service Pack 1
MettleCI IBM OEM Release 1.3
MettleCI IBM OEM Release 1.2
MettleCI IBM OEM Release 1.1
MettleCI IBM OEM Release 1.0
MettleCI IBM OEM Release 1.0 Fix Pack 1
Entitlement, Procurement, and Licensing
Support Policy
Third Party Code in Data Migrators Products
Procurement Support Documents
Purchasing MettleCI
Company Information
Reseller FAQ
YouTube
LinkedIn
www
Data Migrators
How can we help you?
MettleCI
Documentation supporting the installation, configuration, and use of MettleCI for DataStage v11.x
