Selective Promotion Process

Introduction

MettleCI-enabled CI/CD pipelines are commonly demonstrated providing a model where all committed changes are deployed and tested in a CI environment; successful CI pipelines result in the same comitted design artefacts being automatically depoyed to a downstream QA environment on their journey to a Production environment. Many organisations have CI/CD requirments different to these, such as …

Manual Approval - requiring a manual approval step before automated deployment of DataStage assets beyond the CI environment is performed. This is a capability of your build system, and will offer different capabilities, and require a different approach to configuration, depending upon which build system you are using.
Binary-only deployment - mandating that no design assets may be promoted beyond the CI environment, meaning only DataStage job binaries may deployed to QA and Production enviroments. This is often enforced by ensuring the binary-only environments do not have a compiler installed, thereby making in-situ compilation impossible. The MettleCI capabilities used to support approach this are described here.
Seletive Deployment - requiring the ability for this manual approval step to permit the the 'cherry picking' of assets to be deployed down stream.

The default behaviour of the mettleci datastage deploy command uses an incremental approach to depoying ISX files to a specified target DataStage environment. This incremental behaviour is designed to ensure that a destination environment is fully (and automatically) aligned with the contents of an directory of ISX assets (which itself is usually a local working copy of a remote Git repository.). This page describes how this automated alignment can be managed to support the cherry picking ('selective promotion') of DataStage assets.

Video

This process described here is illustrated in two video …

The Selective Promotion process for design assets

This example suits organisations who wish to deploy DataStage design assets to downstream environments, such as QA and PROD, where they are re-compiled before usage.

The Selective Promotion process for binary assets

This example suits organisations who do not permit the compilation of DataStage assets in downstream environments, such as QA and PROD, and require a solution to selectively deploying only pre-compiled binary assets.

A step-by-step process walkthrough

The process to achieve selective promotion is virtual identical

You make changes in your DEV project and commit the design assets to Git using MettleCI Workbench.
Your build system deploys those design assets to your DEV_CI project where they are compiled and tested.
Once testing passes, …
1. Design Assets: the design assets supplied to the CI project are automatically committed to a Git ci branch where they are tagged.
2. Binary Assets: the compiled binary assets are exported from your DEV_CI project and automatically committed to a Git ci branch where they are tagged.
Using a local clone of the repository you then perform a manual selective promotion of design/binary assets downstream (to a qa branch, for example) using the MettleCI Selective Promotion utility, described below. This promotion can optionally include both DataStage and non-DataStage assets, or assets not automatically identified by MettleCI’s incremental deployment.
This commit in the ci branch triggers an automatic deployment pipeline which uses MettleCI commands to deploy your design/binary assets to the QA project. The promoted commit is then tagged in the qa branch in Git.
After QA testing has completed satisfactorily you can then use the same technique to perform a selective promotion of your design/binary assets from the qa branch to the prod branch.
This commit to the prod branch triggers a deployment pipeline to deploy your design/binary assets to the PROD_RC (Release Candidate) project. The commit to the prod branch is then tagged with a rc tag.
After final QA checks you can manually trigger a MettleCI-enabled deployment of your prod branch’s design/binary assets to the DataStage PROD project at your discretion. This deployment process tags the deployed commit with a prod tag.

Your repository now has a set of tags describing, in each branch, the testing and promotions applied to each version of your codebase!

The Selective Promotion utility

MettleCI ships with a Selective Promotion utility which enables you to specify which assets should be promoted. e.g.,

The utility is currently delivered as a shell script named mci-promote.sh and is used like this:

Bash

#
# Create a local clone of our repository
#
$> git clone user@bitbucket.org:myorganisation/myrepository.git
Cloning into '53b-test'...
Receiving objects: 100% (2310/2310), 41.33 MiB | 5.34 MiB/s, done.
Resolving deltas: 100% (904/904), done.

# Copy the mci-promote.sh out of the repository into your path
# Note that subsequent steps will involve us switching our session to branches where
# the mci-promote.sh script does not exists, so we need to put it somewhere on our
# path where we can always access it)
$> cd myrepository
$> cp ./pipelines/scripts/mci-promote.sh ~

#
# Switch to the DESTINATION git branch (the branch into which you wish to promote assets) 
#
$> git switch qa
Switched to branch 'qa'
Your branch is up to date with 'origin/qa'.

#
# Run the mci_promote script specifying the mandatory parameters ...
# Usage: mci-promote.sh -s <source branch> -a <datastage assets>
#
$> ~/mci-promote.sh -s ci -a ./datastage

#
# At this stage you'll use yoyr terminal's default editor to select your asset for promotion
# (See the viudeo above)
#

Fetching and tracking 'origin/ci'
remote: Enumerating objects: 42, done.
remote: Counting objects: 100% (39/39), done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 24 (delta 14), reused 0 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (24/24), 473.08 KiB | 1.01 MiB/s, done.
From bitbucket.org:myorganisation/myrepository
 * branch            ci         -> FETCH_HEAD
   3268f21..8bbba66  ci         -> origin/ci
branch 'ci' set up to track 'origin/ci'.
Resetting assets folder 'datastage' to 'qa'
Promoting assets from 'ci' (8bbba66)
  datastage/Jobs/Transform/TR_ORDERS.isx

#
# Once you're happy you've promoted the correct assets between the
# correct branches you can commit and push your changes as normal
#
$> git commit -m "Promoted TR_ORDERS to qa"
[qa 71ad047] Promoted to qa
 1 file changed, 0 insertions(+), 0 deletions(-)

$> git push
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 10 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 385.89 KiB | 22.70 MiB/s, done.
Total 6 (delta 4), reused 0 (delta 0), pack-reused 0
remote:
remote: Create pull request for qa:
remote:   https://bitbucket.org/myorganisation/53b-test/pull-requests/new?source=qa&t=1
remote:
To bitbucket.org:myorganisation/myrepository.git
   19769ba..71ad047  qa -> qa

# Done!