Strategies for Managing .apt Files in MettleCI Pipelines

MettleCI provides capabilities for automatically deploying DataStage assets into nominated target environments, but the management of DataStage node configuration (.apt) files presents a unique set of challenges.

The issue lies in the ownership of the .apt files, where there are three basic ownership models:

Exclusive - Your team uses a directory containing .apt files managed and used exclusively by a specific DataStage project,
Shared - Your organisation uses a network filesystem location to share .apt files among a number of different DataStage projects, or
Hybrid - Your organisation uses a single network filesystem location to store disparate groups of .apt files, each used exclusively by one DataStage project.

Inside the datastage/deploy.sh script, there is a section of the script set up as placeholder to add code as appropriate, but by default is left with only a reference to this document to reduce the risk of unintentionally deleting or overwriting configuration files.

Bash

# Projects APT Config files
set -- $BUILDHOME/config/*.apt
if [ -f "$1" ]
then
    ###############
    #
    # .apt files have been included in this MettleCI source repository.
    # Due the variety of ways .apt files may be managed within DataStage, please refer to
    # https://datamigrators.atlassian.net/wiki/spaces/MCIDOC/pages/2342748163/Strategies+for+Managing+.apt+Files+in+MettleCI+Pipelines
    # for assistance on how to plan their migration.
    #
    ###############

    grep -e "resource disk" -e "resource scratchdisk" $BUILDHOME/config/*.apt | awk -F "[\"']" '{print $2}' | sort -u | xargs -n1 mkdir -p
fi

Shared Scenario

In the case of Scenario 1, we suggest that the shared .apt files are managed outside of MettleCI. If you want to use MettleCI capabilities to manage them, the deployment of these files must be controlled in their entirety by a single DataStage project and MettleCI repository that deploys them (as if they were the exclusive owner) for all other projects to use. In that case, they could be treated as if they were Exclusive to that MettleCI repository.

Exclusive Scenario

In the case where the directory and .apt files contained within are owned by the DataStage project (Scenario 2), use the MettleCI Properties Configuration facility to instruct the deploy script of the location of the directory, by adding a variable into each var.* file in your repository:

DatastageConfigPath=/path/to/configuration/file/directory

This value is used to update datastage/deploy.sh, where the following code will delete all .apt files in DatastageConfigPath and copy over the latest .apt files from the repository.

Bash

# Projects APT Config files
set -- $BUILDHOME/config/*.apt
if [ -f "$1" ]
then
    mkdir -p ${DatastageConfigPath}

    rm -f ${DatastageConfigPath}/*.apt
    cp $BUILDHOME/config/*.apt ${DatastageConfigPath}

    grep -e "resource disk" -e "resource scratchdisk" $BUILDHOME/config/*.apt | awk -F "[\"']" '{print $2}' | sort -u | xargs -n1 mkdir -p
fi

Hybrid Scenario

In this scenario, the directory is shared, but a group of the .apt files contained within are owned by the DataStage project. The simplest way, then, involves using a naming schema to determine ownership of the individual .apt files, in conjunction. with the solution for Scenario 2 above.

DatastageConfigPath=/path/to/configuration/file/directory
DatastageConfigPrefix=MyAPTPrefix

These values are used to update datastage/deploy.sh, where the following code will delete all .apt files in DatastageConfigPath starting with the specified prefix, and copy over the latest .apt files from the repository.

Bash

# Projects APT Config files
set -- $BUILDHOME/config/*.apt
if [ -f "$1" ]
then
    mkdir -p ${DatastageConfigPath}

    rm -f ${DatastageConfigPath}/${DatastageConfigPrefix}*.apt
    cp $BUILDHOME/config/*.apt ${DatastageConfigPath}

    grep -e "resource disk" -e "resource scratchdisk" $BUILDHOME/config/*.apt | awk -F "[\"']" '{print $2}' | sort -u | xargs -n1 mkdir -p
fi

Some projects may involve exclusive .apt files in a shared directory without project-specific prefixes. This too is possible by removing the line to delete selected pre-existing .apt files, but does not provide MettleCI with a way to identify and delete .apt files that are no longer in use.