Difference between revisions of "Software Development (WG1) Version Control (T3)"

Revision as of 19:56, 22 November 2005

Setting up a VistA Development Environment

Overview

This is a description of how VistA development environments should be configured.

Monospaced text designates program names, directory names, command line parameters, etc., that are intended to be typed in at a Linux shell or a GT.M prompt, e.g. /bin/bash. Italic text is a descriptor of something whose actual value must be determined when the command is typed, e.g., gtmver.

Theory

A release is a collection of routines and global variables. The canonical packaging of a release, which is suitable for importation into any standard M distribution, is a collection of M source code routines and a database extract in ZWRite format. When installed on a system, to run with an M implementation, a release may have pre-compiled object files, database files, shell scripts, global directories, etc.

Below is a diagram of the “roll-up” by which a VistA release (“Release A”) gets converted into another VistA release (“Release B”) by a development team. A and B could be from different families, e.g., Release A could be a FOIA release and Release B could be a Leonardo da VistA release.

Once Release A is installed, it remains frozen and unchanged except for any essential patches that may become available independent of the development process for creating Release B and without which the release lacks some required functionality. This means, for example, that KIDS patches should be applied in an integration environment rather than a release environment.

At any given time, there can be multiple development projects underway based on Release A. In other words, there may well be a Release C that is also based on Release A, but which has its own integration and development areas.

To start the development process, an integration directory is created and initialized with a copy of the global variables from Release A, as well as shell scripts, global directories, etc. The integration directory may well have a different name from Release A and Release B. For example, Release A may be FOIAVistA_20051021, Release B may become LeonardoDaVistA_0.4 (at the time that the integration area is created, the name and version of Release B may not even be decided yet), and the integration and development areas may be named Greenbelt_200510. There is no need to copy routines, since the GT.M search path will be set up to look for routines in the integration area before looking in Release A; hence only routines that are different between Release A and the new release need to be duplicated.

Actual development does not take place in the integration area. Development occurs in development areas, typically in the directories of individual developers; however, there may be a shared development area if two developers are collaborating on making changes to the very same module (i.e., their code changes overlap). A development area is set up by creating a directory structure, shell scripts, etc. Note that development areas can be hierarchical so that development subtasks 1a and 1b can be developed separately and integrated into development task 1, prior to integration into the integration area, etc.

During this development process, developers develop code in individual sand boxes (development areas), but normally share the database (global variables) in the integration area – there may typically be no need to create a separate database. However, if some global variable changes are tied to code changes being made in a development area, a copy of those global variables can be placed in a separate database file in that development area, and a global directory for that development area can map those global variables to the database in that development area, with all other global variables mapped to the database in the integration area.

When a developer has code that is ready for integration, the new versions of the modified routines are promoted to the integration area. Any global variables tied to the code being promoted that are in a database in the development area must be merged, reconciled or otherwise integrated with the global variables in the database in the integration area and the global directory in the development area modified accordingly. Note that there needs to be a process, e.g., involving code reviews, by which a development task is considered complete enough to be promoted. These, and other quality gates, are not discussed herein.

Creating a release is minimally the process of creating the directory structure for Release B, copying the routines from Release A, overlying the routines with those from the integration area, and copying the global variables from the integration area.

Normally, Release A, the integration area and development areas would all use the same GT.M version. However, this is not a requirement.

It is strongly recommended that database files in integration and development environments be journaled, and backed up regularly. It is also strongly recommended that a version control system such as CVS or subversion be used for integration areas.

Practice

Shell

The standard shell for VistA community scripting is bash (installed as /bin/bash on Debian GNU/Linux systems).

GT.M Release Directories

Each release of GT.M is installed its own subdirectory of /opt (preferred) or /usr/local (acceptable). Each directory name starts with gtm followed by a release number separated from the release name by an underscore (_). Thus, for example, GT.M V5.0-000C would be installed in /opt/gtm_V5.0-000C. /opt/gtm is a relative symbolic link to the latest version, set up with a command sequence such as:

 cd /opt
 rm -f gtm
 ln -s gtm_V5.0-000C gtm

When installing GT.M, the file gtmprofile should be edited so that the line:

 EDITOR="/bin/vi"; export EDITOR

is modified to read:

 export EDITOR=${EDITOR:=/bin/vi}

VistA Release Directories

Each release is installed in its own sub-directory of a standard system location, such as /opt (preferred) or /usr/local (acceptable). Each directory has a release family name followed by a release / version number suffix separated from the release name by an underscore. Directories for releases without version numbers, such as FOIA releases, have a date suffix of the form yyyymmdd (and where appropriate just yyyymm). For example, an August 25, 2005 FOIA release might be in /opt/FOIAVistA20050825, an October 21, 2005 release in /opt/FOIAVistA20051021, an OpenVistA 0.3 release in /opt/OpenVistA0.3 and a Leonardo da VistA 0.45 release in /opt/LeonardoDaVistA0.45.

The normal protection on all directories and files in a release directory should be read-only. To install files in the p subdirectory, it should be made read-write, the source file(s) for the patch copied in, the object file(s) generated, and the directory and its contents again made read-only.

For each release family, the release directory contains subdirectories g, o, r and p, symbolic link gtm, a script install, scripts run, and cprs_direct that are not executable in a release directory, but which are used when copied to integration directories and made executable. A file env is used to set up the execution environment.

Other files and shell scripts may be created for various maintenance purposes (and ideally would be documented here).

• Subdirectory g contains a global directory file mumps.gld and a compressed database file mumps.dat.gz. The global directory maps all global variables to a single database file $vista_home/g/mumps.dat (i.e., the environment variable $vista_home is used at run-time to point a GT.M process to the location of the database file, so that different processes can actually use the same global directory to refer to entirely different database files).

• Subdirectories o and r correspond to directories for files containing routine object code and routine source code respectively. Each file in r ends in .m and has a corresponding file in o with a newer time stamp and which ends in .o.

• Subdirectory p contains source and object code for patches to a release (generated outside the scope of the work for creating Release B) and which are deemed to be essential for the proper functioning of the release.

• The symbolic link gtm points to the GT.M directory used for this VistA directory.

• The file env sets up the GT.M environment:

# env - file to be sourced to create VistA environment
#
# This temporary version of the commands to set up the VistA
# environment assumes that the parent and child use the same
# version of GT.M.

if [[ -d parent ]] ; then
  pushd parent 1>/dev/null
  source ./env
  popd 1>/dev/null
fi

if [[ -n $routines ]] ; then
  export routines="$PWD/p $PWD/o($PWD/r) $routines"
else
  export routines="$PWD/p $PWD/o($PWD/r)"
fi
if [[ -f $PWD/g/mumps.dat ]] ; then export vista_home=$PWD ; fi

source gtm/gtmprofile
export gtmgbldir=$PWD/g/mumps.gld
export gtmroutines="$routines $gtm_dist"

• The script install creates a new integration directory to be used in creating a new release from this release. Normally, the initial database file of the integration directory is created by uncompressing mumps.dat.gz. Here is the script install:

#!/bin/bash
#
# install - create new VistA environment as child of current environment
#
# This temporary version of the install script assumes that both
# have the same GT.M version.  Also, there is no error handling yet,
# and no provision yet for a separate database on the child.
#
# Set up standard environment to run VistA

# Set up the environment for VistA
pushd `dirname $0` 1>/dev/null
export parent=$PWD
source ./env
popd 1>/dev/null
echo "Parent is" $parent

# Create the directory structure for the child
mkdir -p $1/{g,o,r,p,tmp}

# Link the child to the parent
ln -s $parent $1/parent

# Give the child the same GT.M version as the parent
ln -s $parent/gtm $1/gtm

# Copy other files in this directory
find $parent/ -maxdepth 1 -type f \! -name \*~ -exec cp {} $1/ \;

# Make run & cprs_direct executable because it may not be executable for the parent
chmod u+x $1/run $1/cprs_direct
find $1 -maxdepth 1 \( -name run -o -name cprs_direct \) -perm -4 -exec chmod o+x {} \;
find $1 -maxdepth 1 \( -name run -o -name cprs_direct \) -perm -40 -exec chmod g+x {} \;

# Give the child a global directory, for now the same as the parent
cp $parent/g/mumps.gld $1/g/mumps.gld

# Give the child a database
if [[ -e $parent/g/mumps.dat.gz ]] ; then
  echo "Unzipping from parent to create initial database"
  gzip -d <$parent/g/mumps.dat.gz >$1/g/mumps.dat
  $gtm_dist/mupip set -journal="enable,on,before" -file $1/g/mumps.dat
  chmod o+r,o-w,g+w $1/g/mumps.{dat,mjl}
else
  echo "Using parent database in this environment"
fi

# Make files except database files, journal files & directories read-only
find $1 -type f \! -name \*.dat \! -name \*mjl -exec chmod a-w {} \;
find $1 -type d -exec chmod 775 {} \;

echo "Default permission for development environment is for all to read and group to write - please fix if needed"

In a future enhancement, option --newgtm gtmver, will specify that the integration directory is to be configured to run a different version of GT.M (as specified by the directory gtmver) from the one used by the release. If --newgtm gtmver is used, the default assumption will be that the database will need to be extracted with the version of GT.M used by Release A, and loaded into a new database created with the version of GT.M to be used in the integration directory. When --newgtm gtmver is used, a an additional optional option --nonewdb will be available to specify that the releases of GT.M have compatible database formats, and that database from Release A can be used unchanged in the integration directory. Note that the use of --newgtm gtmver will almost certainly result in a command that takes a long time, because the database may need to be extracted and reloaded, and the routines recompiled.

Here is a listing of a top level release directory:

bhaskar@Makaira:~$ ls -l /opt/FOIAVistA20051021
total 912
dr-xr-sr-x  2 root staff   4096 Sep 15 13:19 CPRS_Gui
-r-xr-xr-x  1 root staff    216 Nov 22 09:21 cprs_direct
-r--r--r--  1 root staff    583 Nov 22 09:21 env
dr-xr-xr-x  2 root staff   4096 Oct 27 14:00 g
lrwxrwxrwx  1 root staff     18 Nov 22 09:26 gtm -> /opt/gtm_V5.0-000C
-r-xr-xr-x  1 root staff   1832 Nov 22 10:44 install
dr-xr-xr-x  2 root staff 446464 Nov 22 07:10 o
dr-xr-sr-x  2 root staff   4096 Nov 22 09:22 p
dr-xr-xr-x  2 root staff 446464 Oct 27 13:38 r
-r--r--r--  1 root staff    252 Nov 22 09:21 run
-r-xr-xr-x  1 root staff   3633 Oct 30 06:42 vista

VistA Integration Directories

Here is an example of creating an integration environment from a release environment:

vista@Makaira:~$ /opt/FOIAVistA20051021/install LeonardoDaVistA0.1
Parent is /opt/FOIAVistA20051021
Unzipping from parent to create initial database
%GTM-I-JNLCREATE, Journal file /home/vista/LeonardoDaVistA0.1/g/mumps.mjl created for database file /home/vista/LeonardoDaVistA0.1/g/mumps.dat with BEFORE_IMAGES
%GTM-I-JNLSTATE, Journaling state for database file /home/vista/LeonardoDaVistA0.1/g/mumps.dat is now ON
Default permission for development environment is for all to read and group to write - please fix if needed
vista@Makaira:~$

A VistA integration directory is similar to a VistA release directory, with the following differences.

• Subdirectory g contains a database file, mumps.dat, rather than a compressed database file, mumps.dat.gz.

• The symbolic link parent points to the release directory from which this integration directory was created. A release directory does not have a symbolic link called parent. (The recommended way for a script to determine whether a directory is a release directory or an integration / development directory is to test for the existence of parent.)

• A script run which invokes and runs VistA in that integration environment. Note that a VistA release may physically include a run script, but it will not work because it is not possible to directly run VistA from a release directory. Here is the run script:

#!/bin/bash
#
# run - run this instance of VistA

# Set up the environment for VistA
pushd `dirname $0` 1>/dev/null
export parent=$PWD/parent
source ./env
popd 1>/dev/null

if [[ -n $1 ]] ; then
  $gtm_dist/mumps -run $1
else
  $gtm_dist/mumps -dir
fi

• The cprs_direct script handles connection requests from CPRS GUI client processes that come in via inetd/xinetd:

#!/bin/bash
#
# cprs_direct - start a process to serve the CPRS Gui client

# Set up the environment for VistA
export HOME=/home/`whoami`
cd `dirname $0`
export parent=$PWD/parent
source ./env

# Run the server for the CPRS GUI client
cd tmp
$gtm_dist/mumps -run GTMLNX^XWBTCPM

• When the install script is executed, it will create a development directory.

• Except for database files, journal files, and directories, the normal protection on all directories and files in an integration directory should be read-only. When promoting code to the integration directory from a development directory, or to install externally generated patches in the p subdirectory, needed directories should be made read-write, sources copied in, object file(s) generated, and directories again made read-only.

Below is a listing of the directory of an integration environment created from the release in the example above. Notice parent link pointing to the directory of the parent environment. (In this case, the gtm link points to the gtm link in the parent, rather than /opt/gtm_V5.0-000C. It doesn't matter either way.)

bhaskar@Makaira:~$ ls -l /home/vista/LeonardoDaVistA0.1
total 48
-r-xr-xr-x  1 vista vista  216 Nov 22 09:28 cprs_direct
-r--r--r--  1 vista vista  583 Nov 22 09:28 env
drwxrwxr-x  2 vista vista 4096 Nov 22 10:11 g
lrwxrwxrwx  1 vista vista   26 Nov 22 09:28 gtm -> /opt/FOIAVistA20051021/gtm
-r-xr-xr-x  1 vista vista 1832 Nov 22 10:44 install
drwxrwxr-x  2 vista vista 8192 Nov 22 09:33 o
drwxrwxr-x  2 vista vista 4096 Nov 22 09:28 p
lrwxrwxrwx  1 vista vista   22 Nov 22 09:28 parent -> /opt/FOIAVistA20051021
drwxrwxr-x  2 vista vista 8192 Nov 22 09:32 r
-r-xr-xr-x  1 vista vista  252 Nov 22 09:28 run
drwxrwxr-x  2 vista vista 4096 Nov 22 09:28 tmp
-r-xr-xr-x  1 vista vista 3633 Nov 22 09:28 vista

VistA Development Directories

A development directory is like an integration directory, except that it does not normally have its own database file or global directory. However, as discussed above, it may be entirely appropriate for a development directory to have database files and global directories.

Indeed, since development directories are hierarchical – a development directory is an integration directory for development directories below it, and itself contributes to an integration directory above it, an integration directory is really nothing more than a development directory that does not itself contribute to an integration directory above it, but instead is used to create a release.

Note that although a GT.M database file can be opened by only one GT.M version at a time, it is quite easy to create a directory structure where Release A, the integration area, and each development area use different versions of GT.M. In the most common case, where the integration and development areas use a different version of GT.M from Release A, the only additional configuration requirement is to create a directory in the integration area for object files generated from the source files in Release A with the version of GT.M used for integration and development of Release B