
         C3 version 5:   Cluster Command & Control Suite
           Oak Ridge National Laboratory, Oak Ridge, TN,
     Authors: M.Brim, R.Flanery, G.A.Geist, B.Luethke, S.L.Scott
          Thomas Naughton, Geoffroy Vallee, Wesley Bland

          Copyright (c) 2000-2011    Oak Ridge National Laboratory
                                     All rights reserved.

                             NOTICE

 Permission to use, copy, modify, and distribute this software and
 its documentation for any purpose and without fee is hereby granted
 provided that the above copyright notice appear in all copies and
 that both the copyright notice and this permission notice appear in
 supporting documentation.
 
 Neither the Oak Ridge National Laboratory nor the Authors make any
 representations about the suitability of this software for any
 purpose.  This software is provided "as is" without express or
 implied warranty.
 
 The C3 tools were funded by the U.S. Department of Energy.



OVERVIEW
--------
Currently a few solutions are available for the installation of C3, 
the use of:
  - Binary packages for Debian (.deb) or RedHat-based (.rpm) systems,
  - a Makefile, which is for installation from source, or
  - a make_dist script, which is the original source installation mechanism. 

The remainder of this file gives details on pre-reqs, install and
configuration details for C3.




I. REQUIRED SOFTWARE
--------------------

Before C3 can be installed on a system, you must ensure that the following 
software is installed on your system.  If you are using the binary packages
these requirements will be resolved by the package management system, e.g.,
Debs, RPMs.  The following seven software packages are required:  
the C3 tools suite, Rsync, SSH (or OpenSSH), Python, and Perl (used by
'ckillnode' on clients).  

You must also configure that system to support host name resolution of the
machines listed in the configuration file (either through DNS or /etc/hosts).

Finally, if you wish to use the C3 pushimage command, which pushes system
images across a cluster, you must install SystemImager.

Instructions for obtaining each of these software packages are given below.

  C3 tools may be obtained from http://www.csm.ornl.gov/torc/C3

  Rsync, Perl, SSH, and Python should be included with your distribution.
  If they are not then download the source or binaries from their respected
  web sites.

  Python may be obtained from 
  http://www.python.org/
    C3 version 3 requires Python 2.0 or greater

    * Additionally C3 requires either the binary or a link to the python 
      interpreter to be in your PATH, which must be v2.0 or greater. 
      To check the version type, "python -V". 

  Perl may be obtained from http://www.perl.com
    C3 requires 5.005 or greater

  SystemImager may be obtained from 
  http://www.systemimager.org/


II. C3 INSTALLATION
-------------------

The simplest way to install C3 is via the binary packages.  C3 packages are
available for both Debian (.deb) and RedHat (.rpm) based system.  The
following details describe the install from source using the basic "tarball"
archive.  

Regardless of whether C3 is installed from source or via binary packages 
"Step C. configuration" should be performed to configure the system to use
C3.  [Note, this configuration is automated by tools like OSCAR.]

 NOTE: if you are wanting to use the scalable model of the C3 tools then
  follow steps A and B, read C as it still pertains to the scalable model,
  then see the README.scale file for the scalable instructions.

A. pre-install
    Begin by making sure that Rsync, OpenSSL, OpenSSH, PERL, and
    Python are installed.  Install Systemimager, if needed.  Install DNS
    or /etc/hosts as needed, and make sure that hostname resolution is 
    supported.

    Directions for downloading each of these packages are given in 
    Section I above.  Perl, Python, Rsync, OpenSSH, and OpenSSL are included 
    with most distributions

    You will need root access to install these packages on your system. 
        Follow the instruction in each package if you need to install them.

B. C3 install 

  (i) Installation from source (using Makefile)

      After you complete the pre-install (step A), install the Cluster
      Command & Control (C3) tools.  This installs the C3 command suite, but
      does not configure the commands or any local clusters for operation.
      Directions for the remaining tasks are given in 'Step C. configuration'.

      To install C3, it is possible to use the standard 'configure' and
      'make' tools at the top of the source tree. To compile and install C3
      from sources, execute, 

        ./autogen.sh && ./configure && make

      It is also possible to specify to target for the install using the
      DESTDIR parameter. For example, the following will install C3 into
      "/tmp/my_test",

        make install DESTDIR=/tmp/my_test
         
      Note that we decided to use autotools because the Python Distutils
      tool assumes that the piece of software you want to install is part of
      the Python installation. In our case, C3 is more concidered as a
      third-party self-contained piece of software (it is for instance
      installed in "/opt"), therefore the autotools are more suitable.

      Uninstallation using Makefile -- run the following to clean and
      uninstall C3,

        make clean

      If C3 has been installed in a specific directory, please use the
      DESTDIR parameter. For example, will uninstall C3 from "/tmp/my_test",

        make clean DESTDIR=/tmp/my_test


 (ii) Installation for Debian 

       You can install the packages using 'apt-get' from
       online repositories by adding the following to your 
       '/etc/apt/sources.list' file for the appropriate version/release, 
	   
         deb https://https://bison.csm.ornl.gov/repos/<distro_id>

       where <distro_id> refers to the Debian distribution your local
       Debian based distribution is compatible with. The support distro, and
       the respective distro_id, are:
           - debian 4 for x86 architecture: debian-4-i386
           - debian 4 for x86_64 architecture: debian-4-x86_64
           - debian 5 for x86_64 architecture: debian-5-x86_64
           - ubuntu 9.10 for x86_64 architecture: debian-5-x86_64
           - ubuntu 10.4 for x86_64 architecture: debian-5-x86_64
           - ubuntu 10.10 for x86_64 architecture: debian -5-x86_64
       For any other Debian based distribution, please install c3 from sources.

	   Then run the following command,

           # Server side
         apt-get update
         apt-get install c3

           # Client side
         apt-get install c3-ckillnode c3-c3cmd-filter


       You can also install the packages directly using 'dpkg', 
       
           # Where VER is the C3 version and architecture
           #
           # Server side
         dpkg -i c3-VER.deb 

           # Client side
         dpkg -i c3-ckillnode-VER.deb c3-c3cmd-filter-VER.deb



 (iii) Installation for RedHat-based systems (using YUM)

       You can install the packages using 'yum' from
       online repositories by cerating a file named ORNL.repo to your 
       '/etc/yum.repos.d/' directory, the content of the file being:

           # ORNL.repo
           #

           [onrl]
           name=CentOS-$releasever - ORNL_SRT
           baseurl=http://bison.csm.ornl.gov/repos/rhel-5-<arch>
           gpgcheck=0
       where <arch> is i386 or x86_64 based on your hardware architecture. Note
       that we are currently supporting only the RPM based distributions
       compatible with RHEL-5. For other RPM distributions, please install from
       sources.
       then run the following command,

           # Server side
         yum update
         yum install c3 c3-profiled

           # Client side
         yum install c3-ckillnode c3-c3cmd-filter


       You can also install the packages directly using 'rpm', 

           # Where VER is the C3 version and architecture
           #
           # Server side
         rpm -ivh c3-VER.rpm c3-profiled-VER.rpm

           # Client side
         rpm -ivh c3-ckillnode-VER.rpm c3-c3cmd-filter-VER.rpm


C. C3 configuration
    Specific instances of C3 commands identify their compute nodes with 
    the help of **cluster configuration files**:  files that name a set
    of accessible clusters, and that list and describe the set of 
    machines in each accessible cluster.  Cluster configuration files 
    are accessed in one of two ways:

    -.  explicitly:  an instance of a C3 command names a specific 
    configuration file, using a command-line switch.

    -.  implicitly:  an instance of a C3 command fails to name a specific
    configuration file, and the command defaults to the list of cluster
    descriptions given in /etc/c3.conf.
    
    When you install C3, you should create a default configuration file 
    that is appropriate to the site.  This file, which should be named
    /etc/c3.conf, should consist of a list of **cluster descriptor
    blocks**:  syntactic objects that name and describe a single cluster 
    that is accessible to that system's users.  

    The following is an example of a default configuration file that 
    contains exactly one cluster descriptor block:  a block that 
    describes a cluster of 64 nodes:

        cluster local {
            htorc-00:node0  #head node
            node[1-64]  #compute nodes
        }

    Cluster description blocks consist of the following basic elements:

    -.  a **cluster tag**:  the word "cluster", followed by a label,
        which assigns a name to the cluster.  This name--here, "local"--
        can be supplied to C3 commands as a way of specifying the cluster
        on which a command should execute.

    -.  an open curly brace, which signals the start of the cluster's
        declaration proper.

    -.  a **head node descriptor**:  a line that names the interfaces
        on the cluster's head node.  The head node descriptor shown here
        has two parts:

        -.  The string to the left of the colon identifies the head 
        node's **external** interface: a network card that links 
        the head node to computers outside the cluster.  This string
        can be the interface's IP address or DNS-style hostname.
        -.  The string to the right of the colon identifies the head 
        node's **internal** interface: a network card that links the
        head node to nodes inside the cluster.  This string can be 
        the interface's IP address or DNS-style hostname.

        Here, the head node descriptor names a head node with an external
        interface named htorc-00, and an internal interface named node0.

        A cluster that has no external interface--i.e., a cluster that is
        on a closed system--can be specified by either

        -.  making the internal and external name the same, or
        -.  dropping the colon, and using one name in the specifier. 

    -.  a list of **compute node descriptors**:  a series of individual
        descriptors that name the cluster's compute nodes.  

        The example given here contains exactly one compute node 
        descriptor.  This descriptor uses a **range qualifier** to 
        specify a cluster that contains 64 compute nodes, named node1, 
        node2, etc., up through node64.  A range qualifier consists of 
        -.  a first, nonnegative integer, followed by 
        -.  a dash, followed by 
        -.  a second integer that is at least as large as the first.

        In the current version of the C3 tools et, these range values are 
        treated as numbers, with no leading zeroes.  A declaration like

            cluster local {
            htorc-00:node0  #head node
            node[01-64] #compute nodes
            }

        expands to the same 64 nodes as the declaration shown above.  To 
        specify a set of nodes with names like node01, node09, node10, ...
        node64, use declarations like

            cluster local {
            htorc-00:node0  #head node
            node0[1-9]  #compute nodes node01..node09
            node[10-64] #compute nodes node10..node64
            }

    -.  a final, closing curly brace.

    Configuration files that specify multiple clusters are constituted as
    a list of cluster descriptor blocks--one per accessible cluster.
    The following example of a cluster configuration file contains three 
    blocks that specify configurations for clusters named local, torc, 
    and my-cluster, respectively:

        cluster local {
            htorc-00:node0  #head node
            node[1-64]  #compute nodes
            exclude 2
            exclude [55-60]
        }

        cluster torc {
            :orc-00b
        }

        cluster my-cluster {
            osiris:192.192.192.2
            woody
            dead riggs
        }

    The first cluster in the file has a special significance that is
    analogous to the special significance accorded to the first 
    declaration in a make file.  Any instance of a C3 command that fails
    to name the cluster on which it should run executes, by default, on 
    the first cluster in the configuration file.  Here, for example, any
    command that fails to name its target cluster would default to local.

    The cluster configuration file shown above illustrates three final
    features of the cluster definition language:  **exclude qualifiers**,
    **dead qualifiers**, and **indirect cluster** descriptors.

    **Exclude qualifiers** allow nodes to be excluded from a cluster's
    configuration: i.e., to be identified as offline for the purpose of
    a command execution.  Exclude qualifiers may only be applied to 
    range declarations, and must follow immediately after a range 
    declaration to which they are being applied.  A series of exclude 
    declarations is ended by a non-exclude declaration, or the final "}" 
    in a cluster declaration block.  
    
    An exclude qualifier can be written in one of three ways:
    -.   "exclude n", where n is the number of a node to exclude from the
         cluster;
    -.   "exclude[m-n]", where m, m+1, m+2, ..., n-1, n is the range of 
         nodes to exclude; or as
    -.   "exclude [m-n], which has the same effect as "exclude[m-n]".
    Note that a string like "exclude5" is parsed as a node name, rather 
    than as an exclude qualifier.

    In the above example, the two exclude qualifiers have the effect of
    causing node2, node55, node56 node57, node58, node59, and node60 to 
    be treated as offline for the purpose of computation.

    **Dead qualifiers** are similar to exclude qualifiers, but apply to
    individual machines.  In the example given above, the machine named
    "riggs" in the cluster named "my-cluster" is excluded from all
    computations.

    "Dead", like "exclude", is not a reserved word in the current version
    of the C3 suite.  A specification block like

        cluster my-cluster {
            alive:alive
            dead
        }

    for example, declares a two-machine cluster with a head node named 
    "alive" and a compute node named "dead".

    An **indirect cluster descriptor** is treated as a reference to 
    another cluster, rather than as a characterization of a cluster per 
    se.  In the example shown above, the descriptor

        cluster torc {
            :orc-00b
        }

    is an indirect cluster descriptor.  An indirect descriptor consists 
    of

    -.  a cluster tag, followed by, 
    -.  an **indirect head head node descriptor**, followed by
    -.  an empty list of compute node descriptors.

    An indirect head node descriptor consists of an initial colon, 
    followed by a string that names a **remote** system.  This name, 
    which can either be an IP address or a DNS-style hostname, is checked
    whenever a C3 command executes to verify that that the machine being
    referenced is **not** the machine on which that command is currently 
    executing.

    A command that is destined for an indirect cluster is executed by 

    -.  first forwarding that command to the remote cluster's head node
    -.  next, executing that command, relative to the remote machine's 
        default configuration file.

    For this feature to work properly, the remote machine must also 
    support a fully operational C3 suite (version 5.0) placed in the 
    /opt/c3-5 directory.  

    The indirect cluster descriptors can be used to construct **chains** 
    of remote references:  that is, multi-node configurations where an
    indirect cluster descriptor on a machine A references an indirect 
    cluster descriptor on a machine B.  Here, it is the system 
    administrator's responsibility to avoid circular references.

D. Post-install
    For the C3 ckill command to work properly, ckillnode must be copied 
    to a directory on each compute node on every supported cluster.  The 
    easy way to install ckillnode is to use cexec and cpush.  After 
    installing and configuring C3 (cf. steps A-C above), use the 
    following two commands to push ckillnode to each node in the default 
    cluster.

    cexec mkdir /opt/c3-5
    cpush /opt/c3-5/ckillnode

    For the scalable version a full C3 install is needed on each node.
    This can be accomplished by either installing the RPM on each node
    or pushing the tarball out and using cexec (non-scalable at this point)
    to run the install script on each node.
    
This completes the installation of the C3 tools.

E. Notes
    The relative positions of nodes in c3.conf files can be significant
    for C3 command execution.  Version 3 of the C3 suite allows the use 
    of node ranges on the command line.  The command line parameters used
    to specify the indices of compute nodes refer to relative node 
    positions in c3.conf.  

    Consider, for example, the semantics of node range parameters, 
    relative to the following c3.conf file:

    cluster local {
        htorc-00:node0  #head node
        node[1-64]  #compute nodes
        exclude 60
        node[129-256]
    }

    This cluster is made up of 192 nodes.  Here, 

    -.  the 64 nodes named node1 through node64 correspond to slots 0-63
    -.  the 128 nodes named node129 through node256 correspond to slots
        64-191--and **not**, for example, to slots 129-256.

    Note also that the excluded node--node60--acts as a place holder in 
    the range of indices: node60 is a relative index of 59, which allows
    nodes node61, node62, node63, and node64 to correspond to 60, 61, 62,
    and 63, respectively.  This "place holder" effect is an important 
    reason for explicitly specifying that a node is dead or excluded--as
    opposed to simply dropping that line from the specification.

    Two new tools in version 3.1 of the C3 tools suite support the 
    management of node numbers.  The first, cname, inputs a node name,
    and outputs that node's relative position (slot number).  The second,
    cnum, inputs a range of slot numbers, and outputs the names of the
    corresponding compute nodes.


III.  C3 SUITE DOCUMENTATION
---------------------------
    
C3 command documentation may be found in two locations.
  1. Quick Usage Info - enter "<command> --help" at the command line
  2. Full Man Page - enter "man <command>" at the command line

