NetNews Usenet Archive 1993 #3/NN_1993_3.iso/spool/comp/unix/cray/349

Text File | 1993-01-21 | 11.5 KB | 406 lines

Newsgroups: comp.unix.cray
Path: sparky!uunet!zaphod.mps.ohio-state.edu!magnus.acs.ohio-state.edu!ggawboy
From: [e-mail] (Galen F Gawboy)
Subject: Re: Where do you put the home directories!
Message-ID: <[e-mail]>
Sender: [e-mail]
Nntp-Posting-Host: bottom.magnus.acs.ohio-state.edu
Organization: The Ohio State University
References: <[e-mail]> <[e-mail]>
Date: Fri, 22 Jan 1993 01:09:06 GMT
Lines: 393

In article <[e-mail]> [e-mail] ("Sharad Gandhi") writes:
>When I was working on PSC Cray Y/MP. I was able to come up with
>a different jobfile to run jobs on CRAY without telnet'ing and
>ftp'ing from your local machine. The primary requirement is
>that your local machine should support remote command such as
>rsh, rcp etc. If it does then do following:
>
>1. open a .rhosts file on your local machine and add the
>following line:
>
>pscymp.psc.edu <your-user-id-on-cray>
>
>2. open a .rhosts file in your cray account with the following
>line:
>
><address of your local machine> <your user id on local machine>
>
>4. to run submit job on cray:
>
>rcp a.nqs <cra-id>@pscymp.psc.edu:<directory>
>rsh pscymp.psc.edu -l <cray-id> <job-submit-command> a.nqs
>
>See if it works and help you in saving your time.
>
>sharad

 Or better yet you may want to use craysub. This script will also send you
an email message on your home machine when the job is done. It requires the
existance of a crayqueues file which varies from site to site. For example if
you often use use two queues, fourmegb and universe the crayqueues file would
be:

cat> crayqueues <<\eof
LOCALNAME QNAME  MEM  CPU
fourmegb  fourmeg 4Mw 5000
universe  universe 32Mw 26000
eof

It also requires the existence of a .temparms file, which specifies the
pathnames on the cray. A sample .parms file is:

cat > .parms <<\ eof
#
#  Variables used on the cray.
#
set -ka
PATH=/where/my/executables/live ; export PATH
TEMPPATH=/where/my/huge/intermediate/files/will/be
FORTOUT=$TEMPPATH/$$.out
export TEMPPATH FORTOUT
#
#  Test for and create,if necessary, various directories used
#  on the cray.
#
if test -d $TEMPPATH
then
    :            # some times you might want to clean this directory.
else
    mkdir $TEMPPATH
fi
cd $TEMPPATH
set +v
eof

The usage would be

m4 jcl | craysub -b .parms -q universe -r bj -o bj.out -T 12000 -M 22Mw

  The jcl file would be the same jcl you would use on for your typical job.

  The shell script was originally written by Eric Stahlberg. It has since
been modified by Bob Zellmer, Gary Kedziora, and to an extremely minor extent
myself. Hope someone finds this helpful.

-Regards

Galen

(snip here) 8-<--------------------------------------------------------------
#! /bin/sh
#
# created 1/4/89 Eric Stahlberg
# updated 20/nov/90 Eric Stahlberg
#
# This is an experimental script to modify a given
# shell script to be able to run under NQS on the OSC Cray
#
# Various shell variables set
crayuser=$CRAYUSER      # cray userid
craypass=$CRAYPASS      # cray passwd
jobname=$USER           # job name -- default to sun userid
cpulim="T 20"           # time limit -- default to 20 sec
timeset=0               # flag indicating explicit time set
memlim="m 2Mw"          # memory limit
memset=0                # flag indicating explicit memory set
nqslog=NQS.LOG$$        # default to keep login output
scriptfile=NQS.OUT$$    # script output run through nasa
sunfile=`pwd`/'$QSUB_REQID' # default file name
setopt=akx              # default set options
beforefile=/dev/null    # file to include as first executed by NQS
afterfile=/dev/null     # file to include as last processed by NQS
frontend=`hostname`.mps.ohio-state.edu
device=`tty`
nqsstream=nqs.$$
craysubtmp=/tmp/craysub$$
queuelist=/usr/local/localadm/bin/crayqueues
#
if [ -n "$QUEUEFILE" ] ; then
   qfile=$QUEUEFILE
else
   qfile=$queuelist
fi
#
# Notice that the queue will specified to express by default on the time
# Variables CRAYUSER CRAYPASS are set by the user and exported.
# USER is a shell system variable.
#
# preprocess command line
lineopts="$*"
set -- `getopt edfnzZs:l:u:p:r:t:T:m:M:o:q:a:b:c $lineopts`
if [ $? != 0 ] ; then
   echo "Incorrect options specified"
   exit 1
fi
for opt in $* ; do
    case "$opt"
    in
    -d | -e | -f | -n | -z | -Z | -c ) shift 1;;
    -s | -l | -u | -p | -r | -t | -m | -T | -M ) shift 2 ;;
    -o | -q | -a | -b ) shift 2 ;;
    -- ) shift ; break;;
    esac
done
#
# check for suffix .m4 on files to run macro preprocessor
# from /usr/5bin
if [ -f $1.m4 ] ; then
  /usr/5bin/m4 $1.m4 >> $craysubtmp
else
  if [ `expr "$1" : ".*\.m4" ` -ne 0 ] ; then
     /usr/5bin/m4 $1 >> $craysubtmp
  else
     cat $1 > $craysubtmp
  fi
fi
defaults=`egrep '#craysub' $craysubtmp | head -1 |cut -c9-`
lineopts=" $CRAYOPTS $defaults $lineopts"
echo craysub  $lineopts
# check various flags
set -- `getopt edfnZzs:l:u:p:r:t:T:m:M:o:q:a:b:c $lineopts`
if [ $? != 0 ] ; then
   echo "usage error"
   exit 1
fi
#
for opt in $* ; do
    case "$opt"
    in
        -c ) colparms=yes;
             if [ ! -f $HOME/.colparms ] ; then
               echo " No .colparms file in " $HOME
               exit  1
             fi
             shift ;;
        -d ) nqslog=/dev/null; shift ;;
        -e ) echocode=yes; shift;;
        -f ) delfortout=yes; shift;;
        -n ) dest=">/dev/null"; shift ;;
        -Z ) if [ "$zapme" != "yes" ] ; then
                zapme="yes"
             else
                zapme="no"
             fi
             shift;;
        -z ) if [ "$mailme" != "yes" ] ; then
                mailme="yes"
             else
                mailme="no"
             fi
             shift;;
        -a ) if [ `expr "$2" : "-.*"` -ne 0 ] ; then
              echo "specify last execute file. "
              exit 1
            else
              if [ `expr "$2" : "/.*"` -eq 0 ] ; then
                afterfile=`pwd`/$2
              else
                afterfile=$2
              fi ;
             fi ; shift 2;;
        -b ) if [ `expr "$2" : "-.*"` -ne 0 ] ; then
              echo "specify first execute file. "
              exit 1
            else
              if [ `expr "$2" : "/.*"` -eq 0 ] ; then
                beforefile=`pwd`/$2
              else
                beforefile=$2
              fi
             fi ; shift 2;;
        -s ) if [ `expr "$2" : "-.*` -ne 0 ] ; then
               echo "specify set parms"
               exit 1
             else
               setopt="$2"
             fi ;  shift 2;;
   -u | -l ) if [ `expr "$2" : "-.*"` -ne 0 ] ; then
              echo "specify cray userid. "
              exit 1
            else
              crayuser="$2"
            fi ; shift 2;;
        -p) if [ `expr "$2" : "-.*"` -ne 0 ] ; then
              echo "specify cray password."
              exit 1
            else
              craypass="$2"
            fi ; shift 2;;
        -r ) if [ `expr "$2" : "-.*"` -ne 0 ] ; then
              echo "specify cray jobname."
              exit 1
            else
              jobname="$2"
            fi ; shift 2 ;;
        -T ) if [ `expr "$2" : "-.*"` -ne 0 ] ; then
              echo "specify cray cputime. "
              exit 1
            else
              if [ `expr "$2" : "[0-9]*"` -eq 0 ] ; then
                 echo "Invalid time specification."
                 exit 1
              else
                   timeset=1
                   cpulim="T $2"
              fi
            fi ; shift 2;;
        -t ) if [ `expr "$2" : "-.*"` -ne 0 ] ; then
              echo "specify cray per process cputime. "
              exit 1
            else
              if [ `expr "$2" : "[0-9]*"` -eq 0 ] ; then
                 echo "Invalid time specification."
                 exit 1
              else
                   timeset=1
                   cpulim="t $2"
              fi
            fi ; shift 2;;
        -M ) if [ `expr "$2" : "-.*"` -ne 0 ] ; then
              echo "specify cray memory limit. "
              exit 1
            else
              if [ `expr "$2" : "[0-9]*.*"` -eq 0 ] ; then
                 echo "Invalid memory specification."
                 exit 1
              else
                   memset=1
                   memlim="M $2"
              fi
            fi; shift 2;;
        -m ) if [ `expr "$2" : "-.*"` -ne 0 ] ; then
              echo "specify cray per process memory limit. "
              exit 1
            else
              if [ `expr "$2" : "[0-9]*.*"` -eq 0 ] ; then
                 echo "Invalid memory specification."
                 exit 1
              else
                   memlim="m $2"
                   memset=1
              fi
            fi; shift 2;;
        -o ) if [ `expr "$2" : "-.*"` -ne 0 ] ; then
              echo "specify output file. "
              exit 1
            else
              if [ `expr "$2" : "/.*"` -eq 0 ] ; then
                sunfile=`pwd`/$2
              else
                sunfile=$2
              fi
            fi ; shift 2;;
         -q ) if [ `egrep -c "$2" $qfile` -eq 0 ] ; then
                echo "Invalid queue specification"
                exit 1
              else
               qinfo=`egrep "$2" $qfile | head -1 `
               qname=`echo $qinfo | cut -f2 -d" "`
               if [ $memset -eq 0 ] ; then
                 memlim=`echo $qinfo | cut -f3 -d" "`
                 memlim="m $memlim"
               fi
               if [ $timeset -eq 0 ] ; then
                 cpulim=`echo $qinfo | cut -f4 -d" "`
                 cpulim="t $cpulim"
               fi
              fi
              shift 2 ;;
        -- ) shift; break
      esac
done
echo "# user=$crayuser pw=$craypass
#QSUB-r $jobname
#QSUB-eo
#QSUB-l$cpulim
#QSUB-l$memlim" > $nqsstream
if [ -n "$qname" ] ; then
  echo "#QSUB-q $qname" >> $nqsstream
fi
echo "#QSUB-o $nqslog
#QSUB-nr" >> $nqsstream
# insert pre execute file
cat $beforefile >> $nqsstream
# execute .colparms if necessary
if [ "$colparms" = "yes" ] ; then
#  echo ".colparms" >> $nqsstream
cat $HOME/.colparms >> $nqsstream
fi
#echo "(
echo "sh <<\EOF--sh--craysub 2>&1 | nasa >> $scriptfile
ja
date
logname
hostname
echo \" \$QSUB_REQID queue info cpulim=$cpulim memlim=$memlim \"  " >> $nqsstrea
m
#
if [ -n "$setopt" ] ; then
 echo "set -$setopt " >> $nqsstream
fi
# place commands into jobstream
  cat $craysubtmp >> $nqsstream
  rm $craysubtmp
#
if [ -n "$setopt" ] ; then
 echo "set +$setopt" >> $nqsstream
fi
echo "ja -st
EOF--sh--craysub
#) 2>&1 | nasa >> $scriptfile
echo " " >> $scriptfile
cat \$FORTOUT >> $scriptfile
mv $scriptfile \$FORTOUT" >> $nqsstream
#
#
echo "cat >ftp.xfer <<ENDFTP
user $USER $SUNPASS
status
put \$FORTOUT $sunfile
bye
ENDFTP
ftp -n $frontend < ftp.xfer
rm ftp.xfer " >> $nqsstream
#
#
# remove $FORTOUT if requested
if [ "$delfortout" = "yes" ] ; then
  echo "rm \$FORTOUT" >> $nqsstream
fi
#
# insert post execute file in jobstream
cat $afterfile >> $nqsstream
# check for mailback response
if [ "$mailme" = "yes" ] ; then
cat >> $nqsstream <<ENDMAIL
 echo " OSCA JOB $jobname [\$QSUB_REQID]
returned to file $sunfile" | \
 mail  $USER@$frontend
ENDMAIL
fi
# check for zap response to job completion
if [ "$zapme" = "yes" ] ; then
cat >> $nqsstream <<ENDZAP
  remsh $frontend -l $USER \
  "echo ^G +++ [\$QSUB_REQID] $jobname complete. +++ > $device "
ENDZAP
fi
#
if [ "$echocode" = "yes" ] ; then
cat $nqsstream
rm $nqsstream
else
 eval echo "Submitting job to the cray" $dest
 eval rcp $nqsstream $[e-mail]:$nqsstream
 eval rsh osca -l $CRAYUSER -n qsub $nqsstream $dest
 eval rsh osca -l $CRAYUSER -n rm $nqsstream
 eval echo "Job submitted"   $dest
 rm $nqsstream
fi