Chapter 21. Performance options for REXX.

This final chapter of the Telecourse focuses on the performance and security aspects of your REXX procedures.

Introduction.

1. Readability, ease of writing and execution performance are three aspects that not always follow the same rules.  We think you should give them importance in that order.  Only when you have procedures that have to be 'super-optimized' (e.g. routines running in servers or frequently used by many users), the performance aspect should take precedence.  (The BENCH goodie described in appendix C can help you with this). 

   Of course, when readability, ease of writing and execution performance can go together, it's clear that you should opt for that solution.

2. More than optimizing the code, other techniques will enhance the performance for you:
   • Use the REXX Compiler to compile your procedures
   • Load your procedures in storage permanently
   • Bundle all your routines for one application in one file instead of calling external subroutines, to avoid the overhead associated with CMS file search and I/O to load the procedure in storage.

In Appendix G. "VM/ESA REXX Performance Guidelines" we reproduce a very interesting article that appeared in the Washington Systems Center Technical Bulletin VM/ESA Release 2.1 Performance Report (GC24-5673).  The main part is devoted to comparisions between coding styles and techniques, with the purpose of finding the best performer.

If you don't want to spend the time reading the whole text, then these are the general conclusions that you should remember:

1. use address COMMAND to execute CP, CMS or external commands (see lesson 1 for details);
2. enclose literal data in quotes;
3. avoid switching between environments;
4. combine as much commands or statements as possible in one statement;
5. use VALUE() instead of GLOBALV.  In general, prefer the REXX functions above other CMS or CP commands (e.g. Time() instead of CP Q TIME) ;
6. code the most probable condition first in a SELECT structure, although for readability or coding ease, it may appear more natural to code the easy or obvious conditions first and keep the remaining condition for the OTHERWISE section;
7. eliminate unused or useless parts in the procedures;
8. avoid to code a statement separator (;) at the end of a line (especially for PL/I or PASCAL programmers...);
9. minimize wildcards in CMS file identifications (this is more a general CMS performance issue than a REXX issue).

Most of these rules correspond with what we have stressed throughout this course, don't they ?

We can now look into details of other methods that enhance the performance of your procedures.  These are:

1. Load procedures in storage
2. Use the REXX Compiler

Load procedures in storage.

The sequence of operations done by the system to prepare the execution of a REXX procedure are:

1. Search for the procedure (see Lesson 1, Chapter 3 for the CMS search order).
2. Load the procedure from disk into storage (involves I/O)

Both steps can be considered system overhead for the user.

In this chapter, we will see that it is possible to reduce this overhead if we use one of the methods that store the procedure in storage(footnote 1).

If you review the CMS Command Search order, then you'll see that CMS first looks for the procedures stored in storage.  So, loading in storage reduces the overhead both for the search step as for the load step.

There are two major techniques by which a procedure can be loaded into storage:

1. in the users' private storage via EXECLOAD ;
2. in storage shared by many or all users on the system, the so-called shared segments.

It's clear that shared segments give an advantage to all users, while the EXECLOAD method profits only to that one specific user.  Let's look at the details now.

EXECLOAD

                           +-*--=--=----------------------------+
 >>--+-EXECLoad-+--fn--ft--+------------------------------------+------->
     +-EXLoad---+          !         +-=--=-------------------+ !
                           +-+-fm-+--+------------------------+-+
                             +-*--+  !           +-=--------+ !
                                     +-execname--+----------+-+
                                                 +-exectype-+
    +-(--User---------------------------+
 >--+-----------------------------------+------------------------------><
    !   (1) +-User---+                  !
    +-(-----³--------³--+------+--+---+-+
            +-SYstem-+  +-Push-+  +-)-+

Use the EXECLOAD command to load an exec or XEDIT macro into storage and prepare it for execution.

Operands

fn	is the file name of the exec to be loaded.
ft	is the file type of the exec to be loaded.
fm	is the file mode of the exec to be loaded.  The default for file mode is an asterisk (*).  You must specify fm (or *) if you want to specify an execname and exectype.
execname	is the name to be assigned to the loaded exec.  The default is '=', which means the exec's present file name is to be used.
exectype	is the type to be assigned to the loaded exec.  The default is '=', which means the exec's present file type is to be used.

Options

user	specifies that the storage for the loaded exec is allocated from user free storage.  This is the default.
SYstem	specifies that the storage for the loaded exec is allocated from nucleus free storage.
Push	specifies that the exec is loaded whether an exec by the same name already exists in storage.  This loaded exec does not replace the existing exec.  Subsequent invocation of this execname and exectype executes the most recently loaded version.  Also, a subsequent EXECDROP of this execname and exectype drops the most recently loaded version.

Usage Notes

1. If SET INSTSEG is ON and you attempt to load an exec into storage with the same execname and exectype as a shared exec - hence one that is loaded in a shared segment - then you must specify the PUSH option.
2. To list the execs in storage and in a CMS installation saved segment, use the EXECMAP command.  To remove an exec from storage or to discontinue use of an exec in a CMS installation saved segment, use the EXECDROP command.  Use the SET INSTSEG OFF command to discontinue use of all shared execs temporarily.  To determine the status of a specific exec, use the EXECSTAT command.
3. The amount of storage required to load an exec includes system overhead for the control blocks and I/O buffers required to maintain the exec in storage.

Specifying the following:

   execload tphone exec a = xedit (system

loads the TPHONE EXEC A into nucleus free storage and assigns to it the name TPHONE XEDIT.  This is the example as given in the manual, but we find it not so good.  In lesson 4 we learned about the self-contained EXEC procedures, and there is thus no need to load an EXEC as if it were an XEDIT macro.

The difference between the USER and SYSTEM attributes is important.  When the procedure is loaded in the user free storage (the default), then this storage will be flushed when you abend or issue HX.  If the procedure is loaded in the system or nucleus free storage, this procedure will be preserved in storage at abend or HX.

Storing EXECs in Shared Segments

In addition to the advantages of pre-loading the procedure in storage, the use of shared segments makes the same storage area to be shared by many users on the system.  This reduces the load on the paging subsystem.

EXEC procedures and XEDIT macros can be stored in shared segments.  A shared segment can contain many different procedures.

There are two different implementations of shared segments that can contain procedures:

1. The CMS Installation Segment (commonly called CMSINST).  This segment is automatically loaded by CMS at IPL, unless you specify IPL CMS PARM INSTSEG NO.
2. A logical shared segment.  Such a segment is loaded via a SEGMENT LOAD command (could be automated in the SYSPROF EXEC or PROFILE EXEC).

The first type existed long before the second type was implemented, and is probably the most used and best known.  The CMS Installation Segment can only contain procedures.  Logical shared segments on the other hand can also contain

• modules
• CSL routines
• Language Message Repositories
• Minidisk File Directories
• User data

The CMSINST segment is generated via the DCSSGEN command, while the logical shared segments are generated via the SEGGEN command.  SEGGEN needs these segment definition files:

• Physical Segment definition file (PSEG), lists the logical segments it should contain
• Logical Segment definition file(s) (LSEG), lists the elements that are contained in the segment(s).

The result of the SEGGEN command is that the shared segments are generated, but also that the names and relations are stored in the SYSTEM SEGID S file, which is used by CMS when SEGMENT LOAD/RESERVE/RELEASE/PURGE commands are executed.

CMS Search order.

We have to come back on CMS' search order for procedures, especially in the case of the procedures stored in the CMS Installation Segment (CMSINST) or Logical Shared Segments.

You should first understand this:

1. We mentioned that the USER and SYSTEM attributes that can be specified on the EXECLOAD command.  Procedures stored in shared segments get the attribute SHARED.
2. By default, CMSINST is automatically loaded at IPL of CMS.  The command SET INSTSEG ON S is also implicitly executed.  This means that the segment is loaded in storage and is placed in the CMS search order just before the S-disk.  You can for example, issue the SET INSTSEG ON B command to place the shared procedures of CMSINST before the B-disk in the search order.
3. Logical Segments are loaded via a SEGMENT LOAD segname  command.

The complete search order for procedures is thus:

1. EXECLOADed procedures and logical segment members.
2. CMS minidisks, SFS directories and CMSINST segment (if INSTSEG is ON) in alphabetical access search order.

Members of a logical shared segment can also be defined as extensions of the CMSINST segment, and are then found in step 2 instead of 1.

So, if you have a procedure FILELIST EXEC on the S-disk, on the Y-disk, in the CMSINST shared segment and on your B-disk, the version on the B-disk will be executed, unless you issue the SET INSTSEG B command, in which case the CMSINST version will be executed.

If however, you EXECLOAD the version on the Y-disk, then this version will be executed.

The EXECMAP command can inform you of any procedure loaded in storage.  Here is an example:

Name        Type   Usage    Records      Bytes       Attribute   Segname

EXECUTE     XEDIT      0        865      37632       SHARED      CMSINST
FILELIST    EXEC       0        388      15152       SHARED      CMSINST
PARSE       XEDIT      7        224      10080       SHARED      CMSINST
PEEK        EXEC       1        570      25688       SHARED      CMSINST
PROFFLST    XEDIT      0        232      11288       SHARED      CMSINST
PROFILE     XEDIT      1         30      31016       USER
PROFILE     XEDIT      0        140       5312       SHARED      CMSINST
QUERY       XEDIT      1        127       5824       USER
RECEIVE     XEDIT      0         35       1688       SHARED      CMSINST
RZLINK      EXEC       2        590     608936       SHARED      RZLINK
RZLNKM      EXEC       2         70      72296       SHARED      RZLINK
SETSYN      XEDIT      5         94       4720       SYSTEM
SPLTJOIN    XEDIT      3         45       2176       USER
SYSPROF     EXEC       1        231      11816       SHARED      CMSINST

From this output we can learn that:

1. CMSINST is active and there is also an logical shared segment called RZLINK.  All procedures in those segments have the SHARED attribute.
2. PROFILE XEDIT exists both in CMSINST and in the user free storage (through EXECLOAD).  The latter of the two will be executed.
3. PROFILE, QUERY and SPLTJOIN XEDIT macros with USER attribute are EXECLOADed in user free storage.  This was in fact done by XEDIT itself.  Indeed, you must know that XEDIT will EXECLOAD any macro that is used during the XEDIT session and keep it there for subsequent use.  (The EXECMAP command was executed from an XEDIT session in this example).
4. The SETSYN XEDIT macro has the attribute SYSTEM. 

When a procedure is loaded in storage, the minidisk (or SFS directory) that contains the 'source' procedure is not required anymore.  It is therefore possible to EXECLOAD a procedure through the SYSPROF EXEC, or to save it in a logical shared segment and have no other version available to the users on any of the accessed disks.  This can be considered a security feature, as end-users will have great difficulty in changing the procedure (it is still possible).  As we will see, compiled procedures can also be loaded in storage, and then the protection from modification is complete as a compiled procedure is totally unreadable by an end-user.

Virtual disks.

To complete this discussion, we have to add yet another possibility for loading procedures in storage.  Since VM/ESA Release 1.2.1, it is possible to define virtual minidisks which are implemented in storage.  DEFINE VFB-512 is the command that allows you to do this.

Would that be the best solution to improve the performance of the procedures ?  Think a while before reading on...

Virtual Disks have following drawbacks:

1. You have to FORMAT the virtual disk before use,
2. You have to copy the files to the disk, before you use the files,
3. Virtual disks are seen as regular minidisks by CMS, so regular, costly I/O requests are issued.  These requests have then to be translated by CP to read from the Virtual Disk in storage, and a page-in operation may be required to bring parts of the Virtual Disk into real storage.

For procedures in shared segments, in the worst case, only a page-in operation may be required.

REXX Compiler

We hope you were convinced about the real strong points of REXX:

• Excellent string handling, many built-in functions
• Easy to debug
• Arithmetic with unlimited precision
• No variable declarations needed
• Easy to learn, easy to read, easy to use
  (we hope, even more now, after this course...)
• Easy interfacing with the operating system

There are however a few weak points that make that REXX is not yet considered widely as a complete and general programming language:

• Interpreting isn't the fastest for some applications
• Source code must be exposed (danger for unauthorized modifications).

These weak points can be addressed by using the IBM Compiler and Library for SAA REXX/370  (program number 5695-013)(footnote 2)

The compiler effectively can make your procedures run as fast as other compiled programs, and the source code gets hidden from the user.

To give you an idea of the gains the compiler can give, see next table:

Compiled programs that include many...	Run this much faster...	Measured
Arithmetic operation	6 to 10 times	9.7
String and word processing	6 to 10 times
Constants and variables	4 to 6 times	5.8
References to procedures and built-in functions	4 to 6 times	4.9
Changes to values of variables	4 to 6 times	8.7
Assignments	2 to 4 times	25.2
Reused compound variables	2 to 4 times	4.4
Host commands	minimal	1.0

Note: For the measured values, no coding tricks were used to favor the compiler !

Rule of thumb: expect a 3 to 4 times improvement for a REXX program that is

• not too small
• spending most of its time in REXX

The results of the measurements are rather obvious when you think about it a while.

• Host commands (CMS, CP, XEDIT, ...) are not executed by REXX but by the Host, so, compilation will change almost nothing to this situation.
• On the other side, pure arithmetics can be highly optimized by a compiler.  You know that all variables are considered literal strings, until REXX discovers arithmetics must be done.  The interpreter will then do the numeric data conversion, calculate the result, and return this result to a literal string again.  The compiler, on the other hand, will, in most cases, detect that a variable is only used for arithmetic and store it immediately in binary format in the program.
• In the middle range of the table, we find the manipulation of constants and variables.  This is merely storage management where Host functions are required again.  The compiler can (and will in fact) optimize these operations too, but can not do it to the same extent as for the arithmetic operations.

The REXX compiler has a series of inherent optimizations:

• No tokenizing and statement parsing at run-time.  Understanding and analyzing what the programmer has coded is indeed done once at compile time.
• Direct addressing of both simple and compound variables.  The interpreter has to look up each variable each time it is referenced.
• Compiled output is optimized
  • common sub-expressions are recognized and become new temporary internal functions.  We had an example of this in Lesson 3:

     do queued()
        parse pull line
        select
          when left(line,1)='*' then iterate          /* ignore comments */
          when word(line,1)='KING' then call chess substr(line,10,20)
          when word(line,1) word(line,3)='FROM BOSS' then call myboss
          when word(line,1)='SKIP' then iterate
          otherwise Say 'Invalid card:' line
        end /* select */
     end /* queued */
    

    The word(line,1) function is executed several times, and the compiler may be able to discover this and optimize it for you.

  • Constant folding.  When a literal string is written as a variable (hence not inside quotes), the compiler may recognize this as not being a variable and make it constant for you.
  • Knowledge about the state of variables (initialized or not), numeric digits settings, and types of operands is used to suppress generation of unnecessary machine code.
  • Register notebook is used to suppress loading of addresses already in a register.
• Fast linkage to library routines.
• Optimized storage management
  • Different kinds of storage for different elements (e.g. single and compound variables, strings and numbers).
  • Optimized number of storage get and put.
• Arithmetic
  • Binary arithmetic (whenever possible)
  • String arithmetic optimized for large numbers
• Compound variable access
  • Lookup not always from top
  • Keep address for subsequent access
  • Optimization for integer tails

We reproduce the most pertinent parts from a document that discusses benchmark results comparing different coding techniques:

code	Interpreted	Compiled	Factor	Note
a='1 ' or a=1e0 a='1' or a=1	0.555 0.550	0.165 0.009	3.4 60	1
Numeric digits 8;a=1 Numeric digits 9;a=1 Numeric digits 10;a=1	0.550 0.550 0.550	0.151 0.009 0.009	3.6 58.5 58.5	2
a=hello world a=hello_world	0.505 0.283	0.102 0.023	5.0 12.3	3
a=hello world a='HELLO WORLD'	0.505 0.212	0.102 0.010	5.0 20.8	4
parse value 1 with a 1 b 1 c 1 d parse value 1 1 1 1 with a b c d a=1;b=1;c=1;d=1	0.938 1.182 1.009	0.413 0.296 0.043	2.3 4.0 23.4	5
a=left('1',1) a=left('1',1);a=a+0 a=left('1',1);a=(a+0)+0	0.559 0.558 0.558	0.161 0.009 0.008	3.5 63.2 72.4	6
call to inner subroutine call to outer subroutine	0.11 2.31	0.02 2.45		7

Notes:

1. Avoid fancy ways to code integers
2. NUMERIC DIGITS < 9 suppress binary arithmetic
3. First case, two variables have to be looked up, while one one in second case.  For interpreter, this halves the time, which is logical, but the compiler optimizes even further.
4. Assignment of a quoted string performs better.  The gains with the compiler are very high.  In any case, place constants between quotes to follow our rules.
5. Compiled assignment is faster than PARSE.  This is logical as the PARSE is done only at execution time.
6. Binary representation can be enforced.  By adding 0 to a character string '1', the compiler knows the variable will be used for arithmetic and can store it in binary format immediately from then on.
7. From the figures in the table it's clear that you should code frequently used subroutines inside your procedure.

Almost all examples show that when you follow good coding rules, both the interpreter as the compiler profit from it.  A compiled procedure is however always larger than the source version, and it will thus cost extra I/O to load it, unless you have loaded it into storage.

One final, but important remark: the compiler can not only generate compiled procedures (commonly called CEXECs(footnote 3)), but also text decks.  The latter can then be generated to modules (GENMOD) or can be linked by other programs (REXX or other languages).

The compiled procedures (CEXECs) can be manipulated as regular REXX procedures (e.g. EXECLOADed or stored in shared segments).

This is the end of the course.  Not everything is said about REXX, and the more you use it, the more you will discover new techniques for yourself.  If you liked our course, then you may consider taking the CMS Pipelines Telecourse too.  We would also appreciate some feedback from you, see our e-mail addresses in the Instructions.

(1) If there is only one user for the procedure, and he/she executes it only once in a session, there will be of course no gain in first loading the procedure in storage.
Back to text

(2) This product allows you to compile and run REXX procedures.  To run compiled REXX procedures you need only the IBM Library for SAA REXX/370 (program number 5695-014), and this is included free of charge in VM/ESA Version 2, Release 3.0 and subsequent.
Back to text

(3) When a source procedure is compiled, the output has a filetype CEXEC (or CXEDIT), and that explains the terminology.  CMS, however, does not recognize these filetypes, and it is therefore necessary to rename the source to SEXEC and the compiled version to EXEC.
Back to text

(4) In this discussion, XEDIT macro is an editor command that is implemented in an exec file with a file type of XEDIT.  XEDIT subcommands are the other commands that XEDIT understands, but are not in a file with file type XEDIT.
Back to text