CCAPrint Newsletter for Model 204 and System 1032 Users

You can use the User Language statements MQPUT and MQGET to put messages on an MQ queue and to get messages from an MQ queue. A message can be anything that you can assign to a %variable or put in an image. Furthermore, prior to issuing the MQPUT statement you can write multiple images to the Model 204 MQ_BUFFER area, which has a maximum size of 16 megabytes. You can place extremely long messages or data objects on an MQ queue. These features provide the capability of transferring entire files or datasets between Model 204 and any other MQ application using MQ/204 and IBM's WebSphere MQ, formerly known as MQSeries.

The most straightforward example of this implementation is the transfer of a sequential dataset to an MQ queue for subsequent MQGET processing from another MQ application. The receiving application can be Model 204 or another MQ-capable application running on the same or a different platform. Since a Model 204 file is a sequential dataset, Figure 1 shows how you can transfer an entire Model 204 file to another system using MQ/204.

MODIFY MQ_BUFFER SIZE=618400 OPEN QUEUE CSQ1.Q01 OUTPUT PERSISTENT CALL CHECKER ('OPEN QUEUE') OPEN EXTERNAL VEHICSEQ FOR INPUT CALL CHECKER ('OPEN EXTERNAL') REPEAT FOREVER REPEAT 100 TIMES

READ IMAGE PAGE FROM VEHICSEQ
CALL CHECKER ('READ IMAGE') /? MUST HANDLE EOF IN VEHICSEQ ?/
%PAGECOUNT = %PAGECOUNT + 1
WRITE IMAGE PAGE ON MQ_BUFFER
CALL CHECKER ('WRITE IMAGE')
END REPEAT

MQPUT MQ_BUFFER ON CSQ1.Q01
CALL CHECKER ('MQPUT')
MQCMIT CSQ1
END REPEAT

First, allocate the Model 204 file as a sequential dataset using the ALLOCATE command:

The pages or blocks in a Model 204 file have the DCB attributes of BLKSIZE=6184,RECFM=U. You can use the following image to describe those pages:

The file can then be read one page at a time using a READ IMAGE statement, written to the MQ_BUFFER area, and then placed on an MQ queue by an MQPUT statement. For improved performance, Figure 1 loads the MQ_BUFFER area with 100 pages prior to each MQPUT statement. You must include $STATUS code to interrogate and handle nonzero return codes from each external process. We recommend a subroutine, in our example, CHECKER.

Figure 1 illustrates that MQ/204 combined with WebSphere MQ can transfer any kind of data between any system or platform that supports WebSphere MQ. Code translation, such as ASCII to EBCDIC, is optionally handled by WebSphere MQ. Transfer can include data from:

MQ/204 is an extremely powerful facility for transferring data between operational or business units within an organization, between business partners, or between suppliers and customers. MQ/204 is a very easy-to-use extension to User Language, which can be used to enhance and streamline existing applications and processes in any organization.

In Part 2 of Query Efficiency, printed in CCAprint July 2001, We saw how the key tables are read using a variety of simple FIND commands. Now let's look at the internals of the FIND command processing. When you enter a FIND command System 1032 parses and compiles the command, verifying the syntax. The compiled code is passed to the Back End Execution Routine where all the processing takes place.

Many FIND commands that select a range of values scan the lower-level keys. To assemble the resulting selection set, the FIND command merges each record list that meets the selection criteria via logical OR on the record list bitmaps.

All the examples in Part 2 ignored Missing values. Missing values for a given attribute form a separate B-tree, following the same rules as the other keys. The B-tree structure is primarily for update efficiency; it has little to do with any query operations. For queries involving Missing values, the B-tree for the Missing values is read as a full scan-across. The scan reads as many blocks as required by the lower-level keys to store the record list for Missing.

Some FIND commands never access key tables. These are special cases where the information required to establish the selection set is actually given in the command itself.

The FIND ALL command never reads the key tables. System 1032 always keeps track of the number of records in the dataset. Given that selection sets are represented by bitmaps, a FIND ALL command is easily represented by a bitmap with as many bits as records, then turning all the bits on. As you'll see further on, this is subject to a few factors such as deleted records and consider sets.

FIND command processing reads the key tables to collect the $IDs of qualified records, which it uses to create the selection set. If you specify $IDs in your FIND command, you have done part of the processing. Your command is processed that much faster. There is no $ID key; $ID just provides an offset into the dataset records. For example, the following command tells FIND processing to create a bitmap where the sixth, 21st, and 334th bits are set, and all others are zero.

The remaining doesn't-use-keys FIND command uses the LAST keyword, which is most often issued with an AND or OR clause to expand or contract the current selection set. You can use a FIND LAST command by itself to eliminate the current sorting or to remove records deleted by someone else from your current selection set. System 1032 always knows what the current selection set is; the FIND LAST command essentially re-establishes the current set. Previous FIND commands are not re-executed and no keys are read. The current selection set is simply reset in selection set mode without any sort order, as opposed to single record mode.

The NOT operator reverses the results of the FIND expression it is attached to. A FIND NOT command processes itself as if it were a FIND command without a NOT keyword. When the selection set is determined, it is swapped for the not-selected records, including everything else within the dataset-subject to deleted records and consider sets, of course.

In commands with another negative operator, such as NBET, NCT, NE, and NMATCHES, using NOT with the corresponding positive operator generates the same result. For example the following commands are equivalent:

However, using NOT to negate GE, GT, LE, or LT provides slightly different results from using the opposite operator. The difference is whether to include Missing values. For example, the opposite operator of LT is GE. The command:

You can use a FIND NOT ALL command to create an empty selection set. Perhaps more commonly used, the FIND NOT LAST command returns all the records not in your current selection set.

The DUPLICATES keyword does some processing of the key data that no other FIND option can, but it still boils down to a straight-forward, full scan-across read of the lower-level keys comparable to the CT operator. To determine which records have duplicate values, the FIND DUPLICATES command scans the lower-level keys looking at each record list to determine how many records have that value. If there is:

The scan of the lower-level keys then continues adding qualified records to the selection set using a logical OR.

A consider set is the result of issuing a CONSIDER ON command. The consider set limits the scope of records for all queries from that point forward to the records currently in the selection set. The bitmap of selected records making up the consider set is saved and used for subsequent FIND commands. A CONSIDER OFF command restores the previous selection set.

Record Level Security operates the same way. Although various options control how often your security set is updated, to restrict access to outside records the basic approach is the same as a CONSIDER ON command.

Note: When a consider set or security set is in place, commands that read the key tables are not faster because of the smaller subset of records used. All commands that read the key tables read the entire key table. Which key table blocks are read is determined by:

The consider set is applied to filter out unwanted records only after the FIND processing is complete. The results of FIND processing and the consider set are joined using the logical AND to produce the finished selection set.

All FIND commands, even FIND ALL and FIND LAST commands, must weed out deleted records. System 1032 does not automatically remove deleted records from the dataset or from the key tables. This enables you to issue a DFIND command at a later time to retrieve deleted records.

The deleted record list is yet another bitmap which identifies deleted records. The last step before returning the selection set is to perform a logical IMPLIES between the deleted record list and the list of records selected from the FIND command.

Because the deleted record list is scanned for every FIND command, as well as for other commands such as MAP, an extremely large list of deleted records can impact FIND performance. We advise you to rebuild your datasets periodically to remove deleted records. Rebuilding reduces the deleted record list to a minimum and creates fresh key tables without the data pertaining to those deleted records. The resulting keys are smaller and, therefore, faster to access-especially for the ranged FIND commands.

Usually We want to select data on more than one condition. For example, a Multiple Condition that involves:

FIND processing breaks Multiple Condition commands into individual selection operations and executes them asynchronously. When the individual selections are complete, the results are logically combined based on the operators specified in the command. In Multiple Condition 1, the results of LAST_NAME EQ "Fuller" is merged with the results of LAST_NAME EQ "Vais" via logical OR. In Multiple Condition 2, the results of LAST_NAME EQ "Fuller" are merged with the results from the AGE GE 30 clause of the command.

Because the individual FIND conditions are executed asynchronously, if your key tables are in separate DMK files and split across disks, you may be able to decrease the amount of elapsed time it takes for a FIND command, such as the previous Multiple Condition 2, to complete. If you are not competing with yourself for disk I/O, disk reads may complete faster when reading different disks at the same time. The amount of CPU processing and the number of I/Os done will not significantly change, but the elapsed time may be reduced.

It is more efficient to issue a single FIND command with multiple conditions rather than a series of individual FIND LAST commands. Each FIND command has the overhead of stripping out the deleted records and respecting the consider or security set, if one exists. So, for a series of n FIND LAST commands, add the overhead of n-1 additional bitmap merges just to deal with deleted records. Although merges are very fast, large numbers of them certainly add overhead, which you can avoid by combining the conditions in a single command.

Key tables are initially built with some empty space in them to allow for expansion as updates occur. Filling a key block requires the block to be split, which is a relatively slow process. The amount of null space is determined when you create the dataset by specifying the NULL option in the KEY_DEFAULTS section of the DMD.

The FIND command uses the record lists in the attribute key tables in a variety of ways to generate the desired selection set. In general, the mechanism of this manipulation is a logical operation, such as AND, NOT, or OR, upon the individual attribute values record list to form conglomerate record lists to reflect the directives of the FIND command.

Certain low-level FIND options such as $ID, ALL, LAST, and NOT have no need of key tables; instead they process directly upon the entire dataset and/or current selection set.

Since the FIND command automatically incorporates the deleted record list as part of its processing, We recommend that you combine all known conditions into a single command for more efficient processing. You can achieve additional efficiency by manipulating the null space within a key table block, as well as moving the key tables outside of the dataset file.


Home DownloadsownFeedback

500 Old Connecticut Path, Framingham, MA 01701
Contact CCA Webmaster
Copyright 2005