Loading PC-Format Data into System 1032
By Tym Stegner

Data files created on non-VMS systems often share a particular characteristic: the file format is represented on a VMS system as a stream file. For example, Windows and UNIX systems usually treat their files as a stream of bytes, in contrast to the record-oriented nature of most files found on the VMS operating system. To load a data file created on a PC or from a UNIX operating system into a System 1032 dataset, you must convert the stream-format file into a fixed-format or variable-format file, which means a file with end-of-record marks.

Recently I received the following e-mail from a customer, which I have condensed.

I am trying to load some rather large text files into System 1032 without success.

One file has about 1.3 million records with a text file size of about 800,000 KB. When I try to run the EDT or EVE editor--just to save the text file with the DMI extension, not to perform edits--I get the following error message:

Insufficient virtual memory to allocate a new cache.
Fatal DECTPU internal error, please submit an SPR…
%TPU-F-ACCVIO, access violation, reason mask=00,…

Is there a way to read in only a part of the file at a time?

Diagnosing the Difficulty
System 1032 is not particular about the file extension of data files and it is not necessary to use an editor to rename a file. The error seen by the customer is the result of a resource problem using the editor, and is unrelated to System 1032 operation.

The customer’s predecessor had left procedural instructions that all text files extracted from their PC database must be processed by the TPU/EVE editor. Otherwise, the data would not load into System 1032 correctly. Attempts to load the data without this processing step resulted in the following type of error:

%S1032-E-LOAREADER, error reading field <name>
      Couldn't read record 1 in file.....
-RMS-W-RTB, nnn byte record too large for user's buffer

The messages suggest that the file’s format is a non-record format file. This may happen when the file is created using a PC application, or if the transfer process does not understand VMS file formats. A check of the data file by the customer using the DIRECTORY/FULL command confirmed this:

…
Record format: Stream, maximum 0 bytes, longest 0 bytes
Record format: Carriage return carriage control
…

Using the RECORD_STREAMS Option
Therefore, the problem became how to make System 1032 correctly read the data file into the required dataset. My first idea was to add the RECORD_STREAMS option to the record descriptor (RD) that would be used to load the data file. The RECORD_STREAMS option for an RD enables the RD to process data from non-record oriented files. The RD intended for this dataset had correctly loaded smaller data files from the same source in the past.

After adding the RECORD_STREAMS option to the RD, the dataset was re-created. A trial load of 10 records was attempted from the 1.3 million record input file using the MAXIMUM option of the LOAD command to limit the number of records read from the input file. Unfortunately, the same error (-RMS-W-RTB) was received.

Using VMS File Conversion
A VMS system supports three types of stream files: plain stream, Stream_LF, and Stream_CR. The latter two types have embedded characters used to denote an internal record, while the former is a true stream of bytes with no inherent record delineation. In plain stream files, null characters are often used to delineate data, or the program reading or writing the file has intrinsic knowledge of the data format.

A stream format file can have characters that precede the data stream that an RD cannot handle (ignore), which causes columns to misalign. I expected this was the case with this file. Therefore, we returned to the original solution: edit the input file to inspect the record format to possibly adjust the RD to match, while at the same time change the file format to a VMS common format.

We decided to use the VMS CONVERT command with a VMS File Definition Language (FDL) file to translate the stream file into a variable format file with end-of-record carriage-return control. The VMS FDL feature allows the creation of files with specific qualities, without using a higher-level programming language to define the file characteristics.

Creating an FDL File
Our first task was to create an FDL file of the desired output file type. This is done using the VMS ANALYZE/RMS_FILE command against an existing data file, specifying the /FDL=filename qualifier. The result is the following FDL file: