Answer to question 21

This question was meant primarily to let your brains work a bit.

It is important to understand that reading file records one by one is never good for performance.  So, we have to find other ways.

If we have to handle very large files, then we may have problems if we want to load all records into storage, although storage is not so much a constraint these days.  Extremely large files are still exceptions however, so, do not get the wrong impression that reading large files is always a problem that has to be solved differently.  Extremely large files may be log and dump files, images and sound recordings.  In many cases, these types of files will be processed by specialized programs that should be able to deal with their sizes.

So, the easiest solution is to read the file completely, but to test the return code.  If that is indicating a storage full condition, then the user should be informed so that he/she can define a larger virtual machine size.

Let's summarize:

• For files of normal sizes, the best solution is to use Data In Memory techniques, or CMS Pipelines.
• For files you expect to become very large, you have to use a method by which you don't hit the storage full condition.  If you can, use CMS Pipelines, as this normally only needs storage for a few records at a time.
• Never process records one by one as you multiply the overhead of CMS.

Return to the lesson by hitting the backward navigation button of your browser.