External file affect on memory

[DSweet]

When I envoke the commands

var myNewFile = new ExternalDataFileEx(FusionPro.Composition.inputFileName, ",");
recTotal = myNewFile.recordCount;

in an OnJobStart rule to count the number of records in the data file, I understand that the data file is fully read into the RAM memory of the computer first then the file size is determined by the command in the second line. However, does the ExternalDataFile command still use up RAM memory in the computer until the composition is over and then released? Is there another way to count the number of records without having to use up memory, or a way to release that memory after the recordCount is taken?

 

If the file is only a few thousand then the memory usage is not to taxing for most computers, but when the data file is a few hundred thousand or a few million this could cause serious memory issues.

.

[Dan Korn]

It is true that the entire file is read into the JavaScript object, in memory, and that it does hang around for the entire composition. You could delete the object when you're done with it, but because JavaScript uses deferred garbage collection internally, there's no guarantee of when the memory will actually be released.

 

Let's take a step back and see if we can figure out another way to do what you're trying to do. What is it that you're ultimately trying to accomplish with the number of records?

[DSweet]

We are in the middle of a job that entails processing several million pieces of mail per month over several different pre-printed stock shells. The problem is that we are NOT allowed to put any type of sequence numbering on them so it is quite scary having all those sheets of multi-up stocks printed without having some kind of check to keep them in their sorted order. I came up with a method of putting a sheet number on the shell-sheet outside of any of the pre-printed areas of the postcards so after they are cut down you don't see the sequence number. The only way I had of doing this was to put a text box off the edge of the piece and use a counting number that only printed (via a rule) for every 8th record that was processed (the stock is imposed 8-up).

 

This method worked fine when the entire sheet was full and my "sheet-number" would appear at the bottom right position (the 8th record) on the sheet. The only problem was that when there was not a full sheet filled (towards the end of the file) the 8th position might not be at the bottom right. However the "sheet-number" would still be placed below the 8th printed piece which would step it's way back through the sheets until the end of the file. This would now have an extra number printing on top of potentially important information of the postcard itself.

 

We normally wouldn't worry about the potential of at most seven cards out of 3 million being ruined in this fashion, but it would just be our luck that one of those cards would be a seed name that would find its way back to our customer and they would question it.

 

I found a way around this as well by trying to think like FP Imposer and how it would break up the data files into the imposed file and when it would start to not fill up an entire sheet. The only method that I found was using a set of rules in an OnJobStart rule to determine how many records the file was and doing an integer division by the imposition factor (in this case 8) and then stepping back by a remainder of that division to set the upper boundary of the "full sheet-limit". Then in my counter rule I would only place the incrementing sheet-counter on every 8th record only when that number was less than the sheet-limit as determined in the OJS rule.

 

But when you are counting 3 million records at a time (even the 250K that we broke it down to) that eats up quite a bit of memory and the whole composition processed is a bit slowed. I want to be able to open the file into a temporary location, count the records and then close the file to free up the memory for speed.

 

I suppose the best way to do this would be to be able to do some type of simple programming within FP Imposer itself to put some kind of sheet-counter number on it that way, but as for now you can't do any programming like that in Imposer. Imposer hasn't REALLY been changed since it came out around version 3.2 or 3.4.

.

[Dan Korn]

Hi David,

 

That's a fairly complex custom workflow scenario, and I'm not sure I can answer your questions about it completely, at least not in the context of this user-to-user forum (although it's possible that another user may have a similar workflow and would be able to chime in). I suggest contacting Support; they can route your request to the appropriate folks here who can help you, possibly negotiating an enhancement to the product and/or some custom consulting. Thanks!

[esmith]

ON the first read through, I wondered why you wouldn't just have a slug line on each record (rather than every 8th) and set it up as explained in mhilger's post here, but perhaps you do not have enough gutter space between imposed pieces to allow for a slug line?

 

What about placing the slug line off the top edge of the first item on the page (record 1 then every 8th record beginning with #9)? The first spot in the imposition should be used throughout the stack and the number would appear at the top portion of that position on the sheet.

[DSweet]

That is exactly what I do use Eric, only my "slug line" as you call it is a counter I created that increments by 1 for every 8 records that FP processes (a fully imposed sheet). My only problem occurs when that sheet is not fully imposed such as at the end of the file and the "8th record" that is processed is now somewhere in the middle of the sheet. The only place that I am allowed to put any text is at the bottom left edge of the full 12x18 pre-printed shell. That is where I put my counter value.

 

I suppose the only option that is available to me for the moment would be to append blank records to the end of the data file to force it to always come out to even sheets. I just seems a bit odd to me that since the imposer pre-counts the records anyway why some type of counting number could not be placed on the imposed sheet somewhere.

.

[aaronsupremeworks.com]

if all you want to do is number each sheet, why not use the page info checkbox under the imposition tab

[DSweet]

Thanks for the tip on that Aaron, and I have tried it with others like this. However we are not really able to control where that page info slug goes on the sheet. I've included two samples which illustrate this. The page info on one of them is even imprinted on top of the card itself! Another problem is that some printers can not even print the full area that close to an edge so you get a cut-off number or no number at all depending on the imaging area of that machine.

 

I know that I can shift or rotate the imposition page as needed, but we've had problems sending pdf files to our Indigo machines that are not in a specific orientation...for some reason our Indigo rips rotate the sheet 90 degrees and the information is printed all wrong. Operators are still trying to figure that one out.

 

Maybe if we could just somehow be given the ability to specify the placement of that page info on the sheet when that option is selected...

.

[Dan Korn]

Thanks for the tip on that Aaron, and I have tried it with others like this. However we are not really able to control where that page info slug goes on the sheet. I've included two samples which illustrate this. The page info on one of them is even imprinted on top of the card itself! Another problem is that some printers can not even print the full area that close to an edge so you get a cut-off number or no number at all depending on the imaging area of that machine.

 

I know that I can shift or rotate the imposition page as needed, but we've had problems sending pdf files to our Indigo machines that are not in a specific orientation...for some reason our Indigo rips rotate the sheet 90 degrees and the information is printed all wrong. Operators are still trying to figure that one out.

 

Maybe if we could just somehow be given the ability to specify the placement of that page info on the sheet when that option is selected...

.

FusionPro 8.2 will introduce a new feature where you can define a background page for imposition sheets, which will allow you to put information anywhere you want on the imposed output sheets. You will be able to place the current sheet number, the total number of sheets, and other similar information, either as text or as a barcode.