Variabe page sizes for each record

[DSweet]

Have a bit of a dilemma...have a project that has the possibility of multiple page output for each record in the data (record 1 may have 2 pages, record 2 may have 8 pages, record 3 may have 7 pages, etc...). We want to impose this 2-up to save money, time and print streams produced; however since FusionPro can only impose the input data-stream and not the output print-stream I don't believe that it is possible. I have done some tests in the past with the imposer on similar situations and imposer would either 1-Crash and Flame, or 2-Add pages to the imposition of one side (the smaller output page side) to match up with the other larger output count side thus giving us a huge amount of blank pages that need to be weeded out by hand increasing our time and squandering any potential savings to begin with.

Using a bit of calculations we can tell how many pages are produced based on the information in each record; we currently split up all data files by output record count keeping the output counts together (all 3's together all 8's together, etc.). One data file containing as little as only 50 records may have up to a dozen or more data streams depending on the number of output pages that each individual record may produce. Then each "matched sub-set" data sream is fed through the template and sent to a printer ... that way we have matching output counts for the imposition to work with. Very lengthy, very tedious, the possibility of error to mix a batch set from one file with another increases with each new daily data file set we receive ... we receive between 50 to 60 files daily.

It's been a while since I've had the chance to look into some of the newer "chunking features" of FusionPro or maybe as part of the imposer itself but is there anything new that would help with this...or has anyone else been down this road and found "a path less travelled", or like Wiley Coyote just hit that same brick wall time and time again?

Any help and insight would be appreciated. Thank you.
.

[Dan Korn]

When you say "Variable page sizes," that makes me think of the physical sizes of the pages (such as 8.5 x 11 inches). And yes, all pages need to be the same size to impose them.

Originally Posted by DSweet

Have a bit of a dilemma...have a project that has the possibility of multiple page output for each record in the data (record 1 may have 2 pages, record 2 may have 8 pages, record 3 may have 7 pages, etc...).

But it seems you mean the number of pages in a record, not their sizes. Here again, the number of pages output per record needs to match the the number of pages per record specified in FP Imposer. (Although for a saddle-stitch, single-gate, or perfect bound booklet, the number of pages can be a multiple of the signature size. For other types, such as simplex and duplex, the number of pages has to be an exact match.

The reason for this should be obvious. If you are composing a two-sided postcard, you need to have something on both the front and the back, and if you leave off the back, FusionPro has to leave a blank space for that in the imposed output.

Originally Posted by DSweet

We want to impose this 2-up to save money, time and print streams produced; however since FusionPro can only impose the input data-stream and not the output print-stream I don't believe that it is possible.

I'm not sure what you mean here. FusionPro imposes the output, not the input. It doesn't impose your tab-delimited data file, it imposes your composed PDF/PPML/whatever output.

Originally Posted by DSweet

I have done some tests in the past with the imposer on similar situations and imposer would either 1-Crash and Flame, or 2-Add pages to the imposition of one side (the smaller output page side) to match up with the other larger output count side thus giving us a huge amount of blank pages that need to be weeded out by hand increasing our time and squandering any potential savings to begin with.

Yes, again, if you have a two-sided imposition, FusionPro needs to be able to output two sides for each record.

Originally Posted by DSweet

Using a bit of calculations we can tell how many pages are produced based on the information in each record; we currently split up all data files by output record count keeping the output counts together (all 3's together all 8's together, etc.). One data file containing as little as only 50 records may have up to a dozen or more data streams depending on the number of output pages that each individual record may produce. Then each "matched sub-set" data sream is fed through the template and sent to a printer ... that way we have matching output counts for the imposition to work with. Very lengthy, very tedious, the possibility of error to mix a batch set from one file with another increases with each new daily data file set we receive ... we receive between 50 to 60 files daily.

This is all very difficult to follow in the abstract. An example job would be worth many words.

Originally Posted by DSweet

It's been a while since I've had the chance to look into some of the newer "chunking features" of FusionPro or maybe as part of the imposer itself but is there anything new that would help with this...or has anyone else been down this road and found "a path less travelled", or like Wiley Coyote just hit that same brick wall time and time again?

You can certainly "chunk" your output into any arbitrary number of output files, each of which contains an arbitrary number of output records. But you can't split an individual record, if that's what you mean.

[esmith]

When I have dealt with this in the past, I usually output non-imposed data from FusionPro and then impose the resulting file using a separate PDF imposition plug-in (you could use FP for the second pass, but you wouldn't be composing anything; just imposing). The latter plug-in doesn't care about records; it's just imposing an n-page PDF.

[Dan Korn]

Originally Posted by esmith

When I have dealt with this in the past, I usually output non-imposed data from FusionPro and then impose the resulting file using a separate PDF imposition plug-in (you could use FP for the second pass, but you wouldn't be composing anything; just imposing). The latter plug-in doesn't care about records; it's just imposing an n-page PDF.

Can you elaborate on the use case for this?

[DSweet]

Sorry to be so long in getting back...

First off Dan you are correct and I wasn't quite clear enough...the length of the output varies from one record to another but all output pages are 8.5x11. Apologies.

The use for this project (for a product warranty company) is that each 2-page letter refers to any number of informational sheets that may accompany the letter depending which product(s) the end customer purchased a warranty for. These info-sheets are pre-built pdf files that simply grow from the end of each letter (using in-line pdf files through an overflow page) depending on the number of "packets" required. Each info-packet may contain anywhere from 3 to 12 pages and the end-user might get any number of these packets attached to their letter. Therefore record 1 might produce 14 pages, record 2 might produce 4 pages, record 3 might produce 20 pages, and so on through the file. Accordingly this output file can not be imposed by FusionPro Imposer as it currently runs.

I'm not sure what you mean here. FusionPro imposes the output, not the input. It doesn't impose your tab-delimited data file, it imposes your composed PDF/PPML/whatever output.

I don't believe you are quite right here. It might be that I just don't understand the process but when I use a stacked imposition and look at the resulting .msg file I see ...

.....
Preprocessing record #233, input record 233
Preprocessing record #234, input record 234
Preprocessing record #235, input record 235
Preflight ended 13:58:33 - 1476381513.
Composing record #1, input record 1
    Sheet #1, record #1
Composing record #2, input record 119
    Sheet #1, record #2
Composing record #3, input record 2
    Sheet #2, record #1
Composing record #4, input record 120
    Sheet #2, record #2
......

To me this appears that FusionPro has "pre-processed the data file" into memory, starts with record 1 and composes the resulting pdf output on the right side of the output sheet...then it jumps in the data file, reads in record 119, composes that data record in my template and places the output pdf file on the left side of the output sheet. Even though I have chosen "stacked" as my imposition scheme, FusionPro still can only perform a right-left imposition on the output sheet ... it just jumps through the data file picking out the correct records for that output placement. That's why all output information from a template MUST be the same length in order to impose properly.

If FusionPro were to impose the "output" for a stacked pdf file I would expect FusionPro to 1.. process a temporary pdf file in a single page format (in this case 8.5x11 pages) then 2.. count the number of pages within that document and finally 3.. step through that temporary file and place the proper pages on the final imposed sheet much in the same manner it currently steps through the data file records. That way even if the bottom of the right stack contains a page or two from say "record 13" and the top of the left side contains the remaining 8 pages from "record 13", when the imposed output sheets are "cut and stacked" the pages come out in their proper order.

I realize that this is may be a somewhat unique situation and is only useful for a stacked output, but I do think that others might have the same frustration as I do. My bosses continually scream to impose all jobs to save money and clicks on the machine, but I have to tell them that FusionPro can't, or we have to preprocess all the data files and split them up into several files based on the output pages and feed up to as many as 25 data files for a single job. Which again takes up time and wastes recources thus negating any potential savings from the lower number of clicks on the machines.

Sorry to be ranting here because I am a proponent of the FusionPro and would like to see it grow even more in areas that I personally feel like it is lacking. Charts are another big concern but that needs a whole new subject thread.
.

[Dan Korn]

Originally Posted by DSweet

The use for this project (for a product warranty company) is that each 2-page letter refers to any number of informational sheets that may accompany the letter depending which product(s) the end customer purchased a warranty for. These info-sheets are pre-built pdf files that simply grow from the end of each letter (using in-line pdf files through an overflow page) depending on the number of "packets" required. Each info-packet may contain anywhere from 3 to 12 pages and the end-user might get any number of these packets attached to their letter. Therefore record 1 might produce 14 pages, record 2 might produce 4 pages, record 3 might produce 20 pages, and so on through the file. Accordingly this output file can not be imposed by FusionPro Imposer as it currently runs.

True, you can't impose that output exactly as described. But I think there is a way to get the output you want. More on that in a bit...

Originally Posted by DSweet

I don't believe you are quite right here. It might be that I just don't understand the process but when I use a stacked imposition and look at the resulting .msg file I see ...

.....
Preprocessing record #233, input record 233
......
Composing record #4, input record 120
    Sheet #2, record #2
......

To me this appears that FusionPro has "pre-processed the data file" into memory, starts with record 1 and composes the resulting pdf output on the right side of the output sheet...then it jumps in the data file, reads in record 119, composes that data record in my template and places the output pdf file on the left side of the output sheet. Even though I have chosen "stacked" as my imposition scheme, FusionPro still can only perform a right-left imposition on the output sheet ... it just jumps through the data file picking out the correct records for that output placement. That's why all output information from a template MUST be the same length in order to impose properly.

Well, I suppose that whether you want to say that it's the input being imposed, rather than being composed to output and then imposed, that it's a matter of semantics.

It's true that for stacked imposition, in particular, FusionPro does basically compose the pages in the imposed sheet order, by first doing a preprocessing pass through the data to determine (a) how many records and pages, and therefore how many sheets, will be composed, and (b) the offset to each record in the input data file so that it can randomly access ("jump" to) records in the data as needed in the second (composition) pass. But technically, it still composes each record first, in the sense that it runs all the rules, typesets all the text, and saves off an "image" of each page in memory, then it takes those images and places them onto imposed sheets.

Also, it's not because of stacked imposition that all records have to have the same number of pages; this is true of non-stacked imposition as well. You still need a front and a back for each postcard in a non-stacked duplex imposition.

Originally Posted by DSweet

If FusionPro were to impose the "output" for a stacked pdf file I would expect FusionPro to 1.. process a temporary pdf file in a single page format (in this case 8.5x11 pages)

Perhaps, but PDF is not the only output format that FusionPro generates. It also needs to be able to do stacked imposition for output to PostScript, PPML, JLYT, AFP, etc.

To get probably way more technical than is necessary here, all of that composition, typesetting, and laying out pages onto imposed sheets happens before FusionPro even writes the output to a particular output format. It's all output format agnostic, if you will (other than some low-level stuff about embedding fonts and processing graphics). Everything is laid out, completely composed and imposed, in memory, for each output file (chunk), then that abstract in-memory representation of the output is sent to a particular output module (OM) to turn it into a PDF or PostScript or AFP or whatever kind of file.

Originally Posted by DSweet

then 2.. count the number of pages within that document and finally 3.. step through that temporary file and place the proper pages on the final imposed sheet much in the same manner it currently steps through the data file records. That way even if the bottom of the right stack contains a page or two from say "record 13" and the top of the left side contains the remaining 8 pages from "record 13", when the imposed output sheets are "cut and stacked" the pages come out in their proper order.

I hear what you're saying. The thing is, FusionPro wants to maintain the integrity of each output record. Consider that you could be chunking the output to multiple output files, where the chunk breaks occur at stack breaks, and in that case, the pages from your record 13 may end up in different output files. That's generally considered a bad thing. Though perhaps you don't care about that in this particular case, so you don't really need to output these distinct "records" in the first place. (We're getting to that, I promise.)

Originally Posted by DSweet

I realize that this is may be a somewhat unique situation and is only useful for a stacked output, but I do think that others might have the same frustration as I do.

It's possible. My impression is that the vast majority of people using imposition are either using fixed-length records (i.e. all records output the same number of pages), or are using impositions such as saddle-stitch where you can have variable numbers of pages, or are composing non-imposed output from FusionPro and doing imposition via third-party software or on their RIPs.

Originally Posted by DSweet

My bosses continually scream to impose all jobs to save money and clicks on the machine, but I have to tell them that FusionPro can't, or we have to preprocess all the data files and split them up into several files based on the output pages and feed up to as many as 25 data files for a single job. Which again takes up time and wastes recources thus negating any potential savings from the lower number of clicks on the machines.

What kind of machine are we talking about? Or more importantly, what RIP? Can't you send it non-imposed data and tell it to impose it?

Originally Posted by DSweet

Sorry to be ranting here because I am a proponent of the FusionPro and would like to see it grow even more in areas that I personally feel like it is lacking.

Sure, and I appreciate that. And your feedback is heard, even if we don't act on it right away. Frankly, we have indeed grown FusionPro into many areas. FusionPro 10 has new features to handle multi-line records, for statement jobs, new graphic clipping abilities, ad resizing, and lots more. It's just that the number of possible directions we can take are too numerous to count, and everyone seems to have a different idea or a new requirement.

Originally Posted by DSweet

Charts are another big concern but that needs a whole new subject thread.

That feedback would be valuable as well. This is another area we've been looking to enhance for some time. It's also another area where 20 different people would probably have 20 different ideas for exactly what to enhance. I can say that there may be more options than you realize. We have the ability to make custom rules to basically create any kind of chart you want, by effectively programmatically creating a graphic file, like we do for graphic QR barcodes. This, of course, is custom consulting work, but we've done some things that you wouldn't recognize as FusionPro output.

Okay, now I did promise I would tell you how you can get the output you want. That will be in my next reply.

[Dan Korn]

Okay, so to continue from my previous reply, here's how I think you can get the output you want.

In order to do what you describe here:

Originally Posted by DSweet

These info-sheets are pre-built pdf files that simply grow from the end of each letter (using in-line pdf files through an overflow page) depending on the number of "packets" required.

Presumably you have some JavaScript code that looks something like this:

var result = [];
var r = CreateResource("insert.pdf", "graphic");
var numpages = r.countPages;
for (var p = 1; p <= numpages; p++)
    result.push('<graphic file="' + r.name + '" pagenumber=' + p + '>');
return result.join("<p>");

You're basically adding all of those pages to a record, by outputting a series of inline graphics to a text flow with an overflow page.

So instead of doing that, why not just add a whole new record for each page, and impose what FusionPro considers to be a bunch of single-page records? If you really don't care about preserving the integrity of the output record boundaries, then why deal with output "records" in that sense at all?

It's really just a matter of setting up the job with a page with a graphic frame for the inserts (information sheets), instead of the overflow page, and refactoring the code above somewhat and putting it into the OnRecordStart rule. Something like this:

var repeat = FusionPro.Composition.repeatRecordNumber;
//var allPages = []; // defined in JavaScript Globals
var numStaticPages = 2;
if (repeat <= numStaticPages)
{
    FusionPro.Composition.SetBodyPageUsage("static page " + repeat, true);
    
    if (repeat == 1)
    {
        allPages = [];
        for (var i = 1; i <= numStaticPages; i++)
            allPages.push([]);

        var r = CreateResource("insert.pdf", "graphic");
        var numpages = r.countPages;
        for (var p = 1; p <= numpages; p++)
            allPages.push([r.name, p]);
        
        FusionPro.Composition.repeatRecordCount = allPages.length;
    }
}
else
{
    FusionPro.Composition.SetBodyPageUsage("insert page", true);
    var pagerec = allPages[repeat-1];
    var r = CreateResource(pagerec[0], "graphic");
    r.pagenumber = pagerec[1];
    var insertFrame = FindGraphicFrame("insert");
    insertFrame.SetGraphic(r);
}

This assumes you have two "static" body pages for your letter, named "static page 1" and "static page 2", and another body page named "insert page" with a graphic frame named "insert", and all the pages set to unused. It should output one record of output for each page. Then you can use any FPI file you want with the Pages Per Record set to 1.