Jump to content

Bloated file size when outputting in batches


Stack

Recommended Posts

I'm working on a calendar with about 400 records on it. I'm batching files in groups of 20 to keep the file size down. The first batch of 20 I did came out to be about 710MB in size; the second batch of 20 went way up to 4.1GB. This will undoubtedly crash our iGen so it's a no-go. I ran into this same issue last year during our yearly calendar project and I can't remember what the workaround was. Anybody know what would cause such a huge disparity in file size, and better yet what I can do to prevent it from happening?
Link to comment
Share on other sites

What output format are you composing to? If it's PDF, it could be the embedded fonts. Also, what are the versions of FusionPro and your operating system? And are you composing locally in FP Desktop (Acrobat), or via FP Producer or Server?

 

EDIT: I just noticed that you posted this in the FP Expression forum. Is this an Expression job that you're running through the Expression Producer or Producer CL? Or does the FusionPro VDP job use Expression images? Or did you mean to post this somewhere else?

Link to comment
Share on other sites

Whoops, forgot to include those details. Was in a hurry to post before I left for the day... I'm running everything locally. It's a FusionPro VDP job in Acrobat that uses 8 different Expression images. I'm outputting to PDF. Don't know if it's worth mentioning but the FP VDP template and the final files being output are not actually on my machine, but rather a local network where we host our files.

 

On both files I output so far -- the 700MB and the 4GB -- I actually did receive an error about a font not embedding. Weirdly though, it says "*** Bold" is not embedded so it's embedding the bold version of the font? Either way, I ran the Preflight tool in Acrobat afterwards and on both files, neither reported missing fonts.

 

I figured this was more a question specific to FP Expression since I only ever have it happen when using Expression images.

Link to comment
Share on other sites

I'm assuming that expression images will vary in size based on what's composed....

 

We'd had issues with higher volume PDFs print runs and had to use PPML for our iGen and although the file sizes aren't that drastic in size, the RIP was significantly faster. Printing is slower while RIP'ng.

Link to comment
Share on other sites

A few things: First, you probably want to use a VDP-optimized output format such as PPML, VDX, or PDF/VT instead of PDF. That said, it's difficult to optimize something like a personalized image, which is likely different for each record. Perhaps the size of the Expression images can be reduced? You may need to save lower-resolution versions of your FPX templates so that the generated images are smaller. If I could see the files, I might be able to make more specific suggestions.
Link to comment
Share on other sites

Thanks for the replies, guys.

 

A few things: First, you probably want to use a VDP-optimized output format such as PPML, VDX, or PDF/VT instead of PDF.

 

I tried PDF/VT but it made no difference. And unfortunately we're not really well versed in using PPML, VDX or basically anything other than PDF so we can't just switch it up at this time.

 

Perhaps the size of the Expression images can be reduced?

 

The FPX templates all use 300dpi RGB .jpg images, 9.25" x 9.75" for the background. Most use a mask file of the same dimension, in .png format. One template uses a fill path for the personalized part, the rest use simple masking text. The templates all range in size from 3.7 to 12.7MB. I'm looking now at all the unique .jpg files that have been output for each name when I try to compose my FP VDP template, and they are all consistent in size for each correlating FPX template, regardless if the final PDF was normal or bloated in size. For example, my FPX template for the month of August is 5.6MB; All 133 unique .jpgs that have been output so far for each name come in at 4.3MB. So basically, all the personalized .jpgs are consistent in size from template to template, so I doubt that it's an issue with one (or more) of the Expression images.

 

One peculiar thing that happened this morning is one of the batches that initially bloated this morning came out to the correct size when I selected "Use RGB Color Space" on the "Output" tab when I Composed. I tried this setting on another batch that had bloated earlier but did not have the same luck.

 

At this point I've tried closing all programs (except Acrobat, Suitcase and FusionPro, of course) while composing, composing directly to my desktop as opposed to the network and done a complete restart of my computer. Nothing has worked. It's frustrating because I can't pinpoint anything that might be causing the problem. It seems totally random, but yet consistent in that once it happens to a batch, it happens to all batches that follow. I was able to squeeze two batches out this morning (after my computer had set on overnight) and then that rogue RGB batch mentioned earlier, but everything since then keeps coming out bloated.

 

I'm really starting to wonder if the issue is with FP Expression at all, maybe it is just a FP VDP issue? Maybe there's a cache or something somewhere that should be trashed, something along those lines? I'm at a loss.

Link to comment
Share on other sites

The FPX templates all use 300dpi RGB .jpg images, 9.25" x 9.75" for the background. Most use a mask file of the same dimension, in .png format. One template uses a fill path for the personalized part, the rest use simple masking text. The templates all range in size from 3.7 to 12.7MB.

Yeah, those are some big graphics you're asking FusionPro VDP to process. Are the graphic frames in the FusionPro VDP template really 9.25" x 9.75"? Or are you having FusionPro squeeze those huge Expression graphics into smaller frames? If it's the latter, then you should redesign the Expression templates to generate smaller graphics. Or perhaps make some of the content a static picture to bring in behind whatever the variable part is that Expression is generating.

I'm really starting to wonder if the issue is with FP Expression at all, maybe it is just a FP VDP issue? Maybe there's a cache or something somewhere that should be trashed, something along those lines? I'm at a loss.

Well, yes, it is an issue with FusionPro VDP. Expression is doing what you're asking it to do. FP VDP is also trying to do what you're asking, but you're asking a lot.

 

Again, though, it's hard to troubleshoot this, let alone make suggestions, purely in the abstract like this. A peek at, say, a few pages of one of your VDP output files would be quite illuminating.

Link to comment
Share on other sites

Yeah, those are some big graphics you're asking FusionPro VDP to process. Are the graphic frames in the FusionPro VDP template really 9.25" x 9.75"? Or are you having FusionPro squeeze those huge Expression graphics into smaller frames? If it's the latter, then you should redesign the Expression templates to generate smaller graphics. Or perhaps make some of the content a static picture to bring in behind whatever the variable part is that Expression is generating.

 

Yes, that is the actual size of the graphic frame and template. Or actually it's 9.25" x 9.625", I forgot to deduct bleeds on the bind. I had to set the template up imposed because I couldn't figure out how to setup an imposition with FP Imposer that would let me bind head-to-head without rotating pages. I figured I'd just set the template up as spreads if I would have to rotate pages anyway.

 

This is a whole other issue but before I even went this route, inserting the FPX template into a graphic frame, I tried to use FPX Producer to create the personalized graphics in advance. But Producer wouldn't recognize my tab delimited text file, it'd see all 400+ names in one cell and then just fail.

 

Well, yes, it is an issue with FusionPro VDP. Expression is doing what you're asking it to do. FP VDP is also trying to do what you're asking, but you're asking a lot.

 

Again, though, it's hard to troubleshoot this, let alone make suggestions, purely in the abstract like this. A peek at, say, a few pages of one of your VDP output files would be quite illuminating.

 

I attached a .zip file with a sample spread from each file (bloated and non-bloated). I tried to include a few pages of each but the filesize way over exceeded what's allowed on this forum. Interestingly, just extracting and saving the page I can see the bloated file is significantly bigger. And the quality looks degraded, at least on my screen.

Archive.zip

Edited by Stack
Added missing attachment
Link to comment
Share on other sites

Yes, that is the actual size of the graphic frame and template. Or actually it's 9.25" x 9.625", I forgot to deduct bleeds on the bind. I had to set the template up imposed because I couldn't figure out how to setup an imposition with FP Imposer that would let me bind head-to-head without rotating pages. I figured I'd just set the template up as spreads if I would have to rotate pages anyway.

Well, the imposition issue is a separate question. But you can do head-to-head or "chop cut" imposition by rotating the rows and columns in FP Imposer.

This is a whole other issue but before I even went this route, inserting the FPX template into a graphic frame, I tried to use FPX Producer to create the personalized graphics in advance. But Producer wouldn't recognize my tab delimited text file, it'd see all 400+ names in one cell and then just fail.

The data file format to run a batch through Expression Producer is different than the data file format for a FusionPro VDP job. You need to read the Expression User Guide carefully.

 

Also, you say you wanted to "create the personalized graphics in advance," and (if you get the data file format right) you could do that, and it would speed up the main VDP composition because it wouldn't have to wait for the Expression images to render, but it won't have any effect on how those images are used in the VDP output. Or is the Expression template basically the entire job, and you're just using FP VDP to put the images into a PDF and impose them?

I attached a .zip file with a sample spread from each file (bloated and non-bloated). I tried to include a few pages of each but the filesize way over exceeded what's allowed on this forum. Interestingly, just extracting and saving the page I can see the bloated file is significantly bigger. And the quality looks degraded, at least on my screen.

Sorry, I don't see any attachments.

Link to comment
Share on other sites

Also, you say you wanted to "create the personalized graphics in advance," and (if you get the data file format right) you could do that, and it would speed up the main VDP composition because it wouldn't have to wait for the Expression images to render, but it won't have any effect on how those images are used in the VDP output. Or is the Expression template basically the entire job, and you're just using FP VDP to put the images into a PDF and impose them?

 

The entire job is a mix of FPX images and regular FP VDP elements.

 

I did read the manual pretty closely but I'll look over it again. I also watched the webinar video posted on Vimeo and followed the same steps presented there to setup my data file. Though rather than manually typing all 400+ names, I just extracted them from the data file I'm using for the FP VDP template and exported each personalized field as a tab delimited .txt file from Excel.

 

Strangely, if I only exported like 3 or 4 names and ran producer, it'd work fine. But when my list is 400+ names, it shows that it's trying to create only one personalized file and then it eventually errors out.

 

Sorry, I don't see any attachments.

 

Hmm, guess it didn't save. I edited my previous post and attached it there, it shows up now.

Link to comment
Share on other sites

Well after days of fighting with the template, I finally stumbled on something that allowed me to consistently output batches of files that were not bloated: From the Composition screen, click on the Graphics tab, then the Settings button next to "Personalized Images". Leave everything as is -- "Enable in Preview Mode" and "Preprocess in Batch Mode" both checked and hit OK.

 

That's it.

 

If I didn't do the steps above, I could see in the FusionPro Composition System window that it was dragging it's way thru all 20 records, and I'd end up with a bloated file. But when I did the above steps, I could see in the same window that that it flew thru all 20 records and output a perfectly fine PDF file in 10-15 seconds.

 

Kinda wish I hadn't struggled for days but oh well, I got what I needed eventually.

Link to comment
Share on other sites

Well after days of fighting with the template, I finally stumbled on something that allowed me to consistently output batches of files that were not bloated: From the Composition screen, click on the Graphics tab, then the Settings button next to "Personalized Images". Leave everything as is -- "Enable in Preview Mode" and "Preprocess in Batch Mode" both checked and hit OK.

 

That's it.

 

If I didn't do the steps above, I could see in the FusionPro Composition System window that it was dragging it's way thru all 20 records, and I'd end up with a bloated file. But when I did the above steps, I could see in the same window that that it flew thru all 20 records and output a perfectly fine PDF file in 10-15 seconds.

 

Kinda wish I hadn't struggled for days but oh well, I got what I needed eventually.

Okay, so a few thoughts on this.

 

First, turning off preprocessing will probably make the job run more slowly, because FusionPro VDP has to wait for each Expression image to be made in turn. With preprocessing, Expression makes the images in parallel to the VDP composition. However, generated graphics should be cached, so if you have already made all of the graphics for a particular data set, and then you run the job again, you should be able to turn preprocessing off, since the VDP job won't have to wait for the Expression images to be generated. (The Preview mode setting has no effect on non-Preview compositions.)

 

Second, it seems most likely that the reason some outputs are bigger than others is because some of the Expression graphics were created and cached at a larger size than others. This is likely because you ran some of them separately with the Expression Producer. I can't verify this without poking around on your machine, though.

 

Third, all of this can work a bit differently depending on your setup. In my very first post in this thread, I asked if you were composing locally in FP Desktop (Acrobat), or via FP Producer or Server, but you never answered that.

 

The bottom line is that the way that FusionPro VDP and FusionPro Expression work together is complicated, and depends on how you have set things up, and on how you're composing. So it's almost impossible to diagnose what's going on through a keyhole here on the forum, even with your files. I'm glad you got things working though.

Link to comment
Share on other sites

Okay, so a few thoughts on this.

 

First, turning off preprocessing will probably make the job run more slowly, because FusionPro VDP has to wait for each Expression image to be made in turn.

 

So do you have to click the Settings button to enable preprocessing? When I clicked Settings, it was already checked, I never turned the setting off or on. But the files would bloat if I didn't first click the Settings button.

Link to comment
Share on other sites

So do you have to click the Settings button to enable preprocessing? When I clicked Settings, it was already checked, I never turned the setting off or on. But the files would bloat if I didn't first click the Settings button.

Preprocessing is on by default, because it's usually more optimal. That's why I said that the job may run more slowly if you turn it off.

 

I have no idea why simply going into that dialog and clicking OK without changing anything would make a difference in the output. That seems like a coincidence. But it might be worthwhile to compare the DEF (data definition) files from the two composition runs.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...