Go Back   MarcomCentral (PTI) and FusionPro User Communities > Software-Related Talk > FusionPro® VDP Creator

Notices

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old January 29th, 2020, 05:03 PM
cdaters cdaters is offline
Junior Community Member
 
Join Date: Aug 2019
Posts: 10
Default Reading in external text files...

I have a question.

In some rules that I have, I am reading in a tab-delimited external data file containing some 60K+ records in it with the following code:

Code:
var transactionList = new ExternalDataFileEx("/jobs/job1/data_files/transaction_history_data.txt", "\t");
var recipient_id = Field("Recipient ID");

for (var i = 0; i <= transactionList.recordCount; i++) {

  var idVariable = transactionList.GetFieldValue(i, "Recipient ID");
  var dateVariable = transactionList.GetFieldValue(i, "Date");
  var paymentVariable = transactionList.GetFieldValue(i, "Payment Amount");

  if (recipient_id == idVariable) {
    matches.push(dateVariable + paymentVariable);
  }

}

return matches.join("<br>\n");
Does it make since to do it this way, maybe load it into JavaScript Globals so that I can use it across multiple rules?

Or would it make more since to load it in as a text file resource and then reference it in my rule like:

Code:
var transactionList = Resource("transactionHistoryFile");
I am then running this through an Array matching up relative data between each database (my main input file to external data file) using a "key" column in each and returning a list that shows a sort of "transaction history" for each record in my input file.

My main input file is my mail-list/recipient file containing 9K+ records, or recipients. My external data file is a file of about 60K+ records of transaction data, We are generating letter/statements for each recipient.

As you might imagine, it is taking a while to compose. I am chunking these out into 500 record PDFs for printing and I am trying to get an idea of a better way that might help speed composition up...?
Reply With Quote
  #2  
Old January 29th, 2020, 07:53 PM
Dan Korn's Avatar
Dan Korn Dan Korn is offline
FusionPro Senior Engineer / Forum Moderator
 
Join Date: Aug 2008
Location: Chicago, IL
Posts: 4,462
Default Re: Reading in external text files...

There are a couple of inefficiencies I can see in your code. First, you have to iterate all 60K records of the XDF for every 9K records of your main data. Multiplying 60,000 by 9000, you're talking over half a billion (with a "B") iterations overall! Also, on each iteration, you're fetching the date and payment amount even if you're not actually adding them to your result array. So you're calling transactionList.GetFieldValue over 1.5 billion times.

This should speed things up quite a bit:
Code:
var matches = [];
var transactionList = new ExternalDataFileEx("/jobs/job1/data_files/transaction_history_data.txt", "\t");
var RecipientCursor = transactionList.SortBy("Recipient ID");
var recs = RecipientCursor.FindRecords(Field("Recipient ID"));
for (var i in recs)
{
  var dateVariable = transactionList.GetFieldValue(recs[i], "Date");
  var paymentVariable = transactionList.GetFieldValue(recs[i], "Payment Amount");
  matches.push(dateVariable + paymentVariable);
}

return matches.join("<br>\n");
The cursor caches the numbers of the records which match the selected field to various values, so that you don't have to iterate all the records again each time, and you don't have to do anything at all with the records that don't match.

I don't think there will be any additional advantage to moving this to the JavaScript Globals or OnRecordStart, since external data files and cursors are automatically cached for the whole job.

Adding the XDF as a resource won't speed anything up either, but it will ensure that the XDF gets collected up with the job, which can be handy.

Beyond that, you could speed things up more if you could combine the two data files into a single multi-line record data file.
__________________
Dan Korn
FusionPro Developer / JavaScript Guru / Forum Moderator
PTI Marketing Technologies | Printable | MarcomCentral
LinkedIn

I am a not a Support engineer, and this forum is not a substitute for Support. My participation on this forum is primarily as a fellow user (and a forum moderator). I am happy to provide help and answers to questions when I can; however, there is no guarantee that I, or anyone else on this forum, will be able to answer all questions or fix any problems. If I ask for files to clarify an issue, I might not be able to look at them personally. I am not able to answer private messages, emails, or phone calls unless they go through proper Support channels. Please direct any sales or pricing questions to your salesperson or inquiries@marcom.com.

Complex template-building questions, as well as all installation and font questions or problems, should be directed to FusionProSupport@marcom.com. Paid consulting work may be required to fulfill your template-building needs.

This is a publicly viewable forum. Please DO NOT post fonts, or other proprietary content, to this forum. Also, please DO NOT post any "live" data with real names, addresses, or any other personal, private, or confidential data.

Please include the specific versions of FusionPro, Acrobat, and your operating system in any problem reports or help requests. I recommend putting this information in your forum signature. Please also check your composition log (.msg) file for relevant error or warning messages.

Please post questions specific to the MarcomCentral Enterprise and Web-to-Print applications in the MarcomCentral forum. Click here to request access. Or contact your Business Relationship Manager (BRM/CPM) for assistance.

Please direct any questions specific to EFI's Digital StoreFront (DSF) to EFI support.

How To Ask Questions The Smart Way

The correct spellings are JavaScript, FusionPro, and MarcomCentral (each with two capital letters and no spaces). Acceptable abbreviations are JS, FP, and MC (or MCC). There is no "S" at the end of "Expression" or "Printable"! The name of the product is FusionPro, not "Fusion". "Java" is not is not the same as JavaScript.

Check out the JavaScript Guide and JavaScript Reference! FusionPro 8.0 and newer use JavaScript 1.7. Older versions use JavaScript 1.5.

return "KbwbTdsjqu!spdlt\"".replace(/./g,function(w){return String.fromCharCode(w.charCodeAt()-1)});

Last edited by Dan Korn; January 29th, 2020 at 08:13 PM..
Reply With Quote
  #3  
Old January 30th, 2020, 04:54 PM
cdaters cdaters is offline
Junior Community Member
 
Join Date: Aug 2019
Posts: 10
Default Re: Reading in external text files...

Dan, thank you for your help. I have had to tweak a couple of things to work between the data that I have and the other (transaction total) rule that I have also incorporated with your help.

Here are some interesting observations.

On our now primary VDP person's Mac running macOS High Sierra (Mac Mini) with 8GB RAM, this actually took a little longer than with your code above (using my original rules) but let's just call it a wash and say that a 500 record output file took about an hour!

On my PC, using those same earlier rules, it took a little over 30 minutes for the same result.

Using the streamlined code you were able to assist me with, I now get a 500 record output file in 6 minutes! (mind blown!)

So this job actually consists of about 23,000 records (mail recipients) with a second transaction history database containing a little over 65,000 records of transaction data. Some recipients had only 1 record of transaction history while others maybe had 30 or 40 records of transaction history. I was brought into this project (Monday) and by yesterday, we were utilizing 1 PC and 2 Macs to power through this job (it had to drop in the mail and be date-stamped by today), Ugh!

Needless to say, I am trying to move our primary VDP person over to a PC to do this work moving forward instead!

Thank you for you help!
Reply With Quote
  #4  
Old January 31st, 2020, 08:29 AM
Dan Korn's Avatar
Dan Korn Dan Korn is offline
FusionPro Senior Engineer / Forum Moderator
 
Join Date: Aug 2008
Location: Chicago, IL
Posts: 4,462
Default Re: Reading in external text files...

Great! Just out of curiosity, when you say you're running it on Windows (PC), is that directly in Acrobat? Or is that through FP Producer or Server? Also, what version of FusionPro?
__________________
Dan Korn
FusionPro Developer / JavaScript Guru / Forum Moderator
PTI Marketing Technologies | Printable | MarcomCentral
LinkedIn

I am a not a Support engineer, and this forum is not a substitute for Support. My participation on this forum is primarily as a fellow user (and a forum moderator). I am happy to provide help and answers to questions when I can; however, there is no guarantee that I, or anyone else on this forum, will be able to answer all questions or fix any problems. If I ask for files to clarify an issue, I might not be able to look at them personally. I am not able to answer private messages, emails, or phone calls unless they go through proper Support channels. Please direct any sales or pricing questions to your salesperson or inquiries@marcom.com.

Complex template-building questions, as well as all installation and font questions or problems, should be directed to FusionProSupport@marcom.com. Paid consulting work may be required to fulfill your template-building needs.

This is a publicly viewable forum. Please DO NOT post fonts, or other proprietary content, to this forum. Also, please DO NOT post any "live" data with real names, addresses, or any other personal, private, or confidential data.

Please include the specific versions of FusionPro, Acrobat, and your operating system in any problem reports or help requests. I recommend putting this information in your forum signature. Please also check your composition log (.msg) file for relevant error or warning messages.

Please post questions specific to the MarcomCentral Enterprise and Web-to-Print applications in the MarcomCentral forum. Click here to request access. Or contact your Business Relationship Manager (BRM/CPM) for assistance.

Please direct any questions specific to EFI's Digital StoreFront (DSF) to EFI support.

How To Ask Questions The Smart Way

The correct spellings are JavaScript, FusionPro, and MarcomCentral (each with two capital letters and no spaces). Acceptable abbreviations are JS, FP, and MC (or MCC). There is no "S" at the end of "Expression" or "Printable"! The name of the product is FusionPro, not "Fusion". "Java" is not is not the same as JavaScript.

Check out the JavaScript Guide and JavaScript Reference! FusionPro 8.0 and newer use JavaScript 1.7. Older versions use JavaScript 1.5.

return "KbwbTdsjqu!spdlt\"".replace(/./g,function(w){return String.fromCharCode(w.charCodeAt()-1)});
Reply With Quote
  #5  
Old February 3rd, 2020, 10:38 AM
cdaters cdaters is offline
Junior Community Member
 
Join Date: Aug 2019
Posts: 10
Default Re: Reading in external text files...

I am composing directly in Acrobat using FP v10.1.11, no FusionPro Producer/Server.
Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 05:54 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
(c) 2011, PTI Marketing Technologies™, Inc.