Jump to content

XDF - Out of Memory Error


bkurzbuch

Recommended Posts

Mac Sierra 10.12.6, Acrobat 11.0.23, FusionPro 9.3.36. 32 Gigs memory

Dan

I'm having trouble with a large XDF and error line 10: out of memory. The jobs loads a list of providers and them compares the mailing list to the XDF and returns the 3 closets office using Long & Lat. This file has worked fine in the past with the provider list being around 13,000 records. The new provider list from the customer is 194,092 and won't process. Customer has assured me that this list is correct. Since this is using Long. and Lat I'm not sure how and if the provider file can be split. File attached. Any help is greatly appreciated. Thank You

DD Prevent Services2 No Current Dentist Ltr_ccg.zip

Link to comment
Share on other sites

Mac Sierra 10.12.6, Acrobat 11.0.23, FusionPro 9.3.36. 32 Gigs memory

Thank you for that information. I can reproduce the problem on Windows as well, in FusionPro 10.1.

Dan

"Help me Obi-Wan Kenobi, you're my only hope."

 

While I do end up answering a lot of the questions here, this isn't really the "Ask Dan" forum. Like my signature notes, there's no guarantee that I, or anyone else, will respond. If you really need urgent help, you still need to go through Support.

 

That said...

I'm having trouble with a large XDF and error line 10: out of memory. The jobs loads a list of providers and them compares the mailing list to the XDF and returns the 3 closets office using Long & Lat. This file has worked fine in the past with the provider list being around 13,000 records. The new provider list from the customer is 194,092 and won't process. Customer has assured me that this list is correct. Since this is using Long. and Lat I'm not sure how and if the provider file can be split. File attached. Any help is greatly appreciated. Thank You

The problem isn't loading the XDF. That loads up just fine, into FusionPro's memory. The problem is building a JavaScript array with 194,092 entries.

 

I know this because, when I go to validate the rules, the OnJobStart rule, which actually loads the XDF, has no errors. The error is when I validate the "Rule_Closest Offices" rule, and after I close the error popup, the cursor is on line 10 of the rule, and the error message in the Building Blocks also denotes that the error occurred on line 10.

 

But you don't need to add every single item to an array, and then sort that giant array with every item in it. Since you only need the three closest, you can iterate the XDF, and if the current entry you're on isn't any closer than the three you already have in your array, you can just throw it away. The code is only very slightly more complicated:

var XDF = new ExternalDataFileEx("PROVIDER DATA FINAL.csv", ",");

if (!XDF.valid)
   ReportWarning("Warning: The XDF file could not be found/read. Results may be unexpected.");

//return XDF.recordCount;

var arrayDistances = [];

for (var row = 1; row <= XDF.recordCount; row++)
{
   var thisRowDistance = CalculateDistance(XDF.GetFieldValue(row, "LAT"),XDF.GetFieldValue(row, "LONG"));
   arrayDistances.push({row:row,city:XDF.GetFieldValue(row, "City"),distance:thisRowDistance})       
   arrayDistances.sort(function(a, b) {return parseFloat(a.distance) - parseFloat(b.distance);});
   arrayDistances.splice(3); // max. number to return
}

//return arrayDistances.map(function(item) { return item.row + "\t" + item.distance; }).join('\n');

var ClosestOffices = [];

for (var x in arrayDistances)
{
   var row = arrayDistances[x].row;
   var thisOffice = ["Dr. " + ToUpper(XDF.GetFieldValue(row, "NAME"))];
   thisOffice.push(XDF.GetFieldValue(row, "TEL"));
   //thisOffice.unshift(arrayDistances[x].row); // For debugging, to see what row we found.
   ClosestOffices.push(thisOffice.join('\t'));
}

return ClosestOffices.join('\n');


function CalculateDistance(Doctors_LAT,Doctors_LONG)
{
   var latitude1 = Field("LAT");
   var longitude1 = Field("LONG");
   var latitude2 = (Doctors_LAT);
   var longitude2 = (Doctors_LONG);
   var Radius = 3959; //in miles
   var dLat = (latitude2-latitude1)*Math.PI/180;  
   var dLon = (longitude2-longitude1)*Math.PI/180;   
   var a = Math.sin(dLat/2) * Math.sin(dLat/2) +  
           Math.cos(latitude1*Math.PI/180) * Math.cos(latitude2*Math.PI/180) *   
           Math.sin(dLon/2) * Math.sin(dLon/2);   
   var c = 2 * Math.asin(Math.sqrt(a));   
   var distance = Radius * c;

   return distance;
}

Basically, after we add an item to the array, we do the sorting, then we remove any items that aren't in the first three. So the array never has more than four items in it at any given time, and you don't run out of memory.

 

(Though it's still kind of slow to iterate hundreds of thousands of XDF records, but there's not much to be done about that. You're running into a practical limitation of what can be done with relational databases in JavaScript, and in Creator. A more robust solution would probably do these calculations "server side" with SQL and some other programming language, then generate the data for a FusionPro Server composition accordingly, as part of a larger application.)

 

I did verify that I get the same results with both your code and my updated code, with a 13,000 record XDF. But I do get different results with the 194,092-record XDF, so you'll want to double-check the results.

 

Note that I've made some other "best practice" changes to the rule as well: I've moved the opening of the XDF directly into the rule, and removed the OnJobStart rule, as XDFs are cached in FusionPro 9.3 and later, so there's no penalty for creating the ExternalDataFileEx object for every record. Also, I've changed the tags to newline and tab literals, so this will work with the "Treat returned strings as tagged text" box UNchecked, even if there are "special" characters in the XDF data. And I'm using Array.join in place of some of the manual concatenation of strings with tab and newline delimiters.

Link to comment
Share on other sites

Dan

 

Thank You too much. Ya, I know its not the "Ask Dan Forum". With a problem this complex I figured you where the only one who could help. Thank You Obi-Wan. My next stop was support. Thanks again for taking the time to look at this and the explanation. Seems the more I learn about javascript the more i realize how much I don't know. You are the man.

Link to comment
Share on other sites

Thank You too much. Ya, I know its not the "Ask Dan Forum". With a problem this complex I figured you where the only one who could help.

No problem. Though I'll bet one of our forum superstars, like the Classic Graphics guys, would have been able to figure it out too. It's just about who has time.

Works great Dan. One quick question. The old rules returned a tab in front of "Dr." and another tab in front of "Tel". I've tried coding them back in but I can't seem to get the syntax correct. Any advise? Thanks Again.

Just add another item at the start of the array, either by changing this line:

    var thisOffice = ['', "Dr. " + ToUpper(XDF.GetFieldValue(row, "NAME"))];

Or by adding this line:

    thisOffice.unshift('');

Or just add the tab here:

    ClosestOffices.push('\t' + thisOffice.join('\t'));

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...