inefficent code? suggestions?

[Talco]

I have tried to search through the forums and take the suggestions others have offered and asked regarding this issue. But I'm still learning the coding end and I know enough to know I still don't know enough.

 

I just started running this new project and it builds tables from external data files. Problem is the size of the external files are very large and taxing the limits of FPDesktop. (we will be testing the direct and or server versions soon). The main data is around 10,000, but the external file can be in excess of 100,000 records. when I process the entire file at one time I compose 1,500 records in about 90 minutes.

 

right now we are breaking the data into smaller batches and putting the external data in the same order as the 'main data' file. having that external file small really helps the composing times. doing this brings them down to about 30-35 minutes per 1,500 records. BUT it is an extra step in massaging the data and is has caused us a couple issues down the line.

 

Breaking the data up might be the only way for us the realistically compose at this time, but if someone with a much better javascript/FP eye can take a look and see if I'm looping through the external data more than once (or any other suggestions that might streamline my code) I would appreciate it.

 

I currently have a switch set up to determine how each different line item is entered into the table. The code below just has a small sample, there are dozens of different code options.

 

Thanks for taking a look and any suggestions.

 

if(FusionPro.Composition.isPreview == true || FusionPro.inValidation == true)
{
   Rule("OnJobStart");
}


numRecsExtBTR = externalBTR.recordCount;



var BTRTable = '<table columns = 4 margins=left:13;right:13;top:28;bottom:28><column width=14000><column width=6100><column width=4900><column width=4900>';

recordWalker=1
externalBTR.FindRecord(recordWalker, Field("KEY"));

while (recordWalker <= numRecsExtBTR)
{

var TransID_BTR;
TransID_BTR = (externalBTR.GetFieldValue(recordWalker, 'KEY'));


  if (TransID_BTR == Field("KEY")) 
   {
switch ((externalBTR.GetFieldValue(recordWalker, 'RANK')))
{     
// Prior Bill Amount
case "1":
       BTRTable += '<row><cell><p br=false quad=L>' + (externalBTR.GetFieldValue(recordWalker, 'Trans_Descript'))
       BTRTable += '<cell>'
       BTRTable += '<cell><p br=false quad=R>' + "$" + (externalBTR.GetFieldValue(recordWalker, 'Amount'))
       BTRTable += '<cell>'
       break;

// Sewer Charge Adjustment
case "33": 
       val = ReplaceSubstring(externalBTR.GetFieldValue(recordWalker, 'Amount'), ",", "");
           if (val < 0)
           {
               negval = ReplaceSubstring(val, "-", "");
               BTRTable += '<row><cell><p br=false quad=L>' + (externalBTR.GetFieldValue(recordWalker, 'Trans_Descript'))
               BTRTable += '<cell>'
               BTRTable += '<cell><p br=false quad=R>' + "$" + FormatNumber("###,###.##", negval) + "CR"
               BTRTable += '<cell>'
           }   

         else

         {
               BTRTable += '<row><cell><p br=false quad=L>' + (externalBTR.GetFieldValue(recordWalker, 'Trans_Descript'))
               BTRTable += '<cell>'
               BTRTable += '<cell><p br=false quad=R>' + "$" + (externalBTR.GetFieldValue(recordWalker, 'Amount'))
               BTRTable += '<cell>'
           }
           break;

// Dozens of other switch/case options here....

// Do Not Print on bill
case "9999":
   Print("");
       break;

default:
Print("MISSING DESCRIPTIONS");
//close switch
}

   }
recordWalker++  


}


BTRTable += '<row><cell><p br=false quad=L>' + "Total Current Monthly Activity";
BTRTable += '<cell>';
BTRTable += '<cell>';
BTRTable += '<cell><p br=false quad=R>' + "$" + Field("CURRMONTHL");

// Total Amount Due
       val = ReplaceSubstring(Field("BILLAMT"), ",", "");

           if (val < 0)
           {
       negval = ReplaceSubstring(val, "-", "");
       BTRTable += '<row><cell><p br=false quad=L>' + "Total Amount Due" + " "+"  "+"  "+"       " + "(Credit)";
       BTRTable += '<cell>';
       BTRTable += '<cell>';
       BTRTable += '<cell><p br=false quad=R>' + "$"+FormatNumber("###,###.##", negval)+"CR";

           }   
         else
         {
       BTRTable += '<row><cell><p br=false quad=L>' + "Total Amount Due";
       BTRTable += '<cell>';
       BTRTable += '<cell>';
       BTRTable += '<cell><p br=false quad=R>' + "$"+FormatNumber("###,###.##", val);
         }


BTRTable += '</table>';


//Print(BTRTable);
return BTRTable;

[Alex Marshall]

You don't want to declare a new ExternalDataFileEx object every time you process a rule, because FusionPro has to open the file and read the file's contents into memory each time, which is an expensive process.

So it's not surprising that your composition is slowed down.

Instead, what you want to do is declare the ExternalDataFileEx object as a global, either in your JavaScript Globals or in your OnJobStart callback rule, like so:

MyFile = new ExternalDataFileEx("FileName.txt", ",").

And then reference the global object as needed in individual rules:

return MyFile.GetFieldValue(Field("Data Match"), "Word");

This way, you're only loading the file's contents into memory once. Once it's loaded, accessing that memory is a much less expensive operation.

[Talco]

I do have my OnJobStart callback rule set up to create the file externalBTR

externalBTR = new ExternalDataFileEx('/Production/BTR_1500.txt',',');

This is where I'm getting confused. I thought I was just loading the external file in once in my OnJobStart and then using that data in my code above. Where in my code am I declaring the new external file each time?

 

Thanks for your help Alex.

[Talco]

Not to push the issue but I just ran my first full test on this project using FPDirect. The composition times are down greatly from using Desktop but still almost 3 hours to compose the 8,000 records.

 

I know the slow down is due the accessing the external files which have approx 100,000 records (used to build tables in main file). Alex mentioned I might be loading the external file in every time that my rules runs. I'm not sure I am. I hope I am, then it could be sped up 'easily'.

 

As I have mentioned, I'm fairly new to javascripting and I'm still learning. But to my eye I am loading the external data in my OnJobStart rule and then I using the data in my rule. If anyone sees where I'm declaring the data each time the rule runs, please let me know and I can try and fix my speed issue.

 

Thanks again for any assistance.

 

Currenty using FPDirect v7.2P1k - demo