Go Back   MarcomCentral (PTI) and FusionPro User Communities > Software-Related Talk > The JavaScript Library

Notices

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old December 14th, 2017, 02:54 PM
ScottHillock ScottHillock is offline
Regular Contributor
 
Join Date: Mar 2010
Posts: 94
Default 2d Datamatrix performance

If you've used the FusionPro datamatrix encoder you know it's painfully slow.

Our usual solution for increasing performance on simple jobs is to have a 100,000 page pdf with a datamatrix barcode on each page, counting from 1-100,000. Then we use a variable image to grab the page number we need that matches the number we need. So pull in page 27, if you need a datamatrix of "27".

We've come into a new project that though requires a more complex datamatrix code, so with a bunch of searching and testing I landed on using an open source encoder that would deliver the barcodes in a get request to an http server.

Here's the code that ends up being used in FusionPro. You could also CreateResource on it, but for us this is the easiest drop in replacement.

Code:
var VariableText = Field("FIELD");
var dataMatrixApi = NormalizeEntities("http://127.0.0.1:3030/?text=" + VariableText);
Pic = '<graphic file="' + dataMatrixApi + '">';
return Pic;
Simply replace the red text with your requirements and ip address.

Attached is the Mac command line application that I use, which will automatically listen on port 3030, and return a png of the barcode. It's a golang application so if you're on windows you can compile it yourself. I have mine set to autorun using "LaunchControl.app". You can also use Windows Task Scheduler to perform the same thing to auto launch at login/startup.


Here's the code to compile yourself (you'll need to download golang to do so):
Code:
package main

import (
	"bytes"
	"strconv"
	"net/http"
	"image/png"
	"github.com/boombuler/barcode/datamatrix"
)

func handler(w http.ResponseWriter, r *http.Request) {
	datamatrixCode, _ := datamatrix.Encode(r.URL.Query().Get("text"))
	buffer := new(bytes.Buffer)
	png.Encode(buffer, datamatrixCode)
	w.Header().Set("Content-Type", "image/jpeg")
	w.Header().Set("Content-Length", strconv.Itoa(len(buffer.Bytes())))
	w.Write(buffer.Bytes())
}

func main() {
    http.HandleFunc("/", handler)
    http.ListenAndServe(":3030", nil)
}

Moving from the FusionPro encoder to this solution we saw an increase of 15x to our composition. This is on FusionPro 9.3.36. Not sure if 10 has the native JavaScript encoder from IDAutomation that would have even faster performance. But, this solution puts us super close to the speed we would get if we upgraded to Producer.
Attached Files
File Type: zip golang-barcode.zip (2.19 MB, 5 views)
Reply With Quote
  #2  
Old December 14th, 2017, 05:06 PM
Dan Korn's Avatar
Dan Korn Dan Korn is offline
FusionPro Senior Engineer / Forum Moderator
 
Join Date: Aug 2008
Location: Chicago, IL
Posts: 4,592
Default Re: 2d Datamatrix performance

That's a very clever solution, and a welcome contribution here. Thanks!

I feel compelled to offer a couple of caveats, though. First, I would carefully check the licensing of any third-party code before utilizing it in production, especially for printed output. Second, MarcomCentral/PTI cannot vouch for the accuracy or reliability of using such third-party code, nor can we assume any responsibility for problems that may arise from such usage.

Also, I have to quibble slightly with this statement:
Quote:
Originally Posted by ScottHillock View Post
If you've used the FusionPro datamatrix encoder you know it's painfully slow.
That statement is not universally true. It may be true on some versions of Mac OS X, but not necessarily on all of them, and certainly not on Windows. On Mac, the Data Matrix barcode encoder (along with the PDF 417 barcode encoder) uses Java (not JavaScript), the performance of which isn't great on some versions of OS X. On Windows, it calls a native DLL, which takes no noticeable time at all.
Quote:
Originally Posted by ScottHillock View Post
Moving from the FusionPro encoder to this solution we saw an increase of 15x to our composition. This is on FusionPro 9.3.36.
And what version of OS X?
Quote:
Originally Posted by ScottHillock View Post
Not sure if 10 has the native JavaScript encoder from IDAutomation that would have even faster performance.
No, there has been no change to the Data Matrix encoder from FusionPro 9 to 10. But the new JavaScript encoder is something we'll look into. Thanks for the heads up.
Quote:
Originally Posted by ScottHillock View Post
But, this solution puts us super close to the speed we would get if we upgraded to Producer.
That's fine. However, I would caution others to not necessarily expect similar results. Obviously the total impact on composition speed depends on many factors, not just the Data Matrix barcode generation. And, as noted, the barcode generation speed depends on the operating system.

Frankly, the biggest advantage of creating barcodes this way is that they are images, not text, so they can be more easily scaled. We added the ability to create QR barcodes as graphics a while back. Perhaps it's time to do that for Data Matrix as well.
__________________
Dan Korn
FusionPro Developer / JavaScript Guru / Forum Moderator
PTI Marketing Technologies | Printable | MarcomCentral
LinkedIn

I am a not a Support engineer, and this forum is not a substitute for Support. My participation on this forum is primarily as a fellow user (and a forum moderator). I am happy to provide help and answers to questions when I can; however, there is no guarantee that I, or anyone else on this forum, will be able to answer all questions or fix any problems. If I ask for files to clarify an issue, I might not be able to look at them personally. I am not able to answer private messages, emails, or phone calls unless they go through proper Support channels. Please direct any sales or pricing questions to your salesperson or inquiries@marcom.com.

Complex template-building questions, as well as all installation and font questions or problems, should be directed to FusionProSupport@marcom.com. Paid consulting work may be required to fulfill your template-building needs.

This is a publicly viewable forum. Please DO NOT post fonts, or other proprietary content, to this forum. Also, please DO NOT post any "live" data with real names, addresses, or any other personal, private, or confidential data.

Please include the specific versions of FusionPro, Acrobat, and your operating system in any problem reports or help requests. I recommend putting this information in your forum signature. Please also check your composition log (.msg) file for relevant error or warning messages.

Please post questions specific to the MarcomCentral Enterprise and Web-to-Print applications in the MarcomCentral forum. Click here to request access. Or contact your Business Relationship Manager (BRM/CPM) for assistance.

Please direct any questions specific to EFI's Digital StoreFront (DSF) to EFI support.

How To Ask Questions The Smart Way

The correct spellings are JavaScript, FusionPro, and MarcomCentral (each with two capital letters and no spaces). Acceptable abbreviations are JS, FP, and MC (or MCC). There is no "S" at the end of "Expression" or "Printable"! The name of the product is FusionPro, not "Fusion". "Java" is not is not the same as JavaScript.

Check out the JavaScript Guide and JavaScript Reference! FusionPro 8.0 and newer use JavaScript 1.7. Older versions use JavaScript 1.5.

return "KbwbTdsjqu!spdlt\"".replace(/./g,function(w){return String.fromCharCode(w.charCodeAt()-1)});
Reply With Quote
  #3  
Old December 15th, 2017, 09:15 AM
ScottHillock ScottHillock is offline
Regular Contributor
 
Join Date: Mar 2010
Posts: 94
Default Re: 2d Datamatrix performance

Dan, the source code of the library on this is MIT licensed. Which is pretty permissive. But, I agree others should always double check licensing and any possible copyright issues with code pushed out to the public domain.

We've had the same performance from the FusionPro datamatrix encoder on macOS 10.9.x, 10.11.6, 10.13.1, and on Windows 10. I had PTI support run the same tests on Producer and Composer on Windows, and Composer on Windows had the same performance as I've experienced across all platforms.

I agree many things effect composition speed. On this particular project that had multiple variable images, copyfitting, page selections, split outputs, and for 1.2 million records the composition time was reduced from 65 hours to 7 hours.
Reply With Quote
  #4  
Old December 15th, 2017, 11:58 AM
Dan Korn's Avatar
Dan Korn Dan Korn is offline
FusionPro Senior Engineer / Forum Moderator
 
Join Date: Aug 2008
Location: Chicago, IL
Posts: 4,592
Default Re: 2d Datamatrix performance

Quote:
Originally Posted by ScottHillock View Post
We've had the same performance from the FusionPro datamatrix encoder on macOS 10.9.x, 10.11.6, 10.13.1, and on Windows 10. I had PTI support run the same tests on Producer and Composer on Windows, and Composer on Windows had the same performance as I've experienced across all platforms.
First of all, what is "Composer"? Do you mean FusionPro VDP Creator (aka Desktop)?

Anyway, I haven't seen your job, nor what exactly you're encoding into these barcodes. For me, a simple Data Matrix barcode takes almost no time at all to create on Windows 10. I added a Data Matrix barcode to the Cell Phone tutorial job, and it added one second for 100 records compared to composing without it. So that's 1/100th of a second to make each barcode. I have no idea why your results are different.

Also, did all of these tests you mention isolate the barcode generation, or were they jobs doing other typesetting? It's possible that the composition of your job is generally faster on Mac in FusionPro 9.3, even though the barcode generation may be slower there. I can also tell you that Producer composes much faster in FusionPro 10 than in FusionPro 9.

So again, while you may have run into a particular edge case where barcode generation is slow, those results are not typical. Anyone else is welcome to chime in with their results, of course.
Quote:
Originally Posted by ScottHillock View Post
I agree many things effect composition speed. On this particular project that had multiple variable images, copyfitting, page selections, split outputs, and for 1.2 million records the composition time was reduced from 65 hours to 7 hours.
In Producer? Where the only change was how the barcode is generated?
__________________
Dan Korn
FusionPro Developer / JavaScript Guru / Forum Moderator
PTI Marketing Technologies | Printable | MarcomCentral
LinkedIn

I am a not a Support engineer, and this forum is not a substitute for Support. My participation on this forum is primarily as a fellow user (and a forum moderator). I am happy to provide help and answers to questions when I can; however, there is no guarantee that I, or anyone else on this forum, will be able to answer all questions or fix any problems. If I ask for files to clarify an issue, I might not be able to look at them personally. I am not able to answer private messages, emails, or phone calls unless they go through proper Support channels. Please direct any sales or pricing questions to your salesperson or inquiries@marcom.com.

Complex template-building questions, as well as all installation and font questions or problems, should be directed to FusionProSupport@marcom.com. Paid consulting work may be required to fulfill your template-building needs.

This is a publicly viewable forum. Please DO NOT post fonts, or other proprietary content, to this forum. Also, please DO NOT post any "live" data with real names, addresses, or any other personal, private, or confidential data.

Please include the specific versions of FusionPro, Acrobat, and your operating system in any problem reports or help requests. I recommend putting this information in your forum signature. Please also check your composition log (.msg) file for relevant error or warning messages.

Please post questions specific to the MarcomCentral Enterprise and Web-to-Print applications in the MarcomCentral forum. Click here to request access. Or contact your Business Relationship Manager (BRM/CPM) for assistance.

Please direct any questions specific to EFI's Digital StoreFront (DSF) to EFI support.

How To Ask Questions The Smart Way

The correct spellings are JavaScript, FusionPro, and MarcomCentral (each with two capital letters and no spaces). Acceptable abbreviations are JS, FP, and MC (or MCC). There is no "S" at the end of "Expression" or "Printable"! The name of the product is FusionPro, not "Fusion". "Java" is not is not the same as JavaScript.

Check out the JavaScript Guide and JavaScript Reference! FusionPro 8.0 and newer use JavaScript 1.7. Older versions use JavaScript 1.5.

return "KbwbTdsjqu!spdlt\"".replace(/./g,function(w){return String.fromCharCode(w.charCodeAt()-1)});
Reply With Quote
  #5  
Old December 18th, 2017, 09:39 AM
ScottHillock ScottHillock is offline
Regular Contributor
 
Join Date: Mar 2010
Posts: 94
Default Re: 2d Datamatrix performance

Yes, sorry, by Composer I meant Creator. And my long 65 hour composition that reduced down to 7 hours was in Creator.

Attached is a simple job that only creates the datamatrix barcode.

My results, all using FusionPro Creator 9.3.36, from local ssd hard disk, all 16GB ram:
macOS 10.11.6 (iMac Late 13, 3.5GHz i7)
- 745 seconds

macOS 10.13.1 (imac Late 13, 2.9GHz i5)
- 845 seconds

And PTI support's Windows FusionPro Creator result:
- 635 seconds


Using the datamatrix-as-image solution with FusionPro Creator, my time went down to 38 seconds.
Attached Files
File Type: zip Archive.zip (45.3 KB, 4 views)
Reply With Quote
  #6  
Old December 18th, 2017, 11:50 AM
Dan Korn's Avatar
Dan Korn Dan Korn is offline
FusionPro Senior Engineer / Forum Moderator
 
Join Date: Aug 2008
Location: Chicago, IL
Posts: 4,592
Default Re: 2d Datamatrix performance

On my computer running Windows 10 and FusionPro VDP Creator 10.0.26, your 5000-record job composed in 10 minutes and 30 seconds. I then removed the DataMatrix barcode rule, and it composed in 10 minutes and 26 seconds. That supports my contention that the time to generate the barcodes on Windows is negligible. It's also very close to the time that Support reported, although you seem to have jumped to the conclusion that the barcode generation was a significant part of that time, which it clearly wasn't.
__________________
Dan Korn
FusionPro Developer / JavaScript Guru / Forum Moderator
PTI Marketing Technologies | Printable | MarcomCentral
LinkedIn

I am a not a Support engineer, and this forum is not a substitute for Support. My participation on this forum is primarily as a fellow user (and a forum moderator). I am happy to provide help and answers to questions when I can; however, there is no guarantee that I, or anyone else on this forum, will be able to answer all questions or fix any problems. If I ask for files to clarify an issue, I might not be able to look at them personally. I am not able to answer private messages, emails, or phone calls unless they go through proper Support channels. Please direct any sales or pricing questions to your salesperson or inquiries@marcom.com.

Complex template-building questions, as well as all installation and font questions or problems, should be directed to FusionProSupport@marcom.com. Paid consulting work may be required to fulfill your template-building needs.

This is a publicly viewable forum. Please DO NOT post fonts, or other proprietary content, to this forum. Also, please DO NOT post any "live" data with real names, addresses, or any other personal, private, or confidential data.

Please include the specific versions of FusionPro, Acrobat, and your operating system in any problem reports or help requests. I recommend putting this information in your forum signature. Please also check your composition log (.msg) file for relevant error or warning messages.

Please post questions specific to the MarcomCentral Enterprise and Web-to-Print applications in the MarcomCentral forum. Click here to request access. Or contact your Business Relationship Manager (BRM/CPM) for assistance.

Please direct any questions specific to EFI's Digital StoreFront (DSF) to EFI support.

How To Ask Questions The Smart Way

The correct spellings are JavaScript, FusionPro, and MarcomCentral (each with two capital letters and no spaces). Acceptable abbreviations are JS, FP, and MC (or MCC). There is no "S" at the end of "Expression" or "Printable"! The name of the product is FusionPro, not "Fusion". "Java" is not is not the same as JavaScript.

Check out the JavaScript Guide and JavaScript Reference! FusionPro 8.0 and newer use JavaScript 1.7. Older versions use JavaScript 1.5.

return "KbwbTdsjqu!spdlt\"".replace(/./g,function(w){return String.fromCharCode(w.charCodeAt()-1)});

Last edited by Dan Korn; December 18th, 2017 at 12:00 PM..
Reply With Quote
  #7  
Old December 18th, 2017, 12:05 PM
ScottHillock ScottHillock is offline
Regular Contributor
 
Join Date: Mar 2010
Posts: 94
Default Re: 2d Datamatrix performance

Weird, if I change the datamatrix code to just return ""; I get a speed of 34 seconds... Maybe you didn't edit the code correctly, or there's something wrong hardware wise with your workstation.

Last edited by ScottHillock; December 18th, 2017 at 12:23 PM..
Reply With Quote
  #8  
Old December 18th, 2017, 02:57 PM
Dan Korn's Avatar
Dan Korn Dan Korn is offline
FusionPro Senior Engineer / Forum Moderator
 
Join Date: Aug 2008
Location: Chicago, IL
Posts: 4,592
Default Re: 2d Datamatrix performance

Quote:
Originally Posted by ScottHillock View Post
Weird, if I change the datamatrix code to just return ""; I get a speed of 34 seconds... Maybe you didn't edit the code correctly, or there's something wrong hardware wise with your workstation.
On Mac? I was talking about Windows.

Yes, Creator composition is faster overall on Mac than on Windows, without any 2D barcodes. That's not in dispute. It's also not in dispute that the composition on Mac is faster without the barcodes than it is on Mac with the barcodes.

My quibble was with with your assertion that the barcode generation was slow on Windows. I've shown that's not the case.
__________________
Dan Korn
FusionPro Developer / JavaScript Guru / Forum Moderator
PTI Marketing Technologies | Printable | MarcomCentral
LinkedIn

I am a not a Support engineer, and this forum is not a substitute for Support. My participation on this forum is primarily as a fellow user (and a forum moderator). I am happy to provide help and answers to questions when I can; however, there is no guarantee that I, or anyone else on this forum, will be able to answer all questions or fix any problems. If I ask for files to clarify an issue, I might not be able to look at them personally. I am not able to answer private messages, emails, or phone calls unless they go through proper Support channels. Please direct any sales or pricing questions to your salesperson or inquiries@marcom.com.

Complex template-building questions, as well as all installation and font questions or problems, should be directed to FusionProSupport@marcom.com. Paid consulting work may be required to fulfill your template-building needs.

This is a publicly viewable forum. Please DO NOT post fonts, or other proprietary content, to this forum. Also, please DO NOT post any "live" data with real names, addresses, or any other personal, private, or confidential data.

Please include the specific versions of FusionPro, Acrobat, and your operating system in any problem reports or help requests. I recommend putting this information in your forum signature. Please also check your composition log (.msg) file for relevant error or warning messages.

Please post questions specific to the MarcomCentral Enterprise and Web-to-Print applications in the MarcomCentral forum. Click here to request access. Or contact your Business Relationship Manager (BRM/CPM) for assistance.

Please direct any questions specific to EFI's Digital StoreFront (DSF) to EFI support.

How To Ask Questions The Smart Way

The correct spellings are JavaScript, FusionPro, and MarcomCentral (each with two capital letters and no spaces). Acceptable abbreviations are JS, FP, and MC (or MCC). There is no "S" at the end of "Expression" or "Printable"! The name of the product is FusionPro, not "Fusion". "Java" is not is not the same as JavaScript.

Check out the JavaScript Guide and JavaScript Reference! FusionPro 8.0 and newer use JavaScript 1.7. Older versions use JavaScript 1.5.

return "KbwbTdsjqu!spdlt\"".replace(/./g,function(w){return String.fromCharCode(w.charCodeAt()-1)});
Reply With Quote
  #9  
Old December 18th, 2017, 04:02 PM
ScottHillock ScottHillock is offline
Regular Contributor
 
Join Date: Mar 2010
Posts: 94
Default Re: 2d Datamatrix performance

Dan, I think there was something wrong with your test. If the difference between with or without the barcode is only 4 seconds, I think you are still processing the barcode rule, especially if it is taking over 10 minutes.

Composing without the barcode gives me a baseline of 34 seconds. Adding in the barcode to use the go app is 38 seconds, with FusionPro creating the barcode it is 745 seconds.

That shows me that FusionPro creating the barcode is very slow.
Reply With Quote
  #10  
Old December 18th, 2017, 04:12 PM
Dan Korn's Avatar
Dan Korn Dan Korn is offline
FusionPro Senior Engineer / Forum Moderator
 
Join Date: Aug 2008
Location: Chicago, IL
Posts: 4,592
Default Re: 2d Datamatrix performance

Quote:
Originally Posted by ScottHillock View Post
Dan, I think there was something wrong with your test. If the difference between with or without the barcode is only 4 seconds, I think you are still processing the barcode rule, especially if it is taking over 10 minutes.
Not on Windows! On Windows, the time difference is negligible.

Yes, I'm sure I'm doing the right test, and not calculating any barcodes. Again, I'm on Windows.

Windows Creator compositions are generally slower than Mac Creator compositions. The Windows composition of your job happens to be about as slow as it is on Mac, but for a different reason, which has nothing to do with the barcodes.
Quote:
Originally Posted by ScottHillock View Post
Composing without the barcode gives me a baseline of 34 seconds. Adding in the barcode to use the go app is 38 seconds, with FusionPro creating the barcode it is 745 seconds.
Yes, and you are composing on Mac. All the numbers you are quoting are from Mac. They have nothing to do with the numbers I'm quoting from Windows.
Quote:
Originally Posted by ScottHillock View Post
That shows me that FusionPro creating the barcode is very slow.
Only on Mac!

You don't seem to be understanding that you're talking about results from Mac, and I'm talking about results from Windows. I don't know how else to explain that.
__________________
Dan Korn
FusionPro Developer / JavaScript Guru / Forum Moderator
PTI Marketing Technologies | Printable | MarcomCentral
LinkedIn

I am a not a Support engineer, and this forum is not a substitute for Support. My participation on this forum is primarily as a fellow user (and a forum moderator). I am happy to provide help and answers to questions when I can; however, there is no guarantee that I, or anyone else on this forum, will be able to answer all questions or fix any problems. If I ask for files to clarify an issue, I might not be able to look at them personally. I am not able to answer private messages, emails, or phone calls unless they go through proper Support channels. Please direct any sales or pricing questions to your salesperson or inquiries@marcom.com.

Complex template-building questions, as well as all installation and font questions or problems, should be directed to FusionProSupport@marcom.com. Paid consulting work may be required to fulfill your template-building needs.

This is a publicly viewable forum. Please DO NOT post fonts, or other proprietary content, to this forum. Also, please DO NOT post any "live" data with real names, addresses, or any other personal, private, or confidential data.

Please include the specific versions of FusionPro, Acrobat, and your operating system in any problem reports or help requests. I recommend putting this information in your forum signature. Please also check your composition log (.msg) file for relevant error or warning messages.

Please post questions specific to the MarcomCentral Enterprise and Web-to-Print applications in the MarcomCentral forum. Click here to request access. Or contact your Business Relationship Manager (BRM/CPM) for assistance.

Please direct any questions specific to EFI's Digital StoreFront (DSF) to EFI support.

How To Ask Questions The Smart Way

The correct spellings are JavaScript, FusionPro, and MarcomCentral (each with two capital letters and no spaces). Acceptable abbreviations are JS, FP, and MC (or MCC). There is no "S" at the end of "Expression" or "Printable"! The name of the product is FusionPro, not "Fusion". "Java" is not is not the same as JavaScript.

Check out the JavaScript Guide and JavaScript Reference! FusionPro 8.0 and newer use JavaScript 1.7. Older versions use JavaScript 1.5.

return "KbwbTdsjqu!spdlt\"".replace(/./g,function(w){return String.fromCharCode(w.charCodeAt()-1)});
Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 04:54 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.
(c) 2011, PTI Marketing Technologies™, Inc.