ScottHillock Posted December 14, 2017 Posted December 14, 2017 If you've used the FusionPro datamatrix encoder you know it's painfully slow. Our usual solution for increasing performance on simple jobs is to have a 100,000 page pdf with a datamatrix barcode on each page, counting from 1-100,000. Then we use a variable image to grab the page number we need that matches the number we need. So pull in page 27, if you need a datamatrix of "27". We've come into a new project that though requires a more complex datamatrix code, so with a bunch of searching and testing I landed on using an open source encoder that would deliver the barcodes in a get request to an http server. Here's the code that ends up being used in FusionPro. You could also CreateResource on it, but for us this is the easiest drop in replacement. var VariableText = [color="red"]Field("FIELD")[/color]; var dataMatrixApi = NormalizeEntities("http://[color="Red"]127.0.0.1[/color]:3030/?text=" + VariableText); Pic = '<graphic file="' + dataMatrixApi + '">'; return Pic; Simply replace the red text with your requirements and ip address. Attached is the Mac command line application that I use, which will automatically listen on port 3030, and return a png of the barcode. It's a golang application so if you're on windows you can compile it yourself. I have mine set to autorun using "LaunchControl.app". You can also use Windows Task Scheduler to perform the same thing to auto launch at login/startup. Here's the code to compile yourself (you'll need to download golang to do so): package main import ( "bytes" "strconv" "net/http" "image/png" "github.com/boombuler/barcode/datamatrix" ) func handler(w http.ResponseWriter, r *http.Request) { datamatrixCode, _ := datamatrix.Encode(r.URL.Query().Get("text")) buffer := new(bytes.Buffer) png.Encode(buffer, datamatrixCode) w.Header().Set("Content-Type", "image/jpeg") w.Header().Set("Content-Length", strconv.Itoa(len(buffer.Bytes()))) w.Write(buffer.Bytes()) } func main() { http.HandleFunc("/", handler) http.ListenAndServe(":3030", nil) } Moving from the FusionPro encoder to this solution we saw an increase of 15x to our composition. This is on FusionPro 9.3.36. Not sure if 10 has the native JavaScript encoder from IDAutomation that would have even faster performance. But, this solution puts us super close to the speed we would get if we upgraded to Producer.golang-barcode.zip Quote
Dan Korn Posted December 15, 2017 Posted December 15, 2017 That's a very clever solution, and a welcome contribution here. Thanks! I feel compelled to offer a couple of caveats, though. First, I would carefully check the licensing of any third-party code before utilizing it in production, especially for printed output. Second, MarcomCentral/PTI cannot vouch for the accuracy or reliability of using such third-party code, nor can we assume any responsibility for problems that may arise from such usage. Also, I have to quibble slightly with this statement: If you've used the FusionPro datamatrix encoder you know it's painfully slow. That statement is not universally true. It may be true on some versions of Mac OS X, but not necessarily on all of them, and certainly not on Windows. On Mac, the Data Matrix barcode encoder (along with the PDF 417 barcode encoder) uses Java (not JavaScript), the performance of which isn't great on some versions of OS X. On Windows, it calls a native DLL, which takes no noticeable time at all. Moving from the FusionPro encoder to this solution we saw an increase of 15x to our composition. This is on FusionPro 9.3.36. And what version of OS X? Not sure if 10 has the native JavaScript encoder from IDAutomation that would have even faster performance. No, there has been no change to the Data Matrix encoder from FusionPro 9 to 10. But the new JavaScript encoder is something we'll look into. Thanks for the heads up. But, this solution puts us super close to the speed we would get if we upgraded to Producer. That's fine. However, I would caution others to not necessarily expect similar results. Obviously the total impact on composition speed depends on many factors, not just the Data Matrix barcode generation. And, as noted, the barcode generation speed depends on the operating system. Frankly, the biggest advantage of creating barcodes this way is that they are images, not text, so they can be more easily scaled. We added the ability to create QR barcodes as graphics a while back. Perhaps it's time to do that for Data Matrix as well. Quote
ScottHillock Posted December 15, 2017 Author Posted December 15, 2017 Dan, the source code of the library on this is MIT licensed. Which is pretty permissive. But, I agree others should always double check licensing and any possible copyright issues with code pushed out to the public domain. We've had the same performance from the FusionPro datamatrix encoder on macOS 10.9.x, 10.11.6, 10.13.1, and on Windows 10. I had PTI support run the same tests on Producer and Composer on Windows, and Composer on Windows had the same performance as I've experienced across all platforms. I agree many things effect composition speed. On this particular project that had multiple variable images, copyfitting, page selections, split outputs, and for 1.2 million records the composition time was reduced from 65 hours to 7 hours. Quote
Dan Korn Posted December 15, 2017 Posted December 15, 2017 We've had the same performance from the FusionPro datamatrix encoder on macOS 10.9.x, 10.11.6, 10.13.1, and on Windows 10. I had PTI support run the same tests on Producer and Composer on Windows, and Composer on Windows had the same performance as I've experienced across all platforms. First of all, what is "Composer"? Do you mean FusionPro VDP Creator (aka Desktop)? Anyway, I haven't seen your job, nor what exactly you're encoding into these barcodes. For me, a simple Data Matrix barcode takes almost no time at all to create on Windows 10. I added a Data Matrix barcode to the Cell Phone tutorial job, and it added one second for 100 records compared to composing without it. So that's 1/100th of a second to make each barcode. I have no idea why your results are different. Also, did all of these tests you mention isolate the barcode generation, or were they jobs doing other typesetting? It's possible that the composition of your job is generally faster on Mac in FusionPro 9.3, even though the barcode generation may be slower there. I can also tell you that Producer composes much faster in FusionPro 10 than in FusionPro 9. So again, while you may have run into a particular edge case where barcode generation is slow, those results are not typical. Anyone else is welcome to chime in with their results, of course. I agree many things effect composition speed. On this particular project that had multiple variable images, copyfitting, page selections, split outputs, and for 1.2 million records the composition time was reduced from 65 hours to 7 hours. In Producer? Where the only change was how the barcode is generated? Quote
ScottHillock Posted December 18, 2017 Author Posted December 18, 2017 Yes, sorry, by Composer I meant Creator. And my long 65 hour composition that reduced down to 7 hours was in Creator. Attached is a simple job that only creates the datamatrix barcode. My results, all using FusionPro Creator 9.3.36, from local ssd hard disk, all 16GB ram: macOS 10.11.6 (iMac Late 13, 3.5GHz i7) - 745 seconds macOS 10.13.1 (imac Late 13, 2.9GHz i5) - 845 seconds And PTI support's Windows FusionPro Creator result: - 635 seconds Using the datamatrix-as-image solution with FusionPro Creator, my time went down to 38 seconds.Archive.zip Quote
Dan Korn Posted December 18, 2017 Posted December 18, 2017 (edited) On my computer running Windows 10 and FusionPro VDP Creator 10.0.26, your 5000-record job composed in 10 minutes and 30 seconds. I then removed the DataMatrix barcode rule, and it composed in 10 minutes and 26 seconds. That supports my contention that the time to generate the barcodes on Windows is negligible. It's also very close to the time that Support reported, although you seem to have jumped to the conclusion that the barcode generation was a significant part of that time, which it clearly wasn't. Edited December 18, 2017 by Dan Korn Quote
ScottHillock Posted December 18, 2017 Author Posted December 18, 2017 (edited) Weird, if I change the datamatrix code to just return ""; I get a speed of 34 seconds... Maybe you didn't edit the code correctly, or there's something wrong hardware wise with your workstation. Edited December 18, 2017 by ScottHillock Quote
Dan Korn Posted December 18, 2017 Posted December 18, 2017 Weird, if I change the datamatrix code to just return ""; I get a speed of 34 seconds... Maybe you didn't edit the code correctly, or there's something wrong hardware wise with your workstation. On Mac? I was talking about Windows. Yes, Creator composition is faster overall on Mac than on Windows, without any 2D barcodes. That's not in dispute. It's also not in dispute that the composition on Mac is faster without the barcodes than it is on Mac with the barcodes. My quibble was with with your assertion that the barcode generation was slow on Windows. I've shown that's not the case. Quote
ScottHillock Posted December 18, 2017 Author Posted December 18, 2017 Dan, I think there was something wrong with your test. If the difference between with or without the barcode is only 4 seconds, I think you are still processing the barcode rule, especially if it is taking over 10 minutes. Composing without the barcode gives me a baseline of 34 seconds. Adding in the barcode to use the go app is 38 seconds, with FusionPro creating the barcode it is 745 seconds. That shows me that FusionPro creating the barcode is very slow. Quote
Dan Korn Posted December 18, 2017 Posted December 18, 2017 Dan, I think there was something wrong with your test. If the difference between with or without the barcode is only 4 seconds, I think you are still processing the barcode rule, especially if it is taking over 10 minutes. Not on Windows! On Windows, the time difference is negligible. Yes, I'm sure I'm doing the right test, and not calculating any barcodes. Again, I'm on Windows. Windows Creator compositions are generally slower than Mac Creator compositions. The Windows composition of your job happens to be about as slow as it is on Mac, but for a different reason, which has nothing to do with the barcodes. Composing without the barcode gives me a baseline of 34 seconds. Adding in the barcode to use the go app is 38 seconds, with FusionPro creating the barcode it is 745 seconds. Yes, and you are composing on Mac. All the numbers you are quoting are from Mac. They have nothing to do with the numbers I'm quoting from Windows. That shows me that FusionPro creating the barcode is very slow. Only on Mac! You don't seem to be understanding that you're talking about results from Mac, and I'm talking about results from Windows. I don't know how else to explain that. Quote
ScottHillock Posted December 19, 2017 Author Posted December 19, 2017 I installed FusionPro on a Windows VM. With or without generating the barcode, you are correct, the results are negligible. I didn't realize that the Windows version had such a wide performance difference than the mac version on composing. Sorry for jumping to the conclusion the 2d performance was to blame. I ran a loop to generate the barcodes so only 1 record is composed, but the barcodes are generated many times. And, on Windows my results were for 10,000 iterations: 0.50 seconds - FusionPro Datamatrix 0.03 seconds - Golang image generator 1.60 seconds - QR code On mac they were: 1570.15 seconds - FusionPro Datamatrix 0.03 seconds - Golang image generator 3.00 seconds - QR code So, you are correct Dan, on Windows Datamatrix performance is quite good. But on the mac it is quite slow. Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.