Message Processors!

Hello out there! Its Brendan again, and I'd like to share some information with you about message processors, which are pretty mysterious, unless you know how they work. These executables are the heart and soul of Scribe Insight, and I'll bet you never quite understood how they operated before. Well, fear not! I'm here to shed some light on this subject.

Now some of you might be wondering what a message processor is? Well, it's an executable that Scribe Insight uses to automatically run a DTS file. These programs are what make your integrations run in the background. There are two different types of message processors when it comes to Scribe; those owned by the Scribe Event Manager and those owned by the Scribe Message Server.

The message processor owned by the Event Manager is what controls the running of all your Time, File, and Query integration processes. There is only one message processor allocated for this purpose, meaning that these types of integrations are all single threaded. Now, what does that mean to you? It means if you have 5 Time based integration processes that each take 10 minutes to complete, and they are all set to run at the exact same time, it will take 50 minutes until these all finish. This can lead to some bottlenecks in your integration design, if you have one integration process behaving badly, it can hold up every other integration on the machine.

The message processors owned by the Message Server are what controls the running of all Queue based integrations. There are several of these that can be allocated, based on your Scribe license. The standard Scribe license has 8 message processors, 1 dedicated to the Time, File, and Query based integrations, and the other 7 dedicated to queue based integrations. Professional licenses and Enterprise licenses have 64 Message Processors, again 1 is dedicated to Time, File, and Query integrations, and 63 dedicated to Queue. So the first thing you can gather from this is that Queue based integrations can really be tuned up to boost performance having so many more threads than the other 3 types of integration processes. But, what else can they give you?

Well, the short answer to that is a lot. Not only do these Queue based integrations give you many threads; they also are smart enough to share connections. So that means, if a message processor has a DTS open, it will try to hold it open as long as possible. This means we cut out the overhead cost of opening these connections over and over again. When you are working with adapters like Dynamics CRM and Salesforce.com, that benefit really starts to shine.

Queue message processors also allow you to build in retry capabilities, so that if something fails, it does not take user intervention to fix it. Let's take a salesorder as an example. With multiple threads it's possible the account and salesorder message will get picked up out of sequence, meaning the order will attempt to be inserted before its corresponding account exists. Without automatic retry capability, this would simply fail, and have to be re-submitted. BUT, with Queue based integrations, we can set up an End Job Retry command in the DTS file when the Account seek finds no results, which causes the message to be forwarded into the Scribe Retry Queue. Each integration has a number of retries it will allow, 120 by default. What this means to you is that the order message will retry that set number of times before finally giving up, so we've given the account message sufficient time to get picked up by another thread and processed. Pretty cool, huh?

 

So that will do it for me, keep an eye out for more training offerings this year and stay tuned for more blog posts from yours truly!

Comments

[chris markiewicz]

Hello Brendan. Thanks for the article. Quick question related to Queue Message Processor. I have a Queue process that generates an XML file for each message (the filename is set with the timestamp). I seem to be running into an issue where - since threads run concurrently as you describe - Scribe tries to create multiple files with the same name. (The message I'm seeing is: 20 – msxml3.dll …. The process cannot access the file because it is being used by another process.) So my question is: is there any way to configure the Message Processor so that it avoids this issue?
Thanks
Chris

[Brendan]

Thanks for the comment Chris, unfortunately the multi threading is definitely hampering you here, the only way I could guarantee the XML time-stamp issue would go away would be to ratchet the threads down to 2, 1 for queue integrations and 1 for time, file and query integrations. But, to help alleviate the issue, make sure that your DTS is set to have a failure on the XML insert do a End Job Retry. That would throw the message into the retry queue, which would give it a chance to get created with a real unique time-stamp. You would still get the error, but, the XML should get created in one of the retry loops.

Brendan

[chris markiewicz]

Ah - i wasn't sending the failure to End Job - that should take care of it. As you say, there will still be failures, but they'll make it before long. if i'm still seeing issues, i'll change the number of threads, but i'd like to leave that where it is so that i don't slow other processes down.
thanks so much for your response.
chris