Scribe Software Blog

Are you ready to upgrade Scribe Insight?

Are you using a Supported Version of Scribe Insight? If not, you are at risk of not being able to receive help when you have an issue. The Scribe Insight 6.5.x platform will be discontinued the end of June 2011 (**please see update below). What does this mean to you? If you are running on Scribe Insight 6.5.x or older, you will not receive support for your installation. Scribe Technical Support will not be able to deliver new hotfixes and will have limited ability to replicate your issues in house. Simply put, if you value your Scribe Insight deployment, you need to upgrade. Why not upgrade and take advantage of the innovative features that the Insight 7 platform provides, especially the coveted Multi-Target functionality that will dramatically simplify your integrations. As long as you are on current Maintenance, you can upgrade to Insight 7.0.2.

**UPDATE: Due to the number of customers who responded to this upgrade reminder, we are pleased to extend support through the end of August, 2011. This will allow ample time for those who have delayed their upgrade to complete it over the upcoming months.

In general, it is important to plan on doing regular upgrades to stay current with all of your business applications. Plan for it. Budget for it. And please refer to the Scribe Support Policy to learn more about how we determine when a product will become discontinued or end of life. You can also refer to the Scribe Supported Versions to verify the support plans for each version.

Converting a Time/File/Query based integration to a Queue based integration

Queue based integrations are the way to go when designing a new integration. As you can tell from reading other posts in this Blog and from our training, we recommend that you make use of queue based integrations when you can. Queue based integrations can take advantage of the multi-threading capability of your Scribe Insight server and can have an automatic retry for rows that fail to be processed.

But, what if you created an integration using file, time, or query integration processes before you knew how great queue based integrations were? We don't want you to have wasted all the time you spent creating that integration solution. So here are some steps to follow to preserve all your work if you want to transition your existing integration to one that uses message queues.

The first thing you need to do is create a publisher to feed the new queue based integration. Queue based integrations utilize XML as the source in the DTS as opposed to using a connection directly to a data store. You can review this blog post to find out more info on how to setup and use query publishers. The most important thing that you should do is use the same exact query from the original DTS file in your new query publisher. This will ensure that you get the same data, in the same format, as you did in your original DTS file. It makes your conversion much smoother.

Once your query publisher is set up, you are ready to start changing the DTS file over to use XML as the source. First, create a schema file (XDR) and a sample XML file from the query publisher. Then open your DTS file and use the Connection Manager to add a connection to XML . Define this XML connection as a source, and select Dynamic as the connection type.

Once the connection has been established its time to swap it in as the new source for your DTS file.

When go through this process, you will get a LOT of errors, but, in the end they are worth it. When given the option, select that you want to use the same data links and source query; so that it does not wipe out your data links or source fields. Once you get to the point of changing the source query, you will have to set the XML object as the new source, and define any repeating nodes that you may have in the XML. Again, you may get some errors about re-defining links, source fields going missing, etc., but stay with it, you are almost finished.

When you come to the end of the warning messages, you will see that the source is changed, but, all your data links are now invalid. This occurs because the Workbench cannot validate what field belongs to which source field reference number. For example, let's say you used to have a data link using S3, which was Name, the Workbench cannot figure out what field is now the Name field. This happens because of the way the XML is formatted. Even though the field name is the same to you, to Scribe it's slightly different. What was once a Name on the Account table is now <Account.Name> under the covers; so it does not match 100% to what it was before. Also, on top of this, 7 extra fields are added to the beginning of your field list, so S3 is actually now S10.

DTS Query connected to SQL

DTS Query now connected to XML

This is where some manual labor comes into play, as you must correct each invalid link to change the ?# (?1, ?2, ?3, etc.) to the new reference (S8, S9, S10). This does sound like a lot of work, but it's a whole lot easier than having to re-design every single data link and formula that is in your DTS file. Once you get all the links, including your calculated variables and lookup links, corrected, you are ready to start testing your newly updated DTS.

The last piece of this puzzle is to create a queue based integration process that calls this DTS. When you create the integration process, make sure the message label matches the message label created by the query publisher you created.

At that point, you should be ready to start testing your query publisher to send data to the message queue, and testing your integration process and related DTS file to see if they process messages properly.

If you want to learn more detailed steps about how this process works, please check out the Scribe Inside Track service, which gives you access to a vBook called "Message Queuing" including a step-by-step video guide to converting your DTS files. A subscription to the Inside Track gets you access to premium content including deep dive educational videos, monthly interactive expert panel discussions, and a quarterly Inside Track newsletter. Learn more here - http://www.scribesoft.com/InsideTrack/.

 

Microsoft mainstream support ending for CRM 3.x

With their upcoming release of CRM 2011, Microsoft is preparing to sunset support for v3.x.  You can find their timeline here: http://support.microsoft.com/lifecycle/?LN=en-us&p1=9776&x=8&y=11, but to save you the round trip: official mainstream support will end in April '11 and extended support ending in April '16.

As Scribe moves forward with CRM 2011, we will be following Microsoft's example & will be discontinuing support for Microsoft CRM 3.x as of the upcoming CRM Adapter release in March '11. While we will not be making any technical decisions that would prohibit Scribe Solutions from working with MS CRM 3.x, we will not be testing our solutions on MS CRM v3.x & Scribe Support will not be fielding questions about how to resolve issues on MS CRM 3.x.

If you have any questions or concerns, please let us know via a reply to this post or by emailing us at ProductManagment at Scribesoft.com (<-I've obfuscated the email address here to fool them web bots).

Message Processors!

Hello out there! Its Brendan again, and I'd like to share some information with you about message processors, which are pretty mysterious, unless you know how they work. These executables are the heart and soul of Scribe Insight, and I'll bet you never quite understood how they operated before. Well, fear not! I'm here to shed some light on this subject.

Now some of you might be wondering what a message processor is? Well, it's an executable that Scribe Insight uses to automatically run a DTS file. These programs are what make your integrations run in the background. There are two different types of message processors when it comes to Scribe; those owned by the Scribe Event Manager and those owned by the Scribe Message Server.

The message processor owned by the Event Manager is what controls the running of all your Time, File, and Query integration processes. There is only one message processor allocated for this purpose, meaning that these types of integrations are all single threaded. Now, what does that mean to you? It means if you have 5 Time based integration processes that each take 10 minutes to complete, and they are all set to run at the exact same time, it will take 50 minutes until these all finish. This can lead to some bottlenecks in your integration design, if you have one integration process behaving badly, it can hold up every other integration on the machine.

The message processors owned by the Message Server are what controls the running of all Queue based integrations. There are several of these that can be allocated, based on your Scribe license. The standard Scribe license has 8 message processors, 1 dedicated to the Time, File, and Query based integrations, and the other 7 dedicated to queue based integrations. Professional licenses and Enterprise licenses have 64 Message Processors, again 1 is dedicated to Time, File, and Query integrations, and 63 dedicated to Queue. So the first thing you can gather from this is that Queue based integrations can really be tuned up to boost performance having so many more threads than the other 3 types of integration processes. But, what else can they give you?

Well, the short answer to that is a lot. Not only do these Queue based integrations give you many threads; they also are smart enough to share connections. So that means, if a message processor has a DTS open, it will try to hold it open as long as possible. This means we cut out the overhead cost of opening these connections over and over again. When you are working with adapters like Dynamics CRM and Salesforce.com, that benefit really starts to shine.

Queue message processors also allow you to build in retry capabilities, so that if something fails, it does not take user intervention to fix it. Let's take a salesorder as an example. With multiple threads it's possible the account and salesorder message will get picked up out of sequence, meaning the order will attempt to be inserted before its corresponding account exists. Without automatic retry capability, this would simply fail, and have to be re-submitted. BUT, with Queue based integrations, we can set up an End Job Retry command in the DTS file when the Account seek finds no results, which causes the message to be forwarded into the Scribe Retry Queue. Each integration has a number of retries it will allow, 120 by default. What this means to you is that the order message will retry that set number of times before finally giving up, so we've given the account message sufficient time to get picked up by another thread and processed. Pretty cool, huh?

 

So that will do it for me, keep an eye out for more training offerings this year and stay tuned for more blog posts from yours truly!

Message Queuing Tips

Microsoft Message Queue (MSMQ) is a well adopted tool and part of many Scribe based integrations. It is also baked in to the integration templates that we publish because it enables faster throughput of data by enabling the multi-message processor capability of Scribe and because of the relatively easy to implement automatic retry of failed rows.

However, what is not well known is how to manage message queues. I realize that for some of our users, their first exposure to message queuing is when they have to implement it as a requirement for setting up a Scribe template. I recently did some research on MSMQ and I thought I would share some information about it, hoping it would shed some light on what might be a new area for you.

The 4 MB Limit

You might have heard about an absolute 4 GB limit on a message queue. That's not exactly the story. What I learned that there is a 4 MEGABYTE limit is on the size of an individual message.

Storage Limits

 

Storage limits can be set on message queues. If you want to do some reading on this, also search for the word quota, because storage limits are called quotas in the MSMQ documentation. However, the word quota is not used the MSMQ user interface.

You can put a quota on all the cumulative space occupied by all message queues in one implementation of MSMQ or you can put a quota on an individual queue. For the cumulative option, you do that as a property of Message Queuing in Computer Management. For the queue option, you do it as a property of a queue.

You can increase a quota or disable quotas altogether. Disabling quotas is not a good idea because you will then allow MSMQ to use up more than its fair share of resources and you will likely slow down other programs on the server.

Express Messages vs. Recoverable Messages

Messages can be one of two types, express or recoverable. Express messages are stored in memory, unless your server runs out of RAM allocated to MSMQ, then the messages are stored on disk.

Recoverable messages are always stored on disk.

Scribe publishers create recoverable messages. If you are using a DTS file to create messages in MSMQ, you have the option to create express messages or recoverable messages.

Message Storage

 

The files in which the messages are stored are "memory mapped" which means they do take up some memory, but not as much as the actual message that is stored in the message queue file. Messages use some kernel memory because of being memory mapped, although the amount of memory they use is small. It would take 1 million messages to use up 75MB of kernel memory.

By default messages are stored in C:\WINDOWS\system32\msmq\storage. You might want to have a separate volume dedicated to message storage, rather than leaving it at this default setting.

This article is about the limits on MSMQ storage, which it says are virtually limitless, if you can buy all the hardware you want. But in the real world of budgets, there are limits. J There are some tips in the about how to estimate how much storage space you will need.

This article has information about the files in the storage folder and what type of data the files store.

Troubleshooting

There is a lot of good information about troubleshooting the "insufficient resources" error in this article. I would expect that points #3, #6, and #7 in this article to be the most likely situations that you will run into.

Monitoring

To be proactive in monitoring how much storage space message queues are taking up, there is a performance counter for Total Bytes in All Queues that you could use. For example, in Microsoft System Center Operations Manager, you could create a monitor that creates an alert when the storage space used for all queues reaches 80% of the storage limit.

More Information About Administering MSMQ

It would be a lot to read at one time, though is a lot valuable information about how to administer MSMQ in this Microsoft TechNet article.

Creating Connections with Insight 7

Insight 7 includes the new capability to add steps for multiple targets in one DTS file. This is a big departure from the way things used to work and as you might imagine, adding a major capability like this will impact other features in the product and the user interface. One of the changes is Connection Manager, which I will give you an introduction to in this article.

The concept of connections has always been there in the product, but it was behind the scenes and we never openly referred to connections because we already had a name for each of them - the source, the target, and internal. Now with the multi-target capability of Insight 7, the concept of a connection comes into play because we need to be able to have names for all the connections that you could have in a DTS file; and there might be more than three of them.

In Insight 7 there is a new Connection Manager feature, in which you set up connections, give the connections the name you want, and edit the settings for connections. A little later, I will show you some screenshots and give you some tips on how to use the Connection Manager.

Overview of the connection process

First, I would like to spend a moment on something that could be easily overlooked. I think this is worth pointing out because it could help you get used to a new way of setting up your DTS files that has an easy and logical flow to it. In earlier versions of Insight, you would normally follow steps in this order:

You can still follow the old pattern you like, though I have found a slightly different order to work best for me when using Insight 7, and maybe it will work for you too:

Creating a connection

To create a connection in Insight 7, on the View menu, click Connections. The Connection Manager appears. Note that there is already a connection there for the Scribe Internal Database. This is the database that Scribe Insight uses to store its settings and logs. You should use InternalDB.exe if you need to make changes to the password. Also, be careful if using with this connection as a source or a target because doing so could make unwanted changes to the SCRIBEINTERNAL database.

To add a connection, click Add. The Add a Connection dialog box appears.

This dialog should look familiar to you. It's the same dialog that you used in earlier versions of Insight. From there, select the thing you want to connect to. In this example, I'll connect to the ODBC DSN called Scribe Sample. After clicking Scribe Sample and clicking OK, the login dialog box appears. This dialog box will differ depending on what you are connecting to. After logging in, the Connection Settings dialog appears.

On the General tab, you can enter a name for the connection. Type something that is short and easy to remember. Tip: Make it short because you might be using this connection name as a parameter in a function, such as DBLOOKUP. The old S, T, I, parameters in functions have been replaced with entering the connection name. So the old

On the Connection tab, you can click Change Connection to change the login parameters for the connection. Note that there are a couple of features that have migrated to this dialog box. 1.) Adapter Settings used to be located on the Source or Target menu in the Workbench, depending on whether you were using an adapter as a source or as a target. 2.) Prompt for reconnect when opening this DTS and Keep this connection open if it was used in the previous DTS used to be located in the Source and Target tabs in the Properties dialog box on the File menu. 3.) SQL scripts used to be in the Translation Spec Settings dialog box, located on the Run menu.

Once you have added some connections and have used them in your DTS file, you will see that the Usage column tells you how the connections are being used. This helps you keep track of your connections, and where you might need to go to make adjustments.

Upgrade tips

You may be wondering what happens with your DTS files that were created before Insight 7 when you open them in the Insight 7 Workbench. Here's a quick rundown on what happens.

• A connection called Source is automatically created.
• A connection called Target is automatically created.
• Formulas and other settings in the DTS file that used to use S, T, or I as parameters are replaced with Source, Target, and Scribe Internal Database respectively.

I hope you find this information helpful in getting familiar with some of the changes related to the multi-target capability of Insight 7.

Update to Scribe Support Policy

The Support team and I have been working hard to improve your experience when dealing with Scribe Technical Support. Part of that experience is knowing what to expect when contacting Scribe for Technical Support. Do I have to pay for Incidents? When can I expect my Support Incident to be resolved? I don't feel like my issue is getting enough attention, how do I escalate my Incident? These are all great questions. I addressed several of them with an update to the Support Policy in June 2009 and this week published a new revision which closed the gap. The new revision combines a clarified version of the previous Support Policy and a revised Service Level Agreement. What are a few highlights in this update?

• Explanation of who is eligible to receive technical support
• Explanation of how to open Support Incidents
• Explanation of how Support Incidents are prioritized
• Service Level Targets
• Conditions of Service
• The difference between Consulting and Support
• Reporting and Escalation

I think many of you will feel that the Service Targets have the most relevant day-to-day impact on your Scribe experience. The document explains that each Support Incident is given a priority and that that priority dictates how much attention (read: how quickly) the Incident will get resolved. You can imagine the challenge that every support organization has: Every caller states that their issue is critical and must be resolved in minutes. This presents a bit of challenge as we then have all Incidents being a P1 or Critical Incident – how do we choose which to tackle first? In addition to feedback from the caller, we use the following table as a guideline to set the call priority:

Priority		Description
Critical		Production System, Materially Impacted, No Reasonable Workaround in place
High		Production System, Materially Impacted, Reasonable Workaround in place - OR – Non-Production System, Project is Materially Impacted, No Reasonable Workaround in place
Medium		Production System, No Material Impacts - OR - Non-Production System, Project is Materially Impacted, Reasonable Workaround in place - OR - Non-Production System, Project is Not Materially Impacted, No Reasonable Workaround in place
Low		Non-Production System, Project is Not Materially Impacted, Reasonable Workaround in place - OR - Request for Information

 

This gives us the guidance we need and sets the expectation that if you open a Support Incident and your production system is materially impacted (such as no orders can be processed because Scribe stopped working) and there is no short term-workaround in place, you'll be at top of the queue. Compare that with a person who is working with the Scribe Demonstration Virtual PC and does not understand how the template works. The person with the production system down will always be helped before the person asking about a template in a VPC Demo.

Please take some time to review the Support Policy and let me know what you think.

Scribe Insight User Guide

Over the last few months, I've been working on a project with our technical writer, Miriam Lezak, on one of our "must do" projects – a user guide for Scribe Insight. I'm pleased to announce that the user guide is ready and you can download it by clicking on the thumbnail of the cover page below. 

Before I go into detail about what is in the user guide, I'll give you a little background on what we are trying to accomplish with it. While we do not plan for the user guide to ever be a replacement for the online help for Insight, we believe it is important to have a user guide as part of the product documentation set. Specifically, our goal for the user guide is to provide a documentation resource that gives a good overview of the features in the Workbench and the Console, and on how to put them into action. We are trying to present the information in a more linear fashion for what sometimes is a non-linear process you follow when using Insight to design an integration solution.

We also wanted to provide a documentation resource that is printable. Because of the way online help is organized and because of the tools used to create it, it does not lend itself to printing. So creating the user guide in PDF format gives us something that can be read online, searched, and can also be printed if needed.

We expect that the user guide will be helpful for users that are new to the product and are looking for more in depth information than you can find on a data sheet. We also think it will be something that can be sent to prospective customers if they are looking for more information about how Insight works and what it looks like.

Here's a quick overview of what you will find in the user guide.

• An overview of Workbench features and concepts.
• How to use the Workbench to create a DTS file.
• An overview of Console features.
• A chapter highlighting several use case scenarios and how to design solutions for them using Insight.

While I have your attention for this announcement, let me give you a little more information about additional documentation resources that are available. There is the Scribe Insight Installation Guide, which includes the step-by-step instructions for installing and configuring Insight including how to set it up in a failover environment. There is the Scribe help library, which contains a Web help version of all of our product help, including Insight, adapters, templates, and template components.

The last resource I'd like to mention is the Technical Resources vBook. I know it does not fit the usual definition of documentation, but it does contain valuable product information in primarily a video format; so you can think of it as a talking book. Register for the Technical Resources vBook and use the registration code "technical" if you want to check it out. I've set the goal of adding at least six new chapters to the book in 2010. So stay in touch if you want to see what new content we will be publishing.

I'm always open to feedback about what we can do to improve our documentation and training resources. The goal is to make it easier for you to use our software to provide solutions for customers. I keep in touch with our tech support group and I monitor the activity on our forums. I'm also open to your direct feedback.

Scribe 2010 Technical Support Calendar

New Year's Day – 1 January, 2010

Scribe Technical Support and Scribe offices will be closed.

Martin Luther King Day – 18 January, 2010

Scribe US offices will be closed. Scribe Technical Support and non-US offices will be open.

Presidents Day – 15 February, 2010

Scribe US offices will be closed. Scribe Technical Support and non-US offices will be open.

Memorial Day – 31 May, 2010

Scribe Technical Support and Scribe US offices will be closed. Scribe non-US offices will be open.

Independence Day – 5 July, 2010

Scribe Technical Support and Scribe US offices will be closed. Scribe non-US offices will be open.

Labor Day – 6 September, 2010

Scribe Technical Support and Scribe US offices will be closed. Scribe non-US offices will be open.

Columbus Day – 11 October, 2010

Scribe Technical Support will have limited support hours on October 11, 2010 from 8:30 am EST to 5:30 pm EST. Scribe US offices will be closed. Scribe non-US offices will be open.

Thanksgiving – 25 - 26 November, 2010

Scribe Technical Support will be closed on November 25, 2010 and will have limited Support hours on November 26, 2010 from 08:30 AM EST to 05:30 PM EST. Scribe US offices will be closed November 25 – November 26. Scribe non-US offices will be open both days.

Christmas – 24 - 26 December, 2010

Scribe Technical Support will have limited support hours on December 24, 2010, closing at 3:00 pm EST and will re-open on Monday December 27, 2010. Scribe offices will close at 3:00 pm December 24 and will reopen on Monday December 27, 2010.

New Years Eve – 31 December, 2010

Scribe Technical Support will have limited support hours on December 31, 2010 closing at 3:00pm EST. Scribe offices will close at 3:00 PM on December 31, 2010.

 

Are you Entitled?

 My name is Eric Wahl, I am the Director of Services at Scribe. I've been on board with Scribe Software for almost a year now. During the past year, I've been working hard with my team to improve the level of service that you receive from Scribe Technical Support. We have revamped our Incident Management processes, improved our tooling and reporting, grown our OpenMind Knowledge Base, and through the efforts of Scribe's Mark Walker, delivered training in the vBooks format (see an example of the vBook format). Last week we recorded our 2^nd of 3 sessions that will become the CRM Adapter training. I'm thrilled about this new training medium and the revamped content that is being presented. Why am I thrilled? It's not just because the content is solid, that it is on-demand or even that it gives everyone a chance to see a few of the faces here at Scribe. It's more selfish than that. This training will ultimately reduce the number of Support calls we receive here. We receive numerous "how-to" calls per day, calls that might not have to be made if more people used the vBooks training.

I am challenged with setting the response time expectations of all of our callers and ultimately where they sit in the queue of Support Incidents. Let me give you a bit of background: We address our Support Incidents by Incident Priority where a Priority 1 (P1) is considerably more important than a Priority 4 (P4). On a P1, we drop everything and work until the Incident is resolved. On a P2 we spend significant effort to get the Incident resolved within 3 days. P3s, which are where most of our Incidents sit, are tackled after the P1's and P2's. We use the guidelines below to sort things out.

Priority		Description
P1		Production System, Materially Impacted, No Reasonable Workaround in place
P2		Production System, Materially Impacted, Reasonable Workaround in place - OR - Non-Production System, Project is Materially Impacted, No Reasonable Workaround in place
P3		Production System, No Material Impacts - OR - Non-Production System, Project is Materially Impacted, Reasonable Workaround in place - OR - Non-Production System, Project is Not Materially Impacted, No Reasonable Workaround in place
P4		Non-Production System, Project is Not Materially Impacted, Reasonable Workaround in place - OR - Request for Information

The issue lies with the fact that most Support Incidents are a P3. With most Incidents being a P3, how do we prioritize the P3's? How does Scribe ensure that your support calls are resolved in a timely fashion to meet your needs? Scribe has Service Levels that I have revised and am ready to publish in the next version of our Support Policy. However, in order for us to make the new targets, we need to have a reasonable call volume so we can be responsive to the people that really need the help. To get where I need to be, I am re-enabling our Partner Entitlements program. As part of the Scribe reseller agreement, our partners are obligated to provide level-1 support to all of their customers and depending on their partner status with Scribe they may also receive a number of free Support Incidents per certified consultant per calendar year. What does this really mean to you?

• If you are a partner and are calling about something other than Installation or Registration issues, we'll log your call, deduct that Support Incident from your remaining annual entitlements and help you out. If you have used all of your annual entitlements or never had any to start with, we will help you out and our Sales Team will contact you after the fact to purchase a Support Incident.
• If you are a customer and are calling about something other than Installation or Registration issues, we'll refer you to your Scribe partner for assistance. If you do not want use your partner, we'll direct you to Sales to purchase a Support Incident.
• If you are calling about Installation or Registration issues, we'll always help you free of charge.

I've seen several very responsible partners who have asked to have access to all of the Support Incidents that their consultants have logged with Scribe so that they can address these questions as training issues internally. I think this is a great approach and I am making changes to our SupportWeb pages to give partners full access to all Support Incidents logged for that Partner.

I'm really hoping that re-enabling our Entitlements Program, the changes in our training, and changes to our SupportWeb system will enable you to have a better support experience.