Scribe Software Blog

NAV and AX Adapter Release Candidates Available for Download

Last week I posted to the blog that the Insight 7.0 Release Candidate was available for download, along with most adapters except for AX and NAV (see "Steelhead Release Candidate Available for Download"). As of today, release candidates for the Scribe Adapter for Dynamics NAV v1.4 and the Scribe Adapter for Dynamics AX v1.1 are available for download as well.

To access the download page for these pre-release versions of the adapters, please view the original web cast at http://blog.scribesoft.com/2010/05/get-early-access-to-insight-70-with-the-steelhead-technical-preview.html.

Insight 7.0 Makes Multi-Threaded Integrations Safer

Designing integrations to operate on multiple message processors is a powerful capability of the Scribe Insight product that can drive large performance gains. In fact, using message processors to improve your performance was the subject of a technical workshop vBook which you can read more about at https://openmind.scribesoftware.com/topics/best-practice-for-performance-all-on-one-or-prefer.

There is one potential caveat to multi-threaded integrations. To understand the caveat, consider the following scenario in which you need to synchronize company information from your ERP system with accounts in your CRM system. You have configured a publisher that writes record change events to the ScribeIn queue. Your AccountSync DTS then pulls records off the ScribeIn queue, and executes the following logic:

The logic for the decision whether to update the record or create a record would use an XREFLOOKUP function such as:

IF(ISERROR(XREFLOOKUP_BKEY("CUSTOMER", S1), TRUE(), GOTOSTEP(3))

The problem can arise if two events for the same record are written to the ScribeIn queue in rapid succession. In this case, message processor #1 and message process #2 may attempt to synchronize the same company record at the same time:

In this case, message processor #1 picks up the first message and examines the key cross reference table to determine whether that record already exists in the target system. Since this is a new record, the answer comes back as no. At that exact moment, message process #2 picks up the second message for the same company. It also determines that the company does not yet exist. Message process #1 then creates a new record in the CRM system. Message processor #2 also creates a new record in the CRM system. The result is that a duplicate record has been created in the CRM system.

While this can happen, in most situations its occurrence is exceedingly rare – it depends first on duplicate events having been recorded in the message queue in rapid succession, and second on the two message processors processing the events at almost exactly the same time. But it can happen. For example, OpenMind idea #303 advocated dedicating a message processor to a specific message type in part in order to avoid this scenario. We think we came up with a better solution.

Insight 7.0 provides semaphore logic to avoid this situation. In Insight 7.0, when message processor #1 performs an XREFLOOKUP, Scribe will create a row level lock for the record in question in a special table in the internal database called KEYCROSSREFERENCE_LOCK. Insight will clear the lock automatically when message process #1 finishes executing the DTS for that record. This is a new table that was added in Insight 7.0 and its sole purpose is to act as a locking semaphore. If message process #2 grabs another record which is different from the message processed by message processor #1, message process #2 will process that record right away. However, if message processor #2 attempts to process an event for the same record as message process #1, message processor #2 will be blocked until message processor #1 completes:

With this new behavior, there is no longer a situation in which multi-threading that utilizes the key cross reference logic to determine whether to create a new record can result in duplicate messages being created. At the same time, the performance of the integration is able to benefit from multi-threading in all cases except those few in which the danger exists for duplicates. In those situations, the message processor logic will automatically be sequenced to avoid the possibility for duplicate records being created.

Scribe has created several new functions for working with this logic:

XREFLOOKUPLOCK_AKEY() and XREFLOOKUPLOCK_BKEY() are used to return values from the KEYCROSSREFERENCETWOWAY table. They work exactly as the XREFLOOKUP_AKEY() and XREFLOOKUP_BKEY() did in Insight 6.5.x except that these new methods employ the locking logic as described above.

XREFLOOKUPNOLOCK_AKEY() and XREFLOOKUPNOLOCK_BKEY() work exactly like XREFLOOKUP__AKEY() and XREFLOOKUP_BKEY() did in Insight 6.5.x. They do not perform any locking logic. If you are not using message queuing and thus are not worried about the duplicate record scenario as described above, you will get a minor performance gain by using the "NOLOCK" version of these functions.

For backward compatibility, XREFLOOKUP_AKEY() and XREFLOOKUP_BKEY() functions still exist. However, as a best practice Scribe recommends using the "LOCK" and "NOLOCK" flavors of these functions to make the intended behavior explicitly clear in your DTS. As such, the XREFLOOKUP_AKEY() and XREFLOOKUP_BKEY() functions will not appear in the list of functions in the formula editor window.

Lastly, the KEYCROSSREFERENCE_LOCK table will grow over time. If you'd like to perform maintenance on this table, you can safely delete records from this table at any time without impacting integrations.

I hope this new logic will be useful to you. As always, I look forward to you feedback.

Bob

Steelhead Release Candidate Available for Download

We have just published Release Candidate #1 for the Steelhead. The download includes:

• Scribe Insight 7.0
• Scribe Adapter for Dynamics CRM 4.4
• Scribe Adapter for Dynamics GP 4.1
• Scribe Adapter for Salesforce 2.5
• Scribe Adapter for SalesLogix 4.9
• Scribe Adapter for WebServices 1.2
• Scribe Adapter for ACT!4-6 7.0
• Scribe Adapter for ACT!2007-2008 7.0
• Scribe Adapter for Exchange 7.0
• Scribe Adapter for GoldMind 7.0

Scribe Adapter for Dynamics NAV 1.4 and Scribe Adapter for Dynamics AX 1.1 are not part of the RC1 release. Preview versions of those adapters will be made available in the near future.

Release candidate #1 is released as part of our technical preview program. Access is open to all Scribe partners and customers. Details on how you can download the RC1 release can be found at http://blog.scribesoft.com/2010/05/get-early-access-to-insight-70-with-the-steelhead-technical-preview.html.

Technical Preview 2 for Scribe Insight v7.0

In a previous post, I outlined our Scribe Insight v7.0 pre-release program; what it is & how to participate.  We are getting closer to the Technical Preview 2 hand-off & wanted to make a "last call" for anyone who would be interested in participation.

We have quite a few participants already. There are a few cases of users who want to get a heads-start on solving real-world problems with the new Multi-target feature set.  However there are others who are just looking to get an advance copy to educate themselves with the version. And still others are looking to take advantage of GP2010 support - or Salesforce.com's new endpoint (winter 2010; v18.0).

So - there is no "right answer" to tell if are you qualified to participate or not.  The more hands-on, the better. If you are interested in participating, please send an email to productmanagement at scribesoft.com (I've obfuscated the email a bit to try and fool those pesky bots out there). The hand off itself is this Thursday May 6th - so don't wait!

A Tour of Some of the Changes in Insight 7.0

I'd like to take this opportunity to show you some of the major new features in Scribe Insight 7.0 and how the process of working with Sources and Targets will change accordingly.

One of the main features of Insight 7.0 is the ability to use multiple targets for a single job. You can read more about how you might use this new capability in the earlier blog post "Announcing Insight 7.0". In order to provide the best user experience for this feature we have made changes to the process of connecting and configuring the source and target. I'd like to take you through some of these changes.

Consolidation of Connection and Configure Buttons

The first thing you'll notice is that the source and target connection buttons are missing. The process of connecting and configuring a source or target has been consolidated from separate connect and configure options in Insight 6.5 (Figure 1) to a single option to configure in Insight 7.0 (Figure 2). Connecting to the source or target is now an option on the configuration dialog.

Figure 1 - Insight 6.5 has separate connection and configuration buttons

Figure 2 - Insight 7.0 has configure step button

 

Connection Management

A connection manager has been added to centralize the setup and maintenance of these multiple connections (Figure 3). You can access it from the View menu or by pressing the "Plug" icon on the toolbar. All of the connections created for this DTS are displayed; you can select a connection and change its settings or delete it. Now you can setup all of your connections here at once if you like.

One of the nice additions is that you'll be able to provide your own name and a comment for each connection (Figure 5). For example, if you are accessing two different Dynamic CRM systems for a multi-tenant scenario, each connection can have it own unique name "CRM ABC Company" and "CRM XYZ Company".

Figure 3 - Insight 7.0 Connection Manager

 

Adding a Connection

Let's add a connection to get a feel for using this new feature. Pressing the "Add..." button displays the familiar connection selection screen. For this example we're going to select the "Scribe Sample" ODBC Data Source.

Figure 4 - Adding a connection in Insight 7.0

 

Once you've made your selection and entered the connection information you are brought to the new "Connection Settings" dialog (Figure 5). This is your one stop shopping for all connection options. When adding a new connection a unique name is provided for you – in this case it was "Scribe Sample" but I didn't like that so I changed it to "My Scribe Sample Data" and added a comment. To complete adding a new connection press OK. That's it…now you have a named connection that you can easily reference anywhere in the product.

Figure 5 - Naming a connection in Insight 7.0

 

Changing the Connection Settings

Very often after a connection is created some of the information needs to be changed such as the password or connection reuse. In the past this information was located in different areas of the product. Now it is consolidated in one place (Figure 6).

Changing connection settings has moved from the Source and Target Connection option to the "Connection" tab. Connection Reuse and Optimization have moved from the "File Properties" menu option to the connection tab.

Figure 6 - Changing connection setting in Insight 7.0

 

Assigning Pre and Post Job Scripts

The "Edit Settings…" option on the "Run" menu was the place to go to assign a script to before and/ or after the job was run (Figure 7). Now, because of the possibility of multiple targets, it is now defined per connection on the "Scripts" tab (Figure 8) rather for the source, target and Scribe internal database.

Figure 7 - Script assignment in Insight 6.5

Figure 8 - Script assignment in Insight 7.0

Configuring the Source

The source configuration dialog now has a drop down to select the named connection from a list of available connections (Figure 10). Now that connection creation is independent from its usage, the source connection can be assigned and changed without having to enter connection information each time. Another nice feature is that the list of connections only allows the user to select a valid source connection. For example, if an XML connection is already being used as a target, the user will not be able to select it as a source since it can only be used as a source or target but not both. If you don't see the connection you want to use select press the "Connection…" button and you can add a connection in the Connection Manager.

Figure 9 - Source Configuration in Insight 6.5

Figure 10 - Source Configuration in Insight 7.0

 

Configuring a Target… I Mean Steps

In the multi target world you are no longer configuring a target – you are really configuring steps. Now you can select multiple targets from a drop down list of available connections (Figure 12) in the Step Configuration dialog. The Data Object tree now contains a node for each target from which steps can be assigned (Figure 13). The connection list behaves like the source in that it only allows connections that can be valid targets to be selected.

Figure 11 - Configuring a Target in Insight 6.5

Figure 12 – Selecting targets in Insight 7.0

Figure 13 - Configuring steps in Insight 7.0

 

Filtering the Mapping View

When setting up data mappings it's very useful to filter on a step (Figure 14) to avoid confusing which fields need mapping. Now with multiple targets this can become even more difficult so target filtering has been added above the step filtering option (Figure 15).

Figure 14 - Filtering in Insight 6.5

Figure 15 - Filtering in Insight 7.0

That wraps up our tour for this post. I hope you like these new features and that they provide a more streamlined approach to developing integrations and migrations.

Security for Scribe Insight

I would like to discuss the various security considerations when using Scribe Insight. As a long time veteran of Scribe, I have found that as Scribe is integrated into enterprise environments, access to various resources within the customer's network and the cloud becomes increasingly more difficult due to security measures taken by network administrators.

In addition to Insight, it is important to understand any additional security models around specific applications that you will be connecting to, such as Microsoft Dynamics products (CRM, GP, NAV, AX), or any of the many applications for which Scribe provides an adapter. Understanding Insight security should allow an application administrator to easily connect Scribe products to their applications.

Understanding how Insight interacts with the various security systems is critical to troubleshooting. Today I will discuss the core Insight security items. These include…

• Windows Vista and Windows Server 2008
• Scribe Internal Database
• Scribe Services
• Message Queues
• Scribe Console User Group (SCUG)
• Scribe Console

Windows Vista and Windows Server 2008

The introduction of UAC (user Access Control) in these operating systems, has made it very difficult for enterprise applications such as Insight to function without heavily customizing the UAC permissions settings. This is because applications like Insight touch so many facets of your operating system… ODBC, SQL Server, directories with DTS files and scripts, MSMQ, IIS, Services, etc….

At this time Scribe does not support using the application with UAC enabled. UAC must be disabled prior to installing Insight.

Stay tuned for information on how to leave UAC enabled with some custom settings in future releases of Insight.

Scribe Internal Database

After creating the Scribe Internal Database (ScribeInternal), you will notice that the objects created in the database are created within a schema called 'Scribe'. This schema exists regardless of whether you chose to use SQL Authentication or Windows Authentication during the setup of ScribeInternal. All of the tables in this database will be prefixed with the schema name. Underneath the covers, Scribe Workbench and Scribe Console reference this prefix heavily.

SQL Authentication

If you installed ScribeInternal using SQL Authentication, you can view the properties of the Scribe schema and see that the 'Scribe' schema is owned by a SQL user named 'Scribe'. The 'scribe' user is created during a SQL Authentication install with a default password 'integr8!'. These screenshots show the properties of the schema and the user.

It is important to note that the Scribe schema is owned by the Scribe user. This may seem obvious, but this is very important to remember if you ever wish to revert from 'Windows Authentication' back to 'SQL Authentication'. After making the changes, some admins forget to actually make the new user an owner of the Scribe schema.

Changing the Default User and Password

For obvious security reasons, SQL admins may wish to change the default password of the Scribe user from 'integr8!'. In order to continue to have access to the internal database, Scribe must be made aware of this change. To do this, there is an executable in the Scribe program files directory called 'InternalDB.exe'. This program allows pointing Scribe to the correct ODBC source containing the internal database, as well as changing the password that Scribe uses to access it.

On the second tab, hit the 'Change Password' button to access the password change dialog.

This screen is self explanatory. If your db admin has already changed the password in SQL, you need only change the password locally for the sake of Scribe. If the password has not yet been changed on SQL Server, you may make the change in one fell swoop by also providing an admin login and password.

Windows Authentication

If you installed ScribeInternal using Windows Authentication, the Scribe schema will still exist, but the owner of the schema will be 'dbo'. The Scribe user is not created.

When using the Workbench, access to the internal database will be under the permissions of the user that is logged into Windows. You will not be able to run the workbench unless your Windows user has been given explicit access to ScribeInternal within SQL server. By default, the installer user is mapped to the 'dbo' user in the database.

If you ever wish to disable Windows Authentication on ScribeInternal, make sure you create a Scribe user that has privileges as outlined in the SQL Authentication section above.

Scribe Services

Workbench vs. Console

As I stated earlier, when using the Workbench with Windows Authentication, access to the internal database will be under the permissions of the user that is logged into Windows. If you are using SQL authentication, ScribeInternal will be accessed using the Scribe user permissions. Accessing directories with scripts and DTS files will be based on the permissions of the user that is logged into Windows.

But what about the Scribe Console? If the collaboration folder containing scripts and DTS's exists on a network share, how does the console gain access to the files if the user with permissions to the folder is not logged in? It is important to understand that although YOU may be able to run a DTS from the Workbench, the console may not. The console operates under its own security context.

There are 5 Scribe services installed in the Windows Services Control Panel. These services run all of the features that you have set up in the Scribe Console (Monitors, Alerts, Publishers, and Integration Processes). By default these services log themselves in as 'Local System'. If you r database server and all of your collaboration files exist on the same server as Scribe, 'Local System' most likely has access to everything it needs to run your console integrations. If you find that you receive errors indicating that DTS's files don't exist or SQL scripts can't be run, it is a very good bet that you need to change the log on user associated with these services.

This user should be an Active Directory user with all of the appropriate network permissions. A good troubleshooting technique is to start with domain admin privileges and move backwards from there. This allows you to see immediately that domain admin allows the Scribe processes to run, and that you are experiencing permissions issues. Conversely, you may be chasing a red herring if you attempt to 'baby step' the permissions up until you have liftoff. (If you call Scribe Support, you WILL be asked to log these services with domain admin privileges to troubleshoot)

Always change all 5 services to use the same log on. You will cause way too much confusion if you try to get cute with the services permissions.

Windows Authentication

If you chose to use Windows Authentication when you installed ScribeInternal, you MUST immediately change the 5 Scribe services to log on as an Active Directory user with 'dbo' permissions to the ScribeInternal database. The Scribe services will not have access to the internal database as 'Local System'

Message Queues

The use of Microsoft Message Queues is what allows Insight to multithread large scale integrations, up to 64 simultaneous processes for enterprise licenses.

After installing Scribe Insight, you will see 3 new message queues appear in the Message Queuing control panel. ScribeDeadMessage, ScribeIn, and ScribeRetry.

If for some reason, you installed MSMQ after you installed Scribe Insight, these message queues will not be present. You can manually create these queues and set the 'SYSTEM' permissions as they appear here.

Security on these queues is already set for use by the system. If your integration is having trouble accessing message queues, check the properties of these queues and add the Active directory user that the 5 Scribe services uses, and add 'Full Control' permissions.

Scribe Console User Group (SCUG)

After installing Insight, you will notice that there is a new user group on the Scribe server. The Scribe Console User Group allows the members to access and control the Scribe Console.

The Scribe Console is used to automate your entire integration solution. So while your organization may have several 'workbench users' who create integration packages, you will probably want to restrict access to the automation tools to administrators.

After adding a user to this Windows group, they must log off of their session and log back in for the permissions to be enabled.

This is a very common initial install step that is skipped, resulting in an unnecessary call to Scribe Support.

Scribe Console

Once your user has been added to the SCUG, you should be able to open the Scribe Console without being prompted to create a site. Here, there are two main areas that deal with security.

Security Settings

Browse to '(server name)>Administration>Security'. Here you will see 3 tabs which allow the Console and Services access to areas of the server.

File Management

Here is where you allow your Console and Services access to files in your environment. You may need your DTS's and integration process to access SQL scripts, Source text and XML files, and batch scripts in addition to the DTS files. By default, this setting is set to allow the console access to all of the files on the Scribe server. Notice that mapped drives are NOT listed here. The Scribe Console has no access to mapped drives by drive letter. This default setting is thus limited because it cannot access network resources.

To limit the Scribe Console access to certain areas of the file system, choose the option to select specific folders. Select only those folders on the Scribe server to which your integrations will need access. Keep in mind any ODBC sources that utilize text file drivers. The directories needed by these ODBC source will have to be included here.

Once you have selected all of your server directories, you may also add a list of network resources by using the UNC path option. Simply type the fully qualified path to the shared resource and add it to the right side of the screen.

Keep in mind that although you are setting up resources the Scribe Console can manage, the console is still limited further by the permissions of the account that the 5 Scribe services are logged in as. When setting up access to network resources, make sure you think about network shares, permissions on the network shares, domain trusts, one way trusts, and time limited accounts. Active Directory Accounts can be limited to access during certain times of the day. So although you may be able to set up and run your tests during the work day, the Scribe Services may not be able to access the necessary resources at midnight on Saturday due to corporate security policies. It is always best to ensure that the services account has access 24/7 for maintenance reasons.

Services

If you have a need to restart any services that your DTS's or integration processes may need, for instance the NAV NAS service when using the Scribe Adapter for Microsoft Dynamics NAV, this tab allows you to select services that can be controlled from within Scribe Console rather than moving to your Services Control Panel.

Message Queues

This tab allows you to add MSMQ queues for use with the publisher feature. You need to make sure that all 3 Scribe queues are added to the Shared Message Queue side of this screen. Any additional queues that you wish to manage from within the console will need to be added as well. I.E. Microsoft Dynamics CRM or NAV queues.

Site Settings

Browse to '(server name)>Administration>Site Settings'.

Collaborations and Queues

Here on the General tab you may need to change the Collaborations root. This path should exist in the list of Shared Folders you set up under the File Management Security tab. As long as you are using the default queues suggested by Scribe for input, retry, and dead messages, the Site Main Queues settings will be automatic.

Note: You can change these queues around for troubleshooting and recovery purposes. For instance, if you have thousands of messages in ScribeDeadMessage, you may want to set the 'input queue' to ScribeDeadMessage temporarily in order to process those messages without having to manually transfer every message to ScribeIn.

Conclusion

Understanding how Insight interacts with the various systems on the Scribe server and the network domain will prevent a lot of unnecessary panic and allow you to save that call to Scribe Support for a more pressing matter.

Using Command Line switches to reuse existing work

Did you know that you can change which database you are integrating to via command line parameters? Or how about set values of variables you’ve created right from the command line? ;The command line switches available to you can allow you some very flexible options that will make creation and maintenance of your integration solutions much easier… I’m going to outline a few scenarios here – but please feel free to suggest others by adding comments to this post.

Supporting Multi-Company Scenarios

In some ERP systems (like Microsoft Dynamics GP), each company’s data is kept in separate databases (usually on the same server). It’s not uncommon for an integration to be applied to each company; however many folks resort to copying their integration and changing the connection information for each. ;However… any changes to the integration must be manually applied to all the copies, which can be a hassle. Enter Command Line parameters!

Using command line parameters, you can direct your integration (e.g. DTS file) to connect using different connection parameters. For example, the /TD= switch will change the database parameter for the “target” connection. Similarly, /SD= would change the “source” connection. Using this feature will allow you to create and maintain one DTS file; and run the same integration against different company databases:

 C:\>TWorkbench.exe “C:\Data\MyIntegration.dts” /TD=TWO
 C:\>TWorkbench.exe “C:\Data\MyIntegration.dts” /TD=ABC
 C:\>TWorkbench.exe “C:\Data\MyIntegration.dts” /TD=OB1

Voila! One DTS & multi-company support all through a few simple keystrokes. In this example, the same integration will be run against all 3 company databases (TWO, ABC and OB1).

But what if the company name is actually part of the XML document I am integrating?

You can combine some predefined keywords available to integration processes with command line switches to support a wide scenario of situations. In this case, the %XPath keyword can be combined with command line switches to leverage content in an XML document with a command line switch:

 C:\>TWorkbench.exe “C:\Data\MyIntegration.dts” /TD=%XPath=//Root/@CompanyID

This XPath expression “//Root/@CompanyID” will be evaluated on the source xml document and applied as the target database via the /TD switch.

What was the file that started the integration?

With the Scribe Console, you can set up your integration solution to start when a file is added to a certain folder (by using an integration process of type “File”). If your integration process uses wild cards for the name or type of file (e.g. *.pdf) that trigger this event, it may be useful to know what the file name actually was as part of your integration solution.

For example, suppose that orders are created in a .pdf file –where the name of that file was the order number (e.g. Order1501.pdf). Your integration solution should start once the any pdf is added to a specific folder – however you want to know the name of the pdf file (for obvious reasons). Never fear – you can pass the name of the file to your integration through the use of - - you guessed it - - command line parameters. In this example, I’ll assume that you’ve created a user variable of type “Prompted” called PDFFileName. In your DTS file, you should expect the fully qualified file name that triggered the integration. You can then use the /V command line switch to set the value of a prompted variable; and the %Filename variable available to integration processes to send the name of the file to the PDFFileName variable:

 /VPDFFileName=%Filename

Using prompted variables to allow you to capture the name of the file with you can then parse to find the order number within the DTS itself.

Integration processes and command line switches

Usually, you will be using command line switches as part of an integration process vs. running the integration through the workbench. When you create your integration process, there is a spot available for you to define any command line switches:

 

In these cases, there is no need to specify the DTS file or the workbench executable (TWorkbench.exe). Only the switches themselves are needed (e.g. “/TD=ABC” or “/VPDFFileName=%Filename”).

Searching the Insight help on “Command Line” and “DTS Parameters” will outline the supported parameters and related keywords. Happy scripting!

Recorded Webcast; Scribe Adapter Version 4.0 for Microsoft Dynamics GP

Scribe VP of Development Bob Sturim and Product Manager Tom O'Brien recently held a webcast to demonstrate and discuss the newly released Scribe Adapter for Microsoft Dynamics GP. To view the recording of our recent webcast click here.

The Scribe Adapter version 4.0 for Microsoft Dynamics GP focuses on three major areas:

1. Subscribing to data changes out of Microsoft Dynamics GP: Just as with other Scribe adapters, the latest Scribe Adapter for GP adapter allows users to subscribe to changes from the target system through an easy point‐and‐click user interface. Subscribing to these changes in real‐time allows them to propagate throughout the organization; keeping all of the systems in‐synch with the same data, with little to no custom code.
2. Improving usability: When using Microsoft Dynamics GP as a source or as a target, users will be able to interact with Dynamics GP data by referencing the same fields as they would within the Dynamics GP user interface.
3. More integration points: Users will to be able to make use of new integration points added in eConnect 10.0; specifically the Fixed Asset module's Asset Book item.

To read the adapter overview document click here

The Health of your Scribe Insight system

Hi, my name is Joe Dotson and I'm the Software Development Manager here at Scribe. I've been on the dev team since Scribe's beginning, and I've seen and been involved in a lot of changes in the Insight products over the years. And with healthcare all abuzz in the news lately, I thought I'd tell you about a few of the Insight system "healthcare" features that have been added over the past few years.

Failover

Most recently, in Insight v6.5, Scribe has added support for Failover environments. Scribe Failover uses a two-node active/passive configuration of Microsoft's Clustering Services. In this configuration, Scribe services are installed on two nodes in the cluster, however only one is active at any given time. The Microsoft Cluster Server periodically checks the health of the active Scribe node. If, for some reason, the active node is unavailable, the passive Scribe node is made active. For seamless transition between the active and passive nodes, these nodes need to share information. The active and passive node solution concept is not a backup or snapshot strategy. Instead, both active and passive nodes share the same information. This information is managed by Microsoft's Clustering Services through its shared resources.

The Failover feature of Scribe Insight is only available with Enterprise licenses of Scribe Insight. The Failover feature is not available for non-Enterprise licenses. See the Scribe Insight Installation Guide to learn more about Scribe Failover.

Services Auto Recovery

In non-Failover configuration, the five Insight services (AdminServer, EventManager, MessageServer, MonitorServer and BridgeServer) have all been modified to check for the well-being of the connection to the Scribe Internal Database ("IDB"). This feature was added in Insight v6.0.9. As you probably know, the IDB contains, among other things, all of the metadata for Integration Processes, Monitors, Publishers/Bridges, Execution and Transaction Error information, licensing details, etc., that are the heart of the integration system. If the connection to the IDB, which each of the services maintains, is lost or interrupted due to a number of environmental issues (network and/or network traffic problems, SQL Server service failure, etc.), the Scribe services had previously gone into a tailspin and ultimately failed. Now, the services periodically check the health of the IDB connection and gracefully shutdown. And coupled with the fact that these services are now installed with auto-restart options configured, the services will attempt to restart in a repeating fashion until the connection to the IDB is available.

The services are installed and configured as follows:

• restart on first and second failure
• take no action on subsequent failures
• reset the failure count after one day and to wait 10 minutes before restarting.

The snapshot of the Windows Services Properties dialog for the AdminServer, included below, shows these default settings; the same settings are used for all 5 services. These properties can be changed in the Windows Services applet; you may want to review these settings and adjust them for your requirements.

Process Auto Recovery

Another system health-related topic I'd like to share with you is the Auto Recovery feature for Integration Process, Monitors, Publishers and Bridges. This feature was also added in Insight v6.0.9. Settings for this feature are configured in the Insight Console: navigate to the Administration node, and then to the Site Settings node. You will notice a fifth tab on the far right – "Auto Recovery" – I've got it selected in this screenshot:

Here, there are 3 choices:

• No auto recovery
• Attempt auto recovery for N hours/minutes
• Attempt auto recovery indefinitely

No auto recovery – this is fairly straight-forward; Insight will not attempt any automatic recovery of the processes mentioned (IPs, Publishers, Monitors) beyond what you can already define for the specific processes (more on that exactly a bit later.)

Attempt auto recovery for N hours/minutes – here you can tell Insight how long you want the system to try to get those failed processes to work again. If a process fails for any reason (lost network connection, file not found, etc., there could be many reasons for process failure), Insight will try the process again after a pre-defined wait period, specifically, at increasing intervals of 1, 2, 4, 8, 16, and 32 minutes, and every 60 minutes thereafter until it reaches the end of the user specified period. When the process is successfully run, the recovery interval will be reset to start at 1 minute on the next failure.

At each failed attempt, normal alerting for the process will occur as defined with the following exception: when recovery attempts have been tried without success for the user specified period, an alert will be raised indicating that no more recovery attempts will be made and that user intervention is required.

Attempt auto recovery indefinitely – this option is similar to the previous except that the recovery attempts do not end until success is achieved; ultimately, the system retries every 60 minutes.

These three system-level auto recovery settings can be enabled or overridden for each Integration Process, Publisher/Bridge, or Monitor defined in your system. In the Console, the "Activation" step in the each component's properties "dialog" contains 3 options for auto-recovery (for Integration Processes, go to "Step 4 – Activation"; for Publisher/Bridge and Monitor components, go to "Step 5 – Define the Activation Properties.")

The Auto Recovery settings determine how the process reacts if it encounters an error that prevents the process from running.

• Select No auto recovery to have the process go immediately into a failed state.
• Select Use system settings to use the auto recovery settings configured in Site Settings as described above.
• Select On schedule to have the process attempt recovery on the schedule defined for the process.

Note:  The Auto Recovery option is enabled for File, Time, and Query Integration Processes and disabled for Queue Integration Processes. Also, this option is enabled for Query Publishers and Email Bridges and is disabled for application publishers.

So What's Next

OK, so some of these features were added back in 2007 – "do you have anything new planned for me" you might be asking. Good question and the answer is yes. We are closely tracking new feature requests – we call them "Ideas" – in OpenMind. We encourage you to review existing Ideas, add your own Ideas, and vote for those that are most important to you. We are currently devoting the next release of Insight, codenamed "Steelhead", to the top vote-getters currently in OpenMind – so your Ideas and Votes definitely count!

Some of the Ideas we are working on in Steelhead:

• 7: Allow multiple target connections
• 66: Allow lookup into a database outside of the source, internal, or target.
• 142: Using Update Source with Custom Queries
• 156: Improve the update source functionality to not be so quirky
• 215: Update multiple source fields
• 249: Place "something" on the screen while Workbench is starting
• 280: Add support for Windows 7
• 306: Workbench Test Dialog should remember the current window state
• 239: Always create the MSMQs if they are not there when the MessageServer starts
• 139: Start new message processors only if a reasonable amount of system resources are available

And as for healthcare features, these are the Ideas we are considering for future releases:

• 315: SFDC Adapter - retry connection when it's down...
• 250: Enable email alert if Scribe loses communication with its internal database
• 192: Provide a error threshold to define when a DTS should stop executing
• 163: Support for failover on Windows 2008
• 152: Add more granularity to alerting notifications
• 94: System and Configuration Audit/Test
• 34: Get Insight to automatically reconnect DTS's after a SQL reboot where SQL is on a different box

That about wraps up my discussion on Insight system healthcare, so until next time - be well.

Support for Windows7

We just completed a survey in OpenMind which asked:

Regarding Windows 7…

• I plan to upgrade immediately. Scribe should support it ASAP

• Windows 7 is not a high priority for me at this time

Of 249 respondents, 41% indicated that they planned to update immediately.

This result was somewhat surprising to me. Frankly, I had not imagined that so many business users would jump to a new version of an operating system so quickly. All the same, I was happy to have my assumptions invalidated – OpenMind is all about using feedback from our partners and customers to make better product decisions.

Based upon your collective feedback, we have decided to add support for Windows 7 to Steelhead release of Scribe Insight. Steelhead is schedule for release in Q1 2010.

Your feedback is welcome.

Bob