Scribe Software Blog

Multi-Target Use Cases: an alternative approach for product catalog integration between Dynamics NAV and Dynamics CRM

In this post I want to share an idea with you I came about during my experimentation with the multi-target feature of Scribe Insight 7. My company, Succentric, specializes in the support and enablement of both new and existing Scribe partners, particularly regarding Dynamics NAV and Dynamics CRM. As such we get a lot of feedback around the topic of the Adapter for NAV. When the NAV/CRM template was released back in 2009, one of the first remarks we got was this: This is a great solution, but what about customers with large item master data?

Those of you that are familiar with the NAV/CRM template will know that in order to match the different data models of Dynamics NAV and Dynamics CRM a custom table was built in NAV that would store all combinations of items, variants, and sales prices available the NAV Company, calculated by Scribe Integration processes called ScribeSalesPrices. Let's imagine a company with just 125,000 products with on average 5 variants and also on average 3 different prices per variant. That'll give you 1.875 million records in this table. It is not prevalent scenario, but it is quite obvious that if the situation is like this, we could get into trouble with this approach. At the time the template was originally built, there was scarcely an alternative. But the multi-target-feature introduced in Scribe Insight 7 allows a completely new approach, which also may fit to other scenarios where the differences in the data model require a high degree of calculation and/or expansion of the data during the integration.

The following text is not a step-by-step guide, but rather an outline to discuss the possibilities and it will require considerable NAV skills to implement. However, it may be inspiring to some of you, and it shows the direction to a solution for a whole class of complex problem.

If we have a static list structure on the CRM side, and a dynamic, engine-driven structure on the ERP side, we have to execute the ERP logic in order to build all the data as a static snapshot. But can't we avoid storing the result in an extra table just for the sake of integrating it? What we would need to do is execute the price calculation for every n-tuple from inside a DTS, and immediately write the result to the CRM price list item entity. In our example this can be achieved in the following way.

First, we have to design a new DTS that would use the "Sales Price"-Table of Dynamics NAV as a source. This table contains a record with a sales price for each combination of

• Item
• Item Variant
• Price Group or Customer Code
• Date
• Currency
• Unit
• Minimum Quantity

It is also used by the current NAV/CRM template, but the result of the integration process that is processing those records is just written back to NAV. But actually the Sales Price table provides all we need to create a sales order, which is the most convenient way to execute NAV's pricing engine. We can retrieve the price as it was calculated by Dynamics NAV immediately after the execution of our step by storing the "Unit Price" field of the Sales Line table in a target variable. As a result, we'd get a particular price that would be valid as a specific CRM price list item, and with the multi-target feature, we can within the same DTS both create a sales order and price lists and price list items on the target side. And we can get completely rid of the ScribeSalesPrice table that can become so large.

Now you may say, "That's great, but wait a sec. What about all those useless sales orders we create all around the place? We will have to delete them afterwards, which is costly in terms of NAV system resources, and it will leave huge gaps in our sales doc. No. series." That is why we would only create those sales orders temporarily that means without actually storing the Header and Line records to the database and without creating a document no. Doing this is actually a Dynamics NAV standard procedure that is used in lot of circumstances if specific prices are needed.

To do this with Scribe, we have two options. The first one is less work, but it will require a change in the NAV adapter C/AL code, in the adapter's NAV objects. We would enhance the adapter to allow us to choose if a step would create permanent or temporary records. To do this we have to perform the following actions:

1. Add a field to the Sales Header table that acts as a semaphore we can use to indicate what we want to do. It can be any data type really, but an option field will probably be the best fit.
2. Change the code in the function "InsertRecord" of Codeunit "Scribe Adapter commands":
   
   IF (<Use a function here to retrieve the value of the XML-Tags representing our semaphore from the XmlDocument-variable here> = 'yes') THEN BEGIN
   RecRef.OPEN(tmpInt,TRUE);
   xRecRef.OPEN(tmpInt,TRUE);
   END ELSE BEGIN
   RecRef.OPEN(tmpInt);
   xRecRef.OPEN(tmpInt);
   END;
   
3. Make sure that we apply a dummy value to the No. / Document No. fields of the Sales Header / Sales Line table in the data mapping of our Insert steps.

The beauty of this option is that it is less effort to implement, but the downside is that you have to modify actual adapter code. Only those that have high level of NAV coding experience should do that. You should make this decision very explicitly by weighing the gain and the risks.

The second option gives you more security and you omit the risk of screwing up your adapter, but it'll mean more effort. However, I would strongly recommend this option to most users. The idea is simply to code a custom action that calls a custom Codeunit that creates the temporary Sales Order, and returns the calculated line values as they are needed. To do this we have to perform the following actions:

1. Following the Steps in the Help file for the Adapter for Dynamics NAV (Section "Working with actions"), create a new Codeunit. Prepare a "GetInputParameters"-Function as described in the help file, and make sure that all of the above listed parameters are distributed to global variables.
2. Create your "ExecuteCustomAction"-Function and put your code for creating your Sales Header and Line there. Make sure that you are using temporary variables for Sales Header and Sales Line, and that you predefine a Sales Doc. by filling a value in the field "No." of the Sales Header (can be '0' for example). Also, you need to create your Sales Line in the following way:
   SalesLine.SetSalesHeader(VerkKopf);
   SalesLine.INIT;
   SalesLine."Line No." := SalesHeader."Line No." + 10000;
   SalesLine."Document Type" := SalesHeader."Document Type";
   SalesLine ."Document No." := SalesHeader."No.";
   <Your code to fill in Sales Line Details like Type, No., Qty. and so on>
   SalesLine.INSERT;
3. Prepare your "SetOutputParameters"-Function as described in the Adapter help, and put all relevant values from the Sales Line table record there.
4. In your DTS with the Sales Price Table as Source, add an Execute Step to call your custom action.

I hope this information about these two approaches will help you determine the best strategy for your NAV integrations that use Scribe Insight. As part of my participation the October edition of the Scribe Inside Track web meeting, I led a discussion about these approaches of using the Adapter for NAV.

The Five Capabilities of a Collaborative Integration Platform – 4. Supportability

Last time I discussed the need for easy transferability of the ongoing management of integration solutions from development to production. I will now discuss the fourth capability essential to a collaborative integration platform, Supportability.

Moving an integration process into production introduces the same set of challenges as moving any application or solution out of QA into production. Users always use application in ways that were not contemplated by the original integration design, resulting in unexpected errors and exceptions. Modifying the design of the end point applications after the integration is implemented may disrupt the integration processes. Changes to the overall IT environment may interrupt connections between the various applications. In short, business happens and things stop working or not work as intended. Your integration approach should include a robust set of error detection, diagnosis and remediation tools.

Is there a way to proactively monitor the health of the integration and raise alerts when things are not functioning properly? Are there meaningful execution and error logs that enable someone to quickly identify the root cause of any issue? Does the integration process have built in re-try mechanisms that address any timing issues (for example, attempting to process an order transaction before the related customer record has been created in your target application)?

A major area to consider with any integration solution is the ability to handle gracefully upgrades to your business applications. For example, if a customer moves from Dynamics CRM 4.0 to Dynamics CRM 2011, what will the impact be on the integration? Is the upgrade simply a testing exercise where things for the large part continue to work or does it require a re-write of the integration? The best integration solutions create an abstraction layer between the integration processes themselves and the API of the endpoint applications. The process only sees the abstracted interface and when the endpoint application is upgraded, it is simply a matter of modifying the interface between the API and the abstraction layer. If you are looking at 3^rd party adapter products, make sure you interrogate this question fully. You will be glad you did later.

At the risk of stating the obvious, the integration approach needs to be very reliable. In other words, "It just needs to work". The best integration is one that no one ever knows about or notices (of course, until it stops working and then it gets a lot of attention). Integration solutions that are not rock solid are disruptive and costly to your business as well a significant drain on your resources. Users may not trust the data they are looking at and go to other methods that may be more time consuming and costly to the business. They may stop using the CRM system all together and cobble their own "system" which completely defeats the purpose of your CRM implementation and integration. Your resources that could be on other, more important projects are now working on fixing that integration – again.

In the next and final installment, I conclude with a discussion of the need for expandability.

The 5 Capabilities of a Collaborative Integration Platform – 3. Transferability

Last time I discussed the need for flexibility when designing and implementing integration solutions. In this installment, I cover the third capability essential to a collaborative integration platform, Transferability.

In many cases, the people responsible for designing and implementing the integration processes are not the same people who will be administering them once they are put into production. Integration design and development is generally performed by business and/or data analysts who gather business requirements, create a design that meets those requirements, and develop the integration itself. The design of the integration could include defining the events that initiate an integration process, capturing, filtering and/or merging the relevant data affected by the events, mapping and transforming that data, establishing rules for data ownership, and designing error capture and remediation. The implementation itself involves completing the unit and system testing to ensure that the integration processes are ready for production. The final task of the integration developer is ensuring the integration is working in production and transitioning the ongoing ownership.

This hand off of the solution is very critical to the success of the integration project. How do you transfer ownership of the solution to someone that does not have intimate knowledge of the detailed design? Are there tools and dashboards that exist that can enable this person to know when things are working well and when they are not? Is there a straightforward mechanism in place for them to repair data issues? It is easy to identify which issues they can resolve or which issues they should contact the integration developer? If the original integration developer is no longer available, can someone else with integration design skills quickly understand the integration design and make modifications or enhancements to it?

This issue of transferability is also critical to the partner. It is very costly for a partner to pull their integration designers off other their other customer projects and have them fix an issue at an existing deployment. Depending on the issue, having to go back and fix or constantly maintain the integration can negatively impact the customer and partner relationship. A good partner makes sure their customer understands the integration software, the design of their particular integrations, and that the customer can maintain it.

Next up…Supportability.

The 5 Capabilities of a Collaborative Integration Platform – 2. Flexibility

This is the next installment of the posts I started in April about the key capabilities of a collaborative integration platform. It has been over a month since I blogged about the first in the series about Reusability. Things got a bit away from me over the last five weeks but I hope to pick up the pace. So here it goes. The second big capability is Flexibility.

Flexibility

Integration requires a flexible approach. There are infinite varieties of integration needs as there are customers and business systems. One key driver of this variability is industry. The integration processes across different vertical markets can be dramatically different. Take for example, the integration of an order process between CRM and ERP in a distribution company versus a capital goods manufacturer. The integration for the distribution company is going to very transaction intensive with paramount emphasis on ensuring that the sales and services teams have up to date information on the location and delivery schedules of goods, while the capital goods manufacturer will likely focus on leveraging back office data about their customers to up-sell and cross-sell additional products and services. Integration processes can also vary greatly within industries based on differences in business models from company to company. A simple example is the difference between a direct selling and a channel selling organization.

Typical areas of variability in integration design include:

• The data design of each business application involved and the transformations needed to resolve the design differences between those business applications,
• The rules of data ownership: which system is the master of what data; which systems can insert or update certain records or fields; etc.,
• The actions that trigger integration events and the rules to process those integrations.

Beware of pre-packaged integration solutions, particularly around this issue of flexibility. They may seem a lot easier to implement, but may not give the user close to what their business requires. Even if a customer can take advantage of a pre-packaged integration solution, the solution will almost always require modification to address the customer's needs. These changes can range from minimal to extensive depending on how unique the business requirements are.

Custom code to achieve integration has a major advantage where it offers unlimited flexibility to meet a customer's needs. There is no limit to what can be accomplished provided that one has the resources with the skills and time necessary. The downside is cost in people, time, and money including the initial cost of the implementing the integration project and the ongoing costs of supporting and modifying the integration over time. The cost of custom code is always more expensive and can be out of reach for some customers. For many customers, the right answer is a third party integration tool that combines a graphical user interface, templates for common integration scenarios, and significant depth of configuration capabilities. An integration tool gives customers to the ability get exactly what they need while significantly reducing (or in some cases, eliminating) the costs and complexity of a custom coded integration.

Next up…Transferability.

 

The 5 Capabilities of a Collaborative Integration Platform – 1. Reusability

In my previous post I introduced the 5 Capabilities of a Collaborative Integration Platform. Now I'll get into the good stuff and begin with capability #1, Reusability.

Reusability

Partners invest heavily in developing the depth and expertise around the unique business requirements of their specific industry focus to provide solutions that deliver competitive advantage for their clients. A key question for any partner is how to take advantage of this industry knowledge and business process and distribute that quickly and easily to a broader set of clients without having to "reinvent the wheel" at every client site. A packaged solution offered by a partner benefits the customer by offering a lower cost and lower risk integration option, as the they gain the benefit of the partner's expertise not only in the industry but also with the integration needs typical of customers in that industry. Customers may do an integration project once or not at all; the partner brings their experience and insight from many integration projects.

This sounds all well and good, but is not so easy to implement in practice. Invariably things get in the way. First, the project teams at partners are generally under a lot of pressure to complete their project and then move on to the next customer. They do not have the time to package the solution into something that can be re-used. Even if they did have the time, not all of the elements of a customer's specific solution are transferable to another customer. There are a number of elements that are very specific to the particular deployment. There may also be unique intellectual property within the solution that the partner may want to protect. The solution may also be implemented by an individual lacking the same depth of skills as the original designer of the solution and so, requires a more streamlined deployment experience. All of these things stand in the way of achieving the great benefits of reusability.

A collaborative integration approach:

• Provides partners the ability to repackage an integration solution in a simple and easy manner. It should not add significant incremental effort to any project.
• Makes it easy to pick and choose the common elements of a solution implementation that can be packaged and re-used.
• Protects the partner's intellectual property. A partner may invest significant effort to develop unique and innovative methods or approaches to solving particular industry or application integration challenges. Without some way to protect this intellectual property, partners may be reluctant to package the integration and spend the time re-implementing the integration at each client site which increases cost and time for both the partner and customer.
• Allows customization which may be something as simple as a wizard based UI where configuration choices are driven by metadata provided by the user to the use of a full suite of integration design tools depending on the complexity and uniqueness of the integration needed.

That is all for now. Next up…Flexibility.

Introducing the 5 Capabilities of a Collaborative Integration Platform

In my previous blog post, I spoke of the changing role of the partner brought on by the advent of cloud computing. In this new world, partners are moving away from a reliance on 'systems integration' and moving towards creating value-added industry focused solutions. A critical component to these solutions is the need to help companies make sense of the integration requirements to other business applications and data sources. When developing an integration approach, it is important to recognize the different and evolving roles between the partner and customer throughout the phases of an application deployment. Even if there isn't a partner involved, in many cases IT organizations act in a similar capacity as partner to the business to deliver a successful CRM project. A collaborative approach to integration between "partner" and "customer" best serves the integration needs of companies deploying these business applications.

It is rare that one person has all of the skills and knowledge necessary to fully implement an integration solution. In a CRM integration project for example, the development of the integration requires a solid understanding of the design of the CRM system and the key customer processes that drive the integration need. It also requires detailed knowledge of the other applications, many of them complex back office applications. The ability to allow multiple people to participate in the development and design of the integration solution in a coordinated manner is essential. Each of the parties should be able to design their discrete piece and simply fold it into the broader solution. The other members of the project team should be able to easily understand the work others have done and should be able add their input and comments. Team members that have moved onto other projects should be able to easily move back into an active role.

So beyond these team development capabilities what does a collaborative approach to integration really mean? Over the new few weeks I will be exploring the 5 capabilities of a collaborative integration platform. This exploration should shed some light on the direction we are headed with our new integration platform, Scribe Online, as well as some of the continued investments we will be making in Scribe Insight. So watch for my next post as we drill down on the first capability; Reusability.

Scribe Partner Excellence Results in Customer Excellence

As Director of Sales for Scribe Software, I am continually impressed by the professionalism and dedication to customer satisfaction of our reselling partners. Nowhere is this more evident than at Microsoft Convergence when each year, Microsoft honors their Dynamics customers for their excellence in various award categories. 

This year Scribe customers won excellence awards in 7 of the 19 categories. This is no fluke, as the Scribe Partners involved in these Dynamics/Scribe implementations are not only top Dynamics partners, but are also perennial participants in Scribe’s top ten performing partners! 

The winning customers certainly deserve congratulations and recognition.  I would also like to add a special Scribe ‘congratulations’ to our partners Hitachi Consulting, McGladrey, Tectura Corporation, Columbus IT, Tribridge, and Customer Effective - their recognition and rewards are well deserved! Read more about these customers and partners and the awards won, in our press release.

Dynamics NAV to Dynamics CRM Template Video Guide

I'm pleased to announce that we have a new vBook offering available to you, a guide to implementing the integration template for Dynamics NAV and Dynamics CRM. This vBook is offered free of charge as a resource to help you through the implementation of the template with ease. You can find the link to this video on the template download page in OpenMind, or use the link here.

This vBook is designed to be used hand-in-hand with the supplied help file guide that comes with the template. By following the steps in the guide and the video we hope to provide you with the detail to get this template operational the first time through. The video walks you through visually what the guide does, but adds some additional explanations along the way.

The video covers the data sharing model used with the template, how to make use of the template as a modular integration component, and the step-by-step process of getting the template up and running. The discussion about the data model is to help you understand why some of the choices were made to design the template, and also to help you compare the template's capability to the business requirements you have. This will allow you to know where you need to modify the template to suit your specific business needs.

The template is a modular integration component in the sense that you can pick and choose what parts you want to use, and simply not use the rest. If you are just looking to move account and customer data back and forth you can review the template and implement those pieces, leaving the rest inactive. This can be a powerful starting point for you when beginning a new integration project, as the design details have been taken care of for you already.

The implementation steps we detail take you through the setup of the template. Inside the help file, we list all the prerequisites for the template, and include links to the setup for those individual requirements. Then, we walk through the setup of the template piece by piece, from localizing the DTS files to suit your CRM system, to the initial testing and syncing of data between the two systems. The end result is, by following the directions in the guide and watching the video, you should be able to follow our steps and have a working two way integration at the end.

So please, take a look at this new resource and let us know what you think. We're always looking to create more content, and want to meet the needs of our loyal users.

Scribe Adapter for Dynamics NAV 1.5 released

Scribe is pleased to announce the release of version 1.5 of the Scribe Adapter for Dynamics NAV. Version 1.5 features support for 64 bit environments, increased reliability, and various other improvements and bug fixes. 

To learn more about this Adapter and the release, please visit:

• The NAV Adapter v1.5 OpenMind page
• Download the release notes
• Read the latest Blog describing highlights of this release

Upcoming Scribe NAV Adapter release

Here at Scribe it seems we always have a couple of product releases underway at any one time (in fact it's usually closer to 4 or 5) and now is certainly no different. One of our upcoming releases is the Scribe NAV Adapter v1.5. This release will be focusing on a couple of main items:

• Support for the Adapter w/64 bit systems. The adapter had been using a Microsoft technology called "C/Front" to read data out of Microsoft Dynamics NAV. Unfortunately, this technology was not 64bit compliant. This release of the adapter will use the NAS (NAV Application Server) for both reads and writes. Users will also be able to scale performance to their needs by bringing additional NAS's to bear.
• Make Scribe NAV Adapter more stable when the NAS blurps. Occasionally, the NAS will get into an unusable/unstable state (e.g. if it lost connectivity to SQL Server). Historically, the Scribe NAV Adapter would require that you cycle the Scribe Bridge Service to get all things working again. This release will make the NAV Adapter more resiliant to these types of events by doing periodic health checks and adjusting itself as needed so that you don't need to manually intervene to get things working again.

As with all our products, these "features" are driven mainly by you - the user community.  Our goals are to make you more efficient in both new installations of Scribe Insight (as represented in the first item listed above) and to improve the product for those who are investing in our maintenance program (as represented in our second item).

User feedback is crucial to us & would like to solicit feedback from all our users. In addition to input from OpenMind, we discussed the NAV adapter with our partners and customers at the recent WWPC and from individual partners via one-on-one conversations.

If you are available for one of these one-on-one conversations and would like to influence the direction of the product line, please let us know. You can contact the product management team at our scribe email alias of ProductManagement at Scribesoft.com (I've obfuscated the email address a bit to fool those tricky web-bots). Comments attached to this post will also make it to the product management team.

Thanks in advance for any feedback!

[TOB 11/2/2010: As a quick follow up - the NAV 1.5 Adapter has been officially released and is available for download from our download page: https://openmind.scribesoftware.com/html/insight_download]