Scribe Software Blog

How to Enable the Dynamics GP to Dynamics CRM Template for use with CRM OnLine

We have just released our latest version of the Dynamics GP to Dynamics CRM template which you can read about here. Out of the box, this template supports On Premise deployments of CRM 2011, but what about CRM Online? Below you will find the steps needed to deploy this template for Dynamics CRM Online, and some of the things to watch out for.

The biggest difference between the deployments of this template for On Premise versus Online is the use of the INTEGRATION user and the impersonation feature. On Premise deployments allow you to use the impersonation feature of Dynamics CRM, which lets you make changes in Dynamics CRM as if you were logged in as some other user. Using impersonation, your Scribe jobs make changes as if they were logged in as the user named INTEGRATION, a system user that is built into Dynamics CRM. This user is allowed to make changes to fields that normally would be locked, such as total fields on a submitted order. Without this user, our jobs would be unable to make these changes. Dynamics CRM Online does not allow us to use impersonation, so we need a different way to enable the Scribe jobs to update the protected fields. There is functionality built into Dynamics CRM Online that lets you enable a user so that they can edit these protected fields. You change the user through the API and therefore through a Scribe job. So, what needs to be done first is create a DTS that uses CRM Online on both the source and target, pulling from the SystemUser object and updating the SystemUser object. There is a field on that object named isintegrationuser, you must set this to 1 on the user you wish to use as your integration user. You can make an easy lookup link on systemuserid = systemuserid.

 

The implication of deploying the integration with Dynamics CRM Online is that you cannot use the built in system user called INTEGRATION and you must use a real user, which requires a full CAL. The user can still be utilized use in CRM, but the changes made by that user will not be sent to GP, as configured in the integration template.

When you create your DTS file, I would highly recommend using the Custom Query option on the source to filter down to the one user account you wish to change. I did this by adding a filter on the internalemailaddress field for the SystemUser object, which contains the Live ID you had to use to create this CRM user. That way, you are not changing a user to become the integration user when you do not intend to. Once this has been done, run the DTS to update the user and you now have a system user with the ability to update those protected fields.

.

Now that we have our integration user defined, it's time to use it. You need to change all of the connections to Dynamics CRM in the template DTS Files to use this special user, with the exception of one DTS file. The CustomerToCRM.dts file has two connections to Dynamics CRM. You will change one of them to use this special user. For the other connection, labeled "non-INTEGRATION CRM", set it to connect to Dynamics CRM as some other user who does not have the "isintegrationuser" property set to 1.

 

The last thing you must do is change the CRM Publisher settings to ignore changes from this integration user. This is done within step 4 of the publisher settings, and we recommend doing this at the Entity Level as opposed to the publisher level. By only ignoring at the entity level, you can add your own custom published entities and allow that user to have their changes sent down via Scribe.

 

 

Aside from the modifications to the process noted in this blog post, you can use the rest of the template implementations steps in the template help. Once you have made the changes outlined in this post, you can follow the step by step directions provided with the template to deploy and test the integration between Dynamics GP and Dynamics CRM Online. You can do your testing and initial sync according to the doc and have your integration up and running in short order.

The Five Capabilities of a Collaborative Integration Platform – 4. Supportability

Last time I discussed the need for easy transferability of the ongoing management of integration solutions from development to production. I will now discuss the fourth capability essential to a collaborative integration platform, Supportability.

Moving an integration process into production introduces the same set of challenges as moving any application or solution out of QA into production. Users always use application in ways that were not contemplated by the original integration design, resulting in unexpected errors and exceptions. Modifying the design of the end point applications after the integration is implemented may disrupt the integration processes. Changes to the overall IT environment may interrupt connections between the various applications. In short, business happens and things stop working or not work as intended. Your integration approach should include a robust set of error detection, diagnosis and remediation tools.

Is there a way to proactively monitor the health of the integration and raise alerts when things are not functioning properly? Are there meaningful execution and error logs that enable someone to quickly identify the root cause of any issue? Does the integration process have built in re-try mechanisms that address any timing issues (for example, attempting to process an order transaction before the related customer record has been created in your target application)?

A major area to consider with any integration solution is the ability to handle gracefully upgrades to your business applications. For example, if a customer moves from Dynamics CRM 4.0 to Dynamics CRM 2011, what will the impact be on the integration? Is the upgrade simply a testing exercise where things for the large part continue to work or does it require a re-write of the integration? The best integration solutions create an abstraction layer between the integration processes themselves and the API of the endpoint applications. The process only sees the abstracted interface and when the endpoint application is upgraded, it is simply a matter of modifying the interface between the API and the abstraction layer. If you are looking at 3^rd party adapter products, make sure you interrogate this question fully. You will be glad you did later.

At the risk of stating the obvious, the integration approach needs to be very reliable. In other words, "It just needs to work". The best integration is one that no one ever knows about or notices (of course, until it stops working and then it gets a lot of attention). Integration solutions that are not rock solid are disruptive and costly to your business as well a significant drain on your resources. Users may not trust the data they are looking at and go to other methods that may be more time consuming and costly to the business. They may stop using the CRM system all together and cobble their own "system" which completely defeats the purpose of your CRM implementation and integration. Your resources that could be on other, more important projects are now working on fixing that integration – again.

In the next and final installment, I conclude with a discussion of the need for expandability.

The 5 Capabilities of a Collaborative Integration Platform – 3. Transferability

Last time I discussed the need for flexibility when designing and implementing integration solutions. In this installment, I cover the third capability essential to a collaborative integration platform, Transferability.

In many cases, the people responsible for designing and implementing the integration processes are not the same people who will be administering them once they are put into production. Integration design and development is generally performed by business and/or data analysts who gather business requirements, create a design that meets those requirements, and develop the integration itself. The design of the integration could include defining the events that initiate an integration process, capturing, filtering and/or merging the relevant data affected by the events, mapping and transforming that data, establishing rules for data ownership, and designing error capture and remediation. The implementation itself involves completing the unit and system testing to ensure that the integration processes are ready for production. The final task of the integration developer is ensuring the integration is working in production and transitioning the ongoing ownership.

This hand off of the solution is very critical to the success of the integration project. How do you transfer ownership of the solution to someone that does not have intimate knowledge of the detailed design? Are there tools and dashboards that exist that can enable this person to know when things are working well and when they are not? Is there a straightforward mechanism in place for them to repair data issues? It is easy to identify which issues they can resolve or which issues they should contact the integration developer? If the original integration developer is no longer available, can someone else with integration design skills quickly understand the integration design and make modifications or enhancements to it?

This issue of transferability is also critical to the partner. It is very costly for a partner to pull their integration designers off other their other customer projects and have them fix an issue at an existing deployment. Depending on the issue, having to go back and fix or constantly maintain the integration can negatively impact the customer and partner relationship. A good partner makes sure their customer understands the integration software, the design of their particular integrations, and that the customer can maintain it.

Next up…Supportability.

The 5 Capabilities of a Collaborative Integration Platform – 2. Flexibility

This is the next installment of the posts I started in April about the key capabilities of a collaborative integration platform. It has been over a month since I blogged about the first in the series about Reusability. Things got a bit away from me over the last five weeks but I hope to pick up the pace. So here it goes. The second big capability is Flexibility.

Flexibility

Integration requires a flexible approach. There are infinite varieties of integration needs as there are customers and business systems. One key driver of this variability is industry. The integration processes across different vertical markets can be dramatically different. Take for example, the integration of an order process between CRM and ERP in a distribution company versus a capital goods manufacturer. The integration for the distribution company is going to very transaction intensive with paramount emphasis on ensuring that the sales and services teams have up to date information on the location and delivery schedules of goods, while the capital goods manufacturer will likely focus on leveraging back office data about their customers to up-sell and cross-sell additional products and services. Integration processes can also vary greatly within industries based on differences in business models from company to company. A simple example is the difference between a direct selling and a channel selling organization.

Typical areas of variability in integration design include:

• The data design of each business application involved and the transformations needed to resolve the design differences between those business applications,
• The rules of data ownership: which system is the master of what data; which systems can insert or update certain records or fields; etc.,
• The actions that trigger integration events and the rules to process those integrations.

Beware of pre-packaged integration solutions, particularly around this issue of flexibility. They may seem a lot easier to implement, but may not give the user close to what their business requires. Even if a customer can take advantage of a pre-packaged integration solution, the solution will almost always require modification to address the customer's needs. These changes can range from minimal to extensive depending on how unique the business requirements are.

Custom code to achieve integration has a major advantage where it offers unlimited flexibility to meet a customer's needs. There is no limit to what can be accomplished provided that one has the resources with the skills and time necessary. The downside is cost in people, time, and money including the initial cost of the implementing the integration project and the ongoing costs of supporting and modifying the integration over time. The cost of custom code is always more expensive and can be out of reach for some customers. For many customers, the right answer is a third party integration tool that combines a graphical user interface, templates for common integration scenarios, and significant depth of configuration capabilities. An integration tool gives customers to the ability get exactly what they need while significantly reducing (or in some cases, eliminating) the costs and complexity of a custom coded integration.

Next up…Transferability.

 

The 5 Capabilities of a Collaborative Integration Platform – 1. Reusability

In my previous post I introduced the 5 Capabilities of a Collaborative Integration Platform. Now I'll get into the good stuff and begin with capability #1, Reusability.

Reusability

Partners invest heavily in developing the depth and expertise around the unique business requirements of their specific industry focus to provide solutions that deliver competitive advantage for their clients. A key question for any partner is how to take advantage of this industry knowledge and business process and distribute that quickly and easily to a broader set of clients without having to "reinvent the wheel" at every client site. A packaged solution offered by a partner benefits the customer by offering a lower cost and lower risk integration option, as the they gain the benefit of the partner's expertise not only in the industry but also with the integration needs typical of customers in that industry. Customers may do an integration project once or not at all; the partner brings their experience and insight from many integration projects.

This sounds all well and good, but is not so easy to implement in practice. Invariably things get in the way. First, the project teams at partners are generally under a lot of pressure to complete their project and then move on to the next customer. They do not have the time to package the solution into something that can be re-used. Even if they did have the time, not all of the elements of a customer's specific solution are transferable to another customer. There are a number of elements that are very specific to the particular deployment. There may also be unique intellectual property within the solution that the partner may want to protect. The solution may also be implemented by an individual lacking the same depth of skills as the original designer of the solution and so, requires a more streamlined deployment experience. All of these things stand in the way of achieving the great benefits of reusability.

A collaborative integration approach:

• Provides partners the ability to repackage an integration solution in a simple and easy manner. It should not add significant incremental effort to any project.
• Makes it easy to pick and choose the common elements of a solution implementation that can be packaged and re-used.
• Protects the partner's intellectual property. A partner may invest significant effort to develop unique and innovative methods or approaches to solving particular industry or application integration challenges. Without some way to protect this intellectual property, partners may be reluctant to package the integration and spend the time re-implementing the integration at each client site which increases cost and time for both the partner and customer.
• Allows customization which may be something as simple as a wizard based UI where configuration choices are driven by metadata provided by the user to the use of a full suite of integration design tools depending on the complexity and uniqueness of the integration needed.

That is all for now. Next up…Flexibility.

Introducing the 5 Capabilities of a Collaborative Integration Platform

In my previous blog post, I spoke of the changing role of the partner brought on by the advent of cloud computing. In this new world, partners are moving away from a reliance on 'systems integration' and moving towards creating value-added industry focused solutions. A critical component to these solutions is the need to help companies make sense of the integration requirements to other business applications and data sources. When developing an integration approach, it is important to recognize the different and evolving roles between the partner and customer throughout the phases of an application deployment. Even if there isn't a partner involved, in many cases IT organizations act in a similar capacity as partner to the business to deliver a successful CRM project. A collaborative approach to integration between "partner" and "customer" best serves the integration needs of companies deploying these business applications.

It is rare that one person has all of the skills and knowledge necessary to fully implement an integration solution. In a CRM integration project for example, the development of the integration requires a solid understanding of the design of the CRM system and the key customer processes that drive the integration need. It also requires detailed knowledge of the other applications, many of them complex back office applications. The ability to allow multiple people to participate in the development and design of the integration solution in a coordinated manner is essential. Each of the parties should be able to design their discrete piece and simply fold it into the broader solution. The other members of the project team should be able to easily understand the work others have done and should be able add their input and comments. Team members that have moved onto other projects should be able to easily move back into an active role.

So beyond these team development capabilities what does a collaborative approach to integration really mean? Over the new few weeks I will be exploring the 5 capabilities of a collaborative integration platform. This exploration should shed some light on the direction we are headed with our new integration platform, Scribe Online, as well as some of the continued investments we will be making in Scribe Insight. So watch for my next post as we drill down on the first capability; Reusability.

Revelations on Efficiency with Scribe 7 and Multi Target

One of my clients here at Infinity Info Systems has been involved in an elaborate integration between Microsoft Dynamics GP and Microsoft Dynamics CRM that I built in Scribe 6.4 and have been tweaking ever since. The client is a medical component parts supplier with over 9000 component parts in stock, and thousands more that they source on a yearly basis.

The original GP to CRM integration I designed used an hourly batch process to migrate their data from GP (SQL direct) to CRM. This, while acceptable, was not the preferred solution due to lag-time between data entry in GP and usage in CRM, as well as a lack of true net-change management.

With the additional stability offered in Scribe 6.5.2 and the GP adapter enhancements in version 3.9, the client finally agreed to implement the adapter in their production environment. Earlier this year, I moved the client to a transactional integration using the GP publisher and MSMQ for Accounts, Contacts, Products, Units, Unit Groups and Price Lists. The effectiveness of the GP Publisher instilled confidence in the client and allowed us to move forward, tackling some of the more complex problems that we were facing.

One of the more challenging tasks I was assigned was the concept of implementing a “New Product Release” process. This process would allow the client to replace an aging database used to source new products that they do not currently carry in stock. Since this customization set was already in place in CRM, the client wanted to leverage it to store these “non-inventory” items and then convert them to GP inventory when a certain threshold was met.  This would reduce repetitive data entry in multiple systems, and allow end users to work in CRM where they felt comfortable, rather than the legacy database.

The two major challenges with this process were that the client wanted the process to be nearly instantaneous using MSMQ, and the client wanted to empower the end user to troubleshoot conversion issues themselves to reduce IT Department burden.

Unfortunately, if there was a problem on any step in the process, I was limited to one set of data from the Message published from the MSMQ to write error specific information to the end user. This frustrated me to no end, because in order to be successful with this process, the client wanted to be able to write back detailed failure information to CRM if there was a problem, such as:

• Error Reason
• (CRM) Item ID
• (CRM) Item Description
• (CRM) Item Created On
• (GP) Item Number
• (GP) Item Description
• (GP) Item Created On
• (CRM) Conversion Attempted On
• (CRM) Conversion Attempted By

This information was difficult to write back, because of the limited source update capabilities using MSMQ. In addition to the limitations, the process felt extremely clunky and prone to error due to the back and forth of the message handoffs between DTS files on success of each job. The only way for me to do this was to create multiple jobs to handle failures, creating a temporary table in Scribe Internal to gather information about the failure, get information from GP and CRM, and then push that information to CRM after the processing was complete.

With the release of Scribe 7, I attended one of the multi-target webinars, and one night while in the shower (where my brain tends to do most of its intense thinking), had one of those “Holy Cow!” moments where I realized the power of multi-target and the huge reduction in overhead it could provide. I decided to install and configure Scribe 7 in our test environment to see if I could reduce the complexity of this process by leveraging multi-target.

After redesigning the process, I was able to take 8 individual DTS files performing multiple tasks, and merge them down into one DTS that has multiple steps going against GP and CRM. The new DTS gets its source from the same message as the old process; however the resulting complexity has been reduced by a significant factor, cutting out temp tables and multiple message handoffs.

Multi-target has also provided us the ability to write back to CRM with failure info from GP (Steps 7 and 8 in Exhibit 1), should we find duplicates in GP or if the insert fails. Using temporary variables (Exhibit 2) I can capture CRM and GP data points for each record, writing back robust error information to CRM if the job encounters issues.

Additionally, upgrading to 7 from 6.5.2 for the existing processes was seamless. I took the 20+ DTS files and two publishers related to the other pieces of the GP-CRM integration and updated them all in a manner of an hour. The only issue I ran into was that I had to update my DBLOOKUP statements, because with the addition of multi-target, you now do DBLOOKUP against the connection name, not “S, T or I”.

Once I realized the power of multi-target, I haven’t looked back. I am pushing all my clients to Scribe 7.0.1 if they need any kind of new DTS development, because it makes their systems more efficient and makes my life significantly easier.

Exhibit 1

Exhibit 2

More than you could ever want to know about integrating with Dynamics GP

Many of you are aware about the upcoming Scribe User Conference this September. One of the topics that will be covered by me (as the Product Manager here at Scribe) and Mike Bates (Scribe Template Guru Extraordinaire) is a detailed walk through of the Scribe Microsoft Dynamics GP Adapter. Mike has been with Scribe for nearly a decade and is the author of many the templates you use (including the GP<>CRM Template). I am the Product Manager here at Scribe and, over the past 15 years or so, have written and worked with many integration solutions with Microsoft Dynamics products. Between the two of us, we will be walking you through the details of the Scribe GP Adapter and integration solutions to Microsoft Dynamics GP.

This session will be a deep-dive on the various moving parts of the adapter including the role of Metadata in the Scribe GP Adapter and how to expose customizations in Microsoft Dynamics GP fields for integration.

There have also some dramatic changes around how we publish data out of Microsoft Dynamics GP in the recent months, so there will a particular emphasis on how the new Scribe GP Publisher works and options you have to customize when and how data is published out of Microsoft Dynamics GP.

To illustrate the various technical points, we will be doing demo's that use the product in action. With this real-time interaction with the product, we will be able to react to questions and scenarios that you bring with you - so please do come prepared with any specific questions or real-world use cases.

We look forward to seeing you in Sunny NH at the end of September!

Customer realizes over $100,000 savings per year with the help of Scribe Insight

The Apple Gold Group, a tri-state organization founded in 1984 and operating more than 70 Applebee's Neighborhood Grill and Bar restaurants in North Carolina, Oklahoma and Arkansas, has realized incredible cost and time savings using Scribe Insight for data integration with their Microsoft Dynamics® GP application.

A year ago Apple Gold's new employee and payroll processes were time consuming and expensive. With over 4,000 employees at any given time working different jobs and shifts, Apple Gold's Payroll Manager, Angelia Parrish, was dealing with a lot of data. The W-2 and per paycheck processing costs of their third-party payroll provider were expensive, and the turnover typical of the restaurant industry meant that a significant amount of new hire data had to be double-keyed; both at the restaurant and headquarters locations.

Reducing costs and increasing efficiency
Angelia and her team knew they needed to reduce costs, but with the amount of employee and time tracking data their business managed, their search for a more cost-effective payroll solution had to involve an efficient data integration strategy. They turned to Massey Consulting who recommended Microsoft Dynamics® GP, specifically because of its seamless integration capabilities when used with Scribe Insight. Read the full Customer Success Story here.

Steelhead Release Candidate Available for Download

We have just published Release Candidate #1 for the Steelhead. The download includes:

• Scribe Insight 7.0
• Scribe Adapter for Dynamics CRM 4.4
• Scribe Adapter for Dynamics GP 4.1
• Scribe Adapter for Salesforce 2.5
• Scribe Adapter for SalesLogix 4.9
• Scribe Adapter for WebServices 1.2
• Scribe Adapter for ACT!4-6 7.0
• Scribe Adapter for ACT!2007-2008 7.0
• Scribe Adapter for Exchange 7.0
• Scribe Adapter for GoldMind 7.0

Scribe Adapter for Dynamics NAV 1.4 and Scribe Adapter for Dynamics AX 1.1 are not part of the RC1 release. Preview versions of those adapters will be made available in the near future.

Release candidate #1 is released as part of our technical preview program. Access is open to all Scribe partners and customers. Details on how you can download the RC1 release can be found at http://blog.scribesoft.com/2010/05/get-early-access-to-insight-70-with-the-steelhead-technical-preview.html.