Breaking the SOURCE -TARGET Paradigm

My name is Eddie Cole and I'm the Software Architect here at Scribe. I wanted to share with you a big change we are planning to make to our product, and also solicit some feedback about what you think of our plans.

Due to its roots as an ETL tool, at its core Scribe's technology has always been based on a very simple concept - Moving data from a SOURCE to a TARGET. While complex multi-point integrations can be developed using our product, when you drill down to the point at which data is actually being moved, you will find a DTS file that moves data from a SOURCE to a TARGET. We are planning to begin breaking this paradigm with Insight 6.6 (code named Steelhead) by allowing users to design DTS files that include 'steps' that can operate on data sources other than the target.

Why would anyone want to do that!

The SOURCE-TARGET paradigm has served us well over the years, but we have come to realize that by breaking this paradigm we would enable Scribe users to overcome two product "features" that have at times caused problems for users:

• Publisher based integrations are more complicated than they need to be because their DTS files have XML as their source. This prevents users from doing lookups or updates against the true data source, often resulting in extra DTS files or lacking template functionality. It's this feature that causes us to require those "loopback" IPs in our templates.
• Our Web Services Adapter can only be used as a source or target. Users want to be able to use it to make Web Service calls in conjunction with a source and target. This will allow users to create jobs that do tasks like correct addresses as part of a migration or call out to Hoovers to perform a credit check or performing address validation as part of an integration.

So we created an OpenMind idea (OpenMind #7) to ask the community what they thought. It has received 33 votes to date. As we reviewed other Openmind ideas we found a number of others that would also be resolved by adding this functionality:

• Allow lookup into a database outside of the source, internal, or target (OpenMind #66 - 21 Votes)
• Update multiple source fields. (OpenMind #215 - 16 votes)
• Using Update Source with Custom Queries (OpenMind #142 - 5 votes)
• Improve the update source functionality to not be so quirky (OpenMind #156 - 3 votes)

All told this feature represents 78 OpenMind votes! This signifies that support for multiple targets is in fact the most requested feature by a wide margin. The bottom line is that by adding this new functionality users should be able to develop richer integrations in a more straight forward manner.

Well how would it work?

Our basic plan is to simply expand the data object browser in the target configure window to include data objects from connections other than the target. The user would then be able to add steps for any data object in the same way they do today. Of course that's just the basic plan. Unfortunately, our folks in QA and Support have told us that we also have to rework the UI in a whole bunch of ways so this functionality will make sense to the users. Most significantly there are many things that we now refer to as Target This or Target That that now need to be renamed Step This or Step That. You'll understand what I'm talking about the first time you open the new Step Configure dialog.

All of our existing functionality will remain relatively unchanged. The UI may change a bit here and there but we are not removing any features such as Update Source or Key Cross Reference that are becoming a bit redundant. DTSs created with earlier versions will continue to run unchanged.

So what would I like to know?

I mentioned at the top I wanted to solicit some feedback. Please make liberal use of this blog's comment feature, or participate on the discussion threads of the related ideas in OpenMind. What I am looking for most is real life use cases that you see this functionality being a good fit to solve. This information will help us ensure our design is on the right path and we will also incorporate these use cases in both our development and testing processes. I'd also love to hear any other thoughts or comments you may have about this feature, pro or con.

Comments

[Mangesh Lolge]

Thanks a lot for coming up with this great feature !! By allowing additional data source objects in the target window and steps would enable us minimize number of detached DTS packages, where the source needs to be updated once data has been synced into the target and/or satisfies certain business criteria.

I am working on a project where Dynamics CRM (Single Database with multiple organization) needs to be synced with Dynamics GP (Multiple Company Databases) for certain transactions, with current product version I have to repeat and copy same business logic for each company based DTS package, So If I have 5 Transactions and 5 companies, I have code, maintain and manage an 25 DTS packages.
Can I assume that with these enhancements, I can simply repeat the steps pointing to GP databases as required target and build only single DTS package per transaction.

[Eckart Fischer]

We are using SCRIBE to move data between our ERP and CRM installations (14 installations world wide so far).
We were forced to use INAPORT in the Asian installations as SCRIBE can not handle unicode languages.
When will SCRIBE be able to handle unicode languages, too?

[Scribe Software]

Eckart, I am VP, Product Marketing at Scribe. Our product will handle Unicode no later than June 30, 2010. Thanks for the post and thanks for using Scribe.

[Eddie Cole]

Hi Mangesh,
Thanks for the feedback. And yes, this new feature will allow you to create multi-company integrations as you described.

The one thing I'd add is that if your companies all have matching schemas and therefore have identical DTS logic, we have some features that may allow you to accomplish this in a single DTS file already. Check out the following link for more info https://openmind.scribesoftware.com/topics/790

[crmright.wordpress.com]

I have wanted to update source fields based on information in the target very often. It sounds like this would meet the need. Well done.

[Bernd Schwenk]

Eckart Fischer asked for handling of Unicode in 2009 and John Gravely gave the answer "Our product will handle Unicode no later than June 30, 2010".

One year later the same question: When will SCRIBE be able to handle unicode languages?
Wish version do we need?

[Scribe Software]

Bernd, the current plan is to have Unicode support no later than the end of 2011. We had previously slated it for a 2010 release but that plan was changed and I had forgotten that my blog comment still provided that information. I have since removed my comment so as not to keep the old information published.