An ETL Tool vs. a Home Grown Solution

[Huge archives of historical data, growing volumes of transactional data, informative external data when harvested into business Intelligence systems and data warehouses for decision making activities provide a competitive edge to business houses. Yet data warehousing is an expensive proposition and often organizations hope to reduce costs by economizing on specific stages of the process. There is always a raging debate on the economics of creating in-house software versus buying off-the-shelf expensive tools. Each critical process is evaluated in this context and the use of ETL tools and software does not escape from this eternal debate. However, off-the-shelf products have their own advantages and though the programming is generic they are flexible enough for customizations. Now there is even a new breed of off-the-shelf ETL tools priced for the small and mid size market such as VisualETL.]

The off-the-shelf advocates insist that business intelligence solutions have enough challenges without having to worry about possible lacunae in the home grown software that may corrupt target systems and make data inconsistent. Software like VisualETL should be used as they have proven track records of success and have taken care of most of the data issues by a long drawn process of experimentation and implementation.

The home grown solution advocates, on the other hand, dismiss the off the shelf solutions on the ground that they cannot really cater to the specific needs of a business entity. They point out that high costs and generic programming do not really address the specific needs of the business entities. Home grown solutions have distinct advantages. The low cost; the customization of the code; optimization of the program to suit the needs; the pace at which the solution can be built and the large knowledgebase of the programmers all make the possibility of a home grown solution attractive. On the other hand, it must be acknowledged that the disadvantages out weigh the advantages. Home grown solutions are difficult to manage and maintain. Any change to the data warehouse would impact on the ETL solution. There would be no centralized repository of code and the metadata capabilities would be limited. The development cycle is large and debugging is more difficult. Audit trials are limited or non existent. Moreover, a small mistake in the process of Extraction, transformation and load or a lack of foresight in the creation of the program would result in a crippling impact on the data analysis and interpretation. Fractures in the integration of the various data sources would corrupt the target system and cause incorrect representation of facts.

Off-the-shelf ETL solutions like VisualETL provide an attractive user interface and have a centralized storage for programs. Version control of programs is possible and customizations of transforms become fairly simple. Metadata support is optimal. Transformations can be quickly deployed and transform scheduling, auditing are possibilities. Debugging is easy and user friendly.

Though the debate continues to rage, more and more organizations are opting for off the shelf solutions for ETL. They reason that ETL is an important stage in the process of creating the data warehouse. It is the process that determines the integrity and accuracy of data. It is foolishness to risk data integrity for cutting costs or for reasons of customization. Off the shelf products have been tested on a variety of data sources and has enough inbuilt flexibilities to accommodate customizations.

With the new breed of reasonably priced ETL tools such as VisualETL, there is even more of a reason to use an off-the-shelf extract, transform and load software tool.