Research and Practice of Kettle in the Field of Distributed Data Warehouses

Abstract

To address the diverse and often conflicting data extraction and loading requirements that arise across heterogeneous business systems—ranging from enterprise resource planning (ERP) and manufacturing execution systems (MES) to customer relationship management (CRM) and supply chain platforms—this paper proposes a Kettle-based implementation scheme for the systematic construction of a distributed data warehouse, grounded in simulated data generated by the industry-standard TPC-DS benchmarking tool. Through a rigorous comparative analysis of three distinct data extraction methodologies—namely manual scripting, direct extraction (which pulls data straight from source systems with minimal intermediate processing), and indirect extraction (which employs staging areas and intermediate transformations prior to loading)—the indirect approach is selected as the optimal strategy due to its superior balance of performance, maintainability, and fault tolerance, particularly when handling the complex, multi-source data landscapes typical of modern enterprises