Picture this situation: your business has spent a considerable amount of time and effort building a Data Warehouse, designing the ETL package and constructing a rich layer of Business Intelligence queries to access the data, but now the volume of data is growing enormously. To compound the performance problems this is causing, the user community have embraced the BI solution that has been provided and now more complex queries, more ad-hoc queries and an increased usage of the standard BI suite is causing contention and other performance problems. What can you suggest?
Well, you could offer a major re-design of the Warehouse, accompanied by a significant investment in hardware upgrades, more disk space and an end-to-end performance tuning exercise for your BI suite. Or you could suggest hosting the Data Warehouse on Netezza.
The Netezza solution is a purpose built Data Warehouse server with all hardware and software required in a tightly integrated and highly specialized configuration. Data is stored in compressed format on disk arrays with mirroring and failover built in. CPU processing is aligned to your specified data distribution keys, providing rapid and intelligent pre-processing and filtering of the data for your BI queries. Migrating to Netezza is straight-forward:
• Your existing warehouse design can be imported into a Netezza database with minimal change (removal of indexes, triggers and other complex structures, and the selection of a distribution key for each table).
• Data can be loaded into the Netezza database from your existing source (at up to 2TB per hour)
• Your existing ETL package can be configured to load into Netezza with a change to the connection (ODBC, OLE-DB or JDBC) configuration
• Your existing BI suite can run against the Netezza database with a similar configuration change
The performance improvement you can expect to see would be in the region of 10x. Customers migrating from Oracle, Teradata and DB2 warehouses have experienced gains in this region, or better. IBM offers a Proof of Concept that will demonstrate this level of performance gain if your Warehouse is 2TB+.
Of course, you don’t need to consider Netezza only as a rescue package; you can design a Data Warehouse implementation from scratch and make use of this high-performance technology. But if you do find yourself in the nightmare scenario, shouldn’t this be an option to consider?