Why ETL Tools Might Hobble Your Real-Time Reporting & Analytics

Companies report large investments in their data warehousing (DW) or Business Intelligence (BI) systems with a large portion of the software budget spent on extract, transformation and load (ETL) tools that provide ease of populating such warehouses, and the ability to manipulate data to map to the new schemas and data structures.
Companies do need DW or BI for analytics on sales, forecasting, stock management and so much more, and they certainly wouldn’t want to run the additional analytics workloads on top of already taxed core systems, so keeping them isolated is a good idea and a solid architectural decision. Considering, however, the extended infrastructure and support costs involved with managing a new ETL layer, it’s not just the building of them that demands effort. There’s the ongoing scheduling of jobs, on-call support (when jobs fail), the challenge of an ever-shrinking batch window when such jobs are able to run without affecting other production workloads, and other such considerations which make the initial warehouse implementation expense look like penny candy.
So not only do such systems come with extravagant cost, but to make matters worse, the vast majority of jobs run overnight. That means, in most cases, the best possible scenario is that you’re looking at yesterday’s data (not today’s). All your expensive reports and analytics are at least a day later than what you require. What happened to the promised near-realtime information you were expecting?
Contrast the typical BI/DW architecture to the better option of building out your analytics and report processing using realtime routing and transformation with tools such as IBM MQ, DataPower and Integration Bus. Most of the application stacks that process this data in realtime have all the related keys in memory –customer number or ids, order numbers, account details, etc. – and are using them to create or update the core systems. Why duplicate all of that again in your BI/DW ETL layer? If you do, you’re dependent on ETLs going into the core systems to find what happened during that period and extracting all that data again to put it somewhere else.
Alongside this, most organizations are already running application messaging and notifications between applications. lf you have all the data keys in memory, use a DW object, method, function or macro to drop the data as an application message into your messaging layer. The message can then be routed to your DW or BI environment for Transformation and Loading there, no Extraction needed, and you can get rid of your ETL tools.
Simplify your environments and lower the cost of operation. If you have multiple DW or BI environments, use the Pub/Sub capabilities of IBM MQ to distribute the message. You may be exchanging a nominal increase in CPU for eliminating problems, headaches and costs to your DW or BI.
Rethinking your strategy in terms of EAI while removing the whole process and overhead of ETL may indeed bring your whole business analytics to the near-realtime reporting and analytics you expected. Consider that your strategic payoff. Best regards in your architecture endeavors!
Image by Mark Morgan.

IBM Cast Iron Options: Appliance Vs. Live (SaaS)

Software as a Service (SaaS) is fast becoming the deployment du jour, but appliances haven’t lost favor yet. The decision over an on-premise install or appliance versus an SaaS solution should still be made on a case-by-case basis. IBM Cast Iron offers both options and each enables cloud-to-cloud, cloud-to-on-premise and on-premise-to-on-premise integration and real-time, near-real-time and batch.
Here’s a brief look at both options.

Cast Iron As Appliance

Cast Iron can be deployed as either physical hardware or a virtual machine. With this style of deployment, the Integration Appliance is installed on-premise – normally behind the firewall, but not within a DMZ.
The runtime environments for the dev, test and prod lifecycles are typically separated, each with its own Integration Appliance to access necessary endpoints within the environment.
The user maintains full control of projects and their orchestrations on the appliance through the WMC. The orchestrations are started through activities which include polling, scheduling or an incoming request – an HTTP Receive Request, for example. Data flows through the Integration Appliance and is stored internally as XML variables. Users control the logging for each orchestration.

Cast Iron Live

This multi-tenant, browser-accessed deployment model includes key components for users to design, run and manage integrations all in the cloud. Those components are:

  • A clustered runtime engine that runs the integrations with built-in fault-tolerance and recovery mechanisms
  • A multi-tenant, highly available system to store the designed integrations
  • A load-balancer to intelligently manage the loads throughout the various runtime engines
  • Highly available file systems to store and manage logs that are related to the integrations

The Live version also includes a Design environment with the same capabilities as the on-premise Studio environment
TxMQ specializes in application integration. Initial consultations are free and communications are always confidential. Contact vice president Miles Roty for more information: (716) 636-0070 x228, miles@txmq.com.

Internet Of Things In Focus At Europe's IFA Fair

Stroll through any box electronics or homestore nowadays and you’ll see the beginnings of the Internet of Things (IoT) – formal slang for the coming world of interconnected everyday devices. Best Buy is selling a smartphone-controlled garage-door openers and smartphone-enabled deadbolts as just a few examples.
Some true action on the IoT front took place this week at Europe’s IFA tradefair, held this year in Berlin, Germany. It concluded today and is touted as “the world’s leading tradeshow for consumer electronics and home appliances.” German engineering is always front and center at the show.
Early reports out of the show point to several highlights:

  1. A robotic vacuum cleaner from British sensation Dyson
  2. A growing horde of smartwatch competitors, led by Samsung’s Galaxy Gear S (Apple had its own homegrown Watch rollout concurrently in California)
  3. TVs with convex screens to better mimic the cinematic experience
  4. A new Sony glass to compete with Google Glass in the market of aug-reality
  5. Appliances for the interconnected home

About No. 5: Bosch and its partner Siemens debuted their first Home Connect oven and dishwasher, which should be available before the 2014 holidays. The Home Connect app, announced last spring, will run on iPhone and Droid and control compatible devices through the app.
The important part of this news is to track the competition for the overall IoT standard. Home Connect is an open standard that Bosch-Siemens developed. Both  brands use it, and curiously, it supports other competitive brands. Bosch and Siemens in fact passed on the Apple HomeKit connection standard in favor of their own proprietary system.
The Home Connect appliances are definitely cool. The engineering quality is crazy good – you’d expect nothing less from brands like Bosch and Siemens. But in terms of the IoT, they still only talk to each other. We’ll see in the coming year whether other manufactures develop for the Home Connect standard, or whether Google Nest, or Apple HomeKit or other emerging platforms can successfully attract brands into their big tent.

[youtube id="a1xtXM5zjdA" width="600" height="350"]