Why ETL Tools Might Hobble Your Real-Time Reporting & Analytics

Companies report large investments in their data warehousing (DW) or Business Intelligence (BI) systems with a large portion of the software budget spent on extract, transformation and load (ETL) tools that provide ease of populating such warehouses, and the ability to manipulate data to map to the new schemas and data structures.
Companies do need DW or BI for analytics on sales, forecasting, stock management and so much more, and they certainly wouldn’t want to run the additional analytics workloads on top of already taxed core systems, so keeping them isolated is a good idea and a solid architectural decision. Considering, however, the extended infrastructure and support costs involved with managing a new ETL layer, it’s not just the building of them that demands effort. There’s the ongoing scheduling of jobs, on-call support (when jobs fail), the challenge of an ever-shrinking batch window when such jobs are able to run without affecting other production workloads, and other such considerations which make the initial warehouse implementation expense look like penny candy.
So not only do such systems come with extravagant cost, but to make matters worse, the vast majority of jobs run overnight. That means, in most cases, the best possible scenario is that you’re looking at yesterday’s data (not today’s). All your expensive reports and analytics are at least a day later than what you require. What happened to the promised near-realtime information you were expecting?
Contrast the typical BI/DW architecture to the better option of building out your analytics and report processing using realtime routing and transformation with tools such as IBM MQ, DataPower and Integration Bus. Most of the application stacks that process this data in realtime have all the related keys in memory –customer number or ids, order numbers, account details, etc. – and are using them to create or update the core systems. Why duplicate all of that again in your BI/DW ETL layer? If you do, you’re dependent on ETLs going into the core systems to find what happened during that period and extracting all that data again to put it somewhere else.
Alongside this, most organizations are already running application messaging and notifications between applications. lf you have all the data keys in memory, use a DW object, method, function or macro to drop the data as an application message into your messaging layer. The message can then be routed to your DW or BI environment for Transformation and Loading there, no Extraction needed, and you can get rid of your ETL tools.
Simplify your environments and lower the cost of operation. If you have multiple DW or BI environments, use the Pub/Sub capabilities of IBM MQ to distribute the message. You may be exchanging a nominal increase in CPU for eliminating problems, headaches and costs to your DW or BI.
Rethinking your strategy in terms of EAI while removing the whole process and overhead of ETL may indeed bring your whole business analytics to the near-realtime reporting and analytics you expected. Consider that your strategic payoff. Best regards in your architecture endeavors!
Image by Mark Morgan.

How Do I Support My Back-Level IBM Software (And Not Blow The Budget)?

So you’re running outdated, obsolete, out-of-support versions of some of your core systems.? WebSphere MQ maybe? or WebSphere Process Server or Datapower…the list is endless.
Staff turnover may be your pain point – a lack of in-house skills – or maybe it’s lack of budget to upgrade to newer, in-support systems. A lost of times it’s just a matter of application dependencies, where you can’t get something to work in QA, and you’re not ready to migrate to current versions just yet.
The problem is that management requires you to be under support. So you get a quote from IBM to support your older software, and the price tag is astronomical – not even in the same solar system as your budget.
The good news is you do have options.

We were able to offer a 6-month support package, which eventually ran 9 months in total. Total cost was under $1,000 a month.

Here at TxMQ, we have a mature and extensive migration practice, but we also offer 24×7 support (available as either 100% onshore, or part onshore, part offshore) for back-level IBM software and product support – all at a fraction of IBM rates.
Our support starts at under $1,000 a month and scales with your size and needs.
TxMQ has been supporting IBM customers for over 35 years. We have teams of architects, programmers, engineers and others across the US and Canada supporting a variety of enterprise needs.
Case Studies
A medical-equipment manufacturer planned to migrate from unsupported versions of MQ and Message Broker. The migration would run 6 to 9 months past end-of-support, but the quote from IBM for premium support was well beyond budget.
The manufacturer reached out to TxMQ and we were able to offer a 6-month support package, which eventually ran 9 months in total. Total cost was under $1,000 a month.
Another customer (a large health-insurance payer) faced a similar situation. This customer was running WebSphere Process Server, Ilog, Process Manager, WAS, MQ, WSRR, Tivoli Monitoring, and outdated DataPower appliances. TxMQ built an executed a comprehensive “safety net” plan to support this customer’s entire stack during a very extensive migration period.
It’s never a good idea to run unsupported software – especially in these times of highly visible compliance and regulatory requirements.
In addition to specific middleware and application support, TxMQ can also work to build a compliance framework to ensure you’re operating within IBM’s license restrictions and requirements – especially if you’re running highly virtualized environments.
Get in touch today!
(Image from Torkild Retvedt)

DataPower Firmware V7.2 Upgrade Details (More Cloud!)

One of the coolest things about IBM DataPower is how it provides a single security and integration gateway solution using a simple, policy-driven interface. And it’s ultra-flexible, so it’s easily deployed to multiple cloud environments. And that’s really the trick, isn’t it? The need to deploy across today’s growing mix of public, private and hybrid cloud mixes. It’s not as easy as it seems.
The good news is that the newest DataPower Gateway Firmware V7.2 upgrade extends multiple-cloud support with enhanced security and integration capabilities for mobile, API and web workloads. Want to run DataPower to gate Amazon Web Services? Great, this firmware update is definitely for you.
Cloud environments like Amazon AWS and IBM SoftLayer are hosting environments that allow a business to deploy an optimal set of supported infrastructure resources. Using DataPower, businesses can then quickly build gateway solutions and rely upon the cloud provider to maintain the underlying infrastructure. The result is an overall lower cost to deploy and secure workloads (especially mobile workloads), plus additional flexibility for faster responses to market trends and business opportunities.
The DataPower Firmware V7.2 enables this type of process improvement through several key upgrades:

  • Use GatewayScript to easily integrate Systems of Record with Systems of Engagement
  • Deploy IBM DataPower Gateways on Amazon Elastic Compute Cloud (EC2) and IBM SoftLayer CCI
  • Enhanced hybrid cloud integration: Use the Secure Gateway service to securely connect between IBM Bluemix applications and on-premise services secured by DataPower Gateways
  • Enhanced message-level security for mobile, API and web workloads with JSON web encryption for message confidentiality, JSON Signature for message integrity, plus JSON Web Token and JSON Web Key
  • Protect mission-critical applications with enhanced TLS protocol using Elliptic Curve Cryptography (ECC)
  • Build automation, deployment and migration scripts more quickly using the new Representational State Transfer (REST)-based management API
  • Enjoy the increased high availability with enhanced load-balancer group healthchecks
  • Support for IMS Commit mode 0 for more flexible integration with systems of record.
  • IBM WebSphereeXtreme Scale 8.6+ support for distributed caching

TxMQ specializes in DataPower upgrades and services. Need some quick advice about deploying this firmware upgrade into production? Want to decouple from core IT and speed your time to market? Want to integrate your systems of record with your new systems of engagement? That’s what we do. Let’s start a conversation today: Click the chat icon below, or email our president chuck@txmq.com.
(Image from Paul)