I would say historically that 99% of my conversations around business intelligence have focused on the “business first” angle. I do feel that this will continue to be the trend as we engage with the business for better insights, self-service, and user empowerment. However, I recently had the opportunity to attend a Parallel Data Warehouse (PDW) class offered by Microsoft in Redmond and a very interesting conversation started around BI for IT.
This conversation wasn’t about what type of IT dashboard can I create or what metrics do IT look at, but rather, what are the driving factors that make IT review its current strategy, or force them to look in a new direction. So what are these drivers? How a business conversation is deeply rooted in IT? Let’s look at a couple of examples.
- Short or shrinking processing window. This is a great example of how BI is changing the view of IT from the business perspective. The insatiable appetite for data is making IT rethink how we process and load information. What was almost always considered a batch or MOLAP process is now becoming much closer to real time. Not only are they asking for the frequency to change, but also the volume of information has spiked. Sadly, we can’t always go back in and put in a bigger pipe, so how do we do more with what we have before trying to sneak a cool six figures into the capital budget.
- Big Data + Dev Tools = Big Problems. I am not specifically talking about semi / unstructured text data here, but rather reiterating that lots of data flowing through a small pipe reaches critical mass very quickly. Now combine this with dev tools and we have just created a Martha Stewart worthy recipe for disaster. Simple real world example: I worked with a reatailor that needed to first extract from binary, transform to CSV, then load to a data warehouse every day from about 255 POS system in less than two hours. When the solution was first being worked through, simple SSIS tasks (such as the lookup task) worked great. When I then tested it with multiday files that were each 10 million plus rows, we sort of ran into an issue – sadly, nobody liked my idea to just sell 100 products and cull the remaining 400,000 from inventory. Now, this is where I could make a very good push for PDW and a bigger pipe, but we will save that for a future article.
- Heterogeneous data and data sources. Has the cloud impacted your life yet? What about that pesky FTP site from the 3rd party company? How about Mary in Accounting that has her formula riddled spreadsheet that just might contain the secrets of the universe? The sheer number of ETL sources is also now causing some heartburn in the trenches. How and where you handle your business logic, is also now a bigger slice of the pie. Another plug for PDW here, but their stance is ELT vs. ETL – so does this mean that our methodology should also be changing for standard systems?
The list goes on (trust me), but I thought I would call out just a few of the common themes that I have heard in conversation lately. Do any of these sound like your organization? Which one did I miss that should be on the top 10 list? If you are an IT shop struggling with some of these choices – or if you are a business minded person who is tired of beating up your IT staff with a wet noodle, there is help! Aspect has helped navigate and advise our customers in these scenarios. Please reach out so we can row the life boat in your direction!
The post More Data, More Problems appeared first on Aspect Blogs.