Integration and Analytics in the Cloud (100% Cloud)

I work as an Integration Architect.
 

Few years ago, life of senior management guys was not easy. One of my senior manager had to run a BI report everyday in the morning at 7 AM for doing some analysis and decision making. So he used to reach office before 7 AM, start his workstation, open the appropriate tools and then run the report. And this whole process used to take good 30 minutes to an hour. Apart from it, he used to have concerns regarding elasticity, availability, flexibility and cost of the tools and technologies.
 

He used to envision that one day he would be able to do this process by click of a button from anywhere, anytime. “No constraints whatsoever.”

We did an integration implementation for a large Canadian Telecom Giant. The name of the project was Marketing Data Mart. 
 

The Marketing Data Mart consisted of an integrated architecture of heterogeneous data stores and technologies to support the ultimate analysis of data. 
 

We needed to integrate the data from the following source systems:

• Salesforce.com
• Eloqua
• Harte Hanks
• Dun and Bradstreet Optimizer
• Jigsaw Dun and Bradstreet Contacts

To make it happen, we had used the following tools and technologies:

• Informatica Cloud - cloud based integration tool (http://www.informaticacloud.com/)
• Amazon EC2 (Elastic Cloud Compute) – Cloud based hosting (http://aws.amazon.com/ec2/)
• Amazon RDS (Relational Database Service) – Cloud based database (http://aws.amazon.com/rds/)
• GoodData – Cloud based reporting and analytics (http://www.gooddata.com/)

Data from all the source systems was loaded and transformed in Amazon RDS and this data was fed into GoodData which enabled complex and analytical reports creation.
 

There were some initial challenges while configuring the Informatica on Amazon EC2, setting up secure FTP on Amazon EC2 and configuring Amazon RDS and GoodData because of our minimal exposure on these technologies, but we had the vision in front of us that enabled us to overcome all the hurdles and implement the entire integration on cloud. By cloud, I mean 100% on Cloud.
 

Some of the salient features are:

• The complete integration was implemented on 100% cloud based technologies.
• Informatica Cloud was configured on Amazon EC2 UNIX instance successfully.
• Data Volumes to the tune of 2-3 million records were integrated successfully.
• 83 separate tables in Amazon RDS, containing data from 6 source systems, are part of the data mart solution.
• Complex analytical reports and dashboards were generated using GoodData.
• The client previously had to use 3+ separate systems to get reports which then had to be consolidated via spreadsheets & other tools. The reporting from GoodData is a one-stop shop for reporting across multiple systems, all accessible via a web browser.  For deeper dives into the data, using sophisticated SQL queries, the client can run reports on the Amazon RDS database.
• There was no compromise on the security aspect and the data of the client was stored in highly secure cloud platform.
• Amazon EC2 and RDS are highly scalable and there are no concerns with respect to availability and flexibility

We have successfully proved that cloud technologies can be used for complex integrations and now senior managers can feel relieved as they can run the BI reports by click of a button anytime, anywhere.