Multi-cloud is growing in popularity as a key enterprise architecture strategy as organisations progress their digital transformation to assure business continuity with seamless access to applications. This approach also helps avoid vendor lock-in, enabling businesses to move between different cloud environments to maximise their flexibility.
A multi-cloud strategy enables an organisation to leverage multiple cloud computing and storage vendors within a single architecture, to advance its data maturity to achieve business goals more efficiently. For example, an organisation may have its data on a Snowflake database hosted on Microsoft Azure Cloud and at the same time, also leverages the AWS Cloud and some of its data services and tools to help stream, land and ingest data into Snowflake and perform downstream data analytics and data science activities to inform growth strategies.
Determining the right cloud service for the job is no small task. CIOs need to ensure optimal user experiences and application availability whilst managing CapEx, scalability, governance, security, and other considerations. Cloud platform choices truly come to life with integration – the ability to run different applications, workloads, and business processes is critical.
Tridant was approached by a large sports organisation facing several data challenges that prevented business stakeholders from getting to the information they needed to make agile, informed, and validated decisions:
After intensive discovery sessions with the client, Tridant cloud architects recommended a multi-cloud architecture that would address their organisational requirements and critical goals, and comply with the client’s overarching vendor strategy.
The diagram below depicts the recommended multi-cloud architecture to resolve our client’s unique challenges.
This multi-cloud architecture was partly dictated by the organisation’s requirement to have MS Azure Cloud as its main cloud vendor, but also leverage AWS Cloud and some of its services and tools to achieve specific outcomes.
With Microsoft Active Directory and Microsoft Azure Active Directory used for Identity and Access Management (IAM), and Microsoft Power BI for their reporting and analytics needs, the organisation sourced data from a mix of internal and external data sources, including on-premise and cloud-based data sources. Some of the cloud data sources require an Amazon Kinesis stream and s3 bucket to send across data.
Over time, the solution can easily evolve to include other tools and technologies to achieve specific outcomes. For example, a data integration tool like Matillion may be considered if the number and complexity of data sources to be stored and analysed increase, and its orchestration needs to increase accordingly. Amazon SageMaker can also be considered to meet some of the data science outcomes.
Figure 1: Cloud data architecture highlighting a multi-cloud approach
From the Data Landing Zone, we used a combination of Snowflake’s internal capabilities like Bulk Load, SnowPipe, Stages, Tasks to load the data into a Staging Area within Snowflake. The Data Landing Zone and the Snowflake Staging Area establishes a reconciliation process which can be both automated and exposed to a business intelligence tool like Power BI or Tableau for continuous monitoring of the state of the incoming data from the data source, the quality of that data and the health of the data load process. This report can then be used to alert stakeholders as needed.
From the Staging Area, we used data modelling techniques to model the data into a data foundation layer that will form a strong and robust foundation to all downstream data science and data analytics applications.
An important part of this solution is integration with the on-premise Active Directory, Azure Active Directory, Snowflake and any downstream data analytics applications like Power BI and Tableau, to alleviate data governance concerns faced by the organisation. We also enabled Single Sign On (SSO) so that a user logs in only onto their workstation and is automatically granted access to all approved applications for that user.
Today, Snowflake provides a powerful and flexible data cloud solution for our client. It is able to ingest large amounts of data within minutes (if not seconds). We conducted a benchmark test of the time taken to load the entire historical data by the SQL Datawarehouse using Microsoft SSIS vs. the time taken by Snowflake to load the same data. Figure 2 depicts some results:
Figure 2: Benchmark test of data load time using SQL Server/SSIS solution on-premise data warehouse vs. a Snowflake solution
A multi-cloud strategy has complex logistical and operational considerations, with inherent challenges that need to be understood and worked through:
The value of a single vendor-specific cloud architecture is in question, particularly as contracts, costs, and capacity issues struggle to keep pace with the flexibility embedded in a multi-cloud strategy.
As organisations assess their cloud architecture to optimally manage cost, performance and scalability, multi-cloud will continue to gain momentum.
Assure uninterrupted availability, mitigate risk, and optimise security in your cloud strategy. Talk with Tridant cloud platform architects today.