Tag Archives: azure

Azure Data Factory Region Detection

23 Dec

If you are building an Azure Data Factory (ADF) pipeline and receive an error that contains this message when your pipeline is executed:

Failed to detect region of linked service

or

Failed to detect the region for

… then you may be running into a situation where the Data Movement Service (DMS) feature of ADF is either not able to detect the region of the data store or there is no DMS in that region.

The Data Movement Service of ADF is the Azure-managed cloud service (PaaS) that performs scale-out data movement at elastic scale. Azure handles all of the plumbing for moving Big Data for your data pipelines. You can see the locations available for Data Movement on the Azure Regions page (https://azure.microsoft.com/en-us/regions/services/). On the screenshot of that page below you’ll see that the Data Factory service has several sub-services. The Data Factory service stores your factory account metadata while the Movement, Activity Dispatch and SSIS IR are separate managed services that have their own region deployments. It is the Data Movement service in those regions that perform the heavy lifting of moving your data and that is where you should focus to bypass the error.

regions

In the V1 original ADF service, there is a property on the Linked Services definition that allows you to explicitly tell ADF the location of your data store (executionLocation). This is taken directly from the online Azure documentation for ADF (https://docs.microsoft.com/en-us/azure/data-factory/v1/data-factory-data-movement-activities#global):

For example, to copy between Azure stores in Korea, you can specify "executionLocation": "Japan East" to route through Japan region (see sample JSON as reference).
Note

If the region of the destination data store is not in preceding list or undetectable, by default Copy Activity fails instead of going through an alternative region, unless executionLocation is specified. The supported region list will be expanded over time.

In the new V2 ADF service, the Integration Runtime (IR) feature is the primary way to move data in the cloud or on-prem. So, you may have to explicitly tell ADF about the location of your data source by creating an IR in your data store region and then reference that IR in your Linked Services definition using the new “connectVia” property. If you do not specify an explicit IR reference, then ADF will use a “default IR” which may not be able to resolve the location.

First, create an Integration Runtime in the region where your data store is located:

https://docs.microsoft.com/en-us/azure/data-factory/create-azure-integration-runtime#create-azure-ir

Then add the

connectVia

property to your Linked Service using a reference to that new IR:

https://docs.microsoft.com/en-us/azure/data-factory/concepts-datasets-linked-services#linked-service-json

Advertisements

Azure Big Data Analytics in the Cloud

3 Nov

Hi All … I’m BAAAACK! Now that I’ve settled into my new role in the Microsoft Azure field team as a Data Solution Architect, I’m getting back out on the speaker circuit. Here are my next 2 speaking engagements:

Tampa SQL BI Users Group

Global Big Data Conference Dec 9 Tampa

In each of those, I will be presenting Azure Big Data Analytics in the Cloud with Azure Data Platform overviews, demos and presentations.

I am uploading some of the demo content on my GitHub here

And the presentations on Slideshare here

 

Big Data + Cloud = Perfect Storm

3 Apr

Is this the perfect storm for those of us who live every day in the data world?

Two of the biggest buzzwords and changes to IT in the way that we manage data assets are occurring at just about the same time: data processing is moving from on-premises to the cloud … and the size and techniques that we use to manage and analyze that data is turning to Big Data distributed approaches.

Mobile is also a big focus for IT executives and probably fits in well as a 3 leg of the data platform and also part of this industry inflexion point. Microsoft is moving in this direction with BI tools in Excel and SharePoint in the Cloud with Office 365, Google has their Cloud-based productivity tools as well. But traditional business intelligence tools like Tableau, QlikView and Business Objects are still primarily on-premises products. Moving those from laptops to mobile devices like tablets and phones is where Big Data Analytics meets mobile. More on that in a later posting …

The ability to utilize cloud providers massive infrastructures to shard your data, process it in parallel and then analyze it is very compelling to control costs, complexity and maintenance of your own clusters.

The proof that Big Data in the Cloud can be the primary use case for Big Data Analytics becomes apparent when you look at what 3 of the biggest software companies, who also happen to be 3 of the largest consumers of Big Data Analytics, are taking to market:

  1. Microsoft HDInsight is Hadoop on Windows Azure
  2. Google’s BigQuery, which provides REST access into query across huge data sets
  3. Amazon’s Hadoop in the Cloud is Elastic MapReduce

Amazon is far & away the leader in this market today. They had the advantage of being early to embrace these approaches and used Big Data & NoSQL techniques internally for many years before taking their platforms to the public as a service with Amazon Web Services (AWS).

Google has also been a Big Data leader and user for a long time, but has a long way to go before they become a platform of choice for Big Data Analytics.

Microsoft is interesting in that they are investing heavily in Azure and their partnership with Hortonworks on the Hadoop for Windows platform. Microsoft’s REST-based object store (ASV) is similar to Amazon’s S3 and is something to consider when you look at future Big Data projects. Just keep in mind that HDInsight is still in preview (beta) at this time.

cbailiss

Microsoft SQL/BI and other bits and pieces

TIME

Current & Breaking News | National & World Updates

Tech Ramblings

My Thoughts on Software

SQL Authority with Pinal Dave

SQL Server Performance Tuning Expert

Insight Extractor - Blog

Paras Doshi's Blog on Analytics, Data Science & Business Intelligence.

The SQL Herald

Databases et al...

Chris Webb's BI Blog

Microsoft Analysis Services, MDX, DAX, Power Pivot, Power Query and Power BI

Bill on BI

Info about Business Analytics and Pentaho

Big Data Analytics

Occasional observations from a vet of many database, Big Data and BI battles

Blog Home for MSSQLDUDE

The life of a data geek