Archive | Cloud Computing RSS feed for this section

Monitoring Azure SQL Data Warehouse from SSMS

13 Apr

For those of us who lived through the Microsoft lifecycle of bringing to market a scale-out MPP data warehouse offering from DatAllegro to Parallel Data Warehouse (PDW) to Analytics Platform System, the technology behind that offering has evolved tremendously and were all happy to see it elevated to new heights in the cloud as Azure Data Warehouse.

But it’s important to understand the lineage from the perspective of same of the namings of the DMVs that you’ll use in SSMS. And, yes, those of us who had to evolve from the early PDW v1 days using Nexus because SSMS didn’t work with PDW are very excited with new T-SQL compatibility and SSMS compatibility.

Just make sure that when you are using SSMS to monitor your Azure Data Warehouse that you recognize that many of the DMVs from SQL Server land do not work in PDW or ADW and that many of the names of similar DMVS will have PDW in their names. But these will work with ADW.

For example, here is the documentation on monitoring your ADW workloads and grabbing SQL command syntax, similar to using DM EXEC REQUEST in SQL Server, but with PDW DMVs: https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-manage-monitor.

-- Find queries 
-- Replace request_id with value from Step 1.

SELECT waits.session_id,
      waits.request_id,  
      requests.command,
      requests.status,
      requests.start_time,  
      waits.type,
      waits.state,
      waits.object_type,
      waits.object_name
FROM   sys.dm_pdw_waits waits
   JOIN  sys.dm_pdw_exec_requests requests
   ON waits.request_id=requests.request_id
WHERE waits.request_id = 'QID####'
ORDER BY waits.object_name, waits.object_type, waits.state;

So now you know why you have to look for PDW for ADW DMVs!

Advanced Analytics Going Mainstream in 2017

8 Jan

Well, I finally feel comfortable saying it: Advanced Analytics is going mainstream this year. Even the term “Advanced Analytics” is a recent amalgam of long-time analytical disciplines that includes predictive analytics, descriptive analytics, data mining, machine learning and more. And now we refer to these techniques at Big Data scale as “Deep Learning”.

Here is Microsoft’s Joseph Sirosh talking about “Deep Learning in Every Software“. I would probably state it instead as “Advanced Analytics everywhere”. Not all scenarios require Big Data scale techniques, but most every application can gain an advantage by including cognitive capabilities as a natural aspect of the end-user experience.

Having spent years in the wilderness working on projects that included predicitve, data mining and machine learning, I wondered what are some of the recent technology and business drivers that have led us to the current inflection point in which advanced analytics begins finally breaking through into mainstream applications.

At Pentaho, we struggled for years to break through with machine learning projects using the popular Weka ML platform and retrofitted Weka to Big Data platforms Hadoop & Spark. At Microsoft, we had data mining built into the mainstream SQL Server database product for a long time, but it was a niche capability.

To me, these 5 factors have most impacted the recent turn, which is also the next-step result of US businesses focusing a lot of time, attention and resource on hiring, training and mentoring the Data Science role in their organizations.

  1. Open source projects, tools and libraries eliminated both the high-cost requirements of advanced analytics tools as well as making pre-built, trained and tested models available to non-math PhDs.
  2. R, Python, CRAN, TensorFlow, Cognitive Toolkit. I’ll also throw in my affinity to Weka because it was a trailblazer in the open source ML market and is still taught in many academic classes.
  3. Data quality and governance maturity: Decades of collecting data for business intelligence by the business and IT communities has raised awareness of the need to curate data, meaning that there are more quality data marts available for advanced analytical projects that can mine and optimize those marts.
  4. Artificial intelligence in everyday life: The more comfortable and familiar people become with AI, the more they will come to expect that in business applications as well. Everyday exposure to AI, ie. recommendation engines (Amazon, Netflix), face recognition (Facebook)
  5. Cloud Computing: Without needing to put resources into acquiring, standing-up and maintaining complex analytics architectures on-prem, I can just build machine learning experiments, explore data sets and operationalize learning as web services from my broswer or client tool using Azure Machine Learning, R Studio or Spark/R notebooks from an on-demand Hadoop cluster.

 

 

Azure Big Data Analytics in the Cloud

3 Nov

Hi All … I’m BAAAACK! Now that I’ve settled into my new role in the Microsoft Azure field team as a Data Solution Architect, I’m getting back out on the speaker circuit. Here are my next 2 speaking engagements:

Tampa SQL BI Users Group

Global Big Data Conference Dec 9 Tampa

In each of those, I will be presenting Azure Big Data Analytics in the Cloud with Azure Data Platform overviews, demos and presentations.

I am uploading some of the demo content on my GitHub here

And the presentations on Slideshare here

 

Big Data + Cloud = Perfect Storm

3 Apr

Is this the perfect storm for those of us who live every day in the data world?

Two of the biggest buzzwords and changes to IT in the way that we manage data assets are occurring at just about the same time: data processing is moving from on-premises to the cloud … and the size and techniques that we use to manage and analyze that data is turning to Big Data distributed approaches.

Mobile is also a big focus for IT executives and probably fits in well as a 3 leg of the data platform and also part of this industry inflexion point. Microsoft is moving in this direction with BI tools in Excel and SharePoint in the Cloud with Office 365, Google has their Cloud-based productivity tools as well. But traditional business intelligence tools like Tableau, QlikView and Business Objects are still primarily on-premises products. Moving those from laptops to mobile devices like tablets and phones is where Big Data Analytics meets mobile. More on that in a later posting …

The ability to utilize cloud providers massive infrastructures to shard your data, process it in parallel and then analyze it is very compelling to control costs, complexity and maintenance of your own clusters.

The proof that Big Data in the Cloud can be the primary use case for Big Data Analytics becomes apparent when you look at what 3 of the biggest software companies, who also happen to be 3 of the largest consumers of Big Data Analytics, are taking to market:

  1. Microsoft HDInsight is Hadoop on Windows Azure
  2. Google’s BigQuery, which provides REST access into query across huge data sets
  3. Amazon’s Hadoop in the Cloud is Elastic MapReduce

Amazon is far & away the leader in this market today. They had the advantage of being early to embrace these approaches and used Big Data & NoSQL techniques internally for many years before taking their platforms to the public as a service with Amazon Web Services (AWS).

Google has also been a Big Data leader and user for a long time, but has a long way to go before they become a platform of choice for Big Data Analytics.

Microsoft is interesting in that they are investing heavily in Azure and their partnership with Hortonworks on the Hadoop for Windows platform. Microsoft’s REST-based object store (ASV) is similar to Amazon’s S3 and is something to consider when you look at future Big Data projects. Just keep in mind that HDInsight is still in preview (beta) at this time.

cbailiss

Microsoft SQL/BI and other bits and pieces

TIME

Current & Breaking News | National & World Updates

Tech Ramblings

My Thoughts on Software

SQL Authority with Pinal Dave

SQL Server Performance Tuning Expert

Insight Extractor - Blog

Paras Doshi's Blog on Analytics, Data Science & Business Intelligence.

The SQL Herald

Databases et al...

Chris Webb's BI Blog

Microsoft Analysis Services, MDX, DAX, Power Pivot, Power Query and Power BI

Bill on BI

Info about Business Analytics and Pentaho

Big Data Analytics

Occasional observations from a vet of many database, Big Data and BI battles

Blog Home for MSSQLDUDE

The life of a data geek