Data solutions and data architectures encompass key areas in data management and provide guidance on how we ingest, process, store, maintain, consume, secure, and manage data. That is the lifecycle of most data analytics projects and systems.
Microsoft is fully invested in making Azure one of the most robust data platforms in the industry. It has developed tools and offerings on cloud that can operate at scale and provide the necessary data ingestion, processing, storing, and consumption options. From adding key capabilities in Azure Data Factory to making Power BI the most versatile BI tool in the industry, Microsoft has most of the ingredients for data processing.
The industry has also witnessed a new Microsoft philosophy in how the Azure data platform has evolved in recent years. Microsoft is not shy of deprecating products it has built — Azure Data Lake Analytics (ADLA) is no longer recommended as the processing option of choice for big data processing. Microsoft adopted Spark in HDInsight and gave customers the option to process data using open source frameworks. With the adoption and full-scale integration of Databricks, HDInsight has also taken a back seat.
However, there are some obvious gaps in the data management and maintenance space in Azure.
Metadata Management
When it comes to managing and maintaining data in the ecosystem, there is something Azure can do better. Managing data can be correlated to Metadata Management, which encompasses glossary, metadata management, and data lineage.
Microsoft does have a solution called Azure Data Catalog, but it seriously lacks features and capabilities that enterprises need. It does connect with most Azure sources but does not have full-fledged metadata tags. It has no capability in data lineage, and data discovery and source system scans are almost nonexistent. The surprising part is that the tool had not seen any updates in almost 2 years at the time of writing.
Is Microsoft going to refresh the tool any time soon? Is there a possible acquisition? We will have to wait for the answer.
Update: Microsoft has since released Microsoft Purview (formerly Azure Purview), which addresses many of these gaps with data cataloging, lineage, and governance capabilities.
Master Data Management
There is no service in Azure that provides Master Data Management (MDM) capabilities. Microsoft’s SQL Server Master Data Services (MDS) is for SQL Server on-premises only and cannot be ported to Azure.
The industry has seen growth in the number of tools in this space, but there are very few offerings that customers can use on Azure. Azure Marketplace has very few master data offerings. This feels like a space Microsoft is not planning to put energy into, leaving it to industry leaders.
Microsoft needs offerings in this area — whether through partnerships, acquisitions, or building something from the ground up. It will be interesting to see how things play out.