зеркало из https://github.com/Azure/Storage.git
autobuild
This commit is contained in:
Shawn Weisfeld Build
Родитель
b1ccba81f1
Коммит
64b6edf31d
|
|
@ -14,7 +14,7 @@
|
|||
<link rel="manifest" href="/manifest.json">
|
||||
<link rel="icon" href="/favicon.png" type="image/x-icon">
|
||||
<link rel="stylesheet" href="/book.min.6c7c6446dfdee7c8c933e9bbc6e80ee3ed6c913b2a59519f2092c3c6a9d63e55.css" integrity="sha256-bHxkRt/e58jJM+m7xugO4+1skTsqWVGfIJLDxqnWPlU=">
|
||||
<script defer src="/en.search.min.2781fc6c1b60cdb041c21ffb3dcfa81dd7146701076b8d5b74ac6dc760c9e031.js" integrity="sha256-J4H8bBtgzbBBwh/7Pc+oHdcUZwEHa41bdKxtx2DJ4DE="></script>
|
||||
<script defer src="/en.search.min.a092390ed9ca69153376eaacab4624d67749500866440e4365296f3fa207eb56.js" integrity="sha256-oJI5DtnKaRUzduqsq0Yk1ndJUAhmRA5DZSlvP6IH61Y="></script>
|
||||
<!--
|
||||
Made with Book Theme
|
||||
https://github.com/alex-shpak/hugo-book
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@
|
|||
<link rel="manifest" href="/manifest.json">
|
||||
<link rel="icon" href="/favicon.png" type="image/x-icon">
|
||||
<link rel="stylesheet" href="/book.min.6c7c6446dfdee7c8c933e9bbc6e80ee3ed6c913b2a59519f2092c3c6a9d63e55.css" integrity="sha256-bHxkRt/e58jJM+m7xugO4+1skTsqWVGfIJLDxqnWPlU=">
|
||||
<script defer src="/en.search.min.2781fc6c1b60cdb041c21ffb3dcfa81dd7146701076b8d5b74ac6dc760c9e031.js" integrity="sha256-J4H8bBtgzbBBwh/7Pc+oHdcUZwEHa41bdKxtx2DJ4DE="></script>
|
||||
<script defer src="/en.search.min.a092390ed9ca69153376eaacab4624d67749500866440e4365296f3fa207eb56.js" integrity="sha256-oJI5DtnKaRUzduqsq0Yk1ndJUAhmRA5DZSlvP6IH61Y="></script>
|
||||
<link rel="alternate" type="application/rss+xml" href="https://azurestorage.com/categories/index.xml" title="Azure Storage" />
|
||||
<!--
|
||||
Made with Book Theme
|
||||
|
|
@ -247,7 +247,7 @@ https://github.com/alex-shpak/hugo-book
|
|||
Azure Storage ©2021 <br />
|
||||
Visit the <a href="https://azure.microsoft.com/services/storage/">Azure Storage homepage</a> or read our <a href="https://docs.microsoft.com/azure/storage/">getting started guide</a> or the <a href="https://azure.microsoft.com/blog/topics/storage-backup-and-recovery/">Azure Storage Blog</a>. <br />
|
||||
Contact us: <a href="mailto:azurestoragefeedback@microsoft.com?subject=AzureStorage.com%20Feedback">azurestoragefeedback@microsoft.com</a>.<br />
|
||||
Generated on Fri, Feb 05 2021 20:15:27 UTC
|
||||
Generated on Mon, Feb 08 2021 23:18:57 UTC
|
||||
</footer>
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,442 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en" dir=>
|
||||
|
||||
<head>
|
||||
<meta name="generator" content="Hugo 0.80.0" />
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<meta name="description" content="Gen1 and Gen2 ACL Behavior Analysis # Overview # Azure Data Lake Storage is Microsoft’s optimized storage solution for big data analytics workloads. ADLS Gen2 is the combination of the current ADLS Gen1 and Blob storage.
|
||||
Azure Data Lake Storage Gen2 is built on Azure Blob storage and provides a set of capabilities dedicated to big data analytics. Data Lake Storage Gen2 combines features from Azure Data Lake Storage Gen1, such as file system semantics, directory, and file level security and low cost scalability, tiered storage, high availability/disaster recovery capabilities from Azure Blob storage.">
|
||||
<meta name="theme-color" content="#FFFFFF"><meta property="og:title" content="Gen1 and Gen2 ACL Behavior Analysis" />
|
||||
<meta property="og:description" content="Gen1 and Gen2 ACL Behavior Analysis # Overview # Azure Data Lake Storage is Microsoft’s optimized storage solution for big data analytics workloads. ADLS Gen2 is the combination of the current ADLS Gen1 and Blob storage.
|
||||
Azure Data Lake Storage Gen2 is built on Azure Blob storage and provides a set of capabilities dedicated to big data analytics. Data Lake Storage Gen2 combines features from Azure Data Lake Storage Gen1, such as file system semantics, directory, and file level security and low cost scalability, tiered storage, high availability/disaster recovery capabilities from Azure Blob storage." />
|
||||
<meta property="og:type" content="article" />
|
||||
<meta property="og:url" content="https://azurestorage.com/docs/analytics/adls-gen1-to-gen2-migration/adls-gen1-and-gen2-acl-behavior/" />
|
||||
|
||||
<title>Gen1 and Gen2 ACL Behavior Analysis | Azure Storage</title>
|
||||
<link rel="manifest" href="/manifest.json">
|
||||
<link rel="icon" href="/favicon.png" type="image/x-icon">
|
||||
<link rel="stylesheet" href="/book.min.6c7c6446dfdee7c8c933e9bbc6e80ee3ed6c913b2a59519f2092c3c6a9d63e55.css" integrity="sha256-bHxkRt/e58jJM+m7xugO4+1skTsqWVGfIJLDxqnWPlU=">
|
||||
<script defer src="/en.search.min.a092390ed9ca69153376eaacab4624d67749500866440e4365296f3fa207eb56.js" integrity="sha256-oJI5DtnKaRUzduqsq0Yk1ndJUAhmRA5DZSlvP6IH61Y="></script>
|
||||
<!--
|
||||
Made with Book Theme
|
||||
https://github.com/alex-shpak/hugo-book
|
||||
-->
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<body dir=>
|
||||
<input type="checkbox" class="hidden toggle" id="menu-control" />
|
||||
<input type="checkbox" class="hidden toggle" id="toc-control" />
|
||||
<main class="container flex">
|
||||
<aside class="book-menu">
|
||||
<div class="book-menu-content">
|
||||
|
||||
<nav>
|
||||
<h2 class="book-brand">
|
||||
<a href="/"><img src="/images/azure-icon.png" alt="Logo" /><span>Azure Storage</span>
|
||||
</a>
|
||||
</h2>
|
||||
|
||||
|
||||
<div class="book-search">
|
||||
<input type="text" id="book-search-input" placeholder="Search" aria-label="Search" maxlength="64" data-hotkeys="s/" />
|
||||
<div class="book-search-spinner hidden"></div>
|
||||
<ul id="book-search-results"></ul>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<ul>
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/analytics/" class="">Analytics</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/application-and-user-data/" class="">Application and User Data</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/backup-and-archive/" class="">Backup and Archive</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/hpc-iot-and-ai/" class="">HPC IoT and AI</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/storage-partners/" class="">Storage Partners</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/tools-and-utilities/" class="">Tools and Utilities</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
|
||||
<script>(function(){var menu=document.querySelector("aside.book-menu nav");addEventListener("beforeunload",function(event){localStorage.setItem("menu.scrollTop",menu.scrollTop);});menu.scrollTop=localStorage.getItem("menu.scrollTop");})();</script>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
<div class="book-page">
|
||||
<header class="book-header">
|
||||
|
||||
<div class="flex align-center justify-between">
|
||||
<label for="menu-control">
|
||||
<img src="/svg/menu.svg" class="book-icon" alt="Menu" />
|
||||
</label>
|
||||
|
||||
<strong>Gen1 and Gen2 ACL Behavior Analysis</strong>
|
||||
|
||||
<label for="toc-control">
|
||||
|
||||
<img src="/svg/toc.svg" class="book-icon" alt="Table of Contents" />
|
||||
|
||||
</label>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<aside class="hidden clearfix">
|
||||
|
||||
|
||||
<nav id="TableOfContents">
|
||||
<ul>
|
||||
<li><a href="#overview">Overview</a></li>
|
||||
<li><a href="#prerequisites">Prerequisites</a></li>
|
||||
<li><a href="#acl-behavior-in-adls-gen1-and-gen2">ACL Behavior in ADLS Gen1 and Gen2</a>
|
||||
<ul>
|
||||
<li><a href="#account-root-permissions">Account Root Permissions</a></li>
|
||||
<li><a href="#oid-upn-conversion">OID-UPN Conversion</a></li>
|
||||
<li><a href="#rbac-user-role-significance">RBAC User Role Significance</a></li>
|
||||
<li><a href="#store-default-permission">Store Default Permission</a></li>
|
||||
<li><a href="#user-provided-permission-on-filedirectory-creation">User Provided Permission on File/Directory Creation</a></li>
|
||||
<li><a href="#set-permission-with-no-permission-provided">Set Permission with No Permission Provided</a></li>
|
||||
<li><a href="#nested-file-or-directory-creation-for-non-owner-user">Nested File or Directory Creation For Non-Owner User</a></li>
|
||||
<li><a href="#umask-support">Umask Support</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#references">References</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
</aside>
|
||||
|
||||
|
||||
</header>
|
||||
|
||||
|
||||
|
||||
<article class="markdown"><h1 id="gen1-and-gen2-acl-behavior-analysis">
|
||||
Gen1 and Gen2 ACL Behavior Analysis
|
||||
<a class="anchor" href="#gen1-and-gen2-acl-behavior-analysis">#</a>
|
||||
</h1>
|
||||
<h2 id="overview">
|
||||
Overview
|
||||
<a class="anchor" href="#overview">#</a>
|
||||
</h2>
|
||||
<p>Azure Data Lake Storage is Microsoft’s optimized storage solution for big data analytics workloads. ADLS Gen2 is the combination of the current ADLS Gen1 and Blob storage.<br>
|
||||
Azure Data Lake Storage Gen2 is built on Azure Blob storage and provides a set of capabilities dedicated to big data analytics. Data Lake Storage Gen2 combines features from Azure Data Lake Storage Gen1, such as file system semantics, directory, and file level security and low cost scalability, tiered storage, high availability/disaster recovery capabilities from Azure Blob storage. Azure Data Lake Storage Gen1 implements an access control model that derives from HDFS, which in turn derives from the POSIX access control model. Azure Data Lake Storage Gen2 implements an access control model that supports both Azure role-based access control (Azure RBAC) and POSIX-like access control lists (ACLs).</p>
|
||||
<p>This article summarizes the behavioral differences of the access control models for Data Lake Storage Gen1 and Gen2.</p>
|
||||
<h2 id="prerequisites">
|
||||
Prerequisites
|
||||
<a class="anchor" href="#prerequisites">#</a>
|
||||
</h2>
|
||||
<ul>
|
||||
<li><strong>Active Azure Subscription</strong></li>
|
||||
<li><strong>Azure Data Lake Storage Gen1 and Gen2</strong></li>
|
||||
<li><strong>Azure Key Vault</strong>. Required keys and secrets to be configured here.</li>
|
||||
<li><strong>Service principal</strong> with read, write and execute permission to the resource group, key vault, data lake store Gen1 and data lake store Gen2. To learn more, see <a href="https://docs.microsoft.com/azure/active-directory/develop/howto-create-service-principal-portal">create service principal account</a> and to provide SPN access to Gen1 refer to <a href="https://docs.microsoft.com/azure/data-lake-store/data-lake-store-service-to-service-authenticate-using-active-directory">SPN access to Gen1</a></li>
|
||||
<li><strong>Java Development Kit (JDK 7 or higher, using Java version 1.7 or higher)</strong> for Filesystem operations on Azure Data Lake Storage Gen1 and Gen2</li>
|
||||
</ul>
|
||||
<h2 id="acl-behavior-in-adls-gen1-and-gen2">
|
||||
ACL Behavior in ADLS Gen1 and Gen2
|
||||
<a class="anchor" href="#acl-behavior-in-adls-gen1-and-gen2">#</a>
|
||||
</h2>
|
||||
<h3 id="account-root-permissions">
|
||||
Account Root Permissions
|
||||
<a class="anchor" href="#account-root-permissions">#</a>
|
||||
</h3>
|
||||
<p>Check GetFileStatus and GetAclStatus APIs with or without permissions on root Account path</p>
|
||||
<ul>
|
||||
<li><strong>GEN1 Behavior</strong>: Permission required on Account root- RX(minimum) or RWX , to get an account root content view</li>
|
||||
<li><strong>GEN2 Behavior</strong>: A user with or without permissions on container root can view account root content</li>
|
||||
</ul>
|
||||
<h3 id="oid-upn-conversion">
|
||||
OID-UPN Conversion
|
||||
<a class="anchor" href="#oid-upn-conversion">#</a>
|
||||
</h3>
|
||||
<p>Check the identity inputs for UPN format APIs (Eg:GetAclStatus, Liststatus ,GetFileStatus) and OID format APIs (Eg:SetAcl, ModifyAclEntries, RemoveAclEntries)</p>
|
||||
<ul>
|
||||
<li><strong>GEN1 Behavior</strong>: OID <-> UPN conversion is supported for Users, Service principals and groups
|
||||
<blockquote>
|
||||
<p>Note: For groups, as there is no UPN, conversion is done to Display name property</p>
|
||||
</blockquote>
|
||||
</li>
|
||||
<li><strong>GEN2 Behavior</strong>: Supports only User OID-UPN conversion.
|
||||
<blockquote>
|
||||
<p>Note: For service principal or group ,as UPN or Display Name is not unique, the derived OID could end up being an unintended identity</p>
|
||||
</blockquote>
|
||||
</li>
|
||||
</ul>
|
||||
<h3 id="rbac-user-role-significance">
|
||||
RBAC User Role Significance
|
||||
<a class="anchor" href="#rbac-user-role-significance">#</a>
|
||||
</h3>
|
||||
<p>RBAC roles and access control</p>
|
||||
<ul>
|
||||
<li><strong>GEN1 Behavior</strong>: All users in RBAC Owner role are superusers. All other users (non-superusers),need to have permission that abides by File Folder ACL. Refer <a href="https://docs.microsoft.com/azure/data-lake-store/data-lake-store-access-control">here</a> for more details</li>
|
||||
<li><strong>GEN2 Behavior</strong>: All users in ‘RBAC -Storage blob data owner’ role are superusers. All other users can be provided different roles(contributor, reader etc.) that govern their read, write and delete permissions, this takes precedence to the ACLs sent on individual file or folder. Refer <a href="https://docs.microsoft.com/azure/data-lake-store/data-lake-store-access-control">here</a> for more details</li>
|
||||
</ul>
|
||||
<h3 id="store-default-permission">
|
||||
Store Default Permission
|
||||
<a class="anchor" href="#store-default-permission">#</a>
|
||||
</h3>
|
||||
<p>Check if default permission is considered during file and directory creation</p>
|
||||
<ul>
|
||||
<li><strong>GEN1 Behavior</strong>: Permissions for an item(file/directory) cannot be inherited from the parent items. Refer <a href="https://docs.microsoft.com/azure/data-lake-store/data-lake-store-access-control">here</a></li>
|
||||
<li><strong>GEN2 Behavior</strong>: Permissions are only inherited if default permissions have been set on the parent items before the child items have been created. Refer <a href="https://docs.microsoft.com/azure/storage/blobs/data-lake-storage-access-control">here</a></li>
|
||||
</ul>
|
||||
<h3 id="user-provided-permission-on-filedirectory-creation">
|
||||
User Provided Permission on File/Directory Creation
|
||||
<a class="anchor" href="#user-provided-permission-on-filedirectory-creation">#</a>
|
||||
</h3>
|
||||
<p>Create a file/directory with explicit permission</p>
|
||||
<ul>
|
||||
<li><strong>GEN1 Behavior</strong>: File/Directory is created, and the final permission will be same as the user provided permission</li>
|
||||
<li><strong>GEN2 Behavior</strong>: File/Directory is created, and the final permission will be computed as [user provided permission ^ umask (currently 027 in code)]</li>
|
||||
</ul>
|
||||
<h3 id="set-permission-with-no-permission-provided">
|
||||
Set Permission with No Permission Provided
|
||||
<a class="anchor" href="#set-permission-with-no-permission-provided">#</a>
|
||||
</h3>
|
||||
<p>Set permission Api is called with permission = null/space and permission parameter not present</p>
|
||||
<ul>
|
||||
<li><strong>GEN1 Behavior</strong>: A default value of 770 is set for both file and directory</li>
|
||||
<li><strong>GEN2 Behavior</strong>: Gen2 will return bad request as permission parameter is mandatory</li>
|
||||
</ul>
|
||||
<h3 id="nested-file-or-directory-creation-for-non-owner-user">
|
||||
Nested File or Directory Creation For Non-Owner User
|
||||
<a class="anchor" href="#nested-file-or-directory-creation-for-non-owner-user">#</a>
|
||||
</h3>
|
||||
<p>Check if wx permission on parent is copied to nested file/directory when non-owner creates it. (i.e. dir1 exists and user desires to create dir2/dir3/a.txt or dir2/dir3/dir4)</p>
|
||||
<ul>
|
||||
<li><strong>GEN1 Behavior</strong>: Adds wx permissions for owner in the sub directory</li>
|
||||
<li><strong>GEN2 Behavior</strong>: Doesn’t add wx permissions in the sub directory</li>
|
||||
</ul>
|
||||
<h3 id="umask-support">
|
||||
Umask Support
|
||||
<a class="anchor" href="#umask-support">#</a>
|
||||
</h3>
|
||||
<p>Permissions of file/directory can be controlled by applying UMASK on it.</p>
|
||||
<ul>
|
||||
<li><strong>GEN1 Behavior</strong>: Client needs to apply umask on the permission on new file/directory before sending the request to server.
|
||||
<blockquote>
|
||||
<p>Note: Server doesn’t provide explicit support in accepting umask as an input</p>
|
||||
</blockquote>
|
||||
</li>
|
||||
<li><strong>GEN2 Behavior</strong>: Clients can provide umask as request query params during file and directory creations. If client does not pass umask parameter, default umask 027 will be applied on file/directory</li>
|
||||
</ul>
|
||||
<h2 id="references">
|
||||
References
|
||||
<a class="anchor" href="#references">#</a>
|
||||
</h2>
|
||||
<ul>
|
||||
<li><a href="https://docs.microsoft.com/azure/storage/blobs/data-lake-storage-access-control">ACL in ADLS Gen2</a></li>
|
||||
<li><a href="https://docs.microsoft.com/azure/data-lake-store/data-lake-store-access-control">ACL in ADLS Gen1</a></li>
|
||||
<li><a href="https://docs.microsoft.com/azure/data-lake-store/data-lake-store-secure-data">Securing data stored in Azure Data Lake Storage Gen1</a></li>
|
||||
</ul>
|
||||
</article>
|
||||
|
||||
|
||||
|
||||
<footer class="book-footer">
|
||||
|
||||
<div class="flex flex-wrap justify-between">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<hr />
|
||||
Azure Storage ©2021 <br />
|
||||
Visit the <a href="https://azure.microsoft.com/services/storage/">Azure Storage homepage</a> or read our <a href="https://docs.microsoft.com/azure/storage/">getting started guide</a> or the <a href="https://azure.microsoft.com/blog/topics/storage-backup-and-recovery/">Azure Storage Blog</a>. <br />
|
||||
Contact us: <a href="mailto:azurestoragefeedback@microsoft.com?subject=AzureStorage.com%20Feedback">azurestoragefeedback@microsoft.com</a>.<br />
|
||||
Generated on Mon, Feb 08 2021 23:18:57 UTC
|
||||
</footer>
|
||||
|
||||
|
||||
|
||||
<div class="book-comments">
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<label for="menu-control" class="hidden book-menu-overlay"></label>
|
||||
</div>
|
||||
|
||||
|
||||
<aside class="book-toc">
|
||||
<div class="book-toc-content">
|
||||
|
||||
|
||||
<nav id="TableOfContents">
|
||||
<ul>
|
||||
<li><a href="#overview">Overview</a></li>
|
||||
<li><a href="#prerequisites">Prerequisites</a></li>
|
||||
<li><a href="#acl-behavior-in-adls-gen1-and-gen2">ACL Behavior in ADLS Gen1 and Gen2</a>
|
||||
<ul>
|
||||
<li><a href="#account-root-permissions">Account Root Permissions</a></li>
|
||||
<li><a href="#oid-upn-conversion">OID-UPN Conversion</a></li>
|
||||
<li><a href="#rbac-user-role-significance">RBAC User Role Significance</a></li>
|
||||
<li><a href="#store-default-permission">Store Default Permission</a></li>
|
||||
<li><a href="#user-provided-permission-on-filedirectory-creation">User Provided Permission on File/Directory Creation</a></li>
|
||||
<li><a href="#set-permission-with-no-permission-provided">Set Permission with No Permission Provided</a></li>
|
||||
<li><a href="#nested-file-or-directory-creation-for-non-owner-user">Nested File or Directory Creation For Non-Owner User</a></li>
|
||||
<li><a href="#umask-support">Umask Support</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#references">References</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
</main>
|
||||
|
||||
<script src="https://code.jquery.com/jquery-3.5.1.slim.min.js" integrity="sha256-4+XzXVhsDmqanXGHaHvgh1gMQKX40OUvDEBTu8JcmNs=" crossorigin="anonymous"></script>
|
||||
|
||||
<script>
|
||||
$('a[href^="http://"], a[href^="https://"]').not('a[href*="'+location.hostname+'"]').attr('target','_blank');
|
||||
</script>
|
||||
|
||||
<script type="text/javascript">
|
||||
!function(T,l,y){var S=T.location,u="script",k="instrumentationKey",D="ingestionendpoint",C="disableExceptionTracking",E="ai.device.",I="toLowerCase",b="crossOrigin",w="POST",e="appInsightsSDK",t=y.name||"appInsights";(y.name||T[e])&&(T[e]=t);var n=T[t]||function(d){var g=!1,f=!1,m={initialize:!0,queue:[],sv:"4",version:2,config:d};function v(e,t){var n={},a="Browser";return n[E+"id"]=a[I](),n[E+"type"]=a,n["ai.operation.name"]=S&&S.pathname||"_unknown_",n["ai.internal.sdkVersion"]="javascript:snippet_"+(m.sv||m.version),{time:function(){var e=new Date;function t(e){var t=""+e;return 1===t.length&&(t="0"+t),t}return e.getUTCFullYear()+"-"+t(1+e.getUTCMonth())+"-"+t(e.getUTCDate())+"T"+t(e.getUTCHours())+":"+t(e.getUTCMinutes())+":"+t(e.getUTCSeconds())+"."+((e.getUTCMilliseconds()/1e3).toFixed(3)+"").slice(2,5)+"Z"}(),iKey:e,name:"Microsoft.ApplicationInsights."+e.replace(/-/g,"")+"."+t,sampleRate:100,tags:n,data:{baseData:{ver:2}}}}var h=d.url||y.src;if(h){function a(e){var t,n,a,i,r,o,s,c,p,l,u;g=!0,m.queue=[],f||(f=!0,t=h,s=function(){var e={},t=d.connectionString;if(t)for(var n=t.split(";"),a=0;a<n.length;a++){var i=n[a].split("=");2===i.length&&(e[i[0][I]()]=i[1])}if(!e[D]){var r=e.endpointsuffix,o=r?e.location:null;e[D]="https://"+(o?o+".":"")+"dc."+(r||"services.visualstudio.com")}return e}(),c=s[k]||d[k]||"",p=s[D],l=p?p+"/v2/track":config.endpointUrl,(u=[]).push((n="SDK LOAD Failure: Failed to load Application Insights SDK script (See stack for details)",a=t,i=l,(o=(r=v(c,"Exception")).data).baseType="ExceptionData",o.baseData.exceptions=[{typeName:"SDKLoadFailed",message:n.replace(/\./g,"-"),hasFullStack:!1,stack:n+"\nSnippet failed to load ["+a+"] -- Telemetry is disabled\nHelp Link: https://go.microsoft.com/fwlink/?linkid=2128109\nHost: "+(S&&S.pathname||"_unknown_")+"\nEndpoint: "+i,parsedStack:[]}],r)),u.push(function(e,t,n,a){var i=v(c,"Message"),r=i.data;r.baseType="MessageData";var o=r.baseData;return o.message='AI (Internal): 99 message:"'+("SDK LOAD Failure: Failed to load Application Insights SDK script (See stack for details) ("+n+")").replace(/\"/g,"")+'"',o.properties={endpoint:a},i}(0,0,t,l)),function(e,t){if(JSON){var n=T.fetch;if(n&&!y.useXhr)n(t,{method:w,body:JSON.stringify(e),mode:"cors"});else if(XMLHttpRequest){var a=new XMLHttpRequest;a.open(w,t),a.setRequestHeader("Content-type","application/json"),a.send(JSON.stringify(e))}}}(u,l))}function i(e,t){f||setTimeout(function(){!t&&m.core||a()},500)}var e=function(){var n=l.createElement(u);n.src=h;var e=y[b];return!e&&""!==e||"undefined"==n[b]||(n[b]=e),n.onload=i,n.onerror=a,n.onreadystatechange=function(e,t){"loaded"!==n.readyState&&"complete"!==n.readyState||i(0,t)},n}();y.ld<0?l.getElementsByTagName("head")[0].appendChild(e):setTimeout(function(){l.getElementsByTagName(u)[0].parentNode.appendChild(e)},y.ld||0)}try{m.cookie=l.cookie}catch(p){}function t(e){for(;e.length;)!function(t){m[t]=function(){var e=arguments;g||m.queue.push(function(){m[t].apply(m,e)})}}(e.pop())}var n="track",r="TrackPage",o="TrackEvent";t([n+"Event",n+"PageView",n+"Exception",n+"Trace",n+"DependencyData",n+"Metric",n+"PageViewPerformance","start"+r,"stop"+r,"start"+o,"stop"+o,"addTelemetryInitializer","setAuthenticatedUserContext","clearAuthenticatedUserContext","flush"]),m.SeverityLevel={Verbose:0,Information:1,Warning:2,Error:3,Critical:4};var s=(d.extensionConfig||{}).ApplicationInsightsAnalytics||{};if(!0!==d[C]&&!0!==s[C]){method="onerror",t(["_"+method]);var c=T[method];T[method]=function(e,t,n,a,i){var r=c&&c(e,t,n,a,i);return!0!==r&&m["_"+method]({message:e,url:t,lineNumber:n,columnNumber:a,error:i}),r},d.autoExceptionInstrumented=!0}return m}(y.cfg);(T[t]=n).queue&&0===n.queue.length&&n.trackPageView({})}(window,document,{
|
||||
src: "https://az416426.vo.msecnd.net/scripts/b/ai.2.min.js",
|
||||
|
||||
|
||||
|
||||
|
||||
cfg: {
|
||||
instrumentationKey: "0600b273-5440-42cc-9fe4-51924c721ce0"
|
||||
|
||||
}});
|
||||
</script>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,477 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en" dir=>
|
||||
|
||||
<head>
|
||||
<meta name="generator" content="Hugo 0.80.0" />
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<meta name="description" content="Application and Workload Update # Overview # The purpose of this document is to provide steps and ways to migrate the workloads and applications from Gen1 to Gen2 after data migration is completed.
|
||||
This can be applicable for below migration patterns:
|
||||
Incremental Copy pattern
|
||||
Lift and Shift copy pattern
|
||||
Dual Pipeline pattern
|
||||
As part of this, we will configure services in workloads used and update the applications to point to Gen2 mount.">
|
||||
<meta name="theme-color" content="#FFFFFF"><meta property="og:title" content="Application and Workload Update" />
|
||||
<meta property="og:description" content="Application and Workload Update # Overview # The purpose of this document is to provide steps and ways to migrate the workloads and applications from Gen1 to Gen2 after data migration is completed.
|
||||
This can be applicable for below migration patterns:
|
||||
Incremental Copy pattern
|
||||
Lift and Shift copy pattern
|
||||
Dual Pipeline pattern
|
||||
As part of this, we will configure services in workloads used and update the applications to point to Gen2 mount." />
|
||||
<meta property="og:type" content="article" />
|
||||
<meta property="og:url" content="https://azurestorage.com/docs/analytics/adls-gen1-to-gen2-migration/application-update/" />
|
||||
|
||||
<title>Application and Workload Update | Azure Storage</title>
|
||||
<link rel="manifest" href="/manifest.json">
|
||||
<link rel="icon" href="/favicon.png" type="image/x-icon">
|
||||
<link rel="stylesheet" href="/book.min.6c7c6446dfdee7c8c933e9bbc6e80ee3ed6c913b2a59519f2092c3c6a9d63e55.css" integrity="sha256-bHxkRt/e58jJM+m7xugO4+1skTsqWVGfIJLDxqnWPlU=">
|
||||
<script defer src="/en.search.min.a092390ed9ca69153376eaacab4624d67749500866440e4365296f3fa207eb56.js" integrity="sha256-oJI5DtnKaRUzduqsq0Yk1ndJUAhmRA5DZSlvP6IH61Y="></script>
|
||||
<!--
|
||||
Made with Book Theme
|
||||
https://github.com/alex-shpak/hugo-book
|
||||
-->
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<body dir=>
|
||||
<input type="checkbox" class="hidden toggle" id="menu-control" />
|
||||
<input type="checkbox" class="hidden toggle" id="toc-control" />
|
||||
<main class="container flex">
|
||||
<aside class="book-menu">
|
||||
<div class="book-menu-content">
|
||||
|
||||
<nav>
|
||||
<h2 class="book-brand">
|
||||
<a href="/"><img src="/images/azure-icon.png" alt="Logo" /><span>Azure Storage</span>
|
||||
</a>
|
||||
</h2>
|
||||
|
||||
|
||||
<div class="book-search">
|
||||
<input type="text" id="book-search-input" placeholder="Search" aria-label="Search" maxlength="64" data-hotkeys="s/" />
|
||||
<div class="book-search-spinner hidden"></div>
|
||||
<ul id="book-search-results"></ul>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<ul>
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/analytics/" class="">Analytics</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/application-and-user-data/" class="">Application and User Data</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/backup-and-archive/" class="">Backup and Archive</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/hpc-iot-and-ai/" class="">HPC IoT and AI</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/storage-partners/" class="">Storage Partners</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/tools-and-utilities/" class="">Tools and Utilities</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
|
||||
<script>(function(){var menu=document.querySelector("aside.book-menu nav");addEventListener("beforeunload",function(event){localStorage.setItem("menu.scrollTop",menu.scrollTop);});menu.scrollTop=localStorage.getItem("menu.scrollTop");})();</script>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
<div class="book-page">
|
||||
<header class="book-header">
|
||||
|
||||
<div class="flex align-center justify-between">
|
||||
<label for="menu-control">
|
||||
<img src="/svg/menu.svg" class="book-icon" alt="Menu" />
|
||||
</label>
|
||||
|
||||
<strong>Application and Workload Update</strong>
|
||||
|
||||
<label for="toc-control">
|
||||
|
||||
<img src="/svg/toc.svg" class="book-icon" alt="Table of Contents" />
|
||||
|
||||
</label>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<aside class="hidden clearfix">
|
||||
|
||||
|
||||
<nav id="TableOfContents">
|
||||
<ul>
|
||||
<li><a href="#overview">Overview</a></li>
|
||||
<li><a href="#prerequisites">Prerequisites</a></li>
|
||||
<li><a href="#how-to-configure-and-update-azure-databricks">How to Configure and Update Azure Databricks</a></li>
|
||||
<li><a href="#how-to-configure-and-update-azure-datafactory">How to Configure and Update Azure Datafactory</a></li>
|
||||
<li><a href="#how-to-configure-and-update-hdinsight">How to Configure and update HDInsight</a></li>
|
||||
<li><a href="#how-to-configure-and-update-azure-synapse-analytics">How to configure and update Azure Synapse Analytics</a></li>
|
||||
<li><a href="#cutover-from-gen1-to-gen2">Cutover from Gen1 to Gen2</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
</aside>
|
||||
|
||||
|
||||
</header>
|
||||
|
||||
|
||||
|
||||
<article class="markdown"><h1 id="application-and-workload-update">
|
||||
Application and Workload Update
|
||||
<a class="anchor" href="#application-and-workload-update">#</a>
|
||||
</h1>
|
||||
<h2 id="overview">
|
||||
Overview
|
||||
<a class="anchor" href="#overview">#</a>
|
||||
</h2>
|
||||
<p>The purpose of this document is to provide steps and ways to migrate the workloads and applications from <strong>Gen1</strong> to <strong>Gen2</strong> after data migration is completed.</p>
|
||||
<p>This can be applicable for below migration patterns:</p>
|
||||
<ol>
|
||||
<li>
|
||||
<p>Incremental Copy pattern</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Lift and Shift copy pattern</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Dual Pipeline pattern</p>
|
||||
</li>
|
||||
</ol>
|
||||
<p>As part of this, we will <a href="https://docs.microsoft.com/azure/storage/blobs/data-lake-storage-supported-azure-services">configure services in workloads</a> used and update the applications to point to Gen2 mount.</p>
|
||||
<blockquote>
|
||||
<p>NOTE: We will be covering below azure services</p>
|
||||
</blockquote>
|
||||
<ul>
|
||||
<li>Azure Data Factory
|
||||
<ul>
|
||||
<li><a href="https://docs.microsoft.com/azure/data-factory/load-azure-data-lake-storage-gen2">Load data into Azure Data Lake Storage Gen2 with Azure Data Factory</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>Azure Databricks
|
||||
<ul>
|
||||
<li><a href="https://docs.microsoft.com/azure/databricks/data/data-sources/azure/azure-datalake-gen2">Use with Azure Databricks</a></li>
|
||||
<li><a href="https://docs.microsoft.com/azure/storage/blobs/data-lake-storage-quickstart-create-databricks-account">Quickstart: Analyze data in Azure Data Lake Storage Gen2 by using Azure Databricks</a></li>
|
||||
<li><a href="https://docs.microsoft.com/azure/azure-databricks/databricks-extract-load-sql-data-warehouse">Tutorial: Extract, transform, and load data by using Azure Databricks</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>SQL Data Warehouse
|
||||
<ul>
|
||||
<li><a href="https://docs.microsoft.com/azure/data-factory/load-azure-sql-data-warehouse">Use with Azure SQL Data Warehouse</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>HDInsight
|
||||
<ul>
|
||||
<li><a href="https://docs.microsoft.com/azure/hdinsight/hdinsight-hadoop-use-data-lake-storage-gen2">Use Azure Data Lake Storage Gen2 with Azure HDInsight clusters</a></li>
|
||||
<li><a href="https://docs.microsoft.com/azure/storage/blobs/data-lake-storage-tutorial-extract-transform-load-hive">Tutorial: Extract, transform, and load data by using Azure HDInsight</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<h2 id="prerequisites">
|
||||
Prerequisites
|
||||
<a class="anchor" href="#prerequisites">#</a>
|
||||
</h2>
|
||||
<p><strong>The migration of data from Gen1 to Gen2 should be completed</strong></p>
|
||||
<h2 id="how-to-configure-and-update-azure-databricks">
|
||||
How to Configure and Update Azure Databricks
|
||||
<a class="anchor" href="#how-to-configure-and-update-azure-databricks">#</a>
|
||||
</h2>
|
||||
<p>Applies where Databricks is used for data ingestion to ADLS Gen1.</p>
|
||||
<p><strong>Before the migration</strong>:</p>
|
||||
<ol>
|
||||
<li>
|
||||
<p><strong>Mount configured to Gen1 path</strong></p>
|
||||
<p>Sample code showing mount path configured for ADLS Gen1 using service principle:</p>
|
||||
<p><img src="../images/79265180-90c91b80-7e4a-11ea-9000-0f86aa7c6ebb.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Set up DataBricks cluster for scheduled job run</strong></p>
|
||||
<p>Sample snapshot of working code:</p>
|
||||
<p><img src="../images/83693292-ac2ee800-a5aa-11ea-878e-e8f6d72daf72.png" alt="image" /></p>
|
||||
<blockquote>
|
||||
<p>Note: Refer to <a href="../incremental/incremental.zip">Application\IncrementalSampleLoad.py</a> script for more details.</p>
|
||||
</blockquote>
|
||||
</li>
|
||||
</ol>
|
||||
<p><strong>After the migration</strong>:</p>
|
||||
<ol>
|
||||
<li>
|
||||
<p><strong>Change the mount configuration to Gen2 container</strong></p>
|
||||
<p><img src="../images/79016042-dfad4300-7b22-11ea-97c2-274e533a37e7.png" alt="image" /></p>
|
||||
<blockquote>
|
||||
<p>Note: <strong>Stop</strong> the job scheduler and change the mount configuration to point to Gen2 with the same mount name.</p>
|
||||
</blockquote>
|
||||
<p><img src="../images/79009824-49beeb80-7b15-11ea-8d14-ce444f7fd4b8.png" alt="image" /></p>
|
||||
<blockquote>
|
||||
<p>Note: Refer to <a href="../incremental/incremental.zip">Application\MountConfiguration.py</a> script for more details.</p>
|
||||
</blockquote>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Reschedule the job scheduler</strong></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Check for the new files getting generated at the Gen2 root folder path</strong></p>
|
||||
</li>
|
||||
</ol>
|
||||
<h2 id="how-to-configure-and-update-azure-datafactory">
|
||||
How to Configure and Update Azure Datafactory
|
||||
<a class="anchor" href="#how-to-configure-and-update-azure-datafactory">#</a>
|
||||
</h2>
|
||||
<p>Once the data migration using ADF is completed from ADLS Gen1 to Gen2, follow the below steps:</p>
|
||||
<ol>
|
||||
<li>
|
||||
<p><strong>Stop the trigger to Gen1</strong> used as part of Incremental copy pattern.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Modify the existing factory by creating new linked service to point to Gen2 storage</strong>.</p>
|
||||
<p>Go to –> <strong>Azure Data Factory</strong> –> <strong>Click on Author</strong> –> <strong>Connections</strong> –> <strong>Linked Service</strong> –> <strong>click on New</strong> –> <strong>Choose Azure Data Lake Storage Gen2</strong> –> <strong>Click on Continue button</strong></p>
|
||||
<p><img src="../images/79276321-a3e4e700-7e5c-11ea-9908-b013e2d1e12b.png" alt="image" /></p>
|
||||
<p>Provide the details to create new Linked service to point to Gen2 storage account.</p>
|
||||
<p><img src="../images/79276405-cd057780-7e5c-11ea-9c31-95dfd26db5b9.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Modify the existing factory by creating new dataset in Gen2 storage</strong>.</p>
|
||||
<p>Go to –> <strong>Azure Data Factory</strong> –> <strong>Click on Author</strong> –> <strong>Click on Pipelines</strong> –> <strong>Select the pipeline</strong> –> <strong>Click on Activity</strong> –> <strong>Click on sink tab</strong> –> Choose the dataset to point to Gen2</p>
|
||||
<p><img src="../images/83690089-eeedc180-a5a4-11ea-8a57-28a22822a595.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Click on Publish all</strong></p>
|
||||
<p><img src="../images/79280406-21145a00-7e65-11ea-8950-bff27882c4de.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Go to Triggers and activate it</strong>.</p>
|
||||
<p><img src="../images/79280526-66388c00-7e65-11ea-895e-915018092b67.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Check for the new files getting generated at the Gen2 root folder path</strong></p>
|
||||
</li>
|
||||
</ol>
|
||||
<h2 id="how-to-configure-and-update-hdinsight">
|
||||
How to Configure and update HDInsight
|
||||
<a class="anchor" href="#how-to-configure-and-update-hdinsight">#</a>
|
||||
</h2>
|
||||
<p>Applies where HDInsight is used as workload to process the Raw data and execute the transformations. Below is the step by step process used as part of <a href="../dual-pipeline/#how-to-set-up-gen1-data-pipeline">Dual pipeline pattern</a>.</p>
|
||||
<p><strong>Prerequisite</strong></p>
|
||||
<p>Two HDInsight clusters to be created for each Gen1 and Gen2 storage.</p>
|
||||
<p><strong>Before Migration</strong></p>
|
||||
<p>The Hive script is mounted to Gen1 endpoint as shown below:</p>
|
||||
<p><img src="../images/83672012-74b04380-a58a-11ea-89b6-54564aeb52f5.png" alt="image" /></p>
|
||||
<p><strong>After Migration</strong></p>
|
||||
<p>The Hive script is mounted to Gen2 endpoint as shown below:</p>
|
||||
<p><img src="../images/83672806-b8f01380-a58b-11ea-8c16-ae0c662d7de6.png" alt="image" /></p>
|
||||
<p>Once all the existing data is moved from Gen1 to Gen2, Start running the worloads at Gen2 endpoint.</p>
|
||||
<h2 id="how-to-configure-and-update-azure-synapse-analytics">
|
||||
How to configure and update Azure Synapse Analytics
|
||||
<a class="anchor" href="#how-to-configure-and-update-azure-synapse-analytics">#</a>
|
||||
</h2>
|
||||
<p>Applies to the data pipelines having Azure synapse analytics formerly called as <strong>Azure SQL DW</strong> as one of the workloads. Below is the step by step process used as part of <a href="../dual-pipeline/#how-to-set-up-gen1-data-pipeline">Dual pipeline pattern</a></p>
|
||||
<p><strong>Before Migration</strong></p>
|
||||
<p>The stored procedure activity is pointed to Gen1 mount path.</p>
|
||||
<p><img src="../images/84082011-eece3700-a993-11ea-8ba0-f4efab65c0e9.png" alt="image" /></p>
|
||||
<p><strong>After Migration</strong></p>
|
||||
<p>The stored procedure activity is pointed to Gen2 endpoint.</p>
|
||||
<p><img src="../images/84082177-42408500-a994-11ea-84ba-d575ba1e3611.png" alt="image" /></p>
|
||||
<p><strong>Run the trigger</strong></p>
|
||||
<p><img src="../images/84082352-8f245b80-a994-11ea-9132-45e335429145.png" alt="image" /></p>
|
||||
<p><strong>Check the SQL table in the Data warehouse for new data load</strong>.</p>
|
||||
<h2 id="cutover-from-gen1-to-gen2">
|
||||
Cutover from Gen1 to Gen2
|
||||
<a class="anchor" href="#cutover-from-gen1-to-gen2">#</a>
|
||||
</h2>
|
||||
<p>After you’re confident that your applications and workloads are stable on Gen2, you can begin using Gen2 to satisfy your business scenarios. Turn off any remaining pipelines that are running on Gen1 and decommission your Gen1 account.</p>
|
||||
</article>
|
||||
|
||||
|
||||
|
||||
<footer class="book-footer">
|
||||
|
||||
<div class="flex flex-wrap justify-between">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<hr />
|
||||
Azure Storage ©2021 <br />
|
||||
Visit the <a href="https://azure.microsoft.com/services/storage/">Azure Storage homepage</a> or read our <a href="https://docs.microsoft.com/azure/storage/">getting started guide</a> or the <a href="https://azure.microsoft.com/blog/topics/storage-backup-and-recovery/">Azure Storage Blog</a>. <br />
|
||||
Contact us: <a href="mailto:azurestoragefeedback@microsoft.com?subject=AzureStorage.com%20Feedback">azurestoragefeedback@microsoft.com</a>.<br />
|
||||
Generated on Mon, Feb 08 2021 23:18:57 UTC
|
||||
</footer>
|
||||
|
||||
|
||||
|
||||
<div class="book-comments">
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<label for="menu-control" class="hidden book-menu-overlay"></label>
|
||||
</div>
|
||||
|
||||
|
||||
<aside class="book-toc">
|
||||
<div class="book-toc-content">
|
||||
|
||||
|
||||
<nav id="TableOfContents">
|
||||
<ul>
|
||||
<li><a href="#overview">Overview</a></li>
|
||||
<li><a href="#prerequisites">Prerequisites</a></li>
|
||||
<li><a href="#how-to-configure-and-update-azure-databricks">How to Configure and Update Azure Databricks</a></li>
|
||||
<li><a href="#how-to-configure-and-update-azure-datafactory">How to Configure and Update Azure Datafactory</a></li>
|
||||
<li><a href="#how-to-configure-and-update-hdinsight">How to Configure and update HDInsight</a></li>
|
||||
<li><a href="#how-to-configure-and-update-azure-synapse-analytics">How to configure and update Azure Synapse Analytics</a></li>
|
||||
<li><a href="#cutover-from-gen1-to-gen2">Cutover from Gen1 to Gen2</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
</main>
|
||||
|
||||
<script src="https://code.jquery.com/jquery-3.5.1.slim.min.js" integrity="sha256-4+XzXVhsDmqanXGHaHvgh1gMQKX40OUvDEBTu8JcmNs=" crossorigin="anonymous"></script>
|
||||
|
||||
<script>
|
||||
$('a[href^="http://"], a[href^="https://"]').not('a[href*="'+location.hostname+'"]').attr('target','_blank');
|
||||
</script>
|
||||
|
||||
<script type="text/javascript">
|
||||
!function(T,l,y){var S=T.location,u="script",k="instrumentationKey",D="ingestionendpoint",C="disableExceptionTracking",E="ai.device.",I="toLowerCase",b="crossOrigin",w="POST",e="appInsightsSDK",t=y.name||"appInsights";(y.name||T[e])&&(T[e]=t);var n=T[t]||function(d){var g=!1,f=!1,m={initialize:!0,queue:[],sv:"4",version:2,config:d};function v(e,t){var n={},a="Browser";return n[E+"id"]=a[I](),n[E+"type"]=a,n["ai.operation.name"]=S&&S.pathname||"_unknown_",n["ai.internal.sdkVersion"]="javascript:snippet_"+(m.sv||m.version),{time:function(){var e=new Date;function t(e){var t=""+e;return 1===t.length&&(t="0"+t),t}return e.getUTCFullYear()+"-"+t(1+e.getUTCMonth())+"-"+t(e.getUTCDate())+"T"+t(e.getUTCHours())+":"+t(e.getUTCMinutes())+":"+t(e.getUTCSeconds())+"."+((e.getUTCMilliseconds()/1e3).toFixed(3)+"").slice(2,5)+"Z"}(),iKey:e,name:"Microsoft.ApplicationInsights."+e.replace(/-/g,"")+"."+t,sampleRate:100,tags:n,data:{baseData:{ver:2}}}}var h=d.url||y.src;if(h){function a(e){var t,n,a,i,r,o,s,c,p,l,u;g=!0,m.queue=[],f||(f=!0,t=h,s=function(){var e={},t=d.connectionString;if(t)for(var n=t.split(";"),a=0;a<n.length;a++){var i=n[a].split("=");2===i.length&&(e[i[0][I]()]=i[1])}if(!e[D]){var r=e.endpointsuffix,o=r?e.location:null;e[D]="https://"+(o?o+".":"")+"dc."+(r||"services.visualstudio.com")}return e}(),c=s[k]||d[k]||"",p=s[D],l=p?p+"/v2/track":config.endpointUrl,(u=[]).push((n="SDK LOAD Failure: Failed to load Application Insights SDK script (See stack for details)",a=t,i=l,(o=(r=v(c,"Exception")).data).baseType="ExceptionData",o.baseData.exceptions=[{typeName:"SDKLoadFailed",message:n.replace(/\./g,"-"),hasFullStack:!1,stack:n+"\nSnippet failed to load ["+a+"] -- Telemetry is disabled\nHelp Link: https://go.microsoft.com/fwlink/?linkid=2128109\nHost: "+(S&&S.pathname||"_unknown_")+"\nEndpoint: "+i,parsedStack:[]}],r)),u.push(function(e,t,n,a){var i=v(c,"Message"),r=i.data;r.baseType="MessageData";var o=r.baseData;return o.message='AI (Internal): 99 message:"'+("SDK LOAD Failure: Failed to load Application Insights SDK script (See stack for details) ("+n+")").replace(/\"/g,"")+'"',o.properties={endpoint:a},i}(0,0,t,l)),function(e,t){if(JSON){var n=T.fetch;if(n&&!y.useXhr)n(t,{method:w,body:JSON.stringify(e),mode:"cors"});else if(XMLHttpRequest){var a=new XMLHttpRequest;a.open(w,t),a.setRequestHeader("Content-type","application/json"),a.send(JSON.stringify(e))}}}(u,l))}function i(e,t){f||setTimeout(function(){!t&&m.core||a()},500)}var e=function(){var n=l.createElement(u);n.src=h;var e=y[b];return!e&&""!==e||"undefined"==n[b]||(n[b]=e),n.onload=i,n.onerror=a,n.onreadystatechange=function(e,t){"loaded"!==n.readyState&&"complete"!==n.readyState||i(0,t)},n}();y.ld<0?l.getElementsByTagName("head")[0].appendChild(e):setTimeout(function(){l.getElementsByTagName(u)[0].parentNode.appendChild(e)},y.ld||0)}try{m.cookie=l.cookie}catch(p){}function t(e){for(;e.length;)!function(t){m[t]=function(){var e=arguments;g||m.queue.push(function(){m[t].apply(m,e)})}}(e.pop())}var n="track",r="TrackPage",o="TrackEvent";t([n+"Event",n+"PageView",n+"Exception",n+"Trace",n+"DependencyData",n+"Metric",n+"PageViewPerformance","start"+r,"stop"+r,"start"+o,"stop"+o,"addTelemetryInitializer","setAuthenticatedUserContext","clearAuthenticatedUserContext","flush"]),m.SeverityLevel={Verbose:0,Information:1,Warning:2,Error:3,Critical:4};var s=(d.extensionConfig||{}).ApplicationInsightsAnalytics||{};if(!0!==d[C]&&!0!==s[C]){method="onerror",t(["_"+method]);var c=T[method];T[method]=function(e,t,n,a,i){var r=c&&c(e,t,n,a,i);return!0!==r&&m["_"+method]({message:e,url:t,lineNumber:n,columnNumber:a,error:i}),r},d.autoExceptionInstrumented=!0}return m}(y.cfg);(T[t]=n).queue&&0===n.queue.length&&n.trackPageView({})}(window,document,{
|
||||
src: "https://az416426.vo.msecnd.net/scripts/b/ai.2.min.js",
|
||||
|
||||
|
||||
|
||||
|
||||
cfg: {
|
||||
instrumentationKey: "0600b273-5440-42cc-9fe4-51924c721ce0"
|
||||
|
||||
}});
|
||||
</script>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,106 @@
|
|||
#cloud-config
|
||||
|
||||
package_update: true
|
||||
|
||||
disk_setup:
|
||||
ephemeral0:
|
||||
table_type: mbr
|
||||
layout: [66, [33, 82]]
|
||||
overwrite: True
|
||||
fs_setup:
|
||||
- device: ephemeral0.1
|
||||
filesystem: ext4
|
||||
- device: ephemeral0.2
|
||||
filesystem: swap
|
||||
mounts:
|
||||
- ["ephemeral0.1", "/mnt"]
|
||||
- ["ephemeral0.2", "none", "swap", "sw", "0", "0"]
|
||||
|
||||
apt:
|
||||
sources:
|
||||
source1:
|
||||
source: "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
|
||||
key: |
|
||||
-----BEGIN PGP PUBLIC KEY BLOCK-----
|
||||
|
||||
mQINBFit2ioBEADhWpZ8/wvZ6hUTiXOwQHXMAlaFHcPH9hAtr4F1y2+OYdbtMuth
|
||||
lqqwp028AqyY+PRfVMtSYMbjuQuu5byyKR01BbqYhuS3jtqQmljZ/bJvXqnmiVXh
|
||||
38UuLa+z077PxyxQhu5BbqntTPQMfiyqEiU+BKbq2WmANUKQf+1AmZY/IruOXbnq
|
||||
L4C1+gJ8vfmXQt99npCaxEjaNRVYfOS8QcixNzHUYnb6emjlANyEVlZzeqo7XKl7
|
||||
UrwV5inawTSzWNvtjEjj4nJL8NsLwscpLPQUhTQ+7BbQXAwAmeHCUTQIvvWXqw0N
|
||||
cmhh4HgeQscQHYgOJjjDVfoY5MucvglbIgCqfzAHW9jxmRL4qbMZj+b1XoePEtht
|
||||
ku4bIQN1X5P07fNWzlgaRL5Z4POXDDZTlIQ/El58j9kp4bnWRCJW0lya+f8ocodo
|
||||
vZZ+Doi+fy4D5ZGrL4XEcIQP/Lv5uFyf+kQtl/94VFYVJOleAv8W92KdgDkhTcTD
|
||||
G7c0tIkVEKNUq48b3aQ64NOZQW7fVjfoKwEZdOqPE72Pa45jrZzvUFxSpdiNk2tZ
|
||||
XYukHjlxxEgBdC/J3cMMNRE1F4NCA3ApfV1Y7/hTeOnmDuDYwr9/obA8t016Yljj
|
||||
q5rdkywPf4JF8mXUW5eCN1vAFHxeg9ZWemhBtQmGxXnw9M+z6hWwc6ahmwARAQAB
|
||||
tCtEb2NrZXIgUmVsZWFzZSAoQ0UgZGViKSA8ZG9ja2VyQGRvY2tlci5jb20+iQI3
|
||||
BBMBCgAhBQJYrefAAhsvBQsJCAcDBRUKCQgLBRYCAwEAAh4BAheAAAoJEI2BgDwO
|
||||
v82IsskP/iQZo68flDQmNvn8X5XTd6RRaUH33kXYXquT6NkHJciS7E2gTJmqvMqd
|
||||
tI4mNYHCSEYxI5qrcYV5YqX9P6+Ko+vozo4nseUQLPH/ATQ4qL0Zok+1jkag3Lgk
|
||||
jonyUf9bwtWxFp05HC3GMHPhhcUSexCxQLQvnFWXD2sWLKivHp2fT8QbRGeZ+d3m
|
||||
6fqcd5Fu7pxsqm0EUDK5NL+nPIgYhN+auTrhgzhK1CShfGccM/wfRlei9Utz6p9P
|
||||
XRKIlWnXtT4qNGZNTN0tR+NLG/6Bqd8OYBaFAUcue/w1VW6JQ2VGYZHnZu9S8LMc
|
||||
FYBa5Ig9PxwGQOgq6RDKDbV+PqTQT5EFMeR1mrjckk4DQJjbxeMZbiNMG5kGECA8
|
||||
g383P3elhn03WGbEEa4MNc3Z4+7c236QI3xWJfNPdUbXRaAwhy/6rTSFbzwKB0Jm
|
||||
ebwzQfwjQY6f55MiI/RqDCyuPj3r3jyVRkK86pQKBAJwFHyqj9KaKXMZjfVnowLh
|
||||
9svIGfNbGHpucATqREvUHuQbNnqkCx8VVhtYkhDb9fEP2xBu5VvHbR+3nfVhMut5
|
||||
G34Ct5RS7Jt6LIfFdtcn8CaSas/l1HbiGeRgc70X/9aYx/V/CEJv0lIe8gP6uDoW
|
||||
FPIZ7d6vH+Vro6xuWEGiuMaiznap2KhZmpkgfupyFmplh0s6knymuQINBFit2ioB
|
||||
EADneL9S9m4vhU3blaRjVUUyJ7b/qTjcSylvCH5XUE6R2k+ckEZjfAMZPLpO+/tF
|
||||
M2JIJMD4SifKuS3xck9KtZGCufGmcwiLQRzeHF7vJUKrLD5RTkNi23ydvWZgPjtx
|
||||
Q+DTT1Zcn7BrQFY6FgnRoUVIxwtdw1bMY/89rsFgS5wwuMESd3Q2RYgb7EOFOpnu
|
||||
w6da7WakWf4IhnF5nsNYGDVaIHzpiqCl+uTbf1epCjrOlIzkZ3Z3Yk5CM/TiFzPk
|
||||
z2lLz89cpD8U+NtCsfagWWfjd2U3jDapgH+7nQnCEWpROtzaKHG6lA3pXdix5zG8
|
||||
eRc6/0IbUSWvfjKxLLPfNeCS2pCL3IeEI5nothEEYdQH6szpLog79xB9dVnJyKJb
|
||||
VfxXnseoYqVrRz2VVbUI5Blwm6B40E3eGVfUQWiux54DspyVMMk41Mx7QJ3iynIa
|
||||
1N4ZAqVMAEruyXTRTxc9XW0tYhDMA/1GYvz0EmFpm8LzTHA6sFVtPm/ZlNCX6P1X
|
||||
zJwrv7DSQKD6GGlBQUX+OeEJ8tTkkf8QTJSPUdh8P8YxDFS5EOGAvhhpMBYD42kQ
|
||||
pqXjEC+XcycTvGI7impgv9PDY1RCC1zkBjKPa120rNhv/hkVk/YhuGoajoHyy4h7
|
||||
ZQopdcMtpN2dgmhEegny9JCSwxfQmQ0zK0g7m6SHiKMwjwARAQABiQQ+BBgBCAAJ
|
||||
BQJYrdoqAhsCAikJEI2BgDwOv82IwV0gBBkBCAAGBQJYrdoqAAoJEH6gqcPyc/zY
|
||||
1WAP/2wJ+R0gE6qsce3rjaIz58PJmc8goKrir5hnElWhPgbq7cYIsW5qiFyLhkdp
|
||||
YcMmhD9mRiPpQn6Ya2w3e3B8zfIVKipbMBnke/ytZ9M7qHmDCcjoiSmwEXN3wKYI
|
||||
mD9VHONsl/CG1rU9Isw1jtB5g1YxuBA7M/m36XN6x2u+NtNMDB9P56yc4gfsZVES
|
||||
KA9v+yY2/l45L8d/WUkUi0YXomn6hyBGI7JrBLq0CX37GEYP6O9rrKipfz73XfO7
|
||||
JIGzOKZlljb/D9RX/g7nRbCn+3EtH7xnk+TK/50euEKw8SMUg147sJTcpQmv6UzZ
|
||||
cM4JgL0HbHVCojV4C/plELwMddALOFeYQzTif6sMRPf+3DSj8frbInjChC3yOLy0
|
||||
6br92KFom17EIj2CAcoeq7UPhi2oouYBwPxh5ytdehJkoo+sN7RIWua6P2WSmon5
|
||||
U888cSylXC0+ADFdgLX9K2zrDVYUG1vo8CX0vzxFBaHwN6Px26fhIT1/hYUHQR1z
|
||||
VfNDcyQmXqkOnZvvoMfz/Q0s9BhFJ/zU6AgQbIZE/hm1spsfgvtsD1frZfygXJ9f
|
||||
irP+MSAI80xHSf91qSRZOj4Pl3ZJNbq4yYxv0b1pkMqeGdjdCYhLU+LZ4wbQmpCk
|
||||
SVe2prlLureigXtmZfkqevRz7FrIZiu9ky8wnCAPwC7/zmS18rgP/17bOtL4/iIz
|
||||
QhxAAoAMWVrGyJivSkjhSGx1uCojsWfsTAm11P7jsruIL61ZzMUVE2aM3Pmj5G+W
|
||||
9AcZ58Em+1WsVnAXdUR//bMmhyr8wL/G1YO1V3JEJTRdxsSxdYa4deGBBY/Adpsw
|
||||
24jxhOJR+lsJpqIUeb999+R8euDhRHG9eFO7DRu6weatUJ6suupoDTRWtr/4yGqe
|
||||
dKxV3qQhNLSnaAzqW/1nA3iUB4k7kCaKZxhdhDbClf9P37qaRW467BLCVO/coL3y
|
||||
Vm50dwdrNtKpMBh3ZpbB1uJvgi9mXtyBOMJ3v8RZeDzFiG8HdCtg9RvIt/AIFoHR
|
||||
H3S+U79NT6i0KPzLImDfs8T7RlpyuMc4Ufs8ggyg9v3Ae6cN3eQyxcK3w0cbBwsh
|
||||
/nQNfsA6uu+9H7NhbehBMhYnpNZyrHzCmzyXkauwRAqoCbGCNykTRwsur9gS41TQ
|
||||
M8ssD1jFheOJf3hODnkKU+HKjvMROl1DK7zdmLdNzA1cvtZH/nCC9KPj1z8QC47S
|
||||
xx+dTZSx4ONAhwbS/LN3PoKtn8LPjY9NP9uDWI+TWYquS2U+KHDrBDlsgozDbs/O
|
||||
jCxcpDzNmXpWQHEtHU7649OXHP7UeNST1mCUCH5qdank0V1iejF6/CfTFU4MfcrG
|
||||
YT90qFF93M3v01BbxP+EIY2/9tiIPbrd
|
||||
=0YYh
|
||||
-----END PGP PUBLIC KEY BLOCK-----
|
||||
|
||||
packages:
|
||||
- apt-transport-https
|
||||
- ca-certificates
|
||||
- curl
|
||||
- gnupg-agent
|
||||
- software-properties-common
|
||||
- docker-ce
|
||||
- docker-ce-cli
|
||||
- containerd.io
|
||||
|
||||
runcmd:
|
||||
- [ cloud-init-per, once, fetch_docker_compose, curl, -L,
|
||||
"https://github.com/docker/compose/releases/download/1.25.3/docker-compose-Linux-x86_64",
|
||||
-o, /usr/local/bin/docker-compose ]
|
||||
- [ cloud-init-per, once, docker_compose_perms,
|
||||
chmod, +x, /usr/local/bin/docker-compose ]
|
||||
|
||||
system_info:
|
||||
default_user:
|
||||
groups: [docker]
|
||||
|
|
@ -0,0 +1,588 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en" dir=>
|
||||
|
||||
<head>
|
||||
<meta name="generator" content="Hugo 0.80.0" />
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<meta name="description" content="Bi-directional sync pattern Guide: A quick start template # Overview # This manual will introduce WANdisco as a recommended tool to set up bi-directional sync between ADLS Gen1 and Gen2 using the Replication feature.
|
||||
Below will be covered as part of this guide:
|
||||
Data Migration from Gen1 to Gen2 Data Consistency Check Application update for ADF, ADB and SQL DWH workloads Considerations for using the bi-directional sync pattern:">
|
||||
<meta name="theme-color" content="#FFFFFF"><meta property="og:title" content="Bi-directional sync pattern Guide: A quick start template" />
|
||||
<meta property="og:description" content="" />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://azurestorage.com/docs/analytics/adls-gen1-to-gen2-migration/bi-directional/" />
|
||||
|
||||
<title>Bi-directional sync pattern Guide: A quick start template | Azure Storage</title>
|
||||
<link rel="manifest" href="/manifest.json">
|
||||
<link rel="icon" href="/favicon.png" type="image/x-icon">
|
||||
<link rel="stylesheet" href="/book.min.6c7c6446dfdee7c8c933e9bbc6e80ee3ed6c913b2a59519f2092c3c6a9d63e55.css" integrity="sha256-bHxkRt/e58jJM+m7xugO4+1skTsqWVGfIJLDxqnWPlU=">
|
||||
<script defer src="/en.search.min.a092390ed9ca69153376eaacab4624d67749500866440e4365296f3fa207eb56.js" integrity="sha256-oJI5DtnKaRUzduqsq0Yk1ndJUAhmRA5DZSlvP6IH61Y="></script>
|
||||
<link rel="alternate" type="application/rss+xml" href="https://azurestorage.com/docs/analytics/adls-gen1-to-gen2-migration/bi-directional/index.xml" title="Azure Storage" />
|
||||
<!--
|
||||
Made with Book Theme
|
||||
https://github.com/alex-shpak/hugo-book
|
||||
-->
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<body dir=>
|
||||
<input type="checkbox" class="hidden toggle" id="menu-control" />
|
||||
<input type="checkbox" class="hidden toggle" id="toc-control" />
|
||||
<main class="container flex">
|
||||
<aside class="book-menu">
|
||||
<div class="book-menu-content">
|
||||
|
||||
<nav>
|
||||
<h2 class="book-brand">
|
||||
<a href="/"><img src="/images/azure-icon.png" alt="Logo" /><span>Azure Storage</span>
|
||||
</a>
|
||||
</h2>
|
||||
|
||||
|
||||
<div class="book-search">
|
||||
<input type="text" id="book-search-input" placeholder="Search" aria-label="Search" maxlength="64" data-hotkeys="s/" />
|
||||
<div class="book-search-spinner hidden"></div>
|
||||
<ul id="book-search-results"></ul>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<ul>
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/analytics/" class="">Analytics</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/application-and-user-data/" class="">Application and User Data</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/backup-and-archive/" class="">Backup and Archive</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/hpc-iot-and-ai/" class="">HPC IoT and AI</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/storage-partners/" class="">Storage Partners</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/tools-and-utilities/" class="">Tools and Utilities</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
|
||||
<script>(function(){var menu=document.querySelector("aside.book-menu nav");addEventListener("beforeunload",function(event){localStorage.setItem("menu.scrollTop",menu.scrollTop);});menu.scrollTop=localStorage.getItem("menu.scrollTop");})();</script>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
<div class="book-page">
|
||||
<header class="book-header">
|
||||
|
||||
<div class="flex align-center justify-between">
|
||||
<label for="menu-control">
|
||||
<img src="/svg/menu.svg" class="book-icon" alt="Menu" />
|
||||
</label>
|
||||
|
||||
<strong>Bi-directional sync pattern Guide: A quick start template</strong>
|
||||
|
||||
<label for="toc-control">
|
||||
|
||||
<img src="/svg/toc.svg" class="book-icon" alt="Table of Contents" />
|
||||
|
||||
</label>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<aside class="hidden clearfix">
|
||||
|
||||
|
||||
<nav id="TableOfContents">
|
||||
<ul>
|
||||
<li><a href="#overview">Overview</a></li>
|
||||
<li><a href="#prerequisites">Prerequisites</a></li>
|
||||
<li><a href="#login-to-fusion-ui">Login to Fusion UI</a></li>
|
||||
<li><a href="#create-replication-rule">Create Replication Rule</a></li>
|
||||
<li><a href="#consistency-check">Consistency Check</a></li>
|
||||
<li><a href="#migration-using-livemigrator">Migration using LiveMigrator</a></li>
|
||||
<li><a href="#managing-replication">Managing Replication</a></li>
|
||||
<li><a href="#application-update">Application Update</a>
|
||||
<ul>
|
||||
<li><a href="#mount-path-configuration">Mount path configuration</a></li>
|
||||
<li><a href="#beginning-state">Beginning State</a></li>
|
||||
<li><a href="#interim-state">Interim State</a></li>
|
||||
<li><a href="#eventual-end-state">Eventual End State</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#references">References</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
</aside>
|
||||
|
||||
|
||||
</header>
|
||||
|
||||
|
||||
|
||||
<article class="markdown"><h1 id="bi-directional-sync-pattern-guide-a-quick-start-template">
|
||||
Bi-directional sync pattern Guide: A quick start template
|
||||
<a class="anchor" href="#bi-directional-sync-pattern-guide-a-quick-start-template">#</a>
|
||||
</h1>
|
||||
<h2 id="overview">
|
||||
Overview
|
||||
<a class="anchor" href="#overview">#</a>
|
||||
</h2>
|
||||
<p>This manual will introduce <a href="https://docs.wandisco.com/bigdata/wdfusion/adls/">WANdisco</a> as a recommended tool to set up bi-directional sync between ADLS Gen1 and Gen2 using the Replication feature.</p>
|
||||
<p>Below will be covered as part of this guide:</p>
|
||||
<ul>
|
||||
<li>Data Migration from Gen1 to Gen2</li>
|
||||
<li>Data Consistency Check</li>
|
||||
<li>Application update for ADF, ADB and SQL DWH workloads</li>
|
||||
</ul>
|
||||
<p>Considerations for using the bi-directional sync pattern:</p>
|
||||
<ul>
|
||||
<li>Ideal for complex scenarios that involve a large number of pipelines and dependencies where a phased approach might make more sense.</li>
|
||||
<li>Migration effort is high, but it provides side-by-side support for Gen1 and Gen2.</li>
|
||||
</ul>
|
||||
<h2 id="prerequisites">
|
||||
Prerequisites
|
||||
<a class="anchor" href="#prerequisites">#</a>
|
||||
</h2>
|
||||
<ul>
|
||||
<li>
|
||||
<p><strong>Active Azure Subscription</strong></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Azure Data Lake Storage Gen1</strong></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Azure Data Lake Storage Gen2</strong> For more details please refer to :link: <a href="https://docs.microsoft.com/azure/storage/common/storage-account-create">create azure storage account</a></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Licenses for WANdisco Fusion</strong> that accommodate the volume of data that you want to make available to ADLS Gen2</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Azure Linux Virtual Machine</strong> Please refer here to know <a href="./wandisco-set-up-and-installation">How to create Azure VM</a></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Windows SSH client</strong> like <a href="https://www.putty.org/">Putty</a>, <a href="https://gitforwindows.org/">Git for Windows</a>, <a href="https://cygwin.com/">Cygwin</a>, <a href="https://mobaxterm.mobatek.net/">MobaXterm</a></p>
|
||||
</li>
|
||||
</ul>
|
||||
<h2 id="login-to-fusion-ui">
|
||||
Login to Fusion UI
|
||||
<a class="anchor" href="#login-to-fusion-ui">#</a>
|
||||
</h2>
|
||||
<ol>
|
||||
<li>
|
||||
<p><strong>Start</strong> the VM in azure portal if not in Running status.</p>
|
||||
<p><img src="../images/80544309-9b64d400-8965-11ea-9b28-a4e4daf05a3d.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Start the Fusion</strong></p>
|
||||
<p>Go to <strong>SSH Client</strong> <a href="./wandisco-set-up-and-installation/#connect-to-vm">Connect</a> and run below commands:</p>
|
||||
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-scala" data-lang="scala">cd fusion<span style="color:#f92672">-</span>docker<span style="color:#f92672">-</span>compose <span style="color:#75715e">// Change to the repository directory
|
||||
</span><span style="color:#75715e"></span>
|
||||
<span style="color:#f92672">./</span>setup<span style="color:#f92672">-</span>env<span style="color:#f92672">.</span>sh <span style="color:#75715e">// set up script
|
||||
</span><span style="color:#75715e"></span>
|
||||
docker<span style="color:#f92672">-</span>compose up <span style="color:#f92672">-</span>d <span style="color:#75715e">// start the fusion
|
||||
</span></code></pre></div></li>
|
||||
<li>
|
||||
<p><strong>Login to Fusion UI</strong>. Open the web browser and give the path as below</p>
|
||||
<p>URL –> http://{dnsname}:8081</p>
|
||||
<blockquote>
|
||||
<p>Note: The DNS name can be taken from VM Overview details.</p>
|
||||
</blockquote>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Set up ADLS Gen1 and Gen2 storage</strong>. <a href="./wandisco-set-up-and-installation/#adls-gen1-and-gen2-configuration">Click here</a> to know more.</p>
|
||||
</li>
|
||||
</ol>
|
||||
<h2 id="create-replication-rule">
|
||||
Create Replication Rule
|
||||
<a class="anchor" href="#create-replication-rule">#</a>
|
||||
</h2>
|
||||
<p>File system content is replicated selectively by defining Replication Rules.These specify the directory in the file system that will be replicated and the Zones that will participate in that replication.</p>
|
||||
<p>Without any Replication Rules defined, each Zone’s file system operates independently of the others. With the combination of Zones and Replication Rules, WANdisco Fusion gives you complete control over how data is replicated between your file systems and/or object stores.</p>
|
||||
<p>On the dashboard, create a HCFS rule with the following parameters:</p>
|
||||
<ul>
|
||||
<li>
|
||||
<p>Rule Name = migration (Give any unique name)</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Path for all storages = /</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Default exclusions</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Preserve HCFS Block Size = False</p>
|
||||
<p><img src="../images/80546359-44153280-896a-11ea-9e12-bb85b6ceeafc.png" alt="image" /></p>
|
||||
<p>To know more click :link: <a href="https://wandisco.github.io/wandisco-documentation/docs/quickstarts/operation/create-rule">how to create rule</a></p>
|
||||
</li>
|
||||
</ul>
|
||||
<p><strong>Click Finish</strong> and wait for the rule to appear on the dashboard.</p>
|
||||
<h2 id="consistency-check">
|
||||
Consistency Check
|
||||
<a class="anchor" href="#consistency-check">#</a>
|
||||
</h2>
|
||||
<p>Once you have created a <a href="https://wandisco.github.io/wandisco-documentation/docs/quickstarts/operation/create-rule">replication rule</a> as per above mentioned steps, run a consistency check to compare the contents between both zones.</p>
|
||||
<p>On the Rules table, click to View rule.</p>
|
||||
<ol>
|
||||
<li>
|
||||
<p>On the rule page, start consistency check and wait for the Consistency status to update.</p>
|
||||
<p><img src="../images/80934228-0e15eb00-8d7c-11ea-8a8c-1975e2c0c9ba.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>The Consistency Status will determine the next steps:</p>
|
||||
<ul>
|
||||
<li>
|
||||
<p>Consistent - no action needed</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Inconsistent - migration required</p>
|
||||
</li>
|
||||
</ul>
|
||||
<p><strong>Consistency check before migration</strong>:</p>
|
||||
<p><img src="../images/80765875-f418a600-8af8-11ea-9129-0791ccfcba12.png" alt="image" /></p>
|
||||
<p>To know more refer to <a href="https://docs.wandisco.com/bigdata/wdfusion/2.12/#consistency-check">Consistency Check using WANdisco fusion</a></p>
|
||||
<blockquote>
|
||||
<p>Note: <strong>START CONSISTENCY CHECK</strong> is recommended for small set of data volume.</p>
|
||||
</blockquote>
|
||||
</li>
|
||||
</ol>
|
||||
<h2 id="migration-using-livemigrator">
|
||||
Migration using LiveMigrator
|
||||
<a class="anchor" href="#migration-using-livemigrator">#</a>
|
||||
</h2>
|
||||
<p>Once HCFS replication rule is created, migration activity can be started using the LiveMigrator. This allows migration of data in a single pass while keeping up with all changes to the source storage(ADLS Gen1). As an outcome data consistency is gauranteed between source and target.</p>
|
||||
<blockquote>
|
||||
<p>Note: The Gen2 is synchronized with Gen1 source using consistency checks and scheduled migrations.</p>
|
||||
</blockquote>
|
||||
<ol>
|
||||
<li>
|
||||
<p><strong>Get Sample data</strong></p>
|
||||
<p>Upload sample data to the ADLS Gen1 storage account, see the <a href="https://docs.microsoft.com/azure/data-lake-store/data-lake-store-get-started-portal#uploaddata">guide</a> to know more.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Place it within the home mount point.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>On the Fusion UI dashboard, view the HCFS rule.</p>
|
||||
<p><img src="../images/80547216-8c355480-896c-11ea-8adb-1a58d4e1be6c.png" alt="image" /></p>
|
||||
<p>The overwrite settings needs to be configured. This determines what happens if the LiveMigrator encounters content in the target path with the same name and size.</p>
|
||||
<ul>
|
||||
<li>
|
||||
<p><strong>Skip</strong>: If the filesize is identical between the source and target, the file is skipped. If it’s a different size, the whole file is replaced.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Overwrite</strong>: Everything is replaced, even if the file size is identical.</p>
|
||||
</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>
|
||||
<p>Start your migration with the following settings:</p>
|
||||
<p>Source Zone = adls1</p>
|
||||
<p>Target Zone = adls2</p>
|
||||
<p>Overwrite Settings = Skip / Overwrite</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Wait until the migration is complete, and check the contents in the ADLS Gen2 container.</p>
|
||||
<p><strong>Consistency check after migration</strong>:</p>
|
||||
<p><img src="../images/81225979-f8ccd680-8f9e-11ea-9d93-42f8d4ad2c63.png" alt="image" /></p>
|
||||
<blockquote>
|
||||
<p>NOTE: A hidden folder :file_folder: .fusion will be present in the ADLS Gen2 path.</p>
|
||||
</blockquote>
|
||||
<blockquote>
|
||||
<p>Limitation: Client based replication is not supported by Fusion UI , so replication process here is manually driven.</p>
|
||||
</blockquote>
|
||||
</li>
|
||||
</ol>
|
||||
<h2 id="managing-replication">
|
||||
Managing Replication
|
||||
<a class="anchor" href="#managing-replication">#</a>
|
||||
</h2>
|
||||
<p><img src="../images/80843564-7799cc00-8bb9-11ea-89bb-b7009162f6a1.png" alt="image" /></p>
|
||||
<p>To know more visit <a href="https://docs.wandisco.com/bigdata/wdfusion/2.12/#managing-replication">How to manage replication</a></p>
|
||||
<h2 id="application-update">
|
||||
Application Update
|
||||
<a class="anchor" href="#application-update">#</a>
|
||||
</h2>
|
||||
<p>As part of this, we will <a href="https://docs.microsoft.com/azure/storage/blobs/data-lake-storage-supported-azure-services">configure services in workloads</a> used and update the applications to point to Gen2 mount after the migration is complete.</p>
|
||||
<p>We will be covering below azure services</p>
|
||||
<ul>
|
||||
<li>Azure Data Factory
|
||||
<ul>
|
||||
<li><a href="https://docs.microsoft.com/azure/data-factory/load-azure-data-lake-storage-gen2">Load data into Azure Data Lake Storage Gen2 with Azure Data Factory</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>Azure Databricks
|
||||
<ul>
|
||||
<li><a href="https://docs.microsoft.com/azure/databricks/data/data-sources/azure/azure-datalake-gen2">Use with Azure Databricks</a></li>
|
||||
<li><a href="https://docs.microsoft.com/azure/storage/blobs/data-lake-storage-quickstart-create-databricks-account">Quickstart: Analyze data in Azure Data Lake Storage Gen2 by using Azure Databricks</a></li>
|
||||
<li><a href="https://docs.microsoft.com/azure/azure-databricks/databricks-extract-load-sql-data-warehouse">Tutorial: Extract, transform, and load data by using Azure Databricks</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>SQL Data Warehouse
|
||||
<ul>
|
||||
<li><a href="https://docs.microsoft.com/azure/data-factory/load-azure-sql-data-warehouse">Use with Azure SQL Data Warehouse</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<p>This can be achieved by following a phased approach where in the migration of data, work loads and applications will be validated incrementally.</p>
|
||||
<h3 id="mount-path-configuration">
|
||||
Mount path configuration
|
||||
<a class="anchor" href="#mount-path-configuration">#</a>
|
||||
</h3>
|
||||
<p>This will show how to set and configure the mount paths for Gen1 and Gen2 in the MountConfiguration script.</p>
|
||||
<p><strong>Gen1 mount path configuration</strong>:</p>
|
||||
<p><img src="../images/81219887-4e9c8100-8f95-11ea-8b81-e32122547757.png" alt="image" /></p>
|
||||
<p><strong>Gen2 mount path configuration</strong>:</p>
|
||||
<p><img src="../images/81220219-cff41380-8f95-11ea-9295-4df62c7a9afe.png" alt="image" /></p>
|
||||
<h3 id="beginning-state">
|
||||
Beginning State
|
||||
<a class="anchor" href="#beginning-state">#</a>
|
||||
</h3>
|
||||
<p><strong>Before Migration</strong>– The data pipeline is on Gen1</p>
|
||||
<p>In this state the data ingestion from ADB to Raw folder, writing the processed data into the Processed folder and loading the processed data to SQL DW will be happening at Gen1.</p>
|
||||
<p><img src="../images/81105450-c650aa00-8ec8-11ea-917c-619c6909b2c9.png" alt="image" /></p>
|
||||
<p>Sample design:</p>
|
||||
<p><img src="../images/81109171-8a204800-8ece-11ea-9ddc-a8c12007e32a.png" alt="image" /></p>
|
||||
<p>All the ADB notebooks will be pointing to Gen1 Mount path in this state and the data will be ingested, processed and loaded to SQL DW from Gen1.</p>
|
||||
<h3 id="interim-state">
|
||||
Interim State
|
||||
<a class="anchor" href="#interim-state">#</a>
|
||||
</h3>
|
||||
<p>In this state we will start with the migration of the existing Gen1 data to Gen2 using <strong>Wandisco fusion</strong>. The data pipeline will be set to Gen1 and Gen2 partially which will include the data ingestion and processing happening at Gen1 meanwhile writing the processed data to SQL DW at Gen2.</p>
|
||||
<p><img src="../images/81351734-dc519c80-9079-11ea-8a8e-669b65e17b49.png" alt="image" /></p>
|
||||
<p>Follow the steps for the <a href="#migration-using-livemigrator">migration</a> of Gen1 data to Gen2 for the Raw and Processed data.</p>
|
||||
<p>Once the data is migrated, run <strong>Consistency check</strong> in Fusion UI. There should be no inconsistencies.</p>
|
||||
<p><strong>How to change the mount path</strong></p>
|
||||
<p>This will show how to configure the mount path for the work load Azure Databricks which will load processed data to SQL DW at Gen2.
|
||||
In the master pipeline in Azure Datafactory, Go to the notebook <strong>settings</strong> –> <strong>Base parameters</strong> and mount the <strong>RootPathParam</strong> to Gen2.</p>
|
||||
<p><img src="../images/81126936-b4cfc800-8ef1-11ea-86bc-8550b21a8aa3.png" alt="image" /></p>
|
||||
<p><strong>Note</strong>: At the end of this state we will acheive to establish data pipeline partially at Gen1 and Gen2.</p>
|
||||
<h3 id="eventual-end-state">
|
||||
Eventual End State
|
||||
<a class="anchor" href="#eventual-end-state">#</a>
|
||||
</h3>
|
||||
<p><strong>After Migration</strong> – The data pipeline moved to Gen2</p>
|
||||
<p>This state depicts when all the work loads and applications are moved from Gen1 to Gen2 and the bi directional replication is ready to be turned off.</p>
|
||||
<p><img src="../images/81124187-ad0c2580-8ee9-11ea-9ca5-f1e2327dc2ab.png" alt="image" /></p>
|
||||
<p>Below are the steps to change the mount path from Gen1 to Gen2 for the Azure Databricks notebook in Azure Datafactory pipeline</p>
|
||||
<p><img src="../images/81138345-929c7100-8f16-11ea-9e0f-3681d7c76cf1.png" alt="image" /></p>
|
||||
<p>Run the master pipeline consisting of all the ADB notebooks and check the data. The ingestion and processing of raw data should take place at Gen2 now along with writing to SQL DW.
|
||||
This can be verified by checking the data at Gen2 storage. And when <strong>Consistency check</strong> is run using WAndisco Fusion UI , there will be files missing at Gen1.</p>
|
||||
<p><img src="../images/81352652-ed9ba880-907b-11ea-9cf2-c1620b28455a.png" alt="image" /></p>
|
||||
<blockquote>
|
||||
<p>Note: This marks the state where all the workloads are moved to Gen2. Gen1 will not revieve any new data in any form.</p>
|
||||
</blockquote>
|
||||
<p><strong>Cutover from Gen1 to Gen2</strong>
|
||||
After all the applications and workloads are stable on Gen2, Turn off any remaining pipelines that are running on Gen1 and decommission your Gen1 account. This will include deletion of the rules created for the migration and replication process. Shutting down the Azure VM and deleting the resource group.</p>
|
||||
<h2 id="references">
|
||||
References
|
||||
<a class="anchor" href="#references">#</a>
|
||||
</h2>
|
||||
<ul>
|
||||
<li><a href="https://wandisco.github.io/wandisco-documentation/docs/quickstarts/preparation/azure_vm_creation">WANdisco fusion Installation and set up guide</a></li>
|
||||
<li><a href="https://www.wandisco.com/products/live-migrator">WANdisco LiveMigrator</a></li>
|
||||
</ul>
|
||||
</article>
|
||||
|
||||
|
||||
|
||||
<footer class="book-footer">
|
||||
|
||||
<div class="flex flex-wrap justify-between">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<hr />
|
||||
Azure Storage ©2021 <br />
|
||||
Visit the <a href="https://azure.microsoft.com/services/storage/">Azure Storage homepage</a> or read our <a href="https://docs.microsoft.com/azure/storage/">getting started guide</a> or the <a href="https://azure.microsoft.com/blog/topics/storage-backup-and-recovery/">Azure Storage Blog</a>. <br />
|
||||
Contact us: <a href="mailto:azurestoragefeedback@microsoft.com?subject=AzureStorage.com%20Feedback">azurestoragefeedback@microsoft.com</a>.<br />
|
||||
Generated on Mon, Feb 08 2021 23:18:57 UTC
|
||||
</footer>
|
||||
|
||||
|
||||
|
||||
<div class="book-comments">
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<label for="menu-control" class="hidden book-menu-overlay"></label>
|
||||
</div>
|
||||
|
||||
|
||||
<aside class="book-toc">
|
||||
<div class="book-toc-content">
|
||||
|
||||
|
||||
<nav id="TableOfContents">
|
||||
<ul>
|
||||
<li><a href="#overview">Overview</a></li>
|
||||
<li><a href="#prerequisites">Prerequisites</a></li>
|
||||
<li><a href="#login-to-fusion-ui">Login to Fusion UI</a></li>
|
||||
<li><a href="#create-replication-rule">Create Replication Rule</a></li>
|
||||
<li><a href="#consistency-check">Consistency Check</a></li>
|
||||
<li><a href="#migration-using-livemigrator">Migration using LiveMigrator</a></li>
|
||||
<li><a href="#managing-replication">Managing Replication</a></li>
|
||||
<li><a href="#application-update">Application Update</a>
|
||||
<ul>
|
||||
<li><a href="#mount-path-configuration">Mount path configuration</a></li>
|
||||
<li><a href="#beginning-state">Beginning State</a></li>
|
||||
<li><a href="#interim-state">Interim State</a></li>
|
||||
<li><a href="#eventual-end-state">Eventual End State</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#references">References</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
</main>
|
||||
|
||||
<script src="https://code.jquery.com/jquery-3.5.1.slim.min.js" integrity="sha256-4+XzXVhsDmqanXGHaHvgh1gMQKX40OUvDEBTu8JcmNs=" crossorigin="anonymous"></script>
|
||||
|
||||
<script>
|
||||
$('a[href^="http://"], a[href^="https://"]').not('a[href*="'+location.hostname+'"]').attr('target','_blank');
|
||||
</script>
|
||||
|
||||
<script type="text/javascript">
|
||||
!function(T,l,y){var S=T.location,u="script",k="instrumentationKey",D="ingestionendpoint",C="disableExceptionTracking",E="ai.device.",I="toLowerCase",b="crossOrigin",w="POST",e="appInsightsSDK",t=y.name||"appInsights";(y.name||T[e])&&(T[e]=t);var n=T[t]||function(d){var g=!1,f=!1,m={initialize:!0,queue:[],sv:"4",version:2,config:d};function v(e,t){var n={},a="Browser";return n[E+"id"]=a[I](),n[E+"type"]=a,n["ai.operation.name"]=S&&S.pathname||"_unknown_",n["ai.internal.sdkVersion"]="javascript:snippet_"+(m.sv||m.version),{time:function(){var e=new Date;function t(e){var t=""+e;return 1===t.length&&(t="0"+t),t}return e.getUTCFullYear()+"-"+t(1+e.getUTCMonth())+"-"+t(e.getUTCDate())+"T"+t(e.getUTCHours())+":"+t(e.getUTCMinutes())+":"+t(e.getUTCSeconds())+"."+((e.getUTCMilliseconds()/1e3).toFixed(3)+"").slice(2,5)+"Z"}(),iKey:e,name:"Microsoft.ApplicationInsights."+e.replace(/-/g,"")+"."+t,sampleRate:100,tags:n,data:{baseData:{ver:2}}}}var h=d.url||y.src;if(h){function a(e){var t,n,a,i,r,o,s,c,p,l,u;g=!0,m.queue=[],f||(f=!0,t=h,s=function(){var e={},t=d.connectionString;if(t)for(var n=t.split(";"),a=0;a<n.length;a++){var i=n[a].split("=");2===i.length&&(e[i[0][I]()]=i[1])}if(!e[D]){var r=e.endpointsuffix,o=r?e.location:null;e[D]="https://"+(o?o+".":"")+"dc."+(r||"services.visualstudio.com")}return e}(),c=s[k]||d[k]||"",p=s[D],l=p?p+"/v2/track":config.endpointUrl,(u=[]).push((n="SDK LOAD Failure: Failed to load Application Insights SDK script (See stack for details)",a=t,i=l,(o=(r=v(c,"Exception")).data).baseType="ExceptionData",o.baseData.exceptions=[{typeName:"SDKLoadFailed",message:n.replace(/\./g,"-"),hasFullStack:!1,stack:n+"\nSnippet failed to load ["+a+"] -- Telemetry is disabled\nHelp Link: https://go.microsoft.com/fwlink/?linkid=2128109\nHost: "+(S&&S.pathname||"_unknown_")+"\nEndpoint: "+i,parsedStack:[]}],r)),u.push(function(e,t,n,a){var i=v(c,"Message"),r=i.data;r.baseType="MessageData";var o=r.baseData;return o.message='AI (Internal): 99 message:"'+("SDK LOAD Failure: Failed to load Application Insights SDK script (See stack for details) ("+n+")").replace(/\"/g,"")+'"',o.properties={endpoint:a},i}(0,0,t,l)),function(e,t){if(JSON){var n=T.fetch;if(n&&!y.useXhr)n(t,{method:w,body:JSON.stringify(e),mode:"cors"});else if(XMLHttpRequest){var a=new XMLHttpRequest;a.open(w,t),a.setRequestHeader("Content-type","application/json"),a.send(JSON.stringify(e))}}}(u,l))}function i(e,t){f||setTimeout(function(){!t&&m.core||a()},500)}var e=function(){var n=l.createElement(u);n.src=h;var e=y[b];return!e&&""!==e||"undefined"==n[b]||(n[b]=e),n.onload=i,n.onerror=a,n.onreadystatechange=function(e,t){"loaded"!==n.readyState&&"complete"!==n.readyState||i(0,t)},n}();y.ld<0?l.getElementsByTagName("head")[0].appendChild(e):setTimeout(function(){l.getElementsByTagName(u)[0].parentNode.appendChild(e)},y.ld||0)}try{m.cookie=l.cookie}catch(p){}function t(e){for(;e.length;)!function(t){m[t]=function(){var e=arguments;g||m.queue.push(function(){m[t].apply(m,e)})}}(e.pop())}var n="track",r="TrackPage",o="TrackEvent";t([n+"Event",n+"PageView",n+"Exception",n+"Trace",n+"DependencyData",n+"Metric",n+"PageViewPerformance","start"+r,"stop"+r,"start"+o,"stop"+o,"addTelemetryInitializer","setAuthenticatedUserContext","clearAuthenticatedUserContext","flush"]),m.SeverityLevel={Verbose:0,Information:1,Warning:2,Error:3,Critical:4};var s=(d.extensionConfig||{}).ApplicationInsightsAnalytics||{};if(!0!==d[C]&&!0!==s[C]){method="onerror",t(["_"+method]);var c=T[method];T[method]=function(e,t,n,a,i){var r=c&&c(e,t,n,a,i);return!0!==r&&m["_"+method]({message:e,url:t,lineNumber:n,columnNumber:a,error:i}),r},d.autoExceptionInstrumented=!0}return m}(y.cfg);(T[t]=n).queue&&0===n.queue.length&&n.trackPageView({})}(window,document,{
|
||||
src: "https://az416426.vo.msecnd.net/scripts/b/ai.2.min.js",
|
||||
|
||||
|
||||
|
||||
|
||||
cfg: {
|
||||
instrumentationKey: "0600b273-5440-42cc-9fe4-51924c721ce0"
|
||||
|
||||
}});
|
||||
</script>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
|
||||
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
|
||||
<channel>
|
||||
<title>Bi-directional sync pattern Guide: A quick start template on Azure Storage</title>
|
||||
<link>https://azurestorage.com/docs/analytics/adls-gen1-to-gen2-migration/bi-directional/</link>
|
||||
<description>Recent content in Bi-directional sync pattern Guide: A quick start template on Azure Storage</description>
|
||||
<generator>Hugo -- gohugo.io</generator>
|
||||
<language>en-us</language><atom:link href="https://azurestorage.com/docs/analytics/adls-gen1-to-gen2-migration/bi-directional/index.xml" rel="self" type="application/rss+xml" />
|
||||
<item>
|
||||
<title>WANdisco Fusion Set up and Installation Guide</title>
|
||||
<link>https://azurestorage.com/docs/analytics/adls-gen1-to-gen2-migration/bi-directional/wandisco-set-up-and-installation/</link>
|
||||
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
|
||||
|
||||
<guid>https://azurestorage.com/docs/analytics/adls-gen1-to-gen2-migration/bi-directional/wandisco-set-up-and-installation/</guid>
|
||||
<description>WANdisco Fusion Set up and Installation Guide # Overview # This quickstart will help in setting up the Azure Linux Virtual Machine (VM) suitable for the WANdisco Fusion installation. Below will be covered:
|
||||
Azure Linux Virtual Machine (VM) creation using Azure Portal
|
||||
Configuration set up and Installation guide for WANdisco Fusion
|
||||
Prerequisites # Active Azure Subscription
|
||||
Azure Data Lake Storage Gen1</description>
|
||||
</item>
|
||||
|
||||
</channel>
|
||||
</rss>
|
||||
|
|
@ -0,0 +1,612 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en" dir=>
|
||||
|
||||
<head>
|
||||
<meta name="generator" content="Hugo 0.80.0" />
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<meta name="description" content="WANdisco Fusion Set up and Installation Guide # Overview # This quickstart will help in setting up the Azure Linux Virtual Machine (VM) suitable for the WANdisco Fusion installation. Below will be covered:
|
||||
Azure Linux Virtual Machine (VM) creation using Azure Portal
|
||||
Configuration set up and Installation guide for WANdisco Fusion
|
||||
Prerequisites # Active Azure Subscription
|
||||
Azure Data Lake Storage Gen1">
|
||||
<meta name="theme-color" content="#FFFFFF"><meta property="og:title" content="WANdisco Fusion Set up and Installation Guide" />
|
||||
<meta property="og:description" content="WANdisco Fusion Set up and Installation Guide # Overview # This quickstart will help in setting up the Azure Linux Virtual Machine (VM) suitable for the WANdisco Fusion installation. Below will be covered:
|
||||
Azure Linux Virtual Machine (VM) creation using Azure Portal
|
||||
Configuration set up and Installation guide for WANdisco Fusion
|
||||
Prerequisites # Active Azure Subscription
|
||||
Azure Data Lake Storage Gen1" />
|
||||
<meta property="og:type" content="article" />
|
||||
<meta property="og:url" content="https://azurestorage.com/docs/analytics/adls-gen1-to-gen2-migration/bi-directional/wandisco-set-up-and-installation/" />
|
||||
|
||||
<title>WANdisco Fusion Set up and Installation Guide | Azure Storage</title>
|
||||
<link rel="manifest" href="/manifest.json">
|
||||
<link rel="icon" href="/favicon.png" type="image/x-icon">
|
||||
<link rel="stylesheet" href="/book.min.6c7c6446dfdee7c8c933e9bbc6e80ee3ed6c913b2a59519f2092c3c6a9d63e55.css" integrity="sha256-bHxkRt/e58jJM+m7xugO4+1skTsqWVGfIJLDxqnWPlU=">
|
||||
<script defer src="/en.search.min.a092390ed9ca69153376eaacab4624d67749500866440e4365296f3fa207eb56.js" integrity="sha256-oJI5DtnKaRUzduqsq0Yk1ndJUAhmRA5DZSlvP6IH61Y="></script>
|
||||
<!--
|
||||
Made with Book Theme
|
||||
https://github.com/alex-shpak/hugo-book
|
||||
-->
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<body dir=>
|
||||
<input type="checkbox" class="hidden toggle" id="menu-control" />
|
||||
<input type="checkbox" class="hidden toggle" id="toc-control" />
|
||||
<main class="container flex">
|
||||
<aside class="book-menu">
|
||||
<div class="book-menu-content">
|
||||
|
||||
<nav>
|
||||
<h2 class="book-brand">
|
||||
<a href="/"><img src="/images/azure-icon.png" alt="Logo" /><span>Azure Storage</span>
|
||||
</a>
|
||||
</h2>
|
||||
|
||||
|
||||
<div class="book-search">
|
||||
<input type="text" id="book-search-input" placeholder="Search" aria-label="Search" maxlength="64" data-hotkeys="s/" />
|
||||
<div class="book-search-spinner hidden"></div>
|
||||
<ul id="book-search-results"></ul>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<ul>
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/analytics/" class="">Analytics</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/application-and-user-data/" class="">Application and User Data</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/backup-and-archive/" class="">Backup and Archive</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/hpc-iot-and-ai/" class="">HPC IoT and AI</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/storage-partners/" class="">Storage Partners</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/tools-and-utilities/" class="">Tools and Utilities</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
|
||||
<script>(function(){var menu=document.querySelector("aside.book-menu nav");addEventListener("beforeunload",function(event){localStorage.setItem("menu.scrollTop",menu.scrollTop);});menu.scrollTop=localStorage.getItem("menu.scrollTop");})();</script>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
<div class="book-page">
|
||||
<header class="book-header">
|
||||
|
||||
<div class="flex align-center justify-between">
|
||||
<label for="menu-control">
|
||||
<img src="/svg/menu.svg" class="book-icon" alt="Menu" />
|
||||
</label>
|
||||
|
||||
<strong>WANdisco Fusion Set up and Installation Guide</strong>
|
||||
|
||||
<label for="toc-control">
|
||||
|
||||
<img src="/svg/toc.svg" class="book-icon" alt="Table of Contents" />
|
||||
|
||||
</label>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<aside class="hidden clearfix">
|
||||
|
||||
|
||||
<nav id="TableOfContents">
|
||||
<ul>
|
||||
<li><a href="#overview">Overview</a></li>
|
||||
<li><a href="#prerequisites">Prerequisites</a></li>
|
||||
<li><a href="#azure-linux-virtual-machine-creation">Azure Linux Virtual Machine Creation</a></li>
|
||||
<li><a href="#virtual-machine-connection-set-up">Virtual Machine Connection set up</a>
|
||||
<ul>
|
||||
<li><a href="#create-an-ssh-connection-with-the-vm">Create an SSH connection with the VM</a></li>
|
||||
<li><a href="#connect-to-vm">Connect to VM</a></li>
|
||||
<li><a href="#wandisco-fusion-set-up">WANdisco Fusion Set up</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#adls-gen1-and-gen2-configuration">ADLS Gen1 and Gen2 Configuration</a>
|
||||
<ul>
|
||||
<li><a href="#adls-gen1-storage-configuration">ADLS Gen1 storage Configuration</a></li>
|
||||
<li><a href="#adls-gen2-storage-configuration">ADLS Gen2 storage Configuration</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#references">References</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
</aside>
|
||||
|
||||
|
||||
</header>
|
||||
|
||||
|
||||
|
||||
<article class="markdown"><h1 id="wandisco-fusion-set-up-and-installation-guide">
|
||||
WANdisco Fusion Set up and Installation Guide
|
||||
<a class="anchor" href="#wandisco-fusion-set-up-and-installation-guide">#</a>
|
||||
</h1>
|
||||
<h2 id="overview">
|
||||
Overview
|
||||
<a class="anchor" href="#overview">#</a>
|
||||
</h2>
|
||||
<p>This quickstart will help in setting up the Azure Linux Virtual Machine (VM) suitable for the WANdisco Fusion installation. Below will be covered:</p>
|
||||
<ul>
|
||||
<li>
|
||||
<p>Azure Linux Virtual Machine (VM) creation using Azure Portal</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Configuration set up and Installation guide for WANdisco Fusion</p>
|
||||
</li>
|
||||
</ul>
|
||||
<h2 id="prerequisites">
|
||||
Prerequisites
|
||||
<a class="anchor" href="#prerequisites">#</a>
|
||||
</h2>
|
||||
<ul>
|
||||
<li>
|
||||
<p><strong>Active Azure Subscription</strong></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Azure Data Lake Storage Gen1</strong></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Azure Data Lake Storage Gen2</strong>. For more details please refer to <a href="https://docs.microsoft.com/azure/storage/common/storage-account-create">create azure storage account</a></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Windows SSH client</strong> like <a href="https://www.putty.org/">Putty</a>, <a href="https://gitforwindows.org/">Git for Windows</a>, <a href="https://cygwin.com/">Cygwin</a>, <a href="https://mobaxterm.mobatek.net/">MobaXterm</a></p>
|
||||
</li>
|
||||
</ul>
|
||||
<h2 id="azure-linux-virtual-machine-creation">
|
||||
Azure Linux Virtual Machine Creation
|
||||
<a class="anchor" href="#azure-linux-virtual-machine-creation">#</a>
|
||||
</h2>
|
||||
<ol>
|
||||
<li>
|
||||
<p>Go To <strong>Azure Portal Home page</strong></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Click</strong> on <strong>+ Create a resource</strong></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Search for <strong>Ubuntu Server</strong>. Select <strong>Ubuntu Server 16.04 LTS</strong>.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Click</strong> on <strong>Create</strong></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>In the <strong>Basics</strong> tab, under <strong>Project details</strong>, make sure the correct subscription is selected and then choose existing Resource group or <strong>Create new</strong> one.</p>
|
||||
<p><img src="../../images/80261884-9b509580-8640-11ea-9b4e-b4e909ce76ca.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Under <strong>Instance details</strong>, type any name for the Virtual machine name, choose East US for your Region, and choose Ubuntu 18.04 LTS for Image. Leave the other defaults.</p>
|
||||
<p><img src="../../images/80262349-0189e800-8642-11ea-84a5-954f74bc7b2c.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Under <strong>Administrator account</strong>, select <strong>SSH public key</strong> or <strong>password</strong>. Fill the details as required. Under <strong>Inbound port rules</strong> > <strong>Public inbound ports</strong>, choose <strong>Allow selected ports</strong> and then select <strong>SSH (22)</strong> and <strong>HTTP (80)</strong> from the drop-down.</p>
|
||||
<p><img src="../../images/80262658-000cef80-8643-11ea-9007-462dc366e2d3.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Leave the defaults under <strong>Disks</strong>, <strong>Networking</strong>, <strong>Management</strong> . In the <strong>Advanced</strong> tab under <strong>Cloud init</strong> text field, paste the <a href="../cloud-init.txt">cloud init</a> content.</p>
|
||||
<p><img src="../../images/80263267-f2586980-8644-11ea-86b8-4b1714779948.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Leave the remaining defaults and then select the <strong>Review + create</strong> button at the bottom of the page.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>On the Create a virtual machine page, you can see the details about the VM you are about to create. When you are ready, select <strong>Create</strong>.</p>
|
||||
</li>
|
||||
</ol>
|
||||
<h2 id="virtual-machine-connection-set-up">
|
||||
Virtual Machine Connection set up
|
||||
<a class="anchor" href="#virtual-machine-connection-set-up">#</a>
|
||||
</h2>
|
||||
<h3 id="create-an-ssh-connection-with-the-vm">
|
||||
Create an SSH connection with the VM
|
||||
<a class="anchor" href="#create-an-ssh-connection-with-the-vm">#</a>
|
||||
</h3>
|
||||
<ol>
|
||||
<li>
|
||||
<p>Select the <strong>Connect</strong> button on the overview page for your VM.</p>
|
||||
<p><img src="../../images/80268818-407a6680-865f-11ea-818f-3d4dca06ed00.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Go to <strong>Networking</strong> under <strong>Settings</strong>.</p>
|
||||
<p><img src="../../images/80269030-1f1a7a00-8661-11ea-925f-2beb8cca0d2a.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Click on <strong>Add inbound port rule</strong>.</p>
|
||||
<p><img src="../../images/80269048-47a27400-8661-11ea-8aa6-cc65d19bc8ad.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Select <strong>Source</strong> as <strong>IP addresses</strong> from the drop down. Provide your <a href="https://whatismyipaddress.com/">source ip</a> address in <strong>Source IP addresses/CIDR ranges</strong>. Provide list of port ranges as <strong>22,8081,8083,8084</strong> in the <strong>Destination port ranges</strong> field. Choose <strong>TCP</strong> under <strong>Protocol</strong>. Give any <strong>Name</strong></p>
|
||||
<p><img src="../../images/80269123-bd0e4480-8661-11ea-8c92-d8923aaaa324.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Click on <strong>Add</strong> button.</p>
|
||||
</li>
|
||||
</ol>
|
||||
<h3 id="connect-to-vm">
|
||||
Connect to VM
|
||||
<a class="anchor" href="#connect-to-vm">#</a>
|
||||
</h3>
|
||||
<p>To connect to the VM created above, you need a secure shell protocol (SSH) client like <a href="https://www.putty.org/">Putty</a>, <a href="https://gitforwindows.org/">Git for Windows</a>, <a href="https://cygwin.com/">Cygwin</a>, <a href="https://mobaxterm.mobatek.net/">MobaXterm</a></p>
|
||||
<ol>
|
||||
<li>
|
||||
<p>Start the VM if it isn’t already running. Under <strong>Overview</strong> configure the DNSname dynamic and set DNS name.</p>
|
||||
<p><img src="../../images/80270761-ca323000-866f-11ea-94bf-20e02133bb11.png" alt="image" /></p>
|
||||
<p>The above DNS name can be used to login into SSH client.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Open the SSH client (Putty , Git , Cygwin, MobaXterm).</p>
|
||||
<p>:bulb: Note : Here we will be using MobaXterm.</p>
|
||||
<p>Go to <strong>Session</strong> —> <strong>Click</strong> on <strong>SSH</strong></p>
|
||||
<p><img src="../../images/80332096-b406aa00-87fe-11ea-9ebc-4a5c4a757833.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Provide the DNSname into the <strong>Remote host</strong> field along with username defined for SSH client while creating VM.</p>
|
||||
<p><img src="../../images/80333937-86bcfa80-8804-11ea-93ca-cefb660789f2.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Click <strong>OK</strong></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Provide the password for SSH client.</p>
|
||||
<p><img src="../../images/80334350-d18b4200-8805-11ea-881c-866858a255c4.png" alt="image" /></p>
|
||||
</li>
|
||||
</ol>
|
||||
<h3 id="wandisco-fusion-set-up">
|
||||
WANdisco Fusion Set up
|
||||
<a class="anchor" href="#wandisco-fusion-set-up">#</a>
|
||||
</h3>
|
||||
<ol>
|
||||
<li>
|
||||
<p>Clone the Fusion docker repository using below command in SSH Client:</p>
|
||||
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-scala" data-lang="scala">git clone https<span style="color:#66d9ef">:</span><span style="color:#75715e">//github.com/WANdisco/fusion-docker-compose.git
|
||||
</span></code></pre></div></li>
|
||||
<li>
|
||||
<p>Change to the repository directory:</p>
|
||||
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-scala" data-lang="scala">cd fusion<span style="color:#f92672">-</span>docker<span style="color:#f92672">-</span>compose
|
||||
</code></pre></div></li>
|
||||
<li>
|
||||
<p>Run the setup script:</p>
|
||||
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-scala" data-lang="scala"><span style="color:#f92672">./</span>setup<span style="color:#f92672">-</span>env<span style="color:#f92672">.</span>sh
|
||||
</code></pre></div></li>
|
||||
<li>
|
||||
<p>Enter the option <strong>4</strong> for <strong>Custom deployment</strong></p>
|
||||
<p><img src="../../images/80396711-e6012600-8869-11ea-8161-cfbcc25c3170.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Enter the first zone type as <strong>adls1</strong></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Set the first zone name as [adls1] . Hit enter at the prompt.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Enter the second zone type as <strong>adls2</strong></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Set the second zone name as [adls2]. Hit <strong>enter</strong> key at the prompt.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Enter your license file path. Hit <strong>enter</strong> key at the prompt.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Enter the docker hostname. Hit <strong>enter</strong> key at the prompt for setting default name.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Enter the HDI version for adls1 as <strong>3.6</strong></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Enter HDI version for adls2 as <strong>4.0</strong> . Hit <strong>enter</strong> key for rest prompts</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>The docker set up is complete.</p>
|
||||
<p><img src="../../images/80404559-38e0da80-8876-11ea-9cc4-22314a46fedc.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>To start the Fusion run the below command:</p>
|
||||
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-scala" data-lang="scala">docker<span style="color:#f92672">-</span>compose up <span style="color:#f92672">-</span>d
|
||||
</code></pre></div><p><img src="../../images/80407953-4ba9de00-887b-11ea-97a5-2baa8683943d.png" alt="image" /></p>
|
||||
</li>
|
||||
</ol>
|
||||
<h2 id="adls-gen1-and-gen2-configuration">
|
||||
ADLS Gen1 and Gen2 Configuration
|
||||
<a class="anchor" href="#adls-gen1-and-gen2-configuration">#</a>
|
||||
</h2>
|
||||
<h3 id="adls-gen1-storage-configuration">
|
||||
ADLS Gen1 storage Configuration
|
||||
<a class="anchor" href="#adls-gen1-storage-configuration">#</a>
|
||||
</h3>
|
||||
<ol>
|
||||
<li>
|
||||
<p>Log in to Fusion via a web browser.</p>
|
||||
<p>Enter the address in the form of: http://{dnsname}:8081</p>
|
||||
<p><img src="../../images/80413692-b7dd0f80-8884-11ea-82d0-9925706b4522.png" alt="image" /></p>
|
||||
<blockquote>
|
||||
<p>Note: Get the DNS name from portal <strong>–></strong> Go to Virtual machine <strong>–></strong> Overview <strong>–></strong> DNS name</p>
|
||||
</blockquote>
|
||||
</li>
|
||||
<li>
|
||||
<p>Create account and login to the Fusion.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Click on settings icon for the adls1 storage. Select the ADLS Gen1 storage type</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Enter the details for the ADLS Gen1 storage account</p>
|
||||
<p><img src="../../images/80416242-a8f85c00-8888-11ea-851c-d23f38749d56.png" alt="image" /></p>
|
||||
<p><strong>ADLS Gen1 storage account details</strong></p>
|
||||
<ul>
|
||||
<li>
|
||||
<p>Hostname / Endpoint (Example: <account-name>.azuredatalakestore.net)</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Home Mount Point / Directory (Example: / or /path/to/mountpoint)</p>
|
||||
<blockquote>
|
||||
<p>Note: Fusion will be able to read and write to everything contained within the Mount Point.</p>
|
||||
</blockquote>
|
||||
</li>
|
||||
<li>
|
||||
<p>Client ID / Application ID (Example: a73t6742-2e93-45ty-bd6d-4a8art6578ip)</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Refresh URL (Example: <code>https://login.microsoftonline.com/<tenant-id>/oauth2/token</code>)</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Handshake User / Service principal name (Example: fusion-app)</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>ADL credential / Application secret (Example: 8A767YUIa900IuaDEF786DTY67t-u=:])</p>
|
||||
</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>
|
||||
<p>Click on <strong>APPLY CONFIGURATION</strong></p>
|
||||
</li>
|
||||
</ol>
|
||||
<h3 id="adls-gen2-storage-configuration">
|
||||
ADLS Gen2 storage Configuration
|
||||
<a class="anchor" href="#adls-gen2-storage-configuration">#</a>
|
||||
</h3>
|
||||
<ol>
|
||||
<li>
|
||||
<p>Click on settings icon for the adls2 storage. Select the ADLS Gen2 storage type</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Enter the details for the ADLS Gen2 storage account</p>
|
||||
<p><img src="../../images/80425042-c1bc3e00-8897-11ea-96b8-891c6e739935.png" alt="image" /></p>
|
||||
<p><strong>ADLS Gen2 storage account details</strong></p>
|
||||
<ul>
|
||||
<li>
|
||||
<p>Account name (Example: adlsg2storage)</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Container name (Example: fusionreplication)</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Access key (Example: eTFdESnXOuG2qoUrqlDyCL+e6456789opasweghtfFMKAHjJg5JkCG8t1h2U1BzXvBwtYfoj5nZaDF87UK09po==)</p>
|
||||
</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>
|
||||
<p>Click on <strong>APPLY CONFIGURATION</strong></p>
|
||||
</li>
|
||||
</ol>
|
||||
<h2 id="references">
|
||||
References
|
||||
<a class="anchor" href="#references">#</a>
|
||||
</h2>
|
||||
<ul>
|
||||
<li><a href="https://wandisco.github.io/wandisco-documentation/docs/quickstarts/preparation/azure_vm_creation">WANdisco fusion Installation and set up guide</a></li>
|
||||
<li><a href="https://docs.microsoft.com/azure/virtual-machines/linux/ssh-from-windows">How to use SSH key with Windows on Azure</a></li>
|
||||
</ul>
|
||||
</article>
|
||||
|
||||
|
||||
|
||||
<footer class="book-footer">
|
||||
|
||||
<div class="flex flex-wrap justify-between">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<hr />
|
||||
Azure Storage ©2021 <br />
|
||||
Visit the <a href="https://azure.microsoft.com/services/storage/">Azure Storage homepage</a> or read our <a href="https://docs.microsoft.com/azure/storage/">getting started guide</a> or the <a href="https://azure.microsoft.com/blog/topics/storage-backup-and-recovery/">Azure Storage Blog</a>. <br />
|
||||
Contact us: <a href="mailto:azurestoragefeedback@microsoft.com?subject=AzureStorage.com%20Feedback">azurestoragefeedback@microsoft.com</a>.<br />
|
||||
Generated on Mon, Feb 08 2021 23:18:57 UTC
|
||||
</footer>
|
||||
|
||||
|
||||
|
||||
<div class="book-comments">
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<label for="menu-control" class="hidden book-menu-overlay"></label>
|
||||
</div>
|
||||
|
||||
|
||||
<aside class="book-toc">
|
||||
<div class="book-toc-content">
|
||||
|
||||
|
||||
<nav id="TableOfContents">
|
||||
<ul>
|
||||
<li><a href="#overview">Overview</a></li>
|
||||
<li><a href="#prerequisites">Prerequisites</a></li>
|
||||
<li><a href="#azure-linux-virtual-machine-creation">Azure Linux Virtual Machine Creation</a></li>
|
||||
<li><a href="#virtual-machine-connection-set-up">Virtual Machine Connection set up</a>
|
||||
<ul>
|
||||
<li><a href="#create-an-ssh-connection-with-the-vm">Create an SSH connection with the VM</a></li>
|
||||
<li><a href="#connect-to-vm">Connect to VM</a></li>
|
||||
<li><a href="#wandisco-fusion-set-up">WANdisco Fusion Set up</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#adls-gen1-and-gen2-configuration">ADLS Gen1 and Gen2 Configuration</a>
|
||||
<ul>
|
||||
<li><a href="#adls-gen1-storage-configuration">ADLS Gen1 storage Configuration</a></li>
|
||||
<li><a href="#adls-gen2-storage-configuration">ADLS Gen2 storage Configuration</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#references">References</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
</main>
|
||||
|
||||
<script src="https://code.jquery.com/jquery-3.5.1.slim.min.js" integrity="sha256-4+XzXVhsDmqanXGHaHvgh1gMQKX40OUvDEBTu8JcmNs=" crossorigin="anonymous"></script>
|
||||
|
||||
<script>
|
||||
$('a[href^="http://"], a[href^="https://"]').not('a[href*="'+location.hostname+'"]').attr('target','_blank');
|
||||
</script>
|
||||
|
||||
<script type="text/javascript">
|
||||
!function(T,l,y){var S=T.location,u="script",k="instrumentationKey",D="ingestionendpoint",C="disableExceptionTracking",E="ai.device.",I="toLowerCase",b="crossOrigin",w="POST",e="appInsightsSDK",t=y.name||"appInsights";(y.name||T[e])&&(T[e]=t);var n=T[t]||function(d){var g=!1,f=!1,m={initialize:!0,queue:[],sv:"4",version:2,config:d};function v(e,t){var n={},a="Browser";return n[E+"id"]=a[I](),n[E+"type"]=a,n["ai.operation.name"]=S&&S.pathname||"_unknown_",n["ai.internal.sdkVersion"]="javascript:snippet_"+(m.sv||m.version),{time:function(){var e=new Date;function t(e){var t=""+e;return 1===t.length&&(t="0"+t),t}return e.getUTCFullYear()+"-"+t(1+e.getUTCMonth())+"-"+t(e.getUTCDate())+"T"+t(e.getUTCHours())+":"+t(e.getUTCMinutes())+":"+t(e.getUTCSeconds())+"."+((e.getUTCMilliseconds()/1e3).toFixed(3)+"").slice(2,5)+"Z"}(),iKey:e,name:"Microsoft.ApplicationInsights."+e.replace(/-/g,"")+"."+t,sampleRate:100,tags:n,data:{baseData:{ver:2}}}}var h=d.url||y.src;if(h){function a(e){var t,n,a,i,r,o,s,c,p,l,u;g=!0,m.queue=[],f||(f=!0,t=h,s=function(){var e={},t=d.connectionString;if(t)for(var n=t.split(";"),a=0;a<n.length;a++){var i=n[a].split("=");2===i.length&&(e[i[0][I]()]=i[1])}if(!e[D]){var r=e.endpointsuffix,o=r?e.location:null;e[D]="https://"+(o?o+".":"")+"dc."+(r||"services.visualstudio.com")}return e}(),c=s[k]||d[k]||"",p=s[D],l=p?p+"/v2/track":config.endpointUrl,(u=[]).push((n="SDK LOAD Failure: Failed to load Application Insights SDK script (See stack for details)",a=t,i=l,(o=(r=v(c,"Exception")).data).baseType="ExceptionData",o.baseData.exceptions=[{typeName:"SDKLoadFailed",message:n.replace(/\./g,"-"),hasFullStack:!1,stack:n+"\nSnippet failed to load ["+a+"] -- Telemetry is disabled\nHelp Link: https://go.microsoft.com/fwlink/?linkid=2128109\nHost: "+(S&&S.pathname||"_unknown_")+"\nEndpoint: "+i,parsedStack:[]}],r)),u.push(function(e,t,n,a){var i=v(c,"Message"),r=i.data;r.baseType="MessageData";var o=r.baseData;return o.message='AI (Internal): 99 message:"'+("SDK LOAD Failure: Failed to load Application Insights SDK script (See stack for details) ("+n+")").replace(/\"/g,"")+'"',o.properties={endpoint:a},i}(0,0,t,l)),function(e,t){if(JSON){var n=T.fetch;if(n&&!y.useXhr)n(t,{method:w,body:JSON.stringify(e),mode:"cors"});else if(XMLHttpRequest){var a=new XMLHttpRequest;a.open(w,t),a.setRequestHeader("Content-type","application/json"),a.send(JSON.stringify(e))}}}(u,l))}function i(e,t){f||setTimeout(function(){!t&&m.core||a()},500)}var e=function(){var n=l.createElement(u);n.src=h;var e=y[b];return!e&&""!==e||"undefined"==n[b]||(n[b]=e),n.onload=i,n.onerror=a,n.onreadystatechange=function(e,t){"loaded"!==n.readyState&&"complete"!==n.readyState||i(0,t)},n}();y.ld<0?l.getElementsByTagName("head")[0].appendChild(e):setTimeout(function(){l.getElementsByTagName(u)[0].parentNode.appendChild(e)},y.ld||0)}try{m.cookie=l.cookie}catch(p){}function t(e){for(;e.length;)!function(t){m[t]=function(){var e=arguments;g||m.queue.push(function(){m[t].apply(m,e)})}}(e.pop())}var n="track",r="TrackPage",o="TrackEvent";t([n+"Event",n+"PageView",n+"Exception",n+"Trace",n+"DependencyData",n+"Metric",n+"PageViewPerformance","start"+r,"stop"+r,"start"+o,"stop"+o,"addTelemetryInitializer","setAuthenticatedUserContext","clearAuthenticatedUserContext","flush"]),m.SeverityLevel={Verbose:0,Information:1,Warning:2,Error:3,Critical:4};var s=(d.extensionConfig||{}).ApplicationInsightsAnalytics||{};if(!0!==d[C]&&!0!==s[C]){method="onerror",t(["_"+method]);var c=T[method];T[method]=function(e,t,n,a,i){var r=c&&c(e,t,n,a,i);return!0!==r&&m["_"+method]({message:e,url:t,lineNumber:n,columnNumber:a,error:i}),r},d.autoExceptionInstrumented=!0}return m}(y.cfg);(T[t]=n).queue&&0===n.queue.length&&n.trackPageView({})}(window,document,{
|
||||
src: "https://az416426.vo.msecnd.net/scripts/b/ai.2.min.js",
|
||||
|
||||
|
||||
|
||||
|
||||
cfg: {
|
||||
instrumentationKey: "0600b273-5440-42cc-9fe4-51924c721ce0"
|
||||
|
||||
}});
|
||||
</script>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,565 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en" dir=>
|
||||
|
||||
<head>
|
||||
<meta name="generator" content="Hugo 0.80.0" />
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<meta name="description" content="Dual Pipeline Pattern Guide: A quick start template # Overview # The purpose of this document is to provide a manual for the use of Dual pipeline pattern for migration of data from Gen1 to Gen2. This provides the directions, references and approach how to set up the Dual pipeline, do migration of existing data from Gen1 to Gen2 and set up the workloads to run at Gen2 endpoint.">
|
||||
<meta name="theme-color" content="#FFFFFF"><meta property="og:title" content="Dual Pipeline Pattern Guide: A quick start template" />
|
||||
<meta property="og:description" content="Dual Pipeline Pattern Guide: A quick start template # Overview # The purpose of this document is to provide a manual for the use of Dual pipeline pattern for migration of data from Gen1 to Gen2. This provides the directions, references and approach how to set up the Dual pipeline, do migration of existing data from Gen1 to Gen2 and set up the workloads to run at Gen2 endpoint." />
|
||||
<meta property="og:type" content="article" />
|
||||
<meta property="og:url" content="https://azurestorage.com/docs/analytics/adls-gen1-to-gen2-migration/dual-pipeline/" />
|
||||
|
||||
<title>Dual Pipeline Pattern Guide: A quick start template | Azure Storage</title>
|
||||
<link rel="manifest" href="/manifest.json">
|
||||
<link rel="icon" href="/favicon.png" type="image/x-icon">
|
||||
<link rel="stylesheet" href="/book.min.6c7c6446dfdee7c8c933e9bbc6e80ee3ed6c913b2a59519f2092c3c6a9d63e55.css" integrity="sha256-bHxkRt/e58jJM+m7xugO4+1skTsqWVGfIJLDxqnWPlU=">
|
||||
<script defer src="/en.search.min.a092390ed9ca69153376eaacab4624d67749500866440e4365296f3fa207eb56.js" integrity="sha256-oJI5DtnKaRUzduqsq0Yk1ndJUAhmRA5DZSlvP6IH61Y="></script>
|
||||
<!--
|
||||
Made with Book Theme
|
||||
https://github.com/alex-shpak/hugo-book
|
||||
-->
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<body dir=>
|
||||
<input type="checkbox" class="hidden toggle" id="menu-control" />
|
||||
<input type="checkbox" class="hidden toggle" id="toc-control" />
|
||||
<main class="container flex">
|
||||
<aside class="book-menu">
|
||||
<div class="book-menu-content">
|
||||
|
||||
<nav>
|
||||
<h2 class="book-brand">
|
||||
<a href="/"><img src="/images/azure-icon.png" alt="Logo" /><span>Azure Storage</span>
|
||||
</a>
|
||||
</h2>
|
||||
|
||||
|
||||
<div class="book-search">
|
||||
<input type="text" id="book-search-input" placeholder="Search" aria-label="Search" maxlength="64" data-hotkeys="s/" />
|
||||
<div class="book-search-spinner hidden"></div>
|
||||
<ul id="book-search-results"></ul>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<ul>
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/analytics/" class="">Analytics</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/application-and-user-data/" class="">Application and User Data</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/backup-and-archive/" class="">Backup and Archive</a>
|
||||
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/hpc-iot-and-ai/" class="">HPC IoT and AI</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/storage-partners/" class="">Storage Partners</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a href="https://azurestorage.com/docs/tools-and-utilities/" class="">Tools and Utilities</a>
|
||||
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
|
||||
<script>(function(){var menu=document.querySelector("aside.book-menu nav");addEventListener("beforeunload",function(event){localStorage.setItem("menu.scrollTop",menu.scrollTop);});menu.scrollTop=localStorage.getItem("menu.scrollTop");})();</script>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
<div class="book-page">
|
||||
<header class="book-header">
|
||||
|
||||
<div class="flex align-center justify-between">
|
||||
<label for="menu-control">
|
||||
<img src="/svg/menu.svg" class="book-icon" alt="Menu" />
|
||||
</label>
|
||||
|
||||
<strong>Dual Pipeline Pattern Guide: A quick start template</strong>
|
||||
|
||||
<label for="toc-control">
|
||||
|
||||
<img src="/svg/toc.svg" class="book-icon" alt="Table of Contents" />
|
||||
|
||||
</label>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<aside class="hidden clearfix">
|
||||
|
||||
|
||||
<nav id="TableOfContents">
|
||||
<ul>
|
||||
<li><a href="#overview">Overview</a></li>
|
||||
<li><a href="#prerequisites">Prerequisites</a></li>
|
||||
<li><a href="#data-pipeline-set-up-for-gen1-and-gen2">Data pipeline set up for Gen1 and Gen2</a>
|
||||
<ul>
|
||||
<li><a href="#creation-of-linked-service-for-gen1-and-gen2-in-adf">Creation of linked service for Gen1 and Gen2 in ADF</a></li>
|
||||
<li><a href="#how-to-set-up-gen1-data-pipeline">How to set up Gen1 data pipeline</a></li>
|
||||
<li><a href="#how-to-set-up-gen2-data-pipeline">How to set up Gen2 data pipeline</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#steps-to-be-followed">Steps to be followed</a>
|
||||
<ul>
|
||||
<li><a href="#migrate-data-from-gen1-to-gen2">Migrate data from Gen1 to Gen2</a></li>
|
||||
<li><a href="#data-ingestion-to-gen1-and-gen2">Data ingestion to Gen1 and Gen2</a></li>
|
||||
<li><a href="#run-workloads-at-gen2">Run workloads at Gen2</a></li>
|
||||
<li><a href="#cutover-from-gen1-to-gen2">Cutover from Gen1 to Gen2</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#references">References</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
</aside>
|
||||
|
||||
|
||||
</header>
|
||||
|
||||
|
||||
|
||||
<article class="markdown"><h1 id="dual-pipeline-pattern-guide-a-quick-start-template">
|
||||
Dual Pipeline Pattern Guide: A quick start template
|
||||
<a class="anchor" href="#dual-pipeline-pattern-guide-a-quick-start-template">#</a>
|
||||
</h1>
|
||||
<h2 id="overview">
|
||||
Overview
|
||||
<a class="anchor" href="#overview">#</a>
|
||||
</h2>
|
||||
<p>The purpose of this document is to provide a manual for the use of Dual pipeline pattern for migration of data from Gen1 to Gen2. This provides the directions, references and approach how to set up the Dual pipeline, do migration of existing data from Gen1 to Gen2 and set up the workloads to run at Gen2 endpoint.</p>
|
||||
<p>Considerations for using the dual pipeline pattern:</p>
|
||||
<ul>
|
||||
<li>Gen1 and Gen2 pipelines run side-by-side.</li>
|
||||
<li>Supports zero downtime.</li>
|
||||
<li>Ideal in situations where your workloads and applications can’t afford any downtime, and you can ingest into both storage accounts.</li>
|
||||
</ul>
|
||||
<h2 id="prerequisites">
|
||||
Prerequisites
|
||||
<a class="anchor" href="#prerequisites">#</a>
|
||||
</h2>
|
||||
<ul>
|
||||
<li>
|
||||
<p><strong>Active Azure Subscription</strong></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Azure Data Lake Storage Gen1</strong></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Azure Data Lake Storage Gen2</strong>. For more details please refer to <a href="https://docs.microsoft.com/azure/storage/common/storage-account-create?tabs=azure-portal">create azure storage account</a></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Azure Key Vault</strong>. Required keys and secrets to be configured here.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Service principal</strong> with read, write and execute permission to the resource group, key vault, data lake store Gen1 and data lake store Gen2. To learn more, see <a href="https://docs.microsoft.com/azure/active-directory/develop/howto-create-service-principal-portal">create service principal account</a> and to provide SPN access to Gen1 refer to <a href="https://docs.microsoft.com/azure/data-lake-store/data-lake-store-service-to-service-authenticate-using-active-directory">SPN access to Gen1</a></p>
|
||||
</li>
|
||||
</ul>
|
||||
<h2 id="data-pipeline-set-up-for-gen1-and-gen2">
|
||||
Data pipeline set up for Gen1 and Gen2
|
||||
<a class="anchor" href="#data-pipeline-set-up-for-gen1-and-gen2">#</a>
|
||||
</h2>
|
||||
<p>As part of this pattern, Gen1 and Gen2 pipelines will run side by side.</p>
|
||||
<p>Below is the sample pipeline set up for Gen1 and Gen2 using Azure Databricks for data ingestion, HDInsight for data processing and Azure SQL DW for storing the processed data for analytics.</p>
|
||||
<p><img src="../images/83429980-c2417a80-a3e9-11ea-9ab6-4d08b02b51b1.png" alt="image" /></p>
|
||||
<p><img src="../images/83563507-16bf2580-a4d0-11ea-9707-ae659f87eb3d.png" alt="image" /></p>
|
||||
<p><strong>Prerequisite</strong></p>
|
||||
<ul>
|
||||
<li>
|
||||
<p>Create <strong>HDInsight cluster</strong> for Gen1. Refer <a href="https://docs.microsoft.com/azure/data-lake-store/data-lake-store-hdinsight-hadoop-use-portal">here</a> for more details.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Create <strong>HDInsight cluster</strong> for Gen2. Refer <a href="https://docs.microsoft.com/azure/hdinsight/hdinsight-hadoop-use-data-lake-storage-gen2">here</a> for more details.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Create <strong>user assigned managed identity</strong>. Refer <a href="https://docs.microsoft.com/azure/active-directory/managed-identities-azure-resources/how-to-manage-ua-identity-portal#create-a-user-assigned-managed-identity">here</a> to know more.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Permission should be set up for the managed identity for Gen2 storage account. Refer <a href="https://docs.microsoft.com/azure/hdinsight/hdinsight-hadoop-use-data-lake-storage-gen2#set-up-permissions-for-the-managed-identity-on-the-data-lake-storage-gen2-account">here</a> for more details.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Additional blob storage should be created for Gen1 to support HDInsight linked service in ADF. Refer <a href="https://docs.microsoft.com/azure/storage/blobs/storage-blob-create-account-block-blob?tabs=azure-portal">here</a> for more details.</p>
|
||||
</li>
|
||||
</ul>
|
||||
<blockquote>
|
||||
<p>Note: To set up the data pipeline in ADF, two separate HDInsight clusters should be created each for Gen1 and Gen2.</p>
|
||||
</blockquote>
|
||||
<p>Here ADF is used for orchestrating data-processing pipelines supporting data ingestion, copying data from and to different storage types (Gen1 and Gen2) in azure, loading the processed data to datawarehouse and executing transformation logic.</p>
|
||||
<p><img src="../images/83435632-6b3fa380-a3f1-11ea-8639-dba1e217e044.png" alt="image" /></p>
|
||||
<h3 id="creation-of-linked-service-for-gen1-and-gen2-in-adf">
|
||||
Creation of linked service for Gen1 and Gen2 in ADF
|
||||
<a class="anchor" href="#creation-of-linked-service-for-gen1-and-gen2-in-adf">#</a>
|
||||
</h3>
|
||||
<p>As part of this pipeline set up, below linked services needs to be created as first step in ADF:</p>
|
||||
<p>Go to ADF –> Manage –> Linked service –> Click on <strong>+ New</strong></p>
|
||||
<p><img src="../images/84070157-875abc00-a980-11ea-9d27-9764b577e2d0.png" alt="image" /></p>
|
||||
<ol>
|
||||
<li>
|
||||
<p>Create <strong><a href="https://docs.microsoft.com/azure/data-factory/transform-data-using-databricks-notebook#create-an-azure-databricks-linked-service">ADB linked service</a></strong>.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Create <strong><a href="https://docs.microsoft.com/azure/hdinsight/hdinsight-hadoop-create-linux-clusters-adf#create-an-azure-storage-linked-service">HDInsight linked service</a></strong>.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Create <strong><a href="https://docs.microsoft.comazure/data-factory/load-azure-sql-data-warehouse#load-data-into-azure-synapse-analytics">Stored procedure linked service</a></strong>.</p>
|
||||
</li>
|
||||
</ol>
|
||||
<p><strong>How to create HDInsight linked service for Gen1(Blob storage)</strong></p>
|
||||
<ol>
|
||||
<li>
|
||||
<p>Go to <strong>Linked Services</strong> –> <strong>click</strong> on <strong>+ New</strong> –> <strong>New linked service</strong> –> <strong>Compute</strong> –> <strong>Azure HDInsight</strong> –> <strong>Continue</strong></p>
|
||||
<p><img src="../images/83468627-356edf00-a432-11ea-9375-0594ab25b975.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Provide the details from Azure subscription with respect to each field and choose <strong>Blob Storage</strong> under <strong>Azure Storage linked service</strong></p>
|
||||
<p><img src="../images/83469679-335a4f80-a435-11ea-91c3-2d844be17cbe.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Provide the user name and password details.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Click on <strong>Create</strong> button.</p>
|
||||
</li>
|
||||
</ol>
|
||||
<p><strong>How to create HDInsight linked service for Gen2</strong></p>
|
||||
<ol>
|
||||
<li>
|
||||
<p>Go to <strong>Linked Services</strong> –> <strong>click</strong> on <strong>+ New</strong> –> <strong>New linked service</strong> –> <strong>Compute</strong> –> <strong>Azure HDInsight</strong> –> <strong>Continue</strong></p>
|
||||
<p><img src="../images/83468627-356edf00-a432-11ea-9375-0594ab25b975.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Provide the details from Azure subscription with respect to each field and choose <strong>ADLS Gen 2</strong> under <strong>Azure Storage linked service</strong></p>
|
||||
<p><img src="../images/83479945-d79cc000-a44e-11ea-8940-3884606cec7b.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Provide the storage container name in the <strong>File system</strong> field. Give the user name and password.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Click on <strong>Create</strong> button.</p>
|
||||
</li>
|
||||
</ol>
|
||||
<h3 id="how-to-set-up-gen1-data-pipeline">
|
||||
How to set up Gen1 data pipeline
|
||||
<a class="anchor" href="#how-to-set-up-gen1-data-pipeline">#</a>
|
||||
</h3>
|
||||
<p>Create a master pipeline in ADF for Gen1 and invoke all activities listed below:</p>
|
||||
<p><img src="../images/83823632-48c7b780-a689-11ea-903f-068b26ca3741.png" alt="image" /></p>
|
||||
<ol>
|
||||
<li>
|
||||
<p><strong>Raw data ingestion using ADB script</strong></p>
|
||||
<p>Create a pipeline for data ingestion process using ADB activity. Refer <a href="https://docs.microsoft.com/azure/data-factory/transform-data-using-databricks-notebook#create-a-pipeline">here</a> for more details.</p>
|
||||
<p><img src="../images/83448158-63d6c500-a406-11ea-8a29-a1cdd514509c.png" alt="image" /></p>
|
||||
<p><strong>Mount path configured to Gen1 endpoint</strong></p>
|
||||
<p><img src="../images/83800138-cecd0980-a65b-11ea-93de-17b9fc016e90.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Data processing using HDInsight</strong></p>
|
||||
<p>Create a pipeline for data processing using HDInsight activity. Refer <a href="https://docs.microsoft.com/azure/hdinsight/hdinsight-hadoop-create-linux-clusters-adf#create-a-pipeline">here</a> for more details.</p>
|
||||
<p><img src="../images/83450714-a6020580-a40a-11ea-8c99-55c2c9a96104.png" alt="image" /></p>
|
||||
<p><strong>Mount path configured to Gen1 endpoint</strong></p>
|
||||
<p><img src="../images/83800757-da6d0000-a65c-11ea-9060-bbc0ac30a3fe.png" alt="image" /></p>
|
||||
<p><strong>Sample input path</strong>: adl://gen1storage.azuredatalakestore.net/AdventureWorks/Raw/FactFinance/</p>
|
||||
<p><strong>Sample output path</strong>: adl://gen1storage.azuredatalakestore.net/AdventureWorks/ProcessedHDI/FactFinance/</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Loading to Azure synapse analytics (SQL DW) using stored procedure</strong></p>
|
||||
<p>Create a pipeline using Stored Procedure Activity to invoke a stored procedure in Azure SQL data warehouse.</p>
|
||||
<p><img src="../images/83453396-48bc8300-a40f-11ea-8c7d-886097bbc323.png" alt="image" /></p>
|
||||
<p><strong>Stored procedure Settings</strong>:</p>
|
||||
<p><img src="../images/83823216-3c8f2a80-a688-11ea-9a5e-c22097f05389.png" alt="image" /></p>
|
||||
</li>
|
||||
</ol>
|
||||
<h3 id="how-to-set-up-gen2-data-pipeline">
|
||||
How to set up Gen2 data pipeline
|
||||
<a class="anchor" href="#how-to-set-up-gen2-data-pipeline">#</a>
|
||||
</h3>
|
||||
<p>Create a master pipeline in ADF for Gen2 invoking all activities as listed below:</p>
|
||||
<p><img src="../images/83824400-3cdcf500-a68b-11ea-8622-1143e3691707.png" alt="image" /></p>
|
||||
<ol>
|
||||
<li>
|
||||
<p><strong>Raw data ingestion using ADB script</strong></p>
|
||||
<p>Create a pipeline for data ingestion process using ADB activity. Refer <a href="https://docs.microsoft.com/azure/data-factory/transform-data-using-databricks-notebook#create-a-pipeline">here</a> for more details.</p>
|
||||
<p><img src="../images/83466106-ebcec600-a42a-11ea-875a-120cb4e2a821.png" alt="image" /></p>
|
||||
<p>Mount path configured to Gen2 endpoint</p>
|
||||
<p><img src="../images/83802131-14d79c80-a65f-11ea-9dda-87240b9fec97.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Data processing using HDInsight</strong></p>
|
||||
<p>Create a pipeline for data processing using HDInsight activity. Refer <a href="https://docs.microsoft.com/azure/hdinsight/hdinsight-hadoop-create-linux-clusters-adf#create-a-pipeline">here</a> for more details.</p>
|
||||
<p><img src="../images/83466207-39e3c980-a42b-11ea-9ed6-d056b1c1cf0f.png" alt="image" /></p>
|
||||
<p>Mount path configured to Gen2 endpoint</p>
|
||||
<p><img src="../images/83802269-47819500-a65f-11ea-8800-ab4290641beb.png" alt="image" /></p>
|
||||
<p><strong>Sample input path</strong>: abfs://gen2storage@g2hdistorage.dfs.core.windows.net/AdventureWorks/Raw/FactInternetSales/</p>
|
||||
<p><strong>Sample output path</strong>: abfs://gen2storage@g2hdistorage.dfs.core.windows.net/AdventureWorks/ProcessedHDI/FactInternetSales/</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Loading to Azure synapse analytics (SQL DW) using stored procedure</strong></p>
|
||||
<p>Create a pipeline for loading the processed data to SQL DW using stored procedure activity.</p>
|
||||
<p><img src="../images/83466549-43216600-a42c-11ea-9306-e62ad0d6fc67.png" alt="image" /></p>
|
||||
<p><strong>Stored procedure Settings</strong>:</p>
|
||||
<p><img src="../images/83824637-e7edae80-a68b-11ea-887b-db853a4a8600.png" alt="image" /></p>
|
||||
<p><strong>Stored procedures created to load processed data to main tables</strong>:</p>
|
||||
<p><img src="../images/83895821-13ae7a00-a708-11ea-8566-1683a20579b2.png" alt="image" /></p>
|
||||
<p><strong>External Table structure in SQL DW</strong>:</p>
|
||||
<p><img src="../images/83895724-f37ebb00-a707-11ea-9292-4e80b45a0da8.png" alt="image" /></p>
|
||||
</li>
|
||||
</ol>
|
||||
<h2 id="steps-to-be-followed">
|
||||
Steps to be followed
|
||||
<a class="anchor" href="#steps-to-be-followed">#</a>
|
||||
</h2>
|
||||
<p>This section will talk about the approach and steps to move ahead with this pattern once the data pipelines are set up for both Gen1 and Gen2.</p>
|
||||
<h3 id="migrate-data-from-gen1-to-gen2">
|
||||
Migrate data from Gen1 to Gen2
|
||||
<a class="anchor" href="#migrate-data-from-gen1-to-gen2">#</a>
|
||||
</h3>
|
||||
<p>To migrate the existing data from Gen1 to Gen2, please refer to <a href="../lift-and-shift/">lift and shift</a> pattern.</p>
|
||||
<h3 id="data-ingestion-to-gen1-and-gen2">
|
||||
Data ingestion to Gen1 and Gen2
|
||||
<a class="anchor" href="#data-ingestion-to-gen1-and-gen2">#</a>
|
||||
</h3>
|
||||
<p>This step will ingest new data to both Gen1 and Gen2.</p>
|
||||
<ol>
|
||||
<li>
|
||||
<p>Create a pipeline in ADF to execute both data ingestion acitvity for Gen1 and Gen2.</p>
|
||||
<p><img src="../images/83543532-63473880-a4b1-11ea-8267-ee267f4845a9.png" alt="image" /></p>
|
||||
<p>Setting of the Base parameter:</p>
|
||||
<p><img src="../images/84072859-b70bc300-a984-11ea-870b-db59c59584ac.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Check the storage path at Gen1 and Gen2 end points. New data should be ingested simultaneously at both paths.</p>
|
||||
<p><img src="../images/84086549-fe05b280-a99c-11ea-8c89-f574d9ce9d2d.png" alt="image" /></p>
|
||||
</li>
|
||||
</ol>
|
||||
<h3 id="run-workloads-at-gen2">
|
||||
Run workloads at Gen2
|
||||
<a class="anchor" href="#run-workloads-at-gen2">#</a>
|
||||
</h3>
|
||||
<p>This step make sure that the workloads are run at Gen2 endpoint only.</p>
|
||||
<ol>
|
||||
<li>
|
||||
<p>Create a pipeline in ADF to execute the workloads for Gen2. Run the pipeline.</p>
|
||||
<p><img src="../images/83572596-b8e60a00-a4de-11ea-84ff-a4a178fa7575.png" alt="image" /></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Check the Gen2 storage path for the new files. The SQL DW should be loading with new processed data.</p>
|
||||
</li>
|
||||
</ol>
|
||||
<h3 id="cutover-from-gen1-to-gen2">
|
||||
Cutover from Gen1 to Gen2
|
||||
<a class="anchor" href="#cutover-from-gen1-to-gen2">#</a>
|
||||
</h3>
|
||||
<p>After you’re confident that your applications and workloads are stable on Gen2, you can begin using Gen2 to satisfy your business scenarios. Turn off any remaining pipelines that are running on Gen1 and decommission your Gen1 account.</p>
|
||||
<h2 id="references">
|
||||
References
|
||||
<a class="anchor" href="#references">#</a>
|
||||
</h2>
|
||||
<ul>
|
||||
<li><a href="https://docs.microsoft.com/azure/storage/blobs/data-lake-storage-migrate-gen1-to-gen2">Migrate Azure Data Lake Storage from Gen1 to Gen2 </a></li>
|
||||
</ul>
|
||||
</article>
|
||||
|
||||
|
||||
|
||||
<footer class="book-footer">
|
||||
|
||||
<div class="flex flex-wrap justify-between">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<hr />
|
||||
Azure Storage ©2021 <br />
|
||||
Visit the <a href="https://azure.microsoft.com/services/storage/">Azure Storage homepage</a> or read our <a href="https://docs.microsoft.com/azure/storage/">getting started guide</a> or the <a href="https://azure.microsoft.com/blog/topics/storage-backup-and-recovery/">Azure Storage Blog</a>. <br />
|
||||
Contact us: <a href="mailto:azurestoragefeedback@microsoft.com?subject=AzureStorage.com%20Feedback">azurestoragefeedback@microsoft.com</a>.<br />
|
||||
Generated on Mon, Feb 08 2021 23:18:57 UTC
|
||||
</footer>
|
||||
|
||||
|
||||
|
||||
<div class="book-comments">
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<label for="menu-control" class="hidden book-menu-overlay"></label>
|
||||
</div>
|
||||
|
||||
|
||||
<aside class="book-toc">
|
||||
<div class="book-toc-content">
|
||||
|
||||
|
||||
<nav id="TableOfContents">
|
||||
<ul>
|
||||
<li><a href="#overview">Overview</a></li>
|
||||
<li><a href="#prerequisites">Prerequisites</a></li>
|
||||
<li><a href="#data-pipeline-set-up-for-gen1-and-gen2">Data pipeline set up for Gen1 and Gen2</a>
|
||||
<ul>
|
||||
<li><a href="#creation-of-linked-service-for-gen1-and-gen2-in-adf">Creation of linked service for Gen1 and Gen2 in ADF</a></li>
|
||||
<li><a href="#how-to-set-up-gen1-data-pipeline">How to set up Gen1 data pipeline</a></li>
|
||||
<li><a href="#how-to-set-up-gen2-data-pipeline">How to set up Gen2 data pipeline</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#steps-to-be-followed">Steps to be followed</a>
|
||||
<ul>
|
||||
<li><a href="#migrate-data-from-gen1-to-gen2">Migrate data from Gen1 to Gen2</a></li>
|
||||
<li><a href="#data-ingestion-to-gen1-and-gen2">Data ingestion to Gen1 and Gen2</a></li>
|
||||
<li><a href="#run-workloads-at-gen2">Run workloads at Gen2</a></li>
|
||||
<li><a href="#cutover-from-gen1-to-gen2">Cutover from Gen1 to Gen2</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><a href="#references">References</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
</main>
|
||||
|
||||
<script src="https://code.jquery.com/jquery-3.5.1.slim.min.js" integrity="sha256-4+XzXVhsDmqanXGHaHvgh1gMQKX40OUvDEBTu8JcmNs=" crossorigin="anonymous"></script>
|
||||
|
||||
<script>
|
||||
$('a[href^="http://"], a[href^="https://"]').not('a[href*="'+location.hostname+'"]').attr('target','_blank');
|
||||
</script>
|
||||
|
||||
<script type="text/javascript">
|
||||
!function(T,l,y){var S=T.location,u="script",k="instrumentationKey",D="ingestionendpoint",C="disableExceptionTracking",E="ai.device.",I="toLowerCase",b="crossOrigin",w="POST",e="appInsightsSDK",t=y.name||"appInsights";(y.name||T[e])&&(T[e]=t);var n=T[t]||function(d){var g=!1,f=!1,m={initialize:!0,queue:[],sv:"4",version:2,config:d};function v(e,t){var n={},a="Browser";return n[E+"id"]=a[I](),n[E+"type"]=a,n["ai.operation.name"]=S&&S.pathname||"_unknown_",n["ai.internal.sdkVersion"]="javascript:snippet_"+(m.sv||m.version),{time:function(){var e=new Date;function t(e){var t=""+e;return 1===t.length&&(t="0"+t),t}return e.getUTCFullYear()+"-"+t(1+e.getUTCMonth())+"-"+t(e.getUTCDate())+"T"+t(e.getUTCHours())+":"+t(e.getUTCMinutes())+":"+t(e.getUTCSeconds())+"."+((e.getUTCMilliseconds()/1e3).toFixed(3)+"").slice(2,5)+"Z"}(),iKey:e,name:"Microsoft.ApplicationInsights."+e.replace(/-/g,"")+"."+t,sampleRate:100,tags:n,data:{baseData:{ver:2}}}}var h=d.url||y.src;if(h){function a(e){var t,n,a,i,r,o,s,c,p,l,u;g=!0,m.queue=[],f||(f=!0,t=h,s=function(){var e={},t=d.connectionString;if(t)for(var n=t.split(";"),a=0;a<n.length;a++){var i=n[a].split("=");2===i.length&&(e[i[0][I]()]=i[1])}if(!e[D]){var r=e.endpointsuffix,o=r?e.location:null;e[D]="https://"+(o?o+".":"")+"dc."+(r||"services.visualstudio.com")}return e}(),c=s[k]||d[k]||"",p=s[D],l=p?p+"/v2/track":config.endpointUrl,(u=[]).push((n="SDK LOAD Failure: Failed to load Application Insights SDK script (See stack for details)",a=t,i=l,(o=(r=v(c,"Exception")).data).baseType="ExceptionData",o.baseData.exceptions=[{typeName:"SDKLoadFailed",message:n.replace(/\./g,"-"),hasFullStack:!1,stack:n+"\nSnippet failed to load ["+a+"] -- Telemetry is disabled\nHelp Link: https://go.microsoft.com/fwlink/?linkid=2128109\nHost: "+(S&&S.pathname||"_unknown_")+"\nEndpoint: "+i,parsedStack:[]}],r)),u.push(function(e,t,n,a){var i=v(c,"Message"),r=i.data;r.baseType="MessageData";var o=r.baseData;return o.message='AI (Internal): 99 message:"'+("SDK LOAD Failure: Failed to load Application Insights SDK script (See stack for details) ("+n+")").replace(/\"/g,"")+'"',o.properties={endpoint:a},i}(0,0,t,l)),function(e,t){if(JSON){var n=T.fetch;if(n&&!y.useXhr)n(t,{method:w,body:JSON.stringify(e),mode:"cors"});else if(XMLHttpRequest){var a=new XMLHttpRequest;a.open(w,t),a.setRequestHeader("Content-type","application/json"),a.send(JSON.stringify(e))}}}(u,l))}function i(e,t){f||setTimeout(function(){!t&&m.core||a()},500)}var e=function(){var n=l.createElement(u);n.src=h;var e=y[b];return!e&&""!==e||"undefined"==n[b]||(n[b]=e),n.onload=i,n.onerror=a,n.onreadystatechange=function(e,t){"loaded"!==n.readyState&&"complete"!==n.readyState||i(0,t)},n}();y.ld<0?l.getElementsByTagName("head")[0].appendChild(e):setTimeout(function(){l.getElementsByTagName(u)[0].parentNode.appendChild(e)},y.ld||0)}try{m.cookie=l.cookie}catch(p){}function t(e){for(;e.length;)!function(t){m[t]=function(){var e=arguments;g||m.queue.push(function(){m[t].apply(m,e)})}}(e.pop())}var n="track",r="TrackPage",o="TrackEvent";t([n+"Event",n+"PageView",n+"Exception",n+"Trace",n+"DependencyData",n+"Metric",n+"PageViewPerformance","start"+r,"stop"+r,"start"+o,"stop"+o,"addTelemetryInitializer","setAuthenticatedUserContext","clearAuthenticatedUserContext","flush"]),m.SeverityLevel={Verbose:0,Information:1,Warning:2,Error:3,Critical:4};var s=(d.extensionConfig||{}).ApplicationInsightsAnalytics||{};if(!0!==d[C]&&!0!==s[C]){method="onerror",t(["_"+method]);var c=T[method];T[method]=function(e,t,n,a,i){var r=c&&c(e,t,n,a,i);return!0!==r&&m["_"+method]({message:e,url:t,lineNumber:n,columnNumber:a,error:i}),r},d.autoExceptionInstrumented=!0}return m}(y.cfg);(T[t]=n).queue&&0===n.queue.length&&n.trackPageView({})}(window,document,{
|
||||
src: "https://az416426.vo.msecnd.net/scripts/b/ai.2.min.js",
|
||||
|
||||
|
||||
|
||||
|
||||
cfg: {
|
||||
instrumentationKey: "0600b273-5440-42cc-9fe4-51924c721ce0"
|
||||
|
||||
}});
|
||||
</script>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/78856445-ad44fe00-79db-11ea-89e7-c4f89dd62701.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/78856445-ad44fe00-79db-11ea-89e7-c4f89dd62701.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 92 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/78946426-8a682780-7a77-11ea-973b-8f7cad667295.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/78946426-8a682780-7a77-11ea-973b-8f7cad667295.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 32 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/78946760-6fe27e00-7a78-11ea-915e-e716fb1d1c78.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/78946760-6fe27e00-7a78-11ea-915e-e716fb1d1c78.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 40 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/78953831-f1dda180-7a8e-11ea-82e9-07aa66fd2856.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/78953831-f1dda180-7a8e-11ea-82e9-07aa66fd2856.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 26 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/78954784-01121e80-7a92-11ea-8799-1b075e06b29d.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/78954784-01121e80-7a92-11ea-8799-1b075e06b29d.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 44 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/78966833-ad193100-7ab5-11ea-97b6-cf3ca372a451.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/78966833-ad193100-7ab5-11ea-97b6-cf3ca372a451.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 34 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79009824-49beeb80-7b15-11ea-8d14-ce444f7fd4b8.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79009824-49beeb80-7b15-11ea-8d14-ce444f7fd4b8.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 29 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79016042-dfad4300-7b22-11ea-97c2-274e533a37e7.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79016042-dfad4300-7b22-11ea-97c2-274e533a37e7.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 52 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79170712-0a0e3300-7da5-11ea-9268-1462751db77c.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79170712-0a0e3300-7da5-11ea-9268-1462751db77c.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 81 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79265180-90c91b80-7e4a-11ea-9000-0f86aa7c6ebb.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79265180-90c91b80-7e4a-11ea-9000-0f86aa7c6ebb.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 34 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79276321-a3e4e700-7e5c-11ea-9908-b013e2d1e12b.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79276321-a3e4e700-7e5c-11ea-9908-b013e2d1e12b.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 73 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79276405-cd057780-7e5c-11ea-9c31-95dfd26db5b9.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79276405-cd057780-7e5c-11ea-9c31-95dfd26db5b9.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 38 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79280406-21145a00-7e65-11ea-8950-bff27882c4de.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79280406-21145a00-7e65-11ea-8950-bff27882c4de.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 6.9 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79280526-66388c00-7e65-11ea-895e-915018092b67.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79280526-66388c00-7e65-11ea-895e-915018092b67.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 28 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79594064-3e2d7080-8091-11ea-872e-d69052da0ff7.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/79594064-3e2d7080-8091-11ea-872e-d69052da0ff7.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 42 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80261884-9b509580-8640-11ea-9b4e-b4e909ce76ca.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80261884-9b509580-8640-11ea-9b4e-b4e909ce76ca.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 43 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80262349-0189e800-8642-11ea-84a5-954f74bc7b2c.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80262349-0189e800-8642-11ea-84a5-954f74bc7b2c.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 39 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80262658-000cef80-8643-11ea-9007-462dc366e2d3.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80262658-000cef80-8643-11ea-9007-462dc366e2d3.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 59 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80263267-f2586980-8644-11ea-86b8-4b1714779948.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80263267-f2586980-8644-11ea-86b8-4b1714779948.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 43 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80268818-407a6680-865f-11ea-818f-3d4dca06ed00.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80268818-407a6680-865f-11ea-818f-3d4dca06ed00.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 15 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80269030-1f1a7a00-8661-11ea-925f-2beb8cca0d2a.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80269030-1f1a7a00-8661-11ea-925f-2beb8cca0d2a.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 18 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80269048-47a27400-8661-11ea-8aa6-cc65d19bc8ad.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80269048-47a27400-8661-11ea-8aa6-cc65d19bc8ad.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 2.0 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80269123-bd0e4480-8661-11ea-8c92-d8923aaaa324.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80269123-bd0e4480-8661-11ea-8c92-d8923aaaa324.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 30 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80270761-ca323000-866f-11ea-94bf-20e02133bb11.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80270761-ca323000-866f-11ea-94bf-20e02133bb11.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 2.4 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80332096-b406aa00-87fe-11ea-9ebc-4a5c4a757833.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80332096-b406aa00-87fe-11ea-9ebc-4a5c4a757833.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 44 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80333937-86bcfa80-8804-11ea-93ca-cefb660789f2.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80333937-86bcfa80-8804-11ea-93ca-cefb660789f2.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 6.4 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80334350-d18b4200-8805-11ea-881c-866858a255c4.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80334350-d18b4200-8805-11ea-881c-866858a255c4.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 6.1 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80396711-e6012600-8869-11ea-8161-cfbcc25c3170.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80396711-e6012600-8869-11ea-8161-cfbcc25c3170.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 132 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80404559-38e0da80-8876-11ea-9cc4-22314a46fedc.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80404559-38e0da80-8876-11ea-9cc4-22314a46fedc.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 146 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80407953-4ba9de00-887b-11ea-97a5-2baa8683943d.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80407953-4ba9de00-887b-11ea-97a5-2baa8683943d.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 27 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80413692-b7dd0f80-8884-11ea-82d0-9925706b4522.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80413692-b7dd0f80-8884-11ea-82d0-9925706b4522.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 117 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80416242-a8f85c00-8888-11ea-851c-d23f38749d56.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80416242-a8f85c00-8888-11ea-851c-d23f38749d56.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 48 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80425042-c1bc3e00-8897-11ea-96b8-891c6e739935.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80425042-c1bc3e00-8897-11ea-96b8-891c6e739935.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 40 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80544309-9b64d400-8965-11ea-9b28-a4e4daf05a3d.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80544309-9b64d400-8965-11ea-9b28-a4e4daf05a3d.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 12 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80546359-44153280-896a-11ea-9e12-bb85b6ceeafc.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80546359-44153280-896a-11ea-9e12-bb85b6ceeafc.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 33 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80547216-8c355480-896c-11ea-8adb-1a58d4e1be6c.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80547216-8c355480-896c-11ea-8adb-1a58d4e1be6c.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 77 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80765875-f418a600-8af8-11ea-9129-0791ccfcba12.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80765875-f418a600-8af8-11ea-9129-0791ccfcba12.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 70 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80843564-7799cc00-8bb9-11ea-89bb-b7009162f6a1.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80843564-7799cc00-8bb9-11ea-89bb-b7009162f6a1.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 72 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80934228-0e15eb00-8d7c-11ea-8a8c-1975e2c0c9ba.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/80934228-0e15eb00-8d7c-11ea-8a8c-1975e2c0c9ba.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 14 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81105450-c650aa00-8ec8-11ea-917c-619c6909b2c9.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81105450-c650aa00-8ec8-11ea-917c-619c6909b2c9.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 15 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81109171-8a204800-8ece-11ea-9ddc-a8c12007e32a.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81109171-8a204800-8ece-11ea-9ddc-a8c12007e32a.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 23 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81124187-ad0c2580-8ee9-11ea-9ca5-f1e2327dc2ab.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81124187-ad0c2580-8ee9-11ea-9ca5-f1e2327dc2ab.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 16 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81126936-b4cfc800-8ef1-11ea-86bc-8550b21a8aa3.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81126936-b4cfc800-8ef1-11ea-86bc-8550b21a8aa3.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 16 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81138345-929c7100-8f16-11ea-9e0f-3681d7c76cf1.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81138345-929c7100-8f16-11ea-9e0f-3681d7c76cf1.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 23 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81219887-4e9c8100-8f95-11ea-8b81-e32122547757.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81219887-4e9c8100-8f95-11ea-8b81-e32122547757.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 36 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81220219-cff41380-8f95-11ea-9295-4df62c7a9afe.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81220219-cff41380-8f95-11ea-9295-4df62c7a9afe.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 33 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81225979-f8ccd680-8f9e-11ea-9d93-42f8d4ad2c63.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81225979-f8ccd680-8f9e-11ea-9d93-42f8d4ad2c63.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 58 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81351734-dc519c80-9079-11ea-8a8e-669b65e17b49.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81351734-dc519c80-9079-11ea-8a8e-669b65e17b49.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 27 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81352652-ed9ba880-907b-11ea-9cf2-c1620b28455a.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/81352652-ed9ba880-907b-11ea-9cf2-c1620b28455a.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 80 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83429980-c2417a80-a3e9-11ea-9ab6-4d08b02b51b1.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83429980-c2417a80-a3e9-11ea-9ab6-4d08b02b51b1.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 17 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83435632-6b3fa380-a3f1-11ea-8639-dba1e217e044.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83435632-6b3fa380-a3f1-11ea-8639-dba1e217e044.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 24 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83448158-63d6c500-a406-11ea-8a29-a1cdd514509c.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83448158-63d6c500-a406-11ea-8a29-a1cdd514509c.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 32 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83450714-a6020580-a40a-11ea-8c99-55c2c9a96104.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83450714-a6020580-a40a-11ea-8c99-55c2c9a96104.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 37 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83453396-48bc8300-a40f-11ea-8c7d-886097bbc323.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83453396-48bc8300-a40f-11ea-8c7d-886097bbc323.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 32 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83466106-ebcec600-a42a-11ea-875a-120cb4e2a821.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83466106-ebcec600-a42a-11ea-875a-120cb4e2a821.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 31 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83466207-39e3c980-a42b-11ea-9ed6-d056b1c1cf0f.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83466207-39e3c980-a42b-11ea-9ed6-d056b1c1cf0f.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 35 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83466549-43216600-a42c-11ea-9306-e62ad0d6fc67.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83466549-43216600-a42c-11ea-9306-e62ad0d6fc67.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 34 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83468627-356edf00-a432-11ea-9375-0594ab25b975.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83468627-356edf00-a432-11ea-9375-0594ab25b975.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 37 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83469679-335a4f80-a435-11ea-91c3-2d844be17cbe.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83469679-335a4f80-a435-11ea-91c3-2d844be17cbe.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 3.4 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83479945-d79cc000-a44e-11ea-8940-3884606cec7b.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83479945-d79cc000-a44e-11ea-8940-3884606cec7b.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 3.6 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83543532-63473880-a4b1-11ea-8267-ee267f4845a9.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83543532-63473880-a4b1-11ea-8267-ee267f4845a9.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 11 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83554216-4adf1a00-a4c1-11ea-9ea4-ae9284e678c0.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83554216-4adf1a00-a4c1-11ea-9ea4-ae9284e678c0.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 39 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83555204-c8eff080-a4c2-11ea-8162-a8f86b5e9e9e.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83555204-c8eff080-a4c2-11ea-8162-a8f86b5e9e9e.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 38 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83555444-26843d00-a4c3-11ea-9fac-5bd0760aca0b.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83555444-26843d00-a4c3-11ea-9fac-5bd0760aca0b.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 19 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83555536-44ea3880-a4c3-11ea-90d0-ccae337fb531.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83555536-44ea3880-a4c3-11ea-90d0-ccae337fb531.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 232 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83563507-16bf2580-a4d0-11ea-9707-ae659f87eb3d.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83563507-16bf2580-a4d0-11ea-9707-ae659f87eb3d.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 16 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83572596-b8e60a00-a4de-11ea-84ff-a4a178fa7575.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83572596-b8e60a00-a4de-11ea-84ff-a4a178fa7575.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 7.8 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83672012-74b04380-a58a-11ea-89b6-54564aeb52f5.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83672012-74b04380-a58a-11ea-89b6-54564aeb52f5.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 24 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83672806-b8f01380-a58b-11ea-8c16-ae0c662d7de6.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83672806-b8f01380-a58b-11ea-8c16-ae0c662d7de6.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 24 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83690089-eeedc180-a5a4-11ea-8a57-28a22822a595.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83690089-eeedc180-a5a4-11ea-8a57-28a22822a595.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 23 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83693292-ac2ee800-a5aa-11ea-878e-e8f6d72daf72.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83693292-ac2ee800-a5aa-11ea-878e-e8f6d72daf72.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 45 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83800138-cecd0980-a65b-11ea-93de-17b9fc016e90.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83800138-cecd0980-a65b-11ea-93de-17b9fc016e90.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 13 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83800757-da6d0000-a65c-11ea-9060-bbc0ac30a3fe.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83800757-da6d0000-a65c-11ea-9060-bbc0ac30a3fe.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 15 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83802131-14d79c80-a65f-11ea-9dda-87240b9fec97.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83802131-14d79c80-a65f-11ea-9dda-87240b9fec97.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 13 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83802269-47819500-a65f-11ea-8800-ab4290641beb.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83802269-47819500-a65f-11ea-8800-ab4290641beb.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 14 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83807252-9fbc9500-a667-11ea-9f0c-8deeefa742e7.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83807252-9fbc9500-a667-11ea-9f0c-8deeefa742e7.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 33 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83823216-3c8f2a80-a688-11ea-9a5e-c22097f05389.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83823216-3c8f2a80-a688-11ea-9a5e-c22097f05389.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 18 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83823632-48c7b780-a689-11ea-903f-068b26ca3741.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83823632-48c7b780-a689-11ea-903f-068b26ca3741.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 12 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83824400-3cdcf500-a68b-11ea-8622-1143e3691707.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83824400-3cdcf500-a68b-11ea-8622-1143e3691707.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 13 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83824637-e7edae80-a68b-11ea-887b-db853a4a8600.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83824637-e7edae80-a68b-11ea-887b-db853a4a8600.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 19 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83895724-f37ebb00-a707-11ea-9292-4e80b45a0da8.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83895724-f37ebb00-a707-11ea-9292-4e80b45a0da8.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 12 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83895821-13ae7a00-a708-11ea-8566-1683a20579b2.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/83895821-13ae7a00-a708-11ea-8566-1683a20579b2.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 7.9 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/84070157-875abc00-a980-11ea-9d27-9764b577e2d0.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/84070157-875abc00-a980-11ea-9d27-9764b577e2d0.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 4.2 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/84072859-b70bc300-a984-11ea-870b-db59c59584ac.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/84072859-b70bc300-a984-11ea-870b-db59c59584ac.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 17 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/84082011-eece3700-a993-11ea-8ba0-f4efab65c0e9.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/84082011-eece3700-a993-11ea-8ba0-f4efab65c0e9.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 19 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/84082177-42408500-a994-11ea-84ba-d575ba1e3611.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/84082177-42408500-a994-11ea-84ba-d575ba1e3611.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 19 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/84086549-fe05b280-a99c-11ea-8c89-f574d9ce9d2d.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/84086549-fe05b280-a99c-11ea-8c89-f574d9ce9d2d.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 34 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/95709479-5830c000-0c7c-11eb-8111-8c2fc592f168.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/95709479-5830c000-0c7c-11eb-8111-8c2fc592f168.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 50 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/95709529-71d20780-0c7c-11eb-803a-b6a63bdd464e.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/95709529-71d20780-0c7c-11eb-803a-b6a63bdd464e.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 26 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/95709586-8910f500-0c7c-11eb-9f13-c8dea1928a82.png
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/images/95709586-8910f500-0c7c-11eb-9f13-c8dea1928a82.png
Normal file
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 29 KiB |
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/incremental/incremental.zip
Normal file
Двоичные данные
docs/docs/analytics/adls-gen1-to-gen2-migration/incremental/incremental.zip
Normal file
Двоичный файл не отображается.
Некоторые файлы не были показаны из-за слишком большого количества измененных файлов Показать больше
Загрузка…
Ссылка в новой задаче