ADXIoTAnalytics/assets/readme.md

# Assets

* Eval: https://aka.ms/adx.iot.eval (5 Qs Max). Pls, help us improve this workshop.

## Slides
1. [Download](https://github.com/Azure/ADXIoTAnalytics/raw/main/assets/Full%20deck%20-%20ADX%20for%20IoT%20Analytics.pdf) PDF of **Full deck - ADX for IoT Analytics.pdf**

# Resources 

1. Documentation: https://aka.ms/adx.docs
2. Reference Architecture: https://aka.ms/adx.iotrefarch
3. Tech Blogs: https://aka.ms/adx.techblogs
4. PG Ask me Anything: https://aka.ms/ama/azuredataexplorer
5. Free personal cluster: https://aka.ms/adx.free 
6. Cost Estimator: https://aka.ms/adx.cost or https://aka.ms/adx.cost.old


## Agenda
![image](https://user-images.githubusercontent.com/4984616/163595941-74cfd3ce-5ac7-4529-a998-3230a0adf53e.png)

## Module 1 - Intro
![image](https://user-images.githubusercontent.com/4984616/163595966-85ec03f7-2736-464a-ab35-3273b7e7745c.png)

* [How big is a Petabyte](https://www.bing.com/images/search?view=detailV2&ccid=V7WQ9yZ2&id=2511310F4A21ACFBB2D2E8048131F6CAA4B6C387&thid=OIP.V7WQ9yZ2UwCrSZwj1uZdIAHaPj&mediaurl=https%3a%2f%2finfo.cobaltiron.com%2fhs-fs%2fhubfs%2fhow-big-is-a-petabyte-graphic.jpg%3fwidth%3d600%26name%3dhow-big-is-a-petabyte-graphic.jpg&cdnurl=https%3a%2f%2fth.bing.com%2fth%2fid%2fR.57b590f726765300ab499c23d6e65d20%3frik%3dh8O2pMr2MYEE6A%26pid%3dImgRaw%26r%3d0&exph=1260&expw=600&q=how+big+is+a+petabyte&simid=608049176200178479&FORM=IRPRST&ck=1A985FD16C50CC253256647977035BBD&selectedIndex=0&ajaxhist=0&ajaxserp=0)
* https://aka.ms/adx.try
* [Proven Technology](https://aka.ms/adx.kustomers) 
* [Scan operator](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/the-new-scan-operator-process-mining-in-azure-data-explorer/ba-p/2378795)
* Visualizations [Cross-filter](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/adx-dashboards-november-2021-updates/ba-p/2956518?lightbox-message-images-2956518=326461i0206BB959AA3CBAB)
* Provision [lab](https://aka.ms/adx.lab) for self-study
* [KQL cheat-sheet](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/azure-data-explorer-kql-cheat-sheets/ba-p/1057404)
* [Diff(s)](https://docs.microsoft.com/azure/synapse-analytics/data-explorer/data-explorer-compare) between ADX & Synapse DE Pools


## Module 1.2 - Overview
![image](https://user-images.githubusercontent.com/4984616/163596990-f15ead24-3992-4a66-8d1c-f28aaa12e5d7.png)

* [Ingestion Patterns](https://docs.microsoft.com/azure/data-explorer/ingest-data-overview#batching-vs-streaming-ingestion)


## Module 1.3 - Architecture
![image](https://user-images.githubusercontent.com/4984616/163597294-5279e840-4a5f-4ea2-92b7-3a2376fda9a8.png)


## Module 1.4 - Ingestion
![image](https://user-images.githubusercontent.com/4984616/163597332-dc65f90f-d06c-4cb3-b0b3-f316c6142d7f.png)

* [One-click UI](https://docs.microsoft.com/azure/data-explorer/ingest-data-one-click)
* [Free personal cluster](https://aka.ms/adx.free)
* `.ingest` [docs](https://docs.microsoft.com/azure/data-explorer/kusto/management/data-ingestion/ingest-from-storage)


## Module 2 - ADX + IoT
![image](https://user-images.githubusercontent.com/4984616/163597618-8d1db20d-21b8-43ae-8551-f2294273a17c.png)

* [Continuous export](https://docs.microsoft.com/azure/data-explorer/kusto/management/data-export/continuous-data-export)
* [Dashboards video](https://youtu.be/n5wMgkQzbWw?t=640)
* [Python video](https://youtu.be/n5wMgkQzbWw?t=801)
* [Customers video](https://youtu.be/n5wMgkQzbWw?t=917)


## Module 3 - HOL
![image](https://user-images.githubusercontent.com/4984616/163598072-64a98c05-05ed-49b8-8ff0-45be80a110d5.png)

* [IoT Central](https://iotcentralpm10774domain.azureiotcentral.com/devices)
* [ADT](https://digitaltwinpm10774.api.eus.digitaltwins.azure.net)


## Module 4 - ADT
![image](https://user-images.githubusercontent.com/4984616/163598234-60375948-015f-48bd-bd1e-4de87164917d.png)

* [DTDLs](https://github.com/Azure/ADXIoTAnalytics/tree/main/dtconfig)
* https://explorer.digitaltwins.azure.net
- https://explorer.digitaltwins.azure.net/3DScenes/demo


## Module 5 - KQL
![image](https://user-images.githubusercontent.com/4984616/163598419-51586355-1167-4857-9a12-f8dcd25b86e6.png)

* Pluralsight > **Redeem**
* https://aka.ms/adx.youtube > **Query** playlist
* MS Learn: [Write your first query](https://docs.microsoft.com/en-us/learn/modules/write-first-query-kusto-query-language/)
* [Application query statements](https://docs.microsoft.com/azure/data-explorer/kusto/query/statements?pivots=azuredataexplorer#application-query-statements)
* [KQL Cheat-sheet](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/azure-data-explorer-kql-cheat-sheets/ba-p/1057404)
* [arg_max()](https://docs.microsoft.com/azure/data-explorer/kusto/query/arg-max-aggfunction)
* [evaluate](https://docs.microsoft.com/azure/data-explorer/kusto/query/evaluateoperator) plugin operator
* [series_fill_linea()](https://docs.microsoft.com/azure/data-explorer/kusto/query/series-fill-linearfunction)
* [series_decompose_anomalies()](https://docs.microsoft.com/azure/data-explorer/kusto/query/series-decompose-anomaliesfunction)
* [mv-expand](https://docs.microsoft.com/azure/data-explorer/kusto/query/mvexpandoperator) operator
* [materialized view](https://docs.microsoft.com/azure/data-explorer/kusto/management/materialized-views/materialized-view-create#backfill-a-materialized-view) performance tips
* [lookup](https://docs.microsoft.com/azure/data-explorer/kusto/query/lookupoperator) operator
* [external tables](https://docs.microsoft.com/azure/data-explorer/kusto/query/schema-entities/externaltables), **Note:** for parquet format use [nativeParquetWriter](https://docs.microsoft.com/azure/data-explorer/kusto/management/data-export/export-data-to-an-external-table)
* [KQL Best Practices](https://aka.ms/adx/query.bp)


## Module 6 - ML & TS
![image](https://user-images.githubusercontent.com/4984616/163603772-4118557a-4b55-4162-a608-523c207e7665.png)

* series_fit_line()
* series_fit_2lines()
* [series_periods_detect()](https://docs.microsoft.com/azure/data-explorer/kusto/query/series-periods-detectfunction)
* [series_periods_validate()](https://docs.microsoft.com/azure/data-explorer/kusto/query/series-periods-validatefunction)
* [series_fill_linear()](https://docs.microsoft.com/azure/data-explorer/kusto/query/series-fill-linearfunction)
* [series_decompose_forecast()](https://docs.microsoft.com/azure/data-explorer/kusto/query/series-decompose-forecastfunction)
* [series_decompose()](https://docs.microsoft.com/azure/data-explorer/kusto/query/series-decomposefunction)
* [autocluster()](https://docs.microsoft.com/azure/data-explorer/kusto/query/autoclusterplugin)
* [basket()](https://docs.microsoft.com/azure/data-explorer/kusto/query/basketplugin)
* [diffpatterns()](https://docs.microsoft.com/azure/data-explorer/kusto/query/diffpatternsplugin)
* [diffpatterns_text()](https://docs.microsoft.com/azure/data-explorer/kusto/query/diffpatterns-textplugin)
* Python plugin

![image](https://user-images.githubusercontent.com/4984616/163604217-762c65e4-e7f8-4d31-89a0-df566f807d06.png)
* R plugin

![image](https://user-images.githubusercontent.com/4984616/163604313-244b958f-1d28-46ba-b330-ea1537eee760.png)
* externaldata()
* external_artifacts


## Module 7 - Visuals
![image](https://user-images.githubusercontent.com/4984616/163604498-096f834a-71fd-4eaf-88ca-079a5df7ef5a.png)

* [render](https://docs.microsoft.com/azure/data-explorer/kusto/query/renderoperator?pivots=azuredataexplorer) operator
* [Power BI Tutorial](https://docs.microsoft.com/azure/data-explorer/visualize-power-bi)
* [Excel](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/direct-query-from-excel-to-azure-data-explorer-aka-kusto/ba-p/2973596)
* [Excel connector](https://docs.microsoft.com/azure/data-explorer/excel-connector)


## Module 8 - Ops & Mgmt
![image](https://user-images.githubusercontent.com/4984616/163607687-d6739d52-89c4-47c1-888a-23c346e0dd97.png)

* [soft-delete](https://docs.microsoft.com/azure/data-explorer/kusto/concepts/data-soft-delete)  via `.delete` or [retention-policy](https://docs.microsoft.com/azure/data-explorer/delete-data#delete-data-using-a-retention-policy)
* Handle dups, using [any()](https://docs.microsoft.com/azure/data-explorer/kusto/query/take-any-aggfunction)/[arg_min()](https://docs.microsoft.com/azure/data-explorer/kusto/query/arg-min-aggfunction)/[arg_max()](https://docs.microsoft.com/azure/data-explorer/kusto/query/arg-max-aggfunction) agg functions (see example #4 in [materialized view create command](https://docs.microsoft.com/azure/data-explorer/kusto/management/materialized-views/materialized-view-create)).


## Module 9 - Adv
![image](https://user-images.githubusercontent.com/4984616/163608233-29e0c764-b3a4-49d8-9265-571dbed86e0e.png)

* [All about update-policies](https://y0nil.github.io/kusto.blog/blog-posts/update-policies.html)
* hotcache windows
```
set query_datascope="hotcache"; MyTable | limit 10

.alter cluster policy caching hot = 30d

.alter cluster policy caching
    hot = 30d, 
    hot_window = datetime(2021-01-01) .. datetime(2021-02-01)
```
* [RLS & Mask columns](https://docs.microsoft.com/azure/data-explorer/kusto/management/rowlevelsecuritypolicy)
* [Python client library](https://docs.microsoft.com/azure/data-explorer/kusto/api/python/kusto-python-client-library)
* Python ingest test [sample.py](https://github.com/Azure/azure-kusto-python/blob/master/azure-kusto-ingest/tests/sample.py)
* Python install [Kqlmagic](https://docs.microsoft.com/sql/azure-data-studio/notebooks/notebooks-kqlmagic)