CCO Dashboard 4.0 Initial release

README.md

@@ -5,22 +5,49 @@
 
 - [Overview](README.md#overview)
 - [List of resources](README.md#List-of-resources)
-- [Current Dashboard Pages](README.md#Current-Dashboard-Pages)
+- [CCO Dashboard report pages](README.md#CCO-Dashboard-report-Pages)
+- [CCO Dashboard AKS add-on report pages](README.md#CCO-Dashboard-AKS-add-on)
+- [Call for Contribution](README.md#Call-for-Contribution)
 -------------------------------
 
 
 ## Overview
 The Continuous Cloud Optimziation Power BI Dashboard (referred as CCO Dashboard moving forward) is a Power BI Dashboard developed using Power Query M queries that pulls information directly from diferent Azure and Graph REST APIs. It presents the information in a simpliflied format to track potential recommendations from Azure Advisor or Azure Security Center allowing you to filter by subscriptions, resources groups, tags or particular resources.
 
+![OverviewImage](/install/images/OverviewImage.png)
+
 ## List of resources
 This project includes the following resources:
 
-1. **`install folder`:** This folder includes all the files required to successfully deploy the Dashboard in your environment. The deliveryGuide.md file contains a detailed guidance to install and setup your dashboard including the requirements, what REST APIs are in use, what resource providers needs to be enabled or what tabs are included as part of the default Dashboard
-2. **`queries folder`:**
-3. **`docs/assets/pictures`:**
-4. **`Secure the subscription`:**
+1. **install folder**: Includes all the files required to successfully deploy the Dashboard in your environment. The [Deployment Guide](/install/DeploymentGuide.md) file contains a detailed guidance to install and setup your dashboard including the requirements, what REST APIs are in use, what resource providers needs to be enabled or what tabs are included as part of the default Dashboard. The [Troubleshooting Guide](/install/TroubleshootingGuide.md) file contains guidance to solve potential issues that you might encounter during the Dashboard deployment. Errors like Power BI regional settings, or Privacy levels will be documented on this document.
+2. **queries folder**: Includes the M queries used in the Dashboard to pull data from Azure and Graph REST APIs. This content is for reference purposes to facilitate the Data Model comprehension and to enable contributors to expand the Dashboard capabilities. 
+3. **docs/assets/pictures folder**: Contains all the images that the Dashboard will use when loading data from Azure. The content of this folder will be dynamic and we will update the repository regularly. Make sure the computer running the Dashboard that has internet access also have access to this URL https://azure.github.io/ccodashboard/assets/pictures
+4. **dashboards folder**: This parent folder contains sub folders with different versions of the CCO Dashboard depending on the workloads you want to get report from. We expect to see more versions in the future from community contributions.
+    - ***CCODashboard folder*** has the more generic version of the Dashboard that includes information from Azure Advisor, Azure Security Center, Azure Networking REST APIs, Azure Compute REST APIs and Graph
+    - ***CCODashboard folder*** has the add-on report to monitor Azure Kubernetes Services
 
+## CCO Dashboard report pages
+The version 4.0 of the CCO Power BI Dashboard includes 7 report pages. You will be able to navigate, filter and report the following information:
+- Page 1: Overview
+- Page 2: Azure Advisor Recommendations
+- Page 3: Azure Security Center Task recommendations
+- Page 4: Azure Security Center Alerts
+- Page 5: Azure Networking information
+- Page 6: Azure Compute information
+- Page 7: Azure RBAC permissions
+  
+You can find more details about each page information on the [Deployment Guide](/install/DeploymentGuide.md) file
 
+## CCO Dashboard AKS add-on report pages
+
+The version 4.0 of the CCO Power BI Dashboard AKS add-on includes the following information
+
+- Azure Kubernetes Clusters
+- Azure Container Images running on AKS Clusters
+- Azure Container Registries
+- Nodes, Pods, Containers status from Azure Log Analytics
+
+## Call for contribution
 This project welcomes contributions and suggestions.  Most contributions require you to agree to a
 Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
 the rights to use your contribution. For details, visit https://cla.microsoft.com.

install/DeploymentGuide.md

@@ -26,25 +26,28 @@ The Continuous Optimization Power BI Dashboard pulls the information from severa
 <br><br>
 </div>
 
-| API Name| Dashboard API Version | Azure last API version | Using last version|
+| API Name| Dashboard API Version | Last API version | Using latest version|
 | --- | :---: | :---: |:---: |
 | [Azure Advisor](https://docs.microsoft.com/en-us/rest/api/advisor/) | 2017-03-31|2017-04-19|:x:|
-| [Azure Security Center](https://msdn.microsoft.com/en/US/library/mt704034(Azure.100).aspx)  |2015-06-01-preview |2015-06-01-preview|:heavy_check_mark:|
+| [Azure Security Center Alerts](https://msdn.microsoft.com/en/US/library/mt704034(Azure.100).aspx)  |2015-06-01-preview |2015-06-01-preview|:heavy_check_mark:|
+| [Azure Security Center Tasks](https://msdn.microsoft.com/en/US/library/mt704034(Azure.100).aspx)  |2015-06-01-preview |2015-06-01-preview|:heavy_check_mark:|
 | [Azure Kubernetes Service](https://docs.microsoft.com/en-us/rest/api/aks) | 2018-03-31|2018-03-31|:heavy_check_mark:|
 | [Azure Compute](https://docs.microsoft.com/en-us/rest/api/compute) | 2017-12-01|2018-10-01|:x:|
 | [Azure Virtual Networks]( https://docs.microsoft.com/en-us/rest/api/virtual-network) | 2017-09-01|2018-08-01|:x:|
 | [Azure Network Interfaces](https://docs.microsoft.com/en-us/rest/api/virtualnetwork/networkinterfaces) |2017-10-01 |2018-08-01|:x:|
-| [Resource Groups](https://docs.microsoft.com/en-us/rest/api/resources/resourcegroups)  |2017-05-10 |2018-08-01|:x:|
-| [Azure Resources](https://docs.microsoft.com/en-us/rest/api/resources/resources)  |2017-05-10 |2018-08-01|:x:|
+| [Resource Groups](https://docs.microsoft.com/en-us/rest/api/resources/resourcegroups)  |2017-05-10 |2018-09-01|:x:|
+| [Azure Resources](https://docs.microsoft.com/en-us/rest/api/resources/resources)  |2017-05-10 |2018-09-01|:x:|
 | [Azure Subscriptions](https://docs.microsoft.com/en-us/rest/api/resources/subscriptions)  |2016-06-01 |2018-08-01|:x:|
-| [Azure Locations](https://docs.microsoft.com/en-us/rest/api/resources/subscriptions/listlocations)  |2016-06-01 |2018-08-01|:x:|
+| [Azure Locations](https://docs.microsoft.com/en-us/rest/api/resources/subscriptions/listlocations)  |2016-06-01 |2018-09-01|:x:|
 | [Azure Role Assignments](https://docs.microsoft.com/en-us/rest/api/authorization/roleassignments) |2015-07-01 |2018-09-01-preview|:x:|
+| [Azure Role Definitions](https://docs.microsoft.com/en-us/rest/api/authorization/roleassignments) |2018-01-01-preview |2018-01-01-preview|:heavy_check_mark:|
 | [Azure Container Registry](https://docs.microsoft.com/en-us/rest/api/containerregistry/)  | 2017-10-01|2017-10-01|:heavy_check_mark:|
 | <span style="color:#0088cc">Log Analytics Rest API </span> ([1](https://docs.microsoft.com/en-us/rest/api/loganalytics/), [2](https://dev.loganalytics.io/))  |v1 |v1|:heavy_check_mark:|
 | [Azure Active Directory Graph API](https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-graph-api) | 1.6|1.6 |:heavy_check_mark:|
 
 <div style="text-align: justify">
 
+
 ## Resource Providers requirements
 
 Although some of the Resource Providers might be enabled by default, you need to make sure that at least the **Microsoft.Advisor** and the **Microsoft.Security** resource providers are registered across all the  subscriptions that you plan analyze using the Dashboard. Otherwise, the Azure Advisor and the Azure Security Center.<br><br>
@@ -151,21 +154,22 @@ If the permissions and Credentials are properly flushed it should ask you for cr
   
 ![credentials7](/install/images/Credentials7.png)
 
-# Tabs
-## CCO Dashboard overview tab
-In this tab, you will be able to identify the top 5 of recommendations that Azure Advisor and Azure Security Center has identified. You can also locate all the deployed resources in a map.
+# Report Pages
+## CCO Dashboard overview page
+In this page, you will be able to identify the top 5 of recommendations that Azure Advisor and Azure Security Center has identified. You can also locate all the deployed resources in a map.
 It’s important to mention that this tab it’s just only to make a quick view. All the recommendations will be available with more details in the following tabs
 
 ![overview](/install/images/Overview.png)
 
-## Azure Advisor Recommendations Dashboard tab
-In second report tab, you will be able to identify the total amount of recommendations that Azure Advisor has identified, to what resources apply each recommendation and to what subscription is applicable.
+## Azure Advisor Recommendations Dashboard page
+In second report page, you will be able to identify the total amount of recommendations that Azure Advisor has identified, to what resources apply each recommendation and to what subscription is applicable.
 
 You can filter the information by:
 - Subscription
 - Resource Group
 - Resource type
 - Recommendation type
+- Tags
 
 It will also give a high-level overview of what subscriptions require more attention and has more recommendations to snooze or implement.
 
@@ -173,14 +177,15 @@ If you press on an impacted resource you will see a quick description, the solut
 
 ![advisor](/install/images/Advisor.png)
 
-## Azure Security Center Recommendations Dashboard tab
-In this tab, you will be able to identify the total amount of recommendations that Azure Security Center has detected, to what resources apply each recommendation and to what subscription is impacting.
+## Azure Security Center Recommendations Dashboard page
+In this page, you will be able to identify the total amount of recommendations that Azure Security Center has detected, to what resources apply each recommendation and to what subscription is impacting.
 
 You can filter the information by:
 - Subscription
 - Resource Group
 - Task State
 - Resource Type
+- Tags
   
 It will also give a high-level overview of what subscriptions require more attention and has more recommendations to snooze or implement.
 
@@ -193,7 +198,8 @@ You can filter the information by:
 - Data range
 - Subscription
 - Attack type
-  
+- Tags
+    
 ![security Center alerts](/install/images/SecurityCenterAlerts.png)
 
 ## Azure VNETs and Subnets Recommendations Dashboard tab

install/TroubleshootingGuide.md

@@ -0,0 +1,92 @@
+# CCO Dashboard Troubleshooting guide
+<div style="text-align: justify">
+
+## Introduction
+The Continuous Cloud Optimization Power BI Dashboard development started more than 1 year ago. During all this time the project team and some Microsoft Services folks have been piloting and testing different versions of the Dashboards until the latest version published in here (v4.0). We have been able to identify and fix several bugs or product limitations and we would like to encourage you to read this document before running the Dashboard.
+
+- ## List of known issues or limitations
+    - [Which Power BI Desktop version I should use?](TroubleshootingGuide.md#which-power-bi-desktop-version-i-should-use-microsoft-store-or-web-download)  (Microsoft Store or  Web Download)
+    - [Power BI Regional Settings](TroubleshootingGuide.md#power-bi-regional-settings-maps-visualizations-incorrectly-locate-resources-or-vnet-peerings)
+    (Maps Visualizations incorrectly locate resources or VNET peerings)
+    - [Graph REST API credentials error](TroubleshootingGuide.md#graph-rest-api-credentials-error)
+
+    - [Privacy Levels across Data Sources not configured properly](TroubleshootingGuide.md#privacy-levels-across-data-sources-not-configured-properly)
+
+    - [RBAC information is empty or blank](TroubleshootingGuide.md#rbac-information-is-empty-or-blank)
+
+    - [Log Analytics REST API timeout (CCO Dashboard AKS add on only)](TroubleshootingGuide.md#log-analytics-rest-api-timeout-cco-dashboard-aks-add-on-only)
+
+    - [Data Model Relationships missing](TroubleshootingGuide.md#data-model-relationships-missing)
+
+We will keep updating this list of known issues as soon as we get more feedback from the community....
+   
+----------------------------------
+#### Which Power BI Desktop version I should use? (Microsoft Store or Web Download)
+
+Based on our experience we highly recommend to use the Power BI Desktop version from the Microsoft Store to get automatic updates. The following article explains the main difference between both options. https://docs.microsoft.com/en-us/power-bi/desktop-get-the-desktop <br>
+Make sure that you don't have both versions installed on the computer where you plan to run the CCO Dashboard.
+
+#### Power BI Regional Settings (Maps Visualizations incorrectly locate resources or VNET peerings)
+
+It might happen then when you run the Dashboard using different regional settings some coordinates are not calculated properly. The CCO Dashboard development has been based on English US regional settings. Make sure that you set the Regional Settings to use English (United States) on the application Language on both Global and Current File options. If the current file has a different configuration you will need to change to English US, export the file as template again, and then open it from your computer
+
+<p align="center">
+  <img src="/install/images/locale_options_powerBI.PNG">
+</p>
+
+#### Graph REST API credentials error
+
+During the first run of the CCO Dashboard template you should be prompted to enter the credentials for both the Azure Management REST API and the GRAPH REST API. You might get the error message from below if you incorrectly enter your credentials. Also, in some cases, during the first execuction Power BI will not ask for credentials because they are already cached by some other Power BI Dashboard execution accessing the same APIs. 
+
+![graph apil](/install/images/WrongTenantNameError.png)
+
+![graph apil](/install/images/problem_graph_api.png)
+
+If that happens you will need to manually set the proper credentials for the GRAPH REST API Data Source.
+To do this you must follow this steps:
+
+- Go to **File**.
+- Click on **Options and settings**.
+- Click on **Data source settings**.
+- In in **Current file/Global permissions** select https://graph.windows.net and click on **Edit permissions**.
+- Click on **Edit** and enter your credentials.
+
+<p align="center">
+  <img src="/install/images/Credentials5.png">
+</p>
+
+#### Privacy Levels across Data Sources not configured properly
+
+Another credentials issue we identified during our pilots is that in some cases the end user didn't setup the Privacy levels correctly getting the following error
+
+<p align="center">
+  <img src="/install/images/WrongPrivacyLevelError.png">
+</p>
+
+
+This is an example of a wrong Privacy Levels configuration across Data Sources. Makes sure that you set both Data Sources to Organizational
+
+<p align="center">
+  <img src="/install/images/WrongPrivacyLevelConfig.png">
+</p>
+
+
+#### RBAC information is empty or blank
+
+Everytime you run the Dashboard from the .pbit template you will be asked to enter the Tenant parameter. This parameter is critical to properly get your RBAC information. If this parameter is entered incorrectly but the tenant exists it will load the Dashboard information with blank information on the RBAC page or the Subscriptions Owners visualization on the Overview page
+
+![Wrong Tenant Name](/install/images/RBACwrongTenantName.PNG)
+
+#### Log Analytics REST API timeout (CCO Dashboard AKS add on only)
+
+Depending on the number of records we have in log analytics, we can obtain a timeout during the refresh.
+
+The solution is to wait a few minutes and launch a new refresh.
+
+#### Data Model Relationships missing
+
+In some ocasions when having both Power BI Desktop versions installed on the computer we have seen the Data Relationship model to dissapear. If that happens you can use the following reference to manually reconfigure the relationship between tables. It should also help you to understand how we defined the Data Model in case you want to expand the default content
+
+![relationship model](/install/images/RelationshipsModel.PNG)
+
+

install/scripts/CCOCheckAzureApiLastVersion.ps1

@@ -0,0 +1,43 @@
+Login-AzureRmAccount
+Get-AzureRmSubscription | Out-GridView -PassThru
+
+#Resource Types
+$ref = @('^recommendations$', '^tasks$', '^alerts$','^managedClusters$','^virtualMachines$','^virtualNetworks$','^networkInterfaces$','^networkInterfaces$','^resourceGroups$','^subscriptions$','^roleAssignments$','^roleDefinitions$')    
+$refRegex = [string]::Join('|', $ref) 
+#Resource Providers
+$ref2 = @('Microsoft.Resources','Microsoft.Network','Microsoft.Advisor','Microsoft.Compute','Microsoft.ContainerService','Microsoft.Security','Microsoft.Authorization')
+$ref2Regex = [string]::Join('|', $ref2)
+
+#Resource Types (location only for resources)
+$ref3 = @('^resourceGroups$','^subscriptions$','^locations$')    
+$ref3Regex = [string]::Join('|', $ref3) 
+
+$providers = Get-AzureRmResourceProvider 
+$providers | %{
+    if ($_.ProviderNamespace -match $ref2Regex){
+        "******************************************************************"
+        "### Provider:          "+$_.ProviderNamespace
+        $resourcetypes = (Get-AzureRmResourceProvider -ProviderNamespace $_.ProviderNamespace).ResourceTypes
+        #"### Resource Types:    " + ((Get-AzureRmResourceProvider -ProviderNamespace $_.ProviderNamespace).ResourceTypes).count
+        ""
+        #We only want to show location resource API version if the provider is Microsoft.Resources
+        if ($_.ProviderNamespace -eq 'Microsoft.Resources'){ 
+                $resourcetypes | %{
+                    If ($_.ResourceTypeName -match $ref3Regex){
+                        "- Resource Type Name:  " + $_.ResourceTypeName 
+                        "- API last version:    " + ($_.ApiVersions | Select-Object -First 1)
+                        ""
+                    }
+                }
+        }
+        else{
+            $resourcetypes | %{
+                    If ($_.ResourceTypeName -match $refRegex){
+                        "- Resource Type Name:  " + $_.ResourceTypeName 
+                        "- API last version:    " + ($_.ApiVersions | Select-Object -First 1)
+                        ""
+                    }
+            }
+        }
+    }
+}