This commit is contained in:
Tushar D 2021-03-15 00:24:06 -07:00
Родитель e1272be362
Коммит 4be8855319
8 изменённых файлов: 33 добавлений и 33 удалений

4
.github/workflows/docker-publish.yml поставляемый
Просмотреть файл

@ -16,7 +16,7 @@ on:
- main
env:
IMAGE_NAME: mifi
IMAGE_NAME: monitofi
jobs:
# See also https://docs.docker.com/docker-hub/builds/automated-testing/
@ -98,7 +98,7 @@ jobs:
with:
push: true
tags: |
dtushar/mifi:latest
dtushar/monitofi:latest
-
name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}

2
.github/workflows/dockerhub-description.yml поставляемый
Просмотреть файл

@ -17,4 +17,4 @@ jobs:
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
repository: dtushar/mifi
repository: dtushar/monitofi

Просмотреть файл

@ -2,16 +2,16 @@
# // Licensed under the MIT license.
FROM python:3.7-slim
RUN mkdir /opt/nifimonitor
RUN mkdir /opt/monitofi
ENV VIRTUAL_ENV=/opt/nifimonitor/nifimonitor-env
ENV VIRTUAL_ENV=/opt/monitofi/monitofi-env
RUN python -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
COPY ./requirements.txt /opt/nifimonitor/requirements.txt
RUN pip install -r /opt/nifimonitor/requirements.txt
COPY ./requirements.txt /opt/monitofi/requirements.txt
RUN pip install -r /opt/monitofi/requirements.txt
WORKDIR /opt/nifimonitor
COPY ./nifi_monitor.py /opt/nifimonitor/nifi_monitor.py
WORKDIR /opt/monitofi
COPY ./monitofi.py /opt/monitofi/monitofi.py
CMD ["python", "/opt/nifimonitor/nifi_monitor.py"]
CMD ["python", "/opt/monitofi/monitofi.py"]

Просмотреть файл

@ -27,7 +27,7 @@
Using a Simple Instrumentation Key Received when creating an Application Insights Resource in Azure, all the MonitoFi monitoring data can be pushed to Azure. This data can be viewed using Grafana by importing included AIDashboard. Queries for data in Application Insights are written using Kusto Query Language and are easy to modify.
Visit https://github.com/tushardhadiwal/docker-influxdb-grafana for Grafana And InfludDB Support.\
Visit https://github.com/microsoft/MiFi for MonitoFi Source Code.
Visit https://github.com/microsoft/MonitoFi for MonitoFi Source Code.
## Architecture
@ -35,9 +35,9 @@
#### Quick Start
Run following commands on machine which has access to NiFi cluster:
Run following commands on machine which has access to Apache NiFi cluster:
```sh
git clone git@github.com:microsoft/MiFi.git
git clone git@github.com:microsoft/MonitoFi.git
```
Fully Automated Script will take care of running InfluxDB & Grafana container , configuring datasources in grafana, importing pre included dashboards in grafana, Adding notification channels if needed, Running MonitoFi container against your cluster. Please set variables at the top of the script to desired state.
```sh
@ -63,8 +63,8 @@ docker run -d \
-p 3003:3003 \
-p 3004:8083 \
-p 8086:8086 \
-v /home/$(whoami)/nifimonitor/influx:/var/lib/influxdb \
-v /home/$(whoami)/nifimonitor/grafana:/var/lib/grafana \
-v /home/$(whoami)/monitofi/influx:/var/lib/influxdb \
-v /home/$(whoami)/monitofi/grafana:/var/lib/grafana \
dtushar/docker-influxdb-grafana:latest
```
You can now visit http://localhost:3003 to access Grafana with following credentials.
@ -84,20 +84,20 @@ Run this command to run MonitoFi against your NiFi Cluster :
```sh
docker run \
--name=mifi1 \
--name=monitofi1 \
--network=host -d \
-e INFLUXDB_SERVER="influxdb-grafana" \
-e ENDPOINT_LIST="controller/cluster,flow/cluster/summary,flow/process-groups/root,flow/status,counters,system-diagnostics" \
-e SLEEP_INTERVAL=300 \
-e API_URL='http://localhost:8080/nifi-api/' \
--restart unless-stopped \
dtushar/mifi:1.0
dtushar/monitofi:1.0
```
If your NiFi Cluster is SECURE and supports login via certificate then please add the following to the above command, this will mount the certificate into MonitoFi Container:
```sh
-e SECURE=True \
-v $(pwd)/keystore.pkcs12:/opt/nifimonitor/cert.pkcs12 \
-v $(pwd)/keystore.pkcs12:/opt/monitofi/cert.pkcs12 \
-e CERT_PASS="PasswordForCertificate" \
```

Просмотреть файл

@ -1,8 +1,8 @@
#!/bin/bash
# // Copyright (c) Microsoft Corporation.
# // Licensed under the MIT license.
sudo rm -rf /home/$(whoami)/nifimonitor
sudo rm -rf /home/$(whoami)/monitofi
docker stop influxdb-grafana
docker rm influxdb-grafana
docker stop mifi1
docker rm mifi1
docker stop monitofi1
docker rm monitofi1

Просмотреть файл

@ -42,12 +42,12 @@ fi
#Create Folders for Grafana & Influxdb
echo "Creating necessary volumes for Grafana and InfluxDB..."
mkdir -p /home/$(whoami)/nifimonitor/{influx,grafana}
mkdir -p /home/$(whoami)/monitofi/{influx,grafana}
#Run Grafana & InfluxDB Container
#Note in below command since host netowrking is used, -p flags are ignored. They are kept there to show user which ports are running applications. User can access them with localhost:PORT on host machine directly.
echo "Launching InfluxDB & Grafana Container"
docker run -d --name influxdb-grafana --network=host -p 3003:3003 -p 3004:8083 -p 8086:8086 -v /home/$(whoami)/nifimonitor/influx:/var/lib/influxdb -v /home/$(whoami)/nifimonitor/grafana:/var/lib/grafana dtushar/docker-influxdb-grafana:latest
docker run -d --name influxdb-grafana --network=host -p 3003:3003 -p 3004:8083 -p 8086:8086 -v /home/$(whoami)/monitofi/influx:/var/lib/influxdb -v /home/$(whoami)/monitofi/grafana:/var/lib/grafana dtushar/docker-influxdb-grafana:latest
#This Is necessary to provide enough time to Grafana to spin up before making API Calls.
sleep 30
@ -75,19 +75,19 @@ fi
if [ "$USE_AZURE" = true ] ; then
#Run NiFi Monitor container
echo "Running MiFi 1.0 Container with Azure AppInsights & InfluxDB Enabled..."
#Run MonitoFi container
echo "Running MonitoFi 1.0 Container with Azure AppInsights & InfluxDB Enabled..."
echo "Please Make sure to add Instrumentation Key for your Application Insights Resource in the IKEY Variable"
docker run --name=mifi1 --network=host -d -e INFLUXDB_SERVER="localhost" -e ENDPOINT_LIST="controller/cluster,flow/cluster/summary,flow/process-groups/root,flow/status,counters,system-diagnostics,system-diagnostics?nodewise=true" -e SLEEP_INTERVAL=300 -e API_URL=$NiFi_API_URL -e IKEY=$AppInsights_InstrumentationKey --restart unless-stopped dtushar/mifi:1.0
docker run --name=monitofi1 --network=host -d -e INFLUXDB_SERVER="localhost" -e ENDPOINT_LIST="controller/cluster,flow/cluster/summary,flow/process-groups/root,flow/status,counters,system-diagnostics,system-diagnostics?nodewise=true" -e SLEEP_INTERVAL=300 -e API_URL=$NiFi_API_URL -e IKEY=$AppInsights_InstrumentationKey --restart unless-stopped dtushar/monitofi:1.0
elif [ "$USE_AZURE" = false ] ; then
#Run NiFi Monitor container
echo "Running MiFi 1.0 Container..."
docker run --name=mifi1 --network=host -d -e INFLUXDB_SERVER="localhost" -e ENDPOINT_LIST="controller/cluster,flow/cluster/summary,flow/process-groups/root,flow/status,counters,system-diagnostics,system-diagnostics?nodewise=true" -e SLEEP_INTERVAL=300 -e API_URL=$NiFi_API_URL --restart unless-stopped dtushar/mifi:1.0
#Run MonitoFi container
echo "Running MonitoFi 1.0 Container..."
docker run --name=monitofi1 --network=host -d -e INFLUXDB_SERVER="localhost" -e ENDPOINT_LIST="controller/cluster,flow/cluster/summary,flow/process-groups/root,flow/status,counters,system-diagnostics,system-diagnostics?nodewise=true" -e SLEEP_INTERVAL=300 -e API_URL=$NiFi_API_URL --restart unless-stopped dtushar/monitofi:1.0
fi
echo "Importing NiFi Monitor Dashboard to Grafana"
#Please Adjust the queries according to number of nodes in your NiFi cluster after deployment.
#Adding Dashboard For NiFI Monitor
#Adding Dashboard For MonitoFi
curl -X "POST" "http://localhost:3003/api/dashboards/db" -H "Content-Type: application/json" --user $GRAFANA_USERNAME:$GRAFANA_PASSWORD --data-binary @Grafana/Dashboards/NiFiMonitorDashboard.json | jq
if [ "$USE_AZURE" = true ] ; then

Просмотреть файл

@ -4,7 +4,7 @@ SLEEP_INTERVAL=300
MODE ="unlimited" # In limited mode, only NUMBEROFITERATIONS API calls are made before exiting.
NUMBER_OF_ITERATIONS = 2
SECURE=True #If Set to True, Please add following volume mount with certificate -v $(pwd)/keystore.pkcs12:/opt/nifimonitor/cert.pkcs12
SECURE=True #If Set to True, Please add following volume mount with certificate -v $(pwd)/keystore.pkcs12:/opt/monitofi/cert.pkcs12 to docker run command
CERT_PASS="PasswordForCertificate"
IKEY="REPLACE_ME" #AppInsights Instrumentation Key"

Просмотреть файл

@ -22,7 +22,7 @@ IKEY = os.getenv('IKEY', "REPLACE_ME")
SLEEP_INTERVAL = int(os.getenv('SLEEP_INTERVAL', 300))
SECURE = os.getenv('SECURE', False) # Is NiFi Cluster Secure
CERT_FILE = os.getenv('CERT_FILE', '/opt/nifimonitor/cert.pkcs12')
CERT_FILE = os.getenv('CERT_FILE', '/opt/monitofi/cert.pkcs12')
CERT_PASS = os.getenv('CERT_PASS', 'REPLACE_ME')
INFLUXDB_SERVER = os.getenv('INFLUXDB_SERVER', "127.0.0.1") # IP or hostname to InfluxDB server
@ -39,7 +39,7 @@ conditions = {
}
# Sysout Logging Setup
logger = logging.getLogger("nifi-monitor")
logger = logging.getLogger("monitofi")
logger.setLevel(logging.INFO)
syshandler = logging.StreamHandler(sys.stdout)
syshandler.setLevel(logging.INFO)