Day 14: Advanced Logging and Monitoring with Prometheus, Grafana, and Alertmanager (Termux Edition!)

Introduction

Today, we will look deep into the world of advanced logging and monitoring, but with a twist—we’re doing it all on our phones! This is Day 14 of my Devops project, and we’re going mobile with Termux( I don’t have access to my windows machine because I will be on transit all day but I thought it would be fun to see what I can do using termux) to build a powerful monitoring system using Prometheus, Grafana, and Alertmanager. This is a great way to learn and practice these tools, even when you’re on the go.

Objectives

• Create a Node.js application that generates metrics and logs.
• Dockerize the application for easy deployment.
• Configure Prometheus to collect metrics and trigger alerts.
• Set up Alertmanager to send email to notify you.
• Use Grafana to visualize metrics and create dashboards.
• Organize the project for easy sharing on GitHub.

A More detailed Guide

1. Setting up the Node.js Application (Termux)

• Create the Project Directory:

   mkdir Day14-monitoring-app cd Day14-monitoring-app
  

• Create the app Directory:

   mkdir app cd app
  

• Create the index.js File:

   touch index.js
  

• Open index.js in Termux’s Nano editor:

   nano index.js
  

• Paste the following code into index.js:

   // Import necessary modules const express = require('express'); const app = express(); const port = 4000; const client = require('prom-client'); const morgan = require('morgan'); const collectDefaultMetrics = client.collectDefaultMetrics; // Collect default metrics (CPU, memory, etc.) collectDefaultMetrics(); // Middleware for logging requests app.use(morgan('combined')); // Logs HTTP requests // Define routes for our application app.get('/', (req, res) => res.send('Hello World!')); app.get('/api/data', (req, res) => res.json({ data: 'some data' })); // Endpoint to expose Prometheus metrics app.get('/metrics', (req, res) => { res.set('Content-Type', client.register.contentType); res.end(client.register.metrics()); }); // Start the server app.listen(port, () => console.log(`App listening at http://localhost:${port}`));
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

• Create the package.json File:

   touch package.json
  

• Open package.json in Nano:

   nano package.json
  

• Paste the following code into package.json:

   { "name": "advanced-monitoring-app", "version": "1.0.0", "description": "A Node.js application with advanced monitoring", "main": "index.js", "scripts": { "start": "node index.js" }, "author": "Your Name", "license": "MIT", "dependencies": { "express": "^4.18.2", "morgan": "^1.10.0", "prom-client": "^16.0.0" } }
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

• Install Dependencies:

   npm install
  

2. Dockerizing the Application (Termux)

• Create the Dockerfile:

   touch Dockerfile
  

• Open Dockerfile in Nano:

   nano Dockerfile
  

• Paste the following code into Dockerfile:

   # Use the Node.js 14 image as our base FROM node:14 # Set the working directory inside the container WORKDIR /app # Copy the package.json file COPY package*.json ./ # Install dependencies using npm RUN npm install # Copy the rest of the application code COPY . . # Expose port 4000 for our application to listen on EXPOSE 4000 # Run the Node.js application when the container starts CMD ["node", "index.js"] 

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

• Build the Docker Image:

   docker build -t your_dockerhub_username/advanced-monitoring-app:latest .
  

• Log in to Docker Hub:

   docker login
  

• Push the Image to Docker Hub:

   docker push your_dockerhub_username/advanced-monitoring-app:latest
  

3. Configuring Prometheus (Termux)

• Create the prometheus.yml File:

   touch prometheus.yml
  

• Open prometheus.yml in Nano:

   nano prometheus.yml
  

• Paste the following code into prometheus.yml:

   global: scrape_interval: 15s # Scrape metrics every 15 seconds scrape_configs: - job_name: 'node-app' # Name of the job for scraping our app static_configs: - targets: ['host.docker.internal:4000'] # Target to scrape (our app) rule_files: - 'alert.rules.yml'
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

4. Configuring Alertmanager (Termux)

• Create the alertmanager.yml File:

   touch alertmanager.yml
  

• Open alertmanager.yml in Nano:

   nano alertmanager.yml
  

• Paste the following code into alertmanager.yml:

   global: resolve_timeout: 5m # Alerts are considered resolved after 5 minutes of no new alerts route: group_by: ['job'] # Group alerts by job name group_wait: 30s # Wait 30 seconds to group alerts group_interval: 5m # Group alerts every 5 minutes repeat_interval: 12h # Repeat alerts every 12 hours receiver: 'email' # Send alerts to the email receiver receivers: - name: 'email' email_configs: - to: '[email protected]' # Your email address from: '[email protected]' # Sender email smarthost: 'smtp.example.com:587' # SMTP server (replace with your settings) auth_username: 'alertmanager' # SMTP username auth_password: 'password' # SMTP password
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

5. Docker Compose Setup (Termux)

• Create the docker-compose.yml File:

   touch docker-compose.yml
  

• Open docker-compose.yml in Nano:

   nano docker-compose.yml
  

• Paste the following code into docker-compose.yml:

   version: '3' services: prometheus: image: prom/prometheus volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml # Mount the prometheus.yml file ports: - "9090:9090" # Expose port 9090 for Prometheus grafana: image: grafana/grafana ports: - "3000:3000" # Expose port 3000 for Grafana alertmanager: image: prom/alertmanager volumes: - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml # Mount the alertmanager.yml file ports: - "9093:9093" # Expose port 9093 for Alertmanager
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

6. Adding Custom Alerts (Termux)

• Create the alert.rules.yml File:

   touch alert.rules.yml
  

• Open alert.rules.yml in Nano:

   nano alert.rules.yml
  

• Paste the following code into alert.rules.yml:

   groups: - name: example rules: - alert: HighRequestLatency expr: http_request_duration_seconds_mean{job="node-app"} > 0.5 # Check if average request duration is over 0.5 seconds for: 2m # Trigger alert only if the condition is true for 2 minutes labels: severity: "page" # Add a label to the alert annotations: summary: "High request latency" # Alert summary description: "The request latency is above 0.5s for more than 2 minutes." # Detailed description
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

7. Deploying the Stack (Termux)

 # Start the Docker Compose stack in detached mode docker-compose up -d

8. Configuring Grafana Dashboards (Termux)

• Access Grafana: Open your web browser and go to http://localhost:3000.
• Add Prometheus as a data source: In Grafana, add Prometheus as a data source.
• Create custom dashboards: Create dashboards to visualize your metrics.
• Set up alerts in Grafana: Set up alerts in Grafana to trigger notifications via Alertmanager.

9. GitHub Actions Workflow (Termux)

• Create the .github/workflows Directory:

   mkdir .github/workflows cd .github/workflows
  

• Create the main.yml File:

   touch main.yml
  

• Open main.yml in Nano:

   nano main.yml
  

• Paste the following code into main.yml:

   name: Docker Build & Deploy on: push: branches: - main jobs: build-and-deploy: runs-on: ubuntu-latest steps: - name: Checkout Repository uses: actions/checkout@v2 - name: Login to Docker Hub uses: docker/login-action@v2 with: username: ${{ secrets.DOCKER_USERNAME }} password: ${{ secrets.DOCKER_PASSWORD }} - name: Build and Push Docker Image run: | docker build -t ${{ secrets.DOCKER_USERNAME }}/advanced-monitoring-app:latest . docker push ${{ secrets.DOCKER_USERNAME }}/advanced-monitoring-app:latest - name: Deploy Docker Container run: | docker-compose up -d
  

• Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

10. Setting Up Your GitHub Repository

• Create a new repository on GitHub.

• Initialize a Git repository in your project directory:

   git init
  

• Add all files to the repository:

   git add .
  

• Commit your changes:

   git commit -m "Initial commit"
  

• Add the GitHub remote:

   git remote add origin <your_github_repo_url>
  

• Push your changes to GitHub:

   git push origin main
  

Remember:

• Replace your_dockerhub_username with your actual Docker Hub username.
• Replace [email protected] with your email address.
• Replace smtp.example.com:587 with your SMTP server address and port.
• Set the DOCKER_USERNAME and DOCKER_PASSWORD secrets in your GitHub repository settings.

Conclusion

With this setup , you can indeed monitor your application and receive alerts directly on your phone using Termux. Here’s how it works:

1. Data Collection: The Node.js application you’ve created acts as a source of data. It collects metrics about HTTP requests, system resources (CPU, memory), and network connections.

2. Prometheus: Prometheus is the data collector and storage engine. It regularly scrapes metrics from your Node.js application and stores them.

3. Alertmanager: Alertmanager is the notification system. You’ve configured it to send email alerts when certain conditions are met. For example, if your Node.js application experiences high request latency or if your system’s CPU load exceeds a threshold, Alertmanager will send you an email notification.

4. Grafana: Grafana is the visualization tool. It allows you to create dashboards to view and analyze the collected metrics. You can create graphs, charts, and other visualizations to gain insights into your application’s performance.

5. Termux: Termux is your mobile terminal environment. You can use it to run all the necessary commands to build, deploy, and manage your monitoring stack. You can also use Termux to access Grafana’s web interface on your phone’s browser to view your dashboards and monitor your application.

How to Receive Alerts on Your Phone:

• Email: The most straightforward way to receive alerts is through email. You’ve already configured Alertmanager to send emails to your specified address.
• Push Notifications: You can explore integrations with push notification services like Pushover or Prowl. These services allow you to receive alerts directly on your phone as push notifications.

Key Points:

• Termux is your mobile command center: Which can start and stop the Docker Compose stack, view logs, and manage your monitoring system.
• Grafana is your visual dashboard: It gives you a clear and intuitive way to see your application’s performance.
• Alertmanager is your early warning system: It notifies you of potential problems so you can address them quickly.

-To receive emails on your phone, you’ll need to ensure that your email client is configured .

• If you want to use push notifications, you’ll need to set up an account with a push notification service and integrate it with Alertmanager.

With this setup, you have a powerful and flexible monitoring system at your fingertips, even when you’re on the go!