To monitor all aspects related to users, including network bandwidth, across your Proxmox cluster, we'll enhance our setup with additional exporters and configurations. We'll use a combination of node_exporter, SNMP exporter, and a custom Proxmox VE exporter to gather comprehensive data. Here's an expanded setup that will cover user activities and network bandwidth:

1. Follow the previous steps to set up Prometheus and Grafana on your central monitoring server.
2. On each Proxmox node, in addition to node_exporter, we'll install a Proxmox VE exporter:

3. Configure the Proxmox VE exporter:

Add the following content (adjust as needed):

4. Start the Proxmox VE exporter:

5. Install and configure collectd on each Proxmox node for detailed network statistics:

Edit the collectd configuration:

Add or modify these plugins:

Replace "vmbr0" with your main bridge interface and "192.168.1.100" with your Prometheus server IP.

6. On the Prometheus server, install collectd-exporter:

Create a systemd service for collectd-exporter:

Add the following content:

Start the collectd-exporter:

7. Update the Prometheus configuration to scrape these new exporters:

Add these job configurations:

8. Restart Prometheus:

9. In Grafana, import these dashboards:

   • 10347 (Proxmox VE Cluster)
   • 11074 (SNMP Statistics)
   • 14537 (Network Overview)

10. Create custom dashboards in Grafana for user-specific metrics. Here are some useful Prometheus queries:

    • CPU usage per VM:

      sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (instance)

    • Memory usage per VM:

      (node_memory_MemTotal_bytes - node_memory_MemFree_bytes - node_memory_Buffers_bytes - node_memory_Cached_bytes) / node_memory_MemTotal_bytes * 100

    • Network bandwidth per VM:

      rate(node_network_receive_bytes_total{device="vmbr0"}[5m]) rate(node_network_transmit_bytes_total{device="vmbr0"}[5m])

    • Disk I/O per VM:

      rate(node_disk_read_bytes_total[5m]) rate(node_disk_written_bytes_total[5m])
11. For user-specific monitoring, you can use the Proxmox API (which the Proxmox VE exporter utilizes) to get data about user sessions, resource usage per user, etc. You may need to create custom scripts to pull this data and expose it to Prometheus.

This enhanced setup will provide comprehensive monitoring of your Proxmox cluster, including:

• System resources (CPU, memory, disk) per node and per VM
• Network bandwidth and I/O statistics
• User sessions and resource usage
• Proxmox-specific metrics (cluster health, storage, etc.)

You can further customize the Grafana dashboards to display the most relevant information for your use case. Remember to set up alerting in Grafana to notify you of any issues or threshold breaches.

This solution uses only open-source tools and provides deep insights into your Proxmox cluster's performance and user activities. You can continue to refine and expand this setup as your monitoring needs evolve.

I'll guide you through creating a Grafana dashboard that includes these metrics. We'll create a dashboard with panels for CPU usage, memory usage, network bandwidth, and disk I/O.

1. Log in to your Grafana instance.
2. Click on the "+" icon on the left sidebar and select "Dashboard".
3. Click on "Add new panel".
4. Now, let's create panels for each of your metrics:

Panel 1: CPU Usage per Instance

1. Name: "CPU Usage per Instance"

2. In the query editor (PromQL), paste:

   sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (instance)
3. From Visualization, select "Graph"
4. Under Panel Options, set Unit to "Percent (0-100)"
5. Click "Apply"

Panel 2: Memory Usage per Instance

1. Add a new panel
2. Name: "Memory Usage per Instance"

3. In the query editor, paste:

   (node_memory_MemTotal_bytes - node_memory_MemFree_bytes - node_memory_Buffers_bytes - node_memory_Cached_bytes) / node_memory_MemTotal_bytes * 100
4. From Visualization, select "Gauge"
5. Under Panel Options, set Unit to "Percent (0-100)"
6. Click "Apply"

Panel 3: Network Bandwidth

1. Add a new panel
2. Name: "Network Bandwidth"

3. In the query editor, add two queries: Query A:

   rate(node_network_receive_bytes_total{device="vmbr0"}[5m])

   Query B:

   rate(node_network_transmit_bytes_total{device="vmbr0"}[5m])
4. From Visualization, select "Graph"
5. Under Panel Options, set Unit to "bytes/sec"
6. In the Legend, you can rename A to "Receive" and B to "Transmit"
7. Click "Apply"

Panel 4: Disk I/O

1. Add a new panel
2. Name: "Disk I/O"

3. In the query editor, add two queries: Query A:

   rate(node_disk_read_bytes_total[5m])

   Query B:

   rate(node_disk_written_bytes_total[5m])
4. From Visualization, select "Graph"
5. Under Panel Options, set Unit to "bytes/sec"
6. In the Legend, you can rename A to "Read" and B to "Write"
7. Click "Apply"
8. Arrange the panels on your dashboard as desired. You can resize and move them around.

9. At the top of the dashboard, click on the settings icon (gear) and then "Variables" to add some variables that will make your dashboard more interactive:

   Add a new variable:

   • Name: node
   • Type: query
   • Data source: Your Prometheus data source
   • Query: label_values(node_cpu_seconds_total, instance)
   • Select "Include All option"
10. Now you can edit each panel and add {instance=~"$node"} to each of your queries to make them filterable by node.
11. Save your dashboard by clicking the save icon at the top of the dashboard.

Here's an example of how your queries might look after adding the variable:

This dashboard will give you a good overview of CPU usage, memory usage, network bandwidth, and disk I/O across your Proxmox nodes. You can further customize it by adding more panels, adjusting the time range, or setting up alerts based on these metrics.