Problem with Prometheus in Sametime monitoring dashboard
I set up the monitoring dashboard in Sametime, using the provided JSON files; in case you don’t know how to do that, you enable it running this command
docker-compose -f docker-compose.yml -f docker-compose-monitoring.yml up -d
https://help.hcltechsw.com/sametime/1201/admin/Managing_dashboard.html
Today I discovered that my Prometheus container kept crashing, and in the logs I found lines like these:
corrupted block 01GQT2R33ASC29W57NF4WEZKP3: open data/01GQT2R33ASC29W57NF4WEZKP3/chunks: no such file or directory;
corrupted block 01GQW0HNMW76WDHNY6T8CMEVQ0: open data/01GQW0HNMW76WDHNY6T8CMEVQ0/chunks: no such file or directory;
corrupted block 01GQGDRDRKSS062T1762WACX5H: open data/01GQGDRDRKSS062T1762WACX5H/chunks: no such file or directory;
corrupted block 01GQJBHXKJW38754FR7CZGTWZN: open data/01GQJBHXKJW38754FR7CZGTWZN/chunks: no such file or directory;
corrupted block 01GQM9BGT78HDQXM2EZ2BP5P8S: open data/01GQM9BGT78HDQXM2EZ2BP5P8S/chunks: no such file or directory;
corrupted block 01GQP75073Q7FRW2MEGE1T5869: open data/01GQP75073Q7FRW2MEGE1T5869/chunks: no such file or directory;
corrupted block 01GQR4YKN8XJS7FY30VYR24VTG: open data/01GQR4YKN8XJS7FY30VYR24VTG/chunks: no such file or directory"
In fact looking at the directory structure under /tmp/prometheus, the default directory where Prometheus store files I found this: