Data Retention configuration for HM only

📊 Retention Settings for HM-Only Data Storage

🎯 Objective

The customer requires only Hardware Metrics (HM) data to be retained. All other data types — such as functions, raw data, screenshots, and KPIs — must be automatically removed based on defined retention settings.

⚙️ Related Retention Parameters and Their Effects

⚠️ Warnings:

Setting any of the above to 0 causes immediate deletion of respective data.

Negative values should only be used for deleting KPI aggregates (WEEKLY, MONTHLY).

DO NOT set RETENTION_FUNCTIONS_ITEM_DAYS, RETENTION_ITEM_DATA_DAYS, or RETENTION_RAW_DATA_DAYS to negative values — it will cause errors.

🔍Info:

All the retention parameters are described in the Special settings article

🛠️ Recommended Configuration

Frequent Cleanup via Cron

• ⏱️ Set up a cron job to remove non-HM data every 10 minutes.

• 🔁 Ensures quick turnover and aligns with the requirement to retain HM only.

📋 Pipelines to be cleaned up

• kpi-pipeline

• rawdata-pipeline

• process-pipeline

📌 Behaviour Validation

• ✅ HM data is retained and displayed properly on frontend.

• ❌ Screenshots are not stored (SCREENSHOTS_RETENTION_DAYS = 0).

• ♻️ Functions and raw data are cleared every 10 minutes.

• 🧹 KPI data (HOURLY + DAILY) is removed daily via cron.

✅ Final Configuration Summary

The final configuration ensures that only essential Hardware Metrics (HM) data is retained, while all non-HM data (functions, raw data, screenshots, and KPI aggregates) is regularly cleaned up using retention settings and scheduled cron jobs. This approach:

• ✅ Meets the customer’s requirement to retain HM only

• 🧹 Minimizes data storage and system load by removing unnecessary data quickly

• 🛠️ Prevents errors by using valid retention values (0 instead of negative) for non-aggregate data

• 🔁 Keeps the system clean and efficient with a 10-minute cleanup cycle
  

Overall, it balances system performance with compliance to data retention policies.