Question3

Diagnosing and Fixing a Declining Demand Forecasting Model

Introduction

A food delivery company relies on a demand forecasting model to predict order volumes. While the model initially performs well, its accuracy declines over time, causing inefficiencies in inventory management, staffing, and delivery operations. To address this, we must:

1. Diagnose the root cause of performance degradation
2. Fix the model using updated data and techniques
3. Implement a monitoring system to detect future issues early

Step 1: Diagnosing the Issue

(Referring to Phase 5: Communicate Results & Phase 6: Operationalize in the PDF)

1. Check for Data Drift

• Data drift occurs when the statistical properties of input data change over time, making the model's past patterns unreliable.
• Steps to detect drift:
  • Compare historical vs. recent data distributions using Kolmogorov-Smirnov test or Chi-Square test.
  • Use visualization techniques:
    • Boxplots & Histograms to detect shifts in feature distributions.
    • Heatmaps to observe correlations between variables over time.

2. Analyze Model Performance Trends

• Evaluate accuracy metrics (RMSE, MAPE, R²) over time.
• Identify specific time periods when accuracy dropped.
• Compare predicted vs. actual values using:
  • Line graphs to observe trends.
  • Scatter plots to detect prediction deviations.

3. Investigate External Factors