Fix N+1 query pattern in daily reminder cron (3500→5 queries)

[Copilot]

Daily reminder task was executing 6-7 queries per employee, creating severe database load at scale (3500 queries for 500 employees, 30-60s execution time).

Fixes: #766

Changes

Replaced per-employee queries with batch operations:

• Holidays: frappe.get_all() with ["in", holiday_lists] filter
• Leaves: Single SQL query with employee IN (...) clause
• Timesheets: frappe.get_all() with ["in", employee_names] filter
• Daily norms: Inline calculation from pre-fetched custom_working_hours/custom_work_schedule fields

Before:

for employee in employees:
    daily_norm = get_employee_daily_working_norm(employee.name)  # Query
    if is_holiday_or_leave(date, employee.name, daily_norm):     # 3 queries
        continue
    hour = reported_time_by_employee(employee.name, date, ...)   # 3 queries

After:

# Batch fetch once
holidays_on_date = set(frappe.get_all("Holiday", filters={"parent": ["in", ...]}))
leave_map = {leave.employee: [...] for leave in frappe.db.sql(..., employee_names)}
timesheet_map = {ts.employee: hours for ts in frappe.get_all("Timesheet", ...)}

# Process with O(1) lookups
for employee in employees:
    if employee.holiday_list in holidays_on_date: continue
    total_hours = leave_map.get(...) + timesheet_map.get(...)

Removed:

• get_employee_daily_working_norm() - replaced with calculate_daily_norm() using fetched fields
• is_holiday_or_leave() - logic inlined with batch-fetched data
• reported_time_by_employee() - logic inlined with batch-fetched data
• get_leave_info() - replaced with batch SQL query

Impact

Query count: 3500→5 (99.8% reduction)
Execution time: 30-60s→1-2s (estimated)
Scales O(1) vs O(N) with employee count

Original prompt

---

This section details on the original issue you should resolve

<issue_title>N+1 Query Pattern in Daily Reminder Cron</issue_title>
<issue_description>

Severity: Critical

Category: Database/Scheduled Tasks

Location

• File: next_pms/timesheet/tasks/daily_reminder_for_time_entry.py
• Lines: 33-49
• Function: send_reminder()

Description

The daily reminder cron loops through all active employees and makes multiple database queries per employee, resulting in severe N+1 performance issues. For an organization with 500 employees, this task could generate 2000+ database queries.

Problematic Code

employees = frappe.get_all(
    "Employee",
    filters={"status": "Active", "department": ["in", allowed_departments]},
    fields="*",
)

for employee in employees:
    daily_norm = get_employee_daily_working_norm(employee.name)  # DB query
    if is_holiday_or_leave(date, employee.name, daily_norm):  # 2+ DB queries
        continue
    hour = reported_time_by_employee(employee.name, date, daily_norm)  # 2+ DB queries
    if hour >= daily_norm:
        continue
    # ... send mail

Nested N+1 in is_holiday_or_leave():

def is_holiday_or_leave(date: datetime.date, employee: str, daily_norm: int) -> bool:
    holiday_list = get_holiday_list_for_employee(employee)  # DB query
    is_holiday = frappe.db.exists("Holiday", {...})  # DB query
    leave_info = get_leave_info(employee, date, daily_norm)  # SQL query
    # ...

Nested N+1 in reported_time_by_employee():

def reported_time_by_employee(employee: str, date: datetime.date, daily_norm: int) -> int:
    leave_info = get_leave_info(employee, date, daily_norm)  # SQL query (duplicate!)
    if_exists = frappe.db.exists("Timesheet", {...})  # DB query
    timesheets = frappe.get_all("Timesheet", {...})  # DB query
    # ...

Impact

• Database Load: 4-6 queries per employee = 2000-3000 queries for 500 employees
• Execution Time: Task runs daily, taking 30-60 seconds when it should take 1-2 seconds
• Resource Contention: Creates database connection pressure during business hours

Recommended Fix

1. Batch-fetch holiday lists for all employees:

def send_reminder():
    current_date = getdate()
    date = add_days(current_date, -1)
    
    setting = frappe.get_doc("Timesheet Settings")
    if not setting.send_daily_reminder:
        return
    
    allowed_departments = [doc.department for doc in setting.allowed_departments]
    employees = frappe.get_all(
        "Employee",
        filters={"status": "Active", "department": ["in", allowed_departments]},
        fields=["name", "employee_name", "user_id", "holiday_list", 
                "custom_working_hours", "custom_work_schedule"],
    )
    
    if not employees:
        return
    
    employee_names = [e.name for e in employees]
    
    # Batch fetch all holiday lists
    holiday_lists = set(e.holiday_list for e in employees if e.holiday_list)
    holidays_on_date = set()
    if holiday_lists:
        holidays = frappe.get_all(
            "Holiday",
            filters={"parent": ["in", list(holiday_lists)], "holiday_date": date},
            pluck="parent"
        )
        holidays_on_date = set(holidays)
    
    # Batch fetch leaves for all employees on this date
    leave_map = {}
    leaves = frappe.db.sql("""
        SELECT employee, half_day, half_day_date
        FROM `tabLeave Application`
        WHERE %(date)s BETWEEN from_date AND to_date 
        AND employee IN %(employees)s 
        AND status IN ('Approved', 'Open')
    """, {"date": date, "employees": employee_names}, as_dict=True)
    
    for leave in leaves:
        if leave.employee not in leave_map:
            leave_map[leave.employee] = []
        leave_map[leave.employee].append(leave)
    
    # Batch fetch timesheets for all employees on this date
    timesheets = frappe.get_all(
        "Timesheet",
        filters={"employee": ["in", employee_names], "start_date": date, "end_date": date},
        fields=["employee", "total_hours"]
    )
    timesheet_map = {}
    for ts in timesheets:
        if ts.employee not in timesheet_map:
            timesheet_map[ts.employee] = 0
        timesheet_map[ts.employee] += ts.total_hours
    
    # Now process employees without additional queries
    for employee in employees:
        daily_norm = calculate_daily_norm(employee)  # Use fetched fields
        
        # Check holiday
        if employee.holiday_list in holidays_on_date:
            continue
        
        # Check leave
        emp_leaves = leave_map.get(employee.name, [])
        is_full_day_leave = False
        leave_hours = 0
        for leave in emp_leaves:
            if leave.half_day and getdate(leave.half_day_date) == date:
                leave_hours += daily_norm / 2
            else:
                is_full_day_leave = True
                break
        
        if is_full_day_leave:
            c...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes rtCamp/next-pms#766

<!-- START COPILOT CODING AGENT TIPS -->
---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs.