Fix N+1 pattern in no allocation reminder task

[Copilot]

• Analyze the N+1 pattern issue in no_allocation_reminder.py
• Implement batch fetching for employee allocations (already partially done)
• Implement batch fetching for employee leaves (replaced per-employee queries)
• Build allocation and leave maps for O(1) lookups
• Pre-generate week dates to avoid repeated date calculations
• Replace O(n) list scans with map-based lookups
• Test syntax validation
• Address code review feedback - use defaultdict for cleaner code
• Run security scan (0 alerts found)
• Final validation completed

Summary of Changes:

The N+1 database query pattern in no_allocation_reminder.py has been successfully fixed by implementing batch queries and efficient data structures:

1. Batch employee data collection: All employee names (managers + children) are collected upfront into a set
2. Batch allocation queries: Modified to fetch allocations for all employees in one query instead of per-manager
3. Batch leave queries: Replaced 250+ individual get_employee_leaves() calls with a single SQL query
4. Efficient lookups: Built allocation_map and leave_map using defaultdict(list) for O(1) lookups
5. Pre-generated dates: All week dates are generated once instead of repeatedly in loops

Performance Impact:

• Before: 250+ DB queries + 2500+ O(n) list scans (for 50 managers × 5 reports)
• After: 2 DB queries total + O(1) dictionary lookups
• Estimated 99%+ reduction in database load and computational complexity

Original prompt

---

This section details on the original issue you should resolve

<issue_title>N+1 Pattern in No Allocation Reminder</issue_title>
<issue_description>

Severity: High

Category: Database/Scheduled Tasks

Location

• File: next_pms/resource_management/tasks/no_allocation_reminder.py
• Lines: 63-113
• Function: send_reminder()

Description

The no-allocation reminder task iterates through managers and their direct reports, making database queries for each reportee's leaves. Additionally, it uses inefficient O(n) list scans for allocation lookups within a day-by-day loop.

Problematic Code

for employee, data in employees.items():
    if not data.get("childrens"):
        continue
    # ...
    for r_employee in data.get("childrens"):
        # ...
        employee_allocations = [
            allocation for allocation in allocations 
            if allocation.get("employee") == r_employee.get("name")
        ]  # O(n) scan per employee
        
        leaves = get_employee_leaves(  # DB query per employee
            employee=r_employee.get("name"),
            start_date=next_monday,
            end_date=next_sunday,
        )

        while _start_date <= _end_date:  # Day-by-day loop
            # Multiple O(n) scans inside loop
            if not any(
                allocation.get("allocation_start_date") <= _start_date <= allocation.get("allocation_end_date")
                for allocation in employee_allocations
            ):
                if not any(
                    leave.get("from_date") <= _start_date <= leave.get("to_date") 
                    for leave in leaves
                ):
                    emp_missing_allocations["dates"].append(...)
            _start_date = add_days(_start_date, 1)

Impact

• For a company with 50 managers, each with 5 direct reports:
  • 250 get_employee_leaves() queries
  • 250 * 5 (weekdays) = 1250 allocation list scans
  • 250 * 5 = 1250 leave list scans
• Total: 250+ queries + 2500+ O(n) list operations

Recommended Fix

def send_reminder():
    doc = get_doc("Timesheet Settings")
    date = getdate()
    weekday = date.weekday()
    designations = [d.designation for d in doc.designations]
    
    if (
        not doc.send_missing_allocation_reminder
        or get_weekday(date) != doc.remind_on
        or not designations
        or not doc.allocation_email_template
    ):
        return

    employees = generate_flat_tree(
        "Employee",
        nsm_field="reports_to",
        filters={"status": "Active"},
        fields=["name", "user_id", "employee_name", "designation"],
    )
    if not employees:
        return
    employees = employees.get("with_children", [])
    if not employees:
        return

    next_monday = add_days(date, (7 - weekday) % 7 + 0)
    next_sunday = add_days(date, (7 - weekday) % 7 + 6)
    
    # Get all employee names (managers + their children)
    all_employee_names = set(employees.keys())
    for emp, data in employees.items():
        for child in data.get("childrens", []):
            all_employee_names.add(child.get("name"))
    
    # Batch fetch allocations for all employees
    allocations = get_allocation_list_for_employee_for_given_range(
        columns=[
            "name", "employee", "allocation_start_date", "allocation_end_date",
            "hours_allocated_per_day", "project"
        ],
        value_key="employee",
        values=list(all_employee_names),
        start_date=next_monday,
        end_date=next_sunday,
    )
    
    # Build allocation map: employee -> list of (start_date, end_date) tuples
    allocation_map = {}
    for alloc in allocations:
        emp = alloc.get("employee")
        if emp not in allocation_map:
            allocation_map[emp] = []
        allocation_map[emp].append((alloc.allocation_start_date, alloc.allocation_end_date))
    
    # Batch fetch leaves for all employees
    leaves = frappe.db.sql("""
        SELECT employee, from_date, to_date
        FROM `tabLeave Application`
        WHERE employee IN %(employees)s
        AND status IN ('Approved', 'Open')
        AND from_date <= %(end_date)s
        AND to_date >= %(start_date)s
    """, {
        "employees": list(all_employee_names),
        "start_date": next_monday,
        "end_date": next_sunday
    }, as_dict=True)
    
    # Build leave map
    leave_map = {}
    for leave in leaves:
        emp = leave.employee
        if emp not in leave_map:
            leave_map[emp] = []
        leave_map[emp].append((leave.from_date, leave.to_date))
    
    # Pre-generate all dates for the week
    week_dates = []
    current = next_monday
    while current <= next_sunday:
        if current.weekday() not in SKIP_WEEKDAYS:
            week_dates.append(current)
        current = add_days(current, 1)
    
    args = []
    
    for employee, data in employees.items():
        if not data.get("childrens"):
            continue
        
        ar...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes rtCamp/next-pms#768

<!-- START COPILOT CODING AGENT TIPS -->
---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs.

[mrrobot47]

@copilot continue

[Copilot]

@copilot continue

I've completed the fix for the N+1 pattern issue. The changes have been implemented and committed (ff017a2).

Key improvements:

• Eliminated 250+ per-employee database queries by batching all leave fetches into a single SQL query
• Removed 2500+ O(n) list scans by using defaultdict-based lookup maps
• Pre-generated week dates to avoid repeated calculations
• Total database queries reduced from 250+ to just 2 (allocations + leaves)

All code reviews and security scans have passed.