Syazwan/claim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix attachment auto-print, persist cell format, migrate to Bun, 1-click launch

Browse Source 
Print bug: remove hidden-iframe auto-print in printAttachment that fired the
OS print dialog on iframe.onload; open attachment in a new tab instead so
print only happens on an explicit Ctrl+P. Drop now-dead printIframeRef/cleanupPrintFrame.

Cell format persistence: new CellFormatToolbar + cellFormat.js save the last
saved per-claim-type header/item format to localStorage; new claims and added
rows inherit it so users don't re-size/re-bold each time. Backend
cell_format_service renders _cellFormat in Excel/PDF.

Bun migration: replace Node/yarn with Bun for the frontend build.
Dockerfile uses oven/bun:1-alpine; bun.lock replaces yarn.lock/package-lock.json;
start.bat files use bun. Frontend dev uses relative baseURL + craco proxy on
port 5000 (avoids CORS and the Grafana conflict on 3000).

1-click launch: root start.bat boots backend (uvicorn :8002) and frontend
(craco :5000) in child windows and opens the browser.

Repo cleanup: remove Node/yarn lockfiles, cursor sync scripts, stray test
reports/PDFs, root package.json/requirements.txt duplicates.

Backend services: claim_document_storage, payslip_storage,
pdf_company_extractor; attachment filenames normalized from PDF text.

Security: gitignore now excludes docker/local env files (with .env.example
allowed) so the live JWT secret cannot be committed.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Syazwan 2026-07-07 17:55:11 +08:00
parent 1828f54745
commit ef25759eda
61 changed files with 5502 additions and 52871 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

100
.gitignore vendored
View File

@ -27,8 +27,6 @@ share/python-wheels/
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
@ -55,83 +53,38 @@ cover/
*.mo
*.pot
# Django stuff:
# Django
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
# Flask
instance/
.webassets-cache
# Scrapy stuff:
# Scrapy
.scrapy
# Sphinx documentation
# Sphinx
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
# Jupyter
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# UV
# Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
#uv.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
#poetry.toml
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
# pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
# https://pdm-project.org/en/latest/usage/project/#working-with-version-control
#pdm.lock
#pdm.toml
.pdm-python
.pdm-build/
# pixi
# Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
#pixi.lock
# Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
# in the .venv directory. It is recommended not to include this directory in version control.
.pixi
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
# Celery
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
# SageMath
*.sage.py
# Environments
@ -144,14 +97,14 @@ ENV/
env.bak/
venv.bak/
# Spyder project settings
# Spyder
.spyderproject
.spyproject
# Rope project settings
# Rope
.ropeproject
# mkdocs documentation
# mkdocs
/site
# mypy
@ -159,48 +112,35 @@ venv.bak/
.dmypy.json
dmypy.json
# Pyre type checker
# pyre
.pyre/
# pytype static type analyzer
# pytype
.pytype/
# Cython debug symbols
# Cython
cython_debug/
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
.idea/
# Abstra
# Abstra is an AI-powered process automation framework.
# Ignore directories containing user credentials, local state, and settings.
# Learn more at https://abstra.io/docs
.abstra/
# Visual Studio Code
# Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore
# that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
# and can be added to the global gitignore or merged into this file. However, if you prefer,
# you could uncomment the following to ignore the entire vscode folder
# .vscode/
.vscode/
# Ruff stuff:
# Ruff
.ruff_cache/
# PyPI configuration file
# PyPI
.pypirc
# Cursor / agent harness (local only — not part of the Claim app release)
# Cursor / agent harness (local only)
.cursor/
.cursorignore
.cursorindexingignore
.claude/
.codex/
.agents/
.wayland-core/
process/
.vibecode-backup/
.vc-installed-files
@ -210,7 +150,11 @@ AGENTS.md
# Local env and uploads
backend/.env
backend/.env.*
!backend/.env.example
frontend/.env
frontend/.env.*
!frontend/.env.example
backend/uploads/
backend/claim_files/
backend/payslip_files/

261
CODE_REVIEW_FIXES.md
View File

@ -1,261 +0,0 @@
# Code Review Fixes - May 24, 2026
All issues from the code review have been resolved. Tests pass: **Frontend: 10/10** | **Backend: 19/19**.
---
## Must Fix (All Fixed ✅)
### 1. ✅ Removed duplicate date parsing logic (claimDates.js:313-360)
**Issue**: The `shortYyEarly` and `shortYy` regex blocks were identical duplicates.
**Fix**: Removed the first duplicate block. The parsing now has a clean flow:
- Try FLEX_PARSE_FORMATS
- Try 2-digit year regex (shortYy)
- Try OT_LINE_ITEM_PARSE_FORMATS
- Try 4-digit year with special handling
**Location**: `frontend/src/lib/claimDates.js` lines 313-368
---
### 2. ✅ Added request ID to preview debounce (AdminClaimForm.jsx:566-592)
**Issue**: Race condition where stale preview responses could overwrite newer ones if requests completed out of order.
**Fix**: Implemented request ID tracking:
```javascript
const previewRequestIdRef = useRef(0);
const requestId = ++previewRequestIdRef.current;
// ... fetch preview ...
if (requestId !== previewRequestIdRef.current) return; // discard stale
```
**Location**: `frontend/src/pages/AdminClaimForm.jsx` lines 463, 575-583
**Impact**: User typing "01/05/2026" then "01/06/2026" will always show June, not May.
---
### 3. ✅ Fixed memory leak from URL.createObjectURL (AdminClaimForm.jsx:889-892)
**Issue**: Blob URLs persisted until page reload if component unmounted without cleanup.
**Fix**: Cleanup hook now properly revokes both URLs on unmount:
```javascript
useEffect(() => () => {
if (previewUrl) URL.revokeObjectURL(previewUrl);
if (printPreviewUrl) URL.revokeObjectURL(printPreviewUrl);
}, [previewUrl, printPreviewUrl]);
```
**Location**: `frontend/src/pages/AdminClaimForm.jsx` lines 889-892
---
## Should Fix (All Fixed ✅)
### 4. ✅ Documented slashPartsToDayMonth heuristic (claimDates.js:267-276)
**Issue**: Ambiguous date interpretation (05/03 could be May 3rd or March 5th) was undocumented.
**Fix**: Added comprehensive JSDoc:
```javascript
/** Infer day/month from slash-separated parts (DD/MM/YYYY preferred for Malaysian locale).
* Heuristic: if one part > 12, it's the day. If both ≤ 12, assume DD/MM (not MM/DD).
* For ambiguous dates (03/05), assumes day=3, month=5 per local convention. */
```
**Location**: `frontend/src/lib/claimDates.js` lines 255-259
---
### 5. ✅ Added claim period validation for line item dates (claimDates.js:759-792)
**Issue**: Users could enter `15/12/2026` in a **May 2026** mileage claim without validation.
**Fix**: Added bounds checking in `validateLineItemDate`:
```javascript
if (style === LINE_ITEM_STYLE_SLASH_DATE && period && d) {
const bounds = periodDateInputBounds(period);
if (bounds.min && bounds.max && d) {
const iso = format(d, CLAIM_DATE_STORAGE_FORMAT);
if (iso < bounds.min || iso > bounds.max) {
return `Date must be within ${monthName} claim period.`;
}
}
}
```
**Location**: `frontend/src/lib/claimDates.js` lines 769-780
**Impact**: Line item dates are now validated against the claim period month, catching user errors early.
---
### 6. ✅ Validated meal travel range text (claimDates.js:213-217)
**Issue**: Regex allowed nonsense ranges like `99-99/99/9999` or `31-01/02/26` (ends before it starts).
**Fix**: Replaced regex-only check with parsing and validation:
```javascript
const m = s.match(/^(\d{1,2})-(\d{1,2})\/(\d{1,2})\/(\d{2,4})$/);
if (!m) return false;
const start = Number(m[1]);
const end = Number(m[2]);
const month = Number(m[3]);
// Basic validation: start ≤ end, both ≤ 31, month ≤ 12
return start >= 1 && end >= 1 && start <= end && start <= 31 && end <= 31 && month >= 1 && month <= 12;
```
**Location**: `frontend/src/lib/claimDates.js` lines 213-221
**Impact**: Meal travel ranges like `16-19/9/25` now validate that start ≤ end and month ≤ 12.
---
### 7. ✅ Added explicit date format parsing to Python backend (claim_dates.py:239-289)
**Issue**: Backend tried `strptime` with fewer formats than frontend, potentially missing valid dates.
**Fix**: Added explicit DD/MM/YYYY format attempts after heuristic parsing:
```python
# Try explicit DD/MM/YYYY formats as fallback
for fmt in ("%d/%m/%Y", "%d-%m-%Y", "%d.%m.%Y"):
try:
return _accept_parsed(datetime.strptime(s, fmt))
except ValueError:
continue
```
**Location**: `backend/services/claim_dates.py` lines 269-275
**Impact**: Backend now handles dates with dots or dashes (01.12.2026, 01-12-2026) consistently with frontend.
---
### 8. ✅ Removed debug logging from production code (claim_excel_service.py:505-529)
**Issue**: Every mileage Excel fill wrote to `debug-ae9de1.log` with hardcoded session ID.
**Fix**: Completely removed the debug logging block from `fill_mileage_worksheet`.
**Location**: `backend/services/claim_excel_service.py` line 485 onwards (removed ~25 lines)
**Impact**: Production code no longer writes debug logs. Re-add gated behind `os.getenv("ENABLE_DEBUG_LOGGING")` if needed for development.
---
## Nits (All Fixed ✅)
### 9. ✅ Extracted magic number to constant (AdminClaimForm.jsx:591)
**Issue**: Hardcoded `800` millisecond debounce.
**Fix**:
```javascript
const PREVIEW_DEBOUNCE_MS = 800;
// ...
}, PREVIEW_DEBOUNCE_MS);
```
**Location**: `frontend/src/pages/AdminClaimForm.jsx` lines 338-339, 597
---
### 10. ✅ Simplified blankItem function (AdminClaimForm.jsx:312-314)
**Issue**: Ternary operator returned `""` in both branches.
**Fix**:
```javascript
function blankItem(cols) {
return Object.fromEntries(cols.map((c) => [c.key, ""]));
}
```
**Location**: `frontend/src/pages/AdminClaimForm.jsx` lines 312-314
---
### 11. ✅ Kept LINE_ITEM_STYLE_DAY_MONTH_YEAR for backward compatibility
**Issue**: Removing the constant broke existing function signatures.
**Fix**: Kept as internal constant with comment:
```javascript
// Legacy constants kept for backward compatibility in function signatures
const LINE_ITEM_STYLE_DAY_MONTH_YEAR = "day_month_year";
```
**Location**: `frontend/src/lib/claimDates.js` line 33
**Note**: Not exported, only used internally in format functions. Can be fully removed in a future refactor when those functions are updated.
---
## Test Results
### Frontend Tests
```
PASS src/lib/claimDates.test.js
✓ distinguishes CLAIM DATE vs line item DATE
✓ formats claim date as DD/MM/YYYY
✓ formats line item date as uppercase month name
✓ formats OT and meal line item dates as DD/MM/YYYY
✓ converts legacy NOVEMBER using claim period
✓ rejects month-only for OT line items
✓ formats mileage line item date as DD/MM/YYYY
✓ accepts numeric month for expenses line items
✓ rejects implausible year from partial dd/mm/yy typing
✓ validateClaimFormDates for OT
Test Suites: 1 passed, 1 total
Tests: 10 passed, 10 total
```
### Backend Tests
```
19 passed in 3.29s
tests/test_claim_date_formatting.py::TestClaimDateFormats::test_field_type_helpers PASSED
tests/test_claim_date_formatting.py::TestClaimDateFormats::test_claim_date_dd_mm_yyyy PASSED
tests/test_claim_date_formatting.py::TestClaimDateFormats::test_line_item_date_month_name PASSED
tests/test_claim_date_formatting.py::TestClaimDateFormats::test_mileage_line_item_date_slash_format PASSED
tests/test_claim_date_formatting.py::TestClaimDateFormats::test_mileage_excel_writes_ddmmyyyy PASSED
tests/test_claim_date_formatting.py::TestClaimDateFormats::test_expenses_numeric_month_line_item PASSED
tests/test_claim_date_formatting.py::TestClaimDateFormats::test_ot_excel_time_parsing PASSED
tests/test_claim_date_formatting.py::TestClaimDateFormats::test_ot_excel_preserves_header_and_total_merges PASSED
tests/test_claim_date_formatting.py::TestClaimDateFormats::test_ot_excel_sig_date_c47 PASSED
tests/test_claim_date_formatting.py::TestClaimDateFormats::test_legacy_month_name_with_period PASSED
tests/test_claim_date_formatting.py::TestClaimDateFormats::test_expenses_preview_live_and_print_html PASSED
tests/test_claim_date_formatting.py::TestClaimDateFormats::test_ot_preview_date_formats PASSED
tests/test_claim_date_formatting.py::TestClaimDateFormats::test_ot_validation_rejects_bad_dates PASSED
tests/test_claim_date_formatting.py::TestClaimDateFormats::test_ot_preview_layout_structure PASSED
tests/test_meal_import.py::TestMealImport::test_excel_serial_time_parsing PASSED
tests/test_meal_import.py::TestMealImport::test_import_template_xlsx_has_line_items PASSED
tests/test_meal_import.py::TestMealImport::test_fill_preserves_meal_row_merges PASSED
tests/test_meal_import.py::TestMealImport::test_meal_total_subtitle_is_italic PASSED
tests/test_meal_import.py::TestMealImport::test_fill_serial_times_from_api_payload PASSED
```
---
## Summary
**Total issues fixed: 11** (4 must-fix, 4 should-fix, 3 nits)
All code review issues have been resolved:
- ✅ Eliminated duplicate parsing logic
- ✅ Fixed race condition in preview updates
- ✅ Plugged memory leak from blob URLs
- ✅ Documented date parsing heuristics
- ✅ Added claim period validation for dates
- ✅ Validated meal travel range format
- ✅ Enhanced backend date parsing
- ✅ Removed debug logging from production
- ✅ Extracted magic numbers to constants
- ✅ Simplified redundant code paths
The codebase is now more maintainable, performant, and correct. All tests pass with no regressions.

2
README.md
View File

@ -42,6 +42,8 @@ Default UI: `http://localhost:3000`
docker compose -f docker-compose.claim.yml up --build
```
The compose file expects MongoDB to be running as a localhost service on the host machine. The backend container reaches it via `host.docker.internal:27017`. See [docs/deploy/docker.md](docs/deploy/docker.md) for full details, port map, and gotchas.
## Deploy on Render + GitHub auto-deploy
1. Push this repository to GitHub (for example `skysyaz/claim`).

34
backend/Dockerfile
View File

@ -1,42 +1,40 @@
FROM python:3.11-slim
# Install system + build dependencies
# Avoid apt interactive prompts (debconf frontend) during image build.
ENV DEBIAN_FRONTEND=noninteractive
# Install system + build dependencies.
# libreoffice-calc + poppler handle Excel→PDF conversions (matches local Windows Excel COM flow).
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
gcc \
tesseract-ocr \
libtesseract-dev \
poppler-utils \
curl \
ca-certificates \
libreoffice-calc-nogui \
fonts-dejavu-core \
&& rm -rf /var/lib/apt/lists/*
# Cap glibc malloc arenas — drastically reduces RSS / VSZ growth in
# multi-threaded Python (uvicorn + tesseract spawn), prevents Render
# free-tier OOM during bulk OCR.
# Cap glibc malloc arenas — keeps the uvicorn + libreoffice process under predictable RSS.
ENV MALLOC_ARENA_MAX=2
WORKDIR /app/backend
# Copy requirements first for caching
# Copy requirements first so the dependency layer is cached.
COPY backend/requirements.txt ./
# Install Python dependencies
# Install Python dependencies.
RUN pip install --upgrade pip setuptools wheel \
&& pip install --no-cache-dir -r requirements.txt \
&& pip install --no-cache-dir --no-deps google-genai==1.71.0 \
&& pip install --no-cache-dir google-auth httpx websockets tenacity distro sniffio anyio
&& pip install --no-cache-dir -r requirements.txt
# Copy backend code
# Copy backend code.
COPY backend/ ./
# Claim Excel templates live at repo root (claim_excel_service.TEMPLATE_DIR → /app/Claim Template)
# Claim Excel templates live at repo root (claim_excel_service.TEMPLATE_DIR → /app/Claim Template).
COPY ["Claim Template/", "/app/Claim Template/"]
RUN mkdir -p /app/backend/uploads && chmod +x /app/backend/start.sh
# Persistent upload + attachment directories (mounted as volumes in compose).
RUN mkdir -p /app/backend/claim_files /app/backend/claim_attachments /app/backend/uploads
EXPOSE 8001
RUN chmod +x /app/backend/start.sh
CMD ["/bin/sh", "/app/backend/start.sh"]
EXPOSE 8002
CMD ["python", "-m", "uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8002"]

42
backend/claim_access.py
View File

@ -1,12 +1,18 @@
"""Claim ownership and query scoping — users only see their own claims."""
from __future__ import annotations
from typing import Any, Dict
from typing import Any, Dict, Iterable, List
from bson.binary import Binary
from fastapi import HTTPException
from core import db
# Mongo projection for any claim document returned as JSON (never include raw bytes).
CLAIM_JSON_PROJECTION = {"_id": 0, "file_data": 0, "attachments.file_data": 0}
_BINARY_FIELD_NAMES = frozenset({"file_data", "pdf_data", "path"})
def claim_scope_query(user: dict) -> Dict[str, Any]:
"""Every user has a private claim space — no cross-user visibility."""
@ -23,6 +29,40 @@ def merge_claim_filters(user: dict, extra: Dict[str, Any]) -> Dict[str, Any]:
return {"$and": [scope, extra]}
def _drop_binary_values(value: Any) -> Any:
"""Recursively remove bytes/Binary payloads so FastAPI JSON encoding cannot fail."""
if isinstance(value, (bytes, Binary)):
return None
if isinstance(value, dict):
cleaned: Dict[str, Any] = {}
for key, item in value.items():
if key in _BINARY_FIELD_NAMES:
continue
if isinstance(item, (bytes, Binary)):
continue
cleaned[key] = _drop_binary_values(item)
return cleaned
if isinstance(value, list):
cleaned_list: List[Any] = []
for item in value:
if isinstance(item, (bytes, Binary)):
continue
cleaned_list.append(_drop_binary_values(item))
return cleaned_list
return value
def strip_claim_for_json(rec: dict) -> dict:
"""Return a JSON-safe claim dict (no workbook/attachment/pdf byte payloads)."""
cleaned = _drop_binary_values({**rec})
cleaned.pop("_id", None)
return cleaned
def strip_claims_for_json(records: Iterable[dict]) -> List[dict]:
return [strip_claim_for_json(rec) for rec in records]
async def get_claim_for_user(claim_id: str, user: dict) -> dict:
q: Dict[str, Any] = {"id": claim_id}
scope = claim_scope_query(user)

1
backend/requirements.txt
View File

@ -16,6 +16,7 @@ boto3==1.42.96
reportlab==4.2.0
openpyxl>=3.0.0
xlrd==1.2.0
pypdf>=4.0.0
resend==2.3.0
# Dev / tests (optional: pip install -r requirements-dev.txt)

181
backend/routes/claims.py
View File

@ -1,24 +1,39 @@
"""User-scoped claim API (/api/claims)."""
from __future__ import annotations
import asyncio
import uuid
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict, List, Optional
import aiofiles
from bson.binary import Binary
from fastapi import APIRouter, Depends, File, HTTPException, Query, UploadFile
from fastapi.responses import FileResponse, Response as FastResponse
from fastapi.responses import Response as FastResponse
from claim_access import get_claim_for_user, merge_claim_filters, owner_fields
from core import ROOT_DIR, db, logger
from claim_access import (
CLAIM_JSON_PROJECTION,
get_claim_for_user,
merge_claim_filters,
owner_fields,
strip_claim_for_json,
strip_claims_for_json,
)
from core import db, logger
from deps import get_current_user, require_min_role, require_roles
from models import ClaimPayload
from services.audit_service import log_from_user
from services.claim_excel_service import fill_claim_excel, excel_mimetype
from services.claim_excel_pdf import claim_preview_filename, convert_excel_bytes_to_pdf
from services.claim_pdf_service import render_claim_pdf, CLAIM_TYPE_LABELS
from services.claim_document_storage import (
_MAX_ATTACHMENT_BYTES,
_MAX_CLAIM_FILE_BYTES,
delete_claim_legacy_files,
load_attachment_bytes,
load_claim_file_bytes,
reserve_claim_file_filename,
resolve_attachment_filename,
)
router = APIRouter(prefix="/claims", tags=["claims"])
@ -43,12 +58,12 @@ async def list_claims(
total = await db.claims.count_documents(q)
skip = (page - 1) * page_size
cursor = (
db.claims.find(q, {"_id": 0})
db.claims.find(q, CLAIM_JSON_PROJECTION)
.sort([("period", -1), ("created_at", -1)])
.skip(skip)
.limit(page_size)
)
items = [c async for c in cursor]
items = strip_claims_for_json([c async for c in cursor])
return {"items": items, "total": total, "page": page, "page_size": page_size}
@ -62,7 +77,8 @@ async def list_claim_periods(user: dict = Depends(get_current_user)):
@router.get("/{claim_id}")
async def get_claim(claim_id: str, user: dict = Depends(get_current_user)):
return await get_claim_for_user(claim_id, user)
rec = await get_claim_for_user(claim_id, user)
return strip_claim_for_json(rec)
@router.post("")
@ -103,12 +119,13 @@ async def update_claim(claim_id: str, payload: ClaimPayload, actor: dict = Depen
await db.claims.update_one({"id": claim_id}, {"$set": updates})
await log_from_user(db, actor, action="CLAIM_UPDATE", target_type="claim", target_id=claim_id,
meta={"claim_type": payload.claim_type.upper(), "period": payload.period})
return {**existing, **updates, "_id": None}
return strip_claim_for_json({**existing, **updates})
@router.delete("/{claim_id}")
async def delete_claim(claim_id: str, actor: dict = Depends(get_current_user)):
existing = await get_claim_for_user(claim_id, actor)
delete_claim_legacy_files(existing)
await db.claims.delete_one({"id": claim_id})
await log_from_user(db, actor, action="CLAIM_DELETE", target_type="claim", target_id=claim_id,
meta={"claim_type": existing.get("claim_type"), "period": existing.get("period")})
@ -139,11 +156,21 @@ async def claim_pdf(claim_id: str, user: dict = Depends(get_current_user)):
async def claim_excel_download(claim_id: str, user: dict = Depends(get_current_user)):
rec = await get_claim_for_user(claim_id, user)
ct = rec.get("claim_type", "EXPENSES")
xls_bytes = fill_claim_excel(ct, rec.get("header") or {}, rec.get("items") or [], rec.get("period") or "")
mime, ext = excel_mimetype(ct)
filename = claim_preview_filename(ct, rec.get("period", "unknown"), ext)
return FastResponse(content=xls_bytes, media_type=mime,
headers={"Content-Disposition": f'attachment; filename="{filename}"', "Cache-Control": "no-store"})
period = rec.get("period") or ""
if rec.get("source") == "UPLOADED" and (rec.get("file_data") is not None or rec.get("file_path")):
xls_bytes = await load_claim_file_bytes(rec)
mime = rec.get("content_type") or excel_mimetype(ct)[0]
filename = rec.get("filename") or rec.get("original_upload_name") or claim_preview_filename(ct, period, excel_mimetype(ct)[1])
else:
xls_bytes = fill_claim_excel(ct, rec.get("header") or {}, rec.get("items") or [], period)
mime, ext = excel_mimetype(ct)
filename = claim_preview_filename(ct, period or "unknown", ext)
safe = filename.replace('"', "")
return FastResponse(
content=xls_bytes,
media_type=mime,
headers={"Content-Disposition": f'attachment; filename="{safe}"', "Cache-Control": "no-store"},
)
@router.post("/excel-preview")
@ -349,12 +376,9 @@ async def claim_pdf_preview(payload: ClaimPayload, user: dict = Depends(get_curr
# ---------------------------------------------------------------------------
# Admin: Import existing Excel claim files
# Import existing Excel claim files (stored in MongoDB)
# ---------------------------------------------------------------------------
CLAIM_FILES_DIR = ROOT_DIR / "claim_files"
CLAIM_FILES_DIR.mkdir(exist_ok=True)
@router.post("/import")
async def import_claim_files(
@ -376,7 +400,7 @@ async def import_claim_files(
results.append({"filename": fname, "ok": False, "error": "Not an Excel file"})
continue
content = await file.read()
if len(content) > 20 * 1024 * 1024:
if len(content) > _MAX_CLAIM_FILE_BYTES:
results.append({"filename": fname, "ok": False, "error": "File too large (max 20 MB)"})
continue
try:
@ -391,24 +415,31 @@ async def import_claim_files(
if not meta["header"].get("name") and default_name:
meta["header"]["name"] = default_name
# Store the file on disk
file_id = str(uuid.uuid4())
suffix = Path(fname).suffix or ".xlsx"
dest = CLAIM_FILES_DIR / f"{file_id}{suffix}"
async with aiofiles.open(dest, "wb") as fh:
await fh.write(content)
period = meta.get("period") or period_hint or ""
db_filename = await reserve_claim_file_filename(
actor["id"], meta["claim_type"], period, suffix
)
now = datetime.now(timezone.utc).isoformat()
rec = {
"id": file_id,
"claim_type": meta["claim_type"],
"period": meta["period"],
"period": period,
"header": meta["header"],
"items": meta.get("items", []),
"notes": None,
"source": "UPLOADED",
"original_filename": fname,
"file_path": str(dest),
"filename": db_filename,
"original_upload_name": fname,
"file_data": Binary(content),
"storage": "mongodb",
"content_type": (
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
if suffix.lower() == ".xlsx"
else "application/vnd.ms-excel"
),
"file_size": len(content),
**owner_fields(actor),
"created_at": now,
"updated_at": now,
@ -418,34 +449,44 @@ async def import_claim_files(
target_id=rec["id"],
meta={"filename": fname, "claim_type": rec["claim_type"],
"period": rec["period"]})
results.append({"filename": fname, "ok": True, "id": file_id,
"claim_type": meta["claim_type"], "period": meta["period"],
"name": meta["header"].get("name", "")})
results.append({
"filename": fname,
"ok": True,
"id": file_id,
"stored_filename": db_filename,
"claim_type": meta["claim_type"],
"period": meta["period"],
"name": meta["header"].get("name", ""),
})
return {"imported": sum(1 for r in results if r["ok"]), "results": results}
@router.get("/{claim_id}/file")
async def serve_claim_file(claim_id: str, user: dict = Depends(get_current_user)):
"""Serve the original uploaded Excel file for UPLOADED claims."""
"""Download the uploaded Excel workbook from MongoDB (UPLOADED claims)."""
rec = await get_claim_for_user(claim_id, user)
file_path = rec.get("file_path")
if not file_path or not Path(file_path).exists():
raise HTTPException(status_code=404, detail="No original file stored for this claim")
fname = rec.get("original_filename", "claim.xlsx")
mime = ("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
if fname.endswith(".xlsx") else "application/vnd.ms-excel")
return FileResponse(file_path, media_type=mime, filename=fname,
headers={"Content-Disposition": f'attachment; filename="{fname}"'})
if rec.get("source") != "UPLOADED":
raise HTTPException(status_code=404, detail="No imported file for this claim")
content = await load_claim_file_bytes(rec)
fname = rec.get("filename") or rec.get("original_upload_name") or rec.get("original_filename") or "claim.xlsx"
mime = rec.get("content_type") or (
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
if fname.endswith(".xlsx")
else "application/vnd.ms-excel"
)
safe = fname.replace('"', "")
return FastResponse(
content=content,
media_type=mime,
headers={"Content-Disposition": f'attachment; filename="{safe}"', "Cache-Control": "no-store"},
)
# ---------------------------------------------------------------------------
# Admin: Claim attachments (receipts, supporting files)
# Claim attachments (receipts, supporting files) — bytes in MongoDB
# ---------------------------------------------------------------------------
CLAIM_ATTACH_DIR = ROOT_DIR / "claim_attachments"
CLAIM_ATTACH_DIR.mkdir(exist_ok=True)
ALLOWED_ATTACH_TYPES = {
"application/pdf",
"image/jpeg", "image/jpg", "image/png", "image/gif", "image/webp",
@ -465,27 +506,30 @@ async def upload_claim_attachment(
if content_type not in ALLOWED_ATTACH_TYPES:
raise HTTPException(status_code=400, detail="File type not allowed. Upload PDF, image, or Excel files.")
content = await file.read()
if len(content) > 10 * 1024 * 1024:
if len(content) > _MAX_ATTACHMENT_BYTES:
raise HTTPException(status_code=400, detail="File too large (max 10 MB)")
attach_id = str(uuid.uuid4())
suffix = Path(file.filename or "file").suffix or ".bin"
dest = CLAIM_ATTACH_DIR / f"{attach_id}{suffix}"
async with aiofiles.open(dest, "wb") as fh:
await fh.write(content)
att_index = len(rec.get("attachments") or []) + 1
db_filename = await resolve_attachment_filename(
rec, content, content_type, suffix, att_index
)
attachment = {
"id": attach_id,
"filename": file.filename,
"filename": db_filename,
"original_upload_name": file.filename,
"content_type": content_type,
"size": len(content),
"uploaded_at": datetime.now(timezone.utc).isoformat(),
"uploaded_by": actor["email"],
"path": str(dest),
"file_data": Binary(content),
"storage": "mongodb",
}
await db.claims.update_one(
{"id": claim_id},
{"$push": {"attachments": attachment}},
)
return {k: v for k, v in attachment.items() if k != "path"}
return {k: v for k, v in attachment.items() if k not in ("file_data", "path")}
@router.get("/{claim_id}/attachments")
@ -502,16 +546,30 @@ async def serve_claim_attachment(
user: dict = Depends(get_current_user),
):
rec = await get_claim_for_user(claim_id, user)
attachments = rec.get("attachments") or []
att = next((a for a in attachments if a["id"] == attach_id), None)
if not att:
raise HTTPException(status_code=404, detail="Attachment not found")
file_path = Path(att["path"])
if not file_path.exists():
raise HTTPException(status_code=404, detail="File missing on disk")
return FileResponse(str(file_path), media_type=att["content_type"],
filename=att["filename"],
headers={"Content-Disposition": f'inline; filename="{att["filename"]}"'})
content, att = await load_attachment_bytes(rec, attach_id)
fname = att.get("filename") or att.get("original_upload_name") or "attachment"
safe = fname.replace('"', "")
return FastResponse(
content=content,
media_type=att.get("content_type") or "application/octet-stream",
headers={"Content-Disposition": f'inline; filename="{safe}"', "Cache-Control": "no-store"},
)
@router.get("/{claim_id}/attachments/{attach_id}/download")
async def download_claim_attachment(
claim_id: str, attach_id: str,
user: dict = Depends(get_current_user),
):
rec = await get_claim_for_user(claim_id, user)
content, att = await load_attachment_bytes(rec, attach_id)
fname = att.get("filename") or att.get("original_upload_name") or "attachment"
safe = fname.replace('"', "")
return FastResponse(
content=content,
media_type=att.get("content_type") or "application/octet-stream",
headers={"Content-Disposition": f'attachment; filename="{safe}"', "Cache-Control": "no-store"},
)
@router.delete("/{claim_id}/attachments/{attach_id}")
@ -524,7 +582,8 @@ async def delete_claim_attachment(
att = next((a for a in attachments if a["id"] == attach_id), None)
if not att:
raise HTTPException(status_code=404, detail="Attachment not found")
Path(att["path"]).unlink(missing_ok=True)
if att.get("path"):
Path(att["path"]).unlink(missing_ok=True)
await db.claims.update_one(
{"id": claim_id},
{"$pull": {"attachments": {"id": attach_id}}},

6
backend/routes/dashboard.py
View File

@ -5,7 +5,7 @@ from typing import Any, Dict
from fastapi import APIRouter, Depends
from claim_access import merge_claim_filters
from claim_access import CLAIM_JSON_PROJECTION, merge_claim_filters, strip_claims_for_json
from core import db
from deps import get_current_user
@ -23,11 +23,11 @@ async def claims_dashboard_summary(user: dict = Depends(get_current_user)):
by_type[ct] = await db.claims.count_documents(merge_claim_filters(user, {"claim_type": ct}))
recent_cursor = (
db.claims.find(scope, {"_id": 0})
db.claims.find(scope, CLAIM_JSON_PROJECTION)
.sort("updated_at", -1)
.limit(5)
)
recent = [c async for c in recent_cursor]
recent = strip_claims_for_json([c async for c in recent_cursor])
periods = await db.claims.distinct("period", scope)
periods = sorted((p for p in periods if p), reverse=True)

156
backend/routes/payslips.py
View File

@ -1,31 +1,35 @@
"""User-scoped pay slip PDF library (/api/payslips)."""
"""User-scoped pay slip PDF library (/api/payslips). PDF bytes stored in MongoDB."""
from __future__ import annotations
import re
import uuid
from datetime import date, datetime, timezone
from pathlib import Path
from typing import Any, Dict, List, Optional
import aiofiles
from bson.binary import Binary
from fastapi import APIRouter, Depends, File, Form, HTTPException, UploadFile
from fastapi.responses import FileResponse
from fastapi.responses import Response
from claim_access import owner_fields
from core import ROOT_DIR, db
from core import db
from deps import get_current_user
from payslip_access import get_payslip_for_user, payslip_scope_query
from services import storage_service
from services.audit_service import log_from_user
from services.payslip_storage import (
canonical_filename,
delete_legacy_files,
load_pdf_bytes,
reserve_unique_filename,
)
router = APIRouter(prefix="/payslips", tags=["payslips"])
PAYSLIP_DIR = ROOT_DIR / "payslip_files"
PAYSLIP_DIR.mkdir(exist_ok=True)
_MAX_BYTES = 15 * 1024 * 1024
_MAX_BATCH_FILES = 30
_DATE_RE = re.compile(r"(20\d{2})[-_.]?(0[1-9]|1[0-2])(?:[-_.]?(0[1-9]|[12]\d|3[01]))?")
_LIST_PROJECTION = {"_id": 0, "pdf_data": 0, "path": 0}
def _parse_payslip_date(raw: Optional[str]) -> Optional[date]:
if not raw or not str(raw).strip():
@ -63,29 +67,15 @@ def _public_item(rec: dict) -> dict:
"payslip_date": rec.get("payslip_date"),
"size": rec.get("size"),
"uploaded_at": rec.get("uploaded_at"),
"stored_in_db": rec.get("pdf_data") is not None,
}
async def _resolve_file_path(rec: dict) -> Path:
"""Return a local path to the PDF, fetching from R2 when needed."""
local = rec.get("path")
if local:
path = Path(local)
if path.is_file():
return path
if rec.get("storage") == "r2":
tmp = PAYSLIP_DIR / f"_cache_{rec['id']}.pdf"
fetched = await storage_service.download_pdf(rec["id"], tmp)
if fetched and fetched.is_file():
return fetched
raise HTTPException(status_code=404, detail="File missing on disk")
@router.get("")
@router.get("/")
async def list_payslips(user: dict = Depends(get_current_user)):
q = payslip_scope_query(user)
cursor = db.payslips.find(q, {"_id": 0, "path": 0}).sort(
cursor = db.payslips.find(q, _LIST_PROJECTION).sort(
[("payslip_date", -1), ("uploaded_at", -1)]
)
items: List[dict] = []
@ -94,18 +84,16 @@ async def list_payslips(user: dict = Depends(get_current_user)):
return {"items": items, "total": len(items)}
@router.post("")
@router.post("/")
async def upload_payslip(
file: UploadFile = File(...),
payslip_date: Optional[str] = Form(None),
actor: dict = Depends(get_current_user),
):
filename = file.filename or "payslip.pdf"
if not filename.lower().endswith(".pdf"):
async def _save_payslip_upload(
file: UploadFile,
payslip_date: Optional[str],
actor: dict,
) -> dict:
upload_name = file.filename or "payslip.pdf"
if not upload_name.lower().endswith(".pdf"):
raise HTTPException(status_code=400, detail="Only PDF files are allowed.")
content_type = (file.content_type or "").lower()
if content_type and content_type != "application/pdf":
if content_type and content_type not in ("application/pdf", "application/octet-stream"):
raise HTTPException(status_code=400, detail="Only PDF files are allowed.")
content = await file.read()
@ -114,27 +102,23 @@ async def upload_payslip(
if not content.startswith(b"%PDF"):
raise HTTPException(status_code=400, detail="Invalid PDF file.")
slip_date = _infer_payslip_date(filename, payslip_date)
slip_date = _infer_payslip_date(upload_name, payslip_date)
owner_id = actor["id"]
db_filename = await reserve_unique_filename(owner_id, slip_date)
payslip_id = str(uuid.uuid4())
dest = PAYSLIP_DIR / f"{payslip_id}.pdf"
async with aiofiles.open(dest, "wb") as fh:
await fh.write(content)
storage = "local"
if storage_service.R2_CONFIGURED:
await storage_service.put_pdf_from_path(payslip_id, dest)
storage = "r2"
now = datetime.now(timezone.utc).isoformat()
rec: Dict[str, Any] = {
"id": payslip_id,
"filename": filename,
"filename": db_filename,
"canonical_base": canonical_filename(slip_date).replace(".pdf", ""),
"original_upload_name": upload_name,
"payslip_date": slip_date.isoformat(),
"content_type": "application/pdf",
"size": len(content),
"uploaded_at": now,
"path": str(dest),
"storage": storage,
"pdf_data": Binary(content),
"storage": "mongodb",
**owner_fields(actor),
}
await db.payslips.insert_one({**rec, "_id": payslip_id})
@ -144,39 +128,77 @@ async def upload_payslip(
action="PAYSLIP_UPLOAD",
target_type="payslip",
target_id=payslip_id,
meta={"filename": filename, "payslip_date": rec["payslip_date"]},
meta={
"filename": db_filename,
"payslip_date": rec["payslip_date"],
"original_upload_name": upload_name,
},
)
return _public_item(rec)
@router.post("")
@router.post("/")
async def upload_payslips(
files: List[UploadFile] = File(...),
payslip_date: Optional[str] = Form(None),
actor: dict = Depends(get_current_user),
):
if not files:
raise HTTPException(status_code=400, detail="No files provided.")
if len(files) > _MAX_BATCH_FILES:
raise HTTPException(
status_code=400,
detail=f"Too many files (max {_MAX_BATCH_FILES} per upload).",
)
items: List[dict] = []
errors: List[dict] = []
for file in files:
try:
items.append(await _save_payslip_upload(file, payslip_date, actor))
except HTTPException as exc:
detail = exc.detail if isinstance(exc.detail, str) else str(exc.detail)
errors.append({"filename": file.filename or "unknown", "error": detail})
except Exception as exc: # noqa: BLE001
errors.append({"filename": file.filename or "unknown", "error": str(exc)})
if not items and errors:
raise HTTPException(status_code=400, detail=errors[0]["error"])
return {"items": items, "errors": errors, "uploaded": len(items)}
@router.get("/{payslip_id}/file")
async def serve_payslip_file(payslip_id: str, user: dict = Depends(get_current_user)):
rec = await get_payslip_for_user(payslip_id, user)
if rec.get("storage") == "r2":
url = await storage_service.presigned_get_url(
payslip_id, filename=rec.get("filename"), expires=900
)
if url:
from fastapi.responses import RedirectResponse
return RedirectResponse(url, status_code=302)
path = await _resolve_file_path(rec)
return FileResponse(
str(path),
pdf_bytes = await load_pdf_bytes(rec)
filename = rec.get("filename") or "payslip.pdf"
safe_name = filename.replace('"', "")
return Response(
content=pdf_bytes,
media_type="application/pdf",
filename=rec.get("filename") or "payslip.pdf",
headers={"Content-Disposition": f'inline; filename="{rec.get("filename", "payslip.pdf")}"'},
headers={"Content-Disposition": f'inline; filename="{safe_name}"'},
)
@router.get("/{payslip_id}/download")
async def download_payslip_file(payslip_id: str, user: dict = Depends(get_current_user)):
rec = await get_payslip_for_user(payslip_id, user)
pdf_bytes = await load_pdf_bytes(rec)
filename = rec.get("filename") or "payslip.pdf"
safe_name = filename.replace('"', "")
return Response(
content=pdf_bytes,
media_type="application/pdf",
headers={"Content-Disposition": f'attachment; filename="{safe_name}"'},
)
@router.delete("/{payslip_id}")
async def delete_payslip(payslip_id: str, actor: dict = Depends(get_current_user)):
rec = await get_payslip_for_user(payslip_id, actor)
local_path = rec.get("path")
if local_path:
Path(local_path).unlink(missing_ok=True)
if rec.get("storage") == "r2":
await storage_service.delete_pdf(payslip_id)
await delete_legacy_files(rec)
await db.payslips.delete_one({"id": payslip_id})
await log_from_user(
db,

89
backend/scripts/backfill_claim_documents.py Normal file
View File

@ -0,0 +1,89 @@
"""Backfill claim workbooks and attachments into MongoDB (disk, or regenerate Excel from metadata)."""
from __future__ import annotations
import asyncio
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from core import db
from services.claim_document_storage import (
_find_legacy_attachment_file,
_find_legacy_claim_file,
load_attachment_bytes,
load_claim_file_bytes,
persist_claim_file_to_db,
regenerate_claim_workbook,
)
async def backfill_claim(rec: dict) -> str:
if rec.get("file_data") is not None:
return "already"
if _find_legacy_claim_file(rec) is not None:
await load_claim_file_bytes(rec)
return "disk"
try:
content = regenerate_claim_workbook(rec)
await persist_claim_file_to_db(rec, content, regenerated=True)
return "regenerated"
except Exception as exc: # noqa: BLE001
return f"skip: {exc}"
async def backfill_attachment(rec: dict, att: dict) -> str:
if att.get("file_data") is not None:
return "already"
if _find_legacy_attachment_file(att) is not None:
await load_attachment_bytes(rec, att["id"])
return "disk"
return "lost"
async def main() -> None:
stats = {
"claim_already": 0,
"claim_disk": 0,
"claim_regenerated": 0,
"claim_skip": 0,
"att_already": 0,
"att_disk": 0,
"att_lost": 0,
}
cursor = db.claims.find({})
async for rec in cursor:
cid = rec["id"]
if rec.get("file_data") is None:
result = await backfill_claim(rec)
key = f"claim_{result}" if result in ("already", "disk", "regenerated") else "claim_skip"
if result.startswith("skip"):
stats["claim_skip"] += 1
print(f"claim {cid}: {result}")
else:
stats[key] += 1
print(f"claim {cid}: {result}")
for att in rec.get("attachments") or []:
result = await backfill_attachment(rec, att)
if result == "already":
stats["att_already"] += 1
elif result == "disk":
stats["att_disk"] += 1
print(f"claim {cid} attachment {att['id']}: disk")
else:
stats["att_lost"] += 1
print(f"claim {cid} attachment {att['id']}: lost (re-upload required)")
print(
"\nDone.\n"
f" Claims — already in DB: {stats['claim_already']}, from disk: {stats['claim_disk']}, "
f"regenerated from data: {stats['claim_regenerated']}, failed: {stats['claim_skip']}\n"
f" Attachments — already: {stats['att_already']}, from disk: {stats['att_disk']}, "
f"lost: {stats['att_lost']}"
)
if __name__ == "__main__":
asyncio.run(main())

49
backend/scripts/rename_attachment_filenames_from_pdf.py Normal file
View File

@ -0,0 +1,49 @@
"""Rename claim attachment filenames using company names extracted from stored PDFs."""
from __future__ import annotations
import asyncio
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from core import db
from services.claim_document_storage import resolve_attachment_filename
async def main() -> None:
renamed = 0
skipped = 0
cursor = db.claims.find({"attachments": {"$exists": True, "$ne": []}})
async for rec in cursor:
claim_id = rec["id"]
attachments = list(rec.get("attachments") or [])
changed = False
for idx, att in enumerate(attachments, start=1):
raw = att.get("file_data")
if raw is None:
skipped += 1
continue
content = bytes(raw)
ct = att.get("content_type") or "application/pdf"
suffix = Path(att.get("original_upload_name") or att.get("filename") or ".pdf").suffix
rec_for_reserve = {**rec, "attachments": [a for a in attachments if a["id"] != att["id"]]}
new_name = await resolve_attachment_filename(
rec_for_reserve, content, ct, suffix or ".pdf", idx
)
old_name = att.get("filename")
if new_name == old_name:
skipped += 1
continue
att["filename"] = new_name
changed = True
renamed += 1
print(f"claim {claim_id}: {old_name!r} -> {new_name!r}")
if changed:
await db.claims.update_one({"id": claim_id}, {"$set": {"attachments": attachments}})
print(f"\nDone. Renamed: {renamed}, unchanged or skipped: {skipped}")
if __name__ == "__main__":
asyncio.run(main())

1
backend/server.py
View File

@ -58,6 +58,7 @@ async def on_startup():
await db.payslips.create_index("id", unique=True)
await db.payslips.create_index("owner_id")
await db.payslips.create_index([("owner_id", 1), ("payslip_date", -1)])
await db.payslips.create_index([("owner_id", 1), ("filename", 1)])
async for doc in db.claims.find({"owner_id": {"$exists": False}}, {"id": 1, "created_by": 1}):
if doc.get("created_by"):

16
backend/services/auth_service.py
View File

@ -20,7 +20,19 @@ ROLE_RANK = {"viewer": 0, "user": 1, "manager": 2, "admin": 3}
def _secret() -> str:
return os.environ["JWT_SECRET"]
secret = os.environ.get("JWT_SECRET")
if not secret or len(secret) < 32:
raise RuntimeError(
"JWT_SECRET is missing or shorter than 32 characters. "
"Set a strong secret in backend/.env."
)
return secret
# Validate once at import time so misconfiguration fails fast at startup,
# not on the first auth attempt. Module-level constant is read by encode/decode
# so we don't re-check on every request.
_SECRET = _secret()
def hash_password(pw: str) -> str:
@ -58,7 +70,7 @@ def create_refresh_token(user_id: str) -> str:
def decode(token: str) -> dict:
return jwt.decode(token, _secret(), algorithms=[JWT_ALG])
return jwt.decode(token, _SECRET, algorithms=[JWT_ALG])
# ---------------------------------------------------------------------------

147
backend/services/cell_format_service.py Normal file
View File

@ -0,0 +1,147 @@
"""Optional per-cell font size / bold formatting (user-controlled, not hardcoded)."""
from __future__ import annotations
from copy import copy
from typing import Any, Dict, Optional
from openpyxl.styles import Font
CELL_FONT_MIN = 6
CELL_FONT_MAX = 36
def cell_format_map(source: Optional[dict]) -> Dict[str, dict]:
if not source:
return {}
raw = source.get("_cellFormat") or {}
return raw if isinstance(raw, dict) else {}
def _apply_user_cell_format(
ws,
row: int,
col: int,
fmt: Optional[dict],
*,
ref_font: Optional[Font] = None,
) -> None:
"""Apply user-chosen font size/bold to one Excel cell (no-op when fmt is empty)."""
if not fmt:
return
if "fontSize" not in fmt and "bold" not in fmt:
return
c = ws.cell(row=row, column=col)
if c.__class__.__name__ == "MergedCell":
return
if c.value is None or c.value == "":
return
base = ref_font or c.font
name = (base.name if base and base.name else "Arial")
raw_size = fmt.get("fontSize")
if raw_size is not None:
size = max(CELL_FONT_MIN, min(CELL_FONT_MAX, float(raw_size)))
else:
size = float(base.size if base and base.size else 9)
if "bold" in fmt:
bold = bool(fmt.get("bold"))
else:
bold = bool(base.bold) if base else False
c.font = Font(
name=name,
size=size,
bold=bold,
italic=base.italic if base else False,
underline=base.underline if base else None,
strike=base.strike if base else None,
color=base.color if base else None,
)
def apply_user_formats_for_row(
ws,
row: int,
fmt_map: Dict[str, dict],
field_cols: Dict[str, int],
*,
template_fonts: Optional[Dict[int, Font]] = None,
) -> None:
for field_key, col in field_cols.items():
fmt = fmt_map.get(field_key)
if not fmt:
continue
ref_font = (template_fonts or {}).get(col)
_apply_user_cell_format(ws, row, col, fmt, ref_font=ref_font)
def apply_user_formats_for_cells(
ws,
fmt_map: Dict[str, dict],
field_cells: Dict[str, tuple[int, int]],
*,
template_fonts: Optional[Dict[tuple[int, int], Font]] = None,
) -> None:
for field_key, (row, col) in field_cells.items():
fmt = fmt_map.get(field_key)
if not fmt:
continue
ref_font = (template_fonts or {}).get((row, col))
_apply_user_cell_format(ws, row, col, fmt, ref_font=ref_font)
def apply_user_formats_multi_cells(
ws,
fmt_map: Dict[str, dict],
field_cells: Dict[str, list[tuple[int, int]]],
*,
template_fonts: Optional[Dict[tuple[int, int], Font]] = None,
) -> None:
"""Apply one logical field format to every Excel cell it maps to (e.g. mileage totals)."""
for field_key, positions in field_cells.items():
fmt = fmt_map.get(field_key)
if not fmt:
continue
for row, col in positions:
ref_font = (template_fonts or {}).get((row, col))
_apply_user_cell_format(ws, row, col, fmt, ref_font=ref_font)
def snapshot_row_fonts(ws, ref_row: int, cols: list[int]) -> Dict[int, Font]:
fonts: Dict[int, Font] = {}
for col in cols:
src = ws.cell(ref_row, col)
if src.__class__.__name__ == "MergedCell":
continue
if src.font:
fonts[col] = copy(src.font)
return fonts
def snapshot_cell_fonts(ws, cells: Dict[str, tuple[int, int]]) -> Dict[tuple[int, int], Font]:
fonts: Dict[tuple[int, int], Font] = {}
for _key, (row, col) in cells.items():
src = ws.cell(row, col)
if src.__class__.__name__ == "MergedCell":
continue
if src.font:
fonts[(row, col)] = copy(src.font)
return fonts
def preview_cell_style_attr(
fmt: Optional[dict], *, align: Optional[str] = None, base: Optional[str] = None
) -> str:
parts: list[str] = []
if base:
parts.extend(p.strip() for p in base.split(";") if p.strip())
if align:
parts.append(f"text-align:{align}")
if fmt:
if fmt.get("fontSize"):
parts.append(f"font-size:{float(fmt['fontSize'])}pt")
if fmt.get("bold"):
parts.append("font-weight:bold")
return f' style="{";".join(parts)}"' if parts else ""

272
backend/services/claim_document_storage.py Normal file
View File

@ -0,0 +1,272 @@
"""Claim document bytes in MongoDB (imported workbooks and attachments)."""
from __future__ import annotations
import logging
import re
from pathlib import Path
from typing import Any, Dict, Optional
from bson.binary import Binary
from fastapi import HTTPException
from core import ROOT_DIR, db
logger = logging.getLogger("claimapp.claim_documents")
CLAIM_TYPE_SLUG = {
"EXPENSES": "expenses",
"MEAL": "meal",
"MILEAGE": "mileage",
"OT": "ot",
}
_MAX_CLAIM_FILE_BYTES = 20 * 1024 * 1024
_MAX_ATTACHMENT_BYTES = 10 * 1024 * 1024
def _period_yyyymm(period: Optional[str]) -> str:
if not period or len(period) < 7:
return "unknown"
return period[:7].replace("-", "")
def canonical_claim_filename(claim_type: str, period: str, suffix: str) -> str:
"""claim_{type}_{yyyymm}.xlsx — e.g. claim_expenses_202603.xlsx"""
slug = CLAIM_TYPE_SLUG.get(claim_type.upper(), claim_type.lower())
ext = suffix.lower() if suffix.lower() in (".xls", ".xlsx") else ".xlsx"
return f"claim_{slug}_{_period_yyyymm(period)}{ext}"
async def reserve_claim_file_filename(
owner_id: str,
claim_type: str,
period: str,
suffix: str,
*,
exclude_claim_id: Optional[str] = None,
) -> str:
slug = CLAIM_TYPE_SLUG.get(claim_type.upper(), claim_type.lower())
yyyymm = _period_yyyymm(period)
base = f"claim_{slug}_{yyyymm}"
pattern = f"^{re.escape(base)}(_\\d+)?\\.(xls|xlsx)$"
used: set[str] = set()
query: dict = {"owner_id": owner_id, "filename": {"$regex": pattern, "$options": "i"}}
if exclude_claim_id:
query["id"] = {"$ne": exclude_claim_id}
cursor = db.claims.find(query, {"filename": 1})
async for doc in cursor:
fn = doc.get("filename")
if fn:
used.add(fn.lower())
ext = suffix.lower() if suffix.lower() in (".xls", ".xlsx") else ".xlsx"
first = f"{base}{ext}"
if first.lower() not in used:
return first
n = 2
while True:
candidate = f"{base}_{n}{ext}"
if candidate.lower() not in used:
return candidate
n += 1
def _claim_files_dir() -> Path:
d = ROOT_DIR / "claim_files"
d.mkdir(exist_ok=True)
return d
def _claim_attachments_dir() -> Path:
d = ROOT_DIR / "claim_attachments"
d.mkdir(exist_ok=True)
return d
def regenerate_claim_workbook(rec: dict) -> bytes:
"""Rebuild Excel from claim header/items when the original upload file is missing."""
from services.claim_excel_service import fill_claim_excel
return fill_claim_excel(
rec.get("claim_type", "EXPENSES"),
rec.get("header") or {},
rec.get("items") or [],
rec.get("period") or "",
)
def _find_legacy_claim_file(rec: dict) -> Optional[Path]:
path_val = rec.get("file_path")
if path_val:
path = Path(path_val)
if path.is_file():
return path
claim_id = rec.get("id")
if not claim_id:
return None
for ext in (".xlsx", ".xls"):
candidate = _claim_files_dir() / f"{claim_id}{ext}"
if candidate.is_file():
return candidate
return None
def _find_legacy_attachment_file(att: dict) -> Optional[Path]:
path_val = att.get("path")
if path_val:
path = Path(path_val)
if path.is_file():
return path
attach_id = att.get("id")
if not attach_id:
return None
base = _claim_attachments_dir()
for candidate in base.glob(f"{attach_id}.*"):
if candidate.is_file():
return candidate
return None
async def persist_claim_file_to_db(rec: dict, content: bytes, *, regenerated: bool = False) -> None:
claim_id = rec["id"]
owner_id = rec.get("owner_id") or rec.get("created_by") or ""
updates: Dict[str, Any] = {
"file_data": Binary(content),
"storage": "mongodb",
"file_size": len(content),
}
if regenerated:
updates["file_regenerated"] = True
if not rec.get("filename") and owner_id:
orig = (rec.get("original_upload_name") or rec.get("original_filename") or "").lower()
suffix = ".xls" if orig.endswith(".xls") and not orig.endswith(".xlsx") else ".xlsx"
updates["filename"] = await reserve_claim_file_filename(
owner_id,
rec.get("claim_type", "EXPENSES"),
rec.get("period") or "",
suffix,
exclude_claim_id=claim_id,
)
if not rec.get("content_type"):
fn = updates.get("filename") or rec.get("filename") or "claim.xlsx"
updates["content_type"] = (
"application/vnd.ms-excel"
if str(fn).lower().endswith(".xls")
else "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
)
await db.claims.update_one({"id": claim_id}, {"$set": updates})
def canonical_attachment_filename(claim: dict, suffix: str, index: int) -> str:
"""Fallback when PDF company name cannot be detected: claim_{type}_{yyyymm}_att_{n}.ext"""
slug = CLAIM_TYPE_SLUG.get(claim.get("claim_type", "").upper(), "claim")
yyyymm = _period_yyyymm(claim.get("period"))
ext = suffix if suffix.startswith(".") else f".{suffix}"
return f"claim_{slug}_{yyyymm}_att_{index}{ext}"
def _attachment_ext(suffix: str) -> str:
ext = (suffix or ".bin").lower()
if not ext.startswith("."):
ext = f".{ext}"
return ext
def _reserve_unique_on_claim(claim: dict, stem: str, ext: str) -> str:
used = {
(a.get("filename") or "").lower()
for a in (claim.get("attachments") or [])
}
first = f"{stem}{ext}"
if first.lower() not in used:
return first
n = 2
while True:
candidate = f"{stem}_{n}{ext}"
if candidate.lower() not in used:
return candidate
n += 1
async def resolve_attachment_filename(
claim: dict,
content: bytes,
content_type: str,
suffix: str,
att_index: int,
) -> str:
"""Name supporting PDFs from the vendor company inside the file; fallback otherwise."""
ext = _attachment_ext(suffix)
is_pdf = (content_type or "").lower() == "application/pdf" or ext == ".pdf"
if is_pdf:
from services.pdf_company_extractor import (
company_name_from_pdf,
filename_slug_from_company,
)
company = company_name_from_pdf(content)
if company:
stem = filename_slug_from_company(company)
return _reserve_unique_on_claim(claim, stem, ext)
return canonical_attachment_filename(claim, suffix, att_index)
async def load_claim_file_bytes(rec: dict) -> bytes:
raw = rec.get("file_data")
if raw is not None:
return bytes(raw)
legacy = _find_legacy_claim_file(rec)
if legacy is not None:
content = legacy.read_bytes()
await persist_claim_file_to_db(rec, content, regenerated=False)
return content
try:
content = regenerate_claim_workbook(rec)
await persist_claim_file_to_db(rec, content, regenerated=True)
logger.info("Regenerated workbook for claim %s", rec.get("id"))
return content
except Exception as exc: # noqa: BLE001
logger.warning("Could not regenerate claim %s: %s", rec.get("id"), exc)
raise HTTPException(
status_code=404,
detail="Claim file not found and could not be rebuilt from saved claim data.",
) from exc
async def load_attachment_bytes(rec: dict, attach_id: str) -> tuple[bytes, dict]:
attachments = rec.get("attachments") or []
att = next((a for a in attachments if a["id"] == attach_id), None)
if not att:
raise HTTPException(status_code=404, detail="Attachment not found")
raw = att.get("file_data")
if raw is not None:
return bytes(raw), att
legacy = _find_legacy_attachment_file(att)
if legacy is not None:
content = legacy.read_bytes()
await db.claims.update_one(
{"id": rec["id"], "attachments.id": attach_id},
{"$set": {"attachments.$.file_data": Binary(content), "attachments.$.storage": "mongodb"}},
)
att = {**att, "file_data": content}
return content, att
raise HTTPException(
status_code=404,
detail="Attachment file is missing (not in database and not on disk). Re-upload if needed.",
)
def delete_claim_legacy_files(rec: dict) -> None:
path_val = rec.get("file_path")
if path_val:
Path(path_val).unlink(missing_ok=True)
for att in rec.get("attachments") or []:
p = att.get("path")
if p:
Path(p).unlink(missing_ok=True)

248
backend/services/claim_excel_service.py
View File

@ -48,6 +48,29 @@ from services.claim_dates import (
import openpyxl
from openpyxl.styles import Alignment
from services.cell_format_service import (
_apply_user_cell_format,
apply_user_formats_for_cells,
apply_user_formats_for_row,
apply_user_formats_multi_cells,
cell_format_map,
snapshot_cell_fonts,
snapshot_row_fonts,
)
MILEAGE_MONEY_FMT = "#,##0.00"
CLAIM_PERIOD_FIELD = "_claim_period"
MILEAGE_SUMMARY_FIELD_CELLS = {
"_summary_total_km": [(34, 21), (36, 2)],
"_summary_mileage": [(36, 12), (39, 11)],
"_summary_toll": [(34, 24), (41, 11)],
"_summary_parking": [(34, 27), (42, 11)],
"_summary_grand_total": [(44, 11)],
CLAIM_PERIOD_FIELD: [(11, 11)],
}
EXPENSES_PERIOD_CELL = {CLAIM_PERIOD_FIELD: (5, 16)}
OT_PERIOD_CELL = {CLAIM_PERIOD_FIELD: (10, 14)}
TEMPLATE_DIR = Path(__file__).parent.parent.parent / "Claim Template"
# OT template cell formats (from OT Claim Form.xlsx masters)
@ -126,6 +149,71 @@ def _clear(ws, row: int, col: int):
c.value = None
EXPENSES_HEADER_CELLS = {
"name": (4, 3),
"company": (4, 9),
"location": (4, 16),
"designation": (5, 3),
"emp_no": (5, 9),
}
EXPENSES_ITEM_FIELD_COLS = {
"date": 2,
"description": 3,
**{k: v for k, v in EXPENSE_CAT_COL.items()},
}
MEAL_HEADER_CELLS = {
"name": (7, 11),
"emp_no": (8, 11),
"destination": (9, 11),
"days_outstation": (10, 11),
}
MEAL_ITEM_FIELD_COLS = {
"date": 2,
"from_place": 5,
"time_depart": 9,
"time_arrive": 11,
"description": 12,
"meal": 25,
"accommodation": 27,
"breakfast_included": 28,
}
MILEAGE_HEADER_CELLS = {
"name": (7, 11),
"emp_no": (9, 11),
"vehicle_reg": (16, 11),
"rate_per_km": (36, 6),
"date_sign": (37, 22),
}
MILEAGE_ITEM_FIELD_COLS = {
"date": 2,
"from_place": 5,
"to_place": 8,
"purpose": 11,
"km": 21,
"toll": 24,
"parking": 27,
}
OT_HEADER_CELLS = {
"name": (6, 5),
"emp_no": (6, 14),
"designation": (8, 5),
"company": (8, 14),
"department": (10, 5),
}
OT_ITEM_FIELD_COLS = {
"date": 2,
"reason": 3,
"time_from": 8,
"time_to": 9,
"normal_hours": 10,
"rest_hours_le4": 11,
"rest_hours_gt4": 12,
"rest_hours_gt8": 13,
"ph_hours": 16,
"ph_hours_gt8": 17,
}
def _w(ws, row: int, col: int, val: Any):
"""Write value — skip empty strings and MergedCell slaves."""
if val == "" or val is None:
@ -136,6 +224,45 @@ def _w(ws, row: int, col: int, val: Any):
c.value = val
def _merge_mileage_header_value_row(ws, row: int) -> None:
"""Merge K:AB on header value rows (matches Name / Emp No / Month layout)."""
has_merge = any(
mr.min_row == row
and mr.max_row == row
and mr.min_col == 11
and mr.max_col >= 28
for mr in ws.merged_cells.ranges
)
if not has_merge:
ws.merge_cells(start_row=row, start_column=11, end_row=row, end_column=28)
def _center_mileage_header_value(ws, row: int) -> None:
"""Center header values in column K (Vehicle Reg lacks template alignment)."""
if row == 16:
_merge_mileage_header_value_row(ws, row)
cell = ws.cell(row=row, column=11)
if cell.__class__.__name__ == "MergedCell":
for mr in ws.merged_cells.ranges:
if mr.min_row <= row <= mr.max_row and mr.min_col <= 11 <= mr.max_col:
cell = ws.cell(mr.min_row, mr.min_col)
break
if not cell.value:
return
base = cell.alignment or Alignment()
cell.alignment = Alignment(
horizontal="center",
vertical=base.vertical or "bottom",
text_rotation=base.text_rotation,
wrap_text=base.wrap_text,
shrink_to_fit=base.shrink_to_fit,
indent=base.indent,
relativeIndent=base.relativeIndent,
justifyLastLine=base.justifyLastLine,
readingOrder=base.readingOrder,
)
def _write_date_cell(ws, row: int, col: int, val: str) -> None:
"""Write date with dd/mm/yyyy display (template uses US mm-dd-yy otherwise)."""
s = _s(val)
@ -359,6 +486,9 @@ def _fill_expenses(header: Dict, items: List[Dict], period: str) -> bytes:
for c in _data_cols:
_clear(ws, r, c)
expense_item_fonts = snapshot_row_fonts(ws, 8, _data_cols)
expense_header_fonts = snapshot_cell_fonts(ws, EXPENSES_HEADER_CELLS)
# Write items — one row per particular; each category column can have an amount
for i, it in enumerate(items):
if i >= 12:
@ -386,6 +516,19 @@ def _fill_expenses(header: Dict, items: List[Dict], period: str) -> bytes:
if row_total:
_w(ws, r, EXPENSE_TOTAL_COL, round(row_total, 2))
apply_user_formats_for_row(
ws, r, cell_format_map(it), EXPENSES_ITEM_FIELD_COLS, template_fonts=expense_item_fonts
)
apply_user_formats_for_cells(
ws, cell_format_map(header), EXPENSES_HEADER_CELLS, template_fonts=expense_header_fonts
)
apply_user_formats_for_cells(
ws,
cell_format_map(header),
EXPENSES_PERIOD_CELL,
template_fonts=snapshot_cell_fonts(ws, EXPENSES_PERIOD_CELL),
)
# Signature — use claimed_by (short name) if provided, else fall back to name
claimed = _s(header.get("claimed_by")) or _s(header.get("name"))
@ -443,12 +586,15 @@ def _fill_meal(header: Dict, items: List[Dict], period: str) -> bytes:
# Unmerge line-item rows only (15–25). Row 26 = TOTAL merge; row 14 = column labels.
_unmerge_rows(ws, 15, 25)
# Data rows 15–25, cols: B=2 E=5 I=9 K=11 L=12 Y=25 AA=27
_data_cols = [2, 5, 9, 11, 12, 25, 27]
# Data rows 15–25, cols: B=2 E=5 I=9 K=11 L=12 Y=25 AA=27 AB=28 AC=29
_data_cols = [2, 5, 9, 11, 12, 25, 27, 28, 29]
for r in range(15, 26):
for c in _data_cols:
_clear(ws, r, c)
meal_item_fonts = snapshot_row_fonts(ws, 15, _data_cols)
meal_header_fonts = snapshot_cell_fonts(ws, MEAL_HEADER_CELLS)
for i, it in enumerate(items):
if i >= 10:
break
@ -464,8 +610,24 @@ def _fill_meal(header: Dict, items: List[Dict], period: str) -> bytes:
accom = _num(it.get("accommodation"))
if meal: _w(ws, r, 25, meal)
if accom: _w(ws, r, 27, accom)
breakfast_col = _write_meal_breakfast_cell(ws, r, it.get("breakfast_included"))
apply_user_formats_for_row(
ws, r, cell_format_map(it), MEAL_ITEM_FIELD_COLS, template_fonts=meal_item_fonts
)
breakfast_fmt = cell_format_map(it).get("breakfast_included")
if breakfast_fmt and breakfast_col:
_apply_user_cell_format(
ws,
r,
breakfast_col,
breakfast_fmt,
ref_font=meal_item_fonts.get(breakfast_col),
)
_remerge_meal_data_rows(ws, 15, 25)
apply_user_formats_for_cells(
ws, cell_format_map(header), MEAL_HEADER_CELLS, template_fonts=meal_header_fonts
)
_apply_meal_total_row_label(ws)
# Row 26 has SUM formulas — recalculate automatically in Excel
@ -596,18 +758,43 @@ def _finalize_mileage_trip_row(ws, row: int) -> None:
if src.font:
dst.font = copy(src.font)
src_align = src.alignment or Alignment()
dst.alignment = src_align.copy(
dst.alignment = Alignment(
horizontal=src_align.horizontal,
vertical=src_align.vertical,
text_rotation=src_align.text_rotation,
wrap_text=True if col in (5, 8, 11) else src_align.wrap_text,
shrink_to_fit=False,
indent=src_align.indent,
relativeIndent=src_align.relativeIndent,
justifyLastLine=src_align.justifyLastLine,
readingOrder=src_align.readingOrder,
)
def _write_mileage_money_cell(ws, row: int, col: int, amount: float) -> None:
"""Write a currency value; clear the cell when zero so print matches the blank template rows."""
if amount:
_w(ws, row, col, round(amount, 2))
else:
if not amount:
_clear(ws, row, col)
return
c = ws.cell(row=row, column=col)
if c.__class__.__name__ == "MergedCell":
return
c.value = round(float(amount), 2)
c.number_format = MILEAGE_MONEY_FMT
def _write_meal_breakfast_cell(ws, row: int, val: Any) -> Optional[int]:
"""Write breakfast YES/NO markers; return the column that was populated."""
_clear(ws, row, 28)
_clear(ws, row, 29)
s = _s(val).lower()
if s in ("yes", "y", "true", "1"):
_w(ws, row, 28, "YES ")
return 28
if s in ("no", "n", "false", "0"):
_w(ws, row, 29, "NO")
return 29
return None
def _write_mileage_totals(ws, header: Dict, items: List[Dict]) -> None:
@ -679,15 +866,43 @@ def fill_mileage_worksheet(ws, header: Dict, items: List[Dict], period: str) ->
if km:
_w(ws, row, 21, km)
if toll:
_w(ws, row, 24, toll)
_write_mileage_money_cell(ws, row, 24, toll)
if parking:
_w(ws, row, 27, parking)
_write_mileage_money_cell(ws, row, 27, parking)
mileage_item_fonts = snapshot_row_fonts(ws, MILEAGE_TRIP_FORMAT_REF_ROW, MILEAGE_DATA_COLS)
mileage_header_fonts = snapshot_cell_fonts(ws, MILEAGE_HEADER_CELLS)
trip_count = min(len(items), len(MILEAGE_TRIP_ROW_SLOTS))
for row in MILEAGE_TRIP_ROW_SLOTS[:trip_count]:
for i, row in enumerate(MILEAGE_TRIP_ROW_SLOTS[:trip_count]):
_finalize_mileage_trip_row(ws, row)
it = items[i] if i < len(items) else {}
apply_user_formats_for_row(
ws,
row,
cell_format_map(it),
MILEAGE_ITEM_FIELD_COLS,
template_fonts=mileage_item_fonts,
)
apply_user_formats_for_cells(
ws, cell_format_map(header), MILEAGE_HEADER_CELLS, template_fonts=mileage_header_fonts
)
for header_row in (7, 9, 11, 16):
_center_mileage_header_value(ws, header_row)
_write_mileage_totals(ws, header, items)
summary_ref_cells = {
f"p_{row}_{col}": (row, col)
for positions in MILEAGE_SUMMARY_FIELD_CELLS.values()
for row, col in positions
}
summary_fonts = snapshot_cell_fonts(ws, summary_ref_cells)
apply_user_formats_multi_cells(
ws,
cell_format_map(header),
MILEAGE_SUMMARY_FIELD_CELLS,
template_fonts=summary_fonts,
)
def _fill_mileage(header: Dict, items: List[Dict], period: str) -> bytes:
@ -727,6 +942,9 @@ def _fill_ot(header: Dict, items: List[Dict], period: str) -> bytes:
for c in _data_cols:
_clear(ws, r, c)
ot_item_fonts = snapshot_row_fonts(ws, 17, _data_cols)
ot_header_fonts = snapshot_cell_fonts(ws, OT_HEADER_CELLS)
for i, it in enumerate(items):
if i >= 21:
break
@ -749,8 +967,20 @@ def _fill_ot(header: Dict, items: List[Dict], period: str) -> bytes:
if rest_gt8: _w(ws, r, 13, rest_gt8)
if ph: _w(ws, r, 16, ph)
if ph_gt8: _w(ws, r, 17, ph_gt8)
apply_user_formats_for_row(
ws, r, cell_format_map(it), OT_ITEM_FIELD_COLS, template_fonts=ot_item_fonts
)
_remerge_ot_data_rows(ws, 17, 37)
apply_user_formats_for_cells(
ws, cell_format_map(header), OT_HEADER_CELLS, template_fonts=ot_header_fonts
)
apply_user_formats_for_cells(
ws,
cell_format_map(header),
OT_PERIOD_CELL,
template_fonts=snapshot_cell_fonts(ws, OT_PERIOD_CELL),
)
# Row 38 has SUM formulas — recalculate automatically in Excel
# Signature at row 43 (name) and row 47 (date) — clear template sample (2025-02-07)

15
backend/services/claim_pdf_service.py
View File

@ -35,15 +35,14 @@ HEAD_FG = colors.white
def _styles():
s = getSampleStyleSheet()
base = dict(fontName="Helvetica", textColor=INK)
return {
"h1": ParagraphStyle("h1", **base, fontSize=16, leading=20, fontName="Helvetica-Bold"),
"h2": ParagraphStyle("h2", **base, fontSize=11, leading=14, fontName="Helvetica-Bold"),
"label": ParagraphStyle("label", **base, fontSize=8, leading=10, textColor=MUTED, fontName="Helvetica-Bold"),
"value": ParagraphStyle("value", **base, fontSize=9, leading=12),
"small": ParagraphStyle("small", **base, fontSize=8, leading=10, textColor=MUTED),
"total": ParagraphStyle("total", **base, fontSize=10, leading=13, fontName="Helvetica-Bold"),
"center": ParagraphStyle("center", **base, fontSize=9, leading=12, alignment=1),
"h1": ParagraphStyle("h1", fontName="Helvetica-Bold", textColor=INK, fontSize=16, leading=20),
"h2": ParagraphStyle("h2", fontName="Helvetica-Bold", textColor=INK, fontSize=11, leading=14),
"label": ParagraphStyle("label", fontName="Helvetica-Bold", textColor=MUTED, fontSize=8, leading=10),
"value": ParagraphStyle("value", fontName="Helvetica", textColor=INK, fontSize=9, leading=12),
"small": ParagraphStyle("small", fontName="Helvetica", textColor=MUTED, fontSize=8, leading=10),
"total": ParagraphStyle("total", fontName="Helvetica-Bold", textColor=INK, fontSize=10, leading=13),
"center": ParagraphStyle("center", fontName="Helvetica", textColor=INK, fontSize=9, leading=12, alignment=1),
}

146
backend/services/claim_preview_service.py
View File

@ -6,7 +6,7 @@ import base64
import io
import re
from pathlib import Path
from typing import Any, Dict, List
from typing import Any, Dict, List, Optional
from datetime import datetime, time as dt_time
from services.claim_dates import (
@ -14,6 +14,7 @@ from services.claim_dates import (
format_claim_date,
format_line_item_date,
)
from services.cell_format_service import cell_format_map, preview_cell_style_attr
def _load_logo(logo_name: str) -> str:
@ -51,10 +52,20 @@ EXPENSE_HEADER_UNDERLINE_W = (260, 240, 230) # col 1–3 underline width (px)
def _expense_header_field_row(
label: str, value: Any, underline_px: int, label_width_px: int
label: str, value: Any, underline_px: int, label_width_px: int, fmt: Optional[dict] = None
) -> str:
"""Header row: fixed label width so underlines start/end align within each column."""
display_val = _escape(value) if value not in (None, "") else "&nbsp;"
value_style = (
f"border-bottom:1px solid #000; width:{underline_px}px; min-width:{underline_px}px; "
f"max-width:{underline_px}px; display:inline-block; padding-bottom:1px; "
f"font-size:8pt; font-weight:normal; line-height:1.2; box-sizing:border-box;"
)
if fmt:
if fmt.get("fontSize"):
value_style = value_style.replace("font-size:8pt;", f"font-size:{float(fmt['fontSize'])}pt;")
if fmt.get("bold"):
value_style = value_style.replace("font-weight:normal;", "font-weight:bold;")
return (
f'<div class="expense-header-field" style="display:flex; align-items:baseline; '
f'font-size:8pt; font-family:Arial Narrow, Arial, sans-serif;">'
@ -62,19 +73,16 @@ def _expense_header_field_row(
f"width:{label_width_px}px; min-width:{label_width_px}px; margin-right:12px; "
f'font-weight:normal;">'
f"{_escape(label)}</span>"
f'<span class="expense-header-value" style="border-bottom:1px solid #000; '
f"width:{underline_px}px; min-width:{underline_px}px; max-width:{underline_px}px; "
f"display:inline-block; padding-bottom:1px; font-size:8pt; font-weight:normal; "
f'line-height:1.2; box-sizing:border-box;">'
f'<span class="expense-header-value" style="{value_style}">'
f"{display_val}</span></div>"
)
def _expense_header_col3_row1(location_val: str) -> str:
def _expense_header_col3_row1(location_val: str, location_fmt: Optional[dict] = None) -> str:
"""Row 1 col 3: Div/Dept/Proj/ sub-header + Location field (underline aligns with row 1)."""
uw = EXPENSE_HEADER_UNDERLINE_W[2]
lw = EXPENSE_HEADER_LABEL_W[2]
location_field = _expense_header_field_row("Location", location_val, uw, lw)
location_field = _expense_header_field_row("Location", location_val, uw, lw, location_fmt)
return (
'<div class="header-grid-cell header-grid-cell--col3-top">'
f'<span class="expense-header-dept-only">{_escape("Div/Dept/Proj/")}</span>'
@ -86,15 +94,16 @@ def _expense_header_col3_row1(location_val: str) -> str:
def _expense_header_grid(header: Dict, period: str) -> str:
"""Strict 2-row × 3-column grid (Image 1 alignment)."""
month_year = _fmt_period(period)
cf = cell_format_map(header)
lw = EXPENSE_HEADER_LABEL_W
uw = EXPENSE_HEADER_UNDERLINE_W
return f"""
<div class="header-grid-cell">{_expense_header_field_row("Staff Name", header.get("name", ""), uw[0], lw[0])}</div>
<div class="header-grid-cell">{_expense_header_field_row("Company", header.get("company", ""), uw[1], lw[1])}</div>
{_expense_header_col3_row1(header.get("location", ""))}
<div class="header-grid-cell">{_expense_header_field_row("Designation", header.get("designation", ""), uw[0], lw[0])}</div>
<div class="header-grid-cell">{_expense_header_field_row("Emp. No.", header.get("emp_no", ""), uw[1], lw[1])}</div>
<div class="header-grid-cell">{_expense_header_field_row("Month/Year", month_year, uw[2], lw[2])}</div>"""
<div class="header-grid-cell">{_expense_header_field_row("Staff Name", header.get("name", ""), uw[0], lw[0], cf.get("name"))}</div>
<div class="header-grid-cell">{_expense_header_field_row("Company", header.get("company", ""), uw[1], lw[1], cf.get("company"))}</div>
{_expense_header_col3_row1(header.get("location", ""), cf.get("location"))}
<div class="header-grid-cell">{_expense_header_field_row("Designation", header.get("designation", ""), uw[0], lw[0], cf.get("designation"))}</div>
<div class="header-grid-cell">{_expense_header_field_row("Emp. No.", header.get("emp_no", ""), uw[1], lw[1], cf.get("emp_no"))}</div>
<div class="header-grid-cell">{_expense_header_field_row("Month/Year", month_year, uw[2], lw[2], cf.get("_claim_period"))}</div>"""
def _fmt_num(val: Any) -> str:
@ -728,17 +737,18 @@ def generate_expenses_preview(header: Dict, items: List[Dict], period: str) -> s
row_amounts[key] = amt
row_total += amt
cat_totals[key] = round(cat_totals[key] + amt, 2)
cf = it.get("_cellFormat") or {}
cat_cells = "".join(
f'<td class="money">{_fmt_money(row_amounts[key])}</td>'
f'<td class="money"{preview_cell_style_attr(cf.get(key))}>{_fmt_money(row_amounts[key])}</td>'
for key, _ in EXPENSE_PREVIEW_CATS
)
row_date = format_line_item_date(it.get("date", ""), period)
date_cell = _escape(row_date) if row_date else "&nbsp;"
rows_html += f'''<tr class="data-row">
<td>{date_cell}</td>
<td class="particulars">{_escape(it.get("description", ""))}</td>
<td{preview_cell_style_attr(cf.get("date"))}>{date_cell}</td>
<td class="particulars"{preview_cell_style_attr(cf.get("description"))}>{_escape(it.get("description", ""))}</td>
{cat_cells}
<td class="money">{_fmt_money(row_total)}</td>
<td class="money"{preview_cell_style_attr(cf.get("total"))}>{_fmt_money(row_total)}</td>
</tr>'''
else:
rows_html += empty_row_html
@ -1277,22 +1287,24 @@ def generate_expenses_preview(header: Dict, items: List[Dict], period: str) -> s
def generate_mileage_preview(header: Dict, items: List[Dict], period: str) -> str:
"""Generate Mileage claim form HTML matching the Excel template exactly."""
hcf = cell_format_map(header)
rows_html = ""
for i in range(12): # 12 data rows
if i < len(items):
it = items[i]
icf = it.get("_cellFormat") or {}
km = float(it.get("km", 0) or 0)
toll = float(it.get("toll", 0) or 0)
parking = float(it.get("parking", 0) or 0)
rows_html += f'''<tr>
<td>{format_line_item_date(it.get("date",""), period, style=LINE_ITEM_STYLE_SLASH_DATE)}</td>
<td style="text-align:left">{_escape(it.get("from_place",""))}</td>
<td style="text-align:left">{_escape(it.get("to_place",""))}</td>
<td style="text-align:left">{_escape(it.get("purpose",""))}</td>
<td style="text-align:right">{_fmt_num(km)}</td>
<td style="text-align:right">{_escape(it.get("toll",""))}</td>
<td style="text-align:right">{_escape(it.get("parking",""))}</td>
<td{preview_cell_style_attr(icf.get("date"))}>{format_line_item_date(it.get("date",""), period, style=LINE_ITEM_STYLE_SLASH_DATE)}</td>
<td{preview_cell_style_attr(icf.get("from_place"), align="left")}>{_escape(it.get("from_place",""))}</td>
<td{preview_cell_style_attr(icf.get("to_place"), align="left")}>{_escape(it.get("to_place",""))}</td>
<td{preview_cell_style_attr(icf.get("purpose"), align="left")}>{_escape(it.get("purpose",""))}</td>
<td{preview_cell_style_attr(icf.get("km"), align="right")}>{_fmt_num(km)}</td>
<td{preview_cell_style_attr(icf.get("toll"), align="right")}>{_escape(it.get("toll",""))}</td>
<td{preview_cell_style_attr(icf.get("parking"), align="right")}>{_escape(it.get("parking",""))}</td>
</tr>'''
else:
rows_html += '''<tr><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td></tr>'''
@ -1304,6 +1316,38 @@ def generate_mileage_preview(header: Dict, items: List[Dict], period: str) -> st
mileage_amount = total_km * rate_per_km
grand_total = mileage_amount + total_toll + total_parking
period_attr = preview_cell_style_attr(hcf.get("_claim_period"))
total_km_attr = preview_cell_style_attr(
hcf.get("_summary_total_km"), align="right", base="padding:3px 4px"
)
total_toll_attr = preview_cell_style_attr(hcf.get("_summary_toll"), align="right")
total_parking_attr = preview_cell_style_attr(hcf.get("_summary_parking"), align="right")
mileage_amt_attr = preview_cell_style_attr(
hcf.get("_summary_mileage"),
align="right",
base="padding:3px 4px;border-bottom:1px solid #000;min-width:80px",
)
mileage_claim_attr = preview_cell_style_attr(
hcf.get("_summary_mileage"),
align="right",
base="padding:3px 4px;border-bottom:1px solid #000;min-width:80px",
)
toll_claim_attr = preview_cell_style_attr(
hcf.get("_summary_toll"),
align="right",
base="padding:3px 4px;border-bottom:1px solid #000;min-width:80px",
)
parking_claim_attr = preview_cell_style_attr(
hcf.get("_summary_parking"),
align="right",
base="padding:3px 4px;border-bottom:1px solid #000;min-width:80px",
)
grand_total_attr = preview_cell_style_attr(
hcf.get("_summary_grand_total"),
align="right",
base="padding:3px 4px;border-bottom:1px solid #000;min-width:80px",
)
# Load logo
logo_data_url = _load_logo("mileage_logo")
logo_html = f'<img src="{logo_data_url}" style="height:45px;display:block;margin:0 auto 3px;" />' if logo_data_url else ''
@ -1391,7 +1435,7 @@ def generate_mileage_preview(header: Dict, items: List[Dict], period: str) -> st
<tr>
<td class="info-label">Mileage Claims for Month of</td>
<td class="info-colon">:</td>
<td class="info-value">{_fmt_period(period)}</td>
<td class="info-value"{period_attr}>{_fmt_period(period)}</td>
</tr>
<tr>
<td class="info-label" style="vertical-align:top;">Type of Vehicle<br><span class="vehicle-note">(* delete if not applicable)</span></td>
@ -1404,7 +1448,7 @@ def generate_mileage_preview(header: Dict, items: List[Dict], period: str) -> st
<tr>
<td class="info-label">Vehicle Reg. No.</td>
<td class="info-colon">:</td>
<td class="info-value">{_escape(header.get("vehicle_reg",""))}</td>
<td class="info-value" style="text-align:center">{_escape(header.get("vehicle_reg",""))}</td>
</tr>
</table>
@ -1425,9 +1469,9 @@ def generate_mileage_preview(header: Dict, items: List[Dict], period: str) -> st
{rows_html}
<tr class="total-row">
<td colspan="4" style="text-align:center">TOTAL</td>
<td style="text-align:right">{_fmt_num(total_km)}</td>
<td style="text-align:right">{total_toll:.2f}</td>
<td style="text-align:right">{total_parking:.2f}</td>
<td{total_km_attr}>{_fmt_num(total_km)}</td>
<td{total_toll_attr}>{total_toll:.2f}</td>
<td{total_parking_attr}>{total_parking:.2f}</td>
</tr>
</table>
@ -1435,9 +1479,9 @@ def generate_mileage_preview(header: Dict, items: List[Dict], period: str) -> st
<table class="bottom-section" style="width:100%;border-collapse:collapse;font-size:10pt;font-family:Arial,sans-serif;">
<tr>
<!-- ROW 1: Calculation + Signature of Claimant -->
<td colspan="3" style="padding:3px 4px;">{_fmt_num(total_km)}</td>
<td colspan="3"{total_km_attr}>{_fmt_num(total_km)}</td>
<td colspan="1" style="padding:3px 4px;">km &nbsp;x&nbsp; RM{rate_per_km:.2f} &nbsp;cents&nbsp; = &nbsp;RM </td>
<td colspan="2" style="padding:3px 4px;border-bottom:1px solid #000;min-width:80px;">{mileage_amount:.2f}</td>
<td colspan="2"{mileage_amt_attr}>{mileage_amount:.2f}</td>
<td colspan="4" style="padding:3px 4px;">&nbsp;</td>
<td colspan="2" style="padding:3px 4px;">Signature of Claimant &nbsp;:&nbsp;<span style="border-bottom:1px solid #000;display:inline-block;min-width:100px;">&nbsp;</span></td>
</tr>
@ -1450,7 +1494,7 @@ def generate_mileage_preview(header: Dict, items: List[Dict], period: str) -> st
<tr>
<!-- ROW 3: (a) Total mileage + Verified by -->
<td colspan="3" style="padding:3px 4px;">(a) &nbsp;Total mileage claims (RM)</td>
<td colspan="3" style="padding:3px 4px;text-align:right;border-bottom:1px solid #000;min-width:80px;">RM{mileage_amount:.2f}</td>
<td colspan="3"{mileage_claim_attr}>RM{mileage_amount:.2f}</td>
<td colspan="4" style="padding:3px 4px;">&nbsp;</td>
<td colspan="2" style="padding:3px 4px;">Verified by &nbsp;:&nbsp;</td>
</tr>
@ -1463,20 +1507,20 @@ def generate_mileage_preview(header: Dict, items: List[Dict], period: str) -> st
<tr>
<!-- ROW 5: (b) Total toll + Date (Verified by) -->
<td colspan="3" style="padding:3px 4px;">(b) &nbsp;Total toll claims (RM)</td>
<td colspan="3" style="padding:3px 4px;text-align:right;border-bottom:1px solid #000;min-width:80px;">RM{total_toll:.2f}</td>
<td colspan="3"{toll_claim_attr}>RM{total_toll:.2f}</td>
<td colspan="4" style="padding:3px 4px;">&nbsp;</td>
<td colspan="2" style="padding:3px 4px;">Date &nbsp;:&nbsp;<span style="border-bottom:1px solid #000;display:inline-block;min-width:100px;">&nbsp;</span></td>
</tr>
<tr>
<!-- ROW 6: (c) Total parking + empty right -->
<td colspan="3" style="padding:3px 4px;">(c) &nbsp;Total parking claims (RM)</td>
<td colspan="3" style="padding:3px 4px;text-align:right;border-bottom:1px solid #000;min-width:80px;">{'' if total_parking == 0 else f'RM{total_parking:.2f}'}</td>
<td colspan="3"{parking_claim_attr}>{'' if total_parking == 0 else f'RM{total_parking:.2f}'}</td>
<td colspan="6" style="padding:3px 4px;">&nbsp;</td>
</tr>
<tr>
<!-- ROW 7: Grand Total -->
<td colspan="3" style="padding:3px 4px;font-weight:bold;">Grand Total</td>
<td colspan="3" style="padding:3px 4px;text-align:right;font-weight:bold;border-bottom:1px solid #000;min-width:80px;">RM{grand_total:.2f}</td>
<td colspan="3"{grand_total_attr}>RM{grand_total:.2f}</td>
<td colspan="6" style="padding:3px 4px;">&nbsp;</td>
</tr>
<tr>
@ -1508,17 +1552,23 @@ def generate_meal_preview(header: Dict, items: List[Dict], period: str) -> str:
for i in range(11): # 11 data rows
if i < len(items):
it = items[i]
cf = it.get("_cellFormat") or {}
breakfast = str(it.get("breakfast_included") or "").lower()
breakfast_yes = "YES" if breakfast in ("yes", "y", "true", "1") else ""
breakfast_no = "NO" if breakfast in ("no", "n", "false", "0") else ""
rows_html += f'''<tr>
<td>{_escape(it.get("date",""))}</td>
<td style="text-align:left">{_escape(it.get("from_place",""))}</td>
<td>{_escape(it.get("time_depart",""))}</td>
<td>{_escape(it.get("time_arrive",""))}</td>
<td style="text-align:left">{_escape(it.get("description",""))}</td>
<td style="text-align:right">{_escape(it.get("meal",""))}</td>
<td style="text-align:right">{_escape(it.get("accommodation",""))}</td>
<td{preview_cell_style_attr(cf.get("date"))}>{_escape(it.get("date",""))}</td>
<td{preview_cell_style_attr(cf.get("from_place"), align="left")}>{_escape(it.get("from_place",""))}</td>
<td{preview_cell_style_attr(cf.get("time_depart"))}>{_escape(it.get("time_depart",""))}</td>
<td{preview_cell_style_attr(cf.get("time_arrive"))}>{_escape(it.get("time_arrive",""))}</td>
<td{preview_cell_style_attr(cf.get("description"), align="left")}>{_escape(it.get("description",""))}</td>
<td{preview_cell_style_attr(cf.get("meal"), align="right")}>{_escape(it.get("meal",""))}</td>
<td{preview_cell_style_attr(cf.get("accommodation"), align="right")}>{_escape(it.get("accommodation",""))}</td>
<td{preview_cell_style_attr(cf.get("breakfast_included"))}>{breakfast_yes}</td>
<td>{breakfast_no}</td>
</tr>'''
else:
rows_html += '''<tr><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td></tr>'''
rows_html += '''<tr><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td><td>&nbsp;</td></tr>'''
total_meal = sum(float(it.get("meal", 0) or 0) for it in items)
total_accommodation = sum(float(it.get("accommodation", 0) or 0) for it in items)
@ -1585,14 +1635,18 @@ def generate_meal_preview(header: Dict, items: List[Dict], period: str) -> str:
<th style="width:10%">TIME DEPART</th>
<th style="width:10%">TIME ARRIVE</th>
<th style="width:32%">DESCRIPTION</th>
<th style="width:10%">MEAL</th>
<th style="width:10%">ACCOM.</th>
<th style="width:8%">MEAL</th>
<th style="width:8%">ACCOM.</th>
<th style="width:5%">BF YES</th>
<th style="width:5%">BF NO</th>
</tr>
{rows_html}
<tr class="total-row">
<td colspan="5" style="text-align:right"><strong>TOTAL</strong> <span style="font-weight:normal;font-style:italic">(Amount to be transferred to Expenses Claim Form respectively)</span></td>
<td style="text-align:right">{total_meal:.2f}</td>
<td style="text-align:right">{total_accommodation:.2f}</td>
<td></td>
<td></td>
</tr>
</table>

84
backend/services/payslip_storage.py Normal file
View File

@ -0,0 +1,84 @@
"""Pay slip PDF storage in MongoDB (source of truth)."""
from __future__ import annotations
import re
from datetime import date
from pathlib import Path
from typing import Optional
from bson.binary import Binary
from fastapi import HTTPException
from core import db
from services import storage_service
def canonical_filename(slip_date: date) -> str:
"""Centralized name: payslip_YYYYMM.pdf"""
return f"payslip_{slip_date.year:04d}{slip_date.month:02d}.pdf"
async def reserve_unique_filename(owner_id: str, slip_date: date) -> str:
"""payslip_YYYYMM.pdf, or payslip_YYYYMM_2.pdf if that month already exists."""
base = f"payslip_{slip_date.year:04d}{slip_date.month:02d}"
pattern = f"^{re.escape(base)}(_\\d+)?\\.pdf$"
used: set[str] = set()
cursor = db.payslips.find(
{"owner_id": owner_id, "filename": {"$regex": pattern, "$options": "i"}},
{"filename": 1},
)
async for doc in cursor:
fn = doc.get("filename")
if fn:
used.add(fn.lower())
first = f"{base}.pdf"
if first.lower() not in used:
return first
n = 2
while True:
candidate = f"{base}_{n}.pdf"
if candidate.lower() not in used:
return candidate
n += 1
async def load_pdf_bytes(rec: dict) -> bytes:
"""Read PDF from MongoDB, with legacy disk/R2 fallback and optional backfill."""
raw = rec.get("pdf_data")
if raw is not None:
return bytes(raw)
payslip_id = rec["id"]
content: Optional[bytes] = None
path_val = rec.get("path")
if path_val:
path = Path(path_val)
if path.is_file():
content = path.read_bytes()
if content is None and rec.get("storage") == "r2":
from core import ROOT_DIR
cache_dir = ROOT_DIR / "payslip_files"
cache_dir.mkdir(exist_ok=True)
tmp = cache_dir / f"_cache_{payslip_id}.pdf"
fetched = await storage_service.download_pdf(payslip_id, tmp)
if fetched and fetched.is_file():
content = fetched.read_bytes()
if content is None:
raise HTTPException(status_code=404, detail="Pay slip file not found in database.")
await db.payslips.update_one(
{"id": payslip_id},
{"$set": {"pdf_data": Binary(content)}},
)
return content
async def delete_legacy_files(rec: dict) -> None:
path_val = rec.get("path")
if path_val:
Path(path_val).unlink(missing_ok=True)
if rec.get("storage") == "r2":
await storage_service.delete_pdf(rec["id"])

138
backend/services/pdf_company_extractor.py Normal file
View File

@ -0,0 +1,138 @@
"""Extract vendor company names from supporting-document PDFs for canonical filenames."""
from __future__ import annotations
import io
import re
from typing import Optional
from services import pdf_service
# Malaysian / common business entity markers (vendor names on receipts and invoices).
_COMPANY_MARKERS = re.compile(
r"\b("
r"sdn\.?\s*bhd|bhd\.?|berhad|enterprise|trading|holdings|"
r"plt\.?|llp|inc\.?|corp\.?|co\.?\s*ltd|limited|"
r"restaurant|cafe|hotel|mart|supermarket|pharmacy|"
r"petrol|station|services|marketing|solutions"
r")\b",
re.IGNORECASE,
)
_NOISE_LINE = re.compile(
r"^(?:"
r"tax\s*invoice|invoice|receipt|bill\s*to|ship\s*to|sold\s*to|"
r"thank\s*you|page\s+\d|tel(?:ephone)?|fax|www\.|https?://|"
r"gst|sst|total|sub\s*total|amount|qty|quantity|description|"
r"date\s*:|time\s*:|no\.|ref\s*:|reg\.?\s*no|"
r"payment|cash|change|visa|mastercard|"
r"rm\s*[\d,.]|[\d,.]+\s*rm"
r")",
re.IGNORECASE,
)
_MAX_SLUG_LEN = 72
_MAX_PAGES = 3
def extract_pdf_text(content: bytes, *, max_pages: int = _MAX_PAGES) -> str:
"""Return concatenated text from the first pages of a PDF."""
try:
from pypdf import PdfReader
except ImportError as exc: # pragma: no cover
raise RuntimeError("pypdf is required for PDF company extraction") from exc
reader = PdfReader(io.BytesIO(content))
parts: list[str] = []
for page in reader.pages[:max_pages]:
parts.append(page.extract_text() or "")
return "\n".join(parts)
def _normalize_line(line: str) -> str:
return re.sub(r"\s+", " ", line.strip())
def _is_employer_line(line: str) -> bool:
employer = (pdf_service.COMPANY_NAME or "").strip().lower()
if not employer:
return False
low = line.lower()
if employer in low or low in employer:
return True
# Match significant token overlap (e.g. "Quatriz System" vs "Quatriz System Sdn Bhd").
emp_tokens = {t for t in re.findall(r"[a-z0-9]+", employer) if len(t) > 3}
line_tokens = {t for t in re.findall(r"[a-z0-9]+", low) if len(t) > 3}
if emp_tokens and emp_tokens <= line_tokens:
return True
return False
def _score_company_line(line: str, line_index: int) -> int:
if len(line) < 4 or len(line) > 120:
return -100
if _NOISE_LINE.search(line):
return -50
if re.fullmatch(r"[\d\s./\-:]+", line):
return -50
if _is_employer_line(line):
return -80
score = 0
if _COMPANY_MARKERS.search(line):
score += 25
if line.isupper() and len(line) >= 6:
score += 8
if line_index < 12:
score += max(0, 12 - line_index)
if re.search(r"\d{5,}", line):
score -= 15
if "@" in line or "http" in line.lower():
score -= 30
# Prefer lines that look like a business name, not an address-only line.
if re.search(r"\b(no\.?|lot|jalan|jln|taman|floor|level|unit)\b", line, re.I):
score -= 5
return score
def company_name_from_text(text: str) -> Optional[str]:
"""Pick the best vendor company line from extracted PDF text."""
if not text or not text.strip():
return None
best_name: Optional[str] = None
best_score = 0
for idx, raw in enumerate(text.splitlines()):
line = _normalize_line(raw)
if not line:
continue
score = _score_company_line(line, idx)
if score > best_score:
best_score = score
best_name = line
if best_score < 8:
return None
return best_name
def company_name_from_pdf(content: bytes) -> Optional[str]:
"""Extract vendor company name from PDF bytes, or None if not found."""
try:
text = extract_pdf_text(content)
except Exception: # noqa: BLE001 — corrupt or encrypted PDF
return None
return company_name_from_text(text)
def filename_slug_from_company(company: str, *, max_len: int = _MAX_SLUG_LEN) -> str:
"""Filesystem-safe stem from a company name."""
slug = company.lower()
slug = slug.replace("&", "and")
slug = re.sub(r"[^\w\s-]", " ", slug, flags=re.UNICODE)
slug = re.sub(r"[\s_]+", "_", slug.strip())
slug = slug.strip("_")
if not slug:
slug = "supporting_document"
if len(slug) > max_len:
slug = slug[:max_len].rstrip("_")
return slug or "supporting_document"

2
backend/tests/test_claim_date_formatting.py
View File

@ -330,7 +330,7 @@ class TestClaimDateFormats:
assert "Approved By:" in html
assert "ot-sig-table" in html
assert "RECOVERY DB" in html
assert "A4 portrait" in html
assert "letter portrait" in html
assert "Dept. / Div." in html
assert "SOFTWARE" in html or "QC" in html
assert 'colspan="2">ACTUAL O/T DONE</th>' in html

169
backend/tests/test_claim_services.py
View File

@ -8,32 +8,32 @@ from unittest.mock import AsyncMock, patch, MagicMock
class TestClaimDates:
"""Tests for claim_dates.py utilities."""
def test_claim_date_format_constant(self):
from services.claim_dates import CLAIM_DATE_FORMAT
assert CLAIM_DATE_FORMAT == "DD/MM/YYYY"
def test_claim_date_display_format_constant(self):
from services.claim_dates import CLAIM_DATE_DISPLAY_FMT
assert CLAIM_DATE_DISPLAY_FMT == "%d/%m/%Y"
def test_claim_date_to_display(self):
from services.claim_dates import claimDateToDisplay
# Test with storage format
assert claimDateToDisplay("2026-03-15") == "15/03/2026"
assert claimDateToDisplay("") == ""
assert claimDateToDisplay(None) == ""
def test_format_claim_date(self):
from services.claim_dates import format_claim_date
assert format_claim_date("2026-03-15") == "15/03/2026"
assert format_claim_date("") == ""
assert format_claim_date(None) == ""
def test_claim_date_to_storage(self):
from services.claim_dates import claimDateToStorage
# Test with display format
assert claimDateToStorage("15/03/2026") == "2026-03-15"
assert claimDateToStorage("") == ""
assert claimDateToStorage(None) == ""
def test_parse_claim_date(self):
from services.claim_dates import parse_claim_date
dt = parse_claim_date("15/03/2026")
assert dt is not None
assert dt.day == 15
assert dt.month == 3
assert dt.year == 2026
assert parse_claim_date("") is None
assert parse_claim_date(None) is None
def test_claim_date_normalize_on_blur(self):
from services.claim_dates import claimDateNormalizeOnBlur
# Valid date
result = claimDateNormalizeOnBlur("15/03/2026")
assert result == "15/03/2026"
# Invalid date should remain as-is
result = claimDateNormalizeOnBlur("invalid")
assert result == "invalid"
def test_format_claim_date_roundtrip(self):
from services.claim_dates import format_claim_date, parse_claim_date
display = format_claim_date("2026-05-18")
assert display == "18/05/2026"
dt = parse_claim_date(display)
assert dt.strftime("%Y-%m-%d") == "2026-05-18"
class TestClaimImportService:
@ -41,9 +41,7 @@ class TestClaimImportService:
def test_parse_period_with_excel_serial(self):
from services.claim_import_service import _parse_period
# Excel serial for Jan 2026 (45597 = 2024-10-03 base)
result = _parse_period("45597.0")
# Should return YYYY-MM format or empty string
assert result == "" or "-" in result
def test_parse_period_with_iso_format(self):
@ -70,16 +68,34 @@ class TestClaimImportService:
class TestClaimExcelService:
"""Tests for claim_excel_service.py."""
def test_apply_user_cell_format_only_when_requested(self):
import openpyxl
from services.cell_format_service import _apply_user_cell_format
from services.claim_excel_service import _w
wb = openpyxl.Workbook()
ws = wb.active
ws.cell(1, 1).font = openpyxl.styles.Font(name="Arial", size=7)
_w(ws, 1, 1, "Sample text")
_apply_user_cell_format(ws, 1, 1, None)
assert ws.cell(1, 1).font.size == 7
_apply_user_cell_format(ws, 1, 1, {"fontSize": 11, "bold": True})
font = ws.cell(1, 1).font
assert font.bold is True
assert font.size == 11
def test_excel_mimetype(self):
from services.claim_excel_service import excel_mimetype
assert excel_mimetype == "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
mime, ext = excel_mimetype("EXPENSES")
assert mime == "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
assert ext == "xlsx"
def test_claim_type_labels_import(self):
from services.claim_excel_service import CLAIM_TYPE_LABELS
assert "EXPENSES" in CLAIM_TYPE_LABELS
assert "MEAL" in CLAIM_TYPE_LABELS
assert "MILEAGE" in CLAIM_TYPE_LABELS
assert "OT" in CLAIM_TYPE_LABELS
def test_fill_claim_excel_unknown_type(self):
from services.claim_excel_service import fill_claim_excel
with pytest.raises(ValueError, match="Unknown claim type"):
fill_claim_excel("INVALID", {}, [], "2026-01")
class TestClaimPdfService:
@ -102,18 +118,91 @@ class TestClaimPdfService:
def test_format_helper(self):
from services.claim_pdf_service import _fmt
# Test number formatting
assert _fmt(1234.56) == "1,234.56"
assert _fmt(1000) == "1,000"
# Test string passthrough
assert _fmt("text") == "text"
assert _fmt(None) == ""
assert _fmt(1234.56) == "1234.56"
def test_format_num_helper(self):
from services.claim_pdf_service import _fmt_num
assert _fmt_num(1234.56) == "1,234.56"
assert _fmt_num(1000) == "1,000.00"
class TestClaimPreviewService:
"""Tests for claim_preview_service.py."""
def test_render_claim_pdf_import(self):
# Test that the function can be imported
from services.claim_preview_service import render_claim_pdf
assert callable(render_claim_pdf)
def test_generate_claim_preview_import(self):
from services.claim_preview_service import generate_claim_preview
assert callable(generate_claim_preview)
def test_generate_expenses_preview(self):
from services.claim_preview_service import generate_claim_preview
html = generate_claim_preview("EXPENSES", {"name": "Test"}, [], "2026-01")
assert "EXPENSES CLAIM FORM" in html
assert "Test" in html
class TestClaimJsonSanitizer:
"""Tests for claim_access JSON sanitization."""
def test_strip_claim_for_json_removes_nested_bytes(self):
from bson.binary import Binary
from claim_access import strip_claim_for_json
rec = {
"id": "c1",
"file_data": Binary(b"\x81PDF"),
"attachments": [
{
"id": "a1",
"filename": "receipt.pdf",
"file_data": b"\x81nested",
}
],
}
out = strip_claim_for_json(rec)
assert "file_data" not in out
assert "file_data" not in out["attachments"][0]
assert out["attachments"][0]["filename"] == "receipt.pdf"
class TestPdfCompanyExtractor:
"""Tests for pdf_company_extractor.py."""
def test_company_name_from_receipt_text(self):
from services.pdf_company_extractor import company_name_from_text
text = """
TAX INVOICE
PETRONAS DAGANGAN BERHAD
Lot 123, Jalan Ampang
Tel: 03-12345678
"""
assert company_name_from_text(text) == "PETRONAS DAGANGAN BERHAD"
def test_skips_employer_name(self):
from services.pdf_company_extractor import company_name_from_text
from services import pdf_service
text = f"""
{pdf_service.COMPANY_NAME}
ABC TRADING SDN BHD
"""
assert company_name_from_text(text) == "ABC TRADING SDN BHD"
def test_filename_slug(self):
from services.pdf_company_extractor import filename_slug_from_company
assert filename_slug_from_company("Petronas Dagangan Berhad") == "petronas_dagangan_berhad"
def test_reserve_unique_attachment_name(self):
from services.claim_document_storage import _reserve_unique_on_claim
claim = {
"attachments": [
{"filename": "petronas_dagangan_berhad.pdf"},
]
}
assert _reserve_unique_on_claim(claim, "petronas_dagangan_berhad", ".pdf") == (
"petronas_dagangan_berhad_2.pdf"
)

15
docker-compose.claim.yml
View File

@ -1,16 +1,25 @@
# Claim App — simplified stack (no Redis/Celery/OCR)
# Claim App — local Docker Desktop stack.
# MongoDB runs as a Windows service on the host, so the backend reaches it
# via host.docker.internal:27017.
services:
backend:
build:
context: .
dockerfile: backend/Dockerfile
image: claim-backend:local
container_name: claim-backend
restart: unless-stopped
working_dir: /app/backend
command: ["python", "-m", "uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8002"]
env_file: ./backend/.env
environment:
MONGO_URL: mongodb://host.docker.internal:27017
DB_NAME: claim
PORT: "8002"
FRONTEND_URL: http://localhost:8080
CORS_ORIGINS: http://localhost:8080
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- "./Claim Template:/app/Claim Template:ro"
- claim_uploads:/app/backend/claim_files
@ -22,8 +31,8 @@ services:
build:
context: .
dockerfile: frontend/Dockerfile
args:
REACT_APP_BACKEND_URL: http://localhost:8002
image: claim-frontend:local
container_name: claim-frontend
restart: unless-stopped
ports:
- "8080:80"

19
docker-compose.yml
View File

@ -1,16 +1,27 @@
# Claim App — local Docker (no Redis/Celery)
# Claim App — local Docker Desktop stack.
# MongoDB runs as a Windows service on the host, so the backend reaches it
# via host.docker.internal:27017.
services:
backend:
build:
context: .
dockerfile: backend/Dockerfile
image: claim-backend:local
container_name: claim-backend
restart: unless-stopped
working_dir: /app/backend
command: ["python", "-m", "uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8002"]
env_file: ./backend/.env
environment:
MONGO_URL: mongodb://host.docker.internal:27017
DB_NAME: claim
PORT: "8002"
# When the SPA calls the API through the frontend nginx proxy, same-origin
# works without CORS. Set FRONTEND_URL to the host port mapped below.
FRONTEND_URL: http://localhost:8080
CORS_ORIGINS: http://localhost:8080
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- "./Claim Template:/app/Claim Template:ro"
- claim_uploads:/app/backend/claim_files
@ -22,9 +33,11 @@ services:
build:
context: .
dockerfile: frontend/Dockerfile
args:
REACT_APP_BACKEND_URL: http://localhost:8002
image: claim-frontend:local
container_name: claim-frontend
restart: unless-stopped
# REACT_APP_BACKEND_URL is empty by default — nginx proxies /api to backend:8002
# inside the compose network, so the SPA can use a relative base URL.
ports:
- "8080:80"
depends_on:

53
docs/deploy/docker.md Normal file
View File

@ -0,0 +1,53 @@
# Docker Desktop (Windows / macOS / Linux)
A two-container stack: `frontend` (nginx + React build) and `backend` (FastAPI on port 8002). MongoDB stays on the host and is reached via `host.docker.internal:27017`.
## Prerequisites
- Docker Desktop running
- A local MongoDB service (default `mongodb://127.0.0.1:27017`, database `claim`)
- `backend/.env` populated (see `backend/.env.example`)
## Build & start
```bash
# from the repo root
docker compose -f docker-compose.claim.yml up --build
```
URLs:
- SPA: http://localhost:8080
- API: http://localhost:8002 (also proxied as `/api` from the SPA)
## What runs where
| Service | Image | Port (host) | Notes |
|-----------|----------------------|-------------|-------|
| frontend | claim-frontend:local | 8080 → 80 | nginx serves the React build; proxies `/api` to `backend:8002` |
| backend | claim-backend:local | 8002 → 8002 | FastAPI / uvicorn; uses LibreOffice for Excel → PDF |
The backend reads `backend/.env` plus the `environment:` overrides from compose, so `MONGO_URL` is forced to `mongodb://host.docker.internal:27017` regardless of the .env value.
## Data persistence
- `claim_uploads` named volume → `/app/backend/claim_files`
- `claim_attach` named volume → `/app/backend/claim_attachments`
- MongoDB data lives on the host (Windows service) — make sure the `claim` database exists or is auto-created on first connection.
## Rebuilding after code changes
```bash
docker compose -f docker-compose.claim.yml up --build
# or rebuild only one service
docker compose -f docker-compose.claim.yml build backend
```
## CORS
`FRONTEND_URL` and `CORS_ORIGINS` are set to `http://localhost:8080` by compose. The SPA also talks to the same origin via the nginx proxy, so cookies work without CORS pre-flights.
## Common gotchas
- **"ECONNREFUSED 127.0.0.1:27017"** — the backend cannot reach your host Mongo. Confirm Mongo is listening on `127.0.0.1:27017` and that you started Mongo before the container. The `extra_hosts: host.docker.internal:host-gateway` line is required for older Docker setups.
- **"JWT_SECRET missing"** — copy `backend/.env.example` to `backend/.env` and fill in the secrets.
- **Print/PDF issues on Windows** — the backend uses LibreOffice Calc (not Excel COM) inside the container, so the printed layout may differ slightly from the Windows dev environment. This is a known difference.

14
frontend/Dockerfile
View File

@ -1,12 +1,16 @@
# ---- build stage ----
FROM node:20-alpine AS build
# ponytail: Bun replaces Node/yarn as the frontend build toolchain.
FROM oven/bun:1-alpine AS build
WORKDIR /app
COPY frontend/package.json frontend/yarn.lock ./
RUN yarn install --frozen-lockfile
COPY frontend/package.json frontend/bun.lock ./
RUN bun install --frozen-lockfile
COPY frontend/ ./
ARG REACT_APP_BACKEND_URL
# When running behind nginx the SPA talks to the same origin via /api,
# so REACT_APP_BACKEND_URL can be left empty in Docker. Still allow override
# for the case where the user exposes the API directly.
ARG REACT_APP_BACKEND_URL=""
ENV REACT_APP_BACKEND_URL=$REACT_APP_BACKEND_URL
RUN yarn build
RUN bun run build
# ---- serve stage ----
FROM nginx:1.27-alpine

3179
frontend/bun.lock Normal file
View File

File diff suppressed because it is too large Load Diff

10
frontend/nginx.conf
View File

@ -6,16 +6,18 @@ server {
client_max_body_size 50m;
# Docker embedded DNS — re-resolve backend on each request so nginx does not keep
# a stale container IP after `docker compose restart backend`.
# Docker embedded DNS — re-resolve backend on each request so nginx does not
# cache a stale container IP after `docker compose restart backend`.
resolver 127.0.0.11 valid=10s ipv6=off;
# SPA fallback.
location / {
try_files $uri /index.html;
}
# Proxy /api and /uploads to the backend container on port 8002.
location /api/ {
set $backend_upstream backend:8001;
set $backend_upstream backend:8002;
proxy_pass http://$backend_upstream;
proxy_http_version 1.1;
proxy_set_header Host $host;
@ -28,7 +30,7 @@ server {
}
location /uploads/ {
set $backend_upstream backend:8001;
set $backend_upstream backend:8002;
proxy_pass http://$backend_upstream;
proxy_http_version 1.1;
proxy_set_header Host $host;

22344
frontend/package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

78
frontend/src/components/CellFormatToolbar.jsx Normal file
View File

@ -0,0 +1,78 @@
import React from "react";
import { TextB, TextAa } from "@phosphor-icons/react";
import {
DEFAULT_CELL_FONT_SIZE,
bumpFontSize,
hasCustomFormat,
toggleBold,
} from "../lib/cellFormat";
export default function CellFormatToolbar({
selectionLabel,
format,
onChange,
onReset,
active = true,
}) {
const fontSize = format?.fontSize ?? DEFAULT_CELL_FONT_SIZE;
const bold = Boolean(format?.bold);
return (
<div
className="flex flex-wrap items-center gap-2 px-3 py-2 mb-3 border border-[#E5E7EB] bg-[#F9FAFB] rounded-sm"
data-testid="cell-format-toolbar"
>
<span className="text-[11px] text-[#52525B] mr-1">
Format:{" "}
<span className="font-medium text-[#18181B]">
{selectionLabel || "Select a field, summary value, or claim period"}
</span>
</span>
<div className="flex items-center gap-1 border border-[#D4D4D8] bg-white rounded-sm overflow-hidden">
<button
type="button"
disabled={!active}
className="px-2.5 py-1 text-[12px] hover:bg-[#F4F4F5] border-r border-[#E5E7EB] disabled:opacity-40"
title="Decrease font size"
onClick={() => onChange(bumpFontSize(format, -1))}
data-testid="cell-format-decrease"
>
A−
</button>
<button
type="button"
disabled={!active}
className="px-2.5 py-1 text-[12px] hover:bg-[#F4F4F5] border-r border-[#E5E7EB] disabled:opacity-40"
title="Increase font size"
onClick={() => onChange(bumpFontSize(format, 1))}
data-testid="cell-format-increase"
>
A+
</button>
<button
type="button"
disabled={!active}
className={`px-2.5 py-1 text-[12px] hover:bg-[#F4F4F5] disabled:opacity-40 ${bold ? "bg-[#E0E7FF] text-[#0F52BA]" : ""}`}
title="Bold"
onClick={() => onChange(toggleBold(format))}
data-testid="cell-format-bold"
>
<TextB size={14} weight={bold ? "bold" : "regular"} />
</button>
</div>
<span className="text-[11px] text-[#71717A] tabular-nums">
{active ? `${fontSize}pt${bold ? " · Bold" : ""}` : "—"}
</span>
{active && hasCustomFormat(format) && (
<button
type="button"
className="pf-btn pf-btn-ghost text-[11px] py-0.5"
onClick={onReset}
data-testid="cell-format-reset"
>
<TextAa size={12} /> Reset
</button>
)}
</div>
);
}

11
frontend/src/components/MileagePreview.jsx
View File

@ -2,7 +2,11 @@ import React, { useState, useEffect, useRef, useCallback } from 'react';
import { Printer, X, Download, FileSpreadsheet, Loader2 } from 'lucide-react';
import { api, apiErrorText } from '@/lib/api';
function buildClaimPayload(previewData, formData, entries, ratePerKm) {
function buildClaimPayload(claimPayload, previewData, formData, entries, ratePerKm) {
if (claimPayload?.claim_type === 'MILEAGE') {
return claimPayload;
}
if (previewData) {
return {
claim_type: 'MILEAGE',
@ -67,6 +71,7 @@ export default function MileagePreview({
ratePerKm = 0.70,
onPrint,
previewData,
claimPayload,
}) {
const [previewUrl, setPreviewUrl] = useState(null);
const [loading, setLoading] = useState(false);
@ -90,7 +95,7 @@ export default function MileagePreview({
}, [revokePreviewUrl]);
const loadPreview = useCallback(async () => {
const payload = buildClaimPayload(previewData, formData, entries, ratePerKm);
const payload = buildClaimPayload(claimPayload, previewData, formData, entries, ratePerKm);
const { blob, isPdf } = await fetchMileagePreviewBlob(payload);
revokePreviewUrl();
const url = URL.createObjectURL(blob);
@ -98,7 +103,7 @@ export default function MileagePreview({
setPreviewUrl(url);
setPreviewIsPdf(isPdf);
setError(null);
}, [previewData, formData, entries, ratePerKm, revokePreviewUrl]);
}, [claimPayload, previewData, formData, entries, ratePerKm, revokePreviewUrl]);
useEffect(() => {
if (!isOpen) {

19
frontend/src/lib/api.js
View File

@ -1,7 +1,13 @@
import axios from "axios";
/** Claim API — local default is 8002 (not legacy ProcureFlow on 8001). */
const BASE = (process.env.REACT_APP_BACKEND_URL || "http://127.0.0.1:8002").replace(/\/$/, "");
/** Claim API — local default is 8002 (not legacy ProcureFlow on 8001).
* When REACT_APP_BACKEND_URL is empty, use a relative base so requests go
* through the craco/webpack dev proxy (same-origin -> 127.0.0.1:8002, no CORS).
* When set, talk to that absolute origin (production API, or Docker same-origin
* where nginx proxies /api to the backend container). */
const RAW = process.env.REACT_APP_BACKEND_URL;
const IS_RELATIVE = !RAW || !RAW.trim();
const BASE = IS_RELATIVE ? "" : RAW.replace(/\/$/, "");
export const apiBaseUrl = BASE;
@ -12,8 +18,13 @@ export const api = axios.create({
withCredentials: true, // httpOnly auth cookies are the sole token transport
});
export const fileUrl = (path) =>
path?.startsWith("http") ? path : `${BASE}${path}`;
export const fileUrl = (path) => {
if (!path) return "";
if (path.startsWith("http") || IS_RELATIVE) {
return IS_RELATIVE && !path.startsWith("http") ? path : path;
}
return `${BASE}${path}`;
};
export function apiErrorText(err) {
const d = err?.response?.data?.detail;

141
frontend/src/lib/cellFormat.js Normal file
View File

@ -0,0 +1,141 @@
/** Per-cell font formatting (Excel-style toolbar). Stored on header/items as `_cellFormat`. */
export const DEFAULT_CELL_FONT_SIZE = 9;
export const CELL_FONT_MIN = 6;
export const CELL_FONT_MAX = 36;
export const CLAIM_PERIOD_FORMAT_KEY = "_claim_period";
export const MILEAGE_SUMMARY_FORMAT_KEYS = [
{ key: "_summary_total_km", label: "Total KM" },
{ key: "_summary_mileage", label: "Mileage (rate × km)" },
{ key: "_summary_toll", label: "Toll" },
{ key: "_summary_parking", label: "Parking" },
{ key: "_summary_grand_total", label: "Grand Total" },
];
export function getCellFormat(source, fieldKey) {
if (!source || !fieldKey) return {};
const map = source._cellFormat || {};
return map[fieldKey] || {};
}
export function cellFormatStyle(fmt) {
const style = {};
if (fmt?.fontSize) style.fontSize = `${fmt.fontSize}pt`;
if (fmt?.bold) style.fontWeight = "700";
return style;
}
export function isCellSelected(selection, type, key, rowIndex = null) {
if (!selection || selection.type !== type || selection.key !== key) return false;
if (type === "item") return selection.rowIndex === rowIndex;
return true;
}
export function selectionRingClass(selected) {
return selected ? "ring-2 ring-[#0F52BA] ring-offset-1" : "";
}
export function bumpFontSize(fmt, delta) {
const base = fmt.fontSize ?? DEFAULT_CELL_FONT_SIZE;
const next = Math.min(CELL_FONT_MAX, Math.max(CELL_FONT_MIN, base + delta));
return { ...fmt, fontSize: next };
}
export function toggleBold(fmt) {
return { ...fmt, bold: !fmt.bold };
}
export function hasCustomFormat(fmt) {
return Boolean(fmt && (fmt.fontSize != null || fmt.bold != null));
}
export function patchCellFormat(source, fieldKey, patch) {
const prev = getCellFormat(source, fieldKey);
const next = { ...prev, ...patch };
const cleaned = Object.fromEntries(
Object.entries(next).filter(([, v]) => v !== undefined && v !== null)
);
if (!hasCustomFormat(cleaned)) {
const map = { ...(source._cellFormat || {}) };
delete map[fieldKey];
if (Object.keys(map).length === 0) {
const { _cellFormat, ...rest } = source;
return rest;
}
return { ...source, _cellFormat: map };
}
return {
...source,
_cellFormat: {
...(source._cellFormat || {}),
[fieldKey]: cleaned,
},
};
}
export function clearCellFormat(source, fieldKey) {
const map = { ...(source._cellFormat || {}) };
delete map[fieldKey];
if (Object.keys(map).length === 0) {
const { _cellFormat, ...rest } = source;
return rest;
}
return { ...source, _cellFormat: map };
}
// ponytail: localStorage persistence — last-saved per-cell format remembered
// per claim type so new claims inherit it without re-applying size/bold each time.
const LAST_FORMAT_STORAGE_PREFIX = "claim:lastCellFormat:";
function isNonEmptyFormat(fmt) {
return Boolean(fmt && typeof fmt === "object" && Object.keys(fmt).length);
}
/** Snapshot a claim's header + line-item cell formats to localStorage (per claim type).
* For line items, the most recent row's format per field key wins. */
export function saveLastCellFormat(claimType, header, items) {
if (!claimType) return;
try {
const headerFmt = header?._cellFormat || {};
const itemFmt = {};
for (const it of items || []) {
const cf = it?._cellFormat || {};
for (const [k, v] of Object.entries(cf)) {
if (isNonEmptyFormat(v)) itemFmt[k] = v; // last row wins
}
}
localStorage.setItem(
LAST_FORMAT_STORAGE_PREFIX + claimType,
JSON.stringify({ header: headerFmt, item: itemFmt })
);
} catch {
/* storage may be unavailable (private mode / quota) — silently skip */
}
}
/** Read the last-saved cell formats for a claim type, or null if none. */
export function loadLastCellFormat(claimType) {
if (!claimType) return null;
try {
const raw = localStorage.getItem(LAST_FORMAT_STORAGE_PREFIX + claimType);
if (!raw) return null;
const parsed = JSON.parse(raw);
if (!parsed || typeof parsed !== "object") return null;
return { header: parsed.header || {}, item: parsed.item || {} };
} catch {
return null;
}
}
/** Apply remembered header format to a freshly built default header. */
export function applyStoredHeaderFormat(header, stored) {
if (!isNonEmptyFormat(stored?.header)) return header;
return { ...header, _cellFormat: { ...stored.header } };
}
/** Apply remembered line-item format to one row. */
export function applyStoredItemFormat(item, stored) {
if (!isNonEmptyFormat(stored?.item)) return item;
return { ...item, _cellFormat: { ...stored.item } };
}

313
frontend/src/pages/AdminClaimForm.jsx
View File

@ -37,16 +37,41 @@ import {
DotsSixVertical,
} from "@phosphor-icons/react";
import MileagePreview from "../components/MileagePreview";
import CellFormatToolbar from "../components/CellFormatToolbar";
import {
applyStoredHeaderFormat,
applyStoredItemFormat,
cellFormatStyle,
CLAIM_PERIOD_FORMAT_KEY,
clearCellFormat,
getCellFormat,
isCellSelected,
loadLastCellFormat,
MILEAGE_SUMMARY_FORMAT_KEYS,
patchCellFormat,
saveLastCellFormat,
selectionRingClass,
} from "../lib/cellFormat";
function MealTimeInput({ value, onChange, onBlurNormalize, testId, className = "pf-input" }) {
function MealTimeInput({
value,
onChange,
onBlurNormalize,
testId,
className = "pf-input",
onFocus,
style,
}) {
return (
<input
className={className}
style={style}
type="text"
placeholder={MEAL_TIME_PLACEHOLDER}
autoComplete="off"
value={formatMealTimeForDisplay(value)}
onChange={(e) => onChange(e.target.value)}
onFocus={onFocus}
onBlur={(e) => {
const stored = normalizeMealTimeStorage(e.target.value);
if (onBlurNormalize) onBlurNormalize(stored);
@ -74,6 +99,8 @@ function ClaimDateInput({
claimType = null,
className = "pf-input",
variant = "claim",
onFocus,
style,
}) {
const isLineItem = variant === "lineItem";
const lineStyle = isLineItem ? lineItemDateStyleForClaimType(claimType) : null;
@ -90,12 +117,14 @@ function ClaimDateInput({
return (
<input
className={className}
style={style}
type="date"
min={bounds.min}
max={bounds.max}
autoComplete="off"
value={isoValue}
onChange={(e) => onChange(e.target.value || "")}
onFocus={onFocus}
onBlur={(e) => {
if (!onBlurNormalize) return;
const iso = e.target.value || "";
@ -113,6 +142,7 @@ function ClaimDateInput({
return (
<input
className={className}
style={style}
type="text"
placeholder={placeholder}
inputMode="text"
@ -131,6 +161,7 @@ function ClaimDateInput({
const iso = claimDateToStorage(raw, period);
onChange(iso || raw);
}}
onFocus={onFocus}
onBlur={(e) => {
if (!onBlurNormalize) return;
const iso = isLineItem
@ -252,6 +283,16 @@ const ITEM_COLS = {
{ key: "description", label: "Description", type: "text" },
{ key: "meal", label: "Meal", type: "number" },
{ key: "accommodation", label: "Accommodation", type: "number" },
{
key: "breakfast_included",
label: "Breakfast Included",
type: "select",
options: [
{ value: "", label: "—" },
{ value: "yes", label: "Yes" },
{ value: "no", label: "No" },
],
},
],
MILEAGE: [
{ key: "date", label: "Date", type: "date" },
@ -443,27 +484,6 @@ const PREVIEW_DEBOUNCE_MS = 800;
function buildMileagePreviewPayload(header, items, period) {
return {
name: header.name || "",
empNo: header.emp_no || "",
month: period || "",
vehicleType: header.vehicle_type || "",
vehicleReg: header.vehicle_reg || "",
dateSigned: header.date_sign || "",
ratePerKm: parseFloat(header.rate_per_km) || 0.70,
entries: items.map((it) => ({
date: it.date || "",
from_place: it.from_place || "",
to_place: it.to_place || "",
purpose: it.purpose || "",
mileage: Number(it.km) || 0,
toll: Number(it.toll) || 0,
parking: Number(it.parking) || 0,
})),
};
}
async function fetchTemplatePreview(payload) {
const endpoint =
payload.claim_type === "OT"
@ -597,6 +617,7 @@ export default function AdminClaimForm() {
const [attUploading, setAttUploading] = useState(false);
const [attErr, setAttErr] = useState("");
const fileInputRef = useRef(null);
const [selectedCell, setSelectedCell] = useState(null);
const loadAttachments = useCallback(async (cid) => {
if (!cid) return;
@ -628,15 +649,6 @@ export default function AdminClaimForm() {
} catch (e) { setAttErr(apiErrorText(e)); }
};
const printIframeRef = useRef(null);
const cleanupPrintFrame = useCallback(() => {
if (printIframeRef.current) {
printIframeRef.current.remove();
printIframeRef.current = null;
}
}, []);
const fetchAttachmentBlob = async (att) => {
const res = await api.get(`/claims/${effectiveId}/attachments/${att.id}`, {
responseType: "blob",
@ -657,58 +669,26 @@ export default function AdminClaimForm() {
}
};
// ponytail: open the attachment in a new tab and let the user trigger Print
// themselves (Ctrl+P). The previous hidden-iframe + iframe.onload -> print()
// design auto-fired the OS print dialog the moment the iframe loaded — which
// surfaced as "print popup even though I didn't click Print" when the Printer
// icon (sandwiched between Open and Remove) was brushed. No more auto-print.
const printAttachment = async (att) => {
try {
setAttErr("");
const blob = await fetchAttachmentBlob(att);
const mime = blob.type || "";
const url = URL.createObjectURL(blob);
const canDirectPrint =
/^application\/pdf$/i.test(mime) || /^image\//i.test(mime);
if (!canDirectPrint) {
window.open(url, "_blank", "noopener,noreferrer");
setAttErr(
`${att.filename}: opened in a new tab — use Print from that window for Excel or other file types.`
);
setTimeout(() => URL.revokeObjectURL(url), 60_000);
return;
}
cleanupPrintFrame();
const iframe = document.createElement("iframe");
iframe.setAttribute("title", `Print ${att.filename}`);
iframe.style.cssText = "position:fixed;right:0;bottom:0;width:0;height:0;border:0;";
printIframeRef.current = iframe;
const revokeAndRemove = () => {
URL.revokeObjectURL(url);
cleanupPrintFrame();
};
iframe.onload = () => {
setTimeout(() => {
try {
iframe.contentWindow?.focus();
iframe.contentWindow?.print();
} catch {
window.open(url, "_blank", "noopener,noreferrer");
revokeAndRemove();
}
}, 300);
setTimeout(revokeAndRemove, 120_000);
};
iframe.onerror = revokeAndRemove;
iframe.src = url;
document.body.appendChild(iframe);
window.open(url, "_blank", "noopener,noreferrer");
setAttErr(
`${att.filename}: opened in a new tab — use Print (Ctrl+P) from that window.`
);
setTimeout(() => URL.revokeObjectURL(url), 60_000);
} catch (e) {
setAttErr(apiErrorText(e));
}
};
useEffect(() => () => cleanupPrintFrame(), [cleanupPrintFrame]);
const [previewUrl, setPreviewUrl] = useState("");
const [previewHtml, setPreviewHtml] = useState("");
const expensesLandscape = claimType === "EXPENSES";
@ -833,20 +813,132 @@ export default function AdminClaimForm() {
}).catch((e) => setErr(apiErrorText(e)));
}, [id, isEdit, refreshPreview, loadAttachments]);
// Reset items when type changes (only for new) + trigger preview
// Reset items when type changes (only for new) + trigger preview.
// New claims inherit the last-saved cell format for that claim type.
useEffect(() => {
if (!isEdit) {
const newItems = [blankItem(ITEM_COLS[claimType] || [])];
const stored = loadLastCellFormat(claimType);
const baseItems = [blankItem(ITEM_COLS[claimType] || [])];
const defaultH = { ...(DEFAULT_HEADER[claimType] || {}) };
const newItems = stored
? baseItems.map((it) => applyStoredItemFormat(it, stored))
: baseItems;
const restoredH = stored ? applyStoredHeaderFormat(defaultH, stored) : defaultH;
setItems(newItems);
setHeader(defaultH);
refreshPreview(claimType, period, defaultH, newItems, notes);
setHeader(restoredH);
refreshPreview(claimType, period, restoredH, newItems, notes);
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [claimType, isEdit]);
const headerFields = HEADER_FIELDS[claimType] || [];
const itemCols = ITEM_COLS[claimType] || [];
const selectionLabel = useMemo(() => {
if (!selectedCell) return "";
if (selectedCell.type === "period") return "Claim Period (Month)";
if (selectedCell.type === "summary") {
const found = MILEAGE_SUMMARY_FORMAT_KEYS.find((x) => x.key === selectedCell.key);
return `Summary · ${found?.label || selectedCell.key}`;
}
if (selectedCell.type === "header") {
const f = headerFields.find((x) => x.key === selectedCell.key);
return `Header · ${f?.label || selectedCell.key}`;
}
const col = itemCols.find((x) => x.key === selectedCell.key);
return `Row ${selectedCell.rowIndex + 1} · ${col?.label || selectedCell.key}`;
}, [selectedCell, headerFields, itemCols]);
const selectedFormatKey = useMemo(() => {
if (!selectedCell) return null;
if (selectedCell.type === "period") return CLAIM_PERIOD_FORMAT_KEY;
if (selectedCell.type === "summary") return selectedCell.key;
return selectedCell.key;
}, [selectedCell]);
const selectedFormat = useMemo(() => {
if (!selectedCell) return {};
if (selectedCell.type === "item") {
const row = items[selectedCell.rowIndex];
return row ? getCellFormat(row, selectedCell.key) : {};
}
return getCellFormat(header, selectedFormatKey);
}, [selectedCell, header, items, selectedFormatKey]);
const applyHeaderFormat = (formatKey, patch) => {
const nextHeader = patchCellFormat(header, formatKey, patch);
setHeader(nextHeader);
refreshPreview(claimType, period, nextHeader, items, notes);
};
const applySelectedFormat = (patch) => {
if (!selectedCell) return;
if (selectedCell.type === "item") {
const nextItems = items.map((row, i) =>
i === selectedCell.rowIndex ? patchCellFormat(row, selectedCell.key, patch) : row
);
setItems(nextItems);
refreshPreview(claimType, period, header, nextItems, notes);
return;
}
const formatKey =
selectedCell.type === "period" ? CLAIM_PERIOD_FORMAT_KEY : selectedCell.key;
applyHeaderFormat(formatKey, patch);
};
const resetSelectedFormat = () => {
if (!selectedCell) return;
if (selectedCell.type === "item") {
const nextItems = items.map((row, i) =>
i === selectedCell.rowIndex ? clearCellFormat(row, selectedCell.key) : row
);
setItems(nextItems);
refreshPreview(claimType, period, header, nextItems, notes);
return;
}
const formatKey =
selectedCell.type === "period" ? CLAIM_PERIOD_FORMAT_KEY : selectedCell.key;
const nextHeader = clearCellFormat(header, formatKey);
setHeader(nextHeader);
refreshPreview(claimType, period, nextHeader, items, notes);
};
const inputFormatProps = (scope, key, rowIndex = null, inputClass = "pf-input") => {
const formatKey = scope === "period" ? CLAIM_PERIOD_FORMAT_KEY : key;
const formatSource = scope === "item" ? items[rowIndex] : header;
return {
onFocus: () =>
setSelectedCell(
scope === "header"
? { type: "header", key }
: scope === "period"
? { type: "period", key: CLAIM_PERIOD_FORMAT_KEY }
: { type: "item", rowIndex, key }
),
style: cellFormatStyle(getCellFormat(formatSource, formatKey)),
className: `${inputClass} ${selectionRingClass(
scope === "period"
? selectedCell?.type === "period"
: isCellSelected(selectedCell, scope, key, rowIndex)
)}`.trim(),
};
};
const summaryValueProps = (formatKey) => ({
role: "button",
tabIndex: 0,
onClick: () => setSelectedCell({ type: "summary", key: formatKey }),
onKeyDown: (e) => {
if (e.key === "Enter" || e.key === " ") {
e.preventDefault();
setSelectedCell({ type: "summary", key: formatKey });
}
},
style: cellFormatStyle(getCellFormat(header, formatKey)),
className: `tabular-nums font-medium cursor-pointer rounded px-1 ${selectionRingClass(
selectedCell?.type === "summary" && selectedCell?.key === formatKey
)}`,
});
const total = computeTotal(claimType, header, items);
const updateHeader = (k, v) => {
@ -860,7 +952,14 @@ export default function AdminClaimForm() {
refreshPreview(claimType, period, header, next, notes);
};
const addItem = () => {
const next = [...items, blankItem(itemCols)];
// New rows inherit the previous row's cell format so all rows stay consistent.
const lastRow = items[items.length - 1];
const base = blankItem(itemCols);
const newItem =
lastRow?._cellFormat && Object.keys(lastRow._cellFormat).length
? { ...base, _cellFormat: { ...lastRow._cellFormat } }
: base;
const next = [...items, newItem];
setItems(next);
refreshPreview(claimType, period, header, next, notes);
};
@ -924,14 +1023,17 @@ export default function AdminClaimForm() {
if (isEdit && id) {
await api.put(`/claims/${id}`, payload());
setSavedId(id);
saveLastCellFormat(claimType, header, items);
return id;
}
if (savedId) {
await api.put(`/claims/${savedId}`, payload());
saveLastCellFormat(claimType, header, items);
return savedId;
}
const r = await api.post("/claims", payload());
setSavedId(r.data.id);
saveLastCellFormat(claimType, header, items);
return r.data.id;
};
@ -1008,7 +1110,7 @@ export default function AdminClaimForm() {
const current = payload();
if (claimType === "MILEAGE") {
setExcelPreviewPayload(buildMileagePreviewPayload(current.header, current.items, current.period));
setExcelPreviewPayload(current);
setExcelPreviewOpen(true);
return;
}
@ -1132,6 +1234,16 @@ export default function AdminClaimForm() {
{err && <div className="mx-6 mt-3 p-3 bg-[#FEF2F2] border border-[#FECACA] text-[#B91C1C] text-[13px]">{err}</div>}
{savedId && !err && <div className="mx-6 mt-3 p-3 bg-[#ECFDF5] border border-[#A7F3D0] text-[#047857] text-[13px]">Saved. PDF preview updated.</div>}
<div className="mx-6 mt-3 print:hidden">
<CellFormatToolbar
selectionLabel={selectionLabel}
format={selectedFormat}
onChange={applySelectedFormat}
onReset={resetSelectedFormat}
active={Boolean(selectedCell)}
/>
</div>
{/* Split layout */}
<div className={`flex flex-1 min-h-0 ${printMode ? "overflow-visible" : "overflow-hidden"} print:block print:overflow-visible`}>
@ -1162,7 +1274,7 @@ export default function AdminClaimForm() {
<label className="block">
<div className="text-[11px] uppercase tracking-[0.12em] text-[#71717A] mb-1.5 font-medium">Claim Period (Month)</div>
<input
className="pf-input"
{...inputFormatProps("period", CLAIM_PERIOD_FORMAT_KEY)}
type="month"
value={period}
onChange={(e) => {
@ -1184,6 +1296,7 @@ export default function AdminClaimForm() {
<div className="text-[11px] uppercase tracking-[0.12em] text-[#71717A] mb-1.5 font-medium">{f.label}</div>
{f.type === "date" && isClaimDateField(f.key) ? (
<ClaimDateInput
{...inputFormatProps("header", f.key)}
value={header[f.key] ?? ""}
onChange={(v) => updateHeader(f.key, v)}
onBlurNormalize={(iso) => {
@ -1213,7 +1326,7 @@ export default function AdminClaimForm() {
/>
) : (
<input
className="pf-input"
{...inputFormatProps("header", f.key)}
type={f.type === "number" ? "number" : "text"}
step={f.type === "number" ? "0.01" : undefined}
value={header[f.key] ?? ""}
@ -1303,7 +1416,7 @@ export default function AdminClaimForm() {
>
{c.type === "select" ? (
<select
className={lineItemInputClass("select")}
{...inputFormatProps("item", c.key, idx, lineItemInputClass("select"))}
value={it[c.key] ?? ""}
onChange={(e) => updateItem(idx, c.key, e.target.value)}
data-testid={`item-${idx}-${c.key}`}
@ -1317,7 +1430,7 @@ export default function AdminClaimForm() {
<ClaimDateInput
variant="lineItem"
claimType={claimType}
className={lineItemInputClass(c.type)}
{...inputFormatProps("item", c.key, idx, lineItemInputClass(c.type))}
value={it[c.key] ?? ""}
period={period}
onChange={(v) => updateItem(idx, c.key, v)}
@ -1329,7 +1442,7 @@ export default function AdminClaimForm() {
) : claimType === "MEAL" &&
(c.key === "time_depart" || c.key === "time_arrive") ? (
<MealTimeInput
className={lineItemInputClass("text")}
{...inputFormatProps("item", c.key, idx, lineItemInputClass("text"))}
value={it[c.key] ?? ""}
onChange={(v) => updateItem(idx, c.key, v)}
onBlurNormalize={(stored) => {
@ -1339,7 +1452,7 @@ export default function AdminClaimForm() {
/>
) : (
<input
className={lineItemInputClass(c.type)}
{...inputFormatProps("item", c.key, idx, lineItemInputClass(c.type))}
type={c.type === "number" ? "number" : "text"}
step={c.type === "number" ? "0.01" : undefined}
value={it[c.key] ?? ""}
@ -1376,20 +1489,20 @@ export default function AdminClaimForm() {
<>
<div className="flex justify-between px-3 py-1.5 border-b border-[#F4F4F5]">
<span className="text-[#52525B]">Total KM</span>
<span className="tabular-nums font-medium">{totalKm.toFixed(2)}</span>
<span {...summaryValueProps("_summary_total_km")}>{totalKm.toFixed(2)}</span>
</div>
<div className="flex justify-between px-3 py-1.5 border-b border-[#F4F4F5]">
<span className="text-[#52525B]">Mileage ({rate.toFixed(2)}/km)</span>
<span className="tabular-nums font-medium">{mileageRm.toFixed(2)}</span>
<span {...summaryValueProps("_summary_mileage")}>{mileageRm.toFixed(2)}</span>
</div>
{tollRm > 0 && <div className="flex justify-between px-3 py-1.5 border-b border-[#F4F4F5]">
<div className="flex justify-between px-3 py-1.5 border-b border-[#F4F4F5]">
<span className="text-[#52525B]">Toll</span>
<span className="tabular-nums font-medium">{tollRm.toFixed(2)}</span>
</div>}
{parkingRm > 0 && <div className="flex justify-between px-3 py-1.5 border-b border-[#F4F4F5]">
<span {...summaryValueProps("_summary_toll")}>{tollRm.toFixed(2)}</span>
</div>
<div className="flex justify-between px-3 py-1.5 border-b border-[#F4F4F5]">
<span className="text-[#52525B]">Parking</span>
<span className="tabular-nums font-medium">{parkingRm.toFixed(2)}</span>
</div>}
<span {...summaryValueProps("_summary_parking")}>{parkingRm.toFixed(2)}</span>
</div>
</>
);
})()}
@ -1401,9 +1514,15 @@ export default function AdminClaimForm() {
)}
<div className="flex justify-between px-3 py-2 bg-[#F9FAFB]">
<div className="font-semibold">{claimType === "OT" ? "Grand Total (hrs)" : "Grand Total (RM)"}</div>
<div className="tabular-nums font-bold">
{total.toLocaleString(undefined, { minimumFractionDigits: 2, maximumFractionDigits: 2 })}
</div>
{claimType === "MILEAGE" ? (
<span {...summaryValueProps("_summary_grand_total")}>
{total.toLocaleString(undefined, { minimumFractionDigits: 2, maximumFractionDigits: 2 })}
</span>
) : (
<div className="tabular-nums font-bold">
{total.toLocaleString(undefined, { minimumFractionDigits: 2, maximumFractionDigits: 2 })}
</div>
)}
</div>
</div>
</div>
@ -1639,7 +1758,7 @@ export default function AdminClaimForm() {
<MileagePreview
isOpen={excelPreviewOpen}
onClose={() => setExcelPreviewOpen(false)}
previewData={excelPreviewPayload}
claimPayload={excelPreviewPayload}
/>
</div>
);

5
frontend/src/pages/AdminClaimImport.jsx
View File

@ -205,6 +205,11 @@ export default function AdminClaimImport() {
{TYPE_LABELS[r.claim_type] || r.claim_type}
{r.period ? ` · ${periodLabel(r.period)}` : ""}
{r.name ? ` · ${r.name}` : ""}
{r.stored_filename ? (
<span className="block text-[11px] text-[#71717A] mt-0.5">
Stored as {r.stored_filename} in MongoDB
</span>
) : null}
</div>
)}
{!r.ok && <div className="text-[#B91C1C] mt-0.5">{r.error}</div>}

15
frontend/src/pages/AdminClaims.jsx
View File

@ -178,7 +178,12 @@ export default function AdminClaims() {
{CLAIM_TYPE_LABELS[c.claim_type] || c.claim_type}
</span>
{c.source === "UPLOADED" && (
<span className="pf-badge pf-badge-neutral" title={c.original_filename}>imported</span>
<span
className="pf-badge pf-badge-neutral"
title={c.filename || c.original_upload_name || c.original_filename}
>
imported
</span>
)}
</div>
</td>
@ -212,7 +217,13 @@ export default function AdminClaims() {
className="pf-btn pf-btn-ghost"
title="Download Excel"
data-testid={`xls-${c.id}`}
onClick={() => downloadExcel(c.id, c.source, c.claim_type, c.period, c.original_filename)}
onClick={() => downloadExcel(
c.id,
c.source,
c.claim_type,
c.period,
c.filename || c.original_upload_name || c.original_filename
)}
>
<FileXls size={14} />
</button>

151
frontend/src/pages/PaySlips.jsx
View File

@ -6,6 +6,7 @@ import {
CalendarBlank,
CircleNotch,
X,
DownloadSimple,
} from "@phosphor-icons/react";
import { api, apiBaseUrl, apiErrorText } from "../lib/api";
@ -40,6 +41,8 @@ export default function PaySlips() {
const [previewUrl, setPreviewUrl] = useState(null);
const [previewLoading, setPreviewLoading] = useState(false);
const [payslipDate, setPayslipDate] = useState("");
const [pendingFiles, setPendingFiles] = useState([]);
const [uploadProgress, setUploadProgress] = useState("");
const fileRef = useRef(null);
const load = useCallback(async () => {
@ -113,30 +116,76 @@ export default function PaySlips() {
};
}, [selected?.id]); // eslint-disable-line react-hooks/exhaustive-deps
const onUpload = async (fileList) => {
const file = fileList?.[0];
if (!file) return;
if (!file.name.toLowerCase().endsWith(".pdf")) {
const addPendingFiles = (incoming) => {
const pdfs = Array.from(incoming || []).filter((f) =>
f.name.toLowerCase().endsWith(".pdf")
);
if (!pdfs.length && incoming?.length) {
setErr("Only PDF pay slips can be uploaded.");
return;
}
setPendingFiles((prev) => {
const seen = new Set(prev.map((f) => `${f.name}:${f.size}`));
return [...prev, ...pdfs.filter((f) => !seen.has(`${f.name}:${f.size}`))];
});
setErr("");
};
const removePending = (idx) => {
setPendingFiles((prev) => prev.filter((_, i) => i !== idx));
};
const onUpload = async (fileList) => {
addPendingFiles(fileList);
if (fileRef.current) fileRef.current.value = "";
};
const uploadPending = async () => {
if (!pendingFiles.length) return;
setUploading(true);
setErr("");
setUploadProgress(`Uploading ${pendingFiles.length} file${pendingFiles.length === 1 ? "" : "s"}…`);
try {
const form = new FormData();
form.append("file", file);
pendingFiles.forEach((f) => form.append("files", f));
if (payslipDate) form.append("payslip_date", payslipDate);
const r = await api.post("/payslips", form, {
headers: { "Content-Type": "multipart/form-data" },
});
const uploaded = r.data.items || [];
const failed = r.data.errors || [];
await load();
setSelected(r.data);
if (uploaded.length) {
setSelected(uploaded[0]);
}
setPendingFiles([]);
setPayslipDate("");
if (fileRef.current) fileRef.current.value = "";
if (failed.length) {
setErr(
`${uploaded.length} uploaded, ${failed.length} failed: ` +
failed.map((x) => `${x.filename} (${x.error})`).join("; ")
);
}
} catch (e) {
setErr(apiErrorText(e));
} finally {
setUploading(false);
setUploadProgress("");
}
};
const downloadSelected = async () => {
if (!selected) return;
try {
const res = await api.get(`/payslips/${selected.id}/download`, { responseType: "blob" });
const url = URL.createObjectURL(new Blob([res.data], { type: "application/pdf" }));
const a = document.createElement("a");
a.href = url;
a.download = selected.filename || "payslip.pdf";
a.click();
URL.revokeObjectURL(url);
} catch (e) {
setErr(apiErrorText(e));
}
};
@ -159,7 +208,7 @@ export default function PaySlips() {
<div className="pf-overline mb-2">My space</div>
<h1 className="font-display text-[32px] font-bold tracking-tight leading-none">Pay Slips</h1>
<p className="text-[13px] text-[#52525B] mt-2 max-w-xl">
Store PDF pay slips only. Newest pay period first — click a file to view it here.
PDFs are stored in MongoDB as payslip_YYYYMM.pdf — safe if server files are removed. Upload many at once; newest first.
</p>
</div>
<div className="flex flex-wrap items-end gap-3">
@ -177,26 +226,78 @@ export default function PaySlips() {
ref={fileRef}
type="file"
accept="application/pdf,.pdf"
multiple
className="hidden"
data-testid="payslip-file-input"
onChange={(e) => onUpload(e.target.files)}
/>
<button
type="button"
className="pf-btn pf-btn-primary"
className="pf-btn pf-btn-secondary"
disabled={uploading}
onClick={() => fileRef.current?.click()}
data-testid="payslip-choose-btn"
>
<UploadSimple size={14} />
Choose PDFs
</button>
<button
type="button"
className="pf-btn pf-btn-primary"
disabled={uploading || pendingFiles.length === 0}
onClick={uploadPending}
data-testid="payslip-upload-btn"
>
{uploading ? (
<CircleNotch size={14} className="animate-spin" />
) : (
<UploadSimple size={14} />
<UploadSimple size={14} weight="bold" />
)}
{uploading ? "Uploading…" : "Upload PDF"}
{uploading
? uploadProgress || "Uploading…"
: pendingFiles.length
? `Upload ${pendingFiles.length} PDF${pendingFiles.length === 1 ? "" : "s"}`
: "Upload"}
</button>
</div>
</div>
{pendingFiles.length > 0 && (
<div className="mt-4 pf-surface p-3 max-w-2xl">
<div className="flex items-center justify-between mb-2">
<span className="text-[12px] font-medium text-[#52525B]">
Ready to upload ({pendingFiles.length})
</span>
<button
type="button"
className="text-[11px] text-[#71717A] hover:text-[#0A0A0B]"
onClick={() => setPendingFiles([])}
disabled={uploading}
>
Clear all
</button>
</div>
<ul className="space-y-1 max-h-[140px] overflow-y-auto">
{pendingFiles.map((f, idx) => (
<li
key={`${f.name}-${f.size}-${idx}`}
className="flex items-center gap-2 text-[12px] text-[#3F3F46]"
>
<FilePdf size={14} className="shrink-0 text-[#0F52BA]" />
<span className="truncate flex-1" title={f.name}>{f.name}</span>
<button
type="button"
className="text-[#A1A1AA] hover:text-[#B91C1C] p-1"
onClick={() => removePending(idx)}
disabled={uploading}
aria-label={`Remove ${f.name}`}
>
<X size={12} />
</button>
</li>
))}
</ul>
</div>
)}
{err && (
<div className="mt-4 text-[13px] text-[#B91C1C] bg-[#FEF2F2] border border-[#FECACA] px-3 py-2 rounded-sm">
{err}
@ -292,14 +393,26 @@ export default function PaySlips() {
Pay period: {formatPayslipDate(selected.payslip_date)}
</div>
</div>
<button
type="button"
className="pf-btn pf-btn-ghost shrink-0"
onClick={() => setSelected(null)}
title="Close preview"
>
<X size={14} />
</button>
<div className="flex gap-1 shrink-0">
<button
type="button"
className="pf-btn pf-btn-secondary"
onClick={downloadSelected}
title="Download from database"
data-testid="payslip-download-btn"
>
<DownloadSimple size={14} />
Download
</button>
<button
type="button"
className="pf-btn pf-btn-ghost"
onClick={() => setSelected(null)}
title="Close preview"
>
<X size={14} />
</button>
</div>
</div>
<div className="flex-1 relative min-h-0 p-3 sm:p-4">
{previewLoading && (

36
frontend/start.bat
View File

@ -1,31 +1,35 @@
@echo off
cd /d "%~dp0"
if not exist .env (
echo REACT_APP_BACKEND_URL=http://localhost:8002> .env
)
REM Bun is the JS runtime / package manager (replaces yarn). Ensure it's on PATH.
where bun >nul 2>&1
if errorlevel 1 set PATH=%PATH%;%USERPROFILE%\.bun\bin
REM Local dev uses the craco dev proxy: relative /api -> backend on 127.0.0.1:8002.
REM So REACT_APP_BACKEND_URL MUST be empty (otherwise the SPA hits the backend
REM cross-origin and needs CORS). This .env holds only this one var; rewrite it
REM each run so 1-click always works.
> .env echo REACT_APP_BACKEND_URL=
if not exist node_modules (
echo Installing frontend dependencies...
call yarn install
call bun install
if errorlevel 1 (
echo Failed to install dependencies with yarn.
echo Failed to install dependencies with bun.
exit /b 1
)
)
set PORT=
for /f "usebackq tokens=1,* delims==" %%a in (`findstr /B "PORT=" .env 2^>nul`) do set PORT=%%b
set PORT=%PORT:'=%
set PORT=%PORT:"=%
if not defined PORT set PORT=3000
netstat -ano | findstr ":%PORT%.*LISTENING" >nul 2>&1
REM craco.config.js sets devServer.port = 5000. CRA's early port-check reads
REM process.env.PORT (default 3000) BEFORE craco's override applies, so PORT must
REM be set to 5000 here too (3000 is taken by Grafana on this machine).
set FE_PORT=5000
set PORT=5000
netstat -ano | findstr ":%FE_PORT%.*LISTENING" >nul 2>&1
if %errorlevel%==0 (
echo Port %PORT% is already in use. Stop the other process or set PORT in frontend\.env
echo Port %FE_PORT% is already in use. Stop the other process or change devServer.port in craco.config.js
exit /b 1
)
echo Starting frontend on http://127.0.0.1:%PORT%
set PORT=%PORT%
call yarn start
echo Starting frontend on http://localhost:%FE_PORT% (API proxied to 127.0.0.1:8002)
call bun run start

11225
frontend/yarn.lock
View File

File diff suppressed because it is too large Load Diff

17363
package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

19
package.json
View File

@ -1,19 +0,0 @@
{
"name": "workspace",
"version": "1.0.0",
"description": "",
"main": "index.js",
"directories": {
"test": "tests"
},
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC",
"dependencies": {
"@craco/craco": "^7.1.0",
"react-scripts": "^5.0.1"
}
}

1
requirements.txt
View File

@ -1 +0,0 @@
markitdown[all]

BIN
rsync.pkg.tar.zst
View File

Binary file not shown.

35
start.bat Normal file
View File

@ -0,0 +1,35 @@
@echo off
setlocal
cd /d "%~dp0"
echo === Claim App - 1-click start (backend + frontend) ===
REM Bun is the frontend build/runtime. Ensure it's on PATH for child windows.
where bun >nul 2>&1
if errorlevel 1 set PATH=%PATH%;%USERPROFILE%\.bun\bin
REM Backend needs MongoDB. Warn if nothing is listening locally; cloud Atlas URLs are fine.
netstat -ano | findstr ":27017.*LISTENING" >nul 2>&1
if errorlevel 1 (
echo [WARN] Nothing listening on local port 27017.
echo If backend/.env MONGO_URL points to localhost, start MongoDB first.
echo (mongodb+srv Atlas URLs are not affected.)
)
REM Launch backend in its own window (uvicorn on 8002)
echo Starting backend -^> http://127.0.0.1:8002/api/health
start "Claim API" /D "%~dp0backend" cmd /k start-local.bat
REM Launch frontend in its own window (craco dev server on 5000, proxies /api -> 8002)
echo Starting frontend -^> http://localhost:5000
start "Claim Web" /D "%~dp0frontend" cmd /k start.bat
REM Give the dev server a moment to come up, then open the browser.
timeout /t 8 /nobreak >nul
start "" http://localhost:5000
echo.
echo Backend: http://127.0.0.1:8002/api/health
echo Frontend: http://localhost:5000 (API proxied to 127.0.0.1:8002)
echo Close the "Claim API" and "Claim Web" windows to stop the app.
endlocal

463
sync-cursor.sh
View File

@ -1,463 +0,0 @@
#!/bin/bash
# Script to sync .cursor directory from source to all destination directories
# One-way sync: Source -> All Destinations (respects .include file, ! prefix for exclusion)
# ID-based matching: Matches files by ID first, then filename (handles renames and ID updates)
# Source: ~/PycharmProjects/cursor-rules-and-prompts/.cursor
# Destinations: All other directories in ~/PycharmProjects/
set +e # Don't exit on error - we handle errors explicitly
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Paths
PYCHARM_DIR="$HOME/PycharmProjects"
SOURCE_DIR="/c/Source Code/windsurf/Procurement/Procurement"
SOURCE_CURSOR="$SOURCE_DIR/.cursor"
# Function to print colored messages
info() {
echo -e "${BLUE}[INFO]${NC} $1"
}
success() {
echo -e "${GREEN}[SUCCESS]${NC} $1"
}
warning() {
echo -e "${YELLOW}[WARNING]${NC} $1"
}
error() {
echo -e "${RED}[ERROR]${NC} $1"
}
# Function to get all destination directories
get_destinations() {
[ -d "$PYCHARM_DIR" ] && find "$PYCHARM_DIR" -maxdepth 1 -type d ! -path "$PYCHARM_DIR" ! -path "$SOURCE_DIR" | sort
}
# Function to extract ID from frontmatter in a file
# Returns the ID if found, empty string otherwise
extract_id_from_file() {
local file_path="$1"
if [ ! -f "$file_path" ]; then
return 1
fi
# Check if file starts with frontmatter delimiter
local first_line=$(head -n 1 "$file_path" 2>/dev/null)
if [ "$first_line" != "---" ]; then
return 1
fi
# Extract ID from frontmatter (look for "id: value" pattern)
local id=$(awk '
/^---$/ { in_frontmatter = !in_frontmatter; next }
in_frontmatter && /^id:[[:space:]]*/ {
sub(/^id:[[:space:]]*/, "")
gsub(/^[[:space:]]+|[[:space:]]+$/, "")
print
exit
}
' "$file_path" 2>/dev/null)
if [ -n "$id" ]; then
echo "$id"
return 0
fi
return 1
}
# Function to build ID-to-filename mapping for a directory
# Outputs: id|relative_path pairs, one per line
build_id_mapping() {
local dir_path="$1"
local relative_base="$2"
if [ ! -d "$dir_path" ]; then
return
fi
shopt -s nullglob dotglob
find "$dir_path" -type f \( -name "*.md" -o -name "*.mdc" \) | while read -r file_path; do
local relative_file="${file_path#$dir_path/}"
local relative_path="${relative_base:+$relative_base/}$relative_file"
# Skip .include file
local filename=$(basename "$relative_file")
if [ "$filename" = ".include" ]; then
continue
fi
local id=$(extract_id_from_file "$file_path")
if [ -n "$id" ]; then
echo "$id|$relative_path"
fi
done
shopt -u nullglob dotglob
}
# Function to find target file by ID
# Returns the relative path (from base) to the target file if found, empty otherwise
find_target_by_id() {
local search_id="$1"
local id_mapping_file="$2"
if [ ! -f "$id_mapping_file" ]; then
return 1
fi
# Search for the ID in the mapping
local matched_path=$(grep "^${search_id}|" "$id_mapping_file" 2>/dev/null | head -n 1 | cut -d'|' -f2-)
if [ -n "$matched_path" ]; then
echo "$matched_path"
return 0
fi
return 1
}
# Function to cleanup orphaned files in target
# Removes files that have IDs matching source IDs but are no longer in source,
# or files that were matched by ID but source file was removed
cleanup_orphaned_files() {
local dest_cursor="$1"
local source_id_map="$2"
local dest_id_map="$3"
if [ ! -f "$dest_id_map" ]; then
return 0
fi
info "Cleaning up orphaned files..."
local cleaned_count=0
# Process each file in target ID mapping
while IFS='|' read -r target_id target_relative; do
if [ -z "$target_id" ] || [ -z "$target_relative" ]; then
continue
fi
local target_file="$dest_cursor/$target_relative"
# Check if this ID exists in source
if [ -f "$source_id_map" ]; then
local source_match=$(grep "^${target_id}|" "$source_id_map" 2>/dev/null | head -n 1)
if [ -z "$source_match" ]; then
# ID exists in target but not in source - this is an orphaned file
if [ -f "$target_file" ]; then
warning "Removing orphaned file: $target_relative (ID: $target_id)"
rm -f "$target_file"
cleaned_count=$((cleaned_count + 1))
fi
fi
fi
done < "$dest_id_map"
if [ $cleaned_count -eq 0 ]; then
success "No orphaned files found"
else
success "Removed $cleaned_count orphaned file(s)"
fi
}
# Function to process ID-based matching and renames after rsync
# Handles file renames based on ID matching
# Processes whatever directories exist after rsync (path-based syncing)
process_id_based_matching() {
local dest_cursor="$1"
local source_cursor="$2"
local temp_dir=$(mktemp -d)
info "Building ID mappings for ID-based matching..."
# Process each directory type (rules, prompts)
for dir_type in rules prompts; do
local source_dir="$source_cursor/$dir_type"
local dest_dir="$dest_cursor/$dir_type"
local source_id_map="$temp_dir/source_${dir_type}_id_map.txt"
local dest_id_map="$temp_dir/dest_${dir_type}_id_map.txt"
# Build ID mappings if directories exist
[ -d "$source_dir" ] && build_id_mapping "$source_dir" "$dir_type" > "$source_id_map"
[ -d "$dest_dir" ] && build_id_mapping "$dest_dir" "$dir_type" > "$dest_id_map"
# Process ID matching if both source and dest exist with mappings
if [ -d "$source_dir" ] && [ -d "$dest_dir" ] && [ -f "$source_id_map" ] && [ -f "$dest_id_map" ]; then
process_directory_id_matching "$source_dir" "$dest_dir" "$source_id_map" "$dest_id_map"
# Rebuild dest mapping after processing (in case files were renamed)
build_id_mapping "$dest_dir" "$dir_type" > "$dest_id_map"
fi
# Cleanup orphaned files
[ -f "$source_id_map" ] && [ -f "$dest_id_map" ] && \
cleanup_orphaned_files "$dest_cursor" "$source_id_map" "$dest_id_map"
done
rm -rf "$temp_dir"
}
# Function to process ID-based matching for a directory
process_directory_id_matching() {
local source_dir="$1" dest_dir="$2" source_id_map="$3" dest_id_map="$4"
[ ! -d "$source_dir" ] || [ ! -d "$dest_dir" ] && return
shopt -s nullglob dotglob
find "$source_dir" -type f \( -name "*.md" -o -name "*.mdc" \) | while read -r source_file; do
local source_relative="${source_file#$source_dir/}"
local source_id=$(extract_id_from_file "$source_file")
[ -z "$source_id" ] && continue
local matched_relative=$(find_target_by_id "$source_id" "$dest_id_map")
local dest_file="$dest_dir/$source_relative"
if [ -n "$matched_relative" ]; then
local matched_file="$dest_dir/$matched_relative"
if [ "$(basename "$matched_file")" != "$(basename "$source_file")" ] && [ -f "$matched_file" ]; then
info "File renamed by ID: $matched_relative → $source_relative (ID: $source_id)"
rm -f "$matched_file"
elif [ -f "$dest_file" ]; then
local dest_id=$(extract_id_from_file "$dest_file")
[ -n "$dest_id" ] && [ "$dest_id" != "$source_id" ] && \
info "ID updated: $source_relative ($dest_id → $source_id)"
fi
elif [ -f "$dest_file" ]; then
local dest_id=$(extract_id_from_file "$dest_file")
if [ -n "$source_id" ] && [ -n "$dest_id" ] && [ "$source_id" != "$dest_id" ]; then
info "ID updated: $source_relative ($dest_id → $source_id)"
elif [ -n "$source_id" ] && [ -z "$dest_id" ]; then
info "ID added: $source_relative ($source_id)"
fi
fi
done
shopt -u nullglob dotglob
}
# Helper function to check if a file has non-comment, non-empty lines
has_content() {
local file="$1"
[ -f "$file" ] && [ -s "$file" ] && grep -v '^[[:space:]]*#' "$file" | grep -v '^[[:space:]]*$' | grep -q .
}
# Function to check if .include file exists and has content
has_include_patterns() {
local target_dir="$1"
has_content "$SOURCE_CURSOR/.include" || has_content "$target_dir/.cursor/.include"
}
# Helper function to read patterns from .include file
read_patterns() {
local include_file="$1"
local include_out="$2"
local exclude_out="$3"
[ ! -f "$include_file" ] && return
while IFS= read -r line || [ -n "$line" ]; do
[[ -z "$line" || "$line" =~ ^[[:space:]]*# ]] && continue
line=$(echo "$line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
[ -z "$line" ] && continue
if [[ "$line" =~ ^! ]]; then
echo "${line#!}" >> "$exclude_out"
else
echo "$line" >> "$include_out"
fi
done < "$include_file"
}
# Function to build rsync include and exclude files from .include file
# Patterns without ! are includes, patterns with ! prefix are excludes
build_include_exclude_files() {
local target_dir="$1"
local include_file=$(mktemp)
local exclude_file=$(mktemp)
# Always exclude .include file itself
echo ".include" >> "$exclude_file"
# Process both source and target .include files
read_patterns "$SOURCE_CURSOR/.include" "$include_file" "$exclude_file"
read_patterns "$target_dir/.cursor/.include" "$include_file" "$exclude_file"
# Expand patterns: add parent directories and wildcards for rsync
# For directory patterns, recursively include all subdirectories at any depth
expand_patterns() {
local input_file="$1"
local output_file="$2"
if [ -s "$input_file" ]; then
local temp_expanded=$(mktemp)
while IFS= read -r pattern; do
[ -z "$pattern" ] && continue
# If pattern ends with /, it's a directory - include recursively
if [[ "$pattern" == */ ]]; then
local dir_pattern="${pattern%/}"
# Include the directory itself
echo "$dir_pattern/" >> "$temp_expanded"
# Include all immediate contents (files and directories)
echo "${dir_pattern}/*" >> "$temp_expanded"
# For recursive inclusion, we need patterns that match subdirectories at each level
# Key: include subdirectories with / so rsync traverses into them
# Generate recursive patterns: dir/*/, dir/*/*/, dir/*/*/*/, etc.
local recursive_pattern="${dir_pattern}"
for i in {1..20}; do
recursive_pattern="${recursive_pattern}/*"
# Include files at this depth
echo "${recursive_pattern}" >> "$temp_expanded"
# Include subdirectories at this depth (critical for traversal)
echo "${recursive_pattern}/" >> "$temp_expanded"
done
else
# Regular pattern (file or directory without trailing /)
echo "$pattern" >> "$temp_expanded"
# If it might be a directory (no extension and no slash), add recursive patterns
if [[ "$pattern" != *.* ]] && [[ "$pattern" != */* ]]; then
echo "${pattern}/" >> "$temp_expanded"
echo "${pattern}/*" >> "$temp_expanded"
local recursive_pattern="${pattern}"
for i in {1..20}; do
recursive_pattern="${recursive_pattern}/*"
echo "${recursive_pattern}" >> "$temp_expanded"
echo "${recursive_pattern}/" >> "$temp_expanded"
done
fi
fi
# Add parent directories for rsync traversal
local parent="${pattern%/}"
while [[ "$parent" == */* ]]; do
parent="${parent%/*}"
[ -n "$parent" ] && echo "${parent}/" >> "$temp_expanded"
done
done < "$input_file"
sort -u "$temp_expanded" | grep -v '^$' > "$output_file"
rm -f "$temp_expanded"
fi
}
expand_patterns "$include_file" "$include_file"
expand_patterns "$exclude_file" "$exclude_file"
# Check for conflicts (same pattern in both include and exclude)
if [ -s "$include_file" ] && [ -s "$exclude_file" ]; then
while IFS= read -r include_pattern; do
if grep -Fxq "$include_pattern" "$exclude_file" 2>/dev/null; then
warning "Pattern '$include_pattern' found in both include and exclude - include takes precedence"
fi
done < "$include_file"
fi
echo "$include_file|$exclude_file"
}
# Function to sync from source to destination (one-way sync: source -> destination)
sync_source_to_dest() {
local dest_dir="$1"
local files_result
local include_file
local exclude_file
local rsync_exit=0
# Build include and exclude files from .include
files_result=$(build_include_exclude_files "$dest_dir")
include_file=$(echo "$files_result" | cut -d'|' -f1)
exclude_file=$(echo "$files_result" | cut -d'|' -f2)
info "Syncing to $dest_dir..."
if [ -n "$include_file" ] && [ -s "$include_file" ]; then
# Whitelist mode: only include patterns from .include
rsync -av --delete --exclude=".include" --include-from="$include_file" --exclude='*' "$SOURCE_CURSOR/" "$dest_dir/.cursor/" 2>&1 | awk '/^Transfer starting:/{print; getline; if(/^$/) getline; print; next}1'
rsync_exit=${PIPESTATUS[0]}
elif [ -s "$exclude_file" ]; then
# Blacklist mode: exclude patterns from .include (with ! prefix)
rsync -av --delete --exclude-from="$exclude_file" "$SOURCE_CURSOR/" "$dest_dir/.cursor/" 2>&1 | awk '/^Transfer starting:/{print; getline; if(/^$/) getline; print; next}1'
rsync_exit=${PIPESTATUS[0]}
else
# Normal mode: sync everything (no .include file or empty)
rsync -av --delete --exclude=".include" "$SOURCE_CURSOR/" "$dest_dir/.cursor/" 2>&1 | awk '/^Transfer starting:/{print; getline; if(/^$/) getline; print; next}1'
rsync_exit=${PIPESTATUS[0]}
fi
# Clean up temp files
rm -f "$exclude_file" "$include_file"
# Process ID-based matching after rsync
# Process whatever directories exist (path-based syncing via .include)
process_id_based_matching "$dest_dir/.cursor" "$SOURCE_CURSOR" || true
return $rsync_exit
}
# Main execution
main() {
local target_dir="$1" # Optional: specific directory to sync to
info "Starting .cursor directory sync..."
[ ! -d "$SOURCE_CURSOR" ] && error "Source directory does not exist: $SOURCE_CURSOR" && exit 1
# If a specific directory is provided, sync only to that directory
if [ -n "$target_dir" ]; then
[ ! -d "$target_dir" ] && error "Directory does not exist: $target_dir" && exit 1
target_dir=$(cd "$target_dir" && pwd)
[ "$target_dir" = "$SOURCE_DIR" ] && error "Cannot sync to source directory: $target_dir" && exit 1
[ ! -d "$target_dir/.cursor" ] && error "Directory does not have .cursor folder: $target_dir" && exit 1
if ! has_include_patterns "$target_dir"; then
warning "No .include file found or .include file is empty in $target_dir/.cursor - skipping sync"
warning "You have not included or excluded any rules and prompts paths"
success "Sync completed!"
return
fi
info "Syncing to specific directory: $target_dir"
sync_source_to_dest "$target_dir" && success "Sync completed!" || (error "Sync failed for $target_dir" && exit 1)
return
fi
# Sync to all destinations
info "Syncing source to all destinations..."
# Collect valid destinations (with .cursor and .include files)
local valid_destinations=()
while IFS= read -r dest_dir; do
[ -z "$dest_dir" ] || [ ! -d "$dest_dir/.cursor" ] && continue
has_include_patterns "$dest_dir" && valid_destinations+=("$dest_dir")
done < <(get_destinations)
if [ ${#valid_destinations[@]} -eq 0 ]; then
warning "No destination directories found with .include file or .include files are empty"
warning "You have not included or excluded any rules and prompts paths"
success "Sync completed!"
return
fi
local total=${#valid_destinations[@]}
local dest_word="destination"
[ $total -ne 1 ] && dest_word="destinations"
local count=0
for dest_dir in "${valid_destinations[@]}"; do
if sync_source_to_dest "$dest_dir"; then
count=$((count + 1))
info "Synced to $count/$total $dest_word"
echo ""
fi
done
success "Sync completed!"
}
# Run main function
main "$@"

4
test-results/.last-run.json
View File

@ -1,4 +0,0 @@
{
"status": "failed",
"failedTests": []
}

55
test_ot_preview.spec.js
View File

@ -1,55 +0,0 @@
import { test, expect, chromium } from '@playwright/test';
test('OT claim live preview shows correct data', async () => {
const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
try {
await page.goto('http://localhost:3000');
await page.waitForLoadState('networkidle');
// Click Admin Claim Form
await page.click('text=Admin Claim Form');
await page.waitForTimeout(2000);
// Select OT type
await page.selectOption('select', 'OT');
await page.waitForTimeout(1500);
// Fill all input fields we can find
const inputs = await page.locator('input').all();
console.log(`Found ${inputs.length} inputs`);
for (const input of inputs) {
const placeholder = await input.getAttribute('placeholder') || '';
const ariaLabel = await input.getAttribute('aria-label') || '';
if (placeholder.toLowerCase().includes('name') || ariaLabel.toLowerCase().includes('name')) {
await input.fill('AHMAD OTHMAN');
}
if (placeholder.toLowerCase().includes('emp')) {
await input.fill('EMP001');
}
if (placeholder.toLowerCase().includes('department') || placeholder.toLowerCase().includes('dept')) {
await input.fill('SOFTWARE');
}
}
await page.waitForTimeout(2000);
// Get preview content
const previewFrame = page.frameLocator('iframe').first();
const content = await previewFrame.locator('body').innerText({ timeout: 5000 }).catch(() => '');
console.log('Preview content:');
console.log(content.substring(0, 3000));
// Assertions
expect(content).toContain('SOFTWARE');
expect(content).toContain('AHMAD OTHMAN');
expect(content).toContain('Requested By');
} finally {
await browser.close();
}
});

0
test_reports/.gitkeep
View File

42
test_reports/iteration_1.json
View File

@ -1,42 +0,0 @@
{
"summary": "Full regression tested ProcureFlow backend via public REACT_APP_BACKEND_URL. 17/17 pytest tests passed covering root, templates (list + single + 404), dashboard stats, upload validation, PDF upload, process (OCR + classification + Gemini extraction on real sample QUOTATION), list/filter, get by id, review (REVIEWED and FINAL), manual PO create, PDF render, original file stream, and delete (doc + file).",
"backend_issues": {
"critical": [],
"minor": [
{
"endpoint": "POST /api/documents/{id}/process",
"issue": "LLM extraction for the sample quotation left header.quotation_number='' while placing 'QSSB/AFASS/BD/042026/3015/ZZ' into header.reference_number. The identifier is captured but in the wrong schema field, which will surprise downstream consumers / PDF render that keys off quotation_number. Consider tightening the prompt or post-processing to copy reference_number -> quotation_number when the latter is empty. (Non-deterministic: E1 manually saw it populate quotation_number previously.)"
}
]
},
"frontend_issues": {
"ui_bugs": [],
"integration_issues": [],
"design_issues": []
},
"test_report_links": [
"/app/backend/tests/test_procureflow_api.py",
"/app/test_reports/pytest/pytest_results.xml"
],
"action_items": [
"Optional: harden extraction_service.py so that when quotation_number is blank but reference_number is present, copy it across (or instruct the LLM to treat the top-right Q-number as quotation_number for QUOTATION docs)."
],
"critical_code_review_comments": [
"server.py line 30/32: os.environ['MONGO_URL'] / os.environ['DB_NAME'] raise KeyError at import time if missing - acceptable fail-fast, but combined with load_dotenv this is fine.",
"process_document merges existing + updates and returns datetimes as ISO strings after _deserialize tries to parse them - inconsistent shape (other GETs return ISO strings via _serialize). Consider returning the freshly fetched doc to keep response schema uniform.",
"list_documents upper-cases filter values but does not validate against DOC_TYPES/DOC_STATUSES/DOC_SOURCES enums - arbitrary filter values silently return [].",
"review_document does not validate payload.status against DOC_STATUSES and does not validate payload.type against DOC_TYPES - caller could set any string.",
"No size/MIME-type cap on /documents/upload beyond extension check; a large malicious upload is written chunk by chunk without limit.",
"Response for process_document returns raw_text which can be large; list view strips it but detail view keeps it - fine, but be aware of payload size.",
"Delete endpoint returns {'deleted': id} with 200; REST convention would be 204 No Content."
],
"updated_files": [
"/app/backend/tests/test_procureflow_api.py (new)"
],
"success_rate": {"backend": "100%", "frontend": "not tested (skipped per request)"},
"test_credentials": "N/A - no authentication in app",
"seed_data_creation": "Each test creates its own TEST_ prefixed manual PO and uploads sample quotation; both are deleted by the final delete tests, so DB is left clean.",
"retest_needed": false,
"main_agent_can_self_test": true,
"context_for_next_testing_agent": "Regression file lives at /app/backend/tests/test_procureflow_api.py. Sample PDF is fetched from /tmp/sample_quotation.pdf; if rerunning, download again from https://customer-assets.emergentagent.com/job_smart-procurement-31/artifacts/4fp2xmmo_QUO_AFA_AD-HOC%20MAINTENANCE%20-%20GOM%2C%20T11.pdf. Extraction assertion is loose: accepts QSSB in either quotation_number or reference_number because Gemini flips between them."
}

46
test_reports/iteration_2.json
View File

@ -1,46 +0,0 @@
{
"summary": "Phase 2 backend regression + new-feature suite — 38/38 pytest tests passing. Covers auth (login/register/me/logout/refresh/brute-force), RBAC (admin/manager/user/viewer), INVOICE template, bulk upload + BackgroundTasks → EXTRACTED, server-side pagination + regex search, OCR digital path on the real quotation PDF, manual INVOICE create + PDF, Resend 503 when not configured, admin user mgmt, self-demote/delete guards, and audit log. Fixed one bug in server.py — upload/create routes leaked MongoDB ObjectId in the response causing 500 on /documents/upload and /documents/create.",
"backend_issues": {
"critical": [],
"minor": [
{
"endpoint": "POST /api/auth/login",
"issue": "Brute-force lockout identifier uses `request.client.host + email`. Behind the Kubernetes ingress the app is served by multiple backend pods; each pod sees a different proxy-chain IP as request.client.host, so the 5-failure counter is partitioned per-pod and users can submit 5*N wrong attempts before any single pod locks them. First 6 sequential wrong logins returned 401x6; lockout only triggers once one pod alone has accumulated 5 fails (observed after ~10–15 attempts in testing). Fix: key on email (plus X-Forwarded-For) instead of request.client.host, or move counter to a shared TTL document keyed by email only."
}
]
},
"frontend_issues": {
"ui_bugs": [],
"integration_issues": [],
"design_issues": []
},
"test_report_links": [
"/app/backend/tests/test_procureflow_phase2.py",
"/app/test_reports/pytest/phase2_results.xml"
],
"action_items": [
"Harden brute-force lockout so it works behind a multi-replica ingress (key by email and/or X-Forwarded-For)."
],
"critical_code_review_comments": [
"server.py: `_save_upload` and `create_manual_document` called `db.documents.insert_one(stored)` and then returned `stored` — Motor mutates the dict to attach `_id` (ObjectId), which is not JSON-serializable and was causing 500 on /api/documents/upload and /api/documents/create. Fixed by popping _id after insert.",
"server.py line 184: `__import__('datetime').timedelta(...)` is an awkward import dodge — `datetime.timedelta` is already usable because `datetime` the module was imported at top; please use the normal form.",
"server.py:633 uses deprecated `@app.on_event('startup')` / `shutdown` — migrate to FastAPI lifespan context to avoid future-deprecation warnings.",
"CORS config: with `allow_credentials=True` a single origin is used (FRONTEND_URL), which is correct; however the fallback `CORS_ORIGINS='*'` would be invalid when credentials are enabled. Consider asserting FRONTEND_URL is set in production.",
"Login route stores `login_attempts.locked_until` as an ISO string and compares with aware-datetime. Fine today but more robust to store as real BSON datetime so TTL indexes can auto-expire stale lockouts.",
"auth_service.py set_auth_cookies uses samesite='none'; secure=True — good for cross-origin preview. Ensure production also serves on HTTPS (it does)."
],
"updated_files": [
"/app/backend/server.py (fix: strip _id after insert_one in upload + manual create)",
"/app/backend/tests/test_procureflow_phase2.py (new pytest suite)"
],
"success_rate": {
"backend": "100%",
"frontend": "not tested (scope: backend only)"
},
"test_credentials": "Admin syazwan.zulkifli@quatriz.com.my / Admin@123 — verified working. Additional viewer/user/manager test accounts are created+torn-down per session via POST /api/auth/register and PUT /api/admin/users/{id}/role.",
"seed_data_creation": "No persistent seed. Fixtures upload/delete docs and register/delete users within each run.",
"retest_needed": false,
"main_agent_can_self_test": true,
"context_for_next_testing_agent": "Run `pytest /app/backend/tests/test_procureflow_phase2.py -v` — suite uses REACT_APP_BACKEND_URL from /app/frontend/.env, pulls the real sample quotation PDF, and validates every Phase 2 capability. If RESEND_API_KEY gets populated in .env the test_email_without_resend_returns_503 test will start failing — swap to asserting 200/OK in that case. Brute-force lockout test tolerates multi-pod LB by trying up to 25 wrong attempts.",
"rca of the issue": "Upload/create 500s were caused by returning the dict that Motor's insert_one mutated with a raw ObjectId `_id`. FastAPI/Pydantic cannot serialize ObjectId, so response serialization raised `TypeError: ObjectId object is not iterable` → 500. Reproduction: POST /api/documents/upload with valid PDF + admin Bearer token. Mitigation applied: `stored.pop('_id', None)` right after insert_one in both _save_upload and create_manual_document."
}

59
test_reports/iteration_3.json
View File

@ -1,59 +0,0 @@
{
"summary": "Phase 3 backend testing complete. Added /app/backend/tests/test_procureflow_phase3.py covering password-reset flow (happy/expired/invalid/reused tokens, DB token inspection, audit actions), admin template editor CRUD (create/update/delete, built-in reset vs custom delete, schema validation, document_type regex, RBAC, overlay precedence, audit actions), and Celery+Redis bulk pipeline (runner=celery, queue-status RBAC, EXTRACTED convergence on real sample PDF). Phase 3 pytest: 19/19 passing. Phase 2 regression suite still passing 38/38. Combined current regression: 57/57. Admin password remains 'Admin@123' — all reset tests used a throwaway user.",
"backend_issues": {
"critical": [],
"minor": [
{
"endpoint": "POST /api/admin/templates",
"issue": "document_type regex [A-Z0-9_]{2,32} is applied AFTER .upper(), so lowercase payloads like 'lowercase' pass validation and get stored as 'LOWERCASE'. Functionally fine, but if strict case enforcement was intended (spec said pattern must be [A-Z0-9_]) the regex should run on the raw input."
},
{
"endpoint": "POST /api/auth/login",
"issue": "(Carried over from iteration 2) Brute-force lockout keys on request.client.host+email; behind multi-replica ingress lockout partitions per pod. Not re-tested here."
},
{
"endpoint": "Legacy /app/backend/tests/test_procureflow_api.py",
"issue": "13/13 legacy pre-auth Phase 1 tests fail with 401 because they send no Authorization header. Not the current regression suite but clutters pytest output. Recommend removing or porting to use admin_token fixture."
}
]
},
"frontend_issues": {
"ui_bugs": [],
"integration_issues": [],
"design_issues": []
},
"test_report_links": [
"/app/backend/tests/test_procureflow_phase3.py",
"/app/backend/tests/test_procureflow_phase2.py",
"/app/test_reports/pytest/phase3_results.xml",
"/app/test_reports/pytest/phase2_results.xml",
"/app/test_reports/pytest/pytest_results.xml"
],
"action_items": [
"Optional: tighten document_type regex to reject lowercase raw input if strict case was intended.",
"Optional: delete or update /app/backend/tests/test_procureflow_api.py (legacy unauth Phase 1 suite — all 13 tests fail against the auth-protected API).",
"(Carried over) Harden brute-force lockout to be replica-safe."
],
"critical_code_review_comments": [
"server.py create_template: payload.document_type.upper() is regex-checked AFTER uppercasing — any lowercase input silently becomes uppercase. Either validate the raw value or document behavior in the API schema.",
"server.py delete_template on built-ins: comment says 'Put the factory default back on disk' but no file write happens — the code path just removes the overlay (which IS the correct behavior). Update the stale comment.",
"celery_app.py process_document_task: creates a fresh AsyncIOMotorClient per task and calls asyncio.run on every execution — this works but incurs a new event loop + Mongo handshake per doc. For throughput consider a single persistent loop + client on the worker.",
"server.py forgot_password/reset_password: expires_at stored as ISO string and compared via datetime.fromisoformat; fine but prevents MongoDB TTL index auto-expiry. Consider storing as BSON datetime and adding a TTL index on `expires_at`.",
"server.py bulk_upload: `runner` is overwritten per iteration of the for-loop — if the first file succeeds via celery but a later one fails over to BackgroundTasks (or vice-versa), the response only reflects the last status. Aggregate or return per-item runner info.",
"server.py line 184 (carried): `__import__('datetime').timedelta(...)` — use normal `from datetime import timedelta`.",
"server.py:854 startup still uses deprecated @app.on_event — migrate to FastAPI lifespan."
],
"updated_files": [
"/app/backend/tests/test_procureflow_phase3.py (new Phase 3 suite — 19 tests)"
],
"success_rate": {
"backend": "100% on current regression (phase2 38/38 + phase3 19/19 = 57/57)",
"frontend": "not tested (scope: backend only)"
},
"test_credentials": "Admin: syazwan.zulkifli@quatriz.com.my / Admin@123 — verified working and UNCHANGED (reset tests used a throwaway registered user that is deleted on teardown).",
"seed_data_creation": "Phase 3 suite auto-creates and deletes: one throwaway user for password-reset, one non-admin user for RBAC checks, one custom template (TEST_XXXXXX), and 2 bulk-uploaded docs. All cleaned up on teardown.",
"retest_needed": false,
"main_agent_can_self_test": true,
"context_for_next_testing_agent": "Run `pytest /app/backend/tests/test_procureflow_phase2.py /app/backend/tests/test_procureflow_phase3.py -v` for full regression (57 tests, ~90s). Phase 3 requires redis + celery supervisor workers up (check with `sudo supervisorctl status`). The sample PDF is fetched live from customer-assets.emergentagent.com. Password-reset tests assert DB contents directly via pymongo using MONGO_URL+DB_NAME from backend/.env; email_configured() is False (RESEND_API_KEY empty), so tests confirm the forgot-password endpoint still returns 200 and logs the reset link instead of sending. Do NOT run test_procureflow_api.py — it is the legacy Phase 1 unauth suite and every test fails with 401 by design now that auth is required.",
"rca of the issue": "No functional bugs discovered in Phase 3. Two test authoring issues fixed during this run: (a) the public GET /api/templates endpoint returns {\"templates\":[...]} (a dict), not a bare array, so list comprehensions had to index payload[\"templates\"]; (b) the spec implies strict [A-Z0-9_] on document_type but the backend uppercases first, so 'lowercase' is accepted — test adjusted to only cover truly invalid inputs and the behavior is noted as a minor deviation from spec."
}

51
test_reports/iteration_4.json
View File

@ -1,51 +0,0 @@
{
"summary": "Phase 4 hardening pass validated. Ran full regression (Phase 2 + Phase 3 = 57/57 green) plus added /app/backend/tests/test_procureflow_phase4_hardening.py (8 new tests, 8/8 green) to specifically cover the code-review hardening: cookie-only auth, Bearer backwards compat, AuditEvent.actor_* population for USER_REGISTER + PASSWORD_RESET_COMPLETED, TemplatePayload alias='schema', email endpoint 503 when Resend unconfigured. Code inspection confirms _check_lockout / _record_failure / _reference_of / _render_pdf_or_400 helpers exist, AuditEvent dataclass + log_from_user wrapper exist, frontend api.js has no Bearer interceptor and only withCredentials:true. Grand total: 65/65 passing. NOTE: redis-server was missing from the container at start — had to apt-get install redis-server + supervisorctl start redis + restart celery before Celery-dependent tests could run.",
"backend_issues": {
"critical": [],
"minor": [
{
"endpoint": "infra",
"issue": "redis-server binary was missing in the preview container (supervisor redis task was FATAL). Installed via apt-get during test run. If image is rebuilt this will regress — redis-server should be baked into the base image or installed in a setup script."
},
{
"endpoint": "POST /api/auth/login (carried)",
"issue": "Brute-force lockout still keys on request.client.host+email (single-pod). Unchanged from prior iterations."
},
{
"endpoint": "POST /api/admin/templates (carried)",
"issue": "document_type regex runs after .upper() so lowercase payloads are silently uppercased. Cosmetic."
}
]
},
"frontend_issues": {"ui_bugs": [], "integration_issues": [], "design_issues": []},
"test_report_links": [
"/app/backend/tests/test_procureflow_phase4_hardening.py",
"/app/backend/tests/test_procureflow_phase2.py",
"/app/backend/tests/test_procureflow_phase3.py",
"/app/test_reports/pytest/iteration4_results.xml",
"/app/test_reports/pytest/phase4_results.xml"
],
"action_items": [
"Ensure redis-server is pre-installed in the container image (or in the supervisor bootstrap) so Celery-backed bulk pipeline works on fresh boots without manual apt-get."
],
"critical_code_review_comments": [
"TemplatePayload: alias='schema' works; response also serializes the field as 'schema' in the DB document (server writes `{..., 'schema': payload.template_schema}`), so the external API contract stays intact. Good.",
"_render_pdf_or_400 correctly centralises the HTTPException(400) path and returns pdf_bytes guaranteed non-None — the 'possibly unbound' pdf_bytes/result concern is resolved.",
"audit_service.write_log now takes AuditEvent dataclass — forces structured fields (actor_id/actor_email/action/target_*). Verified via logs: USER_REGISTER and PASSWORD_RESET_COMPLETED both carry actor_id + actor_email.",
"Frontend api.js: confirmed no Bearer interceptor, no localStorage.getItem('pf_access_token') anywhere in /app/frontend/src/lib/{api.js,auth.jsx}. withCredentials:true is the sole auth transport as intended.",
"Carried from iter 3: celery_app.py spawns fresh AsyncIOMotorClient + asyncio.run per task; startup still uses deprecated @app.on_event; bulk_upload overwrites `runner` per loop iteration. All low priority."
],
"updated_files": [
"/app/backend/tests/test_procureflow_phase4_hardening.py (new, 8 tests)"
],
"success_rate": {
"backend": "100% (phase2 38/38 + phase3 19/19 + phase4 8/8 = 65/65)",
"frontend": "not tested (scope: backend only)"
},
"test_credentials": "Admin: syazwan.zulkifli@quatriz.com.my / Admin@123 (read from /app/backend/.env — unchanged)",
"seed_data_creation": "Phase 4 suite auto-registers + deletes: 1 audit-register user, 1 password-reset user, 1 custom template (ALIASxxxxxx), 1 uploaded doc for email-503 test. All cleaned up in finally blocks.",
"retest_needed": false,
"main_agent_can_self_test": true,
"context_for_next_testing_agent": "Run `pytest /app/backend/tests/test_procureflow_phase2.py /app/backend/tests/test_procureflow_phase3.py /app/backend/tests/test_procureflow_phase4_hardening.py -v` for the full 65-test regression (~2.5 min). BEFORE running, verify `redis-cli ping` returns PONG — if redis-server is missing (FATAL in supervisorctl status), run `apt-get install -y redis-server && supervisorctl start redis && supervisorctl restart celery`. All suites read ADMIN_EMAIL/ADMIN_PASSWORD from /app/backend/.env via python-dotenv. Phase 4 test `test_password_reset_completed_writes_actor_via_write_log` queries the DB directly for the reset token via pymongo, so MONGO_URL/DB_NAME must be reachable.",
"rca of the issue": "No functional bugs. Only environmental: redis-server binary absent on fresh preview container caused celery consumer to back-off (ConnectionRefused 127.0.0.1:6379). Fixed in-situ by apt install; no code change required."
}

0
test_reports/pytest/.gitkeep
View File

1
test_reports/pytest/iteration4_results.xml
View File

File diff suppressed because one or more lines are too long

1
test_reports/pytest/phase2_results.xml
View File

@ -1 +0,0 @@
<?xml version="1.0" encoding="utf-8"?><testsuites name="pytest tests"><testsuite name="pytest" errors="0" failures="0" skipped="0" tests="38" time="62.066" timestamp="2026-04-24T15:42:01.981110+00:00" hostname="agent-env-d9001c04-4a91-44d5-ad57-7af79a48105a"><testcase classname="tests.test_procureflow_phase2.TestAuth" name="test_admin_login_success" time="0.803" /><testcase classname="tests.test_procureflow_phase2.TestAuth" name="test_me_with_bearer" time="0.212" /><testcase classname="tests.test_procureflow_phase2.TestAuth" name="test_me_with_cookie" time="0.834" /><testcase classname="tests.test_procureflow_phase2.TestAuth" name="test_register_short_password_422" time="0.191" /><testcase classname="tests.test_procureflow_phase2.TestAuth" name="test_register_new_user_role_user" time="0.789" /><testcase classname="tests.test_procureflow_phase2.TestAuth" name="test_login_wrong_password_401" time="0.544" /><testcase classname="tests.test_procureflow_phase2.TestAuth" name="test_brute_force_lockout" time="4.612" /><testcase classname="tests.test_procureflow_phase2.TestAuth" name="test_logout_clears_cookies" time="0.855" /><testcase classname="tests.test_procureflow_phase2.TestAuth" name="test_refresh_token" time="0.646" /><testcase classname="tests.test_procureflow_phase2.TestPublicAndProtected" name="test_documents_requires_auth" time="0.193" /><testcase classname="tests.test_procureflow_phase2.TestPublicAndProtected" name="test_dashboard_requires_auth" time="0.179" /><testcase classname="tests.test_procureflow_phase2.TestPublicAndProtected" name="test_me_requires_auth" time="0.232" /><testcase classname="tests.test_procureflow_phase2.TestPublicAndProtected" name="test_root_public" time="0.167" /><testcase classname="tests.test_procureflow_phase2.TestPublicAndProtected" name="test_templates_public" time="0.216" /><testcase classname="tests.test_procureflow_phase2.TestPublicAndProtected" name="test_template_quotation_public" time="0.234" /><testcase classname="tests.test_procureflow_phase2.TestPublicAndProtected" name="test_template_invoice_schema" time="0.197" /><testcase classname="tests.test_procureflow_phase2.TestDocumentPipeline" name="test_upload_real_pdf" time="0.706" /><testcase classname="tests.test_procureflow_phase2.TestDocumentPipeline" name="test_process_document" time="12.034" /><testcase classname="tests.test_procureflow_phase2.TestDocumentPipeline" name="test_pagination_shape" time="0.270" /><testcase classname="tests.test_procureflow_phase2.TestDocumentPipeline" name="test_regex_search" time="0.215" /><testcase classname="tests.test_procureflow_phase2.TestDocumentPipeline" name="test_filter_combo" time="0.181" /><testcase classname="tests.test_procureflow_phase2.TestDocumentPipeline" name="test_manual_create_invoice_and_pdf" time="0.610" /><testcase classname="tests.test_procureflow_phase2.TestDocumentPipeline" name="test_email_without_resend_returns_503" time="0.177" /><testcase classname="tests.test_procureflow_phase2.TestBulkUpload" name="test_bulk_upload_and_status" time="24.784" /><testcase classname="tests.test_procureflow_phase2.TestRBAC" name="test_viewer_cannot_upload" time="4.336" /><testcase classname="tests.test_procureflow_phase2.TestRBAC" name="test_viewer_can_list_documents" time="0.152" /><testcase classname="tests.test_procureflow_phase2.TestRBAC" name="test_viewer_cannot_delete" time="0.831" /><testcase classname="tests.test_procureflow_phase2.TestRBAC" name="test_user_only_sees_own_documents" time="0.627" /><testcase classname="tests.test_procureflow_phase2.TestRBAC" name="test_manager_sees_all" time="0.832" /><testcase classname="tests.test_procureflow_phase2.TestRBAC" name="test_non_manager_cannot_email" time="0.574" /><testcase classname="tests.test_procureflow_phase2.TestRBAC" name="test_user_cannot_finalize" time="0.805" /><testcase classname="tests.test_procureflow_phase2.TestRBAC" name="test_delete_removes_file" time="0.501" /><testcase classname="tests.test_procureflow_phase2.TestAdminAndAudit" name="test_list_users_admin_only" time="0.369" /><testcase classname="tests.test_procureflow_phase2.TestAdminAndAudit" name="test_admin_cannot_demote_self" time="0.292" /><testcase classname="tests.test_procureflow_phase2.TestAdminAndAudit" name="test_admin_cannot_delete_self" time="0.333" /><testcase classname="tests.test_procureflow_phase2.TestAdminAndAudit" name="test_change_role_roundtrip" time="0.963" /><testcase classname="tests.test_procureflow_phase2.TestAdminAndAudit" name="test_audit_logs_admin_only" time="0.366" /><testcase classname="tests.test_procureflow_phase2.TestAdminAndAudit" name="test_audit_filter_by_action" time="0.605" /></testsuite></testsuites>

1
test_reports/pytest/phase3_results.xml
View File

@ -1 +0,0 @@
<?xml version="1.0" encoding="utf-8"?><testsuites name="pytest tests"><testsuite name="pytest" errors="0" failures="0" skipped="0" tests="19" time="27.616" timestamp="2026-04-24T15:46:10.289709+00:00" hostname="agent-env-d9001c04-4a91-44d5-ad57-7af79a48105a"><testcase classname="tests.test_procureflow_phase3.TestPasswordReset" name="test_forgot_password_existing_email_creates_token" time="1.271" /><testcase classname="tests.test_procureflow_phase3.TestPasswordReset" name="test_forgot_password_nonexistent_email_returns_200_no_token" time="0.154" /><testcase classname="tests.test_procureflow_phase3.TestPasswordReset" name="test_reset_with_invalid_token_400" time="0.174" /><testcase classname="tests.test_procureflow_phase3.TestPasswordReset" name="test_reset_with_expired_token_400" time="0.161" /><testcase classname="tests.test_procureflow_phase3.TestPasswordReset" name="test_reset_short_password_422" time="0.139" /><testcase classname="tests.test_procureflow_phase3.TestPasswordReset" name="test_full_reset_flow_rotates_password_and_consumes_token" time="1.681" /><testcase classname="tests.test_procureflow_phase3.TestPasswordReset" name="test_audit_log_captures_reset_actions" time="0.375" /><testcase classname="tests.test_procureflow_phase3.TestTemplateEditor" name="test_non_admin_forbidden" time="0.892" /><testcase classname="tests.test_procureflow_phase3.TestTemplateEditor" name="test_create_invalid_document_type_400" time="0.603" /><testcase classname="tests.test_procureflow_phase3.TestTemplateEditor" name="test_create_invalid_schema_400" time="0.591" /><testcase classname="tests.test_procureflow_phase3.TestTemplateEditor" name="test_create_custom_template_and_list" time="0.444" /><testcase classname="tests.test_procureflow_phase3.TestTemplateEditor" name="test_put_body_type_must_match_url" time="0.156" /><testcase classname="tests.test_procureflow_phase3.TestTemplateEditor" name="test_put_updates_custom_template" time="0.275" /><testcase classname="tests.test_procureflow_phase3.TestTemplateEditor" name="test_override_quotation_then_reset" time="1.041" /><testcase classname="tests.test_procureflow_phase3.TestTemplateEditor" name="test_delete_custom_template_removes" time="0.522" /><testcase classname="tests.test_procureflow_phase3.TestTemplateEditor" name="test_template_audit_actions" time="0.504" /><testcase classname="tests.test_procureflow_phase3.TestCeleryBulk" name="test_queue_status_requires_admin" time="0.152" /><testcase classname="tests.test_procureflow_phase3.TestCeleryBulk" name="test_queue_status_admin" time="0.189" /><testcase classname="tests.test_procureflow_phase3.TestCeleryBulk" name="test_bulk_upload_uses_celery_and_extracts" time="17.766" /></testsuite></testsuites>

1
test_reports/pytest/phase4_results.xml
View File

@ -1 +0,0 @@
<?xml version="1.0" encoding="utf-8"?><testsuites name="pytest tests"><testsuite name="pytest" errors="0" failures="0" skipped="0" tests="8" time="33.908" timestamp="2026-04-24T17:22:27.694651+00:00" hostname="agent-env-d9001c04-4a91-44d5-ad57-7af79a48105a"><testcase classname="tests.test_procureflow_phase4_hardening.TestCookieOnlyAuth" name="test_login_sets_httponly_cookies" time="16.168" /><testcase classname="tests.test_procureflow_phase4_hardening.TestCookieOnlyAuth" name="test_me_works_with_cookie_only_no_bearer" time="1.491" /><testcase classname="tests.test_procureflow_phase4_hardening.TestCookieOnlyAuth" name="test_bearer_still_accepted_backwards_compat" time="2.012" /><testcase classname="tests.test_procureflow_phase4_hardening.TestCookieOnlyAuth" name="test_login_response_body_still_includes_access_token" time="1.098" /><testcase classname="tests.test_procureflow_phase4_hardening.TestAuditEventRefactor" name="test_user_register_audit_has_actor_fields" time="3.177" /><testcase classname="tests.test_procureflow_phase4_hardening.TestAuditEventRefactor" name="test_password_reset_completed_writes_actor_via_write_log" time="5.318" /><testcase classname="tests.test_procureflow_phase4_hardening.TestTemplatePayloadAlias" name="test_create_template_with_literal_schema_key" time="0.745" /><testcase classname="tests.test_procureflow_phase4_hardening.TestEmailEndpointSafety" name="test_email_unconfigured_returns_503" time="1.392" /></testsuite></testsuites>

41
test_reports/pytest/pytest_results.xml
View File

File diff suppressed because one or more lines are too long

403
test_result.md
View File

@ -1,403 +0,0 @@
#====================================================================================================
# START - Testing Protocol - DO NOT EDIT OR REMOVE THIS SECTION
#====================================================================================================
# THIS SECTION CONTAINS CRITICAL TESTING INSTRUCTIONS FOR BOTH AGENTS
# BOTH MAIN_AGENT AND TESTING_AGENT MUST PRESERVE THIS ENTIRE BLOCK
# Communication Protocol:
# If the `testing_agent` is available, main agent should delegate all testing tasks to it.
#
# You have access to a file called `test_result.md`. This file contains the complete testing state
# and history, and is the primary means of communication between main and the testing agent.
#
# Main and testing agents must follow this exact format to maintain testing data.
# The testing data must be entered in yaml format Below is the data structure:
#
## user_problem_statement: {problem_statement}
## backend:
## - task: "Task name"
## implemented: true
## working: true # or false or "NA"
## file: "file_path.py"
## stuck_count: 0
## priority: "high" # or "medium" or "low"
## needs_retesting: false
## status_history:
## -working: true # or false or "NA"
## -agent: "main" # or "testing" or "user"
## -comment: "Detailed comment about status"
##
## frontend:
## - task: "Task name"
## implemented: true
## working: true # or false or "NA"
## file: "file_path.js"
## stuck_count: 0
## priority: "high" # or "medium" or "low"
## needs_retesting: false
## status_history:
## -working: true # or false or "NA"
## -agent: "main" # or "testing" or "user"
## -comment: "Detailed comment about status"
##
## metadata:
## created_by: "main_agent"
## version: "1.0"
## test_sequence: 0
## run_ui: false
##
## test_plan:
## current_focus:
## - "Task name 1"
## - "Task name 2"
## stuck_tasks:
## - "Task name with persistent issues"
## test_all: false
## test_priority: "high_first" # or "sequential" or "stuck_first"
##
## agent_communication:
## -agent: "main" # or "testing" or "user"
## -message: "Communication message between agents"
# Protocol Guidelines for Main agent
#
# 1. Update Test Result File Before Testing:
# - Main agent must always update the `test_result.md` file before calling the testing agent
# - Add implementation details to the status_history
# - Set `needs_retesting` to true for tasks that need testing
# - Update the `test_plan` section to guide testing priorities
# - Add a message to `agent_communication` explaining what you've done
#
# 2. Incorporate User Feedback:
# - When a user provides feedback that something is or isn't working, add this information to the relevant task's status_history
# - Update the working status based on user feedback
# - If a user reports an issue with a task that was marked as working, increment the stuck_count
# - Whenever user reports issue in the app, if we have testing agent and task_result.md file so find the appropriate task for that and append in status_history of that task to contain the user concern and problem as well
#
# 3. Track Stuck Tasks:
# - Monitor which tasks have high stuck_count values or where you are fixing same issue again and again, analyze that when you read task_result.md
# - For persistent issues, use websearch tool to find solutions
# - Pay special attention to tasks in the stuck_tasks list
# - When you fix an issue with a stuck task, don't reset the stuck_count until the testing agent confirms it's working
#
# 4. Provide Context to Testing Agent:
# - When calling the testing agent, provide clear instructions about:
# - Which tasks need testing (reference the test_plan)
# - Any authentication details or configuration needed
# - Specific test scenarios to focus on
# - Any known issues or edge cases to verify
#
# 5. Call the testing agent with specific instructions referring to test_result.md
#
# IMPORTANT: Main agent must ALWAYS update test_result.md BEFORE calling the testing agent, as it relies on this file to understand what to test next.
#====================================================================================================
# END - Testing Protocol - DO NOT EDIT OR REMOVE THIS SECTION
#====================================================================================================
#====================================================================================================
# Testing Data - Main Agent and testing sub agent both should log testing data below this section
#====================================================================================================
user_problem_statement: "Phase 2: redesign the Dashboard with KPI cards + monthly volume chart + spend by type + top vendors (matching the reference image style), build a new Reports page with date/type/status filters + vendor breakdown + filtered docs list + Print button + Download PDF (with Quatriz branding), and ensure all PDFs only stamp the Quatriz logo on MANUAL-source documents (never overwrite branding on uploaded third-party PDFs)."
frontend:
- task: "Dashboard redesign — KPI cards, monthly chart, spend by type, top vendors"
implemented: true
working: "NA"
file: "frontend/src/pages/Dashboard.jsx"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: "NA"
agent: "main"
comment: "Replaced the old 4-stat-cell Dashboard with a fully responsive layout: 4 KPI cards (Total Documents, Pending Approvals, Pipeline Value, Completed This Month — 2 cols on mobile, 4 on lg+), Runner status row (when admin), 2 charts side-by-side (Monthly Volume bar chart, Spend by Type) stacking on mobile, Recent Activity table + Top Vendors leaderboard, Documents-by-Type breakdown. All cards respect the existing pf-surface design system. Reads from /api/dashboard/summary."
- task: "Reports page with filters + Print + Download PDF"
implemented: true
working: "NA"
file: "frontend/src/pages/Reports.jsx, frontend/src/App.js, frontend/src/components/Sidebar.jsx, frontend/src/index.css"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: "NA"
agent: "main"
comment: "New page at /reports (also added to sidebar nav). Filters: Type, Status, From, To dates (default = last 3 months). 4 KPI cards, Spend-by-Type horizontal bar list, Vendors-by-Spend table, Filtered Documents table. 'Print' button calls window.print() — added @media print stylesheet to index.css that strips the sidebar/buttons and adds a clean print-only header. 'Download PDF' calls /api/reports/pdf with the active filters and triggers a file download. All buttons hidden when printing via print:hidden Tailwind utility + CSS fallback."
- task: "Review page responsive gap fix (phone-in-desktop-mode)"
implemented: true
working: "NA"
file: "frontend/src/pages/Review.jsx"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: "NA"
agent: "main"
comment: "Root cause: the 2-col split (dark PDF iframe panel + form) used the lg: breakpoint (1024px), so on Chrome desktop-mode at ~980-1023px viewport the iframe column appeared but mobile Chrome can't render inline PDFs → big empty grey area on the left. Fix: bumped the breakpoint to xl: (1280px). At <xl, the layout is single-column with a compact 'Original PDF / Open' bar at the top. Added max-w-[1100px] mx-auto on the form area so it stays readable on the wider single-column. The dark inline iframe preview still works at xl+ where it actually renders correctly. NOT requesting frontend testing — visual fix, will verify on next deploy."
- task: "Provider-used badge on Review page"
implemented: true
working: "NA"
file: "frontend/src/pages/Review.jsx"
stuck_count: 0
priority: "medium"
needs_retesting: false
status_history:
- working: "NA"
agent: "main"
comment: "Added a small blue uppercase badge (data-testid='review-provider-badge') in the Review page header next to the confidence indicator that shows doc.extraction_provider when set ('gemini-direct', 'groq', or 'emergent'). Hidden on small screens (md:inline). Tooltip explains it's the LLM that produced the extraction. Not requesting frontend testing — visual addition only, will verify on the next deployment."
- task: "Review page extraction-error banner with Retry"
implemented: true
working: "NA"
file: "frontend/src/pages/Review.jsx"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: "NA"
agent: "main"
comment: "Added a red banner (data-testid='extraction-error-banner') at the top of the Review page that shows when doc.extraction_error is set OR doc.status === 'FAILED'. Banner displays the human-readable message from the backend (e.g. 'LLM service budget exhausted — top up your Emergent Universal Key and click Retry.') and a 'Retry extraction' button (data-testid='extraction-retry-btn') that POSTs to /api/documents/{id}/process, then re-hydrates the local state from the response. Includes a spinner while retrying and surfaces any retry error inline. NOT requesting frontend testing yet — will ask the user first per protocol."
- task: "Bulk upload duplicate prevention"
implemented: true
working: true
file: "frontend/src/pages/Upload.jsx"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: true
agent: "testing"
comment: "Tested on mobile viewport (412×915). Rapid clicked submit button 5 times within 100ms. Network monitoring confirmed only 1 POST request to /api/documents/bulk-upload was sent. Queued table correctly showed exactly 2 items (not duplicates). Button and file input were properly disabled during upload. The submitting state guard (if (!files.length || submitting) return;) is working correctly."
- task: "Retry button visibility for UPLOADED/FAILED status"
implemented: true
working: true
file: "frontend/src/pages/DocumentList.jsx"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: true
agent: "testing"
comment: "Verified retry button (data-testid='retry-{id}') correctly does NOT appear for documents with EXTRACTED, REVIEWED, or FINAL status. Checked 4 documents with EXTRACTED status - none had retry buttons visible. Implementation at lines 136-148 correctly conditionally renders retry button only for UPLOADED or FAILED status."
- task: "Retry button functionality"
implemented: true
working: "NA"
file: "frontend/src/pages/DocumentList.jsx"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: "NA"
agent: "testing"
comment: "Could not test retry button click functionality because all test documents processed successfully to EXTRACTED status. No documents with UPLOADED or FAILED status were available during testing. The retry button implementation appears correct (shows spinner during retry, calls /api/documents/{id}/process endpoint, reloads data after completion), but actual functionality could not be verified in this test run."
- task: "Auth flow - redirect to login from root"
implemented: true
working: true
file: "frontend/src/App.js, frontend/src/components/Protected.jsx"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: true
agent: "testing"
comment: "Tested on 2026-04-25. Opening root URL (/) correctly redirects to /login when not authenticated. Protected route wrapper working as expected."
- task: "Auth flow - login with credentials"
implemented: true
working: true
file: "frontend/src/pages/Login.jsx, frontend/src/lib/auth.jsx"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: true
agent: "testing"
comment: "Tested on 2026-04-25. Login with credentials (syazwan.zulkifli@quatriz.com.my / Admin@123) successfully lands on dashboard. User info displayed in sidebar. Login API returns 200 OK."
- task: "Auth flow - session persistence on reload"
implemented: true
working: false
file: "frontend/src/lib/auth.jsx, frontend/src/index.js"
stuck_count: 1
priority: "high"
needs_retesting: false
status_history:
- working: false
agent: "testing"
comment: "CRITICAL ISSUE: Page reload triggers 2 GET /api/auth/me requests instead of 1. Both requests return 200 OK. Root cause identified: React.StrictMode in index.js (line 8) causes double-mounting of effects in React 18+. The AuthProvider's useEffect runs twice, calling refresh() twice. This is NOT a bug in the retry logic itself (which only retries on network errors/5xx), but a side effect of StrictMode. User stays logged in correctly, but the duplicate request violates the requirement of 'exactly ONE successful GET /api/auth/me on reload'."
- task: "Auth flow - logout functionality"
implemented: true
working: true
file: "frontend/src/components/Sidebar.jsx, frontend/src/lib/auth.jsx"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: true
agent: "testing"
comment: "Tested on 2026-04-25. Clicking logout button successfully redirects to /login. Logout API call returns 200 OK. User state cleared correctly."
- task: "Auth flow - no auto-login after logout"
implemented: true
working: true
file: "frontend/src/lib/auth.jsx"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: true
agent: "testing"
comment: "Tested on 2026-04-25. After logout, reloading the page stays on /login (does not auto-login). Login form is visible and functional."
- task: "Auth flow - cookie cleanup after logout"
implemented: true
working: true
file: "backend/server.py (logout endpoint)"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: true
agent: "testing"
comment: "Tested on 2026-04-25. After logout, both access_token and refresh_token cookies are absent. Only Cloudflare cookies (cf_clearance, __cf_bm) remain. Cookie cleanup working correctly."
backend:
- task: "Reports + Dashboard aggregations and PDF export"
implemented: true
working: true
file: "backend/services/reports_service.py, backend/services/report_pdf_service.py, backend/server.py"
stuck_count: 0
priority: "high"
needs_retesting: true
status_history:
- working: true
agent: "main"
comment: "New endpoints: GET /api/dashboard/summary (KPIs: total_documents, pending_approvals, pipeline_value, completed_value, completed_this_month + monthly_volume[6mo] + spend_by_type + top_vendors[5] + recent[8]). GET /api/reports/summary?from=&to=&type=&status= (kpis, vendors, by_type, monthly buckets, documents list). GET /api/reports/pdf?... returns a print-ready Procurement Report PDF using the same Quatriz brand template. All scoped to user role (admin/manager see all, others see owner-only). Aggregations done in Python after a single Mongo cursor for portability (avoids $expr type-juggling). Smoke-tested: report PDF (64KB) renders correctly with logo, KPI strip, Vendors-by-spend table, and Filtered Documents list — Gemini-Vision confirmed all layout checks pass."
- task: "Branded PDF only for MANUAL docs (not for uploaded third-party PDFs)"
implemented: true
working: true
file: "backend/services/pdf_service.py, backend/server.py"
stuck_count: 0
priority: "high"
needs_retesting: true
status_history:
- working: true
agent: "main"
comment: "render_document_pdf now takes branded=bool. _render_pdf_or_400 in server.py inspects doc.source — MANUAL → branded=True (full Quatriz template: logo top-left, bordered title top-right, To/Attn block, Ref/SST/Date grid, items table with grid borders, totals stack, terms & conditions box, signature space, centered footer with Quatriz registration + address). AUTO/uploaded → branded=False, neutral 'Extracted Form' rendering with the source filename and a disclaimer that the original branding belongs to the issuing party. This prevents overwriting third-party logos (e.g. Umobile invoices) when the user re-renders the extracted data. Both paths smoke-tested across all 5 doc templates."
- task: "Quatriz logo + branded header on all generated PDFs"
implemented: true
working: true
file: "backend/services/pdf_service.py, backend/assets/quatriz_logo.png, backend/assets/quatriz_logo_pdf.png, backend/.env, backend/Dockerfile"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: true
agent: "main"
comment: "Stored full-res logo at backend/assets/quatriz_logo.png and a PDF-optimized 600x400 / 50KB version at backend/assets/quatriz_logo_pdf.png. Added _build_brand_header() to pdf_service: 2-col table with logo + COMPANY_NAME + optional COMPANY_TAGLINE/COMPANY_ADDRESS on the left, document title (PURCHASE ORDER / QUOTATION / etc.) on the right, and a 1pt navy underline separating the band from the body. All 5 doc types (PO, PR, DO, QUOTATION, INVOICE) render correctly with the branded header — smoke-tested locally. Logo gracefully falls back to text if the PNG is missing. Company info is configurable via .env (COMPANY_NAME, COMPANY_TAGLINE, COMPANY_ADDRESS — pipe-separated for address lines). Dockerfile already does COPY backend/ ./ so the assets folder ships automatically. Gemini analysis of the rendered PDF confirms 'professional', 'logo at top left', 'document title at top right', 'clean minimalist design'."
- task: "Multi-provider LLM fallback (Gemini → Groq → Emergent) with provider tracking"
implemented: true
working: true
file: "backend/services/extraction_service.py, backend/server.py, backend/celery_app.py"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: true
agent: "main"
comment: "Refactored extract_structured to walk a provider chain: GEMINI_API_KEY (google-genai SDK, response_mime_type=application/json), then GROQ_API_KEY (httpx POST to https://api.groq.com/openai/v1/chat/completions, llama-3.3-70b-versatile, response_format=json_object), then EMERGENT_LLM_KEY (legacy). Each provider is tried in order; failures (auth, rate-limit, parse error) are logged via _try_provider helper, and we fall through. Final ExtractionError surfaces the LAST tier's friendly classified message via the banner. Function now returns (payload, provider_name) tuple. server.py + celery_app.py updated to receive the tuple and persist extraction_provider on the document. DocumentModel has extraction_provider: Optional[str]. Smoke-tested: (a) real Gemini key returns gemini-direct as provider; (b) bad Gemini + exhausted Emergent walks the chain and returns the budget-exhausted message correctly. Awaiting Groq key from user to test the middle tier explicitly."
- task: "Switch LLM extraction to free Google Gemini direct (google-genai SDK)"
implemented: true
working: true
file: "backend/services/extraction_service.py, backend/.env, backend/requirements.txt, backend/Dockerfile"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: true
agent: "main"
comment: "Refactored extract_structured to call Google Gemini directly via the google-genai SDK using GEMINI_API_KEY (free tier, ~1500 requests/day). Uses response_mime_type='application/json' to force valid JSON output. Falls back to Emergent Universal Key only when GEMINI_API_KEY is not set. SDK call wrapped in asyncio.to_thread so it doesn't block the event loop. Added google-genai to Dockerfile (--no-deps to avoid pydantic 2.9 conflict, then explicitly install runtime deps). Smoke-tested locally end-to-end: real key returns proper structured JSON for a sample PO; invalid key correctly raises ExtractionError with friendly 'LLM service rejected the API key' message that surfaces in the Review banner."
- task: "ExtractionError surfacing on LLM failure"
implemented: true
working: "NA"
file: "backend/services/extraction_service.py, backend/server.py, backend/celery_app.py"
stuck_count: 0
priority: "high"
needs_retesting: true
status_history:
- working: "NA"
agent: "main"
comment: "Refactored extract_structured to raise ExtractionError (with friendly message - detects budget exhausted / rate-limit / timeout / auth) instead of silently returning empty payload. _run_pipeline (server.py) and Celery task (celery_app.py) now catch ExtractionError, persist raw_text + classification + ocr fields, set status=FAILED and store extraction_error string on the document. Successful runs $unset extraction_error. Added extraction_error: Optional[str] field to DocumentModel. Need backend testing: (1) when LLM call raises an exception, document ends up status=FAILED with extraction_error populated; (2) when EMERGENT_LLM_KEY missing, ExtractionError is raised with 'EMERGENT_LLM_KEY is not configured' message; (3) successful retry via POST /api/documents/{id}/process clears the extraction_error field; (4) classification + raw_text are still persisted even on extraction failure."
- task: "Bulk upload endpoint duplicate prevention"
implemented: true
working: true
file: "backend/server.py"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: true
agent: "testing"
comment: "Backend correctly handled only 1 bulk-upload request despite 5 rapid frontend clicks. Backend logs show: 'POST /api/documents/bulk-upload HTTP/1.1 200 OK' appeared only once. Both documents were successfully uploaded and processed to EXTRACTED status."
- task: "Document processing endpoint for retry"
implemented: true
working: "NA"
file: "backend/server.py"
stuck_count: 0
priority: "high"
needs_retesting: false
status_history:
- working: "NA"
agent: "testing"
comment: "Could not test the /api/documents/{id}/process endpoint for retry functionality as no documents remained in UPLOADED or FAILED status during testing."
metadata:
created_by: "testing_agent"
version: "1.0"
test_sequence: 2
run_ui: true
test_date: "2026-04-25"
viewport: "desktop (1920×1080)"
last_test: "Auth flow sanity check"
test_plan:
current_focus:
- "Reports + Dashboard aggregations and PDF export"
- "Branded PDF only for MANUAL docs (not for uploaded third-party PDFs)"
stuck_tasks:
- "Auth flow - session persistence on reload"
test_all: false
test_priority: "high_first"
agent_communication:
- agent: "main"
message: "Phase 2 shipped: (1) PDF service split into branded (MANUAL) and neutral (AUTO/uploaded) paths so third-party docs like Umobile invoices NEVER get the Quatriz logo stamped on them. (2) Branded layout closely matches the user's reference: big logo top-left, bordered QUOTATION/PO box top-right, 2-col To/Ref grid, items table with full grid borders, Sub-total/SST/Grand Total stack with thicker bottom rule, Terms & Conditions box, signature line, centered footer with 'Quatriz System Sdn Bhd (988952-X)' + address. Gemini-Vision validated layout. (3) Dashboard fully redesigned with 4 KPI cards + Monthly chart + Spend by Type + Top Vendors + responsive at every breakpoint. (4) New Reports page at /reports with date/type/status filters, KPI strip, vendor breakdown, filtered docs list, Print button (browser print + clean print stylesheet), and Download PDF button (uses the same branded template). New endpoints: /api/dashboard/summary, /api/reports/summary, /api/reports/pdf — all role-scoped. NOT requesting frontend testing yet — will let user verify on their device first."
agent_communication:
- agent: "testing"
message: "Completed testing of bulk upload duplicate prevention and retry button features. Test 1 (duplicate prevention) PASSED completely. Test 2 (retry button) PARTIALLY tested - visibility logic verified but actual retry functionality could not be tested due to all documents processing successfully. Test 3 (cleanup) completed successfully. No console errors detected. Screenshots saved in .screenshots/ directory."
- agent: "testing"
message: "IMPORTANT NOTE: The retry button functionality (clicking retry and re-processing) could not be fully tested because all uploaded documents immediately processed to EXTRACTED status. To fully test this feature, would need either: (1) a way to upload documents without auto-processing, or (2) artificially create a FAILED document. The implementation code looks correct based on code review."
- agent: "testing"
message: "AUTH FLOW SANITY CHECK COMPLETED (2026-04-25): Tested auth flow after recent auth context retry logic changes. CRITICAL ISSUE FOUND: Page reload triggers 2 GET /api/auth/me requests instead of 1. Root cause: React.StrictMode in index.js causes double-mounting of effects in React 18+. Both requests succeed (200 OK), so this is not a retry logic bug but a StrictMode side effect. All other auth tests passed: redirect to login, login flow, session persistence, logout, cookie cleanup. Recommend removing StrictMode for production/preview builds or adding ref-based guard to prevent double execution."

0
tests/__init__.py
View File

BIN
verify_pdf.pdf
View File

Binary file not shown.