Reports & Analytics - Backend Documentation

1. Documentation Evidence

2. Backend Scope and Boundaries

Owns

• The MongoDB analytics store: 13 fact/state streams projected from PostgreSQL, the daily rollups, sync-state documents, pending work, and export records.
• The ETL control plane: watermarks, locks, run statistics, the pending-work queue.
• The read surface: dashboard, KPIs, trends, funnel, splits, rankings, the report registry.
• The export pipeline: request, build (CSV/XLSX), sweep, download.
• The operator sync surface: state inspection and refresh/backfill/reconcile triggers.

Does Not Own

• PostgreSQL — it reads the transactional store and writes nothing to it.
• The source domains' semantics: orders, payments, refunds, reviews, POS all remain owned by their modules; analytics projects their rows.
• Cache invalidation: no Redis domain is touched.
• Storage of user uploads — export files deliberately bypass StorageManager (see §9).

Source of Truth

3. Module Composition

4. File and Directory Map

apps/api/src/modules/analytics/
  analytics.module.ts
  analytics-shared.module.ts
  analytics-worker.module.ts
  admin/
    analytics-admin-aggregate.module.ts
    dashboard/
      analytics-dashboard-admin.controller.ts
      analytics-dashboard-admin.module.ts
      analytics-dashboard.service.ts
    reports/
      analytics-reports-admin.controller.ts
      analytics-reports-admin.module.ts
    exports/
      analytics-exports-admin.controller.ts
      analytics-exports-admin.module.ts
    sync/
      analytics-sync-admin.controller.ts
      analytics-sync-admin.module.ts
    dto/
      analytics-query.dto.ts
      analytics-sync.dto.ts
      report-export.dto.ts
      report-run.dto.ts
  shared/
    analytics.constants.ts
    analytics-freshness.service.ts
    analytics-lifetime.service.ts
    analytics-period.service.ts
    analytics-permission.service.ts
    analytics-query.service.ts
    analytics-rollup-aggregate.service.ts
    analytics-rollup.service.ts
    reports/
      analytics-export-builder.service.ts
      analytics-export.service.ts
      analytics-report.service.ts
      report-registry.ts
      report-registry.types.ts
    sync/
      analytics-catalog.projector.ts
      analytics-commerce.projector.ts
      analytics-order.projector.ts
      analytics-operator.service.ts
      analytics-pending-drain.service.ts
      analytics-refund.projector.ts
      analytics-return.projector.ts
      analytics-sync.service.ts
      analytics-sync-state.service.ts
      analytics-writer.service.ts
  workers/
    analytics-maintenance.scheduler.ts
    analytics.processor.ts

5. Data Model

5.1 Schema Source

packages/mongodb/src/schemas/analytics/
  analytics.shared.ts        # ANALYTICS_CHANNELS, ANALYTICS_PAYMENT_METHODS
  analytics-sync-state.schema.ts
  analytics-pending-work.schema.ts
  analytics-order-fact.schema.ts
  analytics-order-line-fact.schema.ts
  analytics-cart-fact.schema.ts
  analytics-checkout-fact.schema.ts
  analytics-payment-fact.schema.ts
  analytics-pos-sale-fact.schema.ts
  analytics-return-fact.schema.ts
  analytics-refund-fact.schema.ts
  analytics-review-fact.schema.ts
  analytics-product-fact.schema.ts
  analytics-customer-fact.schema.ts
  analytics-promotion-fact.schema.ts
  analytics-promotion-usage-fact.schema.ts
  analytics-inventory-movement-fact.schema.ts
  analytics-daily-rollup.schema.ts
  analytics-report-export.schema.ts

5.2 Collections

analytics_sync_state

Retention: none, never TTL'd — deleting a document forces a full backfill.

analytics_pending_work

Retention: none — an entry lives until its work commits; resolvePending runs AFTER the work, never before.

analytics_order_fact

The central money document. Fields include orderNumber, invoiceNumber, channel (enum online/pos), status, customerId, customerName, paymentMethod, dateKey, yearMonth/yearQuarter/year, isCancelled, netRevenue, approvedRefundedAmount, settledRefundedAmount, grossRevenue-family. Retention: none — the historical business record every period comparison reads.

analytics_order_line_fact

Per-line projections: productPublicId, productName, sku, brandName, categoryName (snapshot via $setOnInsert), netQuantity, netLineRevenue, orderStatus, orderPublicId. Retention: none.

analytics_pos_sale_fact

Till dimensions only — saleNumber, status, fulfilment, customerId, customerName, customerCreated, createdByAdminId, createdByAdminName, dateKey. Never a revenue source — POS revenue lives on the order facts via channel: "pos".

analytics_refund_fact

orderPublicId, orderNumber, status, dateKey, line items with amount/quantity. Retention: none — money movement is a permanent finance record.

analytics_daily_rollup

One document per (dateKey, channel) (_id = buildRollupId(dateKey, channel)), rebuilt via replaceOne(upsert): orders { created, completed, cancelled, returned, refunded }, revenue { gross, net, subtotal, discount, shipping, shippingDiscount, paid, settledRefunded, approvedRefunded, cancelled, completed }, units, customers, payments { byMethod }, funnel { cartsCreated, checkoutsStarted, reachedPayment, paymentsSucceeded, ordersCompleted, checkoutsAbandoned }, returns, refunds, promotions, reviews, averageOrderValue, computedAt, calendar keys.

analytics_report_export

_id (uuid), reportType, format (csv/xlsx), filters (Mixed), periodPreset, periodFrom/periodTo, sortBy/sortDirection, timeZone, status (queued → processing → ready/failed/expired), requestedByAdminId/requestedByAdminName, requestedAt, startedAt, completedAt, rowCount, truncated, fileSizeBytes, fileKey, error, expiresAt, idempotencyKey. Unique index on (requestedByAdminId, idempotencyKey). TTL on expiresAt.

5.3 Relationship Diagram

6. Services and Responsibilities

6.1 AnalyticsSyncStateService

The ETL control plane. Every method is safe to call concurrently from more than one process — single-document atomicity is the only atomicity available.

6.2 AnalyticsSyncService

The watermarked scan loop. Reads rows since watermark - overlap, projects, and loops until a page comes back short — one page per run would skip everything past the cap permanently. A null watermark routes to backfill rather than scanning all of history.

6.3 Projectors (order, refund, return, catalog, commerce)

Each stream has its own projector. Order projector drives both orders and order_lines (parent-driven, no independent watermark). Refund projector, on settlement, queues {kind: "order"} and {kind: "rollup_day"} pending work — the mechanism that carries a settlement into net revenue without touching the order row.

6.4 AnalyticsRollupService

Rebuilds (day, channel) rollup documents with replaceOne(upsert) — recomputed, never incremented. Trailing 7 days plus queued rollup_day entries (deduplicated into one Map). resolvePending runs only after every rebuild committed. No Math.max(0) clamp on net — a negative result means the two populations drifted and must surface.

6.5 AnalyticsQueryService

The read side. totalsForPeriod sums pre-aggregated days (exact, because each day is a complete recompute); averageOrderValue is recomputed from the summed pair, never averaged from daily averages. Rankings (top products, districts, ratings, stock) read facts directly, always bounded by a period and a limit. byDistrict groups by id, never by name — provinces have genuinely been renamed.

6.6 AnalyticsReportService

One code path for all 19 reports: resolve descriptor → assert permission (before the pipeline runs — a check after the aggregation is a DoS surface) → validate filters against the report's own allowlist → build pipeline → count + page in parallel → wrap in the standard envelope. Filter coercion is the injection boundary: field names come from the descriptor, never the request; search terms are escaped and prefix-anchored (^escaped on a lowercased field).

6.7 AnalyticsExportService

request writes the record FIRST (Mongo), then enqueues — the reverse order would let a job run against a record that does not exist. isDuplicateKey on the idempotency index returns the existing export. findForAdmin scopes to the requester; another admin's export is the same 404 as a missing one. reEnqueueStuck (hourly) is the compensation for having no outbox. assertStillPermitted re-checks at download time.

6.8 AnalyticsOperatorService

describeStreams maps 13 stream states + pending backlog into a healthy rollup. forceSync / backfill / reconcile enqueue and return queued immediately. 409 ANALYTICS_SYNC_ALREADY_RUNNING is an honest answer, not a race guard (the lock is the guard). assertKnownStream rejects typos with the valid list.

6.9 AnalyticsFreshnessService

forStreams takes the OLDEST contributing stream's lastSuccessAt; a stream with no control document means never synced. assertSynced throws 503 before any aggregation over an empty collection could return a plausible zero.

6.10 AnalyticsPeriodService

Resolves presets and custom windows, applies the 3-year cap, builds the calendar-aligned comparison. referenceInstant is injectable so an export rebuilt from a stored request resolves the same window it did when queued — a last_7_days export must not silently cover a different week when the worker runs it.

7. Runtime Flows

7.1 The incremental sync

7.2 The refund-settlement path (why pending work exists)

7.3 The export pipeline

8. Caching

None. This module writes no Redis cache domain and no PostgreSQL table. Its read model is MongoDB itself; the "cache" is the daily-rollup layer, and its invalidation is the 5-minute recompute.

9. BullMQ, Schedulers, and Async Work

AnalyticsProcessor is the one @Processor on the queue (the codebase hard rule), with an exhaustive switch over AnalyticsJob — a second decorated class would race and silently drop jobs. Deterministic HttpException failures are returned (not retried); transient failures rethrow.

Outbox exemption, declared: exports have no outbox because the module writes no PostgreSQL table — the export record is Mongo, written first, and reEnqueueStuck (in the hourly sweep) compensates for a lost enqueue. The crons also enqueue directly (the documented cron-with-no-write exemption).

Export files are written to storage/analytics-exports/, not through StorageManager.handleUpload — that path sniffs magic bytes and has no CSV entry; it protects user uploads and must not be loosened for a server-generated artefact. The trade (files served by the API rather than object storage) is documented in analytics.constants.ts.

10. Realtime and Events

No realtime surface exists. The module's "events" are the Mongo collections and queue jobs described above.

11. Security, Auth, and Abuse Controls

• Guards: JwtAuthGuard + RoleGuard on all four controllers; IpThrottlerGuard per route.
• Permissions: Analytics_READ gates every read; Analytics_UPDATE gates the three sync triggers — a backfill walks every row a stream has ever produced and must not be reachable by a read-only account. Ten modules: Analytics + nine Reports*; _CREATE, _DELETE, _RESTORE unused (catalogue cross product).
• Report permissions: each report descriptor names its own Reports*_READ; GET /reports filters to the caller's grants; running a report checks BEFORE the aggregation runs.
• No id enumeration: an unpermitted report is the SAME 404 as a nonexistent one; another admin's export is the SAME 404 as a missing one.
• Download re-check: the export download route resolves the caller's LIVE role and re-checks the report's permission — queueing before a role change must not preserve access.
• Injection boundary: report filters are validated against the descriptor allowlist; field names come from the descriptor, never the request; search terms escaped and prefix-anchored.
• Rate limits (per admin account, not IP): ADMIN_ANALYTICS_READ 60/min; export submission ADMIN_ANALYTICS_EXPORT 5/hour; sync triggers ADMIN_HEAVY_OP 20/min.
• DoS bounds: 3-year period cap; report limit 500; export cap 50k rows; search prefix-anchored so it stays index-served.
• PII: customer facts carry names/emails for reporting; lookup-style surfaces are admin-gated and the export records are scoped to their requester.

13. Error Handling

14. Observability

15. Testing and Validation

Validation: pnpm check-types · pnpm check (Biome) · pnpm build · pnpm test (serial) · pnpm rules — all passed in the feature's verification run.

16. Mandatory Backend Deep-Dive Pack

16.1 Submodule Coverage Matrix

16.2 UML and Architecture Diagram Pack

• Component diagram: §3 composition graph.
• Deployment/runtime diagram: §9 queue topology.
• Sequence diagrams: §7.1 sync, §7.2 refund settlement, §7.3 export.
• ER diagram: §5.3.
• State diagrams: feature §6.2 (export), feature §7.2 (stream).

16.3 Code Flow Narrative

The three critical paths — incremental sync (§7.1), refund settlement (§7.2) and export (§7.3) — have full narratives. The shared shape of every other flow: resolve → validate → read → wrap with meta (freshness + period + duration). The watermark advance (completeRun) is the module's most delicate logic and its 4-mechanism design (overlap, conditional advance, reconcile, backfill) is documented at ANALYTICS_WATERMARK_OVERLAP_SECONDS and in the sync-state service comments.

16.4 Data Layer Deep Dive

Field-level detail for the control-plane and money collections is in §5.2. Collection rationale:

16.5 Business Logic and Invariant Catalog

16.6 Tradeoffs, Alternatives, and ADR Notes

16.7 Operational Runbook

16.8 Backend Risk Register

17. Zero-Omission Backend Checklist

• Every file in the module directory is represented or explicitly marked non-runtime (§4, §16.1).
• Every controller, service, provider, processor, scheduler, helper, mapper, DTO, enum, and schema is documented.
• Every method with business behavior has a code-flow narrative (§6, §7, §16.3).
• Every collection and job payload has field-level detail (§5.2, §9).
• Every index, constraint, relation, and retention behavior has rationale (§5.2, §16.4).
• Every lifecycle/status transition has a state diagram and transition table (feature §7).
• Every read/write/action/job flow has sequence and activity diagrams (§7, §9).
• Every business invariant is cataloged (§16.5).
• Every cache key, invalidation path, queue job, realtime event, and external call is documented (§8, §9, §10) — no cache or realtime exists by design.
• Every architectural tradeoff is documented with alternatives and revisit triggers (§16.6).
• Every operational failure mode has a runbook entry (§16.7).

18. Backend Completion Checklist

• Module boundaries are documented (§2).
• Every controller, service, DTO, schema file, job, cache key, and event is covered (§3–§11).
• Every database collection has a field table and relationship diagram (§5).
• Every runtime flow has a diagram and branch notes (§7).
• API and features/flows docs are linked (See Also).
• No claim is made without a source file or documented source reference.

See Also

• API doc: /docs/developer/analytics/api
• Features and flows doc: /docs/developer/analytics/feature
• TDD: not yet published