Maintain logging and audit trails

Maintain Logging and Audit Trails

Capture the right events in a structured, centralized, and tamper-evident way. Minimize personal data in logs, restrict access, and keep enough history to investigate incidents and prove accountability.

Scope and Strategy

• Define a logging policy: what to log, where to store it, who can access it, and how long to retain it.

• Use structured JSON logs and a centralized sink or SIEM. Prefer append-only or object-lock storage.

• Use UTC timestamps, synchronized time (NTP), and consistent schemas across services.

What to Log

• Authentication: logins, MFA prompts, successes, failures, password resets.

• Authorization: allow and deny decisions with role or permission evaluated.

• Personal data access: reads, writes, exports, deletes, and bulk queries.

• Admin and config: role changes, policy edits, feature flags, key and secret use.

• Data lifecycle: retention changes, erasure requests, data subject request workflows.

• Integrations: outbound transfers to vendors, webhook deliveries, and retries.

• System health: deployment events, errors, rate limit triggers, anomaly detections.

Log Format and Fields

Use a consistent schema so queries and alerts work across services.

{
  "ts": "2025-09-03T18:24:10Z",
  "event_type": "authz.decision",
  "request_id": "2b5f3c0f-9e6c-47fd-8c1e-5ef1f5f5c2a1",
  "trace_id": "00-4bf92f3577b34da6a3ce929d0e0e4733-00f067aa0ba902b7-01",
  "actor": { "id": "user_123", "type": "user", "ip": "203.0.113.10" },
  "subject": { "id": "cust_987", "type": "data_subject_hash" },
  "resource": "report:456",
  "action": "read",
  "decision": "deny",
  "reason": "missing_permission:report.read",
  "role": "analyst",
  "mfa_present": true,
  "pii_logged": false,
  "user_agent": "Mozilla/5.0"
}

Correlation and Traceability

• Generate and propagate a request_id and trace_id through gateways and services.

• Include them in every log line to reconstruct cross-service flows.

# NGINX: add a request ID header and log it
map $request_id $reqid { default $request_id; }
proxy_set_header X-Request-Id $reqid;
log_format json escape=json '{ "ts":"$time_iso8601","request_id":"$reqid","status":$status,"path":"$request_uri","ip":"$remote_addr" }';
access_log /var/log/nginx/access.json json;

Privacy-Safe Logging

• Do not log plaintext personal data, secrets, access tokens, or full card numbers.

• Mask or tokenize values. Hash persistent identifiers with a secret salt for linkability without reversibility.

• Enable log scrubbing in apps and ingestion pipelines. Block stack traces from dumping request bodies.

// Example: hash a subject ID with secret salt for logs
import crypto from "crypto";
function subjectHash(userId, salt) {
  return crypto.createHmac("sha256", salt).update(userId).digest("hex");
}

Storage, Integrity, and Access Control

• Send logs to a dedicated project or account. Restrict with least privilege and break-glass access.

• Enable immutability: object lock or WORM where supported. Keep a secondary copy in a separate region.

• Make logs tamper-evident: chain records or sign batches. Record integrity metadata.

-- Append-only audit table with simple hash chaining
CREATE TABLE audit_log (
  id bigserial PRIMARY KEY,
  ts timestamptz NOT NULL DEFAULT now(),
  event_type text NOT NULL,
  actor_id text,
  resource text,
  action text,
  details jsonb,
  prev_hash bytea,
  hash bytea
);

-- Application computes: hash = sha256(ts||event_type||...||prev_hash)
-- Do not allow UPDATE or DELETE; enforce with a rule
CREATE RULE audit_no_update AS ON UPDATE TO audit_log DO INSTEAD NOTHING;
CREATE RULE audit_no_delete AS ON DELETE TO audit_log DO INSTEAD NOTHING;

Detection and Alerting

• Build rules for unusual patterns: spikes in 403 denials, MFA disabled, export volume jumps, decrypt spikes, logins from new countries, admin role grants.

• Send high-severity alerts to on-call with runbooks and auto-ticket creation.

Retention and Deletion

• Keep security and access logs long enough to investigate incidents, commonly 12 to 24 months, with legal hold support.

• Separate short-lived verbose app logs from long-term audit records to control cost and exposure.

• Document retention schedules and purge workflows. Verify deletion in both hot and archive stores.

Example: Postgres Row-Change Auditing

Capture who changed what without storing plaintext values.

CREATE TABLE customer_audit (
  at timestamptz DEFAULT now(),
  actor text,
  op text,         -- insert, update, delete
  table_name text,
  row_id uuid,
  cols_changed text[],
  before_hash text,
  after_hash text
);
-- App computes before_hash/after_hash over selected fields with a secret salt

Quick Logging and Audit Checklist

• Use structured JSON logs with UTC timestamps and propagated request IDs

• Centralize logs in a restricted, immutable store with multi-region copies

• Log auth, data access, admin actions, exports, and lifecycle events

• Avoid plaintext PII and secrets; hash or tokenize identifiers

• Build alerts for high-risk patterns and maintain incident runbooks

• Enforce retention schedules and verify purge and legal hold behavior

• Make audit stores append-only and tamper-evident

Conclusion

Reliable logs and auditable trails turn surprises into timelines. With structured, privacy-safe logging, immutable storage, and clear retention policies, you can detect abuse quickly, investigate confidently, and meet GDPR and CPRA expectations for accountability and security.