Working Notes •

Payment Processing Patterns in Lending Systems

Understanding idempotency, retry logic, and reconciliation patterns when handling payments at scale in lending platforms.

Writing payments lending reconciliation

The Reality of Payment Processing

Payment processing in lending systems is messier than it looks. You’re coordinating between your loan management system, payment processors, banks, and third-party services. Each has its own failure modes, retry behaviors, and reconciliation requirements.

The textbook says “just make it idempotent” but that glosses over the real complexity. What happens when a payment succeeds but the callback fails? What if the bank confirms but your database update times out? These aren’t edge cases; they’re Tuesday morning.

Idempotency Tokens Done Right

Every payment request needs an idempotency key. That’s obvious. The tricky part is deciding what makes a request “the same” request.

For loan payments, we use a composite key: {loan_id}_{payment_date}_{amount}_{source}. This handles the case where a borrower makes multiple payments on the same day (different amounts) while preventing duplicate charges from retry logic.

Store these tokens with a TTL. We keep them for 24 hours which covers our retry window without bloating the database. After 24 hours, if someone retries with the same key, treat it as a new payment.

The Three-Phase Commit Pattern

Here’s what works in production:

  1. Reserve: Create a payment intent with status PENDING. This locks the borrower’s payment method and reserves funds but doesn’t move money.

  2. Execute: Process the actual payment. If this fails, the intent stays PENDING and you can retry safely.

  3. Confirm: Update loan balance, mark intent as COMPLETE, trigger notifications.

If anything fails between Execute and Confirm, you’re in a gray area. The money moved but your system doesn’t know it yet. This is where reconciliation saves you.

Reconciliation as a First-Class Feature

Don’t treat reconciliation as a batch job you run monthly. Build it into your payment flow from day one.

We poll payment provider APIs every 5 minutes for transaction status updates. If we find a payment that succeeded but isn’t marked complete in our system, we trigger the confirmation flow. This catches 90% of gray-area cases within minutes.

The other 10% gets flagged for manual review. That’s fine. Better to have a human sort out the ambiguous cases than to guess wrong and double-charge someone.

Retry Logic That Actually Works

Exponential backoff is fine for transient failures. But payment failures aren’t always transient. Insufficient funds isn’t going to fix itself in 30 seconds.

We categorize failures:

  • Retriable: Network timeouts, 503s, processor busy signals. Retry with backoff.
  • Temporary: Insufficient funds, account locked. Mark as FAILED but allow re-attempt after 24 hours.
  • Permanent: Invalid account, fraud block. Mark as FAILED and escalate.

This keeps your retry queues clean and prevents burning through API limits on payments that won’t succeed.

Testing the Unhappy Paths

You can’t test this stuff in staging. Payment processors don’t fail the same way in test environments. Instead:

  • Use feature flags to simulate failures in production (with limits).
  • Monitor your reconciliation pipeline obsessively. Unexplained discrepancies are your early warning system.
  • Keep detailed logs. When a payment goes sideways, you need the audit trail.

The goal isn’t zero payment failures. The goal is knowing exactly what state every payment is in, even when things break.

← Back to all topics