Multi-Site High-Availability Retail Platform

The challenge

Retail operations depended on application and data services that needed to remain available through host maintenance, component failure, and site-level disruption. A single oversized server would have concentrated risk, while a complex hyperscale design would have been expensive and difficult to service.

The goal was practical high availability — distributing failure domains without burdening a small engineering team with hyperscale complexity.

Requirements

• Distribute compute, database, proxy, and application responsibilities across serviceable nodes
• Support controlled failover and maintenance without taking down the entire environment
• Connect multiple sites securely while defining what continues locally during WAN impairment
• Provide consistent deployment for custom retail services and their supporting databases
• Make backup, monitoring, recovery, and replacement procedures operationally realistic

Architecture

Clustered Proxmox virtualization hosts provide a common compute and lifecycle layer. Segmented management, application, database, and service networks limit unnecessary trust and traffic. A synchronous multi-node relational database cluster protects transactional application data. Redundant reverse proxies and virtual service addresses provide stable application entry points.

┌──────── Site A ─────────┐          ┌──────── Site B ─────────┐
│ Proxmox cluster         │◀────────▶│ Proxmox cluster         │
│ ┌─────┐ ┌─────┐         │   WAN    │ ┌─────┐ ┌─────┐         │
│ │ App │ │ App │         │          │ │ App │ │ App │         │
│ └──┬──┘ └──┬──┘         │          │ └──┬──┘ └──┬──┘         │
│    └───┬───┘            │          │    └───┬───┘            │
│   ┌────▼────┐           │          │   ┌────▼────┐           │
│   │ HA DB   │           │          │   │ HA DB   │           │
│   └────┬────┘           │          │   └────┬────┘           │
│   ┌────▼────┐           │          │   ┌────▼────┐           │
│   │ Proxy / │           │          │   │ Proxy / │           │
│   │ VIP     │           │          │   │ VIP     │           │
│   └─────────┘           │          │   └─────────┘           │
└─────────────────────────┘          └─────────────────────────┘

Major capabilities

• Virtual-machine and container lifecycle management
• Highly available transactional database services
• Redundant web and API entry points
• Custom retail application hosting
• Network segmentation and controlled inter-site connectivity
• Automated database and configuration backups
• Cluster, node, and service health monitoring
• Planned maintenance and failure recovery procedures

Security and privacy

Management interfaces are separated from user and application networks. Database access is restricted to authorized services, and public traffic terminates through controlled proxy layers. Administrative identities, backups, secrets, and inter-site links receive separate controls rather than inheriting broad retail-network access.

Implementation

The environment was assembled as cooperating layers, with health and failure testing at each boundary. Legacy services were migrated in controlled phases, database state was validated before cutover, and stable virtual endpoints reduced the number of application changes required. Documentation focused on both normal maintenance and degraded operation.

Difficult problems solved

• Avoiding false high availability where one shared dependency can still stop every site
• Maintaining database consistency while keeping application failover practical
• Separating infrastructure management from retail and application traffic without creating an unserviceable network
• Designing recovery priorities around business operations rather than infrastructure elegance

Operational results