Production Architecture

Why bother?

If you're providing software as a service(who isn't) you need to invest in a proper production architecture.

Do a rough back of the napkin calculation:

  • Every prod release involves preparing the code for production, testing, etc
  • For a small team (4 devs, 1 support analyst/tester & 1 PM)
  • Say it takes 2-3 days to fix an issue (including troubleshooting and patch)
  • If it costs $1k/day per resource (replace w/ your real #s)
  • That's $6k/day x 3 days = $18k

    If you're agile and release every 2 week sprint - its 2 releases per month.

    Instead if you spend those 3 days at the begining of every project setting up your CI/CD, it'll pay for itself in no time.

What we want to achieve

You want a system you call production to be securely available anytime from anywhere.

You can count on it to not lose your data.

Bonus for auditing and bonus for reusable process and framework.

It means:

  • Security
  • High availability
  • Disaster Recovery
  • Auditing




  • Keep your entire network behind a firewall (Route all requests thru it)
  • Restrictions
    • Webapp only accepts requests from the firewall
    • DB in restricted network (only accessible by protected network or jumpbox)
    • Can only connect to jumpbox from a known IP address
    • Put your audit logs in a different location from your app
    • For DR, spin up similar system in another geo location
    • Use DB Peering between the two locations
    • Use load balancers to handle cutover

Eating your own dogfood

You can repeat this in Google Cloud or AWS or Azure using

  • Blobs for storage container (audit logs)
  • VMs and PAAS
  • VNETs or VPCs

I run a dedicated bsd server.

The goal is to rely on it as much as I rely on Google Drive or Dropbox.

This means:

  • 100 % uptime
  • Multiple backups to protect against data loss
  • Prevent unauthorized access
  • Audit logs so I know who accessed what and when
  • Available anytime, anywhere there's an internet connection.

This is my setup:


Things I am working on:

  • Secure remote logging (syslogd over stunnel)
  • DB Backups
  • Encryption at Rest
  • Setup buildbot to restart on system reboot