Python in Production on BSD

1. Prepare the system
2. Checkpoint
3. Clone the project to your server
4. Setup a virtual environment (inside your jail)
5. Test Django app locally
6. Setup Nginx
7. Setup uWSGI
8. Test website from public url
9. Setup Supervisord to manage uWSGI processes
10. Database Setup
11. Reference Links

1 Prepare the system

The notes are for django framework but can be adapted for any wsgi based app.

1.1 Setup a BSD Jail

See how to setup a bsd jail notes here.

1.2 Install necessary packages

pkg update
pkg upgrade
pkg install python36 py36-pipenv sqlite3 py36-sqlite3 nano git nginx

# used to manage uwsgi processes later
pkg install py36-supervisor

2 Checkpoint

Jail ( with root access)
IP Address for jail
installed packages

3 Clone the project to your server

Add ssh public key to your git repo with read only access
Instead of running django project as root, create a user and clone

the project into the user's home. e.g. /home/djangouser/cloned-django-proj

4 Setup a virtual environment (inside your jail)

cd /home/djangouser/cloned-django-proj
pipenv shell   # activate virtual environment
pipenv install # install packages in Pipfile.lock

5 Test Django app locally

Be sure the virtualenv is activated. If packages are missing you can catch the errors here.

python manage.py runserver  # test you can run locally

6 Setup Nginx

6.1 Verify nginx is running.

service nginx status
Cannot 'status' nginx. Set nginx_enable to YES in /etc/rc.conf or use 'onestatus' instead of 'status'.

6.2 If you get a message "Cannot 'status'…"

Add nginx enable to /etc/defaults/rc.conf

nginx_enable="YES"  # make nginx part of system startup

6.3 Verify nginx status again

service nginx status
nginx is not running.

6.4 Start nginx

Test nginx is up and serving a page by doing a curl or wget of localhost.

6.5 Configure Nginx

6.6 Setup main nginx.conf

Listen for nginx config files (virtualhosts) in the sites folder.

cat /usr/local/etc/nginx/nginx.conf
worker_processes  1;
error_log  /var/log/nginx/error.log;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  65;

    include       /usr/local/etc/nginx/sites/*;
}

6.7 Create an nginx config for your Django app

 server {
  listen 80;
  server_name domainname.com;         # Replace with your domain name

  location /static/ {
      root /home/jailuser/djangoproj; # Serve static files
  }

  location / {
      include         uwsgi_params;
      uwsgi_pass      unix:/home/jailuser/djangoproj/djangoproj.sock;
  }
}

7 Setup uWSGI

Figure 1: WSGI flow

The interface between a webserver and a python app is defined by WSGI (Web Server Gateway Interface)

WSGI is an interface. Implementations vary (i.e. uWSGI, Gunicorn…)

I use uWSGI. It's as good as any other and over time I've gotten better at understanding and tweaking the settings.

7.1 Sample uWSGI file

I keep it in a config folder in my django project.

  [uwsgi]
  # Abort on unknown configuration options.
  strict              = true
  uid                 = jailuser
  gid                 = jailuser
  umask               = 022


  # Report memory usage to check for leaks; optional.
  memory-report       = true


  #base                = /home/%(uid)
  chdir               = /home/jailuser/djangoproj
  virtualenv          = /home/%(uid)/.local/share/virtualenvs/djangoproj-qXZ

  module = djangoproj.wsgi:application

  master = true
  processes = 5

  # Create a UNIX socket that the web server can access.
  # 'www'is the group (not the user) that your web server runs as.
  socket = /home/jailuser/djangoproj/djangoproj.sock
  chown-socket = %(uid):www
  chmod-socket = 660
  vacuum = true


  # The number of worker processes to create.
  workers             = 5
  
  # Create multiple threads per worker.  This is more memory-efficient if your
  # application is thread-safe.
  enable-threads      = true
  threads             = 5

  # Use cheaper to kill off idle workers.  This doesn't always work well; for
  # example, on Debian 7 with recently uWSGI versions it appears to sometimes
  # deadlock the application when killing a worker.  If this isn't specified,
  # uWSGI will just create the maximum number of workers at all times.
  cheaper-algo        = spare
  cheaper             = 1
  cheaper-initial     = 1
  cheaper-step        = 1
  
  # Log errors and requests.
  logto               = /var/log/djangoproj-uwsgi.log
  log-date            = true
  log-prefix          = [djangoproj]
logfile-chown       = true

7.2 Test uWSGI ini file works

Do a uwsgi djangoproj/config/djangoproj.ini

Make sure everything starts up without errors.

8 Test website from public url

If everything is working properly, you should be able to see your django app from the public url.

If not, trace the route using Figure-1 as a guide.

Did the request make it to the BSD www jail nginx?

Did the request route correctly to your python jail?

Did the request inside python jail hit the nginx?

Did nginx reverse proxy and connect to the uWSGI app properly?

9 Setup Supervisord to manage uWSGI processes

So far the uWSGI process was kicked off from the command line.

If you wanted to start the uWSGI process at system startup, system reboot and manage like any system process, you need a uWSGI manager.

Supervisord (which was installed as part of initial setup) does the job.

9.1 Copy /usr/local/etc/supervisord.conf.sample

Copy to supervisord.conf and modify.

[program:djangoprojname]
directory=/home/jailuser/djangoproj
command=/usr/local/bin/uwsgi-3.6 --ini config/django-proj-uwsgi.ini
autostart=true
autorestart=true
stdout_logfile=NONE
stderr_logfile=NONE
; Handy for debugging:
; stderr_logfile = /var/log/djangoproj-err.log
; stdout_logfile = /var/log/djangoproj-out.log
stopsignal=INT

9.2 Setup supervisord to start at system startup

supervisord_enable="YES"  # add to rc.conf file

9.3 Verify supervisord is up

sudo service supervisord status                                                    
Password:
supervisord is running as pid 946.

9.4 Start supervisord

service supervisord res

9.5 Check uWSGI process in supervisord.conf is running

supervisorctl status

9.6 Update code using git pull

9.7 Restart the uwsgi process

supervisorctl restart djangoprojname

10 Database Setup

10.1 Create Postgres user and database

# swith to pgsql user
$ su pgsql

# Connect to database
$ psql template1

OR
# Connect as root specifying user and database
$ psql -d template1 -u pgsql


# Create user w/ perm to create databases. (man createuser to see options)
$ createuser -sdrP username 

# Createdb 
$ createdb <dbname>

# Connect as user create
$ psql -U <username> -h <hostname> -d <database_name>