Buildbot CI on freeBSD
Table of Contents
1 Buildbot concepts
- Buildbot is a job scheduling system - e.g. CI/CD.
- For installation, you need a master and worker.
- Master and Worker can start on the same machine.
- A buildbot master can have 1 to N workers distributed across multiple nodes.
1.1 Builders
A builder (represented as BuildFactory) is an action or a group of actions to build a unit of software.
A builder has a list of workers it can use to execute the build.
You can use a Scheduler to schedule a build.
A build can have a source. (source repo)
2 Setup
2.1 As root
$ pkg install git py36-pip py36-pipenv py36-buildbot py36-buildbot-www
2.2 Create user to run buildbot master
$ pw useradd -n buildbot-master -m -w random
2.3 Directory to hold buildbot master config
E.g. /var/www/buildbot-ci/bb-master
2.4 Switch to directory (e.g. bb-master)
Turn on virtualenv
$ pipenv-3.6 shell
2.5 Give ownership of directory to bb-master user
$ chown buildbot-master:buildbot-master /var/www/buildbot-master
2.6 Switch as unprivileged user
$ su -l buildbot-master
2.7 Create master configuration
$ buildbot-3.6 create-master /var/www/buildbot-master
3 Configure Buildbot Master
3.1 Create master.cfg file
The create-master command creates a master.cfg.sample file you can use to start.
Create a copy.
$ cp /var/www/buildbot-master/master.cfg.sample /var/www/buildbot-master/master.cfg
3.2 Edit master.cfg file
Only relevant parts shown for brevity
c['workers'] = [worker.Worker("example-worker", "pass")] # Worker
c['change_source'] = [] # Source control
c['change_source'].append(changes.GitPoller(
'git://github.com/buildbot/hello-world.git', # replace with your repo
workdir='gitpoller-workdir', branch='master',
pollInterval=300))
# There are different types of schedules. Poll, force, periodic...
c['schedulers'] = []
c['schedulers'].append(schedulers.SingleBranchScheduler(
name="all",
change_filter=util.ChangeFilter(branch='master'),
treeStableTimer=None,
builderNames=["personalsite"]))
c['schedulers'].append(schedulers.ForceScheduler(
name="force",
builderNames=["personalsite"]))
psf = util.BuildFactory() # Build
psf.addStep(steps.ShellCommand(command=["git", "checkout", "--", "."],
workdir="/var/www/site"))
psf.addStep(steps.ShellCommand(command=["git", "pull"], workdir="/var/www/site"))
psf.addStep(steps.ShellCommand(command=["chmod", "-R", "775", "kiranbusi.com"], workdir="/var/www/personal_site"))
c['builders'] = [] # Add buildfactor to builders
c['builders'].append(
util.BuilderConfig(name="personalsite",
workernames=["example-worker"],
factory=psf))
# Password protect buildbot GUI
c['www']['auth'] = util.UserPasswordAuth([('username','changepassword')])
3.3 Reload configuration
$ buildbot reconfig master
# Run Buildbot Master $ buildbot start master # view logs $ cat master/twistd.log
3.4 Go back to root user
4 Run buildbot master
4.1 Manually
As buildbot-master user in virtual env
In buildbot-master directory (e.g. /var/www/buildbot-master)
$ buildbot start
4.2 Automatically 1
To start on system startup
Define location of master directory
$ sysrc buildbot_basedir=/var/buildbot-master
Specify the user the service should run under
$ sysrc buildbot_user=buildbot-master
Start service at system bootup
$ sysrc buildbot_enable=YES
Start buildbot service
$ service buildbot start
4.3 Verify by checking log file
Look at twistd.log in master config directory
5 Setup buildbot worker
Create a folder to hold buildbot worker config
$ mkdir /var/www/buildbot-worker
Install pkgs
$ pkg install py36-buildbot-worker
Create buildbot-worker user
$ pw useradd -n buildbot-worker -m -w random
Give directory permissions for buildbot-worker user
$ chown buildbot-worker:buildbot-worker /var/buildbot-worker
Swith to buildbot-worker user
5.1 Create buildbot-worker
$ buildbot-worker-3.6 create-worker /var/buildbot-worker localhost worker1 'pass'
Make sure information in master.cfg matches
$ pipenv-3.6 shell # Create a virtualenv using pipenv $ pip install buildbot-worker' # Install buildbot-worker $ pip install setuptools-trial # required for `runtests` build #+ENDSRC
—
# Create the buildbot worker in the bb-worker virtualenv # Name of worker: worker # Host: localhost (name of computer where buildbot master is running ) # Username: worker-A # Password: pass $ buildbot-worker create-worker worker localhost worker-A pass
—
Be sure to match bb-master/master/master.cfg workers
'c['workers'] = [worker.Worker("worker-A", "pass")]'
—
# Start the worker # In var/www/buildbot-ci/bb-worker virtualenv run: buildbot-worker start worker Following twistd.log until startup finished.. The buildbot-worker appears to have (re)started correctly.
6 Run buildbot-worker
6.1 Manually
As buildbot-worker in buildbot-worker config directory
$ buildbot-worker start
6.2 Automatically
Leverage the rc script by using the service utility to manage the service. Else on every reboot you'll notice your build is down
sysrc buildbot_worker_basedir=/var/buildbot-worker sysrc buildbot_worker_uid=buildbot-worker sysrc buildbot_worker_gid=buildbot-worker sysrc buildbot_worker_enable=YES
Start buildbot worker
$ service buildbot-worker start
Check twistd.log in buildbot-worker directory for success/failure/debug
7 Checkpoint
Verify buildbot is running on port 8010 on localhost
curl http://localhost:8010
8 Nginx Setup
Host buildbot behind nginx
Create a conf file (e.g. /usr/local/etc/nginx/sites/buildbot-nginx.conf
server {
listen 80;
server_name domain_name.com;
include /usr/local/etc/nginx/snippets/letsencrypt.conf;
# Redirect other HTTP connections to HTTPS
location / {
return 301 https://domain_name.com$request_uri;
}
}
# HTTPS
server {
listen 443 ssl;
server_name domain_name.com;
ssl_certificate /usr/local/etc/letsencrypt/live/domain_name.com/fullchain.pem;
ssl_certificate_key /usr/local/etc/letsencrypt/live/domain_name.com/privkey.pem;
access_log /var/log/nginx/domain_name.com-access.log;
error_log /var/log/nginx/domain_name.com-error.log;
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/local/www/nginx-dist;
}
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 10m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
# Use gzip compression
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 5;
gzip_buffers 16 8k;
gzip_http_version 1.0;
# Set a variable to work around the lack of nested conditionals
#set $cache_uri $request_uri;
# you could use / if you use domain based proxy instead of path based proxy
location / {
proxy_set_header HOST $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-Host $host;
proxy_pass http://127.0.0.1:8010/;
}
location /sse/ {
# proxy buffering will prevent sse to work
proxy_buffering off;
proxy_pass http://127.0.0.1:8010/sse/;
}
# required for websocket
location /ws {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://127.0.0.1:8010/ws;
# raise the proxy timeout for the websocket
proxy_read_timeout 6000s;
}
}
9 Test deployment
9.1 Login via GUI and force build a project
10 Security
SSL Certs
Rerun omitting dry run if dry run works
$ certbot certonly -d domain_name.com --dry-run