# 1. Create virtual machine via VMware's "Easy Install" easy install option. Use the username "bootstrap" with the password "bootstrap". (We'll create other accounts via chef.) 2. Login via the VMware console, then update packages: $ sudo apt-get update && sudo apt-get upgrade 3. Install openssh: $ sudo apt-get install openssh-server 5. Change hostname: $ sudoedit /etc/hostname # must be unique across chef-managed nodes! $ sudoedit /etc/hosts # change "ubuntu" to whatever the hostname is 4. Get IP address: $ ifconfig 5. Reboot: $ sudo shutdown -r now 6. Create `deploy` account on git server: $ sudo useradd -m deploy $ sudo -u deploy ssh-keygen -N '' -f /home/deploy/.ssh/id_rsa $ sudo -u deploy cp /home/deploy/.ssh/id_rsa.pub /home/deploy/.ssh/authorized_keys $ sudo cat /home/deploy/.ssh/id_rsa # copy the public key into the deploy_to key of the apps/beebo # data bag via 'knife data bag edit apps beebo' # ... or into /var/www/beebo.org/id_deploy directly Note that the `deploy` account needs read-only access to the `deploy@grace.beebo.org:/var/lib/git/beebo.org` repository. 7. Bootstrap from host: $ knife bootstrap 172.16.152.131 -x bootstrap -Pbootstrap -r 'role[base],role[beebo]' --distro ubuntu10.04-gems --sudo (Enter "bootstrap" again as the sudo password when prompted.) # Getting Beebo (and other sites) running on Ubuntu What follows is mostly instructions for my own use that tell me how to get `beebo.org` running on Ubuntu, with separate instructions for both a virtual machine running under VMware Fusion, and on gandi's virtual server. However, they may be somewhat helpful to anyone trying any of these applications on their own Ubuntu server. ## Prerequisites You must have `knife` installed, and be able to run `knife node list` before configuring a server to run Beebo. To do this, you need to do two things: install `chef`, and download and install the configuration files associated with the beebo chef organisation. ### Install Chef To install chef on OS X or Ubuntu, follow Opscode's [Quick Start Instructions](http://wiki.opscode.com/display/chef/Quick+Start). On other platforms, follow Opscode's [Verbose Instructions](http://wiki.opscode.com/display/chef/Installation#Installation-InstallingChefClientandChefSolo). You probably want to install the `net-ssh-multi` rubygem at the same time as installing chef itself; this is needed for `knife bootstrap`. ### Download Config Files The config files are stored in git. Simply clone the `$GITROOT/chef` repository into `~/.chef`: $ cd ~ $ git clone $GITROOT/chef .chef At this point `knife node list` should show you a list of nodes connect to the beebo organisation: $ knife node list [ "babyboom.localdomain", "syd.beebo.org" ] ## Initial System Setup That is, what you need to do, under different environments, to get the system to a point where chef can be bootstrapped via `knife bootstrap`. In the following sections, it is assumed that the name of the regular user account is `mjs`. ### VMware Fusion 3.1 1. Use the "Easy Install" process, and create the account `mjs`. 1. Log in to the virtual machine via VMware's terminal using the username and password you supplied in the "Easy Install" dialog. 1. Install the `openssh-server` package via $ sudo apt-get install openssh-server 1. Set the name of the server: 1. Change `/etc/hostname` to the hostname of the machine. (e.g. `babyboom`--this shouldn't include a dot.) 1. Modify `/etc/hosts` replacing every instance of the string `ubuntu` with the hostname. (e.g. `babyboom`.) 1. Once the ssh server is running, modify `/etc/hosts` **on the host** and point the following hostnames at the IP address of the virtual machine: beebo.local www.beebo.local static.beebo.local scratch.local (Use `ifconfig` on the VM to find the IP address.) It should now be possible to `ssh mjs@beebo.local`. 1. Enable shared folders, and make `~/workspace` on the host available as `/mnt/hgfs/workspace` on the VM. (If this doesn't "just work", you'll probably need to recompile and reinstall the HGFS kernel modules. Do this by running the following commands in order: * sudo apt-get update && sudo apt-get -u upgrade * sudo apt-get install build-essential linux-headers-$(uname -r) * sudo vmware-config-tools.pl 1. Ensure that all source code dependencies (e.g. Zend Framework) have been downloaded and installed in the expected location. (See `README.md` in the root directory of the `beebo` repository.) ### VirtualBox 3.x (Note: I don't use VirtualBox anymore; this hasn't been tested in a while.) 1. Download Ubuntu Server Edition from . We're going to install the [JeOS](http://www.ubuntu.com/products/whatisubuntu/serveredition/jeos) edition (minimal install designed for Virtual Machines), but this ISO is now combined with the standard server ISO. 1. Follow the standard installation instructions to boot and install Ubuntu from the `*.iso`. 1. The virtual machine will probably boot into the graphical Ubuntu installer. Choose your language, then hit F4 (Fn-F4 on Macs), choose "Install a minimal virtual machine", and then choose "Install Ubuntu Server." (This selects the [JeOS](http://www.ubuntu.com/products/whatisubuntu/serveredition/jeos) edition on Ubuntu Server Edition 8.10 or later. If you have Ubuntu 8.04 you need to download the (separate) 8.04 ISO.) 1. Continue through the (text mode) installer as usual. Note that if you want to be offered United Kingdom time zones you need to choose "United Kingdom" for your language. 1. Update the package list, and install any updates: $ sudo apt-get update && sudo apt-get -u upgrade 1. Install/reinstall "Linux Guest Additions". 1. Install various packages necessary for compiling the drivers (if reinstalling the guest additions after an upgrade, check that these are actually installed, especially the kernel headers): $ sudo apt-get install dkms # "Dynamic Kernel Module Support Framework" $ sudo apt-get install build-essential # compiler, compiler tools $ sudo apt-get install linux-headers-$(uname -r) 1. From the "Devices" menu, select "Install Guest Additions...". This does the equivalent of putting a CDROM in disk drive. 1. Mount this CDROM: $ mount /dev/cdrom 1. Run the installer: $ cd /cdrom && sudo sh ./VBoxLinuxAdditions-x86.run 1. Recompile the guest kernel modules (is this actually necessary?): $ sudo /etc/init.d/vboxadd setup 1. Install the ssh server: $ sudo apt-get install openssh-server 1. Shutdown so that we can configure networking: $ sudo shutdown -h now 1. Configure networking. 1. Even though the networking is running at this point (you should be able to ping `google.com` from your virtual machine, for example), for some reason VirtualBox arranges networking so that you can't ssh in to your virtual machine from the host. You need to configure VirtualBox for forward ports on the host to ports on the guest. You do this by editing the virtual machine's configuration file, which is probably stored in a location like `~/Library/VirtualBox/Machines/Ubuntu JeOS 8.04/Ubuntu JeOS 8.04.xml`. In the `` element (should be near the top), add the following lines: This maps `localhost:8080` to port 80 on the virtual machine (for http) and `localhost:8022` to port 22 on the virtual machine (for ssh). 1. Restart the virtual machine. You should now be able to ssh in to the guest with: $ ssh -p 8022 mjs@localhost 1. Configure shared folders. 1. In the VirtualBox GUI on the host, configure "Shared Folders", so that `/User/mjs/workspace` is made available under the name `workspace`. (You may need to shut the guest down first.) 1. At this point, it should be possible to mount the shared folder to the directory `/mnt` via: $ sudo mount -t vboxsf workspace /mnt 1. To arrange for this to happen automatically at boot, add the following line to `/etc/fstab`: workspace /mnt vboxsf uid=1000,gid=1000 This will mount the drive as if it is owned by the user with user id 1000, and group id 1000; check to see what ids your user has with `id`. 1. Reboot, or apply the mounts manually: $ sudo mount -a For more information see the in-application help. (A PDF file.) ### Gandi 1. In the VPS management interface, create an Ubuntu 10.10 64-bit virtual machine, with `ssh` access via the username `mjs`. 1. Allow `mjs` to execute `sudo`: $ su - # become root $ echo "mjs ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers NOTE: If editing this file by hand, use `sudoedit` not `sudo $EDITOR ...`. 1. Edit `/etc/default/gandi` and change `HOSTNAME_TYPE` from `short` to `full`: $ vim /etc/default/gandi (This sets `/etc/hostname` to the full hostname, and ensures that `hostname -f` returns the fully qualified domain name--these are needed for chef.) 1. Reboot, and then check that `hostname -f` does in fact return the full domain name: $ sudo shutdown -r now # ... $ hostname -f syd.beebo.org ## Install Chef To install everything else needed to get the site running, use `knife bootstrap`: this installs `chef-client` on the server, and assigns it a particular role. Before doing this, check that the value of the `node[:beebo][:servers]` attribute for the host you're bootstrapping is a list of the virtual hosts that the server is supposed to provide. Do this via e.g. $ knife node edit babywhimsy.local For example, to install `chef-client` on the virtual machine, and get it to "converge" to the `vm` role, run the following command **on the host**: $ knife bootstrap $HOSTNAME -x $USERNAME -P$PASSWORD -r 'role[vm]' --distro ubuntu10.04-gems --sudo # e.g. $ knife bootstrap beebo.local -x mjs -PXXX -r 'role[vm]' --distro ubuntu10.04-gems --sudo # (remember to specify the password!) See the [chef documentation](http://wiki.opscode.com/display/chef/Knife+Bootstrap) for more information, including the distributions that are supported. (10.04 works with 10.10.) (We're using the `gems` distro instead of `apt` because otherwise we seem to get 64-bit binaries installed on 32-bit systems.) Note that the settings that will be applied are a function of the role (passed in the `knife bootstrap` command line) and the attributes assigned to the node (i.e. hostname). In particular, if you're using a completely new hostname, this probably won't do what you expect until you set the attributes as appropriate. For example, the location of Apache's document root is configured via the node's `node[:beebo][:servers]` attribute. This must be set correctly for the server to be configured correctly. ## Install password, source, data, symlinks At this point, the server is configured to run `beebo.org`, but the source code, databases, and a few other things are missing. ### Password So that the bulk of the source, including config files, can be placed on a public server, the passwords stored within the config are encrypted. The decryption procedure reads the file `config/password` to decrypt them; add the current password to this file. ### Source code and symlinks 1. The location of the source code is configured via the `node[:beebo][:servers]` attribute in chef. (See `chef/cookbooks/beebo/attributes/default.rb`.) To arrange for the source code to be in the correct location, the following commands may be useful. $ git clone --branch dev ../git/beebo.org dev.beebo.org This clones the `live` branch of the git repository in `../git/beebo.org` into `dev.beebo.org`, using hardlinks to save space and increase performance. Currently, the source code is cloned into `/srv/slammer/www/beebo.org` on `syd`, and a `post-receive` script in `/srv/slammer/git/beebo.org/hooks` does: #!/bin/bash unset GIT_DIR pushd /var/www/dev.beebo.org >/dev/null git pull cd lately jekyll --no-auto popd pushd /var/www/beebo.org >/dev/null git pull cd lately jekyll --no-auto popd (The `unset GIT_DIR` is very important! (Why?) Also, make sure the script is executable.) 1. Check out the latest version of Zend Framework: # In /srv/slammer/www $ svn co http://framework.zend.com/svn/framework/standard/tags/release-1.11.2/library/Zend Then symlink this into the source at `lib/Zend`. 1. Similarly, create a symlink from `data/quotes` to `misc/quotes`. 1. And finally, create a symlink from `web/stuff/tech` to `misc/tech`. ### MongoDB MongoDB is expecting to find its databases in `/var/lib/mongodb` (see `/etc/init/mongodb`). (Though you probably want this to be a symlink to a larger drive.) To get the database loaded into MongoDB, the following commands may be useful. * Dump existing database to `backup` directory in current directory: $ mongodump -o backup * Import contents of `backup` directory into running MongoDB instance: $ mongorestore backup * Stop MongoDB: $ sudo stop mongodb * Start MongoDB: $ sudo start mongodb ### SQLite The location of the databases is store in config files (e.g. in `config/beebo.local/config.ini`), but it's recommended that they actually go into `/var/lib/sqlite` (which again, can be a symlink). To move existing SQLite databases, the following commands may be useful. * Export SQLite database: $ sqlite3 tracker.db .dump | gzip tracker.sql.gz * Import SQLite database: $ gunzip -c tracker.sql.gz | sqlite3 tracker.db ## FAQs and HOWTOs ### HOWTO: Back Up 1. Use the script called `backup` in `beebo/bin` to back the various databases up into `/srv/backup`. 1. Copy the backups onto an offsite machine with a command like `rsync -CLvax mel.beebo.org:/srv/backup .` ### HOWTO: Restore You'll have to do it manually, I'm afraid. ### HOWTO: Create SQLite database files from `*.sql.bz2` dumps export SRC=~/Documents/Backup/beebo/2010-06-24/sqlite export DST=~/workspace/beebo/db for f in $SRC/*.bz2 ; do bzip2 -dc $f | sqlite3 $DST/$(basename $f .sql.bz2).db ; done ### HOWTO: Encrypt passwords From the root directory, run something like: $ DOCUMENT_ROOT=web php bin/encrypt jjjj encrypted:v2P+7KCtn+3zojavwjbpnw== ### HOWTO: Decrypt passwords The config contains a number of encrypted passwords. To decrypt these you need a password file that is not contained in the repository. It needs to go in `config/password`, should simply be the password itself, as text: $ echo "ajaafsadsoj" > config/password ### FAQ: Why did the shared folders stop working? Perhaps because the kernel has been upgraded--to fix it, run $ sudo /usr/bin/vmware-config-tools.pl to reinstall VMware's kernel modules. If it complains about missing headers, add the appropriate headers with $ sudo apt-get install linux-headers-`uname -r` ### HOWTO: Copy SQLite databases from live to another machine Do something like this: function copy-sqlite-from-live { if [ "$1" = "" ]; then echo "usage: $FUNCNAME database.db" return fi rm -f /var/lib/sqlite/$1 ssh -C mjs@grace.beebo.org sqlite3 /var/lib/sqlite/$1 .dump | sqlite3 /var/lib/sqlite/$1 # Make sure apache can read/write the database chgrp www-data /var/lib/sqlite/$1 chmod 664 /var/lib/sqlite/$1 } copy-sqlite-from-live beebo.db # creates /var/lib/sqlite/beebo.db copy-sqlite-from-live tracker-current.db # creates /var/lib/sqlite/tracker-current.db Restart Apache after copying databases! Some (and only some) Apache processes might have a connection open to the old database.