Archives for posts with tag: development

postgresqlI’ve been working with postgres in my free time for a while now. I have been playing with fire, so to speak. I haven’t had any sort of backup solution on my postgres server. So after a bit of googling and reading postgres docs, I’ve came up with a very basic backup script.

What it does is backup each database separately and zips them up in order to save on space. It puts an hour – minute stamp on the database file and backs the DB up to a directory that is the current date. I’m going to add more to the script later, like only keeping so many backups, etc. But at this point, I just wanted to get a script running that creates backups just in case the server’s main drive dies or I botch something.

So without further ado, here’s the code of the script. I do have to give credit to the author of this post, as I heavily modified that script to do what I needed it to do. The big thing was the sed-awk line, as my sed-fu is quite poor.

#! /bin/bash
# PGSQL backup script

# logfile loc
logfile="/var/log/pgbackup.log"
# backup loc
backup_dir="/backup/pg_backups/"
touch $logfile
timeslot=`date +%H-%M`
monthday=`date +%F`

# mount /backup, create the dir and get a list of dbs
/bin/mount /backup
/bin/mkdir $backup_dir/$monthday
databases=`psql -U postgres -q -c "\l" | sed -n 4,/\eof/p | grep -v rows\) | awk {'print $1'}`

#backup the dbs
for i in $databases; do
        timeinfo=`date '+%T %x'`
        echo "Backup complete at $timeinfo for time slot $timeslot on database: $i " >> $logfile
	/usr/bin/pg_dump -U postgres $i | gzip > "$backup_dir/$monthday/postgre-$i-$timeslot-db.gz"
done

#unmount the backup drive
/bin/umount /backup

And that’s that. It’s a pretty simple script, but it gets the job done. I’m obviously running the script on the machine that postgres is on and the server I’m running it on is Ubuntu LTS with a backup drive in the server, so your milage may vary depending on your distro and your setup. You may want to test the script first before setting it as a cron on your DB server. Speaking of the cron, here’s the line you’ll want to set in your root crontab.

00 06 * * * /root/bin/pgbackup.sh > /dev/null 2>&1

I have the cron set to run at 6am since I’m asleep at 6am, so no one will be accessing the DBs. And now my DBs are safe from me and my postgres nubness.

A few jobs ago I was a Work Study in college, and I was the head of IT’s assistant. He was in charge of the PBX phone system along with the network and workstations.

We had to do some desk juggling and rather than change assignments in the PBX we decided to just move the cables that connected the patch panel from the desk sto the PBX. Easy right? Nope. My boss’s predecessor when he set up the PBX system, along with all of the other patch panels to labs and offices didn’t document anything. He also used 15 foot cables when a 1 foot cable would have worked, but that’s a rant for another day.

My boss and I spent 20 hours over the course of a weekend documenting which ports on the patch panel matched which port in which office. 40 man-hours wasted because my boss’s predescor didn’t take a few minutes to document as he set everything up.

This kind of behavior is hard for me to wrap my head around. And my PBX example above isn’t an isolated experience. It seems that non-documentation like this occurs in IT departments across the country. It appears to be a mixture of pure laziness this and a “If I know everything, I’m irreplaceable” mentality.

SPOILERS: Everyone is replaceable.

The replacement may not be better, but everyone is replaceable. Case in point, Michael Jordan, the greatest basketball player of all time, retired from the Bulls twice. After both times, the Bulls had to find another guy to start at Shooting Guard. Were either of them as good as Jordan? No, not at all, but they were his replacement.

My advice is this: You’re not going to be at your current job forever, so do yourself and everyone else a favor and document everything. I use a mediawiki install for personal stuff, but even if its just a collection of Word docs in a network share, find something that works for you and run with it.

I’m setting this up for my own python development, and so far it’s working LIKE A BOSS. Using my workstation as a development environment, and a server as the production environment, I’m synergizing the deliverables by maximizing throughput in the signal/noise ratio.

Erm.

That is, I’m using virtualenv (and a set of popular virtualenv wrapper scripts) along with git to keep development and deployment of python projects with varying dependencies in sync.

It’s disturbingly easy to set up:

  1. Install virtualenv
  2. Install virtualenvwrapper
  3. Put the virtualenvwrapper directory in version control
  4. create a virtualenv for each project

It’s that easy!!!

I’m a fan of using pip to install everything. It’s the future of python package management because it’s so flippin’ easy to use, as well as being sanely implemented.  The copypasta for the above setup runs something like this:

pip install virtualenv
pip install virtualenvwrapper
mkdir ~/.virtualenvs

Add the following to your .bashrc:

export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper_bashrc

then:

source ~/.bashrc
mkvirtualenv projectname
cd ~/.virtualenvs
git init

Now feel free to start using pip to build up the particular python environment the project will need, commiting the ~/.virtualenv directory to git as needed. When you’ve got your python libraries how you like, install virtualenv on the production server, and add it to the repository for your project. I’m thinking I’ll set it as a git submodule for Django projects, but so far I’ve just been keeping it as a separate repository for non-public in-development non-website projects.