Archives for posts with tag: bash

Accessing remote file systems with sshfs

August 12, 09 //
0

Like anyone has data stored accross multiple servers, keeping it organized can be a headache. At my house, I’m using a server running Samba to store my data for my laptop and desktop. That way, when I edit file X, I know it’ll be the same on either system.

Samba works great at home. However, opening a Samba share to entire world is a Bad Idea™.I wanted to be able to access some of my files off of my server at work that was secure and didn’t require setting up too much extra software. And this is where sshfs comes into play. Sshfs is a Filesystem client based off of the SSH Protocol. I picked sshfs over something like webdav because the server in question doesn’t have Apache installed and all of the servers I work on run ssh.

Setting up and using sshfs is pretty straight forward. First, install sshfs on the client. That’s one of the best part of sshfs; as long as the server has ssh running, you don’t need have to do anything on the server.  All of my boxes are running a varient of Debian, so I just used aptitude to install sshfs.

sudo aptitude install sshfs

I have not tried installing this on other linux distros, but I’m going to guess that using their package managers will give you similiar results. If you want to do things the hard way, source can be downloaded here. You’ll need FUSE2.2 and the latest version of glibc devel installed also. Your package manager of choice should take care of the dependencies for you.

Once you’ve got the software installed, you’ll need to add your normal user to the fuse group so they can mount and unmount the sshfs shares. You can use your GUI of choice, of the following CLI command:

sudo adduser user fuse

That’s it! That’s all you need to do to setup sshfs. Like with any file system, make a directory to mount the system to and then run the following command:

sshfs remoteuser@remotesystem:/path/to/dir localdir

You’ll now be able to access that directory remotely using sshfs. When running that above command, you may get the following error:

fusermount: failed to open /dev/fuse: Permission denied

Try running the command in a fresh shell. Chances are the current shell hasn’t picked up on the refreshed permissions. To unmount the sshfs directory, run the following command:

fusermount -u mountpoint

Like any filesystem, sshfs shares can be automagically mounted via the server’s fstab. Please note that if you do want to have the sshfs automount due to fstab, you will need to have SSH keys setup between the client and the server. Below is the syntax you’d want to use:

sshfs#USERNAME@REMOTE_HOST:REMOTE_PATH MOUNT_POINT fuse SSHFS_OPTIONS 0 0

It’s a pretty straightforward solution to a problem that had been bugging me for a while. I am able to access my remote files on a server with the ease of a samba share without having to install any extra software on the server itself. To me, something like sshfs is more convenient than webdav or sftp, because it allows the remote system to integrate seamlessly into my current system. I hope you find sshfs as useful as I have.

Backing up postgreSQL DBs with bash and cron

July 21, 09 //
1

postgresqlI’ve been working with postgres in my free time for a while now. I have been playing with fire, so to speak. I haven’t had any sort of backup solution on my postgres server. So after a bit of googling and reading postgres docs, I’ve came up with a very basic backup script.

What it does is backup each database separately and zips them up in order to save on space. It puts an hour – minute stamp on the database file and backs the DB up to a directory that is the current date. I’m going to add more to the script later, like only keeping so many backups, etc. But at this point, I just wanted to get a script running that creates backups just in case the server’s main drive dies or I botch something.

So without further ado, here’s the code of the script. I do have to give credit to the author of this post, as I heavily modified that script to do what I needed it to do. The big thing was the sed-awk line, as my sed-fu is quite poor.

#! /bin/bash
# PGSQL backup script

# logfile loc
logfile="/var/log/pgbackup.log"
# backup loc
backup_dir="/backup/pg_backups/"
touch $logfile
timeslot=`date +%H-%M`
monthday=`date +%F`

# mount /backup, create the dir and get a list of dbs
/bin/mount /backup
/bin/mkdir $backup_dir/$monthday
databases=`psql -U postgres -q -c "\l" | sed -n 4,/\eof/p | grep -v rows\) | awk {'print $1'}`

#backup the dbs
for i in $databases; do
        timeinfo=`date '+%T %x'`
        echo "Backup complete at $timeinfo for time slot $timeslot on database: $i " >> $logfile
	/usr/bin/pg_dump -U postgres $i | gzip > "$backup_dir/$monthday/postgre-$i-$timeslot-db.gz"
done

#unmount the backup drive
/bin/umount /backup

And that’s that. It’s a pretty simple script, but it gets the job done. I’m obviously running the script on the machine that postgres is on and the server I’m running it on is Ubuntu LTS with a backup drive in the server, so your milage may vary depending on your distro and your setup. You may want to test the script first before setting it as a cron on your DB server. Speaking of the cron, here’s the line you’ll want to set in your root crontab.

00 06 * * * /root/bin/pgbackup.sh > /dev/null 2>&1

I have the cron set to run at 6am since I’m asleep at 6am, so no one will be accessing the DBs. And now my DBs are safe from me and my postgres nubness.

Deleting a Crapton of Small Files from a Directory Using Find

July 14, 09 //
1

One of the fun situations you find yourself in when administrating people’s email (at least, I find myself in this situation frequently enough, and I am an uninteresting person, so this is what passes for a fun situation in my world) is the dreaded I-have-magically-hit-my-email-quota problem. The conversation usually runs something like this:

User: HALP I cannot send any email on this address!

Me: checks logs The mail logs are telling me that it can’t write to file due to quota issues. Looks like you’ve hit your $BIGNUM quota.

User: I don’t know why the system quota is saying I have $BIGNUM of mail when $MAILCLIENT informs me I have $MUCHSMALLERNUM of mail.

Me: cd’s into user’s maildir directory, starts running ll -ah like a boss

User: I mean, holy crap, I friggin’ hate email, why does the world have to hate me, I have $VERYIMPORTANTATTACHMENT to send to $BIGCLIENT and then this happens. Why don’t computers run on happiness and unicorn farts?

Me: Ah, it looks like this email address has $BIGNUM – $MUCHSMALLERNUM worth of mail in its .spam directory.

User: I have a dot spam directory? How come I can’t see it in $MAILCLIENT?

Me: You know, I could explain the concept of how hidden directories work in linux, and bemoan the fact that the mailserver uses them and most mail clients don’t, but in lieu of that, how about I delete all of that spam for you, thereby fixing your quota issue?

User: in a monotone Thank you sir may I have another

Here is the bash one-liner for removing the mails from this directory:

for file in $(find -type f -maxdepth 1); do rm -fv $file; done

“But sir! Sir!” you find yourself saying, “why on earth are you not using xargs? Surely piping find’s output to xargs is more memory-efficient than a naive for loop!”

While it is true that a for loop is less memory efficient than piping find to xargs, I always use it in this situation. The reason is simple; if a directory is $BIGNUM large, and most of the files in it are measured in tens of kilobytes, your find command will more often than not send too many arguments xarg’s way. Your time is too precious a commodity to see whether or not this is one of the situations when the stars align and the mail quota is small enough that the directory doesn’t have too many files for xargs. Just use the for loop. If the server is reasonably fast, it will handle it gracefully.

Every once in a while you’ll run into an enlightened user who realizes, just as you’re about to go and mow through the .spam directory, that hey, there was an important email last week that somehow never got through. They’d like to be able to poke at the last week’s .spam emails. Which means you can’t just delete all the mails willy-nilly. This is why God invented the -mtime flag.

for file in $(find -type f -maxdepth 1 -mtime +7); do rm -fv $file; done

That -mtime +7 causes find to snag only the files that haven’t been modified in the last seven days.

I really can’t get over how useful a little command like find can be. I mean, I’ve tried to get over it. Brought it into therapy a couple of times. Still can’t shake how useful it is. I usually refer to this find tutorial when I trip over the syntax.

One final tip: before you start deleting data with reckless abandon, I highly recommend substituting “do echo $file;” for "do rm -fv $file;" in the above one-liners. It never hurts to make sure that you’re actually grabbing the files you think you are before you delete them.