sysadmin

Using Smugmug as a Photo Backup Location

I’ve finally gotten around to uploading my photo collection. I’ve decided to pay for a subscription to Smugmug.com and so far I’m happy with it. I chose not to use Flickr or other options because I wanted a site which I could use as a private one-to-one backup of my personal store of photos. I can make albums or photos public using other services or through Smugmug. I also wanted to be able to synchronize tags and metadata between the on-line store and my digikam-managed local version. If I ever lose my local copy, I should be able to download a full copy of everything in its original form. Smugmug appears to be able to do all of that (and has other good features, too).

To do the synchronization, I’m using the php uploader found here. It seems to be working, although I had some trouble at first pointing it at the right directory on my system. Running the upload program is simple:

php /path/to/multi-album-upload.php /path/to/pictures

NOTE: I initially had a little trouble because I was pointing to the wrong level in the hierarchy of my picture directory tree. The program assumes an arrangement something like this:

home
user
photo_library
category
album
image.jpg

My reading of the instructions implied that you could point to “/home/user/photo_library/category” but that doesn’t work. You need to point to “/home/user/photo_library.”

Scripts for Monkeying with Cinelerra XML

I’ve moved my Cinelerra projects from one directory path to another and scrambled a few things in the process. I’ve written a few bash scripts which helped me sort things out.

In the process, I’ve learned a bit about the xml format which Cinelerra uses to store references to files and edits. When you add a resource, it appears twice between opening and closing “ASSET” tags with the format <ASSET SRC=”/path/to/filename”>[asset details]</ASSET>. Then every time you include that file in a track of your project, the filename appears again in an “EDIT” tag complex in the format <FILE SRC=”/path/to/filename”>[edit details]</FILE>

First, I needed to check the paths for all the resources in the file to make sure they are still valid. I called this program test_xml_paths.sh. I corrected the incorrect paths with a text editor, but found that Cinelerra was still crashing with a memory error when I loaded the project.

Second, I decided to find and eliminate all of the unused assets in the file. I originally had a forty minute video in one file, but some time ago I split the project into four ten minute chunks. When I created the original project, I had added a couple of thousands of clips for possible inclusion, so now that I’m working on the final cut, I don’t need all that bloat.

I created a function to figure out which assets were being used in the edit list and which were not: list_unused_assets.sh.

Finally, I wrote a script which would recreate a new xml project file with all the redundant assets removed: remove_unused_assets.sh.

I can’t offer any warranty on these scripts, of course, but I hope they may help somebody else. If nothing else, there’s a great example of multi-line sed editing in the last one.

Remove Unused Assets from a Cinelerra XML Project File

This is a bash script to remove all the unwanted assets from a Cinelerra project XML file. I needed it to cut out bloat from a project I was working on which had hundreds of unused resources after I split the project up into ten minute chunks. Doing so stopped Cinelerra from crashing when I loaded the project: YMMV.

Save this script as “remove_unused_assets.sh”. It needs the function in “list_unused_assets.sh” which I posted here.

#! /bin/bash
#kk remove_unused_assets.sh
#kk
#kk For cinelerra xml files
#kk this script removes the contents of asset tags which are not also used for edits

if [ -z "$1" -o "$1" = "-h" ]
then
echo “Usage: remove_unused_assets [xml_filename]“
echo “For cinelerra xml files, this script removes the contents of asset tags which are not being used for edits”
exit
fi

TMPDIR=”/tmp”
SCRIPTDIR=”$HOME/Video/video_new/common/scripts”

. $SCRIPTDIR/list_unused_assets.sh
. $SCRIPTDIR/include.inc

#assign the parameter to a variable
XMLFILE=$1

#location of the temporary files
ASSETS_FILE=”$TMPDIR/ASSETS_FILE”
EDITS_FILE=”$TMPDIR/EDITS_FILE”
XMLFILE_NEW=”$1″_new
XMLFILE_TMP=”$TMPDIR/XMLFILE_TMP”

#check if the xml file exists and exit if it does not
if [ ! -e $XMLFILE ]; then error “XML file does not exist”; fi

list_unused_assets $XMLFILE
if [ $? != 0 ] ; then error “list_unused_assets failed”; fi

cp $XMLFILE $XMLFILE_TMP

echo
echo Removing Unused Assets…
echo
echo “Working…”
echo

# loop through the list of unused assets and remove the ASSET tag lines for each
# sed command from here: http://ilfilosofo.com/blog/2008/04/26/sed-multi-line-search-and-replace/

for f in $( comm -23 $ASSETS_FILE $EDITS_FILE ); do

#to make the filename string work in sed, I have to escape all of the periods and slashes
FORMATTED_f=`echo “$f” | sed -e ‘s:\/:\\\/:g’ -e ‘s:\.:\\\.:g’`

#this complicated sed script came from the sed FAQ: http://sed.sourceforge.net/sedfaq4.html#s4.21
#in order to make the variable substitution work, I had to close the sed script with a single quote
#then enclose the variable in double quotes, then re-open the sed script with a single quote
#the -i option is for editing a file “in place”
sed -i -e ‘
# sed script to delete a block if /regex/ matches inside it
:Top
/\/ { # For each line between these block markers..
/\/ASSET\>/!{ # If we are not at the /end/ marker
$!{ # nor the last line of the file,
N; # add the Next line to the pattern space
b Top
} # and branch (loop back) to the :Top label.
} # This line matches the /end/ marker.
/’”$FORMATTED_f”‘/d; # If /regex/ matches, delete the block.
} # Otherwise, the block will be printed.
#—end of script—
‘ $XMLFILE_TMP

#diff $XMLFILE $XMLFILE_TMP

echo -n ‘ \r’
echo -n $f’\r’

done

cp $XMLFILE_TMP $XMLFILE_NEW

echo
list_unused_assets $XMLFILE_NEW

echo
echo Finished.
echo
echo The new version of the xml file has been saved as $XMLFILE_NEW.
echo

This program depends on include.inc which contains the following:

#!/bin/bash

#use colour in the output just for fun
GREEN=$(printf “\033[32m”)
RED=$(printf “\033[31m”)
NC=$(printf “\033[0m”)

error()
{
echo ERROR: $1
exit 1
}

I tried to make these scripts as tidy and portable as possible — you will, however, need to change the path for $SCRIPTDIR to the location you are using (and $TMPDIR if you don’t want to put your temporary files in /tmp).

List Unused Assets in a Cinelerra XML Project File

This is a bash script which contains a function used to figure out which assets in a Cinelerra project are not required for the edit list. I created it because I had pulled in a couple of thousand media resources for a project and wanted to eliminate bloat. I moved the project and found it kept crashing on me until I eliminated the crud. I wrote another program to call this function and then actually clean out the xml. You can view “remove_unused_assets.sh” here.

Running this script doesn’t do anything until you call the enclosed function. This function can be called from the command line with:

. list_unused_assets.sh;list_unused_assets
#! /bin/bash
#kk list_unused_assets.sh
#kk
#kk check a cinelerra xml file to see which assets have been loaded, but not used in the project
#kk provide filename to check as parameter
#kk from the command line, type “. list_unused_assets.sh;list_unused_assets

list_unused_assets()
{

SCRIPTDIR=”$HOME/Video/video_new/common/scripts”
TMPDIR=”/tmp”

. $SCRIPTDIR/include.inc

#location of the temporary files
ASSETS_FILE=”$TMPDIR/ASSETS_FILE”
EDITS_FILE=”$TMPDIR/EDITS_FILE”

#assign the parameter to a variable
XMLFILE=$1

#check if the xml file exists and exit if it does not
if [ ! -e $XMLFILE ]; then error “XML file does not exist”; fi

#if the temp file exists, delete it
if [ -e $ASSETS_FILE ]; then
rm $ASSETS_FILE
fi

if [ -e $EDITS_FILE ]; then
rm $EDITS_FILE
fi

echo
echo Checking $XMLFILE for assets unused by the project…
echo

#find all of the filenames in the xml file and strip the extra junk
grep -E “ASSET SRC=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/”>.*$//g’ | sort | uniq > $ASSETS_FILE
grep -E “FILE SRC=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/”>.*$//g’ | sort | uniq > $EDITS_FILE

#output the counts
echo `grep -E “ASSET SRC=|FILE SRC=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/”>.*$//g’ | sort | uniq | wc -l` Total Assets
echo $RED`comm -23 $ASSETS_FILE $EDITS_FILE | wc -l` Unused Assets$NC
echo $GREEN`comm -12 $ASSETS_FILE $EDITS_FILE | wc -l` Used Assets$NC

}

This program depends on include.inc which contains the following:

#!/bin/bash

#use colour in the output just for fun
GREEN=$(printf “\033[32m”)
RED=$(printf “\033[31m”)
NC=$(printf “\033[0m”)

error()
{
echo ERROR: $1
exit 1
}

I tried to make these scripts as tidy and portable as possible — you will, however, need to change the path for $SCRIPTDIR to the location you are using (and $TMPDIR if you don’t want to put your temporary files in /tmp).

Test File Paths in a Cinelerra XML File

This is a bash script to test all the filenames and paths in a Cinelerra project to make sure they are still valid.

#!/bin/bash
#kk test_xml_paths.sh
#kk
#kk check a cinelerra xml file to make sure the file references are all valid
#kk provide filename to check as parameter

if [ -z "$1" -o "$1" = "-h" ]
then
echo “Usage: test_xml_paths [xml_filename]“
echo “For cinelerra xml files, this script checks that file paths are valid”
exit
fi

SCRIPTDIR=”$HOME/Video/video_new/common/scripts”
TMPDIR=”/tmp”

. $SCRIPTDIR/include.inc

#assign the parameter to a variable
XMLFILE=$1

#check if the xml file exists and exit if it does not
if [ ! -e $XMLFILE ]; then error “XML file does not exist”; fi

#location of the temporary file
FILENAMES=”$TMPDIR/TEST_XML_PATHS”

#set counters to zero
GOODCOUNT=0
BADCOUNT=0

#if the temp file exists, delete it
if [ -e $FILENAMES ]; then
rm $FILENAMES
fi

echo
echo Checking $XMLFILE for valid file paths…
echo

#find all of the filenames in the xml file and strip the extra junk
#send the output to a file
grep -E “SRC=|PATH=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/^.*PATH=”//g’ -e ‘s/”>.*$//g’ > $FILENAMES
if [ $? != 0 ] ; then error “grep failed”; fi

# loop through the output and test if the filename is either a link or a file
for f in $( cat $FILENAMES ); do
if [ ! -h $f -a ! -e $f ]; then
BADCOUNT=`expr $BADCOUNT + 1`
echo ERROR: Missing $f
else
GOODCOUNT=`expr $GOODCOUNT + 1`
fi
done

#output the counts
echo
echo $RED$BADCOUNT Total Bad Pathss $NC
echo $GREEN$GOODCOUNT Total Good Paths$NC
echo

This program depends on include.inc which contains the following:

#!/bin/bash

#use colour in the output just for fun
GREEN=$(printf “\033[32m”)
RED=$(printf “\033[31m”)
NC=$(printf “\033[0m”)

error()
{
echo ERROR: $1
exit 1
}

I tried to make these scripts as tidy and portable as possible — you will, however, need to change the path for $SCRIPTDIR to the location you are using (and $TMPDIR if you don’t want to put your temporary files in /tmp).

Reinstalling A Corrupted Ubuntu Karmic Upgrade

I have previously posted about my own Karmic upgrade problems. In my case, the system crashed in the middle of the upgrade (due to my own fault, nothing to do with the upgrade itself as far as I know), causing my half-upgraded system to be unable to mount the root filesystem and dropping me into a maintenance shell.

I’ve always tried to promote Linux use among friends and family which means I also provide support as required. Yesterday, a friend of mine brought me her laptop with a similar problem to the one I had: in her case, she thinks that she lost her internet connection in the middle of the Jaunty to Karmic upgrade. In any case, her half-upgraded system would neither boot nor allow a root prompt. The error message on the console was:

mountall: symbol lookup error: mountall: undefined symbol: udev_monitor_filter_add_match_subsystem_devtype
init: mountall main process (310) terminated with status 127

I tried a couple of things: first, I used an old Jaunty disk to boot the machine, mounted the old root filesystem and attempted to use chroot so that I could complete the upgrade.

mkdir /media/disk
mount /dev/sda5 /media/disk
chroot /media/disk

Unfortunately, this didn’t work because I had no network devices available in the chroot jail, and I couldn’t find any obvious way to sort that out. Similarly, I downloaded a new iso of Ubuntu Karmic hoping to use it as the apt source for the upgrade, but I couldn’t access the cd drive either, even using apt-cdrom to set it up. As before, a bit more effort might have sorted this out, but I decided to solve the problem by doing a complete re-install of Karmic over top of the previous one.

I backed up the /etc and /home directories to an external hard drive first to be safe. To my delight, the reinstall worked fine and didn’t clobber any of the existing user data on the drive. During the partioning step, I selected the manual option. I changed /dev/sda5 (which in this case was the old Ubuntu root filesystem) to mount as “/” but didn’t check the box for it to be formatted. In the “import settings” step, I also told the installer to import user settings from the old Ubuntu install. When I rebooted, everything was up to date, and the user settings and documents were all in place as expected. Good to know!

As a final note, I should add that when I set up this system for my friend, I configured Dropbox and told her how to use it. That meant that I could’ve easily restored everything even if I’d had to do a complete format and reinstall. On my own machine, I’ve placed a symbolic link to /etc in my Dropbox folder (it doesn’t take up much space) along with linked versions of all of my important dot files — that means that, with a little effort, I can reimage and restore at any time. I will set up all my friends the same way from now on.

Logging out with the Cairo dock

I’ve decided to take advantage of my new powerful laptop and enable a little eye candy. I’ve tweaked Compiz to my liking and enabled the Cairo dock (I used the Avant Window Navigator for a couple of days and liked it as well, but Cairo is even better). I’ve been able to set up my Cairo dock to give me everything I appreciated in the old gnome panels with one exception: the user switcher applet. Cairo gives you a shut-down menu, but no way to log out of your session or switch to another user.

I have discovered the command “gdmXnestchooser” which at least lets me switch to the face browser and log in as another user, but I’d like to be able to log out of my current session sometimes in order to refresh things. I have also enabled restarting the X server (similar results to logging out) as described here. Essentially, you must go to the preferences/keyboard/layout and enable the ctrl-alt-backspace key combination. Another tool, “gdmXnestchooser” allows you to create a nested X session, but you need to install gdm-2.20 to make it work and I don’t want to risk buggering up my current setup.

One thing I love about the combination of Compiz and Cairo is the ability to dump all of my widgets (weather, clock, conky, system tray etc.) onto the compiz widget layer. I have it configured to pop up when I move the mouse to the top edge of the screen so I can instantly see the status of everything without having to use precious screen real estate or flip back to the desktop. I have the dock on auto-hide, so I can use the full screen without sacrificing usability. It is an elegant solution to the problem of trying to use a single display for many simultaneous purposes.

UPDATE: Turns out it’s easy to log out from the Cairo dock…see comment below.

Ubuntu Karmic Koala Upgrade

I have upgraded from Jaunty to Karmic on my Dell XPS 1610 and everything seems to work great.

My upgrade was interrupted (my own fault for running another application at the same time and having it crash) and caused some serious problems for a few hours while I tried to fix it. The symptom was that the system wouldn’t boot and told me that it couldn’t read from the root filesystem. In the end I solved the problem by dropping to a shell, re-mounting the root disk so I could write to it (“mount -o remount,rw /”), then running apt-get and sorting out dependency problems with python2.6 and mono-common (I finally had to delete /var/lib/binfmts/python2.5 in order for python to install properly). This process took hours and finally I was able to perform a full “apt-get dist-upgrade” from the shell, then reboot. I still had a disk error which prevented booting, but I discovered a great tip someplace to force a disk check on the root filesystem on reboot with “tune2fs -C 30 /dev/sda5″ (or whatever device).

The experience reminded me that Linux is great for people like me, but maybe not so great for others. It isn’t that Windows doesn’t have similar problems (like I said, it was my fault for crashing the system in the middle of a major upgrade), but when it does, a less-savvy user can take the machine to the corner computer shop to get it fixed.

In spite of that, Karmic has lots of great features that make it perfect for any user. I’m still noticing small changes, but so far, so good. I like the new face browser login window and I’m thrilled with the new audio control panel, which is simpler and clearer than before with new features — for example, you can now see which active applications are using the sound server and turn them on and off with a single click.

Display Video Thumbnails in DigiKam

Digikam is working great for me and I’m diligently tagging all my old photos. I was disappointed that, although DigiKam could play them, it wouldn’t display thumbnails for video files in my albums. Turns out that I just needed to install the mplayerthumbnails package and now I have thumbnails. In Ubuntu Jaunty, ignore instructions which suggest adding libarts1-xine and libxine-extracodecs — neither package is available in Jaunty.

Virtualbox Setup to Run Windows XP Inside Ubuntu Jaunty

I had no trouble with the initial virtualbox-3.0 install from Sun’s repositories. I popped in my old Windows XP Home Edition Service Pack 1a CD and installed the software on a virtual disk — all of this was easy to do and required no special tricks.

I did learn something important, however: I tried to install SP2 of XP and kept getting a blue screen on reboot. I tried every combination of updates and rollbacks I could think of, but could not get SP2 to install. I wouldn’t have bothered, since my Epson scanner software runs fine on SP1a, but I couldn’t install my TV tuner drivers without upgrading.

After a lot of experimenting, I realized that I had to start over and wait to install the VirtualBox Guest Additions until AFTER installing all the XP updates and Service Packs. Everything seems to be fine now.