List Unused Assets in a Cinelerra XML Project File
Nov 30th
This is a bash script which contains a function used to figure out which assets in a Cinelerra project are not required for the edit list. I created it because I had pulled in a couple of thousand media resources for a project and wanted to eliminate bloat. I moved the project and found it kept crashing on me until I eliminated the crud. I wrote another program to call this function and then actually clean out the xml. You can view “remove_unused_assets.sh” here.
Running this script doesn’t do anything until you call the enclosed function. This function can be called from the command line with:
#kk list_unused_assets.sh
#kk
#kk check a cinelerra xml file to see which assets have been loaded, but not used in the project
#kk provide filename to check as parameter
#kk from the command line, type “. list_unused_assets.sh;list_unused_assets
list_unused_assets()
{
SCRIPTDIR=”$HOME/Video/video_new/common/scripts”
TMPDIR=”/tmp”
. $SCRIPTDIR/include.inc
#location of the temporary files
ASSETS_FILE=”$TMPDIR/ASSETS_FILE”
EDITS_FILE=”$TMPDIR/EDITS_FILE”
#assign the parameter to a variable
XMLFILE=$1
#check if the xml file exists and exit if it does not
if [ ! -e $XMLFILE ]; then error “XML file does not exist”; fi
#if the temp file exists, delete it
if [ -e $ASSETS_FILE ]; then
rm $ASSETS_FILE
fi
if [ -e $EDITS_FILE ]; then
rm $EDITS_FILE
fi
echo
echo Checking $XMLFILE for assets unused by the project…
echo
#find all of the filenames in the xml file and strip the extra junk
grep -E “ASSET SRC=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/”>.*$//g’ | sort | uniq > $ASSETS_FILE
grep -E “FILE SRC=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/”>.*$//g’ | sort | uniq > $EDITS_FILE
#output the counts
echo `grep -E “ASSET SRC=|FILE SRC=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/”>.*$//g’ | sort | uniq | wc -l` Total Assets
echo $RED`comm -23 $ASSETS_FILE $EDITS_FILE | wc -l` Unused Assets$NC
echo $GREEN`comm -12 $ASSETS_FILE $EDITS_FILE | wc -l` Used Assets$NC
}
This program depends on include.inc which contains the following:
#use colour in the output just for fun
GREEN=$(printf “\033[32m”)
RED=$(printf “\033[31m”)
NC=$(printf “\033[0m”)
error()
{
echo ERROR: $1
exit 1
}
I tried to make these scripts as tidy and portable as possible — you will, however, need to change the path for $SCRIPTDIR to the location you are using (and $TMPDIR if you don’t want to put your temporary files in /tmp).
Test File Paths in a Cinelerra XML File
Nov 30th
This is a bash script to test all the filenames and paths in a Cinelerra project to make sure they are still valid.
#kk test_xml_paths.sh
#kk
#kk check a cinelerra xml file to make sure the file references are all valid
#kk provide filename to check as parameter
if [ -z "$1" -o "$1" = "-h" ]
then
echo “Usage: test_xml_paths [xml_filename]“
echo “For cinelerra xml files, this script checks that file paths are valid”
exit
fi
SCRIPTDIR=”$HOME/Video/video_new/common/scripts”
TMPDIR=”/tmp”
. $SCRIPTDIR/include.inc
#assign the parameter to a variable
XMLFILE=$1
#check if the xml file exists and exit if it does not
if [ ! -e $XMLFILE ]; then error “XML file does not exist”; fi
#location of the temporary file
FILENAMES=”$TMPDIR/TEST_XML_PATHS”
#set counters to zero
GOODCOUNT=0
BADCOUNT=0
#if the temp file exists, delete it
if [ -e $FILENAMES ]; then
rm $FILENAMES
fi
echo
echo Checking $XMLFILE for valid file paths…
echo
#find all of the filenames in the xml file and strip the extra junk
#send the output to a file
grep -E “SRC=|PATH=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/^.*PATH=”//g’ -e ‘s/”>.*$//g’ > $FILENAMES
if [ $? != 0 ] ; then error “grep failed”; fi
# loop through the output and test if the filename is either a link or a file
for f in $( cat $FILENAMES ); do
if [ ! -h $f -a ! -e $f ]; then
BADCOUNT=`expr $BADCOUNT + 1`
echo ERROR: Missing $f
else
GOODCOUNT=`expr $GOODCOUNT + 1`
fi
done
#output the counts
echo
echo $RED$BADCOUNT Total Bad Pathss $NC
echo $GREEN$GOODCOUNT Total Good Paths$NC
echo
This program depends on include.inc which contains the following:
#use colour in the output just for fun
GREEN=$(printf “\033[32m”)
RED=$(printf “\033[31m”)
NC=$(printf “\033[0m”)
error()
{
echo ERROR: $1
exit 1
}
I tried to make these scripts as tidy and portable as possible — you will, however, need to change the path for $SCRIPTDIR to the location you are using (and $TMPDIR if you don’t want to put your temporary files in /tmp).
Creating Subtitles/Captions for a Video Project
Nov 25th
I’m working on a quick 20 minute video for a non-profit group (and I do mean quick — less than two weeks from initial contact to the premiere). Everything has come together pretty nicely and I’ve finished the editing to my satisfaction in Cinelerra. Because some of the interviewees in the video have impaired speech, we’ve decided that it would be good to add subtitles in English to help viewers to follow the audio. Since we don’t want to stigmatize the people with the speech difficulties, I’m subtitling the whole video. This is fine since it isn’t too long. In any case, it also has the benefit of making the video accessible to the hard of hearing.
Rather than use Cinelerra’s title effect to add the captions, I decided to teach myself how to do it the “proper” way with a separate subtitle file.
Transcribing the Subtitles
Surprisingly, it isn’t completely obvious how and when to place the subtitles in the video. How long should they appear on screen? Should they be synchronized precisely with the video? How do you punctuate people’s verbal stream-of-consciousness ramblings? Do you include the speaker’s pauses and “um”s and so on? What do you do about words you simply cannot decipher? There’s lots of technical info out there on closed captioning, but not much in the way of tips for the beginner who is creating do-it-yourself video. The best synopsis I could find was a quote from this thread about live streaming closed captioning:
The real keys are timing and readability. Captioning is not just typing what the person is saying. You also decide when the caption appears and disappears. Tying a new caption to a shot change or any sort of movement that indicates that a person is about to speak is a good idea. The idea is that the caption itself should work with the rest of the visuals and not be too distracting. Don’t let any captions be too short or too long either. Right around two seconds a piece is a good rule of thumb. As far as readability, be wary of where you break a sentence if a caption is more than one line long. Again, the idea here is that the captions should flow.
Based on these thoughts, plus some common sense, I’ve devised the following informal guidelines for my project:
1) Try to transcribe precisely all of the actual words. Indicate long pauses by creating a new caption, or with ellipses or whatever punctuation seems appropriate. Ignore interjected particles such as “uh” or “um” unless they seem to be essential to the meaning of the utterance.
2) Synchronize the caption to start and end at the same time as the audio, unless the audio is less than two seconds long, in which case let the caption linger for two seconds.
3) Try to break captions on cuts in the video where possible.
4) Watch out for dramatic conclusions to sentences — keep the parts synchronized separately, so you don’t give away the ending in advance.
5) When a speaker is talking at length, use two line captions as required and try to break on sentence clauses or on pauses in the speech. Shorter is usually better.
Creating the Subtitle File
The technical process of creating subtitles is fairly easy, if tedious. I used the subtitleeditor application (tips here and here and here) which can be easily installed in Ubuntu to transcribe most of the titles. Unfortunately, I found it frustrating that the application sometimes seemed to arbitrarily ignore what I had typed, forcing me to re-enter all or most of many subtitles. Just for this step, I switched to the “Gaupol Subtitle Editor” which worked much the same way and didn’t seem to have the same bug. Next time, I’d investigate using a spreadsheet and then importing the data into the subtitle editor program. I found that transcribing and synchronizing had to be done as two separate steps anyhow.
For the synchronizing, I found the easiest method was to go back to the first Subtitle Editor app, use it to generate a waveform from the audio track of my video, then use the mouse and keyboard to set the correct start and end points for each subtitle. I set the “play/pause” shortcut to Super (Windows key) plus space, “short” (skip backward) to Super-left arrow, and “short” (skip forward) to Super-right arrow (yes, there are two keyboard shortcuts with the same name, but you can tell which is which by clicking on them in the shortcuts preferences). The default is for left-mouse-click to set the start point for the selected subtitle line, and right-mouse-click to set the end point. Middle-click restarts the playback at the point in the wave timeline where you clicked. Since I already had all the words transcribed, I just had to run through the audio track, listening and watching the waveform graph, using the shortcuts to move around, and click to select the right in and out points for each line. I found I had to split some of the subtitles, but that was also easy using the Subtitle Editor tool. Overall, synchronization went pretty smoothly.
There are many different subtitle formats, but the default format worked fine for me.
Combining the Subtitles with the Video
According to the Cinelerra manual, there are three obvious choices for combining subtitles with your video:
# Distribute it with your video. People will have to load the appropriate subtitle file in their video player to actually see the subtitles.
# Use it with dvdauthor, to add the subtitles in a DVD. Read dvdauthor’s documentation for more information.
# Incrust the subtitles into the video using mencoder.
I used the second method, which seems to be the most flexible option, but also the most complicated. In my case, I wanted to force the subtitles to appear rather than allowing the user to select whether to play them. Dvdauthor provides a tool called spumux, which allows you to add a subtitle track to the video. Then, you use dvdauthor (as you normally would) to build the dvd filesystem. From there, the process is the same as for burning a non-subtitled video (I’ll write up my own process at some point, but there’s already a very good explanation from Crazed Mule: look here for exporting from Cinelerra and look here for creating a DVD). You’ll need to create an xml file for spumux and another for dvdauthor.
Here is my subtitle xml file for the first step:
<stream>
<textsub filename=”/home/kevin/Video/video_ykacl/project_xml/en_subtitles.sub” characterset=”UTF-8″
fontsize=”32.0″ font=”Arial.ttf”
horizontal-alignment=”center”
vertical-alignment=”bottom” left-margin=”60″
right-margin=”60″
top-margin=”20″ bottom-margin=”30″ subtitle-fps=”29.97″
movie-fps=”29.97″ movie-width=”720″ movie-height=”480″
force=”yes”/>
</stream>
</subpictures>
Note that for spumux to work, you MUST have copy or symbolic link to the truetype font file (in this case, Arial.ttf, but use whatever you like) in the .spumux folder in your home directory. I did it by running this command:
The above command assumes you have the msttcorefonts package installed. Also note the force=”yes” option, which tells the dvd player to always display these subtitles.
Once you’ve created the xml, you run spumux to combine your subtitle file with your video:
I put this command in a script in which I defined $VIDEO_PROJECT and $VIDEO_TMP — you can subsitute whatever path is appropriate.
Now you need to create another xml file for dvdauthor to use when building the dvd menu and filesystem (in my case the video is supposed to play automatically, so there is no actual menu):
<vmgm />
<titleset>
<titles>
<subpicture lang=”en” />
<pgc>
<pre> subtitle=64; </pre>
<vob file=”/home/kevin/Video/video_tmp/video_ykacl_tmp/full_render_subtitle.mpg” />
</pgc>
</titles>
</titleset>
</dvdauthor>
The most important option in this file is the subtitle=64, which tells the dvd player to automatically display subtitle track 0 (see the links below for details on this). Now you just run dvdauthor:
At this point you should have a working dvd filesystem ready to be burned onto an actual dvd. Before burning, you can test your output by running
More Information About Subtitles
Explanation of how to add subtitles in dvdauthor is here.
Useful details on spumux is here.
Good examples here.
Reinstalling A Corrupted Ubuntu Karmic Upgrade
Nov 25th
I have previously posted about my own Karmic upgrade problems. In my case, the system crashed in the middle of the upgrade (due to my own fault, nothing to do with the upgrade itself as far as I know), causing my half-upgraded system to be unable to mount the root filesystem and dropping me into a maintenance shell.
I’ve always tried to promote Linux use among friends and family which means I also provide support as required. Yesterday, a friend of mine brought me her laptop with a similar problem to the one I had: in her case, she thinks that she lost her internet connection in the middle of the Jaunty to Karmic upgrade. In any case, her half-upgraded system would neither boot nor allow a root prompt. The error message on the console was:
mountall: symbol lookup error: mountall: undefined symbol: udev_monitor_filter_add_match_subsystem_devtype
init: mountall main process (310) terminated with status 127
I tried a couple of things: first, I used an old Jaunty disk to boot the machine, mounted the old root filesystem and attempted to use chroot so that I could complete the upgrade.
mount /dev/sda5 /media/disk
chroot /media/disk
Unfortunately, this didn’t work because I had no network devices available in the chroot jail, and I couldn’t find any obvious way to sort that out. Similarly, I downloaded a new iso of Ubuntu Karmic hoping to use it as the apt source for the upgrade, but I couldn’t access the cd drive either, even using apt-cdrom to set it up. As before, a bit more effort might have sorted this out, but I decided to solve the problem by doing a complete re-install of Karmic over top of the previous one.
I backed up the /etc and /home directories to an external hard drive first to be safe. To my delight, the reinstall worked fine and didn’t clobber any of the existing user data on the drive. During the partioning step, I selected the manual option. I changed /dev/sda5 (which in this case was the old Ubuntu root filesystem) to mount as “/” but didn’t check the box for it to be formatted. In the “import settings” step, I also told the installer to import user settings from the old Ubuntu install. When I rebooted, everything was up to date, and the user settings and documents were all in place as expected. Good to know!
As a final note, I should add that when I set up this system for my friend, I configured Dropbox and told her how to use it. That meant that I could’ve easily restored everything even if I’d had to do a complete format and reinstall. On my own machine, I’ve placed a symbolic link to /etc in my Dropbox folder (it doesn’t take up much space) along with linked versions of all of my important dot files — that means that, with a little effort, I can reimage and restore at any time. I will set up all my friends the same way from now on.
Looking at Video File Metadata
Nov 16th
Three choices as per the details here:
Info on file types here.
I’m trying to understand video formats to make it easier to transcode for various purposes. I’ve found some useful links:
adamwilt.com > the DV, DVCAM & DVCPRO Formats
Logging out with the Cairo dock
Nov 15th
I’ve decided to take advantage of my new powerful laptop and enable a little eye candy. I’ve tweaked Compiz to my liking and enabled the Cairo dock (I used the Avant Window Navigator for a couple of days and liked it as well, but Cairo is even better). I’ve been able to set up my Cairo dock to give me everything I appreciated in the old gnome panels with one exception: the user switcher applet. Cairo gives you a shut-down menu, but no way to log out of your session or switch to another user.
I have discovered the command “gdmXnestchooser” which at least lets me switch to the face browser and log in as another user, but I’d like to be able to log out of my current session sometimes in order to refresh things. I have also enabled restarting the X server (similar results to logging out) as described here. Essentially, you must go to the preferences/keyboard/layout and enable the ctrl-alt-backspace key combination. Another tool, “gdmXnestchooser” allows you to create a nested X session, but you need to install gdm-2.20 to make it work and I don’t want to risk buggering up my current setup.
One thing I love about the combination of Compiz and Cairo is the ability to dump all of my widgets (weather, clock, conky, system tray etc.) onto the compiz widget layer. I have it configured to pop up when I move the mouse to the top edge of the screen so I can instantly see the status of everything without having to use precious screen real estate or flip back to the desktop. I have the dock on auto-hide, so I can use the full screen without sacrificing usability. It is an elegant solution to the problem of trying to use a single display for many simultaneous purposes.
UPDATE: Turns out it’s easy to log out from the Cairo dock…see comment below.
Ubuntu Karmic Koala Upgrade
Nov 15th
I have upgraded from Jaunty to Karmic on my Dell XPS 1610 and everything seems to work great.
My upgrade was interrupted (my own fault for running another application at the same time and having it crash) and caused some serious problems for a few hours while I tried to fix it. The symptom was that the system wouldn’t boot and told me that it couldn’t read from the root filesystem. In the end I solved the problem by dropping to a shell, re-mounting the root disk so I could write to it (“mount -o remount,rw /”), then running apt-get and sorting out dependency problems with python2.6 and mono-common (I finally had to delete /var/lib/binfmts/python2.5 in order for python to install properly). This process took hours and finally I was able to perform a full “apt-get dist-upgrade” from the shell, then reboot. I still had a disk error which prevented booting, but I discovered a great tip someplace to force a disk check on the root filesystem on reboot with “tune2fs -C 30 /dev/sda5″ (or whatever device).
The experience reminded me that Linux is great for people like me, but maybe not so great for others. It isn’t that Windows doesn’t have similar problems (like I said, it was my fault for crashing the system in the middle of a major upgrade), but when it does, a less-savvy user can take the machine to the corner computer shop to get it fixed.
In spite of that, Karmic has lots of great features that make it perfect for any user. I’m still noticing small changes, but so far, so good. I like the new face browser login window and I’m thrilled with the new audio control panel, which is simpler and clearer than before with new features — for example, you can now see which active applications are using the sound server and turn them on and off with a single click.
Cinelerra Audio Sync Problem on Ubuntu Jaunty
Nov 13th
Everytime I try to set up Cinelerra on a new project or on a new system configuration, it takes a while to get everything working well. In my case, I have several new variables to contend with: I’m starting with HDV footage from a new camera, I have a new machine, I’m using a new version of Ubuntu and of Cinelerra.
This time around, I’ve had trouble with audio sync during playback. The symptom is that when I play a dv clip (which is fine on its own in VLC) in the Compositor Window in Cinelerra, I get a 10 second audio time lag which seems impossible to correct by nudging or setting the audio offset value.
Fortunately, this doesn’t seem to affect rendering, but it still makes it tough to edit properly. After a great deal of fiddling, I finally discovered (via a note here) that I could use the Pulseaudio sound driver in Ubuntu (select “Esound” and port 7007 in the Cinelerra Playback settings) instead of ALSA. Suddenly, my project sounded much better. The audio track was still out slightly, but I used the technique described in Secrets of Cinelerra to sync the sound using the nudge value of the audio tracks. There is additional helpful info on audio syncing here.
I was surprised that this simple matter took so much searching to solve. Here’s hoping that this post will pop up in future searches.
To summarize, if you are running Cinelerra on Ubuntu and have audio sync problems, try switching your audio driver to “esound” on port 7007 and you may be pleasantly surprised.
UPDATE: This is very weird, but the above config works fine when I run Cinelerra as a user, but NOT when I run it as root! Obviously, I’ve got the PulseAudio settings messed up somewhere, but at least it works.
Capturing Old VHS Tapes
Nov 13th
Yes, I’m digitizing everything — it’s like all of my belongings are slowly being vacuumed up into the computer.
This time it is VHS tape I’m working on. I’m not concerned about commercially recorded tapes — I got rid of all of those a while ago. Rather, I want to capture the tapes we have which are irreplaceable. In our case that is mostly videos which feature people in our family that we recorded from TV or other sources.
I figured the simplest method would be to feed the video out from a VCR to the AV IN on one of my cameras, then sent the output via firewire to my computer for capture using dvgrab. Turns out it worked like a charm.
I found a great explanation of the process here.
I used my old Panasonic PV-GS700 which no longer records properly — it works just fine to convert the analog signal to digital output. I had to switch the camera’s “DV OUT” setting to “ON” and it was ready to go. Here is the dvgrab command I used to capture the video (where “foo” is the base output filename):
Note: if you are keeping an eye on it, you can stop the capture at the actual end of the tape by pressing ctrl-c and the output file will still be fine. You don’t have to wait until the 2:10 mark — that is just for convenience so the capture doesn’t continue to run indefinitely.
As per the instructions in the link, the following command is a handy way to trim off the excess from the end of each file (where the time element is the place you want to end the clip):