Posts tagged cinelerra

Canon Vixia HV40 HDV Format Notes

So far I have always converted my HDV footage to NTSC before editing. For a small project I’m working on now, I’ve decided to try working directly with the raw HDV footage.

This thread explains why my m2t files show up as 1440×1080 even though they can be edited as 1920×1080 (aka 1080i). It has to do with square vs. rectangular pixels — or something — it doesn’t really matter. More general info on formats is here.

In Cinelerra, you must set the project format to 1440×1080, 16:9 ratio, then make sure you select “match output size” by right-clicking on the video track(s). This looks okay in the compositor window in Cinelerra, but the resulting file plays by default at 4:3 aspect ratio. You can set this manually when you open the video in VLC and see that everything looks okay. The only method I have found so far for getting the rendered video to play as 16:9 by default is by using a command such as this on to re-encode it after the fact:

ffmpeg -i full_render_test.mov -aspect 16:9 -sameq full_render_test_16-9.mov

For rendering in various formats, follow the instructions here as usual.

Scripts for Monkeying with Cinelerra XML

I’ve moved my Cinelerra projects from one directory path to another and scrambled a few things in the process. I’ve written a few bash scripts which helped me sort things out.

In the process, I’ve learned a bit about the xml format which Cinelerra uses to store references to files and edits. When you add a resource, it appears twice between opening and closing “ASSET” tags with the format <ASSET SRC=”/path/to/filename”>[asset details]</ASSET>. Then every time you include that file in a track of your project, the filename appears again in an “EDIT” tag complex in the format <FILE SRC=”/path/to/filename”>[edit details]</FILE>

First, I needed to check the paths for all the resources in the file to make sure they are still valid. I called this program test_xml_paths.sh. I corrected the incorrect paths with a text editor, but found that Cinelerra was still crashing with a memory error when I loaded the project.

Second, I decided to find and eliminate all of the unused assets in the file. I originally had a forty minute video in one file, but some time ago I split the project into four ten minute chunks. When I created the original project, I had added a couple of thousands of clips for possible inclusion, so now that I’m working on the final cut, I don’t need all that bloat.

I created a function to figure out which assets were being used in the edit list and which were not: list_unused_assets.sh.

Finally, I wrote a script which would recreate a new xml project file with all the redundant assets removed: remove_unused_assets.sh.

I can’t offer any warranty on these scripts, of course, but I hope they may help somebody else. If nothing else, there’s a great example of multi-line sed editing in the last one.

Remove Unused Assets from a Cinelerra XML Project File

This is a bash script to remove all the unwanted assets from a Cinelerra project XML file. I needed it to cut out bloat from a project I was working on which had hundreds of unused resources after I split the project up into ten minute chunks. Doing so stopped Cinelerra from crashing when I loaded the project: YMMV.

Save this script as “remove_unused_assets.sh”. It needs the function in “list_unused_assets.sh” which I posted here.

#! /bin/bash
#kk remove_unused_assets.sh
#kk
#kk For cinelerra xml files
#kk this script removes the contents of asset tags which are not also used for edits

if [ -z "$1" -o "$1" = "-h" ]
then
echo “Usage: remove_unused_assets [xml_filename]“
echo “For cinelerra xml files, this script removes the contents of asset tags which are not being used for edits”
exit
fi

TMPDIR=”/tmp”
SCRIPTDIR=”$HOME/Video/video_new/common/scripts”

. $SCRIPTDIR/list_unused_assets.sh
. $SCRIPTDIR/include.inc

#assign the parameter to a variable
XMLFILE=$1

#location of the temporary files
ASSETS_FILE=”$TMPDIR/ASSETS_FILE”
EDITS_FILE=”$TMPDIR/EDITS_FILE”
XMLFILE_NEW=”$1″_new
XMLFILE_TMP=”$TMPDIR/XMLFILE_TMP”

#check if the xml file exists and exit if it does not
if [ ! -e $XMLFILE ]; then error “XML file does not exist”; fi

list_unused_assets $XMLFILE
if [ $? != 0 ] ; then error “list_unused_assets failed”; fi

cp $XMLFILE $XMLFILE_TMP

echo
echo Removing Unused Assets…
echo
echo “Working…”
echo

# loop through the list of unused assets and remove the ASSET tag lines for each
# sed command from here: http://ilfilosofo.com/blog/2008/04/26/sed-multi-line-search-and-replace/

for f in $( comm -23 $ASSETS_FILE $EDITS_FILE ); do

#to make the filename string work in sed, I have to escape all of the periods and slashes
FORMATTED_f=`echo “$f” | sed -e ‘s:\/:\\\/:g’ -e ‘s:\.:\\\.:g’`

#this complicated sed script came from the sed FAQ: http://sed.sourceforge.net/sedfaq4.html#s4.21
#in order to make the variable substitution work, I had to close the sed script with a single quote
#then enclose the variable in double quotes, then re-open the sed script with a single quote
#the -i option is for editing a file “in place”
sed -i -e ‘
# sed script to delete a block if /regex/ matches inside it
:Top
/\/ { # For each line between these block markers..
/\/ASSET\>/!{ # If we are not at the /end/ marker
$!{ # nor the last line of the file,
N; # add the Next line to the pattern space
b Top
} # and branch (loop back) to the :Top label.
} # This line matches the /end/ marker.
/’”$FORMATTED_f”‘/d; # If /regex/ matches, delete the block.
} # Otherwise, the block will be printed.
#—end of script—
‘ $XMLFILE_TMP

#diff $XMLFILE $XMLFILE_TMP

echo -n ‘ \r’
echo -n $f’\r’

done

cp $XMLFILE_TMP $XMLFILE_NEW

echo
list_unused_assets $XMLFILE_NEW

echo
echo Finished.
echo
echo The new version of the xml file has been saved as $XMLFILE_NEW.
echo

This program depends on include.inc which contains the following:

#!/bin/bash

#use colour in the output just for fun
GREEN=$(printf “\033[32m”)
RED=$(printf “\033[31m”)
NC=$(printf “\033[0m”)

error()
{
echo ERROR: $1
exit 1
}

I tried to make these scripts as tidy and portable as possible — you will, however, need to change the path for $SCRIPTDIR to the location you are using (and $TMPDIR if you don’t want to put your temporary files in /tmp).

List Unused Assets in a Cinelerra XML Project File

This is a bash script which contains a function used to figure out which assets in a Cinelerra project are not required for the edit list. I created it because I had pulled in a couple of thousand media resources for a project and wanted to eliminate bloat. I moved the project and found it kept crashing on me until I eliminated the crud. I wrote another program to call this function and then actually clean out the xml. You can view “remove_unused_assets.sh” here.

Running this script doesn’t do anything until you call the enclosed function. This function can be called from the command line with:

. list_unused_assets.sh;list_unused_assets
#! /bin/bash
#kk list_unused_assets.sh
#kk
#kk check a cinelerra xml file to see which assets have been loaded, but not used in the project
#kk provide filename to check as parameter
#kk from the command line, type “. list_unused_assets.sh;list_unused_assets

list_unused_assets()
{

SCRIPTDIR=”$HOME/Video/video_new/common/scripts”
TMPDIR=”/tmp”

. $SCRIPTDIR/include.inc

#location of the temporary files
ASSETS_FILE=”$TMPDIR/ASSETS_FILE”
EDITS_FILE=”$TMPDIR/EDITS_FILE”

#assign the parameter to a variable
XMLFILE=$1

#check if the xml file exists and exit if it does not
if [ ! -e $XMLFILE ]; then error “XML file does not exist”; fi

#if the temp file exists, delete it
if [ -e $ASSETS_FILE ]; then
rm $ASSETS_FILE
fi

if [ -e $EDITS_FILE ]; then
rm $EDITS_FILE
fi

echo
echo Checking $XMLFILE for assets unused by the project…
echo

#find all of the filenames in the xml file and strip the extra junk
grep -E “ASSET SRC=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/”>.*$//g’ | sort | uniq > $ASSETS_FILE
grep -E “FILE SRC=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/”>.*$//g’ | sort | uniq > $EDITS_FILE

#output the counts
echo `grep -E “ASSET SRC=|FILE SRC=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/”>.*$//g’ | sort | uniq | wc -l` Total Assets
echo $RED`comm -23 $ASSETS_FILE $EDITS_FILE | wc -l` Unused Assets$NC
echo $GREEN`comm -12 $ASSETS_FILE $EDITS_FILE | wc -l` Used Assets$NC

}

This program depends on include.inc which contains the following:

#!/bin/bash

#use colour in the output just for fun
GREEN=$(printf “\033[32m”)
RED=$(printf “\033[31m”)
NC=$(printf “\033[0m”)

error()
{
echo ERROR: $1
exit 1
}

I tried to make these scripts as tidy and portable as possible — you will, however, need to change the path for $SCRIPTDIR to the location you are using (and $TMPDIR if you don’t want to put your temporary files in /tmp).

Test File Paths in a Cinelerra XML File

This is a bash script to test all the filenames and paths in a Cinelerra project to make sure they are still valid.

#!/bin/bash
#kk test_xml_paths.sh
#kk
#kk check a cinelerra xml file to make sure the file references are all valid
#kk provide filename to check as parameter

if [ -z "$1" -o "$1" = "-h" ]
then
echo “Usage: test_xml_paths [xml_filename]“
echo “For cinelerra xml files, this script checks that file paths are valid”
exit
fi

SCRIPTDIR=”$HOME/Video/video_new/common/scripts”
TMPDIR=”/tmp”

. $SCRIPTDIR/include.inc

#assign the parameter to a variable
XMLFILE=$1

#check if the xml file exists and exit if it does not
if [ ! -e $XMLFILE ]; then error “XML file does not exist”; fi

#location of the temporary file
FILENAMES=”$TMPDIR/TEST_XML_PATHS”

#set counters to zero
GOODCOUNT=0
BADCOUNT=0

#if the temp file exists, delete it
if [ -e $FILENAMES ]; then
rm $FILENAMES
fi

echo
echo Checking $XMLFILE for valid file paths…
echo

#find all of the filenames in the xml file and strip the extra junk
#send the output to a file
grep -E “SRC=|PATH=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/^.*PATH=”//g’ -e ‘s/”>.*$//g’ > $FILENAMES
if [ $? != 0 ] ; then error “grep failed”; fi

# loop through the output and test if the filename is either a link or a file
for f in $( cat $FILENAMES ); do
if [ ! -h $f -a ! -e $f ]; then
BADCOUNT=`expr $BADCOUNT + 1`
echo ERROR: Missing $f
else
GOODCOUNT=`expr $GOODCOUNT + 1`
fi
done

#output the counts
echo
echo $RED$BADCOUNT Total Bad Pathss $NC
echo $GREEN$GOODCOUNT Total Good Paths$NC
echo

This program depends on include.inc which contains the following:

#!/bin/bash

#use colour in the output just for fun
GREEN=$(printf “\033[32m”)
RED=$(printf “\033[31m”)
NC=$(printf “\033[0m”)

error()
{
echo ERROR: $1
exit 1
}

I tried to make these scripts as tidy and portable as possible — you will, however, need to change the path for $SCRIPTDIR to the location you are using (and $TMPDIR if you don’t want to put your temporary files in /tmp).

Cinelerra Audio Sync Problem on Ubuntu Jaunty

Everytime I try to set up Cinelerra on a new project or on a new system configuration, it takes a while to get everything working well. In my case, I have several new variables to contend with: I’m starting with HDV footage from a new camera, I have a new machine, I’m using a new version of Ubuntu and of Cinelerra.

This time around, I’ve had trouble with audio sync during playback. The symptom is that when I play a dv clip (which is fine on its own in VLC) in the Compositor Window in Cinelerra, I get a 10 second audio time lag which seems impossible to correct by nudging or setting the audio offset value.

Fortunately, this doesn’t seem to affect rendering, but it still makes it tough to edit properly. After a great deal of fiddling, I finally discovered (via a note here) that I could use the Pulseaudio sound driver in Ubuntu (select “Esound” and port 7007 in the Cinelerra Playback settings) instead of ALSA. Suddenly, my project sounded much better. The audio track was still out slightly, but I used the technique described in Secrets of Cinelerra to sync the sound using the nudge value of the audio tracks. There is additional helpful info on audio syncing here.

I was surprised that this simple matter took so much searching to solve. Here’s hoping that this post will pop up in future searches.

To summarize, if you are running Cinelerra on Ubuntu and have audio sync problems, try switching your audio driver to “esound” on port 7007 and you may be pleasantly surprised.

UPDATE: This is very weird, but the above config works fine when I run Cinelerra as a user, but NOT when I run it as root! Obviously, I’ve got the PulseAudio settings messed up somewhere, but at least it works.

Setting up/Moving a Cinelerra Project

I finally decided to reorganize my current video project to make things easier to find.  For starters, I realized that all the raw dv footage I’ve captured should not be in the same folder as the project itself.  That is a recipe for creating copies of the same footage later when I want to use it in another project.

Instead, I created a folder for my footage named by date and subject and stored it with the rest of my similar captured footage.  That way it is ready for me to grab for any project.  Next, I created a project folder (I called it “project_wbp” in my case).  Inside that I added another folder called “media_rawdv”.  Then I used this command in a terminal:

ln -s -t /path/to/media_rawdv /path/to/original/*.dv

That command creates symbolic links (i.e. shortcuts) to all the original files.  Cinelerra doesn’t care whether its resources are real files or links, so this works fine.  Later, if I move the original files, I can just recreate the links and I’m all set.

There are two important steps  you need to do when you move things around or rename them in your project:

1. You should edit your project .xml file and change the paths as necessary (note that Cinelerra doesn’t seem to like relative paths, so you should use absolute ones).  Don’t forget to change the path to the xml file itself if that has moved.  Any text editor can handle this.

2. To save a lot of processing on large projects with many resources, you should rename all the project’s index files (you can change the location in Cinelerra’s settings, but they’re usually in ~/.bcast”) and change the internal paths in each file.  The following program should do the trick:

#!/bin/bash
#first rename the files in .bcast
rename ‘s/home_video_oldname/home_video_project_wbp_media_rawdv/’ ~/.bcast/*.idx
#next use this to fix the internal paths in the index files
OLD=”\/home\/video\/oldname\/”
NEW=”\/home\/video\/project_wbp\/media_rawdv\/”
DPATH=”~/.bcast/*.idx”
BPATH=”~/.bcast/bak/”
TFILE=”/tmp/out.tmp.$$”
[ ! -d $BPATH ] && mkdir -p $BPATH || :
for f in $DPATH
do
if [ -f $f -a -r $f ]; then
/bin/cp -f $f $BPATH
echo $f
sed “s/$OLD/$NEW/g” “$f” > $TFILE && mv $TFILE “$f”
else
echo “Error: Cannot read $f”
fi
done
/bin/rm $TFILE

The only things to change in the preceding script are the two paths in the rename command and the same in the OLD and NEW variables.  Note the the index files include real paths (hence the \ escapes to make the literal / characters work), but in their names they replace the / characters with underscores.

For smaller projects, you could probably skip this step, as Cinelerra will regenerate the files anyhow.  For my 40 minute video, which has hundreds of clips, it takes hours to do this.   The script does the job in a couple of minutes.

Auditioning Multimedia Distros (part 3 – Ubuntu Studio)

After trying dyne:bolic and pure:dyne, I reverted to my last resort:  Ubuntu Studio (Intrepid Ibex version).

It isn’t that I dislike Ubuntu.  I’ve had it installed on my laptop and on our family computer for years.  I’ve also recommended it to friends several times with great success.  There were, however, two reasons why I wanted to avoid it for my multimedia workstations:

  1. Ubuntu uses the Gnome desktop, which, although it is great, needs more system resources than, for instance, Xfce.  I wanted to leave as much CPU and memory as possible for Cinelerra.
  2. Ubuntu Studio comes pre-configured with loads of good multimedia applications, but Cinelerra itself isn’t one of them.

It turns out that I needn’t have worried on either score.  Installing Cinelerra turned out to be painless.  I used the community version (see Cinelerra :: a video editor and compositor for Linux for information on adding the repository to apt and installing the package).  Further tips for installing Cinelerra on UbuStu can be found here.

Once I had Cinelerra installed, it seemed to be working, but I still needed to do some tweaking of Cinelerra’s own settings to make it usable.  I’ll cover that in another post.

Beyond that, UbuStu worked like a charm without any performance issues on my aging hardware.  I’m also really enjoying the artwork and styling of the default UbuStu desktop.  Very snazzy.

Auditioning Multimedia Distros (part 2 – pure:dyne)

So, after my brush with dyne:bolic, I decided to try pure:dyne. PD was originally a revised version of dyne:bolic, but the producers of it decided to go upstream and recreate their distro based on Debian — The Universal Operating System (the ultimate ancestor of many specialized distros, including Ubuntu).

I’ve been using Debian for nearly ten years now, and I believe it is a good foundation to start from.  Unfortunately, the pure:dyne folks have a lot of work to do in order to customize their version to the same extent as dyne:bolic.  For example, where dyne:bolic has organized its apps and utilities into a very hierarchical menu system which makes it easy to find the function you want, even if you don’t know the name of the application, pure:dyne dumps everything into a single menu folder.  Unless you already know all of the multimedia apps you want to use, you have to spend a long time exploring to begin to be productive.  This is a minor complaint, of course, and otherwise pure:dyne looks like it would suit my purposes just fine.

When I tried to run Cinelerra on “leek and potato,” the latest pure:dyne release, I couldn’t get past loading my project.  This was due to the USB hard drive problem I’ve been having with certain kernel versions.  Pure:dyne, through no fault of its own, inherited the same issue.

At this point, not yet understanding the source of the problem, I gave up and moved on to Ubuntu Studio.  Since my drive worked with Ubuntu Hardy on another machine, I reasoned that UbuStu might work better.  Meanwhile, PD looks like a good distro which I would happily use again.  I expect it will be even better in future versions.

Auditioning Multimedia Distros (part 1 – dyne:bolic)

Cinelerra is a full-featured non-linear video editor for Linux. It has capabilities which, while nowhere near as polished, are roughly comparable to Final Cut Pro, the only commercial NLVE I am familiar with.

I’ve had a lot of trouble with Cinelerra over the years. I managed to successfully create a 40 minute rough cut of my latest project over a year ago, but then, due to upgrades on my Debian box, I couldn’t do any further edits. Cinelerra would hang or crash at times. Other times, I could load my project, but I couldn’t playback anything or make edits. In short it was an unusable pain and I nearly gave up.

Fortunately, I had faith that it was indeed possible to use Cinelerra successfully (in fact, I think it is a very good NLE tool which I enjoy using). Now that I’ve finally decided to tackle setting it up again, I decided rather than invest a lot of time messing with my existing Debian install (which refused to let me install Cinelerra and Kino at the same time due to library conflicts), I decided to start fresh with a setup intended for multimedia production.

I started with d y n e : b o l i c version 2.5.2.  I loved the XFCE desktop and the piles of cool multimedia software.  Dyne:bolic (made by Rastafarians! — I think that should be the distro’s motto) also allows you to copy the live CD to your hard drive without dual booting or disrupting your regular OS.  Cool.

Unfortunately, the live CD doesn’t include apt for upgrades (apt is a package manager in Debian-based distros which allows simple, one-command upgrades of all installed applications and easy installs of new packages from on-line repositories), so it isn’t really easy to fix problems with the pre-installed apps.  I was happy to use dyne:bolic anyhow until I discovered that Cinelerra simply doesn’t work in this version of the distro (at least on my crappy old hardware).

dynebolic screenshot

dynebolic screenshot

Due to a configuration problem with the kernel (or something? see Cinerella dont work :-(: msg#00020)) Cinelerra will load, but won’t allow the user to interact with the menus or controls.  This was a showstopper for me.



So, I’m not using dyne:bolic at this point.  I gave the CD to my musically-inclined son, however, and he used the “dock” method to install on his Windows laptop (to dock, simply copy the “dyne” folder from the CD onto the root of your hard drive — see Install on harddisk? Dock! — then use the “Nest” function to save your settings to the same folder, reboot with the CD, and you’re running db from the hard drive).  He’s had good fun using the drum machine and synth programs so far.

Note: one quirk which I noticed about db is that all of your drives are mounted with an extra layer of subfolder called simply “1.”  No doubt there’s a simple and good explanation for this, but it seems a bit odd to me.