Posts tagged command line

Web Video

I’ve been experimenting with video codecs and formats for uploading video to the web (Facebook, Youtube, etc.).

Encoding for the Web

To make a tiny file, you can encode files this way:

ffmpeg -i foo.dv foo.mp4

For better quality output, there are a number of variables you can control. I’m starting with two assumptions for the time being: I’m using the h264 codec and my target output format is mp4. For YouTube, flv would be better.

ffmpeg -i foo.dv -ab 128k -ar 44100 -b 1200k -vc h264 -qscale 1 -s 480×360 foo.mp4

“-i foo.dv” just identifies the input file, and the rest of the options apply to the output file.

For my own future benefit, I will break down the options I’ve used: “-ab 128k” is the audio bitrate, “-ar 44100″ is the audio sampling frequency (44100 is the default anyhow); “-b 1200k” is the video bitrate; “-vc h264″ is the video codec; “-qscale 1″ sets the video quantizer scale (lower is better quality, use “-sameq” for the same quality); “-s 480×360″ is the target width and height. All of these variables can be changed to create a higher/lower quality file which is smaller/larger in size. These settings create a decent output file for people to download, but it will be resampled on uploading to websites for streaming.

FFMPEG Tricks

If you just want to do a quick trim of the clip before uploading, I’ve found this is an easy way to do it. First play the video:

mplayer -osdlevel 3 -menu -loop 0 -dr -ni -framedrop -autosync 30 -msglevel all=0 -really-quiet -nolirc foo.dv

Note the start and end points you want, then run the ffmpeg command with “-ss NN” for the starting point in seconds and “-t NN” for the duration. That allows you to trim either the start or end of the clip without having to break out Cinelerra or Kino. By the way, I’ve got a pile of switches enabled on mplayer, but the only one that really matters is “-osdlevel 3″ so you can see the counter.

Some Youtube guidance on formats here: http://www.google.com/support/youtube/bin/answer.py?answer=132460&safe=off

Facebook format details are found here. You could use this command and vary the qscale and bitrate to get a reasonable file size:

ffmpeg -i foo.dv -acodec libfaac -ab 128k -b 1200k -vc h264 -qscale 6 foo.mp4

Vimeo format recommendations give specific values for bit rates and codecs as follows for high quality:

ffmpeg -i foo.dv -acodec libfaac -ab 320k -b 2000k -vc h264 -sameq foo.mp4

For HD video on Vimeo:

ffmpeg -i foo.m2t -acodec libfaac -ab 320k -b 3000k -vc h264 -sameq -s hd720 foo.mp4

“-s hd720″ tells ffmpeg to create 1280×720 output.

Sources of Information

This video is a good ffmpeg intro: http://www.linuxjournal.com/video/linux-howto-video-editing-magic-ffmpeg

For Youtube uploads, they use the following command. The presenter explains that these parameters are good for Youtube because they will not be re-encoded on upload:

ffmpeg -i foo.mov -ar 22050 -acodec libmp3lame -ab 32k -r 25 -s 320×240 -vcodec flv -qscale 9.5 foo.flv

He also shows how to crop and letterbox using ffmpeg and even creates a moving spotlight. He also uses the “-loop_input” ffmpeg switch in order to make a 10 second clip of a still frame. Very handy!

ffmpeg -loop_input -i still.png -t 10 -r 29.97 -qscale 2 extendedstill.mpg

Here is an explanation of how to use ffmpeg for screencasting.

Extract Audio Track from a Video File

I needed to isolate the audio track from an HDV video I captured from my camera. This command worked great:

mplayer -dumpaudio video.m2t -dumpfile audio.mp3

as discussed here: http://linux.byexamples.com/archives/229/extract-audio-from-video-or-online-stream/

UPDATE: To extract wav audio data from a dv file, I had to use this command:

ffmpeg -i video.dv -vn -acodec copy audio.wav

Details here: http://www.tuxradar.com/content/ffmpeg-made-easy

Scripts for Monkeying with Cinelerra XML

I’ve moved my Cinelerra projects from one directory path to another and scrambled a few things in the process. I’ve written a few bash scripts which helped me sort things out.

In the process, I’ve learned a bit about the xml format which Cinelerra uses to store references to files and edits. When you add a resource, it appears twice between opening and closing “ASSET” tags with the format <ASSET SRC=”/path/to/filename”>[asset details]</ASSET>. Then every time you include that file in a track of your project, the filename appears again in an “EDIT” tag complex in the format <FILE SRC=”/path/to/filename”>[edit details]</FILE>

First, I needed to check the paths for all the resources in the file to make sure they are still valid. I called this program test_xml_paths.sh. I corrected the incorrect paths with a text editor, but found that Cinelerra was still crashing with a memory error when I loaded the project.

Second, I decided to find and eliminate all of the unused assets in the file. I originally had a forty minute video in one file, but some time ago I split the project into four ten minute chunks. When I created the original project, I had added a couple of thousands of clips for possible inclusion, so now that I’m working on the final cut, I don’t need all that bloat.

I created a function to figure out which assets were being used in the edit list and which were not: list_unused_assets.sh.

Finally, I wrote a script which would recreate a new xml project file with all the redundant assets removed: remove_unused_assets.sh.

I can’t offer any warranty on these scripts, of course, but I hope they may help somebody else. If nothing else, there’s a great example of multi-line sed editing in the last one.

Remove Unused Assets from a Cinelerra XML Project File

This is a bash script to remove all the unwanted assets from a Cinelerra project XML file. I needed it to cut out bloat from a project I was working on which had hundreds of unused resources after I split the project up into ten minute chunks. Doing so stopped Cinelerra from crashing when I loaded the project: YMMV.

Save this script as “remove_unused_assets.sh”. It needs the function in “list_unused_assets.sh” which I posted here.

#! /bin/bash
#kk remove_unused_assets.sh
#kk
#kk For cinelerra xml files
#kk this script removes the contents of asset tags which are not also used for edits

if [ -z "$1" -o "$1" = "-h" ]
then
echo “Usage: remove_unused_assets [xml_filename]“
echo “For cinelerra xml files, this script removes the contents of asset tags which are not being used for edits”
exit
fi

TMPDIR=”/tmp”
SCRIPTDIR=”$HOME/Video/video_new/common/scripts”

. $SCRIPTDIR/list_unused_assets.sh
. $SCRIPTDIR/include.inc

#assign the parameter to a variable
XMLFILE=$1

#location of the temporary files
ASSETS_FILE=”$TMPDIR/ASSETS_FILE”
EDITS_FILE=”$TMPDIR/EDITS_FILE”
XMLFILE_NEW=”$1″_new
XMLFILE_TMP=”$TMPDIR/XMLFILE_TMP”

#check if the xml file exists and exit if it does not
if [ ! -e $XMLFILE ]; then error “XML file does not exist”; fi

list_unused_assets $XMLFILE
if [ $? != 0 ] ; then error “list_unused_assets failed”; fi

cp $XMLFILE $XMLFILE_TMP

echo
echo Removing Unused Assets…
echo
echo “Working…”
echo

# loop through the list of unused assets and remove the ASSET tag lines for each
# sed command from here: http://ilfilosofo.com/blog/2008/04/26/sed-multi-line-search-and-replace/

for f in $( comm -23 $ASSETS_FILE $EDITS_FILE ); do

#to make the filename string work in sed, I have to escape all of the periods and slashes
FORMATTED_f=`echo “$f” | sed -e ‘s:\/:\\\/:g’ -e ‘s:\.:\\\.:g’`

#this complicated sed script came from the sed FAQ: http://sed.sourceforge.net/sedfaq4.html#s4.21
#in order to make the variable substitution work, I had to close the sed script with a single quote
#then enclose the variable in double quotes, then re-open the sed script with a single quote
#the -i option is for editing a file “in place”
sed -i -e ‘
# sed script to delete a block if /regex/ matches inside it
:Top
/\/ { # For each line between these block markers..
/\/ASSET\>/!{ # If we are not at the /end/ marker
$!{ # nor the last line of the file,
N; # add the Next line to the pattern space
b Top
} # and branch (loop back) to the :Top label.
} # This line matches the /end/ marker.
/’”$FORMATTED_f”‘/d; # If /regex/ matches, delete the block.
} # Otherwise, the block will be printed.
#—end of script—
‘ $XMLFILE_TMP

#diff $XMLFILE $XMLFILE_TMP

echo -n ‘ \r’
echo -n $f’\r’

done

cp $XMLFILE_TMP $XMLFILE_NEW

echo
list_unused_assets $XMLFILE_NEW

echo
echo Finished.
echo
echo The new version of the xml file has been saved as $XMLFILE_NEW.
echo

This program depends on include.inc which contains the following:

#!/bin/bash

#use colour in the output just for fun
GREEN=$(printf “\033[32m”)
RED=$(printf “\033[31m”)
NC=$(printf “\033[0m”)

error()
{
echo ERROR: $1
exit 1
}

I tried to make these scripts as tidy and portable as possible — you will, however, need to change the path for $SCRIPTDIR to the location you are using (and $TMPDIR if you don’t want to put your temporary files in /tmp).

List Unused Assets in a Cinelerra XML Project File

This is a bash script which contains a function used to figure out which assets in a Cinelerra project are not required for the edit list. I created it because I had pulled in a couple of thousand media resources for a project and wanted to eliminate bloat. I moved the project and found it kept crashing on me until I eliminated the crud. I wrote another program to call this function and then actually clean out the xml. You can view “remove_unused_assets.sh” here.

Running this script doesn’t do anything until you call the enclosed function. This function can be called from the command line with:

. list_unused_assets.sh;list_unused_assets
#! /bin/bash
#kk list_unused_assets.sh
#kk
#kk check a cinelerra xml file to see which assets have been loaded, but not used in the project
#kk provide filename to check as parameter
#kk from the command line, type “. list_unused_assets.sh;list_unused_assets

list_unused_assets()
{

SCRIPTDIR=”$HOME/Video/video_new/common/scripts”
TMPDIR=”/tmp”

. $SCRIPTDIR/include.inc

#location of the temporary files
ASSETS_FILE=”$TMPDIR/ASSETS_FILE”
EDITS_FILE=”$TMPDIR/EDITS_FILE”

#assign the parameter to a variable
XMLFILE=$1

#check if the xml file exists and exit if it does not
if [ ! -e $XMLFILE ]; then error “XML file does not exist”; fi

#if the temp file exists, delete it
if [ -e $ASSETS_FILE ]; then
rm $ASSETS_FILE
fi

if [ -e $EDITS_FILE ]; then
rm $EDITS_FILE
fi

echo
echo Checking $XMLFILE for assets unused by the project…
echo

#find all of the filenames in the xml file and strip the extra junk
grep -E “ASSET SRC=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/”>.*$//g’ | sort | uniq > $ASSETS_FILE
grep -E “FILE SRC=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/”>.*$//g’ | sort | uniq > $EDITS_FILE

#output the counts
echo `grep -E “ASSET SRC=|FILE SRC=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/”>.*$//g’ | sort | uniq | wc -l` Total Assets
echo $RED`comm -23 $ASSETS_FILE $EDITS_FILE | wc -l` Unused Assets$NC
echo $GREEN`comm -12 $ASSETS_FILE $EDITS_FILE | wc -l` Used Assets$NC

}

This program depends on include.inc which contains the following:

#!/bin/bash

#use colour in the output just for fun
GREEN=$(printf “\033[32m”)
RED=$(printf “\033[31m”)
NC=$(printf “\033[0m”)

error()
{
echo ERROR: $1
exit 1
}

I tried to make these scripts as tidy and portable as possible — you will, however, need to change the path for $SCRIPTDIR to the location you are using (and $TMPDIR if you don’t want to put your temporary files in /tmp).

Test File Paths in a Cinelerra XML File

This is a bash script to test all the filenames and paths in a Cinelerra project to make sure they are still valid.

#!/bin/bash
#kk test_xml_paths.sh
#kk
#kk check a cinelerra xml file to make sure the file references are all valid
#kk provide filename to check as parameter

if [ -z "$1" -o "$1" = "-h" ]
then
echo “Usage: test_xml_paths [xml_filename]“
echo “For cinelerra xml files, this script checks that file paths are valid”
exit
fi

SCRIPTDIR=”$HOME/Video/video_new/common/scripts”
TMPDIR=”/tmp”

. $SCRIPTDIR/include.inc

#assign the parameter to a variable
XMLFILE=$1

#check if the xml file exists and exit if it does not
if [ ! -e $XMLFILE ]; then error “XML file does not exist”; fi

#location of the temporary file
FILENAMES=”$TMPDIR/TEST_XML_PATHS”

#set counters to zero
GOODCOUNT=0
BADCOUNT=0

#if the temp file exists, delete it
if [ -e $FILENAMES ]; then
rm $FILENAMES
fi

echo
echo Checking $XMLFILE for valid file paths…
echo

#find all of the filenames in the xml file and strip the extra junk
#send the output to a file
grep -E “SRC=|PATH=” $XMLFILE | sed -e ‘s/^.*SRC=”//g’ -e ‘s/^.*PATH=”//g’ -e ‘s/”>.*$//g’ > $FILENAMES
if [ $? != 0 ] ; then error “grep failed”; fi

# loop through the output and test if the filename is either a link or a file
for f in $( cat $FILENAMES ); do
if [ ! -h $f -a ! -e $f ]; then
BADCOUNT=`expr $BADCOUNT + 1`
echo ERROR: Missing $f
else
GOODCOUNT=`expr $GOODCOUNT + 1`
fi
done

#output the counts
echo
echo $RED$BADCOUNT Total Bad Pathss $NC
echo $GREEN$GOODCOUNT Total Good Paths$NC
echo

This program depends on include.inc which contains the following:

#!/bin/bash

#use colour in the output just for fun
GREEN=$(printf “\033[32m”)
RED=$(printf “\033[31m”)
NC=$(printf “\033[0m”)

error()
{
echo ERROR: $1
exit 1
}

I tried to make these scripts as tidy and portable as possible — you will, however, need to change the path for $SCRIPTDIR to the location you are using (and $TMPDIR if you don’t want to put your temporary files in /tmp).

Looking at Video File Metadata

Three choices as per the details here:

tcprobe -i filename
ffmpeg -i filename
midentify filename

Info on file types here.

I’m trying to understand video formats to make it easier to transcode for various purposes. I’ve found some useful links:

adamwilt.com > the DV, DVCAM & DVCPRO Formats

HDV vs HD: A Primer

Digital Video Formats and Video Conversion Explained

Video File Formats 101 : Formats Explained

Scanning via USB from a Brother MFC-8500C

I’m still working on digitizing all my old media, including paper. I have an old Brother laser multi-function with an automatic document feeder that I’ve only ever used as a printer and fax. I tried to set it up for scanning years ago, but had no success. Times have changed and I had no problem getting it working this time around so I can scan all my old paper files.

Scanning only works for the root user at the moment, but I don’t care. Discussion here.

Step 1. Download and install the linux drivers which are now available from Brother here. The instructions provided worked flawlessly on Ubuntu Jaunty.

Step 2. As root, Install the sane-utils and libtiff-tools packages.

sudo apt-get install sane-utils libtiff-tools

Step 3. Scan. XSane worked fine for basic scans, but I want to be able to do double-sided scanning on a simplex scanner, so I decided to use the command line for this. There are other tools available if you prefer a gui.

To scan letter-size single sided:

scanimage -y 279 –format=tiff –batch –progress

The “-y 279″ is the page length in mm — without it, scanimage seemed to create legal size scans by default. To scan a pile of pages on both sides, start with side one:

scanimage -y 279 –format=tiff –batch –progress –batch-double

Then, flip over the pile and substitute the last page number (which the previous command will tell you) minus one for “[last page - 1]” below:

scanimage -y 279 –format=tiff –batch –batch-double –progress –batch-increment=-2 –batch-start [last page - 1]

Finally, put the whole thing together in one big tiff and convert to pdf:

tiffcp out*.tif out_all.tif;tiff2pdf out_all.tif -o filename.pdf;rm -f out*.tif

The final rm is added to the previous command so that you can continue scanning jobs without either getting your tiff files mixed up or inadvertently trying to include a finished multipage tiff in the next pdf conversion.

I have written a script which prompts the user for single or double-sided printing and an output filename, then scans and converts documents:

#!/bin/bash
#kk a script to scan a batch of pages on both sides, then combine and convert them to pdf

WORKDIR=”/tmp”
OUTPUTDIR=”/home/kevin/Dropbox/tmp/scans”
DATESTAMP=$(date ‘+%Y%m%d%H%M%S’)

#clean up
rm -f $WORKDIR/out*

#prompt for single or double-sided scanning
read -n 1 -p “How many sides ([1]/2)? ” SIDES
if [ "$SIDES" == "2" ]
then

#scan the first sides
#the end of this command sends progress output to both stdout and to a file
scanimage –format=tiff –batch=”$WORKDIR/out%d.tif” –progress –batch-double -y 279 3>&1 >&2 2>&3 3>&- | tee $WORKDIR/out_count.txt

#figure out what page to start the flip side scan on
LASTPAGE=`grep “Scanning page ” $WORKDIR/out_count.txt | tail -n 1 | sed ‘s/Scanning page //g’`
BATCHSTART=`echo $LASTPAGE – 1 | bc -l`

#give user a chance to flip and refeed the pages
read -n 1 -p “Flip over the pages (counting down from $BATCHSTART) and press any key to continue…”

#scan the second sides
scanimage –format=tiff –batch=”$WORKDIR/out%d.tif” –progress –batch-increment=-2 –batch-start $BATCHSTART -y 279

else

#scan the pages
scanimage –format=tiff –batch=”$WORKDIR/out%d.tif” –progress -y 279

fi

#combine all the tiff files into one document
tiffcp $WORKDIR/out*.tif $WORKDIR/out_all.tif

#prompt for a filename
read -p “What should I name the document (extension will be added automatically)? ” NAMER
if [ "$NAMER" == "" ]
then
NAMER=”scan$DATESTAMP”
fi

#convert to pdf
tiff2pdf $WORKDIR/out_all.tif -o $OUTPUTDIR/$NAMER.pdf

The script is pretty basic, but should be easily adaptable for your own purposes. You can save it as scan.sh, make it executable, and run it with “./scan.sh”

Creating a Shared Photo Directory

In order to allow my wife and myself to both have access to our common photo directory, I created a folder called /home/photos and set permissions as follows:

As root, create a new group and add users to it:

addgroup photos
adduser user1
adduser user2

Create the folder:

mkdir /home/photos

Set the group:

chown root:photos /home/photos

Set the permissions:

chmod g+rwx /home/photos
chmod a+s /home/photos

The last command sets the setgid bit so that all subfolders and files will inherit the photos group.

Convert Raw MiniDV Footage to Upload to Facebook

I just shot a bit of footage I wanted to upload to Facebook. According to this Facebook question, mpeg4 video is a good choice.

This command seems to do the trick to convert my rawdv footage to a Facebook friendly format:

cat foo.dv | ffmpeg -f dv -i pipe: -acodec libfaac -vcodec libx264 -s 640×360 -aspect 16:9 foo.mp4

Note that the names of the codecs in this command seem to vary according to your version of ffmpeg.  If this version doesn’t work, you might try faac and h264 instead of libfaac and libx264, respectively.