Casual Hacking

Main Menu

Batch import Multimedia Stream Data into MongoDB with FFprobe

FFprobe is a stream analyzer that optionally reports in JSON. We will create a PHP script that reads JSON from STDIN, makes an object using json_decode, and inserts the object into a MongDB database. This script could be used with any program that outputs a JSON stream. A bash script will be used to batch process all files within the current directory. For example, the data may be used for analysis and logging of a day's shoot.

Example JSON output from FFprobe

Note: In this example I'm using GNU GCC 4.6.2 compiler I built and installed with --prefix=/home/devcc and PHP installed with --prefix=/home/devcc

 

Download

Code available at github.

git@github.com:creamy/in-ffprobe-mongodb.git

 

You can shorten build times using -j4, -j8, -j16 etc on make (depending on number of processors available).

yasm

download source from http://yasm.tortall.net

$ ./configure --prefix=/home/devcc --with-gnu-ld \
	--with-libiconv-prefix=/home/devcc --with-libintl-prefix=/home/devcc
$ /home/devcc/bin/make
$ /home/devcc/bin/make install

libx264

$ git clone git://git.videolan.org/x264.git
$ cd x264
$ ./configure --prefix=/home/devcc --enable-shared
$ /home/devcc/bin/make
$ /home/devcc/bin/make install

ffmpeg

$ git clone git://source.ffmpeg.org/ffmpeg.git
$ cd ffmpeg
$ ./configure --prefix=/home/devcc --enable-libx264 --enable-gpl
$ /home/devcc/bin/make
$ /home/devcc/bin/make install


$ ffmpeg -version
ffmpeg version N-35180-g31a1342, Copyright (c) 2000-2011 the FFmpeg developers
  built on Nov 25 2011 22:20:21 with gcc 4.6.2
  configuration: --prefix=/home/devcc --enable-libx264 --enable-gpl
  libavutil    51. 29. 1 / 51. 29. 1
  libavcodec   53. 37. 1 / 53. 37. 1
  libavformat  53. 21. 0 / 53. 21. 0
  libavdevice  53.  4. 0 / 53.  4. 0
  libavfilter   2. 49. 1 /  2. 49. 1
  libswscale    2.  1. 0 /  2.  1. 0
  libpostproc  51.  2. 0 / 51.  2. 0

Install MongoDB PHP driver

$ git clone https://github.com/mongodb/mongo-php-driver.git
$ cd mongo-php-driver
$ /home/devcc/bin/phpize
$ ./configure --prefix=/home/devcc --with-gnu-ld --with-pic
$ /home/devcc/bin/make
$ /home/devcc/bin/make install

Make sure reported installation directory matches extension_dir in php.ini

$ php -i | grep extension_dir
extension_dir => /home/devcc/lib/php/extensions/no-debug-non-zts-20090626 
	=> /home/devcc/lib/php/extensions/no-debug-non-zts-20090626
sqlite3.extension_dir => no value => no value

Enable extension in php.ini

$ vi /home/devcc/lib/php.ini

extension=mongo.so

Create jtom.php

This script reads json data from STDIN and inserts a BSON object into MongoDB

<?php

$x=fopen ("php://stdin","r");
$d='';
while ($s=fgets($x)) {
        $d.=$s;
}
fclose($x);

$u=json_decode($d);
$m = new Mongo("mongodb://127.0.0.1:27017");
$parm = $argv[1];
$in = $m->motion->$parm;
$in->save($u);
$m->close();

?>

chmod 755 jtom.php

Create inmongo.sh

We use $1 to specify the project (MongoDB collection) name.

#!/usr/bin/env bash

# $2 is movie filename
# $1 is collection name within mongodb motion

ffprobe $2 -v quiet -print_format json -show_format -show_streams -show_packets | php jtom.php $1

chmod 755 inmongo.sh

Test the script

$ ./inmongo.sh testone MVI_4449.MOV
$ mongo motion
> db.testone.find();

[...output...]

Batch import

$ for i in *.MOV; do ./inmongo.sh testfour $i; done;
$ mongo motion
> db.testfour.find();

[...output...]

 

References

MongoDB
http://www.mongodb.org/

Mongo PHP Driver
http://php.net/manual/en/mongo.installation.php

FFmpeg
http://ffmpeg.org/

FFprobe
http://sourceforge.net/projects/ffprobe/
http://forums.creativecow.net/thread/291/71

x264
http://www.videolan.org/developers/x264.html

NoSQL
http://nosql-database.org/

BSON - Binary JSON
http://bsonspec.org/

PHP - Personal Home Page
http://www.php.net/
http://php.net/manual/en/book.mongo.php

Git
http://git-scm.com/
https://github.com/
http://forums.freebsd.org/showthread.php?t=10810

 


Copyright 2011 Waitman Gobble · waitman@waitman.net