curious onloooker

A few things, mostly technical notes...

Monday, January 27, 2014

lockfile in python

Alternate Link to this article

lockfile in python

lockfile is one of my must haves, as it is a very useful semaphore lock file creator.
# lockfile --help
Usage: lockfile -v | -nnn | -r nnn | -l nnn | -s nnn | -! | -ml | -mu | file ...
# rpm -qf `which lockfile`
procmail-3.22-25.1.el6.x86_64
As can be seen above, /usr/bin/lockfile is part of procmail, which comes with its own dependencies (usually sendmail or postfix etc.). I did not like having to install those MTA type dependencies to setup procmail for locking purposes. This is the reason why I wrote lockfile.py.

lockfile.py

#!/usr/bin/python

# Author Evuraan Gmail dot com

import optparse,sys,os,time

def usage():
 print """
Usage   : lockfile.py 
Usage   : lockfile.py -l  
Example : lockfile.py -l 1300 /tmp/my.lockfile.txt
Example : lockfile.py /tmp/my.lockfile.txt

"""


def remove_lockfile(lock_file):
 os.remove(lock_file)


def set_lockfile(lock_file):
 write_file = open(lock_file,"w",)
 write_file.write(str(time.time()))
 write_file.close()
 if os.path.isfile(lock_file):
  # file exists, good enough for us.
  return 0
 else:
  return 1



#if the lockfile exists, check for locktimeout
def checklock(lock_file):
 if os.path.isfile(lock_file):
  # lock file exists. check how old etc.
  mtime = float(os.path.getmtime(lock_file))
  delta = time.time() - mtime
  if float(delta) >= float(locktimeout):
   # file is stale, over write it
   if set_lockfile(lock_file) != 0:
    sys.exit(1)
  else:
   #file is not stale. new. spanking new.
   sys.exit(1)
 else:
  # no lock file.
  if set_lockfile(lock_file) != 0:
   sys.exit(1)



if __name__ == "__main__":

 parser = optparse.OptionParser()
 parser.add_option('-l', '--locktimeout',
  dest="locktimeout",
  type="int",
  default="3489999999",
  help="""If you specify a locktimeout then a lockfile will be
removed by force after locktimeout seconds have passed since the lockfile was last
modified/created (most likely by some other program that unexpectedly
died a long time ago, and hence could not clean up any leftover lockfiles)."""
  )

 options,remainder = parser.parse_args()

 #print "locktimeout       :", options.locktimeout
 #print "remainder         :", remainder

 global locktimeout
 locktimeout = options.locktimeout

 try:
  if len(remainder) == 0:
   print "Invalid usage"
   usage()
   sys.exit(2)
  else:
   global lock_file
   lock_file = remainder[0]
 except:
  sys.exit(2)


 #print "Ahem", lock_file
 checklock(lock_file)


Usage:

$ ./lockfile.py
Invalid usage

Usage   : lockfile.py 
Usage   : lockfile.py -l  
Example : lockfile.py -l 1300 /tmp/my.lockfile.txt
Example : lockfile.py /tmp/my.lockfile.txt


$ ./lockfile.py --help
Usage: lockfile.py [options]

Options:
  -h, --help            show this help message and exit
  -l LOCKTIMEOUT, --locktimeout=LOCKTIMEOUT
                        If you specify a locktimeout then a lockfile will be
                        removed by force after locktimeout seconds have passed
                        since the lockfile was last modified/created (most
                        likely by some other program that unexpectedly  died a
                        long time ago, and hence could not clean up any
                        leftover lockfiles).


$ ./lockfile.py  /tmp/mylockfile ; echo $?
0

$ ./lockfile.py  /tmp/mylockfile ; echo $?
1

Download

lockfile.py can be downloaded as:
$ wget https://evuraan.info/evuraan/stuff/lockfile.py.txt --no-check-certificate -O lock.py

License

Software License:

Copyright (c) 2013, evuraan All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  • Neither the name of evuraan nor the names of other contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Sunday, December 22, 2013

mythtv - Auto-delete duplicate recordings with myth-remove-duplicates.sh


Question:

I have a large archive of recordings on a well-used mythbox and I know I have accumulated a lot of duplicates. I need to free some space, but going through and manually getting rid of the duplicates would be a monumental task.

Is there any mechanism or script available to track down these duplicates, and preferably delete all but the newest recording? I've searched the web and poked at my interface with no luck.

Answer:

You're welcome to use the script I wrote; I've been running it regularly to remove the duplicates.

$ wget http://evuraan.info/evuraan/stuff/myth-remove-duplicates.sh.txt -O myth-remove-duplicates.sh
$ chmod +x myth-remove-duplicates.sh 

Edit to replace RECORDINGDIR, user_name and pass_word with what's applicable to your setup and run myth-remove-duplicates.sh.

Running it :

$ ./myth-remove-duplicates.sh
1091_20131221090100.mpg is duplicate
removed `/var/lib/mythtv/recordings/1091_20131221090100.mpg'
1091_20131221070100.mpg is duplicate
removed `/var/lib/mythtv/recordings/1091_20131221070100.mpg'

Here's the script, myth-remove-duplicates.sh:

#!/bin/bash 
# Authored by Evuraan_AT_gmail_DOT_com
# ABSOLUTELY NO WARRANTY, to the extent permitted by
# applicable law.
# YMMV.
# Use at your own risk.

user_name="mythtv"
pass_word="yourpassword" 
RECORDINGDIR="/var/lib/mythtv/recordings"
list="/tmp/recordings.txt-$RANDOM-$RANDOM"
SQLSCRIPT="/tmp/recordings.sql-txt-$RANDOM-$RANDOM"

gen_lists(){
mysql -u "$user_name" -p"$pass_word" -e "select starttime,basename,title,description from recorded order by starttime" mythconverg | tac > $list
}

verify_duplicate(){
# if verbatim repeats..
sum_a=$(egrep "${a:0:16}" $list | awk -F".mpg\t" 'NR>1 {print $NF}' |md5sum) 
[ ! -z "$sum_a" ] && egrep "${a:0:16}" $list | awk -F".mpg\t" 'END {print $NF}' | md5sum |egrep -q  ${sum_a//-}
}

remove_duplicates(){
[ -s "${RECORDINGDIR}/${a}" ] && ( rm -v "${RECORDINGDIR}/${a}" ; 
	echo "DELETE FROM mythconverg.recorded WHERE basename = '$a';" > $SQLSCRIPT
	)
[ -s $SQLSCRIPT ] && mysql -u "$user_name" -p"$pass_word"   mythconverg < $SQLSCRIPT
}




gen_lists
awk {'print $3'} $list  |grep mpg$ | while read a ; do 
	[[ $(egrep -c ${a:0:16} $list ) -ge 2 ]]  && verify_duplicate && sed -i /"${a:0:16}"/d $list && echo $a is duplicate && remove_duplicates
done

rm $SQLSCRIPT $list 1>/dev/null 2>&1 || : 

Tuesday, July 30, 2013

mplayer - Cannot find codec for audio format 0x6134706D

mplayer is awesome!

 
I've been happily living with mplayer leveraging vdpau on my Ubuntu Lucid 10.04 LTS. Until today that is, when I tried to play videos from my SLR, and there was no audio. Upon close look, mplayer was complaining thusly:

FAAD: Failed to initialize the decoder!
ADecoder init failed :(
[aac @ 0x4442a60]Audio object type 0 is not supported.
Could not open codec.
ADecoder init failed :(
Cannot find codec for audio format 0x6134706D.
I did STFW, and tried a couple of things, none of those worked for me. That's when I decided to give mplayer2 a try. mplayer2 is a fork of mplayer, I had read elsewhere that it contains a number of features not available in the original MPlayer.

Since I use vdpau, the mplayer2 website indicated that it'd be a better bet for me:

Much better support for VDPAU functionality

  • Removed limitations that prevented switching frames more than once per monitor refresh. With MPlayer2 you can play high-FPS content or use fast forward on a 60 Hz monitor without breaking playback.
  • Added support for the frame timing functionality of VDPAU.
  • Improved performance by better buffer handling and smarter subtitle texture uploads (both VDPAU hardware decoding and displaying software decoded video with VDPAU perform better).
  • Added logic to reduce frame timing jitter in some situations.
  • Handle frames added by deinterlacing properly.
  • Several bugfixes.
  • Various minor improvements (studio level output support, set default deinterlace mode, ...)

Installing mplayer2 on  Ubuntu 10.04 LTS 

Here's how I found to install mplayer2 on ubuntu 10.04 LTS:

$ sudo add-apt-repository ppa:ripps818/coreavc
$  sudo apt-get update 
$  sudo apt-get install mplayer2 
 
For reference, here's my ~/.mplayer2/config file. (See archlinux wiki on more about these entries..)

$ cat ~/.mplayer2/config  
# Write your default config options here!
vo=vdpau,
vc=ffh264vdpau,ffmpeg12vdpau,ffodivxvdpau,ffwmv3vdpau,ffvc1vdpau,
 

mplayer2 and 0x6134706D

mplayer2 seem to be able to play all those files mplayer was complaining about. I think I like the vdpau quality better with mplayer2, or it could just be a phsycological placebo effect!

Either way, I love mplayer, and mplayer2!


 

Saturday, April 06, 2013

Comparing Calling Rates : Skype vs Fring Vs Rebtel

Comparing Calling Rates : Skype vs Fring Vs Rebtel

Rebtel wins hands down. (Best quality too!) (i use their android app) 



9.2c/min, 3.4c/min and 1.69c/min @Rebtel Wins!


http://evuraan.info/screenshots/images/skype_vs_fring_vs_rebtel.jpg

Saturday, March 09, 2013

install xbmc frodo on ubuntu 10.04 (Lucid)

xbmc does not seem to provide PPAs for Ubuntu Lucid at the moment. Like many others, I am not upgrading from Lucid LTS for another 2 years until its LTS span runs out, if I can help it.

Here's how to install xbmc 12.0 (aka Frodo) on Ubuntu Lucid:



sudo add-apt-repository ppa:nathan-renniewaldock/xbmc-nightly 
sudo apt-get update 
sudo apt-get install xbmc 
sudo apt-get install libtag1-dev 


Enjoy!

(reference)


Saturday, October 13, 2012

mplayer and vdpau



So I was just using mplayer -vo vdpau... and here's my cpu util:

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6172 XXX      20   0  155m  50m  14m R 22.9  2.5   0:06.86 mplayer
 
Thats when I stumbled upon this ArchLinux Wiki post, and added these to my ~/.mplayer/config file:

vo=vdpau,
vc=ffh264vdpau,ffmpeg12vdpau,ffodivxvdpau,ffwmv3vdpau,ffvc1vdpau, 
 
And now,
  
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
6397 XXX      20   0  157m  51m  15m S  2.3  2.6   0:03.73 mplayer
 
my %cpu dropped from 22.9% to 2.3%..! Amazing, isn't it?


Alternate link

Saturday, June 23, 2012

script to download latest firefox aurora

script to download latest firefox aurora and extract to mydir
wget -q "https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/?C=M;O=A" -O - | awk '/en-US.linux-i686.tar.bz2">/ {gsub(/href="/,"",$6)+gsub(/">.*/,"",$6); print "https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/"$6}' | while read a 
do  
 wget -c "$a" -q -O /tmp/firefox-aurora-en-US.linux-i686.tar.bz2 && cd mydir  && tar -xvjf /tmp/firefox-aurora-en-US.linux-i686.tar.bz2 
done

script to download latest firefox nightly build




script to download latest firefox nightly build and extract to mydir

wget http://nightly.mozilla.org/ -q -O - |awk '/en-US.linux-i686/ {gsub(/href="/,"",$2)+gsub(/">/,"",$2);print $2}' | while read a 
do   
   wget -c "$a" -q -O /tmp/firefox-nightly-en-US.linux-i686.tar.bz2  && cd mydir && tar -xvjf /tmp/firefox-nightly-en-US.linux-i686.tar.bz2 
done 

Thursday, May 24, 2012

xbmc as myth frontend on macosx

xbmc these days can function as mythtv frontend. I found xbmc easier to setup on other OS'es other than my regular front and back ends.

For instance, my regular mythtv back and front ends are Ubuntu lucid LTS (10.04); but I also have a macbook pro (running mac osx), and a Lenovo U400 running Ubuntu precise.

Here's how I setup xbmc as my mythtv frontenv.

Assumptions:

My myth backend is 192.168.1.100

Step  1.

Install xbmc. Refer http://xbmc.org/download/ on howto, for your platform.

Step 2.

Install MythBox addon. Go to Videos -> Add-ons -> Get More.. and select and install MythBox.

Step 3.

Go into MythBox to where you can define the MySQL details.  Here's how I've configured mine:




Step 4.


Spawn xbmc. Before we do that, lets route 6543 traffic to the myth backend server.

(Note: you can use socat easier than ssh, there is an example elsewhere in this blog to use socat)

mac


$ /usr/X11/bin/xterm -title "Close_me_after_myth" -e "ssh -c blowfish -L 6543:localhost:6543 192.168.1.100 cat - " 


linux
 
$ /usr/bin/xterm  -title "Close_me_after_myth" -e "ssh -c blowfish -L 6543:localhost:6543 192.168.1.100 cat - " & 


Start MythBox, enjoy



Tuesday, November 22, 2011

the thin myth-tv frontend

Running a thin myth-tv frontend:
-------------------------------

Here's how to setup a thin frontend for mythtv with minimal config and filesets.

Install mythtv frontend and socat on your machine.


# apt-get install mythtv-frontend socat


Create a file called ~/bin/thin-myth.sh:
(in my case, the IP of my backend server is 192.168.1.100)


#! /bin/bash
log="~/myth-tv.log"

#set the IP of your backend server.
IP=192.168.1.100 

xrandr >> $log
socat -b 128000 -d -d -lmlocal2 TCP4-LISTEN:3306,fork,reuseaddr  TCP4:$IP:3306 &
socat -b 128000 -d -d -lmlocal2 TCP4-LISTEN:6543,fork,reuseaddr  TCP4:$IP:6543  &
/usr/bin/mythfrontend -l $log


Ensure that DBHostname in ~/.mythtv/config.xml and ~/.mythtv/mysql.txt is set to 127.0.0.1:

$ grep 127.0.0.1 config.xml mysql.txt
config.xml:        127.0.0.1
mysql.txt:DBHostName=127.0.0.1


Launch ~/bin/thin-myth.sh and enjoy.

How does it work?
-----------------
socat listens on 3306,6543 for mysql and mythbackend ports respectively and relays those onto your 
mysql and mythbackend ports on your backend server. 
 
 
more here.. 

Monday, February 07, 2011

Wifi recommendation for HD streaming

I stream HD media over wifi, lots and lots of it. I live in a rented apartment where there're no wired ports available. I tried Belkin Ethernet over power adapter to carry my HD traffic, it failed miserably. Then I tried to bridge my wireless G usb dongle along with my wired nic - it never worked.

Then I went onto buy a "NETGEAR WNDR3400-100NAS IEEE 802.11a/b/g/n 2.4/5GHz Simultaneous Dual Band N600 Wireless Router" and a "NETGEAR WNCE2001 Wireless 802.11b/g/n Ethernet Port Universal WiFi Internet Adapter" as my bridge.

Sadly the bridge is in 2.4Ghz spectrum, and is susceptible to interference, even to microwaves. (2.4Ghz is such a crowded space - microwaves, garage door openers, cordless phones all use it - compared to that, 5.x GHz spectrum is squeaky clean..!)

I returned the 2.4Ghz WNCE2001 bridge and got a "Cisco-Linksys WET610N Dual-Band Wireless-N Gaming and Video Adapter" from Amazon - as implied, this bridge can talk in the 5.x Ghz spectrum. It has been almost a month now, and my HD streaming is working smooth.

Gist: For HD streaming over wifi - go for Wireless N devices in the 5.x Ghz spectrum, you will be happy!

keywords: mythtv wifi recommendation, hd streaming over wifi

Monday, January 24, 2011

rsync delta-transfer algorithm

rsync delta-transfer algorithm is pretty cool, reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.


export alias scpr='rsync -av --partial --progress --rsh="ssh -c arcfour -o compression=no -o StrictHostKeyChecking=no"'

myhost:/tmp# ls -ltrh somefile
-rw-r----- 1 root root 2.3G Jan 24 11:53 somefile

myhost:/tmp# time scpr somefile somehost:/tmp/
somefile
2452194584 100% 67.76MB/s 0:00:34 (xfer#1, to-check=0/1)

sent 2452494012 bytes received 31 bytes 65399841.15 bytes/sec
total size is 2452194584 speedup is 1.00

real 0m37.408s
user 0m22.341s
sys 0m5.624s



append some data to somefile:

myhost:/tmp# dmesg >> somefile


myhost:/tmp# time scpr somefile somehost:/tmp/

somefile
2452249209 100% 137.71MB/s 0:00:16 (xfer#1, to-check=0/1)

sent 266589 bytes received 396255 bytes 20395.20 bytes/sec
total size is 2452249209 speedup is 3699.59

real 0m32.525s
user 0m16.521s
sys 0m0.484s
myhost:/tmp#


rsync just sends out the delta(!), 2452494012 bytes the very first time vs. 266589 to accomodate delta.

scp: transfer speed & compression

With compression:

$ time scp -c arcfour -C somefile somehost:/somewhere

real 47m30.974s
user 46m18.974s
sys 1m4.040s


Without compression:

$ time scp -c arcfour somefile somehost:/somewhere
real 10m41.195s
user 2m41.142s
sys 0m51.611s


details of somefile:

$ file somefile
somefile: MySQL MISAM compressed data file Version 1


Gist: if a file is already compressed, scp-ing with -C (compression) rightly slows down the transfer.

Friday, December 31, 2010

suggestion to shareholic/boxee

I use shareholic to add items to boxee, and have a suggestion to make:

currently shareholic cannot find boxee-able itmes in pages like http://www.asianetglobal.com/ShowVdo.aspx?GlHID=2008, but can be easily fixed.


if you look closely, from,


http://www.asianetglobal.com/flvplayer.swf?videoURL=http://videos.asianetglobal.com/flv/asianet/VdoGallery/KKandathum2712.flv&thumbURL=Images/VdoGallery/Kettathum%20Kandathum.jpg


you should be able to extract just the flv url fairly easily,

http://videos.asianetglobal.com/flv/asianet/VdoGallery/KKandathum2712.flv


I'd love to see this added to shareholic

Saturday, December 11, 2010

Microsoft LifeCam Cinema

http://ecx.images-amazon.com/images/I/41da5eR2zlL._AA300_.jpg


So far this has been the best webcam I've ever used. It's an HD webcam, 720p HD widescreen, even works on ubuntu (lucid) linux right away under uvcvideo. It also has a microphone, and the audio quality is very clear.


[ 26.861845] uvcvideo: Found UVC 1.00 device Microsoft® LifeCam Cinema(TM) (045e:075d)
[ 26.911241] input: Microsoft® LifeCam Cinema(TM) as /devices/pci0000:00/0000:00:0b.1/usb1/1-3/1-3:1.0/input/input6

$ lsusb |grep -i microsoft
Bus 001 Device 003: ID 045e:075d Microsoft Corp.

Above all, it is priced reasonably.

vital specs:
  • 720p HD Widescreen Video
  • Auto Focus
  • High-precision glass element lens
  • ClearFrame Technology for smooth, detailed video
  • Digital, noise-canceling microphone
more here..


tl;dr: here's a good HD webcam for almost all operating systems. works with skype and other IMs.

Followers


Creative Commons License
This work is licensed under a Creative Commons License.