Recent Comments
Octopi camera commands
sudo v4l2-ctl --set-ctrl=focus_automatic_continuous=0
sudo v4l2-ctl --set-ctrl=focus_absolute=40
sudo v4l2-ctl --set-ctrl=power_line_frequency=1
All camera settings:
User Controls
brightness 0x00980900 (int) : min=0 max=255 step=1 default=128 value=128
contrast 0x00980901 (int) : min=0 max=255 step=1 default=128 value=128
saturation 0x00980902 (int) : min=0 max=255 step=1 default=128 value=128
white_balance_automatic 0x0098090c (bool) : default=1 value=1
gain 0x00980913 (int) : min=0 max=255 step=1 default=0 value=0
power_line_frequency 0x00980918 (menu) : min=0 max=2 default=2 value=2
0: Disabled
1: 50 Hz
2: 60 Hz
white_balance_temperature 0x0098091a (int) : min=2000 max=6500 step=1 default=4000 value=4000 flags=inactive
sharpness 0x0098091b (int) : min=0 max=255 step=1 default=128 value=128
backlight_compensation 0x0098091c (int) : min=0 max=1 step=1 default=0 value=0
Camera Controls
auto_exposure 0x009a0901 (menu) : min=0 max=3 default=3 value=3
1: Manual Mode
3: Aperture Priority Mode
exposure_time_absolute 0x009a0902 (int) : min=3 max=2047 step=1 default=250 value=250 flags=inactive
exposure_dynamic_framerate 0x009a0903 (bool) : default=0 value=1
pan_absolute 0x009a0908 (int) : min=-36000 max=36000 step=3600 default=0 value=0
tilt_absolute 0x009a0909 (int) : min=-36000 max=36000 step=3600 default=0 value=0
focus_absolute 0x009a090a (int) : min=0 max=250 step=5 default=0 value=0 flags=inactive
focus_automatic_continuous 0x009a090c (bool) : default=1 value=1
zoom_absolute 0x009a090d (int) : min=100 max=500 step=1 default=100 value=100
Posted in Computers
Leave a comment
MariaDB
Useful commands
docker exec -it mariadb bash
create database DATABASE_NAME;
grant all privileges on DATABASE_NAME.* TO 'USER_NAME' identified by 'PASSWORD';
flush privileges;
CREATE OR REPLACE TABLE table_name (a int);
docker exec mariadb mariadb-dump -uroot -ppassword --all-databases > /mnt/backup/database/mariadb-$(date +%Y-%m-%d).sql
Posted in Computers
Leave a comment
Home Assistant in a Docker container
I’ve managed to move my current Home Assistant install from a python virtual environment to Docker. This will allow me to upgrade the host operating system or move to a new computer without too much difficulty. The most difficult thing to get working was the networking and it took me a while to work out which containers needed to be on the host network and which could be on the default docker network.
My Docker stack has the following components:
- Mosquitto
- Zigbee2MQTT
- Home Assistant
- NodeRed
- ESPhome
- Lets-encrypt
- My own personal power monitor Perl script
- Watchtower
- MySQL
- phpMyAdmin
Eventually I’ll move my Plex install into the same environment but I’m waiting for me new 8Tb hard drive.
The docker-compose file looks like this:
version: '3.5'
services:
mosquitto:
container_name: mqtt
hostname: mqtt
image: eclipse-mosquitto
labels:
- com.centurylinklabs.watchtower.enable=true
restart: always
ports:
- 1883:1883
- 8883:8883
- 9001:9001
volumes:
- /mnt/data/docker/mosquitto/config:/mosquitto/config
- /mnt/data/docker/mosquitto/data:/mosquitto/data
- /mnt/data/docker/mosquitto/log:/mosquitto/log
- /etc/localtime:/etc/localtime:ro
zigbee2mqtt:
container_name: zigbee2mqtt
hostname: zigbee2mqtt
image: koenkk/zigbee2mqtt:latest
volumes:
- /mnt/data/docker/zigbee2mqtt:/app/data
- /run/udev:/run/udev:ro
- /etc/localtime:/etc/localtime:ro
devices:
- /dev/ttyACM0:/dev/ttyACM0
depends_on:
- mosquitto
restart: always
privileged: true
environment:
- TZ=Australia/Hobart
homeassistant:
container_name: homeassistant
hostname: homeassistant
image: homeassistant/home-assistant
volumes:
- /mnt/data/docker/homeassistant:/config
- /etc/localtime:/etc/localtime:ro
depends_on:
- mosquitto
restart: always
network_mode: host
nodered:
container_name: nodered
hostname: nodered
labels:
- com.centurylinklabs.watchtower.enable=true
network_mode: host
image: nodered/node-red-docker:latest
ports:
- 1880:1880
volumes:
- /mnt/data/docker/node-red:/data
- /etc/localtime:/etc/localtime:ro
depends_on:
- mosquitto
- homeassistant
restart: always
environment:
- TZ=Australia/Hobart
esphome:
container_name: esphome
hostname: esphome
image: esphome/esphome
labels:
- com.centurylinklabs.watchtower.enable=true
volumes:
- /mnt/data/docker/esphome:/config
- /etc/localtime:/etc/localtime:ro
restart: always
network_mode: host
lets-encrypt:
container_name: letsencrypt
hostname: letsencrypt
image: linuxserver/letsencrypt
labels:
- com.centurylinklabs.watchtower.enable=true
restart: always
volumes:
- /mnt/data/docker/homeassistant/letsencrypt:/config
- /etc/localtime:/etc/localtime:ro
ports:
- 433:433
cap_add:
- NET_ADMIN
environment:
- PUID=1000
- PGID=1000
- EMAIL=chris.jennings@riscy.biz
- URL=#####
- VALIDATION=duckdns
- TZ=Australia/Hobart
- DUCKDNSTOKEN=#######
powerlog:
container_name: powerlog
hostname: powerlog
image: powerlog
restart: always
privileged: true
volumes:
- /mnt/data/docker/powerlog:/usr/src/myapp
- /run/udev:/run/udev:ro
- /etc/localtime:/etc/localtime:ro
devices:
- /dev/ttyUSB0:/dev/ttyUSB0
links:
- mysql:mysql
command: perl /usr/src/myapp/powerlog.pl
watchtower:
container_name: watchtower
image: containrrr/watchtower
command: --cleanup --label-enable
restart: always
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
mysql:
container_name: mysql
hostname: mysql
image: mysql
ports:
- 3306:3306
restart: always
volumes:
- /mnt/data/docker/mysql:/var/lib/mysql
- /etc/localtime:/etc/localtime:ro
phpmyadmin:
container_name: phpmyadmin
hostname: phpmyadmin
image: phpmyadmin/phpmyadmin
ports:
- 8080:80
restart: always
volumes:
- /mnt/data/docker/phpmyadmin:/sessions
- /etc/localtime:/etc/localtime:ro
links:
- mysql:db
I have all the docker mounted volumes in a folder /mnt/data/docker. To migrate MySQL I had to export the data and then import it into the new docker container. Copying the files over didn’t work, maybe it was because the MySQL version was incompatible. Best practice usually requires a mysqldump and import instead of just copying the MySQL files over.
Ensure that host names are set correctly so that containers can talk to each other. I also mounted /etc/localtime on each container to ensure the time is correct. Initially my Perl script was time stamping incorrectly because I didn’t map the time through.
Posted in Computers
Leave a comment
Quadcopters
Well a while ago I went to a friends place and there was someone with a quadcopter and they had it hooked up to their laptop and mucking around with it. I thought that looks like great fun so I bought some stuff and made my own. I got all the parts from a combination of RCTimer and HobbyKing but I think there is some good advice required for people who want to do the same.
I chose the RCTimer Spider UAV quadcopter:
RCTimer Spider
This is a good medium size quadcopter which has the capability of having a gimbal added to it later. These are the things I learnt that hopefully will help others to make the right decisions for their first quadcopter.
The XT60 connectors seem to be the new standard for all battery connections. You will need to buy male and female connectors to put on your power distribution board and batteries:
XT60 Connectors
Use 3.5mm connectors between your motors and ESCs, they can come loose but it makes field repairs easier. Just make sure you check ALL connections before each flight.
3.5mm connectors
The screws that come with the RCTimer quadcopter are total rubbish, they have ruined at least 4 hex key drivers. I decided to use screws from Element14 (hex head and torx).
Cheap hex head
Stainless steel hex head
Torx head
You need heatshrink, lots of it and some wire as well so you can extend the length of the ESC cables. You should twist the ESC cables that run back to the power distribution board. This will reduce the magnetic interference. AWG16 should be sufficient for a quadcopter this size:
Black cable
Red cable
Get a range of different sizes
Clear heatshrink is good for protecting some components that you need to see LEDs on (transmitters and receivers etc). I’d go into Jaycar and pick some up that looks the right size. You don’t save that much from getting it off eBay.
Make sure you get some nyloc nuts for your props. It is guaranteed that the nuts will come off otherwise, perhaps midair. Check the tightness of props and nuts before every launch. I got my nyloc nuts from Mitre 10.
These are other parts you should buy from RCTimer as well:
Telemetry kit
Spare frame (you will need it)
LiPo bag
Battery straps
Battery monitor
Spare props (plastic is better to start with, it cuts less OUCH!)
Cable ties, you can never have too many
Power monitor (and power supply for APM)
These instructions are great for putting it all together:
Build photos
This is a great guide with more detailed instructions:
Build tutorial
Now for parts you want from HobbyKing, they have a warehouse in Australia so they can send LiPo batteries by truck. You will want batteries, a charger and a transmitter. Start cheap and then work up to good stuff if you want:
Transmitter
4000mAh 4S battery (get two)
Battery charger
You will need a DC power supply for the battery charger. Jaycar has some, they are more expensive than they should be and I’ve had two blow up (one DOA the other after 3 months). But I couldn’t find anything better in Hobart for the price.
Power supply
In my next post I’ll discuss hints on how to make the build go smoothly.
Posted in Arduino, Drone
Leave a comment
Power monitor update
Well after setting the original system up it only lasted 4 days before the battery went flat. It appears I shouldn’t leave the radio on all the time on the remote node otherwise it chews through the battery very quickly. So I searched through the JeeNode examples to look for low power options.
One of the issues with low power mode for the ATmega is that it doesn’t maintain time accuracy, and the JeeNode doesn’t have a RTC (I’m thinking of getting one). So the best thing I think I can do is to just turn the radio off when not used. The other option is to tone down update rate, I’ll probably look at that after I’ve collected more baseline data.
So after working with the RadioBlip code from the Jeelib examples I came up with this:
#include <Ports.h>
#include <RF12.h>
#include <avr/sleep.h>
#include “kWh.h”#define SEND_MODE 1
volatile bool adcDone;
// for low-noise/-power ADC readouts, we’ll use ADC completion interrupts
ISR(ADC_vect) { adcDone = true; }// this must be defined since we’re using the watchdog for low-power waiting
ISR(WDT_vect) { Sleepy::watchdogEvent(); }static byte vccRead (byte count =4) {
set_sleep_mode(SLEEP_MODE_ADC);
ADMUX = bit(REFS0) | 14; // use VCC as AREF and internal bandgap as input
bitSet(ADCSRA, ADIE);
while (count– > 0) {
adcDone = false;
while (!adcDone)
sleep_mode();
}
bitClear(ADCSRA, ADIE);
// convert ADC readings to fit in one byte, i.e. 20 mV steps:
// 1.0V = 0, 1.8V = 40, 3.3V = 115, 5.0V = 200, 6.0V = 250
return (55U * 1024U) / (ADC + 1) – 50;
}class Port;
Port inputPort(1);
static unsigned long last;
static unsigned long totalblinks;void setup() {
Serial.begin(57600);
inputPort.mode2(INPUT); // Set AIO mode as input
inputPort.digiWrite2(1); // Activate pull-up resistor for AIO//rf12_config();
rf12_config(); // Apparently this is necessary
rf12_easyInit(1); // Send value at most every 3 seconds
last = millis();
rf12_control(0xC040); // set low-battery level to 2.2V i.s.o. 3.1V
rf12_sleep(RF12_SLEEP);
}void loop () {
static boolean ledOn = false; // Variable to indicate LED status
int data = inputPort.anaRead();
rf12_easyPoll();
if (!ledOn && data > 760) { // After testing I found the switching point was 750
ledOn = true;
} else if (ledOn && data < 740) {
ledOn = false;
ledBlink();
totalblinks++;
}
}void ledBlink() {
static int nBlinks = 0;
unsigned long time = millis();
unsigned long interval = time – last;nBlinks++;
if (interval < 0) { // millis() overflow
last = time;
nBlinks = 0;
} else if (interval > 1000) { // 1+ sec passed
// Blinks are 1000 per kWh, or 1 Wh each
// One hour has 3.6M milliseconds
long watts = nBlinks * 1 * 3.6E6 / interval;byte vcc = vccRead();
wattSend(watts,totalblinks,vcc);last = time;
nBlinks = 0;
}
}static void wattSend(long watts,long blinks, long vcc) {
Packet_t packet;
packet.lang = LANG_ELECTRICITY;
packet.mesg = MESG_ELEC_CURRENT;
packet.data = watts;
packet.blinks = blinks;
packet.vcc = vcc;
rf12_sleep(RF12_WAKEUP);
while (!rf12_canSend())
rf12_recvDone();
rf12_sendStart(0, &packet, sizeof packet);
rf12_sendWait(SEND_MODE);
delay(10);
rf12_sleep(RF12_SLEEP);
}
The code also contains parts which measures the Vcc voltage so I can determine when the battery pack is starting to get low.
The receiver code needed to change too. I wanted to include not just power, but the number of blinks and the new Vcc measurement. Here is the new code:
#include <Ports.h>
#include <RF12.h>
#include “kWh.h”void setup() {
Serial.begin(57600);
//rf12_config();
rf12_config();
}void loop() {
if (rf12_recvDone() && rf12_crc == 0 && rf12_len == sizeof (Packet_t)) {
Packet_t packet = *(Packet_t *) rf12_data;if (packet.lang == LANG_ELECTRICITY) {
if (packet.mesg == MESG_ELEC_CURRENT) {
wattShow(packet.data,packet.blinks,packet.vcc);
}
}
}
else {
delay(10);
}
}static void wattShow(long watts,long blinks,long vcc) {
Serial.print(watts);
Serial.print(“,”);
Serial.print(blinks);
Serial.print(“,”);
Serial.println(vcc);
}
I also changed the receiving Perl script to put the data into a MySQL database instead of dumping to a text file.
use strict;
use warnings;
use DBI;
use LWP::Simple;use Win32::SerialPort qw( :STAT 0.19 );
use POSIX qw/strftime/;my $port = Win32::SerialPort->new(‘COM9’);
my $time = time;
# MYSQL CONFIG VARIABLES
my $host = “192.168.1.101”;
my $database = “powermonitor”;
my $tablename = “data”;
my $user = “powermonitor”;
my $pw = [the password];if( ! defined($port) ) {
die(“Can’t open COM9: $^E\n”);
}my $outfd;
open ($outfd, “>>”, “log.txt”) or die “Failed to open output file – $!n”;my $output = select(STDOUT);
$|++;
select($outfd);
$|++;
select $output;$port->initialize();
$port->baudrate(57600);
$port->parity(‘none’);
$port->databits(8);
$port->stopbits(1);
$port->write_settings();
$port->are_match(“\n”);while(1) {
my $char = $port->lookfor();
if ($char) {
$char =~ s/\xd//g;
if($char =~ m/(\d+),(\d+),(\d+)/){
my $dbh = DBI->connect(‘DBI:mysql:’.$database.’;host=’.$host, $user, $pw, { RaiseError => 1});
my $thedate=strftime(‘%Y-%m-%d %H:%M:%S’,localtime);
my $pvdate = strftime(‘%Y%m%d’,localtime);
my $pvtime = strftime(‘%H:%M’,localtime);
my $pvoutput = “get http://pvoutput.org/service/r2/addoutput.jsp?key=[the api number]&sid=[the sid number]&d=$pvdate&pt=$pvtime&ip=$1”;#my $content = get $pvoutput;
my $sth=$dbh->prepare(“INSERT INTO $tablename VALUES(‘$thedate’,’$1′,’$2′,’$3′)”)
or die “Can’t prepare SQL statement: $DBI::errstr\n”;
$sth->execute
or die “Can’t execute SQL statement: $DBI::errstr\n”;
$dbh->disconnect();
}
sleep (1);
}
}
$port->close();
exit(0);
This is the database structure:
— Server version: 5.5.24
—
— Database: `powermonitor`— Table structure for table `data`
—
CREATE TABLE IF NOT EXISTS `data` (
`date` datetime NOT NULL,
`watts` int(11) NOT NULL,
`blinks` int(11) NOT NULL,
`vcc` int(11) NOT NULL,
PRIMARY KEY (`date`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Now finally I wanted to graph the data but I haven’t found a good graph program for the front-end. So I’m using PVoutput, I’m not sure how it will work but I wrote this script to dump usage data for 5 minute intervals. It calculates averages and power usage data for the last 5 minute period, I initiate it using cron.
use strict;
use warnings;
use DBI;
use LWP::Simple;
use LWP::UserAgent;
use HTTP::Request::Common qw(POST);use POSIX qw/strftime/;
# MYSQL CONFIG VARIABLES
my $host = “192.168.1.101”;
my $database = “powermonitor”;
my $tablename = “data”;
my $user = “powermonitor”;
my $pw = “powermonitor”;my $thetime=int(time() / 300) * 300;
my $thetime_5=$thetime-300;
my $thetimestring=strftime(‘%Y-%m-%d %H:%M:%S’,localtime($thetime));
my $thetime_5string=strftime(‘%Y-%m-%d %H:%M:%S’,localtime($thetime_5));my $dbh = DBI->connect(‘DBI:mysql:’.$database.’;host=’.$host, $user, $pw, { RaiseError => 1});
my $sth=$dbh->prepare(“select DATE_FORMAT(MIN(date), ‘%Y-%m-%d’) as day, max(blinks)-min(blinks) as Wh, round(avg(watts)) as Watts, Date_format(Min(date), ‘%H:%i’) as time from data where date > ‘$thetime_5string’ AND date < ‘$thetimestring’ GROUP BY ( 12 * HOUR( date ) + FLOOR( MINUTE( date ) / 5 ));”)
or die “Can’t prepare SQL statement: $DBI::errstr\n”;
$sth->execute
or die “Can’t execute SQL statement: $DBI::errstr\n”;
my @result = $sth->fetchrow_array();
my $pvdate= substr($result[0], 0, 4).substr($result[0],5,2).substr($result[0],8,2);
my $pvwatth= $result[1];
my $pvwatt= $result[2];
my $pvtime= $result[3];
print “$pvdate $pvwatth $pvwatt $pvtime\n”;
$sth->finish();
$dbh->disconnect();my $url = “http://pvoutput.org/service/r2/addstatus.jsp”;
my $ua = LWP::UserAgent->new( timeout => 120 );
$ua->default_header(‘X-Pvoutput-Apikey’=> “edaf9593007119df34f393acbd36703b7476457b”);
$ua->default_header(‘X-Pvoutput-SystemId’=> “12265”);my $req = POST $url, [
d => $pvdate,
t => $pvtime,
v3=> $pvwatth,
v4=> $pvwatt
];my $content = $ua->request($req)->as_string;
print $content;
I’ll post a link to my PVOutput data once it looks ok.
Posted in Arduino, Computers, JeeNode
Leave a comment
Home Power Monitor
I’ve wanted to do a home power monitoring project for some time. I was using a Clipsal Cent-a-meter a while ago to track power usage when I lived in Victoria but I didn’t have much luck with the newer model. It didn’t have a good range and seemed to lose connection between the base and sending unit. So I thought I would go DIY.
I looked at lots of different systems including PIC microcontrollers but I decided to give the JeeNode and JeeLink combination a try. It seemed like a good choice for the following reasons:
- Arduino compatible
- Low power (3.3V)
- Integrated wireless
- Easy interfacing with a PC (JeeLink)
- Expandable
In our house we have a smart meter which has a blinking red light for every Watt Hour of power used. So by measuring the length of the pulse you can have an indication of instantaneous power usage. This saves using clip on CTs which need to be installed by an electrician. The other advantage of using the smart meter is that this is the actual power usage I am being charged for. The only problem with this is that in Tasmania we have two different tariffs, one for normal usage and one for heating (cheaper). My system doesn’t take into account the tariffs and can only measure total usage. It also could not be used with electricity generation because the smart meter only blinks for imported power, if you are generating then the actual power usage will be higher than the imported.
So I went to Modern Device and ordered the following:
The JeeNode will be used to monitor the blinking light, which will send data to the JeeLink (plugged into my HTPC running Ubuntu) and the USB BUB II is required for programming the JeeNode.
I soldered the JeeNode and JeeLink up when they arrived on the coffee table while watching something mindless on TV. It had been a while since I had done proper soldering and with a good temperature controlled iron it was pretty easy. I should get some good quality solder though!
I tested the JeeNode and JeeLink and they worked first time. Now to look for some code.
I ended up using this code http://jeelabs.net/projects/cafe/wiki/Electricity_consumption_meter I modified it to suit the 1Wh blinks and using a LDR.
Here is my version of the transmitter code:
// Reading Comparator input #include <Ports.h> #include <RF12.h> #include "kWh.h" class Port; Port inputPort(1); static unsigned long last; void setup() { inputPort.mode2(INPUT); // Set AIO mode as input inputPort.digiWrite2(1); // Activate pull-up resistor for AIO rf12_config(); rf12_config(); // Apparently this is necessary rf12_easyInit(3); // Send value at most every 3 seconds last = millis(); } void loop () { static boolean ledOn = false; // Variable to indicate LED status int data = inputPort.anaRead(); rf12_easyPoll(); if (!ledOn && data > 750) { // After testing I found the switching point was 750 ledOn = true; } else if (ledOn && data < 750) { ledOn = false; ledBlink(); } } void ledBlink() { static int nBlinks = 0; unsigned long time = millis(); unsigned long interval = time - last; nBlinks++; if (interval < 0) { // millis() overflow last = time; nBlinks = 0; } else if (interval > 1000) { // 1+ sec passed // Blinks are 1000 per kWh, or 1 Wh each // One hour has 3.6M milliseconds long watts = nBlinks * 1 * 3.6E6 / interval; wattSend(watts); last = time; nBlinks = 0; } } static void wattSend(long watts) { Packet_t packet; packet.lang = LANG_ELECTRICITY; packet.mesg = MESG_ELEC_CURRENT; packet.data = watts; rf12_easySend(&packet, sizeof packet); }
And here is my version of the JeeLink receiver code
#include <Ports.h> #include <RF12.h> #include "kWh.h" void setup() { Serial.begin(57600); rf12_config(); rf12_config(); } void loop() { if (rf12_recvDone() && rf12_crc == 0 && rf12_len == sizeof (Packet_t)) { Packet_t packet = *(Packet_t *) rf12_data; if (packet.lang == LANG_ELECTRICITY) { if (packet.mesg == MESG_ELEC_CURRENT) { wattShow(packet.data); } } } else { delay(10); } } static void wattShow(long watts) { Serial.print("Usage: "); Serial.print(watts); Serial.println(" W"); }
I connected 4 AA NiMh batteries as the power supply to the JeeNode and sat it on top of the smart meter. The LDR is connected between the AI pin and ground of one of the ports.
Pretty isn’t it?
The LDR is right next to the LED
The JeeLink is connected to my PC running Mythbuntu, which is my HTPC.
Ubuntu is running Perl code to read the serial port and dump it to a file which timestamps the incoming data.
use strict; use warnings; use Device::SerialPort; use POSIX qw/strftime/; my $port = Device::SerialPort->new('/dev/ttyUSB0'); my $time = time; if( ! defined($port) ) { die("Can't open /dev/ttyUSB0 $^E\n"); } my $outfd; open ($outfd, ">>", "log.txt") or die "Failed to open output file - $!n"; my $output = select(STDOUT); $|++; select($outfd); $|++; select $output; $port->baudrate(57600); $port->parity('none'); $port->databits(8); $port->stopbits(1); $port->write_settings(); $port->are_match("\n"); while(1) { my $char = $port->lookfor(); if ($char) { $char =~ s/\xd//g; my ($watts)= $char =~ /(\d+) W/; print strftime('%d-%m-%Y %H:%M:%S',localtime); print ",$watts\n"; print $outfd strftime('%d-%b-%Y %H:%M:%S',localtime); print $outfd ",$watts\n"; } } $port->close(); exit(0);
It works and the next step is to make a nice graphing utility. I’ll post again when I have some pretty graphs. At the moment I’m just dumping all the data into Excel.
Posted in Arduino, Computers, JeeNode, Personal
2 Comments
MythTV Dynamic Range Equalisation
I found this on the net recently after trying to work out how to fix problems watching movies with very low dialogue volume levels and loud sound effects:
http://blog.trenchcoatsoft.com/2009/11/mythtv_volume_leveling.html
After following the instructions I am now able to watch all my movies using MythTV without having to change the volume once, and Claire doesn’t complain about the movie being too loud when she is in bed. Win win!
P.S. Saved from the Internet Archive because the original page is dead
MythTV: Volume Leveling
Back at the other end of this year, I decided to retire my aging TiVo, and become a full-time user of the magic that is MythTV. For those of you who don’t know, MythTV is a software package you can run on a computer (Linux based. I won’t swear that there isn’t a Windows port, but of this, I know little) which does more or less what a Digital Video Recorder does — it can record TV to its hard drive so you can watch it whenever you like. But because you’ve got a whole computer which is under your control, it can also do, well, anything else you want. For my purposes, the most useful thing that it does is to act as a sort of video jukebox: I can back up all my DVDs to a network hard drive, and thereby avoid all the hassle of (a) having to keep piles of DVDs in the living room, (b) risking scratches, and (c) dealing with temperamental DVD players. Another of its nice features is that, with an add-on called MythNetTV, you can subscribe to video podcasts via MythTV, and it will deliver new episodes to you just as if they’d been broadcast over the air, allowing you to watch grainy, low-resolution YouTube quality video of cats doing amusing things on your 40 inch HDTV. I’ve long found it ironic that as TVs get bigger and resolutions increase, we’re increasingly willing to huddle around a laptop monitor to watch a 320×200 viral video. Well, suck it, losers, because I’m watching The Spoony Experiment and The Nostalgia Critic on the big screen.
Now, like most Linux projects, it’s not all sunshine. I’ve got two cheap TV tuner dongles, which don’t work with it (There’s several very nice tuners which work with it, but I really wanted just a cheap one to use as a secondary tuner). And the usability is not nearly as polished as, say, TiVo (That said, it’s miles beyond most cable box DVRs in the UI department). There’s a few annoyances that I have yet to be able to overcome (The size at which subtitles render is hard-coded, which means that it displays at a size which was plainly selected for a Standard Definition screen, making it slightly microsocopic at 1080p), but, like I said, it’s a whole computer, and you can bring to bear all that implies.
I’d been meaning for some time to write a series of articles about the cool things I’ve written to bend the Mythtv to my will, but actually banging any of my hacks into a presentable state has required a bit more time than I’ve been willing to invest. But this week, I found something so handy and so elegant that I thought it was time to share it.
So, MythTV trick Number One:
One problem with playing back video from various disparate sources is the volume level. You know how when you’re watching regular old-fashioned TV, more often than not, the commercials will be about a million decibels louder than the show? The volume will be different from one channel to the next. When you’re also downloading New Media from The Intertubes, those too will be at radically different levels from TV, and from each other. DVDs are usually at a much lower level than TV (I think this may be caused by the downmix from 5.1 to stereo). And if, say, you’re watching a third generation rip from a grainy VHS of a film so rare that no one involved in it will even admit to having heard of it, you’re talking borderline inaudible.
With months of training, I’ve got Leah to the point where she’ll actually give me a fair chance to reach the remote control and turn the volume down before she yells at me to turn it down the instant the sound starts, but it’s still not really an optimal solution for me to keep having to adjust the volume from one video to the next.
If you are a modern person who keeps all your music in digital format, you may be familiar with the concept of volume normalizing, which analyses a whole song and works out how to adjust the overall volume to the song so that you don’t blow out your ear drums if Shuffle Play puts a John Tesh song right after one by Alice in Chains (Which is not to say that you don’t deserve deafness for your taste in music).
But the tools for doing this to video are less mature, and besides, you might be willing to spend 2 minutes preprocessing a 4 minute song you’re going to keep for the rest of your life, but I’m not willing to spend 30 minutes processing an episode of Stargate Universe which I’m going to delete as soon as I’ve finished watching it.
As it turns out, though, since you’re running a whole computer, and it’s Linux, the Magical World Where You Can Basically Do Anything You Want So Long As You’re Willing To Carve It From the Solid Granite of the OS With Your Bare Hands, it’s possible to just order your sound card to do that normalization for you as it plays — in this case, it’s called Compression and Limiting.
I could just about muddle through the science of how it works, but probably not well enough to explain it to anyone in detail. The general gist of it is that a “compressor” squishes audio such that it reduces the difference between the loudest sounds and the softest. When a sound is louder than some threshhold, it reduces the volume, but it does it in a very smooth way that sounds good. This is something radio stations do so that you can turn the volume up loud enough to hear the soft bits without blowing out your speakers for the loud bits. A “limiter” is the same basic process, but it’s much more powerful and lacks the subtlety of a lower-rate compressor. Basically, the purpose of the compressor is to make the audio all “fit” within a certain range of loud-to-soft, and then the limiter boosts the gain (ie. “Turns the volume up”) while keeping it from exceeding a certain threshold.
In Linux’s ALSA sound system, you can create plugins which (long story short) basically act like virtual sound devices. You tell an application to use that sound device, and any audio the application tries to put out will be sent through the plugin before it’s turned into sweet delicious audio. Here’s an audio compressor that I threw together based on some stuff I found on the ALSA wiki:
pcm.ladcomp { type plug slave.pcm "ladcomp_compressor"; } pcm.ladcomp_compressor { type ladspa slave.pcm "ladcomp_limiter"; path "/usr/lib/ladspa"; plugins [ { label dysonCompress input { controls [0 1 0.5 0.99] } } ] } pcm.ladcomp_limiter { type ladspa slave.pcm "default"; path "/usr/lib/ladspa"; plugins [ { label fastLookaheadLimiter input { controls [ 15 0 0.8 ] } } ] }
This code can be put in your /etc/asound.conf
, then just tell MythTV to use the sound device ALSA:ladcomp (It’s under Utilities / Setup -> Setup -> General if you’re using the default menus. I’ve hacked mine up a bit, so it took me longer to find it). It should have defaulted to something like ALSA:default. Restart MythFrontend, and voila: all your audio should play at around and about the same level. To use this, you’ll need the ladspa plugins. If your MythTV is running on Ubuntu Linux (I use Mythbuntu, a version of Ubuntu oriented toward MythTV (For the non-Linux experienced, you can basically run any Linux software on any Linux box. The major difference between various Linux distributions is basically which software it installs by default, as opposed to which ones you have to download and install on your own. Ubuntu is a distribution which takes the radical step of assuming that its users may include actual human beings and might want to spend more time actually using their computer than assembling it.)), you can get them by running “sudo apt-get install ladspa-sdk swh-plugins”.
If you want this trick to apply to other applications, you can tell them to use ladcomp as their audio device too. For instance, with mplayer, try mplayer -ao alsa:device=ladcomp
.
If you want to do some fine tuning, you can try changing that 15 to other numbers to change the range for the final audio (You have to restart MythFrontend before the changes will be honored). I haven’t found quite the right setting for me personally yet — 15 is a bit higher than I want, I think, since it makes the “comfortable” position on my stereo’s volume dial around 6 out of 30 — I think somewhere in the 10-15 range would be better. But, at least for me, it does put MythTV in about the same volume zone as the Nintendo Wii, so I’m not racing to turn the volume down when we turn the game consoles on.
So, with any luck, and a little bit of work, you too can bend the sound system to your will, and watch whatever you like without fear of getting yelled at by your fiancee for having the TV turned up too high.
New addition
On Thursday the 15th of September we had a new addition to the family. Ada Jean Jennings.
All of the family are happy and well, and I think Jasper is enjoying having a new baby sister.
Converting from Drupal to WordPress
Well after looking at this site I modified the existing Drupal database to remove all references to the img_assist module. What a pain in the arse that module is, there is no way to easily migrate away from it so I wrote a Perl script to change the tags to <img> tags instead.
use DBI;
my $databasename = “drupal”;
my $servername = “192.168.1.1”;
my $port = “3306”;
my $dsn = “dbi:mysql:$databasename:$servername:$port”;# set the user and password
my $user = ‘xxxxx’;
my $pass = ‘xxxxx’;# now connect and get a database handle
my $dbh = DBI->connect($dsn, $user, $pass)
or die “Can’t connect to the DB: $DBI::errstr\n”;my $query1=”Select * from node_revisions LEFT JOIN node ON (node_revisions.nid=node.nid) WHERE node.type=’blog'”;
my $action = $dbh->prepare($query1);
$action->execute();while (@row = $action->fetchrow()) {
$body=$row[4];
while ($body=~m/\[img_assist\|nid=(\d+)\|.*\]/){
$vid=$row[1];
$width=450;
my $query2=”Select filepath FROM files LEFT JOIN image ON (files.fid=image.fid) WHERE nid=$1 AND image_size=’_original'”;
my $action2= $dbh->prepare($query2);
$action2->execute();
@row2 = $action2->fetchrow();
$body=~s/\[img_assist\|nid=(\d+)\|.*\]/<p style=”text-align: center;”><img src=”$row2[0]” width=”$width”><\/p>/;($escapequotes=$body)=~s/\’/\’\’/g;
my $query3=”UPDATE node_revisions SET body=’$escapequotes’,teaser=’$escapequotes’ WHERE vid=$vid”;
print “$vid\n”;
my $action3 = $dbh->prepare($query3);
$action3->execute();
}
while ($body=~m/\<img.*src\=\”\?q\=image\/view\/(\d+).*\>/){$vid=$row[1];
$nid=$1;
$width=450;
my $query2=”Select filepath FROM files LEFT JOIN image ON (files.fid=image.fid) WHERE nid=$nid AND image_size=’_original'”;
my $action2= $dbh->prepare($query2);
$action2->execute();
@row2 = $action2->fetchrow();
$body=~s/\<img.*src\=\”\?q\=image\/view\/$nid.*\>/<img src=”$row2[0]” width=”450″>/;($escapequotes=$body)=~s/\’/\’\’/g;
my $query3=”UPDATE node_revisions SET body=’$escapequotes’,teaser=’$escapequotes’ WHERE vid=$vid”;
print “$vid\n”;
my $action3 = $dbh->prepare($query3);
$action3->execute();
}
}
print “Finished\n”;
It’s a bit ugly but works, I had to go through twice because some of the tags were using older versions of img_assist, again what a dumb idea. If only the entries had been migrated as the module was updated. Anyway that was done. Then I modified the import script linked at the start of this post to this:
# This set of instructions was updated by 9seeds.com based on
# a post by Mike Smullin at mikesmullin.com
#
# Mike’s original post can be found here:
# http://www.mikesmullin.com/development/migrate-convert-import-drupal-5-to-wordpress-27/
#
# 9seeds’ updated post can be found here:
# http://9seeds.com/news/drupal-to-wordpress-migration# Clear all existing WordPress content
TRUNCATE TABLE wordpress.wp_comments;
TRUNCATE TABLE wordpress.wp_links;
TRUNCATE TABLE wordpress.wp_postmeta;
TRUNCATE TABLE wordpress.wp_posts;
TRUNCATE TABLE wordpress.wp_term_relationships;
TRUNCATE TABLE wordpress.wp_term_taxonomy;
TRUNCATE TABLE wordpress.wp_terms;# Create Categories
INSERT INTO wordpress.wp_terms (term_id, `name`, slug, term_group)
SELECT
d.tid, d.name, REPLACE(LOWER(d.name), ‘ ‘, ‘_’), 0
FROM drupal.term_data d
INNER JOIN drupal.term_hierarchy h
USING(tid);# Add Taxonomies
INSERT INTO wordpress.wp_term_taxonomy (term_id, taxonomy, description, parent)
SELECT
d.tid `term_id`,
‘category’ `taxonomy`,
d.description `description`,
h.parent `parent`
FROM drupal.term_data d
INNER JOIN drupal.term_hierarchy h
USING(tid);# Import posts/pages
INSERT INTO wordpress.wp_posts (id, post_date, post_content, post_title, post_excerpt, post_name, post_modified, post_type, `post_status`)
SELECT DISTINCT
n.nid `id`,
FROM_UNIXTIME(n.created) `post_date`,
r.body `post_content`,
n.title `post_title`,
r.teaser `post_excerpt`,
IF(SUBSTR(a.dst, 11, 1) = ‘/’, SUBSTR(a.dst, 12), a.dst) `post_name`,
FROM_UNIXTIME(n.changed) `post_modified`,
n.type `post_type`,
IF(n.status = 1, ‘publish’, ‘private’) `post_status`
FROM drupal.node n
INNER JOIN drupal.node_revisions r
USING(vid)
LEFT OUTER JOIN drupal.url_alias a
ON a.src = CONCAT(‘node/’, n.nid)
WHERE n.type IN (‘blog’);# Turn articles in to posts
update wordpress.wp_posts set post_type=’post’ where post_type=’blog’;# Add post to category relationships
INSERT INTO wordpress.wp_term_relationships (object_id, term_taxonomy_id)
SELECT nid, tid FROM drupal.term_node;# Update category count
UPDATE wordpress.wp_term_taxonomy tt
SET `count` = (
SELECT COUNT(tr.object_id)
FROM wordpress.wp_term_relationships tr
WHERE tr.term_taxonomy_id = tt.term_taxonomy_id);# Import comments
INSERT INTO wordpress.wp_comments (comment_post_ID, comment_date, comment_content, comment_parent, comment_author, comment_author_email, comment_author_url, comment_approved)
SELECT nid, FROM_UNIXTIME(timestamp), comment, thread, name, mail, homepage, status FROM drupal.comments;# Update comment counts
use wordpress;
UPDATE `wp_posts` SET `comment_count` = (SELECT COUNT(`comment_post_id`) FROM `wp_comments` WHERE `wp_posts`.`id` = `wp_comments`.`comment_post_id`);# Fix breaks in post content
UPDATE wordpress.wp_posts SET post_content = REPLACE(post_content, ”, ”);# fix images in post content
UPDATE wordpress.wp_posts SET post_content = REPLACE(post_content, ‘”files/’, ‘”wp-content/uploads/’);
UPDATE wordpress.wp_posts SET post_content = REPLACE(post_content, ‘”/files’, ‘”wp-content/uploads/’);
UPDATE wordpress.wp_posts SET post_content = REPLACE(post_content, ‘”images/’, ‘”wp-content/uploads/images/’);# fix images in post_excerpt content
UPDATE wordpress.wp_posts SET post_excerpt = REPLACE(post_excerpt, ‘”files/’, ‘”wp-content/uploads/’);
UPDATE wordpress.wp_posts SET post_excerpt = REPLACE(post_excerpt, ‘”/files’, ‘”wp-content/uploads/’);
UPDATE wordpress.wp_posts SET post_excerpt = REPLACE(post_excerpt, ‘”images/’, ‘”wp-content/uploads/images/’);# fix links
UPDATE wordpress.wp_posts SET post_content = REPLACE(post_content, ‘href=”?q=node/’, ‘href=”?p=’);
UPDATE wordpress.wp_posts SET post_excerpt = REPLACE(post_excerpt, ‘href=”?q=node/’, ‘href=”?p=’);#remove dud characters
UPDATE wordpress.wp_posts SET post_content = REPLACE(post_content, ‘Â’, ”);
UPDATE wordpress.wp_posts SET post_excerpt = REPLACE(post_excerpt, ‘Â’, ”);
I added some extra replacements which fixed any links to old nodes and I also had to include post type ‘blog’ so that all the entries got moved over. I don’t care about pages or stories.
This is by no means a completed product, it works just enough for me to have a WordPress blog running without too much effort. The migration went better on littlefishcreations.com this blog still has almost all the images failing. I might get around to fixing it one day.