Resizing partitions within an image file

I wanted to backup a 32GB SD Card to a 16GB SD Card. The 32 GB SD Card only contained 10GB of data so it should be possible.

I started by creating an image file of the 32GB SD Card

$ dd if=/dev/sdb of=SDCard.img

To complicate things, the original SD Card had 2 partitions, a FAT32 boot partition and and ext4 root partition. The key to making this work would be to shrink the root partition. I used this guide to accomplish the task.

Create a new loopback device

$ sudo losetup -f

Attach the image file to the new loopback device

$ sudo losetup /dev/loop9 SDCard.img

Run GParted on the attached loopback device

$ sudo gparted /dev/loop0

Use GParted GUI to resize the partitions to suit your need

Then disconnect the loopback device

$ sudo losetup -d /dev/loop0

Use Fdisk to findout the last used sector on the image 

$ fdisk -l SDCard.img

Disk SDCardBackup.img: 29.6 GiB, 31724666880 bytes, 61962240 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x32e07f87

Device Boot Start End Sectors Size Id Type
SDCardBackup.img1 8192 93236 85045 41.5M c W95 FAT32 (LBA)
SDCardBackup.img2 94208 20574207 20480000 9.8G 83 Linux

Calculate the last byte and use truncate to chop the file down to size!

$ truncate --size=$[(20574207+1)*512] SDCardBackup.img

Hope this helps others in the same predicament!

Fast boot with Raspberry Pi

I am hoping to have a raspberry pi power a wildlife camera. This camera will have to rely on battery and solar power. As a result, it would be beneficial if the camera was off when no wildlife is present. To aid in this regard, I hope to use a motion sensor that can trigger the raspberry pi to turn on and take a picture. For this to work, the time from motion detection to picture snap is heavily influenced by the boot time of the raspberry pi. Here is a video of what I've been able to accomplish:

I am starting with the stock Raspbian Stretch Lite distribution on a Pi 3B. Boot times out of the box are on the order of 1 minute. Boot time is influenced by the following:

1. Hardware
2. Bootloader
3. Kernel
4. Userspace

The Raspberry Pi hardware and bootloader are essentially out of my control. There was an effort to open source the boot loader, however the proprietary binary blob is the only reasonable option at this point. The Hardware and bootloader take approximately a minimum of 1.5-2 seconds to run. This is explained in an excellent post on the Raspberry Pi Forums. The author tested boot times with various minimal boot loaders. The fastest any code could be run on the ARM processor was around 1.5 seconds.

I was able to get the kernel and userspace boot times down to about 0.6 second and 0.8 seconds respectively. As a result my total boot time is on the order of 3.5 to 4 seconds (from power on to picture taken).

To be able to control the Raspberry Pi without SSH, I used serial (UART) communications. See my previous post to learn how.

I reduced the kernel and userspace boot times by doing the following (in order highest yield to lowest yield):

1. Editing the /boot/config.txt with the following changes:

# Disable the rainbow splash screen
disable_splash=1

# Disable bluetooth
dtoverlay=pi3-disable-bt

#Disable Wifi
dtoverlay=pi3-disable-wifi
 
# Overclock the SD Card from 50 to 100MHz
# This can only be done with at least a UHS Class 1 card
dtoverlay=sdtweak,overclock_50=100
 
# Set the bootloader delay to 0 seconds. The default is 1s if not specified.
boot_delay=0

# Overclock the raspberry pi. This voids its warranty. Make sure you have a good power supply.
force_turbo=1

2. Make the kernel output less verbose by adding the "quiet" flag to the kernel command line in file /boot/cmdline.txt 

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=32e07f87-02 rootfstype=ext4 elevator=deadline fsck.repair=yes quiet rootwait

3. Use systemd-analyze blame, systemd-analyze critical-chain to disable services I didn't need

sudo systemctl disable dhcpcd.service
sudo systemctl disable networking.service
sudo systemctl disable ssh.service
sudo systemctl disable ntp.service
sudo systemctl disable dphys-swapfile.service
sudo systemctl disable keyboard-setup.service
sudo systemctl disable apt-daily.service
sudo systemctl disable wifi-country.service
sudo systemctl disable hciuart.service
sudo systemctl disable raspi-config.service
sudo systemctl disable avahi-daemon.service
sudo systemctl disable triggerhappy.service

See the references below to learn about a primer on systemd and the new linux init system to learn about how to interpret and write the above services.

4. Add a service that runs the code you would like to run as fast as possible. For example if you wanted to add a service called "1ylapse", create the following file: /etc/systemd/system/1ylapse.service

[Unit]
Description=Starts 1 Year Lapse Service

[Service]
ExecStart=/home/pi/foo.sh
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=piservice
User=pi
Group=pi
WorkingDirectory=/root/1ylapse/

[Install]
WantedBy=basic.target

5. Analyze the kernel for unnecessary work being done at boot.

To do this you need to compile your kernel with "CONFIG_PRINTK_TIME" and "CONFIG_KALLSYMS". This should be enabled on the default raspberry pi kernel. This allows you to add "initcall_debug" to the kernel command line. The kernel will now output start and end time information for every init call. You can use "bootgraph.pl" which is included with the linux kernel to analyze the output of dmesg.

On the raspberry pi:

$ dmesg > boot.log

On the cross-compile host:

$ linux/scripts/bootgraph.pl boot.log > boot.sv

This will output an graph of what is taking the most time when initializing the kernel. I noticed that a routine used by the USB driver was taking around 0.3s. I don't need USB for my project so I disabled USB support when re-compiling the  kernel (see below). This saved around 0.3s.

6. Re-compile the Linux kernel

Remove stuff that is wasting time during initialization. I used the guide from the Raspberry Pi Foundation to learn how to re-compile the kernel.

7. Use LZO compression for kernel

When compiling the Linux kernel, select "LZO" compression instead of "GZip". This saved around 0.3s.

8. Don't re-mount the /boot partition

Edit the /etc/fstab file and comment out the line that re-mounts the /boot partition. This saved around 0.2s.


The final systemd-analyze shows:

Startup finished in 669ms (kernel) + 1.225s (userspace) = 1.894s

It should be noted that my camera service starts before systemd is finished initializing. You can find out when your service starts by using systemd-analyze crritical-chain. You can see below that my service starts at 836ms after the kernel is finished initializing, rather than the total of 1.225s.


$ systemd-analyze critical-chain 1ylapse.service
1ylapse.service @836ms
└─basic.target @832ms
  └─sockets.target @832ms
    └─dbus.socket @831ms
      └─sysinit.target @826ms
        └─systemd-update-utmp.service @784ms +41ms
          └─systemd-tmpfiles-setup.service @748ms +33ms
            └─systemd-journal-flush.service @658ms +87ms
              └─systemd-remount-fs.service @585ms +64ms
                └─systemd-fsck-root.service @444ms +137ms
                  └─systemd-journald.socket @433ms
                    └─-.slice @376ms

9. Remove plymouth to disable systemd init messages

sudo apt-get purge --remove plymouth

I haven't seen anyone boot a raspberry pi faster than this using full Raspbian. Bare metal is obviously faster however. However having full Raspbian available at this boot up speed is a good compromise.

Things that failed to improve boot time included making the root partition read only.


Hopefully this helps others in my predicament.

References:

1. Presentation by Jan Altenberg on booting linux in less than 1 second. Powerpoint here. Youtube of presentation here.
2. Excellent powerpoint on boot time optimization using a beagle bone as a prototype here.
3. Excellent powerpoint on speeding up raspberry pi boot time here.
4. Excellent primer on systemd-anzlyze.
5. Good stackoverflow question on using sytemd-analyze. 

Serial Communications with Raspberry Pi

Running a headless raspberry pi can be challenging. Until now I've been using SSH to control my raspberry pi. This works well if your raspberry pi has wifi (namely the 3 and Zero W). However, i'm hoping to use the plain Raspbery Pi Zero for my current project which has no wifi built-in. Thus I needed a means to control and debug my Pi without wifi. This is where serial communication is beneficial.

I purchased a USB<->Serial adapter/cable from Buyapi.ca. It is based on the PL2303HX chipset. It uses 3.3v to drive the RX and TX lines which is compatible with the Raspberry Pi.

The connections are:

1. Red - GPIO2 (5V)
2. Black - GPIO6 (Ground)
3. White (RX into USB) - GPIO8 (TXD from Raspberry Pi)
4. Green (TX out of USB) - GPIO10 (RXD to Raspberry Pi)

Please note that the above connection will cause the raspberry pi to draw power from the USB<->Serial cable. This is usually enough for a Pi Zero, however will cause the Pi 3 to brownout. To handle this, supply the Pi 3 with an external power supply and disconnect the red (5V) wire. Make sure to keep ground (black) connected, however, to prevent ground loops.

My Raspberry Pi did not have the default Raspbian Linux console (the console that prints on a screen if you have one) broadcasting on the serial interface. To enable it you can run:

sudo raspi-config

Look for "Interfacing options", then option P6, Serial, and select "Yes". To use the serial console for other purposes you can set it to "No". For more information see the documentation on the raspberry pi website.

To use the console, fire up a terminal in Ubuntu and type:

sudo screen /dev/ttyUSB0 115200

The device "/dev/ttyUSB0" maybe different depending on your host kernel. Just look for something in "/dev" that looks similar. You can double check if it is correct by removing the PL2303HX device and see if the device you suspect disappears from the list in "/dev".

PiJuice - Mobile Power for the Raspberry Pi (First Impressions)

Ever since the Raspberry Pi came out I've had an idea to make a time lapse camera that would take pictures over weeks, months, or even a whole year. For this purpose I needed a mobile power management solution for the Raspberry Pi.

Over two years ago I supported a Kickstarter campaign for the PiJuice. The PiJuice was delayed multiple times for various technical and non-technical reasons that are nicely outlined in this review. After all that time my PiJuice has finally arrived and now I'll outline my first impressions!

Top View with Raspberry Pi Zero W connected

Bottom View with Raspberry Pi Zero W connected

The PiJuice is designed for the Raspberry Pi A+, B+, 2B, 3B, and is compatible with the Zero, and Zero Wireless. I am testing it with the Zero Wireless and 3B right now. Documentation is somewhat lacking right now. Your best bet is to look at the PiJuice's GitHub Page. I found the hardware page the most useful to get an overview of how it works.

The PiJuice product page touts many features. The features I think most people will care about are the following:

• Charging from weak sources:
• Batteries can be charged from weak and unreliable sources such as solar and wind because of a feature called Dynamic Power Management (DPM)
• Low power mode
• Real Time Clock
• Low power deep-sleep state with wake on interrupt/calendar event
• Hardware watchdog timer
• Software
• Python based API
• Python based system daemon
• Nice to have
• Programmable multi-colored RGB led (x2) and buttons (x3)

I bolded the first feature because I feel like this is what sets the PiJuice apart from other competitors.

I will have many more reactions regarding the PiJuice in future posts. However I'll leave you with this graph which shows the discharge curve of the BP7X 1820mAh battery provided with the PiJuice. I was able to get nearly 7 hours of usable time from it on the Pi 3B! I had to make sure to throttle down the Pi to achieve this but it is possible.

To get the Pi 3B to use less power I performed the following:

#!/bin/bash

echo 0 | sudo tee /sys/devices/platform/soc/3f980000.usb/buspower >/dev/null

sudo tvservice --off

echo gpio | sudo tee /sys/class/leds/led1/trigger

echo 0 | sudo tee /sys/class/leds/led1/brightness

The first line powers off the USB/LAN expansion chip. The second line powers off HDMI, and the last lines turn off the power indication LED. Doing all of these things I can reduce current consumption from 500-600mA to 200-300mA!

I'll be posting more about how to use it with the Solar Panel, as well as my reactions to the available software API including using the real time clock and more!

Raspberry Pi Camera Module HD Time-lapse

So a bunch of camera modules were made available last week and so I finally went ahead and ordered one for my Raspberry Pi. It arrived in less than a week! However the shipping and tax brought the cost up to $36 Canadian from $25 retail. Oh well, at least I have it now!

The obvious first project for the camera module was wireless, battery powered timelapse! I did a similar project last summer with a webcam. However, the camera module is a million times better! It does 1080p video at 30FPS. The quality of the stills is also much better, and its all hardware accelerated!

I used this guide for making the time-lapse. The result is beautiful:

 I was able to capture the sun setting over my cul-de-sac. I think the moving shadows are really cool.

The setup is very ghetto. I used a makeshift lego case, and a cheap 11200mAh battery from eBay.

In my next post I will post how to capture live streaming HD video from the Raspberry Pi. Until then keep tinkering!

Wireless, Battery Powered, Time-lapse Video with a Raspberry Pi

I was looking for cool projects for my Raspberry Pi and came across a really cool project by Jeremy Blythe: Battery powered, Wireless, Motion detecting Raspberry Pi.

It inspired me to build something similar, but instead of using the Pi as a security camera, I wanted to do some outdoor time-lapse photography. Here is an animated GIF that the Pi produced. I'm working on longer and higher resolution versions.

And here is the device itself, the "Lapse-Pi":

I used the following hardware to build it:

1. Microsoft Lifecam Show 2.0MP - $13 from NCIX
2. Power Bank 12000mAh External Battery Charger USB - $25 from eBay
3. Ultra Mini USB Wireless LAN Adapter 802.11n - $7 from Monoprice.com
4. Lego Bricks, Raspberry Pi, SD Card etc.

Below is a close-up of the battery. I've let my Raspberry Pi idle on it for around 24 hours before the juice runs out.

I've had lots of trouble getting webcams working on Linux, fortunately, the Microsoft Lifecam Show 2.0MP is UVC compatible. It was basically plug and play. I highly recommend getting a UVC compatible webcam. It will be almost guaranteed to work with your Raspberry Pi. See the Linux UVC driver webpage for compatible webcams.

On the software side I am using:

1. Official Raspbian Wheezy Image, "2012-07-15-wheezy-raspbian.zip"
2. Streamer
3. Imagemagick Convert
4. sshfs

Step by step instructions:

1. Install Streamer and SSHFS

sudo apt-get install streamer
sudo apt-get install sshfs

2. Mount your network share to store the time lapse images wirelessly. I use SSHFS, however you can use SAMBA, NFS, or whatever else you want.

mkdir server_mount
sshfs username@server:/home/username /home/pi/server_mount

3. Run streamer to capture the frames.

mkdir -p server_mount/timelapse
cd server_mount/timelapse/
streamer -o 0000.jpeg -s 352x248 -j 100 -t 00:20:00 -r 0.1 -c /dev/video0

Some notes on the command line parameters:

• "-o 0000.jpeg" specifies the filename format for the individual frames
• "-s 352x248" specifies the resolution. I think the Pi can handle much higher than this.
• "-j 100" is the JPEG quality"
• "-t 00:20:00" instructs it to capture for 20 minutes
• "-r 0.1" indicates the frame rate of 0.1 frames/second
• "-c /dev/video0" specifies the video device. It should be the same for you.

4. On the server, convert the captured frames to an animated GIF using Imagemagick's "convert" utility

convert -delay 5 *.jpeg -loop 0 animated.gif

Future Plans:

I have recently purchased a 4.5V OCV / 450mA SCC solar panel from eBay for about $20. I am in the process of figuring out how to charge the battery with it. If that happens then I can take time-lapses over multiple weeks and months rather than just 1 day.

I also will be trying higher resolution time-lapses. The Lifecam Show is capable of 2MP or 1600x1200. And hopefully in October the official Raspberry Pi camera module will come out that will be capable of 5MP or 1080p at 30fps!

Raspberry Pi Camera Module Information

I am very excited about the Raspberry Pi Camera module. So excited that I've compiled the latest information I could find on the camera module.

• The camera will be available in 3-4 months (October, 2012) [1]
• The camera will cost $20-$25 USD [1]
• Specifications [2]: 
• Rolling shutter, 1/4" lens
• Resolution 2592x1944
• Frame rates: (sensor specs, not necessarily possible on Raspi)
• QSVGA 15fps
• 1080p 30fps
• 720p 60fps
• VGA 90fps
• Can support 8-/10 bit raw RGB
• Power : approx 100mA at 1.5v

[1] Interview with Eben Upton

[2] Post on Raspberry Pi forum by jamesh