Category Archives: Raspberry Pi

How to (not) set up a RPi V3 server

A few months ago I set up a server running Archlinux on a RPi V3 using a 2.5′ USB drive for root. It is now dead.

One day I innocently did “pacman -Suy” as usual, and it didn’t restart. After that it was very unstable for days until it – the RPi V3 itself – appears broken. That is, I get very random errors, like different kernel panics, trying to boot it. I let a friend try to boot LibreELEC on it (his power, SD-card, TV) and it displayed the 4-color-splash-screen and the little lightning indicating power problem.

There are different ways to connect a USB HDD to a RPi.

  1. Let the RPi power the USB drive.
    (it works sometimes, you can try max_usb_current=1 in config.txt)
  2. Connect the USB drive to a “usb extra power cable” (google it) to take the load off the RPi
  3. Connect the USB drive to a USB hub that has external power
    (optionally, also power the RPi from the same USB hub)

With my unfortunate broken RPi V3, I used method (1) for a hard drive rated at 1.0A (the USB<->SATA-chip probably consumes some power as well). I did use a proper original RPi power supply though, but I believe I somehow stressed some component of the RPi V3 continously taking more than 1.0A from a single USB port.

Findings using USB hub to power RPi + HDD

I have an old USB Hub with a 2.0A rated power supply. I now use it as the only power source for a RPi + USB HDD (The USB Hub and RPi are connected both ways in a loop).

  • RPi V2 + 1.0A 1TB HDD: frequent under-voltage warnings
  • RPi V1 + 0.6A 320GB HDD: works perfectly

So, that particular USB Hub will drive my RPi V1, and I will find another solution for the RPi V2.

Conclusion

I have written several articles about using RPi as a server.

My sober and responsible conclusion must be: don’t (use Raspberry Pi for production Linux servers). It is simply not worth it. First it is not so cheap as you think when you have bought all cables, adapters, cases and chargers. Second, your time my not be for free. Third, performance is bad. Forth, stability is limited and don’t expect very long service time.

The cheapest NUC setup is a more rational choice.

I also believe a QNAP or Synology with virtualisation technology could be a better choice than running (multiple) RPi.

Nevertheless, I never learn, and I am now replacing my broken RPi V3 with two old Raspberry Pi (V1 + V2). I mostly use them for Syncthing and backup, I guess two is better than one, and I have unused USB Harddrives.

Archlinux vs Raspbian

I have come to like Archlinux for RPi. However, the frequent and relatively large upgrades that come with a rolling distribution feels somewhat unoptimal for a low-performance system living on an SD-card.

This time, I am back with Raspbian, because

  1. Raspbian is now based on Debian 10 (buster) right from start (I believe the Debian 9 release of Raspbian was kind of late)
  2. There is simple minimal Raspbian Buster Lite image suitable for servers or headless systems
  3. Creating an empty file named “ssh” in /boot before starting the first time lets you ssh into the brand new raspbian system, so you can easily install with neither keyboard/display or serial

I simply have nothing to complain about with Raspbian anymore.

ArchLinux on RPi with USD Harddrive

Update 20190624: Today I ran “pacman -Suy” and ended up with a system that does not start. Two blinking green leds indicate failure to read the SD card. That is less than 5 months after I used SD-card for only /boot. I really consider not running RPi as headless servers anymore.

I have found that one of the weakest parts of a Raspberry Pi server or workstation is the SD card: it is slow and it will break sooner rather than later. There may be industrial SD cards or better SD cards, but a good old USB hard drive is just better.

With RPi v3 it shall be possible to boot straight off a USB drive! That sounded great so I got a brand new RPi v3 B+, a USB hard drive, and I installed ArchLinux on the hard drive, just as if it was a memory card. Fail. That did not work (with ArchLinux, Raspbian may be another story).

But there are levels of pain:

  1. All SD-card
  2. SD-card, but /home on USB harddrive
  3. USB harddrive, but /boot on SD-card
  4. All USB harddrive

I decided to try #3.

It turns out that when the RPi boots it runs u-boot (its like the BIOS of RPi, and many other embedded devices). At one point u-boot reads boot.scr (from the first VFAT partition of the SD card). It had the lines:

part uuid ${devtype} ${devnum}:2 uuid

setenv bootargs console=ttyS1,115200 console=tty0 root=PARTUUID=${uuid} rw rootwait smsc95xx.macaddr="${usbethaddr}"

I figured that I could do this instead:

# part uuid ${devtype} ${devnum}:2 uuid

setenv bootargs console=ttyS1,115200 console=tty0 root=/dev/sda2 rw rootwait smsc95xx.macaddr="${usbethaddr}"

However, boot.scr has a checksum so you cant just edit it. But it tells you what to do: run ./mkscr. But it is dependent on mkimage, so the procedure is:

  1. Install uboot tools
    1. ARCH: pacman -S uboot-tools
    2. Ubuntu/Debian: apt-get install u-boot-tools
  2. Edit boot.txt (not boot.scr) to your liking
  3. Run: ./mkscr

Now only /boot is on SD-card. That is quite ok with me. There is very little I/O to boot so the SD-card should survive. If I want to I can make a regular simple backup by simple file copy of /boot to a zip-file or something, and just restore that zip-file to any SD-card.

There seems to be no need to edit anything else (like fstab).

Well, the bad thing is it did not work out 100% as I hoped. The good thing is that this should work with any RPi, not just the RPi v3 that supports USB boot.

Syncthing crashes on RPi and Arch Linux

One of my Syncthing servers started crashing (again). It is Rapsberry Pi v2 running Arch Linux. Syncthing was 0.14.44.? I upgraded and got 0.14.48.1. Still not stable.

So I downloaded the Syncthing binary from Syncthing instead of using the one that comes with Arch Linux. That seems to work better.

During trying different things I did a database reset:

$ syncthing -reset-database     (does not start syncthing)
$ syncthing

This is not the first time Syncthing misbehaves on Raspberry Pi and I am beginning to question if it is so smart to store my files on a Raspberry Pi with a USB drive.

Raspbian – kerberos not found

I have this very strange error on my RPi V2 with Raspbian (8.0). I suspect I will throw away the memory card and never fix it, but I will document the error for future reference.

My problem was that curl, ssh, sshd suddenly did not work. When I start the web browser I get “I/O error”. This screenshot shows (at least a symptom of) the problem.

I tried to reinstall ssh and curl:

$ apt-get install --reinstall curl

and that did not help.

Apart from this the system works ok. It shutdowns and starts properly. No I/O errors from dmesg. I doubt I will ever figure this one out. It seems to me the system is corrupt at disk level, probably an SD-card problem, and that a new install on a new SD-card is the only way forward.

Syncthing v0.14.40, Raspberry Pi, 100% CPU

I think Syncthing is an amazing piece of software, but I ran into problem last week.

I have a library of 10 different folders, 120000 files, 42000 directories and 428GB of data.

I thought that was a little bit too much for my RPi V1 (Syncthing 0.14.40, Arch Linux), because it constantly ran at 100%. I raised Rescan Interval to several hours (so it would finish before staring over).

After startup it took about 10-15 min to get the web GUI up, and about an hour to scan all folders for the first time. Well, that is ok, but after that it still constantly used 100% CPU despite all folders were “up to date”.

It turned out it crashed and started over. I found panic logs in .config/syncthing and error messages in ./config/syncthing/index-v0.14.0.db/LOG.

Some errors indicated Bad Magic Number and Checksum Corruption. The usual reason for this seems to be hardware problem (!?!).

I upgraded my RPi V1 to an RPi V2, with little success. Then I found that I had similar problems on another RPi V2. So after shutting down Syncthing I tried the quite scary:

  $ syncthing -reset-database      ( does not start syncthing )      
  $ syncthing                      ( start syncthing )

After several hours of scanning everything seems to work perfectly!
Let us see how long that lasts.

Review: NUC vs Raspberry Pi

I like small, cheap, quiet computers… perhaps a little too much. For a long time I have used a Raspberry Pi V2 (QuadCore@900MHz and 1GB RAM) as a workstation. To be honest, I have not used it for web browsing, that is just too painful. But I have used it for programming and running multiple Node.js services, and a few other things.

Despite there are so many single board computers it is hard to find really good alternatives to the Raspberry Pi. And when I look into it, I find that Intel NUCs are very good options. So, I just decided to replace my RPi2 workstation with the cheapest NUC that money can currently buy: the NUC6CAY with a Celeron J3455 CPU. It sounds cheap, particularly for something server like. The interesting thing with the J3455 CPU is that it is actually Quad Core, with no hyper threading. To me it sounds amazing!

I also have an older NUC, a 54250WYKH with an i5 CPU.

Raspberry Pi V2:   ARMv7    4 Cores      900MHz                  1GB RAM
NUC                Celeron  4 Cores      1500MHz (2300 burst)    8GB RAM
NUC                i5       2 Cores (HT) 1300MHz (2600 burst)   16GB RAM

I/O is obviously superior for the NUCs (both using SSD) versus the RPI v2 having a rotating disk connected to USB. But for my purposes I think I/O and (amount of) RAM makes little difference. I think it is more about raw CPU power.

Node.js / JavaScript
When it comes to different Node.js applications, it seems the older i5 is about twice as fast as the newer Celeron (for one Core and one thread). I would say this is slightly disappointing (for the Celeron). On the other hand the Celeron is about 10x faster than the RPi V2 when it comes to Node.js code, and that is a very good reason to use a NUC rather than a Raspberry PI.

Update 2018-02-11: after a few months
I came back to my RPi2 from my cheap NUC. The difference is… everything. I really like Raspberry PIs. I have built cases for them, bought cases for them, worked on them, made servers of them. But I really must say that a NUC makes more sense: it contains everything nicely and it is so much more powerful.

You can get a Celeron NUC with 2GB RAM and a 2.5′ disk for quite little money. And from there you can go to Core i7, 32GB RAM and two hard drives: M.2 + 2.5′. And check out the Hades Canyon NUC.

I feel sorry there is basically nothing in the market like a NUC with ARM, AMD, PowerPC or Mips. The only competition is the 4 year old MacMini, which is completely an Intel machine. If you find something cool, NUC-like, not Intel, feel free to post below.

Update 2018-02-28
I ran into a new problem on my RPi. It could be anything. My guess, that I will never be able to prove, is that it is a glitch made possible by using an SD-card as root device (and possibly questionable drivers/hardware for SD on the RPi).

Update 2018-04-09
Premier Farnell has introduced a Desktop Pi. Especially promising is that together with a recent RPi you can get rid of the SD-card entirely, and only use SSD/HDD or even mSATA (over USB i presume).

Raspberry PI performance and freezes

On a daily basis I use a Raspberry Pi v2 (4x900MHz) with Raspian as a work station and web server. It is connected to a big display, I edit multiple files and it runs multiple Node.js instances. These Node.js processes serve HTTP and access (both read and write) local files.

I experienced regular freezes. Things that could take 2-3 seconds were listing files in a directory, opening a file, saving a file and so on.

I moved my working directory from my (high performance) SD-card to a regular spinning USB hard drive. That completely solved the problem. I experience zero freezes now, compared to plenty before.

My usual experience with Linux is that the block caching layer is highly effective: things get synced to disk when there is time to do so. I dont know if Linux handles SD-cards fundamentally different from other hard drives (syncing more often) or if the SD card (or the Raspberry Pi SD card hardware) is just slower.

So, for making real use of a Raspberry Pi I would clearly recommend a harddrive.

Raspberry Pi Server

The Raspberry Pi has been around for some years now and it has been used in unbelievable projects. As a budget desktop computer it has not quite had the required performance (although v2 and v3 are much improving the situation over v1). However, for simple hobby server tasks the RPi can work very well.

A simple RPi (any version) setup typically requires:

  • RPi
  • SD Card
  • USB PSU + USB cable
  • Network Cable
  • External USB Drive + USB Cable (+power adapter)
  • A case

That is without display, mouse and keyboard, and you dont have a power button. It gets a bit messy.

The market is full of RPi cases that all do the same thing: nothing. They just contain the board. The market is full of mini/micro-towers for MiniITX. There are rather expensive NAS devices that come without hard drives. Why are there no small tower cases that comes with:

  • PSU
  • Slots for 1-2 hard drives (+USB to SATA converters)
  • Cabling that makes everything tidy and neat

Powering the RPi using an external hard drive
I happened to have an external USB drive with an integrated USB hub (an Iomega Minimax that was left alone when its Mac Mini died). With some wood and glue I built a simple stand for the hard drive and the RPi:

DSCN5193

DSCN5194

DSCN5196

As you can see:

  • the hard drive powers the RPi, and I can even use the hard drive power switch
  • the Ethernet and USB ports are conveniently available on the back side
  • the footprint is just slightly larger (just taller) than the hard drive itself
  • the two USB cables between RPi and harddrive are nicely contained
  • heat/ventilation should be pretty good

I have experienced no problems powering the RPi from a USB drive that it itself is connected to. It may not be a supported or recommended configuration, but for practical purposes it works for me.

Performance
I mostly run Syncthing on this RPi. The bottleneck is very much the 700MHz ARMv6 CPU, not the USB2-to-SATA-overhead.

hdparm gives me:

$ sudo /sbin/hdparm -t /dev/sda
/dev/sda:
 Timing buffered disk reads:  82 MB in  3.03 seconds =  27.09 MB/sec

$ sudo /sbin/hdparm -T /dev/sda
/dev/sda:
 Timing cached reads:   496 MB in  2.01 seconds = 247.36 MB/sec

Of course it sucks compared to what you can get in 2016, but it is not remarkably bad in anyway. And it is not so fun to live on an SD card.

The Western Digital Kit
The other day Western Digital announced both a special 314GB hard drive and accessories to make it all nice.

Plusberry Pi
There is also the interesting Plusberry Pi project.

Picocluster
Picocluster is clearly bringing new options! Not so much focus on support for a USB Hard Drive though. Some models have an HDMI port. I am not sure, but I think the idea is that you connect one RPi to the external USB/HDMI ports, and then that RPi can control the other one, if you run Picoclusters custom distribution. Not so bad, but not a KVM either. Perhaps a little modification to make a serial switch so RPi #2-5 can be controlled from #1 over serial?

Best Raspberry Pi Server Linux Distribution

Since I got my first Raspberry Pi have have wondered: how to turn it into a proper server. Options that I have not been entirely satisfied with:

  • Arch Linux: probably a great option if you know Arch… I have been too lazy to learn.
  • Gentoo Linux: is Gentoo still relevant? Building everything on the RPi sounds very painful (slow)
  • OpenWrt: nice, but slightly too minimal for a server
  • Raspbian: nice, but a little bit too big standard installation (perhaps it does not really matter, but every apt-get upgrade takes longer time, and so on)
  • NetBSD: such a disappointment 🙁

I now found, and tested, Raspbian Unattended Netinstaller. For me, this is the shit.

If is really this simple:

  1. Format your SD-card with FAT32 (just as usual)
  2. Unpack (unzip) the raspbian-ua-netinst on your SD-card
  3. Connect the SD-card, ethernet and power to your Raspberry Pi
  4. Wait (about 25 minutes, they say, that was ok with me)
  5. SSH into your new lean Raspbian system (root/raspbian).
  6. Read under “first boot” what to do next

Clearly, you need a properly configured network (DHCP, allow fetching of packages, and you need to know what IP address it got).

The entire experience is much enhanced if you connect to your Raspberry Pi with a serial cable during the entire procedure. Jokes aside, I used a serial with my first installation. Second time when I felt confident with the process I did not bother with the serial cable.

First boot quick guide

#dpkg-reconfigure locales
#dpkg-reconfigure tzdata

/boot/config.txt: add the line
gpu_mem=16

Upgrade to jessie
For some reason, Raspbian installation is still based on wheezy, not jessie (you don’t get the latest version of Debian). I suggest, upgrade immediately:

/etc/apt/sources.list (replace wheezy with jessie, two places)

# apt-get update
# apt-get dist-upgrade

It is almost as fast as the installation itself 😉

Conclusion
I think, for the Raspberry Pi V1, Raspbian installed this way is the best server system you can have (perhaps Arch is better if you know it). For a Raspberry Pi V2, perhaps standard Debian is better (I have never used an RPi v2). Everthing I have written applies perfectly to the RPi v2 as well.

Notes on Raspberry Pi and Serial

I experimented with my Raspberry Pi (v1 B) and a serial cable, a USB-serial identified as:

[85907.504415] usb 4-5: new full-speed USB device number 19 using ohci-pci
[85907.730850] usb 4-5: New USB device found, idVendor=0403, idProduct=6001
[85907.730863] usb 4-5: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[85907.730871] usb 4-5: Product: TTL232R-3V3
[85907.730877] usb 4-5: Manufacturer: FTDI
[85907.730882] usb 4-5: SerialNumber: ********
[85907.737978] ftdi_sio 4-5:1.0: FTDI USB Serial Device converter detected
[85907.738070] usb 4-5: Detected FT232RL
[85907.744057] usb 4-5: FTDI USB Serial Device converter now attached to ttyUSB1

My USB-serial-device has six cables: black-brown-red-orange-yellow-green.
Connected to the RPi from the corner pin: none-none-black-yellow-orange-none8x.

At this point I have no success with minicom. Screen works though:

sudo minicom -b 115200 -o -D /dev/ttyUSB1
sudo screen /dev/ttyUSB1 115200

When serial works, my procedure is:

  1. Connect everything except power
  2. Start screen
  3. Connect power
  4. Within a few seconds i get output

If I start a fresh default NOOBS (v1.4):

Uncompressing Linux... done, booting the kernel.

Welcome to the rescue system
recovery login: 

You can log in with root/raspberry, but I don’t know if you are meant to (can) install Raspbian this way.

NOTE: The Raspberry Pi itself prints nothing to the serial console. Only with a properly installed SD-card inserted, you get output.

Already installed System
For an already installed Raspbian, I got a normal login prompt over serial.
For an already installed OpenWRT (14.07), I got a root prompt, no password required, over serial.

Formatting SD-card using Linux
Sometimes it is hard to produce an SD-card that the Raspberry Pi wants to boot from.
This partitioning and formatting works:

$ sudo /sbin/fdisk -l /dev/sde

Disk /dev/sde: 7,4 GiB, 7948206080 bytes, 15523840 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00055f28

Device     Boot Start      End  Sectors  Size Id Type
/dev/sde1        2048 15523839 15521792  7,4G  e W95 FAT16 (LBA)

gt@oden:~/Downloads$ sudo mkfs.vfat /dev/sde1
mkfs.fat 3.0.27 (2014-11-12)

To be on the safe side, before using fdisk:

$ sudo dd if=/dev/zero of=/dev/sde bs=1024 count=10240