Tag Archives: ownCloud

Upgrading ownCloud 7.0.4 to 8.1.1

I am running ownCloud on a Debian machine with Mysql and I have been a little behind with upgrading it. Today I upgraded from 7.0.4 to 8.1.1 following the standard instructions. A few more notes on my environment:

  1. I don’t use encryption for files
  2. I don’t use https/ssl (I am behind an openvpn server)
  3. I did the upgrade in one step (7.0.4 to 8.1.1, not via intermediate versions)

It basically went fine. When I ran:

$ sudo -u www-data php occ upgrade
ownCloud or one of the apps require upgrade - only a limited number of commands are available
Checked database schema update
Checked database schema update for apps
Updated database
Disabled 3rd-party app: calendar
Disabled 3rd-party app: contacts
Disabled 3rd-party app: documents
Updating  ...
Updated  to 0.7
Updating  ...
Updated  to 0.6
Updating  ...
Updated  to 1.1.10
Updating  ...
Updated  to 2.0.1
Updating  ...
Updated  to 0.6.2
Updating  ...
Updated  to 0.6.3
Updating  ...
Updated  to 0.6.0
Update 3rd-party app: calendar
Exception: App does not provide an info.xml file
Update failed
Maintenance mode is kept active

I am a little surprised, because I don’t remember calendar, contacts and documents being 3rd party apps before (?). Anyway, the server did not come up, so I ran the command again:

$ sudo -u www-data php occ upgrade
ownCloud or one of the apps require upgrade - only a limited number of commands are available
Turned on maintenance mode
Checked database schema update
Checked database schema update for apps
Updated database
Update successful
Turned off maintenance mode

Now it worked. Logged in, no traces of the three 3rd-party apps. Whatever, I use ownCloud for the files.

Performance after upgrade
ownCloud is not particularly fast. I did a very quick and unscientific performance check: before upgrading I uploaded a folder (17 files, 4.1 MB) to ownCloud: it took 30 seconds (for the desktop client to complete syncing). After the upgrade the same folder took 19 seconds to sync. This proves nothing of course, but at least it seems promising.

Ubuntu Client
My Ubuntu client used ownCloud vs 1.7. It does not work with 8.1.1. Installing ownCloud client from external repository worked fine. Same thing for Debian, obviously.

Upgrading ownCloud 6.0.3 to 7.0.1

I am running ownCloud on a Debian machine with Apache and Mysql, as I have written about before. OwnCloud has released a new version, 7.0.1, and it is possible to update via the Web GUI. I did that, it took a few minutes, and it worked perfectly.

I have written about Performance of ownCloud before. It appears upgrading to 7.0.1 has not changed the performance of the platform at all.

Owncloud client on Raspbian

I found that Raspbian comes with a very old version (1.2 something) of the Owncloud client. I found no prebuilt more up to date versions, so I built one myself:

$ sudo apt-get install cmake qt4-dev-tools build-essential
$ sudo apt-get install libneon27 libneon27-dev qtkeychain-dev
$ sudo apt-get install sqlite3 libsqlite3-dev libsqlite3-0
$ tar -xjf mirall-1.6.1rc1.tar.bz2
$ mkdir mirall-build
$ cd mirall-build/
$ cmake ../mirall-1.6.1rc1

The owncloud client is now in the bin folder.

Note: I took the commands above from my history, so there is a slight risk of a mistake. Also, I might have installed other packages before, that I am not aware of are not required for owncloud. Feel free to give feedback!

It is quite useful to put a Raspberry Pi with a USB-drive in someone elses home, and let it syncronize your files. That way, you have an off-site backup for worst case scenarios.

Testing ownCloud performance

Update 2014-04-28: Upgrading to ownCloud 7.0.1 has not changed the performance of the platform at all.

Update 2014-04-28: I have found that downloading files is quite fast. At about 10% CPU load, the server can saturate my 10MBit/s internet connection, when I download my files to another computer, over https. When uploading files, top shows mostly waiting time. When downloading, top shows mostly idle time. I suspect the SQL communication/overhead is limiting upload performance, and that ownCloud keeps a lot of book keeping in the database. If it does so for a good reason, and download is reasonably fast, I can live with it. I anyway keep my original article below (on upload performance), but I find the performance quite acceptable for my real-world application, on my not so powerful hardware.

Update 2014-04-26: Tried FastCGI/mod-fcgid, see below.

Ubuntu announced that they will cancel the Ubuntu One service, and Condoleezza Rice will start working for Dropbox. So, how am I going to share my files among different computers and devices?

ownCloud appears like a nice option. It is like Dropbox, but I can run it myself, and it works not only for files, but also for contacts/calenders and smartphones.

Buying ownCloud as a service is possible, but as soon as I want to put my pictures (and perhaps some video and music) it gets pretty expensive. If I host myself several hundreds of GB of disk is no problem.

So, I installed ownCloud (6.0.2) on my QNAP TS 109 running Debian (7.4). Horrible performance – it took a minute to log in. Ok – the QNAP has a 500MHz ARM, but even worse, just 128MB of RAM and quite slow disk access. What device to put ownCloud on? A new nice QNAP (TS-221) is quite pricy, and a Raspberry Pi accesses both disk and network over its USB bus. I came to think of buying a used G4 Mac Mini – they are really cheap! Then I came to think of my old Titanium PowerBook G4 that has been gathering dust last year, and I decided to try running ownCloud on it. Perhaps not as a long term solution, but as a learning/testing machine it could work fine.

ownCloud Server configuration
CPU: G4 866MHz
RAM: 1024Mb
HD: 320GB ATA
OS: Debian, 7.4 (PPC) fresh install on entire hard drive
DB: mysql 5.5 (the std version for Debian)
https: apache 2.2 (the std version for Debian)

To improve performance, I enabled APC for PHP, and disabled full text search.

Performance measurements
For the performance tests, I tried to transfer 1x100MB, 10x10Mb and 100x1Mb files. I measured the times with a regular stopwatch, and occationaly I repeated a test when the result was strange. The below measurements are not exactly accurate, but the big picture is there.

Transfers are made from a Windows 7 PC over a Gbit network.

1x100Mb 10x10Mb 100x1Mb
Encryption and checksum on G4 / server
(1): ssl encrypt aes ; sync 7s
(2): md5sum 1s
File transfer using other methods
(3): ftp/Filezilla 3s 3s 4s
(4): sftp/Filezilla 14s 15s 17s
ownCloud
(5): No SSL, NO APC 15s 32s 234s
(6): No SSL, APC 16s 27s 197s
(7): SSL, APC 34s 43s 263s
(8): SSL, APC, encryption 46s 69s 438s

Comments on Performance
(1): tells me that the server is capable of encrypting 100Mb of data, and sync output to local disk, in 7 seconds. The sync is less than a second.
(2): tells me that the server is capable of processing 100Mb of data in a second.
(3): tells me that simply sending the files over the network with a proven protocol takes 3-4 seconds, slightly slower for more smaller files.
(4): tells me that sending the files in an encrypted stream with a proven protocol over the network takes about 15 seconds, slightly slower for more smaller files.
(5): shows that the overhead for many files in ownCloud is massive.
(6): shows that APC helps, but not in a very significant way.
(7): shows the extra cost of SSL (transferring over a secure channel).
(8): shows the extra cost of encrypting the files for the user, on the server (using openssl/AES, according to ownCloud documentation.

It makes sense to compare row (3) and (6), indicating that with no encryption whatsoever the overhead of ownCloud is 5-50x the actual work. Or, the resources used for actually transferring and storing files are 20%-2%, the rest of the resources, 80%-98% are “wasted”. Now ownCloud has some syncroniziation and error handling capacities not found in FTP, but I dont think that justifies this massive overhead.

In the same way it makes sense to compare row (4) and (7), indicating a waste of 60%-94% of resources, for using a secure channel (and I believe that SSH uses stronger encryption than TLS).

For average file size smaller than 1MB, the waste will be even bigger.

I suspect the cost is related to executing php for each and every file. It could also be the use of the database for each file that is expensive. Somewhere I read that there are “hundreds” of database calls for each web server request handled by ownCloud.

Suggestions
It is of course a bit arrogant to suggest solutions to a problem in an Open Source project that I have never contributed to, without even reading the code. Anyway, here we go:

  • Find a way to upload small directories (<10MB, or <60s transfer) as tarballs or zipfiles. This should of course happen transparantly to the user (and only work via the client, not the web). This way hundreds or thousands of small files could be uploaded in a few seconds instead of very long time - and the load on the server would decrease a lot.
  • Similar to the first suggestion, allow files to be uploaded in fragments, to allow upload of 2GB+ files on all server platforms (it is ridiculus that an ARM server, like a QNAP, can not handle 2GB+ files, as I have read in the documentation is the case).
  • Alternatively, allow ownCloud to use ssh/sftp as transfer protocol. It will not work in all situations, but when client and server are allowed to communicate on port 22, and ownCloud is installed on a server with ssh enabled, it could be an option.

I kind of presume that the problem is one-file-per-request and WebDav limitations. Perhaps it is the database that is the problem? Nevertheless, I think som kind of batch-handling of uploads/downloads is the solution in that case too.

LAMP
ownCloud is built on LAMP, and I doubt the performance problems are related to the LA in LAMP. Also, I dont think that the M should be the problem if the databas calls are kept at a reasonable level. The problem must be with P(HP). I understand and appreciate that PHP is simple and productive, and probably 95% of ownCloud can be perfectly written in PHP. But perhaps there are a few things that should be written in something more high-performing (I am talking about C, of course)?

Conclusion
I really like the ambition of ownCloud, and mostly, the software is very nice. The server has many features, and the clients are not only nice, but also available for several platforms.

ownCloud is a quite mature product, at version 6. I wish some effort is put into improving performance. I believe there are possible strategies that would not require very much rewriting, and not need to brake compability. And I also believe it makes much sense to optimize the ownCloud server code: not only because people may run it on Raspberry Pis, QNAPs or old hardware, but also because it would improve the usefulness on more powerful servers.

2014-04-26: FastCGI / mod-fcgid
I decided to try PHP via FastCGI to see if it could improve performance. Very little difference – I disabled it and got back to “recommended” configuration. For details, read on.

I mostly followed this guide (as apache2-mod-fastcgi seems to be replaced by apache2-mod-fcgid lately, other high-ranking guides were out of date). The following options need to be added to /etc/apache2/apache2.conf:

FcgidFixPathinfo 1               (not in the site definition as suggested in guide)
FcgidMaxRequestLen 536870912     (effectively limits maximum file size)