Category Archives: Programming

Simple Password Hashing with Node & Argon2

When you build a service backend you should keep your users’ passwords safe. That is not so easy anymore. You should

  1. hash and salt (md5)
  2. but rather use strong hash (sha)
  3. but rather use a very expensive hash (pbkdf2, bcrypt)
  4. but rather use a hash that is very expensive on GPUs and cryptominers (argon2)

Argon2 seems to be the best choice (read elsewhere about it)!

node-argon2

Argon2 is very easy to use on Node.js. You basically just:

$ npm install argon2

Then your code is:

/* To hash a password */
hash = await argon2.hash('password');

/* To test a password */
if ( await argon2.verify(hash,'password') )
  console.log('OK');
else
  console.log('Not OK');

Great! What is not to like about that?

$ du -sh node_modules/*
 20K  node_modules/abbrev
 20K  node_modules/ansi-regex
 20K  node_modules/aproba
 44K  node_modules/are-we-there-yet
348K  node_modules/argon2
 24K  node_modules/balanced-match
 24K  node_modules/brace-expansion
 24K  node_modules/chownr
 20K  node_modules/code-point-at
 40K  node_modules/concat-map
 32K  node_modules/console-control-strings
 44K  node_modules/core-util-is
120K  node_modules/debug
 36K  node_modules/deep-extend
 40K  node_modules/delegates
 44K  node_modules/detect-libc
 28K  node_modules/fs-minipass
 32K  node_modules/fs.realpath
104K  node_modules/gauge
 72K  node_modules/glob
 20K  node_modules/has-unicode
412K  node_modules/iconv-lite
 24K  node_modules/ignore-walk
 20K  node_modules/inflight
 24K  node_modules/inherits
 24K  node_modules/ini
 36K  node_modules/isarray
 20K  node_modules/is-fullwidth-code-point
 48K  node_modules/minimatch
108K  node_modules/minimist
 52K  node_modules/minipass
 32K  node_modules/minizlib
 32K  node_modules/mkdirp
 20K  node_modules/ms
332K  node_modules/needle
956K  node_modules/node-addon-api
240K  node_modules/node-pre-gyp
 48K  node_modules/nopt
 24K  node_modules/npm-bundled
 36K  node_modules/npmlog
172K  node_modules/npm-normalize-package-bin
 28K  node_modules/npm-packlist
 20K  node_modules/number-is-nan
 20K  node_modules/object-assign
 20K  node_modules/once
 20K  node_modules/osenv
 20K  node_modules/os-homedir
 20K  node_modules/os-tmpdir
 20K  node_modules/path-is-absolute
 32K  node_modules/@phc
 20K  node_modules/process-nextick-args
 64K  node_modules/rc
224K  node_modules/readable-stream
 32K  node_modules/rimraf
 48K  node_modules/safe-buffer
 64K  node_modules/safer-buffer
 72K  node_modules/sax
 88K  node_modules/semver
 24K  node_modules/set-blocking
 32K  node_modules/signal-exit
 88K  node_modules/string_decoder
 20K  node_modules/string-width
 20K  node_modules/strip-ansi
 20K  node_modules/strip-json-comments
196K  node_modules/tar
 28K  node_modules/util-deprecate
 20K  node_modules/wide-align
 20K  node_modules/wrappy
 36K  node_modules/yallist

That is 69 node modules of 5.1MB. If you think that is cool for your backend password encyption code (in order to provide two functions: encrypt and verify) you can stop reading here.

I am NOT fine with it, because:

  • it will cause me trouble, one day, when I run npm install, and something is not exactly as I expected, perhaps in production
  • how safe is this? it is the password encryption code we are talking about – what if any of these libraries are compromised?
  • it is outright ugly and wasteful

Well, argon2 has a reference implementation written in C (link). If you download it you can compile, run test and try it like:

$ make
$ make test
$ ./argon2 -h
Usage: ./argon2-linux-x64 [-h] salt [-i|-d|-id] [-t iterations] [-m log2(memory in KiB) | -k memory in KiB] [-p parallelism] [-l hash length] [-e|-r] [-v (10|13)]
Password is read from stdin
Parameters:
 salt The salt to use, at least 8 characters
 -i   Use Argon2i (this is the default)
 -d   Use Argon2d instead of Argon2i
 -id  Use Argon2id instead of Argon2i
 -t N Sets the number of iterations to N (default = 3)
 -m N Sets the memory usage of 2^N KiB (default 12)
 -k N Sets the memory usage of N KiB (default 4096)
 -p N Sets parallelism to N threads (default 1)
 -l N Sets hash output length to N bytes (default 32)
 -e   Output only encoded hash
- r   Output only the raw bytes of the hash
 -v (10|13) Argon2 version (defaults to the most recent version, currently 13)
 -h   Print ./argon2-linux-x64 usage

It builds to a single binary (mine is 280kb on linux-x64). It does most everything you need. How many lines of code do you think you need to write for node.js to use that binary instead of the 69 npm packages? The answer is less than 69. Here comes some notes and all the code (implementing argon2.hash and argon2.verify as used above):

  1. you can make binaries for different platforms and name them accordingly (argon2-linux-x64, argon2-darwin-x64 and so on), so you can move your code (and binaries) between different computers with no hazzle (as JavaScript should be)
  2. if you want to change argon2-parameters you can do it here, and if you want to pass an option-object to the hash function that is an easy fix
  3. options are parsed from hash (just as the node-argon2 package) when verifying, so you don’t need to “remember” what parameters you used when hashing to be able to verify
/* argon2-wrapper.js */

const nodeCrypto = require('crypto');
const nodeOs    = require('os');
const nodeSpawn = require('child_process').spawn;
/* NOTE 1 */
const binary    = './argon2';
// st binary    = './argon2-' + nodeOs.platform() + '-' + nodeOs.arch();

const run = (args,pass,callback) => {
  const proc = nodeSpawn(binary,args);
  let hash = '';
  let err = '';
  let inerr = false;
  proc.stdout.on('data', (data) => { hash += data; });
  proc.stderr.on('data', (data) => { err += data; });
  proc.stdin.on('error', () => { inerr = true; });
  proc.on('exit', (code) => {
    if ( err ) callback(err);
    else if ( inerr ) callback('I/O error');
    else if ( 0 !== code ) callback('Nonzero exit code ' + code);
    else if ( !hash ) callback('No hash');
    else callback(null,hash.trim());
  });
  proc.stdin.end(pass);
};

exports.hash = (pass) => {
  return new Promise((resolve,reject) => {
    nodeCrypto.randomBytes(12,(e,b) => {
      if ( e ) return reject(e);
      const salt = b.toString('base64');
      const args = [salt,'-id','-e'];
/* NOTE 2 */
//    const args = [salt,'-d','-v','13','-m','12','-t','3','-p','1','-e'];
      run(args,pass,(e,h) => {
        if ( e ) reject(e);
        else resolve(h);
      });
    });
  });
};

exports.verify = (hash,pass) => {
  return new Promise((resolve,reject) => {
    const hashfields = hash.split('$');
    const perffields = hashfields[3].split(',');
/* NOTE 3 */
    const args = [
        Buffer.from(hashfields[4],'base64').toString()
      , '-' + hashfields[1].substring(6) // -i, -d, -id
      , '-v', (+hashfields[2].split('=')[1]).toString(16)
      , '-k', perffields[0].split('=')[1]
      , '-t', perffields[1].split('=')[1]
      , '-p', perffields[2].split('=')[1]
      , '-e'
    ];
    run(args,pass,(e,h) => {
      if ( e ) reject(e);
      else resolve(h===hash);
    });
  });
};

And for those of you who want to test it, here is a little test program that you can run. It requires

  • npm install argon2
  • argon2 reference implementation binary
const argon2package = require('argon2');
const argon2wrapper = require('./argon2-wrapper.js');

const bench = async (n,argon2) => {
  const passwords = [];
  const hashes = [];
  const start = Date.now();
  let errors = 0;

  for ( let i=0 ; i<n ; i++ ) {
    let pw = 'password-' + i;
    passwords.push(pw);
    hashes.push(await argon2.hash(pw));
  }
  const half = Date.now();
  console.log('Hashed ' + n + ' passwords in ' + (half-start) + ' ms');

  for ( let i=0 ; i<n ; i++ ) {
    // first try wrong password
    if ( await argon2.verify(hashes[i],'password-ill-typed') ) {
      console.log('ERROR: wrong password was verified as correct');
      errors++;
    }
    if ( !(await argon2.verify(hashes[i],passwords[i]) ) ) {
      console.log('ERROR: correct password failed to verify');
      errors++;
    }
  }
  const end = Date.now();
  console.log('Verified 2x' + n + ' passwords in ' + (end-half) + ' ms');
  console.log('Error count: ' + errors);
  console.log('Hash example:\n' + hashes[0]);
};

const main = async (n) => {
  console.log('Testing with package');
  await bench(n,argon2package);
  console.log('\n\n');

  console.log('Testing with binary wrapper');
  await bench(n,argon2wrapper);
  console.log('\n\n');
}
main(100);

Give it a try!

Performance

I find that in Linux x64, wrapping the binary is slightly faster than using the node-package. That is weird. But perhaps those 69 dependencies don’t come for free after all.

Problems?

I see one problem. The node-argon2 package generates binary hashes random salts and sends to the hash algorithm. Those binary salts come out base64-encoded in the hash. However, a binary value (a byte array using 0-255) is not very easy to pass on the command line to the reference implementation (as first parameter). My wrapper-implementation also generate a random salt, but it base64-encodes it before it passes it to argon2 as salt (and argon2 then base64-encodes it again in the hash string).

So if you already use the node-package the reference c-implementation is not immediately compatible with the hashes you already have produced. The other way around is fine: “my” hashes are easily consumed by the node package.

If this is a real problem for you that you want to solve I see two solutions:

  1. make a minor modification to the C-program so it expects a salt in hex format (it will be twice as long on the command line)
  2. start supplying your own compatible hashes using the option-object now, and don’t switch to the wrapper+c until the passwords have been updated

Conclusion

There are bindings between languages and node-packages for stuff. But unix already comes with an API for programs written in different languages to use: process forking and pipes.

In Linux it is extremely cheap. It is quite easy to use and test, since you easily have access to the command line. And the spawn in node is easy to use.

Argon2 is nice and easy to use! Use it! Forget about bcrypt.

The best thing you can do without any dependencies is pbkdf2 which comes with node.js and is accessible in its crypto module. It is standardized/certified, that is why it is included.

SonarQube – I disagree

For someone like me working in a (very) small development team SonarQube is good. It reviews my code and it teaches me about both old and new things that I did not know about.

There are some things I don’t agree about though (all JavaScript below).

“switch” statements should have at least 3 “case” clauses

I understand the point. But I have my reasons. Let us say I have orders with different statuses (DRAFT,OPEN,CLOSED). Then often I have switch statements like:

switch (order.status) {
case 'DRAFT':
  ...
  break;
case 'OPEN':
  ...
  break;
case 'CLOSED':
  ...
  break;
default:
  throw new Error(...);
}

This switch over order.status happens in many places. Sometimes it is just:

switch (order.status) {
case 'CLOSED':
  ...
  break;
}

I don’t want to refactor the (rewrite if with switch) code if/when I need to do something for other states. And I like a consistent form.

Another case can be that I allow the user to upload files in different formats, but for now I just allow PDF.

switch ( upload.format ) {
case 'pdf':
  process_pdf();
  break;
default:
  throw new Error('Unsupported upload.format: ' + upload.format);
}

A switch is much more clear than an if-statement. However, if the above code was in a function called uploadPdf() then the code would have been:

if ( 'pdf' !== upload.format ) throw new Error(...);
...
...

But in conclusion, I think switch statements with few (even 1) case clause are often fine.

This branch’s code block is the same as the block for the branch on line …

This happens to me often. A good rule is to never trust user input. So I may have code that looks like this (simplified):

if ( !(user = validateToken(request.params.token)) ) {
  error = 401;
} else if ( !authorizeAction(user,'upload') ) {
  error = 401;
} else if ( !reqest.params.orderid ) {
  error = 400;
  errormsg = 'orderid required';
} else if ( user !== (order = getOrder(request.params.orderid)).owner ) {
  error = 401;
} else if ( !validateOrder(order) ) {
  error = 400;
  errormsg = 'order failed to validate';
} else ...

The point is that important part of the code is not the code blocks, but rather the checks. The content of the code blocks may, or may not, be identical. That hardly counts as duplication of code. The important thing is that the conditions are executed one-by-one, in order and that they all pass.

I could obviously write the first two ones as the same (with a more complex condition), since both gives 401. But code is debugged, extended and refactored and for those purposes my code is much better than:

if ( !(user = validateToken(request.params.token) ) ||
     !authorizeAction(user.upload) ) {
  error = 401;
} else ...

My code is easier to read, reorganize, debug (add temporary breakpoints or printouts) or improve (perhaps all 401 are not equal one day).

Remove this …… label

The reason I get this warning is mostly that I use what i learnt in my own post Faking a good goto in JavaScript.

SonarCube also complained when I wanted to break out of a nested loop this way:

break_block {
  for ( i=0 ; i<10 ; i++ ) {
    for ( j=0 ; j<10 ; j++ ) {
      if ( (obj=lookForObjInMatrix(matrix,i,j) ) ) break break_block;
    }
  }
  // obj not found, not set
}

It is a stupid simplified example, but it happens.

Web Components 2020

Web Components is standard technology (as opposed to libraries, frameworks and tools that come and go) for web development.

It allows you to simply write custom html elements in JavaScript. Such components become framework-independent and can be shared and reused. They are simple to use. In your html-file, simply:

<html>
  ...
  <script src="wc-blink.js"></script>
  ...
  <wc-blink>This should blink</wc-blink>
  ...
</html>

The code of wc-blink is quite simple (view), I stole it (here) and modified it slightly.

Another advantage of Web Components is the Shadow DOM allowing private CSS: the styles wont escape, override or be overridden.

This technology has been around for a while… but the bad news… it still does not work with Microsoft browsers (IE11, Edge).

I wrote a simple demo page (link).

With Polyfill you can use it with Edge. IE11 seems to be out of luck because the keyword class from ES6 must work (see code for wc-blink above) and this is not simply a matter of polyfill. There is technology to compile/transpile stuff to ES5 but in that case I can rather keep building my components in Vue.js.

Conclusion

It is actually sad that work that goes into web browser standards (to make it possible to build applications and components for the web in a good way) gets delayed by MS simply not supporting it.

I dont think web development should be more complicated than including scripts and writing html. Web Components allow just that.

If you need to support IE11, simply ignore Web Components until you don’t need to support IE11 anymore.

If you are fine supporting just Edge, there are ways not to need to include the 117kb polyfill for everyone.

I can not afford to break IE11 at this point, and neither am I willing to transpile stuff. I stick to Vue.js components.

JavaScript: Fast Numeric String Testing

Sometimes I have strings that (should) contain numbers (like ‘31415’) but I want/need to test them before I use them. If this happens in a loop I could start asking myself questions about performance. And if it is a long loop an a Node.js server the performance may actually matter.

For the purpose of this post I have worked with positives (1,2,3,…), and I have written code that finds the largest valid positive in an array. Lets say there are a few obvious options:

// Parse it and test it
const nv = +nc;
pos = Number.isInteger(nv) && 0 < nv;

// A regular expression
pos = /^[1-9][0-9]*$/.test(nc);

// A custom function
const strIsPositive = (x) => {
   if ( 'string' !== typeof x || '' === x ) return false;
   const min = 48; // 0
   const max = 57; // 9
   let   cc  = x.charCodeAt(0);
   if ( cc <= min || max < cc ) return false;
   for ( let i=1 ; i<x.length ; i++ ) {
     cc = x.charCodeAt(i);
     if ( cc < min || max < cc ) return false;
   }
   return true;
 }
pos = strIsPositive(nc);

Well, I wrote some benchmark code and ran it in Node.js, and there are some quite predictable findings.

It is no huge difference between the alternatives above, but there are differences (1ms for 10000 validations, on a 4th generation i5).

There is no silver bullet, the optimal solution depends on.

If all you want is validation, it is wasteful to convert (+nc). A Regular expression is faster, but you can easily beat a Regular expression with a simple loop.

If most numbers are valid, converting to number (+nc) makes more sense. It is expensive to parse invalid values to NaN.

If you are going to use the number, converting to number (+nc) makes sense (if you convert only once).

The fastest solution, both for valid and invalid numbers, is to never convert to number (but use the custom function above to validate) and find the max using string compare.

if ( strIsPositive(nc) &&
     ( max.length < nc.length ) || ( max.length === nc.length && max < nc )
   )
  max = nc; 

This is obviously not a generally good advice.

Other numeric formats

My above findings are for strings containing positives. I have tested both code that only validates, and code that use the value by comparing it.

You may not have positives but:

  • Naturals, including 0, which creates a nastier regular expression but an easier loop.
  • Integers, including negative values, which creates even nastier regular expressions.
  • Ranged integers, like [-256,255], which probably means you want to parse (+nc) right away.
  • Decimal values
  • Non standard formats (with , instead of . for decimal point, or with delimiters like spaces to improve readability)
  • Hex, scientific formats, whatever

In the end readability is usually more important than performance.

Force Vue Update ($forceUpdate)

Occationally you want to force some part of your Vue application to update. One situation is that I have a “big” web application not written in Vue, and somewhere I add a Vue component. Something in the world around it changes but it is not aware of it, so I want to force it to update.

It seems, the vm.$forceUpdate method is just updating the component itself, and children with slots. I didn’t use slots, and $forceUpdate was useless.

So, if you do myVue.$forceUpdate() without success, try:

    myVue.$children.forEach(c => c.$forceUpdate());

It might do what you want. It did for me.

Xcode findings

As I start experimenting with Xcode I realise that it is a tricky beast.

Xcode 10.2.1

I realised Xcode 10.2.1 used 100%+ CPU. I fixed that by reinstalling it completely.

Reainstalling Xcode I had managed to mess upp the simulators.
Error: Unable to boot device because it cannot be located on disk
Solution: Run in Terminal: xcrun simctl erase all

Xcode 7.3.1

Xcode 7.3.1 Fails to start on macOS 10.14.5.

A first iOS app with Xcode 10.2.1

Ten years too late I decided to look into iOS development. It is too late, because the Klondyke era of becoming a millionaire on simple apps is probably over. On the other hand Swift has arrived and reached version 5 so it should be a good time to get started.

What I have is

  • Mac OS 10.14.5
  • Xcode 10.2.1
  • iPhone 6s, iOS 12.2 to deploy to
  • iPad 3, iOS 9.3.5 (obsolete by Apple standard)
  • 20 years of programming experience
  • Very limited experience with Swift 5
  • No experience with Xcode, Objective-C or macOS development

I am mostly a backend-programmer, who have to do HTML/CSS/JavaScript as well. Xcode is creepy. I have thought about a few appoaches

  1. Buying a book (but a challenge to find a book with relevant complexity, mix of tutorial/reference, for Xcode 10 / Swift 5)
  2. Apples obsolete tutorial (but I was put off by the fact that it is written for Swift 3)
  3. Just playing around with Xcode (just kidding – that is too scary)
  4. Some online course, like Udemy (but it is not my way)
  5. A simple trumpet tutorial

I went for (5). It was good, because in a few hours it took me all the way from starting Xcode to running something on my iPhone.

Building for the simulator and running works. And I managed to deploy to my iPhone (it is actually quite self explanatory: connect the iPhone, select it as destination in Xcode, and later in the iPhone under settings -> general -> device management you allow the app to run).

The short version is that it all went well! But…

Obsolete iPad 3

I failed to build for my obsolete iPad 3. What happens is that all is fine, and then I come to this screen:

I type my password, and immediately it (building/signing) “Failed with exit code 1”. I can imagine two options right away

  1. I need a real developer license (not Personal Team) to do this
  2. I need an older version of Xcode to build for 9.3
    (and in that case I might need to use older project format, and perhaps not even Swift 5, I don’t know)
  3. I got some indication that with a Personal (free) developer license I can only deploy to a single test device, that would perhaps not include old devices

It actually only builds for Deployment target 12.2, no older versions in the list.

Update: Page 60 of the free Apple Book “App Development With Swift” tells clearly that a free account only supports a single device. So it is clearly a waste of time to ignore that restriction and try to deploy to my iPad.

Xcode

I have spent a few hours with this now. I wrote 4 lines of code. I have ctrl-clicked on things, dragged-and-dropped-things, added properties to things, added resources, opened panels and used shortcuts. If you are used to things like Visual Studio it will probably feel somewhat familiar. But for me, who mostly use Vim, it is very scary.

Update: Xcode turned out to use 100%+ CPU constantly. I completely removed it and reinstalled it, and it seemed to help.

Computer Requirements / Performance

I did these experiments on a MacBook Pro 6,2 (that officially does not support macOS 10.14). It has an SSD drive and 8GB or RAM. Building takes almost 10 seconds, but starting the simulator and loading the app takes almost a minute. The computer clearly gets warm. Neither Xcode nor the simulator consumes much memory (Activity Monitory says about 200Mb each). Obviously, if you run the simulator much in your daily work, a faster CPU is worth it.

I think my 1440×900 display may be the biggest problem if I want to do anything real thought.

Conclusion

I have mixed feelings, it could be worse and better. I clearly need to find a way to be quickly guided through building different types of apps. I think I need a few days being guided through Xcode until both Xcode and the different project artifacts feel somewhat natural.

I have a simple app I want to build for myself, but right now it feels much to intimidating.

I found that Apple has released a free online book (available in their Books application) called App Development with Swift. That seems to be a good option.

Simple Mobile First Design

If you build a web site today you need to think about the experience on mobiles, tablets and desktops with different screen sizes. This is not very easy. In this article I have applications (SPAs) in mind rather than sites/pages.

If you are a real, ambitious, skilled designer with a significant budget, there is nothing stopping you from doing it right. Responsive design is dead, because most often you have no choice, so it is just design.

However, you may not have that budget, skill, time and ambition, but you still need to think about vastly different screen sizes. Or perhaps you just need to build a simple native-app-like website.

Two separate implementations

In many cases I would argue that it makes sense to simply make a separate site for mobile and desktop. There are many arguments but I will give one: use cases are often very different. A desktop app is often opened, kept open for a long time, and much data may be presented and analysed on screen, in memory. A mobile app is often opened shortly, to accomplish a single task, and then closed. This means that you probably want to manage state, data and workflow very differently as well.

Bootstrap (or similar)

There are frameworks (like Bootstrap) and technologies like Flexbox to allow you to build a responsive app. Before using those, I think you should ask yourself a question.

How do you want to take advantage of more screen space?

Think of regular desktop applications (Word, Photoshop, Visual Studio) or your operating system: when you have more screen available you can have more stuff next to each other. You can have more windows and more panels at the same time. Mostly. Also, but less so, small things get larger (when they benefit from it). It helps to be able to see an entire A4 page when you work with Word. But when you have an Excel sheet with 4 used columns, those don’t use your entire screen just because they can.

Bootstrap tends to create larger space between elements, and larger elements where it is not needed (dropdown <select>, input fields). I say tends to, because if you are good and very careful, you can probably do a better job than I can. But it is not automatic and it is not trivial, to make it good

What I mean is that if my calendar/table looks gorgeous when it is 400px wide, what good does it make to make it larger if the screen gets larger? So I think a better approach to responsiveness is to say that my calendar/table takes 400px. If I have more space available, I can show something else as well.

Mobile Screen Sizes

To complicate things further, mobile phones have different screen sizes, different screen resolutions, and then there are hi-resolution screens that have different virtual and physical resolutions.

So you have your table that looks good on a “standard” mobile with 320px width. What do you want to do if the user has a better/larger screen?

  1. make it look exactly the same (just better/larger)?
  2. reactively change the way your app looks and works?

If you are opting for (2), I need to wonder why, really?

I argue that if you pick (1) you can make development, testing, documentation and support easier. And your users will have a more consistent experience. At the expense that those with a large mobile may not get the most out of it when using your app.

I propose a simple Mobile First Responsive design

What I propose is not for everyone and everywhere. It may suck for your product and project. That is fine, there are different needs.

I propose a Mobile First (Semi-)Responsive design:

  1. Pick a width (320px is fine).
  2. Design all parts, all pages, all controllers of your app for that width.
  3. On mobile, set the viewport to your width for consistent behaviour on all mobiles.
  4. Optionally, on desktop (and possibly tablets), allow pages to open next to each other rather than on top of (and hiding) each other to make some use of more screen when available.

Seems crazy? Please check out my Proof of Concept and decide for yourself! It is only a PoC. It is not a framework, not a working app, not demonstrating Vue best practices, and it is not very pretty. Under Settings (click ?) you can check/change between Desktop, Tablet and Mobile mode (there is a crude auto-discover mechanism in place but it is not perfect). You can obviously try it with “Responsive Design Mode” in your browser and that should work quite fine (except some elements don’t render correctly).

Implementation Details

First, I set (despite this is not normally a recommended thing to do):

<meta id="viewport" name="viewport" content="width=320">

Later I use JavaScript to change this to 640 on a tablet, to allow two columns. Desktops should ignore it.

Second, I use a header div fixed at the top, a footer div fixed at the bottom, and the rest of the page has corresponding margins (top/bottom).

.app_headers {
   position: fixed;
   top: 0;
   left: 0;
 }
 .app_header {
   float: left;
   height: 30px;
   width: 320px;
 }
 .app_footers {
   position: fixed;
   bottom: 0;
   left: 0;
 }
 .app_footer {
   float: left;
   height: 14px;
   width: 320px;
 }
 .app_pages {
   clear: both;
 }
 .app_page {
   margin-top: 30px;
   margin-bottom: 12px;
   width: 320px;
   float: left;
 }

In mobile mode I just add one app_header, app_footer and app_page (div with class). But for Tablets and Desktops I can add more of them (equally many) as the user navigates deeper into the app. It is basically:

<div class="app_headers">
  <div class="app_header">
    Content of first header (to the left)
  </div>
  <div class="app_header">
    Content of second header (to the right)
  </div>
</div>
<div class="app_pages">
  <div class="app_page">
    Content of first page (to the left)
  </div>
  <div class="app_page">
    Content of second page (to the right)
  </div>
</div>
<div class="app_footers">
  <div class="app_footer">
    Content of first footer (to the left)
  </div>
  <div class="app_footer">
    Content of second footer (to the right)
  </div>
</div>

I use little JavaScript to not add too many pages side-by-side should the display/window not be large enough.

It is a good idea to reset margins, paddings and borders to 0 on common items.

I also found that you need a font size of 16px on iPhone, otherwise the Apple mobile Safari browser will immediately zoom when user edits <input> and <select>.

Most effort when I wrote my Proof of Concept was

  1. Getting the HTML/CSS right and as simple as possible (I am simply not good enough with HTML/CSS to just get it right)
  2. Implementing a “router” that supports this behaviour

Being able to scroll the different pages separately would be possible, a bit more complicated, and perhaps not so desirable.

Conclusions

Exploiting the viewport you can build a web app that works fine on different mobiles, and where the issue with different screen sizes and screen resolution is quite much out of your way.

The site will truly be mobile-first, but with the side-by-side-strategy presented, your users can take advantage of larger screens on non-mobiles as well.

This way, you can build a responsive app, with quite little need for testing on different devices as the app grows. You just need to keep 320px in mind, and have a clear idea about navigating your site.

First look at Swift

Apple invented the Swift programming language to make application programming for iOS and macOS a better experience. If you are new to all this (as I am), I guess there are three approaches (depending on your background):

  1. Learn with the Swift Playground App for iOS
  2. Find a book/guide/tutorial to build actual iOS apps (learning Swift along the way)
  3. Use tools that you are used to, solving problems you are familiar with, using Swift (a programmers’ approach)

I decided to just write some Swift code. There is a cool web page called Rosettacode.org with implementations of different “problems” in different languages. I started looking at Swift code there to see if I could learn anything, and decided I could to better. (Admittedly, that is quite arrogant: I have never written a line of Swift code before, and now contribute Swift code)

I started looking at the problem Caesar Encryption and solved it for Swift. The full code comes below (in case someone changes it on Rosettacode)

I have a C/C#/Java/JavaScript background. This is what I find most notable about Swift.

Backward declaration of variables, arguments and function return types. Type comes after the name (with colon in between).

Named parameters to function, unless you prepend an _ to the name.

Closures can be written (quite just) like in JavaScript. (see charRotateWithKey in the caesar function)

Wrapping/optional: a normal variable, after it is declared must have a valid value. The language ensures this for you. Look at the first line in the function charRotate below: the ! means that if the parameter c does not have an ascii value the program will terminate right there. Look at the line starting with guard in main. The language guarantees that key is a valid integer after the guard, otherwise the function (program) must exit. I am far from an expert on this, find a better source! But you can’t do what you do in C/C#/Java/JavaScript – just hope it goes well, and if it does not catch an exception or deal with it afterwards.

ARC rather than garbage collection or explicit memory management. This matters not in my program, but it is worth mentioning. I first thought Swift and Rust were very similar and that it is more or less an incident that they are different languages, but I don’t really think so anymore.

The swift command can be used not only to compile a source file. It can be used to set up a swift project (directory), run tests, run the REPL (read-eval-print-loop) and more things. This seems quite nice, but I will write no more of it here.

My program below demonstrates type conversions, command arguments, usage of map and closures, string and ascii low level operations and output.

I think Swift is a quite fine language that I would be happy to use. I notice that the language has evolved quite much over the few years it has exited. So when you find things on the web or stackoverflow, you might not find current best practices.

func usage(_ e:String) {
   print("error: \(e)")
   print("./caeser -e 19 a-secret-string")
   print("./caeser -d 19 tskxvjxlskljafz")
 }
  
 func charIsValid(_ c:Character) -> Bool {
   return c.isASCII && ( c.isLowercase || 45 == c.asciiValue ) // '-' = 45
 }
  
 func charRotate(_ c:Character, _ by:Int) -> Character {
   var cv:UInt8! = c.asciiValue
   if 45 == cv { cv = 96 }  // if '-', set it to 'a'-1
   cv += UInt8(by)
   if 122 < cv { cv -= 27 } // if larget than 'z', reduce by 27
   if 96 == cv { cv = 45 }  // restore '-'
   return Character(UnicodeScalar(cv))
 }
  
 func caesar(_ enc:Bool, _ key:Int, _ word:String) -> String {
   let r = enc ? key : 27 - key
   func charRotateWithKey(_ c:Character) -> Character {
     return charRotate(c,r)
   }
   return String(word.map(charRotateWithKey))
 }
  
 func main() {
   var encrypt = true
  
   if 4 != CommandLine.arguments.count {
     return usage("caesar expects exactly three arguments")
   }
  
   switch ( CommandLine.arguments[1] ) {
   case "-e":
     encrypt = true
   case "-d":
     encrypt = false
   default:
     return usage("first argument must be -e (encrypt) or -d (decrypt)")
   }
  
   guard let key = Int(CommandLine.arguments[2]) else {
     return usage("second argument not a number (must be in range 0-26)")
   }
  
   if key < 0 || 26 < key {
     return usage("second argument not in range 0-26")
   }
  
   if !CommandLine.arguments[3].allSatisfy(charIsValid) {
     return usage("third argument must only be lowercase ascii characters, or -")
   }
  
   let ans = caesar(encrypt,key,CommandLine.arguments[3])
   print("\(ans)")
 }
  
 func test() {
   if ( Character("a") != charRotate(Character("a"),0) ) {
     print("Test Fail 1")
   }
   if ( Character("-") != charRotate(Character("-"),0) ) {
     print("Test Fail 2")
   }
   if ( Character("-") != charRotate(Character("z"),1) ) {
     print("Test Fail 3")
   }
   if ( Character("z") != charRotate(Character("-"),26)) {
     print("Test Fail 4")
   }
   if ( "ihgmkzma" != caesar(true,8,"a-zecret") ) {
     print("Test Fail 5")
   }
   if ( "a-zecret" != caesar(false,8,"ihgmkzma") ) {
     print("Test Fail 6")
   }
 }
  
 test()
 main()

Performance, Node.js & Sorting

I will present two findings that I find strange in this post:

  1. The performance of Node.js (V8?) has clearly gotten consistently worse with newer Node.js versions.
  2. The standard library sort (Array.prototype.sort()) is surprisingly slow, often slower than a simple textbook mergesort.

My findings in this article are based on running a simple program mergesort.js on different computers and different node versions.

You may also want to read this article about sorting in Node.js. It applies to V8 version 7.0, which should be used in Node.js V11.

The sorting algorithms

There are three sorting algorithms compared.

  1. Array.prototype.sort()
  2. mergesort(), a textbook mergesort
  3. mergesort_opt(), a mergesort that I put some effort into making faster

Note that mergesort is considered stable and not as fast as quicksort. As far as I understand from the above article, Node.js used to use quicksort (up to V10), and from V11 uses something better called Timsort.

My mergesort implementations (2) (3) are plain standard JavaScript. Nothing fancy whatsoever (I will post benchmarks using Node.js v0.12 below).

The data to be sorted

There are three types of data to be sorted.

  1. Numbers (Math.random()), compared with a-b;
  2. Strings (random numbers converted to strings), compared with default compare function for sort(), and for my mergesort simple a<b, a>b compares to give -1, 1 or 0
  3. Objects, containing two random numbers a=[0-9], b=[0-999999], compared with (a.a-b.a) || (a.b-b.b). In one in 10 the value of b will matter, otherwise looking at the value of a will be enough.

Unless otherwise written the sorted set is 100 000 elements.

On Benchmarks

Well, just a standard benchmark disclaimer: I do my best to measure and report objectively. There may be other platforms, CPUs, configurations, use cases, datatypes, or array sizes that give different results. The code is available for you to run.

I have run all tests several times and reported the best value. If anything, that should benefit the standard library (quick)sort, which can suffer from bad luck.

Comparing algorithms

Lets start with the algorithms. This is Node V10 on different machines.

(ms)     ===== Numbers =====   ===== Strings =====   ==== Objects =====
sort() merge m-opt sort() merge m-opt sort() merge m-opt
NUC i7 82 82 61 110 81 54 95 66 50
NUC i5 113 105 100 191 130 89 149 97 72
NUC Clrn 296 209 190 335 250 196 287 189 157
RPi v3 1886 1463 1205 2218 1711 1096 1802 1370 903
RPi v2 968 1330 1073 1781 1379 904 1218 1154 703

The RPi-v2-sort()-Numbers stand out. Its not a typo. But apart from that I think the pattern is quite clear: regardless of datatype and on different processors the standard sort() simply cannot match a textbook mergesort implemented in JavaScript.

Comparing Node Versions

Lets compare different node versions. This is on a NUC with Intel i5 CPU (4th gen), running 64bit version of Ubuntu.

(ms)     ===== Numbers =====   ===== Strings =====   ==== Objects =====
sort() merge m-opt sort() merge m-opt sort() merge m-opt
v11.13.0 84 107 96 143 117 90 140 97 71
v10.15.3 109 106 99 181 132 89 147 97 71
v8.9.1 85 103 96 160 133 86 122 99 70
v6.12.0 68 76 88 126 92 82 68 83 63
v4.8.6 51 66 89 133 93 83 45 77 62
v0.12.9 58 65 78 114 92 87 55 71 60

Not only is sort() getting slower, also running “any” JavaScript is slower. I have noticed this before. Can someone explain why this makes sense?

Comparing different array sizes

With the same NUC, Node V10, I try a few different array sizes:

(ms)     ===== Numbers =====   ===== Strings =====   ==== Objects =====
sort() merge m-opt sort() merge m-opt sort() merge m-opt
10 000 10 9 11 8 12 6 4 7 4
15 000 8 15 7 13 14 11 6 22 7
25 000 15 35 12 40 27 15 11 25 18
50 000 35 56 34 66 57 37 51 52 30
100 000 115 107 97 192 138 88 164 101 72
500 000 601 714 658 1015 712 670 698 589 558

Admittedly, the smaller arrays show less difference, but it is also hard to measure small values with precision. So this is from the RPi v3 and smaller arrays:

(ms)     ===== Numbers =====   ===== Strings =====   ==== Objects =====
sort() merge m-opt sort() merge m-opt sort() merge m-opt
5 000 34 57 30 46 59 33 29 52 26
10 000 75 129 64 100 130 74 63 104 58
20 000 162 318 151 401 290 166 142 241 132
40 000 378 579 337 863 623 391 344 538 316

I think again quite consistently this looks remarkably bad for standard library sort.

Testing throughput (Version 2)

I decided to measure throughput rather than time to sort (mergesort2.js). I thought perhaps the figures above are misleading when it comes to the cost of garbage collecting. So the new question is, how many shorter arrays (n=5000) can be sorted in 10s?

(count)  ===== Numbers =====   ===== Strings =====   ==== Objects =====
sort() merge m-opt sort() merge m-opt sort() merge m-opt
v11.13.0 3192 2538 4744 1996 1473 2167 3791 2566 4822
v10.15.3 4733 2225 4835 1914 1524 2235 4911 2571 4811
RPi v3 282 176 300 144 126 187 309 186 330

What do we make of this? Well the collapse in performance for the new V8 Torque implementation in Node v11 is remarkable. Otherwise I notice that for Objects and Node v10, my optimized algorithm has no advantage.

I think my algorithms are heavier on the garbage collector (than standard library sort()), and this is why the perform relatively worse for 10s in a row.

If it is so, I’d still prefer to pay that price. When my code waits for sort() to finish there is a user waiting for a GUI update, or for an API reply. I rather see a faster sort, and when the update/reply is complete there is usually plenty of idle time when the garbage collector can run.

Optimizing Mergesort?

I had some ideas for optimizing mergesort that I tried out.

Special handling of short arrays: clearly if you want to sort 2 elements, the entire mergesort function is heavier than a simple function that sorts two elements. The article about V8 sort indicated that they use insertion sort for arrays up to length 10 (I find this very strange). So I implemented special functions for 2-3 elements. This gave nothing. Same performance as calling the entire mergesort.

Less stress on the garbage collector: since my mergesort creates memory nodes that are discarded when sorting is complete, I thought I could keep those nodes for the next sort, to ease the load on the garbage collector. Very bad idea, performance dropped significantly.

Performance of cmp-function vs sort

The relevant sort functions are all K (n log n) with different K. It is the K that I am measuring and discussing here. The differences are, after all, quite marginal. There is clearly another constant cost: the cost of the compare function. That seems to matter more than anything else. And in all cases above “string” is just a single string of 10 characters. If you have a more expensive compare function, the significance of sort() will be even less.

Nevertheless, V8 is a single threaded environment and ultimately cycles wasted in sort() will result in overall worse performance. Milliseconds count.

Conclusions

Array.prototype.sort() is a critical component of the standard library. In many applications sorting may be the most expensive thing that takes place. I find it strange that it does not perform better than a simple mergesort implementation. I do not suggest you use my code, or start looking for better sort() implementations out there right away. But I think this is something for JavaScript programmers to keep in mind. However, the compare function probably matters more in most cases.

I find it strange that Node v11, with Timsort and V8 Torque is not more of an improvement (admittedly, I didnt test that one very much).

And finally I find it strange that Node.js performance seems to deteriorate with every major release.

Am I doing anything seriously wrong?