Without network operations, running a website for booking accommodation online would be nearly impossible. Network operations can be anything from simple actions like talking to a browser with a user at the other end, to providing an API for our affiliates, or even writing the internal services that help maintain our systems.

Network operations are everywhere, and these are only a few examples of where we use them.

What is a network socket

Network communication typically happens through the use of sockets.

A network socket is one of the core software components of all modern operating systems. It is a software interface for network services provided by the operating system. It provides a uniformed way of opening a connection to this service and sends and receives data.

Network sockets are used everywhere in the IT world, and they allow us to communicate between different hosts or different programs on the same host. Despite having different kinds of network services (TCP and UDP being prominent examples), network sockets provide a common way to interact with them.

Here is an example of interacting with the Google.com website using IO::Socket::INET, the standard Perl socket module. (IO means Input/Output, and INET means Internet.)

# example 1my$socket=IO::Socket::INET->new('google.com:80');print{$socket}"GET / \n";my$html=join'',<$socket>;

Interestingly, IO::Socket::INET is mostly used for its Object Oriented capable interface. The following example performs the same operations as the previous one, but in an object oriented way:

# example 2my$socket=IO::Socket::INET->new(PeerHost=>'www.booking.com',PeerPort=>80,);$socket->print("GET / \n");my$html=join'',$socket->getlines();

Why sockets timeouts are important

At Booking.com, the handling of requests in a timely manner is critical to the user experience, to operations, and ultimately to our business. To achieve speed and low latency, our platform involves different subsystems and resources are constantly requested.

It is essential that these systems reply quickly. It is also vital that we detect when one of them isn't replying fast enough, that way it can be flagged and a mitigating strategy can be found (such as using an alternative subsystem).

We use Redis in a lot of places: as a cache layer, queue system, and for storage. Redis offers very low latency, and we make use of this feature. However, with sockets, we don't always know immediately when a connection has been lost. Knowing a Redis server is unreachable 30 seconds after the fact - is 30 seconds too late. Our goal is to know this in under a second. For other cases it might be possible (or even mandatory) to allow a longer timeout. It really depends on the subsystems involved as well as the usage.

Most of the time these subsystems are queried using a network socket. So being able to detect that a subsystem is not reachable implies that the sockets provide a way to specify timeouts.

This is why having a fast and reliable platform relies on having sockets that support timeouts. Using a socket involves three main steps: connecting to the external server, reading and writing data from and to it, and, at some point, closing the connection. A socket timeout implementation should allow for setting the timeout at connections, and both reading and writing steps at the very least.

Connection timeout

IO::Socket provides a timeout method, and IO::Socket::INET provides a Timeout option. The Timeout option can be used to set a timeout on the connection to the server. For example, this is how we connect to a local HTTP server on port 80 with a connection timeout of 3 seconds:

my$socket=IO::Socket::INET->new(PeerHost=>'127.0.0.1',PeerPort=>80,Timeout=>3,);

So far so good, but how do we deal with read or write timeouts? What if the server accepts the connection, but then at some point stops communicating? The client socket needs to realize this quickly. We need timeouts for this.

Read/Write timeouts via setsockopt

It is relatively easy to change the option of a socket to supply these timeouts. This is an example that works on GNU/Linux, given $timeout in (optionally fractional) seconds:

my$seconds=int($timeout);my$useconds=int(1_000_000*($timeout-$seconds));my$timeout=pack('l!l!',$seconds,$useconds);$socket->setsockopt(SOL_SOCKET,SO_RCVTIMEO,$timeout)# then use $socket as usual

The only problem is that it only works on some architecture and operating systems. A generic solution is better. Let's look at the available options on systems that do not support setsockopt.

Read/Write timeouts via select

Another more portable (albeit slower) way to simulate a timeout on a socket is to check if the socket is readable/writable with a timeout in a non-blocking way. select(2) can do this, and the Perl select() function can provide access to it.

Here is a simplified version of a function that returns true if we can read the socket with the given timeout:

sub _can_read{my($file_desc,$timeout)=@_;vec(my$fdset='',$file_desc,1)=1;my$nfound=select($fdset,undef,undef,$timeout);}

Using an external library

Yet another way is to use external modules or system calls, like epoll (via IO::Poll), libevent, or libev. To simplify things, it's common to use higher-level event-based modules like AnyEvent and POE. They make it easy to specify a timeout to any IO (Input/Output) operations.

This is an example using AnyEvent, which will set a connection timeout of 0.5 second and a read or write timeout of 0.01 second:

my$handle=AnyEvent::Handle->new(connect=>[$host,$port],on_prepare=>sub {0.5},# ...);$handle->on_timeout(sub {say'timeout occurred'});$handle->timeout(0.01);

While completely valid, it applies only to programs that use these event-based modules. It is useless to standard imperative programs. We need a method for providing timeout features to the standard socket API without changing the operation needed to require an event loop.

Provide a nice API

Let's step back for a moment. We have two ways to setup a timeout on a socket:

• A one-time setting on the socket with setsocket.
• A change to the way we interact with the socket with select.

We need to abstract these two ways of setting timeouts behind a simple and easy-to-use API. Let's consider this example:

my$socket=IO::Socket::INET->new(...);print{$socket}'something';

(Please note that we don't use object-oriented notations on the socket.)

What we want is an easier way to set timeout on the $socket. For example this:

my$socket=IO::Socket::INET->new(...);# set timeouts$socket->read_timeout(0.5);# use the socket as beforeprint{$socket}'something';# later, get the timeout valuemy$timeout=$socket->read_timeout();

when using setsockopt

If we can use setsockopt, setting the timeout using ->read_timeout(0.5) is easy. It can be implemented as a method that we add to IO::Socket::INET class, possibly by using a Role.

This method would just fire setsockopt with the right parameters, and save the timeout value into $socket for later retrieval. Then we can carry on using $socket as before.

One subtlety is that, because the $socket is not a classic hash reference instance, but an anonymous typeglob on a hash reference, instead of doing $socket->{ReadTimeout} = 0.5 we need to do ${*$socket}{ReadTimeout} = 0.5 ... but that's just an implementation detail.

when using select

If however the program is running in a situation where setsockopt can't be used, we have to resort to using the select method. That poses a problem. Because we're not using object oriented programming, the operation on the socket is not done via a method we could easily override, but directly using the built-in function print.

Overwriting a core function is not a good practice for various reasons. Luckily, Perl provides a clean way to implement custom behavior in the IO layer.

PerlIO layers

Perl Input/Output mechanism is based on a system of layers. It is documented in the perliol(1) man page.

What's the PerlIO API? It's a stack of layers that live between the system and the perl generic file-handle API. Perl provides core layers (such as :unix, :perlio, :stdio, and :crlf). It also provides extension layers (such as :encoding and :via).

These layers can be stacked and removed in order to provide more features (when layers are added) or more performance (when layers are removed).

The huge benefit is that no matter which layers are setup on a file handle or socket, the API doesn't change and the read/write operations are the same. Calls to them will go through the specified layers attached to the handle until they potentially reach the system calls.

Here is an example:

openmy$fh,'filename';# for direct binary non-buffered accessbinmode$fh,':raw';# specify that the file is in utf8, and enforce validationbinmode$fh,':encoding(UTF-8)';my$line=<$fh>;

The :via layer is a special layer that allows anyone to implement a PerlIO layer in pure Perl. Contrary to implementing a PerlIO layer in C, using the :via layer is rather easy: it is just a Perl class, with some specific methods. The name of the class is given when setting the layer:

binmode$fh,':via(MyOwnLayer)';

Many :via layers already exist. They all start with PerlIO::via:: and are available on CPAN. For instance, PerlIO::via::json will automatically and transparently decode and encode the content of a file or a socket from and to JSON.

Back to the problem. We could implement a :via layer that makes sure that read and write operations on the underlying handle are performed within the given timeout.

Implementing a timeout PerlIO layer

A :via layer is a class that should start with PerlIO::via:: and implement a set of methods, like READ, WRITE, PUSHED, and POPPED - (see the PerlIO::via manual for more details).

Let's take the READ method as an illustration. This is a very simplified version. The real version handles EINTR and other corner cases.

packagePerlIO::via::Timeout;sub READ{my($self,$buf,$len,$fh)=@_;my$fd=fileno($fh);# we use the same can_read as previouslycan_read($fd,$timeout)orreturn0;returnsysread($fh,$buf,$len,0);}

The idea is to check if we can read on the filesystem using select in the given timeout. If not, return 0. If yes, call the normal sysread operation. It's simple and it works great.

We've just implemented a new PerlIO layer using the :via mechanism! A PerlIO layer works on any handle, including file and socket. Let's try it on a file-handle:

usePerlIO::via::Timeout;openmy$fh,'<:via(Timeout)','foo.html';my$line=<$fh>;if($line==undef&&0+$!==ETIMEDOUT){# timed out reading...}else{# we read one line fast enough, success!...}

I'm sure you can see that there is an issue in the code above. At no point do we set the read timeout value. The :via pseudo-layer doesn't allow us to easily pass a parameter to the layer creation. Though we can technically, we would not be able to change the parameter afterwards. If we want to be able to set, change, or remove the timeout on the handle at any time, we need to somehow attach this information to the handle, and we need to be able to change it.

Add a properties to a Handle using InsideOut OO

A handle is not an object. We can't just add a new timeout attribute to a handle and then set or get it.

Luckily, the moment a handle is opened it receives a unique ID: its file descriptor. A file descriptor is not always unique because they are recycled and reused. Yet, if we know when a handle is opened and closed we can be sure that between these actions a file descriptor is given that uniquely identifies it.

The :via PerlIO layer allows us to implement PUSHED, POPPED, and CLOSE. These functions are called when the layer is added to the handle, when it's removed, and when the handle is closed. We can use these functions to detect if and when to consider the file descriptor as a unique ID for the given handle.

We can create a hash table as a class attribute of our new layer. Here the keys are file descriptors and the values are a set of properties on the associated handle -- essentially a basic implementation of Inside-Out OO - with the object not being its data structure only an ID. Using this hash table, we can associate a set of properties to a file descriptor and set the timeout value when the PerlIO layer is added. Like this:

my%fd_properties;sub PUSHED{my($class,$mode,$fh)=@_;$fd_properties{fileno($fh)}={read_timeout=>0.5};# ...}

By doing this when we remove the layer too, we can also implement a way to associate timeout values to the file-handle.

Wrapping up all the bits of code and features, the full package that implements this timeout layer, PerlIO::via::Timeout, is available on Github and CPAN.

Implement the API

We now have all the ingredients we need to implement the desired behavior. enable_timeouts_on will receive the socket and modify its class (which should be IO::Socket::INET or inherited from it) to implement these methods:

• read_timeout: get/set the read timeout
• write_timeout: get/set the write timeout
• disable_timeout: switch off timeouts (while remembering their values)
• enable_timeout: switch back on the timeouts
• timeout_enabled: returns whether the timeouts are enabled

In order to modify the IO::Socket::INET class in a clean way, let's create a role and apply it to the class. In fact, let's create two roles: one that implements the various methods using setsockopt and another role that uses select (with PerlIO::via::Timeout).

A Role (sometimes known as Trait) provides additional behavior to a class in the form of composition. Roles provide introspection, mutual exclusion capabilities, and horizontal composition instead of the more widely used inheritance model. A class simply consumes a Role, receiving any and all behavior the Role provides, whether these are attributes, methods, method modifiers, or even constraints on the consuming class.

Detailing the implementation of the role mechanism here is a bit out of the scope, but it's still interesting to note that to keep IO::Socket::Timeout lightweight, we don't use Moose::Role or Moo::Role, but instead we apply a stripped down variant of Role::Tiny, which uses a single inheritance of a special class crafted in real time specifically for the targeted class. The code is short and can be seen here.

wrap it up

Use IO::Socket::Timeout to add read/write timeouts to any network socket created with IO::Socket::INET, on any platform:

# 1. Creates a socket as usualmy$socket=IO::Socket::INET->new(...);# 2. Enable read and write timeouts on the socketIO::Socket::Timeout->enable_timeouts_on($socket);# 3. Setup the timeouts$socket->read_timeout(0.5);$socket->write_timeout(0.5);# 4. Use the socket as usualmy$data=<$socket>;# 5. Profit!

Conclusion

IO::Socket::Timeout provides a lightweight, generic, and portable way of applying timeouts on sockets, and it plays an important role in the stability of the interaction between our subsystems at Booking.com. This wouldn't be possible without Perl's extreme flexibility.

Please be aware that there is a performance penalty associated with implementing IO layers in pure Perl. If you are worried about this, we recommend benchmarking when making the decision on whether to use it.

IO::Socket::Timeout is available on GitHub and CPAN.

Parallel replication is a much-expected feature of MySQL. It’s available in MariaDB 10.0 and in MySQL 5.7. Yet, both lose efficiency when replicating through intermediate masters. In this post, we’ll explain how parallel replication works and why it does not play well with intermediate masters. We’ll also offer a solution (hint: it involves Binlog Servers).

In MySQL 5.5 and MariaDB 5.5, replication is single threaded: on a slave, the previous transaction must complete (commit) before the next transaction can start. This is not the case on a master where many transactions can make progress at the same time. The consequence is that a slave has less transaction processing capabilities than its master (for writes). This means that care must be taken to avoid loading a master with more writes than its slaves can execute. If this is not done well, the slaves will start lagging.

This single threaded implementation is disappointing:

• Year after year, servers get an increasing number of cores but only one can be used on slaves to apply transactions from the master
• MySQL (and its storage engines) gets better at using many cores but those improvements do not speed up replication on slaves

On the other hand, running transactions in parallel on a slave is not simple. If done badly, conflicting transactions could be run at the same time, which risks getting different results on master and slave. In order to be able to run transactions in parallel, a way to prevent that from happening must be implemented. For parallel replication in MySQL 5.6, transactions in different schemas are assumed to be non-conflicting and can be run in parallel on slaves. This is a good start but what about transactions in the same schema?

To provide parallel execution of transactions in the same schema, MariaDB 10.0 and MySQL 5.7 take advantage of the binary log group commit optimization. This optimization reduces the number of operations needed to produce the binary logs by grouping transactions. When transactions are committing at the same time, they are written to the binary log in a single operation. But if transactions commit at the same time, then they are not sharing any locks, which means they are not conflicting thus can be executed in parallel on slaves. So by adding group commit information in the binary logs on the master, the slaves can safely run transactions in parallel.

So far we have:

• The master identifying potential parallelism with group commit and making this information available to the slaves via the binary logs
• The slaves executing transactions in parallel using the information provided by the master

What happens when some of those slaves are intermediate masters? What kind of parallelism information will they pass to their slaves? To answer these questions, the first thing that needs to be highlighted is the fact that transactions committing together might have started at different points of time in the past. This situation is illustrated below where T1 and T2 started (B for begin) after T3 and T4, even if they are all committing at the same time (C for commit).

----Time---->T1:B---CT2:B---CT3:B-------CT4:B-------C

Because those transactions commit together on the master, they start together on the slaves but do not commit at the same time:

----Time---->T1:B---CT2:B---CT3:B-------CT4:B-------C

This will cause the group of 4 transactions from the master to be split into 2 groups of 2 transactions on slaves. This is not a big problem on leaf slaves but it is a major issue on intermediate masters. As an intermediate master will write those transactions as 2 commit groups in the binary logs, its slaves will not run 4 transactions in parallel but only 2 transactions at a time with a barrier between the groups as shown below.

----Time--------->T1:B---CT2:B---CT3:B-------CT4:B-------C

So, when using parallel replication (based on the group commit) with transactions of different execution times, the intermediate masters show less parallelism to their slaves than what is shown by their master. This is not only a supposition, this is clearly observed in our test environments.

To test parallel replication at Booking.com we set up four test environments, each consisting of a master and a slave, where each node is running MariaDB 10.0.16 (we’ve also done similiar tests with MySQL 5.7 and found results close to the ones you’ll see below). On each master, we run transactions from production and identify parallelism using group commit (more on the exact methodology in a follow up post). To monitor parallelism identification, the global status BINLOG_COMMITS and BINLOG_GROUP_COMMITS are checked regularly. The graphs below show the collected data (number of commits and group commits) for both master and slave in the four tests environments.

For the most part, the master and the slave are executing the same number of transactions (commits). But, the slave is executing more group commits than the master (less is better). If we divide the number of commits by the number of group commits, we get the average group commit size. Those are shown on the following graphs with the number of applier threads on the slave:

A smaller group size means that, if those slaves were intermediate masters, parallelism is lost when replicating through them. Looking at ratios, we can see how bad this is (slave divided by master, less than one for group commit means that the slave identifies less parallelism than its master). Note: we expect the commit ratio to be 1, as if it were not, the slave would be lagging.

This is very disappointing. In our test environments, using true production workload, intermediate masters divide the parallelism available to their slaves by at least 2 and sometimes by almost 4.

What can be done to improve that? Having all slaves replicating directly from the master could be a solution. But that would increase the load on masters. It would also be impractical for remote site replication because it consumes too much WAN bandwidth as we often have 50 and sometimes more than 100 slaves per site.

People that have read previous posts from this blog might have guessed the solution:

Replace intermediate masters by Binlog Servers.

(Reminder: a Binlog Server sends its slaves exactly the same binary logs it fetched from its master.)

By doing this, all slaves replicate using the same binary logs (the one from the master) and all slaves, either directly connected to the master or replicating from a Binlog Server, see the same parallelism information.

To take full advantage of parallel replication, and while intermediate masters are unable to offer this information to their slaves, intermediate masters must disappear! The tool that will allow us to do that at Booking.com is the Binlog Server.

If you're interested in this topic and would like to learn more, I’ll be giving a talk about Binlog Servers at Percona Live Santa Clara in April. I am looking forward to meeting you there.

(The previous post, Better Parallel Replication for MySQL, is Part 1 of the series.) Parallel replication is a much expected feature of MySQL. It is already available in MariaDB 10.0 and in MySQL 5.7. In this post, a very nice side effect of the MariaDB implementation is presented: Slave Group Commit.

(If you are not familiar with parallel replication and its implementation, read the previous post in this series to better understand this post.)

To allow better parallelization on slaves, MariaDB 10.0 and MySQL 5.7 have parameters that control the commit group sizes on the master. Those parameters are binlog_commit_wait_count / binlog_commit_wait_usec in MariaDB and binlog_group_commit_sync_delay / binlog_group_commit_sync_no_delay_count in MySQL. The purpose of those parameters is to delay the commit of transactions on the master to allow more transactions to join a commit group. This increases the commit group size, and thus the number of transactions that can be run in parallel on slaves. Those parameters can be seen as a way to speed up slaves at the expense of slowing down the master.

When using those parameters on a MariaDB slave with parallel replication enabled (slave_parallel_threads> 1) and when this slave has a MariaDB 10.0 master running a single-threaded workload, transactions will be run sequentially with their commits delayed to try grouping transactions, hence achieving group commit on the slave.

Let's take the previous sentence one element at a time and explain each step in more detail.

In the replication hierarchy shown below:

• X is the master (any of MySQL 5.5, MySQL 5.6, MariaDB 5.5 or MariaDB 10.0)
• Y is an intermediate master running MariaDB 10.0
• Z is a slave running MariaDB 10.0 with binary logs and log-slave-updates enabled

-----     -----     -----
| X | --> | Y | --> | Z |
-----     -----     -----

The Y intermediate master is configured with the following parameters (single threaded slave):

• SET GLOBAL slave_parallel_threads = 0;
• SET GLOBAL binlog_commit_wait_count = 0;
• SET GLOBAL binlog_commit_wait_usec = 0;

And the Z slave is configured with the following parameters:

• SET GLOBAL slave_parallel_threads ="value greater than 1";
• SET GLOBAL binlog_commit_wait_count ="value greater than 1";
• SET GLOBAL binlog_commit_wait_usec ="value greater than 0";

Let's follow two transactions (T1 and T2) on Y and Z. On Y, the transactions are executed sequentially as shown below (B for begin and C for commit).

   ------Time----->
T1:  B----C
T2:        B----C

Once those are in the binary logs of Y, Z will execute them. As Y is single-threaded, Z cannot run the transactions in parallel (T1 and T2 are each in their own commit group in the binary logs of Y). However, as Z is delaying commit (binlog_commit_wait_count> 1 and binlog_commit_wait_usec> 0), T2 will start executing in another thread once T1 is ready to commit (slave_parallel_threads> 1). If T2 completes before the delay expires, T1 and T2 will group commit as shown below.

   -----Time----->
T1:  B---- . . C
T2:       B----C

Group committing is not limited to two transactions, it can extend to the transactions that follow and can result in much larger groups as shown below.

   ----------Time---------->
T1:  B-- . . . . . . . . C
T2:     B----- . . . . . C
T3:           B----- . . C
T4:                 B----C

Transaction grouping will stop in the following three situations:

1. The group size reaches binlog_commit_wait_count, grouping stops and all transactions get committed together.
2. binlog_commit_wait_usec expires, grouping stops, T1 and all the following completed transactions commit together.
3. While the commit of T1 is delayed, a subsequent transaction is blocked by a lock held by a previous transaction that got delayed at the commit stage, a transaction dependency has been found and the group commit pipeline will stall until binlog_commit_wait_usec expires (MariaDB 10.0.16).

Situation 3 above must be understood clearly. Each time a transaction dependency is found, the slave stops applying transactions until binlog_commit_wait_usec expires. This means that if transaction dependencies are frequent and binlog_commit_wait_usec is too big, the slave throughput will suffer. This could result in slave lag as shown below (T3 depends on T1 or T2).

   ----------------Time----------->
T1:  B-- . . . . . . . . C
T2:     B----- . . . . . C
T3:                       B-----C

This could be optimized in one of the next versions of MariaDB by detecting dependencies and triggering commit omitting the waiting [1].

(Note: in MariaDB 10.1, slave group commit can be enabled with the minimalslave_parallel_mode.)

Back to group committing on slaves, it allows the following:

1. Identifying transaction parallelism on a slave.
2. Speeding up the slave by reducing the number of disk syncs needed to write the binary logs when strong durability constraints are set (sync_binlog = 1 and innodb_flush_log_at_trx_commit = 1).

With reference to 1 above, this way of identifying parallelism will be the starting point of the next post in this series. Part 3 of the series will present speedups of parallel applying on slaves using Booking.com production workload. Moreover, you now know how the graphs presented in part 1 of the series were obtained.

With reference to 2 above, this could be a solution to a problem we are facing at Booking.com. To understand this problem, let's look at a Booking.com typical MySQL deployment:

-----
| M |
-----
  |
  +------+- ... -+---------------+
  |      |       |               |
-----  -----   -----           -----
| S1|  | S2|   | Sn|           | M1|
-----  -----   -----           -----
                                 |
                                 +- ... -+
                                 |       |
                               -----   -----
                               | T1|   | Tm|
                               -----   -----

• M is the standard master
• Si are the slaves of this master in the same datacenter
• M1 is an intermediate master on a remote datacenter
• Tj are the slaves of this intermediate master in the remote datacenter

To be able to survive the hardware failure of M1, the database is hosted on shared disks. Such shared disks could be implemented using DRBD or using filer-based storage accessed over fiber channel (SAN) or iSCSI (LAN). To be able to recover the database (and have the Tj slaves recover replication), high durability constraints need to be set on M1 (sync_binlog = 1 and innodb_flush_log_at_trx_commit = 1).

However, high durability constraints on a single-threaded slave mean that, after each transaction, a sync to the binary logs and a sync to the REDO logs are required. When a sync takes 1 millisecond (network round-trip), we cannot run more than 500 (or 1000 [2]) transactions per second.

The other leaf slaves (Si and Tj) do not have this limitation as their databases are hosted on the local disks with a battery backed-up write cache on the RAID controller (syncs are in fact just writes in the controller memory, so they are less expensive). Moreover, those slaves do not need binary logs and we can relax durability on them (innodb_flush_log_at_trx_commit = 0 or 2).

So M1 is an embarrassing bottleneck in the replication, and bursts of transactions on M occasionally cause replication delay in the remote datacenter.

Slave group commit could be a solution to alleviate this bottleneck, but would this work in the real life? To validate that, we did the following test:

-----     -----     -----
| A | --> | B | --> | C |
-----     -----     -----

• A is our true production master running MySQL 5.6
• B is an intermediate master running MariaDB 10.0.16 without parallel replication enabled (slave_parallel_threads = 0)
• C is a MariaDB 10.0.16 slave with binary log and log-slave-updates enabled, and with high durability enforced (sync_binlog = 1 and innodb_flush_log_at_trx_commit = 1)

The database on C is hosted on local disks with a battery backed-up write cache on the RAID controller. We ran the C slave with the following configuration:

• Until 17:05 in the graphs below, C is running in single-threaded mode (slave_parallel_threads = 0)
• At 17:05, slave group committing is enabled (slave_parallel_threads = 20, binlog_commit_wait_count = 15 and binlog_commit_wait_usec = 40.000)
• At 17:10, the write cache of the RAID controller is disabled (the slave is still group committing)
• At 17:15, group committing is disabled (slave_parallel_threads = 0 binlog_commit_wait_count = 0 and binlog_commit_wait_usec = 0)
• Finally at 17:17, the write cache is re-enabled

During those tests, the BINLOG_COMMITS and BINLOG_GROUP_COMMITS global statuses are gathered regularly giving the following graphs:

As we can see on the graphs above, in single-threaded mode and with the write cache enabled, the slaves are able to keep up with the flow of transactions from the master. When enabling group commit (at 17:05), the slaves are still able to keep up with the flow of transactions (binlog_commit_wait_usec does not cause a delay). We can see small behaviour changes after 17:10 when the write cache is disabled but three out of four slaves are able to keep up with the flow of transactions (the 3rd graph shows some slow down because lots of dirty page writes are happening at the same time, but no major delays happen). When the slaves are put back in single-threaded mode and stop group committing (17:15), we can see that their transaction processing capabilities are greatly impaired and that they are not able to keep up with the masters (they are committing much less than the master).

This shows that slave group commit is a good way of increasing throughput on slaves that have expensive disk sync (no write cache or SAN/LAN-attached storage) when binary logs, log-slave-updates and high durability settings are needed.

You might have noticed that the B intermediate master does not play a very important role in these tests. It is still needed as the parallel replication code path in MariaDB 10.0.16 is only used when the master is also running MariaDB 10.0. This restriction could be lifted in one of the next versions of MariaDB and then B could be removed.

This concludes this post on slave group commit. The next post will present speedup results of MariaDB 10.0 slaves running parallel replication on true production workload (with parallelism identified by a group committing intermediate master).

One last thing, if you want to know more about the cool things we do with MySQL at Booking.com and if you are at Percona Live Santa Clara in April, we are giving three talks:

• Booking.com: Evolution of MySQL system design
• Binlog Servers at Booking.com
• Pseudo GTID and easy replication management

You can also come to meet us at booth 315. We are looking for smart people to work with us on MySQL and help us to solve hard problems. See our careers page for more details.

[1] After completing this post, it was brought to our attention that the not yet released MariaDB versions 10.0.18 and 10.1.4 include MDEV-7249 that implement skipping the delay when a transaction dependency is found.

[2] 500 transactions if two syncs are needed per transaction, 1000 transactions if only one is needed: the exact details are not obvious, if you knows more about that, please post a comment below.

With more than 80 designer brains at Booking.com we “think” we know quite a lot. However, being a data-loving company, we have figured out that 986 UXers know even more – 986 is the number of members in the Amsterdam UX Meet-up Group!

At Booking.com, professional growth and development take place outside as well as within the office walls. Of course we have training courses and workshops, but in this constantly changing world of the web – training is often out of date once almost as soon as it’s set up and ready to be conducted. At Booking.com, learning also takes place when we get out and talk to people, attend talks and presentations given by others in the industry and open our minds to new ideas.

And opening our minds and our doors is what we did on 7 January when we hosted the Amsterdam UX Meet-up Group. We have worked a lot with this group in the past and are proud to be a well-integrated part of the Netherlands’ Design and UX community.

After a successful 2014 spent networking, sponsoring, speaking and attending meet-ups and conferences, the Booking.com design team wanted to assure a great start to 2015. So we invited more than 120 Amsterdam UXers to head office...along with Janne Jul Jensen–who came all the way from Denmark to speak about “UI in an Agile Process”.

Janne Jul Jensen specializes in usability and interaction design, and applies her expertise primarily to the mobile app projects within Trifork. These include apps for Danske Bank, Bilka, the Roskilde Festival, KMD and DSB. She is a much sought-after speaker by educational institutions, conferences, the public sector and private companies. She also gives courses on her areas of expertise, and is the founder of the Design & Usability user group (in Danish), where peers can meet and learn from each other.

After Janne shared her knowledge with the UX community our designers still had some questions for her. Here is our interview with Janne.

How did you get into design?

I was originally being educated to be a Software Engineer at university, but was beginning to question whether that was the right choice as I thought it was lacking purpose for me. I asked some of my fellow male students why they liked the education, and often the answer was “I think it is fun to tinker with software”. But to me it wasn’t enough of a reason that I myself was having fun. I wanted a bigger purpose with my education than it just being for my own sake. I wanted an education that would have an impact on peoples' lives. So I started looking into how I could combine the two, and discovered that the software engineering degree had a specialization in Human Computer Interaction (HCI), and the rest, as they say, is history. I chose that specialization and have never looked back since.

Compared to a lot of other designers, you have quite an academic background, what was the most important thing that you learned for the practice of design?

Through my academic background I got to equip my UX toolbox with a lot more tools than most other UX’ers, as I also got to try out some of the more specialized and perhaps less applicable tools within UX. This has its advantages when I’m faced with a project that has an unusual problem or a different profile. I also find that because I have done UX research, which has had to be very rigid to achieve the academic validity required, I have a good understanding of where I can scale the various methods back without the results being affected – and if I scale back in a way that affects the results, I know HOW it affects the results.

What is the most stimulating and challenging project you have accomplished?

I have worked on many different projects, all which have had their own challenges. But one of the most stimulating projects I have worked on, was the Danske Bank Mobile Bank project. It was very rewarding to work with a client who puts quality first! Most clients are very preoccupied with cost, and usually cuts and compromises are made within UX, when the budget is under pressure. This has never been the case with Danske Bank, as they value the quality of the app as the primary objective. This allows designers to dream big and to do things the way they should be done, rather than the way they can be done within a tight budget.

Since this was your first experience creating a mobile app, how did you evaluate your agile process to ensure it was effective in this project context, and is there anything you would change?

We were using SCRUM in the project and part of the SCRUM process is the retrospective. We used this part very actively in that we ended each sprint with a retrospective, during which we looked at what had worked well in the process. Based on this, we decided which parts of the process we should keep and which we should change in the next sprint. For those things that didn’t work well, we discussed why, and came up with a new way of dealing with them that we would then try out in the next sprint.

In addition to starting the UX process a sprint ahead of the rest of the development, are there any other tips for embedding UX into the SCRUM more easily?

Another important aspect of embedding UX into SCRUM is to make it explicitly a part of the backlog. You do this by making UX points on the list for each feature (and prioritizing the feature and the UX together)! It is also important to have a product owner who explicitly values and prioritizes UX. Finally, independent of the development process, it is important to have UX as an integral part of your company culture; to have developers who value and respect UX and who will actively seek UX advice; and to have leadership who prioritize UX and find it important.

Do you think that the Agile methodology is better suited to the development of web-based or native apps?

I think it fits both! The world in general is characterized by never being static and being in a state of constant change. Humans learn constantly and are always accumulating experience. Neither of these givens is taken into account by, for example, the waterfall model, as that is based on an assumption that once a requirement specification is written, the scope of the project will never change (and no one will want it to), and that we will not learn anything new in the development process that will change our idea of what should be developed and how. Rarely in a development project is either of these assumptions true.

At Booking.com we love A/B testing, did you do A/B testing for the Danske Bank app?

I am very much in favor of A/B testing as a way to decide between two possible solutions. However, because we don’t do releases very often in mobile banking, we could potentially have two different versions of the app out for quite a long time. We feel that this would create other challenges, so we opted out of A/B tests in this project. Instead we carry out other activities to address any doubts between potential solutions.

How much do you trust user surveys?

User surveys can never tell you the full picture, and it is always important to be aware of the limitations of a survey. That said, if a survey has been well produced and correctly distributed, it can give you very valuable insight, especially as long as you keep in mind what it cannot tell you.

For the Danske Bank app, you used Facebook and other social media to get feedback and ideas from the users. What other ways do you use to find out which new features users want in their apps?

Facebook is not the only way we listen to users. We use the reviews in App Store and Google Play (yes, someone actually reads those), input from the Danske Bank support, all the customer feedback the bank gets regarding the app, surveys, formulate questionnaires, keep an eye on competitors, and follow general trends in the field. In general, we keep our eyes and ears open in any relevant context that will inform our knowledge of our users’ needs and wants.

Because the Booking.com app is translated into 40 languages, we deal with big translation challenges at Booking.com. Your initial app was developed for Deutsche Bank in Denmark, and in the meantime you were building apps for other countries. Do they all share the same code base or are they completely different apps? How did you manage translations?

The Danske Bank Mobile Bank app exists in six versions (Danish, Swedish, Norwegian, Finnish, Northern Irish, and English) and the code base is largely the same for all six apps, with a few exceptions because of a variation in regulations from country to country. Translation is handled using a large spreadsheet that contains translations for every piece of text in the app. Because of this, it is very important that absolutely none of the graphics contains any text whatsoever. One of the challenges that still remain is if, for example, a label on a button will fit that button in any language. The key here is to find the language that tends to have the longest words. In our case, if it will fit in Finnish, the other languages will almost always fit too. That said, there is no shortcut for checking this, you have to go over every screen to be sure.

And as a last question, you are teaching UX – what book would you recommend as a “Must read” to a junior designer?

There are general books and specialized books, and when it comes to someone fairly new in the trade, I would recommend the more general books. These could include:

• Interaction Design: Beyond Human-Computer Interaction by Jenny Preece and Helen Sharp (Coming out February 2015)
• Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests by Jeffrey Rubin
• The inmates are running the asylum by Alan Cooper
• Don’t make me think by Steve Krug
• Surgery made easy by Steve Krug

Booking.com is getting ready for a UX conference in April, a front-end hackathon in May and another design meet-up in June. These will happen both in and outside our office walls, but everyone’s always welcome. We can only truly learn if we open our doors for more knowledge and experience.

This is a story about a significant new optimization to the Perl interpreter. It is a story about battling code complexity. And it is a story about wanting to have our cake and eat it too.

A recent Booking.com hackathon provided us the opportunity to investigate speeding up integer allocation in the Perl interpreter. If successful, this could optimize nearly every program we run. We discovered that a naive implementation could work, but would make the code a lot more difficult to maintain. Our path lead us to attempt to leverage the C preprocessor to improve code clarity while opening doors to real gains in program execution speed.

First, some background

As described in perlguts and PerlGuts Illustrated, representations of variables in Perl are usually composed of two parts: a head struct and (optionally) a body struct. The head contains the essential “internal book-keeping” common to all variables (regardless of type) including a pointer to the (optional) body struct. Below is an image of the layout of fields in a head struct as depicted in PerlGuts Illustrated:

The body struct can be quite different, depending upon the type of the variable. The most simple type of variable is the SvNULL, which represents undef and does not need a body struct at all.

A string (called PV for “pointer value”) has a body type of XPV:

But the body struct of a PV is different to a body struct of a PVNV. A PVNV can hold a floating point number and a string representation of that same value:

One benefit of this type of design is that all references to the value point to the memory location of the head. As such, Perl is free to change which memory is being used to represent a value by changing the body struct, without having to update any pointer except the pointer contained within the head.

Changing types

Naturally, Perl has an internal function to convert between types of variable. This function is sv_upgrade (“scalar value upgrade”). In essence, whenever we have a variable of some type in Perl (for example, a simple integer) and we need to access it as a different type (for example, a string), the sv_upgrade will convert the type of the variable (for example, into a type that contains both integer and string representations of a value). This change may involve replacing the body struct with a larger struct.

To see how sv_upgrade is implemented, we can look at the Perl_sv_upgrade function in sv.c. We see that this function encapsulates a lot of complexity. There are many comments, which describe subtle corner cases. Since it can take a scalar value of essentially any type and convert it to something capable of representing any other type, perhaps it’s unsurprising that there’s a lot of code here.

Without going line-by-line, an overview of how this function is implemented may be useful. First, there is a switch based on the current type of variable, to determine what it needs for the new type. Shortly thereafter, there is a second switch based on the new type. Inside of the second switch block, there are numerous if blocks for doing different things depending on the old type. Finally, after the new body struct has been set up, and the head struct contains all the correct flags, the memory used by the old body is freed.

Still with me? Good.

A naive optimization

The sv_upgrade function is called from a number of places. Not only is it called in places such as printing integers as strings, but it’s also called when assigning an integer value to a previously cleared variable.

A previously cleared variable is always an undef with no body struct. The reason sv_upgrade is called is to allow the correct setup for the body to occur. This was a very sensible design decision, as it centralizes a lot of “internal book-keeping” behavior, rather than duplicating it. The cost of this centralization is performance: some generic (and in this case superfluous) code is executed. For example, when creating a new integer, sv_upgrade will run an unnecessary check for a conversion of a larger type to a smaller type.

Integer assignment to a cleared variable occurs so frequently that one would imagine it might be worth duplicating the required complexity to achieve the performance improvement. Thus we decided to evaluate that trade-off for integer allocation. After a review of the nearly 300 lines of sv_upgrade internals, we saw that we could remove the call entirely if we could “hoist out” just the essential two lines of code! However, there is a very good reason this had not been done before. Let's look at the two lines.

The first line (since we know the new type) is easy:

SvFLAGS(sv) |= new_type;

But the other line is quite complex:

SvANY(sv) = (XPVIV*)((char*)&(sv->sv_u.svu_iv) - STRUCT_OFFSET(XPVIV, xiv_iv));

If your head is swimming after looking at that last line of code, don’t worry: you’re in good company. Granted, this is described in Illustrated perlguts:

Since 5.10 for a raw IV (without PV) the IVX slot is in the HEAD, there is no xpviv struct ("body") allocated. The SvIVX macro abuses SvANY pointer arithmethic to point to a compile-time calculated negative offset from HEAD-1 to sv_u.svu_iv, so that PVIV and IV can use the same SvIVX macro.

But, even once I thought I maybe understood what that complex line of code supposedly did, I made Steffen patiently sit for more than 15 minutes as I convinced myself with paper and pencil that it actually did what he had described. After that, the drawing from Illustrated Perl Guts made more sense to me:

What’s more, I finally understood that this complexity exists in order to avoid executing an if statement that would otherwise be called every time the value is retrieved!

Thus, the previous design trade-offs seemed like the right ones. Yes, we could make Perl faster, but at the cost of leaking a lot of complexity into another part of the code. That complexity would make any additional development much more difficult in any future work.

Having our cake and eating it too

We wanted to encapsulate that very complex bit of code, but without any added runtime performance cost. Since this is C, we looked at using the preprocessor to push the complexity behind a macro, in much the same way that with other languages we might move some hairy lines of code behind a well named function or method:

#define SET_SVANY_FOR_BODYLESS_IV(sv) \
    SvANY(sv) = (XPVIV*)((char*)&(sv->sv_u.svu_iv) - STRUCT_OFFSET(XPVIV, xiv_iv))

And, of course, the advantage of using a macro (rather than a function) is that the cost is paid entirely at compile time, and thus there is zero additional runtime performance lost.

By introducing the macro, the readability of the code would be improved in several places.

So, how did this change our situation? With the macro, the two lines we wanted to “hoist out” looked a lot less complex. All we would need would be a patch which replaces this one computationally heavy function call:

sv_upgrade(dstr, SVt_IV);

with just these two lines:

SET_SVANY_FOR_BODYLESS_IV(dstr);
SvFLAGS(dstr) |= SVt_IV;

By factoring the complexity differently, we opened the doors to a different set of performance/complexity cost trade-offs. We would leak only a small amount of complexity relative to the gain in performance. But is it worth it? With these changes, the cost would be low, but what would the actual benefit be?

Measuring the gain

The micro-benchmark we used is very heavy on exercising one particular code path, however it is a common code path.

$dumbbench-i50--pin-frequency--\./perl-Ilib-e\'for my $x (1..1000){my @a = (1..2000);}'

Here are the results of the micro-benchmark before the optimization:

  Rounded run time per iteration: 2.4311e-01 +/- 1.4e-04

And here are the results after the optimization:

  Rounded run time per iteration: 1.99354e-01 +/- 5.5e-05

That’s 18% faster.

Results

Overall, this hackathon project was a success.

With these measurements, we demonstrated that the optimization was worthwhile. The complexity cost is pretty close to zero: in some places the Perl core is slightly more complex, but the internals of sv_upgrade are certainly more clear. Additionally, we found there were other similar potential optimizations that could leverage the same technique. In the end, five related patches were pushed to Perl:

• Refactor bodyless-IV/NV hacks into define
• Speed up assigning an IV to a previously cleared SV
• Speed up newSViv()
• Repeat newSViv optimization for newSVuv
• Attempt to bring newSViv/uv optimization to newRV

When Perl 5.22 ships, in part because of this work, many real-world programs will be faster.

It’s no secret that Booking.com has a strong data-driven culture. We validate our work through A/B experimentation, allowing millions of customers to have their say in what works best. But quantitative research is not our answer to everything; we adjust our toolset to the problem at hand.

When we set out to build Booking.com for Business, it immediately felt like we were in a startup. Suddenly, the wealth of experimentation data at our disposal wasn’t enough to start building the application for businesses. To kick-off the design work, we needed to know more about our business users’ needs, motivations, and current frustrations. We needed to get out of the building and talk to them.

Initial research

We performed a series of user interviews in several countries with a significant share of business travel. We met with business travellers of all types (from interns to CEOs), along with the people who organise the business trips for them.

We learned that business users have unique set of needs. Booking a holiday can be a fun pastime on its own, but booking a business trip is part of a job—it needs to be as efficient as possible. There’s also more to business travel than making the booking itself. Companies need an overview of who is going where and how budgets are spent. Existing business travel solutions either don’t satisfy such needs, or they’re expensive and complex.

These and many other insights became the foundation for our work. They were synthesized into a set of user personas that shaped the design of the product as we went on.

Product vision

Armed with knowledge about potential users, we started brainstorming. Our aim was to create a vision of the product that would help our personas accomplish their goals. The outcome of these brainstorms was a list of high-level requirements that were later visualised as a set of wireframes. This tangible representation of our ideas enabled us to have fruitful conversations with stakeholders. The wireframes were high-level enough that we could temporarily set aside details of technical implementation and visual design, but also detailed enough to convey the purpose of each screen.

Minimal viable product (MVP)

The product vision was exciting, but it was just a hypothesis. We didn’t want to spend months implementing something ultimately not useful for customers, and not beneficial to our business. It was important to get the product out there as soon as possible, and to start learning from real world usage. We knew that we already had a great product—Booking.com itself—and we could build on its strengths. With this in mind, we defined the minimal scope that would be sufficient to validate our ideas, and started mapping out the user journey.

Filling the user journey with designs felt like finishing a puzzle. As we progressed, we could see how complete the whole picture was from the design perspective.

However, soon after we started implementation, we noticed a problem. Separate design mockups didn’t provide a true feeling of the user experience. They were static. It wasn’t always clear how the application would respond to user actions, and how one page would transition into another. We found ourselves figuring out these details along the way.

It was also important to get user feedback on design decisions we had made so far. Unfortunately, the actual product was still at the early stage of development, and putting mockups in front of users gave us limited feedback.

Prototype

As a response to these issues, we created an interactive prototype that simulated the end-to-end user flow. For example, it was possible to land on the product page, go through the sign-up process, view transactional emails, experience key application features, sign-out, and sign back in again. In this way, we solved two problems at once: we had created a tool that would better guide product development and procure high-quality user feedback.

We kept it lean and didn’t spend much time creating the prototype. We simply placed mockups in HTML files and connected them with hyperlinks. In-page interactions were triggered by bits of basic Javascript code that showed images of various UI states on click. For example, when the user clicked on an area of the mockup that had a button, the image changed simulating interface response.

Some design solutions that previously looked good as static mockups didn’t work so well when presented in the dynamic prototype. Acknowledging this helped us to fix design issues before they reached the product. Participants in user testing sessions were also more engaged with the prototype because it felt like a real product.

But prototypes are not without limitations. It’s hard to do full-blown usability tests with them. They may look real, but not every possible scenario is supported, so facilitators need to carefully steer participants. Maintaining the prototype also becomes tedious over time. We tried to make it easier by separating reusable parts like header, footer, navigation, etc. into include files. We accomplished this by using Jekyll, a static website generator.

The good news is that we didn’t have to rely solely on the prototype for long. The product quickly took shape and it soon became possible to put the real thing in front of users.

User feedback

After the product reached the MVP state, it became easier to get user feedback. Although the product wasn’t yet ready to be publicly announced, we were able to start gathering usage data and feedback from early adopters. We also continued usability testing, because the usage data told us what was happening, but often left us wondering why.

Even with the working product at our disposal, we continued using prototypes to fill the gaps during usability tests. We seamlessly integrated feature prototypes into the live product and switched them on specifically for usability session participants. This helped us to establish whether mocked-up ideas were worth implementing, and also to test parts of the application that were still in development.

Usability labs were not our only test environment. We visited company offices and observed how the product was performing in the real world. These office visits were highly valuable, providing us with insight from observation of users in their natural environment. We had a chance to see what tools they used, what workarounds they developed, and how our product would fit their work process. This was absolute gold. Some things that performed well in the lab set-up, failed during office visits.

We saw that our users were working in a very busy environment and were constantly distracted. The time they spent making a decision was very short. Plus, they were sceptical about introducing new tools into their work. All this posed a particular challenge for a crucial step in the user journey: the product page. Our users needed tangible proof that the product would do as it promised, and that it was reliable. They wanted to explore the product before making any commitments.

Product page

The research findings informed new product page designs. But we needed confirmation that they would actually solve the problems we had observed. We opted for remote surveys as a method to gather feedback quickly and on a large scale. This enabled us to cover several markets and bring a quantitative component into the research.

Survey participants were presented with various versions of the page and were asked to click and comment on page elements that stood out to them. Afterwards they were asked a series of questions that helped us gauge how well they understood our offering, and how likely they were to sign-up. 

It took us a few survey iterations to arrive at a version that proved to work well for our users. Had we gone with one of the new designs without testing, we would have ended-up with a sub-optimal page upon product launch. 

Now that we had the fully-tested user journey in place, we could reveal the product to the world.

Final thoughts

Fast forward to today. The product is up and running. We can now make decisions through A/B experimentation as there is an established base and a sufficient number of users that continues to grow. If we look back on the process that brought us here, this is what comes to mind:

• In an environment of uncertainty, it was important to remain open to change. We had to think creatively not just about the product itself, but also about how to get there.
• We were focused on the user from day one and all the way through. Even when we didn’t yet have the complete product, we used the prototype to get user feedback.
• By combining quantitative and qualitative research methods, we got the best of both worlds. This was and will remain our recipe to continuously improve the user experience.

This is the annex to Evaluating MySQL Parallel Replication Part 3: Benchmarks in Production.

There is no introduction or conclusion to this post, only landing sections: reading this post without its context will probably be very hard. You should start with the main post and come back here for more details.

Environments

As in the two previous posts (Part 1 and Part 2), we are using the same four environments. Each environment is composed of four servers:

-----     -----     -----     -----
| A | --> | B | --> | C | --> | D |
-----     -----     -----     -----

• A is a true production master running MySQL 5.6,
• B is an intermediate master running MariaDB 10.0.16 without parallel replication enabled (slave_parallel_threads = 0),
• C is an intermediate master running MariaDB 10.0.16 with slave group commit enabled (slave_parallel_threads> 1, binlog_commit_wait_count> 1 and binlog_commit_wait_usec> 1),
• D is the slave where parallel replication tests are run with different parameters.

The slave group commit parameters on C are the following:

To monitor parallelism identification (group committing) on C, the binlog_commits and binlog_group_commits global statuses are gathered regularly.

The D slave was stopped on a Thursday between 17:20 and 17:30 and a backup of the database was taken. Every test starts by recreating the filesystem that hosts the database and restoring the backup on this empty filesystem.

When stopping the database (for backup), the buffer pool was dumped using innodb_buffer_pool_dump_at_shutdown. When starting MariaDB (after backup restoration), innodb_buffer_pool_load_at_startup was used to load the buffer pool. No test was started before the buffer pool was fully loaded.

Every test then started by running the D slave until Friday at 06:00 (a little more than 12 hours). This allowed further warming up of InnoDB, including giving work to purge and write threads.

Then, the D slave ran 24 hours of transactions where parallelism was identified using slave group commit on the C intermediate master. The number of commits and the group commit sizes on C for that period are shown in the graphs below for the four test environments.

Graphs # 0: Commits and Group Commit Sizes in the four Environments

The D servers have the following properties:

To monitor the commit rate on D, the COM_COMMIT global status is gathered regularly.

To prevent D from filling up its disk, relay_log_space_limit is set to 4 GB (some D do not have enough disk space to store all the binary logs of C).

Group Commit: Slave vs. Master

It is briefly mentioned in the main post that slave group commit identifies less parallelism than a true master would. More details are given here.

Let's consider this execution timeline which shows a master where T5 is conflicting with one of the four previous transactions:

   ------Time----->
T1:      B--C
T2:   B-----C
T3:   B-----C
T4:    B----C
T5:  B-- . . --C
T6:       B----C

Using slave group commit and from the binary logs of the timeline above, parallelism can be identified in two groups as shown below.

   ---------------Time---------------->
T1:  B-- . . . . . . . . C
T2:     B----- . . . . . C
T3:           B----- . . C
T4:                 B----C
T5:                       B---- . . C
T6:                            B----C

But if we run the master with delayed commit, we could get the execution timeline below, in which T6 passed T5 to join the first commit group.

   ------Time----->
T1:      B-- . C
T2:   B----- . C
T3:   B----- . C
T4:    B---- . C
T5:  B-- . . . .--C
T6:       B----C

So on a master, transactions blocked in their execution do not block other non-conflicting transactions from joining a commit group, but those block parallelism identification in slave group commit. This is why slave group commit is not as efficient at maximizing group size as a parallel execution on the master would be.

Results

As outlined in the main post, our tests are run in the following binary log configurations:

• Intermediary Master (IM): binary logs and log-slave-updates enabled.
• Slave with Binary Logs (SB): binary logs enabled and log-slave-updates disabled.
• Standard Slave (SS): binary logs and log-slave-updates disabled.

And in the following durability configurations:

• High Durability (HD): sync_binlog = 1 and innodb_flush_log_at_trx_commit = 1.
• No Durability (ND): sync_binlog = 0 and innodb_flush_log_at_trx_commit = 2 (also described/known as relaxed durability).

For each of those configurations (6 in total: IM-HD, IM-ND, SB-HD, SB-ND, SS-HD, and SS-ND), the tests are run with different values of slave_parallel_threads (SPT). The full results are presented in the following four tables. The times presented are in the format hours:minutes.seconds. Below the time taken to process 24 hours of transactions, the speedup achieved from the single-threaded run is presented (in bold) and when applicable, the speedup from the previous run is also presented (in normal font).

Normally, the tests are run at least twice and the shorter time is kept. This is relaxed for E1, E3, and E4 in the SS-HD and SS-ND configurations where only one run is performed: the results from E2 show that SS-HD and SS-ND are very similar to SB-HD and SB-ND respectively, and the limited results with E1, E3, and E4 are consistent with this observation. Moreover, again for E1, E3, and E4 in SS-HD and SS-ND and for the same reasons, some SPT values are skipped (the skipped values are shown as N/A in the tables below).

Graphs during Tests

If you spot something we might have missed in the graphs below, please post a comment. Those graphs include the number of commits per second, CPU stats and Read IOPS for all environments, for the Slave with Binary Logs configuration (log-slave-updates disabled), in both durability settings (high and no/relaxed).

Graphs # 1a: E1 Stats - Slave with Binary Logs - High Durability

Graphs # 1b: E1 Stats - Slave with Binary Logs - Relaxed Durability

Graphs # 2a: E2 Stats - Slave with Binary Logs - High Durability

Graphs # 2b: E2 Stats - Slave with Binary Logs - Relaxed Durability

Graphs # 3a: E3 Stats - Slave with Binary Logs - High Durability

Graphs # 3b: E3 Stats - Slave with Binary Logs - Relaxed Durability

Graphs # 4a: E4 Stats - Slave with Binary Logs - High Durability

Graphs # 4b: E4 Stats - Slave with Binary Logs - Relaxed Durability

Workloads

It is mentioned briefly in the main post that the four test environments have different workloads:

• E2 is a CPU-bound workload (the dataset fits in RAM).
• E1 is also mostly CPU-bound but with some cache misses in the InnoDB buffer pool, needing a page fetch from disk before doing a write.
• E3 is a mixed CPU and IO workload (more cache misses in the InnoDB buffer pool but still with enough cache hit to get a good commit throughput).
• E4 is an IO-bound workload (mostly cache misses).

We can confirm that E2 does not do much IO looking at IOWait and Read IOPS in the graphs below (commit throughput is also good).

Graphs # 5: E2 is CPU-bound
(good commit throughput, low IOWait and few Read IOPS).

E1 has a little more IOWait and Read IOPS but still not that much as you can see in the graphs below. However, commit throughput is smaller than E2, but still good (the transaction sizes are bigger).

Graphs # 6: E1 is also mostly CPU-bound
(good commit throughput, low IOWait and few Read IOPS).

E4 is clearly an IO-bound workload: commit throughput is low, IOWait is high, and Read IOPS are high as shown in the graphs below.

Graphs # 7: E4 is IO-bound
(low commit throughput, high IOWait and high Read IOPS)

E3 has a much higher commit throughput than E4 but with lots of Read IOPS as shown in the graphs below. This makes us put this workload in a mixed CPU and IO-bound category (confirmed by a high IOWait but still lower than E4).

Graphs # 8: E3 is mixed CPU and IO-bound
(respectable commit throughput, high IOWait and high Read IOPS)

Note: the Write IOPS are not as important because they are buffered in the RAID controller write cache. The Read IOPS are important because a write operation to the database that needs an InnoDB page fetch from disk is bounded by the disk seek time.

Additional Discussions

Another Problem with Long-Running Transactions

In the main post, we discussed the impact of long-running transactions on the parallel replication pipeline (idle threads while the long transaction is running). When the number of slave threads is not smaller than the group commit size, the time taken to execute the group is bound by the time taken to run the longest transaction, as shown below.

   --------Time-------->
T1:  B-----------C
T2:           B--C
T3:           B--C
T4:           B--C
T5:              B--C
T6:              B--C

The transactions above on the master are executed as below on the slave with SPT=4.

   --------Time-------->
T1:  B-----------C
T2:  B-- . . . . C
T3:  B-- . . . . C
T4:  B-- . . . . C
T5:               B--C
T6:               B--C
              1
     12345678901234567

But if the number of slave threads is smaller than the group commit size, we get a longer run, as shown below (SPT=2):

   ---------Time-------->
T1:  B-----------C
T2:  B-- . . . . C
T3:               B--C
T4:               B--C
T5:                   B--C
T6:                   B--C
              1         2
     123456789012345678901

So we lost 4 units of time in the global run by decreasing SPT. What is interesting is that there is enough time for executing both T3 and T4 while T1 executes (and while the thread running T2 is idle), so maybe it is possible to avoid this situation.

Note that MySQL 5.7 solves this problem by allowing out-of-order commit. With slave-preserve-commit-order set to 0 (the default), MySQL will commit T2 without waiting for T1 to commit. This will allow T3 to start executing while T1 is still running. However, this changes the behavior of the slave: T2 (and eventually T3 and T4) becomes visible before T1, which is not possible in single-threaded replication. Some people (including the author of this post) might be nervous about this and would prefer to set slave-preserve-commit-order to 1.

A future solution to manage long-running transactions and group size larger than SPT could be to delegate committing transactions. If the thread running T2 delegates its commit to another thread, it can start running T3 event if T2 is not committed. This could be an interesting optimization to implement in MariaDB or in MySQL.

(Note that the problem described above - group size larger than SPT - cannot explain the modest speedup observed in our tests as we identify parallelism with binlog_commit_wait_count = 35 and we run tests with SPT up to 40.)

Relaxed Durability Burns CPU for E1 and E2

As presented in the main post, E1 and E2 have respective speedups of ~1.06 and ~1.04 for SB-ND. However, as shown in the graphs below, while increasing SPT, CPU consumption also increases noticeably without significantly improving the commit throughput.

Graphs # 9: E1 Stats - Slave with Binary Logs - Relaxed Durability

Graphs # 10: E2 Stats - Slave with Binary Logs - Relaxed Durability

Either we hit a synchronization bottleneck in the parallel applier or there is room for optimization: there is probably interesting work to be done here.

Parallel replication is a much-expected feature of MySQL. It is available in MariaDB 10.0 and in MySQL 5.7. In this 3rd post of the series, we present benchmark results from Booking.com production environments.

Note: this post has an annex: Under the Hood. Benchmarking is a complex art and reporting results accurately is even harder. If all the details were put in a single article, it would make a very long post. The links to the annex should satisfy readers eager for more details.

Parallel replication is on its way and with it comes the hope that more transactions can be run on a master without introducing slave lag. But it remains to be seen whether this dream will come true - will parallel replication hold its promise or will there be surprises after its deployment? We would like to know whether or not we can count on that feature in the future.

To get answers, nothing is better than experimenting with the technology. Benchmark results have already been published (MariaDB 10.0: 10 times improvement; and MySQL 5.7: 3 to 6 times improvement) but results might be different in our production environments. The best test would be to run parallel replication on our real workloads but this is not trivial. To be able to run transactions in parallel, a slave needs parallelism information from the master (for more details, see Part 1). With a master in an older version (MySQL 5.6 in our case), slaves do not have this information.

Luckily, we can use slave group commit on an intermediate master to identify transactions that can be run in parallel. The trick is to execute transactions sequentially on a slave, but to delay their commit. While the commit is delayed, the next transaction is started. If the two transactions are non-conflicting, the second could complete and the two transactions would commit together. In this scenario, grouping has succeeded (the two transactions committed in a single group) and parallelism is identified (as they commit together, the transactions are non-conflicting, thus can be run in parallel on slaves). For more details on slave group commit, see Part 2.

Our benchmarks are established within four production environments: E1, E2, E3, and E4. Each of these environments is composed of one production master, with some intermediate masters, and a leaf slave. A complete description of those environments can be found in the annex.

Slave group commit identifies less parallelism than a parallel execution on a master would identify (details in the Group Commit: Slave vs. Master section of the annex). Even so, decent group sizes are obtained, as shown in the group commit size graphs in the annex. Usually we have group sizes of at least 5, most of the time they are larger than 10, and sometimes they are as big as 15 or even 20. These will allow us to test parallel replication with real production workload.

Before closing this long introduction, let's talk a little about our expectations. InnoDB is getting better at using many cores (RO and RW benchmark results) but single-threaded replication gets in the way of pushing more writes in a replicating environment. Without parallel replication, a single core can be used on a master to perform writes without incurring slave lag. This is disappointing as servers come with 12 and more cores (only one can be used for writes). Ideally, we would like to use a significant percentage of the cores of a server for writes (25% could be a good start). So speedups of 3 would be good results at this point (12 cores), and speedups of 6 and 10 would be needed in the near future (24 and 40 cores).

The Test: Catching Up with 24 Hours of Transactions

Our benchmark scenario is as follows: after restoring a backup of the database, starting MariaDB, waiting for the buffer pool to be loaded, and running the slave for at least 12 hours, we measure the length of time it has taken to process 24 hours of production transactions.

The tests are run in the following binary log configurations:

• Intermediary Master (IM): both binary logs and log-slave-updates are enabled.
• Slave with Binary Logs (SB): binary logs are enabled but log-slave-updates is disabled.
• Standard Slave (SS): both binary logs and log-slave-updates are disabled.

And in the following durability configurations:

• High Durability (HD): sync_binlog = 1 and innodb_flush_log_at_trx_commit = 1.
• No Durability (ND): sync_binlog = 0 and innodb_flush_log_at_trx_commit = 2 (also described/known as relaxed durability).

For each of those configurations (6 in total: IM-HD, IM-ND, SB-HD, SB-ND, SS-HD, and SS-ND), the tests are run with different values of slave_parallel_threads (SPT). The full results are presented in the annex and the most interesting results are presented below (SB-HD and SB-ND). The times presented are in the format hours:minutes.seconds. Below the time taken to process 24-hours of transactions, the speedup achieved from the single-threaded run is presented in bold.

In the Graph during Tests section of the annex, you can find many details about the different test runs.

Discussion

There are lots of things to say about those results. Let's start with observations that are not related to parallel replication (SPT=0):

• Obs1: Standard Slave results (without binary logs) are very close to the results of Slave with Binary Logs (without log-slave-updates) (details in the annex).
• Obs2: log-slave-updates has visible cost for E1 (time difference between IM-HD and SB-HD), a less obvious but still noticeable cost for E2 and E3, and it is a win for E4 (that last one is disturbing, the numbers are in the annex).
• Obs3: relaxing durability is a huge win for E1 and E2, a more limited win for E3, and a much smaller one for E4.

With reference to Obs1 above, this shows that binary logs should probably not be disabled on slave: the cost is almost inexistent and the wins are big (tracing errant transactions and being a candidate for master promotion). However, slaves with log-slave-updates are slower than slaves with only binary logs enabled (Obs2 above), so log-slave-updates should be avoided when possible. Binlog Servers can be used to replace log-slave-updates for intermediate masters, see MySQL Slave Scaling for more details (see also Better Parallel Replication for MySQL for an explanation why log-slave-updates is bad on intermediate masters for parallel replication).

With reference to Obs3 above, this can be explained by the different workload of the four environments (more details about the workloads can be found in the annex):

• E2 is a CPU-bound workload (the dataset fits in RAM).
• E1 is also mostly CPU-bound but with some cache misses in the InnoDB buffer pool, so it needs a page fetch from disk before doing a write.
• E3 is a mixed CPU and IO workload (more cache misses in the InnoDB buffer pool but still with enough cache hit to get a good commit throughput).
• E4 is an IO-bound workload (mostly cache misses).

Relaxing durability on CPU-bound workloads achieves good throughput improvements, but this does not happen on IO-bound workloads.

Now, let's focus on parallel replication. The Standard Slave results (SS-HD and SS-ND) are not worth discussing as they are very close to the Slave with Binary Logs results (Obs1 above). We will also not discuss Intermediate Master results (IM-HD and IM-ND) as they should be replaced by Binlog Servers. So all observations below are made on the results of Slave with Binary Logs (SB-HD and SB-ND):

• Obs4: the best speedup (~2.10) is in E2 with high durability. E1 follows with a speedup of ~1.56 (always with high durability).
• Obs5: the speedups for E4 are modest (~1.29) and the results are almost identical for both durability settings.
• Obs6: for E1 and E2, the speedups with no durability are almost non-existent (less than 1.10).
• Obs7: for both E1 and E2, relaxing durability with single-threaded replication leads to faster execution than enabling parallel replication.
• Obs8: the results for E3 are halfway between E1/E2 and E4: both SB-HD and SB-ND get some modest speedups from parallel replication (like E4 and opposite to E1/E2) and relaxing durability makes things run a little faster (like E1/E2 and opposite to E4), but not to the point where single-threaded low durability is faster than multi-threaded high durability.

All those observations point to the importance of the workload in parallel replication speedups:

• CPU-bound workloads seem to get modest speedups in high-durability configurations.
• Relaxing durability for CPU-bound workloads looks like a better option than enabling parallel replication on a high-durability configuration.
• IO-bound workloads get more limited speedups.

Our first reaction is disappointment: the speedups are not as high as expected. Don't get us wrong: faster is always better, especially when the only thing to do is to upgrade the software, which we will do anyway eventually. However, having only 25% more writes on a master (or 110% depending on which environment we look at) will not help us in the long term. Parallel replication is not the solution (at least not the only solution) that will allow us to stop/avoid sharding.

Ideas and Future Work

We have a hypothesis explaining the modest speedups: long-running transactions. In the presence of long-running transactions, the parallel replication pipeline on the slave stalls. Let's take the following six transactions committing on the master in two commit groups (B for begin and C for commit):

   --------Time-------->
T1:  B-----------C
T2:           B--C
T3:           B--C
T4:           B--C
T5:              B--C
T6:              B--C

Running those transactions on a single-threaded slave takes 33 units of time (time scale is at the bottom):

   ----------------Time---------------->
T1:  B-----------C
T2:               B--C
T3:                   B--C
T4:                       B--C
T5:                           B--C
T6:                               B--C
              1         2         3
     123456789012345678901234567890123

Running those transactions on a multi-threaded slave with SPT=4 takes 17 units of time:

   ---------Time-------->
T1:  B-----------C
T2:  B-- . . . . C
T3:  B-- . . . . C
T4:  B-- . . . . C
T5:               B--C
T6:               B--C
              1
     12345678901234567

So we barely achieve a speedup of 2 (and the second commit group does not even contain a large transaction). The low speedup is explained by T1 being much bigger than the other transactions in the group. So our intuition is that to get better speedup with parallel replication, all transactions should be of similar size, and bigger transactions should be broken down into smaller ones (when possible).

We have many of those big transactions in our workload at Booking.com. Most of our design choices predate MySQL 5.6, where a commit was expensive. Reducing the number of commits was a good optimization at that time, so doing many changes in a single transaction was a good thing. Now, with binary log group commit, this optimization is less useful but does not harm. However, this optimization is very bad for parallel replication.

There are at least two other things to discuss from those results but this post is already too long, so you will have to go in the annex to read the Additional Discussions.

Conclusion

It is possible to test parallel replication with true production workload, even if the master is running an old version of MySQL. Thanks to slave group commit in MariaDB 10.0, we can identify parallelism on intermediate master and enable parallel replication on a slave. Even if this parallelism identification is not as good as it would be on a master, we get decent group sizes.

Our CPU-bound workloads are getting speedups of ~1.56 to ~2.10 with high-durability constraints. This is a little disappointing: we would like to have more than two cores busy applying writes on slaves. Our guess is that better speedup could be obtained by hunting down large transactions, but that still needs to be verified. At this point, and for this type of workload, our tests show that relaxing durability is a better optimization than enabling parallel replication. Finally, with relaxed durability, parallel replication shows almost no improvement (4% to 6% improvement), and it is still unknown if hunting down large transactions and splitting them would result in better speedups.

Our IO-bound workloads are getting speedups of ~1.23 to ~1.29, which is also disappointing but expected because it is hard to fight against seek time of magnetic disks. In this type of workload, relaxed durability setting benefits from parallel replication. However, at high enough parallelism on the slave and for this type of workload, relaxing durability is not very beneficial. It is hard to tell what types of improvement would come from hunting down large transactions for this type of workload.

The next step on parallel replication evaluation would be to try optimistic parallel replication. This will make a good fourth part in the series.

In a MySQL replication deployment, the master is a single point of failure. To recover after the failure of this critical component, a common solution is to promote a slave to be the new master. However, when doing so using classic methods, the slaves need to be reconfigured. This is a tedious operation in which many things can go wrong. We found a more simple way to achieve master promotion using Binlog Server. Read on for more details.

When a master fails in a MySQL replication deployment, the classic way to promote a slave to be the new master is the following:

1. Find the most up-to-date slave.
2. If the most up-to-date slave is not a good candidate master, level a suitable candidate with the most up-to-date slave [1].
3. Repoint the remaining slaves to the new master.

The procedure above needs to contact all slaves in step #1, and to reconfigure all slaves in step #3. This becomes increasingly complex in Booking.com environments where we have very wide, and still growing, replication topologies; it is not uncommon to have more than fifty (and sometimes more than a hundred) slaves replicating from the same master. Many things can go wrong when tens of slaves need to be contacted and reconfigured:

• some slaves might be down for maintenance or for taking a backup,
• some slaves could be temporarily unreachable for other reasons,
• and a few slaves could be processing a big backlog of relay logs (including delayed slaves), which will make them hard/unsuitable to reconfigure.

A way to reduce the complexity of master promotion is presented below, but to get there, we must first give some context about Binlog Servers and abstract them into a service.

Reminders about Binlog Servers

In a previous post, I described how to take advantage of Binlog Server to perform master promotion without GTIDs and without log-slave-updates, while still requiring to reconfigure all slaves. To do this, the slaves must replicate through a Binlog Server. This gives us the following deployment with a single Binlog Server:

+---+
| A |
+---+
  |
 / \
/ X \
-----
  |
  +----------+----------+----------+----------+----------+
  |          |          |          |          |          |
+---+      +---+      +---+      +---+      +---+      +---+
| B |      | C |      | D |      | E |      | F |      | G |
+---+      +---+      +---+      +---+      +---+      +---+

or with redundant Binlog Servers:

+---+
| A |
+---+
  |
  +--------------------------------+
  |                                |
 / \                              / \
/ X \                            / Y \
-----                            -----
  |                                |
  +----------+----------+          +----------+----------+
  |          |          |          |          |          |
+---+      +---+      +---+      +---+      +---+      +---+
| B |      | C |      | D |      | E |      | F |      | G |
+---+      +---+      +---+      +---+      +---+      +---+

or with more than one site with redundant Binlog Servers.

  +---+
  | A |
  +---+
    |
    +-----------+------------------------+
    |           |                        |
   / \         / \                      / \         / \
  / X \       / Y \                    / Z \------>/ W \
  -----       -----                    -----       -----
    |           |                        |           |
  +-+-----------+-+                    +-+-----------+-+
  |               |                    |               |
+---+           +---+                +---+           +---+
| S1|    ...    | Sn|                | T1|    ...    | Tm|
+---+           +---+                +---+           +---+

These schemas are becoming increasingly complex - let's simplify them by abstracting the Binlog Servers.

Binlog Server Abstraction

By hiding the Binlog Servers in an abstracted layer, which I call the Distributed Binlog Serving Server (DBSS), a deployment on three sites becomes the following:

   +---+
   | M |
   +---+
     |
+----+----------------------------------------------------------+
|                                                               |
+----+---------+-----------+---------+-----------+---------+----+
     |         |           |         |           |         |
   +---+     +---+       +---+     +---+       +---+     +---+
   | S1| ... | Sn|       | T1| ... | Tm|       | U1| ... | Uo|
   +---+     +---+       +---+     +---+       +---+     +---+

Of course, the DBSS is built with many Binlog Servers. One way to build the layer above minimizing the number of slaves served by the master is described below. Other ways to build this layer can be imagined [2], but let's stick to this one, for now.

+----|----------------------------------------------------------+
|    +---------------------+---------------------+              |
|    |                     |                     |              |
|   / \                   / \                   / \             |
|  / X1\----->/ \        / X2\----->/ \        / X3\----->/ \   |
|  -----     / Y1\       -----     / Y2\       -----     / Y3\  |
|    |       -----         |       -----         |       -----  |
+----|---------|-----------|---------|-----------|---------|----+

In the deployment above, using one DNS A record per site resolving to both Xi and Yi, if a Binlog Server fails, its slaves will reconnect to the other one. If the Yi Binlog Server fails, nothing more needs to be done. If the Xi Binlog Server fails, the corresponding Yi must be repointed to the master. This repointing is easy, as, by design, a Binlog Server is identical to its master. Only the destination server must be changed, and the binary log filename and position stay the same.

When the Master Fails...

Equipped with the above implementation of the DBSS, in a situation when the master fails, we end up with the state below; each site might be at a different position in the binary log stream of the failed master.

+---------------------------------------------------------------+
|                                                               |
|   / \                   / \                   / \             |
|  / X1\----->/ \        / X2\----->/ \        / X3\----->/ \   |
|  -----     / Y1\       -----     / Y2\       -----     / Y3\  |
|    |       -----         |       -----         |       -----  |
+----|---------|-----------|---------|-----------|---------|----+

The first step of master promotion is to level the Binlog Servers in the DBSS. To do so, the most up-do-date Binlog Server must be found and all other Binlog Servers must be chained to it. In the deployment above, only three servers must be contacted, which is much easier than tens of slaves. If the most up-to-date Binlog Server is X2, levelling the Binlog Servers consists of the temporary replication architecture below.

+---------------------------------------------------------------+
|                                                               |
|   / \ <-----------------/ \-----------------> / \             |
|  / X1\----->/ \        / X2\----->/ \        / X3\----->/ \   |
|  -----     / Y1\       -----     / Y2\       -----     / Y3\  |
|    |       -----         |       -----         |       -----  |
+----|---------|-----------|---------|-----------|---------|----+

Levelling should happen very quickly (if it does not, one of the Binlog Servers is lagging, which should not happen). After that, the slaves will quickly follow. Once a slave is up to date (this actually does not need levelling, a slave of X2 or Y2 could have been promoted before levelling), master promotion can be performed. Shown below, a slave from the third site on the right has been chosen to be the new master, but any slave on any of the three sites could have been used.

+------------------------------------------------|--------------+
|    +---------------------+---------------------+              |
|    |                     |                     |              |
|   / \                   / \                   / \             |
|  / X1\----->/ \        / X2\----->/ \        / X3\----->/ \   |
|  -----     / Y1\       -----     / Y2\       -----     / Y3\  |
|    |       -----         |       -----         |       -----  |
+----|---------|-----------|---------|-----------|---------|----+

Note that the other slaves have not been touched: they are still connected to their Binlog Server. This means that this solution works well even if one of the slaves is unavailable during master promotion. This solution also works very well with delayed or lagging slaves, as those slaves are simply not good candidates for becoming the new master. For some time, the lagging slaves will process the binary logs of the old master that are still stored on the Binlog Servers.

The Trick for not Reconfiguring every Slave

Promoting a slave to be the new master in a DBSS deployment requires working some magic on a slave to make its binary log position (SHOW MASTER STATUS) matches what is expected by the Binlog Servers. Let's take an example: if the last binary log stored on the levelled Binlog Servers is binlog.000163, we could repoint the Binlog Servers to a new master if the SHOW MASTER STATUS of this new master is at the beginning of binary log filename binlog.000164.

When doing that promotion, from the point of view of the Binlog Servers, their master is simply restarted with a different server_id and server_uuid. From the point of view of the slaves, they are processing the binary logs of the old master (up to and including binlog.000163) followed by the binary logs of the new master (starting at binlog.000164).

So, the trick is to have our candidate master at the right binary log position. This can be made possible by:

1. configuring all nodes with binary logging enabled,
2. with all identical log-bin value (binlog in the example above),
3. and without enabling log-slave-updates.

Configuration #3 above allows us to assume that the master will consume binary log filenames much faster than the slaves. This way, the slaves will always be behind the master in their binary log filenames [3]. As such, bringing a slave to the right binary log filename is as simple as doing FLUSH BINARY LOGS in a loop until the slave is in the correct position. To avoid this loop from taking too much time, we can run a cron job on our slaves that makes sure they are not too far away from their master (maximum ten binary logs away, for an example).

Summary of Master Promotion

In the following replication deployment, with log-bin=binlog and with log-slave-updates disabled:

   +---+
   | M |
   +---+
     |
+----+----------------------------------------------------------+
|                                                               |
+----+---------+-----------+---------+-----------+---------+----+
     |         |           |         |           |         |
   +---+     +---+       +---+     +---+       +---+     +---+
   | S1| ... | Sn|       | T1| ... | Tm|       | U1| ... | Uo|
   +---+     +---+       +---+     +---+       +---+     +---+

If M fails, we first level the Binlog Servers in the DBSS.

Once this is done, and let's take T1 as our candidate master, we need to perform the following on it:

1. FLUSH BINARY LOGS until the binary log filename follows the last one from the levelled DBSS,
2. PURGE BINARY LOGS TO <latest binary log file>,
3. RESET SLAVE ALL.

The step #2 above drops all binary logs on the new master that could conflict with the one from the previous master. The binary logs of the old master are stored on the DBSS and we must be sure to avoid having similar, but misleading data, on the new master.

We now have this:

   +\-/+
   | X |
   +/-\+

+---------------------------------------------------------------+
|                                                               |
+----+---------+---------------------+-----------+---------+----+
     |         |                     |           |         |
   +---+     +---+       +---+     +---+       +---+     +---+
   | S1| ... | Sn|       | T1| ... | Tm|       | U1| ... | Uo|
   +---+     +---+       +---+     +---+       +---+     +---+

where we repoint the DBSS to T1 to get the following:

   +\-/+                 +---+
   | X |                 | T1|
   +/-\+                 +---+
                           |
+--------------------------+------------------------------------+
|                                                               |
+----+---------+-----------+---------+-----------+---------+----+
     |         |           |         |           |         |
   +---+     +---+       +---+     +---+       +---+     +---+
   | S1| ... | Sn|       | T2| ... | Tm|       | U1| ... | Uo|
   +---+     +---+       +---+     +---+       +---+     +---+

and we have achieved master promotion without reconfiguring all slaves.

A Cleaner Way

The trick above works well, but preforming FLUSH BINARY LOGS in a loop is not the cleanest of solutions. It would be much better if there was a way to set the binary log to the desired filename in a single operation. With this idea in mind, we created the following two feature requests:

• https://bugs.mysql.com/bug.php?id=77438
• https://mariadb.atlassian.net/browse/MDEV-8355

MariaDB 10.1.6 is already implementing a RESET MASTER TO syntax. Let's hope that Oracle will provide something similar in MySQL 5.7.

What about the Software?

This idea and procedure is all well and good, but it is not very useful if you cannot use it yourself. The currently available version of the Binlog Server, the MaxScale Binlog Router plugin, does not yet implement all the configuration hooks needed to make this procedure easy. Booking.com is currently working with MariaDB to implement the missing hooks in a new version of MaxScale. We are in the last testing phase of a Binlog Router plugin that support the following:

• STOP SLAVE, START SLAVE, SHOW MASTER STATUS, SHOW SLAVE STATUS, CHANGE MASTER TO: these new commands allow easier configuration of the Binlog Server.
• The CHANGE MASTER TO command not only allows to easily chain Binlog Servers, but also to bootstrap a Binlog Server without editing the configuration file. Moreover, this command allows to repoint MaxScale to a new master at binary log filename N+1, effectively enabling to perform master promotion.
• Transaction safety: when the master fails, the Binlog Server could have downloaded a partial transaction. If we replace the master with a slave, this transaction should not be sent to slaves. So this feature of the next version of MaxScale will make sure such partial transactions are not sent downstream.
• DBSS identity: the initial design of the Binlog Server was intended to impersonate the master, and did not consider swapping the master at the top of the hierarchy. In a DBSS deployment, swapping the master should not be made visible to slaves, so the Binlog Servers should present the slave with a different server_id and server_uuid to those of the master. The next version of the MaxScale Binlog Router supports that virtual master feature.

This next version of the MaxScale Binlog Router will be generally available once we are done with the testing. Stay tuned on the MariaDB web site for the announcement and the failover procedure. In the meantime, you can still experiment with master promotion without reconfiguring all slaves by using the current version of MaxScale and following this proof of concept procedure.

If you are interested in this topic and would like to learn more, I am giving a talk about Binlog Servers at Percona Live Amsterdam. Feel free to grab me after the talk, catch me at the Booking.com booth (#205) or share a drink with me at the Community Dinner, to exchange thoughts on this subject. (You can also post a comment below.)

I will also be giving a talk about Binlog Servers at Oracle Open World in San Francisco at the end of October.

One last thing: if you want to know more about other cool things we do at Booking.com, I suggest you come to our other talks at Percona Live Amsterdam in September:

• The Virtues of Boring Technology
• Combining Redis and MySQL to store HTTP cookie data
• Events storage and analysis with Riak at Booking.com
• Encrypted MySQL Backups and instant recoverability on large scale
• Unicode and MySQL
• Managing and Visualizing your replication topologies with Orchestrator
• Your Clone Army: Better scalability through more database servers
• Riding the Binlog: an in Deep Dissection of the Replication Stream

[1] Slave levelling can be done with MHA, with MySQL 5.6 or MariaDB 10.0 GTIDs, or with Pseudo-GTIDs when using earlier versions of MySQL and MariaDB.

[2] If we were not concerned about WAN bandwidth, all Binlog Servers could be directly connected to the master. Another solution could be to connect all master-local Binlog Servers directly to the master and to use the chained strategy for remote Binlog Servers. (This hybrid deployment could be well-suited to a semi-sync deployment, but I am diverging from the subject of this post.)

[3] The same can be achieved when using log-slave-updates, by using smaller max_binlog_size on the master than on all the slaves.

Are you afraid to have your designs critiqued? If so you’re not alone, but it’s worth considering how this affects your ability to iterate on your designs and improve the product for your customers.

As designers, we sometimes struggle to ask for feedback during the design process. We get attached to our ideas because of the long hours we spend perfecting our designs. We may also be afraid to show our work in progress, because we don’t want to be judged too harshly on these early attempts. Society reinforces a fear of failure throughout our lives, highlighting achievements but rarely discussing the failures it took to reach them.

There are steps we can take to overcome these mental blocks, that inhibit us from delivering the best experience to our customers. The first step is changing your own attitude towards failure and criticism. The second is to build a culture within your company which recognises failure and iteration as necessary to create a great product for your customers, which is something we value at Booking.com.

“You are not your idea, and if you identify too closely with your ideas, you will take offense when they are challenged.” Catmull, E. and Wallace, A. (2014) Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration

While you may feel that your initial design is a reflection of your skill, it’s easier to view it as a concept to be shaped through several iterations. Then you can see it as being constantly in flux and you will continue to adapt your ideas based on new information, data or feedback. At Booking.com we encourage this, along with a healthy dose of humility, because we must let our designs be part of the whole experience and not something to be exclusively owned.

When we are young we learn by doing, children will test multiple possibilities to solve a problem. No one tells them that it’s wrong to experiment or that they must find the correct answer on the first try. Taking a cue from our younger selves, we can be open to experimentation, allowing us to iterate through various concepts and continually improve.

Experimentation is the key to validating our ideas at Booking.com. Only a small percentage of experiments are successful, but if we aren’t willing to fail, we also won’t be able to learn how to improve the product for our customers.

“This is one of the most important lessons of the scientific method: if you cannot fail, you cannot learn.” Ries, E. (2011). The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses.

We can experiment on everything: from the designs on our website to new internal processes. If someone has an idea, they are empowered to implement it, get feedback and iterate. Experimentation can include: A/B testing, creating a prototype during a Hackathon, qualitative user testing or trying out a new process on your team members. Being in a constant state of change means we must be agile and take on new challenges, both individually and in our teams.

We are constantly receiving feedback from our users. So when an idea fails, we can’t take it personally, we must rethink our hypothesis or implementation and try again. We see all user feedback as good feedback, because it takes us closer to delivering what the customer wants.

A work-in-progress design is often not beautiful. You may feel that sharing a design in this embryonic stage could be damaging to your credibility. That people may believe you’re an inferior designer from your rough sketches, wireframes or unpolished layouts. But the upside to getting feedback early is that you can respond to problems and change your idea, interaction pattern or layout quickly, which reduces rework time.

“The trick to having good ideas is not to sit around in glorious isolation and try to think big thoughts. The trick is to get more parts on the table.” Johnson, S. (2010). Where Good Ideas Come From: The Natural History of Innovation

We often have the image of a creative genius on their own, conjuring amazing designs. However, most of us know this is rarely the case. By asking for feedback on your idea or design from your peers, you access different perspectives. If you find holes in your own logic because you can’t justify your ideas, this could be a red flag that your concept needs to be reworked.

With the spirit of experimentation, it becomes important for all of us to support each other on the journey. At Booking.com there are many ways for designers to get feedback and encouragement from their peers. These can include: face to face chats with peers, an internal chatroom, a designer email list, meeting with designers in your track for feedback and a design sharing tool akin to Dribbble (which was produced during several Hackathons). All of these produce a similar result - quick feedback that can be used to improve your idea or design. This decreases the time between getting our ideas in front of customers, collecting their feedback through experimentation and continuing to iterate on the design.

“Failure isn’t a necessary evil. In fact, it isn’t evil at all. It is a necessary consequence of doing something new.” Catmull, E. and Wallace, A. (2014). Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration

By accepting that failing is part of learning, we decrease our fear of failure and become more willing to experiment with new ideas. As we experiment and seek feedback, we will see how this benefits our customers, by creating a great product that is built on data and not opinions. Innovation won’t happen without failure. We must embrace it to continue learning and grow!

Booking.com constantly monitors, inspects, and analyzes our systems in order to make decisions. We capture and channel events from our various subsystems, then perform real-time, medium and long-term computation and analysis.

This is a critical operational process, since our daily work always gives precedence to data. Relying on data removes the guesswork in making sound decisions.

In this series of blog posts, we will outline details of our data pipeline, and take a closer look at the short and medium-term storage layer that was implemented using Riak.

Introduction to Events Storage

Booking.com receives, creates, and sends an enormous amount of data. Usual business-related data is handled by traditional databases, caching systems, etc. We define events as data that is generated by all the subsystems on Booking.com.

In essence, events are free-form documents that contain a variety of metrics. The generated data does not contain any direct operational information. Instead, it is used to report status, states, secondary information, logs, messages, errors and warnings, health, and so on. The data flow represents a detailed status of the platform and contains crucial information that will be harvested and used further down the stream.

To put this in numerical terms - we have more than billions of events per day, streaming at more than 100 MB per second, and adding up to more than 6 TB per day.

Here are some examples of how we use the events stream:

• Visualisation: Wherever possible, we use graphs to express data. To create them, we use a heavily-modified version of Graphite.
• Looking for anomalies: When something goes wrong, we need to be notified. We use threshold-based notification systems (like seyren) as well as a custom anomaly detection software, which creates statistical metrics (e.g. change in standard deviation) and alerts if those metrics look suspicious.
• Gathering errors: We use our data pipeline to pass stack traces from all our production servers into ElasticSearch. Doing it this way (as opposed to straight from the web application log files) allows us to correlate errors with the wealth of the information we store in the events.

These typical use-cases are made available in less than one-minute after the related event has been generated.

High Level overview

This is a very simplified diagram of the data flow :

--------------senders|box2|...|boxn|(e.g.Webapps)--------------||----------------------aggregators|collect1|...|collectm|----------------------||--------------storage|Riak|======>|Hadoop|--------------||||-----------------------|shortterm||longterm|datacrunching|consumers||consumers|-----------------------

We can generate events by using literally any piece of code that exists on our servers. We pass a HashMap to a function, which packages the provided document into a UDP packet and sends it to a collection layer. This layer aggregates all the events together into "blobs", which are split by seconds (also called epochs) and other variables. These event blobs are then sent to the storage layer running Riak. Finally, Riak sends them on to Hadoop. The Riak cluster is meant to safely store around ten days of data. It is used for near real-time analysis (something that happened seconds or minutes ago), and medium-term analysis of relatively small amounts of data. We use Hadoop for older data analysis or analysis of a larger volume of data.

The above diagram is a simplified version of our data flow. In practical application, it's spread across multiple datacenters (DC), and includes an additional aggregation layer.

Individual Events

An event is a small schema-less[1] piece of data sent by our systems. That means that the data can be in any structure with any level of depth, as long as the top level is a HashTable. This is crucial to Booking.com - the goal is to give as much flexibility as possible for the sender, so that it's easy to add or modify the structure, or the type and number of events.

Events are also tagged in four different ways:

• the epoch at which they were created
• the DC where they originated
• the type of event
• the subtype.

Some common types are:

• WEB events (events produced by code running under a web server)
• CRON events (output of cron jobs)
• LB events (load balancer events)

The subtypes are there for further specification and can answer questions like: "Which one of web server systems are we talking about?".

Events are compressed Sereal blobs. Sereal is possibly the best schema-less serialisation format currently available. It was also written at Booking.com.

An individual event is not very big, but a huge number of them are sent every second.

We use UDP as transport because it provides a fast and simple way to send data. Despite some (very low) risk of data loss, it doesn't impact senders sending events. We are experimenting with an UDP-to-TCP relay that will be local to the senders.

Aggregated Events

Literally every second, events from this particular second (called epoch), DC number, type, and subtype are merged together as an Array of events on the aggregation layer. At this point, it's important to try and get the smallest size possible, so the events of a given epoch are re-serialized as a Sereal blob, using these options:

compress=>Sereal::Encoder::SRL_ZLIB,dedupe_strings=>1

dedupe_strings increases the serialisation time slightly. However it removes strings duplications which occur a lot since events are usually quite similar between them. We also add gzip compression.

We also add the checksum of the blob as a postfix, to be able to ensure data integrity later on. The following diagram shows what an aggregated blob of events looks like for a given epoch, DC, type, and subtype. You can get more information about the Sereal encoding in the Sereal Specification.

This is the general structure of an events blob:

<----------- Sereal part -----------><------ suffix ----->3D F3 726C 3300 DE 36 FD 1F78...983D 63686B 34...33`----+----' |  |  `-+-' `-+-' `--+--' `---+-----' `------'|||||||||||||||   SHA1 of Sereal
     ||||||     string "=chk"|||||      `-> gz compressed payload
     ||||     `-> varint compressed size: 32031|||    `-------> varint uncompressed : 24118||  `------------> header-suffix-size: 0|      `---------------> version-type: version 3 gzip
     `----------------------> magic string

The compressed payload contains the events themselves. It's an Array of HashMaps, Serialized in a Sereal structure and gzip-compressed. Here is an example of a trivial payload of two events, as follows:

[{cpu=>5},{cpu=>99}]

And the gzipped payload would be the compressed version of this binary string:

,---------------------------->ARRAYREF_2,Arrayoflength2|,------------------------->REFN,areferenceon...||,---------------------->...aHASH,ofsize...|||,------------------->...POS_1,thevalue1.||||,---------------->FirstkeyisaSHORT_BINARY_3|||||,---------->thestring"cpu"||||||,---->POS_5,thevalue5.|||||||,->asbefore,HASH_1:||||||||,->keysisCOPYof...|||||||||,->the6thbyte:cpu||||||||||,->aVARINT|||||||||||99|||||||||||||||||--+---|--+---|||-+-42282A016363707505282A012F0520630A`--------------+---------------' `-----+------'||firstelementoftheArraysecondelement

It can be hard to follow these hexdigits [2], yet it's a nice illustration of why the Sereal format helps us to reduce the size of serialised data. The second array element is encoded on far fewer bytes than the first one, since the key has already be seen. The resulting binary is then re-compressed. The Sereal implementation offers multiple compression algorithms, including Snappy and gzip.

A typical blob of events for one second/DC/type/subtype can weight anywhere from several kilobytes to several megabytes, which translates into a (current) average of around 250 gigabytes per hour.

Side note: smaller subtypes on this level of aggregation aren't always used, because we want to minimise the data we transmit over our network by having good compression ratios. Therefore we split types into subtypes only when the blobs are big enough. The downside to this approach is that consumers have to fetch data for the whole type, then filter out only subtypes they want. We're looking at ways to find more balance here.

Data flow size and properties

Data flow properties are important, since they're used to decide how data should be stored:

• The data is timed and all the events blobs are associated with an epoch. It’s important to bear in mind that events are schema-less, so the data is not a traditional time series.
• Data can be considered read-only; the aggregated events blobs are written every second and almost never modified (history rewriting happens very rarely).
• Once sent to the storage, the data must be available as soon as possible

Data is used in different ways on the client side. A lot of consumers are actually daemons that will consume the fresh data as soon as possible - usually seconds after an event was emitted. A large number of clients read the last few hours of data in a chronological sequence. On rare occasions, consumers access random data that is over a few days old. Finally, consumers that want to work on larger amounts of older data would have to create Hadoop jobs.

There is a large volume of data to be moved and stored. In numerical terms:

• Once serialized and compressed into blobs, it is usually larger than 50 MB/s
• That's around 250 GB per hour and more than 6 TB per day
• There is a daily peak hour but the variance of the data size is not huge: There are no quiet periods
• Yearly peak season stresses all our systems, including events transportation and storage, so we need to provision capacity for that

Why Riak

In order to find the best storage solution for our needs, we tested and benchmarked several different products and solutions.

The solutions had to reach the right balance of multiple features:

• Read performance had to be high as a lot of external processes will use the data.
• Write security was important, as we had to ensure that the continuous flow of data could be stored. Write performance should not be impacted by reads.
• Horizontal scalability was of utmost importance, as our business and traffic continuously grows.
• Data resilience was key: we didn't want to lose portions of our data because of a hardware problem.
• Allowed a small team to administrate and make the storage evolve.
• The storage shouldn't require the data to have a specific schema or structure.
• If possible, it would be able to bring code to data, perform computation on the storage itself, instead of having to get data out of the storage.

After exploring a number of distributed file systems and databases, we chose Riak over distributed Key-Value stores. Riak had good performance and predictable behavior when nodes fail and when scaling up. It also had the advantage of being easy to grasp and implement within a small team. Extending it was very easy (which we'll see in the next part of this series of blog posts) and we found the system very robust - we never had to face dramatic issues or data loss.

Disclaimer: This is not an endorsment for Riak. We compared it carefully to other solutions over a long period of time and it seemed to be the best product to suit our needs. As an example, we thoroughly tested Cassandra as an alternative: it had a larger community and similar performance but was less robust and predictable; it also lacked some advanced features. The choice is ultimately a question of priorities. The fact that our events are schema-less made it almost impossible for us to use solutions that require knowledge of the data structures. Also we needed a small team to be able to operate the storage, and a way to process data on the cluster itself, using MapReduce or similar mechanisms.

Riak 101

The Riak cluster is a collection of nodes (in our case physical servers), each of which claims ownership of a given key. Depending on the chosen replication factor, each key might be owned by multiple nodes. You can ask any node for a key and your request will be redirected to one of the owners. Same goes for writes.

On closer inspection of Riak, we see that keys are grouped into virtual nodes. Each physical node can own multiple virtual nodes. This simplifies data rebalancing when growing a cluster. Riak does not need to recalculate the owner for each individual key; it will only do it per virtual node.

We won't cover Riak architecture in a great detail in this post, but we recommend reading the following article for further information.

Riak clusters configuration

The primary goal of this storage is to keep the data safe. We went with the regular replication number value of three. Even if two nodes owning the same data will go down, we won't lose our data.

Riak offers multiple back-ends for actual data storage. The main three are Memory, LevelDB, and Bitcask. We chose Bitcask, since it was suitable for our particular needs. Bitcask uses log-structured hash tables that provide very fast access. As data gets written to the storage, Bitcask simply appends data to a number of opened files. Even if a key is modified or deleted, the information will be written at the end of these storage files. An in-memory HashTable maps the keys with the position of their (latest) value in files. That way, at most one seek is needed to fetch data from the file system.

Data files are then periodically compacted, and Bitcask provides very good expiration flexibility. Since Riak is a temporary storage solution for us, we set it up with automatic expiration. Our expiration period varies. It depends on the current cluster shape, but usually falls between 8-11 days.

Bitcask keeps all of the keys of a node in memory, so keeping large numbers of individual events as key value pairs isn't trivial. We sidestep any issues by using aggregations of events (blobs), which drastically reduce the number of needed keys.

More information about Bitcask can be found here.

For our conflict resolution strategy, we use Last Write Wins. The nature of our data (which is immutable as we described before) allows us to avoid the need for conflict resolution.

The last important part of our setup is load balancing. It is crucial in an enviromnent with a high level of reads, and only 1 gigabyte network. We use our own solution for that based on Zookeeper. Zooanimal daemons are running on the riak nodes, and collect information about system health. The information is then aggregated into simple text files, where we have an ordered list of IP addresses, plus up and running Riak nodes, which we can connect to. All our Riak clients simply choose a random node to send their requests to.

We currently have two Riak clusters in different geographical locations, each of which have more than 30 nodes. More nodes equates to more storage space, CPU power, RAM, and more network bandwidth available.

Data Design

Riak is primarily a key-value store. Although it provides advanced features (secondary indexes, MapReduce, CRDTs), the simplest and most efficient way to store and retrieve data is to use the key-value model.

Riak has three concepts - a bucket is a namespace, in which a key is unique. A key is the identifier of the data; and has to be stored in a bucket. A value is the data; it has an associated mime-type, which can enable Riak awareness of its type.

Riak doesn't provide efficient ways to retrieve the list of buckets or the list of keys by default [3]. When using Riak, it's important to know the bucket and key to access. This is usually resolved by using self-explanatory identifiers.

In our case, our events are stored as Sereal-encoded blobs. From these, we know the datacenter, type, subtype, and of course the time at which it was created.

When we need to retrieve data, we always know the time we want. We are also confident in the list of our datacenters. It doesn't change unexpectedly so we can make it static for our applications. We are not always sure about what types or subtypes will appear in a given epoch for a given datacenter. On some seconds events of certain types may not arrive.

We came up with this simple data design:

• events blobs are stored in the events bucket, keys being <epoch>:<dc>:<type>:<subtype>:<chunk>
• metadata are stored in the epochs bucket, keys being <epoch>:<dc> and values being the list of events keys for this epoch and DC combination

The value of chunk is an integer, starting at zero, which keeps event blobs smaller than 500 kilobytes each. We use the integer to split big events blobs into smaller ones, so that Riak can function more efficiently.

We'll see this data design in action when pushing data to Riak, in the next blog post of this series.

Next post: data processing outside of Riak

The next part of this blog posts series will explain how we enter and fetch data from Riak, in order to do real-time processing and batch processing.

Stay tuned!

Notes

[1] It is not strictly true that our events are schema-less. They obey the structure that the producers found the most useful and natural. But they are so many producers which each of them sending events that have a different schema, so it's almost equivalent to considering them schema-less. Our events can be seen as structured, yet with so many schemas that they can't be traced. There is also complete technical freedom to change the structure of an event, if it's seen as useful by a producer.

[2] After spending some time looking at and decoding Sereal blobs, the human eye easily recognizes common data structures like small HashMaps, small Arrays, small Integers and VarInts, and of course, Strings, since their content is untouched. That makes Sereal an almost human readable serialisation format, especially after a hexdump.

[3] This can be worked around by using secondary indexes (2i) if the backend is eleveldb or Riak Search, to create additional indexes on keys, thus enabling listing them in various ways.

In this post, we'll see how to push data to Riak, how to read it later on, and how to perform data processing out of Riak. As a reminder, we saw in the previous part that the events are schema-less HashTable data structures, grouped by epochs, data centers, types and subtypes, then serialised using Sereal and highly compressed.

If you missed part 1

We strongly recommend that you read part one of this blog series. The previous part explains how Booking.com collects and stores events from its backend into a central storage, and why Riak was chosen to do so.

Pushing to Riak

Pushing data to Riak is done by a number of relocators, which are daemons running on the aggregation layer that then push events blobs to Riak.

Side note: it's not recommended to have keys more then 1-2MB in Riak (see this FAQ). And since our blobs can be 5-10MB in size, we shard them into chunks, 500KB each. Chunks are valid Sereal documents, which means we do not have to stich chunks together in order to retrieve data back.

This means that we have quite a lot of blobs to send to Riak, so to maximise our usage of networking, I/O, and CPU, it's best to send data in a mass-parallel way. To do so, we maintain a number of forked processes (20 per host is a good start), in which each of them push data to Riak.

Pushing data to Riak can be done using the HTTP API, or the Protocol Buffers Client (PBC) API. PBC has a slighly better performance.

Whatever protocol is used, it's important to maximise I/O utilisation. One way is to use an HTTP library that parallelises the requests in term of I/O (YAHC is an example). Another method is to use an asynchronous Riak Client like AnyEvent::Riak.

We use an in-house library to create and maintain a pool of forks, but there are more than one existing libraries on CPAN, like Parallel::ForkManager.

PUT to Riak

Writing data to Riak is rather simple. For a given epoch, we have the list of events blobs, each of them having a different DC/type/subtype combination (remember, DC is short for Data Center). For example:

1413813813:1:type1:subtype11413813813:1:type1:subtype21413813813:1:type2:1413813813:2:type1:subtype11413813813:2:type3:`----+---'|`-+-'`---+--'|||`---->optionalsubtype||`------------>type|`---------------->DCnumber`---------------------->epoch

The first task is to slice the blobs into 500 KB chunks and add a postfix index number to their name. That gives:

1413813813:1:type1:subtype1:01413813813:1:type1:subtype1:11413813813:1:type1:subtype1:21413813813:1:type1:subtype2:01413813813:1:type2::01413813813:1:type2::11413813813:2:type1:subtype1:01413813813:2:type1:subtype1:11413813813:2:type3::0

Next, we can store all the event blobs in Riak in the events bucket. We can simulate it with curl:

curl-d<data>-XPUT"http://node:8098/buckets/events/keys/1413813813:1:type1:subtype1:0"# ...curl-d<data>-XPUT"http://node:8098/buckets/events/keys/1413813813:2:type3::0"

Side note: we store all events in each of the available Riak clusters. In other words, all events from all DCs will be stored in the Riak cluster which is in DC 1, as well as in the Riak cluster which is in DC 2. We do not use cross DC replication to achieve that - instead we simply push data to all our clusters from the relocators.

Once all the events blobs are stored, we can store the metadata, which is the list of the event keys, in the epochs bucket. This metadata is stored in one key per epoch and DC. So for the current example, we will have 2 keys: 1413813813-1 and 1413813813-2. We have chosen to store the list of events blobs names as pipe separated values. Here is a simulation with curl for DC 2:

curl-d"type1:subtype1:0|type1:subtype1:1|type3::0"-XPUT"http://riak_host:8098/buckets/epochs/keys/1413813813-2"

Because the epoch and DC are already in the key name, it's not necessary to repeat that in the content. It's important to push the metadata after pushing the data.

PUT options

When pushing data to the Riak cluster, we can use different attributes to change the way data is written - either by specifying which ones when using the PBC API, or by setting the buckets defaults.

Riak's documentation provides a comprehensive list of the parameters and their meaning. We have set these parameters as follows:

"n_val":3,"allow_mult":false,"last_write_wins":true,"w":3,"dw":0,"pw":0,

Here is a brief explanation of these parameters:

• n_val:3 means that the data is replicated three times
• allow_mult and last_write_wins prohibit siblings values; conflicts are resolved right away by using the last value written
• w:3 means that when writing data to a node, we get a success response only when the data has been written to all the three replica nodes
• dw:0 instruct Riak to wait for the data to have reached the node, not the backend on the node, before returning success.
• pw:0 is here to specify that it's OK if the nodes that store the replicas are not the primary nodes (i.e. the ones that are supposed to hold the data), but replacement nodes, in case the primary ones were unavailable ().

In a nutshell, we have a reasonably robust way of writing data. Because our data is immutable and never modified, we don't want to have siblings or conflict resolution on the application level. Data loss could, in theory, happen if a major network issue happened just after having acknowledged a write, but before the data reached the backend. However, in the worst case we would lose a fraction of one second of events, which is acceptable for us.

Reading from Riak

This is how the data and metadata for a given epoch is laid out in Riak:

bucket:epochskey:1428415043-1value:1:cell0:WEB:app:chunk0|1:cell0:EMK::chunk0bucket:eventskey:1428415043:1:cell0:WEB:app:chunk0value:<binaryserealblob>bucket:eventskey:1428415043:1:cell0:EMK::chunk0value:<binaryserealblob>

Fetching one second of data from Riak is quite simple. Given a DC and an epoch, the process is as follow:

• Read the metadata by fetching the key <epoch>-<dc> from the bucket "epochs"
• Parse the metadata value, split on the pipe character to get data keys, and prepend the epoch to them
• Reject data keys that we are not interested in by filtering on type/subtype
• Fetch the data keys in parallel
• Deserialise the data
• Data is now ready for processing

Reading a time range of data is done the same way. Fetching ten minutes of data from Wed, 01 Jul 2015 11:00:00 GMT would be done by enumerating all the epochs, in this case:

143574840014357484011435748402...1435749000

Then, for each epoch, fetch the data as previously mentioned. It should be noted that Riak is specifically tailored for this kind of workload, where multiple parallel processes perform a huge number of small requests on different keys. This is where distributed systems shine.

GET options

The events bucket (where the event data is stored) has the following properties:

"r":1,"pr":0,"rw":"quorum","basic_quorum":true,"notfound_ok":true,

Again, let's look at these parameters in detail:

• r:1 means that when fetching data, as soon as we have a reply from one replica node, Riak considers this as a valid reply, it won't to compare it with other replicas.
• pr:0 remove the requirement that the data comes from a primary node
• notfound_ok:true makes it so that as soon as one node can't find a key, Riak considers that the key doesn't exist (notfound_ok:true).

These parameter values allow to be as fast as possible when fetching data. In theory, such values don't protect against conflicts or data corruption. However, in the "Aggregated Events" section (see the first post), we've seen that every event blob has a suffix checksum. When fetching them from Riak, this enables the consumer to verify that there is no data corruption. The fact that the events are never modified ensures that no version conflict can occur. This is why having such "careless" parameter values is not an issue for this use case.

Real time data processing outside of Riak

After the events are properly stored in Riak, it's time to use it. The first usage is quite simple: extract data out of it and process it on dedicated machines, usually grouped in clusters or aggregations of machines that perform the same kind of analysis. These machines are called consumers, and they usually run daemons that fetch data from Riak, either continuously or on demand. Most of the continuous consumers are actually small clusters of machines spreading the load of fetching data.

------------------|Riak||eventsstorage|------------------/|\70MB/s  /70MB/s |    . . .  \ 70MB/s/|\/|\/|\------------------------|--------|--------|--------||--------||--------||--------|||||||||||||||fetchers||||fetchers|...|||fetchers||||||||||||||||------------------------consumerconsumerconsumerclusterclustercluster

Some data processing is required at near real-time. This is the case for monitoring, and building graphs. Booking.com heavily uses graphs at every layer of its technical stack. A big portion of graphs are generated from Events. Data is fetched every second from the Riak storage, processed, and dedicated graphing data is sent to an in-house Graphite cluster.

Other forms of monitoring also consume the events stream- fetched continuously and aggregated in per-second, per-minute, and daily aggregations in external databases, which are then provided to multiple departments via internal tools.

These kind of processes try to be as close as possible to real-time. Currently there are 10 to 15 seconds of lag. This lag could be shorter: a portion of it is due to the collection part of the pipeline, and an even bigger part of it is due to the re-serialisation of the events as they are grouped together, to reduce their size. A good deal of optimisation could be done there to reduce the lag down to a couple of seconds [1]. However, there was no operational requirement for reducing it and 15 seconds is small enough for our current needs.

Another way of using the data is to stick to real-time, but accumulate seconds in periods. One example is our Anomaly Detector, which continuously fetches events from the Riak clusters. However, instead of using the data right away, it accumulates it on short moving windows of time (every few minutes) and applies statistical algorithms on it. The goal is to detect anomalous patterns in our data stream and provide the first alert that prompts further action. Needless to say, this client is critical.

Another similar usage is done when gathering data related to A/B testing. A large number of machines harvest data from the events' flow before processing it and storing the results in dedicated databases for use in experimentation-related tooling.

There are a number of other usages of the data outside of Riak, including manually looking at events to check new features behaviours or analysing past issues / outages.

Limitations of data processing outside of Riak

Fetching data outside of the Riak clusters raises some issues that are difficult to work around without changing the processing mechanism.

First of all, there is a clear network bandwidth limitation to the design: the more consumer clusters there are, the more network bandwidth is used. Even with large clusters (more than 30 nodes), it's relatively easy to exhaust the network capacity of all the nodes as more and more fetchers try to get data from them.

Secondly, each consumer cluster tends to use only a small part of the events flow. Even though consumers can filter out types, subtypes, and DCs, the resulting events blobs still contain a large quantity of data that is useless to the consumer. For storage efficiency, events need to be stored as large compressed serialised blobs, so splitting them more by allowing more subtyping is not possible [2].

Additionally, statically splitting the events content is too rigid since use of the data changes over time and we do not want to be a bottleneck to change for our downstream consumers. Part of an event from a given type that was critical 2 years ago might be used for minor monitoring now. A subtype that was heavily used for six month may now be rarely used because of a technical change in the producers.

Finally, the amount of CPU time needed to uncompress, load, and filter the big events blobs is not tiny. It usually takes around five seconds to fetch, uncompress, and filter one second's worth of events. Which means that any real-time data crunching requires multiple threads and likely multiple hosts - usually a small cluster. It would be much simpler if Riak could provide a real-time stream of data exactly tailored to the consumer need.

Next post: data filtering and processing inside Riak

What if we could remove the CPU limitations by doing processing on the Riak cluster itself? What if we could work around the network bandwidth issue by generating sub-streams on the fly and in real-time on the Riak cluster?

This is exactly what we implemented, using simple concepts, and leveraging the ease of use and hackability of Riak. These concepts and implementations will be described in the next part of this blog posts series!

Notes

[1] Some optimisation has been done, the main action was to implement a module to split a sereal blob without deserialising it, thus speeding up the process greatly. This module can be found here: Sereal::Splitter. Most of the time spent in splitting sereal blobs is now spent in decompressing it. The next optimization step would be to use compression that decrunches faster than the currently used gzip; for instance LZ4_HC.

[2] At that point, the attentive reader may jump in the air and proclaim "LevelDB and snappy compression!". It is indeed possible to use LevelDB as Riak storage backend, which provides an option to use Snappy compression on the blocks of data stored. However, this compression algorithm is not good enough for our need (using gzip reduced the size by a factor of almost 2). Also, Leveldb (or at least the eleveldb implementation that is used in Riak) doesn't provide automatic expiration which is critical to us, and had issues with reclaiming free space after key deletions, with versions below 2.x

At Booking.com, hundreds of developers and designers contribute daily to our codebase, which leads to potential complications with code discoverability, maintenance, and reuse. In this post, we're going to focus on the client-side aspect of these challenges, and introduce some of the techniques we use to tackle them.

Prefixing and Namespacing

Because of the size of our codebase and the number of people introducing changes to it daily, it can be difficult to maintain unique identifiers for all of our pages' components.

Both JavaScript and Cascading Style Sheets make use of identifiers to work properly, by means of variable names and selectors respectively, both on a global execution scope.

Let's start with JavaScript. We make use of namespaces, which are object properties of a global variable. (We also implement modules, which will be discussed later in the article.)

// The only global variablevarB={};// The search namespaceB.search={};// Check-in date of the searchB.search.checkIn=newDate(2015,3,1);// Travel purpose of the searchB.search.travelPurpose='business';

In the example above, B.search is the namespace we're using to visually identify our code. Notice how travelPurpose has its scope within search, clarifying its context and meaning.

CSS is different. Because CSS doesn't provide a scalable way of grouping selectors, we make use of prefixes instead. We also make sure all selectors are as specific as possible - to prevent collisions. For example, in our files we already have about 4,000 class selectors containing the term item in their declaration.

Imagine the following simple case: a list of facilities on the hotel page.

<ulclass="facilities"><liclass="item">Wi-Fi</li><liclass="item">Breakfast</li></ul>

That might interfere with another team's work that is adding a universal menu on the website's header.

<ulclass="menu"><liclass="item">Home</li></ul>

On both cases, .item will have specific CSS rules that could be overridden, thus generating unexpected results. Sometimes these interactions happen on a specific page that was beyond the scope of the developer's tests.

To prevent these conflicts we often use prefixing:

<ulclass="hp-facilities"><liclass="hp-facilites__facility">Wi-Fi</li><liclass="hp-facilites__facility">Breakfast</li></ul>

Since we invest so much into experimentation through A/B testing, a considerable amount of code becomes irrelevant when its related experiment expires.

Because we want to avoid code rot in our codebase, we want to keep only the parts that we actually need, and those irrelevant pieces of code must be removed periodically. Being able to quickly search the code for a specific token, such as a CSS class name, is a key requirement for this clean up.

Control of Execution

It is very important that our JavaScript code runs in a controlled fashion. Our code needs to be precise by only executing when it is necessary for a certain feature, page, or event. It also needs to be robust, preventing interference between unrelated features.

Suppose we have three script files concatenated, and the resulting file is added to every page on our website.

// from tooltip.js$('.tooltip').addTooltip();// from available_rooms.jsvarprices=$('#prices .price');prices[0].scrollTop=0;// from sticky_user_bar.js$(window).scroll(function(){$('.user_bar').css('top',document.body.scrollTop);});

Any part that fails will prevent the next part from executing. For instance, if there is no element on the page that matches #prices .price there will be an error stopping everything else from executing, breaking the user bar behavior.

Also, there might be code that shouldn't be running, which is undesirable. In the example, if no element matches .user_bar on the page, the scrolling event is still monitored, wasting CPU cycles.

To mitigate this, we chose to develop an internal flow controller that provides an execution control API based on AMD, built on top of Almond. It uses setTimeout to provide a separate context, which then isolates failures and prevents the blockage of the main thread.

As a bonus, because the code is wrapped, we can easily label those calls for profiling and find features that might be wasting resources.

This also helps isolate the experimented part of our A/B testing, making it easy to find and clean up failed tentatives, as mentioned in the previous section.

Restricting Your Execution

While CSS code avoids clashes by using namespaces, JavaScript should not leak behavior to DOM elements or to other unrelated components in the same page.

Part of that problem can be prevented by following well-established coding principles, such as avoiding global variables (enforced by using the strict mode), modularizing the specific parts of your code, and so forth.

We also encourage our teams to develop context specific software to avoid side-effects.

$('form').on('submit',function(){vardestination=$('.destination');});

Here, every form element will have a submit event handler attached to it. Also, it searches the entire document DOM tree for the .destination selector, which might stop working as soon as someone inadvertently adds an element that matches the same selector.

An even better approach stresses specifying targets in more detail, aiming to only affect what needs to be affected.

$('.js-searchbox-form').on('submit',function(event){varform=$(event.currentTarget);vardestination=form.find('.js-destination-input');});

In this scenario, the class names are clear and specific, and the code will only look for elements inside of its own form, preventing possible leaking.

Modularization

Having multiple teams working independently at the same time allows different development styles across the codebase. One developer might like wrapping her own code on IIFE, and another might prefer the prototype pattern.

While this is not a problem if the code is achieving a simple task, more complex solutions might become too big to understand, to manipulate, or to maintain.

functionshowTheGallery(hotelId){/* Here goes 31415 lines of code involving all event handlers, the overlay behind the gallery, capturing the keyboard events to navigate and close the gallery, logic to preload the images, logic to center the main image relative to another element, scrolling of the thumbnails, obtaining the images from a specific variable in the code, showing the gallery, etc. */}showTheGallery(42);

As you can see, parts of the code can become too complicated and isolated, making it difficult to understand and debug. It also prevents any kind of reusability.

However, we can break the code into smaller blocks that serve a specific purpose, as described in the DRY principle as "Every piece of knowledge must have a single, unambiguous, authoritative representation within a system".

define('photo-gallery',['component-maker','dom-position','dom-scroll','env-variables','image-preload','keyboard-events','overlay'],function(...){// Tying them all together nicely, exporting an extensible component});

Here, every dependency is self-contained, specific enough, and totally reusable by others, and the resulting object allows quick extension and behavior changing, so the original code can be adapted.

Components

Following the principles of restricting the behavior of your code to exactly where you want it to run, and the fact that we want to build a modularized and reusable codebase, we developed a simple solution called B.components.

The principle behind it is to add behavior to one or more DOM nodes. It only executes the code when the element exists, and allows one component to extend the features of another, facilitating reusability.

<buttontype="button"data-component="alert">Alert</button>

In this example, we add behavior to a specific button in the code. The JavaScript doesn't need to know which exact element to target in the document, since it's the button that requests a behavior, not the other way around.

The code receives a pointer to the DOM node, and can perform the necessary action, such as listening to click events on this reference and triggering an alert window.

The benefit of this approach is its DOM-based flexibility. We might change every aspect of the HTML, and even add more elements, while preserving the same behavior.

<adata-component="alert">Alert Anchor</a><spandata-component="alert">Alert Span</span>

We use AMD as the foundation to store the component definitions as modules, the same setTimeout technique mentioned before for containing the execution, and to create a new instance for each component definition found in the DOM or any specified node.

This solution frees us from knowing exactly what to do when rendering dynamic HTML on the page.

vartemplateCode='Complex HTML structure';$('body').append(templateCode);// We added tooltips, so initialize them$('.tooltip').tooltip();// We also added a lightbox, but that uses another APILightboxFactory.addLightbox('#lightbox-a',{lightbox:'options'});// Did we forget something? Did some API change?

This is all replaced by a one-liner:

$(templateCode).appendTo('body').loadComponents();

The method $.fn.loadComponents will take care of finding the necessary components, and each component will be given the opportunity to initialize itself, all under the same predictable API.

The Big Picture

Because we are a big company with hundreds of developers, we exercise care so that our contributions leave our code better than we found it, keeping it more maintainable in the long run.

Organizing and namespacing our CSS and JavaScript blocks helps to make our code easily findable and robust. Controlling our executed code scope and only running it when really necessary makes it predictable and fast. Thinking about modules and UI components ensures we are able to quickly understand and reuse our solutions. All of this is done while keeping a good sense of community and respect for each other.

These are just some of the techniques we adopted and created at Booking.com to help us deal with the challenges of growth.

The important lesson is to always look a the big picture, never assume you are in a perfectly controlled environment.

Code must be resilient.

When we introduce new features on our website, sometimes it’s not simply the behaviour of our users that changes. The behaviour of our own systems can change, too.

For example: A new feature might improve conversion (changing a user’s behaviour) while also slowing down our site rendering (changing the behaviour of our systems). This becomes interesting when you realize that the second effect might influence the first – a rendering slowdown might decrease conversion, for instance.

Sometimes, the opposing effects that turn up in our results make for interesting investigation and developments. At Booking.com, we’ve found a way to use multivariant testing to quantify these opposing effects.

Moving forward from traditional A/B testing

We don't change anything on our website without first validating it (traditionally through A/B split testing) and all but a select few of our experiments run on 100% of our eligible traffic. Small CSS changes, the introduction of new features, and even infrastructure changes like a new availability search engine, a new translation infrastructure, or a backend Perl version upgrade all must first go through the same testing.

It shouldn't therefore be any surprise that we run in excess of a thousand experiments in parallel. Our in-house experimentation platform is very well integrated into the foundations of our stack to make this possible. And many of the new features that get put through the experimentation process come with changes to a few layers.

Suppose we want to A/B test the effects of adding new useful data to our country landing page (the page where we hope you land after searching your favourite search engine for "Hotels in Italy"). We think that our visitors might be interested in having some Italian visa information available on that page. Such a feature needs a new data store for the visa information, extra effort to collect that data, UX design, and it won’t even be available for all our visitors as it probably won't be fully-formed in the beginning.

For the sake of argument, let's say that getting the Italian visa information is a kind-of-expensive operation. Let's also say that we don't have a whole lot of this content created yet – maybe we only have visa information written up for 10% of the countries our visitors want.

So, before investing effort into optimizing the data store or extending the data set, we'd like to run an A/B test to find out whether our visitors actually even like the feature. Here's the first way we could choose to implement this experiment (let's call it 'Experiment 1'):

Experiment 1: The simple A/B test

# data store query, introduces some performance overhead
have_data= get_visa_data()

# track_experiment() returns False if this user is in the control group, and
# True for the treatment group
if have_data and track_experiment("show_visa_data"):
    render_visa_data()

What's good about this setup is that the difference in metrics like the number of bookings made or customer service interactions between the unchanged original and the experiment variant can tell us whether the users liked this feature.

Unfortunately, it doesn't tell us anything about the business impact of having this feature: the performance impact of ‘get_visa_data()’ may be driving some users away from our website, and we’re not measuring that in the way we’ve decided to set up Experiment 1.

You might then choose to address this issue by implementing the experiment in a different way.

Experiment 2: The A/B test refined

if track_experiment("show_visa_data"):
    have_data= get_visa_data()
    if have_data:
        render_visa_data()

This experiment accounts for and measures business impact, which Experiment 1 couldn’t do. But there's a downside: if we have visa data for only 10% of our visitors, we would expose only 5% of the visitors in this experiment to the visible change that might entice them to convert better. It’s likely that this dilutes the effect so much as to be unmeasurable.

In other words, Experiment 2 is very likely to come out inconclusive. It might even be negative, due to the performance cost incurred for all users in the variant. If it is inconclusive (or negative), that might mean one of these two cases:

1. Our users don't care about visa information at all;
2. Our users love visa information, but that effect was diluted by the low availability of the data and/or negated by the negative impact of the performance cost.

It would be very valuable if we could tell (1) from (2) as it would influence our decision-making process and point at what we should do next. In case (1), we would abandon the idea and decide to better focus our effort on the next idea that comes along. In case (2) however, we can invest time in extending the data set and/or optimizing the data store.

We found a solution to that which is now quite commonplace in our organization. Here's what we do: instead of running an A/B test, we run a multivariant test (an "A/B/C" test):

Experiment 3: multivariant test

1. No change;
2. Get the visa data. Even if there's data, don't render it;
3. Get the visa data. If there's data for this visitor, render it.

The comparison between (A) and (C) now gives the same data as Experiment 2 from the example above and tells you about the complete business impact (the one drawback of this three-way split is that it comes with a slight loss of statistical power compared to a 50/50 A/B split).

The comparison between (A) and (B) tells us about the impact of doing the data lookup. In other words, it tells us whether there may be something to gain by improving performance there.

The comparison between (B) and (C) tells us whether users for whom we had data liked the feature. As it is, we are quite unlikely to detect this difference as the visual change is only available for 10% of visitors in both groups. Fortunately, our platform makes it very easy to "zoom in" on this 10% subgroup and have a full experiment report that only includes them, giving us much better odds at detecting a difference – It actually gives us the same comparison as what Experiment 1 yields without any of the downsides.

This Experiment 3 multivariant setup adds another exciting possibility. While normal A/B tests can only give us a yes/no decision about enabling the feature for all our users, the new setup has an additional possible outcome akin to saying that a feature is promising but needs improvement.

As a company, we take pride in taking small steps towards optimizing our website, measuring along the way and learning from every result. This multivariant experimentation setup has proved to be a great resource in our toolbox with which to do just that.

In this post, we'll see how to apply transformations to the events data stored in Riak without the data leaving the cluster. We saw in the previous parts how to gather, aggregate and store events in Riak, and how to fetch them for external processing. We'll see now how to reduce bandwidth usage by applying data transformation without moving the events outside of the cluster.

If you missed Part 2

We strongly recommend that you read part 2 of this blog series. The previous parts explains how Booking.com collects and stores events from its backend into a central storage, and how we use it to do events analysis.

The Theory

The reasoning is actually very simple. The final goal is to perform data processing of the events blobs that are stored in Riak in real-time. Data processing usually produces a very small result, and it appears to be a waste of network bandwidth to fetch data outside of Riak to perform data analysis on consumer clusters, as in this example::

This diagram is equivalent to:

So instead of bringing the data to the processing code, let's bring the code to the data:

This is a typical use case for MapReduce. We're going to see how to use MapReduce on our dataset in Riak, and also why it's not a usable solution.

For the rest of this post, it’s important to establish a reference for all the events that are stored for a time period of exactly one second. Because we already happen to store our events by a second (and call it an “epoch”), using this unit of measure is a practical consideration that we’ll refer to as epoch-data.

A first attempt: MapReduce

MapReduce is a very well known (if somewhat outdated) way of bringing the code near the data and distributing data processing. There are excellent papers explaining this approach for further background study.

Riak has a very good MapReduce implementation. MapReduce jobs can be written in Javascript or Erlang. We highly recommend using Erlang for better performance.

To perform events processing of an epoch-data on Riak, the MapReduce job would look like the following list. Metadata and data keys concepts are explained in the part 2 of the blog series. Here are the MapReduce phases:

• Given a list of epochs and DCs, the input is the list of metadata keys, and as additional parameter, the processing code to apply to the data.
• A first Map phase reads the metadata values and returns a list of data keys.
• A second Map phase reads the data values, deserialises it, applies the processing code and returns the list of results.
• A Reduce phase aggregates the results together

This works just fine. For one epoch-data, one data processing code is properly mapped to the events, the data deserialised and processed in around 0.1 second (on our initial 12 nodes cluster). This is by itself an important result: it's taking less than one second to fully process one second worth of events. Riak makes it possible to implement a real-time MapReduce processing system [1].

Should we just use MapReduce and be done with it? Not really, because our use case involves multiple consumers doing different data processing at the same time. Let's see why this is an issue.

The metrics

To be able to test the MapReduce solution, we need a use case and some metrics to measure.

The use case is the following: every second, multiple consumers (say 20) need the result of one of the data processing (say 10) of the previous second.

We'll consider that an epoch-data is roughly 70MB, data processing results are around 10KB each. Also, we'll consider that the Riak cluster is a 30 nodes ring with 10 real CPUs available for data processing on each node.

The first metric we can measure is the external network bandwidth usage. This is the first factor that encouraged us to move away from fetching the events out of Riak to do external processing. External bandwidth usage is the bandwidth used to transfer data between the cluster as a whole, and the outside world.

The second metric is the internal network bandwidth usage. This represents the network used between the nodes, inside of the Riak cluster.

Another metric is the time (more precisely the CPU-time) it takes to deserialise the data. Because of the heavily compressed nature of our data, decompression and deserialising one epoch-data takes roughly 5 sec.

The fourth metric is the CPU-time it take to process the deserialized data, analyze it and produce a result. This is very fast (compared to deserialisation), let’s assume 0.01 sec. at most.

Note: we are not taking into account the impact of storing the data in the cluster (remember that events blobs are being stored every second) because it’s impacting the system the same way in both external processing and MapReduce.

Metrics when doing external processing

When doing standard data processing as seen in the previous part of this blog series, one epoch-data is fetched out from Riak, and deserialised and processed outside of Riak.

External bandwidth usage

The external bandwidth usage is high. For each query, the epoch-data is transferred, so that's 20 queries times 70MB/s = 1400 MB/s. Of course, this number is properly spread across all the nodes, but that's still roughly 1400 / 30 = 47 MB/s. That, however, is just for the data processing. There is a small overhead that comes from the clusterised nature of the system and from gossiping, so let's round that number to 50 MB/s per node, in external output network bandwidth usage.

Internal bandwidth usage

The internal bandwidth usage is very high. Each time a key value is requested, Riak will check its 3 replicas, and return the value. So 3 x 20 x 70MB/s = 4200 MB/s. Per node, it's 4200 MB/s / 30 = 140 MB/s

Deserialise time

Deserialise time is zero: the data is deserialised outside of Riak.

Processing time

Processing time is zero: the data is processed outside of Riak.

Metrics when using MapReduce

When using MapReduce, the data processing code is sent to Riak, included in an ad hoc MapReduce job, and executed on the Riak cluster by sending the orders to the nodes where the epoch-data related data chunks are stored.

External bandwidth usage

When using MapReduce to perform data processing jobs, there is certainly a huge gain in network bandwidth usage. For each query, only the results are transferred, so 20 x 10KB/s = 200 KB/s.

Internal bandwidth usage

The internal usage is also very low: it's only used to spread the MapReduce jobs, transfer the results, and do bookkeeping. It's hard to put a proper number on it because of the way jobs and data are spread on the cluster, but overall it's using a couple of MB/s at most.

Deserialise time

Deserialise time is high: for each query, the data is deserialised, so 20 x 5 = 100 sec for the whole cluster. Each node has 10 CPUs available for deserialisation, so the time needed to deserialise one second worth of data is 100/300 = 0.33 sec. We can easily see that this is an issue, because already one third of all our CPU power is used for deserialising the same data in each MapReduce instance. It's a big waste of CPU time.

Processing time

Processing time is 20 x 0.01 = 0.2s for the whole cluster. This is really low compared to the deserialise time.

Limitations of MapReduce

As we've seen, using MapReduce has its advantages: it's a well-known standard, and allows us to create real-time processing jobs. However it doesn't scale: because MapReduce jobs are isolated, they can't share the deserialised data, and CPU time is wasted, so it's not possible to have more than one or two dozens of real-time data processing jobs at the same time.

It's possible to overcome this difficulty by caching the deserialised data in memory, within the Erlang VM, on each node. CPU time would still be 3 times higher than needed (because a map job can run on any of the 3 replicas that contains the targeted data) but at least it wouldn't be tied to the number of parallel jobs.

Another issue is the fact that writing MapReduce jobs is not that easy, especially because — in this case — it’s a prerequisite to know Erlang.

Last but not least, it's possible to create very heavy MapReduce jobs, easily consuming all the CPU time. This directly impacts the performance and reliability of the cluster, and in extreme cases the cluster may be unable to store incoming events at a sufficient pace. It's not trivial to fully protect the cluster against MapReduce misuse.

A better solution: post-commit hooks

We explored a different approach to enable real-time data processing on the cluster that scales properly by deserialising data only once, allows us to cap its CPU usage, and allows us to write the processing jobs in any language, while still bringing the code to the data, removing most of the internal and external network usage.

This technical solution is what is currently in production at Booking.com on our Riak events storage clusters, and it uses post-commit hooks and a companion service on the cluster nodes.

We'll explore in detail this solution in the next blog post, so stay tuned!

Notes

[1] Using MapReduce on Riak is usually somewhat discouraged because most of the time it's being used in a wrong way, for instance when performing bulk fetch or bulk insert or traversing a bucket. The MapReduce implementation in Riak is very powerful and efficient, but must be used properly. It works best when used on a small number of keys, even if the size of data processed is very large. The fewer keys the less bookkeeping and the better performance. In our case, there are only a couple of hundred keys for one second worth of data (but somewhat large values, around 400K), which is not a lot. Hence the great performance of MapReduce we've witnessed. YMMV.

Have you ever dreamt of embarking on a voyage through space with the USS Enterprise?

The date is October 1st, 2015. For the last 3 hours I’ve been researching as much as I can about Mars occasionally stopping to drool over those beautiful red-tinted landscapes. I’m working on an email campaign titled Add Mars to your travel bucket-list — funny for a designer at Booking.com, “Planet Earth’s #1 Accommodation Site”.

First, let me tell you how I got myself into this project. This morning I was riding the metro to work. I’m in my own world, head tilted downwards and browsing on Flipboard ¹ to get my daily dose of news. That’s when it found me: NASA Confirms Evidence That Liquid Water Flows on Today’s Mars². Water! On Mars?! Mind blown. I devoured the entire article, washing it down with anything else I could find about Mars — hydrated salts, perchlorate, and various other terms that sounded cool (but were too scientific for me to grasp). I suddenly had this longing desire to visit that place, to explore, to get my boots covered in red dust.

At Booking.com, ideas flow freely and come, mostly, from the ground up. Everyone is encouraged to contribute ideas and to follow through with them. Even so, when I stole a few minutes from the Product Owner of Email Marketing to pitch a travel campaign for Mars, her reaction was better than I could have anticipated. Not only did she agree to change our Sunday campaign, but she rearranged her whole day so that we could start searching for otherworldly holiday destinations right here on Earth. Here’s some of the gems that we found:

Atacama, the driest desert on Earth. This is where NASA tests some of it’s rovers before they fly to Mars.

The iron-stained Rio Tinto river bears an uncanny resemblance to Mars’ hue.

These places are scattered all over the globe, some only 3 hours away from our office here in Amsterdam.

Once the list was complete, we took the idea to our copywriters and they eagerly jumped on board. After that, all we needed was a back-end developer and 14 hours later we’d created an email and this landing page from scratch.

My hand was shaking as I hovered with the cursor over the ‘send’ button. What we’d created was to be shared with millions of people out there. Happily, the response carried some really positive energy — overwhelming and humbling at the same time.

Today I look to the seven-year-old me, the one who had a dream to achieve something great (only knowing it would have to involve a keyboard somehow). I know he would be smiling — against all odds his dream came true. I’m grateful for this day; even more so because it’s not the first time I’ve felt like this while working at Booking.com.

All of this was possible thanks to an open environment and stellar talent. Even with such a fast-growing department, individuals are still trusted to make the best decisions for the company. We’re constantly encouraged to think outside the box, to take chances, and discover new possibilities to grow our business.

Without personal contributions we wouldn’t be standing here today. So I want to challenge you to step back for 15 minutes and let your imagination fly. If sending people to Mars wasn’t too crazy an idea for a company specializing in accommodations on Earth, then what can you come up with to positively impact your company?

If you would like to experience the same feelings of freedom and achievement, join Booking.com. We can always use more dreamers :)

In this post we will discuss Progressive Web Apps and Service Workers. How can they help modern-day mobile web users, and how are we experimenting with them at Booking.com? We will share some challenges we've encountered, as well as some of our learnings.

What is a Progressive Web App?

A Progressive Web App (PWA) is a term Google coined to describe its prospect of app-like web experiences, in which web pages are able to offer many features once deemed app-only—connectivity control, push notifications, home screen icons, and the like.

Before this initiative, some of the features in discussion were already available for mobile web users (although to a limited extend):

• Add to home¹ screen² (requires manual actions)
• Fullscreen mode³
• Application Cache for offline access⁴
• Notifications API⁵

Web pages, however, are still not the first choice, when it comes to delivering the best possible experience on a mobile device (despite being more discoverable in search engines and potentially saving the nuisance of downloading and installing megabytes, especially important for first-time visitors and visitors in 2G/3G connections). All too often do we see websites adding banners or interstitial popups⁶, begging users to download their apps, even going so far as to drop their mobile version completely⁷ (only to be resurrected⁸ 5 months later). The justifying arguments that recur: native apps run more smoothly and have better means to re-engage with customers, and the web environment simply lacks graceful fallbacks in flaky network conditions.

A Progressive Web App addresses all these issues except the rendering performance part. Building a Progressive Web App does not force you to drastically change your current front-end architecture or the way your work; it only gives you a set of tools to enhance the web experience progressively. At the end of the day, you'll be able to have:

• A home screen icon that opens the website in fullscreen
• Native dialogs to let users add your app to their home screens with one click
• A fast and always-usable site even in flaky network connections
• Push notifications just like native apps

Most of these features are made possible by service workers.

What is a Service Worker?

Service workers essentially act as proxy servers that sit between web applications, and between the browser and network (when available). They are intended to (amongst other things) enable the creation of effective offline experiences, intercepting network requests and taking appropriate action based on whether the network is available and updated assets reside on the server. They will also allow access to push notifications and background sync APIs. - MDN

In short, a service worker is an asynchronous background thread that takes control of all network requests in a page.

Quick Facts

• Service Workers run in a different context, thus have no access to DOM elements or JavaScript variables in the main thread
• For security reasons the client page (the main thread) must be in https and the service worker script must be in the same origin, but all requests originated from that page can be intercepted by service workers even if they are not in https or served from a different domain
• A CacheStorage is provided in the worker so that you can store server responses (including headers and response body) locally, and serve them to future requests.
• Server responses can be forged at the client side if necessary.
• Everything is asynchronous, and most APIs return a Promise

Browser support

For now, only Chrome, Firefox and Opera have adequate support for service workers. For mobile devices, that means only Android is supported. Since features like homescreen icons and push notifications are integrated in the OS, the whole Progressive Web App initiative really depends on how enthusiastic OS vendors are about it.

Regarding service workers, Apple's attitude is:

People think they want it, some of them actually do want it. We should probably do it.⁹

(it seems, then, we won't wait for too long before service workers are available in iPhones.)

For a detailed compatibility table of all features of service workers, check out this document: Is ServiceWorker ready?

What can Service Workers do?

The ServiceWorker API provides very granular methods for developers to intercept requests, to cache and forge responses, opening doors for all kinds of interesting activities like:

• Offline access to certain pages (an order confirmation, an e-ticket, etc)
• Precaching assets based on predictions of next user actions (predictions do not rely on service workers per se, but cache manageable can be more programmable with service workers. You can even introduce an expiration time or the LRU algorithm if you want)
• Serving a cached version when it takes too long to load some resources
• Rewriting URLs to always be requested with a canonical url¹⁰

Check the Offline Cookbook for more details about the caching strategies.

In addition, service workers are also used for arranging background communication with servers (think of it as a "service"). Features like push notifications, background sync, task scheduler all depends on service workers in some extend.

Service Workers in Action

Now, let's get our hands dirty and get to grips with the service worker in action.

Registration

Since service workers run in a different context, you'll need to put the code for the worker in a separate file, then register it in the client page:

if('serviceWorker'innavigator){navigator.serviceWorker.register('service-worker.js',{scope:'./'}).then(function(){if(navigator.serviceWorker.controller){console.log('The service worker is currently handling network operations.');}else{console.log('Failed to register.');}});}

This snippet registers a service worker with the file service-worker.js. Once registered, code in this file will be able to control all requests originated from any page within the scope parameter.

By default, the scope is the base location of the service worker script. For example, if you registered "/static/js/serviceworker.js", then the default scope would be "/static/js/". The script itself must be within the same origin as the client page, so it's not possible to serve service worker scripts with CDNs in different domains. But it is possible to override the scope to be outside of the script's base location:

navigator.serviceWorker.register('/scripts/service-worker.js',{scope:'/'})

This code enables the service worker to control all pages under the root path of the origin ({ scope: '/' }). But you'll need to add an extra response header Service-Worker-Allowed to make it work.

For instance, in an nginx configuration, it can be done like this:

Server{listenwww.example.com:443ssl;...location/scripts/service-worker.js{add_header'Service-Worker-Allowed''/';}}

(Note that this header is added for the service worker script itself, not the page it was registered to.)

Inside the worker

Once registered, a service worker will reside in the background intercepting all requests originated from its client pages and staying active until being unregistered.

The script runs in a context called ServiceWorkerGlobalScope¹¹. Several global variables and methods are available in this context:

• clients - Information about client pages, used to claim control over them
• registration - Represents the state of the registration
• cache - The CacheStorage object in which you can store server responses
• skipWaiting() - Allowing registration to process from waiting to active state
• fetch(..) - Part of the GlobalFetch API, also available in the main thread
• importScripts(..) - Import JS scripts synchronously, ideal for loading a service worker library

The Google Chrome team has provided a nice high-level library¹² to help you handle service worker tasks. It ships with a router for expressively applying common caching patterns to different resources, as well as a toolkit for precaching and namespaced cache management. It is highly recommended to use this library if you want to build something production-ready; it saves you a lot of work and is also a good start for you to get familiar with basic concepts in a ServiceWorker. Check out the recipes for example usages.

If you are really after the details, refer to MDN Service Worker API document and pay extra attention to CacheStorage¹³, and FetchEvent¹⁴.

Service Workers at Booking.com

At Booking.com, we are always open to new technologies, and encourage any innovation that improves customer satisfaction. We are currently working closely with the PWA advocate team from Google on applying some of the core features of Progressive Web Apps to our mobile website to see where it helps our customers.

Having service workers installed for users is relatively easy—you simply need them to be using a supported browser (currently this means using Chrome in Android). The real challenge, however, lies in how to introduce meaningful features while carefully measuring the impact. At Booking.com, we do every customer-facing project in A/B test experiments, and try to achieve things in the "smallest viable steps." The purpose is to ship the right things as fast as we can. Even for something as holistic as a Progressive Web App, we work in small steps in order to tackle issues one by one, and learn things quickly.

We have gathered some important learnings on this topic. What follows are some of the learnings we found which might be interesting to the general public.

Caching Strategy Examples

The Offline Cookbook¹⁵ summarized a few caching strategies for different use cases.

• cacheFirst - Serve cache if it exists, requests will still fire, and new responses will update the cache
• cacheOnly - Respond with the cache only, never fire the actual request
• networkFirst - Always try fetching from the network first and save the latest successful response into the cache, which will be served when the network fails
• networkOnly - Never uses local cache

Let's see some examples of how to apply each of them in real life.

For static files that never change, we can safely serve them with "cacheFirst":

toolbox.router.get(/static\/(css|js|images|img)\//,toolbox.cacheFirst,{cache:{name:'static-files'}});

They seldom change and even if they do, we would've updated the URLs. One might ask, what's the use of this technique if we already set the expiration date in the headers? A service worker gives you more granular control over how much cache you want to store and when to expire them. For instance, sw-toolbox provides very easy configurations for maxEntries¹⁶ and maxAgeSeconds¹⁷.

For ordinary HTML documents, we can use "networkFirst":

toolbox.router.get(/\/(confirmation|mybooking|myreservations)/i,toolbox.networkFirst,{networkTimeoutSeconds:10,cache:{name:'booking-confirm'}});

We configured the networkTimeoutSeconds parameter here. If it is acceptable to show this page to offline visitors, then it must be also acceptable to offer the cached version for users with very slow network connections and save them some waiting time. But of course, the length of the timeout seconds depends on your type of business and the common connectivity quality of your users.

For requests used for user behavior data collection, you might want to use "networkOnly":

toolbox.router.any(/www.google-analytics.com/,toolbox.networkOnly);

There's no point to return cache for a tracking request, right? If the request fails, it fails. If you want, you can even monitor the status of a tracking request, and resend it when it fails. This won't be possible if (somehow) the cache in service workers kicks in.

Local Shortcuts

Wouldn't it be nice if users can save a permanent link in bookmarks which will always redirect them to the last booking confirmation they saw?

Let's add a custom handler for the confirmation page:

toolbox.router.get("/confirmations/(.*)",function(request,values,options){varurl=request.url;varpromise=toolbox.networkFirst(request,values,options);varconfirmationId=values[0];if(confirmationId){// when the request finishespromise.then(function(response){if(!response||response.status!==200)return;self.caches.open('last-confirmation').then(function(cache){// save a 302 Redirect response to "/confirmation"varredirectResponse=newResponse('Redirecting',{status:302,statusText:'Found',headers:{Location:url}});cache.put('/confirmation',redirectResponse);});});}returnpromise;},{networkTimeoutSeconds:10,cache:{name:'confirmations',}});toolbox.router.get('/confirmation',toolbox.cacheOnly,{cache:{name:'last-confirmation'}});

Each time users visit a confirmation page, we will return the response as normal, with the strategy "networkFirst". But in addition to that, we forge a 302 redirect response locally, pointing to the current url, then save the fake response in a cache storage named last-confirmation with URL key /confirmation.

We've also added a rule in the router for this path and this cache storage, so that the next time users visit the URL "/confirmation", they will always be redirected to the last confirmation page they visited.

The forged response was put into a separate cache storage namespace, and is served with strategy cacheOnly. Because, apparently, the URL is only valid locally. We certainly don't want to mix it with normal requests.

The Secure Domain Problem

To protect users' data, all parts of our booking process and user account management pages are served via HTTPS, under a separate domain—"secure.booking.com", instead of "www.booking.com"—the one used for public content such as the Search Results and Hotel Details page.

You can't register one service worker across two different domains however, even if they are subdomains of the same root domain. And (for now at least) there's no way to let two service workers communicate with each other.

What if you want to pre-cache assets for secure.booking.com when users are still in www.booking.com, or the other way around? We have a lot of people jumping between two domains, especially when they are making a reservation. Also, with important functionalities spread across different domains, a service worker for one single domain simply cannot offer an uninterrupted offline experience.

Because of this, we are unifying all basic functionalities under one domain and this will give users full HTTPS access to their whole Booking.com journey. Meanwhile, experts of the Service Worker Specs group are working on a new API called "foreign fetch"¹⁵, which will give service workers authorities to intercept any requests for resources within their scopes (as defined when they were registered). These requests may be originated from any page, even if the page is under another domain.

Final Thoughts

The ServiceWorker API targets a long-standing problem for the mobile web—connectivity. It has the potential to make user experience bearable even when connectivity is bad. It empowers modern web apps with the ability to engage users in more intimate ways, and definitely increases web apps' competitiveness over native ones.

The vision of Progressive Web App is nice, but for a large-scale website steering at a very high speed, you can't implement everything and ship them in one go. Constantly experimenting, learning, and improving things with small steps, is the key to success.

Resources

1. Progressive Web Apps
2. Service Worker Spec
3. ServiceWorker API doc on MDN
4. Service Worker Debugging
5. Recipes 1
6. Recipes 2
7. Demos by W3C web mobile group

Development and navigation in a large codebase can prove quite the challenge. A common tool when tackling such a large codebase is an Integrated Development Environment (IDE). Unfortunately, the main programming language at Booking.com, Perl, has limited IDE support. This is the story of the joint effort with the open source community to address this need of IDE support for the Perl programming language.

The Search

In the past I had worked on large projects in a Java-based environment and learned the importance and the value of a good IDE supporting you in your daily tasks, while helping you navigate your way through the endless lines of code in your projects. When I first started to work at Booking.com (about 2 years ago), I was new to Perl and began searching for an IDE for Perl.

I started with Perl oriented IDEs and moved to plugins for Eclipse and IntelliJ, but these attempts came up empty.

Most of the solutions only provided syntax highlighting while others suggested basic text-based or token-based autocompletion. None of them provided full integration in which the IDE understands the relationships between variables, subroutines, and their arguments and packages. I also wanted support for patterns like inheritance.

For a while I resorted to working with Sublime Text but even Sublime Text was simply an enhanced text editor rather than a complete IDE.

_{Searching for a package is a breeze with the 'Navigate to Class' feature}

The Seeding Grounds for Innovation

Luckily at Booking.com we have hackathons (spanning across multiple days). They allow us to experiment with ideas and projects. They enable the business to explore various initiatives which might include adding new tools to making the workflow more effective, exploring new systems or tackling any of the issues for which you normally don't have time in your daily work. I took this opportunity to start working on a proper plugin to provide us with a full IDE for Perl.

_{With referencing: you can click on a package and open it's file or click on a subroutine call to go to it's declaration}

Starting from Scratch

While I encountered some discussions regarding the challenges of parsing perl or expressing it withBNF (a notation technique often used to describe syntax for programming languages), I knew that with enough resourcefulness the majority of use cases can be covered with regular expressions.

Since I was familiar with IntelliJ's capabilities and its plugin development workflow, I decided to develop a plugin that will integrate into the custom languages support, that will make up for the limited support it offers Perl right off the shelf.

One of the challenges when parsing a language (especially a dynamic one like Perl) is that it requires a lot of iterations. Every time you add one thing, it breaks two other things. However, after several hackathons and continuing to work on the plugin at home, I managed to cover most use cases.

I also added many important features like:

• Object Oriented autocompletion for packages (including inherited subroutines from parent packages)
• "Go to" implementation of a package by clicking on an icon in its row
• Searching by package name
• Syntax highlighting
• Creation of Perl project/module and SDK selection
• Support for running .pl files
• Subroutines autocompletion with arguments

_{Scalars, subroutines (and their arguments), packages and inheritance packages - all get autocompleted with the Perl intelliJ Plugin}

First Impressions

I showed the plugin around the company and several developers started using the plugin and contributing to the project. This led to a substantially increased development velocity for the plugin, which enabled adding more functionality to the project.

As time passed, I started to realise that without a full BNF there will always be some functionalities in IntelliJ that the plugin could never support (creating references between the various components of the language, syntax error checks, etc.).

I also knew that the more this project would grow, it would need more time and work than a single person could provide. Knowing Booking.com supports employees sharing and contributing to the open source community, I uploaded the project to GitHub and began to roam the IntelliJ plugin forums.

_{MySQL queries autocompletes databases, tables, and columns}

Working with the Open Source Community

Shortly after releasing the plugin, I was contacted by Alexandr Evstigneev, a talented developer from the open source community, who had already created a similar plugin.

Since Alexandr managed to express nearly the entire Perl language in BNF in his plugin and the Booking.com plugin already contained so many features, we knew we could benefit from merging our projects. I also saw this as a great opportunity to give back to the open source community and started migrating our code to his repository, feature by feature.

After a couple of months of integration, along with more people contributing from the open source community, we got IntelliJ to work as a fully pledged Perl IDE that understands Perl as a language. Referencing, Find by Usage, Going to subroutine declaration, Database Integration and autocompletion, Package Hierarchy - these (and more) are all part of the support this plugin provides from within IntelliJ.

While Alexandr continued to work heavily on the BNF structure and adding features with other contributors that joined the GitHub project, I presented the plugin at an AmsterdamX.pm Perl Mongers group meetup for English-speaking Perl developers in Amsterdam and The Netherlands. The presentation generated great feedback and excitement that motivated us to continue our work on the plugin. You can install the plugin directly through IntelliJ's plugin menu (look for Perl5 Support).

_{the GitHub project of the plugin}

Conclusion

Today, the project belongs to the community and is managed by Alexandr, the main contributor. The work on this project during our hackathons with the open source community has been fruitful and beneficial both for us at Booking.com and for the community as a whole.

Looking ahead, there are still more features to be added like Mason support, enabling the client debugging and server debugging. So if you're interested in the plugin, you're welcomed to join the development effort at GitHub!

Everyone who starts working at Booking.com gets the technology talk. “We don’t want to hold you back,” they say. “We don’t want to restrict your creativity. You’re free to use whatever software you need to get the job done — we’re so relaxed about your technology choices we’ll even let you choose between a MacBook and a Dell!”

So there you are, on your first day, sitting at your new desk and clutching a brand new MacBook (what? I prefer shiny things), trying to decide what to install first. Because it’s true — we don’t care what code editor you prefer, whether you’re an Outlook person or a Thunderbird person; Photoshop or Sketch, Powerpoint or Keynote, GitHub or the command line… We really don’t care. Productivity is the priority.

And for the first few weeks (maybe even the first few months), that’s great. In fact, it’s awesome. But when you’re working in a department of over 1,000 people with a technology stack that encompasses at least half-a-dozen different languages – not to mention interacting with a broader company over 10,000 strong – eventually you’re going to run up against a problem that can’t be solved with existing tools.

Where other large organizations might decide it’s time to start looking for outside vendors to plug a hole in the process, at Booking.com we live by a simple rule: See a problem, own a problem. When we find an issue, it’s time to get hacking.

A Few Small Things We Built In Our Spare Time

Like many other companies built around technology, Booking.com provides a space for developers and designers to tackle those annoying issues that don’t get attention in regular “Hackathon” sessions. For us, that means forming small teams of sympatico colleagues for a couple of days each month to scratch a communal itch – either to improve one of our existing systems or developing something that’s missing but would make everyone’s lives easier.

Some of the things we have come up with (either during hackathons or individually, because we encountered the same damn problem too many times) are relatively simple command line tools, shortcuts that let us reduce the amount of time we spend writing code to free up time so we can focus on what matters. A few examples:

Bash scripts

• Boo, created by client-side developer Ricardo Tomasi, is a Grunt-like set of tools designed to allow us to work on code locally while testing on remote servers. It handles automatic uploads of changed files, auto-runs concatenation scripts or server restarts, and can even tell us whether a particular commit has been deployed to the live site.
• Database engineer Simon Mudd created the queryprofiler and ps-top tools to make it easier for our DBAs to measure query performance, and to see in real time exactly what was going on in the depths of MySQL. Both have been released publicly.
• Ever looked at a git history and wished it would just tell you who the right people are to talk to about a file? Eric Herman’sowner.pl script does just that, identifying the core contributors of any file.
• mysqly is a wrapper around the MySQL command line prompt that automatically handles user authentication and connects you to the correct cluster of database machines.

Browser Extensions

But bash scripts, no matter how complex, just scrape the surface of what’s available to a motivated engineer determined to do no more work than they absolutely have to. Browser extensions are another great option when looking to shave precious seconds from repetitive or common tasks (that foosball isn’t going to play itself, after all).

Here are some of the browser extensions and other related bits and pieces we have built for ourselves:

• Running thousands of A/B tests across a multi-lingual website can often involve some delicate juggling of cookie variables and query parameters. Deepak Gulati'sSilver Hammer was the first in a series of Chrome extensions aimed at reducing the pain of development and testing by allowing us to easily toggle tests and languages on or off, as well as to jump quickly between our development, staging and live environments. Eventually it was superseded by B.quick by Marcelo Oliveira and Renato Costa, which in addition to the original features also hooked into our server-side debugging mechanism. This made the development and testing of new features much easier.
• For our Customer Service department, we built the Blue Phone Extension, a handy widget that not only allows our call centre staff to interact with it as if it is their physical phone, but also integrates with our backend job queue management system. (Look out for a longer write-up on this by client side developer Mat Swainson, coming to this blog in the near future.)
• Although not strictly an extension, the B.Home replacement ‘New Tab’ page for Google Chrome, by Wesley Souza contains a summary of information relevant to the user, plus useful links to common tools and monitors.
• Developed by Aziz Yuldoshev, Blueprint is an extension for Chrome devtools which adds the ability to toggle experiments on or off, as well as search through all available server-side variables and arrays.
• Anyone who has ever worked on a complex enterprise-level site will not be surprised to learn that we have many hundreds, possibly thousands, of different templates. It can often be tricky to figure out where a particular piece of the page was actually generated, so Developer Eli Abramovitch whipped up a lengthy but incredibly useful JavaScript bookmarklet that allows us to click on any part of the page to find the responsible template source.

Terminal goodies

Terminal one-liners and browser extensions can achieve some extremely cool results, but sometimes they’re just not enough. That’s when we break out the mini web apps and standalone services. Some of the useful things that have resulted include:

• Booker, an iOS app that integrates with our intranet and SAP database to let you find colleagues in the main Amsterdam offices. Blueprint maps of the building’s floors and desk layout let you pinpoint where to find the person you need, and you can also book meeting rooms from within the app.
• Lingo is a tool for managing copy translations. As the world’s largest translator of content, we needed a flexible tool that enabled our copywriters to effectively communicate with specialists in over forty different languages… so we built one!
• A room in a hotel might seem like a simple thing to model, but in reality there are many different moving parts — hotel policies, room policies, group sizes, child ages, facilities, prices and so on — so finding and testing every single edge case is a tough job. Give.me is our internal tool for quickly locating properties that fit a specific profile.
• Blinchik, Artem Tyurin and Aziz Yuldoshev, is a plugin for the Sublime text editor that groks our heavily modified template language, flagging up syntax and stylistic errors before they become a problem; developers Valerii Iatsko and Esteban Beltran also created a similar plugin for the Atom editor. (And speaking of Atom, developer Angelo Di Donato added HQL support to Atom too.)
• Recently we experimented with Slack as an internal communication tool within the Technology department, in addition to our existing Jabber platform. During the transition, we needed a way to communicate across platforms, so senior developer Manuel Ceron built Slabber, a Jabber-to-Slack posting tool, so that no critical communications were missed due to people using the wrong chatroom to share important gif-based communication.
• It’s not just web developers who need help. Android developer Aliaksandr Jeihala developed a plugin for Android Studio to enable our mobile developers to easily monitor the state of experimentation on that platform, integrating with our testing system to flag whether tests are currently running or need to be removed.
• Although we have a plethora of OSX and web-based helpers, Linux and Windows users aren’t completely left out in the cold. Thomas Shone created an Ubuntu tray app named Blerty to provide handy access to tools for those users, while Dan-el Khen built fsync for Windows.
• Since moving from CSS sprites to font icons, we discovered the need for a quick reference guide for our icon set. Designer Michel Ferreira whipped up a fancy searchable list, complete with light/dark colour scheme options and automatic copying of icon names to the clipboard.
• The exceedingly well-named “Where Is My Table?” tool, created by Jonathan Zhang, is a search engine for databases that allows us to search for tables based on their name, columns or schema.
• For new starters, configuring a new laptop is often an unnecessary time-suck. Ricardo Tomasi and Mauricio Wolff created Bookstrap, a Ruby-based mini-script that handles the download and installation of the most commonly used software and tools a new hire might need to get to work.

.dotfiles

Finally, a special mention has to be made of our “dotfiles” git repositories. These per-user repos contain our personal bash preferences, aliases, and any other bits and pieces we frequently use while working. The clever part? Whenever we login to any of our servers – even ones that we’ve never used before – our personal set of files is copied to the server. This provides us with an instant set of familiar scripts and shortcuts wherever we need to work. (Want to know more about this? Check out our previous blog post on the subject.)

Of course, this is not even close to a complete list. New tools and programs have been developed since I started writing this article, and I’m sure most coders at Booking.com have some sneaky secret shortcuts they’re keeping to themselves. So if your boss or company hasn’t yet seen the light and embraced the creative chaos of a regular hackathon, why not propose that you start?

(You might have noticed that we’re quite fond of the letter B when it comes to christening our creations, and that’s not even counting the numerous B.branded laptop stickers adorning every available surface, such as b.proud, b.awesome, b.epic, and so on. Coming up with a new word that adequately describes your creation and that also starts with a B can sometimes be the hardest part of the process. Hmm. There’s probably an app idea in there somewhere…)

When events sizes started to matter

In the beginning, four long years ago, simply having every data set in Hadoop was good enough. We were taking baby steps into the big data world and had a limited user base migrating from other sytems into the big data clusters, enabling them to execute heavy queries on large data sets in a timely manner. As the number of users and size of the data sets grew in size, we were challenged by new performance problems.

The majority of those data sets were all sorts of server-generated events stored in JSON — a format which has since become the norm for ease of use and development. We had gone the typical route followed by all NoSQL new starters in allowing our developers maximum flexibility and get rid of nearly all schemas (which had never really existed for these events).

To make this bunch of JSON objects easier to query and process, we used Hive¹, making a big partitioned table with only a few columns for UUID², datacenter IDs, timestamps and a few other things, plus a very fat column containing the whole JSON. Using Hive was an obvious choice for us as we already had many MySQL tables imported there for the analysts to use. Putting the events in the same pool allowed for some powerful scenarios, where you could easily join anything to anything. We could also use Perl and all the business logic we wrote in it directly in Hive thanks to the TRANSFORM construct ³.

This was already billions of events per day coming from a wide range of application types and servers, the bulk of it coming from the web and mobile front-ends. Traffic kept growing steadily and more event types were added. Individual records became fatter, too. Why limit yourself to a few metrics when you can store everything and keep it forever? Analysts and developers could, and would, scan ranges of several weeks, or even months, in a single Hive query — and everyone was pretty happy.

It worked quite well for a long time: Hive would hide the absurdly massive resources needed for querying petabytes behind a friendly face, and nobody (except the handful of people in charge of maintaining and expanding the whole thing) would know about it. But, due to this inflation combined with the rapidly growing number of users, queries soon started getting much slower.

Cool and simple ideas

We scratched our heads for a long time. How could we make it as efficient as it used to be, without imposing a fixed schema? How could we have the tons of scripts and queries that rely on these ugly blobs still work, but regain the efficiency they once had? All the papers that we read and the experience of the big players in the industry told us one thing: you definitely need a schema for readability by analysts (at the discovery phase) and for efficient querying.

Imposing this is also at odds with how things are done at Booking.com. We always go out of our way to make everyone's job easier, and not apply restrictions, even when it implies adopting creative solutions. Some of these may occasionally make your skin crawl, but as long as they do the job, that's what matters. The approach has served us well so far.

So we had this simple idea: instead of splitting the JSONs according to some schema kept in a registry, which would have been the natural thing to do, why not... do nothing? Keep the events as they always were, and live with it. They'd still be usable by most queries, just not the heaviest ones. And for these fat queries, we would make something brand new: a faster ORC⁴ table, with the JSONs split according to the most common use patterns that we see in actual queries.

This means a table whose structure would be driven by the actual needs of our users. It would mutate by itself over time to fit the evolution of these needs, avoiding breaking legacy scripts (by keeping the old table) while allowing new scripts (or those that would need adjustments for performance reasons) to use the new one. A self-mutating table, as silly as it sounds.

What our users were looking for

Getting a full list of all the queries that are run on a cluster wasn't too difficult. We have loads of monitoring scripts, some of which collect data about individual jobs and put them in databases with columns having information about:

user ID
user name
job name
number of maps and reducers
resources used in CPU and MB-seconds
the query (in the case of Hive jobs)
(...)

YARN archives most of this on job completion in one XML file per job, so that's easy enough⁵, and joining this with the extra information we needed wasn't difficult.

Once we had this covered, and true to our usual ways, we went for the simplest thing that could possibly work: figure out what was usually fetched inside the JSONs by extracting all the get_json* calls (and including a faster custom JSON UDF that we've developed) contained in the query strings. For this we used some Perl text processing modules (like Text::Balanced) that are part of the usual junk-processing arsenal. Plugging something in Hive's query parser would have been marginally neater, but the Perl-based solution got the job done in no time.

The extracted data sets gave us a long list of JSON keys in dot notation, like "foo.bar.baz", that would tell us what people were looking for in these events. We had some constraints in mind for the exercise: force some keys that we knew we'd need later to be counted as "seen" even though they were not present in the queries (= a whitelist) and prevent too much fragmentation (foo.bar.baz1 and foo.bar.baz2 should be seen as foo.bar and kept together as a "leaf" JSON fragment). We made a script that would do just that. It checks what the people needed, when they needed it and keeps that resulting list of keys with timestamps (telling when the corresponding query ran) in a database table.

We'd give them everything they wanted

We needed to run something continuously on the stream of real events to:

1. Figure out which of those keys would return actual results, because everyone makes mistakes in their queries from time to time and it's easy to misspell "bar" for "baz". If we only trusted the queries without checking what they returned, we'd soon find ourselves populating many columns with NULLs and making the schema bloated and unusable.
2. Figure out what data type the returned data could fit in once ingested in Hive, in order to specify the most efficient storage format for the destination columns at the splitting phase.
3. Create destination partitions with the proper columns and types, cleanup the schema when needed, and insert the split JSONs into the destination.

A handful of simple tricks

The script that we wrote does all of this in one pass.

1. Obtain the keys from step #1, append them to the key log table in MySQL and merge the set with the keys found during the previous runs.
2. Run a Hive query that splits the JSON blobs for the processed hour in tiny fragments, according to this list of keys. Then for every fragment of actual data, figure out what data type it would best fit into (about 8 types, from TINYINT to TEXT). This is done using a special UDF that does both the JSON splitting and the data type check in one go, outputting two columns for each fragment: the data, and it's guessed type. As should be obvious, the guessed data type for a specific key is dependent on what JSON object is being analyzed, so it's really important that this is done for all JSONs and all keys.
3. Out of the temporary table we just created in the previous step, extract the "fattest" data type that was found for every destination data column / JSON key. This will be the data type for our destination column. This also allows us to detect which keys are just mistakes, as they returned no data at all and their detected data type should be NULL. To figure this all out we run a simple auto-generated aggregate query on the datatype columns. Here's a picture that will hopefully make this clearer (click on the image to see a larger version):
4. Use this information to modify the destination table's schema, stored in ORC format, on the fly. One of the nice things about ORC is that partitions of a table don't have to all use the exact same schema: the column order has to stay the same, but the data types can vary. When creating the partition and inserting, the data type specified in the table definition is used, so what is basically text will be properly converted to numeric or other types and stored accordingly. But when reading, the partition definition (not the table) becomes authoritative. This means that in a single query hitting several partitions, the same column can provide different data types, but they will be properly cast at runtime. Which means you can have for instance INTs of various widths in the same column spread over several partitions, starting with TINYINT when you began gathering data, all the way up to BIGINT as your IDs increased. And it will just work.
5. Finally, populate the destination partition with the contents of the temporary table, minus the data type columns.
6. Update the table definition with per-column comments in JSON format, recording meta information like the first and last time some actual data was seen for the column. This allows us to sort-of nullify some columns that are not used anymore. Since columns cannot be dropped, we do it this way to reuse them for other keys once the time they've been empty is longer than the predefined time window. Efficient in terms of storage space and for keeping a slim schema.
7. Drop partitions older than the time window. Done.

Was it worth it?

This is the stage where you start questioning the sanity of this whole enterprise. After all, we found ourselves building a table that is mutating by itself on-demand (since it only uses information that users provide through their queries), and there ought to be a million ways this could fail horribly. You usually carefully craft ALTER TABLE DDL statements, not leave it to a script without direct supervision, right?

It turns out it seems to work quite well. Besides, this is "only" derived data, and we could go back to the source in the event of something gone wrong.

The script that orchestrates all this is only a handful of lines long, including loads of comments related to some of ORC's youth issues (i.e. bugs) that prevented us making it even niftier, and the logic is pretty straightforward.

Let's have a look at a query example; this is the type of scan on a partition that gets run routinely. Before the new table appeared, the query would look like this. It uses event_parser, a UDF we wrote to make processing of our JSONs easier and faster. The syntax is even uglier than that of a query using stock get_json_object(), but it runs quicker:

CREATETEMPORARYFUNCTIONevent_parserAS'com.booking.hive.udtf.EventParser';createtablemydb.sample_rawASSELECTevent.epoch,response.body_parameters,response.experiment,response.languageFROMdefault.raw_eventsaseventLATERALVIEWevent_parser(event.json,'http.body_parameters','action.experiment','action.language','action.name')responseASbody_parameters,experiment,language,action_nameWHEREyyyy_mm_dd='2016-04-18'ANDhh=21ANDis_frontend=1ANDaction_name='explorer';

Here's the syntax for running the same query on the new table, undoubtedly a whole lot better:

createtablemydb.sample_flatASSELECTepoch,`http/body_parameters`,`action/experiment`,`action/language`FROMdefault.events_flat_webWHEREyyyy_mm_dd='2016-04-18'ANDhh=21ANDis_frontend=1AND`action/name`='explorer';

In terms of storage, removing the lesser used data sets and switching to ORC⁴+ZLIB instead of RCFile⁶+Snappy brought the volume down by 60%. For the hourly partition we're querying here, it means going from nearly 500GB to less than 200GB. But this is big data, and size doesn't matter that much, as usual (unless you're the one buying the hard drives). What about performance, then?

Here is the CPU time for the old version (the result is a table of nearly 100 million rows):

1 days 1 hours 25 minutes 10 seconds 20 msec

And for the new version:

0 days 2 hours 41 minutes 12 seconds 420 msec

Which is 9,672 seconds vs 91,510, close to a 90% drop. What are we going to do with all these free CPU cycles on our hands?

We may want to keep it

Experimenting was fun, but the only thing that really matters is the benefits for our users and infrastructure. In short:

• Query runtimes were divided by about four
• CPU time were divided by up to ten
• Queries are easier to write and read, and more expressive

Which makes it a nice hack, but first and foremost a successful experiment.

The funny thing is, this was initially meant to solve a short-term problem for a few queries, hence the limited time window to save storage space. But it turns out our users liked the new table so much they didn't want us to drop any old partitions, and started using the new table as a full replacement for the old one.

As has happened many times before, what started its life as a quick hack (in search of a better solution) is going to be sticking around quite a bit longer.

Abstract: Building the best travel experience for our customers in Booking.com often involves solving very challenging problems. One that appears very frequently is the $k$-Nearest Neighbours problem ($k$-NN). In simple words it can be stated as follows: Given a thing, find the $k$ most similar things. Depending on how thing and similar are defined, the problem becomes more or less complex. In this post we’ll treat the case where the things are represented by vectors, and the similarity of two things is defined by their angle. We’ll discuss solutions and present a practical trick to make it fast, scalable, and simple. All of it, thanks to maths.

1. Things and Similarity

Suppose that we are given a database with pictures of handwritten digits, and the task of finding similar digit handwriting. Every picture contains exactly one digit, but we don't know which one. For every given picture we want to find other pictures that contain the same digit written with a similar style.

First, we need to find a representation for the pictures that we can use to operate. In this case it will be simply a vector, the length d of the vector is the number of pixels in the picture, and the components are the RGB values of the pixels. This representation has the advantage of working both at the computational level, but also at the mathematical level.

Second, we need to define similarity. Since the pictures are represented by vectors, we can compute the angle between any two of them; if the angle is small, the vectors point in the same direction and the pictures are similar. On the other hand, if the angle is big, the vectors diverge from each other and the pictures are not similar. More formally, the similarity between two pictures represented by vectors X and Y is given by:

\begin{equation} \label{eq:sim} sim(X, Y) = cos(X, Y) = \dfrac{\langle X, Y \rangle}{||X|| ||Y||} = \dfrac{\sum_i^d{X_i Y_i}}{\sqrt{\sum_i^d{X_i^2}}\sqrt{\sum_i^d{Y_i^2}}} \text{ (1)} \end{equation}

This quantity has a very nice property: It will never be more than 1, or less than -1. Given two vectors, if their similarity is close to 1 then they are very similar; if it is close to -1 they are completely different. 0 is in the middle – not very similar, but not completely different either.

Let’s see this in action:

Fig. 1: Two very similar 4s, their similarity according to equation 1 is 0.85

Fig. 2: Two very different 4s, their similarity according to equation 1 is 0.11

Figure 1 shows a graphical representation of the vectors computed from two handwritten digit pictures. These two digits are very similar, and when their similarity is computed by their angle using equation 1 it gives 0.85, which is, accordingly, quite high. On the other hand, Figure 2 shows two quite different numbers; this time their similarity is 0.11, which is quite low but still positive – even though the writing style is very different, both pictures are still a 4.

These pictures were handpicked to illustrate the vector representation and the cosine similarity. We now move on to finding an algorithm that finds similarly handwritten digits.

2. A simple solution

Now that we know how to represent things and compute their similarity, let’s solve the k-NN problem:

1. For every picture in our database, compute its associated vector.
2. When the $k$-Nearest Neighbours for a picture are requested, compute its similarity to every other picture in the database.
3. Sort the pictures by ascending similarity.
4. Return the last $k$ elements.

This is a very good solution (especially because it works). Figure 3 shows this algorithm in action. Every row shows the top 9 most similar pictures to the first picture in the row. The first line captures very rounded 3s, the second inclined 3’s, the fourth line shows 2’s with a loop on the bottom, and the 5th line shows Z like 2’s. Notice that this algorithm has no information about what digit is in the picture (nor, for that matter, anything about what kind of things the picture has), but it nevertheless succeeds to group by digit, and even by typographic style.

But let’s take a closer look by analysing its computational complexity. Consider n pictures with d pixels each:

1. Computing a feature vector is $O(d)$. As this is done for every picture, the first step is $O(nd)$.
2. The similarity function is $O(d)$, and again this happens $n$ times, then the second step is also $O(nd)$.
3. The third step – sort $n$ elements – is $O(nlogn)$
4. Finally returning the last $k$ elements could be constant, but let’s consider it $O(k)$.

In total we get:

\begin{equation} O(nd) + O(nd) + O(nlogn) + O(k) \text{ (2)} \end{equation}

$k$ is very small compared to $n$ and $d$, so we can neglect the last term. We can also collapse the two first terms into one single $O(nd)$. Now, note that $logn$ is much smaller than $d$, for example, if we have 10 million pictures with 256 ($d$) pixels each, $logn$ would be 7, much smaller than $d$. That means we can turn the $O(nlogn)$ into another $O(nd)$. Therefore the computational complexity of this algorithm is $O(nd)$, that is, linear in both total the number of images in our database and the the number of pixels per picture.

Fig 3: k nearest neighbours for some pictures. Every row depicts the top 9 most similar pictures to the first picture in the row

3. Can we do better?

If we do computational complexity analysis, it is natural to ask ourselves whether we can improve. So let’s try.

One idea to consider is to use a heap to keep track of the $k$ most similar items. The heap would never be larger than $k$ so every insertion involves $O(logk)$ similarity computations ($O(d)$), so an insertion is $O(dlogk)$. Since there will be $n$ insertions, in total we get $O(ndlogk)$, which is not an improvement.

We could also try to exploit the fact that we do not need to sort the $n$ elements, just to get the top $k$. The algorithm would be exactly the same, but step 3 would be replaced by applying quick-select instead of sorting. This would change the $O(nlogn)$ term to $O(n)$, that gives $O(nd) + O(n)$ which is $O(nd)$, again, not an improvement.

The last idea we will consider is to use a Space Partitioning Tree (SPT). An SPT is a data structure that allows us to find the closest object to another object in logarithmic time. A priori this seems to be the right solution but there is a problem: SPTs can only operate under certain distance functions, specifically metric distances.

SPTs work with distances, not with similarities. But there is a very close relationship between similarity and distance. In the context of $k$-NN, for every similarity function there exists a distance function such that searching the $k$ most similar items is equivalent to searching the $k$ closest items using that distance function. Just multiplying the similarity by −1 gives such a distance. So now we have a cosine distance that we could use in an SPT, but unluckily this cosine distance is not a metric distance.

A metric distance is a distance that complies the following conditions:

1. $distance(x, y) \geq 0$
2. $distance(x, y)=0 \iff x=y$
3. $distance(x, y)=distance(y, x)$
4. $distance(x, z)\leq distance(x, y)+ distance(y, z)$

Cosine distance clearly violates the first condition, but this is easy to fix by just adding 1. The second and the third conditions are met. Finally the fourth condition is violated and this time we cannot fix it. Here is an example of 3 vectors that violate the fourth condition:

$x=(1,0), y=(1,1), z=(0,1)$. Then:

$distance(x,y)=1-\dfrac{\langle x, y \rangle}{||x||||y||} = 1-\dfrac{1}{\sqrt{2}}$

$distance(y, z)=1-\dfrac{\langle y, z \rangle}{||y||||z||} = 1-\dfrac{1}{\sqrt{2}}$

$distance(x, z)=1-\dfrac{\langle x, z \rangle}{||x||||z||} = 1$

$distance(x,y)+distance(y, z)= 1-\dfrac{1}{\sqrt{2}} + 1-\dfrac{1}{\sqrt{2}} = 2-\dfrac{2}{\sqrt{2}} = 0.58$

And then:

$distance(x, z) \not \leq distance(x,y) + distance(y,z)$

which proves that condition 4 is not met.

In the following sections we are going to show a trick to overcome this limitation.

4. Maths

Let’s introduce some properties of vectors that we’ll exploit later.

Cosine distance is invariant under Normalization

First, let’s make a few definitions:

1. Norm of a vector $X$: Denoted by $\lVert X \rVert$ and computed as $\sqrt{\sum_i^d X_i^2}$. This is exactly the geometric length of the vector.
2. Normalization of vector: Denoted by $\hat X$ and computed as $\dfrac{X}{\lVert X \rVert}$ which is just dividing every component of the vector $X$ by the norm of $X$.

A consequence of these two definitions is the following:

\begin{equation} \lVert \hat X \rVert =1 \end{equation}

Which says that the norm of a normalized vector is always 1. This property is quite obvious, but here is a proof:

$\strut \lVert\hat X \rVert^2=\strut\sum_i^d \hat{X_i}^2 ={\sum_i^d {\dfrac{X_i^2}{\lVert X \rVert^2}}} = {\dfrac{1}{{\lVert X \rVert}^2}\sum_i^ d X_i^2 } =\dfrac{1}{\lVert X \rVert}\lVert X \rVert = 1 $

And since $\strut \lVert\hat X \rVert$ must be positive we conclude that $\lVert \hat X \rVert =1$.

Another consequence is the following:

\begin{equation} \label{eq:invariant} cos(\hat X, \hat Y) = cos(X, Y) \text{ (4)} \end{equation}

In words, this means that the angle between two vectors doesn’t change when the vectors are normalized. Normalization only changes the length (the norm of the vector), not its direction, and therefore the angle is always kept. Again, here is the proof:

$cos(\hat X, \hat Y) = \dfrac{\langle \hat X, \hat Y \rangle}{\lVert \hat X \rVert \lVert \hat Y \rVert} = \langle \hat X, \hat Y \rangle = \langle \dfrac{X}{\lVert X \rVert}, \dfrac{Y}{\lVert Y \rVert} \rangle = \sum_i^d \dfrac{X_i}{\lVert X \rVert}\dfrac{Y_i}{\lVert Y \rVert} = \dfrac{\sum_i^d X_i Y_i}{\lVert X \rVert \lVert Y \rVert} = \dfrac{\langle X, Y \rangle}{\lVert X \rVert \lVert Y \rVert} = cos(X, Y)$

Now since $cos(\hat X, \hat Y) = cos(X, Y)$ then also $1-cos(\hat X, \hat Y) = 1-cos(X, Y)$, which means that the cosine distance between $X$ and $Y$ is exactly the same as the cosine distance between the normalized version of $X$ and $Y$.

From Euclidean to Cosine

The second property we need for this trick is the following:

If $X$ and $Y$ are vectors with norm 1 (unit vectors) then: \begin{equation} \label{eq:euclideaniscosine} \lVert X-Y \rVert = \sqrt{2-2cos(X,Y)} \text{ (5)} \end{equation}

This states that if $X$ and $Y$ are unit vectors then there is an exact relationship between the euclidean distance from $X$ to $Y$ and the angle between them.

The proof:

\begin{equation} {\lVert X-Y \rVert}^2 = X^2 - 2\langle X, Y \rangle + Y^2 \text{ (6)} \end{equation}

Since $X$ is a unit vector then $X^2 = \langle X,X \rangle = \sum_i^d X_i X_i = {\lVert X \rVert}^2 = 1$, likewise, $Y^2=1$ then we have: \begin{equation} {\lVert X-Y \rVert}^2 = X^2 - 2\langle X, Y \rangle + Y^2 = 2-2\langle X, Y \rangle \text{ (7)} \end{equation}

And since $X$ and $Y$ are unit vectors, dividing by their norm is dividing by one:

\begin{equation} {\lVert X-Y \rVert}^2 = X^2 - 2\langle X, Y \rangle + Y^2 = 2-2\langle X, Y \rangle = 2-2\dfrac{\langle X, Y \rangle}{\lVert X \rVert \lVert Y \rVert} = 2-2cos(X,Y) \text{ (8)} \end{equation}

Cosine ranking is equivalent to Euclidean ranking

By looking at equation 5 we can already see that if $1-cos(X,Y)$ is bigger, then $\lVert X-Y \rVert$ must be bigger. That means that if $Y$ gets away from $X$ in the euclidean space, it also does in the cosine space, provided both $X$ and $Y$ are unit vectors. This allows us to establish the following:

Consider three arbitrary $d$-dimensional vectors $X$, $A$ and $B$. (they don't need to be unit vectors). Then the following holds: \begin{equation} \textit{if } 1-cos(X,A)<1-cos(X,B) \textit{ then } \lVert \hat X- \hat A \rVert < \lVert \hat X- \hat B \rVert \text{ (9)} \end{equation}

This equation says that if the cosine distance between $X$ and $A$ is less than the cosine distance between $X$ and $B$ then the euclidean distance between $X$ and $A$ is also less than the euclidean distance between $X$ and $B$. In other words, if $A$ is closer to $X$ than $B$ in the cosine space, it is also closer in the euclidean space.

The proof: We start from the left hand side expression and apply operations to get to the right hand side expression.

\begin{equation} 1-cos(X,A)<1-cos(X,B) \implies 1-cos(\hat X,\hat A)<1-cos(\hat X,\hat B) \text{ (10)} \end{equation} cosine is invariant under normalization (see equation 4)

\begin{equation} 1-1cos(\hat X,\hat A)<1-1cos(\hat X,\hat B) \implies \sqrt{2-2cos(\hat X,\hat A)} < \sqrt{2-2cos(\hat X,\hat B)} \text{ (11)} \end{equation} doubling and taking squared root keeps the inequality

\begin{equation} \sqrt{2-2cos(\hat X,\hat A)} < \sqrt{2-2cos(\hat X,\hat B)} \implies \lVert \hat X- \hat A \rVert < \lVert \hat X- \hat B \rVert \text{ (12)} \end{equation} normalized vectors are unit vectors and equation 5

This is all the maths we need to apply the trick. Let’s see what is it about.

The $k$-NN Trick

The goal of this trick is to find a way to be able to use cosine similarity with a Space Partitioning Tree, that would give us $O(log n)$ time complexity, which is a huge improvement.

The idea is actually very simple: Since cosine similarity is invariant under normalization, we can just normalize all our feature vectors and the $k$-nearest neighbours to X will be exactly the same; but now our vectors are all unit vectors, which means that sorting them by cosine distance to X is exactly the same as sorting them by Euclidean distance to X, and since Euclidean distance is a proper metric we can use a Space Partitioning Tree and enjoy the logarithm of n. Here’s the recipe:

1. Normalize all the feature vectors in the database
2. Build a Space Partitioning Tree using the normalized vectors
3. When the $k$ nearest neighbours to an input vector $X$ are requested:
   1. Normalize $X$
   2. look up the $k$-NN from the Space Partitioning Tree

6. Experiments

Experimentation is at the core of Product Development at Booking.com. Every idea is welcomed, turned into a hypothesis and validated through experimentation. And Data Science doesn’t escape that process.

In this case, the idea has been thoroughly described and supported with practical examples and even maths. But let’s see if reality agrees with our understanding. Our hypothesis is the following: We can improve the response time of the algorithm described in section 2 by applying the trick described in section 5 guaranteeing exactly the same results.

To test this hypothesis we designed an experiment that compares the time needed to solve the $k$-NN problem using the full scan solution with the time needed by the $k$-NN trick solution. The $k$-NN trick is implemented using two different Space Partitioning Trees: Ball Tree, and KD-Tree.

The database consists of handwritten digits pictures from MNIST. For n ranging from 5000 to 40000 randomly sampled n pictures from the original database; then applied the different solutions to the same sample, computing the 10 most similar pictures for 20 input pictures.

7. Results

The results of our experiment are summarized by Figure 4:

Fig 4: Comparison of the full scan solution (brute force) and the $k$-NN trick (norm euclidean ball tree, and kd tree) for different database sizes n

From the chart we can make several conclusions: First, the time complexity of the full scan solution is indeed linear in n as suggested by the blue dots. This confirms the theoretical analysis in section 2. Second, although it is hard to say if the $k$-NN trick based solution is logarithmic, it is clearly much better than the full scan, as suggested by the green and red dots. Third, the Ball Tree based solution is better than KD-Tree solution, though the reason for this fact is not clear and requires further analysis and experimentation. Overall, the experiment strongly supports the hypothesis.

8. The Trap

Every trick sets up a trap, and every gain in one aspect hides a loss in another. Being aware of these traps is key to successfully apply these tricks. Let’s see what trap the $k$-NN trick sets, or, in more technical words, what kind of trade-off are we dealing with?

In the simple solution, before being able to answer a $k$-NN query all we need to do is to compute the feature vectors of each object in the database. On the other hand, when using the trick, before we are able to answer a query we not only need to compute the feature vectors, but also we need to build the Space Partitioning Tree. In the experiment we run, we also recorded the time it takes to be able to answer queries. The results are displayed in Figure 5 and show that the trick-based solutions scale much worse than the simple solution. This means that when using the trick we are trading off query response time with start-up time.

This trade-off must be taken carefully, and for big databases this can have very negative consequences. Consider an e-commerce website that goes down for whatever reason; imagine that this e-commerce uses $k$-NN to serve some recommendations, (a very important yet not critical part of the system). As soon as we fix the problem, we want the system to reboot as soon as possible, but if the booting process depends on the $k$-NN system we fall into the trap – users won’t be able to purchase anything until our Space Partitioning Tree is built. Not good.

This can be easily solved by breaking the dependence using parallel or asynchronous processes to boot different parts of the system, but the simple solution is clearly more robust in this instance, up to a point where we don’t even need to care. The $k$-NN trick forces us to consider this very carefully and act properly. For many applications, this isn’t a bad price to pay for the speed and scalability we get at query time.

9. Conclusion

In this post we described a trick to speed up the standard solution for the $k$-NN problem with cosine similarity. The mathematical rationale for the trick was presented, as well as experiments that prove its validity. We consider this as a good example of a scalability problem overcome by applying elementary maths. This is also a good example of Reductionism: The trick is a reduction from cosine similarity $k$-NN problem to a Euclidean distance $k$-NN problem which is a much more studied and solved problem. Maths and Reductionism are two concepts sitting at the core of applied Data Science at Booking.com, always at the service of the best travelling experience.

Fig 5: Ready-time comparison of the full scan solution and the $k$-NN trick for different database sizes n

This is the annex to Evaluating MySQL Parallel Replication Part 4: More Benchmarks in Production.

There is no introduction or conclusion to this post, only landing sections: reading this post without its context will probably be very hard. You should start with the main post and come back here for more details.

Implementation Details of MariaDB Optimistic Parallel Replication

Rollbacks and Retries

When transactions T1 to T20 are run concurrently by optimistic parallel replication, if T19 blocks T2, T19 will be killed (rolled-back) for unblocking T2 (T2 must commit before T19). In the current implementation, T19 will be retried once T18 completed. It looks like this could be optimized.

I thought that retrying T19 as early as T2 completes could improve optimistic replication speed. Kristian Nielsen, the implementer of parallel replication in MariaDB, was kind enough to implement a patch with more aggressive retries. However, with quicker retries, I got slower results than with delayed retries. So it looks like once a conflict is detected (T19 blocks T2), the probability of another conflict is high, and the gain in retrying T19 earlier is outweighed by the cost of other rollbacks of T19.

DML vs DDL and Non-Transactional Storage Engines

The assumption for optimistic parallel replication to work is that a transaction that causes a conflict can be killed and retried. This is the case for InnoDB DML (Data Manipulation Language: INSERT, UPDATE, DELETE, ...) but it is not the case with MyISAM.

As a transaction involving a MyISAM table (or another non-transactional storage engine) cannot be rolled-back, it is not safe to run those transactions optimistically. When such transaction enters the optimistic parallel replication pipeline, the replication applier will wait for all previous transactions to complete before starting the transaction that cannot be rolled-back. The following transactions could still be run optimistically if they are exclusively using a transactional storage engine (if they can be rolled-back). This means that DMLs that cannot be rolled-back act as a pre-barrier in the parallel replication pipeline.

In MariaDB, DDL (Data Definition Language: [CREATE | ALTER | TRUNCATE | DROP | ...] TABLE, ...) are also (still) impossible to rollback. So they will also act as a pre-barrier in the parallel replication pipeline. Moreover, DDL are also preventing all next transactions to be optimistically applied because a DML is not safe to run at the same time as a DDL on the same table. So, not only DDLs act as a pre-barrier, but they are also acting as a post-barrier.

Different Optimistic Parallel Replication Modes

MariaDB 10.1 optimistic parallel replication can be run in two slave_parallel_mode: optimistic and aggressive. In the optimistic mode, some heuristics are used to avoid needless conflicts. In the aggressive mode, those heuristics are disabled.

One of the heuristics of the optimistic mode is the following: if a transaction executed a row-lock wait on the master, it will not be run in parallel on the slave. The behavior is unclear when intermediate masters are used:

• An intermediate master with slave_parallel_mode=none (single threaded) will not have any row-lock wait. So it looks like for a slave of such intermediate master, the optimistic mode would behave the same way as the aggressive mode.
• An intermediate master with slave_parallel_mode=minimal (slave group committing) will have a row-lock wait for each group commit. So it looks like for a slave of such intermediate master, the optimistic mode would behave the same as the conservative mode.
• An intermediate master with slave_parallel_mode=conservative should generate very few row-lock wait (only for conflicts that will generate retries). So it looks like for a slave of such intermediate master, the optimistic mode will behave mostly the same as the aggressive mode.
• The number of row-lock waits is hard to predict on an intermediate master in optimistic or aggressive mode. So the behavior of a slave of such intermediate master is hard to predict.

As we are doing tests on a slave of an intermediate master, the optimistic mode is not very interesting to test. It would generate results similar to the aggressive mode if the intermediate master was running in single-threaded or conservative mode, or similar to the conservative mode if the intermediate master was running in minimal mode. Without a true master running MariaDB 10.1, the only tests that we think make sense are with slave_parallel_mode=aggressive.

This is a good opportunity to remind that intermediate masters are bad for parallel replication. As shown in a Part 1, intermediate master are doing a poor job at transmitting parallelism information from their master to their slaves. The solution presented in the previous post still applies: use Binlog Servers.

Environments

As in the previous posts (Part 1, Part 2 and Part 3), we are using the same four environments. Each environment is composed of five servers. For slave_parallel_mode=none and slave_parallel_mode=conservative, only four of the five servers are needed and are organized as below:

+---+     +---+     +---+     +---+
| A | --> | B | --> | C | --> | D |
+---+     +---+     +---+     +---+

The A to C servers are strictly the same as before. The D server has the same hardware specification as before but it is now running MariaDB 10.1.8 [1]. This means that the conservative results will use the same parallelism information (group commit) as for the tests from Part 3 (we are re-using the same binary logs as the previous tests).

For optimistic parallel replication to work, a MariaDB 10.1 slave must be connected to a MariaDB 10.1 master [2], hence the introduction of a fifth (E) server. For slave_parallel_mode=aggressive, D is replicating from E as shown below:

+---+     +---+     +---+     +---+     +---+
| A | --> | B | --> | C | --> | E | --> | D |
+---+     +---+     +---+     +---+     +---+

The hardware specifications of E are not important because it is only serving binary logs. It was built as a clone of D that was upgraded to MariaDB 10.1. Replication was then started from C with slave_parallel_mode=none. This way, we produced 10.1 binary logs so slave_parallel_mode=aggressive will work on D.

The full test methodology is the same as for the previous tests and can be found in Part 3. The server and database configurations are mostly the same as in the previous tests with the following modifications:

The motivations for the above changes are the following:

• The InnoDB Buffer Pool Size was reduced for E3 and E4 because we were missing RAM to increase slave_parallel_threads to the number we wanted to test (more threads need more available RAM).
• The InnoDB Log Size was increased because checkpointing was a bottleneck during our tests [3].

Results

In the main post, speedup graphs are presented for each of the four environments. Here, the underlying data for those graphs is presented.

The SB, HD and ND notations are explained in the main post.

The first line of the table below shows the time taken for the single-threaded execution with slave_parallel_mode (SPM) set to none. Then, for slave_parallel_threads (SPT) values of 5, 10, 20 and 40, we have results with both non-optimistic (slave_parallel_mode=conservative) and optimistic (slave_parallel_mode=aggressive) executions. Then, for slave_parallel_threads values of 80, 160, 320, 640, 1280, 2560 and 5120, we have results only for optimistic executions. Note that we cannot have meaningful results for non-optimistic runs with slave_parallel_threads greater than 40 because the maximum group size on C was 35 (see Part 3 for more details).

The times presented below are in the format hours:minutes.seconds and they represent the delay needed to process 24-hours of transactions. The number in bold is the speedup achieved from the single-threaded run.

Graphs during Tests

If you spot something we might have missed in the graphs below, please post a comment. Those graphs include the number of commits per second, CPU stats, Read IOPS and percentage of Retried Transaction for all tests.

Graphs # 1a: E1 Stats - Slave with Binary Logs - High Durability

Graphs # 1b: E1 Stats - Slave with Binary Logs - Relaxed Durability

Graphs # 2a: E2 Stats - Slave with Binary Logs - High Durability

Graphs # 2b: E2 Stats - Slave with Binary Logs - Relaxed Durability

Graphs # 3a: E3 Stats - Slave with Binary Logs - High Durability

Graphs # 4a: E4 Stats - Slave with Binary Logs - High Durability

[1] At the time of the publication of this post, the latest release of MariaDB 10.1 is 10.1.17. Our tests were done with MariaDB 10.1.8 because they were run a long time ago (I am a little embarrassed to be that late in my blog post editing).

[2] In the implementation of optimistic parallel replication in MariaDB 10.1, the master is responsible for flagging DDL and non-transactional DML and to pass this information to slaves via the binary logs. This is why a MariaDB 10.1 master is needed to enable optimistic parallel replication on a slave. This also means that for optimistic parallel replication to work, master and slaves must have compatible storage engines for DML: if a DML is transactional on the master, it must be transactional on the slave. So a master using InnoDB and a slave using MyISAM will not work.

[3] Because the InnoDB Log Size was too small in our previous tests, those tests were run in non-optimal conditions. The results presented in this post should be considered more accurate.

Parallel replication is a highly-expected feature of MySQL available in MariaDB 10.0 and in MySQL 5.7. We already presented benchmark results with MariaDB 10.0 in the previous post of the series; in this post, we will look at a new type of replication introduced in MariaDB 10.1: optimistic parallel replication.

This post has an annex: Under the Hood. Benchmarking is a complex art and reporting results accurately is even harder. If all the details were put in a single article, it would make a very long post. The links to the annex should satisfy readers eager for more details.

Optimistic parallel replication is built on top of the previous implementation of parallel replication. To fully understand optimistic parallel replication in MariaDB 10.1, we must dive back in the implementation details of parallel replication in MariaDB 10.0.

Key Design Decision: In-Order Commit

In Part 1 of the series, we explained how parallel replication works in MariaDB 10.0 and in early version of MySQL 5.7. In short, both MySQL 5.7 and MariaDB 10.0 identify parallelism on the master and send this information to the slaves via the binary logs [1]. However, MariaDB and MySQL differ in the way transactions are committed on slaves.

In MariaDB, transactions run in parallel on slaves are committed in the same order as they appear in the binary logs of the master [2]. So if a slave runs T1 and T2 in parallel and those transactions appear in this order in the binary logs of the master, the thread running T2 will wait for T1 to complete before committing T2 (even if T2 is ready to commit before T1).

By default, a MySQL slave running transactions in parallel is simply committing transactions as they complete without enforcing any ordering. In the above example, T2 will commit before T1. This could (and will most of the time) generate different transactions ordering in the binary logs of the slaves [3][4].

When committing in-order, a transaction that needs to commit first (T1) can be blocked by another transaction that will commit later (T2). It is surprising that such dependencies are generated by transactions committing at the same time on the master, but this can still happen. This is a deadlock situation: T1 is blocked by T2 and T2 is waiting for T1.

The storage engine will not detect this deadlock because both dependencies are not under its control. The first dependency is known to the storage engine: T2 is holding a resource needed by T1. However, the second dependency is in the server, more precisely in the parallel replication code: T2 cannot commit before T1. To resolve this situation, the in-order parallel replication applier must detect that T2 is blocking T1 and it must kill T2. Once killed, T2 will release its resources, allowing T1 to proceed. T2 will then be retried.

This transaction killing and retrying does not happen very often in MariaDB 10.0, but its implementation is essential to avoid blocking replication. The occurrence of those retries can be monitored with the slave_retried_transactions global status. Below is a graph from such monitoring where we can see that three retries were needed for a four-minute interval. This is especially small considering that ~2.5K transactions were run per seconds on this slave (three retries for ~600,000 transactions).

Graph # 0: Retried Transactions on a MariaDB 10.0 Slave

Once this deadlock detection and resolution is implemented, the following becomes very straightforward:

Run all transactions in parallel, with in-order commit making sure that the data is consistent, and with deadlock detection avoiding replication to block.

This is called optimistic parallel replication and is implemented in MariaDB 10.1. This type of parallel replication does not rely on the master to identify parallelism. The slave tries to run as many transactions in parallel as possible to the limit of the slave_parallel_threads parameter while enforcing in-order commit. If no conflict happens, maximum speed is achieved. If a conflict (deadlock) is detected, a transaction that was optimistically run will be rolled-back to unblock in-order commit (this transaction will be retried later).

As long as there are not too many conflicts, or as long as the cost of rollbacks does not outweigh the benefit of running more transactions in parallel, optimistic parallel replication should give good results. What is yet unknown is how those results will compare to the conservative parallel replication (conservative is the MariaDB 10.1 name for the parallel replication in MariaDB 10.0). To answer that question, tests need to be done. The results are presented below.

Of course, the full implementation details are more complex (more details can be found in the annex about rollback and retries, DML vs DDL and non-transactional storage engines and the different optimistic parallel replication modes). However, this introduction is enough to understand the rest of this post, so let us go directly to benchmarking.

The Test

The test is the same as in the previous post: catching up with 24 hours of transactions. The four test environments are also very similar, their description can be found in the annex. In the graphs below, we compare speedups for non-optimistic executions (slave_parallel_mode=conservative) with speedups for optimistic execution (slave_parallel_mode=aggressive). We had to choose aggressive over optimistic because the latter will not give good results replicating via an intermediate master (more details can be found in the annex).

In the graphs, the Y axis shows the speedups with a value of one being the reference time without using parallel replication (slave_parallel_mode=none). The X axis shows the number of threads used: notice the logarithmic scale. The curve stopping at 40 on the X axis is showing the non-optimistic results. The data used to plot those graphs can be found in the annex.

Some reminders from the previous post:

• Parallelism identification (slave group commit) was done with a maximum group size of 35, so increasing the number of threads past 40 for non-optimistic tests is not interesting.
• The binary logging configuration had a noticeable impact on performance. The catch-up times are generally longer when enabling log-slave-updates, and disabling binary logging is not such a big win. The results below are obtained with binary logging enabled but with log-slave-updates disabled, referred to as SB in Part 3.
• The workload of the four environments are different: E2 is a CPU-bound workload, E1 is also mostly CPU-bound but with some cache misses, E3 is a mixed CPU and IO workload, and E4 is an IO-bound workload.
• As E4 is an IO-bound workload (mostly cache misses in the InnoDB buffer pool), reducing durability - referred to as ND in Part 3 - does not get a noticeable improvement (similar behavior has been observed for E3). For that reason, the results presented below only include high durability for E3 and E4 (referred to as HD).

Graph # 1a: E1 SB-HD - Conservative vs Aggressive Speedups
(CPU-bound workload with some cache misses)

Graph # 1b: E1 SB-ND - Conservative vs Aggressive Speedups
(CPU-bound workload with some cache misses)

Graph # 2a: E2 SB-HD - Conservative vs Aggressive Speedups
(CPU-bound workload)

Graph # 2b: E2 SB-ND - Conservative vs Aggressive Speedups
(CPU-bound workload)

Graph # 3a: E3 SB-HD - Conservative vs Aggressive Speedups
(Mixed CPU and IO workload)

Graph # 4a: E4 SB-HD - Conservative vs Aggressive Speedups
(IO-bound workload)

Discussion

The first surprise comes from observing speedups with very high number of threads. Even though the servers used for the tests only have 12 hyper-threaded cores for a total of 24 threads, speedups are still increasing past 80 threads for all configurations and up to 2560 threads for E3. Obviously, we cannot use more processing power when raising slave_parallel_threads from 40 to 80, so those growing speedups cannot be explained simply by using more CPU.

We think that those speedups are caused by replication prefetching. By increasing the number of threads, transactions that need data that is not in cache will trigger a read from disk earlier (prefetching). Triggering that read earlier is a big win and does not consume much CPU because the thread will go in IOWait state. Even if that transaction causes a conflict and is rolled-back, the data will be in cache for the retry. Also in this case, the cost of the rollback is insignificant compared to the gain of having the data in cache for the retry, so extra conflicts are not a problem. This concept is not new; it is known as replication prefetching and has already been discussed before by Baron Schwartz, Domas Mituzas and Yoshinori Matsunobu.

For all environments, aggressive parallel replication can produce much better speedups than conservative parallel replication. It is also very nice to see more significant speedups with low durability on E1 and E2 (conservative parallel replication is not giving good speedups there). It is absolutely obvious that aggressive parallel replication is a great improvement over conservative parallel replication.

I suspect (and I hope) that there are still bottlenecks to be removed. Optimistic parallel replication is a great improvement over conservative parallel replication, but we are still far from the speedup of 6 to 10 that we are looking for (maybe those expectations are unrealistic...).

Something that might slow down our tests are DDL (Data Definition Language: [CREATE | ALTER | TRUNCATE | DROP | ...] TABLE, ...) because DDL instructions are blocking the replication pipeline. As explained in detail in the annex, before being able to run a DDL, all previously started transactions need to commit. Moreover, before starting any transactions after a DDL, the DDL must complete. Below are some extracts from SHOW GLOBAL STATUS during the test:

• E1: 197,499 Com_create_table
• E1: 1148 Com_truncate
• E1: 489 Com_drop_table
• E1: 484 Com_rename_table

So in the quickest run of E1 (7060 seconds), we were doing in average 27 CREATE TABLE per second [5]. That obviously cannot be good for parallel replication. To ease identifying such problems, I opened MDEV-10664 - Add statuses about optimistic parallel replication stalls.

Conclusion

As stated in Part 3 (but is still worth repeating): it is possible to test MariaDB parallel replication even if the master is an old version of MySQL/MariaDB. In our case, our masters were running MySQL 5.6, but the same could be applied to other versions.

Overall, the optimistic parallel replication shows very promising results: almost all speedups are better than conservative parallel replication.

The biggest surprise was to see speedup increasing past 80 threads. We could have thought that more threads than processing units would slow things down, but it is not the case. This is probably caused by threads being most of the time in a waiting state: either waiting for a previous transaction to commit, or waiting for an IO. For a pure CPU workload, we can expect contention and this is probably what explains the thrashing for E2.

Another surprise is that the best speedup (3.78) is achieved for an IO bound workload (E4). In this case, the biggest win of parallel replication seems to be getting more read IOPS from the disks subsystem by scheduling many IOs in parallel. In this environment (and probably in others), optimistic parallel replication with a high number of threads is acting as a replication prefetcher.

Up to now, all our tests were done using magnetic disks. It is unclear how both conservative and optimistic parallel replication would behave with solid state disks (SSDs). More tests are needed to understand how parallel replication will behave with SSDs.

If you are interested in this topic and would like to learn more, I will be giving a talk about MySQL and MariaDB Parallel Replication at Percona Live Amsterdam in October. All talks by Booking.com are:

• Data Diversity at Booking.com
• Datastore axes: Choosing the scalability direction you need
• MySQL/MariaDB Parallel Replication: inventory, use-cases and limitations
• MySQL Time Machine by replicating into HBase
• Using multisource replication in MySQL 5.7 for resharding
• Splitting a Database Without Down-time

We will also be hosting the Percona Live Community Dinner on October 4th. You can also meet us there if you want to know more about the cool things we do at Booking.com.

[1] MySQL 5.7 and MariaDB 10.0 have slightly different implementation of parallelism identification on the master. MariaDB 10.0 uses the binary log group commit optimization as accurately described in Part 1 (the group commit id is shown in MariaDB mysqlbinlog output as cid). From version 5.7.6, MySQL is tagging each transaction with two logical timestamps (last_committed and sequence_number in MySQL mysqlbinlog output).

[2] MariaDB also has an out-of-order parallel replication functionality based on its GTID implementation. This type of parallel replication might not commit transactions in the same order as they appear in the binary logs of the master. To take advantage of out-of-order parallel replication, hints must be given by the application to advertise what can be run in parallel. This is not the type of parallel replication we are discussing in this post (we are focusing on the in-order type).

[3] By allowing out-of-order transaction commit on slaves, MySQL can alleviate a problem previously discussed in Part 3 where too small a value for slave_parallel_threads could give suboptimal speedups.

[4] The slave-preserve-commit-order option allows enabling in-order commit in MySQL but this option needs log-slave-updates. I opened Bug#75396 to have this restriction removed as needing log-slave-updates to enable slave-preserve-commit-order looks like a step backward:

• The goal of parallel replication is to get faster replication
• According to most of our tests, log-slave-updates slows down replication

[5] This number of CREATE TABLE is very high. It looks like the pattern CREATE TABLE IF NOT EXISTS is very common for E1. Checking for table existence and avoiding the CREATE TABLE when it already exists might be a big win for parallel replication in E1. And for the people that do not believe 27 CREATE TABLE per second is possible (I also had to check it by myself):

$ for i in $(seq 4978 5003); do
  mysqlbinlog $(printf "binlog.%06d" $i) |
    grep -i "CREATE TABLE IF NOT EXISTS"; done | wc -l
211753

One of our most valued principles at Booking.com is putting the customer at the center of everything we do. It’s engrained into our day-to-day lives, and we’re always wanting to design experiences that our customers love. And this drive towards always creating user-centered design is made much easier by working within a fast-paced experimentation environment. At Booking.com, we have many ways of getting our customers involved directly — daily user feedback reports, usability tests, user research highlights, and even visiting our customers in their homes to see how our products fit into their daily routine. These kinds of sources are our main idea repository when we brainstorm new features or improve existing ones. And as designers, we use this kind of input to correct design decisions, work towards solving important UX issues and improve our products.

Why do we find it so important to listen?

Real people interact with our products. Real people are on the other side of those screens. Understanding what real customers need is the key to improving our product; the customer is king and keeping your customer happy can have a significant impact on your business.

92% of consumers believe recommendations from friends and family over all forms of advertising ~ source: Nielsen.com

64% of marketing executives indicated that they believe word of mouth is the most effective form of marketing. Only 6% say they master it ~ source: forbes.com

Think about the last time you had a disappointing experience with a product. How many of your friends knew about it within just a day?

We want customers to always be able to talk to us. We aim to reduce the friction our customers are facing in their efforts to find a place to stay and feedback from our users is a crucial source to find these points of friction. Avoiding friction means reducing bouncing, avoiding drops in conversion, frustration and reduction in word of mouth recommendations.

5 ways we listen to our customers

At Booking.com, we use a range of methods on a daily basis, all with the aim of improving user experience:

1. Online Survey Tools
2. Street level user testing
3. Usability tests
4. User research
5. Diary studies

1. Online Survey Tools — concept feedback

Every year, more companies than ever offer online survey tools. They’ve become smarter, better at targeting the right audience and it’s now easier to reach more people with lower implementation costs. It’s an effective way to get feedback at scale from the first day you launch. You can focus on specific parts of the customer journey and run it without leaving the office.

At Booking.com we use tools like Usabilla.com, SurveyGizmo to collect feedback from for our app, email marketing campaigns and web products. Those tools can provide high quality feedback (especially for small website improvements), customer journey friction points and interactions that could be improved. Here’s an example of some high quality feedback:

You can’t always expect the feedback to be constructive and usable to improve your products.

Some of the most popular online tools can be found here.

Site exit surveys (Net promoter score)

Site exit tools let you collect feedback from potential customers who left your product. Customers might need more time than we’d expect to make up their mind, and even if they don’t intend to come back to our site their feedback can provide great insights. Some tools available to try out are Asknicely, Capterra, GetFeedback, 4Screens.

2. Street level user testing

Street user testing is the simplest and least expensive way to get user feedback. You can leave your office anytime with a prototype in your hands and reach out to your potential customers. Inarguable it can be pretty intimidating as you need to approach people and interrupt them from their daily routine, introduce yourself and additionally ask for their feedback. It’s up to you if you can offer some kind of perks when you get valuable insights. From personal experience, in the beginning it was hard to engage the participants to provide feedback. A common mistake was not setting the right expectations meaning if they will need to book a property and if they need to use their own devices. Then with trial and error we adjusted our approach and used the following greeting message:

Hello! We’re from Booking.com and we’d love 10–15' of your time to get your feedback about our website. We’ll provide you with a phone and you don’t have to really book an actual hotel — we just want to see how you use our site and how we can help improve it.

People might easily provide quick less meaningful feedback as a result of wanting to be nice to you or because they probably have more important things to do.

Street user testing is a method we love at Booking.com because no preparation is needed, we have a great park (Rembrandtplein) next to our offices that tourists visit every day and the ideas for improvement have been great so far.

3. Usability tests

Usability tests can provide fast and high quality feedback on your product. The customers follow their natural behavior while the session is being recorded and guided by a moderator.

The success of usability tests depends on many factors, chief of which are the experience of the moderator and the quality of the participants. The most common approach is to outsource the selection of participants to a local recruiting company. Participants are invited into a friendly environment because making them feel as comfortable as possible will promote natural behavior.

If a company doesn’t have an in-house usability lab then this can be a significant additional cost to organize a usability test.

Usability lab is a very powerful tool that we all have access to at Booking.com. During every usability test we work with our researchers to help us extract the best quality feedback possible. A moderation training is essential for everyone that wants to moderate their own sessions. The training is necessary to avoid common mistakes like asking leading or closed questions although extensive experience is needed in order to master the art of moderation.

User testing through remote moderation tools

UserTesting.com allows you to connect with your customers remotely, assign them tasks and observe the results through a recorded video. It’s a great tool to conduct a usability test without having to invest on creating or renting a usability lab. The cost of each video is around $45 at UserTesting.com.

We also use this tool at Booking.com, especially when we want to approach customers in isolated parts of the world where it’s hard to have access to a usability lab.

Neuro-usability testing

At Booking.com, we’re always interested in trying out new, innovative methods of user-testing, and discovering how we can better understand our customers is a never-ending process. Neuro-usability is a way of measuring the Anxiety, Frustration and Excitement of the customer when they interact with our products, and, since this technology is available, we gave it a try:

We have future plans to experiment with other forms of neuro-usability tests monitoring the levels of sweating and skin contractions that can be interpreted into meaningful emotional responses. Scientific and rock solid methodologies to understand user behavior will arise as technology evolves and we want to be on top of it.

4. User research

User research produces excellent quality feedback and in-depth insights. It gives you the opportunity to observe customers using the product for extended periods.

At Booking.com we do more than 120 research studies every year across the planet. Each user research focuses on specific personas depending on our business needs and growth strategies. For example, you can’t perform the same research for family travellers and business travellers. As the needs of different types of users are so different, your conclusions often can’t be easily consolidated. It’s incredible to see how differently the customers in China and India are interacting on the same product. Cultural differences change the perception, understanding, and expectations of the product.

If you are an observer in a user research session, it is very important to focus on the bigger picture of the product. It is easy to draw false conclusions when observing a user just because your observation is aligned with your initial intentions to refine the product. By observing a behaviour you might believe that your idea is validated by the user.

Every user research session comes with a detailed report accessible to everyone in the company. The report that we mostly use as designers at Booking.com is the Research Highlights.

5. Diary Studies

Our research team at Booking.com performs diary studies as well. Diary studies are a form of a long-term research with the same participants. The participants write entries with their activities, their thoughts and frustrations they face along their journeys.

The results of the diary studies are exposed in a prominent place in our headquarters to provide inspirational ideas for the relevant frontend teams.

The exposure of the results is a great example of feeling closer and listening to our customers. As the issues become personified they become more engaging.

Wrap up

All the suggested tools and methods are effective and work. It’s up to you to start with the most affordable and least time consuming method if you need some proof that it can help you create a better product.

A free trial is usually available to test drive the tools available in the industry. Find the most effective for your product or service and give it a go. A mix of several tools would be optimal but at least some kind of feedback tool should be living in your website or app.

Ideas that come from real customers when using your product are already validated and are more likely to give you successful results. Start listening to them.

In the end, it comes down to this simple question: How can you optimize the user experience of your products if you don’t know what’s wrong in the first place?

Some lessons learned from jumping into the deep end.

One of the core principles around which our way of working is based is the concept of the small, multi-disciplinary team. At Booking.com there is no design team, there is no dev team, there is no copy team. We’d rather refer to groups of people in the same field as communities — they scale so much better this way. While some of these communities may number in the hundreds a team is usually around six people working together on a specific problem. Six people with a good mix of skills: designers, developers, usually one product owner, sometimes a copywriter or a data scientist — whatever the scope of the team requires.

In my four years here I’ve been part of 7 teams; choosing a new team is always one of my favourite and yet most challenging moments. It’s not easy leaving behind a bunch of awesome people, even if sometimes you’re just moving a couple desks down. It’s not easy to stop working on problems you’ve been trying to solve for a while and passing them on to someone else. It’s not easy leaving what is already a comfort zone to work on something you may know little about. But, because it’s a great opportunity to learn something new, I almost never pass the chance.

There’s always a learning curve when changing teams. And, to me, that’s essentially the point of doing so, to put fresh eyes on a fresh challenge, bring new perspectives to the team and learn something in the process. If it’s easy — if it’s not a little terrifying — you’re probably doing it wrong.

It’s a quick, informal meeting. There are no chairs in the room, just a tall, round, yellow table standing on three legs. From the other side of the table comes a challenge: Do I want to join the email marketing team? My first thought is that I know absolutely nothing about email marketing and that in fifteen years as a designer I haven’t designed a single email. Sure, it’s a little terrifying but I see that more as a reason to say yes than to say no.

In about twenty minutes the weight of the challenge I’ve just accepted starts to become more and more visible. It’s fine. I’ve done this before, I know how to keep calm in this situation. Take a deep breath and repeat after me: It can’t be that hard. It’s a process. Steps. I can figure out the steps.

What I usually do about now is take a good look at the product. New eyes can spot a lot of things that might be hard to see if you know the context and story behind them too well. I do this first, even before talking to people on the team. In this case I’ve gathered all the email campaigns we send out. I make notes, usually in the form of questions, trying to look at them from the perspective of the user, trying to challenge everything that’s there and identify what might be missing or could be even better.

Kind of like this.

This usually works out very well and gives me a lot of ideas to test and validate (to deliver the best possible customer experience we test everything we do at Booking.com). Sometimes it’s things the team already tried, sometimes my ideas are stupid or impossible. Most of the times I’m wrong. But I can promise there’s a lot of value in a fresh, critical look.

This time though I also tried something else. Remember I knew nothing about email marketing? To fill in a little of that gap I’ve decided to also make note of everything I liked — everything that worked, everything that, as a user, made me happy and told me what I needed to know. When your goal is to improve something it’s easy to focus on the negative — after all you don’t need to improve what’s already great. But when your goal is to learn, looking at what works makes a lot more sense.

I ended up with almost two hundred positive notes. A lot more than I would have thought looking at the long list of things that could be improved. Looking through the tags I made, some patterns emerged pretty quickly. Before I knew it I had six design principles for designing email. I’d like to say I came up with these but they were pretty much already there — they just needed to be put into words. Looking at them now they have value beyond just email marketing (but I’d need 2000 more words to get into that). 
 Anyway, here they are:

1. Relevant

Relevant is hard. Relevant is hard enough , relevant NOW is even harder. But context is important since people started taking email with them everywhere. And I mean everywhere.

But “is this relevant” is not a yes or no question. Relevance is a spectrum that goes all the way from: “I literally couldn’t care less” to “spot-on amazing.” That feeling you get when Google or Facebook know you a little too well.

This is an example of a relevant email. Relevant because summer means I need sunblock and I need sunglasses.

A relevant email

In the next example, because of some extra insight, the email is even more relevant. I have a sunny vacation coming up so I really need these things. I can’t imagine how how they would possibly know this, unless the same company sells flight tickets, sun screen and sunglasses. So maybe not a realistic scenario but it should make my point.

A really relevant email

2. Personal

Speaks to me and fits my needs. I don’t think I need to sell you on this, personal emails work so much better than mass emails. You can look for opportunities to trigger emails based on user behavior or add personalized elements or recommendations in mass campaigns. Feel free to throw in a first name in there every once in a while, it’s a nice touch, but don’t assume that is enough to make the email personal.

A personal email.

This is an example of a mail being personal. This company knows my bike has not been serviced in a year, they know exactly what bike I have, and they remind me to get it checked. Because they know all this they can give me the exact price, I don’t have to figure out myself what price matches my bike type. Throw my name in there and sign it with a person’s name, preferably the guy who will service the bike for me, and it becomes almost impossible to tell this is an automated marketing email, not a personal one.

3. Engaging

Engaging is about making me want to take action but also allowing me to take action. If it triggers my interest but there’s no action to take, I will have an unsatisfying experience. Sometimes I get recommendations for amazing products and then find out they don’t ship them to Europe.

Don’t do this to me! I know you know where I live.

An engaging email

This is an example of engaging elements. I’m reminded of a goal I set, I can see all the progress I made and how close I am to the next milestone. It will be hard to resist taking action and going for that level 3.

4. Clarity

This one I think is clear (see what I did there?). Some examples: I’m subscribed to a lot of email marketing emails and the ones that are infrequent should include a way of reminding me who they are and why I subscribed. Or if I get a recommendation, let’s say for a book, then include a reason why I will like that book.

A clear email.

In this example here I’ve replaced the actual name of the product with what it is for. To me, it seems clear that in one of these scenarios I am more likely to understand why that product is in my email. In the example on the left I’m likely to ignore this recommendation because the product name is so obscure. Stating that clearly, the way a human would speak to me if I walked in a store to buy this, makes the version on the right clear.

5. Control

Control is a tough one. One example from Booking.com is language. We send email in 43 languages. I see a lot of companies offering content in multiple languages and selecting a language for you based on your country. A person’s location hasn’t been an accurate way to determine what language they speak in…ever. We give subscribers control over this, even if it is not explicit. You change the language on the website, the language in your email will also change. No extra steps, it just works.

Giving user control — content preferences.

In this example you can let users control how much email they get, or what content they are interested in. This works great when you have a lot of different content. Your unsubscribe page is a good place to offer this as an alternative to completely opting out. This gets double points for making your emails a lot more relevant to this subscriber and for keeping them around.

6. Inspiration

One way to look at this is by asking what value does this email have to someone who cannot take action on it? What value it has to someone who cannot buy what you are selling or cannot directly benefit from what you are offering? If you take away the chance of a transaction is there any value or meaning left? It’s a good way to keep people opening and reading your emails.

This value can come from beautiful storytelling. It can come from giving some useful information without asking for something in return. It can come from offering a brief moment of delight. Charity Water sends out some great stories. Lonely Planet prioritizes great content over selling travel guides. Kickstarter puts lovely, if random, quotes in the beginning of all their emails.

And if all else fails a kitten GIF will do the trick.

An inspiring email.

In this example, the story, the inspiring content takes the front seat and the selling can happen on the landing page. If they feel you’re not always trying to sell something, subscribers will look forward to your next email.

To sum up — I want to confidently say this about any email we send out: This email is relevant, personal and engaging because it offers clarity, control and inspiration.

And now I’ve got a framework of guiding principles. Yay me!

It took just a couple of days to come up with these principles and just a couple of weeks to refine them. That was a little too easy so I never fully trusted them. I’ve spent the next year putting them to test after test, waiting for the data, the numbers, to tell me I am wrong. To my surprise, I can tell you that didn’t happen.

What did happen is that I’ve found a new challenge, and a new team, and that I’m back working on web products. But if you want to design email, in 43 languages, for tens of millions of subscribers in over 200 countries and one of the largest digital ecommerce companies in the world you can take on the challenge! Don’t worry if email is not your thing we’re also looking for UX designers and mobile app designers.

So, what did I learn from creating my own design principles?

• If you’re going to have design principles, define them early to make sure everyone is on the same page.
• It’s important to get buy-in from everyone involved. Principles will not help much if you are the only one using them.
• Validate your principles through research and experimentation. That will make the above point easier.
• Be flexible, but don’t keep changing or breaking the rules.
• Be positive. If you try to define what your design should not be you will end up with a very long list.

Without defining design principles first, it might seem like there are a million directions in which you can take a product, and a million different things you can try. But be honest with yourself: however much you’d like to, you can’t try every one of those things. You need some direction. It may seem counter–intuitive but having these principles written down is important because of the restrictions they set. These restrictions bring clarity, focus and help you make better design decisions.

In 2003 the Perl development community was made aware of an algorithmic complexity attack on the Perl's hash table implementation[1]. This attack was similar to reports over the last few years of attacks on other languages and packages, such asthe Java, Ruby and Python hash implementations.

The basic idea of this attack is to precompute a set of keys which would hash to the same value, and thus the same storage bucket. These keys would then be fed (as a batch) to a target which would then have to compare each key against each previously stored key before inserting the new key, effectively turning the hash into a linked list, and changing the performance profile for inserting each item from O(1) (amortized) to O(N). This means that the practice of loading arguments such as GET/POST parameters into hashes provided a vector for denial of service attacks on many HTTP-based applications.

As a response to this, Perl implemented a mechanism by which it would detect long chains of entries within a bucket and trigger a "hash split". This meant it would double the number of buckets and then redistribute the previously stored keys as required into the newly added buckets. If, after this hash split, the chain was still unacceptably long, Perl would cause the hash to go into a special mode (REHASH mode) where it uses a per-process random hash seed for its hash function. Switching a normal hash to this special mode would cause Perl to allocate a new bucket array, recalculate all of the previously stored keys using the random seed and redistribute the keys from the old bucket array into the new one. This mitigated the attack by remapping the previously colliding keys into a well distributed set of randomly chosen new buckets.

At this point the Perl community thought we had put the subject of hash collision attacks behind us, and for nearly 10 years we heard little more on the subject.

Memory Exhaustion Attack On REHASH Mechanism

Over the years occasionally the subject of changing our hash function would come up. For instance Jarkko made a number of comments that there were faster hash functions and in response I did a bit of research into the subject, but little came of this work.

In 2012 this changed. I was working on several projects that made heavy use of Perl's hash function, and I decided to invest some efforts to see if other hash functions would provide performance improvements. At the same time other people in the Perl community were becoming interested, partly due to my work and partly due to the publicity from themulti-collision attacks on Python's and Ruby's hash functions (MurmurHash and CityHash). Publicity I actually did not notice until after I had pushed patches to switch Perl to use MurmurHash as its default hash, something that got reverted real fast.

In restructuring the Perl hash implementation so it was easier to test different hash functions, I became well acquainted with the finer details of the implementation of the REHASH mechanism. Frankly it got in the way and I wanted to remove it outright. While arguing about whether it could be replaced with a conceptually simpler mechanism I discovered that the defenses put in place in 2003 were not as strong as had been previously believed. In fact they provided a whole new and, arguably, more dangerous attack vector than the original attack they were meant to mitigate. This resulted in the perl5 security team announcingCVE-2013-1667, and the release of security patches for all major Perls versions since 5.8.x.

The problem was that the REHASH mechanism allowed an attacker to create a set of keys which would cause Perl to repeatedly double the size of the hash table, but never trigger the use of the randomized hash seed. With relatively few keys the attacker could make Perl allocate a bucket array with up to 2^32 hash buckets, or as many as memory would allow. Even if the attack did not consume all the memory on the box there would be serious performance consequences as Perl remapped the keys into ever increasing bucket arrays. Even on fast 2013 hardware, counting from 0 to 2^32 takes a while!

This issue affected all versions of Perl from 5.8.2 to 5.16.2. It does not affect Perl 5.18. For those interested the security patches for these versions are as follows:

    maint-5.8:  2674b61957c26a4924831d5110afa454ae7ae5a6
    maint-5.10: f14269908e5f8b4cab4b55643d7dd9de577e7918
    maint-5.12: 9d83adcdf9ab3c1ac7d54d76f3944e57278f0e70
    maint-5.14: d59e31fc729d8a39a774f03bc6bc457029a7aef2
    maint-5.16: 6e79fe5714a72b1ef86dc890ff60746cdd19f854

At this time most Perl installations should be security-patched. Additionally official Perl maintenance releases 5.16.3, and 5.14.4 were published. But if you would like to know if you are vulnerable you can try the following program:

    perl -le'@h{qw(a h k r ad ao as ax ay bs ck cm cz ej fz hm ia ih is
      iz jk kx lg lv lw nj oj pr ql rk sk td tz vy yc yw zj zu aad acp
      acq adm ajy alu apb apx asa asm atf axi ayl bbq bcs bdp bhs bml)}
      =(); print %h=~/128/ && "not "," ok # perl $]"'

The following are statistics generated by the time program for the full attack (not the one-liner above) against a Perl 5.16 with and without the fix applied (identical/zero lines omitted) on a laptop with 8GB:

Without the fix patch (0ff9bbd11bcf0c048e5b3e4b893c52206692eed2):

         User time (seconds): 62.02
         System time (seconds): 1.57
         Percent of CPU this job got: 99%
         Elapsed (wall clock) time (h:mm:ss or m:ss): 1:04.01
         Maximum resident set size (kbytes): 8404752
         Minor (reclaiming a frame) page faults: 1049666
         Involuntary context switches: 8946

With the fix patch (f1220d61455253b170e81427c9d0357831ca0fac) applied:

         User time (seconds): 0.05
         System time (seconds): 0.00
         Percent of CPU this job got: 56%
         Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.09
         Maximum resident set size (kbytes): 16912
         Minor (reclaiming a frame) page faults: 1110
         Involuntary context switches: 3209

But this doesn't explain all of the changes in Perl 5.18

The observant reader will have realized that if we could patch older Perls to be robust to CVE-2013-1667 that we could also have patched 5.18 and avoided the problems that were caused by changing Perl's hash function. The reason we went even further than those maintenance patches is that we found out that Perl had further, if less readily exploitable, vulnerabilities to attack, and we wanted to do our best to fix them all.

This part of the story starts with Ruslan Zakirov posting a report to the perl5-security mailing list. The report outlined the basis of a key discovery attack on Perl's hash function. At first the Perl security team was not entirely convinced, but he then followed up with more code that demonstrated that his attack was sound. This development meant that the choice of a random seed would not make Perl's hash function robust to attack. An attacker could relatively efficiently determine the seed, and then use that knowledge to construct a set of attack keys that could be used to attack the hash function.

Nicholas Clark then ramped things up a notch further and did some in-depth analysis on the attack and the issues involved. At the same time so did Ruslan and myself. The conclusion of this analysis was that the attack exploited multiple vulnerabilities in how Perl's hash data structure worked and that the response would similarly require a multi-pronged approach.

Changes to the One-At-A-Time function

The first vulnerability was that Bob Jenkins' One-At-A-Time hash, which Perl used, does not "mix" the seed together with the hashed data well enough for short keys. This allows an attacker to mount a key discovery attack by using small sets of short keys and the order they were stored in to probe the "seed" and eventually expose enough bits of the seed that a final collision attack could be mounted.

We addressed this issue by making Perl append a four digit, randomly chosen suffix to every string it hashed. This means that we always "mix" the seed at least 4 times, and we mix it with something that the attacker cannot know. This effectively doubles the number of bits used for "secret" state, and ensures that short keys do not "leak" information about the original seed. The reason we use a suffix is that adding a prefix is the same as starting with a different initial seed state, so does not add any extra security. A suffix modifies the final state after the user input is provided and increases the search space an attacker must consider.

Related to this change was that the original One-At-A-Time function was potentially vulnerable to multi-collision attacks. An attacker could precalculate one or more suffixes such that

H(x) == H( concat(x, suffix) )

which would then allow an attacker to trivially construct an infinite set of keys which would always collide into the same bucket. We hardened the hash by mixing in the length of the key into the seed. We believe that this more or less eliminates the possibility of a multi-collision attack as it means that the seed used to calculate H( concat(x, suffix) ) would not be the same seed as H( concat(x, suffix, suffix) ). Cryptographers are invited to prove us wrong.

Reduce Information Leakage

The second vulnerability was that it is all too easy to leak information about the hash function to an attacker. For instance a web page might accept a set of parameters and respond with information for each of those parameters in the natural key order for the hash. This might provide enough information to mount a key discovery attack.

In order to prevent this information leakage we randomize the element order returned by the keys() and each() functions on the hash. We do this by adding a mask to each hash, and when an insert into the hash occurs we modify the mask in a pseudo-random way. During traversal we iterate from 0 to the k-th bucket and then XOR the iteration value with the mask. The result is that every time a new key is added to the hash the order of keys will change more or less completely. This means that the "natural" key order of the hash exposes almost no useful data to an attacker. Seeing one key in front of another does not tell you anything about which bucket the key was stored in. A nice side effect of this is that we can use this mask to detect people inserting into a hash during each() traversal, which generally indicates a problem in their code and can produce quite surprising results and be very difficult to debug. In Perl 5.18 when this happens we warn the developer about it.

A third vulnerability is related to the case where two keys are to be stored in the same bucket. In this case the order of the keys was predictable: the most recently added key would be "first" out during a keys() or each() traversal. This in itself allows a small amount of data to leak to a potential adversary. By identifying such a case one could find two (or more) strings which had the same least significant bits. By stuffing more keys into the hash and triggering a hash split an attacker could determine that the newly added bit of the hash value was different, or the same, for the two keys. Without the key-order randomization logic mentioned previously the attacker could also determine which of the two had a 1 or 0 in the most significant bit of the used part of the hash value.

While we were not yet able to construct an actual attack based on this information we decided to harden against it anyway. This is done by randomly choosing whether we should insert the colliding key at the top of a bucket chain or if we should insert at the second from top in the chain. Similarly during a hash split we also make such a decision when keys collide while being remapped into the new buckets. The end result is that the order of two keys colliding in a bucket is more or less random, although the order of more than two keys is not.

People complain. Randomization is good anyway!

We introduced randomization into Perl's hash function in order to harden it against attack. But we have discovered that this had other positive consequences that we did not foresee.

The first of these initially appeared to be a downside. Perl's hash function behaved consistently for more than a decade. Over that time Perl developers inadvertently created dependencies on this key order. Most of the examples of this were found in test files of CPAN modules: Many of us got lazy and "froze" a key order into the test code. For example by embedding the output of aData::Dumpercall into one's tests. Some of these however were real bugs in reputable and well tested modules.

By making Perl's key order random these dependencies on key order became instantly visible and a number of bugs that probably manifested themselves as "heisenbugs" became regular occurrences, and much easier to track down and identify. I estimate that for every two "non-bugs" (in things like test code) that we found, there was one "real bug" that was identified as well. Considering one of these was in the Perl core, and the other was in DBI, I personally consider this to be a good result.

Many people object that randomization like this makes debugging harder. The premise is that it becomes difficult to recreate a bug and thus debug it. I believe that in practice it is the opposite. Randomization like this means that a formerly rare bug becomes common. Which in turn means it becomes much more obvious that it is related to subtle dependencies on key order. Effectively making it much easier to find such problems.

A last benefit of randomizing the hash function is that we can now, at any time, change or replace the hash function that Perl is built with. In fact 5.18 is bundled with multiple hash functions including the presumed cryptographically strong Siphash. Since hash order from now on will be random, we don't have to worry if we change the function. External code should already be robust to the hash order being unpredictable.

How tangible are attacks on Perl's hash function?

There has been a lot of discussion on this subject. Obviously erring on the side of caution on security matters is the right course of action. Nevertheless there is a lot of debate on how practical an attack like this is in real production environments where Perl is commonly used, such as web servers. Here are some of the points to keep in mind:

1. Perls hash algorithm uses an array of buckets whose size is always between the number of keys stored in it and a factor of two larger. This means that a hash with 20 keys in it will generally have 32 buckets, a 32 keys hash will be split into 64 buckets, and so on. This means the more keys are inserted in a hash the less likely further keys will be put into the same bucket. So attacking a hash cannot make non-attack keys slower[2]. An attacker basically only slows their own fetches down and except as a by-product of resource consumption they will not affect other requests.
2. For an attack to reach DOS proportions the number of items inserted into the hash would have to be very, very large. On modern CPUs a linked list of thousands to hundreds of thousands of keys would be necessary before there was serious degradation of service. At this point even if the attack was unsuccessful in terms of degrading Perl's hash algorithm, it would still function effectively as a data-flooding denial of service attack. Therefore, focusing on the hash complexity aspect of the attack seems unwarranted.
3. Rudimentary out-of-band measures are sufficient to mitigate an attack. Hard restrictions on the number keys that may be accepted by publicly facing processes are sufficient to prevent an attack from causing any damage. For instance, Apache defaults to restricting the number of parameters it accepts to 512, which effectively hardens it against this type of attack. (This is one reason the attack on the rehash mechanism is so important, a "successful" attack requires relatively few keys.) Similarly, a well designed application would validate the parameters it receives and not put them in the hash unless they were recognized.
4. So long as the hash function chosen is not vulnerable to multi-collision attacks then simple per-process hash seed randomization makes the job of finding an attack keys set prohibitively difficult. One must first perform a hash-seed discovery attack, then generate a large set of keys. If restrictions on the number of keys the process will accept are in place then the keys must be very large before collisions would have a noticeable effect. This also makes the job of finding colliding keys all the more expensive.
5. In many circumstances, such as a web-service provider, the hosts will be behind load balancers. Which will either mean that every different web host uses different hash seeds, making hash seed discovery attacks very difficult. Or it requires an attacker to open a very long-running, persistent session with the server they wish to attack. This should be easily preventable via normal monitoring procedures.

For all these reasons it appears that hash-complexity attacks in the context of Perl and web hosting environments are of limited interest so long as:

1. The hash function does not allow multi-collision attacks.
2. The hash function uses at least a per-process hash seed randomization
3. The interface to untrusted potential attackers uses simple, hard limits on the number of keys it will accept.

These properties are relatively easy to accomplish without resorting to cryptographically strong hash functions (which are generally slow), or other complicated measures to prevent attacks. As the case of Perl's rehashing flaw has shown, the cure may be worse than the disease. The code that this CVE exploits for its attack was added to Perl as part of our attempt to defend against the hypothetical attack vector of excessive bucket collisions. We are at this time unaware of any real attack of this form.

Hash Functions For Dynamic Languages

It seems like the bulk of research on hash functions has focused on finding fast, cryptographically secure hash functions for long strings containing binary data. However, dynamic languages, like Perl, make heavy use of hash functions on small strings, which are often restricted to simple alphanumeric characters. Examples are email addresses, identifiers like variables and method names, single character keys, and lists of numeric ids and similar use cases. Not only are they relatively short strings, but they use restricted sets of bits in their bytes.

So far it appears that Bob Jenkins' One-At-A-Time-Hash with minor modifications provides an acceptable solution. It seems to have good distribution properties, is reasonably fast for short strings, and - with the hardening measures added in Perl 5.18 - it appears to be robust against attack. Analysis done by the Perl5 Security Team suggests that One-At-A-Time-Hash is intrinsically more secure than MurmurHash. However to my knowledge, there is no peer-reviewed cryptanalysis to prove it.

There seems to be very little research into fast, robust, hash algorithms which are suitable for dynamic languages. Siphash is a notable exception and a step forward, but within the Perl community there seems to be consensus that it is currently too slow, at least in the recommended Siphash-2-4 incarnation. It is also problematic that its current implementation only supports 64 bit architectures. (No doubt this will improve over time, or perhaps even already has.)

Universal hashing also seems promising in theory, but unproven in practice for cases like Perl, where the size of the hash table can be very small, and where the input strings are of variable size, and often selected from restricted character sets. I attempted to implement a number of Universal Hashing-style hash functions for Perl, and was disappointed by extremely poor distributions in the bucket hash for simple tasks like hashing a randomly selected set of integers in text form. This may have been a flaw in my implementation, but it appeared that Universal Hashing does not perform particularly well when the input bytes are not evenly distributed. At the very least further work is required to prove the utility of Universal Hashing in a dynamic language context.

The dynamic/scripting language community needs the academic computing community to provide a better tool box of peer reviewed string hash functions which offer speed, good distribution over restricted character sets and on short strings, and that are sufficiently hardened that in practical deployments they are robust to direct attack. Security is important, but theoretical attacks which require large volumes of key/response exchanges cannot trump requirements such as good distribution properties and acceptable performance characteristics. Perl now makes it relatively easy to add and test new hash functions (see hv_func.h), and would make a nice test bed for those interested in this area of research.

Afterwards

I would like to thank Nicholas Clark, Ruslan Zakirov, and Jarkko Hietaniemi for their contributions which led to the changes in Perl 5.18. Nicholas did a lot of deep analysis, and provided the motivation to create my attack proof, Ruslan provided analysis and a working key-discovery attack scripts on Perl's old hash function, and Jarkko motivated me to look into the general subject.

[1] See also:The bug filed against Perl and the original research.

[2] If anything, such an attack might make access to keys that aren't part of the attack faster: The attack keys cluster in one bucket. That means the other keys are much more likely to spread out among the remaining buckets that now have fewer keys on average than without the attack.

The standard mechanism by which one can hook native extensions into Perl, dubbed XS for "eXternal Subroutine", is often criticized for its arcane syntax and for obscuring how the interaction with the Perl VM works while still requiring some familiarity from the programmer user. This article aims to provide a bottom-up approach to understanding how native extensions interact with Perl while eschewing XS altogether, hopefully providing complementary learnings.