Erlang MySQL Driver History

Feb 06, 2011

Open Source Erlang MySQL driver efforts are a fractured matter. You may find yourself digging in the sources to find out about their relationships with each other – and which one to pick. Here is a brief history.

Yxa: The first Erlang MySQL driver, in ~270 lines of code, seems to have been written between 2001 and 2004 by Magnus Ahltorp at the Swedish Royal Institute of Technology. It exposes low level, blocking functions to talk 4.0 protocol with a MySQL server. In 2005 Fredrik Thulin brought the driver into its current modular form to use it for the the SIP proxy Yxa while working at the Stockholm University. It has three process layers: a high level, round-robin connection pooling module; then, middle-man, single-connection, blocking processes that do the bit-level wrangling with the MySQL protocol. This module, mysql_conn, can also be used as a single-connection, stand-alone driver. And below this, there are small, protocol-agnostic receiver processes that handle the socket communication with the server, no matter the contents. Fredrik implemented gen-server behavior, added logging, error messages, upgraded authentication, and thoroughly commented the source. This mysql driver source is working, complete and stable since at least 2007, it is available as part of Yxa 1.0 (hosted on github). It has no support for transactions or stored procedures. It is the basis for the following two packages. Its basic modular division and general functionality were not changed but only enhanced and it had originally been agreed upon that the Yxa branch should receive and merge the contributions of the later forks upstream. Unfortunately, that did not come to pass.

ejabberd: In 2006, a fork of the Yxa driver was created by Mickael Remond at Process One to become part of the successful instant messaging server ejabberd (also hosted on github). It can be assumed to be as stable as the Yxa branch, didn’t change a byte in the lowest level, but only slightly enhanced the higher level. The difference from the original Yxa branch consists mainly of added inspection functions that help with using the query results, along with an independent adoption of the MySQL 4.1 client-server protocol. The original Yxa branch has meanwhile adopted the EDoc comment format, which makes the sources look more different than they actually are. You can find a Feb 2011 diff between Yxa and the ejabberd version here, and one ignoring comments here. These two branches could be merged quite easily, probably without any change in functionality at all.

erlang-mysql-driver: The storied life of this branch began in 2006 when Yariv Sadan created a fork from the ejabberd branch, made it a standalone project, gave it the maximally generic name that stuck, and hosted it at Google Code. Before he moved on to work at Facebook, he added high-level handling of prepared statements and transactions, and at long last completed some loose ends with the connection pooling that had been known to be lagging since the Yxa version. There were changes both in the original Yxa and the ejabberd branch after the forking off that never made their way into this fork, but in all likelihood they must be minor. It is not always obvious if the changes in erlang-mysql-driver had reached their intended final form. The separation of the three process layers seems to have suffered and complicated enhancements as the highest layer module, mysql.erl, started to become entangled with the second, mysql_conn.erl. Maybe that had a part in why the upstream merge never happened. The main repository of this fork lay dormant since Oct ‘07 when in Feb ‘10, Dave Smith started making some updates and put them on github. The driver is now enjoying a couple of active forks that make a convincing case for the github Network graphs.

A parallel fork from Yariv’s branch, not entangled with Dave’s tree, is the one by Nick Gerakines. I suspect it could technically be the strongest of the erlang-mysql-driver forks, with a lot of clean up work by smart guys put in, although it is generally less well known. And much less forked. In the end, the branch was abandoned for Emysql. In all branches, documentation beyond the source comments remains lacking.

Emysql was created from scratch in 2009, specifically to achieve better stability and throughput. It was proposed and written by Jacob Vorreuter at Electronic Arts and deployed at Shawn Fanning’s Rupture.com, a social network site for gamers. Initially, Nick Gerakines, Jacob’s boss at EA, rewrote large parts of erlang-mysql-server to clean it up. But some fundamental problems remained and when half a year in, they still could not overcome performance and stability issues, Nick gave Jacob the green light to rewrite it from the ground up because they felt that, in Jacob’s words, the Yxa branch had been touched by so many people that it had become more complicated than necessary. According to Jacob, Bill Warnecke helped in the early design and testing. They abandoned the separation into three process layers and pulled the socket handling and bit-parsing into one module, coupling the functionality into direct function calls. It looks like they borrowed some chore lines from Magnus but generally created a new, straightforward architecture focused on providing a high performance node. Not only can Emysql open multiple connections, but multiple pools of multiple connections each to multiple database servers, which makes for a strong central OLTP node. Jacob says that Emysql is pretty stable and ran without issues in production at EA. Nick remembers, “The primary way we used it was to have a single Erlang node be the MySQL communication point and allow a number of connected nodes to connect through it to MySQL. We wanted very high throughput with many pids across a grid connecting to it and needed the ability to have a number of MySQL connections open for connection pooling.” Rupture was killed in the consolidations of 2009. But Shawn could probably keep the money and we the fond memory of Napster and now, the glistening Emysql.

The Eonblast Emysql fork is a continuation of Jacob’s work, including all his commits and adding docs, samples, fixes and extensions. Henning Diedrich, Vitaliy Batichko and Chris Rempel have contributed updates to this branch. Support for stored procedures has been added, but the fork is otherwise still very close to the original, which currently lies dormant. The author is currently maintaining that branch.

Fredrik, Nick and Jacob helped shedding light on the matter. Thank you very much! Errors and omissions are mine. Please let me know about any errors you spot. Thanks.

Henning

Links

† maintained at time of writing, Feb 2011.

Tags :                     
trackback
You must be logged in to post a comment.