Socklabs

SitterCity, First Impressions

2009-06-28T00:00:00+00:00

With Vanessa's grandmother moving to another state, Carolyn and I started looking for babysitter for Vanessa with sittercity.com. I'm not sure where she heard about it, but I'm leaning toward believing that it was from another website or blog.

After looking through it briefly, we signed up at the annual membership level ($95.88 USD per year) and posted a job. We weren't really looking for a full time nanny or daily babysitting, but rather someone to watch Vanessa for a few hours either once a week or every other week. Really, Carolyn and I want to get out a bit more now that Vanessa is getting older. For those of you that know me, I really miss going to the movies with her.

Browsing and searching for babysitters on sittercity.com easy. Childcare provides have profiles that lists a bio, their schedule, reviews, information on who they are and where they are from as well as any related skills. When you join, you can specify your zip code and all of the profile searching and browsing is centered on that location. In addition, if you need a childcare provider then you post a "job" and the babysitters are notified via email.

Posting a job is rather simple. The usual title, description, location and rate fields are given and then it goes into the details of when and how often the work lasts and continues to. It also goes into the childcare details like how many children, their age(s) and whatever requirements you may have for the caretaker.

We had tentatively decided beforehand what was going to disqualify people immediately. The criteria used to reject applicants included:

Schedule conflicts or unideal availability.
Already having a full-time nanny job.
The ability to compose full sentences and use correct grammar when communicating.

The job was actually posted on a Monday night and by morning, we had over 30 applications for it. I was blown away. For the next few days the number climbed to over 50 and by then we were already making plans on interviewing some of them. By the following Saturday, we had the list filtered down to 10 or so babysitters that we wanted to meet and introduce to Vanessa. We chose to meet at a nearby park, a nice neutral location, and setup 30-45 minute windows for each of them.

Some of them couldn't make it that so we ended up with 8 people planned. Also, one of them canceled at the last minute so that that took us down to a reasonable 7. Still, we had a pretty full day of interviews. Carolyn and I had a list of questions and a good idea of what we wanted to learn about them. What I wasn't prepared for was how many of them flaked out and just plain didn't show up at all. Only 3 of the 7 people we were expecting actually came. Carolyn and I were puzzled. It's worth saying that the people who didn't show up had, at some point, actually confirmed the time and location. These people acknowledged the time and place but just didn't go. It may be needless to say that those who flaked out did not get rescheduled or contacted again.

For the three that did show up, the interviews weren't bad. One of them is a definite and is starting next week. Another seems like a good fit and we will keep on call as a backup. The third we didn't feel strongly for. Mainly because she didn't engage Vanessa and her transportation situation sounded a bit flaky.

The two that we have are only available for the summer and are off to school in the Fall. That means that we'll have to do another babysitter search in September/October, but that's not so bad. We have a good idea of what to look for when we go through the next round.

In the end, I'm really happy with sittercity.com and we found a babysitter through it fairly quickly with lots and lots of options. I'm really glad that we signed up and I recommend it. There were some hickups but it wasn't with the site itself. And the best news is that next week, I'll finally get to see Star Trek.

Mochevent

2009-06-11T00:00:00+00:00

This past weekend I started a small work-related project called mochevent. It's purpose is to offload the work HTTP request build up, tear down and socket handling onto a c application. As requests come in, the request method, uri, headers and body are dispatched off to an Erlang process on a remote node to do whatever heavy lifting is required to fulfill the request. The Erlang node then sends the results back to the cnode to then be massaged into to an HTTP response which gets sent to the client.

The application is based on the the proof of concept application developed by Richard Jones and posted to http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3/. The goal is to get it as small and tight as possible so we are really looking at performance and memory footprint when looking to improve the application.

The project can be found on GitHub as mochevent and contributions are welcome and encouraged. This is looking like we will be using it in an official capacity at work so keep watching for feature additions, improvements and bug fixes.

Armory3, a proof of concept RabbitMQ application

2009-05-10T00:00:00+00:00

I've been having a lot of fun with RabbitMQ lately, the most recent accomplishment being porting the armory2 module to use a RabbitMQ queue instead of an ets table.

With that said, this is the result of that work. The armory3 module is an extension of the armory module in the erlang_wowarmory project on GitHub. It is a distributed World of Warcraft Armory character and guild retrieval and processing library. The obvious benefit of doing it this way is that you've got the power of a RabbitMQ application powering your queue and work list, allowing you to gut the complexity of queue processing from your application. The downside is that if RabbitMQ goes down or there is some sort of unexpected hiccup along the way, you may feel some pain.

Erlang Factory

2009-05-01T00:00:00+00:00

Erlang-Factory is over and it was great. I'm so happy to see the many project and community leaders coming together to introduce their projects and get involved with each other's work. There were 6 tracks throughout the 2 day conference that included "Erlang Case Studies", "Test-Driven Development", "Enterprise Integration", "Cool Applications and Features", "Give me a break from Erlang" and "Erlang and the Web". My own presentation about test driven development in Erlang went quite well -- I'll post a link to the video once it's up.

On day one keynotes by Robert Virding and Ezra Zygmuntowicz opened up the conference with the idea that there isn't a one true way to work on your projects with Erlang. Robert's talk on the why and how Erlang was designed the way it was, was really interesting. Ezra's talk about blending Ruby and Erlang gave some insight as to how Erlang can play very nicely with and compliment web languages like Ruby.

On day one, I went to these talks:

EUnit How to: Tips, Tricks and News by Richard Carlsson
Integrating OS package management and Erlang by Paul Mineiro
Mixing Erlang and Ruby with Erlectricity by Tom Preston-Werner
Test-Driven Development with Erlang by myself

Day one ended with the Birds of a Feather Sessions followed by an Erlounge at the Palo Alto Gordon Biersch but I couldn't attend.

The day two keynotes where by Damien Katz and Ulf Wiger, both were really really good. Damien's talk shifted and instead of talking about the technical traits of couchdb, he gave a telling and personal story about what its like to put it all on the line to do something you enjoy. His presentation was different because it told the story about how things were kind of crappy before he became "that guy working on the stuff he enjoyed." It was inspirational.

On day two, I went to these talks:

CouchDB from 10,000 feet - An introduction to CouchDB by Jan Lehnardt
CouchDB Apps by Chris Anderson
Everything you wanted to know about Dynomite but was affraid to ask by Cliff Moon
CouchDB Case Studies by Tim Dysinger and Nitin Borwankar

The closing session for day two was really about reaching out to the users, enthusiasts and developers out in the wild to see what they were interested in and wanted. Things like unified package management and better documentation came up.

All of the material from my presentation can be found in my erlang-factory repository on GitHub.

What's been going on lately?

2009-04-02T00:00:00+00:00

Things were getting dusty around here so it's time for me to give a nice life update.

The big news is that we are moving! We found a nice 3 bedroom place just a few miles down from where we live in Mountain View. Carolyn and I are both really excited about moving into a bigger space. Not only will we get a 3rd bathroom but I'll also get an office. It's very exciting.

Vanessa is growing up pretty quickly and starts pre-school in August. Granted, it is a few months away but still very exciting.

We've been extremely busy at work and the progress has been amazing. Without going into too much detail, I've got a really solid Erlang team hacking on some really cool technologies here at EA. Some of the open source technologies we are using include log_roller, erlang_protobuffs and stateless_server. I sincerely hope that we'll be able to disclose more on our official blog soon.

Erlang-Factory is drawing near and that's going to be a lot of fun. I'm looking forward to meeting a lot of the people there. If you plan on attending or will be in the area please drop me a line.

I Play WoW Scaling

2009-02-06T00:00:00+00:00

Having too much traffic is a great problem and I'm fortunate enough to be facing it at the moment. Over the past few weeks, thanks in part to several really great articles on WoW Insider about I Play WoW, I Play WoW traffic and use has been increasing significantly and I've been starting to notice that increase in traffic in a few different ways.

The first has been in queue length. The big feature of the site is the ability to import your World of Warcraft characters and display them on your profile. After you've done that, actions like "dinging" and refreshing your character's data become available. All of these operations require requests to be made to fetch data from the World of Warcraft Armory.

What's great is that this information is freely available and I applaud and thank Blizzard for making it so. What isn't so great is that this can be a lot of traffic coming from my servers to theirs and it often comes in spikes and bursts. To avoid hammering them and drastically reducing my karma, I've implemented queues and throttling when fetching data. Using the erlang_wowarmory module, items consisting of the request type and meta-data about the request get stuffed into request queues that exist on all of the nodes used by the Facebook application. This works really well and has saved me lots of time, money and energy by having a stable and reliable armory data processing system. The caveat with this is that with lots of traffic come large queues.

The issue that I'm seeing is that during peak traffic hours, I may see the queues go up to 2,000+ items. Around the time of the first WoW Insider post the queues got up to 6,000+ items. These queues get processed and in due time everything resolves itself without issue. The real problem is three-fold.

User's data doesn't get processed immediately and they complain. (Rightfully so, in my opinion)
When a user's data doesn't get processed immediately then they tend to think something went wrong (regardless of how clear your messaging is) and try it again, adding more data to the queue.
Some requests are more important than others. Characters should always be processed first, regardless of how large the queue is. Next come guilds, followed by everything else.

There are two, complimenting, solutions to this problem that I've got my eye on. The first is to implement a better queueing system that is less tolerant of duplicate items. The second is to add more crawling capacity to the system.

The first solution can be seen through the development of the armory2 module as part of the erlang_wowarmory project. What I've done here is taken what I've learned about how things are queued and made several key changes. The first big change is the actual queue data structure itself. At first, I was using the 'queue' module that comes in the standard library to manage the FIFO queue. It worked really well and efficiently for small queues but as they start to hit 3,000 to 5,000 items, I'm concerned about efficiency.

Now and ets table is used to manage the queue. The queue manager creates and owns an ordered_set ets table that operates as a first-in/first-out queue based on the item type. When an item is dequeued, it attempts to find character items first, then guilds and then everything else ordered by when it was inserted into the queue. This is working really well so far and I'm pleased with the results.

Another change in the armory2 module is the way queues are distributed. In the armory module, each node would start a gen_server process that would store a local queue and would have a linked process that dequeues items and processes them. I'm experimenting with a different model right now that uses the global module to have a single queue manager that owns the ets table used to keep the entire queue. If at any point a crawler attempts to dequeue something and discovers that the master isn't running, it will attempt to start the master on the local node and the queue is recreated.

The second solution to the problem is essentially "grow with hardware". The reason that this applies here is that I can't increase the actual number of networks requests currently being made on each node. The current throttling model prevents each IP address (slice) from making more than one request against the World of Warcraft Armory per second. There isn't anything that I can do to resolve this. The only real solution here is to add more nodes and crawlers to the grid. Thankfully, Erlang makes this stupid simple. I found this out the hard [read as: easy] way when the WoW Insider article was published.

I'm using SliceHost to host my application and two of the nodes have regular backups made daily and weekly. Each of my slices are the most basic available which keeps cost down for me. When I needed to bring the additional nodes to handle the increase in traffic, it was just a matter of creating new slices based on the backups (saves time in configuration) and running /etc/init.d/ipwcore start and /etc/init.d/ipwfbfe start. When the nodes came up they detected the other nodes using net_adm:world() and starting taking traffic on immediately. I should mention that I also added them to pound, the load balancer that I use.

My experiences scaling the I Play WoW Erlang Facebook application have been great. With good design and a little bit of homework it's easy to take on problems that have traditionally been show stoppers. All of the crawling code that I've referenced in this entry is open source under the MIT license and can be found on GitHub. With the 3 additional slices/nodes (I run both an ipwcore and ipwfbfe node on each slice) things have been running very smoothly.

WoWInsider's 15 Minutes of Fame

2009-02-05T00:00:00+00:00

Recently I was featured on WoW Insider in their 15 minutes of fame column. It was a great chance to talk about and express some of my thoughts and ideas on I Play WoW, social networks, World of Warcraft and mixing the three.

The article can be found on their blog.

Useful Code Coverage Reports with Etap

2009-01-23T00:00:00+00:00

This evening I added the etap_report module to the etap project. This module can be used to create nice looking, well-formatted html code coverage reports. This is a great way to get a quick view into your project's code coverage and determine what areas need to be focused on when creating test code.

Using the etap_report module is simple. It exports the create/0 function which is called in the directory that your coverdata files are created in. The process looks something like this.

$ cd ~/dev/erlang_protobuffs
$ COVER=1 make test
$ erl -eval 'etap_report:create()' -s init stop
$ open protobuffs_report.html

An example can be found here: http://ngerakines.github.com/erlang_protobuffs/coverage/protobuffs_report.html

Just remember: When do I test?

Using processes as caches in Erlang

2009-01-18T00:00:00+00:00

When I'm talking to someone who is new to Erlang, I go out of my way to tell them one very specific thing: Erlang will change the way you write software. I was playing around with a few features in I Play WoW recently, one of which is quasi-dynamic links to wowinsider.com on the My Characters page.

The feature is relatively simple. On the My Characters page, display links for 3 to 5 of the most recently published articles on WoWInsider. This requires some minor RSS/ATOM parsing and a web request. All-in-all, it's a nice feature and something that I think my users would like.

The implementation is pretty simple as well. The module exports a few functions that return articles based on a given RSS feed. The content returned is a list of tuples containing the article title and link. The special sauce of the module is the use of a process to store a cache of articles based on fetch time. When articles are requested, a process is queried to determine if it has a cache and, if it does, should the cached list be refreshed.

This feature went live last night and although the article cache isn't distributed, the data is small enough to cache on each web interface application.

Code Coverage w/ Etap

2009-01-14T00:00:00+00:00

I added a small, somewhat experimental, feature to etap a few days ago. With etap, you can now use the Erlang cover module to determine how much of your code is being testing by your tests. I've already started using it on several of my modules and projects to enhance the test suites. It's pretty easy to use and in this small demo, I'll run through the basics.

In the erlang_protobuffs project I'm using etap to test basic functionality. With code coverage analysis I can increase the module's stability by making sure more of it's functionality is tested.

$ cd ~/dev/erlang_protobuffs
$ make test
... 
All tests successful.
Files=8, Tests=35,  5 wallclock secs ( 0.06 usr  0.03 sys +  3.58 cusr  0.67 csys =  4.34 CPU)
Result: PASS

To enable code coverage analysis, all you have to do is set the COVER=1 environmental variable while running your tests. None of the TAP output will change but it will write a number of .coverdata files, one for each etap test group.

$ COVER=1 make test
... 
All tests successful.
Files=8, Tests=35,  5 wallclock secs ( 0.06 usr  0.03 sys +  3.58 cusr  0.67 csys =  4.34 CPU)
Result: PASS

The .coverdata files hold the exported cover data as per the cover module. With a little bit of Erlang we can aggregate the results from those files and create reports.

$ erl
1> [cover:import(File) || File <- filelib:wildcard("*coverdata")].
[ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok,ok]
2> Cover = fun(M, Out) -> cover:analyse_to_file(M, Out, []) end.
#Fun
3> Mods = cover:imported_modules().
[protobuffs_compile,protobuffs]
4> [Cover(Mod, atom_to_list(Mod) ++ "_coverage.txt") || Mod <- Mods].
Analysis includes data from imported files
[".../dev/erlang_protobuffs/fc4046de9a9bb3efdf53d1d698797bc.coverdata",
 ".../dev/erlang_protobuffs/f27caf5168ce8d77b3fde6a4411cfc11.coverdata",
 ... ]
Analysis includes data from imported files
[".../dev/erlang_protobuffs/fc4046de9a9bb3efdf53d1d698797bc.coverdata",
 ".../dev/erlang_protobuffs/f27caf5168ce8d77b3fde6a4411cfc11.coverdata",
 ... ]
[{ok,"protobuffs_compile_coverage.txt"},
 {ok,"protobuffs_coverage.txt"}]
5> q().

The files protobuffs_compile_coverage.txt and protobuffs_coverage.txt will breakdown your modules line by line and show how much of the code has been executed and how many times. From this I can see that lines that have "0..|" have code that has never been called, hence there is 0 code coverage there.

An example can be seen in the protobuffs_coverage.txt file around the encoding of bools.

   |  %% @doc Encode an Erlang data structure into a Protocol Buffers value.
   |  encode(FieldID, false, bool) ->
0..|      encode(FieldID, 0, bool);
   |  encode(FieldID, true, bool) ->
0..|      encode(FieldID, 1, bool);
   |  encode(FieldID, Integer, enum) ->

To remedy this I'll create a test that calls this code. As I'm writing this, it turns out that there is actually a bug in my implementation of bools and writing the test for it surfaced that bug.

cat protobuffs_t_009.t << EOF
#!/usr/bin/env escript
%% -*- erlang -*-
%%! -pa ./ebin -sasl errlog_type error -boot start_sasl -noshell

main(_) ->
    etap:plan(2),
    etap:is(protobuffs:encode(1, true, bool), [8, 1], "1, true, bool"),
    etap:is(protobuffs:encode(19, false, bool), [<<129,24>>,0], "19, false, bool"),
    etap:end_tests().
EOF

Now that the test exists (and the bug fixed!) I'll clear my cover data and reports and generate new reports.

$ make clean && rm -rfv *.coverdata *.txt
$ COVER=1 make test
$ erl ...

What's different is that now it is visible that the bool encoding functionality has been executed once (as per the newly created test).

   |  encode(FieldID, false, bool) ->
1..|      encode(FieldID, 0, int32);
   |  encode(FieldID, true, bool) ->
1..|      encode(FieldID, 1, int32);

Regular code coverage analysis is important to reducing test debt. It helps find bugs and saves you time, effort and energy.

Erlang and Nginx, a proof of concept

2009-01-12T00:00:00+00:00

I was doing some writing this evening and out of the blue, I was struck with the urge to create what would be the "mod_erlang" Apache module. I've heard of several really cool stories of using c/c++ apps as entry points to Erlang grids and it sounded like a really cool idea to pursue.

Then I remembered what developing Apache modules was like and was immediately discouraged.

Meanwhile, I was also thinking about nginx and the several nginx vs X articles that I've read recently and was curious about nginx's support for plugins and extensions. Low and behold nginx has a pretty nifty extension/module framework and minutes later I was digging through the documentation.

The idea of using a solid c or c++ application for managing the tear down and build up of HTTP requests and using Erlang to actually process the request sounds like a win. I think Erlang is an ideal language and platform for building back-end minded services and application tiers and have been pushing this belief heavily. Using an application like nginx, apache2 or a home-brew interface to provide an entry/access point into and Erlang grid is what I'm going for with this project.

http://github.com/ngerakines/erlang_nginx/tree/master

Right now, it's in the early development stages and exists only as a proof of concept. The features that I'm developing toward include the ability to break down a request into it's URI, headers and body and pass that into an Erlang grid's handler process as defined by a configuration variable. The request would wait for a specific amount of time for a message to be sent back that would include the status code, headers and response body that would be displayed to the user.

Another interesting path would be to compose whatever body is needed by the cnode/application. An example would be having the response from the grid be a tuple representing the type of response and components of the response and having the cnode/application create the json, xml or protocol buffers response bdoy based on that message.

I'm open to your thoughts, questions and comments. I don't claim to be a stellar c developer, so please help me and this project by not only taking a look at the code and contributing, but also telling me what I did wrong and why you changed what you did. As development continues, I'll probably use I Play WoW as a guinea pig and will post results as I can.

I Play WoW hits 100k users

2009-01-11T00:00:00+00:00

I'm really really excited to say that I Play WoW has officially crossed the 100,000 user line. I knew it was getting close and have been checking it every few hours, so about 20 minutes ago I hopped onto Facebook and saw that it happened. The official count was at 100,039 as of 11:45. I think it is the only Facebook application out there with over 100,000 users that is 100% Erlang. Go Erlang!

Erlang Library Best Practices

2009-01-10T00:00:00+00:00

Joe Williams wrote a pretty nifty Erlang memcached client recently and posted it to GitHub. I sent him some pointers and after writing it, decided it was worth posting.

If you've got a library dedicated to interfacing with a particular service, keep the function names simple and direct. Instead of having merle:memcached_set/2, keep it at merle:set/2.
In most cases, it's best to directly interact with the target service as opposed to providing an additional layer such as a gen_server or process. This is something I did with the couchdb, twitter and facebook client libraries. There are some situations where you do want to allow for things like throttling, server settings storage, etc in your client library, but I really don't think that you'll have one of those situations with this one. I'd much rather make a call like merle:set(Server, Key, Value) that directly hits the memcached server than something that may have to sit in a gen_server message inbox/queue before being executed.
Write tests! This sort of thing really needs a test suite. You may even want to investigate creating a mock memcached server that your test suite runs and can push requests against.
Define a good license and add it to the top of each of your source files. I recommend either the MIT or the Mozilla.
Use edoc. At the very least add @doc for public functions and @private for internal functions. It only takes about 5 minutes to set one line descriptions for a module and about 20 functions. With GitHub you can publish all of the docs for the project using the gh-pages branch (see http://ngerakines.github.com/erlang_facebook as an example).
For official releases, publish dist-src packages in your gh-pages branch. I keep source tarballs in the "/sources" for all of the packages that I do that with. Makes it really easy to have your packages integrated in system packages like gentoo ebuilds, rpms or debian packages.
Use a Makefile or Rakefile to make it easy for people to build and install the package. This is a good practice and enables more wide-spread adoption of your library.

There were some other notes but they were specific to the merle library. All of this got me thinking about putting together a Best Practices wiki or git repository where people can contribute their knowledge and snippets.

Welcoming Toby

2009-01-04T00:00:00+00:00

Over the holiday break, Carolyn and I decided that we'd like to adopt another cat. When Picasso first arrived, he came with another cat, Melee, that died soon afterward from FIP. It was kind of rough, especially with Carolyn being pregnant at the time, and we've been a one cat household since then.

On Friday we went to the Silicon Valley Human Society and looked around. We had a good time, although it was slightly depressing to see how many animals go through the place. There was a Volunteer cat petter (what a title eh?) that showed us around and let us interact with some of the cats that were in the cages. While we were looking around there were two that caught my eye. Toby was one and a cat named Artimas was another. We got to hold and pet both with Tom there. Vanessa really loved it.

While we were in the cages, we found out that black cats get adopted far less than other cats. It struck me as really odd but the guy continued to say that a lot of people just don't adopt black cats. Some out of superstition, others for weirder reasons.

I certainly didn't have any reservations going into the place about the type or color of cat that I'd like to adopt. I'm pretty sure that Carolyn didn't either. Our only concern was that it would coexist well with Picasso and Vanessa.

Before we left, we made an appointment for Sunday to come in and interview. When we came in we found that Artimas was being adopted right then. He really is a beautiful cat and we were both happy to see him leave. Thankfully Toby was still there. The entire process took about 2 hours and before we knew it, we were on our way home.

I suspect it'll be a little while before Picasso and Toby start to get along. Right now we have Toby in our bedroom and plan on having him there for a week or so before we start to let him see the rest of the house. Sadly, Toby was brought to the SVHS from a San Jose shelter and has spent most of his life in cages. It might take them some getting used to having each other around and determining territories.

2008 in Review & 2009 Goals

2009-01-02T00:00:00+00:00

2008 was a great year of activity and excitement. I've got the sneaking suspicion that 2009 is going to be no less busy.

Finished and published my first book!
Spoke at CUFP and met a lot of really smart people.
Gave several tech-talks and presentations.
Learned how to develop apps for the iPhone.
Migrated I Play WoW from a Perl app to an Erlang app.
Learned about CouchDB.
Learned Git.
Accepted a lead position at EA.

This is what I'd like to accomplish in 2009.

Finish my second book.
Do more tech-talks and presentations.
Have a 100 day streak of open source commits on GitHub.
Write more technical articles and entries.
Learn Python.
Hit level 80 with Korale and get into tier 9.
Grow I Play WoW to 75,000 monthly active users.

Using Test::Harness with etap

2008-12-30T00:00:00+00:00

This isn't a beginner's walk through. It assumes you know how to use and read Erlang as well as install Erlang and Perl libraries

I love TAP. TAP is a protocol used to create and interact with testing frameworks. The general idea is that your unit tests interact with a TAP client which outputs the TAP protocol that is then consumed by a TAP server. What it boils down to is a really simple text based structure used to convey unit test results, with a few added bonuses here and there.

With etap and Perl's TAP::Harness we can create really slick test result output and move closer to having a continuous integration build environment. Before you get ahead of yourself, you'll need to make sure that you've got etap downloaded and in your Erlang lib path. You'll also have to install TAP::Harness and it's dependancies.

To demonstrate how this is done, we'll use the erlang_protobuffs library. This library is composed of two modules that provide functionality through direct interaction with their exported functions. They don't extend any OTP behaviors and have very few side effects. The project's sources are contained in the src directory and it's tests in the t directory. There is also a support directory that provides the Makefile include for building the actual modules.

The test target in the root Makefile calls the prove command with the verbose flag and a glob of the *.t files in the t diretory.

test: all
    prove -v t/*.t

The actual tests for the project are escript files with the .t extension. Using the escript hash bang lets us treat each file as it's own complete unit test. They can be run in any order and have their own configuration and directives. Below is the protobuffs_t_001.t file.

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -pa ./ebin -sasl errlog_type error -boot start_sasl -noshell

main(_) ->
    etap:plan(8),
    etap_can:loaded_ok(protobuffs, "module 'protobuffs' loaded"),
    etap_can:can_ok(protobuffs, encode),
    etap_can:can_ok(protobuffs, encode, 3),
    etap_can:can_ok(protobuffs, decode),
    etap_can:can_ok(protobuffs, decode, 2),
    etap_can:loaded_ok(protobuffs_compile, "module 'protobuffs_compile' loaded"),
    etap_can:can_ok(protobuffs_compile, scan_file),
    etap_can:can_ok(protobuffs_compile, scan_file, 1),
    etap:end_tests().

Lines 1, 2 and 3 are used by the shell and escript command to set any Erlang vm options necessary. In this case we want to include the ebin directory, start sasl with minimal logging and disable the shell.

The main/0 function is called escript with whatever relevant arguments that may apply, but we don't care about them so we ignore them for now. Here we perform several tests to assert that the project's modules load and export the proper functions.

The etap:plan/1 and etap:end_tests/0 functions wrap the etap*:* function calls and is used to define the test plan. At this point etap is not designed to work without finite plans.

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -pa ./ebin -sasl errlog_type error -boot start_sasl -noshell

-record(person, {name, address, phone_number, age, location}).

main(_) ->
    etap:plan(1),
    etap:is(protobuffs_compile:scan_file("t/simple.proto"), ok, "simple.proto compiled"),
    compile:file("simple_pb.erl"),
    Data = [{1, <<"Nick">>, string}, {2, <<"Mountain View">>, string}, {3, <<"+1 (000) 555-1234">>, string}, {4, 25, int32}],
    BinData = erlang:iolist_to_binary([protobuffs:encode(Pos, Value, Type) || {Pos, Value, Type} <- Data]),
    #person{ name = <<"Nick">>, address = <<"Mountain View">>, phone_number = <<"+1 (000) 555-1234">>, age = 25} = simple_pb:decode_person(BinData),
    BinData = simple_pb:encode_person(#person{ name = <<"Nick">>, address = <<"Mountain View">>, phone_number = <<"+1 (000) 555-1234">>, age = 25}),
    etap:end_tests().

In the protobuffs_t_005.t file you see that you can do things like define records. In this test unit we call several of the library's exported functions and test some of the more complex functionality. The protobuffs_compile:scan_file/1 function actually writes several files which are then compiled by compile:file/1 and used by subsequent test unit function calls.

The output of the make test command, as processed by the TAP::Harness perl module, is very clean.

mbp:erlang_protobuffs ngerakines$ make test
mkdir -p ebin/
(cd src;make)
erlc -W -I ../include  +debug_info -o ../ebin protobuffs.erl
erlc -W -I ../include  +debug_info -o ../ebin protobuffs_compile.erl
prove t/*.t
t/protobuffs_t_001....ok   
t/protobuffs_t_002....ok   
t/protobuffs_t_003....ok   
t/protobuffs_t_004....ok   
t/protobuffs_t_005....ok   
t/protobuffs_t_006....ok   
All tests successful.
Files=6, Tests=17,  1 wallclock secs ( 0.04 usr  0.02 sys +  0.97 cusr  0.27 csys =  1.30 CPU)
Result: PASS

Depending on the error and severity, failed tests will either output in a similarly clean fashion or stop the entire suite from processing.

mbp:erlang_protobuffs ngerakines$ make test
mkdir -p ebin/
(cd src;make)
erlc -W -I ../include  +debug_info -o ../ebin protobuffs.erl
erlc -W -I ../include  +debug_info -o ../ebin protobuffs_compile.erl
prove t/*.t
t/protobuffs_t_001....ok   
t/protobuffs_t_002....ok   
t/protobuffs_t_003.... Failed 1/3 subtests 
t/protobuffs_t_004....ok   
t/protobuffs_t_005....ok   
t/protobuffs_t_006....ok   

Test Summary Report
-------------------
t/protobuffs_t_003 (Wstat: 0 Tests: 3 Failed: 1)
  Failed test:  2
Files=6, Tests=17,  2 wallclock secs ( 0.04 usr  0.02 sys +  0.96 cusr  0.26 csys =  1.28 CPU)
Result: FAIL
make: *** [test] Error 1

The prove command has a number of options and arguments that augment test execution. Please refer to prove --help or the documentation available on CPAN for more information.

There are a few things to take note of. All of the paths to included and referenced files, such as the simple.proto file, are relative to where the prove command is called. In this case it's in the root directory as part of the test target. Also, for some more complex and lengthy tests, please read up on the escript documentation. The compile mode can be used to pre-compile the file before being called but has a few caveats that you should be aware of.

I Play WoW Monthly Active Users

2008-12-24T00:00:00+00:00

I guess the changes that I made back in November have had a positive impact.

In late October / early November I ported over the very last perl piece of the I Play WoW Facebook Application into the Erlang application set. I'm thrilled with the results that I'm seeing. The idea of using Erlang to power a web service has been reaffirmed as a great idea time after time now. The application is supporting well over 100,000 total users and just under 40,000 monthly active users. The system knows about 175,000 unique characters.

I've been telling Janl for a while now that I'd post some stats and information about the app and it's usage patterns. Here you go.

Before I go any further, I want to state the primary and driving reason for doing all of the changes that I've made. The goal was to lower the cost per user. This cost means time developing features, fixing bugs, interacting with user data (storing, fetching, aggregating, crawling, etc) and supporting users.

The application is powered by Erlang, CouchDB, MySQL, Xapian and Amazon S3. Erlang powers the web interface, the interaction with the Facebook API as well as the other services. CouchDB acts as a primary database with writes and updates still being sent to MySQL. For what it's worth, the application entirely relies on CouchDB for document reads and aggregates, but realistically CouchDB is still being developed and there are changes from time to time that warrant not depending on it alone.

Xapian is used internally for a search feature that I'm testing. I decided to go with xapian instead of Lucene because it has a smaller footprint and I don't need anything big and bulky at this point. Amazon S3 is used for image and file storage. Using Amazon S3 was much cheaper (time and money) than rolling my own file storage system.

I Play WoW started as a Perl application using the Catalyst framework using MySQL and Memcached. Since then it's evolved quite a bit. The Erlang component of the application is really a cluster of applications.

ipwapp handles CouchDB, MySQL and Facebook Platform interfacing. This is where the bulk of application's core functionality and features exist.
ipwweb provides the web interface that the Facebook Platform makes requests to for canvas page content.
ipwcache provides a write-through cache and storage system for crawled data. This includes World of Warcraft characters, guilds, etc.
ipwsearch provides the xapian search interface and rebuilds the index from time to time.

Although these applications aren't open source, there are a lot of pieces that are. The erlang_couchdb, erlang_facebook, erlang_wowarmory and etap projects available on GitHub (open source with the MIT license) are all actively used and developed by I Play WoW. This project also uses the extremely powerful MochiWeb application.

There are currently three Erlang nodes running this application. Each node runs one of the services and the other nodes are set as fallback nodes using distributed Erlang configuration. The erlang_wowarmory module is running on all nodes to distributed World of Warcraft Armory crawling evenly. The database, search

The CouchDB data store currently has 537,248 documents. Most of those documents are either character or guild documents. There is one primary document per character, guild and realm as well as a number of supportive documents for storing all sorts of other things. The MySQL database has around 383,000 rows. There are 5 design documents being used and about 25 views. This number has grown and shrunk over the past few months as I've learned to use documents more efficiently with the data structures the contain. It really has been a trial-and-error process with it's ups and downs, but ultimately it's worked out very well for me and I'm very satisfied with the results.

The one thing that I do slightly differently with CouchDB is actively track many document versions. For World of Warcraft characters I don't just keep one document per character, I keep one document per version of a character. This means that document ids are completely arbitrary and aren't used to identify characters unlike guilds and realms in the system. This is done to let me track different configurations of a given character over time and activity. This done through creating a hash of several mutable and immutable fields that represent a character and when that hash changes, track the information separately in a new or different document. This makes me rely very heavily on views to retrieve character documents from the system but it hasn't translated into "more work" yet.

The results of rewriting it have been great. I use fewer servers that cost less to support a larger number of users. At first I had three machines with either one or two gigs of ram as well as one database machine. Now I've got 4 machines with either 256 or 512 megs of ram, one of them still dedicated to MySQL, CouchDB and Xapian. The three nodes on the application grid average about 10% of memory and about 1% CPU. Some simple load tests show that it can sustain an increase of 9x without having to expand, and even at that point expanding is just adding another node to the grid. I'm effectively serving 3x the traffic at a third of the cost. Making changes to the application has been simplified with less downtime and the time to debug and fix errors has decreased.

Message me through Facebook, email or twitter if you've got questions or want to know more about a specific subject.

Protocol Buffers and Pygments

2008-12-23T00:00:00+00:00

I've been working on a small Protocol Buffers definition file parser for Pygments. It's a work in progress and at this point it only supports a very limited number of tokens and features. This is being developed to help the documentation effort for a Protocol Buffers based API that I've been working on.

Deciding on a new phone

2008-12-21T00:00:00+00:00

I'll be getting a new phone in the near future and deciding on which to get is proving to be difficult. I've been using a first generation iPhone for a while. The experience with it has been overwhelmingly good but I want to see what else is out there. It's come down to one of these now: iPhone 3G, Blackberry or Android device.

This is what I'm basing my choice on:

I want to be able to send and receive email. I use Google Apps to manage gerakines.net, so something that ties into Gmail is preferable.
I want to easily send and receive text messages. This includes things like twitter, but also automated system messages.
I want to easily sync contacts between the device and my computer. Syncing with a service like HighRise is preferable.
The web browser has to be good. I don't want to be stuck with a crappy wep interface.
3G/WiFi support. I'm coming from 2G, being an AT&T subscriber, so I'd like to either switch to 3G or be able to use WiFi off and on.
I want to be able to find and use 3rd party applications.

Less important / perks:

I'd be nice to be able to listen to music on it. I've got an iPod that I keep with me, so this isn't terribly important.
I'd be nice to have GPS map software, not required though.

The seemingly obvious choice is to upgrade to an iPhone 3G. The iPhone 3G is in the lead right now because it does nearly everything that I want and I'm already familiar with it. I've also got experience developing apps for it, which is a big bonus. What I don't want to do is spend a fortune on buying one without extending my contract with AT&T by another two years. That is really unattractive right now and is one of the biggest things holding me back from getting one. I haven't been terribly unhappy with AT&T for any particular reason, but I don't like the company and I've been trying to get rid of my dependancies on them. The first step was Comcast, the second was Vonage, the third is not using Cingular (AT&T Wireless).

Having a Blackberry sounds appealing. I've heard that it's support for email and messaging sets the standard for everyone else. The Blackberry Storm is pretty sexy and does what I want. It's gotten good reviews but definitely has it's limitations. Engadget has a really good review that answers a lot of questions and raises some concerns. I suspect that the main thing holding me back is the lack of WiFi support and app store-like service. I've also read, from Engadget as well as others, that the interface isn't as slick and using it feels a bit rigid.

The G1 is another contender. I've never been one to shy away from first generation devices (my iPhone is a first gen) and the Android G1 is no exception. Although I've read a lot of mixed reviews, I think the G1 would make a great phone. It's got all of the features that I'm looking for as well as the perks. The downside to using one of these is that it's off-network for me. I've seen that you can buy a unlocked developer phone, but that's not what I have in mind. There have been a few cases where people have been using it in an unactived mode that is good for doing everything but making calls. Would this be worth moving over to T-Mobile? I've used T-Mobile in the past and my general experience was good. I'm not opposed to this at all.

I've also read about the Kogan Agora, another Android device, coming out in late January. It looks much more attractive than the G1, but is still vaporware. Until I see someone holding it and telling me how awesome it is, pictures don't really do it for me.

For now, I'll just continue to wait. My current iPhone is on it's death bed and I'm not sure how long it'll last. It's got a small crack in the upper-right corner of the screen, the plastic on the bottom-right speaker is gone and the home button sticks. If you've got a 3g, G1 or Blackberry please email me and let me know how you feel about it.

Moving to Jekyll

2008-12-20T00:00:00+00:00

I've had my eye on Jekyll for a short while now and finally decided to make the switch. This is my fourth major CMS change since I started this blog way back when. I'm pleased with the idea of moving to a much simpler publishing system and I'm happy with the results so far.

This all started a few weeks ago when Jekyll caught my eye as a project on GitHub. One of the things I really like about MT is it's publishing model of building out static content. When I saw that Jekyll did the same thing and that it was made with version control in mind, it didn't take long for me to finally start using it.

I also saw today that GitHub will allow user domains using Jekyll as the publishing system for that content. That is such a great idea, using git repositories as a way to allow users to create blogs, documentation sites and whatever else suits them. There are a bunch up so far and mine can be found at ngerakines.github.com. I plan on taking advantage of the ability to create subdirectories for repositories to display project documentation and related material.

What I love about this most is that I'm writing this blog entry from the comfort of TextMate. To publish this entry, I'm just got to commit and push the change to the socklabs-blog repository on GitHub and have it work it's way to the machine that is actually serving the static content.

I'm still debating on storing some of the extra stuff like images and files that technically are a part of my blog but indirectly on GitHub. I think I'll hold off on that for now. The only down side to this transition is that many of the links on the site have changed. I suppose there are worse things though.

The transition from MT to Jekyll wasn't that bad. I created a small Erlang module that dumped the mt_entries table into individual files for the _posts directory. I've been using Markdown to compose blog entries, so that wasn't an issue for me.

January SF Erlounge at EASF-Rupture

2008-12-20T00:00:00+00:00

Last week I announced on the Erlang Questions mailing list that we'd be hosting an SF Erlounge at EASF-Rupture in mid-January. The event will be from 7:00 pm to 8:00 pm at our office 2nd street. Erlang developers, enthusiasts and the curious are invited to this social meetup. We'll be providing light snacks and drinks with the idea that we'll be going out dinner afterward.

This event is open to the public but space is limited.

We are located at Harrison St & 2nd St, just a few blocks north of South Park, south of Market St. Our office is a comfortable 10 minute walk from the Caltrain station for anyone in the south-bay who would like to attend.

400 2nd Street, Suite 350
San Francisco, CA 94107

A quick and easy server demo in erlang

2008-12-07T00:00:00+00:00

I wrote up this quick Erlang module to demonstrate how easy it is to have a process manage state for other functions and processes.

Erlang portage overlay

2008-11-26T00:00:00+00:00

I've posted the portage overlay that I use for my Erlang projects. The sources are downloaded from tagged GitHub repositories and all ebuilds have been tested and verified on a fresh 2008.0 Gentoo environment. It assumes that you are using the most recent dev-lang/erlang package based on Erlang/OTP 12-4. I've decided to build and test all of these packages using the standard dev-lang/erlang package instead of using a custom build, so please keep that in mind when preparing your system.

Current ebuilds include:

erlang_twitter
erlang_couchdb
erlang_facebook
etap
mochiweb

I'm planning on creating ebuilds for the rest of my GitHub projects and then looking at what else is out there.

Check it out: http://github.com/ngerakines/erlang_portage/

etap 0.3

2008-11-25T00:00:00+00:00

etap 0.3 is out and available on GitHub. etap is a TAP testing library with growing support for more complex operations like behaviors and web requests. The README file has more information on installation and use and the included tests provide a solid example of how to use the library.

Hard up for cash? Try a verification process.

2008-11-17T00:00:00+00:00

The Application Verification Process was announced today and at first I thought it was a joke. Here's why:

The benefits are vague and misleading. In the verification FAQ it mentions two different types of benefits but not together. One part says that verified applications get a special badge displayed on the application's about page as well as priority listing in the application directory. Another part of the FAQ mentions an increased number of notifications and requests. Both of these areas also mention a $100 advertising credit but honestly that's like giving bottled water to a fish.

What qualifies is extremely vague and very subjective. When describing the program in the official verification FAQ, they list gems like "Applications should be Meaningful" and "Applications should be Trustworthy." They don't mention anything about the review process or who actually reviews the applications submitted. It could be one person who doesn't like specific things and judges applications with a particular bias. Then again, it could be an extremely lazy process that blindly allows applications that at least look 'OK', however it is defined. For all we know, there could be no review process at all and in the end, it's nothing more than a suped up advertising scheme. This brings us to the next point, fees.

There's a $375 non-refundable processing fee. Seriously? You have to pay a processing fee up-front with initial submission then wait for an email, for up to two weeks, telling you the application will be processed and at that point you'll have to provide more information for the verification process. More information, as said in the FAQ, includes screenshots and detailed descriptions of the application's components and parts. If your application does not pass this verification process then the processing fee you paid is lost. Furthermore, if Facebook finds any issue with your application regarding it's terms of service you may face a penalty. The best part about it is that your verified status lasts only up to one year. Each year you have to go through the same process, all over again.

The Application Verification Process is nothing more than a cheesy marketing scheme. Instead of innovating and paving the way in the social space, they've resorted to accepting bribes for product placements. I'm disappointed in you, Facebook.