bizo developer blog

work at Bizo (looking for some good engineers)

2012-01-30T19:38:00.001-08:00

We’re a small, disciplined team that gets a lot done. Our platform processes billions of page views monthly and 100s of terabytes of data so we have lots of fun problems to tackle. We believe in teamwork and communication: comments, design reviews, code reviews for every change, weekly tech talks. We believe in giving developers ownership over projects. We believe Engineering is more than coding. We have fun and keep the beer fridge well stocked.

We have customers, are well funded and recently named the forth fastest growing private company in the San Francisco Bay Area.

We are looking for motivated problem solvers with an entrepreneurial / hacker spirit.

If you're a reader of this blog, you already know our technology stack. Some highlights: Scala, Java, Javscript, Ruby, AWS (pretty much every service), Hadoop/Hive, GWT, MongoDB, Solr, etc.

If you're interested, please apply on stackoverflow.

Using GenericUDFs to return multiple values in Apache Hive

2012-01-18T17:39:00.000-08:00

A basic user defined function (UDF) in Hive is very easy to write: you simply subclass org.apache.hadoop.hive.ql.exec.UDF and implement an evaluate method. We've previously written about this strategy, and it works well for most simple cases.

The first case where this breaks down is when you want to return multiple values from your UDF. For me, this often arises when we have serialized data stored in a single Hive field and want to extract multiple pieces of information from it.

For example, suppose we have a simple Person object (leaving out all of the error checking code):

case class Person(val firstName: String, val lastName: String)

object Person {
def serialize(p: Person): String = {
p.firstName + "|" + p.lastName
}

def deserialize(s: String): Person = {
val parts = s.split("|")
Person(parts(0), parts(1))
}
}

We want to convert a data table containing these serialized objects into one containing firstName and lastName columns.

create table input(serializedPerson string) ;

load data local inpath ... ;

create table output(firstName string, lastName string) ;

So, what should our UDF and query look like?

Using the previous strategy, we could create two separate UDFs:

insert overwrite table output
select firstName(serializedPerson), lastName(serializedPerson)
from input ;

Unfortunately, the two invocations will have to separately deserialize their inputs, which could be expensive in less trivial examples. It also requires writing two separate implementation classes whose only difference is which field to pull out of your model object.

An alternative is to use a GenericUDF and return a struct instead of a simple string. This requires using object inspectors to specify the input and output types, just like in a UDTF:

class DeserializePerson extends GenericUDF {
private var inputInspector: PrimitiveObjectInspector = _

def initialize(inputs: Array[ObjectInspector]): StructObjectInspector = {

this.inputInspector = inputs(0).asInstanceOf[PrimitiveObjectInspector]

val stringOI = PrimitiveObjectInspectorFactory.getPrimitiveJavaObjectInspector(STRING)

val outputFieldNames = Seq("firstName", "lastName")

val outputInspectors = Seq(stringOI, stringOI)

ObjectInspectorFactory.getStandardStructObjectInspector(outputFieldNames,

outputInspectors)

}

def getDisplayString(children: Array[String]): String = {

"deserialize(" + children.mkString(",") + ")"

}

def evaluate(args: Array[DeferredObject]): Object = {

val input = inputInspector.getPrimitiveJavaObject(args(0).get)

val person = Person.deserialize(input.asInstanceOf[String])

Array(person.firstName, person.lastName)

}

}

Here, we're specifying that we expect a single primitive object inspector as an input (error handling code omitted) and returning a struct containing two fields, both of which are strings. We can now use the following query:

create temporary function deserializePerson as 'com.bizo.udf.DeserializePerson' ;

insert overwrite table output
select person.firstName, person.lastName
from (
select deserializePerson(serializedPerson)
from input
) parsed ;

This query deserializes the person only once but gives you access to both of the values returned by the UDF.

Note that this method does not allow you to return multiple rows -- for that, you still need to use a UDTF.

Clustering of sparse data using python with scikit-learn

2012-01-13T15:14:00.000-08:00

Coming from a Matlab background, I found sparse matrices to be easy to use and well integrated into the language. However, when transitioning to python’s scientific computing ecosystem, I had a harder time using sparse matrices. This post is intended to help Matlab refugees and those interested in using sparse matricies in python (in particular, for clustering)

Requirements:

scikit-learn (2.10+)

numpy (refer to scikit-learn version requirements)
scipy (refer to scikit-learn version requirements)

Sparse Matrix Types:
There are six types of sparse matrices implemented under scipy:

bsr_matrix -- block sparse row matrix
coo_matrix -- sparse matrix in coordinate format
csc_matrix -- compressed sparse column matrix
csr_matrix -- compressed sparse row matrix
dia_matrix -- sparse matrix with diagonal storage
dok_matrix -- dictionary of keys based sparse matrix
lil_matrix -- row-based linked list sparse matrix

For more info see: (http://docs.scipy.org/doc/scipy/reference/sparse.html):

When to use which matrix:
The following are scenarios when you would want to choose one sparse matrix type over the another:

Fast Arithmetic Operation: csc_matrix, csr_matrix
Fast Column Slicing (e.g., A[:, 1:2]): csc_matrix
Fast Row Slicing (e.g., A[1:2, :]) csr_matrix
Fast Matrix vector products: csr_matrix, bsr_matrix, csc_matrix
Fast Changing of sparsity (e.g., adding entries to matrix): lil_matrix, dok_matrix
Fast conversion to other sparse formats: coo_matrix
Constructing Large Sparse Matrices: coo_matrix

Clustering with scikit-learn:

With the release of scikit-learn 2.10, one of the useful new features is the support for sparse matrices with the k-means algorithm. The following is how you would use sparse matrices with k-means:

Full Matrix to Sparse Matrix

Example-1

-----------------------------------------------------------------------------------------------

from numpy.random import random

from scipy.sparse import *

from sklearn.cluster import KMeans

# create a 30x1000 dense matrix random matrix.

D = random((30,1000))

# keep entries with value < 0.10 (10% of entries in matrix will be non-zero)

# X is a "full" matrix that is intrinsically sparse.

X = D*(D<0.10) # note: element wise mult

# convert D into a sparse matrix (type coo_matrix)

# note: we can initialize any type of sparse matrix.

# There is no particular motivation behind using

# coo_matrix for this example.

S = coo_matrix(X)

labeler = KMeans(k=3)

# convert coo to csr format

# note: Kmeans currently only works with CSR type sparse matrix

labeler.fit(S.tocsr())

# print cluster assignments for each row

for (row, label) in enumerate(labeler.labels_):

print "row %d has label %d"%(row, label)

-----------------------------------------------------------------------------------------------

One of the issues with Example-1 is that we are constructing a sparse matrix from a full matrix. It will often be the case that we will not be able to fit a full (although intrinsically sparse) matrix in memory. For example, if the matrix X was a 100000x1000000000 full matrix, there could be some issues. One solution to this is to somehow extract out the non-zero entries of X and to use a smarter constructor for the sparse matrix.

Sparse Matrix Construction

In Example-2, we will assume that we have X's data stored on some file on disk. In particular, we will assume that X is stored in a csv file and that we are able to extract out the non-zero data efficiently.

Example-2

-------------------------------------------------

import csv
from scipy.sparse import *
from sklearn.cluster import KMeans

def extract_nonzero(fname):
"""
extracts nonzero entries from a csv file
input: fname (str) -- path to csv file
output: generator<(int, int, float)> -- generator
producing 3-tuple containing (row-index, column-index, data)
"""
for (rindex,row) in enumerate(csv.reader(open(fname))):
for (cindex, data) in enumerate(row):
if data!="0":
yield (rindex, cindex, float(data))

def get_dimensions(fname):
"""
determines the dimension of a csv file
input: fname (str) -- path to csv file
output: (nrows, ncols) -- tuple containing row x col data
"""
rowgen = (row for row in csv.reader(open(fname)))
# compute col size
colsize = len(rowgen.next())
# compute row size
rowsize = 1 + sum(1 for row in rowgen)
return (rowsize, colsize)

# obtain dimensions of data
(rdim, cdim) = get_dimensions("X.csv")

# allocate a lil_matrix of size (rdim by cdim)
# note: lil_matrix is used since we be modifying
# the matrix a lot.
S = lil_matrix((rdim, cdim))

# add data to S
for (i,j,d) in extract_nonzero("X.csv"):
S[i,j] = d

# perform clustering
labeler = KMeans(k=3)
# convert lil to csr format
# note: Kmeans currently only works with CSR type sparse matrix
labeler.fit(S.tocsr())

# print cluster assignments for each row
for (row, label) in enumerate(labeler.labels_):
print "row %d has label %d"%(row, label)

--------------------------------------------------

What to do when Sparse Matrices aren't supported:

When sparse matrices aren't supported, one solution is to convert the matrix to a full matrix. To do this, simply invoke the todense() method.

Hudson/Jenkins With RVM and PhantomJS

2012-01-13T12:04:00.000-08:00

Setting up Hudson/Jenkins to work RVM (Ruby Version Manager) and PhantomJS (for headless JavaScript testing) can be painful. This post will show you how to easily set them up on your own server.

RVM
At Bizo we have several projects that have dependencies on different versions of Ruby, mostly due to some projects relying on older gems which are incompatible with Ruby 1.9. Installing RVM on a dev machine is almost always a cinch but getting it to play nicely with your ci build server isn't quite so straightforward.

We run our Hudson server off of an Amazon EC2 instance. Our EC2 instances are started up with custom software, but it really boils down to executing a bash start up script. Assuming the Hudson user's $HOME is set to /var/lib/hudson, you can copy/paste the code below to install RVM for you. Otherwise just replace /var/lib/hudson below to the $HOME dir of your Hudson (or Jenkins) user.



# RVM
COMMANDS=$(cat <<EOS
bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)
echo "[ -s \"/var/lib/hudson/.rvm/scripts/rvm\" ] && source \"/var/lib/hudson/.rvm/scripts/rvm\" # loads RVM" > .bashrc

# ensure RVM is loaded
source ~/.bashrc

echo "Installing Ruby 1.9.2"
rvm install 1.9.2

rvm use 1.9.2


echo "Installing gems for Ruby: 1.9.2"
gem install bundler --no-rdoc --no-ri

# add additional ruby versions here

EOS
)
su - hudson -c "$COMMANDS"

Then in your Hudson build go to your project configuration and under "execute shell" you can invoke rvm and run your project like normal. Note* our version of Hudson doesn't automatically load .bashrc, so you might need to source it first to ensure RVM loads, ex:


source ~/.bashrc

# Pick our ruby version
rvm use 1.9.2


# Run your project... ex bundle install && rake test:units for a Rails project

PhantomJS
PhantomJS is our execution environment of choice for running JavaScript unit tests and setting it up to run on Hudson is actually quite easy.

Here is the necessary bash snippet to make it available for use in Hudson.


INSTALL_PATH= # wherever you want
wget http://phantomjs.googlecode.com/files/phantomjs-1.4.1-linux-x86-dynamic.tar.gz ${INSTALL_PATH}/phantomjs.tar.gz
# OR For 64bit machines wget http://phantomjs.googlecode.com/files/phantomjs-1.4.1-linux-x86_64-dynamic.tar.gz

mkdir ${INSTALL_PATH}/phantomjs
tar -zxvf ${INSTALL_PATH}/phantomjs.tar.gz -C ${INSTALL_PATH}

ln -s ${INSTALL_PATH}/phantomjs/bin/phantomjs /usr/local/bin/phantomjs

After running the script above you can invoke PhantomJS as "phantomjs" in the "execute shell" box inside your project configuration. You'll probably want your tests to fail with a non-0 exit status so the Hudson build will fail, if you use the Jasmine testing framework you can use our phantom-jasmine test runner on Github: https://github.com/jcarver989/phantom-jasmine

Interactive Hive sessions, Elastic MapReduce, and GNU screen

2012-01-13T10:56:00.000-08:00

One extremely annoying quality of using Hive interactively on EMR (or any other remote system) is that your sessions will die if you lose your connection to the server. Once this happens, your ssh session will end, terminating both your Hive session and any queries that may currently be running.

In most cases, this happens when I'm waiting for a query to execute and I need to move from one place to another, whether from my desk to a conference room or from the office to home. When I can predict (or know) that I'm going to lose my connection and just want to be able to reconnect to Hive later, the best option I've found is to run Hive inside of GNU screen.

I'm definitely a screen newbie, but there are really only three things you need to know:

1. As soon as you log in for the first time, install screen and start it up:

sudo apt-get install screen
screen
hive

2. When you're (temporarily) done interacting with Hive and want to stick it in the background, tell screen to detach Hive from your current session by pressing "Ctrl-a" then "d". You may now log out from your EMR node.

3. When you're ready to resume your Hive session, simply log back on and tell screen to reconnect to the most recent session:

screen -r

Screen does a whole lot of other stuff, but simply allowing graceful reconnection to Hive sessions is definitely worth the price of entry.

For comparison, some other things you could do to work around this problem are using nohup, suspending/putting the job in the background and using disown, or using an even more advanced tool like tmux.

How to Measure Latency Distribution using Amazon CloudWatch

2012-01-09T05:08:00.000-08:00

By default, web services hosted in AWS behind an Elastic Load Balancer have their response rates automatically tracked in CloudWatch. This lets you easily monitor the minimum, maximum, and average latency of your services.

Unfortunately, none of these metrics are very useful. Minimum and maximum latencies really measure outliers, and averages can easily obscure what's really going on in your services.

To get a better picture of the performance of our backend services, we explicitly have client services track the latency distribution of servers using CloudWatch custom metrics. Instead of having a single metric that measures the number of milliseconds taken by each request, we instead count the number of requests that take a particular number of milliseconds.

More specifically, for each service, we create eleven custom metrics representing the buckets 0-10ms, 11-20ms, 21-30ms, ... , 91-100ms, and 101+ms. (We actually have another set of buckets covering the 100-1000ms range at 100ms intervals, but we've found these to be less useful.) For each request, we then increment the counter for the bucket corresponding to the request's latency. Periodically, an automated process simply pulls the aggregated counts from CloudWatch into a Google Spreadsheet and graph the results in a stacked bar chart.

A stacked bar chart of response times for one of our services. Each layer represents a different response time bucket.

This gives us a nice visual representation of both the overall traffic levels as well as how many of them had response times below each threshold.

Our automated process also converts the totals into percentiles.

A stacked bar chart of response times as percentiles.

Google's chart tools support displaying only a vertical slice of the data, so we can easily show 90th, 95th, and 97.5th percentiles of our response times. This lets us easily see whether we're fulfilling performance requirements like "95% of all requests with response times below N ms."

Our slowest responses.

The combination of latency bucketing, Amazon CloudWatch, and Google Spreadsheets gives us a very lightweight way of tracking our server performance. The only additional overhead on our servers is a bit of logic to do local aggregation and push data into CloudWatch, and the only other moving part is a simple cron task that connects together the CloudWatch and Google Data APIs.

Get Your Company To Blog More With A Game

2011-12-19T13:54:00.000-08:00

At Bizo we try to blog fairly often. But writing blog posts with any degree of frequency at a startup is tough - there are often ten other important tasks that needed to be done yesterday. Finding the time to sit down and write a post when you have features to build, code to review and the occasional meeting is difficult to say the least.

We needed something that would encourage us to blog more frequently, and there's no better way to motivate a bunch of engineers than a game. So during our last Open Source Day we built a Blog Scoreboard that ranks authors based on the number of posts and comments they have. It's setup on our office big screen tv serving as constant reminder that Larry is by a landslide, the king of blogging. For a live demo of the scoreboard and to see just how much Larry is dominating us, checkout the live example here.

Currently we just optimize towards the volume of posts so the points are currently assigned as: 10 points per blog post and 1 point per comment. The weighting scheme is intentionally naive due since this had to be built in a few hours and will undoubtably change as time goes on

The best part about our scoreboard is that it is an open source sinatra (ruby) app, and it works with any Blogger blog! All you have to do is edit a few lines of YAML and you'll have your very own big screen blog scoreboard. You can grab the source code and install instructions on github: https://github.com/jcarver989/blog-scoreboard Happy blogging!

4 tips from the trenches of Amazon Elastic MapReduce and Hive

2011-12-15T13:58:00.000-08:00

Some things you can learn from others, and some things you can only learn from experience. In an effort to move some knowledge from the latter category into the former, here are four things we've learned the hard way while working with big data flows using Hive on Amazon's Elastic MapReduce:

1. To prevent data loss, make sure Hive knows if it owns the data.

All of our data processing follows a simple pattern: servers write logs to S3, and we use the basic EMR/S3 integration to read this data in our Hive scripts. A typical table definition could look something like this:

create external table sample_data(d map<string,string>)
comment 'data logs uploaded from servers'
row format delimited
fields terminated by '\004'
collection items terminated by '\001'
map keys terminated by '\002'
stored as textfile
location 's3://log-bucket/hive/data/files/sample_data/'
;

(The single map column is our way of supporting dynamic columns in Hive.)

The most easily overlooked part of this table definition is the "external" keyword, which tells Hive that the actual underlying data is managed by some other process. If you forget to add this keyword and later issue a "drop table" command, Hive will happily nuke all of your log files.

This can be especially troublesome while doing ad hoc analysis, as these usually involve interactive queries in the Hive console with a workflow that often involves an exploratory style of development that includes deleting and recreating tables.

One common use pattern where dropping tables appears is when running scripts that include a create statement. Normally, trying to recreate a table that already exists in Hive will cause an error, so a script may preemptively issue a drop command in case that table already exists. An alternative is to change the table definition to tell Hive to ignore create statements for preexisting tables:

create external table if not exists sample_data(d map<string,string>)

In the event that you accidentally forget to specify "external" in your table definition, you can add it later by altering the table:

alter table sample_data set tblproperties ('EXTERNAL'='TRUE') ;

Note that the capitalization in the table properties is significant. Additionally, this feature is only available in Hive 0.6.0+. (It was present but buggy prior to 0.6.0.)

2. Use a smart partitioning scheme to create flexible views of the data.

Hive supports partitioning, which can be a huge performance win when querying only a subset of the entire dataset. This is critical when the entire dataset consists of years of request information but you're only interested in analyzing one day of traffic. To take advantage of this, we upload our data into hourly paths in S3:

s3://log-bucket/log-name/year=${year}/month=${month}/day=${day}/hour=${hour}/${logfile}.log.gz

We would then load this data with the following statements:

create external table sample_data(d map<string,string>)
comment 'data logs uploaded from servers'
partitioned by (
year string,
month string,
day string,
hour string)
row format delimited
fields terminated by '\004'
collection items terminated by '\001'
map keys terminated by '\002'
stored as textfile
location 's3://log-bucket/log-name/'
;
alter table sample_data recover partitions ;

The year/month/day/hour partitions are now available to be used in select/where statements just like columns.

We could (and originally did) just use a single partition column for the entire date; however, using multiple partition columns allows Hive to complete ignore the presence of other partitions. The above statement will need to recover metadata for 8-9 thousand partitions of data, which (while less troublesome than not partitioning at all) will still require a lot of time and memory. Multiple partition columns lets us create (for example) a "view" of a single month:

create external table sample_data_2011_12(d map<string,string>)
comment 'data logs uploaded from servers'
partitioned by (
day string,
hour string)
row format delimited
fields terminated by '\004'
collection items terminated by '\001'
map keys terminated by '\002'
stored as textfile
location 's3://log-bucket/log-name/year=2011/month=12/'
;
alter table sample_data recover partitions ;

Now Hive only needs to recover 7-8 hundred partitions. We use a similar strategy for our daily reporting, which makes recovering partition data even faster.

The only time this scheme breaks down is when the report boundaries don't align with the partition boundaries. In these cases, you can still get the benefits of partitioning by manually adding the partition information to the table. For example, to do efficient queries on a week of data, we would replace the "recover partitions" statement with a sequence of statements like these (and tweak the table definition to use only a single partition column):

alter table sample_data add partition(day_hour=2011120600) location "s3://log-bucket/log-name/year=2011/month=12/day=06/hour=00/";

3. Use S3 multipart uploads for large objects.

While S3, Elastic MapReduce, and Hive theoretically allow you easily scale your data storage and analytics, we regularly run up against operational limits as our processing requirements grow. One surprising problem we recently ran into was S3 throttling EMR because our cluster was accessing the source data too quickly. After some back-and-forth with support, they suggested a workaround of uploading the source objects (which were multiple GB in size and created by a previous EMR job) with multi-part uploads.

Enabling multi-part uploads in EMR is simply a matter of flipping a configuration switch in the cluster configuration. When starting a cluster from the command line, simply add the following options (taken from the linked documentation):

elastic-mapreduce --create --alive \
--bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop \
--bootstrap-name "enable multipart upload" \
--args "-c,fs.s3n.multipart.uploads.enabled=true, \
-c,fs.s3n.multipart.uploads.split.size=524288000"

4. Spot instances can dramatically reduce your data costs.

As our data processing requirements have grown, so have our cluster sizes. We've found that we can reduce our costs by over 50% by using a mix of on-demand and spot instances.

Each Elastic MapReduce job can have up to three instance groups: master, core, and task. All data resides on the master and core instances, so jobs can use spot instances to add processing power and additional IO with very little risk. You can add these instances at startup by simply specifying a bid price:

elastic-mapreduce --create --alive --hive-interactive --hive-versions 0.7 \
--instance-group master --instance-type m1.xlarge --instance-count 1 \
--instance-group core --instance-type m1.xlarge --instance-count 9 \
--instance-group task --instance-type m1.xlarge --instance-count 90 --bid-price 0.25

For ad hoc queries, I've also found it easy to prototype my job on a subset of data using only the master and core instances, then add the task instances once I'm satisfied that the query is correct. New task instances are available as Hadoop workers as soon as they're added to the cluster, even if you're already running your MapReduce job. You can modify the size of the task group using the command-line client:

elastic-mapreduce --jobflow j-xxxxxxxxxxxxx --add-instance-group task --instance-type m1.xlarge --instance-count 90 --bid-price 0.25

elastic-mapreduce --jobflow j-xxxxxxxxxxxxx --set-num-task-group-instances 90

Note that while the size of the task group can be changed, you cannot modify the instance type or the bid price after the task group is created.

Promises in Javascript/Coffeescript

2011-12-13T11:55:00.000-08:00

This happens often. You’re humming along writing some awesome Javascript code. At first everything is neat and organized. Then you add a feature here, an AJAX call there and before your once lovely codebase has turned into callback spaghetti.

One scenario where this can become particularly nasty is when you have an arbitrary number of asynchronous tasks, one of which cant run until all the others have completed. Suppose for example you are creating a page that displays information on your organization’s engineering team and open source projects hosted on Github. This might require a few api calls to Github (eg. one to get the repositories, another to get the team members) that can be executed in parallel. We could just append the information to the DOM as each api call finishes, but it would be nice to avoid pieces of the site popping in at different times. Instead it would be ideal if we made the api calls to Github and only after both api calls had completed would we render the information to the DOM. The problem is that in Javascript trying to implement this can get quite messy.

A Bad Solution

The issue here is knowing when it is safe to execute the task that renders the Github information to the page. One strategy might be to store the results from each api call in an array and then wait for a set amount of time before drawing the page.


results = []

githubApiCallOne (response) -> 
  results.push(response)

githubApiCallTwo (response) ->
  results.push(response)

setTimeout(() ->
  drawPage(results)
, 5000)

This turns out to be a poor solution as we can either end up waiting too little, in which case the program would fail or we could set the timeout too high making our users wait longer than necessary to load our page.

Promises to The Rescue

Fortunately there is a better way to implement our Github page. Using a construct called a Promise (sometimes called a Future) allows us to elegantly handle these types of situations. Using promises we can turn our code into something like this:


Promise.when(
  githubApiCallOne(),
  githubApiCallTwo()
).then((apiCallOneData, apiCallTwoData) ->
  renderPage(apiCallOneData, apiCallTwoData)
)

The basic idea is that our async api calls will now return a promise object that functions much like an IOU – they can’t give us the results of the api call immediately but they (probably) can at some time in the future. The Promise.when method takes an arbitrary number of promise objects as parameters and then executes the callback in the “then” method once every promise passed to “when” has been completed.

To do this, our api calls would have to be modified to return promise objects, which turns out to be trivial. Such an implementation might look like so:


githubApiCallOne = () ->
  promise = new Promise()

  # async call
  ajaxGet("/repositories", (repository_data) ->
    # fulfill the promise when async call completes
    promise.complete(repository_data)
  )

  return promise

githubApiCallTwo = () ->
  promise = new Promise()

  ajaxGet("/users", (user_data) ->
    promise.complete(user_data)
  )

  return promise

The githubApiCallOne and githubApiCallTwo make their ajax calls but return a promise object immediately. Then when the AJAX calls complete, they can fulfill the promise objects by calling “complete” and passing in their data. Once both promise objects have been fulfilled the callback passed to Promise.then is executed and we render the page.With jQuery

The good news is if you’re already using jQuery you get Promises for free. As of jQuery 1.5 all the $.ajax methods (eg. $.get, $.post etc) return promises which allows you to do this:


promise1 = $.get "http://foo.com"
promise2 = $.post "http://boo.com"

$.when(promise1, promise2)
 .then (promise1Result, promise2Result) ->
  # do something with the data

What if I cant use jQuery?

Rolling a custom implementation of Promises isn’t recommended for production code but might be necessary if you write a lot of 3rd party Javascript and/or just want to try it for fun. Here’s a very basic implementation to get you started. Error handling, exceptions etc are left as an exercise to the reader.


class Promise
  @when: (tasks...) ->
    num_uncompleted = tasks.length 
    args = new Array(num_uncompleted)
    promise = new Promise()

    for task, task_id in tasks
      ((task_id) ->
        task.then(() ->
          args[task_id] = Array.prototype.slice.call(arguments)
          num_uncompleted--
          promise.complete.apply(promise, args) if num_uncompleted == 0
        )
      )(task_id)

    return promise
    
  constructor: () ->
    @completed = false
    @callbacks = []

  complete: () ->
    @completed = true
    @data = arguments
    for callback in @callbacks
      callback.apply callback, arguments

  then: (callback) ->
    if @completed == true
      callback.apply callback, @data
      return

    @callbacks.push callback

Sharp eyed readers might notice that the code inside the for loop in the Promise.when method looks a bit strange. You might notice that I’m wrapping the promise’s “then” method call inside of a self executing function that passes in the task_id variable. This funkiness is actually required due to the way that closures work in Javascript. If you attempt to reference the task_id without the self executing closure, you’ll actually get a reference to the task_id iterator instead of a copy – which means by the time your “then” methods execute the loop will have finished iterating and all the task_ids will share the same value! To get around this you have to create a new scope and pass in the iterator so we end up with a copy of the value instead of a reference.

And Finally an example using the supplied Promise class to prove it works:


delay = (string) ->
  promise = new Promise()
  setTimeout(() -> 
    promise.complete string
  ,200)
  return promise

logEverything = (fooData, barData, bazData) -> 
  console.log fooData[0], barData[0], bazData[0]

window.onload = () ->
  Promise.when(
    delay("foo"),
    delay("bar"),
    delay("baz")
  ).then logEverything

IAM for DevOps Organizations

2011-12-08T15:53:00.001-08:00

I gave a talk last month at the Advanced AWS Meetup on how we're using IAM.Slides are available here.

SVG Charts, Done Right.

2011-11-21T14:39:00.001-08:00

Here at Bizo we like many companies use charts visualizations in our products. They are fantastic tools for making the lives of our users easier. But unfortunately most charting libraries aren't that great. Some, like Google's visualizations have overly complex apis where you have to contort your data into intermediary objects and zero pad data. Others like flot have poor default options that require you to override almost every option just to get something that looks halfway presentable.

Our initial requirements for a charting library seemed pretty simple:

Good looking defaults
An api that didn't force a developer to do mental gymnastics just to load in data
IE support
No flash

But after looking around for a while it became apparent that none of the existing libraries really met our needs. Every library we looked at fell flat in one of these four requirements. So I set out to write a charting library from scratch during nights and weekends. Eleven days later version 1.0 of Raphy Charts was born. Raphy Charts (http://jcarver989.github.com/raphy-charts/) an html5/canvas charting library built ontop of Raphael (http://raphaeljs.com/) that includes the following features:

Great looking defaults (see example below)
Easy api that allows you to pass a normal 2d Javascript array of x,y points without the need to pad your x-labels.
IE support for version 7+
SVG or VML (older IEs) charts with no Flash.

Get Raphy Charts

5 minute web framework review : reading params

2011-11-01T18:28:00.001-07:00

Through various experiments, hackdays, conversations with other developers, etc. I've found myself experimenting with a few different web frameworks. The focus has been mostly simple webapps / simple REST services written in scala that return html or json. I thought it might be interesting to dive into some focused comparisons in a series of posts.

This is not an exhaustive comparison. I'm going to be focusing on the frameworks I've found the most interesting for my use cases lately, namely scalatra, play, and jersey.

For the first comparison, I want to focusing on reading query and path parameters. Parameter de-serialization has always been a pain. The web uses strings, and strings are messy. Are my required params specified? Are they the right types? Can I easily convert to the types my program expects? Do they pass my validation? etc.

Let's look at how each framework helps us deal with these common concerns.

jersey

I like jersey. It's a reference implementation of JSR-311: Java API for RESTful Web Services. It's also quite nice to work with in scala.

query parameters

With jersey, query parameters are simply specified as method parameters. Simple types are automatically converted, and it's easy to specify defaults. It will also automatically call converters for use with your own complex types. Unfortunately, you must use annotations to map query param names to method params.


 def doGet(@QueryParam("name") name: String,
           @QueryParam("count") @DefaultValue("2") count: Int): String = {
   "name: %s, count: %d\n".format(name, count)
 }

It works pretty much as you'd expect:


$ curl "http://localhost:8080/hello?name=larry&count=5"
name: larry, count: 5

If the types aren't correct, you'll get a 404:


curl "http://localhost:8080/hello?name=larry&count=a" -D -
HTTP/1.1 404 Not Found

path parameters

Path parameters in jersey work pretty much the same way as query params, i.e. typed, with default values and appearing as method arguments. Additionally you can do some simple validation using regular expressions. Their names and path location are specified when defining the route.


@Path("/hello/{userid}")
class Hello {
  def doGet(@PathParam("userid") id: Int)
}

Here's a different example showing some simple regex validation support:


@Path("/hello/{username: [a-zA-Z][a-zA-Z_0-9]+}")

And, again, if the path doesn't match your regex, or type, you will get a 404.

other params

There are also @CookieParam, @HeaderParam annotations for reading cookie and header values, as well as support for pulling in session or request variables using @Context or custom annotations (e.g. I've created @IpAddress for pulling in the ip).

overall thoughts

I really like the automatic de-serialization and type conversion, and having the framework handle errors for incompatible parameters automatically.

I also like the POJO mindset. It's just a function with arguments like any other. All else being equal, this makes testing in any framework super easy.

The annotations do seem a little noisy, especially having to specify the parameter name. I think we can do better.

scalatra

scalatra is also very nice for simple apps, and I've quickly become a fan of scalate which it uses for templating.

When it comes to dealing with parameters though, it feels like a step back. Everything is strings. The fact that it's scala makes it easier to deal with, but it does feel like the framework could go a little further to help you out.

query parameters

To read query parameters, you use the params method from ScalatraServlet. params is a MultiMapHeadView[String, String]. So yes, you are back to dealing with Strings (or a Seq[String] if using multiParams).

E.g.


get("/hello") {
  val name:String = params.getOrElse("name", halt(400))
  val count:Int = params.getOrElse("count", "2").toInt

  "name: %s, count: %d\n".format(name, count)
}

Calling this path without a name will generate a 400, as expected:


$ curl "http://localhost:8080/hello" -D - 
HTTP/1.1 400 Bad Request

If you don't specify count, you will get the default of 2. However, if you specify a non-int, you'll get a 200 where the contents are the stack trace for the toInt call. Again, your validation is all manual -- if you want better type validation, it's up to you.


2:~ larry$ curl "http://localhost:8080/hello?name=larry&count=a" -D -
HTTP/1.1 200 OK
…
<p>
java.lang.NumberFormatException: For input string: "a"
</p>

path parameters

Path params work exactly the same way (including being accessed in params), and are named as part of your route:


get("/hello/:name/:count") {
  val name:String = params.getOrElse("name", halt(400))
  val count:Int = params.getOrElse("count", "2").toInt

  "name: %s, count: %d\n".format(name, count)
}


$ curl "http://localhost:8080/hello/larry/5"
name: larry, count: 5

overall thoughts

The manual de-serialization seems a little dated, and gets old quick. Scala does make it nicer than it would be in java, since you can do things like params.getOrElse("name", halt(400)), but I would like to see more.

I also miss the POJO mindset… when testing you need to do whatever additional setup is necessary to serialize your params as strings and stick them in a map.

I guess I also don't like that barring convention, there's no formal definition of what parameters you are expecting and what their types are - maybe you are calling params.get somewhere in the middle of your method..

play

play the framework feels a little heavy compared to jersey and scalatra, but if definitely shines when it comes to dealing with parameters.

query parameters

Query parameters in play are done really well. It's almost perfect.


def hello(name: String, count: Int = 2) = {
  "name: %s, count: %d\n".format(name, count)
}


$ curl "http://localhost:9000/hello?name=larry"
name: larry, count: 2

You can even use Option for parameters that may be available:


def hello(name: Option[String], count: Int = 2) = {
  "name: %s, count: %d\n".format(name.getOrElse("anon"), count)
}

One problem is that type conversion errors are silently ignored, and defaults will be used:


$ curl "http://localhost:9000/hello?name=larry&count=a"
name: larry, count: 2

Okay, so they're not really ignored. If you call Validation.hasErrors, it will return true, and youcan discover the error. This is the same mechanism you need to use to mark parameter as required:


def hello(name: String, count: Int = 2) = {
  Validation.required("name", name)
  if (Validation.hasErrors) {
    // handle error

path parameters

Path parameters work the same way. They're defined with placeholders in your route, and automatically passed in as the correct argument. In play, routes are defined external to your code, in a routes file. E.g.


GET /hello/{name} Application.hello

Our method looks the same as the first Query param example. Calling it looks like this:

 
$ curl "http://localhost:9000/hello/larry"
name: larry, count: 2

In the case of path parameters, we will get a 404 if missing the parameter (since it won't match our route).


$ curl "http://localhost:9000/hello/" -D -

HTTP/1.1 404 Not Found

overall thoughts

Overall I think parameters in play are done really well.

Like jersey, I really appreciate the POJO approach. play does it even better by eliminating the extraannotations and leveraging scala's default argument support.

Validation does seem a little clunky, though. It seems like more could be done there.

Advanced Amazon Web Services Meetup

2011-10-13T07:49:00.000-07:00

Bizo Engineering, with the help of Amazon Web Servies, has developed a new meetup in San Francisco that focuses on Advanced Amazon Web Services topics. The main concept is that there are many companies who have been operating on AWS for a number of years and have significant experience but do not have a forum to discuss more advanced issues and architectures.

The next meetup is on Tuesday October 18th at 7pm in San Francisco (more details). We have speakers from Bizo, Twilio and Amazon Web Services.

We look forward to meeting other AWS companies and sharing stories from the trenches. Hope to see you there!

the story of HOD : ahead of its time, obsolete at launch

2011-10-10T13:29:00.001-07:00

Last week we shut down an early part of the bizo infrastructure : HOD (Hadoop on Demand). I thought it might be fun to look back on this project a bit.

We've been using AWS as long as bizo has been around, since early 2008. Hadoop has always been a big part of that. When we first started, we were mostly using a shared hadoop cluster. This was kind of a pain for job scheduling, but also was mostly wasteful during off-peak hours… Thus, HOD was born.

From its documentation, "The goal of HOD is to provide an on demand, scalable, sandboxed infrastructure to run Hadoop jobs." Sound familiar? HOD was developed late September and October of 2008, and launched for internal use December 12th, 2008. Amazon announced EMR April of 2009. It's amazing how similar they ended up being… especially since we had no knowledge of EMR at the time.

Even though HOD had a few nice features missing from EMR, the writing was on the wall. For new tasks, we wrote them for EMR. We slowly migrated old reports to EMR when they needed changes, or we had the time.

Architecture

HOD borrowed quite liberally from the design of Alexa's GrepTheWeb.

Users submitted job requests to a controller which managed starting exclusive hadoop clusters (master, slaves, hdfs), retrieving job input from S3 to HDFS, executing the job (hadoop map/reduce), monitoring the job, storing job results, and shutting down the cluster on completion. Job information and status was stored in SimpleDB, S3 was used for job inputs and outputs, and SQS was used to manage the job workflow.

Job Definition

Jobs were defined as thrift structures:


 struct JobRequest {
   1: JobConf job = {},
   2: i32 requested_nodes = 4, // requested number of hadoop slaves
   3: string node_type = "m1.large", // machine size
   4: i32 requested_priority, // job priority
   5: string hadoop_dist, // hadoop version e.g. "0.18.3"
   6: set depends_on = [], // ids of job dependencies
   
   7: list on_success = [], // success notification
   8: list on_failure = [], // failure notification
   9: set flags = [] // everyone loves flags
 }

 struct JobConf {
   1: string job_name,
   2: string job_jar,  // s3 jar path
   3: string mapper_class,
   4: string combiner_class,
   5: string reducer_class,

   6: set job_input,  // s3 input paths
   7: string job_output,  // s3 output path (optional)
   
   8: string input_format_class,
   9: string output_format_class,
   
   10: string input_key_class,
   11: string input_value_class,
   
   12: string output_key_class,
   13: string output_value_class,
   
   // list of files for hadoop distributed cache
   14: set user_data = [],
   
   15: map other_config = {}, // passed directly JobConf.set(k, v)
 }

You'll notice that dependencies could be specified. HOD could hook up the output of 1 or more jobs into the input of a job and wouldn't run the job until all of its dependencies have successfully completed.

User Interaction

We had a user program, similar to emr-client that helped construct and job jobs, e.g.:


JOB="\
 -m com.bizo.blah.SplitUDCMap \
 -r com.bizo.blah.UDCReduce \
 -jar com-bizo-release:blah/blah/blah.jar
 -jobName blah \
 -i com-bizo-data:blah/blah/blah/${MONTH} \
 -outputKeyClass org.apache.hadoop.io.Text \
 -outputValueClass org.apache.hadoop.io.Text \
 -nodes 10 \
 -nodeType c1.medium \
 -dist 0.18.3
 -emailSuccess larry@bizo.com \
 -emailFailure larry@bizo.com \
"

$HOD_HOME/bin/hod_submit $JOB $@

There was also some nice support for looking at jobs, either by status or by name:As well as support for viewing job output, logs, counters, etc.

Nice features

We've been very happy users of Amazon's EMR since it launched in 2009. There's nothing better than systems you don't need to support/maintain yourself! And they've been really busy making EMR more easy to use and adding great features. Still, there are a few things I miss from HOD.

Workflow support

As mentioned, HOD had support for constructing job workflows. You could wire up dependencies amount multiple jobs. E.g. here's an example workflow (also mentioned in this blog previously under hadoop-job-visualization).

It would be nice to see something like this in EMR. For really simple workflows, you can sometime squeeze them into a single EMR job as multiple steps, but that doesn't always make sense and isn't always convenient.

Notification support

HOD supported notifications directly. Initially just email notifications, but there was a plugin structure in place with an eye towards supporting HTTP endpoint, and SQS notifications.

Yes, this is possible by adding a custom EMR job step at the end that checks the status of itself and sends an email/failure notification… But, c'mon, why not just build in easy SNS support? Please?

Counter support

Building on that, HOD had direct support/understanding for hadoop counters. When processing large volumes of data, they become really critical in tracking the health of your reports over time. This is something I really miss. Although, it's less obvious how to fold this in with Hive jobs, which is how most of our reports are written these days.

Arbitrary Hadoop versions

HOD operated with straight hadoop, so it was possible to have it install an arbitrary version/package just by pointing it to the right distribution in S3.

Since Amazon isn't directly using a distribution from the hadoop/hive teams, you need to wait for them to apply their patches/changes and can only run with versions they directly support. This has mostly been a problem with Hive, which moves pretty quickly.

It would be really great if they could get to a point where their changes have been folded back into the main distribution.

Of course, this is probably something you can do yourself, again with a custom job step to install your own version of Hive… Still, they have some nice improvements, and again, it would be nice if it were just a simple option to the job.

The Past / The Future

Of course, HOD wasn't without its problems :). It's become a bear to manage, especially since we pretty much stopped development / maintenance (aside from rebooting it) back in 2009. It was definitely with a sigh of relief that I pulled the plug.

Still, HOD was a really fun project! It was an early project for me at Bizo, and it was really amazing how easy it was to write a program that starts up machines! and gets other programs installed and running! Part of me wonders if there isn't a place for an open source EMR-like infrastructure somewhere? Maybe for private clouds? Maybe for people who want/need more control? Or for cheapskates? :) (EMR does cost money).

Or maybe HOD v2 is just some wrappers around EMR that provides some of the things I miss : workflow support, notifications, easier job configuration…

Something to think about for that next hack day :).

Writing better 3rd party Javascript with Coffeescript, Jasmine, PhantomJS and Dependence.js

2011-10-05T17:07:00.000-07:00

Here at Bizo we recently underwent a major change to our Javascript tags that’s used in our free analytics product (http://www.bizo.com/marketer/audience_analytics). Our code gets loaded by millions of visitors each month across thousands of web sites; so our Javascript has to run reliably in just about any browser on any page.

The Old Javascript:

Unfortunately our codebase had accumulated about three years of cruft,
resulting in a single monolithic Javascript file. The file contained
a single closure, over 600 lines long, with all kinds of edge cases,
some of which no longer existed. Since you can’t access anything in a
closure from the outside, writing unit tests was nearly impossible and
the code had suffered as a result.

That’s not to say we had no tests – there were several selenium tests
that tested functionality at a high level. The problem however was
making the required changes was going to be a time consuming (and
somewhat terrifying) process. The selenium tests provided a very slow
testing feedback loop and debugging a large closure comes with it’s
own challenges. If it’s scary changing your production code, then
you’re doing something wrong.

Modularity and Dependency Management

So we decided to do a complete overhaul and rewrite our Javascript
tags in CoffeeScript (smaller code base with clearer code). The
biggest problem the original code had was that it wasn’t modular and
thus difficult to test. Ideally we wanted to split our project into
multiple files that we could unit test. To do this we needed some
kind of dependency management system for Javascript, which in 2011
surprisingly isn’t standardized yet. We looked at several projects but
none of them really met our needs. Our users are quite sensitive about
the number of http requests 3rd party Javacript makes so solutions
that load several scripts in parallel weren’t an option (ex.
Requirejs). Others like Sprockets were close but didn’t quite support
everything we needed.

We ended up writing Dependence.js, a gem to manage our Javascript
dependencies. Dependence will compile all your files in a module into
a single file. Features include javascript and/or Coffeescript
compilation, dependency resolution (via a topological sort your
dependency graph), allowing you to use an “exports” object for your
modules interface, and optional compression using Google’s Closure
compiler. Check it out on github:
(http://github.com/jcarver989/dependence.js)

Fast Unit testing with Phantom.js

Another way we were looking to improve our Javascript setup was to
have a comprehensive suite of unit tests. After looking at several
possibilities we settled on using the Jasmine test framework
(http://pivotal.github.com/jasmine/) in conjunction with PhantomJS (a
headless webkit browser). So far using Jasmine and PhantomJs together
has been awesome. As our Javascript is inherently DOM coupled, each of
our unit tests executes in a separate iframe (so each test has its own
separate document object). 126 unit tests later the entire suite runs
locally in about 0.1 seconds!

Functional Testing with Selenium

Our functional tests are still executed with Selenium webdriver.
Although there are alternative options such as HtmlUnit, we wanted to
test our code in real browsers and for this Selenium is still the best
option around. A combination of capybara and rspec make for writing
functional tests with a nicer api than then raw selenium. A bonus is
that capybara allows you to swap out selenium in favor of another
driver should we ever want to switch to something else. Lastly a
custom gem for creating static html fixtures allows us to
programmatically generate test pages for each possible configuration
option found in our Javascript module. You can find that here:
(http://github.com/jcarver989/js-fixtures).

Wrapping up

The new code is far more modular, comprehensively tested and way
easier to extend. Overall working with Dependence.js, CoffeeScript,
PhantomJs, Capybara, Rspec and Selenium has been a workflow that works
great for us. If you have a different workflow that you like for
Javascript projects, let us know!

Cloudwatch metrics revisited

2011-08-24T13:58:00.001-07:00

In a previous post, I discussed our initial usage of cloudwatch custom metrics. Since then, we've added more metrics and changed how we're recording them, so I thought it might be helpful to revisit the topic.

Metric Namespaces

Initially we had a single namespace per application. We decided that the stage should be included in the namespace. E.g. api-web-prod, api-web-dev. It seems to make sense to keep metrics from different stages completely separate, especially if you are using them for alerting or scaling events.

Metric Regions

When we started, we were logging all metrics to us-east (may have been a requirement of the beta?). Going forward, it made sense to log to the specific region where the events occurred. There's a little more work if you want to aggregate across regions, but it matches the rest of our infrastructure layout better. Also, if you want to use metrics for auto-scaling events, it's a requirement.

Dropping InstanceId dimensions (by default)

This is something we are currently working on rolling out. When we first started logging events, we would include a metric update tagged with the InstanceId. This mirrors how the built-in AWS metrics work. It seemed like it would be useful to be able to "drill down" when investigating an issue, i.e. the Maximum CPU utilization in this group is at 100%, okay, which instance is it?

In practice, we have started to question the utility versus cost, especially for custom metrics. When you run large services with auto-scaling, you end up generating a lot of metrics for very transient instances. Since the cost structure is based on the number of unique metrics used, this can really add up.

For some numbers, looking at the output of mon-list-metrics in us-east-1 only, we have 31,888 metrics with an InstanceId dimension! That's just for the last 2 weeks. If we were paying for all of those (luckily most of them are for built-in metrics), it would cost us $15k for those 2 weeks of metrics on very transient instances.

It has been useful to have InstanceId granularity metrics in the past, and in a perfect world maybe we'd still be collecting them, but with the current price structure it's just too expensive for most of our (auto-scaled) services.

Metric Dimensions revisited

When we first started using cloudwatch custom metrics, we would log the following dimensions for each event:

Version, e.g. 124021 (svn revision number)
Stage, e.g. prod
Region, e.g. us-west-1
Application, e.g. api-web
InstanceId, e.g. i-201345a

We can drop Stage and Region due to our namespace and region changes above. As mentioned, we've also decide to drop InstanceId for most of our services. This makes our current list of dimension defaults:

Version, e.g. 124021 (svn revision number)
Application, e.g. api-web

We're still tracking stage and region, based on the namespace or region, they just don't need to be expressed as dimensions.

More Metrics!

One of our developers, Darren, put together a JMX->Cloudwatch bridge. Each application can express which JMX stats it would like to export via a JSON config file. Here's a short except that will send HeapMemoryUsage to cloudwatch every 60 seconds:


  {
    "objectName" : "java.lang:type=Memory",
    "attribute" : "HeapMemoryUsage",
    "compositeDataKey" : "used",
    "metricName" : "HeapMemoryUsage",
    "unit" : "Bytes",
    "frequency" : 60,
  },

I'm sure the list will grow, but some of the metrics we've found most useful so far:

NonHeapMemoryUsage
HeapMemoryUsage
OpenFileDescriptorCount
SystemLoadAverage
ThreadCount

I'm hoping Darren will describe the bridge in more detail in a future post. It's made it really easy for applications to push system metrics to cloudwatch.

Of course, we're also sending a lot of application specific event metrics.

Homegrown interfaces

The AWS cloudwatch console is really slow. It also only seems like it will load 5,000 metrics. Our us-east "AWS/EC2" namespace alone has 28k metrics. Additionally, you can only view metrics for a single region at a time. We just haven't had a lot of success with the web console.

We've been relying pretty heavily on the command line tools for investigation, which can be a little tedious.

We've also written some scripts that will aggregate daily metrics for each app and insert them into a google docs spreadsheet to help track trends.

For our last hack day, I started working on a (very rough!) prototype for a custom cloudwatch console.

The app is written using play (scala) with flot for the graphs.

It heavily caches the namespace/metric/dimension/value hierarchies, and queries all regions simultaneously. It certainly feels much faster than the built-in console.

It's great just being able to quickly graph metrics by name, but my main motivation for this console was to provide a place where we could start to inject some intelligence about our metrics. The cloudwatch interface has to be really generic to support a wide range of uses/metrics. For our metrics, we have a better understanding of what they mean and how they're related. E.g. If the ErrorCount metric is high, we know which other metrics/dimensions can help us drill down and find the cause. I'm hoping to build those kinds of relationships into this dashboard.

Summary

So that's how we're currently using cloudwatch at bizo. There are still some rough edges, but we've been pretty happy with it. It's really easy to log and aggregate metric data with hardly any infrastructure.

I'd love to hear any other experiences, comments, uses people have had with cloudwatch.

Report delivery from Hive via Google Spreadsheets

2011-08-19T14:22:00.000-07:00

At Bizo, we run a number of periodically scheduled Hive jobs produce a high level summary as just a few (often, just one) row of data. In the past, we’ve simply used the same delivery mechanism as with larger reports; the output is emailed as a CSV file to the appropriate distribution list. This was less than ideal for a number of reasons:

Managing the distribution lists is difficult. We either needed to create a new list for each type of report, giving us a lot of lists to manage, or just send reports to a generic distribution list, resulting in a lot of unnecessary emails to people who weren’t necessarily interested in the report.

Handling the historical context is manual; the report needs to pull in past results to include in the output or recipients of the output need to find older emails to see trends appear.

Report delivery required an additional step in the job workflow outside of the Hive script.

With the GData storage handler, we now just create a Google Spreadsheet, add appropriate column headers, and do something like this in our script:


add jar gdata-storagehandler.jar ;

create external table gdata_output(
  day string, cnt int, source_class string, source_method string, thrown_class string
)
stored by 'com.bizo.hive.gdata.GDataStorageHandler'
with serdeproperties (
  "gdata.user" = "user@bizo.com",
  "gdata.consumer.key" = "bizo.com",
  "gdata.consumer.secret" = "...",
  "gdata.spreadsheet.name" = "Daily Exception Summary",
  "gdata.worksheet.name" = "My Application",
  "gdata.columns.mapping" = "day,count,class,method,thrown"
)
;

This appends whatever data is written to the table to the specified spreadsheet.

The source code is available here. If you’re running your jobs on Amazon’s Elastic MapReduce, you can access the storage handler by adding the following line to your Hive script:


add jar s3://com-bizo-public/hive/storagehandler/gdata-storagehandler-0.1.jar ;

Note that the library only supports 2-legged OAuth access to Google Apps for Domains, which needs to be enabled in your Google Apps control panel.

Bizo dev team @ TechShopSF

2011-08-12T16:15:00.001-07:00

Every quarter we have an "all hands" week, where the entire company comes to SF (the Bizo team is spread out across the country).

As part of this, we typically spend a day as a development team going over previous accomplishments and upcoming projects, as well as discussing our development process, architecture, etc.

We also spend some time making cool stuff! Last time around we had an internal Arduino workshop. Each developer got an Arduino and various components, and we went through a bunch of exercises from Getting Started with Arduino. We ended the day getting Wii controllers hooked up to our Arduinos (can't beat that).

This time around, we decided to head over to the SF Techshop and learn how to screen print.

We ended up with some great shirts:

They use a really cool process there, where you use a vinyl cutter to create a stencil for your artwork, which you can then just apply to your screen.

It was a lot of fun, and I think we all learned a lot. Special thanks to our instructor, Liz, as well as Devon at TechShop for helping us get this set up.

Check out some more shirts in this photo set.

MongoSF follow-up and contest winners

2011-06-01T15:02:00.001-07:00

Thanks for stopping by and meeting the Bizo engineering team atMongoSF. We had a great time meeting everyone at the conference andthe after-party. Stay in touch!As part of our conference sponsorship, we were able to include a small card in the conference bags. We are hiring, so we decided to use the space to talk a bit about the engineering team. On the back of the card, we included a small puzzle (click though for the full version):If you successfully completed the puzzle, you landed here, with a chance to enter some pirate jokes and win an Amazon gift card.Congratulations to the winners:

Y. Wayne H.
Huy H.
Dan N.

Like I said, we are hiring! We're looking for smart, motivated people who get stuff done. To learn more about the team, check out our mini engineering team site. If you would like to talk more, please get in touch!Finally, here are some of our favorite pirate jokes from the contest:

How does a pirate do calculus?: By taking a derivative with respect to Arrrrrrrrrrrrrrrrrrrr!!!
Where is the hidden treasure map of Silicon Valley?: Legend has it that Captain Zukarrrrburg hides it in his Subvarrrrsion Reparrrrsitory ol' matey!

Bizo @ MongoSF

2011-05-22T16:48:00.001-07:00

This Tuesday, May 24th the Bizo dev team will be attending the MongoSF conference. Hope to see you there. We're also sponsoring the after-party at Oz Lounge. Stop by, say "hello," and let us buy you a drink!

Cloudwatch custom metrics @ Bizo

2011-05-19T11:44:00.001-07:00

Now that Cloudwatch Custom Metrics are live, I wanted to talk a bit about how we're using them here at Bizo.We've been heavy users of the existing metrics to track requests/machine counts/latency, etc. as seen here. We wanted to start tracking more detailed application-specific metrics and were excited to learn about the beta custom metric support.

Error Tracking

The first thing we decided to tackle tracking were application errors. We were able to do this across our applications pretty much transparently by creating a custom java.util.logging.Handler. Any application log message that crosses the specified level (typically SEVERE, or WARNING) will be logged to cloudwatch.For error metrics, we use "ErrorCount" as the metric name, with the following dimensions:

Version, e.g. 124021 (svn revision number)
Stage, e.g. prod
Region, e.g. us-west-1
Application, e.g. api-web
InstanceId, e.g. i-201345a
class, e.g. com.sun.jersey.server.impl.application.WebApplicationImpl
exception, e.g. com.bizo.util.sdb.RuntimeDBException

Each application has its own cloudwatch namespace.This setup allows us to track error counts and rates across our applications/versions/regions, as well as get alerts when they reach specific thresholds.

Other Application Metrics

We expose a simple MetricTracker interface in our applications:


interface MetricTracker {
  void track(String metricName, Number value, List<Dimension> dimensions);
}

The implementation handles internally buffering/aggregating the metric data, and then periodically sending batches of cloudwatch updates.This allows developers to quickly add tracking for any metric they want. Note that with cloudwatch, there's no setup required, you just start tracking.

Wishlist

It's incredibly easy to get up and running with cloudwatch, but it's not perfect. There are a couple of things I'd like to see:

More data - CW only stores 2 weeks of data, which seems too short.
Faster - pulling data from CW (both command line and UI) can be really slow.
Better suport for multiple dimensions / drill down.

Cloudwatch does allow you to track against multiple dimensions, but it doesn't work as you'd probably expect. They're really treated as a single dimension. E.g. If you track against stage=prod,version=123, you can ONLY retrieve stats by querying against stage=prod,version=123. Querying against stage=prod only or version=123 only will not produce any results.You can work around this in your application, by submitting data for all permutations that you want to track against (our MetricTracker implementation works this way). It would be great if couldwatch supported this more fully, including being able to drill down/up in the UI.

Alternatives

We didn't invest too much time into exploring alternatives. It seems like running an OpenTSDB cluster, or something like statds would get you pretty far in terms of metric collection. That's only part of the story though, you would also definitely want alerting, and possible service scaling based on your metrics.

Overall Impressions

We continue to be excited about the custom metric support in Cloudwatch. We were able to get up and running very quickly with useful reports and alarms based on our own application metrics. For us, the clear advantage is that there's absolutely no setup, management or maintenance involved. Additionally, the full integration into alarms, triggers, and the AWS console is very key.

Future Use

We think that we may be able to get more efficient machine usage by triggering scaling events based on application metric data, so this is something we will continue to explore.It's easy to see how the error tracking we are doing can be integrated into a deployment system to allow for more automated rollout/rollback by tracking error rate changes based on version, so I definitely see us heading in that direction.

Synchronizing Stashboard with Pingdom alerts

2011-05-16T09:43:00.000-07:00

First, what's Stashboard? It's is an open-source status page for cloud services and APIs. Here's a basic example:

Alright, now what's Pingdom? It's a commercial service for monitoring cloud services and APIs. You define how to "ping" a service, and Pingdom periodically checks if the service is responding to the ping request and if not, sends email or SMS alerts.

See the connection? At Bizo, we've had Stashboard deployed on Google's AppEngine for a while but we were updating the status of services manually -- only when major outages happened.

Recently, we've been wanting for something more automated and so we decided to synchronize Stashboard status with Pingdom's notification history and came out with the following requirements:

Synchronize Stashboard within 15 minutes of Pingdom's alert.
"Roll-up" several Pingdom alerts into a single Stashboard status (i.e., for a given service, we have several Pingdom alerts covering different regions around the world but we only want to show a single service status in Stashboard)
If any of the related Pingdom alerts indicate a service is currently unavailable, show "Service is currently down" status.
If the service is available but there have been any alerts in the past 24 hours, show "Service is experiencing intermittent problems" status.
Otherwise, display "Service is up" status.

There are several ways we could have implemented this. We initially thought about using AppEngine's Python Mail API but decided against it since we're not familiar enough with Python and we didn't want to customize Stashboard from the inside. We ended up doing an integration "from the outside" using a cron job and a Ruby script that uses the stashboard and the pingdom-client gems.

It was actually pretty simple. To connect to both services,

require 'pingdom-client'
require 'stashboard'

pingdom = Pingdom::Client.new pingdom_auth.merge(:logger => logger)

stashboard = Stashboard::Stashboard.new(
  stashboard_auth[:url],
  stashboard_auth[:oauth_token],
  stashboard_auth[:oauth_secret]
)

then define the mappings between our Pingdom alerts and Stashboard services using a hash of regular expressions,

# Stashboard service id => Regex matching pingdom check name(s)
services = {
  'api' => /api/i,
  'analyze' => /analyze/i,
  'self-service' => /bizads/i,
  'data-collector' => /data collector/i
}

and iterate over all all Pingdom alerts and for each mapping determine if the service is either up or has had alerts in the past 24 hours,

up_services = services
warning_services = {}

# Synchronize recent pingdom outages over to stashboard
# and determine which services are currently up.
pingdom.checks.each do |check|
  service = services.keys.find do |service|
    regex = services[service]
    check.name =~ regex
  end
  next unless service
  
  # check if any outages in past 24 hours
  yesterday = Time.now - 24.hours
  recent_outages = check.summary.outages.select do |outage|
    outage.timefrom > yesterday || outage.timeto > yesterday
  end
  
  # synchronize outage if necessary
  recent_events = stashboard.events(service, "start" => yesterday.strftime("%Y-%m-%d"))
  recent_outages.each do |outage|
    msg = "Service #{check.name} unavailable: " +
    "#{outage.timefrom.strftime(TIME_FORMAT)} - #{outage.timeto.strftime(TIME_FORMAT)}"
    unless recent_events.any? { |event| event["message"] == msg }
      stashboard.create_event(service, "down", msg)
    end
  end
  
  # if service has recent outages, display warning
  unless recent_outages.empty?
    up_services.delete(service)
    warning_services[service] = true
  end

  # if any pingdom check fails for a given service, consider the service down.
  up_services.delete(service) if check.status == "down"
end

Lastly, if any services are up or should indicate a warning then we update their status accordingly,

up_services.each_key do |service|
  current = stashboard.current_event(service)
  if current["message"] =~ /(Service .* unavailable)|(Service operational but has experienced outage)/i
    stashboard.create_event(service, "up", "Service operating normally.")
  end
end

warning_services.each_key do |service|
  current = stashboard.current_event(service)
  if current["message"] =~ /Service .* unavailable/i
    stashboard.create_event(service, "warning", "Service operational but has experienced outage(s) in past 24 hours.")
  end
end

Note that any manually-entered Stashboard status messages will not be changed unless they match any of the automated messages or if there is a new outage reported by Pingdom. This is intentional to allow overriding automated updates if for any reason, some kind of failure isn't accurately reported.

Curious about what the end result looks like? Take a look at Bizo's status dashboard.

When you click on a specific service, you can see individual outages,

We hope this is useful to somebody out there... and big thanks to the Stashboard authors at Twilio, Matt Todd for creating the pingdom-client gem and Sam Mulube for the stashboard gem. You guys rule!

PS: You can download the full Ruby script from https://gist.github.com/975141.

How Bizo survived the Great AWS Outage of 2011 relatively unscathed...

2011-04-21T12:52:00.000-07:00

The twittersphere, techblogs and even some business sites are a buzz with the news that the US East Region of AWS has been experiencing a major outage. This outage has taken down some of the most well known names on the web. Bizo's infrastructure is 100% AWS and we support 1000s of publisher sites (including some very well know business sites) doing billions of impressions a month. Sure, we had a few bruises early yesterday morning when the outage first began, but soon after then we've been operating our core, high volume services on top of AWS but without the East region.

Here is how we have remained up despite not having a single ops person on our engineering team:

1) Our services are well monitored
We rely on pingdom for external verifcation of site availability on a world wide basis. Additionally, we have our own internal alarms and dashboards that give us up to the minute metrics such as request rate, cpu utilization etc. Most of this data comes from AWS Cloudwatch monitoring but we also track error ratesand have alarms setup to alert us when these rates change or go over a certain threshold.

2) Our services have circuit breakers between remote services that trip when other services become unavailable and we heavily cache data
When building our services, we always assume that remote services will fail at some point. We've spend a good deal of time investing in minimizing the domino effect of a failing remote service. When a remote service becomes unavailable the caller detects this and will go into tripped mode occasionally retrying with backoffs. Of course we also rely on caching read-only data heavily and are able to take advantage of the fact that the data needed for most of our services does not change very often.

3) We utilize autoscaling
One of the promises of AWS is the ability to start and stop more servers based on traffic and load. We've been using autoscaling since it was launched and it worked like a charm. You can see the instances starting up based on the new load in the US West region as traffic was diverted over from US East.

(all times UTC)

4) Our architecture is designed to let us funnel traffic around an entire region if necessary
We utilize Global Load Balancing to direct traffic to the closest region based on the end-user's location. For instance, if a user is in California, wedirect their traffic to the US West region. This was extremely valuable in keeping us fully functioning in the face of a regional outage. When we finally decided that the US East region was going to cause major issues, switching all traffic to US West was as easy as clicking a few buttons. You can see how the requests transitioned over quickly after we made the decision. (By the way, quick shout-out to Dynect who is our GSLB service provider. Thanks!)

(all times UTC)

Bumps and Bruises
Of course we didn't escape without sustaining some issues. We'll do another blog post on some of the issues we did run into but they were relatively minor.

Conclusion
After 3 years running full time on AWS across 4 regions and 8 availability zones we design our systems with the assumption that failure will happen and it helped us come through this outage relatively unscathed.

Command Query Responsibility Segregation with S3 and JSON

2011-04-18T09:44:00.000-07:00

We recently tackled a problem at Bizo where we wanted to decouple our high-volume servers from our MySQL database.

While considering different options (NoSQL vs. MySQL, etc.), in retrospect we ended up implementing a SOA-version of the Command Query Separation pattern (or Command Query Responsibility Segregation, which is services/messaging-specific).

Briefly, in our new approach, queries (reads) use an in-memory cache that is bulk loaded and periodically reloaded from a snapshot of the data stored as JSON in S3. Commands (writes) are HTTP calls to a remote JSON API service. MySQL is still the authoritative database, we just added a layer of decoupling for both reads and writes.

This meant our high-volume servers now have:

No reliance on MySQL availability or schema
No wire calls blocking the request thread (except a few special requests)

The rest of this post explains our context and elaborates on the approach.

Prior Approach: Cached JPA Calls

For context, our high-volume servers rely on configuration data that is stored in a MySQL database. Of course, the configuration data that doesn’t have to be absolutely fresh, so we’d already been using caching to avoid constantly pounding the database for data that rarely changes.

There were several things we liked about this approach:

We use Amazon RDS for the MySQL instance, which provides out-of-the-box backups, master/slave configuration, etc., and is generally a pleasure to use. We enjoy not running our own database servers.
We also have several low-volume internal and customer-facing web applications that maintain the same data and are perfectly happy talking to a SQL database. They are normal, chatty CRUD applications for which the tool support and ACID-sanity of a SQL database make life a lot easier.

That being said, we wanted to tweak a few things:

Reduce the high-volume servers’ reliance on MySQL for seeding their cache.
Although RDS is great, and definitely more stable than our own self-maintained instances would be, there are nonetheless limits on its capacity. Especially if one of our other application misbehaves (which has never happened…cough), it can degrade the MySQL instance to the point of negatively affecting the high-volume servers.
Reduce cache misses that block the request thread.
Previously, configuration data (keyed by a pre-request configuration id) was not pulled into cache until it was needed. The first request (after every cache flush) would reload the data for it’s configuration id from MySQL and repopulate the cache.
While not initially a big deal, as Bizo has grown, we’re now running in multiple AWS regions, and cache misses require a cross-region JDBC call to fetch their data from the MySQL server running in us-east.

Illustrated in code, our approach had, very simplified, been:

class TheServlet {
  public void doGet() {
    int configId = request.getParameter("configId");
    Config config = configService.getConfig(configId);
    // continue processing with config settings
  }
}

class ConfigService {
  // actually thread-safe/ehcache-managed, flushed every 30m
  Map<Integer, Config> cached = new HashMap<Integer, Config>();

  public Config getConfig(int configId) {
    Config config = cached.get(configId);
    if (config == null) {
      // hit mysql for the data, blocks the request thread
      config = configJpaRepository.find(configId);
      // cache it
      cached.put(configId, config);
    }
    return config;
  }
}

Potential “Big Data” Approaches

Given our primary concern was MySQL being a single point of failure, we considered moving to a new database platform, e.g. SimpleDB, Cassandra, or the like, all of which can scale out across machines.

Of course, RDS’s master/slave MySQL setup already reduces its risk of single machine point of failure, but the RDS master/slave cluster as a whole is still, using the term loosely, a “single point”. Granted, with this very loose definition, there will always be some “point” you rely on–we just wanted one that we felt more comfortable with than MySQL.

Anyway, for NoSQL options, we couldn’t get over the cons of:

Having to run our own clusters (except for SimpleDB).
Having to migrate our low-volume CRUD webapps over to the new, potentially slow (SimpleDB), potentially eventually-consistent (Cassandra) NoSQL back-end.
Still having cache misses result in request threads blocking on wire calls.

Because of these cons, we did not put a lot of effort into researching NoSQL approaches for this problem–we felt it was fairly apparent they weren’t necessary.

Realization: MySQL is Fine, Fix the Cache

Of course, we really didn’t have a Big Data problem (well, we do have a lot of those, but not for this problem).

We just had a cache seeding problem. Specifically:

All of our configuration data can fit in RAM, so we should be able to bulk-load all of it at once–no more expensive, blocking wire calls on cache misses (basically there are no cache misses anymore).
We can load the data from a more reliable, non-authoritative, non-MySQL data store–e.g. an S3 snapshot (config.json.gz) of the configuration data.
The S3 file then basically becomes our alternative “query” database in the CQRS pattern.

When these are put together, a solution emerges where we can have a in-memory, always-populated cache of the configuration data that is refreshed by a background thread and results in request threads never blocking.

In code, this looks like:

class TheServlet {
  public void doGet() {
    // note: no changes from before, which made migrating easy
    int configId = request.getParameter("configId");
    Config config = configService.getConfig(configId);
    // continue processing with config settings
  }
}

class ConfigService {
  // the current cache of all of the config data
  AtomicReference<Map> cached = new AtomicReference();

  public void init() {
    // use java.util.Timer to refresh the cache
    // on a background thread
    new Timer(true).schedule(new TimerTask() {
      public void run() {
        Map newCache = reloadFromS3("bucket/config.json.gz");
        cached.set(newCache);
      }
    }, 0, TimeUnit.MINUTES.toMillis(30));
  }

  public Config getConfig(int configId) {
    // now always return whatever is in the cache--if a
    // configId isn't present, that means it was not in
    // the last S3 file and is treated the same as it
    // not being in the MySQL database previously
    Map currentCache = cached.get();
    if (currentCache == null) {
      return null; // data hasn't been loaded yet
    } else {
      return currentCache.get(configId);
    }
  }

  private Map reloadFromS3(String path) {
    // uses AWS SDK to load the data from S3
    // and Jackson to deserialize it to a map
  }
}

A Few Wrinkles: Real-Time Reads and Writes

So far I’ve only talked about the cached query/reads side of the new approach. We also had two more requirements:

Very (very) infrequently, a high-volume server will need real-time configuration data to handle a special request.
The high-volume servers occasionally write configuration/usage stats back to the MySQL database.

While we could have continued using a MySQL/JDBC connection for these few requests, this also provided the opportunity to build a JSON API in front of the MySQL database. This was desirable for two main reasons:

It decoupled our high-volume services from our MySQL schema. By still honoring the JSON API, we could upgrade the MySQL schema and the JSON API server at the same time with a much smaller, much less complicated downtime window than with the high-volume services talking directly to the MySQL schema.
The MySQL instance is no longer being accessed across AWS regions, so can have much tighter firewall rules, which only allow the JSON API server (that is within its same us-east region) access it.

The new setup looks basically like:

Scalatra Servlet Example

With Jackson and Scalatra, the JSON API server was trivial to build, especially since it could reuse the same JSON DTO objects that are also serialized in the config.json.gz file in S3.

As an example for how simple Jackson and Scalatra made writing the JSON API, here is the code for serving real-time request requests:

class JsonApiService extends ScalatraServlet {
  get("/getConfig") {
    // config is the domain object fresh from MySQL
    val config = configRepo.find(params("configId").toLong)
    // configDto is just the data we want to serialize
    val configDto = ConfigMapper.toDto(configDto)
    // jackson magic to make json
    val json = jackson.writeValueAsString(configDto)
    json
  }
}

Background Writes

The final optimization was realizing that, when the high-volume servers have requests that trigger stats to be written to MySQL, for our requirements, these writes aren’t critical.

This means there is no need to perform them on the request-serving thread. Instead, we can push the writes onto a queue and have it fulfilled by a background thread.

This generally looks like:

class ConfigWriteService {
  // create a background thread pool of (for now) size 1
  private ExecutorService executor = new ThreadPoolExector(...);

  // called by the request thread, won't block
  public void writeUsage(int configId, int usage) {
    offer("https://json-api-service/writeUsage?configId=" +
      configId +
      "&usage=" +
      usage);
    }
  }

  private void offer(String url) {
    try {
      executor.submit(new BackgroundWrite(url));
    } catch (RejectedExecutionException ree) {
      // queue full, writes aren't critical, so ignore
    }
  }

  private static class BackgroundWrite implements Runnable {
    private String url;

    private BackgroundWrite(String url) {
      this.url = url;
    }

    public void run() {
      // make call using commons-http to url
    }
  }
}

tl;dr We Implemented Command Query Responsibility Segregation

With changing only a minimal amount of code in our high-volume servers, we were able to:

Have queries (most reads) use cached, always-loaded data is that periodically reloaded from data snapshots in S3 (a more reliable source than MySQL)
Have commands (writes) sent from a background-thread to a JSON API that saves the data to MySQL and hides JDBC schema changes.

For this configuration data, and our current requirements, MySQL, augmented with a more aggressive, Command Query Separation-style caching schema, has and continues to work well.

For more reading on CQS/CQRS, I suggest:

Both the Wikipedia article and Martin Fowler’s CommandQuerySeparation, however they focus on CQS as applied to OO, e.g. side-effect free vs. mutating method calls.
For CQS applied to services, e.g. CQRS, Udi Dahan seems to be one of the first advocates of the term. Since then, CQRS even seems to have it’s own site and google group.

Crowdflower Trickery: dynamic tasks

2011-04-15T13:25:00.000-07:00

Here at Bizo, we often use crowdflower to improve the quality of our data. In doing so, we’ve come across some cool, but under-documented, tricks. One trick that we’ve particularly found useful is using liquid for designers to dynamically generate crowdflower tasks. Let us take a look at how to do this with a toy example.

Problem:
We are given a list of deserts from a bakery as shown in data.csv bellow . Our task is to determine the following:

1. If the desert is a cake, is it appropriate for:

Wedding events
Birthday events
Casual eating

2. If the desert is NOT a cake, is it appropriate for:

eating using hands
eating using forks
eating using spoons

file: data.csv


"id","name","desert_type"
0,"Peanut Butter Cake","cake"
1,"Strawberry Donut","donut"
2,"Cookies and crème ice cream cake","cake"
3,"Apple Strudle","strudle"
4,"Chocolate Pie","pie"
5,"Red Velvet Cake","cake"

Initial Approach:
One approach is to create two separate jobs: one which will ask workers questions relevant to cakes and one which will ask the workers questions relevant to non-cake deserts. To create these jobs we would first have to break up our data into two data files: one containing data only for cakes (see cakes.csv) and one for everything else (see non-cakes.csv). Then, we would have to specify the appropriate cml code for each job. For our example, the data and cml code will look like:

Job 1: Cake deserts

file: cake.csv


"id","name","desert_type"
0,"Peanut Butter Cake","cake"
2,"Cookies and crème ice cream cake","cake"
5,"Red Velvet Cake","cake"

file: cake.cml


Desert: {{name}}
<cml:checkboxes label="This desert appropriate for">
<cml:checkbox label="wedding events"/>
<cml:checkbox label="birthday events"/>
<cml:checkbox label="casual eating"/>
</cml:checkboxes>

Job 2: Non-cake deserts

file: non-cake.csv


"id","name","desert_type"
1,"Strawberry Donut","donut"
3,"Apple Strudle","strudle"
4,"Chocolate Pie","pie"

file: non-cake.cml

Desert: {{name}}

<cml:checkboxes label="This desert appropriate for">

<cml:checkbox label="eating using hands"/>

<cml:checkbox label="eating using forks"/>

<cml:checkbox label="eating using spoons"/>

</cml:checkboxes>

Approach using Liquid:

Using liquid, we can solve the same problem with just one job rather than two. To do so, we simply embed liquid logic tags into our cml code to dynamically display tasks to users -- the appropriate checkboxes will appear based on the "desert_type" field. Furthermore, we do not need to break up the data.csv file.

cml code

Desert: {{name}}
<cml:checkboxes label="This desert is appropriate for">
{% if desert_type=='cake' %}
<cml:checkbox label="wedding events"/>
<cml:checkbox label="birthday events"/>
<cml:checkbox label="casual eating"/>
{% else %}
<cml:checkbox label="eating using hands"/>
<cml:checkbox label="eating using forks"/>
<cml:checkbox label="casual using spoons"/>
{% endif %}
</cml:checkboxes>

Hive Unit Testing

2011-04-14T19:39:00.000-07:00

Introduction

Hive has become an extremely important component in our overall software stack. We have numerous ‘mission-critical’ reports that are generated using Hive and want to make sure we can apply our testing processes to Hive scripts in the same way that we apply them to other code artifacts.

A few weeks ago, I was tasked with finding an approach for unit testing our Hive scripts. To my surprise, a Google search for ‘Hive Unit Testing’ yielded relatively few useful results.

I wanted a solution that would allow us to test locally (vs. a solution that would require EMR). Where possible, I prefer local testing because it’s simpler, provides more immediate feedback, and doesn’t require a network.

After reading this post, you will (hopefully) know how to run Hive unit tests in your own environment.

The Approach

After performing some research, I decided on an approach that is part of the Hive project itself. At a high level, the solution works in the following way:

Start up an instance of the Hive CLI
Execute a Hive script (positive or negative case)
Compare the output (from the CLI) of the script compared to an expected output file
Rinse and repeat

The rest of this post discusses the specific steps required to get this solution running in your own environment.

Set up Hive Locally

The first step is to create some Ant tasks for setting up Hive locally. Here’s a snippet of Ant that shows how to do this:

You should now be able to execute ‘ant hive.init’ and have Hive available in the tools directory.

Generate test cases

The developer is responsible for providing the .q Hive files that represent the test cases. There is a code generation step that will create JUnit classes (one for positive test cases, one for negative test cases) given a set of .q files. The Ant snippet below shows how to generate the test classes:

Here are some notes about the key variables above:

hive.test.template.dir - the directory where the velocity templates are located for the code generation step.
target.hive.positive.query.dir - the directory where positive test cases are located.
target.hive.negative.query.dir - the directory where negative test cases are located.
hive.positive.results.dir - the directory where expected positive test results are located. The name of this file must be the name of the query file appened by ‘.out’. For example, if the test query file is named hive_test.q then the results file must be named hive_test.q.out.
hive.negative.results.dir - the directory where expected negative test results are located.
qfile - This variable should be specified if you want to generate a test class with a single test case. For example, if you have a test file named hive_test.q, then you would set the value of this property to hive_test (e.g. ant -Dqfile=hive_test hive.gen.test).
qfile_regex - Similar in functionality to qfile, this variable should be set to a regular expression that will match the test files that you want to generate tests for.

The test classes are generated from velocity template files. You can find examples of the template from the Hive codebase here: https://github.com/apache/hive/blob/trunk/ql/src/test/templates/TestCliDriver.vm
https://github.com/apache/hive/blob/trunk/ql/src/test/templates/TestNegativeCliDriver.vm

The above files can basically be used as-is, but you will need to provide your own Test Helper class, QTestUtil, and update its package location accordingly in the templates.

QTestUtil

QTestUtil contains code for:

starting up hive
executing a query file
comparing the results to expected results
running cleanup between tests
shutting down hive

You can find the one from the Hive project here: https://github.com/apache/hive/blob/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java

The main modifications you will want to make to this file are deletions as there is some Hive project specific set up code that you will not need in your environment.

Executing the tests

After you have generated the tests, you can execute them by creating a target with the junit task. Here is some sample Ant for doing this:

Conclusion

This post outlined a solution for unit testing Hive scripts. Another nice aspect of this approach that I failed to mention is that it’s based on JUnit so you can use your existing code coverage tools with it (we use Cobertura) to get coverage information when testing custom UDFs. Also, I should mention that I used Hive 0.6.0 when putting this together.

Hive 0.7 no longer auto-downloads transform scripts

2011-04-06T09:34:00.000-07:00

I ran into a bit of a surprise moving a Hive 0.5 script to Hive 0.7 the other day.

Previously, in Hive 0.5, we called our Java transform code like:

insert overwrite table the_table

select

transform(...)

using 'java -cp s3://bucket-name/code.jar MapperClassName'

Behind the scenes, before actually calling the "java" executable, Hive would inspect each of the arguments and, if it found an "s3://..." URL, download that file from S3 to a local copy, and then pass the path to the local copy to your program.

This was convenient as then your external "java" executable didn't have to know anything about S3, how to authenticate with it, etc.

However, in Hive 0.7, this no longer works. Perhaps for the understandable reason that if you did want to pass the literal string "s3://..." to your mapper class, Hive implicitly interjecting on your behalf may not be what you want, and, AFAIK, you had no way to avoid it.

So, now an explicit "add file" command is required, e.g.:

add file s3://bucket-name/code.jar

insert overwrite table the_table

select

transform(...)

using 'java -cp code.jar MapperClassName'

The add file command downloads code.jar to the local execution directory (without any bucket name/path mangling like in Hive 0.5), and then your transform arguments can reference the local file directly.

All in all, a pretty easy fix, but rather frustrating to figure out given the long cycle time of EMR jobs.

Also, kudos to this post in the AWS developer forums that describes the same problem and solution:

https://forums.aws.amazon.com/thread.jspa?messageID=225472

On Building a Kick Ass Engineering Team -- Part 1

2011-03-11T11:14:00.000-08:00

We started Bizo just about three years ago with the goal of building a great business and a world-class engineering team. One of the first things I did was write down what I thought it would take to create a kick-ass engineering team.

I figured it would be good to share this list with others and discuss why I thought the items would help build a great engineering team.

Here is what I wrote:

Let's review these in a bit more detail...

Commitment to Discipline

"Commitment to discipline" is perhaps the most important characteristic for any super productive team. Software development is full of moments that try to pull you away from the task at hand or the problem that you should be trying to solve. Those weak moments where you get pulled into refactoring some sub-system end up being a huge time sink. Being able to recognize that while it would be great to refactor foo, it is not actually a requirement of "getting shit done" and can wait.

Must be cultural

I believe that the culture of any startup comes from the founders and early employees and more generally the culture of any business comes from the leaders. Therefore, it is absolutely imperative for these leaders to define what they want the culture to be and furthermore act on that definition. Culture is self-reinforcing: actions create culture which creates actions which strengthen culture. Just like a habit, once the culture is formed it is difficult to change it so it pays to be purposeful when creating your company's culture. You will have to live with it good or bad.

The 3 Cs

Communication, Communication Communication! (I stole this directly from my high school baseball coach Mr. Barden.) A major part of engineering success comes down to communication. What should we build, how should we build it, how can I plug into your system, how does this code work, etc. We spend an enormous amount of time communicating and it shows. We ensure communication through design reviews for all features/projects, and code reviews for every single line of code that makes it into production. Code reviews are an amazing place to learn, ensure quality and share knowledge. We even spend time communicating about non-Bizo related stuff that we find interesting through "Lunch and Learns".

I'll save how we go about managing all this communication for another post but the ramifications of our commitment to designs reviews, code reviews, and the like result in a team of engineers that are eager to get feedback, humble about their skills and strive to provide objective feedback to others. Objectivity is an extremely valuable characteristic of a great engineer and something that we look for in hiring and something that we strive to develop and promote.

Testing

Testing is a huge part of what we believe in here at Bizo. Beyond the obviousness of ensuring that your code works, testing has some great side effects. For one, it is an investment in future development making it easier to develop features on top of an existing product base and making it easier to "pivot" (I used the word "turn" 3 years ago before pivot was in wide use). Being a big believer in having some down time, I also consider testing an investment in one's weekend because things always seem to go wrong when you are not at work. :) Finally, extensive testing ensures a low amount of code debt which should be any dev team's goal. You always have to pay it off and upfront payments are the cheapest...

Visibility

This should really be part of the 3Cs and I would expand on this even more today. Not only is it important for code metrics, build status and task management to be communicated widely it is extremely important for the efforts of engineers to be communicated. As a company, we have a business standup and an engineering standup where group managers can engage with the teams and get visibility into what is being done. We are also experimenting with bringing engineering earlier into the product development process at the scoping level.

Ownership

Ownership is key to getting the most out of any team including engineering. I think productivity and motivation is a function of happiness and a lot of happiness comes from being trusted to do your job. At a certain point this comes back to hiring. If you can hire people who are great cultural fits and great engineering fits than you must trust them to do their jobs. That trust will make them better employees.

In Summary

To wrap this all up, I think we've done well staying true to what I outlined three years ago and I am extremely happy with the results. I've been around a lot of engineering teams and I've never been more happy (or proud) of what we've been able to accomplish both with the products and code we've shipped and with the culture of engineering we've built.

To Be Continued

In a follow up post I'll discuss what (if anything) I would add or remove from this list if I had to start over today and what I think we should focus on as an engineering team for the next few years.

"dynamic" columns in Hive

2011-02-24T17:34:00.001-08:00

One of the presentations at the HBase meetup the other night was on building a query language on top of HBase. No less than 3 people asked "Why not use Hive?". The main reason given was that Hive is too slow for doing simple selects. But, the other thing they really liked about using HBase was that your columns were dynamic -- it's easy to add new fields to your data.

Most of the data we log is in a simple log file format:

One record per line, separated by newline.
Each record can have one or more fields. Fields are separated by ^A (\001).
Each field is a key/value pair separated by ^B (\002). Field order is not specified.

In practice this bascially looks like:


ts=1298598378404/code=403/message=bad referrer: bizo.com/...

(Where / is ^A and = is ^B).

This is a format we chose pretty early on, way before we ever looked at Hive. It turns out to be a great format:

Human readable.
Trivial to parse in any language.
Dynamic -- easy to add/remove fields from your data.

It also turns out that it works really well with Hive. Our typical Hive table looks something like:


  create external table api_logs(d map<string,string>)
  partitioned by (...)
  row format delimited
    fields terminated by '\004'
    collection items terminated by '\001'
    map keys terminated by '\002'
  stored as textfile
;

That is, each row is just a single column, which is a map. At first this seemed a little degenerate to me, but it actually models our data perfectly. There are no guarantees about which fields are available, and it's easy to add/remove fields in the data over time. I should mention that this is really just for our report input, typically our report output will be in a fixed format.

If you're using Hive 0.6 or greater, with Hive View support it's also easy to get the best of both worlds.


  create view api_errors(ts, code, message) as
  select d["ts"], d["code"], d["message"]
    from api_logs
   where d["code"] >= 400
;

You can even change the type information or transform the data by including a cast or a UDF as part of your view. Creating a view doesn't cause anything to run, or create any additional storage. Its query conditions are basically just merged with subsequent queries on that view.

Adventures In GWT-land Part #1: Awkward Baby Steps

2011-01-27T17:09:00.000-08:00

Over the past several months we’ve been working on a (super) secret shiny new GWT application (it’s basically going to rock your socks off). This was my first GWT application and coming from a non-Java, non-GWT background where I was used to writing raw Javascript pretty often - it’s been interesting to say the least. What follows is the first in a multi-part series where I’d like to reflect on life in GWT-land and hopefully provide a few cool tips and code samples along the way.

First Steps:

Stepping into GWT for the first time when your used to pure Javascript - or “normal” CSS/HTML front-end development in general is pretty awkward. The biggest adjustment is that all client-side logic is defined in Java classes which later get compiled into several permutations of obfuscated Javascript. GWT also tries to be helpful and obfuscates all of your CSS classes in an attempt to prevent namespace collisions (more on this in a future post). If your thinking that FireBug becomes much less useful when working with GWT your absolutely right...but thanks to some really good debugging tools in GWT, this isn’t a terrible loss and you really won’t need it.

What’s with this Java -> Javascript stuff?

Why would somebody want to write a compiler for Java -> Javascript? One of the primary arguments for doing this is type-safety...which is fair enough I suppose - compile time checking is nice to have. You also get the benefit of native Java debugging tools, which despite the huge advances in client-side debugging in the past few years, Java’s debugging is still superior (mostly because of static typing). It’s really nice to be able to step through line by line, set breakpoints and inspect typed objects. However despite these benefits writing Java code that compiles into Javascript still feels weird.

The biggest reason for this awkwardness is that normally programmers write code in a language that is more expressive than your compile target, e.x. C/C++ compiles to assembly, Java to the JVM’s bytecode, CoffeeScript to Javascript etc. But with GWT your compile target is actually more expressive than the code you write and it’s not just a little bit more expressive, it’s a lot more expressive - which is a strange feeling indeed. Javascript has lambdas, prototypal inheritance, a less verbose syntax and a dynamic type system - all of which lead to a more expressive language, allowing you to do more with less code. Java on the other hand has none of these - expressing things that are normally trivial in Javascript (like a custom event system, currying etc.) can become a major chore (sometimes 4-5 classes or more) in GWT. Actually the whole process is rather akin to writing C/C++ code that compiles into Ruby or Python (ok, perhaps a slight exaggeration...but really only by a bit). But fear not, below are some tips to help make life easier for you in GWT-land.

Adjusting to life in GWT-land

First use the gwt-mpv framework (http://www.gwtmpv.org/) - it will save your sanity and quite possibly your soul. One of our awesome developers, Stephen has created a very nice model, view, presenter framework on top of GWT that removes a huge amount of boilerplate. It has some really nice stuff like validation, two way data binding, code generation for tedious boilerplate and more. Your fingers and brain will thank you for saving them from the verbosity of vanilla GWT, trust me...I’m an engineer.
Abandon the notion of separation of content (HTML), presentation (CSS), and behavior (Javascript) found in traditional front-end development. GWT, being a framework is opinionated and uses widgets instead. Widgets are generally responsible for all three of these things at once - they know their own CSS classes, keep their own data model and have event handlers. The idea behind this is you can just drop a widget onto any page and have it “just work” with no external dependencies. The disadvantage is the approach is inflexible if you want to change any of those three things while holding the others constant. In practice Widgets work quite well until you need to change or extend one that lives in somebody else’s jar - then they can quickly turn into a pain. So choose your imported widgets carefully or be prepared to fork things (or mash ctrl-c ctrl-v a lot).
Even though you’re writing in Java - you’re still working on the client side. Custom events are great for decoupling and facilitating easy testing of your objects as you only have to talk to a single event handler instead of multiple objects. Don’t go overboard though, you still want to avoid having events that call events that call events etc. as you’ll end up jumping all over the place trying to find out what really was supposed to happen when the original event fired.

EMR/Hive: recovering a large number of partitions

2011-01-26T16:55:00.001-08:00

If you try to run "alter table ... recover partitions" on a table with a large number of partitions, you may run into this error:

FAILED: Error in metadata: org.jets3t.service.S3ServiceException: Failed to sanitize XML document destined for handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListBucketHandler null 'null' -- ResponseCode: -1, ResponseStatus: null, RequestId: null, HostId: null FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

There's some discussion in the aws forums. The underlying cause is that it's running out of memory when trying to build the partition list.

A workaround is to increase the HADOOP_HEAPSIZE. This can be done by modifying hadoop-user-env.sh with an EMR bootstrap action. On an m1.large instance, 2G seems to do the trick for us.

Upload a script like the following somewhere in s3:

You can now run this bootstrap action as part of your job:


elastic-mapreduce --create --alive \
      --name "large partitions..." --hive-interactive \
      --num-instances 1 --instance-type m1.large \
      --hadoop-version 0.20 \
      --bootstrap-action s3://<bucket/path>/set-hadoop-heap.sh

You should now be able to load your partitions.

Spring NamespaceHandler debugging

2010-11-17T09:25:00.000-08:00

While updating a library that used Spring yesterday, I began suffering the dreaded "Unable to locate Spring NamespaceHandler for XML schema namespace" exception.

The net is littered with threads about this exception, all of which end with "put the spring jars in your WEB-INF/lib directory".

I was fairly determined not to do this, as I frequently start a webapp with an embedded instance of Jetty, and use the Eclipse project's classpath for all of the dependencies. Since all of the jars are from the project's classpath, the webapp is always using exactly the same jars that Eclipse is for compiling your code, so you never have the two drift apart.

Too many times, after copying jars to WEB-INF/lib and forgetting about them, I'll upgrade a library, everything compiles fine, but spend an embarrassing amount of time wondering why it's not working in the webapp, before remembering the stale jar in the WEB-INF/lib directory.

Anyway, the real cause of the NamespaceHandler exception in my case was a buggy ClassLoader.getResources implementation.

The way spring works, when doing it's XML parsing/whatever magic, is when it sees "xmlns:tx=...", it wants a new NamespaceHandler that knows how to handle those tags.

To allow extensible NamespaceHandlers, Spring uses ClassLoader.getResources("META-INF/spring.handers") to get a list of all of the spring.handlers files across all of the jar files in the classloader. So if spring-core, spring-tx, spring-etc. all have spring.handlers with NamespaceHandlers in them, each file gets found and loaded.

Here's the rub: whatever Eclipse project classloader that had spring-tx on it only returned spring.handlers files from jars that already had classes loaded from them. The ClassLoader.getResources implementation would not look into jar files that was on its classpath, but had not yet been opened for loading classes from.

Of all things, adding:

Class.forName("org.springframework.transaction.support.TransactionTemplate");

Before spring was initialized fixed the NamespaceHandler error. Everything boots up correctly now.

While it took way too long to figure out, I'm pleased that I can continue using the Eclipse project classloader for the webapp I'm starting and avoid the annoying "copy jars to WEB-INF/lib" solution.

I'd like to know which classloader had the buggy getResources implementation, but I've already spent too much time on this so far.

CSV and Hive

2010-11-12T17:09:00.001-08:00

CSV

Anyone who's ever dealt with CSV files knows how much of a pain the format actually is to parse. It's not as simple as splitting on commas -- the fields might have commas embedded in them, so, okay you put quotes around the field... but what if the field had quotes in it? Then you double up the quotes... "okay, ""great""" -- that was a single CSV field.

We normally use the excellent opencsv (apache2 licensed) library to deal with CSV files.

Hive

We love Hive. Almost all of our reporting is written as Hive scripts. How do you deal with CSV files with Hive? If you know for sure your fields don't have any commas in them, you can get away with the delimited format. There's the RegexSerDe, but as mentioned the format is non-trivial, and you need to change the regex string depending on how many columns you are expecting.

CSVSerde

Enter the CSVSerde. It's a Hive SerDe that uses the opencsv parser to serialize and deserialize tables properly in the CSV format.

Using it is pretty simple:



add jar path/to/csv-serde.jar;

create table my_table(a string, b string, ...)
  row format serde 'com.bizo.hive.serde.csv.CSVSerde'
  stored as textfile
;

This is my first time writing a Hive SerDe. There were a couple of road bumps, but overall I was surprised with how easy it was. I mostly just followed along with RegexSerDe.

I'm sure there are a lot of ways it could be improved, so I'd appreciate any feedback or comments on how to make it better.

Source.
Binary (jar packaged with opencsv).

Rolling out to 4 Global Regional Datacenters in 25 minutes

2010-10-26T17:14:00.000-07:00

Sometimes I just have to sit back and reflect on the amazing operational power available on AWS. As you know, we are hardcore AWS-ers here at Bizo and we've been running in all 4 regions for several months. Recently we needed to roll out a new service which we wanted to be Globally Load Balanced (GSLB) and the rollout was astoundingly quick and easy. The total time it took us to go from 0 to 4 regions was 25 minutes!!! Amazing!

Only 25 minutes to setup a service that is running in 4 regions and 8 datacenters that will autoscale to handle pretty much any amount of load we send it!

Shout out to AWS and Dynect for making it almost too easy...

An experiment in file distribution from S3 to EC2 via bittorrent

2010-10-21T13:11:00.000-07:00

Amazon's autoscaling service is fantastic. It allows you to dynamically scale the number of instances running your application based on a variety of triggers, including CPU usage, request latency, I/O usage, and more. Thus, you can increase your capacity in response to increased demand for your services.

One difficulty with this approach is that your response time is strictly bounded by the time it takes for you to spin up a new instance with your application running on it. This isn't a big deal for most servers, but some of our backend systems need multi-GB databases and indexes loaded onto them at startup.

There are several strategies for working around this, including baking the indexes into the AMI and distributing them via EBS volume; however, I was intrigued by the possibility of using S3's bittorrent support to enable peer-to-peer downloads of data. In an autoscaling situation, there are presumably several instances with the necessary data already running, and using bittorrent should allow us to quickly copy that file to a new instance.

Test setup:

All instances were m1.smalls running Ubuntu Lucid in us-east-1, spread across two availability zones. The test file was a 1GB partition of a larger zip file.

For a client, I used the version of Bittornado available in the standard repository (apt-get install -y bittornado). Download and upload speeds were simply read off of the curses interface.

For reference, I clocked a straight download of this file directly from S3 as taking an average of 57 seconds, which translates into almost 18 MB/s.

Test results:

First, I launched a single instance and started downloading from S3. S3 only gave me 70-75KB/s, considerably less than direct S3 downloads.

As the first was still downloading, I launched a second instance. The second instance quickly caught up to the first, then the download rate on each instance dropped to 140-150KB/s with upload rates at half that. Clearly, what was going on was S3 was giving each instance 70-75KB/s of bandwidth, and the peers were cooperating by sharing their downloaded fragments.

To verify this behavior, I then launched two more instances and hooked them into the swarm. Again, the new peers quickly caught up to the existing instances, and download rates settled down to 280-300KB/s on each of the four instances.

So, there's clearly some serious throttling going on when downloading from S3 via bittorrent. However, the point of this experiment is not the S3 -> EC2 download speed but the EC2 <-> EC2 file sharing speed.

Once all four of these instances were seeding, I added a fifth instance to the swarm. Download rates on this instance maxed out at around 12-13 MB/s. Once this instance was seeding, I added a sixth instance to the swarm to see if bandwidth would continue to scale up, but I didn't see an appreciable difference.

So, it looks like using bittorrent within EC2 is actually only about 2/3rds as fast as downloading directly from S3. In particular, even with a better tuned environment (eg, moving to larger instances to eliminate sharing physical bandwidth with other instances), it doesn't look like we would get any significant decreases in download times by using bittorrent.

Killing java processes

2010-10-01T14:31:00.000-07:00

I often want to kill java processes, be it an unresponsive Eclipse, a blown-out jEdit after I try to open a 2GB file, a stalled JUnit test suite, a borked scalac compiler daemon or a random Tomcat instance.

It gets tiring to write,

$ jps -lv
48231 /opt/eclipse-3.5.1/org.eclipse.equinox.launcher_1.0.201.jar -Xmx1024m
10258 /opt/boisvert/jedit-4.3.2/jedit.jar -Xmx192M
5295 sun.tools.jps.Jps -Dapplication.home=/opt/boisvert/jdk1.6.0_21 -Xms8m

followed by,

$ kill 48231

You know, with the cut & paste in-between ... so I have this Ruby shell script called killjava, a close cousin of killall:

$ killjava -h
killjava [-9] [-n] [java_main_class]

-9, --KILL                       Send KILL signal instead of TERM
-n, --no-prompt                  Do not prompt user, kill all matching processes
-h, --help                       Show this message

that does the job. It's not like I use it everyday but everytime I use it, I'm glad it's there.

Download the script from Github (requires Ruby and UNIX-based OS).

modern IDEs influencing coding style?

2010-10-01T09:37:00.001-07:00

It would be nice if globals, locals, and members could be syntax colored differently. That would be better than g_ and m_ prefixes.
- John Carmack

I saw this from John Carmack last week and thought, what a great idea! It seems very natural and easy to do and makes a lot more sense than crazy prefix conventions. I've been mostly programming in Java, so the conventions are a little different, but I'd love it if we could get rid of using redundant "this" qualifiers to signal member variables, and the super ugly ALL_CAPS for constants... it just seems so outdated.

Eclipse actually provides this kind of highlighting already:

Notice that the member variable "greeting" is always in blue, while the non-member variables are never highlighted. Also, the public static constant "DEFAULT_GREETING" is blue and italicized.

Notice that if you rename DEFAULT_GREETING, it's still completely recognizable as a constant:

I think it's interesting that modern IDEs are able to give us so much more information about the structure of our programs. Stuff you used to have to explicitly call out via conventions like these. How long until we're ready to make the leap and change our code conventions to keep up with our tools?

The main argument against relying on tools to provide this kind of information is that not all tools have caught up. I'm not sure I completely buy this. Hopefully you're not actually remotely editing production code in vi or something. There are a lot of web apps for viewing commits and performing code reviews, and they're unlikely to be as fully featured as your favorite IDE. Still, the context is often limited enough to avoid confusion, and the majority of our time is spent in our IDEs anyway.

So, can we drop the ALL_CAPS already?

emr: Cannot run program "bash": java.io.IOException: error=12, Cannot allocate memory

2010-09-29T16:12:00.001-07:00

Moving one of our jobs from hive 0.4 / hadoop 0.18 to hive 0.5 / hadoop 0.20 on amazon emr, I ran into a weird error in the reduce stage, something like:


java.io.IOException: Task: attempt_201007141555_0001_r_000009_0 - The reduce copier failed 
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:384) 
at org.apache.hadoop.mapred.Child.main(Child.java:170) 
Caused by: java.io.IOException: Cannot run program "bash": java.io.IOException: error=12, Cannot allocate memory 
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:149) 
at org.apache.hadoop.util.Shell.run(Shell.java:134) 
at org.apache.hadoop.fs.DF.getAvailable(DF.java:73) 
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329) 
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124) 
at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputFile.java:160) 
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2622) 
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2586) 
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory 
at java.lang.UNIXProcess.(UNIXProcess.java:148) 
at java.lang.ProcessImpl.start(ProcessImpl.java:65) 
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) 
... 8 more

There's some discussion on this thread in the emr forums.

From Andrew's response to the thread:

The issue here is that when Java tries to fork a process (in this case bash), Linux allocates as much memory as the current Java process, even though the command you are running might use very little memory. When you have a large process on a machine that is low on memory this fork can fail because it is unable to allocate that memory.

The workaround here is to either use an instance with more memory (m2 class), or reduce the number of mappers or reducers you are running on each machine to free up some memory.

Since the task I was running was reduce heavy, I chose to just drop the number of mappers from 4 to 2. You can do this pretty easy with the emr bootstrap actions.

My job ended up looking something like this:

 elastic-mapreduce --create --name "awesome script" \
--num-instances 8 --instance-type m1.large \
--hadoop-version 0.20 \

--bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop \
--args "-s,mapred.tasktracker.map.tasks.maximum=2" \

--hive-script --arg s3://....../script

(relevant parts highlighted).

Salesforce and DART Synchronization

2010-09-21T05:02:00.000-07:00

I’ve recently started some work that involves extending Salesforce for our Ad Ops team. For our most recent Hack Day, I decided to do a little project to continue learning about development with the Salesforce cloud platform, Force.com.

After thinking about what I wanted to work on, I decided to build a custom button that would allow a user to update an Account record in Salesforce with an Advertiser ID from DART, our primary ad serving platform, for the following reasons:

It’s a tool that I could see being used in our live Salesforce instance.
It seems like a typical use case for extending Salesforce (i.e. integrating with a 3rd party SOAP service).

The back of the napkin design looked like this:

At a high-level, I wanted to call DART’s DFP API from within Salesforce and then update an Account object in Salesforce with the Advertiser Id returned from DART. However, I first needed to authenticate with Google’s ClientLogin service in order to get an authentication token for calling the DFP API.

APEX

APEX is the programming language that allows a developer to customize a Salesforce installation. APEX’s syntax, not surprisingly, is very similar to Java. The really interesting thing is that none of the code you write actually compiles or runs on your machine. All compilation and execution happen “in the cloud”.

DART Integration

Salesforce has a strict security model. In order to make a request to a Web Service you actually need to configure any URLs you are accessing as a Remote Site. Instructions for doing this can be found here. For this project, I simply needed to add https://www.google.com as a Remote Site.

There are a couple of options for calling a Web Service via APEX:

Use the Http/HttpRequest/Http APEX classes. These are useful for calling REST style services.
Import a WSDL and use the generated code to make a SOAP request.

In this project, I ended up using both methods.

Here is the APEX code I developed for calling Google’s ClientLogin authentication service:

public class GoogleAuthIntegration {
  private static String CLIENT_AUTH_URL = 'https://www.google.com/accounts/ClientLogin';

  // login to google with the given email and password
  public static String performClientLogin(final String email, final String password) {
    final Http http = new Http();
    final HttpRequest request = new HttpRequest();
    request.setEndpoint(CLIENT_AUTH_URL);
    request.setMethod('POST');
    request.setHeader('Content-type', 'application/x-www-form-urlencoded');
 
    final String body = 'service=gam&accountType=GOOGLE&' + 'Email=' + email + '&Passwd=' + password;
    request.setBody(body);
 
    final HttpResponse response = http.send(request);
    final String responseBody = response.getBody();
    final String authToken = responseBody.substring(responseBody.indexOf('Auth=') + 5).trim();
 
    System.debug('authToken is: ' + authToken);
    return authToken;
  }
}

This piece of code would fetch an authToken for the given username and password. Once I had the authToken, I could then call the DFP API. For this part, I used WSDL/SOAP, the 2nd method for calling web services.

Salesforce provides a way to import a WSDL file via its Admin UI. It then parses and generates APEX code that allows you to call methods exposed by the WSDL. However, when I tried importing DFP’s Company Service WSDL, I ran into some errors:

It turns out that the WSDL contains an element named ‘trigger’ and trigger is a reserved APEX keyword. In any event, I ended up copy/pasting the generated code and fixing it so that it compiled correctly (I also ran into a problem where generated exception classes were not extending Exception).

Once the code to call the DFP Company Service was compiling, I created an APEX controller to perform the update on an Account record.

public class SyncDartAccountController {
  private final Account acct;
 
  public SyncDartAccountController(ApexPages.StandardController stdController) {
    this.acct = (Account) stdController.getRecord();
  }

  // Code we will invoke on page load.
  public PageReference onLoad() {
    String theId = ApexPages.currentPage().getParameters().get('id');

    if (theId == null) {
      // Display the Visualforce page's content if no Id is passed over
      return null;
    }

    // get authToken for DFP API requests
    String authToken = GoogleAuthIntegration.performClientLogin('xxx@xxx.com', 'xxxx');

    // get Account with the given id
    for (Account o:[select id, name from Account where id =:theId]) {
      DartCompanyService.CompanyServiceInterfacePort p = new DartCompanyService.CompanyServiceInterfacePort();
      p.RequestHeader = new DartCompanyService.SoapRequestHeader();
      p.RequestHeader.applicationName = 'sampleapp';
   
      // prepare the DFP query and execute
      DartCompanyService.Statement filterByNameAndType = new DartCompanyService.Statement();
      filterByNameAndType.query = 'WHERE name = \'' + o.Name + '\' and type = \'ADVERTISER\'';
  
      DartCompanyService.CompanyPage page = p.getCompaniesByStatement(filterByNameAndType);
   
      if (page.totalResultSetSize > 0) {
        // update the record if we get a result
        o.Dart_Advertiser_Id__c = page.results.get(0).id;
        update o;
       }
    }

    // Redirect the user back to the original page
    PageReference pageRef = new PageReference('/' + theId);
    pageRef.setRedirect(true);
    return pageRef;
  }
}

UI updates

Then, I created a simple Visuaforce page to invoke the controller:


<apex:page standardController="Account"  extensions="SyncDartAccountController" action="{!onLoad}">
  <apex:sectionHeader title="Auto-Running Apex Code"/>
  <apex:outputPanel >
   You tried calling Apex Code from a button.  If you see this page, something went wrong.  
        You should have been redirected back to the record you clicked the button from.
  </apex:outputPanel>
</apex:page>

Finally, I added a custom button to the Account page which would invoke the Visualforce page. You can do this in the Salesforce UI:

1) Click on ‘Buttons and Links’:

2) Click New:

3) Enter the info for the new button:

4) After clicking on Save, we can add the button to the Account page layout. The final result:

Final Thoughts
This was my first foray into APEX programming in Salesforce and I was pleased with the overall set of tools and ability to be productive quickly. The only hiccup I encoutered was in the WSDL generation step and this issue was fairly easy to overcome. There are good developer docs and there are ways to add debug logging (which I didn’t go over) as well as a framework for unit testing.

quick script: emr-mailer

2010-09-20T13:59:00.001-07:00

We write a lot of hive reports. Frequently we want to email the resulting report to a list. In the past I've usually done this with some one-off post processing scripts, but I thought it would be nice to write a reusable emr job step that will execute as part of the hive job.

The script will download files from an s3 url, concatenate them together, zip up the results and send it as an attachment to a specified email address. It sends email through smtp.mail.com, using account credentials you specify.

I wanted to make it easy to just append an additional step to any existing job, not requiring any additional machine setup or dependencies. I was able do this by making use of amazon's script-runner (s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar). The script-runner.jar step will let you execute an arbitrary script from a location in s3 as an emr job step.

As I mentioned, the intended usage is to run it as a job step with your hive script, passing it in the location of the resulting report.

E.g.:


elastic-mapreduce --create --name "my awesome report ${MONTH}" \
   --num-instances 10 --instance-type c1.medium  --hadoop-version 0.20 \
   --hive-script --arg s3://path/to/hive/script.sql \
   --args -d,MONTH=${MONTH} --args -d,START=${START} --args -d,END=${END} \
   --jar s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar \
   --args s3://path/to/emr-mailer/send-report.rb \
   --args -n,report_${MONTH} --args -s,"my awesome report ${MONTH}" \
   --args -e,awesome-reports@company.com \
   --args -r,s3://path/to/report/results

Above you can see I'm starting a hive report as normal, then simply appending the script-runner step, calling the emr-mailer send-report.rb, telling it where the report will end up, and details about the email.

The full source code is available on github as emr-mailer.

The script is pretty simple, but let me know if you have any suggestions for improvements or other feedback.

Collecting User Actions with GWT

2010-08-13T20:18:00.000-07:00

While I was at one of the Google I/O GWT sessions (courtesy of Bizo), a Google presenter mentioned how one of their internal GWT applications tracks user actions.

The idea is really just a souped-up, AJAX version of server-side access logs: capturing, buffering, and sending fine-grained user actions up to the server for later analysis.

The Google team was using this data to make A/B-testing-style decisions about features–which ones were being used, not being used, tripping users up, etc.

I thought the idea was pretty nifty, so I flushed out an initial implementation in BizAds for Bizo’s recent hack day. And now I am documenting my wild success for Bizo’s first post-hack-day “beer & blogs” day.

No Access Logs

Traditional, page-based web sites typically use access logs for site analytics. For example, the user was on a.html, then b.html. Services like Google Analytics can then slice and dice your logs to tell you interesting things.

However, desktop-style one-page webapps don’t generate these access logs–the user is always on the first page–so they must rely on something else.

This is pretty normal for AJAX apps, and Google Analytics already supports it via its asynchronous API.

We had already been doing this from GWT with code like:

public native void trackInGA(final String pageName) /*-{
  $wnd._gaq.push(['_trackPageview', pageName]);
}-*/;

And since we’re using a MVP/places-style architecture (see gwt-mpv), we just call this on each place change. Done.

Google Analytics is back in action, not a big deal.

Beyond Access Logs

What was novel, to me, about this internal Google application’s approach was how the tracked user actions were much more fine-grained than just “page” level.

For example, which buttons the user hovers over. Which ones they click (even if it doesn’t lead to a page load). What client-side validation messages are tripping them up. Any number of small “intra-page” things that are nonetheless useful to know.

Obviously there are a few challenges, mostly around not wanting to detract from the user experience:

How much data is too much?
Tracking the mouse over of every element would be excessive. But the mouse over of key elements? Should be okay.

How often to send the data?
If you wait too long while buffering user actions before uploading them to the server, the user may leave the page and you’ll lose them. (Unless you use a page unload hook, and the browser hasn’t crashed.)

If you send data too often, the user might get annoyed.

The key to doing this right is having metrics in place to know whether you’re prohibitively affecting the user experience.

The internal Google team had these metrics for their application, and that allowed them to start out batch uploading actions every 30 seconds, then every 20 seconds, and finally every 3 seconds. Each time they could tell the users’ experience was not adversely affected.

Unfortunately, I don’t know what exactly this metric was (I should have asked), but I imagine it’s fairly application-specific–think of GMail and average emails read/minute or something like that.

Implementation

I was able to implement this concept rather easily, mostly by reusing existing infrastructure our GWT application already had.

When interesting actions occur, I have the presenters fire a generic UserActionEvent, which is generated using gwt-mpv-apt from this spec:

@GenEvent
public class UserActionEventSpec {
  @Param(1)
  String name;
  @Param(2)
  String value;
  @Param(3)
  boolean flushNow;
}

Initiating the tracking an action is now just as simple as firing an event:

UserActionEvent.fire(
   eventBus,
   "someAction",
   "someValue",
   false);

I have a separate, decoupled UserActionUploader, which is listening for these events and buffers them into a client-side list of UserAction DTOs:

private class OnUserAction implements UserActionHandler {
  public void onUserAction(final UserActionEvent event) {
    UserAction action = new UserAction();
    action.user = defaultString(getEmailAddress(), "unknown");
    action.name = event.getName();
    action.value = event.getValue();
    actions.add(action);
    if (event.getFlushNow()) {
      flush();
    }
  }
}

UserActionUploader sets a timer that every 3 seconds calls flush:

private void flush() {
  if (actions.size() == 0) {
    return;
  }
  ArrayList<UserAction> copy =
    new ArrayList<UserAction>(actions);
  actions.clear();
  async.execute(
    new SaveUserActionAction(copy),
    new OnSaveUserActionResult());
}

The flush method uses gwt-dispatch-style action/result classes, also generated by gwt-mpv-apt, to the server via GWT-RPC:

@GenDispatch
public class SaveUserActionSpec {
  @In(1)
  ArrayList<UserAction> actions;
}

This results in SaveUserActionAction (okay, bad name) and SaveUserActionResult DTOs getting generated, with nice constructors, getters, setters, etc.

On the server-side, I was able to reuse an excellent DatalogManager class from one of my Bizo colleagues (unfortunately not open source (yet?)) that buffers the actions data on the server’s hard disk and then periodically uploads the files to Amazon’s S3.

Once the data is in S3, it’s pretty routine to setup a Hive job to read it, do any fancy reporting (grouping/etc.), and drop it into a CSV file. For now I’m just listing raw actions:

-- Pick up the DatalogManager files in S3
drop table dlm_actions;
create external table dlm_actions (
    d map<string, string>
)
partitioned by (dt string comment 'yyyyddmmhh')
row format delimited
fields terminated by '\n' collection items terminated by '\001' map keys terminated by '\002'
location 's3://<actions-dlm-bucket>/<folder>/'
;
 
alter table dlm_actions recover partitions;
 
-- Make a csv destination also in S3
create external table csv_actions (
    user string,
    action string,
    value string
)
row format delimited fields terminated by ','
location 's3://<actions-report-bucket/${START}-${END}/parts'
;
 
-- Move the data over (nothing intelligent yet)
insert overwrite table csv_actions
select dlm.d["USER"], dlm.d["ACTION"], dlm.d["VALUE"]
from dlm_actions dlm
where
    dlm.dt >= '${START}00' and dlm.dt < '${END}00'
;

Then we use Hudson as a cron-with-a-GUI to run this Hive script as an Amazon Elastic Map Reduce job once per day.

Testing

Thanks to the awesomeness of gwt-mpv, the usual GWT widgets, GWT-RPC, etc., can be doubled-out and testing with pure-Java unit tests.

For example, a method from UserActionUploaderTest:

UserActionUploader uploader = new UserActionUploader(registry);
StubTimer timer = (StubTimer) uploader.getTimer();
 
@Test
public void uploadIsBuffered() {
  eventBus.fireEvent(new UserActionEvent("someaction", "value1", false));
  eventBus.fireEvent(new UserActionEvent("someaction", "value2", false));
  assertThat(async.getOutstanding().size(), is(0)); // buffered
 
  timer.run();
  final SaveUserActionAction a1 = async.getAction(SaveUserActionAction.class);
  assertThat(a1.getActions().size(), is(2));
  assertAction(a1, 0, "anonymous", "someaction", "value1");
  assertAction(a1, 1, "anonymous", "someaction", "value2");
}

The usual GWT timers are stubbed out by a StubTimer, which we can manually tick via timer.run() to deterministically test timer-delayed business logic.

That’s It

I can’t say we have made any feature-altering decisions for BizAds based on the data gathered from this approach yet–technically its not live yet. But it’s so amazing that surely we will. Ask me about it sometime in the future.

hackday: analog meters

2010-08-13T15:44:00.001-07:00

For this last hackday, I decided to work on something more hardware hacking related. At this year's Maker Fair, I was really inspired by all the cool stuff people were building, so I picked up an arduino and started playing around with a couple of things.

I've always wanted to have some cool old-school analog VU type meters displaying web requests.

Here's my completed hackday project:

Here's a view of the components from the back:

It's battery operated and receives data wirelessly over RF from another arduino I have hooked up via serial to my laptop.

It's pretty simple, but I'm still totally psyched about how it came out.

The main components are some analog panel meters (kinda pricey, but awesome), and an RF receiver. The frame is a piece of scrap acrylic from TAP Plastics that I drilled and cut to size, and the stand is a piece of a wire clothes hanger bent to shape.

Connected to my computer is a another arduino (actually a volksduino) that receives updates over USB and sends the data out over RF:

You may be asking, why bother with wireless if you need a computer hooked up through serial anyway. Or you may ask why not just connect to a wireless network directly.

Well, I wanted the meters to be able to be moved around, or mounted on a wall... I wanted them wireless. But, it turns out that wireless and even ethernet solutions for connecting an arduino to the internet directly are comparatively pretty expensive. Even using bluetooth is expensive. My long term plan is to have a single arduino connected to the internet directly (via ethernet or wireless), and have it serve as a proxy over RF for the others... So this is a bit of work towards that.

I wrote a bit of Java code to connect to amazon's cloudwatch to pull the load balancer statistics for two of our services. I then discovered it's near impossible to connect to anything over USB in Java... It is ridiculous. Luckily, it's REALLY easy to do this with Processing, so I wrote a simple processing program that used my cloudwatch library and wrote it out to serial.

And that's really it. The arduino reads data over serial, and periodically sends it over RF. The arduino hooked up to the meters simply reads the values over RF and sets the meters to display a scaled version of the results. They're showing requests per second. We get a huge amount of requests per second with these services, so the numbers on the dial aren't actually correct (I need to make some custom faceplates). It also flashes an LED every time it gets a RF transmission.

Here's a quick video of it in action:

The one thing I'm not crazy about is that the maximum resolution you can get from cloudwatch is stats per minute, so the meters don't actually change as often as I would like.

Still, pretty cool. I'm looking forward to building some more displays like this in the future.

Extending Hive with Custom UDTFs

2010-07-29T16:52:00.000-07:00

Let’s take a look at the canonical word count example in Hive: given a table of documents, create a table containing each word and the number of times it appears across all documents.

Here’s one implementation from the Facebook engineers:


CREATE TABLE docs(contents STRING);

FROM (
  MAP docs.contents 
  USING 'tokenizer_script' 
  AS 
    word, 
    cnt
  FROM docs
  CLUSTER BY word
) map_output
REDUCE map_output.word, map_output.cnt 
USING 'count_script' 
AS 
  word, 
  cnt
;

In this example, the heavy lifting is being done by calling out to two scripts, ‘tokenizer_script’ and ‘count_script’, that provide custom mapper logic and reducer logic.

Hive 0.5 adds User Defined Table-Generating Functions (UDTF), which offers another option for inserting custom mapper logic. (Reducer logic can be plugged in via a User Defined Aggregation Function, the subject of a future post.) From a user perspective, UDTFs are similar to User Defined Functions except they can produce an arbitrary number of output rows for each input row. For example, the built-in UDTF “explode(array A)” converts a single row of input containing an array into multiple rows of output, each containing one of the elements of A.

So, let’s implement a UDTF that does the same thing as the ‘tokenizer_script’ in the word count example. Basically, we want to convert a document string into multiple rows with the format (word STRING, cnt INT), where the count will always be one.

The Tokenizer UDTF

To start, we extend the org.apache.hadoop.hive.ql.udf.generic.GenericUDTF class. (There is no plain UDTF class.) We need to implement three methods: initialize, process, and close. To emit output, we call the forward method.

Adding a name and description:


@description(name = "tokenize", value = "_FUNC_(doc) - emits (token, 1) for each token in the input document")
public class TokenizerUDTF extends GenericUDTF {

You can add a UDTF name and description using a @description annotation. These will be available on the Hive console via the show functions and describe function tokenize commands.

The initialize method:


  public StructObjectInspector initialize(ObjectInspector[] args) throws UDFArgumentException

This method will be called exactly once per instance. In addition to performing any custom initialization logic you may need, it is responsible for verifying the input types and specifying the output types.

Hive uses a system of ObjectInspectors to both describe types and to convert Objects into more specific types. For our tokenizer, we want a single String as input, so we’ll check that the input ObjectInspector[] array contains a single PrimitiveObjectInspector of the STRING category. If anything is wrong, we throw a UDFArgumentException with a suitable error message.


    if (args.length != 1) {
      throw new UDFArgumentException("tokenize() takes exactly one argument");
    }

    if (args[0].getCategory() != ObjectInspector.Category.PRIMITIVE
        && ((PrimitiveObjectInspector) args[0]).getPrimitiveCategory() != PrimitiveObjectInspector.PrimitiveCategory.STRING) {
      throw new UDFArgumentException("tokenize() takes a string as a parameter");
    }

We can actually use this object inspector to convert inputs into Strings in our process method. This is less important for primitive types, but it can be handy for more complex objects. So, assuming stringOI is an instance variable,


    stringOI = (PrimitiveObjectInspector) args[0];

Similarly, we want our process method to return an Object[] array containing a String and an Integer, so we’ll return a StandardStructObjectInspector containing a JavaStringObjectInspector and a JavaIntObjectInspector. We’ll also supply names for these output columns, but they’re not really relevant at runtime since the user will supply his or her own aliases.


    List fieldNames = new ArrayList(2);
    List fieldOIs = new ArrayList(2);
    fieldNames.add("word");
    fieldNames.add("cnt");
    fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
    fieldOIs.add(PrimitiveObjectInspectorFactory.javaIntObjectInspector);
    return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
  }

The process method:


  public void process(Object[] record) throws HiveException

This method is where the heavy lifting occurs. This gets called for each row of the input. The first task is to convert the input into a single String containing the document to process:


    String document = (String) stringOI.getPrimitiveJavaObject(record[0]);

We can now implement our custom logic:


    if (document == null) {
      return;
    }
    String[] tokens = document.split(“\\s+”);
    for (String token : tokens) {
      forward(new Object[] { token, Integer.valueOf(1) });
    }
  }

The close method:


  public void close() throws HiveException { }

This method allows us to do any post-processing cleanup. Note that the output stream has already been closed at this point, so this method cannot emit more rows by calling forward. In our case, there’s nothing to do here.

Packaging and use:

We deploy our TokenizeUDTF exactly like a UDF. We deploy the jar file to our Hive machine and enter the following in the console:


> add jar TokenizeUDTF.jar ;
> create temporary function tokenize as ’com.bizo.hive.udtf.TokenizeUDTF’ ;
> select tokenize(contents) as (word, cnt) from docs ;

This gives us the intermediate mapped data, ready to be reduced by a custom UDAF.

The code for this example is available in this gist.

Come work at Bizo

2010-06-25T09:31:00.001-07:00

Want to work on some interesting problems with a great development team?

We're looking to hire a junior developer and a quantitative engineer.

Send us your resume! Or if you know someone in the Bay Area that you think might be a good fit, please forward the posting to them.

Accessing Bizo API using Ruby OAuth

2010-06-08T11:34:00.000-07:00

During a recent HackDay, I wrote a search-engine-like frontend to our multi-dimensional business demographic database.

Among other things, the interface provides search term suggestions based on business title classification, using the Classify operation of the Bizo API.

I chose to use Ruby and the lightweight Sinatra web framework for fast prototyping and since the Bizo API uses OAuth for authentication, I reached out for the excellent OAuth gem.

Now, while the documentation is good it took me a little time to grok the OAuth API and figure out how to use it. The Bizo API does not use a RequestToken; instead we use an API key and a shared secret. Since the OAuth gem documentation didn't include any example for this use-case, I figured I'd post my code here as a starting point for other people to reuse.

Without further ado, here's the short code fragment:


require 'rubygems'
require 'oauth'
require 'oauth/consumer'
require 'json'

key = 'xxxxxxxx'
secret = 'yyyyyyyy'

consumer = OAuth::Consumer.new(key, secret, {
  :site         => "http://api.bizographics.com",
  :scheme       => :query_string,
  :http_method  => :get
})

title = "VP of Marketing"
path = URI.escape("/v1/classify.json?api_key=#{key}&title=#{title}")

response = consumer.request(:get, path)

# Display response
p JSON.parse(response.body)

If you're curious, here's the JSON response for the "VP of Market..." title classification,


{
  "usage" => 1,
  "bizographics" => {
  "group" => { "name" => "High Net Worth", "code" => "high_net_worth" },
  "functional_area" => [
    {"name" => "Sales", "code" => "sales" },
    {"name" => "Marketing", "code" => "marketing" }
  ],
  "seniority" => {"name" => "Executives", "code" => "executive" }
  }
}

Hopefully this is useful to Rubyists out there needing quick OAuth integration using HTTP GET and a query string and don't need to go through the token exchange process.

Hackday: dependency searching using scala, jersey, gxp, mongodb

2010-05-11T15:49:00.001-07:00

For my hackday project, I thought I would try to build an internal tool to let us more easily search our dependency repository. We use ivy for dependency management, and maintain our own repository in s3. It can be kind of a pain to track down the latest version of library X, especially if you're not sure what the organization is, or maybe you know the org and not the name. It seemed like a fun, useful project that I could tackle in a day, and that would allow me to play around with a couple of things I was interested in. To build it, I used jersey, gxp, and mongoDB. The whole thing was written using scala.

I borrowed the main layout from the SpringSource Enterprise Bundle Repository. I'm pretty happy with the results:

And the detail view:

There's also a browse view.

I've been really happy using scala and jersey, and I wanted something simple and easy for this project, so I thought it was worth a shot. After adding GXP for templating support, I have to say the combination of scala/jersey/GXP makes a pretty compelling framework for simple web apps.

As an example, here's the beginning of my 'Browse' Controller:


@Path("/b")
class Browse {
  val db = new RepoDB 

  @Path("/o")
  @GET @Produces(Array("text/html"))  
  def browseOrg() = browseOrgLetter("A")
  
  @Path("/o/{letter}")
  @GET @Produces(Array("text/html"))
  def browseOrgLetter(@PathParam("letter") letter : String) = {
    
    val orgs = db.getOrgLetters
    
    val results = db.findByOrgLetter(letter, 30)
    
    BrowseView.getGxpClosure("Organization", "o", orgs, letter, results)
  }
  ...

It's using nested paths, so /b/o is the main browse by organization page, /b/o/G would be all organizations starting with 'G'.

Then, I have a simple MessageBodyWriter that can render a GxpClosure:


@Provider
@Produces(Array("text/html"))
class GxpClosureWriter extends MessageBodyWriter[GxpClosure] {
  val context = new GxpContext(Locale.US)

  override def isWriteable(dataType: java.lang.Class[_], ...) = {
    classOf[GxpClosure].isAssignableFrom(dataType)
  }

  override def writeTo(gxp: GxpClosure, ...) {
    val out = new java.io.OutputStreamWriter(_out)
    gxp.write(out, context)
  }
  ...

And, that's really all there is to it. Nice, simple, and lightweight.

Last but not least, mongodb. It was probably overkill for this project, but I was looking for an excuse to play with it some more. I use it to store and index all of the repository information. I have a separate crawler process that lists everything in our repository s3 bucket, then stores an entry for each artifact. As part of this, it does some basic tokenizing of the organization and artifact names for searching. Searching like this was a little disappointing compared to lucene. Overall though, I'm pretty happy with it. Browsing and searching are both ridiculously fast. Like I said, it was probably overkill for the amount of data we have.... but it can never be too fast. speed is most definitely a feature.

Anyway, that's the wrap-up.

I'd be interested to other thoughts/experiences on mongodb from anyone out there.

Improving Global Application Performance, continued: GSLB with EC2

2010-05-05T12:06:00.000-07:00

This is an unofficial continuation of Amazon's blog post on the use of Amazon CloudFront to improve application performance.

CloudFront is a great CDN to consider, especially if you're already an Amazon Web Services customer. Unfortunately, it can only be used for static content; the loading of dynamic content will still be slower for far-away users than for nearby ones. Simply put, users in India will still see a half-second delay when loading the dynamic portions of your US-based website. And a half-second delay has a measurable impact on revenue.

Let's talk about speeding up dynamic content, globally.

The typical EC2 implementation comprises instances deployed in a single region. Such a deployment may span several availability zones for redundancy, but all instances are in roughly the same place, geographically.

This is fine for EC2-hosted apps with nominal revenue or a highly localized user base. But what if your users are spread around the globe? The problem can't be solved by moving your application to another region - that would simply shift the extra latency to another group.

For a distributed audience, you need a distributed infrastructure. But you can't simply launch servers around the world and expect traffic to reach them. Enter Global Server Load Balancing (GSLB).

A primer on GSLB
Broadly, GSLB is used to intelligently distribute traffic across multiple datacenters based on some set of rules.

With GSLB, your traffic distribution can go from this:

To this:

GSLB can be implemented as a feature of a physical device (including certain high-end load balancers) or as a part of a DNS service. Since we EC2 users are clearly not interested in hardware, our focus is on the latter: DNS-based GSLB.

Standard DNS behavior is for an authoritative nameserver to, given queries for a certain record, always return the same result. A DNS-based implementation of GSLB would alter this behavior so that queries return context-dependent results.

Example:
User A queries DNS for gslb.example.com -- response: 10.1.0.1
User B queries DNS for gslb.example.com -- response: 10.2.0.1

But what context should we use? Since our goal is to reduce wire latency, we should route users to the closest datacenter. IP blocks can be mapped geographically -- by examining a requestor's IP address, a GSLB service can return a geo-targeted response.

With geo-targeted DNS, our example would be:
User A (in China) queries DNS for geo.example.com -- response: 10.1.0.1
User B (in Spain) queries DNS for geo.example.com -- response: 10.2.0.1

Getting started
At a high level, implementation can be broken down into two steps
1) Deploying infrastructure in other AWS regions
2) Configuring GSLB-capable DNS

Infrastructure configurations will vary from shop to shop, but as an example, a read-heavy EC2 application with a single master database for writes should:
- deploy application servers to all regions
- deploy read-only (slave) database servers and/or read caches to all regions
- configure application servers to use the slave database servers and/or read caches in their region for reads
- configure application servers to use the single master in the "main" region for writes

This is what such an environment would look like:

When configuring servers to communicate across regions (app servers -> master DB; slave DBs -> master DB), you will need to use IP-based rules for your security groups; traffic from the "app-servers" security group you set up in eu-west-1 is indistinguishable from other traffic to your DB server in us-east-1. This is because cross-region communication is done using external IP addresses. Your best bet is to either automate security group updates or use Elastic IPs.

Note on more complex configurations: distributed backends are hard (see Brewer's [CAP] theorem). Multi-region EC2 environments are much easier to implement if your application tolerates the use of 1) regional caches for reads; 2) centralized writes. If you have a choice, stick with the simpler route.

As for configuring DNS, several companies have DNS-based GSLB service offerings:
- Dynect - Traffic Management (A records only) and CDN Manager (CNAMEs allowed)
- Akamai - Global Traffic Management
- UltraDNS - Directional DNS
- Comwired/DNS.com - Location Geo-Targeting

DNS configuration should be pretty similar for the vendors listed above. Basic steps are:
1) set up regional CNAMEs (us-east-1.example.com, us-west-1.example.com, eu-west-1.example.com, ap-southeast-1.example.com)
2) set up a GSLB-enabled "master" CNAME (www.example.com)
3) define the GSLB rules:
- For users in Asia, return ap-southeast-1.example.com
- For users in Europe, return eu-west-1.example.com
- For users in Western US, return us-west-1.example.com
- ...
- For all other users, return us-east-1.example.com

If your application is already live, consider abstracting the DNS records by one layer: geo.example.com (master record); us-east-1.geo.example.com, us-west-1.geo.example.com, etc. (regional records). Bring the new configuration live by pointing www.example.com (CNAME) to geo.example.com.

Bizo's experiences
Several of our EC2 applications serve embedded content for customer websites, so it's critical we minimize load times. Here's the difference we saw on one app after expanding into new regions (from us-east-1 to us-east-1, us-west-1, and eu-west-1) and implementing GSLB (load times provided by BrowserMob):

Load times before GSLB:

Load times after GSLB:

Reduced load times for everyone far from us-east-1. Users are happy, customers are happy, we're happy. Overall, a success.

It's interesting to see how the load is distributed throughout the day. Here's one application's HTTP traffic, broken down by region (ELB stats graphed by cloudviz):

Note that the use of Elastic Load Balancers and Auto Scaling becomes much more compelling with GSLB. By geographically partitioning users, peak hours are much more localized. This results in a wider difference between peak and trough demand per region; Auto Scaling adjusts capacity transparently, reducing the marginal cost of expanding your infrastructure to multiple AWS regions.

For our GSLB DNS service, we use Dynect and couldn't be more pleased. Intuitive management interface, responsive and helpful support, friendly, no-BS sales. Pricing is based on number of GSLB-enabled domains and DNS query rate. Contact Dynect sales if you want specifics (we work with Josh Delisle and Kyle York - great guys). Note that those intending to use GSLB with Elastic Load Balancers will need the CDN Management service.

Closing remarks
Previously, operating a global infrastructure required significant overhead. This is where AWS really shines. Amazon now has four regions spread across three continents, and there's minimal overhead to distribute your platform across all of them. You just need to add a layer to route users to the closest one.

The use of Amazon CloudFront in conjunction with a global EC2 infrastructure is a killer combo for improving application performance. And with Amazon continually expanding with new AWS regions, it's only going to get better.

@mikebabineau

GWT MVP Tables

2010-04-02T19:22:00.000-07:00

I'm building a GWT MVP app here at Bizo. Despite its upsides (primarily insanely fast unit test), GWT MVP is new, under documented, and, for me anyway, has had a non-trivial learning curve.

So, in the interest of sharing tricks I've learned, one I've come across is how to elegantly implement dashboard-style tables.

Dashboard-style tables are those where each row has a lot of interactivity, e.g. a user can start/stop/cancel the entity in the row, see the icons change, and other various UI things that you generally want to have under test in presenters.

For awhile, I was stumped because most GWT MVP tables favor data-style tables, where the focus is more on efficiently displaying 100s/1000s of rows of basically non-interactive data. There are a few counter examples, but none I had seen got to the point of pushing testable HasXxx interfaces back into a presenter for business logic to hook up to.

I finally got a suitable solution by realizing that each row is essentially its own view, and so instantiating a new RowXxxPresenter for each domain object in the value. This means there is a RowXxxView, which can expose all the HasXxx interfaces it wants to the presenter. And the RowXxxPresenterTest becomes a treat to write.

For more details and code examples, please jump over to the more elaborate post over on my personal blog.

Bizo Job - Designer

2010-03-24T10:10:00.001-07:00

Position: Web / UI / UX Designer (San Francisco)

Summary:
We’re looking for an out-of-the-box thinker with a good sense-of-humor and a great attitude to join our product development team. As the first in-house Web / UI / UX Designer for Bizo, you will take responsibility for developing easy, powerful, consistent and high velocity web and interaction designs across all Bizo web products as well as marketing materials related to the Bizographic Targeting Platform, a revolutionary new way to target business advertising online. You will be a key player on an incredible team as we build our world-beating, game-changing, and massively scalable bizographic advertising and targeting platform. In a nutshell, you will be the voice of reason in all design and usability aspects of Bizo.

The Team:
We’re a small team of very talented people (if we do say so ourselves!). We use Agile development methodologies. We care about high quality results, not how many hours you’re in the office. We believe in strong design that helps people get stuff done!

The Ideal Candidate:

Self-motivated
Entrepreneurial / Hacker spirit
Experience/Expertise with Adobe Illustrator (and/or similar design tools)
Strong CSS skills
Strong HTML skills
Strong Javascript skills
Understands the value of mock-ups (points for Balsamiq experience)
Flash experience (bonus but not required)
Enjoys working on teams
Educational background or industry experience in Design or related field – points for advanced degrees
Gets stuff done!

Please send a resume, cover letter and link to online portfolio to: donnie@bizo.com

Introducing Cloudviz

2010-03-17T13:31:00.001-07:00

Amazon CloudWatch exposes a variety of useful metrics for EC2 instances, Elastic Load Balancers, and more. Unfortunately, it is tedious to query directly and the results can be difficult to interpret.

Like most operational metrics, CloudWatch data provides the most insight when graphed. While there are existing tools to graph CloudWatch data, they are only available as part of a proprietary suite or service and, generally, they sacrifice customization and flexibility for ease-of-use.

Here at Bizo, we wanted to incorporate CloudWatch data into operational dashboards. Nothing we found was flexible enough to meet our needs, so we decided to write our own. We are now releasing it to for all to use.

I'm pleased to introduce cloudviz, an open source tool for creating embeddable CloudWatch graphs.

Specifically, cloudviz is a data source that exposes CloudWatch data for graphing by Google Interactive Charts (formerly Visualization API). It's written in Python using Google's Data Source library and Mitch Garnaat's excellent AWS interface, boto.

With cloudviz, it's easy to create graphs like these:

I encourage you to check out the project on GitHub here. There's a fairly detailed README and plenty of examples, but feel free to drop me a line if you have any questions, mike@bizo.com.

Happy graphing!

SSH to EC2 instance ID

2010-03-05T11:01:00.000-08:00

I often find myself looking up EC2 nodes by instance ID so I can grab the external DNS name and SSH in. Fed up with the extra “ec2-describe-instance , copy, paste” layer, I threw together a function (basically a fancy alias) to SSH into an EC2 instance referenced by ID.

Assuming you’re on Mac OS X / Linux, just put this somewhere in ~/.profile, reload your terminal, and you’re good to go.

Alternatively, you can use the shell script version.

(note: cross-posted here)

Example git/git-sh config

2010-03-04T11:45:00.000-08:00

I've been using git, git-svn, and git-sh while working on Bizo's internal projects and really enjoying it. Per requests from some other devs, here is my git/git-sh config.

First, you should start with git-sh. It adds some bash shell customizations like a nice `PS1` prompt, tab completion, and incredibly short git-specific aliases. I'll cover some of the aliases later, but this is the thing that started me down the "how cool can I get my git environment" path.

I've included commented versions of my .gitconfig and .gitshrc below, but you can find raw versions here and here. I also cross-posted this on my personal blog if you're so inclined as to read it twice.

Example Shell Session

A lot of my customizations are around aliases, so this is a quick overview, and then the aliases are defined/explained below.

Here is a made up example bash session with some of the commands:


  # show we're in a basic java/whatever project
  $ ls
  src/ tests/

  # start git-sh to get into a git-specific bash environment
  $ git sh

  # change some things
  $ echo "file1" > src/package1/file1
  $ echo "file2" > src/package2/file2
  $ echo "file3" > src/package3/file2

  # see all of our changes
  $ d
  # runs: git diff

  # see only the changes in package1
  $ dg package1
  # runs: git diff src/package1/file1

  # stage any path with 'package' in it
  $ ag package
  # runs: git add src/package1/file1 src/package2/file2 src/package3/file3

  # we only wanted package1, reset package2 and package3
  $ rsg package2
  # runs: git reset src/package2/file2
  $ rsg package3
  # runs: git reset src/package3/file3

  # see what we have staged now (only package1)
  $ p
  # runs: git diff --cached

  # commit it
  $ commit -m "Changed stuff in package1"
  # runs: git commit -m "..."

That is the basic idea.

Most of the magic is from the [alias] section of .gitconfig, along with my .gitshrc allowing the git prefix to be dropped.

.gitconfig

The .gitconfig file is in your home directory and is for user-wide settings.

Here is my current .gitconfig with comments:


  [user]
    name = Stephen Haberman
    email = stephen@exigencecorp.com
  [alias]
    # 'add all' stages all new+changed+deleted files
    aa = !git ls-files -d | xargs -r git rm && git ls-files -m -o --exclude-standard | xargs -r git add

    # 'add grep' stages all new+changed that match $1
    ag = "!sh -c 'git ls-files -m -o --exclude-standard | grep $1 | xargs -r git add' -"

    # 'checkout grep' checkouts any files that match $1
    cg = "!sh -c 'git ls-files -m | grep $1 | xargs -r git checkout' -"

    # 'diff grep' diffs any files that match $1
    dg = "!sh -c 'git ls-files -m | grep $1 | xargs -r git diff' -"

    # 'patch grep' diff --cached any files that match $1
    pg = "!sh -c 'git ls-files -m | grep $1 | xargs -r git diff --cached' -"

    # 'remove grep' remove any files that match $1
    rmg = "!sh -c 'git ls-files -d | grep $1 | xargs -r git rm' -"

    # 'reset grep' reset any files that match $1
    rsg = "!sh -c 'git ls-files -c | grep $1 | xargs -r git reset' -"

    # nice log output
    lg = log --graph --pretty=oneline --abbrev-commit --decorate

    # rerun svn show-ignore -> exclude
    si = !git svn show-ignore > .git/info/exclude

    # start git-sh
    sh = !git-sh
  [color]
    # turn on color
    diff = auto
    status = auto
    branch = auto
    interactive = auto
    ui = auto
  [color "branch"]
    # good looking colors i copy/pasted from somewhere
    current = green bold
    local = green
    remote = red bold
  [color "diff"]
    # good looking colors i copy/pasted from somewhere
    meta = yellow bold
    frag = magenta bold
    old = red bold
    new = green bold
  [color "status"]
    # good looking colors i copy/pasted from somewhere
    added = green bold
    changed = yellow bold
    untracked = red
  [color "sh"]
    branch = yellow
  [core]
    excludesfile = /home/stephen/.gitignore
    # two-space tabs
    pager = less -FXRS -x2
  [push]
    # 'git push' should only do the current branch, not all
    default = current
  [branch]
    # always setup 'git pull' to rebase instead of merge
    autosetuprebase = always
  [diff]
    renames = copies
    mnemonicprefix = true
  [svn]
    # push empty directory removals back to svn at directory deletes
    rmdir = true

.gitshrc

This is my .gitshrc file, heavily based off Ryan Tomayko's original.

Ryan's original comments are prefixed with #, I'll prefix my additions with ###, most of which are aliases to my [alias] entries above and some git-svn aliases.


  #!/bin/bash
  # rtomayko's ~/.gitshrc file
  ### With additions from stephenh

  # git commit
  gitalias commit='git commit --verbose'
  gitalias amend='git commit --verbose --amend'
  gitalias ci='git commit --verbose'
  gitalias ca='git commit --verbose --all'
  gitalias  n='git commit --verbose --amend'

  # git branch and remote
  gitalias  b='git branch -av' ### Added -av parameter
  gitalias rv='git remote -v'

  # git add
  gitalias  a='git add'
  gitalias au='git add --update'
  gitalias ap='git add --patch'
  ### Added entries for my .gitconfig aliases
  alias aa='git aa' # add all updated/new/deleted
  alias ag='git ag' # add with grep
  alias agp='git agp' # add with grep -p
  alias cg='git cg' # checkout with grep
  alias dg='git dg' # diff with grep
  alias pg='git pg' # patch with grep
  alias rsg='git rsg' # reset with grep
  alias rmg='git rmg' # remove with grep

  # git checkout
  gitalias c='git checkout'

  # git fetch
  gitalias f='git fetch'

  # basic interactive rebase of last 10 commits
  gitalias r='git rebase --interactive HEAD~10'
  alias cont='git rebase --continue'

  # git diff
  gitalias d='git diff'
  gitalias p='git diff --cached'   # mnemonic: "patch"

  # git ls-files
  ### Added o to list other files that aren't ignored
  gitalias o='git ls-files -o --exclude-standard'    # "other"

  # git status
  alias  s='git status'

  # git log
  gitalias  L='git log'
  # gitalias l='git log --graph --pretty=oneline --abbrev-commit --decorate'
  gitalias  l="git log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr)%Creset' --abbrev-commit --date=relative"
  gitalias ll='git log --pretty=oneline --abbrev-commit --max-count=15'

  # misc
  gitalias pick='git cherry-pick'

  # experimental
  gitalias mirror='git reset --hard'
  gitalias stage='git add'
  gitalias unstage='git reset HEAD'
  gitalias pop='git reset --soft HEAD^'
  gitalias review='git log -p --max-count=1'

  ### Added git svn asliases
  gitalias si='git si' # update svn ignore > exclude
  gitalias sr='git svn rebase'
  gitalias sp='git svn dcommit'
  gitalias sf='git svn fetch'

  ### Added call to git-wtf tool
  gitalias wtf='git-wtf'

Since I defined most of the interesting aliases in the .gitconfig [alias] section, it means they're all usable via git xxx, e.g. git ag foo, but listing alias ag='git ag' in .gitshrc means you can also just use ag foo, assuming you've started the git-sh environment.

It results in some duplication, but means they're usable from both inside and outside of git-sh, which I think is useful.

Get Your Speed Tracer On!

2010-03-04T01:44:00.000-08:00

I first saw Speed Tracer in action at Google I/O 2009 and was pretty amped about it. While we have been using GWT 2.0 features for a few months now (e.g. OOPHM, UiBinder, ClientBundle), I had not tried out Speed Tracer until tonight. Speed Tracer is a Chrome plugin that is as a web performance profiling tool on steroids. The level of profiling information that you can get from this tool is truly amazing. If you develop web apps then I highly recommend that you check it out. I guarantee it will be something you will want to have in your toolbox. Installation instructions can be found here.

Triggering post-Elastic MapReduce steps as parameterized jobs in Hudson

2010-02-19T11:40:00.000-08:00

Here at Bizo, the combination of Hudson for cron management, Hive for report generation, and Elastic MapReduce for provisioning compute power has greatly simplified our data processing. Periodically and automatically, our Hudson cron instance generates Hive scripts for us and launches them in EC2.

The main inconvenience with this process is that the results of our Hive jobs are left as one or more obscurely named files in S3. These often need some post-processing to put them into a more friendly form. Unfortunately, EMR doesn't have an easy hook for launching these post-processing tasks -- although we could implement them as MapReduce steps, we'd need to write our own workflows, losing the simplicity of using EMR's simple "--hive-script" flag.

Our solution is to use SimpleDB to store some basic metadata about jobs. Using this metadata, a Hudson job periodically checks the EMR API to determine whether tasks have completed. If so, it then triggers other Hudson jobs that are responsible for processing the results.

Here are some tools that make this process work:

Simple script to put data into SimpleDB. Our metadata scheme is to use the jobflow ID as item names and the name/parameters of the jobs to trigger as attributes.

The Hudson parameterized build feature. It's not really feasible to create a new Hudson job for each individual report that runs, so we pass parameters to Hudson so the post-processing step can figure out where the results are in S3 and what to do with them. It's not well-documented how to do this programatically (as opposed to from the web interface); the solution is to send some JSON to the build url.

The Trigger Script. This is the script that periodically runs on our cron server to check if a post-processing step should be triggered. The JSON format for parameterized jobs is described in the comments of this file.

The end result is that a job can run an EMR job and configure a post-processing step for itself with the following commands:


JOB_ID=`elastic-mapreduce --create --hive-script --arg ${s3.location}` | grep "Created job flow" | awk '{ print $4 }' -`

simpledb-put.rb -d ${metadata.domain} -i $JOB_ID "next_on_cron_server_job_name=post-processing-step" "next_on_cron_server_job_params={\"parameter\": [{\"name\":\"PARAM1\", \"value\":\"VALUE1\" }]}" "next_on_cron_server_triggered=false"

This launches the hive script in the specified s3.location and configures the "post-processing-step" job on the cron server to run with the parameter "PARAM1=VALUE1".

Scala Supports Non-Local Returns

2010-01-07T19:42:00.000-08:00

Writing some Scala code today, I found myself using non-local returns without even thinking about it. After realizing what I had done, I dug a little deeper to see what was really going on.

Take this completely made up, nonsensical example:


object Foo {
  def main(args: Array[String]) {
    foo(List(1, 2, 3))
  }

  def foo(l: List[Int]): Int = {
    l.foreach { (i) =>
      println(i)
      return 5
    }
    return 10
  }
}

This code will print "1" and then exit.

Perhaps this is obvious, that the "return 5" applies to the "foo" method, so the values 2 and 3 in "l" will not have a chance to be printed.

However, think about what is going on under the covers--Scala is passing the foreach method an anonymous inner class with a "void apply(int i)" method. And inside of that "apply" method is the code between the "{ (i) => ... }".

So, how does code inside of the "apply" method cause its caller to perform an early exit, without the "foreach" even knowing about it?

Exceptions.

Here is the decompiled version of "print":


 public int print(List l) {
    Object localObject = new Object();
    int exceptionResult1 = 0;
    try {
      l.foreach(new AbstractFunction1() {
        public static final long serialVersionUID = 0L;

        public final Nothing. apply(int i) {
          Predef..MODULE$.println(BoxesRunTime.boxToInteger(i));
          // here is the "return"--it puts "5" into an exception
          throw new NonLocalReturnException(
            this.nonLocalReturnKey1$1,
            BoxesRunTime.boxToInteger(5));
        }
      });
      return 10;
    } catch (NonLocalReturnException localNonLocalReturnException)  {
      if (localNonLocalReturnException.key() == localObject) {
        // get "5" back out of the exception
        return BoxesRunTime.unboxToInt(localNonLocalReturnException.value());
      }
      throw localNonLocalReturnException;
    }
  }

I think the decompiler got a little confused with "nonLocalReturnKey", but you can see the basic idea is that any early return inside of a closure is converted into an exception that is then caught outside of the closure where a proper return call can be done.

I personally think this is handy, once you know what is going on. But from what I've picked up, any closures that make it into Java 7 will not support non-local returns and instead disallow the "return" keyword inside of closures. Which, I guess, at this point any Java closures are better than no closures at all.

amazon ec2 spot instances

2009-12-15T11:22:00.001-08:00

Yesterday Amazon announced EC2 Spot Instances. The idea is that you can bid on unused EC2 instance time. The 'Spot price' is determined periodically by Amazon based on availability and demand for the instances. If your bid is higher than the spot price, you will get an instance and only pay the spot price. Of course, your instance may be terminated at any time, but the nice thing is that unlike the normal ec2 pricing, here you only pay for full hours of usage.

To check out the price history of small linux instances, download the new release of the ec2-api-tools and run:


ec2-describe-spot-price-history --instance-type m1.small -d Linux/UNIX -H

Running this last night, I saw prices that looked like (times PST):

It looks like there's a substantial discount here with prices ranging from $0.025 to $0.035 per hour (the normal ec2 price is $0.085/hr).

Since I'm in the middle of reading How to Cheat at Everything, one of my first thoughts was why not just bid say $0.10/hour? In this way, you're unlikely to get outbid, but you'll probably stand to save significantly for a large part of the day. Now I'm thinking this probably isn't quite a free market... If amazon needs capacity to satisfy reserved instances, or even regular ec2 instances, maybe they'll just kill off these machines to make room.

Still, this is really very cool. A great option for doing a lot of offline batch processing. I hope we start to see support for taking advantage of this type of model in hadoop. It's also exciting to think that one day maybe we'll see something like this across providers -- bid for time across amazon, sun, etc.

Update: Some nice charts by Tim Lossen at cloudexchange.org.

github spam?

2009-12-04T16:17:00.001-08:00

I just happened to land on the github recent repositories page, and noticed a ton of spam:

A bunch of different users and projects advertising movie downloads. There's no project content, of course, just a "homepage" that points to a target url...

At first I was thinking, wow, these are some crazy spammers -- using git as a tool for spam! But on closer look, it seems like they're just hitting the website automating account signup and new repository actions.

Still, spam on github? Crazy! I guess no website is safe these days. If you're hosting user generated content, you need to think about detecting and blocking spam and automation of user activity.

quick script: open hadoop jobtracker UI with elastic map reduce

2009-11-30T18:28:00.001-08:00

If you've ever logged into the hadoop master with amazon's elastic map reduce, you'll see something like:

The Hadoop UI can be accessed via the command: lynx http://localhost:9100/

Great, but lynx?.. not as nice as firefox or safari...

It's easy enough to do some ssh port forwarding so you can use your browser of choice and access the hadoop UI from your machine.

But, after getting tired of typing in the ssh options a bunch of times, I finally put together a short script that automates it a bit. The script takes in the public hostname of your hadoop master (you can get this from elastic-mapreduce --list), then picks a random port number, sets up the ssh forwarding, and opens the page in a new browser window.

I call it hcon for 'hadoop console'. After configuring the script with the path to your emr key file, you run it like:

hcon ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com

Here's the full script, but in case you're curious the magic lines (wrapped) are:


ssh -f -N -o "StrictHostKeyChecking no" \
 -L ${LPORT}:localhost:9100 \
 -i ${KEYFILE} hadoop@${HOST}
$BROWSER http://localhost:${LPORT}

(Yes, for this, I turn off StrictHostKeyChecking).

Anyway, try it out and let me know if it's helpful at all.

Amazon Web Services Start-up Challenge Finalists

2009-11-13T10:08:00.000-08:00

We are excited to be one of nine finalists in the AWS Start-up Challenge!

Using Hudson to manage crons

2009-11-02T09:55:00.000-08:00

We've been using Hudson for several months now to manage our builds -- we probably have 80-90 different projects that it's responsible for. It's an awesome system for continuous integration and testing.

It's also an awesome system for scheduling and managing generic jobs. We've only just begun to use it as a cron server, but it's clear that it has numerous advantages over the more traditional way of using the unix cron service directly.

Notification plugins -- Hudson can be easily configured to send email and Jabber notifications when cron jobs start, succeed, or fail. You can also track your scheduled jobs via RSS.

Stdout/Sterr logging -- Hudson saves the stdout and stderr from each run automatically.

SCM integration -- if you need to update a job, just check the changes into SVN (or whatever SCM system you use). Hudson will automatically pick up the changes the next time your job is run.

Nice web interface -- never underestimate the productivity gains from having a good UI. It can be surprisingly tricky to determine exactly which crons are running on a generic Unix box. Not so with Hudson.

At Bizo, we believe that developers should be getting their hands dirty in the operational aspects of their projects -- Hudson gives us an easy interface for managing our scheduled jobs using the same tools that we're familiar with for managing our build processes. Hudson is such a great tool for continuous integration that it's easy to overlook how good it is at the simpler task of managing generic scheduled jobs.

Clearing Linux Filesystem Cache

2009-10-20T03:52:00.000-07:00

I was doing some performance tuning of our mysql db and was having some trouble consistently reproducing query performance due to IO caching that was occuring in Linux. In case you're wondering, you can clear this cache by executing the following command as root:

echo 1 > /proc/sys/vm/drop_caches

bash, errors, and pipes

2009-10-16T10:53:00.001-07:00

Our typical pattern for writing bash scripts has been to start off each script with:

#!/bin/bash -e

The -e option will cause the script to exit immediate if a command has exited with a non-zero status. This way your script will fail as early as possible, and you never get into a case where on the surface, it looks like the script completed, but you're left with an empty file, or missing lines, etc.

Of course, this is only for "simple" commands, so in practice, you can think of it terminating immediately if the entire line fails. So a script like:


#!/usr/bin/bash -e
/usr/bin/false || true
echo "i am still running"

will still print "i am still running," and the script will exit with a zero exit status.

Of course, if you wrote it that way, that's probably what you're expecting. And, it's easy enough to change (just change "||" to "&&").

The thing that was slightly surprising to me was how a script would behave using pipes.


#!/bin/bash -e
/usr/bin/false | sort > sorted.txt
echo "i am still running"

If your script is piping its output to another command, it turns out that the return status of a pipeline is the exit status of its last command. So, the script above will also print "i am still running" and exit with a 0 exit status.

Bash provides a PIPESTATUS variable, which is an array containing a list of the exit status values from the pipeline. So, if we checked ${PIPESTATUS[0]} it would contain 1 (the exit value of /usr/bin/false), and ${PIPESTATUS[1]} would contain 0 (exit value of sort). Of course, PIPESTATUS is volatile, so, you must check it immediately. Any other command you run will affect its value.

This is great, but not exactly what I wanted. Luckily, there's another bash option -o pipefail, which will change the way the pipeline exit code is derived. Instead of being the last command, it will become the last command with a non-zero exit status. So


#!/bin/bash -e -o pipefail
/usr/bin/false | sort > sorted.txt
echo "this line will never execute"

So, thanks to pipefail, the above script will work as we expect. Since /usr/bin/false returns a non-zero exit status, the entire pipeline will return a non-zero exit status, the script will die immediately because of -e, and the echo will never execute.

Of course, all of this information is contained in the bash man page, but I had never really ran into it / looked into it before, and I thought it was interesting enough to write up.

s3fsr 1.4 released

2009-10-12T20:07:00.000-07:00

s3fsr is a tool we built at Bizo to help quickly get files into/out of S3. It's had a few 1.x releases, but by 1.4 we figured it was worth getting around to posting about.

Overview

While there a lot of great S3 tools out there, s3fsr's niche is that it's a FUSE/Ruby user land file system.

For a command line user, this is handy, because it means you can do:

# mount yourbucket in ~/s3 s3fsr yourbucketname ~/s3 # see the directories/files ls ~/s3/ # upload mv ~/local.txt ~/s3/remotecopy.txt # download cp ~/s3/remote.txt ~/localcopy.txt

Behind the scenes, s3fsr is talking to the Amazon S3 REST API and getting/putting directory and file content. It will cache directory listings (not file content), so ls/tab completion will be quick after the initial short delay.

S3 And Directory Conventions

A unique aspect of s3fsr, and a specific annoyance it was written to fulfill, is that it understands several different directory conventions used by various S3 tools.

This directory convention problem stems from Amazon's decision to forgo any explicit notion of directories in the API, and instead force everyone to realize that S3 is not a file system but a giant hash table of string key -> huge byte array.

Let's take an example--you want to store two files, "/dir1/foo.txt" and "/dir1/bar.txt" in S3. In a traditional file system, you'd have 3 file system entries: "/dir1", "/dir1/foo.txt", and "/dir1/bar.txt". Note that "/dir1" gets its own entry.

In S3, without tool-specific conventions, storing "/dir1/foo.txt" and "/dir1/bar.txt" really means only 2 entries. "/dir1" does not exist of its own accord. The S3 API, when reading and writing, never parses keys apart by "/", it just treats the whole path as one big key to get/set in its hash table.

For Amazon, this "no /dir1" approach makes sense due to the scale of their system. If they let you have a "/dir1" entry, pretty soon API users would want the equivalent of a "rm -fr /dir1", which, for Amazon, means instead of a relatively simple "remove the key from the hash table" operation, they have to start walking a hierarchical structure and deleting child files/directories as they go.

When the keys are strewn across a distributed hash table like Dynamo, this increases the complexity and makes the runtime nondeterministic.

Which Amazon, being a bit OCD about their SLAs and 99th percentiles, doesn't care for.

So, no S3 native directories.

There is one caveat--the S3 API lets you progressively infer the existence of directories by probing the hash table keys with prefixes and delimiters.

In our example, if you probe with "prefix=/" and "delimiter=/", S3 will then, and only then, split & group the "/dir1/foo.txt" and "/dir1/bar.txt" keys on "/" and return you just "dir1/" as what the S3 API calls a "common prefix".

Which is kind of like a directory. Except that you have to create the children first, and then the directory pops into existence. Delete the children, and the directory pops out of existence.

This brings us to the authors of tools like s3sync and S3 Organizer--their users want the familiar "make a new directory, double click it, make a new file in it" idiom, not a backwards "make the children files first" idiom. It is, understandably, different from what users expect.

So, the tool authors got creative and basically added their own "/dir1" marker entries to S3 when users' perform a "new directory" operation to get back to the "directory first" idiom.

Note this is a hack, because issuing a "REMOVE /dir1" to S3 will not recursively delete the child files, because to S3 "/dir1" is just a meaningless key with no relation to any other key in the hash table). So now the burden is on the tool to do its own recursive iteration/deletion of the directories.

Which is cool, and actually works pretty well, except that the two primary tools implemented marker entries differently:

s3sync created marker entries (e.g. a "/dir1" entry) with a hard-coded content that etags (hashes) to a specific value. This known hash is nice because it makes it easy to distinguish directory entries from file entries when listing S3 entries and, S3 knowing nothing about directories, the tool having to infer on its own which keys represent files and which represent directories.
S3 Organizer created marker entries as well, but instead of a known etag/hash, they suffixed the directory name, so the key of "/dir1" is actually "/dir1_$folder$". It's then the job of the tool is recognize the suffix as a marker directory entry, strip off the suffix before showing the name to the user, and use a directory icon instead of a file icon.

So, if you use a S3 tool that does not understand these 3rd party conventions, browsing a well-used bucket will likely end up looking odd with obscure/duplicate entries:

/dir1 # s3sync marker entry file /dir1 # common prefix directory /dir1/foo.txt # actual file entry /dir2_$folder$ # s3 organizer maker entry file /dir2 # common prefix directory /dir2/foo.txt # actual file entry

This quickly becomes annoying.

And so s3fsr understands all three conventions, s3sync, S3 Organizer, and common prefixes, and just generally tries to do the right thing.

FUSE Rocks

One final note is that the FUSE project is awesome. Implementing mountable file systems that users can "ls" around in usually involves messy, error-prone kernel integration that is hard to write and, if the file system code misbehaves, can screw up your machine.

FUSE takes a different approach and does the messy kernel code just once, in the FUSE project itself, and then it acts as a proxy out to your user-land, process-isolated, won't-blow-up-the-box process to handle the file system calls.

This proxy/user land indirection does degrade performance, so you wouldn't use it for your main file system, but for scenarios like s3fsr, it works quite well.

And FUSE language bindings like fusefs for Ruby make it a cinch to develop too--s3fsr is all of 280 LOC.

Wrapping up

Let us know if you find s3fsr useful--hop over to the github site, install the gem, kick the tires, and submit any feedback you might have.

Want to be challenged at work?

2009-10-12T16:13:00.000-07:00

We've got a few challenges and are looking to grow our (kick ass) engineering team. Check out the opportunities below and reach out if you think you've got what it takes...

Efficiently selecting random sub-collections.

2009-10-08T10:24:00.000-07:00

Here's a handy algorithm for randomly choosing k elements from a collection of n elements (assume k < n)


public static <T> List<T> pickRandomSubset(Collection<T> source, int k, Random r) {
  List<T> toReturn = new ArrayList<T>(k);
  double remaining = source.size();
  for (T item : source) {
    double nextChance = (k - toReturn.size()) / remaining;
    if (r.nextDouble() < nextChance) {
      toReturn.add(item);
      if (toReturn.size() == k) {
        break;
      }
    }
    --remaining;
  }
  return toReturn;
}

The basic idea is to iterate through the source collection only once. For each element, we can compute the probability that it should be selected, which simply equals the number of items left to pick divided by the total number of items left.

Another nice thing about this algorithm is that it also works efficiently if the source is too large to fit in memory, provided you know (or can count) how many elements are in the source.

This isn't exactly anything groundbreaking, but it's far better than my first inclination to use library functions to randomly sort my list before taking a leading sublist.

hive map reduce in java

2009-10-07T11:07:00.001-07:00

In my last post, I went through an example of writing custom reduce scripts in hive.

Writing a streaming reducer requires a lot of the same work to check for when keys change. Additionally, in java, there's a decent amount of boilerplate to go through just to read the columns from stdin.

To help with this, I put together a really simple little framework that more closely resembles the hadoop Mapper and Reducer interfaces.

To use it, you just need to write a really simple reduce method:

  void reduce(String key, Iterator<String[]> records, Output output);

The helper code will handle all IO, as well as the grouping together of records that have the same key. The 'records' Iterator will run you through all rows that have the key specified in key. It is assumed that the first column is the key. Each element in the String[] record represents a column. These rows aren't buffered in memory or anything, so it can handle any arbitrary number of rows.

Here's the complete example from the my reduce example, in java (even shorter than perl).

public class Condenser {
  public static void main(final String[] args) {
    new GenericMR().reduce(System.in, System.out, new Reducer() {
      void reduce(String key, Iterator records, Output output) throws Exception {
        final StringBuilder vals = new StringBuilder();
        while (records.hasNext()) {
          // note we use col[1] -- the key is provided again as col[0]
          vals.append(records.next()[1]);
          if (records.hasNext()) { vals.append(","); }
        }
        output.collect(new String[] { key, vals.toString() });
      }
    });
  }
}

Here's a wordcount reduce example:

public class WordCountReduce {
  public static void main(final String[] args) {
    new GenericMR().reduce(System.in, System.out, new Reducer() {
      public void reduce(String key, Iterator<String[]> records, Output output) throws Exception {
        int count = 0;
        
        while (records.hasNext()) {
          count += Integer.parseInt(records.next()[1]);
        }
        
        output.collect(new String[] { key, String.valueOf(count) });
      }
    });
  }
}

Although the real value is in making it easy to write reducers, there's also support for helping with mappers. Here's my key value split mapper from a previous example:

  public class KeyValueSplit {
    public static void main(final String[] args) {
      new GenericMR().map(System.in, System.out, new Mapper() {
      public void map(String[] record, Output output) throws Exception {
        for (final String kvs : record[0].split(",")) {
          final String[] kv = kvs.split("=");
          output.collect(new String[] { kv[0], kv[1] });
        }
      }
    }
  }

The full source code is available here. Or you can download a prebuilt jar here.

The only dependency is apache commons-lang.

I'd love to hear any feedback you may have.

Simple DB Firefox Plugin -- New Release

2009-10-06T13:37:00.000-07:00

I finally got around to updating our open-sourced Simple DB Firefox Plugin creatively named SDB Tool.

obligatory screen shot

The major highlights include:

Runs in Firefox 3.5!
Support for "Select" Queries (e.g. Version 2009-04-15 of the API)
Lots of UI Tweaks and Refactoring...

Please report any issues here.

Click here to install.

reduce scripts in hive

2009-10-06T09:31:00.000-07:00

In a previous post, I discussed writing custom map scripts in hive. Now, let's talk about reduce tasks.

The basics

As before, you are not writing an org.apache.hadoop.mapred.Reducer class. Your reducer is just a simple script that reads from stdin (columns separated by \t) and should write rows to stdout (again, columns separated by \t).

Another thing to mention is that you can't run a reduce without first doing a map.

The rows to your reduce script will be sorted by key (you specify which column this is), so that all rows with the same key will be consecutive. One thing that's kind of a pain with hive reducers, is that you need to keep track of when keys change yourself. Unlike a hadoop reducer where you get a (K key, Iterator<V> values), here you just get row after row of columns.

An example

We'll use a similar example to the map script.

We will attempt to condense a table (kv_input) that looks like:

k1 v1
k2 v1
k4 v1
k2 v3
k3 v1
k1 v2
k4 v2
k2 v2
...

into one (kv_condensed) that looks like:

k1 v1,v2
k2 v1,v2,v3
...

The reduce script

#!/usr/bin/perl                                                                                       

undef $currentKey;
@vals=();

while (<STDIN>) {
  chomp();
  processRow(split(/\t/));
}

output();

sub output() {
  print $currentKey . "\t" . join(",", sort @vals) . "\n";
}

sub processRow() {
  my ($k, $v) = @_;

  if (! defined($currentKey)) {
    $currentKey = $k;
    push(@vals, $v);
    return;
  }

  if ($currentKey ne $k) {
    output();
    $currentKey = $k;
    @vals=($v);
    return;
  }

  push(@vals, $v);
}

Please forgive my perl. It's been a long time (I usually write these in java, but thought perl would make for an easier blog example).

As you can see, a lot of the work goes in to just keeping track of when the keys change.

The nice thing about these simple reduce scripts is that it's very easy to test locally, without going through hadoop and hive. Just call your script and pass in some example text separated by tabs. If you do this, you need to remember to sort the input by key before passing into your script (this is usually done by hadoop/hive).

Reducing from Hive

Okay, now that we have our reduce script working, let's run it from Hive.

First, we need to add our map and reduce scripts:

add file identity.pl;
add file condense.pl;

Now for the real work:

from (
  from kv_input
  MAP k, v
  USING './identity.pl'
  as k, v
 cluster by k) map_output
insert overwrite table kv_condensed
reduce k, v
  using './condense.pl'
  as k, v
;

This is fairly dense, so I will attempt to give a line by line breakdown:

On line 3 we are specifying the columns to pass to our reduce script from the input table (specified on line 2).

As I mentioned, You must specify a map script in order to reduce. For this example, we're just using a simple identity perl script. On line 5 we name the columns the map script will output.

Line 6 specifies the column which is the key. This is how the rows will be sorted when passed to your reduce script.

Line 8 specifies the columns to pass into our reducer (from the map output columns on line 5).

Finally, line 10 names the output columns from our reducer.

(Here's my full hive session for this example, and an example input file).

I hope this was helpful. Next time, I'll talk about some java code I put together to simplify the process of writing reduce scripts.

Developing on the Scala console with JavaRebel

2009-10-05T11:14:00.000-07:00

If you're the type of developer who likes to mess around interactively with your code, you should definitely be using the Scala console. Even if you're not actually using any Scala in your code, you can still instantiate your Java classes, call their methods, and play around with the results. Here's a handy script that I stick in the top-level of my Eclipse projects that will start an interactive console with my compiled code on the classpath:

#!/bin/bash

tempfile=`mktemp /tmp/tfile.XXXXXXXXXX`

/usr/bin/java -jar /mnt/bizo/ivy-script/ivy.jar -settings /mnt/bizo/ivy-script/ivyconf.xml -cachepath ${tempfile} > /dev/null

classpath=`cat ${tempfile} | tr -d "\n\r"`

rm ${tempfile}

exec /usr/bin/java -classpath /opt/local/share/scala/lib:target/classes:${classpath} -noverify -javaagent:/opt/javarebel/javarebel.jar scala.tools.nsc.MainGenericRunner

(Since we already use Ivy for dependency management, this script also pulls in the appropriate jar files from the Ivy cache. See this post for more details.)

The javaagent I'm using here is JavaRebel, a really awesome tool that provides automatic code reloading at runtime. Using the Scala console and JavaRebel, I can instantiate an object on the console and test a method. If I get an unexpected result, I can switch back to Eclipse, fix a bug or add some additional logging, and rerun the exact same method back on the console. JavaRebel will automagically detect that the class file was changed and reload it into the console, and the changes will even be reflected in the objects I created beforehand.

The icing on this cake is that Zero Turnaround (the makers of JavaRebel) is giving away free licenses to Scala developers. How awesome is that?

Running ScalaTest BDD Tests from Eclipse

2009-09-10T11:02:00.000-07:00

At Bizo, we're using Scala for a few things here and there. While investigating testing approaches for Scala, I came across ScalaTest and its Behavior Driven Development (BDD) spec approach.

While its a small thing, I really like the sentence-based it "should do this and that" aspect of the spec approach. You get great readability compared to traditional "testDoThisAndThat" method names.

However, a large downside to the spec approach is that spec tests cannot, on their own, be easily, one-keyboard-shortcut run from within Eclipse. The built-in Eclipse JUnit test runner does not understand the describe/it-based test structure.

To solve this, I wrote a class that can be used with JUnit's "RunWith" annotation to bridge the gap between JUnit and ScalaTest. Its not perfect, but you get back the one-shortcut/greenbar runner in Eclipse. So I can definitely see it being handy if we decide to do any spec-based testing here at Bizo.

GWT hosted mode on snow leopard

2009-09-09T15:41:00.001-07:00

One of the first things I noticed after installing Snow Leopard was that GWT hosted mode no longer worked. You'll see the message "You must use a Java 1.5 runtime to use GWT Hosted Mode on Mac OS X." After spending about 10 minutes convincing myself that I was in fact using jdk1.5 for eclipse, ant, etc., and like, wasn't this working last week? I finally looked at the jdk symlinks in JavaVM.framework and figured out that 1.5 was just pointing to 1.6... interesting.

There was some discussion on the GWT group, along with a proposed fix of downloading someone's packaged leopard JDK and changing the symlinks. Not a great fix...

The Lombardi development team has come up with a great work-around.

I put together a jar with the modified BootStrapPlatform code (contains both .class and .java), or get just the src here.

Here are some step-by-step instructions for getting this working in Eclipse:

add the gwt-dev-mac-snow jar to your Java Build path.

in Java Build Path -> Order and Export, move the gwt-dev-mac-snow jar above the GWT SDK Library.

go to Run->Run Configurations. In Web Applications->(your GWT project), click on Arguments, then add -d32 under VM arguments.

That's it! You should now be able to run GWT hosted mode on Snow Leopard.

Setting up AWS keys for Eclipse

2009-08-11T10:52:00.000-07:00

One somewhat annoying thing about running JUnit tests in Eclipse is that they do not inherit your system's environment variables. There are good reasons for this, but we pass our AWS credentials to all of our applications via system variable, and it's a pain to add these to every single run configuration that needs them. This gets especially tedious when a significant number of your JUnit tests require AWS access.

As a workaround, you can add "Default VM Arguments" to the JVM you use to run your tests. Simply go to "Preferences->Java->Installed JREs" and edit your default JVM. Right under the JRE name is a space to add default VM arguments. I simply added "-DAWS_SECRET_ACCESS_KEY=foo -DAWS_ACCESS_KEY_ID=bar", and now I no longer need to manually edit individual run configurations.

This method seems a bit hacky to me, but until I can get a global run configuration, it definitely beats manually setting common environment variables for individual tests.

Dependency management for Scala scripts using Ivy

2009-07-22T14:15:00.000-07:00

I'm quickly becoming a huge fan of Scala scripting. Because Scala is Java-compatible, we can easily use our existing Java code base in scripts. This is especially convenient as we're moving our reporting to Hive, which supports script-based Hadoop streaming for custom Mappers and Reducers.

The one very annoying thing about Scala scripting is managing dependencies. My initial method was to have my bash preamble manually download the required libraries to the current directory and insert them onto the Scala classpath. So, my scripts looked something like this:


#!/bin/sh

if [ ! -f commons-lang.jar ]; then
s3cmd get [s3-location]/commons-lang.jar commons-lang.jar
fi

if [ ! -f google-collect.jar ]; then
s3cmd get [s3-location]/google-collect.jar google-collect.jar
fi

if [ ! -f hadoop-core.jar ]; then
s3cmd get [s3-location]/hadoop-core.jar hadoop-core.jar
fi

exec /opt/local/bin/scala -classpath commons-lang.jar:google-collect.jar:hadoop-core.jar $0 $@

!#
(scala code here)

This method has some rather severe scaling problems as the complexity of the dependency graph increases. I was about to step into the endless cycle of testing my script, finding the missing or conflicting dependencies, and re-editing it to download and include the appropriate files.

Fortunately, there was an easy solution. We're already using Ivy to manage our dependencies in our compiled projects, and Ivy can be run in standalone mode outside of ant. The key option to use is the "-cachepath" command line option, which causes Ivy to write a classpath to the cached dependencies to a specified file. So, now the preamble of my scripts looks like this:


#!/bin/bash

tempfile=`mktemp /tmp/tfile.XXXXXXXXXX`

/usr/bin/java -jar /mnt/bizo/ivy-script/ivy.jar -settings /mnt/bizo/ivy-script/ivyconf.xml -cachepath ${tempfile} > /dev/null

classpath=`cat ${tempfile} | tr -d "\n\r"`

rm ${tempfile}

exec /opt/local/bin/scala -classpath ${classpath} $0 $@

!#
(scala code here)

Now all I need is a standard ivy.xml file living next to my script, and Ivy will automagically resolve all of my dependencies and insert them into the script's classpath for me.

Crisis averted. Life is once again filled with joy and happiness.

Pruning EBS Snapshots

2009-07-16T11:35:00.000-07:00

We've been using Amazon's Elastic Block Storage (EBS) for some time now. In a nutshell, EBS is like a "hard drive for the AWS cloud". You simply create an EBS volume and then mount it on your EC2 instance. You then read/write to it as if it were local storage. For a good intro to EBS, check out this RightScale blog post.

The snapshots feature of EBS is especially handy as it allows you to easily backup the data on your EBS volume. AWS provides an API that allows you to request a snapshot. The API call will return immediately and then, in the background, the backup will occur and eventually be uploaded to S3.

While the snapshots feature is useful, one of the issues that you will likely run into is the snapshot limit. A standard AWS account allows you to have 500 EBS snapshots at any given time. After this limit has been reached, you will no longer be able to create new snapshots. So, you will need to have a strategy to 'prune' (remove) snapshots.

I wasn't able to find any scripts for pruning EBS snapshots on the web so I ended up writing a little Ruby script to accomplish the task.

You can get the script here. It requires the excellent right_aws ruby gem.

custom map scripts and hive

2009-07-14T17:20:00.001-07:00

First, I have to say that after using Hive for the past couple of weeks and actually writing some real reporting tasks with it, it would be really hard to go back. If you are writing straight hadoop jobs for any kind of report, please give hive a shot. You'll thank me.

Sometimes, you need to perform data transformation in a more complex way than SQL will allow (even with custom UDFs). Specifically, if you want to return a different number of columns, or a different number of rows for a given input row, then you need to perform what hive calls a transform. This is basically a custom streaming map task.

The basics

1. You are not writing an org.apache.hadoop.mapred.Mapper class! This is just a simple script that reads rows from stdin (columns separated by \t) and should write rows to stdout (again, columns separated by \t). It's probably worth mentioning this again but you shouldn't be thinking Key Value, you need to think about columns.

2. You can write your script in any language you want, but it needs to be available on all machines in the cluster. Any easy way to do this is to take advantage of the hadoop distributed cache support, and just use add file /path/to/script within hive. The script will then be distributed and can be run as just ./script (assuming it is executable), or 'perl script.pl' if it's perl, etc.

An example

This is a simplified example, but recently I had a case where one of my columns contained a bunch of key/value pairs separated by commas:


k1=v1,k2=v2,k3=v3,...
k1=v1,k2=v2,k3=v3,...
k1=v1,k2=v2,k3=v3,...

I wanted to transform these records into a 2 column table of k/v:


k1	v1
k2	v2
k3	v3
k1	v1
k2	v2
...

I wrote a simple perl script to handle the map, created the 2 column output table, then ran the following:


-- add script to distributed cache
add file /tmp/split_kv.pl

-- run transform
insert overwrite table test_kv_split
select
  transform (d.kvs)
    using './split_kv.pl'
    as (k, v)
from
  (select all_kvs as kvs from kv_input) d
;

As you can see, you can specify both the input and output columns as part of your transform statement.

And... that's all there is to it. Next time... a reducer?

Load testing with Tsung

2009-07-07T13:44:00.000-07:00

One of the big issues with building scalable software is making tests scale along with the application. A high performance web application should be tested under heavy loads, preferably to the breaking point. Of course, now you need a second application that can generate lots of traffic. You could use something simple like httperf; however, this doesn't work so well with complex systems, since you're only hitting one URL at a time.

Enter Tsung. Tsung is a load testing tool written in Erlang (everybody's favorite scalable language) that can not only generate large amounts of traffic, but it can parametrize requests based on data returned by your web application or with data pulled from external files. It also can generate very nice HTML reports using gnuplot.

Here's how we're running Tsung on Ubuntu in EC2:

Start a new instance. We're using an Ubuntu Hardy instance build by Alestic.

Download, configure, compile, and install Erlang.

Get and install the Tsung dependencies: gnuplot and perl5.
Download, configure, compile, and install Tsung.

Install your favorite web server. I prefer Apache HTTPD...others in this office perfer nginx. If you want to be really Erlang-y, install Yaws or Mochiweb.

Configure your ~/.tsung/tsung.xml configuration file for your test. The Tsung user manual has pretty good documentation about how to do this. Note that you do NOT want to use vm-transport for heavy loads, as this prevents Erlang from spawning additional virtual machines, which limits the number of requests you can use at a time. This does require you to set up passwordless ssh access to localhost.

Point your web server at "~/.tsung/log/". Each test you run will log the results in a subdirectory of this location.

Start your test with the "tsung start" command.

Set the report-generating script /usr/lib/tsung/bin/tsung_stats.pl to run in the appropriate log directory every 10 seconds. You can do this via crons or simply having a "watch" command running in the background.

Now, you can just browse over to your machine to view the latest test report. Tsung exposes all of the statistics you would expect (req/sec, throughput, latency, etc) both in numerical and graphical form. All of the graphs can be downloaded as high quality postscript graphs, too.

If you want to generate truly large amounts of traffic, Tsung supports distributed testing environments (as you might expect from an Erlang testing tool). Just make sure that you have passwordless SSH set up between your test machines and configure the client list in your tsung.xml file appropriately.

custom UDFs and hive

2009-06-23T15:56:00.001-07:00

We just started playing around with Hive. Basically, it lets you write your hadoop map/reduce jobs using a SQL-like language. This is pretty powerful. Hive also seems to be pretty extendable -- custom data/serialization formats, custom functions, etc.

It turns out that writing your own UDF (user defined function) for use in hive is actually pretty simple.

All you need to do is extend UDF, and write one or more evaluate methods with a hadoop Writable return type. Here's an example of a complete implementation for a lower case function:

package com.bizo.hive.udf;

import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;

public final class Lower extends UDF {
  public Text evaluate(final Text s) {
    if (s == null) { return null; }
    return new Text(s.toString().toLowerCase());
  }
}

(Note that there's already a built-in function for this, it's just an easy example).

As you've probably noticed from the import statements, you'll need to add buildtime dependencies for hadoop and hive_exec.

The next step is to add the jar with your UDF code to the hive claspath. The easiest way I've found to do this is to set HIVE_AUX_JARS_PATH to a directory containing any jars you need to add before starting hive. Alternatively you can edit $HIVE_HOME/conf/hive-site.xml with a hive.aux.jars.path property. Either way you need to do this before starting hive. It looks like there's a patch out there to dynamically add/remove jars to the classpath, so, hopefully this will be easier soon.

example:

# directory containing any additional jars you want in the classpath
export HIVE_AUX_JARS_PATH=/tmp/hive_aux

# start hive normally
/opt/hive/bin/hive

Once you have hive running, the last step is to register your function:

create temporary function my_lower as 'com.bizo.hive.udf.Lower';

Now, you can use it:

hive> select my_lower(title), sum(freq) from titles group by my_lower(title);

...

Ended Job = job_200906231019_0006
OK
cmo 13.0
vp 7.0

Although it's pretty simple, I didn't see this documented anywhere so I thought I would write it up. I also added it to the wiki.

Force.com's SOAP/REST library for Google App Engine/Java

2009-06-11T17:30:00.000-07:00

As long as I'm reflecting on our Google I/O experiences, I also want to point out what looks like a very useful library from Salesforce. The Force.com Web Services Connector is a toolkit designed to simplify calling WSDL-defined SOAP and REST services. The best part is that they have a version that works on Google App Engine for Java! (Make sure that you use wsc-gae-16_0.jar, not the regular version.)

I haven't had the chance to do a lot of development on GAE/J, but my colleagues have definitely had some headaches getting SOAP and REST calls working around the GAE/J whitelist. Maybe one of them can comment after we give this toolkit a whirl.

Google Visualizations Java Data Source Library

2009-06-11T11:03:00.000-07:00

As with any data-oriented company, most of our projects revolve around collecting data, processing data, and exposing data to users. In that third category, we've been moving towards Google Visualizations to draw our pretty graphs and charts. So, while the free Android phone and Google Wave were attracting a lot of attention at Google I/O, from a practical standpoint, I was actually most excited about Google's new Data Source Java Library. We had previously written something similar to this in-house, but we were still working on some of the optional parts of the specification when this library was released.

In a nutshell, Google Visualizations is a Javascript library that draws charts and graphs. The data is inserted in one of three ways: programatically in Javascript, via a JSON object, or by pointing the Javascript at a Data Source URL. For example, Google spreadsheets have built-in functionality to expose their contents as a Data Source, so you can just point the Javascript at a special URL, and a graph of your spreadsheet's data will pop up on your webpage. If you use the last method, you can use Gadgets to easily create custom dashboards displaying your data.

The Data Source Java Library makes it very easy to implement a Data Source backed by whatever internal data store you might be using -- it's just a matter of creating a DataTable object and populating it with data. The library provides everything else, up to and including the servlet to drop into your web container. (We ended up implementing a Spring controller instead. The library provides helper code for this; I estimate using a Spring conroller instead of a servlet cost us four lines of code.)

The best part is that it also implements a SQL-like query language for you, so you can expose your data in different forms (which are required by different visualizations) based on the parameters to the URL you call. Dumping data into JSON objects is very straightforward. Writing a parser and interpreter for queries is a real pain.

The library lets you specify how much of the query language you want to implement and which parts you want to make the library worry about. The only (small) complaint I have about this is that this configuration is rather coarsely defined -- we wanted to support basic column SELECTs (to improve performance on our backend) but have the library handle the aggregation functions (which our backend does not support). It wasn't too tough working around this restriction, although it does cost us a bit of extra parsing (so we can get a copy of the complete query) and column filtering (because both our code and the library processes the SELECT phrase).

new version of s3-simple

2009-05-20T18:29:00.001-07:00

Just committed some small changes to the s3-simple library for specifying ACLs, and/or arbitrary request headers/meta-data while storing keys.

Example usage:


S3Store s3 = new S3Store("s3.amazonaws.com", ACCESS_KEY, SECRET_KEY);
s3.setBucket("my-bucket");

// upload an item as public-read
s3.storeItem("test", new String("hello").getBytes(), "public-read");

// upload a js file, with a cache control-header
final Map<String, List<String>> headers = new HashMap<String, List<String>>();
headers.put("Cache-Control", Collections.singletonList("max-age=300, must-revalidate"));
headers.put("Content-Type", Collections.singletonList("application/x-javascript"));

s3.storeItem("test2.js", new String("document.write('hello');").getBytes(), "public-read", headers);

Download it here.

Currently, you can only do this while storing keys, and there's no way to retrieve this data later. Still, it was enough of a pain to get this working correctly with the request signing, so I figured I'd share the code anyway.