I recently wrote an SBT plugin that allows you to resolve and publish using Amazon's S3 service. This is useful if you want to use S3 as a private Maven (or Ivy) repository. I've released the plugin as open source under the Apache 2.0 License.

The source code is available on GitHub: https://github.com/frugalmechanic/fm-sbt-s3-resolver

Usage

The plugin is available on Maven Central so all you have to do is add it as a plugin to your project/plugins.sbt file:

addSbtPlugin("com.frugalmechanic" % "fm-sbt-s3-resolver" % "0.9.0")

You also need to make your S3 credentials available to the plugins via one of the methods specified in the README.md file. The easiest method is to just create a ~/.sbt/.s3credentials file of the following format:

accessKey = XXXXXXXXXX
secretKey = XXXXXXXXXX

Resolving

Just add your s3:// URL to the SBT Resolvers:

resolvers += "FrugalMechanic Snapshots" at "s3://maven.frugalmechanic.com/snapshots"

Note: If your bucket is public then you don't need the plugin and can just use the normal https:// URL.

Publishing

Just use the s3:// URL:

publishTo := Some("FrugalMechanic Snapshots" at "s3://maven.frugalmechanic.com/snapshots")

URL Formats

Just take whatever the https:// (or http://) URL for your S3 bucket would be and replace it with s3://. In the examples above I'm using a maven.frugalmehanic.com as a Static Website Using a Custom Domain where maven.frugalmechanic.com is the bucket name. The long form would also work:

s3://s3-us-west-2.amazonaws.com/maven.frugalmechanic.com/snapshots/

I recently upgraded our HAProxy server from Debian Squeeze to Debian Wheezy and wanted to play around with the HTTP Keep-Alive and SSL support in the latest dev version of HAProxy 1.5 (1.5-dev21 as of this post). I thought maybe wheezy-backports would have the latest 1.5 dev version but couldn't find anything. I started trying to figure out how to install HAProxy from Debian experimental when I found this site: http://haproxy.debian.net/

Instructions (from http://haproxy.debian.net/)

Enable wheezy-backports by adding this to your /etc/apt/sources.list file OR by creating something like /etc/apt/sources.list.d/backports.list:

deb http://ftp.us.debian.org/debian wheezy-backports main

Enable the http://haproxy.debian.net/ repository by creating a /etc/apt/sources.list.d/haproxy.list file with this in it:

deb http://haproxy.debian.net wheezy-backports main

Make sure you have the debian-keyring package installed:

sudo apt-get install debian-keyring

Import the proper GPG keys:

gpg --no-default-keyring --keyring=/usr/share/keyrings/debian-keyring.gpg \
    --export [email protected] | sudo apt-key add -

Perform an apt-get update

sudo apt-get update

Finally, install HAProxy 1.5:

sudo apt-get install haproxy -t wheezy-backports

You should now have the latest HAProxy 1.5 version (which is 1.5-dev21 as of this blog post).

What about Ubuntu?

There is a PPA available for Ubuntu here: https://launchpad.net/~vbernat/+archive/haproxy-1.5

We use a ton of Open Source Software here at Frugal Mechanic. Here is a look at some of our favorites:

The vast majority of our code is written in Scala. We find Scala to be very productive and a mix of the type-safety of a statically typed language with features usually only found in dynamic languages (e.g. Concise Syntax, Higher Order Functions, Type Inference, REPL, String Interpolation, etc...).

Here is a presentation I gave on Scala to the Seattle Tech Startups meeting from back in February 2011. The Powerpoint version of the presentation is available here.

We have used Apache Solr since day 1 here are Frugal Mechanic. After processing all of our part and pricing data feeds on the backend we index the data into Solr which powers the parts catalog on our website.

We have also used HAProxy since day 1 to handle load balancing of all of our traffic (both user traffic to our web servers and backend traffic to Apache Solr). We used to run incoming HTTP/HTTPS traffic through Apache (for HTTP Keep-Alive, SSL, and virtual host handling) before proxying to HAProxy but now that HAProxy 1.5 has both Client Side HTTP Keep-Alive and SSL support we are able to remove Apache and send all HTTP/HTTPS traffic directly to HAProxy.

We use Git for all of our source control needs. We love that it is fast, distributed (which gives multiple complete copies of our repository), has easy branching and is open source.

Server Software

• Linux - Our servers are running a combination of Debian and Ubuntu
• Nagios - For monitoring our servers and services (and even retailer website we link to). It is a pain to figure out and setup but well worth it.
• Munin - Server performance stats with pretty RRD charts

Java Libraries

One of the great things about Scala is that it runs on the Java Virtual Machine and is compatible with existing Java Libraries. Here are some of our favorite Java Libraries:

• Netty - Our web server is built directly on top of Netty for maximum performance and maximum flexibility.
• Woodstox - We deal with a lot of XML feeds and they all get streamed using Woodstox and JAXB.
• HtmlCleaner - We like well-formed HTML (and passing the W3C Markup Validator!) so we run any external HTML (e.g. part descriptions, brand descriptions) and even some of our own HTML (e.g. blog posts) through HtmlCleaner.
• Protocol Buffers - Most of our data is stored using the Protocol Buffers format for it's compactness
• Snappy Java - A JNI-based implementation of Snappy that we use to compress a lot of our data
• Apache POI - Used by our PIES Excel to XML Converter Tool for reading Excel XLS/XLSX files.
• Apache Commons - We use a ton of libraries from the Apache Commons project and a lot of the libraries we depend on also use libraries from Apache Commons
• Joda Time & Joda Money - We use Joda Time and Joda Money (along with some Scala implicit wrappers) for all of our time/money handling.
• HtmlUnit - We use HtmlUnit to automate logging in and downloading reports from our retail partners who don't have APIs.

Scala Libraries

And here are some of our favorite Scala libraries:

• Scalate - Most of our web site templates are rendered using Scalate.
• ScalaTest - For all of our testing needs.

Updates

27 January 2014 - Added Git