Open Source at Frugal Mechanic
Posted on 17 January 2014, 8:10 AM PST by Tim Underwood under At Frugal Mechanic
We use a ton of Open Source Software here at Frugal Mechanic. Here is a look at some of our favorites:
The vast majority of our code is written in Scala. We find Scala to be very productive and a mix of the type-safety of a statically typed language with features usually only found in dynamic languages (e.g. Concise Syntax, Higher Order Functions, Type Inference, REPL, String Interpolation, etc...).
Here is a presentation I gave on Scala to the Seattle Tech Startups meeting from back in February 2011. The Powerpoint version of the presentation is available here.
We have used Apache Solr since day 1 here are Frugal Mechanic. After processing all of our part and pricing data feeds on the backend we index the data into Solr which powers the parts catalog on our website.
We have also used HAProxy since day 1 to handle load balancing of all of our traffic (both user traffic to our web servers and backend traffic to Apache Solr). We used to run incoming HTTP/HTTPS traffic through Apache (for HTTP Keep-Alive, SSL, and virtual host handling) before proxying to HAProxy but now that HAProxy 1.5 has both Client Side HTTP Keep-Alive and SSL support we are able to remove Apache and send all HTTP/HTTPS traffic directly to HAProxy.
We use Git for all of our source control needs. We love that it is fast, distributed (which gives multiple complete copies of our repository), has easy branching and is open source.
Server Software
- Linux - Our servers are running a combination of Debian and Ubuntu
- Nagios - For monitoring our servers and services (and even retailer website we link to). It is a pain to figure out and setup but well worth it.
- Munin - Server performance stats with pretty RRD charts
Java Libraries
One of the great things about Scala is that it runs on the Java Virtual Machine and is compatible with existing Java Libraries. Here are some of our favorite Java Libraries:
- Netty - Our web server is built directly on top of Netty for maximum performance and maximum flexibility.
- Woodstox - We deal with a lot of XML feeds and they all get streamed using Woodstox and JAXB.
- HtmlCleaner - We like well-formed HTML (and passing the W3C Markup Validator!) so we run any external HTML (e.g. part descriptions, brand descriptions) and even some of our own HTML (e.g. blog posts) through HtmlCleaner.
- Protocol Buffers - Most of our data is stored using the Protocol Buffers format for it's compactness
- Snappy Java - A JNI-based implementation of Snappy that we use to compress a lot of our data
- Apache POI - Used by our PIES Excel to XML Converter Tool for reading Excel XLS/XLSX files.
- Apache Commons - We use a ton of libraries from the Apache Commons project and a lot of the libraries we depend on also use libraries from Apache Commons
- Joda Time & Joda Money - We use Joda Time and Joda Money (along with some Scala implicit wrappers) for all of our time/money handling.
- HtmlUnit - We use HtmlUnit to automate logging in and downloading reports from our retail partners who don't have APIs.
Scala Libraries
And here are some of our favorite Scala libraries:
- Scalate - Most of our web site templates are rendered using Scalate.
- ScalaTest - For all of our testing needs.
Updates
27 January 2014 - Added Git