Performant cache structure to save duplicate calculations

Rubyist love the DRY moto (Don’t Repeat Yourself). Here’s how to implement a cache structure so that we Don’t Calculate Twice.

What we want:

  • Don’t compute anything twice, until there is a good reason to think the result might change
  • Any change in the database should be immediatly visible to users (no caching for a few minutes, hoping things won’t change too fast)
  • Don’t compute anything that has not be requested by a user (i.e. don’t pre-calculate everything )

Sounds awesome, how do we proceed ?

Objective:

In this tuto I’ll show you how to process billions of data with minimal efforts and code with elastic mapreduce and hadoop-stream. Our list to Santa is :

  • I want to process an unknown amount of data in a scalable, custom way
  • The same code can be run locally and remotely, so I can debug or process any amount of data without changing anything
  • I should be able to use any language I like, and this does not especially have to be java. In this example I’ll be using ruby because it’s awesome. Simply translate this tuto to python, perl, php or anything you want, it’ll still work.

When building an elaborated api, you might want your users to pass parameters in order to describe exactly what response they expect from the api.

For instance, it is useful to be able to do something as

1
2
3
GET '/users/:id', {users: {name: true}}  # => find a user and get his name only
GET '/users/:id', {users: {email: true, name: true}} # => find a user and get both his name and email
GET '/users/:id', {users: {email: true, name: true, posts: true}, post: {title: true}} # => find a user and get his name, his email, and all his posts titles

instead of presenting all of the model’s attributes at each api call.

Today we’re going to play around with functional programming. Yay !

  • Ok, what’s that exactly ?

Wikipedia says it pretty much:

functional programming is a programming paradigm, a style of building the structure and elements of computer programs, that treats computation as the evaluation of mathematical functions and avoids state and mutable data.

  • Alright, now let’s have some fun with functional programming style, and of course, let’s do that with ruby :)

In this post we’re going to manipulate some (high order) functions, and build a derivative operator in a functional style.

I think the Universe’s laws of physics are now way outdated. Let’s suggest a patch for this.

Universe::Physics v 2.1.0 release notes

  • Quantum tunnel glitch : fixed a bug that randomly caused particles to go through well potentials although they lacked the energy to do it.
  • High velocity engine improved : heavy computations at high speed caused local space-time to bend. The engine is now better and the so-called relativistic effects are now minimized.
  • Improved data structure (Heisenberg’s bug): Variables are stored with more efficiency. It is now possible to know both where a particle is, and at what speed it travels.