Interest and Learning

I’ve been involved in teaching lately (wrote a book, teaching in NYC Monday and Tuesday, mentoring at a Scotiabank hackathon later in the month, and mentoring someone) so I’ve been thinking about how people learn a bit more. I had a chat with one of Kate’s friends last night about the education system and apparently some of the earlier grades are moving toward open ended learning. Instead of having a strict assignment to follow (color in the lines) a bunch of material is placed and a child has to explore that space. Much of the lego today has instructions for building a certain thing. If you follow the instructions, you’ll have your tie fighter just like on the tin.


I saw this article today which is what got me thinking about this more:
“Try to engineer a certain kind of success, and the best you’ll get is an ambitious robot. If you want your children to bring original ideas into the world, you need to let them pursue their passions, not yours.”

I think we’ve been doing it wrong.

Indian music is an interesting case to look at. In our schools of music, we expect our students to be able to take a piece of sheet music and play exactly what is written on the paper. A good musician is one who can play the things on paper very well with expression exactly as described on the sheet. Just like a kid following the instructions for his Lego tie fighter. People who write music are those that have a personal interest that leads them to explore music on their own terms. Everyone else is stuck in a box and wouldn’t be able to improvise. Why can’t the same thing apply to music though? Maybe an open ended approach works there too?

In Indian classical music, a “raga” is a series of 5 to 9 notes in a scale that are a “tonal framework for composition and improvisation.” Someone learns the raga and then they improvise and play with it. There isn’t a strict “here is how you play this” – it’s an open ended problem. Like a bunch of blocks on the table. The student has to explore that space on their own and put it together in their own way. It’s blank canvas and the student gets to paint whatever they fancy. Can you imagine if we took our kids and asked them to do this today? They wouldn’t know what to do because we train them to be obedient robots.

We need to nurture the creativity in our young, to teach them how to think and play. To find their interests and nurture them instead of training them to play the game, to color in the lines, to play the music that is written. One of the biggest complaints from my profs in school was that most people aren’t able to think. Handing people an open ended problem was very challenging to a lot of people. Without strict detail of what was expected, a student would not be able to complete a task. What happens when people get into industry then? They can complete tasks handed to them, but they can’t innovate and drive change toward their vision. Doing things because it’s the way they’ve always been done means a company will stagnate. Once someone questions the status quo and has a vision for change, then they can impact the world. I asked my friend the other day what he looks for when hiring and he replied with “curiosity.” That’s the quality that a person needs to grow and adapt and change. We need our children to grow up understanding that their interests and curiosity are important, not simply that they need to be able to do things exactly as instructed.

Interest and Learning

Brainstorming Circuit Breaker Design


I’m starting to work on an alternative circuit breaker implementation for use with Scala this weekend. I’m hoping to solicit some feedback or get some discussion on what an ideal implementation would look like. I have a few ideas but I’m hoping some experienced eyes might be able to steer me if I should be going in a different direction. I’d really love to see a good simple flexible and efficient circuit breaker implementation but as I work on it I’m seeing my ideas are changing on what the ideal should be.

Akka has a circuit breaker implementation:

I’ll explain why I’m looking to extend the work by reviewing the behaviour of the Akka circuit breaker as I understand it. I believe that the efficiency and simplicity are plusses but I’m not 100% sure that the behaviour is exactly what I would want when looking at scaling systems. I’d like to have a couple different tools ready to use for myself.

The akka circuit breaker takes an int representing the max number of errors it should see before opening. Once it goes above that threshold, then the circuit breaker will open. After a period of time, it changes state to half open where it tries 1 request. If that succeeds, it closes, resuming system behaviour as normal. If it fails, then it re-opens for the period of time specified. When the circuit breaker is open, it will fail all requests. This is to let downstream systems recover if they get overwhelmed. The Akka circuit breaker treats any failure as a +1 to the error count. So if you’re using a database client and it returns a failure on a request because there is no record, instead of a Successful future with a failure inside (eg Future[Try[Record]]) then that causes the error count to increase. The circuit breaker’s intention is to let downstream systems heal if they become over-loaded so business logic failure shouldn’t trip the circuit breaker – only load related failures, such as timeouts. I can see that there may be errors that occur related to load and so 100% failures might be worth tripping the circuit breaker.

The circuit breaker implementation from the Akka team also only counts the errors, and resets the count after it opens. – if you have a 10x increase in traffic over a year, then you might be tripping the circuit breaker 10x more often without going back and changing config, even if systems are healthy. It seems that the errors are counted, rather than using using stats, like 75% of requests sampled must be failures to trip the breaker.

The requests aren’t windowed, (but are sequential), so if 99 errors occurred 10 minutes ago, and then the system is fined and suddenly one request is dropped and times out then the breaker will trip, and you’ll be failing requests even if there is no issue in the system anymore.

The requests must be actual failures – eg they must timeout completely. I would think it would be better to have the breaker trip once the requests are above 80% of the timeout or something similar – so the requests outside of +3 sigma aren’t failing. So I think the latency should be measured instead of failures.

I’m also thinking that if the breaker were to take a % of samples – say every 3rd request, instead of checking every request, that it might be possible to implement more of the features without the cost of synchronous contention of resources like the thread-safe count, then the breaker could be much more efficient. If the breaker takes a sample every x ms, then the cost of the breaker should ceiling as well.

Now the downside of the features is added complexity and overhead. I can implement windowing of the requests by taking samples at the resolution of one second, and just dropping the older samples once they’re rolled out of the window (say after 10 seconds – so we only keep 9 previous samples + the current second’s count). Currently on every ‘tick’ we can check the error rates instead of constantly trying to calculate how many errors have occurred.

My first pass only trips it on the ‘tick’ but I think checking it every request is simpler. If the sample size is only a fraction of the traffic, this should really reduce the cost as the synchronization will be more of an issue without reducing the sample size otherwise.

Let me know what you think – I think the cost of a bit of complexity might make the circuit breaker useful as long as the performance cost is minimal.

Brainstorming Circuit Breaker Design

Building Wireshark on Ubuntu

Wireshark is available in the Ubuntu Software Center, but if you want to build the latest package, you can build it like so;

– Go to the Wireshark Download page
– Navigate to the third party standard packages page for ubuntu (here at time of writing:
– Download the source
– install missing dependencies to build, likely: sudo apt-get install libpcap-dev vsftpd libgtk-3-dev
– untar the source and run: ./autogen
– then run: ./configure –with-gtk3
– run: make
– run: make install

Building Wireshark on Ubuntu

Akka CircuitBreaker Example

Hello fellow Hakkers,

If you’re working with Scala and asynchronous I/O – especially in distributed computing – then you should quickly get familiar with the circuit breaker pattern.

Akka has a simple circuit breaker implementation and it has absolutely nothing to do with Actors. If you’re using Scala futures, you would almost certainly benefit from the implementation of circuit breakers.

Let’s look at what happens when we talk to a service in most normal circumstances.

You may be sending requests to a database or another service, and that service might have a known or estimated maximum throughput – say 100 requests per seconds per instance.

Then, you have the consumers of that service, which could be live users. The known throughput of the input is unknown and can potentially fluctuate 100x throughout the day. If one day your article shows up on hacker news, then you might see 1000x more traffic than you had anticipated.

Normally that fluctuation is tolerable – you’ll end up queuing several requests, and they wait until your system is able to get to your request. Let’s say we want to parse articles for an iphone reader application. You might have some slow consumer in the data pipeline like so:


What happens when you have a large numbers of requests coming in is that the messages start to queue up. It takes longer and longer to process each message.


Eventually your slow consumer will take such a long time that users will get impatient and start hitting retry, only compounding the issue. Services will timeout and users will wait tens of seconds to see errors.

This is a violation of our responsibility to be responsive in all cases.

Ultimately your consumer might end up storing or attempting to process so many messages that it runs out of memory and crashes.

This is a violation of our responsibility to built resilient systems.

Enter the circuit breaker.

If, instead of accepting messages even when the response of downstream systems is slow, we temporarily fail the responses immediately, telling users to come back later, then we start to have a different picture of the problem.


Now the downstream services can “catch up” and heal so we have two benefits:

– We don’t make users wait for an error (responsive)
– We let our systems heal (resiliency)

If the requests are going to timeout and fail anyway, we might as well not compound the problem by throwing more requests on.

The circuit breaker pattern will monitor timeouts and failures between the producer and consumer of requests. Normally it will not affect behaviour. This is the “closed” state for the circuit breaker:


Then, if downstream, a consumer gets overwhelmed and lots of errors or timeouts are occurring, the circuit breaker opens, giving failure messages back to the user quickly.


For timeouts, you can roughly guess the period of time needed based on the the measured max throughput of down service systems against the timeout latency thresh-hold. (ignoring all dependencies, if the system can process 100req/s and the timeout is at 5 seconds, there should be 500 messages queued so it will take 5 seconds to recover.

After waiting a period of time for recovery, the circuit breaker will change states to “half-open” and will try a request to see if the downstream system has recovered. All other requests will still fail with exception.


The response of the sample’d requests is evaluated. If the request fails, we assume the system has not recovered yet, and we flip back to open. If it appears to have recovered, then we can fully close the circuit breaker and continue as usual.

I produced a small scala/akka example here which demonstrates a message producer producing messages faster than the consumer, and shows how the circuit breaker protects against long response times and a total meltdown (eg out of memeory errors). While the Akka toolkit contains a circuit breaker out of the box, you can certainly implement the circuit breaker in any language and system you are using.

Akka CircuitBreaker Example

Fixing Refs in Gerrit

This is just a reminder for myself.

If you push a bunch of commits incorrectly into gerrit, you can correct this fairly easily.

First, backup your local changes with

git checkout -b backupbranch

Then reset to unwrap the commits.

git reset –hard HEAD~2

Then you can commit then again with the correct change id.

git commit –amend

Push the refs and abort the other ones – that should be good now.

Fixing Refs in Gerrit

RHEL nginx init.d Script

I noticed that the behaviour of deamon is a bit different in RHEL – it won’t write a pid file. Here is a very simple init.d script for RHEL.

# nginx Startup script for nginx
# chkconfig: – 85 15
# processname: nginx
# description: nginx is an HTTP and reverse proxy server
# Provides: nginx
# Required-Start: $local_fs $remote_fs $network
# Required-Stop: $local_fs $remote_fs $network
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: start and stop nginx

# Source function library.
. /etc/rc.d/init.d/functions

#if [ -f /etc/sysconfig/nginx ]; then
#. /etc/sysconfig/nginx




start() {
echo -n $”Starting $prog: ”
daemon –user=${user} –pidfile=${pidfile} ${nginx} -c ${conffile}
[ $RETVAL = 0 ] && touch ${lockfile}
return $RETVAL

stop() {
echo -n $”Stopping $prog: ”
#killproc -p ${pidfile} ${prog}
$nginx -s stop
[ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}

configtest() {
if [ “$#” -ne 0 ] ; then
case “$1” in
${nginx} -t -c ${conffile} $FLAG
return $RETVAL

# See how we were called.
case “$1″ in
rh_status >/dev/null 2>&1 && exit 0
configtest -q || exit $RETVAL
echo $”Usage: $prog {start|stop|restart|help|configtest}”

exit $RETVAL

RHEL nginx init.d Script

How to Make An SBT Project that Can Run In Play or a Plain Java Project

It’s not clear from any of the Play documentation, but it’s possible to make plain sbt/maven projects that can optionally plug into the Play API (eg use Play promises or a running Play Application).

You’ll want to isolate places that the play API is touched but you can plug Play into a regular sbt project by adding dependencies like so:

name := "concurrency-utils"

organization := ""

resolvers += "Typesafe repository" at ""

scalaVersion := "2.11.1"

libraryDependencies ++= Seq(
  // Uncomment to use Akka
  //"com.typesafe.akka" % "akka-actor_2.11" % "2.3.6",
  "junit"             % "junit"           % "4.11"  % "test",
  "com.novocode"      % "junit-interface" % "0.10"  % "test",
  "" % "play_2.11" % "2.3.0",
  "" % "guava" % "18.0"
How to Make An SBT Project that Can Run In Play or a Plain Java Project