• Open Sourcing Task Library For Queueing Reliable Task Execution

    •

    Here at Ad Hoc Labs we are open-sourcing one of the projects I started to handle reliable execution of different tasks.

    I wanted to build a library that would:

    • have a higher level of abstraction than queuing
    • allow development of new types of tasks with only config changes
    • be back-end independent, so backends like RabbitMQ and Kafka would work based on config
    • allow flexibility to have publishers running in one project and consumers running in another
    • allow flexibility to decide which projects run what consumers and producers and how many of them.

    Our product Burner is a privacy layer for phone numbers. There are many things we want to run in a reliable manner, and we need to queue tasks and execute things in an asynchronous way. We are a Scala shop and among many Scala libraries we also use Akka.

    In the Akka world, we have actor A sending a message to actor B. There are many things that can go wrong, even if both are running in same JVM. Actor B may be down when actor A is sending the message. Actor B may fail to process the task, so there is a need for retries. Some tasks can take long time to process, so there is a need to run things asynchronously. You may want to scale out and have many instances of actor B, etc.

    This actor model is very clean and well-suited for this type of situation. And if you’re familiar with Akka, you know that Akka handles some of those cases out of the box, some via Akka persistence, and some things simply aren’t there.

    So I wanted to have a very simple library that wouldn’t require a Cassandra cluster, force you to write a lot of boilerplate, or force the use one specific message queue – I wanted it to be extremely flexible and extendable.

    The core idea is the same: actor A sends a task to actor B to process, and actor B will eventually get the task. It’s very similar to the way you send a message from actor A to actor B in Akka; there’s no other complexity there.

    read more

  • Ammonite Modules

    •

    Ammonite is such a great project, especially Ammonite-REPL. It brings many great things to scala console: syntax highlighting, better editing, dynamically loading dependencies, loading scripts, etc. It comes very handy if you want to play with a some library without creating an sbt project. Or you want to try something out without necessarily loading IntelliJ. I happen to do it often and decided to start an open source project called ammonite-modules. The idea is to have pre-defined modules you can load in Ammonite Repl. Modules will have all the dependencies needed at latest versions and do some basic setup.

    read more

  • Akka Http Request Throttling

    •

    Akka Http is a great framework for rest microservices and here at Ad Hoc Labs we’re slowly migrating from Spray to Akka Http. Since we use Spray extensively we ended up having utility classes to complement some functionality here and there or shared in a common jar with other utility classes. With Akka Http we want to keep things little more clean and reduce boilerplate with high level constructs. We decided to start a new project named akka-http-contrib that will host all akka http functionality that we’re building that’s not already in akka-http project. We recently open sourced it since other community members will find it helpful and hopefully will contribute also.

    First functionality we needed for a new project we started with Akka Http was to introduce throttling for some endpoints. We didn’t want to write boilerplate for each endpoint we want to introduce throttling for but instead we wanted to be able to introduce throttling for any endpoint by adding it to typesafe config. Basically we wanted a directive that will wrap the routes and when request comes in, depending on the config and thresholds, either let the route execute or complete the request with 429 - Too Many Requests and prevent the route from being executed. This at high level, which should handle most of the use cases but we also wanted to have it extensible enough that whenever there is a special usecase it still can be handled by writing little code.

    read more

  • Writing Aws Lambdas In Scala

    •

    AWS Lambda supports writing handlers in Nodejs, Python and Java. The first two are really easy - if you don’t have any external dependencies and script is easy to write, you can even write your lambda in AWS console. However when it comes to Java, things get a little more complicated than that (not even talking about Scala yet.) So at the very least you’ll need to create a new project, know what jar dependencies to add for lambda, implement the handler method in a class that potentially implements some interface, know how to generate a deployment package (fat jar or a zip file for java projects), upload to S3 (since that jar would have a decent size), etc. If I want to implement my lambda in Scala now, I would have to deal with all those complexities for Java projects and more - such as more dependencies, how to write an actual Scala code (not Java in Scala syntax), interoperability, immutability, built in features like case classes, futures, etc. I thought it’s not fair for Scala developers and things can be much simpler (maybe not Nodejs simple) but at least it has to be simpler than doing in Java, since we all know Scala has more things to make developer’s job easier.

    read more

  • Moving My Activator Templates To Gitter8

    •

    I had few activater templates for a while now that I use to start different types of Scala projects. I open sourced those and made available via Github and activator website. There was lots of traction from the community and since Activator is deprecated now I decided to take those and convert to Gitter8 templates so sbt new can be used to start projects. The new templates work with sbt 1 and all the dependencies, including sbt plugins in the projects, are upgraded.

    read more