As some of you know, I have now been working for the past few years along with St├ęphane Brossier on Kill Bill, an Open-Source billing platform.

At first, one can wonder how complicated a recurring billing system can be? After all, all it needs is to maintain a list of subscriptions, generate an invoice every month, and trigger a payment? The picture actually becomes complicated quite quickly. The system also needs to know how to compute pro-rations for upgrades and downgrades, manage add-ons (if the user upgrades to a higher tier plan, maybe the add-on is already included and the customer shouldn’t be charged for it), support invoice adjustments and chargebacks, integrate with one or more payment gateways, produce accurate Analytics and financial dashboards, etc. But even if you manage to build it (which is non trivial to get it right and bugs will cost you money), it needs to be flexible enough to meet future requirements (what if the marketing team wants to distribute coupons all of a sudden, how will this logic fit into the rest?).

Killian Bill

Killian Bill, licensed to Bill

After having gone through that cycle a couple of times, we decided to build an Open-Source platform to solve the problem. We learned a lot along the way, not only about financial and accounting concepts, such as generating revenue recognition reports, but also about building and maintaining large software projects with a lot of business logic. Being back-end engineers, we both had extensive experience building distributed and scalable systems and spent our fair share of time in Jetty’s source code, or reading about the latest serialization formats. Kill Bill has been challenging in the sense that, for a billing system, these questions matter less (e.g. you are unlikely to have to manage a throughput of thousands of upgrades per second) compared to writing maintainable and testable code, with a lot of customizable business logic.

We decided to share our experience and have started a series of articles on the architecture of the system and why it is designed that way, called The Kill Bill story. If you are curious, go check it out!


For a little while, I have been maintaining an unofficial Java client library for Recurly. I originally wrote it for Kill Bill, in order to develop the Kill Bill Recurly plugin. I really only needed a few of the API calls to implement the Kill Bill integration, but since Recurly doesn’t seem to provide an official Java binding, I continue to support it beyond my initial needs as it has been getting more and more traction.

This is the type of Open-Source project which can be quite time consuming. Besides the bugs in the code itself, a lot can happen in the integration with the closed-source API. For example, some calls can behave wildly differently depending on the specified parameters, and although Recurly does a pretty good of documenting their endpoints, certain behaviors can fall through the cracks. During a debugging session, it is sometimes difficult to narrow down on the issue: is it a problem with the client code (e.g. NPE, bad serialization) or does the code not follow the expected implementation?

I recently got a bug report from oohira regarding that type of problem: https://github.com/killbilling/recurly-java-library/issues/23.

Long story short, when trying to remove add-ons from a subscription, the following payload was sent to Recurly:


which wasn’t accepted by the servers. Luckily, oohira went further and by trial and error, he realized that the following should be sent instead:

    <subscription_add_ons type="array"></subscription_add_ons> 

This distinction wasn’t specified in the documentation, which simply stated: The original subscription’s add-ons will be removed unless they are specified in your update subscription request.

Having this information, it was quite trivial to change the code to behave that way (and oohira found a client-side serialization bug in the process).

This type of detective work has been invaluable to get a fix in a timely manner. I do have a Recurly test tenant where I run regression suites, but since I don’t use these advanced features in the Kill Bill plugin, I am not as familiar with the API as oohira or others might be. Understanding where the issue was (code or API implementation) could have taken me much longer to fix.

Consider it for your issues. When you submit a bug report, besides providing a necessary reproducible test case, think about digging a little bit in the bug. Don’t assume everybody is as familiar as you are with the feature you are using. This will go a long way with the maintainer, and you’ll likely get a more timely response.


Update: the video is online!

I will be speaking at Ning next Wednesday, April 27, on our approach to Realtime Analytics.

Details here: http://www.meetup.com/Ning-Tech-Talks/events/17221783/.

If you’d like to give a lightning talk, let me know and I’ll put you on the schedule (it doesn’t have to be Analytics related).

For those who can’t make it (any French reader willing to fly-in just for the talk?), the video will be posted online the following week.