From Monolith to Microservice: Lessons learned from building Foxsports automation platform on Mule

I recently left Foxsports after long five eventful years. Over this period, I was very fortunate to be a part of the core backend team that built the foxsports core integration platform / microservice platform from scratch that automates the workflow from controlling the video router switch to delivering realtime sports data to amazon cloud handling 70 million api calls and servicing all live & statistical sports data to mastheads.

The heart of the framework was ScheduAll : A resource and personnel scheduling desktop app thats used widely amongst Broadcast and Network management companies to schedule WorkOrder. This is a tale of how we hooked into the thin integration layer of the system and integrated other network components of the System to give them life. How a WorkOrder booking evolved from a manual spreadsheet job order into self managing , self healing business intelligence framework that needs minimum to no human interaction. Thus transforming the traditional Broadcast infrastructure into a modern Stream based on demand system.

You can see our CTO talking more about it in this Mule Presentation.

This series of blog post will be my humble attempt to document some of the lessons learnt over the process of building a monolith and breaking it up into microservices v.1 and re-writing it again into v.2 over the period of 5 years.

I will try to focus on the following topics:

  1. The self emerging pattern of microservice from trying to do it right
  2. Api is the King – Api Management , Mule support for RAML, Api Gateway , AWS
  3. Splitting the Monolith – Shared api pattern , IPC
  4. Microservice Framework – Spring Integration vs Mule
  5. Event Driven Architecture, Customized persistent Queue Pattern  on Elasticsearch as CQRS implementation
  6. Publish-Subscribe alternative to EventBroker
  7. DataStore – To Share or Not
  8. Mule Flows vs Groovy vs Java Argument
  9. Distributed executors over HazelCast
  10. Troubleshooting – Logging , Event Monitor , Spike Detection , Right Amount of Logging, Logging Policy
  11. Microservice Deployment , Mule MMC , Containers (Docker) & Service discovery , Elastic Container Service , CloudFormation
  12. Testing – Integration vs Functional , Mocking Proxy vs Real Service, How much Unit Test is good, TDD – to do or not to do

By no way, I claim to be right or wrong. This is meant to be my Brain Dump and my thoughts in Retrospect….I am not sure how soon I can finish it though given my busy schedule in a new job and my lack of interest in writing, I have allocated myself 6 month for finishing up this full series 🙂

Configure Docker-client on OSX

So I am a recent MAC convert. Although I am a long time Ubuntu Fan and user but I got tired of switching OS between windows and linux , hacking around virtual box. Also I am more and more working on cloud platform now a days and using my machine as a thin client.

The first thing I wanted on mac is to pull my docker images and use them locally. However, Os X doesn’t support docker images natively. So the docker team came up with this excellent tool called docker-machine. More about docker machine here:

When you install docker toolbox , it also installs a ‘Docker Quickstart Terminal’ which you can use to get started with docker. But if you are like me, you want access to your docker engine from any terminal natively.

#find your local docker-machine ip (it should be

$ docker-machine ip

#verify your certs exists 

~$ ls ~/.docker/machine/
cache certs machines no-error-report

#who are you?

~$ whoami

#Update ~/.bash_profile 

$vi ~/.bash_profile

export DOCKER_HOST=tcp://
export DOCKER_CERT_PATH=/Users/sajid/.docker/machine/machines/default


~$ source ~/.bash_profile




What this does is it lets your local docker client point to your default docker-engine / docker-daemon.

Now you can run docker commands just like you can do on linux natively.

~$ docker images
williamyeh/ansible ubuntu14.04-onbuild ffb3155960ee 3 weeks ago 244.8 MB
ubuntu latest b549a9959a66 3 weeks ago 188 MB
jenkins 2.0-beta-1 a08ca387230c 3 weeks ago 711.9 MB
sameersbn/redis latest 100e6eb3355d 5 weeks ago 196.5 MB
hello-world latest 690ed74de00f 6 months ago 960 B latest 690ed74de00f 6 months ago 960 B
sameersbn/gitlab 7.14.3 ffbbcd99823b 7 months ago 631.6 MB
sameersbn/postgresql 9.4-3 40e7e3862c0c 8 months ago 231.6 MB


Once you configure your docker client, the next thing you would want is to push some local image in your insecure registry (note my environment is running behind a strict firewall, so its ok to run in insecure mode):

$ docker push
The push refers to a repository []
unable to ping registry endpoint
v2 ping attempt failed with error: Get tls: oversized record received with length 20527
v1 ping attempt failed with error: Get tls: oversized record received with length 20527


NOTE that docker-client is just a proxy between you and the docker engine. So any real config (DOCKER_OPTS) needs to be done on the docker daemon. Where is the docker daemon? Its on the virtualbox (you should see a virtual running with name ‘default’).

So you need to connect to it and update the docker OPTS

~$ docker-machine ssh
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
_ _ ____ _ _
| |__ ___ ___ | |_|___ \ __| | ___ ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__| < __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.10.3, build master : 625117e - Thu Mar 10 22:09:02 UTC 2016
Docker version 1.10.3, build 20f81dd

docker@default:~$ sudo vi /var/lib/boot2docker/profile 
--label provider=virtualbox
--insecure-registry mydockerrepo:5000

CTRL-D to exit docker machine

$ docker-machine stop
$ docker-machine start


All set, now you can use your docker environment in OsX just like you would do in native ubunto. No more docker-quickstart-terminal!!!

Docarizing ElasticSearch to Run in Amazon Container Service

Recently I spent almost two days trying to run Elasticsearch in a docker container and hosted under a Elastic Container Service. A word of caution, running a IO-Heavy app like ES under a container like docker might not be a good idea. Specially if you can’t use Native Networking and have to deal with abstracted bridge mode (Currently you can only run in Bridged mode in Elastic Container Service). However, our purpose is to provide search capability on a small set of data (less than 500K) and we were not concerned about the IO overhead, the maintenance benefit outruns the performance penalty so its a good fit for our requirement. Also, if you want to use these scripts for large volume of data , you will have to map larger disk space on the Host services (which is not done here).

While I am not going to describe step-by-step of what ECS is and what Docker is (There is plenty of material from Amazon on the topic), I’ll focus on sharing the code and project that has a working Elastic Image for Docker. Also, I had to re-build a docker image for Elasticsearch of my own because the default docker image that comes from Elasticsearch runs on open-jdk and I wanted Oracle SDK.

Without further Ado , Here is the git-hub project for the elasticsearch:

And the Elasticsearch-ECS container (with some basic plugin and elastic-ec2 plugin installed):

I have setup a build on docker-hub so feel free to use it directly from there:


Now, a note of credit , most of the work is copied from this blog post:

However, I have faced some issues mostly because I am trying to run Elastic 2.2 version. But you must read through this blog to understand the internal working principle of this blog post and because its so nicely detailed there, I am not going to repeat them here.

Cloud Formation:
I have used a custom cloud formation script to make sure my instances ends up in the special VPC structure specific to my organization, you can use the default Elastic ECS Cloud template if you just want to try out the feature.

Feel free to look at it if you have similar requirement:

With the Ready Docker Images and the recipe of how to use them, I think you should have a much smoother time setting it up , Best of luck 🙂

Using GraphDB to manage dynamic subscription

Recently we added functionalities for users to subscribe to their sports of interest so we can send them more personalized content. Given the dynamic nature of sports / match / fixture , I wanted the subscription topics flexible but still hierarchical. The idea was to enable someone to get each and every notification that happens on cricket or to choose a particular team / player etc and get more granular notification.

After trying to model the hierarchy into a RDBMS & document DB , we quickly realized traditional DB is just a bad fit for such recursive model design and the data retrieval queries will be too expensive.

This is finally the big enough problem when you can sell / justify introducing a completely new tool to address the issue. And my chance to use graph-db in production. I was introduced to the graph db concept to a previous project of mine and was really impressed by the ease of natural domain model mapping into the DB via the graph concept.

After some basic R & D, we decided to go with Orientdb. Having used elasticsearch cluster products , Orientdb clustering model seemed more natural compared to neo4j. Having said that, having a graph on a distributed cluster is not a easy problem which we realized later the hard way.

We could easily map our model in a graph schema like the following:


Notice how easily the concept of an unregistered device / user , and a user with multiple device fits into the domain. A user can subscribe to any node on the Tree that is Topic and the events automatically flows through the Subscriber.

This is how it looks like on Database:



With the default setup & no special tuning, the system is looking up 10,000 device-id to push the notification in < 500 ms , which is quite good for us. however, the orientdb-support tells us that it can be made even faster.

Lessons Learned:

>     We went with latest major production release 2.1 (which was released only 1 week ago) , and got burnt by many bugs / corner cases. It took upto 2.1.7 to finally have it stable.

>     Use TinkerPop (think jdbc for database) apis so you can switch between orientdb / neo4j

> If coding in java, use Frames api to maintain the Domain mapping

> Frames api is not properly tuned. I had to re-write most of my queries in gremlin . I am not sure about neo4j but orientdb recommends using gremlin / sql to run the search queries.

> Partition your clusters appropriately, it will result in proper utilization of all nodes in your cluster.

SumoShell – Become a shell ninja

Have you ever seen sys-admins doing awesome grep / awk queries and been amazed by it? Now you can do so too and in a much easier way!

A few months ago, I was writing some parser in sumologic and thinking this is so powerful, I wish I could run it on a regular text file. Well, I got a notification from google about sumoshell today.

And wait for it………they open sourced their parsing engine.

What you can do with it? Wonders!


Here are some examples from their blog:

sudo tcpdump 2>/dev/null | sumo search | sumo parse "IP * > *:" as src, dest | sumo parse "length *" as length | sumo sum length by dest | render




tail -f logfile | sumo search "ERROR" | sumo parse "thread=*]" | sumo count thread | render-basic



NeoPhobia & Docker

Back in 2004 when I was starting my career, I read something about ‘NeoPhobia’ in Kathy Sierra’s blog. NeoPhobia is avoiding something new. Details:

And we tend to be NeoPhobic in learning new things  by making excuse about it. Most common excuse?: ‘We dont need to learn about it because its not good enough for us’..We technologist do it all the time. We do it to save ourselves from the trouble of learning about it. Remember how many people didn’t like Git and wanted to stay in SVN?

I try to watch out and try not to be a NeoPhobic in terms of technology, but every now and then it creeps in! I remember installing Docker back in 2013 and playing around with it. To be fair , docker wasn’t as smooth back then and the experience was not good enough. But still , I discarded it as a viable tool.

Fast forward 2 years and I was forced to look into docker again (Because of licensing issue, we had to run multiple microservice mule container in same server but still wanted isolation)…And my opinion? Its a crime not to know / use docker , and I have been committing it so far!!!

Even if you don’t run multiple containers in production , don’t have  needs for virtualisation, you MUST learn docker. Within my first week of learning docker, I am already using it for so many things. Don’t know a thing about NPM and still want to run a Node server ? Want to test 5 node hazlecast cluster in your mini laptop but dont have enough processing power? Limited by Mule licensing and deploying all your microservice in one container with no isolation?  Look no more..Docker is your friend!

Taking ThreadDump from Java Code

Copied from HERE with some code modification.

import java.util.Scanner;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.LongAdder;

public class StackTraceUtils {
	private static final char PID_SEPERATOR = '@';
	private static String pathToJStack = "jstack";

	private static String acquirePid()
		String mxName = ManagementFactory.getRuntimeMXBean().getName();

		int index = mxName.indexOf(PID_SEPERATOR);

		String result;

		if (index != -1) {
			result = mxName.substring(0, index);
		} else {
			throw new IllegalStateException("Could not acquire pid using " + mxName);

		return result;

	public static String executeJstack()
		ProcessInterface pi = new ProcessInterface();

		int exitCode;

		ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();

		try {
			exitCode = String[] { pathToJStack, "-l", acquirePid(),}, byteArrayOutputStream);
		} catch (Exception e) {
			throw new IllegalStateException("Error invoking jstack", e);

		if (exitCode != 0) {
			throw new IllegalStateException("Bad jstack exit code " + exitCode);

		try {
			String str = new String(byteArrayOutputStream.toByteArray(), "UTF-8");
			return str;
		} catch (UnsupportedEncodingException e) {
			throw new RuntimeException(e);


	public static void main(String[] args) {

		final String s = executeJstack();
		System.out.println("s = " + s);

import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;

public class ProcessInterface
	private final Map<String, String> addModifyEnv;
	private final Set<String> removeEnv;
	static class StreamGobbler extends Thread
		private static int gobblerNumber = 0;
		private final InputStream is;
		private final OutputStream os;
		private static synchronized int getNextGobblerNumber()
			return gobblerNumber++;
		public StreamGobbler(InputStream is, OutputStream os)
			super("StreamGobblerThread-" + getNextGobblerNumber()); = is;
			this.os = os;
		public void run()
				OutputStreamWriter osw = new OutputStreamWriter(os);
				BufferedWriter bw = new BufferedWriter(osw);
				InputStreamReader isr = new InputStreamReader(is);
				BufferedReader br = new BufferedReader(isr);
				String line = null;
				while ((line = br.readLine()) != null)
			catch (IOException ioe)
				throw new IllegalStateException(ioe);
	public ProcessInterface()
		this.addModifyEnv = new HashMap<String, String>();
		this.removeEnv = new HashSet<String>();
	public void removeEnv(String name)
		if (addModifyEnv.containsKey(name))
	public void addEnv(String name, String value)
		if (removeEnv.contains(name))
		addModifyEnv.put(name, value);
	public int run(String[] command) throws InterruptedException, IOException
		return run(command, null, null, null);
	public int run(String[] command, File workingDir) throws InterruptedException, IOException
		return run(command, null, null, workingDir);
	public int run(String[] command, OutputStream os) throws InterruptedException, IOException
		return run(command, os, os, null);
	public int run(String[] command, OutputStream os, File workingDir) throws InterruptedException, IOException
		return run(command, os, os, workingDir);
	public int run(String[] command, OutputStream std, OutputStream err, File workingDir)
			throws InterruptedException, IOException
		StringBuilder sb = new StringBuilder();
		for (String s : command)
			sb.append(" ");
		ProcessBuilder builder = new ProcessBuilder();
		if (workingDir != null)
		Map<String, String> env = builder.environment();
		for (String name : removeEnv)
		for (Entry<String, String> entry : addModifyEnv.entrySet())
			env.put(entry.getKey(), entry.getValue());
		Process process = builder.start();
		OutputStream outStream = ((std == null) ? System.out : std);
		OutputStream errStream = ((err == null) ? System.err : err);
		StreamGobbler outputGobbler = new StreamGobbler(process.getInputStream(), outStream);
		StreamGobbler errorGobbler = new StreamGobbler(process.getErrorStream(), errStream);
		int exitVal = process.waitFor();
		return exitVal;

This can be very helpful, for example in a code block you suspect there is a deadlock, for Example timeout from connection pool , you can log the stacktrace so you can come back later and run a ThreadDump analysis tool on it. Add it to your code before the Operations guy restart your problematic server and report an issue in your name !

Finding the Bad Apple in JVM

If you run into situation where you have to troubleshoot a JVM that consumes high CPU and you have absolutely no idea whats causing it, here are some very useful tricks that saved my life countless time.

Most of us use Top often but there are some handy params that can be even more helpful.

First Do a top to get the Cpu Hungry process id.

top - 14:33:22 up 42 days, 7:11, 2 users, load average: 1.60, 1.62, 1.56
Tasks: 154 total, 1 running, 153 sleeping, 0 stopped, 0 zombie
Cpu(s): 77.8%us, 0.8%sy, 0.0%ni, 21.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8061328k total, 7545548k used, 515780k free, 134776k buffers
Swap: 2097148k total, 162800k used, 1934348k free, 798096k cached</code>

12058 root 20 0 13.3g 4.7g 21m S 302.0 61.5 41219:31 java
18205 root 20 0 4273m 948m 6244 S 11.3 12.0 1704:03 java
12056 root 20 0 22032 1080 920 S 0.7 0.0 62:33.48 wrapper-linux-x
1 root 20 0 19356 1244 1052 S 0.0 0.0 0:03.22 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd

Ok, clearly, the process with PID 12058 isn’t behaving properly.

Lets ask ‘top’ to list the child-process (one to one binding with each thread in JVM) for PID 12058

-H Flags turns on the Threads Toggle
-H : Threads toggle
Starts top with the last remembered ’H’ state reversed. When this toggle is On, all individual threads will be displayed. Other-
wise, top displays a summation of all threads in a process.

## top -p 12058 -H (-p only lists child process for 12058 )


[mule@FSASYDESBPROD02 ~]# top -p $PID -H
top - 15:09:49 up 42 days, 7:48, 2 users, load average: 1.34, 1.50, 1.54
Tasks: 5359 total, 3 running, 5356 sleeping, 0 stopped, 0 zombie
Cpu(s): 77.8%us, 1.2%sy, 0.0%ni, 20.9%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 8061328k total, 7740576k used, 320752k free, 135436k buffers
Swap: 2097148k total, 162652k used, 1934496k free, 833864k cached

28255 root 20 0 13.3g 4.9g 21m R 98.3 63.3 10163:33 java
14064 root 20 0 13.3g 4.9g 21m R 97.7 63.3 14089:00 java
14062 root 20 0 13.3g 4.9g 21m R 97.4 63.3 14073:55 java
925 root 20 0 13.3g 4.9g 21m S 1.9 63.3 279:22.62 java
12060 root 20 0 13.3g 4.9g 21m S 0.6 63.3 108:39.51 java

Ok, so Thread $28255 & $14064 is doing a lot of stuff. How do we know what they are doing??

jstack in rescue !

"AsyncLoggerConfig-1" daemon prio=10 tid=0x00007f45ec2ab000 nid=0x2f27 waiting on condition [0x00007f45e1e22000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
at com.lmax.disruptor.BlockingWaitStrategy.waitFor(
at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(
at java.util.concurrent.ThreadPoolExecutor.runWorker(
at java.util.concurrent.ThreadPoolExecutor$

In the jstack trace, the nid=0x2f27 is the hex of the pid we received before. So lets awk it!

bash# PID=12058
bash# NID=28255
bash# jstack $PID | awk '/ nid='"$(printf '%#x' $NID)"' /,/^$/'</code>

"pool-1140-thread-1" prio=10 tid=0x00007f45706b4000 nid=0x6e5f runnable [0x00007f453247a000]
java.lang.Thread.State: RUNNABLE
at sun.reflect.Reflection.getCallerClass0(Native Method)
at sun.reflect.Reflection.getCallerClass(
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
at java.lang.reflect.Method.invoke(
at org.apache.logging.log4j.util.ReflectionUtil.getCallerClass(
at org.apache.logging.log4j.util.ReflectionUtil.getCallerClass(
at org.apache.logging.log4j.jul.AbstractLoggerAdapter.getContext(
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(
at org.apache.logging.log4j.jul.LogManager.getLogger(
at java.util.logging.LogManager.demandLogger(
at java.util.logging.Logger.demandLogger(
at java.util.logging.Logger.getLogger(
at com.orientechnologies.common.log.OLogManager.log(
at com.orientechnologies.common.log.OLogManager.debug(
at com.orientechnologies.orient.client.remote.OStorageRemote.useNewServerURL(
- locked (a java.util.ArrayList)
at com.orientechnologies.orient.client.remote.OStorageRemote.getAvailableNetwork(
at com.orientechnologies.orient.client.remote.OStorageRemote.openRemoteDatabase(
at com.orientechnologies.orient.core.db.OPartitionedDatabasePool$DatabaseDocumentTxPolled.internalOpen(
at com.orientechnologies.orient.core.db.OPartitionedDatabasePool.openDatabase(
at com.orientechnologies.orient.core.db.OPartitionedDatabasePool.acquire(
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.(
at com.tinkerpop.blueprints.impls.orient.OrientGraphNoTx.(
at com.tinkerpop.blueprints.impls.orient.OrientGraphFactory.getNoTx(
at java.util.concurrent.ThreadPoolExecutor.runWorker(
at java.util.concurrent.ThreadPoolExecutor$

There you go! You can see the 3rd party system eating up the CPU ! In my case its Orientdb. Yes, we faced a lot of issue trying to implement a graph model in orientdb, but thats a story for another day.