Category Theory for programmers

Been trying to get my head around all the hype about Category Theory and Monads and found these really good for programmers:

http://bartoszmilewski.com/2014/10/28/category-theory-for-programmers-the-preface/

I am a bit too late in the game but loving it!

Advertisements

Why twitter needs to be so fast???!!!

Ok, first I must confess, I am not a active twitter user, but I am a regular follower about their System Architecture which always fascinates me. Also they have done some great contribution to the OSS world with products like :

Bootstrap , CloudHopper , Twitter-Server etc.

https://github.com/twitter

Its amazing to see what started as a ruby on rails application turns into this Mega-Structure and does things at a scale that I wouldn’t even believe is possible.
I was watching this excellent presentation in infoQ:

http://www.infoq.com/presentations/real-time-twitter

And I’v been wondering, why Twitter needs to be so fast! I have always had a interest in HighFrequency-LowLatency applications and I understand their importance in Telecom / HFT banking market, but does twitter really need it? Working in broadcast industry, I realised when we say a LIVE Game, its actually a 7 sec delayed FEED, and I don’t hear anyone complaining about it. So would it make a big difference if I got my tweets updates 30 sec later rather than sub-second? I can only imagine how much extra energy / development / money had to be spent to gain sub-second or millisecond level latency.

So the question is, was it actually a business requirement or they did it just because they can ? Or the sub-second latency came as a bonus of their superbly scalable design?

My Journey with Mule

Stage 1 : Drag & Drop, Enterprise Integration Patterns , Oh So Cool!!

Stage 2 : What the **** ¬†$$$!!!??? IDE Sucks, weird behavior , silent xml config alteration ….How can anyone work on this!!! Going back to my core java apps…

Stage 3 (Used to all the quirks of Mule Studio) : Hmmm…I see the point..not so bad after all !!! Just built an entire state machine system app with DB integration / HTTP & TCP Endpoints in 90 mins!

Stage 4: Wait, they hid all the cool features in EE edition ūüė¶ … I am supposed to debug my app only with log messages!! Thats almost as bad as the time when I used to write EJB code! hmmmm….start looking for alternative? How is that spring integration project coming along?¬†

Adding Functional power in Java programs

I am digging the Functional style power in my everyday regular programming. I was implementing some failover connection setup in a pure Java code and thought about adding the ugly java command pattern as an alternative to closure. At the end, the code turned up not so bad so I thought I would share it.

Problem:

We have two Quentin Server – Server A & Server B (A kind of recording server used in broadcast industry). The QuentinApi abstracts the interaction with the server which maintains some persistent TCP connection. Now the requirement is, we have to keep talking to Server A. However, if the connection is not valid / or it went down for some reason, we need to switch back to Server B.

Here is how the Quentin Api Looks like:

/**
 * 
 * Api to create/update/delete clips on Quantel. Clips are actually created from the channel. The quantel system has a pool of server. 
 * Every server has a fixed number of channel. 
 * 
 * @author smoinuddin
 *
 */
public interface QuentinApi {
	
	public String createClip(String clipName, int server, int channel, DateTime startTime, DateTime endTime) throws QuantelServerLookupFailedException, QuentinApiException;
	
	public void updateClip(String clipName, int server, int channel, DateTime startTime, DateTime endTime) throws QuentinApiException;
	
	public void cancelRecording(String clipName, int server, int channel) throws QuentinApiException;
      ...... // some more similar methods....	
}

Now all of these method should have the feature of failure detection and should revert to Server B whenever there is a problem. I have solved similar problem before with Java using Dynamic Proxy and Spring AOP, but I thought using Functional Closure style is simpler and easier to understand.

public class QuentinApiFailoverProxy implements QuentinApi {
	private static final Logger LOGGER = LoggerFactory.getLogger(QuentinApiFailoverProxy.class);
	
	private QuentinApiImpl quantelServerA;
	private QuentinApiImpl quantelServerB;
	
	private QuentinApiImpl currentConnection;
	
	public QuentinApiFailoverProxy(QuentinApiImpl quantelServerA,
			QuentinApiImpl quantelServerB) {
		super();
		this.quantelServerA = quantelServerA;
		this.quantelServerB = quantelServerB;
		
		this.currentConnection = quantelServerA;
	}
	
	private abstract class QuantelApiCall<T> {
		abstract T execute() throws QuentinApiException;
		
		public  String toString() {
			return null;
		}
	}
	
	private <T> T execute(QuantelApiCall<T> call) throws QuentinApiException{
		if(call.toString() != null) {
			LOGGER.debug("***executing Call: {}", call);
		}
		try {
			if(currentConnection.validate()) {
				return call.execute();
			} else {
				LOGGER.warn("Could not validate current connection: " + currentConnection);
				throw new QuantelCommunicationException("Could not validate current connection: " + currentConnection);
			}
		} catch (QuantelServerLookupFailedException | 
				QuantelCommunicationException | 
				NullPointerException| 
				org.omg.CORBA.COMM_FAILURE |
				org.omg.CORBA.OBJECT_NOT_EXIST e) {
			
			LOGGER.warn("Corba Failed to talk to {}", currentConnection, e);
			resetConnections();
			LOGGER.warn("Trying to talk to {}", currentConnection);
			
			return call.execute();
		} finally {
			if(call.toString() != null) {
				LOGGER.debug("***finished executing call: " + call);
			}
		}
	}
	
	private synchronized void resetConnections() throws QuantelCommunicationException {
		boolean validCurrentConnection = false;
		try {
			currentConnection.reset();
			if(currentConnection.validate()) {
				validCurrentConnection = true;
			}
		} catch (Exception e) {	}
		
		if(!validCurrentConnection) {
			currentConnection = getAlternative();			
			currentConnection.reset();
			if(!currentConnection.validate()) {
				LOGGER.warn("Failed to talk to main connection & alternative connection . Alternative connection just tried = {}" , currentConnection);
				throw new QuantelCommunicationException("Failed to talk to main connection & alternative connection . Alternative connection just tried: " + currentConnection);
			}
		} else {
			LOGGER.warn("re-initiated the current connection to {}");
		}
		
	}

	private QuentinApiImpl getAlternative() {
		if(currentConnection.equals(quantelServerA)) {
			return quantelServerB;
		} else {
			return quantelServerA;
		}
	}    

	@Override
	public String createClip(final String clipName, final int serverId, final int channel,
			final DateTime startTime,final  DateTime endTime)
			throws QuantelServerLookupFailedException, QuentinApiException {
		
		return execute(new QuantelApiCall<String>() {
			@Override
			String execute() throws QuentinApiException {
				return getCurrentConnection().createClip(clipName, serverId, channel, startTime, endTime);
			}

			@Override
			public String toString() {
				return "createClip: " + clipName ;
			}});		
	}

	@Override
	public void updateClip(final String clipName,final  int server,final  int channel,
			final DateTime startTime, final DateTime endTime) throws QuentinApiException {
		execute(new QuantelApiCall<Object>() {
			@Override
			Object execute() throws QuentinApiException {
				getCurrentConnection().updateClip(clipName, server, channel, startTime, endTime);
				return null;
			}

			@Override
			public String toString() {
				return "Update Clip: " + clipName;
			}});	
	}

	private QuentinApiImpl synchronized  getCurrentConnection() {
		return currentConnection;
	}

.................. and so on.
}

Notice how the execute method wraps each method call and adds exception handling, reset connection, try alternative connection feature. Introducing the QuantelApiCall lets us bundle our code and pass it as an argument to another method (Code as object). Again the execute() call does not take any parameter, instead it uses static binding functionality with the local variable to carry them over to the new context.

One thing you need to be cautious is to use the getCurrentConnection() in the closures to enable java dynamic binding. If I used the currentConnection instead, the currentConnection at the time of ‘Closing’ (aka Creating the closure object) would be used everytime you call execute(). But methods are always dynamically bounded and will be reevaluated with every call of the execute().

Again, the code can be called by multiple thread at the same time so you need synchronization in the reset() call (You don’t want two threads trying to reset the connection at the same time). Other interesting thing is you need ‘synchronized’ on the getCurrentConnection to avoid Thread data caching issue. I have noticed a lot of java developer mistakenly think synchronized is only used for mutual exclusion but it is also necessary for visibility of the data.

Will scala become obsolete with java 8?

I have been investing a lot of my free time in Scala lately. Although I have been injecting groovy heavily in all my projects whenever I get a chance, I miss the safety of Type system (too much java does that to you).

Even though I had some previous Erlang experience and background knowledge of Actor Model, I must admit Scala has a steeper learning curve than I thought.It requires significant time and energy invested before you can do anything useful with it. I can imagine I’d have a hard time if I wanted to push it to a team of java devs. On the other hand, Groovy is too easy to adopt, you write your java code in a script and in no time fall in love with the power of functional language with Java goodies in your hand.

That brings the question in my mind, will scala be obsolete with java 8? A lot of scala goodies will face serious competition with java 8. Developers will eventually adopt those feature sooner or later if they want to keep their job but I don’t see mass adoption of scala. Don’t get me wrong, I am loving scala and no doubt its an elegant language but I don’t see it as a successful complement language of Java with java 8 in the picture. On the other hand, I can imagine myself doing all my scripting / hacking jobs in Groovy as a replacement of my broken perl skills.

Some taste of functional programming with groovy – 2

Input file:

Ver Duration	House Number	Ver Description
00:15.00	BP0211T0	FS: NRL: BP: NRL Monday Night With Matty Johns TON 2100 (18/3)
00:30.00	CP32336W	CI-April Highlights
01:00.00	CP32119W	LIF-Property Month 60
00:30.00	CP32134W	EV-RioVAlv SunMar31
00:15.00	CP32139W	OD-Argo Sting15
.......... line break.......................................
01:00.00	CP32144W	OD-Argo 60
00:30.00	CP32145W	OD-Argo
01:00.00	CP32149W	OD-Taken 2 60
00:30.00	CP32151W	OD-Taken 2 30B
01:00.00	CP32153W	OD-End Of Watch 60
00:30.00	CP32154W	OD-End Of Watch
00:30.00	CP32159W	OD-Premium Rush VODPrem
00:15.00	CP32224W	OD-Taken 2 STING 15
00:15.00	CP32225W	OD-End Of Watch STING 15
......................erp----------------------------------------
01:00.00	CP32257W	COM-Good God 27/3 60
01:00.00	CP32263W	SHC-GameThrones 1/4 60
00:30.00	CP32269W	A&E-March Highlights
00:15.00	CP32270W	A&E-Toy Hunters 1/4 15
00:45.00	CP32273W	BI-Gene Simmons 27/3
02:00.00	CP32275W	FX-DaVinciDemon 16/4 2min
01:00.00	CP32276W	FX-DaVinciDemon 16/4 60
00:30.00	CP32277W	FX-DaVincis Demons 16/4
^^^^^^^^^^XXXXXXXXXXX^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
00:30.00	CP32278W	NGC-DoomsdayPreppr 8/4
00:30.00	CP32280W	NGC-ToyotaMegafactr8/4
00:30.00	CP32284W	CSA-Aus Podiatry Assoc
00:30.00	CP32285W	CSA-Aus Medicare LocalAll
00:30.00	CP32286W	CSA-RedCross QLD FloodAp
00:45.00	CP32288W	SOHO-Wentworth 1/5 45
00:15.00	CP32324W	OD-The Master
00:30.00	CP32333W	OD-Nitro Circus
01:00.00	CP32335W	CI-April Highlights 60
00:30.00	CP32337W	A&E-TOy Hunters 2/4
02:00.00	CP32341W	FX0-DaVincis Demons 16/4 2m
01:00.00	CP32343W	FX-DaVincis Demons 16/4 60
00:30.00	CP32349W	NGC-WarHerosSkies 10/4
00:30.00	CP32358W	EV-DonVRig Sunday 14/4
00:30.00	JA0086G2	"^FS: FTB: FSD: A League ""iPad  App"" GEN 2012/13 30"" "
01:00.00	JA0106G1	FTV: GEN: Fuel TV is You 2012-13 V2
00:30.00	JA0249D0	SPD: DRA: Andra Drag Racing WED 2130 EDT 2013
00:15.00	JA0251D1	SPD: SPR: World Series Sprintcars WED 1930 (16/1)
00:30.00	JA0447G0	FTV: SUR: ASP Surfing World Tour (2013)
00:30.00	JA0474G1	^FS: RUN: GEN: Rugby Wrap 2013
00:15.00	JA0528G1	"^FS: FSD: NRL: Supercoach ""Prize"" Generic 2013 15"" "
00:15.00	JA0530G1	"^FS: FSD: GEN: Tipping Generic 2013 15"""
01:00.00	JA0574G1	"FS: AFL Who Ever You Love Image 2013 60"" V2"
00:30.00	JA0575G1	"FS: AFL Who Ever You Love Image 2013 30"" V2"
00:15.00	JA0576G1	"FS: AFL Who Ever You Love Image V1 2013 15"" V2"
00:15.00	JA0577G1	"FS: AFL Who Ever You Love Image V2 2013 15"" V2"
01:00.00	JA0582G1	"FS: AFL: Launch Image Games 60"" V2"
00:15.00	JA0583G0	FTV: SUR: ASP Surfing World Tour (2013) 15'  
00:30.00	JA0586G1	"FS: AFL: Launch Image Games 30"" V2"
01:00.00	JA0587G1	"FS: AFL: Image 2013 ""Great Moments"" 60"" V2"
00:30.00	JA0588G1	"FS: AFL: Image 2013 ""Great Moments"" 30"" V2"
00:15.00	JA0619G1	"FS: AFL Who Ever You Love Image V3 2013 15"" V2"
00:15.00	JA0620G1	"FS: AFL Who Ever You Love Image V4 2013 15"" V2"
00:15.00	JA0621G2	"FS: AFL Who Ever You Love Image V5 2013 15"" V3"
00:30.00	JA0622T0	"FS: AFL: On The Couch Launch TON 2030 (18/03) 30"""
00:30.00	JA0627G3	"^FS: FSD: NRL: Supercoach ""Ennis"" 2013 30"" "
01:00.00	JA0648G0	FTV: GEN: March Highlights GEN (2013) 60' 
00:30.00	JA0652T0	"FS: AFL: Open Mike ""Launch"" MON 2130 (18/03) 30"""
01:00.00	JA0681G0	"FS: AFL: Fox Footy Image ""TALENT"" 2013 60"""
00:30.00	JA0682G0	"FS: AFL: Fox Footy Image ""TALENT"" 2013 30"""
00:30.00	JA0699G1	"FS: AFL Who Ever You Love V2 Image 2013 30"" V2"
00:30.00	JA0702T0	"SPD: NAS: Nascar Sprint Cup ""Bristol"" Rply TON 1930 (18/3)"
00:30.00	JA0704D0	"FS: AFL: 2012 GF Recall Syd v Haw SUN 2030 (24/03) 30"""
00:15.00	JA0705D0	"FS: AFL: 2012 GF Recall Syd v Haw ""GOODES"" SUN 2030 (24/03) 15"" "
00:15.00	JA0706D0	"FS: AFL: 2012 GF Recall Syd v Haw ""MALCESKI"" SUN 2030 (24/03) 15"""
00:15.00	JA0707G1	"FS: AFL Who Ever You Love Image V6 ROO 2013 15"" V2"
00:24.00	JA0710D0	^SPD: MAG: Mobil 1 The Grid WED 1900 (20/3)
00:30.00	JA0750G2	FS: CRK: GEN: Cricket in 2013
00:05.00	JA0751G0	"^FS: NRL: NRL Mag Show 5 seconders GEN ""Matty Johns"" 2013"
00:05.00	JA0753G0	"^FS: NRL: NRL Mag Show 5 seconders GEN ""NRL 360 2013"
00:05.00	JA0754G0	"^FS: NRL: NRL Mag Show 5 seconders GEN ""Sterlo"" 2013"
00:05.00	JA0755D0	^FS: GOL: The Golf Show TUE 1930 5 seconders 2013
00:05.00	JA0769G0	"^FS: GEN: Digital Breath  ""HAL"" 2013 "
00:05.00	JA0770G0	"^FS: GEN: Digital Breath  ""CRK"" 2013"
00:05.00	JA0771G0	"^FS: GEN: Digital Breath  ""EPL"" 2013"
00:05.00	JA0772G0	"^FS: GEN: Digital Breath  ""EPL2"" 2013"
00:05.00	JA0773G0	"^FS: GEN: Digital Breath  ""SOCIAL"" 2013 "
00:05.00	JA0774G0	"^FS: GEN: Digital Breath  ""NRL"" 2013"
00:05.00	JA0775G0	"^FS: GEN: Digital Breath  ""NRL2"" 2013 "
00:05.00	JA0776G0	"^FS: GEN: Digital Breath  ""RUN"" 2013"
00:05.00	JA0777G0	"^FS: GEN: Digital Breath  ""SRUN"" 2013 "
00:05.00	JA0778G0	"^FS: GEN: Digital Breath  ""WEB"" 2013 "
00:05.00	JA0779G0	"^FS: GEN: Digital Breath  ""WEB"" 2013"
00:45.00	JA0780G0	"^FS: GEN: FoxSports Relaunch ""Talent"" 2013"
00:05.00	JA0781G0	"^FS: GEN: Digital Breath  ""Foxtel Go 1"" 2013"
00:05.00	JA0782G0	"^FS: GEN: Digital Breath  ""Foxtel Go2"" 2013 "
01:00.00	JA0786G0	"FS: AFL: Friday Night Ritual 60"""
00:30.00	JA0787G0	"FS: AFL: Friday Night Ritual 30"""
00:30.00	JA0788G0	"FS: AFL: Friday Night Ritual ""Ricciuto"" 30"""
00:15.00	JA0789G0	"FS: AFL: Friday Night Ritual 15"""
00:20.00	JA0790G0	"FS: AFL: Friday Night Ritual ""Healy"" 20"""
00:30.00	JA0820C0	"FS: FTB: Socceroos v Oman CSN (26/3) 30"""
00:10.00	JA0821C0	"FS: FTB: Socceroos v Oman CSN (26/3) 10"""
00:30.00	JA0844G0	FS: RUN: FSD: GEN: Rugby Union  Match Centre 2013
00:30.00	JA0850D0	FS: FTB: HAL WK 26 SAT 1700 (23/3)
00:15.00	JA0851D0	FS: FTB: HAL WK 26 SUN 1630 (24/3)
00:30.00	JA0855D0	FS: FSN: AFL: AFL Pre-Season Show WED 1730 (20/3)
00:30.00	XP006929	FX Ultimate Fighter S17 - Wed 1630
00:30.00	XP006930	LSF-AskTheButcher26/3REV

Requirements:

  • Group the lines by 3 category from Description. Description starts with FS / starts with ^FS / starts with ^OD
  • Next group by time for each description group
  • Output with Line number
  • My beginner Groovy code , I am sure the experts will write it even better, comment for improvement will be very much appreciated!

    def out = new File('c:/dev/test/output.txt')
    out.delete()
    new File('c:/dev/test/input.txt').readLines()
                            .findAll { it ==~ /^\d{2}:.+/ } //only select lines that start with number:
                            .collect { it.split ( /\t/ )} //convert string into array of token
                            .collect { lineItems ->           //add an extra column to determine grouping criteria                                 
                                lineItems + [{
                                        switch(lineItems[2]) {
                                            case ~/"FS:.*/ : 
                                                "FS:"
                                                break
                                            case ~/^"\^FS:.*/ : 
                                                "^FS"
                                                break
                                            case ~/^OD-.*/ :
                                                "OD-"
                                                break
                                            default: 
                                                "IGNORE" 
                                        }
                                    }()]
                            }
                            .findAll {!it[3].startsWith("IGNORE")} //drop the rows which doesn't meet our criteria
                            .groupBy { it[3]}  //group by the grouping category (output -> map[string : list[array]]
                            .collect { 
                                       it.value.groupBy{it[0]}  //for each group of Sports Code, perform another grouping on value list based on the duration and sort by keys
                                      .values() //get the values of the groupped by time map
                                      .collectMany{it} //flattens all the result into a single list preserving the order
                            }
                           .collectMany { it } //flatten to a single list of array
                           .eachWithIndex { item , count ->  out << count + '\t' + item[0..2].join('\t') + '\n\r'}
                          
                          
                           println out.text
    

    I think for people familiar with functional coding concepts like Map/Fold/Reduce, the comments are unnecessary and self explanatory. Again the idea is to transform the input file into the output by applying the functions on them. List / Map are the two very important data structure in functional language and you can do magic with them ! And of course all these wouldn’t be possible without the function block/closures. I can’t wait for Java 8 support for closures.

    Anyway, here is the output:

    0	00:15.00	CP32139W	OD-Argo Sting15
    
    1	00:15.00	CP32224W	OD-Taken 2 STING 15
    
    2	00:15.00	CP32225W	OD-End Of Watch STING 15
    
    3	00:15.00	CP32324W	OD-The Master
    
    4	01:00.00	CP32144W	OD-Argo 60
    
    5	01:00.00	CP32149W	OD-Taken 2 60
    
    6	01:00.00	CP32153W	OD-End Of Watch 60
    
    7	00:30.00	CP32145W	OD-Argo
    
    8	00:30.00	CP32151W	OD-Taken 2 30B
    
    9	00:30.00	CP32154W	OD-End Of Watch
    
    10	00:30.00	CP32159W	OD-Premium Rush VODPrem
    
    11	00:30.00	CP32333W	OD-Nitro Circus
    
    12	00:30.00	JA0086G2	"^FS: FTB: FSD: A League ""iPad  App"" GEN 2012/13 30"" "
    
    13	00:30.00	JA0627G3	"^FS: FSD: NRL: Supercoach ""Ennis"" 2013 30"" "
    
    14	00:15.00	JA0528G1	"^FS: FSD: NRL: Supercoach ""Prize"" Generic 2013 15"" "
    
    15	00:15.00	JA0530G1	"^FS: FSD: GEN: Tipping Generic 2013 15"""
    
    16	00:05.00	JA0751G0	"^FS: NRL: NRL Mag Show 5 seconders GEN ""Matty Johns"" 2013"
    
    17	00:05.00	JA0753G0	"^FS: NRL: NRL Mag Show 5 seconders GEN ""NRL 360 2013"
    
    18	00:05.00	JA0754G0	"^FS: NRL: NRL Mag Show 5 seconders GEN ""Sterlo"" 2013"
    
    19	00:05.00	JA0769G0	"^FS: GEN: Digital Breath  ""HAL"" 2013 "
    
    20	00:05.00	JA0770G0	"^FS: GEN: Digital Breath  ""CRK"" 2013"
    
    21	00:05.00	JA0771G0	"^FS: GEN: Digital Breath  ""EPL"" 2013"
    
    22	00:05.00	JA0772G0	"^FS: GEN: Digital Breath  ""EPL2"" 2013"
    
    23	00:05.00	JA0773G0	"^FS: GEN: Digital Breath  ""SOCIAL"" 2013 "
    
    24	00:05.00	JA0774G0	"^FS: GEN: Digital Breath  ""NRL"" 2013"
    
    25	00:05.00	JA0775G0	"^FS: GEN: Digital Breath  ""NRL2"" 2013 "
    
    26	00:05.00	JA0776G0	"^FS: GEN: Digital Breath  ""RUN"" 2013"
    
    27	00:05.00	JA0777G0	"^FS: GEN: Digital Breath  ""SRUN"" 2013 "
    
    28	00:05.00	JA0778G0	"^FS: GEN: Digital Breath  ""WEB"" 2013 "
    
    29	00:05.00	JA0779G0	"^FS: GEN: Digital Breath  ""WEB"" 2013"
    
    30	00:05.00	JA0781G0	"^FS: GEN: Digital Breath  ""Foxtel Go 1"" 2013"
    
    31	00:05.00	JA0782G0	"^FS: GEN: Digital Breath  ""Foxtel Go2"" 2013 "
    
    32	00:45.00	JA0780G0	"^FS: GEN: FoxSports Relaunch ""Talent"" 2013"
    
    33	01:00.00	JA0574G1	"FS: AFL Who Ever You Love Image 2013 60"" V2"
    
    34	01:00.00	JA0582G1	"FS: AFL: Launch Image Games 60"" V2"
    
    35	01:00.00	JA0587G1	"FS: AFL: Image 2013 ""Great Moments"" 60"" V2"
    
    36	01:00.00	JA0681G0	"FS: AFL: Fox Footy Image ""TALENT"" 2013 60"""
    
    37	01:00.00	JA0786G0	"FS: AFL: Friday Night Ritual 60"""
    
    38	00:30.00	JA0575G1	"FS: AFL Who Ever You Love Image 2013 30"" V2"
    
    39	00:30.00	JA0586G1	"FS: AFL: Launch Image Games 30"" V2"
    
    40	00:30.00	JA0588G1	"FS: AFL: Image 2013 ""Great Moments"" 30"" V2"
    
    41	00:30.00	JA0622T0	"FS: AFL: On The Couch Launch TON 2030 (18/03) 30"""
    
    42	00:30.00	JA0652T0	"FS: AFL: Open Mike ""Launch"" MON 2130 (18/03) 30"""
    
    43	00:30.00	JA0682G0	"FS: AFL: Fox Footy Image ""TALENT"" 2013 30"""
    
    44	00:30.00	JA0699G1	"FS: AFL Who Ever You Love V2 Image 2013 30"" V2"
    
    45	00:30.00	JA0704D0	"FS: AFL: 2012 GF Recall Syd v Haw SUN 2030 (24/03) 30"""
    
    46	00:30.00	JA0787G0	"FS: AFL: Friday Night Ritual 30"""
    
    47	00:30.00	JA0788G0	"FS: AFL: Friday Night Ritual ""Ricciuto"" 30"""
    
    48	00:30.00	JA0820C0	"FS: FTB: Socceroos v Oman CSN (26/3) 30"""
    
    49	00:15.00	JA0576G1	"FS: AFL Who Ever You Love Image V1 2013 15"" V2"
    
    50	00:15.00	JA0577G1	"FS: AFL Who Ever You Love Image V2 2013 15"" V2"
    
    51	00:15.00	JA0619G1	"FS: AFL Who Ever You Love Image V3 2013 15"" V2"
    
    52	00:15.00	JA0620G1	"FS: AFL Who Ever You Love Image V4 2013 15"" V2"
    
    53	00:15.00	JA0621G2	"FS: AFL Who Ever You Love Image V5 2013 15"" V3"
    
    54	00:15.00	JA0705D0	"FS: AFL: 2012 GF Recall Syd v Haw ""GOODES"" SUN 2030 (24/03) 15"" "
    
    55	00:15.00	JA0706D0	"FS: AFL: 2012 GF Recall Syd v Haw ""MALCESKI"" SUN 2030 (24/03) 15"""
    
    56	00:15.00	JA0707G1	"FS: AFL Who Ever You Love Image V6 ROO 2013 15"" V2"
    
    57	00:15.00	JA0789G0	"FS: AFL: Friday Night Ritual 15"""
    
    58	00:20.00	JA0790G0	"FS: AFL: Friday Night Ritual ""Healy"" 20"""
    
    59	00:10.00	JA0821C0	"FS: FTB: Socceroos v Oman CSN (26/3) 10"""
    

    Groovy is the new Perl for me, I am not writing any linux scripts with Perl ever again.

    Some taste of functional programming with groovy

    In my day job, we have to do a lot of system integration, a lot of it consists consuming external feeds , process it and feed it to some other system. We use Mule a lot for this. I was writing a groovy script in mule to parse a xml feed and extract some data, here is how my functional programming investment helped me do it in an elegant way. The XML to parse

    <sport reportName="aflRoundMatches" jobId="72451003">
    
      <aflRoundMatches roundNumber="3">
        <match matchId="10370301" matchStatus="complete" matchName="Dockers vs Bombers" matchNumber="1" isPreseason="false" isFinal="false" homeSquadId="60" homeSquadName="Fremantle" awaySquadId="50" awaySquadName="Essendon" venueId="70" venueName="Patersons Stadium" venueTimezone="Australia/Perth" utcStartTime="2013-04-12T10:40:00+00:00" localStartTime="2013-04-12T18:40:00+08:00" period="4" periodCompleted="4" periodSeconds="1877" homeSquadGoals="9" homeSquadBehinds="14" homeSquadScore="68" awaySquadGoals="10" awaySquadBehinds="12" awaySquadScore="72"/>
        <match matchId="10370302" matchStatus="complete" matchName="Kangaroos vs Swans" matchNumber="2" isPreseason="false" isFinal="false" homeSquadId="100" homeSquadName="North Melbourne" awaySquadId="160" awaySquadName="Sydney Swans" venueId="2" venueName="Blundstone Arena" venueTimezone="Australia/Hobart" utcStartTime="2013-04-13T03:45:00+00:00" localStartTime="2013-04-13T13:45:00+10:00" period="4" periodCompleted="4" periodSeconds="1873" homeSquadGoals="13" homeSquadBehinds="14" homeSquadScore="92" awaySquadGoals="20" awaySquadBehinds="11" awaySquadScore="131"/>
        <match matchId="10370303" matchStatus="complete" matchName="Demons vs Eagles" matchNumber="3" isPreseason="false" isFinal="false" homeSquadId="90" homeSquadName="Melbourne" awaySquadId="150" awaySquadName="West Coast Eagles" venueId="40" venueName="MCG" venueTimezone="Australia/Melbourne" utcStartTime="2013-04-13T04:10:00+00:00" localStartTime="2013-04-13T14:10:00+10:00" period="4" periodCompleted="4" periodSeconds="1878" homeSquadGoals="13" homeSquadBehinds="5" homeSquadScore="83" awaySquadGoals="27" awaySquadBehinds="15" awaySquadScore="177"/>
        <match matchId="10370304" matchStatus="complete" matchName="Giants vs Saints" matchNumber="4" isPreseason="false" isFinal="false" homeSquadId="1010" homeSquadName="GWS Giants" awaySquadId="130" awaySquadName="St Kilda" venueId="150" venueName="Manuka Oval" venueTimezone="Australia/Canberra" utcStartTime="2013-04-13T06:40:00+00:00" localStartTime="2013-04-13T16:40:00+10:00" period="4" periodCompleted="4" periodSeconds="1949" homeSquadGoals="10" homeSquadBehinds="8" homeSquadScore="68" awaySquadGoals="21" awaySquadBehinds="14" awaySquadScore="140"/>
        <match matchId="10370305" matchStatus="playing" matchName="Cats vs Blues" matchNumber="5" isPreseason="false" isFinal="false" homeSquadId="70" homeSquadName="Geelong Cats" awaySquadId="30" awaySquadName="Carlton" venueId="190" venueName="Etihad Stadium" venueTimezone="Australia/Melbourne" utcStartTime="2013-04-13T09:40:00+00:00" localStartTime="2013-04-13T19:40:00+10:00" period="3" periodCompleted="2" periodSeconds="421" homeSquadGoals="13" homeSquadBehinds="6" homeSquadScore="84" awaySquadGoals="10" awaySquadBehinds="8" awaySquadScore="68"/>
        <match matchId="10370306" matchStatus="playing" matchName="Suns vs Lions" matchNumber="6" isPreseason="false" isFinal="false" homeSquadId="1000" homeSquadName="Gold Coast Suns" awaySquadId="20" awaySquadName="Brisbane Lions" venueId="81" venueName="Metricon Stadium" venueTimezone="Australia/Brisbane" utcStartTime="2013-04-13T09:40:00+00:00" localStartTime="2013-04-13T19:40:00+10:00" period="3" periodCompleted="2" periodSeconds="773" homeSquadGoals="7" homeSquadBehinds="8" homeSquadScore="50" awaySquadGoals="7" awaySquadBehinds="14" awaySquadScore="56"/>
        <match matchId="10370307" matchStatus="scheduled" matchName="Tigers vs Bulldogs" matchNumber="7" isPreseason="false" isFinal="false" homeSquadId="120" homeSquadName="Richmond" awaySquadId="140" awaySquadName="Western Bulldogs" venueId="190" venueName="Etihad Stadium" venueTimezone="Australia/Melbourne" utcStartTime="2013-04-14T03:10:00+00:00" localStartTime="2013-04-14T13:10:00+10:00" period="0" periodCompleted="0" periodSeconds="0" homeSquadGoals="0" homeSquadBehinds="0" homeSquadScore="0" awaySquadGoals="0" awaySquadBehinds="0" awaySquadScore="0"/>
        <match matchId="10370308" matchStatus="scheduled" matchName="Magpies vs Hawks" matchNumber="8" isPreseason="false" isFinal="false" homeSquadId="40" homeSquadName="Collingwood" awaySquadId="80" awaySquadName="Hawthorn" venueId="40" venueName="MCG" venueTimezone="Australia/Melbourne" utcStartTime="2013-04-14T05:20:00+00:00" localStartTime="2013-04-14T15:20:00+10:00" period="0" periodCompleted="0" periodSeconds="0" homeSquadGoals="0" homeSquadBehinds="0" homeSquadScore="0" awaySquadGoals="0" awaySquadBehinds="0" awaySquadScore="0"/>
        <match matchId="10370309" matchStatus="scheduled" matchName="Power vs Crows" matchNumber="9" isPreseason="false" isFinal="false" homeSquadId="110" homeSquadName="Port Adelaide" awaySquadId="10" awaySquadName="Adelaide Crows" venueId="10" venueName="AAMI Stadium" venueTimezone="Australia/Adelaide" utcStartTime="2013-04-14T06:40:00+00:00" localStartTime="2013-04-14T16:10:00+09:30" period="0" periodCompleted="0" periodSeconds="0" homeSquadGoals="0" homeSquadBehinds="0" homeSquadScore="0" awaySquadGoals="0" awaySquadBehinds="0" awaySquadScore="0"/>
      </aflRoundMatches>
    </sport>
    

    The Groovy script:

    new XmlSlurper().parseText(payload).aflRoundMatches.match
    		.findAll { ['playing','postmatch'].contains(it.@matchStatus.text()) }
    		.collect { match -> 
    			[
    			'matchId': match.@matchId.text() , 
    			'homeSquadId' : match.@homeSquadId.text() , 
                'matchStatus' : match.@matchStatus.text(),
                'awaySquadId' : match.@awaySquadId.text(), 
    		    'HomeTeam/ScoreTotal_C' : match.@homeSquadScore.text(),         
                'AwayTeam/ScoreTotal_C' : match.@awaySquadScore.text() ,            
                'Period_Active' : match.@period.text().toInteger() >= match.@periodCompleted.text().toInteger() ? 0 : 1
                ]
    		}
    

    Here,¬†new XmlSlurper().parseText(payload).aflRoundMatches.match gives me all the <match> elements of the xml, on that I run a filter for only the ‘playing’ & ‘postmatch’ games (incase you are wondering, its for pushing match scores on Live Television for the playing/postmatch games). After that I run a ‘collect’ method to convert the <match> elements into a Map<attribute, value> for that match. At the end of process I get a List<Map<attribute, Value>> Imagine doing this in Java! This is pure transition of data from one state to another through functions. Coming from years of Object Oriented background, I had to really struggle with this concept as we are so used to think in terms of Nouns aka Classes. Here is a really interesting read about it: http://steve-yegge.blogspot.com.au/2006/03/execution-in-kingdom-of-nouns.html I have some more advanced example which I will save for the next blog post.

    Currying with Groovy

    I was writing some functional test case with Cucumber for JVM and opted to use Groovy for my step¬†definitions. While I am not yet comfortable enough to start any big project with an untyped language like groovy,¬†¬†its super cool for writing test cases. Specially when we write functional test cases, we want to write as less code as possible and achieve as much as possible. I think groovy is a perfect candidate for that. I am also writing my shell/perl scripts with Groovy now. Call me old school but I still like my java ūüôā

    Anyway, I was fetching the XML from the server and was asserting various attributes on it. Here is how the power of functional programming made it elegant.

    def xmlFetchedFromWebservice = '''
    <SchedRemote> <!-- Long Form: Room, Service, and Resource -->
     <WorkOrderDetails ConflictLevel="0" SEQNUM="232323">
     <Fields>
     <Field STAT="10" />
     <Field STATUSEX='0'/>
     </Fields>
     </WorkOrderDetails>
     </SchedRemote>
    '''
    
    def xmlFind = { xmlFields, fieldName ->
     def field = xmlFields.find {
     return it['@' + fieldName] != null
     }
     return field?.( '@' + fieldName)
    }
    def selectXmlFields = {xmlText -> new XmlParser().parseText(xmlText).WorkOrderDetails.Fields.Field}
    
    def workOrder = xmlFind.curry(selectXmlFields(xmlFetchedFromWebservice))
    
    assert workOrder('STAT') == 10
    assert workOrder('STATUSEX') == 0
    
    //..... many more assersions like this
    
    

    I refactored some of the implementation details , In a nutshell We have two part task, 1st select the correct section from xml and then perform a search on that section to select the field value.

    So once I get the xml from webservice, I select the right fields with selectXmlFields and curry the result in the xmlFind method.

    Now, the resulting function can be used almost as object like fashion. The core concept of functional programming which took me quite a while to grasp is that functions are the building blocks, not the objects. If I wanted to do the same thing with java, may be I would create a WorkOrder class, parse the XML in the constructor and use workOrder object. In functional language, I am using the function composition from the currying to perform the same thing. In other words, I am using curry here to bind xmlFind to the perticular workorder instance. How cool is that?