Marcin bloguje

 

02

listopada

2011

SoapUI ext libs and its weirdness

Share |

Suppose you want to add some additional jars to your SoapUI installation. It all should work ok if you put them in bin/ext directory. It is scanned at startup, and jars found there are automatically added to classpath.

However if you want to add some JDBC drivers, and happen to be using SoapUI version higher than 3.5.1 it is a bit more tricky.

You may face this NoClassDefFoundError:

An error occured [oracle/jdbc/Driver], see error log for details
java.lang.NoClassDefFoundError: oracle/jdbc/Driver

If so, try registering your drivers with registerJdbcDriver function, like I did in this snippet of code:

What a crappy thing!

 
 

21

września

2011

What is NoSQL good for?

Share |
... or how I ended up writing a CouchDB proof of concept app?

Once upon a time I set out on a journey to discover the NoSQL land. I've decided that doing simple queries wouldn't be interesting enough. That's why I've chose to create an app that would be based on some NoSQL database.

The main idea was to create an app, that would dynamically update itself with geographic data flowing in. Since there are myriads of geo-data that are available on the internet, you can pick your favorite one and load them into your SQL database of choice.

In my case the primary source of data was a proprietary database, or more specifically - one table in it continuously updated with new data. To make that data visible on my map I needed to:

The idea of the front-end HTML page was to show new points on the map. From the moment of opening the page records that appear in database table should be shown interactively on the screen.

Toys used

For the first step I chose to use RabbitMQ broker. A queue on the broker would receive messages - one message per database table's row. Then I'd use some simple groovy middle ware to convert the data to appropriate format and put it onto another db - this time db specific to my app.

You may ask why incorporate another database. It would be good for separating environments - assuming the original data contains some vulnerable content that should be anatomised, or we just don't feel comfortable exposing the whole database of some XYZ-system just to have access to its one table.

Since for my presentation layer I chose HTML+JS without any application server-based back-end I've decided on CouchDB . This seemed like a perfect match for this scenario. Why? - ease of use, REST API, with JSON responses - just great for interacting with my simple front-end.

The flow of things was as shown on the image below:

diagram

Avro - for the beginning

As you can see, I've chosen JSON as my data-format. I've been considering Apache Avro in the first place but using it was a real pain in the ass. Avro itself is used in Apache Hadoop as a serialization layer, so it would seem OK, but it has virtually no documentation. But once you tear through the unintuitive interface and manage to handle all those unthinkable exceptions you get a few pros for this library. It's great in that it does not require code generation - I like it being made on the fly. It also offers sending data in binary format, which was not necessary, but never the less is a nice feature.

What I certainly didn't like about it was its orientation on the files rather than chunks of data - so it was not so obvious how should I send data through the wire.

Than I found out it can produce JSON output, which would work for me, except the output could not have been parsed by other JSON libraries :) (I've asked on stackoverflow about that, but with no luck).

If my whining haven't put you back and still would like to see how to use Avro, try this unit test in project's GitHub repo: AvroSimpleTest.groovy

Svenson

I've dropped Avro in favour of a simple JSON lib called (Svenson and that was painless. The only thing I was forced to do was create my model class in Java - the rest of the project is written in Groovy. I've no idea why was that necessary, and didn't want to look into it.

RabbitMQ

Further on the way is RabbitMQ, to which records are filled by a feeding middle-ware written in Groovy. Since I use ActiveMQ on a day-to-day basis, I've decided to try something new. This broker is a really nice piece of software. Being written in Erlang makes it really fast. What's more it has some extensive capabilities and is easy to approach for anyone similar with messaging (JMS and friends). For such a lightweight product it is really powerful - implements AMQP!

CouchDB

From the broker's queue messages are again fetched by a middle-ware just to be put into CouchDB view. This database is also written in Erlang. It's very reliable, however the way it handles refreshing view isn't the most pleasant one - performance-wise.

Word of advice - if you're on Debian derivative, be cautious with apt-repository version. It's rather _ancient_. Also remember to add allow_jsonp = true to you config file /opt/couchbase/etc/couchdb/local.ini. It's not enabled by default, and not having this set would result with empty responses from the CouchDB server.

The problem here is, that the browser doesn't allow quering a web server with hostname other than the one the script originates. More on this case here. Seems like my problem could be overcame by changing url in index.html and hostname couchdb listens on to the same address.

I've also created a view, that would expose an event by key: view code

Presenting the dots

As a back-end I've done some JQuery based AJAX calls - nothing too fancy. All things necessary for presentation layer are in this file.

Things to consider

Please bear in mind that this whole application is rather a playground, not a full-fledged project!! After creating all the parts I have some doubts about some architectural decisions I made. I don't think the security have been taken into account seriously enough. Also scalability was never an issue ;-)

If you have some thoughts about any of the aspects mentioned in this post, please feel free to comment or contact me directly :)

And also you may try the application by yourself - it's on the GitHub.

 
 

22

sierpnia

2011

5 best things to do with your Kindle

Share |

I bought a Kindle (3rd generation, Wi-Fi only) some time ago - like half a year ago. Read some books, done some web-browsing (awful, quite unpleasant). Gradually I became more and more curious of other things possible to achieve with this slate-looking piece of tech. These are my thoughts and ideas.

Got a Kindle? Use it every day? Feel like modding or extending your ways of usage? Great! Read on, and share your thoughts in comments!

  1. Readability! - This web app is great! Generally this is a simple plug-in for your browser that will show a little button somewhere on the toolbar, and if you click it, the page you're reading now will be transformed into nice and sleek content-only page. Look on the screen below:

    This plug-ins additional function is sending to Kindle account. That's the nicest way to read those loads of RSS-sources articles :) The only limitation is that graphics won't be included if resulting file would exceed allowed size of kindle documents - that's 2MB AFAIR.
  2. Install some hacks! - be that serious hacks or rather some simple software modifications:
    • read all book formats with Calibre - link
    • play Zork on a Kindle!!! - link
    • alternative Kindle keyboard - link
    • custom fonts - link
  3. Install custom screen-savers - do this to be able to install your own images. ...because you've always wanted to have some other things on screen when your kindle is in standby mode. Of course, the original screen-savers look great, but there are only few. Installing this hack gave me an opportunity to have a multitude of new images. Now my Kindle looks even better!
  4. Try out Chinese kindle software - doukan.com - As a matter of fact, I haven't installed that software yet. It doesn't look good enough for me, and has some minor problems. However this is great, that there is actually some other option - I'm not forced to use the official firmware. And this distribution has many nice features like PDF reflow.
  5. Enable Chinese fonts support on your kindle - damn! I'd like a simple, step by step tutorial on how to set up chinese fonts on a kindle. I'd like to put some font file on my device, fire some chinese book and be able to see the actual characters..
  6. Programming for Kindle - with Kindle Official SDK - well, not quite! - unfortunately this is reserved only for the Chosen Ones. I've applied for the SDK but they haven't sent me my developer key yet, and it's been ~2 months. This is not "being supportive" or "supporting the community".

And how do You use your Kindle? Perhaps you're doing some serious, crazy things with it? Share your thoughts!

 
 

07

sierpnia

2011

Great many ways to enjoy life!

Share |
This looks like a very nice art+video project. Look at those clips:
  1. EAT from Rick Mereki on Vimeo.


  2. LEARN from Rick Mereki on Vimeo.


  3. MOVE from Rick Mereki on Vimeo.


 
 

14

lipca

2011

OVal - validate your models quickly and effortlessly!

Share |

Some time ago one of the projects at work required me to validate some Java POJOs. Theses were my model classes and I've been creating them from incoming WebService requests. One would say that XSD would be sufficient for the task, for parts of this validations - sure, it would. But there were some advanced rules XSD would not handle, or would render the schema document very complicated.

Rules I needed to express were like:

The library I've decided to use for this task is OVal and it came out really nice! Read on to find out the details!

Oval is quite mature library that allows POJO validation, but is not JSR303 (bean validation) implementation. It has converters that enable it to understand those annotations, but I'm not sure about the compatibility.

I've tried only a subset of the available checks, among which were:

There are many more, and their attributes give interesting ways to configure the validation process. But using them was rather easy and did not require to much brainstorming. What I really needed were custom checks. And in this area OVal shows it's strength. Implementing a check is really easy.

I needed an annotation that would check a field against some values in a dictionary. If field's value was in the given set, than the validation would succeed, if not, an exception would be thrown. To accomplish this task it is required to implement two classes: annotation class and check class - called by the validation engine on a given field.

Let's start with our new annotation:

In the above snippet I've defined a check-annotation, that would be used like this: You can pass file - containing dictionary values for this field. There is also message field in the annotation which is an error message returned by the validation engine of failed check - pretty handy. And can be expressed in .properties file as:

Placeholder, like context, will be replaced with correct values supplied by the validation engine.

Annotating a field is not enough. It is also needed to create a validator for this kind of check. The name of the class is already defined in DictionaryValue annotation, it is called DictionaryValueCheck and I've done this check this way: What this basically does is:
  1. when file is set - read dictionary content from the file into map
  2. upon check request just lookup value in dictionary parsed from the input file

And that's it!

For me Oval is really great tool. With it at ones disposal it is extremely easy to create any imaginable validation you need. This library is really easy to use and offers lots of handy features.

But perhaps I'm reinventing the wheel and all this can be done easily with some other library? Share Your opinion!

 
 

22

maja

2011

Geecon 2011 - day 2

Share |

And now for part 2 of my visit to Geecon 2011!

1. Jim Webber "Revisiting SOA for the 21st century"

Now this was awesome! Jim Webber, a former ThoughtWorks employee, now Neo4j evangelist (in Neotechnology) described his views on how SOA should look - according to him. This was presented previously, on other occasions as his "Guerilla SOA" talk - generally he advocated for REST based services, loose contracts (stating that WSDLs are too verbose and code generation is evil).

Jim mentioned Martin Fowler's article on integration databases but I couldn't find it anywhere - thou the topic looks interesting. He also recommended BDD and exposing tests on the web for the end user to use them as early as possible.

One big point he made his case with was not relying on enterprise software. Simple tools can do much better job. He compared implementing Web Services security (Secured SOAP over HTTP over TCP IP) to REST based service accessed through HTTPS - basic and easily testable with tools like curl.

Great talk. One of the best!

2. Staffan Noteberg "Regex - the future programming"

I must confess, that this did not go too well. The whole talk was well prepared and laid out but it lacked depth. It was pretty basic introduction to regex. From the presentation's subject I was rather prepared for some novel uses of regex - like for example: showing how to filter big volume of data with simple regex or sth.

But the talk was fun, Staffan is a good speaker. He is also an author of pomodoro technique book - I intend to read sth abut this technique and this may be a nice start

3. Bartosz Kowalewski "Is OSGI ready for wide adoption?"

If it comes to titles I tend to rely on them pretty heavily, however strange it may seem. This time I also did - and the whole talk did not give me a definitive answer to the stated question.

Sure, the presentation was informative, but it described some OSGI specific, quite low level stuff. Of course, if you want to use OSGI - even by leveraging application server with OSGI under the hood - you should know a fair bit about the technology itself. Even thou the AS does a good job of hiding OSGI container specifics from the developer, in case of problems it's better to be well informed. All in all - the talk gave too little information for me.

4. Vaclav Pech "Pick low hanging fruit"

"Parallelism is not hard, multithreading is" - this was the key sentence of the presentation. The speaker showed how to introduce concurrency into normal java/groovy code by sprinkling it with concurrency powder. Easy enough! With GPars library he showed:

Good talk, well received!

5. Anton Arhipov "Bytecode for discriminating developers"

Technical introduction to the world of bytecode, jvm specification details. I've drifted away to some other topics - really - can't recall what this was all about.

6. Andreas Almiray "Polyglot Programming"

This was a nice talk covering Groovy, Scala and Closure. The whole point of it was to show how cool it is to play with emerging JVM languages. They are not only fun but also useful. What's more, they bring freshness to java world, injecting it with some new paradigms and methodologies. It is easier to incorporate new ideas into younger JVM languages than to the mature Java.

7. Jim Webber "A pragmatic introduction to Neo4j"

And Jim Webber again, this time with some Neo4j evangelism. First came some taxonomy information on NoSQL databases (Not Only SQL) as a whole - than some specific examples of problems solvable with graph databases - and Neo4j is a graph database.

Main points of Jim's talk were:

Conclusion

These were all the sessions I attended. On Saturday there was a Hacker-garden, but neither I had time nor will to stay - the topics were very interesting and I'd definitely like to experience such an event, but after 2 days of continuous talks I was rather tired.

To sum up, 2011's Geecon was a great experience, with lots of interesting talks and lots of new inspirations. Keep up the good work guys!

 
 

19

maja

2011

Geecon 2011 - day 1

Share |

Last week's Java conference - Geecon was very interesting. It was well prepared, and gave an insight into the current Java related trends - concurrency, DSLs, polyglot programming. But not only that - there were also some pretty different talks from excellent speakers.

The whole event took 4 days:

I decided to attend only on Thursday and Friday - no time for more. Here is what interesting happened during those days.

Day 1

The morning got me unprepared. After hard enough, after work travel to Krakow on Wednesday, I wasn't in the best shape. However after arriving at the venue, being greeted with breakfast and refreshments I looked at the rest of the day with real hope.

Since the schedule was tight - three parallel tracks of lecture, I had to choose, so bare in mind, that is my account of what I've seen and heard. Others may, of course, differ.

1. Danny Coward "Java SE: The Road Ahead"

Danny, being on Oracle (considering being also former Sun's employee a plus) payroll, gave an insightful talk on new things to came in Java 7. He drew rather serious plans for Java 8. According to Danny, the main trends in today's Java ecosystem are:

and he probably is right :) The great things to come with new versions of Java are:

The talk itself was a nice keynote, but I doubt the road map for Java will be met in its full extent - the goals aren't that small.

2. Juergen Hoeller "Enterprise Java in 2011"

Spring Source as one of the sponsors sent Juergen to evangelize about the world of enterprise and Java's place in it :) He emphasized different kinds of deployment: WAR, cloud deployment - and the latter's rise of importance.

He pointed out how outdated current application servers are - the usually lag ~3 years behind the main trends and developers' needs - good point! He proposed looking under the hood of now-popular cloud environments: Google App Engine or Amazon Elastic Cloud to look for schemas in them, etc - I intend to listen to his advice.

All in all this guy gave a great talk covering wide spectrum of technologies and not focusing on technical stuff too much.

3. Heinz Kabutz "Reflection madness"

Despite living on a Greek island, this guy showed also how to whack ones mind with Java Reflection API. Pure magic! Some highlights of his talk were:

With all this examples he pointed that using SecurityManager will prevent such nasty coding practices.

Since he is an editor of Javaspecialists.eu newsletter, all the answers to problems presented in his talk (and many many more) can be found there.

Well done, not to useful for me, nevertheless - interesting.

4. Michael Figuiere, Cyrille Le Clere "NoSQL & Datagrid from developer perspective"

I don't know what to think of this talk. It consisted an introduction to NoSQL databases but also a bit of problem's description that can be encountered when dealing with them. Notable thoughts were on:

5. Hamlet D'Arcy "New Ideas for old code"

Since a lot of developers (all?) have to deal with legacy code - one way or another, this talk was a must!

. The speaker shared some ideas on how to work with such code and remain sane. The talk was vivid, interesting and entertaining, well, and the notable thought? Here they are:

This was nice! - it assumed arriving at a project with no ( or little) tests.

6. Aslan Knutsen "Arquillian"

The last talk of that day was about some new library from JBoss that would allow to test your components with unit tests - test them in a destination container. The whole point of this library is to run the specific fragment of code as if it was build and deployed to some application server (let's say JBoss AS ;-) ). To be honest, I can't find much application for that - thou I'm not doing any serious work in JEE world.

Party

And the day ended. But there was sth else to do after the official part - party time! It took place at Klub Pauza on Floriańska street. It was a rather nice social event.

... to be continued - stay tuned for part 2

 
 

02

maja

2011

JCE keystore and untrusted sites

Share |

Recently at work I was in need of connecting to a web service exposed via HTTPS. I've been doing this from inside Servicemix 3.3.1, which may seem a bit inhibiting, but that was a requirement. Nevertheless I've been trying my luck with the included servicemix-http-2008.01 component. I've created a simple Service Unit using that component and made connection attempt. Unfortunately I've encountered issues with the SSL conversation negotiation. I had to dig deeper into the servicemix-http code to find out these had something to do with my JCE keystore. Read more to find out what happened!

Ok, so I had my xbean.xml for http component looking like this:

As you can see this is a proxy adapter to some outside service exposed via secured HTTP protocol. Since it's HTTPS I've specified some SSL parameters. It was sufficient in my case to just pass the keystore file and it's password.

I've created my keystore.jks file in smx_home/conf with password servicemix in the following manner:

You can see what's in this file with this command:

At this point I thought, that having a configured keystore and my component would suffice. Wrong! As soon as I've tried to connect to the external service I got an exception:

Hmmm.. this looks pretty nasty, but it's not that bad. As one can read here, it's associated with the other site's having an untrusted (unsigned) certificate. Assuming you actually trust the other end of the communication and this situation is ok for you, you should add the servers certificate to your keystore. The previously mentioned link contained a little java class that would do just that. You can find it here (original code) InstallCert.java or you can look into my slightly changed version here at github.

You should call it as follows, assuming that file keystore.jks is in the current directory:

What you'll probably see, when you execute this app is this:

Please note that there is a prompt (Enter certificate to add to trusted keystore...) in which you can enter the certificate number you wish to add to your keystore.

After all those steps my request got through and I could happily query HTTPS service as long as I wanted to! Great!

Possible problems

In my search for this problem's solution I've encountered this kind of exception:

A little googling led me to this StackOverflow question.

It seems that you cannot have multiple keys with different passwords in the same keystore and use KeyManagerFactory class. Oh well...

.

Ending

To sum up, the solution given works, but in my opinion using the InstallCert.java app is rather dirty. I've been wondering, do you know other ways of doing that thing?

 
 

01

kwietnia

2011

Advisory Messages to the rescue

Share |

The most crucial part of software development is testing. It should ensure us, that our code is correct, works according to given specs, etc. There are many kinds of tests: unit tests, integration, functional. In general you should try to test the smallest possible subset of your code and be able to check the state of the objects after the test.

This seems as rather easy task, but what if you have an integration end-to-end test to perform? In most cases asserting state in integration test is rather hard due to multiple systems interoperability. Let's focus on a specific situation.

What I needed to do the other day was write some integration test for Jms based system. The processing pipeline is easy:

some other system (X-system) polls JMS:

 
 

27

marca

2011

How to run multiple guest OS in QEMU?

Share |
This weekend I've been fiddling with QEMU. I've installed OpenBSD on a single image and wanted to have two instances of it communicating via network. Installing the system was easy, but the networking setup was quite a pain. See how I did that...

To make QEMU instances communicate with each other I needed to plug them to a "network". That's why I've created a bridge to which Virtual Instances would connect to. I've used the following script:

Then I just needed to start Qemu with this command line:

Since I've set up bridge for Qemu instances, I've plugged TAP interfaces into it. That's why I've needed to specify this in my qemu exec line. I've also added macaddress setting since both my instances were getting the same one.

And that's all! It works like a charm. Now on to some harder things!

 
 

20

marca

2011

Me on Hadoop on Parleys

Share |

Finally I've managed to import my WarJUG presentation to parleys.com. See for yourself :)

If you've got problems with opening the parleys' version try the ones uploaded to youtube.

Here is part 1:

And here is part 2:

 
 

23

lutego

2011

After WarJUG

Share |

Some time ago I'd written about my arsaw JUG presentation. I finally presented the the topic yesterday.

I must say I'm fairly content with my yesterday's presentation :) Here are some slides and as soon as the video will be available I'll post it here too.

 
 

17

lutego

2011

Hadoop at Warsaw JUG

Share |

I'll be speaking at the upcoming JUG meeting in Warsaw. The topic will be Apache Hadoop. If you're interested, feel invited. Attend and ask questions. But be aware I'm not advanced in this technology :)

Full invitation text can be found here.

P.S.: Do you know what Hadoop is? Do you use it anywhere? Have you played with it?

 
 

17

lutego

2011

Deus Ex - what a game!

Share |

Some time ago I'd become obsessed with one game. It kept me up late, fed my imagination, sucked all my free time - so basically did what good game should. It was not a new production, quite the contrary -it's release date was year 2000. And the game itself was Deus Ex.

The year is 2050 and you wake up as a rookie named JC Denton - you are an UNATCO (United Nations Anti Terrorism CoOperation) special agent. In the beginning your orders are clear and you're there to fight the terrorists. The first mission takes place in the depths of The Statue of Liberty, and it gets better further into the game. The plot wanders, there seem to be many foreign forces entangled, and it may become clear to you, that actually you're being used as a tool - a blunt one. Gradually the plot unfolds and you become aware of the whole conspiracy theory thing. You interact with Majestic 12 and Illuminati secret organizations and become part of a global terrorist warfare.

As for a game released so many years ago, even before 11 September 2001 - and the terrorist's attacks on WTC - the game is astonishingly predictive. It envisions cyberwarfare, the rise of China's economic importance, threads of terrorism and the whole range of technological topics. Well, mainly some sci-fi visions that became reality the following years since game's release ;-)

Deus Ex has extremely well thought world. Everything is well laid out and there are no major glitches. It also offers multiple solutions to almost each puzzle - you can fight your way through, but you can try to bribe, steal, sneak and stunt. And the engine supports these actions perfectly (I've tried sneaking when playing Fallout: New Vegas but failed miserably). The myriad of solutions are for you to choose from. It would be even better if you could influence the plot in even greater, more substantial degree - but perhaps it would be too much to ask for.

Of course the game has also some weak sides. Because it's a little bit dated the graphics are not up to today's standards. The other thing is that the world seems not too densely populated - but as a matter of fact it may be because of 2000's hardware limitations.

I got the game from Deus Ex Trilogy Pack (3 games in it, just for 20 PLN). The pack contains also other parts called: Deus Ex: Invisible War and Project Snowblind. Invisible War is still an RPG, but Project Snowblind is just a regular FPS - really daunting. I've played the original Deus Ex on Linux with Wine - it worked very well and I haven't experience any problems. I've tried the same approach with Invisible War, but it seems sooooo buugy that I turned it off after just an hour. Perhaps will try to run it under Windows some other time - first to see if the bugs are gone :)

Oh, and there is also a third part coming, which will be a prequel to the first game. It will be called Deus Ex: Human Revolution. Go see some trailers and gameplays on You-tube. It seems well worth the time and money it will cost.

UPDATED: some links from comments below:

 
 

17

stycznia

2011

JMS redelivery with ActiveMQ and Servicemix

Share |

The other day I felt a compelling need to implement a JMS redelivery scenario. The exact scenario I'd been trying to handle was:

  1. my message is in an ActiveMQ queue or topic
  2. its processing fails, because of some exception - ie. database access exception due to server nonavailability
  3. since we get an exception, the message is not handled properly, we may want to retry processing attempt some time later
  4. of course, for the redelivery to happen we need the message to stay in the ActiveMQ queue - fetching messages from the queue will be stopped until the redelivery succeeds or expires
See how this can be done after the jump :)

For this to happen, I've tried implementing Apache Camel route, but as it turns out, Camel fails to deliver facilities for exact JMS redelivery. It is possible to set JMS connection in transacted mode, but the redeliveries happen one after another and fixed times.

What I've ended up doing was implement a servicemix-jms endpoint. I've used this configuration for it:



    
        
            activemq/connectionFactory
        
    

    
        
            activemq/resourceAdapter
        
    

    
        
        
        
        
    

    
    

As you can see, we lookup a couple of things in JNDI registry, so you need to have them configured on the Servicemix side - a sample config presented farther in this entry.

The bean responsible for configuring redelivery settings is activationSpec. You can set various things with it, like:

  • initial redelivery delay
  • maximum number of redeliveries
  • backoff multiplier
  • ...

What is really important in jms:endpoint config for this to work are:

  • processorName="jca"
  • rollbackOnError="true"

Servicemix should have the following entries in its jndi registry:

         

         

(...) 

       xmlns:jencks="http://jencks.org/2.0"
       xmlns:amqra="http://activemq.apache.org/schema/ra" -->

          
          
          

When the redeliveries are exhausted, message is routed to global Dead Letter Queue called ActiveMQ.DLQ. Since this is a single bag for all the failed messages from all queues, you may want to configure this aspect differently. For example you can tell ActiveMQ to create a single DLQ for each queue. Use this config to achieve it - the changes should be made to Broker configuration.


  
    
      
        
        
          
            
            
          
        
      
    
  
  ...

More on the subject of redelivieries in ActiveMQ can be found at http://activemq.apache.org/message-redelivery-and-dlq-handling.html.

 
 

10

grudnia

2010

Ku pamięci - o top-postingu

Share |

W celu niezagubienia i posiadaniu argumentu w przeróżnych dyskusjach:

http://idallen.com/topposting.html
 
 

08

grudnia

2010

Easier and nicer JMS

Share |

JMS seems like a hostile ground. It has all it's quirks and strange behaviours. A couple of defining standards plus esoteric brokers, queues and topics.

At work, we mainly use open source Jms solutions, namely Apache ActiveMQ. This one is usually bundled with Apache Servicemix, as a message broker for this particular ESB. As there are some minor caveats in this scennerio, I'd like to describe here some guidelines for getting to running JMS queues.

Treat this post as a quick cheat sheet with the most common things about JMS I tend to forget :)

Minor glitches encountered during work with embedded broker led to some thoughts about switching to external broker. This is how I configure SMX and AcviteMQ.

Necessary steps:

  • change apache-servicemix/conf/servicemix.properties activemq.port to sth else than standard, for example 61626
  • change apache-activemq/conf/activemq.xml with this settings:
    • change port, the service listens on:
              
                  
              
      
    • setup separate JMX instance:
              
                  
              
      
  • the nicest tool I found for browsing queues and topics is Hermes JMS. Sample config, that connects Hermes to ActiveMQ instance is on the picture below: HermesJMS to ActiveMQ connection config
  • sending simple messages with Hermes is basic, but what if you need to set some headers, send bulk messages, etc. Easy, just use Hermes xml format. Look like this code snippet below and is rather self-explanatory:
    
        
            
                
                
      
        
          105
          1235
        
      
    ]]>
            
        
    
    
  • since we use lots of Apache Camel to consume messages, here is a simple way to start broker in your tests:
    • start a broker
              BrokerService broker = new org.apache.activemq.broker.BrokerService();
              broker.setBrokerName("AMQ-1");
              broker.addConnector("tcp://localhost:51616");
              broker.setPersistent(false);
              broker.start();
      
      Notice it has persistance disabled.
    • initialize Camel's JMS component:
          ctx.removeComponent("jms");
          ctx.addComponent("jms", ActiveMQComponent.activeMQComponent("tcp://localhost:51616"));
      
    • if you want to pass messages to reference endpoints, (like ref:input), use this wrapper method:
      private JmsEndpoint createJmsEndpoint(String endpoint) throws JMSException {
              ActiveMQComponent amqc = (ActiveMQComponent) ctx.getComponent("jms");
              JmsEndpoint endp = JmsEndpoint.newInstance(new ActiveMQTopic(endpoint), amqc);
              return endp;
      }
      
      createJmsEndpoint("ESB/XYZ")
      
These are all the tricks I've got for now! But if you know some other good tools that handle JMS, feel free to comment! Got more advices, again, comment!
 
 

06

listopada

2010

Best pigsticking, EVER!

Share |

This looks like the best pigsticking ever! Look at that little girl! The blood, butcher's knife, and child's soother :)

Photo from Jamie Oliver's cookbook, "Jamie's Italy". Available on Amazon.
 
 

21

października

2010

Schematron to the rescue!

Share |

In an ideal world all the standards fit well into their places. It is sufficient to use just one serious standard, because all the problems can be solved with it - the standardization processes is there for some reason. But that happens only in ideal world, which we're not living in.

In ideal world, when dealing with XML instances you'd be more than fulfilled using XML Schema, or RelaxNG, or any other simple xml formal definition language to declare your data structure. With that you get rigid rules as to how XML documents should look like. There doesn't seem to be much space to deviate from specs. Well, in fact there is.

The main problem of XML, aside its verbosity, is the inability to create concise rules for the input or output document as a whole. Perhaps it's a nice feature, because XML Schema should only be used to describe a data structure, not to infer business rules on it. Perhaps not. Nevertheless it's not what I needed in one of the projects I've worked on.

My need was to actually check the business validity of such documents. This was used in a Web Service environment, a pretty stupid WS, which sole role was to fetch data from database and pack it into appropriate XML structures. Errors might occur in database's views or in WS - as usual. They might be data multiplication or appearance of some elements while they shouldn't. Resulting documents were correctly validated with the xml schema, but the result was simply wrong from the business point of view.

What I needed an XML formalization language, an ability to write rules that would assert some rules, report on not meeting stated rules. I was in need of a tool to write business rules to tame such XML entities.

The simplest way I found to solve this was to use Schematron! - "a language for making assertions about patterns found in XML documents". This neat tool is a set of XSL templates, that you use in conjunction with a rule set on documents to check. As a result of the check you get another XML document with test assertions - whether failed or succeeded.

With Schematron you write a set of rules you expect the document to assert, than you use Schematron XSL template to produce XSL rules specific for your case. Now you only need to use newly generated XSL rules template on your XML document to check rules compliance. Easy, if not, check the diagram below.

How does it look?

The rules' file may look like this:


  TouK Schematron test harness

    



  checking GetMigrationOffers
  
    Report date.
    Unique offers allowed.
    Each offer has to have an @abc attribute 
  
  
    Each offer has to have a tariff
    Each offer has to have a promotion
  



  checking GetAllPhones
  
    
        TACs should be unique. TAC: , 
        handsetId: 
        offerId: 
    
  



Here we see two rules, one named getMigrationOffers and the other getAllPhones. The rules - mainly their asserts seem pretty self explanatory, but for the sake of completeness I'll describe the rules for getAllPhones.

There is one rule, which checks the uniqueness of tac elements. This rule tries to ensure that each handset should have a list of unique tac elements as its children. However there may appear tac elements of the same value in different handset elements.

Given an input XML in the form of:


   
      
         
            12028006
            20070705
            35535302
            01216100
            01216100
         
         
            12028006
            20070705
            35535302
            01216100
         
         
            12028006
            20070705
            35535302
            01216100
         
       
   

And passing those two files through the processing pipeline you get a report:


    
   
   
   
   
   
   
   
   
      
        TACs should be unique. TAC: 01216100, 
        handsetId: 95
        offerId: 103021
   
   
   
      
        TACs should be unique. TAC: 01216100, 
        handsetId: 95
        offerId: 103021
   
   
   
   
   
[...]

After running the validation, the report presents us with the result. It shows that there are actually non-unique tacs. Unfortunately the rule itself is not so optimal, as it is executed for each tac node. The better case would be to create a rule operating only on groups of tacs - having a rule for each handset's tacs would be much better.

Performance consideration

As you may have seen, Schematron gives quite a potential, if it comes to rules building - maybe not the easiest to comprehend, since written with XPath, but good enough.

However, with all the XML processing involved in the process, it may take some considerable amount of time to execute such validations. For example, processing rules for file getMigrationOffers.xml takes about 2.296s - the file has 82 offer elements, which the rules operate on. But validating the other file, getAllPhones.xml takes 5.324s, with 3113 tac elements, and the rule iterating all of them.

This overhead is too much in most of the situations. That's why this solution is rather not for use in normal execution pipeline - it would be unwise to put Schematron to check each request, thus entangle it into my Web Services normal flow.

What may be more desirable is to deploy a continuous integration server, with a project querying such Web Service and checking the rules in this manner.

Conclusion

So, what's so great about having one XML generate another XML? Perhaps nothing, I think it would took just about a day to write some shell, python, <other text processing tool> that would perform equally (or even better). However, we loose technology homogeneity, and employ some other environments, not specific to our primary target platform, and that seems bad. Of course using some powerful text processing tool to impose the same rules might be much more efficient, thou less coherent.

What is your approach to such situations? Have you used Schematron or any other similar tool?

Code for this example is available on GitHub - http://github.com/zygm0nt/schematron-example.

 
 

26

sierpnia

2010

Complex flows with Apache Camel

Share |

At work, we're mainly integrating services and systems, and since we're on a constant lookout for new, better technologies, ways to do things easier, make them more sustainable, we're trying to

Usually we use Apache Camel for this task, which is a Swiss-knife for integration engineer. What's more, this tools corresponds well with our approach to integration solutions:

  • try to operate on XML messages, so you get the advantage of XPaths, XSL and other benefits,
  • don't convert XML into Java classes back and forth and be worried with problems like XML conversion,
  • try to get a simple flow of the process.

However, at first sight Apache Camel seems to have some drawbacks mainly in the area of practical solutions ;-). It's very handy tool if you need to use it as a pipeline with some marginal processing of the data that passes through it. It gets a lot harder to wrap your head around if you consider some branching and intermediate calls to external services. This may be tricky to write properly in Camel's DSL.

Here is a simple pipeline example:

And here the exact scenario we're discussing:

What I'd like to show is the solution to this problem. Well, if you're using a recent version of Camel this may be easier, a little different, but should still more-or-less work this way. This code is written for Apache Camel 1.4 - a rather antic version, but that's what we're forced to use. Oh, well.

Ok, enough whining!

So, I create a test class to illustrate the case. The route defined in TestRouter class is responsible for:

  1. receiving input
  2. setting exchange property to a given xpath, which effectively is the name of the first XML element in the input stream
  3. than, the input data is sent to three different external services, each of them replies with some fictional data - notice routes a, b and c. The SimpleContentSetter processor is just for responding with a given text.
  4. the response from all three services is somehow processed by RequestEnricher bean, which is described below
  5. eventually the exchange is logged in specified category

Here is some code for this:

public class SimpleTest {
    public void setUp() throws Exception {
        TestRouter tr = new TestRouter();
        ctx.addRoutes(tr);
    }

    @Test
    public void shouldCheck() throws Exception {
        ctx.createProducerTemplate().send("direct:in", getInOut("<a/>"));
    }


    class TestRouter extends RouteBuilder {

        public void configure() throws Exception {

            ((ProcessorType<ProcessorType>)from("direct:in")
            .setProperty("operation").xpath("local-name(/*)", String.class)
            .multicast(new MergeAggregationStrategy())
                .to("direct:a", "direct:b", "direct:c")
            .end()
            .setBody().simple("${in.body}"))
            .bean(RequestEnricher.class, "enrich")
            .to("log:pl.touk.debug");
            
            from("direct:a").process(new SimpleContentSetter("<aaaa/>"));
            from("direct:b").process(new SimpleContentSetter("<bbbb param1=\"1\" param2=\"2\" param3=\"3\"/>"));
            from("direct:c").process(new SimpleContentSetter("<cccc/>"));
        }
    }
}

What's unusual in this code is the fact, that what normally Camel does when you write a piece of DSL like:

	.to("direct:a", "direct:b", "direct:c")

is pass input to service a, than a's output gets passed to b, becomes it's input, than b's output becomes c's input. The problem being, you loose the output from a and b, not mentioning that you might want to send the same input to all three services.

That's where a little tool called multicast() comes in handy. It offers you the ability to aggregate the outputs of those services. You may even create an AggregationStrategy that will do it the way you like. Below class, MergeAggregationStrategy does exactly that kind of work - it joins outputs from all three services. A lot of info about proper use of AggregationStrategy-ies can be found in this post by Torsten Mielke.

public class MergeAggregationStrategy implements AggregationStrategy {

	public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
		if (oldExchange.isFailed()) {
			return oldExchange;
		}
		transformMessage(oldExchange.getIn(), newExchange.getIn());
		transformMessage(oldExchange.getOut(), newExchange.getOut());
		return newExchange;
	}
	
	private void transformMessage(Message oldM, Message newM) {
		String oldBody = oldM.getBody(String.class);
		String newBody = newM.getBody(String.class);
		newM.setBody(oldBody + newBody);
	}
	
}

However nice this may look (or not), what you're left with is a mix of multiple XMLs. Normally this won't do you much good. Better thing to do is to parse this output in some way. What we're using for this is a Groovy :). Which is great for the task of parsing XML. A lot less verbose than ordinary Java.

Let's assume a scenario, that the aggregated output, currently looking like this:

	
	
	

is to be processed with the following steps in mind:

  • use <aaaa/> as the result element
  • use attributes param1, param2, param3 from element <bbbb/> and add it to result element <aaaa/>
public class RequestEnricher {
	
	public String enrich(@Property(name = "operation") String operation, Exchange ex) {
		
		use(DOMCategory) {
			def dhl = new groovy.xml.Namespace("http://example.com/common/dhl/schema", 'dhl')
			def pc = new groovy.xml.Namespace("http://example.com/pc/types", 'pc')
			def doc = new XmlParser().parseText(ex.in.body)
			
			def pcRequest   = doc."aaaa"[0]
			
			["param1", "param2", "param3"].each() {
				def node = doc.'**'[("" + it)][0]
				if (node)
					pcRequest['@' + it] = node.text()
			}
			
			gNodeListToString([pcRequest])
		}
		
	}
	
	String gNodeListToString(list) {
		StringBuilder sb = new StringBuilder();
		list.each { listItem ->
			StringWriter sw = new StringWriter();
			new XmlNodePrinter(new PrintWriter(sw)).print(listItem)
			sb.append(sw.toString());
		}
		return sb.toString();
	}
	
}

What we're doing here, especially the last line of enrich method is the conversion to String. There are some problems for Camel if we spit out Groovy objects. The rest is just some Groovy specific ways of manipulating XML. But looking into enrich method's parameters, there is @Property annotation used, which binds the property assigned earlier in a router code to one of the arguments. That is really cool feature and there are more such annotations:

  • @XPath
  • @Header
  • @Headers and @Properties - gives whole maps of properties or headers

This pretty much concludes the subject :) Have fun, and if in doubt, leave a comment with your question!