Wednesday, September 21, 2011

Creating a Virtual Appliance with Karaf: Part 1 (Finding the Right Cloud)

Something I like to do to keep my understanding of technologies up-to-date is to identify a technology on the cusp of widespread acceptance, and to figure it out. This is why I decided to start working on Karaf and OSGi a year ago, and it has served me very well.

The current wave is the cloud and virtualization. Like OSGi, its been around for a while, and is being widely accepted. Many organizations are familiar with "virtual machines", not Java VM's, but the ability to start up an instance of an operating system within another operating system.

Anyhow, as a newcomer to cloud computing, my first small chore was to create a virtual appliance containing Karaf and Celler. Karaf is a small, lightweight OSGi container that allows you to do enterprise stuff. :-) To accomplish this, I used an older Toshiba Satellite computer system as my server, and a newer Dell laptop as my client. The client is planned to be used for development of the virtual appliance.

VMWare: VMWare has a great web-centric application called "Go" that small-to-medium businesses can use to download thier Hypervisor version of software. Using this, businesses can get access to thousands of virtual appliances, and leverage them (for a cost) to handle their business needs. However, there are some drawbacks. Specifically, my laptop needed a gigabit controller. Because my slow computer only had a little ol' nic-card, the driver couldn't be found and I was unable to install Hypervision to test it. Also, remember that installing this app successfully on your system will completely wipe out the previously installed operating system.

Susestudio: SuseStudio has an awesome web-based interface for creating virtual applications. The only drawback I saw to that was the fact that .rpm's needed to be created for any applications I wanted to install into my VApp. I'm still looking to see if this is the standard. In any case, this was a show-stopper for me.

XenServer: This, like VMWare, was a download that wiped out my hard-drive. They were nice-enough to let me know that during installation. The good thing is that this is the first cloud system I've been able to install on my test-system, so this is the server I'll use to buildand install my virtual appliance!

Next time: How to find the XenServer UI!!

Monday, September 19, 2011

Why OSGi?

Recently I answered a question on stackoverflow.com about the difference between component-based and modular architectures. However, what it really does is show one of larger benefits of OSGi, fixing the classpath.

Here is the question and my answer:

Q: OSGi is a modular architecture, JavaBeans is a component architecture. What's the diff?

A: The primary difference between OSGi and Java Beans is in how the classloader works. In a standard .jar file or EJB, the rt.jar file or EJB equivalent maintains the classpath. Additionally, if you are using a container to deploy your application into, you may have multiple classpath maintanance mechanisms which cause problems. As a result, when you make a .war file, for example, you usually create a lib directory with all of your .war's .jar dependencies inside of the .war. If you only have one war or .jar in your application, this isn't so bad. But imagine a large enteprise deployment with 100 EJB's all containing apache-commons! You end up with 100 instances of apache-commons all running inside the same container sucking up resources.

In OSGi, you deploy each .jar file (we'll call them bundles cuz this is OSGi now) into the OSGi container. Each .jar file exposes (exports) the packages it wants other packages to use, and also identifies the version of the bundle. Additionally, each bundle also expressly states (imports) the packages it needs from other bundles to work. The OSGi container will then manage all of these exports and match them up to the appropriate imports. Now you have apache-commons available to each of the EJB's you want to make available. You've done away with your /lib directory and now your application takes up less resources.

In your question you asked the difference between a component architecture and a modular architectures. Modularity refers to this process of making each bundle its own deployment unit and allowing it to talk to other bundles instead of balling them all up into one massive .jar file.

Wednesday, July 20, 2011

Hibernate Antlr and OSGi Fragments

From time to time, you'll run across instances where a bundle of yours in OSGi needs to have access to a class, but can't. The error you'll get is "ClassNotFoundException: foo.Bar" where foo.Bar is the package and class your bundle is trying to access. Then, after reviewing your OSGi environment, you see that the package of that class is available, but for some reason your bundle can't see it. What the heck? Why can't it see it?

This usually happens when you use a pre-bundled .jar file that calls "Class.forName(bar.Foo)". Calling Class.forName() isn't a good practice inside of OSGi, because it requires your bundle to be able to perform "wiring" after it has been active and started. But, seriously, you can't change over a decade of programming practices overnight, so folks still do this. Seeing that this would be an issue, the good folks at OSGi created a neat way of fixing it, the bundle fragment! (applause, Woo HOO!)

What a bundle fragment does is add functionality to an existing bundle. In the OSGi reference the purpose is usually listed as providing localization support. However, it is also a very power mechanism for adding and changing the contents of a bundle's MANIFEST.MF file. For those of you unaware of what this file does, it includes a set of directions for the OSGi environment on how to treat a bundle.

In the above example, you would create a fragment adding the foo package to the Import-Package section of the MANIFEST.MF file.

Most bloggers would stop there. I identified an issue and then told you how to fix it. But not me, nope, I want to show you how a real-life example of this using Hibernate and Antlr.

Hibernate uses a parsing service called AST. This, in turn uses Antlr to help with its parsing. Unfortunately, Antlr needs to use a hibernate class called "org.hibernate.hql.ast.HqlToken". And, of course, Antlr does a Class.forName() at runtime to get an instance of it. This kind of makes sense, Antlr wasn't written just for use with Hibernate. As such, it needs to be told at runtime what token it should use for parsing.

To fix this issue, you create a fragment that adds an Import-Package entry for the package HqlToken is in, "org.hibernate.hql.ast". Doing this is pretty simple. First, you create a normal java project. It doesn't matter what tool you use, as long as you have the following basic structure.

- antlr-hibernate-fragment
   pom.xml
   - src
     - main
       - java
       - resources
         - META-INF

Because we're only adding something to the MANIFEST.MF file, the only file in this project that will have anything in it will be the Maven pom.xml. There are a ton of places folks can go to get smart on Maven, so I'm not going to review how the pom.xml file should look other than the maven-bundle-plugin.

In your build section of your pom.xml, add the following

<plugins>
   <plugin>
     <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-bundle-plugin</artifactId>
     <extensions>true</extensions>
     <configurations>
       <instructions>
         <Fragment-Host>com.springsource.antlr</Fragment-Host>
         <Import-Package>org.hibernate.hql.ast</Import-Package>
       </instructions>
     </configurations>
   </plugin>
<plugins>

Then, run the following from your console, or compile it with your IDE.

mvn clean install

This will create a bundle fragment ready for use with OSGi. In Karaf, after you deploy your original antlr bundle and the fragment, you can run the following console command to see the new import package directive added to the rest of antlr's import-package section. The bundleId referred to in this code is the bundleId of the original antrl bundle, not the new fragment.

headers (bundleId)


I added the new antlr fragment to my hibernate features.xml document (from a previous blog entry) right after my antlr bundle and then deployed it as part of my Hibernate feature.

Please let me know if that helps!

Monday, July 18, 2011

Hibernate and OSGi

This is a copy of a blog post I did about a year ago on Java.net. Of my messages on that blog, this was the most viewed and linked. So, I've improved it, and included it in my new blog to help folks looking for help with Hibernate inside of OSGi.

One of the major contributors to the OSGi movement is Peter Kriens who created an excellent tool called BND. The good folks at apache-felix then created a maven-plugin that makes quick work of bundling apps, but I'll go over that in a later blog. The reason I bring this up here is that a while ago Peter wrote www.aqute.biz/Code/BndHibernate which discusses how to create a mega-bundle with all of Hibernate 3's core .jar files and dependant .jar files wrapped up inside of one huge .jar.

This is an excellent approach, but it has a few drawbacks:

  • It requires the creation of a new .jar file which may confuse new developers,

  • It is pretty complex, and

  • The BND tool, while most excellent and wonderful, takes some time to figure out.

What I'm going to do is show you how to leverage Karaf provisioning and SpringSource bundles to do something similar, but without having to create a massive Hibernate mega-bundle. As Peter says Hibernate is one of the more complicated open source projects to wrap". To do this, I'll:

  • Talk about an excellent source for bundles, Spring Source,

  • Describe what a features.xml file is,

  • Show you a working features.xml file for Hibernate 3, and finally

  • Show to modify one of Karaf's configuration files to automagically start up Hibernate when you start Karaf.

First, I 'd like to talk about Spring Source. These folks are doing an outstanding job of creating bundles out of commonly used java apps, like Hibernate. I would be remiss if I didn't mention the fact that, without their hard work, simply deploying Hibernate using a features.xml file would not be possible. The link points to their bundle repository, you should bookmark it, its a great resource.

Karaf has a few core concepts that this blog will illustrate to you:

  • OSGi bundles,

  • Karaf Features,

  • Features.xml documents, and

  • the org.apache.felix.karaf.feature.cfg file.

When you have an application like Hibernate that has dependencies on other libraries, one way to deploy them is to turn them into OSGi bundles and then create a features.xml file. An OSGi bundle is simply a .jar file with a MANIFEST.MF file containing OSGi goodness. There are a lot of great resources already available on the internet that go into detail about what an OSGi bundle is composed of, so I won't go further into it here.

Deploying a large application composed of bundles using a features.xml file is much less time intensive than deploying them manually. Most of the large bundled open-source programs have created features.xml files for you, Camel being the first one that comes to mind. Unfortunately, at the time of this writing, I was unable to find one for Hibernate, so I made my own using a list of dependencies created by the good folks at Spring Source.

First, start Karaf. Then, to deploy a single OSGi bundle into Karaf, you simply need to invoke the following command (for the uninitiated "karaf@root> " is the prompt, dont' type that.)

karaf@root> osgi:install -s (uri of your bundle, eg mvn:myproject/myapp/version)


In practice, the uri can be anything that can be resolved: a maven URI, a url, or even some file on your local file-system! Once you've typed that, Karaf will print out the name of the bundle you installed on the command-line. Alternatively, you could find out the bundle number of the bundle you installed by typing:


karaf@root> osgi:list grep (bundle symbolic name)



That will give you the number of the bundle as it is installed in Karaf. Finally, you start the bundle by typing:




karaf@root> osgi:start (bundle Id number)


A bundle can have a number of states, it can be Installed, Active, or Active and Failed. What you want is for your bundles to be Active. Usually, anything else is a problem. This is what you'd type to install and start an OSGi bundle from a maven repository:

kara@root> osgi:install mvn:org.dom4j/com.springsource.org.dom4j/1.6.1
89
karaf@root> osgi:start 89


Now, imagine if you have 90 bundles in your application, including all of the dependencies. That's a lot of typing! A features.xml does all that heavy lifting for you. The features.xml file is simply an xml file where you identify a given feature, and then define the OSGi bundles and other features necessary for that feature to work inside of Karaf.

Karaf will read in the features.xml file, and when you install a feature, it will automatically download each bundle listed from its associated URI, install it as an OSGi bundle, and start it. It will do this for each bundle unless it finds a problem. If it finds a problem, it will then stop and uninstall each bundle it started. If a bundle was already installed, the Karaf will "refresh" it, and if there's a problem, it will leave it active. This is the features.xml file I wrote for Hibernate, the file is called hibernate-features-3.3.2-GA.xml.


<!--?xml version="1.0" encoding="UTF-8" ?-->
<features>
<feature name="hibernate" version="3.3.2.GA">
<bundle>mvn:org.dom4j/com.springsource.org.dom4j/1.6.1</bundle>
<bundle>mvn:org.apache.commons/com.springsource.org.apache.commons.collections/3.2.1</bundle>
<bundle>mvn:org.jboss.javassist/com.springsource.javassist/3.9.0.GA</bundle>
<bundle>mvn:javax.persistence/com.springsource.javax.persistence/1.0.0</bundle>
<bundle>mvn:org.antlr/com.springsource.antlr/2.7.7</bundle>
<bundle>mvn:net.sourceforge.cglib/com.springsource.net.sf.cglib/2.2.0</bundle>
<bundle>mvn:org.apache.commons/com/springsource.org.apache.commons.logging/1.1.1</bundle>
<bundle>mvn:javax.xml.stream/com.springsource.javax.xml.stream/1.0.1</bundle>
<bundle>mvn:org.objectweb.asm/com.springsource.org.objectweb.asm/1.5.3</bundle>
<bundle>mvn:org.objectweb.asm/com.springsource.org.objectweb.asm.attrs/1.5.3</bundle>
<bundle>mvn:org.hibernate/com.springsource.org.hibernate/3.3.2.GA</bundle> <bundle>mvn:org.hibernate/com.springsource.org.hibernate.annotations/3.3.1.ga</bundle>
<bundle>mvn:org.hibernate/com.springsource.org.hibernate.annotations.common/3.3.0.ga</bundle>
<bundle>mvn:org.hibernate/com.springsource.org.hibernate.ejb/3.3.2.GA</bundle>
</feature>
</features>


Now, copy that bad-boy into an editor, and save it in your ${karaf-home}/etc directory. Once that's done, you'll be ready for the next step.

We can make Karaf aware of this features.xml file manually by typing:


karaf@root> features:installUrl file:///home/myname/apache-felix-karaf-2.2.1/etc/features/hibernate-features-3.3.2-GA.xml


To verify your new features.xml file is installed, type

karaf@root> features:listUrl


If you see your feature listed there, you can start your feature by typing:

karaf@root> features:install hibernate


Give it a few seconds, and when you get the prompt, if you haven't gotten any errors, you can verify that hibernate is installed by typing:


karaf@root> features:list grep hibernate


Installing your features.xml and starting your feature this way is fun, but what if you've got a ton of features to install? Manually is fun; but lets face it, it's not very sexy... If only there was a way to make Karaf aware of the features.xml on start-up? Well, there is! We just add it to the ${karaf.home}/etc/org.apache.felix.karaf.features.cfg file! This is what it will look like when you download and install karaf:

################################################################################
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.java.net/external?url=http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#################################################################################
# Comma separated list of features repositories to register by default
#
featuresRepositories=mvn:org.apache.felix.karaf/apache-felix-karaf/1.6.0/xml/features
#
# Comma separated list of features to install at startup
#
featuresBoot=ssh,management


This file is composed of two very important items for us, a list of features repositories, and also a list of the features to start up when we start Karaf. Including our features.xml file is pretty straight-forward, we simply add the uri to the end of the featuresRepositories line so that it looks like this:

featuresRepositories=mvn:org.apache.felix.karaf/apache-felix-karaf/1.6.0/xml/features,file:///home/myname/apache-felix-karaf-1.6.0/etc/hibernate-features-3.3.2-GA.xml


When I first did this I tried to put a \ character after the first comma and start each features file on its own line, but that created issues, so now I put all of my features on the same line.

When we start up Karaf, we can verify that the features file is installed by typing:


karaf@root> features:listUrl


This produces a list of each features.xml file Karaf was able to successfully read. If you don't see your features.xml file there, go look at it, there's probably an issue. Also, check out the ${karaf.home}/data/log/karaf.log file and see if any errors or exceptions were reported.

Ok, what if Hibernate is part of a larger set of applications, and you want them all to start up when you start up karaf? Well, that's not too difficult. Simply add your new feature to the featuresBoot line so it looks like this:


featuresBoot=ssh,management,hibernate


If everything works properly, when you start up Karaf, your happy new hibernate feature should be up and running and ready for some abuse!

Please let me know if this helps!

Karaf Exceptions: "Missing Constraint: Import Package"

One of the toughest things about working with Karaf, is that the error messages created by Karaf are not clear to folks new to OSGi. My "Karaf Exceptions" set of blogs are here to provide help in navigating these tough waters. Our first exception is "Missing Constraint: Import Package".

(Blatantly plagiarized and improved from my original post on the Karaf User Mailing List)

Lets set up the scenario. A user asked the dev's this:

I'm unable to activate a bundle and am getting - "could not be resolved. Reason: Missing Constraint: Import-Package: org.springframework.web.util; version="3.0.0"".

What configuration setting am I missing?


This error usually arises when the bundle in question is attempting to wire to a package that is not present in the OSGi environment. To verify this, do the following:
1) ensure that the packages isn't already available within Karaf

root@karaf> exports | grep org.springframework.web.utilhttp://www.blogger.com/img/blank.gifhttp://www.blogger.com/img/blank.gif

2) If that command doesn't return anything, then you should find the bundle that contains the package you need, and install that into your container. Doing a simple search for that package on a sonatype repository should let you see the proper bundle.

Ok, this will work for most cases. But there are corner cases where you'll get this error if a third-party bundle can't access a given package. For example, if you're using Hibernate and C3P0 to access a MySQL database. You'll get the above error when C3P0 attempts to get access to mysql driver. Unfortunately, the C3P0 bundle doesn't wire to the driver, so you need to create a "fragment" that adds an import statement to the C3P0 bundle. This is in itself a sticky issue deserving its own blog, so I'll handle it there.

Please try this and let me know if it helps.

Karaf Logging Overview


A recent post on the Karaf User's mailing list was focused on the topic of Karaf logging. This topic is easily misunderstood, and can take some time to figure out. As is usual with many open-source items, this topic is only dryly documented. Below, hopefully I can help to alleviate this.

First, its good to remember that in log4j, logging can be set on a package level. This can be done in Karaf using the following console command:

karaf@root> log:set LEVEL mypackage.subpackage

This will set the logging for that specific package to whatever the level you provide.

In Karaf 3.x, you can also filter for a specific package using the following command:

karaf@root> log:get mypackage.subpackage

Some caveats with the log console commands. With the exception of log:set, your commands won't affect the logs placed on the file system. For example, if you type log:clear, you won't have access to any log messages written out prior to executing the log:clear command. However, log:clear won't remove those older log messages from the log directory.

This is because all of the logging commands don't actually work against your log files. Instead, the logging commands look for PaxLoggingEvents inside of Karaf. Anytime a log message is generated in Karaf, a PaxLoggingEvent is created. The logging commands look for these and then act on them. So, if you accidentally run log:clear, dont' worry, you won't lose all of your log messages. However, if you accidentally set your logging level to DEBUG or TRACE, this change will result in all of your logging messages being set to the new level and your logs filling up very quickly.

Now, the logging console commands are great, but what if you want to break your logging messages out into different log files? For the most part, the logging commands won't help you. If what you want is a file containing only the messages from a given package, you simply create a logger for your package, and then create a RollingFileAppender for that logger. Here's an example of what to place in your org.ops4j.pax.logging.cfg file.

# mypackage.subpackage appender
log4j.logger.mypackage.subpackage=LEVEL, subpackage
log4j.appender.subpackage=org.apache.log4j.RollingFileAppender
log4j.appender.subpackage.layout=org.apache.log4j.PatternLayout
log4j.appender.subpackage.layout.ConversionPattern=PATTERN
log4j.appender.subpackage.file=${karaf.data}/log/subpackage.log
log4j.appender.subpackage.append=true
log4j.appender.subpackage.maxFileSize=10MB
log4j.appender.subpackage.maxBackupIndex=10

Now, some caveats. If your "subpackage" has camel routes that you'd like logged into your subpackage log file, they won't appear. This is because the messages from your camel routes are generated from the org.apache.camel package, and not by your subpackage.

Also, all messages are still going to be written into your karaf.log file. So, if you are seeing some strangeness and can't diagnose it in your subpackage.log file, check out your karaf.log.

As always, please let me know if you have any questions.

Intro - Open Source Technologies

Over the years I've been involved with a number of emerging open-source technologies including J2EE, OSGi, and a number of Apache Software Foundation projects. The purpose of this blog is to focus on the nitty-gritty details of these new open-source technologies as I start working with them. Currently, I'm working with the Apache Software Foundation's Karaf, Camel, and Cellar projects, along with helping to guide the emerging open-source management framework, OpenAgile.

The majority of this blog will be composed of posts made to various user groups, intended to help out the communities using the technologies I help develop and implement.