Understanding how Java Debug works

I found it surprising that many people that works with Java everyday doesn’t know that there are debugging options that go beyond clicking the “debug” button in your IDE.

You can just attach your IDE to a running application (which has been runned for debug as we’ll see later), or you can even debug it from command line. And the application you debug can even be be in a different machine.

The magic lies in where the debug information actually resides. Apparently people normally think that is the IDE that knows how to debug your programs, but the truth is that is the program who knows how to debug itself, and makes that information available to whoever wants to use it.

The way it works is basically the following. When you compile a program, the .class files get debug information within them, like line numbers or local variables that are made accessible to others who want to access this information. You can then run the program in debug mode passing the following options to your java program execution(you can of course run any java program like this, including mvn goals, appllication servers, etc)

-Xdebug -Xrunjdwp:transport=dt_socket,address=4000,server=y,suspend=y

(you can also use -agentlib:jdwp instead of -Xrunjdwp in latest Java versions)

This line basically says: Run this program in debug mode, use the jdwp protocol, with a socket that listens to port 4000 and waits for connections to continue.

The jdwp protocol is a communication protocol used by the Java application to receive and issue commands and reply to them.

For example, you can connect to this port when the application is running an issue commands like “print variablex” to know the value of a variable, “stop at x” to set a breakpoint, etc. The application issues notification commands like “breakpoint reached”.
The truth is that the protocol is a little more complex than this, but this is enough to know to illustriate the point.

With the previous said, we can see that it would be even possible to debug an application with the use of Telnet! (we'll see later)

Well, enough theory. Let’s see an example Any simple example will do. We’ll make a simple program that takes two parameters from command line and prints the sum. The program won’t be well designed (In the sense that will include some useless temp variables, no validations, etc) but will do to illustrate the example.



class Sum{
    public static void main(String[] args){
        int sum1 = Integer.parseInt(args[0]);
        int sum2 = Integer.parseInt(args[1]);
        int suma= sum1+sum2;
        System.out.println("La suma es "+suma);
    }
}



So we compile it: javac -g Sum.java (the g option adds extra debug info to the class. Like local variable names)
And we run it in debug mode: java -Xdebug -agentlib:jdwp=transport=dt_socket,address=4000,server=y,suspend=y Sum 3 4

Now we have the application listening on port 4000 waiting for connections

We will use the jdb command line debugger that comes with java. But first let’s try this. Run the following (you must run the second line fast after the telnet session starts)

telnet localhost 4000
JDWP-Handshake

That is the handshake to initiate the communication. You now have a debugging session with Telnet !

Ok, that was only to show, you (or i) would have to know the details of the jdwp protocol to use it. Let’s use instead jdb to debug our application. execute the following:

jdb -attach 4000

you’ll get some output like
Initializing jdb ...
>
VM Started: No frames on the current call stack

main[1]

That’s it, you have a debug session started. Now the interesting. Execute the following in your jdb session:

stop at Sum:6 You now have a breakpoint on line 6. execute run on the session, and the program will run until that breakpoint. you’ll get the output

Breakpoint hit: "thread=main", Sum.main(), line=6 bci=18
6 System.out.println("La suma es "+suma);


Now let’s see the value of our variables: run the following commands (one at a time) on the jdb session and see the results.

print sum1
print sum2
print suma
locals
set suma = 10
locals
cont

This is pretty cool stuff. You can debug your program from command line.

Of course if you have the opportunity to use an IDE like Eclipse you should take the advantage of it and still applying what you’ve learnt. So let’s do this.

You need to have the source code of the running application open in your eclipse as a eclipse project for this
Go to the step when you ran the program in debug mode. and run it.

Now go to your eclipse, go to the menu and select RUN -> DEBUG CONFIGURATIONS

In the left panel go to Java Remote Applications, and click new there.

Then select your project, write 4000 in the Port field, and click debug:

That’s it. you have attached your eclipse to the debugging program, now you can put breakpoints, do variable watches, and evaluate expressions from Eclipse.

That’s it. i hope this small article has helped you to understand a little better how the debugging of an application works and how you can debug and application that runs somewhere else.

For more Java Core information you can consult the good official book

Carlo

Intro to Groovy closures for Java developers

People who is starting to work in Groovy usually comes from a Java background. And so they have to deal with the new (sometimes very different) features that the new language provides.

One of the new and different features of Groovy is Closures. Closures are a very powerful feature of Groovy, and one of the must deeply used, so they must definitely be understood to take full advantage of the language.

Sometimes people have trouble understanding closures, because there is no such construct in Java.However they are not really hard to understand, taking into account that Groovy is a 100% Object Oriented Language, and that it compiles to normal Java classes.

I will explain the basics of closures making a comparison between Code using closures, and showing (rather roughly) how this would translate into Java.

First a fast introduction.

A simple definition to Closures, is to say that they are language constructs that encapsulate behaviour, as functions, and can be referenced and passed around the code. Expressing it in Java terms they are like a Method without an associated class (We'll see later that this is not true) that can be referenced and passed around as an object.

The construct of a Closure in Groovy is like this:

{}

Yes, that's it, an open curly brace and the closing curly brace. That is the most simple closure, that actually doesn't do anything.

Closures can take parameters, like this {arg1->}. That closure receives one parameter and doesn't do anything.

A closure can for example return the double of it's argument {arg -> arg*2}

You pass a closure to a method as the last argument of the method without parentheses. For example we can have a method somewhere that receives a closure: def methodFoo(closureArg). And it can be called like
methodFoo{arg ->}

From this point, for the purpose of the explanation, we will use the each method on a Groovy List, which receives a closure as parameter and passes every element of the list to this closure.

Let's suppose we want to print de double of each element of a integer list.

[1,2,3,4].each{element -> print element*2}

That's it we printed the double of each element. To achieve the same in Java, and supposing that there exists an each method on java.util.List and that we already built our list with the four elements, it would be something like:



interface MethodObject{
void execute(Integer element); }

list.each(new MethodObject(){
    public void execute(Integer element){
    System.out.println(element * 2) }
})



As we can see, closures are a really nice way of implementing callbacks at the language level.

In the first case the each method will be something like:



each(closure){
    for(element in this){
    closure.call(element) }
}


the second case will look almost the same:



each(MethodObject callback){
    for(Integer element : this){
        callback.execute(element)
    }
}




Closures in groovy are actually objects of type groovy.lang.Closure. So the constructor def a = {arg ->} actually creates an object of type Closure, which as we can see have the method call.

A common source of confusion for closures is the scope of the variables around the closure. For me the best way to understand how this works is to think that at the time of the closure declaration, al visible variables are passed to the closure. For example, taking the previous example and this time multipliying the element by a number taken for the outer scope, we get the following.

def multiplier = 3
[1,2,3,4].each{element -> print element*multiplier}

We can see that the closure can access the reference multiplier and use it.

As objects, i think that what happens is something like the following(Of course this is not at all the way it is implemented, it's just a way to understand that the closures have access to the local variables in the scope where they there are declared):




Integer multiplier=3;

interface MethodObject{
    void execute(Integer element);
}
class MethodObjectImpl implements MethodObject{
    private Integer multiplier;
    public MethodObjectImpl(Integer multiplier){
        this.multiplier=multiplier;
    }
    public void execute(Integer element){
        System.out.println(element*multiplier)
    }
}

list.each(new MethodObjectImpl(multiplier));



As we can see, the local variable "multiplier" would be passed to the declaration of the closure object, so that it "remembers" the variable when it is executed.

So, i think that the most important thing to remember about closures is that they are a very convenient way of expressing callbacks, that they are objects (even although syntactically they don't look like it), and that they remember the variables on their out local scope.

Using tcpdump and wireshark for debugging

In some of the projects i have worked on i have seen myself in the need of integrating with external systems, in most cases via Web Services, Rest or Soap.

The integration with these external services is usually done with the use of framework help. For example in a couple of projects I used WebServiceTemplate. In other project i had Mule ESB calling to a Web Service Endpoint, etc.

In most cases I pass a pojo to a method and the framework takes care of everything, marshalling, adding headers, sending. And in some cases the debug info that the framework outputs is not enough to see what is going on the wire. I need to see the actual message i’m sending or receiving from the remote host.

For this cases i use tcpdump for capturing traffic, and Wireshark for displaying it. I explain how to do this:

Let’s suppose we are connecting to the Calculator Web Service in

http://soatest.parasoft.com/calculator.wsdl

We want to see the exact contents of our request as it leaves our network interface.

we go to a shell window and execute the following as root

tcpdump -i wlan0 -w /tmp/xxx.dmp -s 1500 dst ws1.parasoft.com

where wlan0 is the network interface in use, /tmp/xxx.dmp the file where we are storing the output, 1500 the capture packet size, and ws1.parasoft.com the destination address for filtering the packets sent (Only packets with this destination will be captured by tcpdump)

We then execute our code that makes the request (In this example i’m doing the request with soapUI, calling the operation add on the web service).

After doing the operation, we terminate the running tcpdump, go to wireshark and open the file /tmp/xxx.dmp. And we see the following:



We can now click on the line with the Protocol HTTP/XML and see the contents of the soap xml message, the http headers, etc:



This way, we have a complete knowledge of what is going on the wire, and maybe this knowledge helps us to debug some problems that can be harder to debug without this knowledge.

Carlo

Hadoop Basics

This summary is not available. Please click here to view the post.

Introduction to Semantic Web (Part 2)

In the first part i talked about the characteristics on which the Semantic Web is built. Now I'll introduce a couple of standards tools that support those characteristics.

The Standards and Tools i’ll cover are:

RDF
Sparql
RDFa


RDF (Resource Description Framewor): Is the De facto standard way of representing graph data for Semantic Werb, in a way that is both understandable by humans and machines. Is a language in which evrything is represented as a Resource (or a literal value), the subject, the predicate and the object. RDF can be represented in many ways. i’ll use the XML representation. An example RDF would be.

<rdf:rdf owl="http://www.w3.org/2002/07/owl#"

cc="http://creativecommons.org/ns#"
dc="http://purl.org/dc/terms/" fb="http://rdf.freebase.com/ns/"
rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xhtml="http://www.w3.org/1999/xhtml/vocab#" about="http://rdf.freebase.com/ns/en.the_other_side_of_midnight">

<fb:m.04l1354>
<dc:description>The Other Side of Midnight is a novel .....</dc:description>
<xhtml:license resource="http://creativecommons.org/licenses/by/3.0/">
<fb:media_common.adapted_work.adaptations resource="http://rdf.freebase.com/ns/m.04jn_x5">

<dc:creator resource="http://rdf.freebase.com/ns/en.sidney_sheldon">
<fb:book.book.genre resource="http://rdf.freebase.com/ns/en.thriller">
......

In this RDF we can see the three parts of every triple. The subject being in our case the rdf:about after the namespace declarations. Objects can be literals or resources, and predicates are resources. So in the previous example one triple would be:

rdf:about="http://rdf.freebase.com/ns/en.the_other_side_of_midnight"
dc:creator
rdf:resource="http://rdf.freebase.com/ns/en.sidney_sheldon"

In this case subject, predicate and object are all resources. As we can see this forms a directed graph.

Namespaces can be mixed as we see, so for example we can create our own custom namespace and mix it with the previous graph definition, extending the knowledge modeled.


SPARQL: Is the language that allow us to query RDF modeled data. From my point of view, it’s a lot like SQL but specifically for querying data in the Triple form. To understand it let’s see an example based on the previous RDF.

PREFIX fb:<”http://rdf.freebase.com/ns/">
PREFIX dc:<”http://purl.org/dc/terms/”>
PREFIX: rdf:<”http://www.w3.org/1999/02/22-rdf-syntax-ns#”>
select ?book
where ?book dc:creator <http://rdf.freebase.com/ns/en.sidney_sheldon>

The previous sparql would return us all the books created by Sidney Sheldon. Notice that all the where clauses would be in the form of Triples. Also binding variables (?book in our case) can be used in more than one triple clause to narrow the results further.


RDFa: Is a set of added constructs that allow us to embbed RDF in XHTML in the form of attributes to normal XHTML tags.For example if we want to expose our knowledge about “The other side of midnight” to the web, allowing potential crawler to see this information we can do something like:

<div xmlns:dc="http://purl.org/dc/elements/1.1/"
about="http://rdf.freebase.com/ns/en.the_other_side_of_midnight">
<span property="dc:description">The Other Side of Midnight is a novel... </span>
<span rel="dc:creator" resource=”http://rdf.freebase.com/ns/en.sidney_sheldon”> Sidney Sheldon</span>
</div>

These are maybe the main standards used in Semantic Web. If you are interested in the subject, you should also take a look at the OWL language and ontologies, and tools like Jena and Sesame for graph data repositories and servers.

Execute an X application remotely.

Today i bought a netbook, and i wanted to do a little programming on a project. I have the project configured on my main computer where i usually develop.

I didn’t want to create a full environment for my project, because of netbook capacity and because i won’t really program from here very often.

So i decided to use the ssh X forwarding facility. This way i could run my main computer eclipse in my netbook.

The Steps followed were very simple:

First on main computer:

- install openssh
- edit file /etc/ssh/sshd_config and make sure there is a non-commented line like X11Forwarding yes

That’s it for the main computer (Where eclipse really resides)

Then in my netbook i did:

- Open A terminal and execute:

ssh -X cscarioni@192.168.0.99 /home/cscarioni/programas/eclipse/eclipse


Where /home/cscarioni/programas/eclipse/eclipse is where eclipse is installed on the main computer.

And that was it!!. i’m working in my home environment from my netbook.

edit: By the way in the home desktop computer xauth must be installed

Simple introduction to oauth protocol at the developement level. (with twitter and foursquare examples)

Oauth is basically a protocol for inter-application exchange of tokens, allowing one application to use resources and APIs of the other application, by authenticating between them with these tokens. One application being the Oauth Server, and the other one being the Oauth client.

The first step in configuring an oauth “contract” between the two applications is in the configuration of the server application, by registering the client application.

The server creates an “ID” for the client application from which it will identify it. This ID is formed by two strings known as the “key” and the “secret” . This ID strings are immediately made available to the client application.

The server must also provide 3 URLs to the client:
Request Token URL
Authorize URL
Access Token URL

It also must register a client callback URL

How it works

Request Token URL: URL where the client will request a first token, to identify a “user” within the client application. When accessing this URL with the id and key of the client application, it will return a token (oauth_token) (a String basically) which will then be used to authorize the client, with that particular token to access the server application. It returns another token (oauth_secret_token) that must be used later, so it has to be stored somehow (usually in the session). The first token will then be used to call the Authorize URL.

Authorize URL: This is the URL that is called from the client to give authorization to a specific account on the server application. This means that by accessing this URL, normally the authentication mechanism of the server application will pop, and after filling it, a dialog will pop requesting the acceptance or denying of the authorizations from the client application. After the permission is granted, the server application will call a callback URL on the client application (defined in the server configuration at the begining of the process) sending it a new token (oauth_token). The final Step will be calling the access token URL and..

Access Token URL: When the client application receives the callback call from the server, it takes the new oauth_token from the request, combine it with the saved oauth_token_secret from the first request and call the access token URL with that. The response is yet another two tokens known as the access tokens. These tokens must be stored persistently somewhere for future use, and usually will be associated with some user within the client application.

Now users within the client application can access their resources on the server application, sending their two tokens in every request for identifying them. And of course sending also the key and secret that identify the application.

Seeing it on the wire

This is the exchange using foursquare as an example of Oauth server

1. First the client request a request token, givinf the key of the application and the encrypted secret

GET /oauth/request_token?oauth_nonce=62603210&oauth_timestamp=1280930229&oauth_consumer_key=NO2ML1KLHEEPWMUQZPZSIOIGWX3LEBOTBYXTSIHECZ02WVL1&oauth_signature_method=HMAC-SHA1&oauth_version=1.0&oauth_signature=PRFQvAJngqdrrj3RIM%

2. The server answers OK and returns the tokens. Also returned Sessions cookies XSESSIONID and more (omited).

Expert Info (Chat/Sequence): HTTP/1.1 200 OK\r\n
oauth_token=2BTJEXPZHWRY2CJD5YVHOQUEMD0WVPNKASLP2WDCLMGYN2ZB&oauth_token_secret=GBMKJYOK0C4EVWMISZHQQPC3POPJADECRNABKT1SX2ZNB0DO


3. The client sends the authorize request with the oauth_token:

/oauth/authorize?oauth_token=2BTJEXPZHWRY2CJD5YVHOQUEMD0WVPNKASLP2WDCLMGYN2ZB


4. The server redirects the client to the server’s authentication page, still referencing the oauth_token.

Request URI: /login?continue=%2Foauth%2Fauthorize%3Foauth_token%3D2BTJEXPZHWRY2CJD5YVHOQUEMD0WVPNKASLP2WDCLMGYN2ZB

5. After the user login into the server application it is redirected to the allow/deny page.

http://foursquare.com/oauth/authorize?oauth_token=2BTJEXPZHWRY2CJD5YVHOQUEMD0WVPNKASLP2WDCLMGYN2ZB

6. When the user allows the permission, the server calls the callback URL on the client sending it the new access token.

http://tripsqr.appspot.com/foursquare/callback?oauth_token=2BTJEXPZHWRY2CJD5YVHOQUEMD0WVPNKASLP2WDCLMGYN2ZB

7. With this new token, and with the secret token from step 2. A call is made to the Access Url on the server.

[truncated]
oauth/access_token
oauth_nonce=83129211&oauth_timestamp=1280932954&oauth_consumer_key=NO2ML1KLHEEPWMUQZPZSIOIGWX3LEBOTBYXTSIHECZ02WVL1&oauth_signature_method=HMAC-SHA1&oauth_version=1.0&oauth_token=URC5OUC55LZPGLMQPBRVKXGDECIH3HVGG3JWJKMPT1H4IUNM

8. The server returns the last two tokens that must be saved in the client for future use.

Te following is an example in python of Oauth against foursquare an twitter

First a couple of classes with the details

__author__ = 'cscarioni' import oauth2 as oauth import cgi as urlparse from django.utils import simplejson from StringIO import StringIO class BaseOauth(object): _oauthKey=None _oauthSecret=None _requestURL=None _accessURL=None _authorizeURL=None def __init__(self,oauthKey,oauthSecret,requestURL,accessURL,authorizeURL): self.oauthKey=oauthKey self.oauthSecret=oauthSecret self.requestURL=requestURL self.accessURL=accessURL self.authorizeURL=authorizeURL def requestAuthorization(self,dictionary): consumer = oauth.Consumer(self.oauthKey, self.oauthSecret) client = oauth.Client(consumer) resp, content = client.request(self.requestURL, "GET") if resp['status'] != '200': raise Exception("Invalid response %s." % resp['status']) request_token = urlparse.parse_qs(content) dictionary["secret"]=request_token["oauth_token_secret"][0] return "%s?oauth_token=%s" % (self.authorizeURL, request_token['oauth_token'][0]) def getAccessTokens(self,oauthtoken,dictionary): consumer = oauth.Consumer(self.oauthKey, self.oauthSecret) token=oauth.Token(oauthtoken,dictionary["secret"]) client = oauth.Client(consumer,token) resp, content = client.request(self.accessURL, "POST") if resp['status'] != '200': raise Exception("Invalid response %s." % resp['status']) access_token = urlparse.parse_qs(content) return access_token["oauth_token"][0],access_token["oauth_token_secret"][0] def requestWithOauthJSON(url, key, secret, http_method="GET", post_body=None,http_headers=None): consumer = oauth.Consumer(self.oauthKey, self.oauthSecret) token = oauth.Token(key, secret) client = oauth.Client(consumer, token) resp, content = client.request( url, method=http_method, body=post_body, headers=http_headers ) return simplejson.load(StringIO(content)) class FoursquareOauth(BaseOauth): def __init__(self): super(FoursquareOauth,self).__init__( "xxxxxxxxxx", "xxxxxxxxx", "http://foursquare.com/oauth/request_token", "http://foursquare.com/oauth/access_token", "http://foursquare.com/oauth/authorize" ) class TwitterOauth(BaseOauth): def __init__(self): super(TwitterOauth,self).__init__( "xxxxxxxx", "xxxxxxxxxx", "https://api.twitter.com/oauth/request_token", "https://api.twitter.com/oauth/access_token", "https://api.twitter.com/oauth/authorize" )

which can be used from client code like:

def associateFoursquareAccount(request): return HttpResponseRedirect(foursquareOauth.requestAuthorization(request.session)) def foursquareCallback(request): token,secretToken=foursquareOauth.getAccessTokens(request["oauth_token"],request.session) user=User.gql("WHERE login = :login",login=request.session["userlogin"]) user[0].foursquareToken=token user[0].foursquareTokenSecret=secretToken user[0].put() return HttpResponseRedirect("/")