Basic searching in Ruby with Solr

Solr is a server application built on top of the Apache Lucene searching engine. It offers a Http interface for storing and querying data.

Internally the way Solr roughly works (and Lucene as it is the engine that powers solr) is by indexing Documents for later searching and retrieval. A Document is described with a collection of Fields, each of this fields can be individually indexed and/or stored on the index.

The index can be built in different ways. The way the index is built is mainly determined by the analyzers used in each field. So an analyzer simply determines the way a particular field will be indexed.

Of course there is a lot of complexity involved in all this, but this is a basic tutorial, and a basic but functional searching solution can be build using defaults for most options.

This tutorial will allow for a search of movies by title and/or Actor using Ruby and Solr. I will assume you already have Ruby installed and the Gem tool as well.

1. Download and install Solr:
 wget http://apache.mirrors.timporter.net/lucene/solr/3.5.0/apache-solr-3.5.0-src.tgz

2. Decompress it:
 tar zxvf apache-solr-3.5.0-src.tgz

3. Modify the index to accept the kind of documents we want (movies).
 
In our example we will be able to query movies by title and actors. The index will also store a summary of the movie although it won’t be searchable by that. So we will have three Fields in our Document representing the movie. To reflect this go to the directory:

cd apache-solr-3.5.0/solr/example/solr/conf/

then open the file schema.xml with your favorite editor, go down to the definitions and replace all the ones that are there with the following ones:

you replace the section with the following

1. <fields>
2.   <field name="id" type="string" indexed="true" stored="true" required="true" />
3.   <field name="title" type="text_general" indexed="true" stored="true"/>
4.   <field name="actor" type="text_general" indexed="true" stored="true" multiValued="true"/>
5.  <field name="summary" type="text_general" indexed="false" stored="true"/>  
6. </fields>

Here we are specifying that our movie Documents will have these four fields for searching purposes. We can see that the type we are using for all of them is "text_general". Going up in the schema.xml file we can find a description of what being "text_general" means,
This is extracted directly from that description:




So this is a default provided analyzer that wil be good enough for our purposes (and for many purposes).

The other two thing worth mentioning in our field definitions, is the fact that the "actor" field is multivalued, meaning that we can associate more than one actor to the field, and the fact that the "summary" is stored but not indexed. This means that the content of the field will be stored (so it can be retrieved when documents are retrieved) but it is not indexed (we can't search on this field).

Ok, so this is all the configuration we need in Solr. let's start the server now.
 From the directory apache-solr-3.5.0/example. Execute: java -jar start.jar.

That will start the server and will listen in the port 8983 by default.

Ok, so let's move to Ruby side now. We will create a little program that will index a couple of movies, and then search to find them. First require the needed gem:

gem install rsolr
Then let's create a Movie class in a file named "moviesearch.rb":

1. class Movie
2.  attr_accessor :id, :title, :actors, :summary
3.  def initialize
4.     @actors = []
5.  end
6. end

And now let’s create the indexer and searcher classes in the same file:

Indexer:

require 'rsolr'

1. class Indexer
2.  def initialize
3.     @solr = RSolr.connect :url => 'http://localhost:8983/solr/collection1/'
4.  end
6.  def index(movies)
7.     movies.each do |movie|
8.      @solr.add :id=>movie.id.to_s, :title=>movie.title, :actor => movie.actors
9.     end
10.     @solr.update :data => '<commit/>'
11.  end
12. end

Searcher:

1. class Searcher
2.  def initialize
3.     @solr = RSolr.connect :url => 'http://localhost:8983/solr/collection1/'
4.  end
6.  def search(term)
7.     term = term.downcase
8.     response = @solr.get 'select', :params => {:q => "title:#{term}* or actor:#{term}*"}
9.     list = response["response"]["docs"]
10.     list
11.  end
12. end

That’s it.

Let’s test it on irb:




1.9.2-p290 :001 > require './moviesearcher'
=> true
1.9.2-p290 :013 >   movie_1 = Movie.new
=> #
1.9.2-p290 :014 > mo
module   movie_1
1.9.2-p290 :014 > movie_1.actors << 'Bruce Willis'
=> ["Bruce Willis"]
1.9.2-p290 :015 > movie_1.actors << "Samuel Jackson"
=> ["Bruce Willis", "Samuel Jackson"]
1.9.2-p290 :016 > movie_1.id = '1'
=> "1"
1.9.2-p290 :017 > movie_1.title='Die Hard 3'
=> "Die Hard 3"
1.9.2-p290 :018 > movie_2 = Movie.new
=> #
1.9.2-p290 :019 > movie_2.actors << 'Mel Gibson'
=> ["Mel Gibson"]
1.9.2-p290 :020 > movie_2.actors << 'Danny Glover'
=> ["Mel Gibson", "Danny Glover"]
1.9.2-p290 :021 > movie_2.id = '2'
=> "2"
1.9.2-p290 :022 > movie_2.title = 'Lethal Weapon'
=> "Lethal Weapon"
1.9.2-p290 :041 >   movie_1.summary = "Great movie"
=> "Great movie"
1.9.2-p290 :042 > movie_2.summary = 'Another great movie'
=> "Another great movie"

Indexing

1.9.2-p290 :061 > idxr=Indexer.new
1.9.2-p290 :080 >   idxr.index [movie_1,movie_2]
=> {"responseHeader"=>{"status"=>0, "QTime"=>50}}

Searching

1.9.2-p290 :085 >   searcher = Searcher.new
1.9.2-p290 :086 > searcher.search 'Die'
=> [{"id"=>"1", "title"=>"Die Hard 3", "actor"=>["Bruce Willis", "Samuel Jackson"]}]
1.9.2-p290 :090 >   searcher.search 'Bru'
=> [{"id"=>"1", "title"=>"Die Hard 3", "actor"=>["Bruce Willis", "Samuel Jackson"]}]

1.9.2-p290 :091 > searcher.search 'Glo'
=> [{"id"=>"2", "title"=>"Lethal Weapon", "actor"=>["Mel Gibson", "Danny Glover"]}]

Private Keys, Public Keys and Certificates

This is a quick tutorial that will cover

- Generate a private key

- Generate a .cert certificate with that private key

- Extract the public key from the certificate.

- Sign a file with private key and verify the signature with the public key

- Import the private key and certificate into a java keystore.


1. Generate a private key


openssl genrsa -out private.key 1024


2. Generate certificate


openssl req -new -x509 -days 365 -key private.key -out certificate.crt


That certificate is a good self signed certificate that is ready to distribute around for testing.


3. Extract public key from certificate


openssl x509 -in certificate.crt -pubkey > public.key


That will copy the certificate and the public key to the file... you need to edit the file and remove the part related to certificate and leave just the public key in the file.


4. We sign a file with private key.

openssl dgst -sha1 -sign private.key -out file_to_sign.sha1 file_to_sign


5. We verify the signature with the public key:


openssl dgst -sha1 -verify public.key -signature file_to_sign.sha1 file_to_sign


6. we import private key and certifcate to a java keystore


first we generate a p12 file


openssl pkcs12 -export -in certificate.crt -inkey private.key > server.p12


then we import this into the keystore


keytool -importkeystore -srckeystore server.p12 -destkeystore keystore.jks -srcstoretype pkcs12

Running a Ruby map-reduce job with Hadoop

I am currently developing and app in my spare time and needed to merge a file with itself to include movies that are both Romance and Comedy.

The file looked somthing like this:

movie-a Comedy
movie-b Comedy
movie-a Romance

I wanted to produce a file of the form

movie-a [Comedy,Romance]

Ignoring the movies that don't include both genres.

I inmediately thought of using hadoop, even if the file was not huge, the map reduce algorithm seems a good fit for the problem.

I have done some small work in hadoop with Java, but in this case my project was Ruby based and I wanted to keep on using Ruby even for my hadoop job, so I used the streaming API of hadoop to solve the problem.
I needed to develop a fast and easy solution. What follows is the code:

Consulting some bibliography and the Web I came to a very easy solution.

map.rb

1. ARGF.each do |line|
2.    begin
3.      parts = line.split("\t")
4.      puts parts[0]+"\t"+ parts[parts.size-1]
5.    rescue
6.      puts 'error'
7.    end
8. end

reduce.rb

1. current_key = nil
2. current_key_values=[]
3. ARGF.each do |line|
4.    line = line.chomp
5.    (key, value) = line.split(/\t/)
6.    if current_key.nil?
7.      current_key=key
8.    end
9.    if current_key==key
10.      current_key_values<<value
11.    else
12.      if current_key_values.include?("Comedy") and current_key_values.include?("Romance")
13.        puts current_key + "\t" + current_key_values.to_s
14.      end
15.      current_key=key
16.      current_key_values=[value]
17.    end
18. end

hadoop_run.sh

1. #!/bin/bash
2.  
3. HADOOP_HOME=/home/cscarioni/programs/hadoop-0.22.0
4. JAR=contrib/streaming/hadoop-0.22.0-streaming.jar
5.  
6. STREAMCOMMAND="$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/$JAR"
7.  
8. $STREAMCOMMAND \
9.  -mapper 'ruby map.rb' \
10.  -reducer 'ruby reduce.rb' \
11.  -file map.rb \
12.  -file reduce.rb \
13.  -input '/home/cscarioni/Downloads/comedy_romance_movies' \
14.  -output /home/cscarioni/Downloads/comedy_romance_movies_results

I consider the main differences between the streaming API and the Java API are:

1. The streaming API works everything in the stdin and stdout between the scripts, (take a look at the ARGF and puts use in both map.rb and reduce.rb) like when we use the pipe in the command line between commands

2. In the Java API the results from the mapper phase are grouped together for example in our case we would actually receive directly on the reduce phase the line (movie-a [Romance,Comedy]). In the streaming API on the other hand, the grouping needs to be done manually, what we get is an ordered list by key (so all the movie-a would be next to each other).

So there we have a small and functional map reduce job in Ruby with Hadoop.

Basic movement game AI

I really love to play video games, and for a while I’ve been wanting to learn a bit about video game programming mainly AI and physics.
I have just read the first 20 or so pages of the good book on AI Artificial Intelligence for Games

and I though I was going to try the first easy algorithms that are explained there. So here I will introduce a little bit of the simplest way to develop the seek and arrive algorithm.
I will develop the example in Ruby using the gosu library to draw some simple characters.

The idea of the seek algorithm we are going to implement is very easy, we will control a character in the screen and another computer controlled character will seek our character on the screen and destroy him when he arrives to our character.

So the first thing to know is what is involved in a very simple movement. In the example we will not consider acceleration forces so we will have what is known as Kinematic movement. We will need 3 variables to express the characters static data at moment in time. So we can start with a Ruby module like this:

1. module Kinematic
2.  attr_reader :velocity, :orientation, :position
3. end
4.  

where all three elements are vectors. The velocity vector will give us the direction and speed of the characters, the orientation vector, in our case will simply be oriented towards the direction, so it will be the velocity vector normalized, and the position is the place where our characters are.

The orientation will also be expressed as a angle in radians using the atan2(-x,y) formula, where y and x are the corresponding coordinates in the velocity vector.

So the position and orientation will be both a function of the velocity like this:

1. orientation = velocity.normalize
2. position = velocity * time

where time is a small unit of time that will be a function of the frame rate we have. When the frame rate is bigger, the update time is smaller. It is calculated in our case something like this:

Let’s suppose our character travels at 2 meters per second, and let the frame rate be 60fps in a particular moment. then our time multiplier will be 1/60 which multiplied by 2 will be 1/30 that will be the length of our vector to be summed to the position vector in each frame. However we will change the value and adjusted to some value that makes the movement look good.

Ok so that is the basic movement, but now we need to implement the seek behaviour. The AI character will need to chase our own character in the screen, in the algorithm terminology we’ll be the target of the seek and arrive algorithm. So logically for implementing this algorithm we need both the character and the target kinematic data. We also need to specify a radius of contact (where the character catches the target) and a speed for our velocity vector.

So to our module we add this max_speed

1. module Kinematic
2.  attr_reader :velocity, :orientation, :position, :max_speed
3. end

We create now two classes that include this module

1. class Target
2.  include Kinematic
3. end
4.  
5. class Character
6.  include Kinematic
7. end

Both character and target include the module, but only the character will be AI controlled, the target will be controlled by ourselves.

So we will create the seek_and_arrive algorithm on the character, in a method that receives the character it is chasing.

First the seek part will be simply to create a velocity of speed ‘max_speed’ and direction pointing to the target’s position. Now for the arrive part we will use a ’radius of impact’ that determines when the character has actually reached the target. We will include this radius as information on the target character. So modifying the algorithm we now have:

1. def seek_and_arrive(target)
2.   @position += @velocity * @time_update
3.   @velocity =  target.position - position
4.   if  @velocity.magnitude < target.radius
5.  EventHandler::add_event(:capture,self,target)
6. end
7. @velocity = @velocity.normalize
8. @velocity *= @max_speed
9. @orientation = @velocity.normalize
10. end

As we see we are simply adding a condition and then sending an event saying that the target has been captured by the character. This event will be handled in the main loop of the game where it will show Game Over.

We will now create the graphics for the game with Gosu, I won’t explain much here as it is not the focus of the post.

The first thing, we create a character wrapper for our characters that will know about gosu, that way our original class remains graphics framework independent:

1. class DrawableCharacter
2.  attr_reader :character
3.  def initialize(character,window,character_img)
4.     @image = Gosu::Image.new(window, character_img, false)
5.     @character = character
6.  end
7.  
8.  def draw
9.     @image.draw_rot(@character.position[0], @character.position[1], 1, @character.orientation_in_radians)
10.  end
11. end

then we create a class for the controlled character and one for the AI Character:

1. class ControllableCharacter < DrawableCharacter
2.  def move(side)
3.     @character.move_ahead if side==:front
4.     change_velocity_according_to_side(side)
5.  end
6.  
7.  def change_velocity_according_to_side(side)
8.     return if side == :front
9.     if side == :right
10.      sin_radians = Math::sin 0.1
11.      cos_radians = Math::cos 0.1
12.     else
13.     sin_radians = Math::sin -0.1
14.     cos_radians = Math::cos -0.1
15.     end
16.     velocity_x = @character.velocity[0]*cos_radians - @character.velocity[1]*sin_radians
17.     velocity_y = @character.velocity[0]*sin_radians + @character.velocity[1]*cos_radians
18.     @character.velocity = Vector[velocity_x,velocity_y]
19.     @character.velocity = @character.velocity.normalize * (@character.max_speed+1)
20.  end
21. end
22.  
23. class AICharacter < DrawableCharacter
24.  def seek_and_arrive(target)
25.     @character.seek_and_arrive(target)
26.  end
27. end

The Controllable character will move depending on input from the keyboard that is captured on the main Game class. The AICharacter delagates its movement to the Character class that contains the seek_and_arrive algorithm.

Now the main Game class:

1. class Game < Gosu::Window
2.  
3.  def initialize
4.     super 1024, 768, false
5.     self.caption = "Seek and Arrive"
6.     @target = ControllableCharacter.new(Target.new(10, 10), self, 'target.gif')
7.     @character1 = AICharacter.new(Character.new(500, 500), self, 'character.gif')
8.     @game_state = :game_started
9.  end
10.  
11.  def manage_ai_characters
12.     @character1.seek_and_arrive(@target.character)
13.  end
14.  
15.  def manage_controllable_character
16.     if button_down? Gosu::KbLeft or button_down? Gosu::GpLeft then
17.      @target.move :left
18.     end
19.     if button_down? Gosu::KbRight or button_down? Gosu::GpRight then
20.      @target.move :right
21.     end
22.     if button_down? Gosu::KbUp or button_down? Gosu::GpButton0 then
23.      @target.move :front
24.     end
25.  end
26.  
27.  def manage_events
28.     EventHandler::each do |event|
29.      if event[0]==:capture
30.        @game_state = :game_over
31.      end
32.     end
33.  end
34.  
35.  def update
36.     manage_events
37.     if @game_state != :game_over
38.      manage_ai_characters()
39.      manage_controllable_character()
40.     end
41.  end
42.  
43.  def draw
44.     if @game_state != :game_over
45.      @target.draw
46.      @character1.draw
47.     else
48.      Gosu::Image.new(self, "game_over.gif", true).draw(0, 0, 0);
49.     end
50.  end
51. end
52.  
53. window = Game.new
54. window.show

The main details to get out from this code are the ‘update’ and ‘draw’ methods. The update method is called 60 times per second by default, and then the show method is called.

The full source code of the example is in github, just download and run the game.rb ruby file.

Of course this introduction is the simplest of the simplest in Game AI, but it is important information to have and very entertaining to learn.

Also of course there are libraries and frameworks that do most of the work for us, but I did this example (and hopefully some following ones) to learn the basics of how it works.