De vanturat

Weekend-ul asta am trecut pe la ITFest. Ce e ITFest? Asta a fost si intrebarea mea si n-as fi ajuns pe-acolo daca nu ma batea Silviu la cap sa mergem. Poate baietii (si fetele) de la OSUT puteau sa faca ceva mai mult tam-tam - au fost ceva cam putini oameni pentru un “festival”. Oricum, din lipsa de timp - care este ocupat cu examene restante - eu am fost decat la doua prezentari.

Vineri am ajuns la o prezentare a lui Gabriel Istrate despre simulare sociala multiagent. Domeniul e foarte interesant si omul probabil ca se pricepe dar prezentarea mi s-a parut putin prea generala. Ne-a povestit despre ce e vorba si ne-a aratat un program folosit in simulari de felul asta. A evitat in schimb sa intre in detalii despre tehnologia din spatele intregii jucarii iar eu tocmai asta vroiam - the “juicy” stuff.

Sambata am fost la prezentarea lui Filip despre proiectele la care lucreaza acum: jobber, jobberbase si jobjob. Dupa cum se aude toate se invart in jurul cuvantului “job”. Jobber e un site gratuit de joburi in IT pentru Romania. Se deosebeste de restul site-urilor de pe piata prin feeling-ul web2.0-ish, adica interfata e curata, usor de folosit iar continutul e la obiect. Pentru cei care cauta sa angajeze, am mentionat ca e gratuit?
jobberbase e o platforma “extrasa” din jobber si care poate fi folosita pentru construirea altor site-uri de nisa.
jobjob e “pe vine” si sper ca, la fel ca si urmeaza, va bucura cat mai multi utilizatori - chiar daca prin asta va fi nevoit sa supere cativa “grei”.

Have you ever tried making zentest autotest use spec_server?

The first guess is to just add “–drb” in spec.opts. Well, guess again because, even if it should, it doesn’t work. The problem is not in the zen garden but in rspec’s.

The “spec” script used for running the tests accepts a bunch of options - including –drb - and also a special option that accepts options from a separate options file. Here is the problem: All the options put in the separate options file are recognized by the spec script with the exception of –drb. It simply doesn’t recognize it if put in the separate options file.

I’ve tried a bit to find the bug in the rspec sources but couldn’t figure it out so I turned to to zentest and made it put –drb directly into the command, not in a separate file. Here’s the little patch script:

class Autotest::Rspec
def spec_command
“script/spec –drb”
end
end

It’s a dirty little hack but it keeps things going until the guys at rspec fix the problem.

I’ve recently stumbled upon  this really cool tool for keeping code DRY. It’s called Simian and it’s from some consulting folks in Australia - they are probably using it to bully their customers on their poor code quality.

In essence, it’s a simple command line similarity analysis tool that shows you the places there are duplications in code in the bulk of files you ask it to go over. Here’s a snippet of the output it gave on one of our projects:

Found 13 duplicate lines in the following files:
Between lines 12 and 24 in C:\ordine_noua\proiecte\Urmeaza\site\app\models\blog_post.rb
Between lines 316 and 328 in C:\ordine_noua\proiecte\Urmeaza\site\app\models\event.rb
Between lines 289 and 301 in C:\ordine_noua\proiecte\Urmeaza\site\app\models\event.rb
Found 19 duplicate lines in the following files:
Between lines 10 and 31 in C:\ordine_noua\proiecte\Urmeaza\site\app\models\blog_post.rb
Between lines 314 and 335 in C:\ordine_noua\proiecte\Urmeaza\site\app\models\event.rb

Ups… not in one of our projects, in a fictional non-existent project that has nothing to do with our unbelievably high code quality.
You can get the tool here. A downside: it’s not free for commercial projects - but might be worth it.

Web scraping “describes any of various means to extract content from a website over HTTP for the purpose of transforming that content into another format suitable for use in another context” (Wikipedia).

This is exactly what I needed for a revived project soon to be revealed so I “crawled” the net a bit for solutions available under Ruby. The following is an overview of alternatives I’ve found - cleaned and published as a short essay for “Web programming” course at university. I’ll start from the simplest to the more complex.

Regex is the first thing that comes to mind when you want to extract data from text and it can also work well for HTML.

Advantages:

• doesn’t depend on ruby as it’s supported in any decent programming environment
• can eat any html, not maddening how garbage it is

Desadvantages:

• you can’t take advantage of the semantics of html structure
• ungly syntax, a week after you write it nobody (including you) will be able to figure out what does it mean

My opinion is that using regex as the single scraping tool is too hard-core but on the other hand it is the best tool for polishing up results from the tools below.

The following approaches profit from the fact html is (or should be) actually a “tree” - or if you want to be particular or idealistic, “xml”. Idealistic because HTML is too rarely XML-clean. There is a solution tho:

HTree is a nice little (i think) tool that is very good at chewing HTML and spitting up valid XML. Considering this this tool is a must but you don’t have to wary about it too much as it’s been already integrated in many of the solutions below.

REXML is the standard XML processing library implemented in ruby and can eat up very well the tidy HTML from HTree and let you play with it like it would be formed by ruby objects. It also has Xpath support, this making it one of the most powerful solutions. So, too keep with the advantages/disadvantages scheme:

Advantages:

• play with html subtrees like with ruby objects, nice way of coding
• full XPath support!
• implemented in ruby so you can tweak it (if you’re into that stuff)
• mature, stable tool

Disadvantages:

• haven’t noticed first-hand but I understand it can be slow on large datasets
• requires Xpath knowledge to work well with it

XMLParser is an alternative to HTree and REXML that uses the expat for parsing xml. The fact that expat is a c library should make it much faster but I cannot tell you very much about it as I haven’t used it. It also requires expat installation - that I think also requires compilation - on the server so if you don’t have good access to your deployment server it is not an option.

Hpricot is a nice HTML parsing tool that makes HTree, JQuery and Ragel work together in a simple and fun to use way.

Advantages:

• nice to use - ruby-ish and everything
• pretty fast
• it supports queries using CSS selectors
• it has some XPath support

Disadvantages:

• it has only some Xpath support (no predicate functions, for example)

Hpricot is the tool I’ve chosen for the “still in stealth mode” project mentioned above and I like it so far.

Rubyful Soup is another HTML parser that looks like a good alternative for somebody not knowing or not wanting to learn xpath. I haven’t used it first-hand but the code looks nice and it brags it won’t “choke if you give it bad markup”. So:
Advantages:

• produce nice-looking code
• accepts bad markup
• ruby is all you need to know - and HTML, of course

Disadvantages:

• no Xpath support

scrAPI that uses CSS selectors as the way of querying the HTML document. It’s suggested way of coding is rather different from the parsers above but it is intuitive and might prove easy to use.

Advantages:

• CSS selectors
• accepts invalid HTML - it actually uses Tidy to clean it up

Disadvantages:

• no XPath support

The above are the main HTML parsers I’ve found. But the purpose is not only parsing HTML, it is actually extracting information from a whole site. In order to do this we also need a way of navigating the site to gather the pages required for parsing. The following tools facilitate this.

WWW::Mechanize does this by providing a layer over the ruby http module that also integrates by default with hpricot (more recently, before was the standard REXML). It emulates the way a browser would crawl pages, including storing cookies, following redirects, etc. It’s a nice tool, it’s only drawback being no support for js interaction with the site in case you need it. This is the tool we are currently using.
scRUBYt is a tool that uses WWW::Mechanize, HPricot and some probably complicated code in order to build web scrapers from an example you give it in a page. The ideea is nice but we (me, Silviu and Cristi) have spent over a day trying to make it work due to it’s crazy complicated dependencies only to finally figure out that it only works at it’s full strength on 32bit linux systems. On other system it either doesn’t work or it don’t support the some feature like the “example to code” module. Anyway, the examples and documentation suggest that it handles greatly 80% of cases but I couldn’t figure out from the documentation how to make complex crawling scenarios (not just nested) so I assume for now they can’t be done. The dependency problems and lack of flexibility in 20% of the cases are the reasons we’ve dropped it.

I’ll move on as criticizing scRUBYit should have been a different post.

Watir is, as they describe it, an “web application testing in ruby” tool so it might not sound like a tool for crawling but it can very well be used for that. It provides the same stuff as WWW:Mechanize with a similar looking code and it can also play with JavaScript. The downsite is that it uses IE’s engine so it only works on Windows for now.

The above solutions are all available for standard ruby. If JRuby is used than this opens access to a whole bunch of Java tools that serve the same purpose. IronRuby will do the same thing for .net soon.

Lastly, the above overview is based partly on my experience with the tools and partly from just reading about them from different sources. Please feel free to correct me if I’ve got something wrong and let me know if there are any other tools that would fit here.

Vreau si eu - sa intru si eu in legalitate - doar ca va trebui sa-mi aleg o varianta dintre banerele alea putin prea agresive. Probabil ca il voi alege pe cel simplu: “TPark - parcare platita cu telefonul mobil”

Pe de alta parte variantele cu exprimarea cea mai … (incerc sa gasesc cuvantul potrivit dar imi vin in minte numai unele cenzurabile): “Ma parchez cu telefonul in timisoara” mi-au dat o idee de reclama tv care ar fi ieftina de realizat si oarecum de efect:

Un nene’ se plimba pe jos, pe mijlocul strazi, pe langa o parcare vorbind la telefon. Putin in fata si in lateral e un loc liber de parcare iar din fata lui vine o masina. Alearga catre masina care vine, vorbind in continuare la telefon, pana se opresete la cativa centimetri fata in fata cu ea. Multumit, merge cu spatele, ca si ar fi cu masina si ar parca. Se opreste. Inchide telefonul. Mai da un beep. Si pleaca. Apoi apare pe ecran un mesaj din asta gen “in timisoara, ma parchez cu telefonul”.