Hi,
I'm trying to set up a personal wiki and I'm having trouble getting search to work. All my searches give zero results. I eventually figured out that the Xapian database that's being created has an empty postlist table. The position and termlist tables are all fine, and when I add new content to the wiki I can see the database is updated and the search terms are in the position table in plaintext. But I can't query for them, even using Xapian's command line tools.
mexon:~/Test/.ikiwiki/xapian/default$ ls -l total 76 -rw-rw-r-- 1 mexon mexon 0 Dec 16 15:56 flintlock -rw-rw-r-- 1 mexon mexon 28 Dec 16 15:55 iamchert -rw-rw-r-- 1 mexon mexon 13 Dec 16 15:55 position.baseA -rw-rw-r-- 1 mexon mexon 49152 Dec 16 15:55 position.DB -rw-rw-r-- 1 mexon mexon 13 Dec 16 15:55 postlist.baseA -rw-rw-r-- 1 mexon mexon 0 Dec 16 15:55 postlist.DB -rw-rw-r-- 1 mexon mexon 13 Dec 16 15:55 record.baseA -rw-rw-r-- 1 mexon mexon 0 Dec 16 15:55 record.DB -rw-rw-r-- 1 mexon mexon 13 Dec 16 15:55 termlist.baseA -rw-rw-r-- 1 mexon mexon 16384 Dec 16 15:55 termlist.DB mexon:~/Test/.ikiwiki/xapian/default$ delve -a . All terms in database: mexon:~/Test/.ikiwiki/xapian/default$
I don't know how to debug from here. Clearly ikiwiki is doing something right when it's building the database, but one of the tables is missing. Can anyone guess what's wrong, or tell me where to start troubleshooting?
I'm using Centos 5. Xapian is version 1.2.5. Ikiwiki version 3.20111107.
I'm using RPMs to install Xapian packages, xapian-omega and xapian-bindings-perl, and they're all 1.2.5. I originally tried building and installing Xapian 1.2.7 from source, but found that ikiwiki failed like this:
Much fruitless googling later I found that there were these 1.2.5 RPMs lying around so I switched to those. If you know a solution to the DB_CREATE_O problem I could give 1.2.7 another go.
I had another go, this time with Xapian 1.2.8.0, and I finally got it working. The errors I was seeing earlier were because Xapian installs itself in /usr/local, but the default CentOS environment doesn't have /usr/local/lib in the LD_LIBRARY_PATH. As usual, the problem and solution are very duh, it's the error messages that make everything hard. It's a lot more clear if you do a "make test" while building Search::Xapian:
So yeah. Worth noting that Xapian 1.2.5 is apparently broken with ikiwiki. Maybe some kind of warning?
As a Xapian database is updated, changes are written out straight away to most of the tables, but the postlist changes are batched up in memory and only written to disk just before they are committed (or to free up memory during a large transaction). So the empty postlist table you're seeing means that some documents were indexed, but the indexer stopped running before anything was committed. By default, there's an auto-commit every 10000 documents added, removed, or changed, so it presumably managed to process less than 10000 documents.
The issue with 1.2.5 RPMs may be down to there being two versions of the Xapian perl bindings, both of which claim to be Search::Xapian and both of which have been packaged up as RPMs. For Xapian 1.2.x, you probably want to use the XS bindings (perl-Search-Xapian) not those generated with SWIG (xapian-bindings-perl). The SWIG-generated ones are aimed to replace the XS ones, but in 1.2.x they're not really ready for prime time. If your perl bindings are described with a 4 coponent version (e.g. 1.2.15.0) then you should be good; if it's 3 components (e.g. 1.2.15) then you probably want the other ones.