Ruby FFI example using #ffi_lib

Ruby FFI is a cross-VM library for calling foreign functions (i.e. C or C++). ┬áIt isn’t obvious from the introductory blog posts how you specify which library to use, but the answer isn’t hard to find in the source.
Examples speak louder than words:
require 'rubygems'
require 'ffi'

class MDB
  extend FFI::Library
  
  # The lib name gets spackled with platform-specific 
  # prefix and suffix. On Mac OS X, e.g., the ffi_lib
  # name turns into 'libmdb.dylib'
  ffi_lib 'mdb'
  
  # Who needs enum, anyway?
  NOFLAGS = 0
  MDB_TABLE = 1
  
  attach_function :mdb_init, [], :void
  attach_function :mdb_exit, [], :void
  
  # In the libmdb headers, you'll find that this function
  # actually returns a pointer to an MDBHandle struct.  
  # FFI::Struct would likely help out here, but just
  # calling the return result a :pointer works for now.
  attach_function :mdb_open, [ :string, :int], :pointer
  attach_function :mdb_close, [ :pointer], :void 
    
  def self.open(path)
    MDB.mdb_init
    db = MDB.mdb_open( path, MDB::NOFLAGS)
    
    yield db
    
    MDB.mdb_close(db)
    MDB.mdb_exit
  end
  
  attach_function :mdb_dump_catalog, [:pointer, :int], :pointer
    
end

MDB.open('mdb_files/sample.mdb') do |db|
  MDB.mdb_dump_catalog(db, MDB::MDB_TABLE)
end

Use Array#pack and String#unpack instead of Base64

Array#pack(“m*”) is your friend. So is String#unpack(“m*”). You can use them instead of the Base64 methods encode64 and decode64.

Hat tip to Rack’s authentication example.

Ruby snippet for flexible blurbing of text

Here’s a cute little method I use for extracting blurbs from text. Specify a minimum character count as the argument, and the method returns a string containing the minimum text plus whatever it takes to reach the next period, question mark, or exclamation point.

def blurb(size=75)
  reg = Regexp.new ".{#{size}}[,;:-_\\w\\s]*[\.\!\?]"
  body.slice(reg)
end

Developer dependencies in RubyGems 1.2.0 using add_development_dependency

As the release note says, RubyGems 1.2.0 now supports two levels of gem dependencies, runtime and developer. If you don’t use add_development_dependency to specify a dependency, RubyGems treats it as a runtime dep.

Google, in its questionable wisdom, tells me that there’s not much interest in this feature. Echoe is already using it, which is simultaneously unsurprising and reassuring.

Perhaps I shouldn’t find the lack of widespread acclaim strange. I suppose Ruby developers don’t hack other people’s code, they just use it. GitHub will change this, perhaps already is changing this. Case in point: I cloned a project today just to see how it works, with no intention of actually using it. The project’s tests failed because I didn’t have three dependencies: one runtime, two developer.

Super. Let’s add some RubyGems 1.2.0 typed dependencies (17 Awesome Points go to anyone who can figure out the project based on this snippet).

  spec = Gem::Specification.new do |s|
    ...
    s.add_runtime_dependency 'right_aws'
    s.add_development_dependency 'Shoulda'
    s.add_development_dependency 'mocha'
    ...
  end

Now, build the gem using whatever means have been provided, and install it with its developer dependencies.

… Wait, what? The release announcement doesn’t tell me how to do trigger the developer dependencies. Google is no help, either. And good luck finding RDocs for RubyGems.

After I started grumbling, but before diving into the rubygems source, I remembered to run gem help install. There I found this beautiful, kissable tip:

 --development                Install any additional development
                              dependencies

Here’s the information you came for: To install a gem with developer dependencies, use gem install --development

Developer dependencies should also be useful with a rake setup task, as Assaf Arkin advises having for all public projects. Somebody’s going to have to dive into the source code to figure out how to make that work.

So now I have added some niceness to a project I cloned from GitHub. It’s not much more work to fork the project, commit my changes, and shoot off a pull request. All for a project I’m not really going to use. Long live GitHub.

Paging in OpenLDAP, or "What, no LIMIT or OFFSET?"

Disclaimer: I’m not an LDAP expert, but I’ve done a whole mess of reading about OpenLDAP lately. Let the knowledgeable correct me where I err.

Paging in LDAP is somewhat of a pain, and by “somewhat” I mean “asymptotically approaching totally”. In the ldapsearch tool, for example, you have to use a “search extension” argument, as paging is not part of the search filter syntax. This is as opposed to SQL queries, where you may specify a LIMIT and OFFSET in the WHERE clause. Thus LDAP clients must implement the pagedResults search control (and the LDAP directory server must support it).

It gets worse. Check out the way the paging is implemented when following RFC 2696 (http://www.faqs.org/rfcs/rfc2696.html). You can only specify the size of the result set, not the offset or a page number. The LDAP server returns a cookie with the search results. The client uses the cookie in the next pagedResults query, and the server uses the cookie to figure out where to start the next set of results. LDAP clients must treat the cookie as opaque, i.e. they shouldn’t know how to do anything other than send the cookie back to the server.

Thus the only way to paginate results on the server side appears to be by looping through all results. The client must retain a cookie from each query for use in the next. Hrmmm. Can you guess who wrote RFC 2696?

At the time of this writing, there are two Ruby libraries for LDAP access, and ActiveLdap can use either as its adapter. To the extent that Net::LDAP supports the pagedResults control, it is only to prevent ActiveDirectory from choking when a query returns more than 1000 results. See ./lib/net/ldap.rb:1158 for the code that handles the pagedResult control.

Ruby/LDAP does support pagedResults, which I should have figured out from the line in the TODO file that started the discussion on the mailing list that started my research: “Add result pagination via LDAP::Controls”. So I think adding support for the control to the Ruby/LDAP adapter for ActiveLdap should be practical.

It might be possible to roll your own pagination, in a very ugly way, by calling the ActiveLdap::Base#search method with a block that throws away results before and after the desired page set. Net::LDAP yields each entry *after* adding it to the result_set array, so you would need to set the entry to nil and compact the result.

Alternatively, perhaps you could override the Net::LDAP search method to yield the entry to the block first, then add it to the result_set only if not nil.

It’s ugly every way you look.

Here’s the link that started my digging:
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/195249

Paging in OpenLDAP, or "What, no LIMIT or OFFSET?"

Disclaimer: I’m not an LDAP expert, but I’ve done a whole mess of reading about OpenLDAP lately. Let the knowledgeable correct me where I err.

Paging in LDAP is somewhat of a pain, and by “somewhat” I mean “asymptotically approaching totally”. In the ldapsearch tool, for example, you have to use a “search extension” argument, as paging is not part of the search filter syntax. This is as opposed to SQL queries, where you may specify a LIMIT and OFFSET in the WHERE clause. Thus LDAP clients must implement the pagedResults search control (and the LDAP directory server must support it).

It gets worse. Check out the way the paging is implemented when following RFC 2696 (http://www.faqs.org/rfcs/rfc2696.html). You can only specify the size of the result set, not the offset or a page number. The LDAP server returns a cookie with the search results. The client uses the cookie in the next pagedResults query, and the server uses the cookie to figure out where to start the next set of results. LDAP clients must treat the cookie as opaque, i.e. they shouldn’t know how to do anything other than send the cookie back to the server.

Thus the only way to paginate results on the server side appears to be by looping through all results. The client must retain a cookie from each query for use in the next. Hrmmm. Can you guess who wrote RFC 2696?

At the time of this writing, there are two Ruby libraries for LDAP access, and ActiveLdap can use either as its adapter. To the extent that Net::LDAP supports the pagedResults control, it is only to prevent ActiveDirectory from choking when a query returns more than 1000 results. See ./lib/net/ldap.rb:1158 for the code that handles the pagedResult control.

Ruby/LDAP does support pagedResults, which I should have figured out from the line in the TODO file that started the discussion on the mailing list that started my research: “Add result pagination via LDAP::Controls”. So I think adding support for the control to the Ruby/LDAP adapter for ActiveLdap should be practical.

It might be possible to roll your own pagination, in a very ugly way, by calling the ActiveLdap::Base#search method with a block that throws away results before and after the desired page set. Net::LDAP yields each entry *after* adding it to the result_set array, so you would need to set the entry to nil and compact the result.

Alternatively, perhaps you could override the Net::LDAP search method to yield the entry to the block first, then add it to the result_set only if not nil.

It’s ugly every way you look.

Here’s the link that started my digging:
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/195249

nginx 0.6.6 – make install fails

I’m trying to deploy a Rails app to a new server with Deprec and Capistrano Server Extensions (capserverext). The capistrano task fails when compiling nginx, during the `make install` bit. The make errors are something like this:

cp: cannot create regular file `/usr/local/nginx/conf/mime.types.default'
No such file or directory

I tried compiling nginx directly, to eliminate deprec and capserverext, and the problem persisted.

After much head beating, and with disbelief, I concluded that the problem was a bug in nginx. Hubristic, I know. But searching the nginx mailing list immediately turned up a message with a patch from the developer. The ‘@@’s in this patch is munged on the web, so I pastied it for your consumption.

The problem stems from the addition of a new configure option, --sysconfdir. This new option means that capserverext is going to need a change to the compile_nginx task.

Once patched, you can run configure with --sysconfdir=/usr/local/nginx/conf to meet capserverext’s assumptions. But having to patch the source breaks the whole install_nginx task anyway.

What you do, though, is bravely pretend that prepare_host is going to work. When it fails:

  1. ssh into the server and cd to /usr/local/src/nginx-0.6.6/
  2. wget http://pastie.caboo.se/84215.txt
  3. patch -p0 < 84215.txt
  4. run the configure script with the arguments from capserverext’s nginx recipe plus –sysconfdir=/usr/local/nginx/conf
    sudo ./configure --sbin-path=/usr/local/sbin \
      --pid-path=/var/run/nginx.pid \
      --error-log-path=/var/log/nginx/error.log \
      --http-log-path=/var/log/nginx/access.log \
      --with-http_ssl_module \
      --sysconfdir=/usr/local/nginx/conf
  5. sudo make
  6. sudo make install
  7. rm /usr/local/nginx/conf/nginx.conf

Now nginx 0.6.6 should be installed on your server. Back on your dev machine run the following tasks to get back on track:

  • cap install_nginx_start_script
  • cap nginx_postgres_rails_setup ( or cap nginx_mysql_rails_setup, if you’re using mysql)

This gets you past the prepare_host task.

Here’s hoping this post becomes obsolete very soon.

FinderColor: A Ruby interface to Finder labels in Mac OS X

I just posted to Rubyforge the first public version of FinderColor, a very small interface to the Finder label colors in Mac OS X. FinderColor sends Apple Events using rb-appscript, bypassing AppleScript entirely. This counts as a good thing.

Install: sudo gem install findercolor

There are only 5 methods to FinderColor:

 
  FinderColor.get_index(full_path_to_file)
  FinderColor.set_index(full_path_to_file, index)
  FinderColor.get_color(full_path_to_file)
  FinderColor.set_color(full_path_to_file, symbol)
  FinderColor.batch_set(hash)
 

The index argument must be between 0 and 7. The hash argument to batch_set expects the keys to be integers or symbols for color names. FinderColor::Labels gives you an array of the color symbols in their index order:

 
  FinderColor::Labels #=> [:none, :orange, :red, :yellow, :blue, :purple, :green, :gray ]
 

Rdocs here.

ActiveLdap wants to find your subschemaSubentry

Today was the second time I had to research the same problem with OpenLDAP and ActiveLdap. I have no idea what happened to the solution that I found and employed, but it’s gone. Can’t find it. No love from grep.

The problem is this error in ActiveLdap

 
 undefined method `[]' for nil:NilClass - (NoMethodError)
 ../active_ldap/adapter/base.rb:99:in `schema'
 

The solution is to add two ACL lines to my slapd.conf or one of its includes:

 
 access to dn.base="" by * read
 access to dn.base="cn=Subschema" by * read
 

The reason why come is that ActiveLdap apparently queries anonymously for the schema, and my acls are too mean and stingy. If you start your development with strict ACLs, you hit the problem early. If you wait until near deployment-time to tighten up the security, you will be surprised when stuff just stops working.

You can see whether your ACLs are preventing access to the schema by running the following ldapsearch command:

ldapsearch -xh http://www.example.com -b '' -s base subschemaSubentry

If the result doesn’t look something like the example below, then you can try adding the two ACL lines above. The important section is the second, where you see that the value of subschemaSubentry is ‘cn=Subschema’

 
 # extended LDIF
 #
 # LDAPv3
 # base  with scope base
 # filter: (objectclass=*)
 # requesting: subschemaSubentry 
 #

 #
 dn:
 subschemaSubentry: cn=Subschema

 # search result
 search: 2
 result: 0 Success

 # numResponses: 2
 # numEntries: 1
 

ActiveLdap, belongs_to, and "invalid filter syntax"

After I started using :adapter => 'net_ldap' in ActiveLdap, my belongs_to associations all broke with when constructing a search filter with Net::LDAP

invalid filter syntax (Net::LDAP::LdapError)

The classes look something like this:


 class Person  'uid', :prefix => 'ou=people', 
       :classes => ['top', 'mozillaAddressBookEntry']
   belongs_to :family, :class => 'Family', 
      :foreign_key => 'dn', :primary_key => 'member'
 end
 
 class Family  'uid', :prefix => 'ou=families', 
       :classes => ['top','family', 'mozillaAddressBookEntry']
   has_many :people, :class => 'Person', 
      :wrap => 'member', :primary_key => 'dn'
 end
 

As I found out per this thread, Ruby Net::LDAP apparently can’t handle search filters with “=” or “,” characters, and doesn’t allow escaping with “\3D” or the like.

I switched back to the default Ruby/LDAP adapter, and the belongs_to associations work as expected.

Capifying a Merb application

UPDATE: I was embarrassed that I actually get hits on this post, so I’m adding a stripped down example deploy.rb.

Create a directory named config in the Merb root (not inside dist).

Create your deploy.rb file inside the config directory. Here’s a stripped down version of what I have used:

# ./config/deploy.rb
# Do your merb daemon configuration in ./dist/conf/merb.yml

set :application, "awesome_thingummy"
set :scm_name, "thingummy"
set :repository, "svn://thelese.com/merb/#{scm_name}/trunk"
set :checkout, "checkout"

# =============================================================================
# ROLES
# =============================================================================

role :web, "slice1.automatthew.com"
role :app, "slice1.automatthew.com"

# =============================================================================
# OPTIONAL VARIABLES
# =============================================================================
set :deploy_to, "/var/www/apps/#{application}"
set :user, "deployer"            # defaults to the currently logged in user

# set :scm, :darcs                  # defaults to :subversion
set :svn, "/opt/local/bin/svn"      
set :merb, "/opt/local/bin/merb --merb-root #{deploy_to}/current"

task :spinner do
  run merb
end

task :restart do
  run "#{merb} -k all"
  run merb
end

task :update_code do
  run "#{svn} --quiet #{checkout} #{repository} #{release_path}"
end

task :after_update_code do
  run "ln -s #{shared_path}/log #{release_path}/log"
end

desc "Deploy the app"
task :deploy do
  transaction do
    update_code
    symlink
  end
  restart
end

Use Net::LDAP with ActiveLdap

Prior to version 0.8.2, ActiveLdap required Ruby/LDAP for its underlying connection. This posed an annoyance, if not a problem, as Ruby/LDAP does not exist as a gem. As of 0.8.2, ActiveLdap supports both Ruby/LDAP and the bona fide, gemified Net::LDAP.

The default connection is still Ruby/LDAP, however, and the official docs aren’t abundantly clear on how to make the connections with Net::LDAP. I couldn’t find an answer with Google until after I had puzzled it out myself and was about to write this post. Googling for the answer found for me this post to the ActiveLdap mailing list, which is mostly correct. The value for :adapter should be ‘net_ldap’, not ‘net-ldap’, according to the Constants list at the very bottom of the front page of the Rdocs for ActiveLdap.

Thus:


ActiveLdap::Base.establish_connection(
 :adapter => 'net_ldap',
 :host => 'localhost',
 :base => 'dc=mycooldomain,dc=com', 
 # doo wa doo shbop
 )