Things are going smoother than I expected. Hibernate 3 already supports the use of dynamic maps and XML documents (these are alternative entity-mode's in Hibernate lingo) through the tuplizer mechanism. By extending Hibernate's AbstractEntityTuplizer class it is remarkably simple to let Hibernate work directly with instances of IRubyObject. Features like inheritance and many-to-one relations seem to work fine. (Haven't tried components and proxies yet.)
Now I'm trying to get collection mapping working (one-to-many). Hibernate wraps the user's collection (which would be a Ruby Array in our case) by its own implementation (of some java.util collection interface like java.util.List when dealing with POJOs) to track additions/removals and to support lazy collection loading. Customizing this is also nicely supported by Hibernate through user collection types. For this we need a wrapper that looks (walks, swims and quacks) like an Array to the Ruby code, and implements org.hibernate.collection.PersistentCollection to keep Hibernate happy on the Java side. I haven't found any other solution than to write a JRuby Java extension (by implementing JRuby's BasicLibraryService interface) to create a RubyObject subclass named PersistentRubyArray (to stay in line with the Hibernate naming convention) that wraps a RubyArray instance.
Anyway, it's fun to manipulate Ruby objects from within Java like puppets on a string, and to see how it all fits naturally with Hibernate.