European Bob's Hula Hoopla!

So, the new debs have been uploaded, and contain my mdbtool. This allows you to munge around your mdb store without playing with the files. Objects can be added and removed, and you can get basic information and attributes from them. I would be interested to hear from those who use it.

Thanks to the help of fatpelt, my problems in the previous post (and a post I made a while ago) are now fixed. The segfault was caused by incorrect argument order, which I should have noticed. The bug getting the right result back was caused by passing an object reference where I should have passed NULL - with a reference, MDB was trying to create a relative name or something. Whatever it was trying to do, it was failing.

So, I've updated the mdbtool source. The manual page is current unwritten, but would look something like:

mdbtool [command] [options]

  getattrs [object] [attribute name]
  
    Example: mdbtool getattrs \\Tree\\Context\\Server netmail:messagingserver
    
    Dumps the value of the attribute for a given object. Double-slashes are
    used to name the object for shell reasons. The result is of the format:
    
    [attribute name]::[value]
    
    This format may change, given values could contain carriage returns.
    
  setattr [object] [attribute name] [new value]
  
    Example: mdbtool setattr \\Tree\\Context\\test Sname newname
    
    Sets the value of an attribute of the object. Attribute names are prefixed
    with 'S' (for attributes that contain strings) or 'D' (for attributes that
    contain the names of objects).
    
  addobject [object] [class] [attr1=value1 attr2=value2 ...]
  
    Example: mdbtool addobject \\Tree\\Context\\test User Sname=foo
    
    Creates a new object of any given type, and initialises some of the 
    attributes. Attribute naming requires 'S'/'D' prefixes (see setattr), 
    you can have as many initial attributes as you like.
    
  setpassword [object]
  
    Example: mdbtool setpassword \\Tree\\Context\\Admin hula
    
    Set the password for any given object; usually used for User objects. The
    example given above resets the password of the admin user.

I have plans for a fair few other commands - we're missing object (re)move, and tree/object introspection in particular - but it seems at least some commands will require work to MDB and the various drivers in order to actually work. Some schema stuff will also be needed at some point. The main thing I next want to achieve is dumping and re-creating complete trees, because I think that's one thing users will really like.

And I can't figure out why, so hopefully lazyweb will help me. So, I have this code:

#include <stdio.h>
#include <stdlib.h>
#include "mdbtool.h"  /* this just pulls in mdb.h */

void leave(char *excuse) { printf("ERROR: %s", excuse); exit(1); }

int main (int argc, char *argv[]) {
	const unsigned char *attrval;
	const unsigned char *object = "\\Tree\\Context\\Server";
	const unsigned char *attr = "netmail:messagingserver";
	
	if (!MemoryManagerOpen("mdbtool"))
		leave("Could not start memory manager\n");
	
	if (!MDBInit()) leave("Could not initialise MDB\n");
	
	MDBHandle *h = MDBAuthenticate("Hula", NULL, NULL);
	if (h == NULL) leave("Could not authenticate to MDB\n");

	MDBValueStruct *vs = MDBCreateValueStruct(h, object);
	MDBEnumStruct *es = MDBCreateEnumStruct(vs);
	
	while ((attrval = MDBReadEx(object, attr, vs, es)) != NULL) {
		printf ("Attribute: %s\n", attrval);
	}
	
	MDBDestroyEnumStruct(es, vs);
	MDBDestroyValueStruct(vs);
	return 0;
}

I run it, and it segfaults:

(gdb) r
Starting program: /home/alex/mdbtool/simple
[Thread debugging using libthread_db enabled]
[New Thread -1210911040 (LWP 28415)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1210911040 (LWP 28415)]
0xb7fccfb9 in MDBReadEx () from /usr/lib/libhulamdb.so.0
(gdb) bt
#0  0xb7fccfb9 in MDBReadEx () from /usr/lib/libhulamdb.so.0
#1  0x080487dc in main (argc=1, argv=0xbf9f48b4) at simple.c:23

According to the API docs, I don't think I'm doing anything particularly wrong. Also, I can't make it work with MDBRead() either - doing something like:

	long l = MDBRead(object, attr, vs);
	
	for (i=0; i<l; i++) {
		printf ("Attr: %s\n", vs->Value[i]);
	}

... returns values, but cut off in bizarre places. Am I doing something remarkably silly, or is there some voodoo dance I need to do?. Frustrating :(

This works:

./mdbtool setpassword \\Tree\\Context\\Admin hula

Easier than felching around for the right file, and trying to encode things and stuff. Also, there's some source up here for now. It's not a patch to Hula, it should compile cleanly outside Hula.

We're getting to the point where the Debian packages of Hula are pretty keen, good enough for people to run, but I'm still worrying somewhat about ensuring people's configurations can be updated sanely. One part of this is reading/writing configuration via the MDB library rather than fiddling files directly - this will be particularly important if anyone starts to use (e.g.) mdb-odbc or another non-file driver.

However, there appear to be API problems which are stymying me somewhat. As an example; looking at some attributes:

$ ./mdbtool getattrs \\Tree\\Context\\Server netmail:messagingserver
netmail:messagingserver::es\Server Messaging Server

$ ./mdbtool addobject \\Tree\\Context\\test User Sname=foo

The first command is basically trying to get an attribute off the Server object - and we can see we get a response which appears to be forshortened. I cannot figure out why; it's just an MDBRead() call which gets immediately printed.

The second command tries to add an object called "test" to the tree, and set an attribute 'name' to the value 'foo'. But, we have to know whether it's a string or a DN, so you have to put S or D in front of the attribute name. Maybe you could figure it out from the schema - that would be nice, but I would think MDB would be perfectly capable of doing this itself. It seems a bizarre API :/

Other stuff has also come up. I want an import and export, so that (for example) someone can back up their configuration. This appears to be a no-go, though, because MDBEnumerateAttributesEx() is barely implemented. Without the introspection, it doesn't seem possible to dump stuff.

I'll put up some source at some point so others can have a play. It looks like it's going to be a fair amount more work before it's in a state to push into Hula..

I think I'm vaguely near having the updated admin UI suitable for committing. It seems to be properly stable - we need to have more checks in the web compiler, it seems - and for most usual uses it seems to be fine.

The main point of this exercise was to make the admin UI fit in a lot better with the main Aurora theme, but it seems like it might also be worth rewriting some of the templates to take more advantage of CSS. That could be a lot of work though, so it probably won't happen soon (unless someone other than me does it, obviously ;)

Micah put forward some great suggestions to improve the design further, and I've done some work on the tabs which I actually ended up doing in Inkscape, so there's an SVG file with the tab designs in: this makes it a lot easier to tweak and get a nice anti-aliased output. Here's a full image of the UI update (933 x 549).

In other news, Phil Cryer and I have been nominated officious Build Sherriffs for the main Hula trunk - the initial task to me seems to be to push some patches in to get BSD building, which will inadvertantly help Mac OS X. Cool :)

The new UI I talked about in my last post is now in my .debs; it all works, though a couple of panels still look a bit brown. I've uploaded a .tar.bz2 of the patch so other can have a look; it's 51K compressed but expands to about 5 times that size. Apply with cd hula/src/ && patch -p0 < webadmin-1.diff or something.

I've also noticed that the default configuration of these packages means it won't pick up SSL certs you put in /etc; I'll try to remedy that soon. I guess I really should run Hula with encrypted protocols, since that's likely to be a common requirement :D

Nat is demoing the prototype of the new Hula calendar UI. Why Nat isn't on Planet Hula I don't know; I guess he doesn't intend to blog much on Hula - so, as a public service, I'm pointing at his entry.

Amusingly enough, I'd also been playing with Hula UI this afternoon, in an effort to make the web administration look a bit more like the Aurora theme:

Apologies if that break the layout for someone feeding off my site. I've a few bugs to iron out, but will probably submit the patch sometime next week if people are interested. Otherwise, it will just live in my .debs :)

I changed over much of my packaging stuff over the last few hours, so that instead of building directly out of SVN, I'm building release tarballs. This has a number of advantages - the main one being that I can publish the source and source packages for my Debian stuff, so the files on my site are basically the latest available. However, this turned out to be harder then I figured.

I've pushed patches to fix `make dist' before, but to be honest they only ever ensured that `make dist' produced something that compiled - I figured if it compiled, there was an extremely good chance it was working. Well, apparently not - modweb, amongst other things, was completely broken and segfaulting all over the place, which is a bug in it's own right.

It basically seems large numbers of files, mostly for the templates, weren't being included in the distribution tarball. I've pushed a patch to -dev to fix some of this, we'll see if it's actually right or not. In the meantime, you can grab source .tar.gzs from me which actually work.

A while ago I posted about updating the Hula schema when upgrading packages, but haven't really had time to look at the various issues since then.

My first problem of translating a diff of the schemas into actual actions is still there, but looks doable. Thing is, it's semi-obvious what to do - but you end up throwing away some of the information that's available to you, which makes me think it might be wrong. I guess the thing to do is try it and see.

The second problem - what information is instantiated by default? Well, I wrote another script to figure that out. Rather than try and work it out from the code, this just looks inside a directory structure that has been created. This isn't necessarily right either, but seems close enough for now. The output looks something like this:

name:root
        object%032class:Top
        t:Tree

name:root/context
        object%032class:Organization
        o:Context

name:root/context/admin
        object%032class:User
        cn:admin
        surname:Admin
        private%032key:aHVsyQ==

These are the first three objects output, in an ldif-ish format with some of my own naming convention :o).

So, I now have a textual version of the schema, and the initial tree. Theoretically, it should now be possible to write a hulasetup which takes those two files and does roughly the same as our current hulasetup (albeit in vastly less code, I suspect). There are three attributes that cannot be stored in the files (they're auto-generated random pads), which I'm not sure what to do with, but although it will be a hack it won't be too bad.

Once we have this new version of hulasetup, it should be possible to do a number of cool things:

• run a tool to repair a directory - fills in the missing parts of the schema and objects, checks the values for validity;
• the "repair" tool would also be pretty good at updating directories to newer schema (there is a potential for dataloss here - e.g., a new attribute semantically replaces another, the latter is an attribute configured by the local admin: blindly putting the new attr in place and deleting the old loses the old configuration. That's not a generally solvable problem IMHO, though);
• opens up the possibility of having hulasetup be made aware of third-party schemas (since the knowledge is no longer programmed in), maybe if we installed stuff into /etc/hula/schemas/ or something (keeping the textual versions around is plainly a good idea);

Next steps: first, see if I can normalise the output formats of these various scripts (maybe even proper LDIF; this would fit in with Pat Felt's ldap plans). Second, write a tool to diff them (diff itself won't cut it for various reasons). Then, write a tool which "actions" the diff via the MDB library, to replace my hulaupdateschema program. Technically, this latter tool could also replace hulasetup (in the special case of comparing a new schema with an empty file). I have a kind-of cunning idea for this last bit too, but we'll see if it's workable.

If you'd like to take a peek at my scripts, they're here (this is a .tar.bz2), including my "hulaupdateschema". Beware with the latter - it was written with my Debian packages in mind, and is a bit hackish - it's certainly not a long-term solution.