Over the years we at Nemein have been experimenting with various ways of keeping our operations managed. Now with some personnel changes including Joe's departure it was a good time to change the way we work again.

I had some goals:

• Keeping status of different projects up-to-date with more accuracy
• Ensuring our sales and project management knows if some project is being blocked by missing materials
• Enabling a more distributed, web working culture

We're a small company of less than 10 people, and as such most project management methods have not been very successful for us. In general they have been made for situations where same person or team keeps on working on a project for several months, whereas in our situation a person typically works on several projects every day.

I discussed this over some beers with Tero Heikkinen from Rohea, and he told me how they were implementing Scrum in their small company. While their number of different projects running at the same time is a bit smaller, their situation otherwise is quite similar: same technologies used, Ajatus for work tracking, etc.

We had a company sauna evening and I presented Tero's ideas there: we would partially implement the Scrum model, and keep tuning it to our needs. At the first phase this means:

• Every morning we have an all-hands 15-20min meeting ("the daily Scrum") where everybody goes over what they have been doing the previous day, and what they were planning to do today. If they are being blocked by something missing: a software bug, missing information or other materials, this is also brought forward
• Every project has a file in Google Docs where we keep the project status and task list (Backlog). This task list is updated based on what comes up in the morning meeting
• Work hours are reported with Ajatus. Rohea also uses it for project burn-down charts, and once their add-on for that is finished we may do the same
• Emilia, the project manager (or Scrum Master) is responsible for resolving possible impediments and maintaining the per-project status files

The approach we have taken should be quite pragmatic and low-tech. Instead of fancy project management software we use simple word processing for status data. And thanks to Google Docs the documents produced are accessible and editable from anywhere.

Similarly the actual meetings are quite easy to manage. The people who are at the office attend there, and others attend either via a Skype or mobile phone conference call, depending on network availability. We decided to have them at 10am so that everybody will be able to participate. Even if there is a Sprint or meeting scheduled for the same time, the short time needed for our all-hands meeting means it can be held over a "cigarette break".

Ajatus is the only more experimental piece of software in our puzzle. In our company, we use it for hour reporting, expense tracking and keeping meeting minutes. For these it works quite well, although more reporting tools are definitely needed. The alpha status of CouchDb, the database software powering Ajatus has bit us a few times by database corruption (caused by OSX-specific erlang bug) or simply difficult installation procedure, but these problems will hopefully improve over time.

We're now in the second week of this model, and at least the gut feel is that this has improved coordination inside the company. The next challenge then is to let the customer get involved in the process. This can mean just sharing the project status files, or even giving them access to actual meetings or the Ajatus data.

Database format changes and migration procedure between CouchDb 0.7 and 0.8

Jerry Jalava presenting the Ajatus Open Source P2P CRM

For those of you who are interested but missed the talk
the video is now available to be downloaded.

Next saturday there is a FOSS event "Avoimien Ohjelmien Päivä" in Turku, Finland.
I will be holding a LightningTalk of Ajatus there somewhere after 3pm.

If you are coming there, let's go for a beer etc. and discuss the topic or offtopic...

On the side of FOSDEM we went today to the XMPP devcon held here in Brussels. In there we started formulating our ideas of XMPP publish/subscribe (XEP-0060) based replication for both Midgard and Ajatus.

This post contains very early ideas, but we would be happy to get some feedback on them.

Basic idea

Each Midgard or Ajatus server runs a "synchronization daemon" which is connected to a XMPP server using some JID identity.

The sync daemon registers a set of pub/sub nodes corresponding to the content structure on the Midgard or Ajatus instance:

• in Midgard: Sitegroups, MgdSchema types, paths (possibly a regexp), approval state
  • /midgard_article/all, /midgard_article/approved
• in Ajatus: tags

The pub/sub nodes can be set up with some access control rules. For example, Ajatus tags would by default require "whitelist" authorization to subscribe.

When content is changed, the sync daemon gets notified about it (via D-Bus signals in Midgard, and via CouchDb external indexer API in Ajatus). The content object (and immediate children like parameters and file attachments) is serialized into the syndication format and sent onward as a XMPP "pub" leaf. If the object appears on multiple nodes (multiple Ajatus tags for instance), the "pub" leaf is sent to all of them.

XMPP server and the federated network will then handle notifying the subscribers of the nodes about the new leaf.

The subscribers will receive the leaf, and unserialize it to the Ajatus or Midgard database.

Communication between sync daemon and application

The sync daemon should exist as an entity separate from the actual user application. Communication between the sync daemon and Midgard or Ajatus should happen via the application database.

This means that XMPP pub/sub whitelists for Ajatus tags would be maintained in the ajatus_db (non-replicated) database, and the sync daemon would read them from there. The Jabber server credentials would also be stored in the same database.

Information about resources from jabber buddies

User's replication partners are stored as local contacts in the application database. In Ajatus this means "Contacts" and in Midgard "midgard_person" objects. Both storage models have optional JID field.

If JID is marked for a contact, the sync daemon should at startup (or at an interval) try to discover if the contact has pub/sub resources available.

Content transformation in replication stage

While mainly intended for Midgard-to-Midgard and Ajatus-to-Ajatus replication, the same mechanisms could work across different systems.

For this, the sync daemon on the subscribing end should support XSLT transformations before content is unserialized into the system. The XSLT transformation templates should be configurable per subscription.

Hermod / Hermóðr (Midgard replication)

• "in Norse mythology, messenger of the gods. He was a son of the principal god, Odin, and his wife, Frigg. Known as Hermod the Swift, he was called upon by the other gods when they had a task requiring speed and urgency." - Encyclopedia Britannica
• Written in Python
  • Using twisted framework or x60br XMPP library
  • using standard Midgard MgdSchema object serialize/unserialize methods
• How to proof-of-concept before D-Bus? Some watcher in MidCOM? (watcher touches spool file)

PillowTalk (Ajatus and CouchDb replication)

• Written on Erlang
• Extendable through plugins
  • Content parsers (Midgard2Ajatus, Ajatus2Midgard)
  • Security
  • CDATA JSON block or convert to XML? (Decision of the content parser?)

Technorati Tags: ajatus, midgard, jabber, replication, synchronization, xmpp