IRC transport migrated to Spectrum

Wednesday, June 30th, 2010, 23:21 by Roi

We just migrated IRC transport to Spectrum. Before it was a core feature of ejabberd, the software we use for the Jabber server itself. Spectrum has more features, so we switched.

If you experience any problems please let us know.

DoS once again yesterday evening

Wednesday, June 9th, 2010, 08:40 by Roi

Once again the server was hit. At around 8pm yesterday evening the services were unreachable for about 30 minutes, the transports for about 60 minutes.

For sure there will be another incident, but this time everything will get logged. From what we then read out of the logs we will choose how we react. If it has something to do with Nimbuzz, we will block Nimbuzz completely.

DoS attack again…

Tuesday, June 8th, 2010, 01:23 by Roi

Again we suffered a DoS attack. The Jabber services and website were down from ~10:45pm to 00:30am.

For sure we have to do something about this. We are already digging into this, also recieving feedback from other Jabber server admins. Maybe abusive Nimbuzz accounts are the reason and, if this turns out to be true, we are also thinking about firewalling Nimbuzz.

What would you think about blocking Nimbuzz? Are you having friends from Nimbuzz in your roster? Tell us your opinion.

Pubsub/PEP migration to MySQL complete

Tuesday, June 1st, 2010, 09:34 by Roi

…and it seems to work. Sorry for the 30 minute downtime because of that.

Planned Pubsub/PEP migration to MySQL

Monday, May 31st, 2010, 22:38 by Roi

We are planning to migrate Pubsub/PEP (Publish-Subscribe / Personal Eventing Protocol) from Mnesia to MySQL in the very near future, which means the next two days.

Unfortunally there is no export/import script, so all data will be lost. It is also impossible to export own data from a client, at least I was told so.

Migration should fix some problems with PEP, also MySQL allows more administration and has less resource overhead.

Server flatlined four times in not even two hours…

Monday, May 31st, 2010, 21:52 by Roi

Due to until now unknown reasons the server which holds the Jabber services crashed four times in the last not even two hours. From one second to the other the ejabberd processes took every resource they could get, and even more. 8 gigs of RAM and 8 gigs of swap, everything gone. Plus a lot of CPU load. The machine was loaded that “top” refreshed just every 5 minutes and in the end just a hardware reset helped to reboot the machine.

For the tech geeks:

top – 19:56:21 up 31 min,  1 user,  load average: 22.86, 13.11, 8.71
Tasks: 240 total,   3 running, 231 sleeping,   0 stopped,   6 zombie
Cpu(s):  1.4%us,  5.8%sy,  0.0%ni, 12.4%id, 80.3%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   8190900k total,  8138972k used,    51928k free,      796k buffers
Swap:  8393848k total,  7276916k used,  1116932k free,    42404k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
3239 ejabberd  20   0 15.7g 6.0g 460 S   23 76.7   3:06.65 beam.smp

We are looking into this issue. Maybe a severe bug with ejabberd, maybe a DoS attack. We don’t know, yet.

Server offline due to maintenance work this morning

Monday, May 31st, 2010, 21:37 by Roi

The server was offline this morning because of Kernel and MySQL upgrade. It would have happened faster if the server rebooted cleanly after “shutdown -r now” which it didn’t. So we had to send someone there to reset the machine manually.

We also upgraded Spectrum to support JID escaping. If this works after our tests (there seem to be some problems with clients who don’t support the unofficial % character which is used for @) I write more about this here.

Moods now also work on secondary domains

Saturday, May 29th, 2010, 14:32 by Roi

We stumbled upon this by accident – moods (or to be precise: everything which Pubsub/PEP offer) did not work on secondary domains.

It works now, it was just a little configuration change. The Pubsub service subdomain (e.g. pubsub.jabber.hot-chilli.eu) has to be the same domain as the Jabber service (e.g. jabber.hot-chilli.eu), not a subdomain of the Jabber server’s primary domain (e.g. jabber.hot-chilli.net).