Author: Roi

Server flatlined four times in not even two hours…

Post author By Roi
Post date May 31, 2010
No Comments on Server flatlined four times in not even two hours…

Due to until now unknown reasons the server which holds the Jabber services crashed four times in the last not even two hours. From one second to the other the ejabberd processes took every resource they could get, and even more. 8 gigs of RAM and 8 gigs of swap, everything gone. Plus a lot of CPU load. The machine was loaded that “top” refreshed just every 5 minutes and in the end just a hardware reset helped to reboot the machine.

For the tech geeks:

top – 19:56:21 up 31 min, 1 user, load average: 22.86, 13.11, 8.71
Tasks: 240 total,   3 running, 231 sleeping,   0 stopped,   6 zombie
Cpu(s): 1.4%us, 5.8%sy, 0.0%ni, 12.4%id, 80.3%wa, 0.0%hi, 0.1%si, 0.0%st
Mem:   8190900k total, 8138972k used,    51928k free,      796k buffers
Swap: 8393848k total, 7276916k used, 1116932k free,    42404k cached

PID USER      PR NI VIRT RES SHR S %CPU %MEM    TIME+ COMMAND
3239 ejabberd 20   0 15.7g 6.0g 460 S   23 76.7   3:06.65 beam.smp

We are looking into this issue. Maybe a severe bug with ejabberd, maybe a DoS attack. We don’t know, yet.

Tags Downtimes, ejabberd, Problems

Downtimes Features Maintenance Transports

Server offline due to maintenance work this morning

Post author By Roi
Post date May 31, 2010
No Comments on Server offline due to maintenance work this morning

The server was offline this morning because of Kernel and MySQL upgrade. It would have happened faster if the server rebooted cleanly after “shutdown -r now” which it didn’t. So we had to send someone there to reset the machine manually.

We also upgraded Spectrum to support JID escaping. If this works after our tests (there seem to be some problems with clients who don’t support the unofficial % character which is used for @) I write more about this here.

Tags Downtimes, Features, JID escaping, Maintenance, MySQL, Spectrum

Features

Moods now also work on secondary domains

Post author By Roi
Post date May 29, 2010
No Comments on Moods now also work on secondary domains

We stumbled upon this by accident – moods (or to be precise: everything which Pubsub/PEP offer) did not work on secondary domains.

It works now, it was just a little configuration change. The Pubsub service subdomain (e.g. pubsub.jabber.hot-chilli.eu) has to be the same domain as the Jabber service (e.g. jabber.hot-chilli.eu), not a subdomain of the Jabber server’s primary domain (e.g. jabber.hot-chilli.net).

Tags Domains, PEP, Pubsub

Transports

Spectrum transports “ghost” contacts issue is fixed

Post author By Roi
Post date May 25, 2010
No Comments on Spectrum transports “ghost” contacts issue is fixed

The thing with contacts appearing to be online after a Spectrum transport crashed (mentioned here) got fixed on May 23rd.

Tags Spectrum, Transports

Problems Transports

ICQ Transport stable again

Post author By Roi
Post date May 22, 2010
No Comments on ICQ Transport stable again

We had stability problems with ICQ Transport starting yesterday morning. This was caused by a certain version of libpurple. Issues should be fixed now.

Please note that with Spectrum transports crashes (not “normal” stops or restarts) it might happen that your client thinks it’s still online. Therefore some contacts appear online even if the client is not really logged into the transport at the moment. Spectrum developers are working on this issue. Until then I just hope that the Spectrum transports run stable. ;-)

With the new libpurple Gadu-Gadu transport should finally be stable, too. We’ll see how this works out in the near feature.

Tags Gadu-Gadu, ICQ, Problems, Spectrum, Transports

Transports

Facebook changed JIDs

It looks like Facebook changed their JIDs from u1234567890@chat.facebook.com to –1234567890@chat.facebook.com. For our Facebook Transport users this means that you might see all Facebook contacts again in your roster, maybe twice, old and new. Sorry for the trouble, but lucky for us this is a Facebook issue.

Old transport contact on facebook.jabber.hot-chilli.net:
u1234567890%chat.facebook.com@facebook.jabber.hot-chilli.net

New transport contact on facebook.jabber.hot-chilli.net:
–1234567890%chat.facebook.com@facebook.jabber.hot-chilli.net

Tags Facebook, Transports

Downtimes Problems

Problem with Jabber database

Post author By Roi
Post date May 12, 2010
No Comments on Problem with Jabber database

Unfortunally there was a major problem with the database for all accounts of the jabber.hot-chilli.net domain (not accounts from other domains, like jabber.hot-chilli.eu).

Finally we decided to restore a backup from 4th/5th of May 2010 (day of the server move) and had to take the Jabber server down for about 2 hours.

Affected are just the contact lists and contact groups. This means that as an affected user of this you have to add/delete all buddies you changed since then.

We really apologize for the trouble caused, especially because the backup is one week old.

The question remains why we just got 20 rows of data inside our current database backup from this morning, missing 150000 (!) other rows. We will take a deep look into the backup process.