Mobydick server /SOURCE/SHARE gets full very fast

Hi Guys,

We are having an issue with our mobydick server disk space. The /SOURCE/SHARE constantly gets full within less than a week.
Please see below:

root@mobydick:/TARGET/SHARE/etc/admin# df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.0G  1.1M 1023M   1% /SOURCE/RAMDISK
/dev/sda1       3.8G  557M  3.1G  16% /SYSTEM
/dev/vg/SHARE    18G   16G  395M  98% /SOURCE/SHARE
/dev/vg/BACKUP   29G   22G  5.3G  81% /BACKUP
/dev/vg/NODE    5.8G  142M  5.3G   3% /SOURCE/NODE
/dev/loop0      415M  415M     0 100% /SOURCE/FIRMWARE
/dev/loop1      358K  358K     0 100% /SOURCE/LINKS
unionfs         1.0G  1.1M 1023M   1% /TARGET/RAM
unionfs         5.8G  142M  5.3G   3% /TARGET/NODE
unionfs          18G   16G  395M  98% /TARGET/SHARE
tmpfs           1.6G  8.0K  1.6G   1% /TARGET/RAM/run
tmpfs           5.0M     0  5.0M   0% /TARGET/RAM/run/lock
devtmpfs         10M     0   10M   0% /dev
tmpfs           3.6G  4.0K  3.6G   1% /TARGET/RAM/run/shm
unionfs         1.6G  8.0K  1.6G   1% /TARGET/RAM/run
unionfs         5.0M     0  5.0M   0% /TARGET/RAM/run/lock
tmpfs           1.6G  8.0K  1.6G   1% /TARGET/RAM/run
tmpfs           5.0M     0  5.0M   0% /TARGET/RAM/run/lock
tmpfs           3.6G  4.0K  3.6G   1% /TARGET/RAM/run/shm
unionfs          18G   16G  395M  98% /TARGET/SHARE/var/spool/hylafax/etc

Deleting the ‘messages’ file under /SOURCE/SHARE/var/log/asterisk and rebooting the server would free up some space but it gets full again very fast causing the database to stop functioning.

I have access to the database but i am not sure if there is anything there needs to be checked.

I am not sure what drives the space to be full
Any help/tips are appreciated.

Thanks

Anybody please ?
I would appreciate any help/advice

Hi,

some possible sources are:

  • excessive logging (asterisk debug log)
  • LOTS of database records. This can happen if such a system is exposed to the internet. Each unauthenticated incoming invite will add a few bytes. This can sum up really quickly
  • Lots of database records due to loops in the dialplan (scripting/actions/redirections). Each call can add 100s of records in few milliseconds
  • excessive call recording, many voicemails etc

Best to run a “du -sch *” within your filesystem to pinpoint the problematic folder.

You’re using some version of a “classic” server wich will be deprecated by end of month. I suggest to invest a bit of time and to perform an update to a more modern version.

Thomas

Thank you Thomas.
Here is what I could find:

/var is taking almost all the space under /SOURCE/SHARE
/var/log is taking 1.7G
/var/lib is taking 14 G

See below:

root@mobydick:/var# du -sch *
4.0K    agentx
1.7M    backups
8.6M    cache
22M     jasperreports
14G     lib
0       local
20K     lock
1.7G    log
76K     mail
0       opt
88K     queuemetrics
48K     rrd
396K    run
11M     spool
100M    tmp
543M    www
155M    xmppd
2.7M    zend
17G     total

I have access to the database, however, I tried to check the tables sizes but most of them were in the KB size. Is there a specific where logs or system events might be accumulating ? Is there a way to clean up the database from any accumulated unneeded records without affecting the system functionality ?
I have deleted all the old voice messages, but that only freed less than 500 Mbytes.
Is there a way to stop Asterisk debug log from building up ? Is Debug mode (if any) required to run ?
And yes, the system is exposed to the internet.

I am sorry for all those questions but I really need to fix this issue.

Best Regards

@ali.kalamchi: sounds maybe strange but did you restart the system recently? I can remember that some version had a issue with the filesystem in a way that we continued to log into already deleted files. The filesystem hides such files but the owning process can still write into them.
Please reboot and check again.

Hi Thomas,

Yes I did. In fact the only way to free up the space taken by the log, is to remove the “messages” file, create it again and set it to the correct owner “asterisk” then reboot the system.
However, the space keeps shrinking. few months ago I had like 3.5 GB free, then it fills up, do the steps above, and the space I get becomes smaller. Now I have only 1.4GB taken by the log.

I also noticed 1GB missing space. please see below:

/dev/vg/SHARE    18G   17G     0 100% /SOURCE/SHARE

the total size is 18GB, used 17GB , and zero remaining (100% usage)

Further to my investigation:

 mobydick=# select pg_size_pretty( pg_database_size('mobydick') );
 pg_size_pretty
 ----------------
 13 GB
(1 row)

But I can’t figure out what is inflating the database size, or which table.
Any hints on that ? is there a way to cleanup the database from old records/logs or whatever is filling it up ?

thanks

Well, as stated above: you probably will have lots of pseudo phonecalls if you exposed your system to the public internet. The table in question is mdphonecallrecord.

many thanks Thomas. Spot on.
there we go:

   mobydick=# select pg_size_pretty( pg_total_relation_size('mdphonecallrecord') );
   pg_size_pretty
   ----------------
   13 GB
   (1 row)

Is there a way to clean / purge this table without affecting the server functionality ?
Best Regards

It’s like a log for all your calls. You can delete all of it or keep N days without any side effect. Be aware that you will cause heavy IO on your host when you remove many/all rows at once. It might be a good choice to use TRUNCATE TABLE instead of a simple DELETE FROM but you must run vacuum afterwards. PostgreSQL Docs will help you.

Hi Thomas,

I am working on a clone VM (no network connection) to avoid interruption/damage to the production VM while I test.
I have tried :

  DELETE FROM mdphonecallrecord WHERE phonecallrecord_timestamp < (now() - '180 days'::interval); 

Howevere that took hours of waiting until I stopped it, and there was no change in size. I ran a VACUUM command:

  VACUUM FULL VERBOSE mdphonecallrecord; 

But gained nothing.

Finally decided to truncate the the table which pointed out that there are pointers to mdjornal table, so I had to add CASCADE

 TRUNCATE mdphonecallrecord CASCADE; 

Which immediately flushed the table and returned the 15 GB of available space.

The question, is this safe to run ? would affect any functionality in the system ? or it just simply empty the call records and any pointers to them ?

Kindly advise.

Best Regards

@ali.kalamchi: like i wrote above: you can delete it without any side effect. Except that you loose your call history, of course :wink:

Hi Thomas,

Thank you very much for your assistance. I have truncated the table as below:

TRUNCATE mdphonecallrecord CASCADE; 

The disk usage dropped to 13%. The server is working fine.

I really appreciate your advice.
I will mark the case as solved.

Stay safe and have a nice day.