[Coco] drive crash

[Gene Heskett]

On Saturday, February 07, 2015 01:58:02 AM Richard E Crislip wrote:
> On Fri, 6 Feb 2015 07:28:17 -0500
> 
> Gene Heskett <gheskett at wdtv.com> wrote:
> > Greets all;
> > 
> > In the process of recovering from a failing terrabytew drive on this
> > box, I thought I would see about converting some huge mailfiles, that
> > were how kmail did it in "the early years".
> > 
> > The quickest way so it seemed was to setup a 2nd Local Account that
> > would process these mailfiles when one was moved to that location,
> > and in the process would seperate the messages into indiviudal
> > directory entries, commonly call maildir.
> > 
> > The mailfile for the coco list, dates back to 2002, and was just shy
> > of 5 gigabytes!  It took kmail about 20 minutes to process that file,
> > and when it was done I had about 82,000 unread messages in the new
> > coco maildir.
> > 
> > And kmail is about 10x faster finding the next unread message now.
> > 
> > Fun & games with a new install specfically for running a simulated
> > linuxcnc.
> > 
> > So, anyone who has tried to access my web page since last friday
> > night, it should be back up now.
> > 
> > Cheers, Gene Heskett
> 
> Hi Gene,
> 
> What do you think of Claws Mail. I've been using it and it has served
> me well. KMail cost me most of my list history 8-/. That's also the
> reason I fired Ubuntu, they broke KMail with one of their upgrades.

I am on the claws list, and it seems to me that Claws will forever suffer 
what we call infant mortality forever. IOW they add features that are not  
well coded, and its usually months before a given new feature Just Works(TM)

Kmail has its warts, but savvy users do not lose old mails.  I just has to 
re-install what was once a Ubu Lucid 10.04.4 LTS due to the main drive 
developing a tummy ache and going read-only.  One of the things I intended 
to do the next time I installed, was to convert some of my older folders, 
which were in mailfile format, into maildir format as kmail seems to be able 
to process that about 2x as fast as it can do a big mailefile.  Kmail has an 
import function, but its slow is an understatement.  So I setup a 2nd local 
account where it would treat that almost 500 megabyte mailfile as incoming 
mail because that is the forat it epects to get from my fetchmail/procmail 
driving clamav and spamassassin, and  what surives that gets put into 
/var/spool/mail named gene.  The second local account expects to seea 
mailfile in that directory named 'import', so after manually making the 
directories the maildir format uses, and setting up what I thought was the 
correct filter criteria to catch the 'incoming' and route it to the coco 
maildir structure, I ticked the mouse on check mail. 20 minutes ;ater I had 
a nearly 81,500 msgs count showing for the coco folder.  But my filter wasn't 
as good as I thought because we have changed the [Coco] in the subject line 
several times over our history, so I spent some time yesterday fine tuning 
the filters to catch the other 2 popular subject labels we've, then wore out 
the ctl-j keys about 3000 times to finally get 99.99% of it into the proper 
folder. Now I admit that is NOT how kmail would allow you to do it as it 
cannot make the folder structures without putting them in a subdir.  Most 
fol;ks would simply accept that and learn to live with it.  The only mails I 
lost were in fact spam that got fed to sa-learn spam about 2:30 this morning 
and then deleted out from under kmail, but its housekeeping will discover 
that in perhaps an hour and refresh the list view.

Yes, kmail has some warts, one of which has caused me to setup background 
processes that do not tie up the machine, off loading the mail fetching from 
your mailserver and making it a background process.  But ALL of those warts 
I am familiar with. and have writen bash scripts that function as background 
daemons that I am never aware are running unless they stop.  To use a 
furinstance, I have a bin directory as a subdir of /home/gene, where I put 
all those scripts, one of which is mailwatcher.  About a 40 line script, it 
loops forever, launching a inotifywait module that watches the 
/var/spool/mail/ directory, and when an incoming mail causes the gene file to 
have the mail appended to it, the append operation has to open and close 
that file.  When the file is closed, inotifywait dies and reports that filename 
back to my script, which in turn sends kmail a message over dbus to go get 
the mail named.  When thats been so done, it relaunches inotifywait to go 
back to watching that directory again.  

This synchonizes the incoming mail so that kmail gets it about 2 
milliseconds after it comes in, all without me having to tell kmail to go 
get it.  The last time I asked, claws had not managed to grow a dbus port so 
you either do it manually, or set up a timer so it does it for you.  But 
that means a 5 minute wait, or whatever you set the timer for, for claws to 
tell you 'you have mail'.  This way is 10 milliseconds after it arrives.  
Fetchmail itself runs as a background daemon, sending mailfilter out to scan 
whats there, and check it against a list of duff sources I maintain, if a 
hit, that mail is deleted off the server and all I see is a DENY line in the 
logs.  Otherwise it returns a count of mails available and fetchmail then 
goes and gets them. 

Thats one heck of a pile of wheel spinning I don't have to do, so my duties 
are to tap the + key to go to the next msg, reply to it if I am so inclined, 
hitting a ctl+return to send my reply. Wash rinse & repeat, that is the 
extent of the work I actually have to do.  And drag-n-drop an errant spam to 
the spam directory where sa-learn will train the bayes database on it in the 
night.

Whats not to like?

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>
US V Castleman, SCOTUS, Mar 2014 is grounds for Impeaching SCOTUS