[Coco] Re: Someone might come to complain

[John E. Malmberg]

[I hope this is not a duplicate, it appears that gmane is not accepting 
posts from my Adelphia I.P. at this time.]

John R. Hogerhuis wrote:
> On Fri, 2005-04-29 at 13:39 -0500, Roger Taylor wrote:
> 
>>I've had this problem trying to talk to a few Portal-9 customers to send 
>>them their key.  SPAM filters just plain suck. My own ISP catches a 
>>tremendous amount of real SPAM, which is great, but they also catch real 
>>e-mails and make me have to weed through their report e-mails to see which 
>>e-mails are not junk.  It takes less time to just hit delete on the obvious 
>>junk letters as they come in.

That is just indicating that your ISP has not implemented anything close
to a state of the art anti-spam system and has just implemented one that
is close to just being a placebo either because they really do not know
what the state of the art is or they do not really care.

State of the art is that over 80% of the spam can be detected with no
false positives from confirmed spam sources before the spam enters the
mail server.  This was also state of the art well over 5 years ago.

You can get close to 99% with most people seeing no false positives if
you refuse e-mail from known DHCP assigned addresses in addition to the
above.

This was also state of the art over 10 years ago.

The above has been state of the art for so long, that it is built into
just about all commercial mail server software.  All the mail server
operator needs to do is set the configuration options, and if they are a
large operation, some of the DNSbl services need compensation for the feed.

And doing so significantly reduces the cost of operating a medium to
large mail system.  In some case we are easily talking thousands of
dollars a month just for bandwidth costs.

If you have a bit more time when you are sorting though the tagged spam,
and especially the mis-tagged mail, plug the I.P. address that your ISP
mail server accepted the e-mail into the box at:

http://www.moensted.dk/spam

It will not take long until you notice a clear trend.  One of CBL, XBL,
DSBL, NJABL, and SPAMHAUS.ORG or SORBSDUL will show up on the majority
of the spam, and if they are not, what ever one is missing or rarely
showing up, your ISP is rejecting based on them before they tag the
mail.  Of the real mail there is only a small chance that you will get a
non-spam e-mail from a SORBSDUL listed address.

And of what spam gets through the above checks, most can be detected by
checking the I.P. of the URLs that the spammer needs the spam victim to
click on, which SpamAssasin 3.0 is able to do.

This is state of the art as of 1 year ago.  Only some mail servers can
take advantage of this before the SMTP transaction is over.

A properly implemented state of the art spam filtering system does not
suck, it is just the multitude of bad systems and the large amount of
people who thing that the bad systems are anything close to state of the
art that is giving spam filtering a bad reputation in some areas.

It amazes me to see NEW anti-spam systems being deployed that only use
methods that for at least the last 5 years have been proven not to work.

With a state of the art system, the average user will not see much if
any spam, and their senders will not get non-delivery notices.

There are many mail servers operating that way.

By the way, for qsl.net to have me "Just Hit Delete" instead of having
them reject the spam, would increase their cash operating costs over
300%.   qsl.net is a non-profit hobby operation that runs on donations.
  Such an increase in costs would shutdown the service.

> I get a lot of spam mails on my jhoger-e+AXbWqSrlAAvxtiuMwx3w(at)public.gmane.org
> account, since I give it out to everyone. It's my universal email address
> so I refuse to give in and try to hide it.

It is quite hidden from the gmane users :-)

If I had not removed the at-sign, gmane would have munged it yet again.

> I have my pobox filters set up to either bounce the mail immediately, or
> let it through.
> 
> That way, at least the sender *knows* that I didn't get the email since
> it is immediately bounced back to them, never stored/delayed anywhere.

Please use the term "reject" instead of "bounce".  With an reject, your
server has simply refused the message and it is the sending server that
generates the bounce.

The difference is significant because mail servers that bounce instead
of rejects end up participating in a DDOS on the domains that spammers
and viruses have forged.  Test.com and others have had their mail
servers brought down because of the volume of those bounces.

Some mail server operators still do not understand the difference and
why having their servers ever generate a bounce after the SMTP
transaction is over is now extremely bad practice.

With spam and viruses, there is no real sending server, so no bounce
message is ever generated, unless an open relay is used.  And most mail
server operators will not accept e-mail from open relays, so those
bounces are never seen.

> If you run your own MTA you can do the same thing.

Yes, unfortunately it is not an option for my area at this time.  There
are no ISPs that will provide me with such service to my home unless I
pay them to run a dedicated line to it.  From what I can see, most of
the major home broadband ISPs prohibit such servers.  A few will allow
it for an extra cost, but not the one that has a monopoly in my area.

However qsl.net and encompasserve.org will deliver a message to me or
they will use an SMTP reject to indicate why it can not be delivered.
Because they are just a mail forwarding service, that reject can also
indicate that the system that I do get the mail on is unreachable for
some reason.

The bounce message will come from the sender's mail server not from
either of my mail servers.

So if someone forges your e-mail address on a spam or a virus and tries
to send it to my e-mail address, your e-mail boxes will not get any
backscatter from either of my primary mail servers.

Most commercial mail servers reject on non-delivery.  If that were not
the case, internet e-mail would have collapsed a long time ago.

> Some people even think that the spammers remove bouncing addresses from
> lists, so it may even help to reduce the volume of spam.

In general though there is no evidence that a spammer ever removes an
address that they get non-delivery notices on.  Look up on google for
the tale of "nadine", and e-mail address that has never accepted e-mail
and is still continuing to get spam.

When people inactivate an account that gets spam, and then reactivated
it some time later and find it spam free, they think the bounces are the
reason.  It is more likely that the mail server operator improved the
spam filtering even if they do not admit to it than the spammer removed
their e-mail address from their list.

Because of people complaining about censorship and related, many mail
server operators have decided to pretend that they are not filtering
even when they are, and if they are blackholing what they detect as spam
by source I.P, there is no way for most users to know what is going on.

When someone complains, and the mail server operator sees that it is an
address they have blackholed, they will whitelist that address and tell
the user to have the sender retry, at which point the problem will be
closed as not reproducible.  I could easily do this on a corporate or
ISP mail server with no one catching on to it that I did not want to,
because I could set up scripting to whitelist domains based on watching
outgoing e-mail.

It seems many people prefer that type of service than knowing what is
really being done.  Of course it is more expensive rejecting, and it
takes longer to get the address whitelisted when the script misses it,
and some lost messages would never be noticed.

> But there's no way I could filter through it all manually. Way too much
> work...

Too high of a chance of error.  Once the SMTP transaction is over, there
is no way to reliably automatically notify the sender of real e-mail it
was not received with out being abusive to the rest of the internet.

With quarantine or tagging system, by the time that a mis-classified
message is detected if it is ever detected, substantial time may have
occurred.

-John
wb8tyw at qsl.network
Personal Opinion Only