Re: [dnswl-users] Query rate limit

[Matthias Leisi <matthias@xxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hello list,

Sorry for the late reply, I'm currently a bit busy :/

>>>> Also, I'm trying to get a feel for the size of operation this is...
>>>> Looking at the yearly graph on
>>>> http://www.dnswl.org/mrtg/dnswl.combined.html I can see that the list
>>>> didn't really start to take off in usage until a few months ago. Is
>>>> there an explanation for this sudden increase? Are there any large users
>>>> such as ISPs or Universities that can be named?

I'm reluctant to actually name individual (corporate/institution) users.
Of course they can come out themselves and admit ;-) What I *can* say is
that not surprisingly some hosting providers are on the top of the usage
charts (eg ev1servers, theplanet, hosteurope), together with large
nameserver farms from larger providers (eg Deutsche Telekom,
interbusiness.it).

The usage started to take off when dnswl.org got included with the
default SpamAssassin ruleset. The increased usage as can be seen in the
mrtg charts is mostly due to this -- and the slope of the increase is
mostly proportionate to the speed of SpamAssassin updates around the world.

The absolute query volume may not be best indicator for dnswl.org usage.
We usually refer to the number of unique /24s that query the nameservers
in the last 24 hours (http://www.dnswl.org/mrtg/dnswl.24husers.html) as
a measure, as this is a proxy variable for "number of mail sites".

As of this week, we have around 20'000 such mail sites. Some use of
dnswl.org data is not visible in those charts; this includes
intermediaries such as karmasphere.com, logsat.com et al., but also
rsync downloads in general (and some HTTP-based download, which I want
to get rid of in the not too distant future).


>  > I even offered to be a dns mirror, and haven't heard a reply in over a
>  > month. The whitelist is still plunty small enough that just

I know, it's still hanging in the admin queue, sorry for that. (Note
that we are currently reaching a maximum of the number of NS records
that can efficiently be returned, and we will unfortunately need to
restructure our NS records before we can add new servers.)


>  > downloading the postfix, or using the rbldnsd file I think is a good
>  > way to go. I would recommend if you start doing around 100k lookups a
>  > day or more.
> 
> I'm probably somewhere around that actually. I'll leave it using the 
> dnsbl over the weekend and look into shifting it over to using the rsync 
> feed next week.

100k lookups/24hours is a good ballpark figure -- for anything above, we
prefer if you use rsync. In a typical SpamAssassin setup, this
translates into roughly 500k mails/24hours (with huge variation
depending on local setups and, especially, correct nameserver cache
configuration). This 5:1 ratio is pretty accurate for three different
mail environments, but may be vastly different for you.

It's not really easy to enforce such limits. We don't want to just shut
the "offenders" off, but it's sometimes difficult to identify a contact
and get them to change to rsync, because all you have is an IP address
and possibly a host/domain name, and it may take some time until one
get's through to a clueful contact (ev1servers and theplanet are my
current pain points in that regard).

We currently have around 100 unique /24s doing more than 100k lookups;
some of the unique /24s in fact belong together (eg ev1servers with
nameservers spread over multiple networks).

I could provide more insight, but maybe nobody is really reading that
far ;-)

- -- Matthias


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFHBmNnxbHw2nyi/okRApJcAKCtuyxEgMQ8fgej5lreEXVCfh/IWwCg1KcR
pJJnzZBqg1xLw0P1uOp+9uM=
=TVuY
-----END PGP SIGNATURE-----