[dnswl-users] dnswl.org - calling for your support

[Matthias Leisi <matthias@xxxxxxxxx>]

[This article is cross-posted to the two blogs at http://news.dnswl.org/ 
and http://matthias.leisi.net/ and to the dnswl-users mailing list 
[http://www.dnswl.org/mailinglist]]


Currently, most of dnswl.org is run by myself with the help of some 
volunteers ("backup" administrators should I ever be run over by a bus) 
and some organisations [http://www.dnswl.org/thanks].

dnswl.org has gained traction since it's inception in October 2006, and 
it's data is now being used by a number of anti-spam solutions (eg 
filtering applications, blocklist providers, reputation aggregation 
services). It has reached a point where more resources are required. 
Since a commercial business model may create unfavourable incentives 
(see the last section of this [http://www.dnswl.org/background] for some 
rationale), I try to stick to a volunteer-based organisation. The 
volunteer, that would be you.

Please let me briefly explain how dnswl.org works so that you can better 
understand the areas where you could be of help.


How dnswl.org works
-------------------

dnswl.org has basically three parts:

1) Data for dnswl.org (ie, identifying domains and their associated IP 
addresses used to send email) comes from log files, requests through the 
website and by mail, and, most importantly, from various import sources 
(all data is manually verified before being added for distribution).

2) Managing the data: We aggregate IP addresses / network ranges by 
"owner". The owner is signified by a domain (with multiple secondary 
domains possibly added). All IP addresses are regularly checked for RBL 
"appearances" and DNS inconsistencies. Over time, scores may be adjusted 
up- or downwards, categories re-assigned, and IP addresses may be added 
or removed. This is usually based on one of the regular checks above or 
when we receive feedback.

3) Using the data: We export data in various formats (for rbldnsd and 
BIND nameservers, in formats usable by Postfix and Lotus Notes, and some 
others), and distribute it via DNS, rsync and HTTP (the latter transfer 
will be discontinued at a not-yet-determined point in time, since it is 
highly inefficient for that purpose).


These things are glued together by a public website (mostly static), a 
request tracker (RT3), a web-based administration interface (based on 
PHP and MySQL), a number of batch jobs (Perl) and standard Unix-based 
tools (rsync, rbldnsd, mrtg, Apache etc).


What should be improved
-----------------------

Given the increased use [http://www.dnswl.org/mrtg/dnswl.combined.html] 
of dnswl.org, the number and (net-) geographical distribution of 
nameservers should be increased. Specifically, DNS mirrors in the 
following locations would be great (but other mirrors would help as well):

* US east coast
* Northern Europe
* Asia/Pacific

CPU and IO load for typical a rbldnsd-based nameserver are minimal (can 
most likely be run in a VMWare/Xen/however virtualised environment), but 
it has some bandwidth requirements (a couple of GB per month).

A DNS mirror must be able to regularly update it's data via rsync, and 
preferrably run rbldnsd (generic DNS servers such as BIND can basically 
be used as well, since our data is rather small -- it will hardly every 
be more than 100k entries). Ideally, a nameserver will do minimal 
logging ("-s" switch to rbldnsd) and regularly run a little Perl script 
to participate in the statistics 
[http://www.dnswl.org/mrtg/dnswl.combined.html]

Handling the automated import queue and the manual request queue would 
be a full-time job. Additional volunteers to manage data that can spend 
at least an hour or two per week would allow to speed up the whole 
process. It would also be worthwhile to have someone chasing large 
providers (of the likes of Yahoo, Postini et al.) to let us know their 
full ranges of mail-sending IP addresses. Possible tasks/focus:

* Working on the import and request queues
* Manual additions (eg from logfiles or personal knowledge)
* Establish relations with large providers
* Review missing/incomplete/old data (eg assign categories, cleanup 
CIDRs, ...)

Volunteers must be trustworthy individuals, and preferrably have a 
background and/or current work experience in handling spam or security 
issues in general.

Having more volunteers for the queues would allow myself to spend more 
time on the improvement of the architecture and implementation. Can you 
spell "historically grown codebase"?  ;)   Some clearly definable work 
could well be done by a knowledgeable person, but it may be 
difficult/take a lot of time for a new person to learn the overall 
environment. Possible tasks include:

* Regular RBL checks for large ranges, where it is not feasible to query 
each individual IP address at each "big" blacklist.
* ASN/riswhois/other checks for large ranges
* Automated feedback loop on spam from dnswl.org-listed sources, eg as a 
plugin for popular spamfilters or something similar.
* Improve features for domain-level whitelisting.

All dnswl.org features are built on Perl (for regular jobs etc) and PHP 
(the admin interface), so some knowledge in either or both of them is 
required (however, rewriting certain parts in eg Java/JSP could be 
considered).

One can probably tell from looking at the public website that no 
talented designer has been involved yet. The following seem the most 
obvious tasks for improvements in design:

* Create a logo for dnswl.org (hey, dnswl.org should be present in the 
logo-bar on karmasphere.com/ [http://www.karmasphere.com/] as well!)
* General design, typographics etc for the public website.
* Since I'm not a native english speaker, the wording and general 
content of the public website may use some improvement.


The future of dnswl.org
-----------------------

I do not regard dnswl.org as an ego-project. It is my goal to establish 
the project as a long-term, stable endeavour. If you are willing and 
able to take responsibility for some parts of dnswl.org, you are more 
than welcome to participate. It's no problem if you want to start small 
-- and grow your duties and responsibilities over time, if you wish to 
do so.

Besides the additional help mentioned above, I intend to evolve and 
stabilise the overall organisation over time. I'd be glad to share and 
pass on responsibility.

I'm convinced that "enumerating goodness" (as opposed to the 
"enumerating badness" still prevalent in today's world of fighting spam) 
will become more important over the coming years, and you can help make 
it work!

-- Matthias