Philosophy

Crawler is an attempt to find a good balance between complexity and flexibility.

Crawler is meant to be 'pokeable': it's OK to poke around in the configuration files. Often, all that is needed to get the desired results is some educated 'poking around'.

The aim of the system's design is to allow a non-experienced user to confidently reconfigure a Crawler personality without needing to digest massive amounts of documentation beforehand.

Crawler's configuration files, templates and formula files are all simple text files.

Users are encouraged to open them and make small modifications to them, and should, in most cases, get the expected results.

It is OK to make changes to a configuration file without fully understanding it; most often, Crawler will behave as expected and produce the expected results.

Users that desire total control or desire to build a custom personality from scratch cannot avoid having to study the Crawler documentation, and spend substantial time trying and experimenting. But unlike many other complex systems, many basic reconfigurations can be performed without much need for a manual.