Difference between revisions of "Granule Acceptance"
(→Visit Counting) |
(→Visit Counting) |
||
| Line 9: | Line 9: | ||
= Visit Counting = | = Visit Counting = | ||
| − | The first default criterium: granules are not | + | The first default criterium: by default, granules are not accepted twice by the same adapter: only one 'visit' is allowed. This behavior can be overridden. |
In some of the more complex personalities, you might see 'adapter loops': networks of adapters where the output of an adapter further down the data flow feeds back into the input of an adapter earlier in the data flow. These loops will often rely on the 'don't accept twice' mechanism to avoid getting caught into endless loops. | In some of the more complex personalities, you might see 'adapter loops': networks of adapters where the output of an adapter further down the data flow feeds back into the input of an adapter earlier in the data flow. These loops will often rely on the 'don't accept twice' mechanism to avoid getting caught into endless loops. | ||
Revision as of 22:53, 29 December 2013
An important mechanism in Crawler is the idea of 'granule acceptance' by adapters.
When a granule is presented to any adapter for processing, the adapter can accept or reject the granule based on a number of criteria.
Some of these criteria are part of the default infrastructure of Crawler, and are checked automatically. However, these automatic criteria can always be overruled by a particular type of adapter or adapter network.
The default criteria are only there for convenience: they will be 'the right thing' in most cases. They can be adjusted for the more uncommon cases where the acceptance criteria need to be different.
Visit Counting
The first default criterium: by default, granules are not accepted twice by the same adapter: only one 'visit' is allowed. This behavior can be overridden.
In some of the more complex personalities, you might see 'adapter loops': networks of adapters where the output of an adapter further down the data flow feeds back into the input of an adapter earlier in the data flow. These loops will often rely on the 'don't accept twice' mechanism to avoid getting caught into endless loops.
A practical example. Below a schematic representation of the adapter network used for document conversion in Crawler:
Note that the ViewAssembler sits at the core of a number of 'adapter loops'.
The 'ViewAssembler' and 'Selector' adapters in this network have been modified to not count visits: they both allow granules to 'pass through' more than once.
On the other hand, the individual 'Processor' sub-adapters of the Selector do use the default visit counting: they only allow one visit. That means that once a granule goes round one of the loops, it'll go back through the ViewAssembler, then the Selector.
The Selector will not send it back to the same Processor adapter because that processor will reject it: it has 'seen' that granule before, and it only allows one visit. That means that the Selector will go down its list of options, and pick the next eligible adapter, and if there aren't any more, it'll pick the Output adapter.
The visit counting is used to set up a mechanism where granules go round and round the network, but take a different path every time.
Each and every granule that ever roams through the network gets assigned a unique identifier when it is created. Once created, a granule never changes: all that can happen to it is that it can be dropped from the data flow, and/or replaced by one or more new granules with different identifiers.
Because of this unique identifier, adapters are able to track how many times they've seen a particular granule.
