STRUDEL · crawl preferences
Set a minimum interval
Create a plain UTF-8 file named robots.txt in your
NomadNet pages directory, beside index.mu. Keep it
non-executable. It must be served by your node at:
/page/robots.txt
For a minimum interval of fourteen days:
User-agent: Strudel
Recrawl-Interval: 14d
For a minimum interval of seven days:
User-agent: Strudel
Recrawl-Interval: 7d
To return to automatic scheduling:
User-agent: Strudel
Recrawl-Interval: auto
What it changes
Whole days from 1d through 14d are accepted.
This is a minimum gap between node visits, not
a promise to refresh each page that often.
Slow connections still receive smaller batches
on a seven-day schedule. A longer operator
interval takes precedence. Intervals above seven
days also reduce the normal batch allowance.
We attempt /page/robots.txt first on every eligible
visit, including the first visit. Settings apply
before fetching content. We do not make extra
visits just to check preferences.
A newly discovered longer interval can end that
visit after the preference check. Changes are
learned when the file is next read. You can also
contact axhoff; requests need proof that you
control the destination.
Parsing rules
User-agent: Strudel takes precedence over
User-agent: *. Field names and agent names are
case-insensitive. Blank lines and # comments
are accepted. Repeated matching groups use the
largest valid numeric interval. auto applies
only if there is no valid numeric interval.
Missing or invalid settings keep the last valid
preference. Without an earlier preference,
automatic scheduling applies. Use auto explicitly
to clear an earlier preference. Zero, negative,
fractional and values above 14d are invalid.
The preference response is limited to 4 KiB and
counts toward the visit's request, size and time
limits. A silent missing handler gets a short,
bounded lookup; there is no repeated lookup in
the same visit. Failed connections end the visit.
Scope
Recrawl-Interval is a Strudel-specific extension
using robots-style syntax. It is not a standard
robots directive. Crawl-delay, Allow and Disallow
are not supported by this preference format.
This controls contact frequency, not inclusion.
Pages remain searchable; full exclusion is not
offered.
Contact