Skip to main content
bestagency.guide
Source and crawler policy

How the agency index is built.

We find possible listings, check official websites and review every record before it is published. The directory does not claim to include every Australian agency, and being listed is not an endorsement.

Bot: BestAgencyGuideBot/0.1Two-second minimum host delayNo login or CAPTCHA bypass
Crawler conduct

How the crawler behaves

The crawler limits requests to each website, stops after a small number of failed attempts and does not retry sources that block it.

The crawler will
  • Identify itself and link back to this page.
  • Follow the most specific applicable robots rule and crawl delay.
  • Prefer canonical pages, sitemaps and high-value service or contact pages.
  • Record retrieval time, HTTP state, source type and supporting field.
  • Suppress opted-out domains and stale or conflicting facts.
The crawler will not
  • Sign in, defeat bot protection, solve CAPTCHAs or bypass access controls.
  • Scrape reviews, employee biographies, personal emails or personal phone numbers.
  • Copy another directory’s descriptions, rankings, prices or creative assets.
  • Auto-publish pricing, performance outcomes, client claims or quality scores.
  • Turn membership, accreditation or panel status into an endorsement.
Before a listing goes live

A crawler result is only a lead

  1. Resolve duplicate brands, offices, rebrands and canonical domains.
  2. Confirm an Australian presence and at least one service through public evidence.
  3. Attach a URL and checked date to every displayed factual field.
  4. Add contacts, specialisms, team bands, prices and work only when a direct source supports the exact field.
  5. Label agency-published work separately from independently reviewed outcomes; leave absent facts unknown.
  6. Pass data checks and a human review before the record enters the live finder.

Need a correction or crawl opt-out?

Select the canonical agency profile, state your authority and identify the affected domain or URL. Submission is reviewed and does not remove a listing or change crawler controls automatically.

Submit an opt-out request