Skip to content
Botscent

Measure agent traffic

Count the share of agent visits, the visits of each agent and each agent's top pages from your log.

Count the agent visits in your verdict log. The counts show how much of your traffic comes from agents, and which agents read which pages.

Before you start

  • Log verdicts for a week, as Log verdicts for a week describes. The log has one request line for each page load and one page line for each page where the page half saw an agent.
  • Copy the log into one file, such as verdicts.log. Lines that are not JSON can stay in the file.
  • Use Node.js 22.18 or later. It runs a TypeScript file without a build step.

1. Write the count script

In count.ts, read the log and count the agent visits:

count.ts
import { readFileSync } from 'node:fs'

type Line = {
  kind: 'request' | 'page'
  path: string
  type: 'agent' | 'human'
  agent_name?: string
  reasons: string[]
  request?: 'agent' | 'human'
  transport?: string
}

const lines: Line[] = readFileSync(process.argv[2], 'utf8')
  .split('\n')
  .filter((text) => text.startsWith('{'))
  .map((text) => JSON.parse(text))

const loads = lines.filter((line) => line.kind === 'request')
const visits = [
  ...loads.filter((line) => line.type === 'agent'),
  ...lines.filter((line) => line.kind === 'page' && line.request === 'human' && line.transport !== 'received'),
]

function print(title: string, keys: string[]) {
  const counts = new Map<string, number>()
  for (const key of keys) counts.set(key, (counts.get(key) ?? 0) + 1)
  console.log(`\n${title}`)
  for (const [key, count] of [...counts].sort((a, b) => b[1] - a[1]).slice(0, 10)) {
    console.log(`${String(count).padStart(6)}  ${key}`)
  }
}

const name = (line: Line) => line.agent_name ?? '(no name)'
const share = ((100 * visits.length) / loads.length).toFixed(1)
console.log(`${loads.length} page loads, ${visits.length} by agents (${share}%)`)
print('Agents', visits.map(name))
print('Pages', visits.map((line) => `${name(line)} ${line.path}`))
print('Reasons', visits.flatMap((line) => line.reasons))

The script counts each agent visit once. A request line counts when its verdict is an agent. A page line adds a visit only when the request's own verdict was human and the page got no verdict from the server.

2. Run the script

Run the script with the log file:

Terminal
node count.ts verdicts.log

The script prints the share of agent visits, then the top 10 agents, pages and reasons.

3. Read the counts by agent name

Count visits by agent_name, not by user agent. One name covers every kind of evidence for one agent. For example, grok-bot comes from Grok Bot's signature and from its page evidence. Agents lists every name, and botscent/names.json gives each name's vendor and kind.

Visits with no name are agents that Botscent found but could not name, such as a browser with the webdriver flag. Keep them in the share of agent visits.

4. Check the reasons

The reasons show which evidence finds your agents: a signature, a user-agent token or a page marker. A minor release can add a reason or a name. See Versions and stability.

So your counts can change after an upgrade, with no change in your traffic. The release notes list each new reason. Count the visits that hold only the new reasons, and you see how much of the change the release made.

Result

You have the share of page loads by agents, the visits of each agent, and each agent's top pages. The output looks like this, with your own numbers:

Output
1840 page loads, 152 by agents (8.3%)

Agents
    71  chatgpt-user
    38  claude-chrome
    24  chatgpt
    19  (no name)

Pages
    30  chatgpt-user /pricing
    17  claude-chrome /docs
    12  chatgpt /pricing

Reasons
    71  ua.declared-agent-token
    38  page.claude.marker.active
    24  signer.web-bot-auth.verified
    19  page.browser.webdriver-flag

Each count is a floor. human means that Botscent saw no agent evidence, so an agent that shows none counts as a person. An agent that signs only its page loads can count twice when the transport is off and the page half also sees it.

Next steps