The Berkeley Function Calling Leaderboard: what a BFCL score means

The short answer

The Berkeley Function Calling Leaderboard (BFCL) measures how accurately language models call functions, or tools. Its first version graded about 2,000 question-function-answer pairs across Python, Java, JavaScript, REST and SQL, mainly by parsing each call and checking the function name, required parameters, types and values against an expected answer.¹ Later versions added relevance detection, where the right answer is to call nothing, and multi-turn tasks scored by the state of a simulated backend.¹²

A high BFCL score tells you a model reliably produces well-formed, correct tool calls in those tests. It doesn’t tell you whether a correct-looking call is safe to run in your system. That’s a different layer.

What is in it

The original blog describes the data: 1,680 Python cases, 100 Java, 50 JavaScript, 70 REST API and 100 SQL.¹ The Python cases are split by shape:

Version 3, released on 19 September 2024, added multi-turn conversations, with extra categories for missing parameters, missing functions, long context, and a composite of all three, 200 cases each.² The current leaderboard describes version 4 as holistic agentic evaluation, and its overall accuracy is the unweighted average of all sub-categories.²

AST matching: how a call is graded

Most single-turn cases are graded by parsing the model’s call into an abstract syntax tree and comparing it with possible answers. The rules are strict:¹

1. The function name must match.
2. Every required parameter must be present.
3. Parameters that aren’t in the function’s documentation count as hallucinations.
4. Types and values are checked strictly; a boolean must be a boolean.
5. For multiple or parallel calls, it’s all or nothing: every expected call must match.

Where the function can actually run, executable evaluation complements the AST check.¹

A worked example. Offered get_weather(city: string, unit: "C" | "F") and asked “What’s the weather in Pune in Celsius?”, these are graded:

The third row is the interesting one for safety: a model that invents parameters is a model that will sometimes send arguments your tool doesn’t expect.

Strict grading can look harsh: a call that is “nearly right” scores zero. For tool use, that’s the correct standard. A tool doesn’t run a nearly right call nearly correctly. It either rejects it, which wastes a turn, or runs it with the wrong meaning, which is worse. A string where a number was expected, a missing unit, a guessed parameter name: each is the kind of small error that becomes a wrong refund or a query against the wrong table. A benchmark that gave partial credit would hide exactly the failures that matter in production.

Knowing when not to call

Relevance detection asks the model to recognise that none of the offered functions fits the request.¹ This matters more in production than it looks. A model that always calls something will, when asked for an action it has no tool for, pick the closest tool and misuse it. “Cancel my subscription” with only a delete_account tool available is the kind of near-miss that turns into an incident.

Multi-turn: checking state

Version 3’s multi-turn cases run two checks at the end of every turn: a state-based check that compares the simulated backend’s state with the ground truth, and a response-based check of the model’s execution path. An entry counts only if it passes both, in every turn.²

That state-based check is the same idea τ-bench uses: grade the effect, not the words.³ And the version 3 post makes an observation worth quoting for anyone deploying agents: even the best models sometimes fail to explore the current state before performing actions, which can be dangerous when actions aren’t reversible.²

What a BFCL score doesn’t cover

This isn’t a criticism of BFCL. It measures tool-calling skill precisely. But a call can be perfectly formed and still be the wrong thing to do: refunding the right order for the wrong amount, or deleting the right table at the wrong time.

Using it to choose a model

1. Use BFCL to shortlist models that call tools accurately in your style: parallel calls, many functions, multi-turn.
2. Check relevance scores if your agent has a narrow tool set. You want a model that says “I can’t do that”.
3. Test on your own tool definitions. Names, descriptions and argument shapes differ from the benchmark’s. Twenty real requests with expected calls make a useful private benchmark.
4. Don’t treat accuracy as permission. Put a tool-call policy between the model and anything that writes, so a correct-looking call still has to satisfy your rules.

For the frameworks’ own hooks where such a policy can live, see nine frameworks’ guardrails compared. For reliability across repeated tries, see τ-bench and pass^k; for coding agents, what SWE-bench measures. To see a policy decide each call on a real database in your browser, try the playground; to grade what an agent can reach on your machine, run npx deedglass scan (the grade, explained).

Written with AI assistance and checked against our claims register.