AI Run Log
A record of every model call your application makes, one run per logical call and one attempt per provider leg, with the usage and cost each reported.
On this page
Quick Reference#
| Item | Value |
|---|---|
| Feature | Features::aiRuns(), then php artisan magic-starter:install --features=ai-runs and php artisan migrate. |
| Contract | AnilcanCakir\MagicStarter\Contracts\RecordsAiRuns; your code makes the call and hands it plain values. |
| Retention | ai_runs.retention_days, 30 by default; the prune blanks bodies and keeps the metadata. |
| Abandoned | ai_runs.abandon_after_minutes, 30 by default, closes a run that lost its caller. |
| Prune | php artisan magic-starter:ai-runs:prune, hourly while the feature is on. |
| Panel | AiRunResource and AiRunStatsWidget, under the key ai_runs. |
Introduction#
The AI run log records every model call your application makes: one run per logical call, one attempt per provider leg (a retry or a fallback is its own attempt), with the usage and cost each reported and, until the prune blanks them, the prompt and the answer. It answers "what did this feature spend, and why did it fail" after the request is gone.
The package never calls a model and has no laravel/ai dependency. Your code makes the call and hands the log plain values through one contract, AnilcanCakir\MagicStarter\Contracts\RecordsAiRuns.
Feature: Features::aiRuns(). While it is off, start() answers null, nothing is written, and no prune, panel resource or widget exists.
Enabling#
// config/magic-starter.php
'features' => [
Features::aiRuns(),
],
Publish the two migrations, create_ai_runs_table.php and then create_ai_run_attempts_table.php, and run them:
php artisan magic-starter:install --features=ai-runs
php artisan migrate
The installer stamps the runs table first because the attempts table references it. A host that upgrades without migrating loses nothing: the recorder skips a missing table and logs one warning per process.
Configuration#
| Key | Default | Meaning |
|---|---|---|
ai_runs.retention_days |
30 |
Days a run and its attempts keep their bodies before the prune blanks them; at least 1 |
ai_runs.abandon_after_minutes |
30 |
Minutes after which a run still running is closed as abandoned; at least 1 |
ai_runs.max_body_bytes |
262144 |
Bytes one stored string may hold; the rest is cut |
ai_runs.mask.extra_patterns |
[] |
Your own regular expressions, applied after the built-in ones |
ai_runs.mask.extra_keys |
[] |
Your own key names, masked like the built-in ones |
Every key is optional. The package merges its config shallowly, so a published ai_runs block replaces the package one whole, and the three numbers fall back to the defaults above when the block leaves them out. The masking defaults are not config at all: they live in SecretMasker and always apply, so a published block can add to them but never switch them off.
Recording a Run#
RecordsAiRuns has three methods, called in this order around one model call:
| Method | Purpose |
|---|---|
start(AiRunStart $start): int|string|null |
Opens a running run and answers its key |
attempt(int|string|null $run, AiRunAttemptData $attempt): void |
Records one finished provider call; attempt counts from 1 and is unique within the run |
finish(int|string|null $run, AiRunOutcome $outcome): void |
Closes the run and folds provider, model, usage and cost from its attempts |
namespace App\Services;
use AnilcanCakir\MagicStarter\Contracts\RecordsAiRuns;
use AnilcanCakir\MagicStarter\Enums\AiCostSource;
use AnilcanCakir\MagicStarter\Enums\AiRunStatus;
use AnilcanCakir\MagicStarter\Support\AiRuns\AiRunAttemptData;
use AnilcanCakir\MagicStarter\Support\AiRuns\AiRunOutcome;
use AnilcanCakir\MagicStarter\Support\AiRuns\AiRunStart;
use AnilcanCakir\MagicStarter\Support\AiRuns\AiUsage;
use Throwable;
class IncidentAnalyst
{
// ModelGateway and the answer it returns are your own; the package never sees a model.
public function __construct(
private readonly RecordsAiRuns $runs,
private readonly ModelGateway $gateway,
) {}
public function analyse(Incident $incident, string $instructions, string $question): string
{
$began = hrtime(true);
// Open the run before the call, so a crash leaves a `running` row for the prune to close.
$run = $this->runs->start(new AiRunStart(
feature: 'incident.analysis',
teamId: $incident->team_id,
subjectType: $incident->getMorphClass(),
subjectId: $incident->getKey(),
promptHash: hash('sha256', $instructions),
model: 'claude-sonnet-4-5',
instructions: $instructions,
input: $question,
));
try {
$answer = $this->gateway->ask($instructions, $question);
} catch (Throwable $exception) {
$this->runs->attempt($run, new AiRunAttemptData(
attempt: 1,
provider: 'anthropic',
model: 'claude-sonnet-4-5',
status: AiRunStatus::FAILED,
errorClass: $exception::class,
errorMessage: $exception->getMessage(),
));
$this->runs->finish($run, new AiRunOutcome(
status: AiRunStatus::FAILED,
errorClass: $exception::class,
errorMessage: $exception->getMessage(),
durationMs: $this->elapsedMs($began),
));
throw $exception;
}
$this->runs->attempt($run, new AiRunAttemptData(
attempt: 1,
provider: 'anthropic',
model: $answer->model,
status: AiRunStatus::SUCCEEDED,
finishReason: $answer->finishReason,
usage: new AiUsage(
inputTokens: $answer->inputTokens,
outputTokens: $answer->outputTokens,
cacheReadTokens: $answer->cacheReadTokens,
cacheWriteTokens: $answer->cacheWriteTokens,
),
costUsd: $answer->costUsd,
costSource: AiCostSource::PROVIDER,
));
$this->runs->finish($run, new AiRunOutcome(
status: AiRunStatus::SUCCEEDED,
output: $answer->text,
durationMs: $this->elapsedMs($began),
));
return $answer->text;
}
private function elapsedMs(int $began): int
{
return (int) ((hrtime(true) - $began) / 1_000_000);
}
}
The run is opened as running, and finish() writes the status you pass: succeeded or failed. AiRunOutcome and AiRunAttemptData accept only those two and throw an InvalidArgumentException for any other status.
The host never branches on the recorder. A null run key (feature off, table missing, or a failed write) makes attempt() and finish() no-ops, so the three calls stay the same. The recorder never throws into your call path:
- Every value is made storable before it is written. Invalid UTF-8 is scrubbed and NUL bytes are stripped from every string,
NANandINFbecomenull, an object becomes the array its JSON form decodes to, andfeature,provider,model,finishReason,errorClassand the other names are cut to the width of their column. - Every write builds its row and runs in its own savepoint, so a failed log row cannot roll back your transaction on PostgreSQL. A
QueryException, aJsonEncodingExceptionor aDeadlockExceptionloses that one row and is logged at error level; a missing table is skipped with one warning. Anything else is a bug and propagates. - The one failure that does throw is configuration: an
extra_patternsentry that does not compile stops the application from booting, with a message naming the pattern, instead of failing every recorded call.
AiRunStart also takes source (AiPromptSource::CODE unless an override supplied the prompt, then AiPromptSource::OVERRIDE), context (named blocks sent with the prompt) and tools (the tool names offered). An attempt takes toolCalls, a list of name, arguments and result.
The ai_runs table also holds a cache_hit status and a source_run_id column for a run that reused another run's answer. hit() writes them.
Recording a Cache Hit#
When your own response cache answers a call, record that nothing was spent with hit():
public function hit(AiRunStart $start, int|string $sourceRunId): int|string|null
$cached = $this->answers->find($hash);
if ($cached !== null) {
$this->runs->hit(new AiRunStart(
feature: 'incident.analysis',
teamId: $incident->team_id,
promptHash: $hash,
), sourceRunId: $cached->run_id);
return $cached->text;
}
$sourceRunId is the key of the run that produced the answer you served; keep it next to the cached answer when you store it. The row is written already closed: status cache_hit, source_run_id set, every token count 0, cost_usd 0, no cost source, and a zero duration. It has no attempts, so there is nothing to attempt() or finish().
hit() answers the new run's key, or null when the feature is off, the table is missing or the write failed, and it never throws, on the same terms as start(). A custom RecordsAiRuns implementation has to provide it.
Tokens and Cost#
AiUsage::$inputTokens includes the cache reads and cache writes: they are subsets of it, the convention laravel/ai and the OpenTelemetry GenAI conventions use. A total is input + output, never input + cacheRead. A count the provider did not report stays null instead of becoming 0, and a negative count throws.
finish() folds the run from its attempts:
- Token counts are summed, and a total stays
nullwhen no attempt reported it. providerandmodelcome from the last attempt that succeeded. When none did,modelkeeps the value fromstart()andproviderstaysnull.attempts_countis the number of recorded attempts.
cost_usd is a decimal string in US dollars with eight places at most, such as 0.00042; never pass a float. The run's cost is the sum of its attempts, added in whole units of 1e-8 dollars so no float rounds it.
costSource is an AiCostSource: PROVIDER when the provider reported the charge, COMPUTED when you priced the reported tokens yourself, UNPRICED when the call had no known price. The run keeps the source its costed attempts share and reads MIXED when they differ.
Masking#
Every body is masked and then capped when it is written, so a prompt that quoted a header never reaches the log in clear. Bodies are the run's instructions, input, context, output, structured and errorMessage, and an attempt's errorMessage and tool call arguments and result. Tool names and the other metadata are stored as given.
SecretMasker works in two passes:
- On an array, every entry whose key is a sensitive name (compared case-insensitively) has its whole value replaced, whatever its type. The other entries are walked.
- On every string, each pattern replaces its named group
secretwhen it has one, and its whole match otherwise.
The replacement is [redacted]. The defaults live in code, as SecretMasker::DEFAULT_PATTERNS and SecretMasker::DEFAULT_KEYS, and always apply:
- The patterns cover OpenAI and Anthropic style
sk-keys, Stripesk_live_,sk_test_andrk_live_keys, GitHub, AWS, Google and Slack keys, JWTs,BearerandBasicvalues (also writtenBearer:), credentials inside a url (an empty user and a password holding@included), secret query parameters (api_key,access_token,refresh_token,id_token,client_secret,passwordand more) and PEM, OpenSSH and PGP private keys. A private key pasted without itsENDline is masked to the end of the text. - The key names are
authorization,proxy-authorization,password,passwd,secret,token,auth_token,x-auth-token,api_key,api-key,apikey,x-api-key,access_token,refresh_token,id_token,client_secret,private_key,cookieandset-cookie. They apply to array keys and to text:password=hunter2,"token": "...",X-Api-Key: ...andAuthorization: Token ...all keep the name and lose the value. A name glued to a longer word, such astokens:, is not one.
The masker fails closed: a string the regex engine gives up on is replaced whole, because returning it would leak exactly the oversized body most likely to carry a pasted secret.
Add your own through extra_patterns and extra_keys, which extend the defaults and can never shorten them. Each pattern is a PCRE pattern with its delimiters; use a (?<secret>...) group to keep the surrounding text. A pattern that does not compile stops the application from booting, with a message naming it:
// config/magic-starter.php
'ai_runs' => [
'mask' => [
'extra_patterns' => [
'/\bacme_live_[A-Za-z0-9]{24}\b/',
'/customer-token:\s*(?<secret>\S+)/i',
],
'extra_keys' => [
'x-tenant-secret',
],
],
],
Each string is first cut to max_body_bytes plus 64 KiB, so masking a multi-megabyte body stays fast, then masked, then cut to max_body_bytes on a character boundary with a [truncated N bytes] marker that counts every byte missing from the original. Masking comes before the final cut, and the extra 64 KiB keeps a secret that crosses the cap whole while the patterns read it. In an array body this applies to each string in it.
Pruning#
php artisan magic-starter:ai-runs:prune
php artisan magic-starter:ai-runs:prune --days=7
The command exists, and is scheduled hourly without overlapping, only while the feature is on, so the application needs the Laravel scheduler running. Hourly, because a daily sweep would leave a dead run running for up to a day. It does two things:
- Blanks old bodies. For runs and attempts older than
retention_days(or--days, at least 1), it nulls the body columns and stampsbodies_pruned_at. A row is never deleted and its metadata (feature, status, usage, cost, durations, timestamps) is never touched, so spend and failure history outlive the prompts. It writes in chunks of 1000 keys. - Closes abandoned runs. A run still
runningafterabandon_after_minutes(at least 1) has lost its caller, for example a worker that was killed betweenstart()andfinish(). It becomesabandoned,finished_atis stamped, and its provider, model, tokens and cost are folded from the attempts it did record, exactly asfinish()would have, so its spend still counts. The stale runs are read in chunks of 1000 and each is closed with its own update, which only applies while the run is stillrunning.
A table that does not exist is skipped, since the command can run before the migrations did.
Admin Panel#
With the admin panel installed and the feature on, the plugin adds two parts behind the panel's own access gate:
AiRunResource, a read-only list and view of every run: started time, feature, status, team, provider, model, attempts, tokens, cost and duration, with filters for feature, status, team and a started date range. Status badges and the status filter use translated labels (admin_ai.statuses.*, English and Turkish). A run is a reading of what a call did, so there is no create, edit or delete.AiRunStatsWidget, on the dashboard: for the last 7 and 30 days, the run count, the failed share, the cost and the cache hit rate. It does not poll; reload the page for fresh figures. The rate iscache_read_tokensoverinput_tokens, taken only over the runs that reported both figures; with no such run, or no run at all for the failed share, it shows a dash.
The resource is registered under the key ai_runs, so MagicStarterPlugin::make()->withoutResources(['ai_runs']) drops it like any other.
Related Documentation#
- AI Prompt Overrides: the prompt source and hash a run records.
- Admin Panel: the panel that hosts the run resource.
- Configuration:
schedule.ai_runs_prune.