SeeTheWork

Synthetic sample / No candidate data

See what the reviewer receives.

Choose a role. Trace a finding to the work, inspect what the checks establish, and find a useful follow-up question. All four briefs use fictional examples; they contain no candidate data.

SeeTheWork / Evidence brief

ML engineering

Synthetic sample

Simulation
Investigate a suspicious model improvement
Work product
Evaluation + model report
Evidence sources
4 illustrative excerpts

Reviewer summary

Leakage is addressed. Deployment behavior remains untested.

The sample identifies entity overlap, rebuilds a group-separated evaluation, and compares a baseline under the same conditions. Production suitability remains a separate question. [04]

Competency evidence

Identifies leakage in evaluation

Supported in this example

[01][02]

Compares models consistently

Supported in this example

[03]

Plans for distribution changes

Follow-up needed

[04]

Source register

All excerpts below are illustrative, including filenames and check results. They explain the review method and contain no real candidate data.

  1. [01]

    Split audit

    Illustrative

    Look for shared entities.

    split_audit.py

    overlap = set(train.customer_id) & set(test.customer_id)
assert len(overlap) == 0
# Original split: assertion fails.
    What it supports
    The audit exposes an entity shared between training and evaluation data.
    What remains open
    No entity overlap alone does not rule out temporal or feature leakage.
    Back to competency evidence
  2. [02]

    Pipeline excerpt

    Illustrative

    Keep training and evaluation separate.

    evaluate.py

    train, holdout = group_split(customer_id)
pipeline.fit(X[train], y[train])
prediction = pipeline.predict(X[holdout])
    What it supports
    The illustrative pipeline fits preprocessing and the model on training records only.
    What remains open
    A reviewer would inspect feature construction and the full split implementation.
    Back to competency evidence
  3. [03]

    Comparison protocol

    Illustrative

    Use one comparison protocol.

    evaluation.md

    Baseline and candidate: same held-out groups
Primary metric: MAE
Report: overall error + error by customer segment
Holdout tuning: not permitted
    What it supports
    The protocol makes the model comparison and reporting scope explicit.
    What remains open
    A single holdout cannot establish stability across future distributions.
    Back to competency evidence
  4. [04]

    Model handoff

    Illustrative

    Keep deployment questions visible.

    model_report.md

    “The revised comparison removes shared customers.
Next: evaluate a future time window.
Drift thresholds are not yet specified.”
    What it supports
    The handoff acknowledges the difference between this evaluation and future behavior.
    What remains open
    Drift thresholds, operational costs, and rollout controls require follow-up.
    Back to competency evidence

Suggested follow-up question

How would you evaluate this model when new entities and data drift appear?

Evidence for human review
Not an automated verdict

What would evidence like this help your team understand about its next hire?

Try this role’s 3-minute walkthroughDiscuss a pilotHow it works