CharacterizationEvaluator

class obsidian.campaign.characterization.CharacterizationEvaluator(campaign: Campaign, seed: int | None = None)

Bases: object

Evaluate classification performance for characterization campaigns.

Classifies regions of parameter space based on GP posterior predictions and thresholds. Optionally computes confusion matrix and Jaccard when ground truth is available.

Parameters:

• campaign – Fitted campaign with thresholds set on targets

• seed – Random seed for Sobol sampling

__init__(campaign: Campaign, seed: int | None = None)

Methods

__init__(campaign[, seed])
classify_confidence_levels(X[, target_names])	Classify points into 4 confidence levels per active target.
classify_points(X[, PI_range, ...])	Classify points based on GP predictions and thresholds.
compute_largest_hypercube(X[, PI_range, ...])	Compute largest inscribed hypercube within feasible region.
evaluate_with_ground_truth(X, y_true[, PI_range])	Compute confusion matrix and Jaccard index for benchmarking.
plan_sample_size([pilot_ratio, epsilon, z, ...])	Plan appropriate sample size using Bernoulli variance on pilot samples.
summarize_confidence(X)	Pass / fail / classified fractions across the standard CI ladder.

Attributes

targets

Active (non-tracking-only) targets with thresholds.

classify_confidence_levels(X: DataFrame | int, target_names: list[str] | None = None) → dict[str, ndarray]

Classify points into 4 confidence levels per active target.

Levels (per cell, per target):

0 — Fail: mean fails the threshold 1 — Uncertain Pass: mean passes; 70% PI lower bound fails 2 — Likely Pass: 70% PI lower bound passes; 95% fails 3 — Confident Pass: 95% PI lower bound passes

Consistent with classify_points by construction: pred_mask of classify_points(PI_range=0.7) equals levels >= 2, and at PI_range=0.95 it equals levels >= 3.

Parameters:

• X – Points to classify (pd.DataFrame) OR number of Sobol samples (int).

• target_names – Restrict to this subset of active targets. The Joint key (when present) is the elementwise min across the subset. Defaults to all active targets.

Returns:

dict keyed by target name. Each value is a length-N int array of levels in {0, 1, 2, 3}. Includes a "Joint" key (elementwise min across targets) when more than one active target exists.

classify_points(X: DataFrame | int, PI_range: float = 0.7, return_samples: bool = False, target_names: list[str] | None = None) → dict

Classify points based on GP predictions and thresholds.

Parameters:

• X – Points to classify (pd.DataFrame) OR number of Sobol samples (int)

• PI_range – Prediction interval coverage (0.7 or 0.95)

• return_samples – If True, include ‘X_samples’ and ‘X_feasible’ in result. Use this when planning to compute multiple hypercubes to avoid redundant GP predictions.

• target_names – Restrict classification (and the Joint key) to this subset of active targets. Defaults to all active targets.

Returns:

dict keyed by target names, plus ‘Joint’ key if multiple targets. Each value is a dict containing pos_frac, neg_frac, classified_frac, pred_mask. Optional keys: ‘X_samples’, ‘X_feasible’ (if return_samples=True)

compute_largest_hypercube(X: DataFrame | int, PI_range: float = 0.7, center: dict[str, float] | Series | None = None, dim_weights: dict[str, float] | None = None, fixed_dims: dict[str, float] | None = None, tolerance: float = 0.001, max_iter: int = 100) → dict

Compute largest inscribed hypercube within feasible region.

Finds the largest axis-aligned hypercube that fits entirely within the Pass region. For continuous parameters only, finds single hypercube. For problems with discrete parameters, enumerates all discrete combinations and returns the one yielding the largest hypercube.

Parameters:

• X – Evaluation points (DataFrame) or number of Sobol samples (int). Recommend 1000*d samples for accuracy.

• PI_range – Prediction interval for classification (0.7 or 0.95).

• center – Optional fixed center. If None, uses centroid of feasible region. Dict format: {param_name: value} or pd.Series

• dim_weights – Relative weights for each dimension. If None, all dimensions weighted equally. Dict format: {param_name: weight}. Example: {‘x’: 2, ‘y’: 1} means x can be twice as wide as y.

• fixed_dims – Parameters to fix at specific values. Dict format: {param_name: value}

• tolerance – Convergence tolerance for binary search.

• max_iter – Max iterations for optimization.

Returns:

• ‘center’: Center point (pd.Series with param names)

• ’volume’: Hypercube volume in parameter space

• ’bounds’: Dict with ‘lower’ and ‘upper’ (pd.Series)

• ’fixed_dims’: Dict of user-specified fixed continuous dimensions

  (empty if none were fixed).

• ’categorical_values’: Dict of categorical parameter assignments,

  user-specified or chosen by enumeration (empty if the campaign has no categorical parameters).

• ’n_corners_feasible’: Number of feasible corners checked

• ’convergence_iters’: Number of iterations to converge

Return type:

dict with

Raises:

ValueError – If no feasible hypercube exists or conflicting constraints.

evaluate_with_ground_truth(X: DataFrame, y_true: ndarray, PI_range: float = 0.7) → dict

Compute confusion matrix and Jaccard index for benchmarking.

Parameters:

• X – Points to evaluate

• y_true – Ground truth values, shape (n_points,) or (n_points, n_active_targets)

• PI_range – Prediction interval coverage

Returns:

dict with per-target and joint Jaccard scores and confusion matrices

plan_sample_size(pilot_ratio: float = 0.1, epsilon: float = 0.01, z: float = 1.96, max_samples: int = 100000) → int

Plan appropriate sample size using Bernoulli variance on pilot samples.

Parameters:

• pilot_ratio – Fraction of max_samples to use as pilot

• epsilon – Desired precision for fraction estimates

• z – Z-score for confidence level (1.96 for 95%)

• max_samples – Maximum allowed samples

Returns:

Recommended sample size (at least n_pilot, at most max_samples)

summarize_confidence(X: DataFrame | int) → dict[str, dict[str, float]]

Pass / fail / classified fractions across the standard CI ladder.

Returns:

{“pass_mean”, “pass_70”, “pass_95”,

”fail_70”, “fail_95”, “classified_70”, “classified_95”}}``

with values in [0, 1]. Includes a "Joint" entry when there is more than one active target.

Return type:

``{target_name

property targets

Active (non-tracking-only) targets with thresholds. Re-evaluated on each access.