ValidationReport

ValidationReport

Define ValidationReport class here.

ValidationReport#

class orbax.export.validate.validation_report.ValidationReport(baseline, candidate, option=None)[source][source]#

Generate validation report based on ValidationSingleJobResult.

This class analyzes the execution results from the baseline (JAX) and candidate (TF SavedModel) runs, comparing their outputs based on configured tolerances. It computes latency percentiles and numerical divergence to assign a final Pass/Fail status.

Example

Configure tolerances and generate a validation report:

from orbax.export.validate.validation_report import ValidationReport,
ValidationReportOption

# Assume `baseline_result` and `candidate_result` were generated by
ValidationJob

# Loosen the floating-point tolerances for validation
options = ValidationReportOption(
    floating_atol=1e-5,
    floating_rtol=1e-5,
    print_debug_info=True
)

report = ValidationReport(
    baseline=baseline_result,
    candidate=candidate_result,
    option=options
)

if report.status == Status.Pass:
    print("Validation Successful!")
outputs#

A dictionary mapping report types to their respective diff reports (FloatingPointDiffReport or NonFloatingPointDiffReport).

Type:

Dict[str, orbax.export.validate.validation_report.FloatingPointDiffReport | orbax.export.validate.validation_report.NonFloatingPointDiffReport]

latency#

A dictionary containing latency statistics for both the ‘baseline’ and ‘candidate’ runs.

Type:

Dict[str, orbax.export.validate.validation_report.LatencyStat]

xprof_url#

Profiling URLs mapped by ‘baseline’ and ‘candidate’.

Type:

Dict[str, str]

metadata#

Contextual metadata mapped by ‘baseline’ and ‘candidate’.

Type:

Dict[str, Any]

status#

The final validation Status (Pass or Fail).

Type:

orbax.export.validate.validation_utils.Status

__init__(baseline, candidate, option=None)[source][source]#

Generate validation result report with users config options.

Parameters:

  • baseline (ValidationSingleJobResult) – The baseline ValidationSingleJobResult.

  • candidate (ValidationSingleJobResult) – The candidate ValidationSingleJobResult. The comparing criterions will be apply on candidate.

  • option (Optional[ValidationReportOption, None]) – ValidationReport options.

Raises:

ValueError – If the baseline and candidate result trees have different structures, or if the flattened floating/non-floating arrays have mismatched lengths.

ValidationReportOption#

class orbax.export.validate.validation_report.ValidationReportOption(floating_atol=1e-07, floating_rtol=1e-07, max_non_floating_mismatch_ratio=0.01, output_report_path=None, print_debug_info=False)[source][source]#

Option for ValidationReport class.

floating_atol#

Absolute tolerance parameter for floating-point comparisons.

Type:

float

floating_rtol#

Relative tolerance parameter for floating-point comparisons.

Type:

float

max_non_floating_mismatch_ratio#

The maximum allowable ratio of mismatches for discrete/non-floating-point tensors before the validation fails.

Type:

float

output_report_path#

Optional path to save the generated report.

Type:

str | pathlib.Path | None

print_debug_info#

If True, prints detailed tensor mismatches to the logs.

Type:

bool

__post_init__()[source][source]#

check if option value is legal.

Raises:

OverflowError – raise if floating_atol < 0 or floating_rtol < 0.

__eq__(other)#

Return self==value.

__hash__ = None#
__init__(floating_atol=1e-07, floating_rtol=1e-07, max_non_floating_mismatch_ratio=0.01, output_report_path=None, print_debug_info=False)#
Contents