Bring Your Own Model (BYOM) Config Guidelines

Matrice.ai supports the Bring Your Own Model (BYOM) feature, allowing users to integrate custom deep learning models into the platform. Below is the configuration specification and guidelines for BYOM integration.

Model Family Configuration

`family_info.json`

The model family configuration is defined in a single JSON file that contains all necessary information about the model family and its variants.

{
    "modelFamily": "string",                // Required: Name of the model family
    "modelInputs": ["string"],             // Required: List of input types (e.g., ["image"])
    "modelOutputs": ["string"],            // Required: List of output types (e.g., ["classification"])
    "modelArchs": [                        // Required: List of model architectures
        {
            "modelKey": "string",          // Required: Unique identifier in snake_case
            "modelName": "string"          // Required: Display name for the model
        }
    ],
    "description": "string",               // Required: Detailed description of the model family
    "references": ["string"],              // Required: List of reference URLs (papers, repos)
    "trainingFramework": "string",         // Required: Primary training framework
    "dataProcessing": {                    // Required: Data processing specifications
        "inputFormat": "string",           // Required: Expected input format
        "input_size": {                    // Required: Input dimensions
            "all": number,                 // Default size for all models
            "model_specific_key": number   // Optional: Override for specific models
        }
    },
    "exportFormats": ["string"],           // Required: Supported export formats
    "supportedMetrics": ["string"],        // Required: List of supported evaluation metrics
    "benchmarkResults": [                  // Required: Performance benchmarks
        {
            "dataset": "string",           // Name of the benchmark dataset
            "splitType": "string",         // Dataset split used (e.g., "val", "test")
            "metric": "string",            // Metric name
            "values": {                    // Results per model architecture
                "model_key": number
            }
        }
    ]
}

Field Descriptions

Core Information

modelFamily: The name of the model family (e.g., “ResNet”, “YOLO”)
modelInputs: Types of inputs the model accepts (e.g., [“image”])
modelOutputs: Types of outputs the model produces (e.g., [“classification”])

Model Architectures

modelArchs: List of model variants within the family
- modelKey: Unique identifier used in system references (snake_case)
- modelName: Human-readable name for display

Documentation

description: Comprehensive description of the model family
references: URLs to relevant papers, repositories, or documentation

Technical Specifications

trainingFramework: Primary framework used for training (e.g., “PyTorch”, “TensorFlow”)
dataProcessing: Input processing requirements
- inputFormat: Expected format of input data (e.g., “ImageNet”)
- input_size: Required dimensions for model input
  - all: Default size for all models
  - [model_key]: Optional model-specific overrides

Deployment & Evaluation

exportFormats: Supported export formats (e.g., [“TorchScript”, “ONNX”])
supportedMetrics: Available evaluation metrics
benchmarkResults: Performance measurements
- dataset: Benchmark dataset name
- splitType: Dataset split used
- metric: Performance metric
- values: Results per model variant

Action Configuration files

Introduction

Config files simplify the management of hyperparameters and export options by defining a structured and reusable JSON schema. They are essential for ensuring consistency and allowing dynamic behavior across different model pipelines.

Training Configuration

`train_config.json`

Example Configuration

{
    "actionType": "train_model",
    "actionConfig": [
        {
            "keyName": "batch_size",
            "valueType": "int32",
            "defaultValue": 4,
            "predefinedValues": [1, 4, 16, 32, 64],
            "minValue": 1,
            "maxValue": 1024,
            "anyValue": true,
            "modelKeys": ["all"]
        },
        {
            "keyName": "epochs",
            "valueType": "int32",
            "defaultValue": 50,
            "predefinedValues": [10, 50, 100, 150, 200],
            "minValue": 5,
            "maxValue": 300,
            "anyValue": true,
            "modelKeys": ["all"]
        },
        {
            "keyName": "learning_rate",
            "valueType": "float32",
            "defaultValue": 0.001,
            "predefinedValues": [1e-05, 0.0001, 0.001, 0.01, 0.05, 0.1],
            "minValue": 1e-05,
            "maxValue": 0.5,
            "anyValue": true,
            "modelKeys": ["all"]
        }
        // Add more parameters as needed...
    ]
}

Key Components

actionType: Defines the action (e.g., "train_model").
actionConfig: An array of hyperparameters:
- keyName: Parameter name (e.g., batch_size, epochs).
- valueType: The data type of the parameter (int32, float32, etc.).
- defaultValue: Default value if not specified.
- predefinedValues: Allowed values for the parameter.
- minValue / maxValue: Numerical bounds for the parameter.
- anyValue: Allows any value if set to true.
- modelKeys: Models applicable for the parameter.

Export Configuration

`export_config.json`

Example Configuration

{
    "actionType": "export_model",
    "actionConfig": [
        {
            "keyName": "dynamic",
            "valueType": "bool",
            "defaultValue": false,
            "predefinedValues": [true, false],
            "anyValue": false,
            "hyperparameter": true,
            "exportFormats": ["ONNX", "TensorRT"],
            "modelKeys": ["all"]
        },
        {
            "keyName": "simplify",
            "valueType": "bool",
            "defaultValue": false,
            "predefinedValues": [true, false],
            "anyValue": false,
            "hyperparameter": true,
            "exportFormats": ["ONNX"],
            "modelKeys": ["all"]
        }
        // Add more parameters as needed...
    ]
}

Key Components

actionType: Defines the action (e.g., "export_model").
actionConfig: An array of export parameters:
- keyName: Parameter name (e.g., dynamic, simplify).
- valueType: The data type (bool, string, etc.).
- defaultValue: Default value for the parameter.
- predefinedValues: Allowed values.
- exportFormats: Formats applicable for this parameter (e.g., ONNX, TensorRT).
- modelKeys: Models applicable for the parameter.

General Info

Key	Description
supportedMetrics	List of supported evaluation metrics in the platform.
exportFormats	List of supported model export formats in the platform.

Supported Metrics

These are the supported accuracy metrics available in the platform for our models - Classification and Detection. You can also use the performance metrics for your models with

metrics = actionTracker.calculate_metrics(split_type='val', outputs=model_outputs, targets=model_targets, metrics_type='classification')
actionTracker.save_evaluation_results(metrics)

Performance Metrics

Accuracy metrics used for classification and detection tasks:

Classification Metrics

Metric	Description
acc@1	Measures the percentage of times the top predicted class matches the true class.
acc@5	Measures the percentage of times the true class is within the top 5 predictions.
MCC	Matthews Correlation Coefficient; measures correlation between predicted and actual binary classifications.
AUC-ROC	Area Under the Receiver Operating Characteristic curve; evaluates model’s ability to distinguish between classes.
AUC-PR	Area Under the Precision-Recall curve; useful for imbalanced datasets.
Cohen’s Kappa	Measures agreement between predicted and actual classifications while accounting for chance.
log_loss	Logarithmic loss metric; penalizes confident but incorrect predictions more heavily.
f1_score	The harmonic mean of precision and recall, providing a balance between them.
recall	The ratio of correctly predicted positive observations to all actual positives.
precision	The ratio of correctly predicted positive observations to the total predicted positives.
specificity	The ratio of correctly predicted negative observations to all actual negatives.
micro_precision	Precision calculated globally by counting total true positives and false positives.
micro_recall	Recall calculated globally by counting total true positives and false negatives.
micro_f1_score	F1 score calculated using micro-averaged precision and recall.
macro_precision	Average of precision calculated independently for each class.
macro_recall	Average of recall calculated independently for each class.
macro_f1_score	Average of F1 scores calculated independently for each class.
weighted_precision	Average of precision for each class, weighted by number of true instances.
weighted_recall	Average of recall for each class, weighted by number of true instances.
weighted_f1_score	Average of F1 scores for each class, weighted by number of true instances.

Detection Metrics

Metric	Description
mAP	Mean Average Precision; the average of the Average Precision (AP) across all classes.
mAP@50	Mean Average Precision at Intersection over Union (IoU) threshold of 50%.
mAP@75	Mean Average Precision at Intersection over Union (IoU) threshold of 75%.
mAP@90	Mean Average Precision at Intersection over Union (IoU) threshold of 90%.
mAP@50-95	Mean Average Precision averaged over IoU thresholds from 50% to 95%.
AP@50	Per-class Average Precision at IoU threshold of 50%.
AP@75	Per-class Average Precision at IoU threshold of 75%.
AP@90	Per-class Average Precision at IoU threshold of 90%.
AP@50-95	Per-class Average Precision averaged over IoU thresholds from 50% to 95%.
mAR@50	Mean Average Recall at Intersection over Union (IoU) threshold of 50%.
mAR@75	Mean Average Recall at Intersection over Union (IoU) threshold of 75%.
mAR@90	Mean Average Recall at Intersection over Union (IoU) threshold of 90%.
mAR@50-95	Mean Average Recall averaged over IoU thresholds from 50% to 95%.
AR@50	Per-class Average Recall at IoU threshold of 50%.
AR@75	Per-class Average Recall at IoU threshold of 75%.
AR@90	Per-class Average Recall at IoU threshold of 90%.
AR@50-95	Per-class Average Recall averaged over IoU thresholds from 50% to 95%.
precision	The ratio of true positive detections to the total detections made (both per-class and mean).
recall	The ratio of true positive detections to the actual number of objects (both per-class and mean).
f1_score	The harmonic mean of precision and recall (both per-class and mean).

These metrics provide a comprehensive evaluation of model performance for different types of tasks, ensuring that the models are assessed thoroughly for their performance and effectiveness in their respective domains.

Supported Export Formats

The platform supports exporting models to various formats optimized for different frameworks and hardware targets, including mobile and edge devices.

Export Format	Description
TorchScript	Export to a serialized PyTorch model format that enables deployment without Python dependencies, optimized for production environments.
ONNX	Open Neural Network Exchange format, allowing models to be transferred between different frameworks.
OpenVINO	Export to a format optimized for Intel’s OpenVINO toolkit, enhancing performance on Intel hardware.
TensorRT	Export to NVIDIA’s TensorRT format for high-performance inference on NVIDIA GPUs.

General Guidelines

Ensure all URLs provided are valid and accessible.
Follow the specified formats and guidelines for each field.
Maintain consistency across related fields (e.g., model keys between different files).
Validate that all required fields are present and correctly filled.

For detailed look into all the types of config files available , feel free to go over the config files of our public Classification Models

This comprehensive guide outlines the necessary JSON file structures and validation rules for integrating models using Matrice.ai’s BYOM feature. Use these templates to ensure compatibility and seamless integration of your custom models into the platform.