Best Practices for Machine Learning Security

Machine learning security defends ML models, the data they learn from, the pipelines that produce them, and the systems that consume their predictions. It blends classical software security, data engineering security, and a set of techniques specific to learned systems.

The MITRE ATLAS knowledge base provides the closest analog to ATT&CK for ML systems. It categorizes adversary behavior through tactics like reconnaissance, initial access, ML model access, execution, persistence, and impact. Familiarity with ATLAS sharpens threat modeling for any ML system.

A typical ML lifecycle includes data collection and labeling, feature engineering, training, evaluation, deployment, monitoring, and retraining. Each stage has its own threats. Data poisoning attacks training. Adversarial examples attack inference. Model extraction attacks intellectual property. Membership inference attacks privacy. Pipeline compromise attacks the entire system.

Three properties drive everything else. Robustness means the model behaves correctly under adversarial conditions. Privacy means the model does not reveal information about training data. Integrity means the model and pipeline have not been tampered with. Mature programs measure each.

Data poisoning manipulates training data so the resulting model misbehaves. Untargeted poisoning aims to degrade accuracy across the board. Targeted poisoning, also called backdoor or trojan attacks, inserts patterns that cause specific inputs to be misclassified while normal performance remains intact. Public datasets, crowdsourced labels, and federated learning are common vectors. Research has demonstrated poisoning attacks against image classifiers, malware detectors, and recommendation systems with very small fractions of poisoned data.

Evasion attacks craft adversarial examples that fool a trained model at inference. They can be untargeted or targeted, white box (full model access) or black box (query only). Famous demonstrations include physical patches on stop signs, eyeglasses that fool facial recognition, and audio that humans hear as one thing but speech systems transcribe as another. Robust models exist but always trade off some clean accuracy.

Model extraction repeatedly queries a deployed model to reconstruct a functional copy. Once an attacker has a local copy, they can craft adversarial examples more efficiently and bypass intellectual property protections. Cloud ML APIs are common targets.

Model inversion and membership inference compromise privacy. Inversion attempts to recover features of training examples from model outputs, sometimes producing recognizable representations of training subjects. Membership inference determines whether a specific record was in the training set, which can be devastating in medical or financial contexts.

Supply chain compromise targets the artifacts and tools of ML. Malicious models on public hubs, backdoored datasets, vulnerable pickle files, and dependencies in ML frameworks have all been observed in the wild. The ML ecosystem evolved quickly, often outpacing the maturity of its security controls.

Pipeline compromise targets the infrastructure: orchestration tools like Airflow and Kubeflow, experiment trackers like MLflow, feature stores like Feast and Tecton, registries, and compute clusters. A compromised pipeline can poison data, swap models, exfiltrate features, or steal credentials with great impact and minimal noise.

Treat training data as a first class asset. Establish clear ownership, classification, lineage, and access control. Tools like DVC, LakeFS, Pachyderm, and proprietary lineage systems track which data went into which model version. Lineage answers compliance and incident questions and makes rollback feasible.

Validate data continuously. Schema checks catch malformed records. Distribution checks catch drift and potential poisoning. Anomaly detection flags suspicious bursts of new examples. Tools like Great Expectations, TFX Data Validation, and Pandera embed these checks into pipelines.

Audit labels. Label noise and adversarial labeling are common entry points. Sample annotations periodically, especially for crowd-sourced labels. Use multiple annotators and consensus mechanisms for important data.

Limit poisoning influence with robust training techniques. Use trimmed loss functions that downweight outliers, ensembles that vote across diverse training subsets, and certified defenses where they fit the threat model. Apply differential privacy to provide mathematical guarantees against memorization of individual records.

Protect training environments. Isolate sensitive training jobs on dedicated networks. Restrict outbound traffic. Enforce least privilege on compute and storage. Audit who can launch training, modify code, and access data. Treat training clusters as production systems, not as data scientist sandboxes.

Sign and verify models. A model registry should track artifact hashes, training metadata, evaluation results, and approval records. Use Sigstore or equivalent to sign artifacts and verify signatures before serving. Reject unsigned or unapproved models in production.

Avoid insecure serialization. Pickle and similar formats execute arbitrary code at load time, which has been exploited repeatedly. Prefer SafeTensors, ONNX, or framework-specific safe loaders. Where pickle is unavoidable, only load from trusted, signed sources.

Defend inference endpoints with classical API security: authentication, authorization, rate limiting, input validation, and observability. Beyond that, add ML-specific protections. Rate limits should consider not just queries per minute but the information value per query. Restrict the precision of returned probabilities. Watermark high-value model outputs to detect theft.

Apply adversarial training where adversarial robustness matters. Generate adversarial examples during training and include them in the loss function. Robustness libraries like the Adversarial Robustness Toolbox and CleverHans provide reference implementations. Combine with input transformation, randomized smoothing, and certified defenses for stronger guarantees.

Detect adversarial inputs at runtime. Statistical detectors compare incoming inputs to the training distribution. Confidence thresholds reject low-confidence predictions. Ensembles that disagree flag suspicious inputs. Monitor reject rates as a signal for emerging attacks.

For privacy, consider differential privacy in training and inference. DP-SGD adds calibrated noise during training and offers formal guarantees that the model does not memorize individual records. Production deployments at Apple, Google, and others have shown the approach is viable, with measurable but acceptable accuracy costs in many cases.

Adopt MLSecOps. Embed security into every stage of the ML lifecycle, not bolted on at the end. Use infrastructure as code for ML platforms, immutable artifacts, signed builds, and automated security testing in pipelines.

Maintain an ML inventory. Catalog every model in production: purpose, owner, training data sources, evaluation metrics, fairness assessments, last update, and risk classification. The inventory drives audits, monitoring, and incident response.

Monitor models in production. Track input distributions, output distributions, business performance metrics, and known abuse patterns. Drift detection catches degraded behavior. Slice-level metrics catch fairness regressions. Combine ML monitoring tools like Evidently, Fiddler, Arize, and WhyLabs with traditional SIEM platforms.

Build cross-functional governance. Bring data science, security, compliance, legal, and business owners together regularly. Review high-risk models before deployment, after material changes, and on a recurring cadence. Document decisions and rationale.

Plan for incident response. Define playbooks for poisoned data discovery, leaked model files, reported adversarial attacks, abuse of inference APIs, privacy complaints, and biased outcomes. Exercise the playbooks with tabletop scenarios.

Address third party risk. ML SaaS, foundation model APIs, labeling vendors, and pretrained model providers all introduce dependencies. Audit their security posture, data handling, and contractual protections. Prefer providers with mature security certifications and transparent practices.

Researchers at Cornell, UC Berkeley, and other institutions have demonstrated practical evasion attacks against image classifiers, speech recognition, and malware detection. The attacks have been transferred across models, including commercial cloud APIs.

The PyTorch ecosystem faced a supply chain compromise in late 2022 when a malicious package was uploaded to a public index, briefly impersonating a real dependency. The incident accelerated investment in package verification and CI controls.

A 2023 audit of public Hugging Face models found pickle-based code execution risks in a significant fraction of community uploads. Hugging Face has since strengthened scanning, but the episode highlighted the gap between research community norms and enterprise security expectations.

Fraud detection teams routinely fight model extraction. Adversaries probe fraud APIs to learn decision boundaries and craft transactions just inside the approved zone. Rate limiting, fine-grained authorization, and adaptive learning are the operational counters.

Healthcare ML has prompted regulators to scrutinize training data sources and patient privacy. Models trained on data that was insufficiently consented or de-identified have faced legal challenges. Differential privacy and federated learning are gaining traction in this domain.