Benchmarking QML Algorithms on Real Quantum Devices

Executing quantum machine learning algorithms within simulators provides valuable insights, but it doesn't capture the full picture of performance on actual quantum processing units (QPUs). As discussed earlier in this chapter, real hardware suffers from noise, limited connectivity, and gate errors. Therefore, rigorously benchmarking QML algorithms on physical devices is an essential step to understand their practical capabilities and limitations in the NISQ era. Benchmarking involves careful experimental design, metric selection, and result analysis to draw meaningful conclusions about feasibility and potential advantages.

Defining the Scope and Goals of Benchmarking

Before running experiments on costly and often queue-time-limited quantum hardware, it's important to define what you aim to measure. Typical goals for benchmarking QML algorithms include:

1. Performance Accuracy: How well does the QML model perform its intended task (e.g., classification accuracy, regression error, fidelity of generated distribution) compared to classical counterparts or ideal quantum simulations?
2. Resource Consumption: What are the hardware requirements? This involves tracking the number of qubits used, the depth of the quantum circuits (especially after transpilation for specific hardware topology), the number of gates, and the required number of measurement shots.
3. Training Dynamics: For variational algorithms, how does the training process (e.g., convergence speed, stability of loss function) differ between simulators and hardware? How effective are chosen optimizers under noisy conditions?
4. Noise Resilience and Mitigation Effectiveness: How significantly does hardware noise degrade performance? How much improvement can be gained by applying the error mitigation techniques discussed previously (like ZNE or PEC)?
5. Scalability: How do accuracy and resource requirements change as the problem size (e.g., number of features, data points, qubits) increases? Where do hardware limitations impose practical boundaries?

These goals dictate the metrics you need to track during your experiments.

Selecting Appropriate Metrics

Choosing the right metrics is fundamental for effective benchmarking. These generally fall into task-specific, resource-related, and noise-related categories:

• Task-Specific Metrics: These depend on the ML task.
  • Classification: Accuracy, Precision, Recall, F1-Score, Area Under the ROC Curve (AUC). Calculated based on measurement outcomes mapped to class labels.
  • Regression: Mean Squared Error (MSE), Mean Absolute Error (MAE). Derived from expectation values of measurement operators.
  • Generative Modeling (QCBMs, QGANs): Kullback-Leibler (KL) divergence, Maximum Mean Discrepancy (MMD), or qualitative assessment of generated samples compared to the target distribution. Often requires estimating probability distributions from measurement counts.
• Resource Metrics: These quantify the computational cost.
  • Qubit Count: The number of qubits required by the algorithm.
  • Circuit Depth: The longest path of gates in the circuit, often reported post-transpilation for a specific device. Deeper circuits are more susceptible to noise.
  • Gate Count: Total number of quantum gates, sometimes broken down by native gate types for specific hardware.
  • Number of Shots: The number of times a circuit is executed to estimate expectation values or sample probabilities. More shots reduce statistical sampling error but increase execution time.
  • Execution Time: Wall-clock time, including queue time and actual QPU execution time.
• Noise and Stability Metrics:
  • Performance Variance: Standard deviation or range of task-specific metrics over multiple identical runs, indicating stability against noise fluctuations and calibration drift.
  • Mitigation Gain: The difference in performance (e.g., accuracy improvement) when error mitigation is applied versus raw hardware execution.

Methodology for Hardware Benchmarking

A systematic approach is needed to obtain reliable benchmarking results.

1. Establish Baselines:

   • Classical Baseline: Implement and evaluate a comparable classical ML algorithm (e.g., classical SVM, a small neural network) on the same dataset and task. This provides a reference point for performance.
   • Ideal Quantum Baseline: Simulate the QML algorithm assuming a perfect, noise-free quantum computer. This represents the theoretical best performance of the chosen quantum model.
   • Noisy Simulation Baseline: Simulate the QML algorithm using a noise model representative of the target hardware (using parameters like $T_1$, $T_2$, gate error rates). This helps isolate the impact of noise compared to the ideal case and can validate error mitigation strategies offline.

2. Select Hardware and Prepare Circuits:

   • Choose the target QPU(s) based on available qubits, connectivity, reported fidelities, and native gate sets. Different providers (e.g., IBM Quantum, Rigetti, IonQ) offer devices with distinct characteristics.
   • Transpile the quantum circuits for the specific hardware. This process maps the logical circuit onto the device's qubit topology and decomposes gates into the hardware's native gate set. Monitor how transpilation affects circuit depth and gate counts, as this directly impacts noise accumulation. Use hardware-efficient ansätze where possible.

3. Design and Execute Experiments:

   • Define the experimental runs. Vary parameters systematically, such as the number of qubits, circuit layers/depth, dataset size, number of measurement shots, and error mitigation settings (e.g., different extrapolation levels for ZNE).
   • Plan for repetition. Run each configuration multiple times (e.g., 5-10 times) to average results and estimate variance, accounting for statistical noise from finite shots and potential fluctuations in device performance (calibration drift).
   • Submit jobs to the quantum hardware provider's platform, being mindful of queue times and execution limits.

4. Collect and Organize Data:

   • Log all experimental parameters meticulously: algorithm configuration, dataset details, hardware used, transpilation settings, error mitigation applied, number of shots.
   • Store the raw measurement counts returned by the hardware for each circuit execution.
   • Calculate the chosen metrics from the raw data.

Analyzing and Interpreting Benchmarking Results

The final step is to analyze the collected data and interpret the findings.

• Quantitative Comparison: Compare the hardware results against the classical, ideal quantum, and noisy simulation baselines using the selected metrics. Visualize these comparisons using plots. For example, plot accuracy versus training epochs or circuit depth for different execution environments (ideal, noisy sim, raw hardware, mitigated hardware).

Comparison of VQC accuracy during training across different execution environments, showing the performance gap and the effect of error mitigation.

• Assess Noise Impact and Mitigation: Quantify the performance drop from ideal simulation to raw hardware execution. Evaluate how effectively error mitigation closes this gap towards the ideal or noisy simulation baseline. Identify which algorithms or circuit structures are more sensitive to noise.
• Evaluate Scalability: Analyze how performance metrics and resource requirements scale with problem size. Identify bottlenecks imposed by qubit count, connectivity, or coherence times. Are there signs of barren plateaus appearing sooner on hardware than in simulations?
• Hardware-Specific Insights: If multiple devices were used, compare their performance. Relate differences to known hardware specifications (e.g., lower error rates or better connectivity might lead to better results).
• Contextualize Findings: Report results clearly, including the experimental setup and limitations (e.g., specific device calibration state, potential biases). Avoid generalizations about "quantum advantage" based on small-scale experiments. Focus on understanding the current state, the effectiveness of techniques like error mitigation, and the challenges that remain for practical QML on near-term hardware.

Benchmarking on real quantum devices is a complex but necessary process for advancing QML. It provides important feedback for algorithm design, error mitigation development, and understanding the true potential and limitations of QML applications in the presence of hardware imperfections.