Methodological Features of Validating Forensic Expert Techniques

The article reviews and summarizes the experience of validating forensic expert techniques in the Russian Federal Centre of Forensic Science of the Ministry of Justice of the Russian Federation. The authors point out the methodological features of practical implementation of the validation procedure. They demonstrate that the specificity, diversity, and complexity of the objects of expert study require the classification of the applied methods in terms of metrology, identification of the main validation parameters of quantitative and qualitative methods, organization of experiments, and evaluation of validation parameters using mathematical analysis methods. They also propose to divide methods into two types: forensic expert measurement methods (FMT) and forensic expert testing methods (FTT). Based on the generalization of information presented in several regulatory documents and scientific publications, the following parameters are identified for FMT: metrological characteristics or properties of the method (specificity, linearity, sensitivity, range of determined values, detection limit, quantitative determination limit) and quality indicators of the method (precision, correctness, accuracy of the analysis result, or uncertainty). When validating FTT, it is proposed to evaluate the reliability of the method and the competence of the expert.
An experiment to assess validation parameters is performed using enough control samples with established characteristics of controlled indicators and with the participation of a sufficient number of experts. Requirements for control samples are provided.
The authors also give examples of probabilistic evaluation of validation parameters for two qualitative testing methods: microscopic examination of textile fibers and detection of gunshot residue using scanning electron microscopy and X-ray microanalysis. The reliability of these methods is assessed by calculating the likelihood ratio, and the specificity of interpreting the results of FMT and FTT validation is noted.
The decision on compliance with the requirements is made if the interval of the established extended uncertainty for the obtained result does not exceed the tolerance field. In the absence of tolerances, FMT is considered suitable for solving forensic expert tasks if the values of the extended uncertainty of the measurement results of the controlled indicator do not exceed the values established during validation. For FTT, a low probabilistic proportion of false positive and false negative results in determining the presence/absence of controlled indicators, as well as experimentally confirmed competence of the expert during validation, are indicators of the suitability of the method for its intended use

1. Smirnova S.A., Usov A.I., Omel’yanuk G.G., Bebeshko G.I., Korol S.G. Practice of Accreditation of Forensic Laboratories of the Ministry of Justice of Russia on Compliance with ISO/MEK 17025. Theory and Practice of Forensic Science. 2011. No. 2 (22). P. 40–56. (In Russ.).

2. Smirnova S.A., Omel’yanuk G.G., Miklyaeva O.V. Expert’s Methodic. In: Smirnova S.A. (ed.). Multimodal Edition “Forensic Expertise: Reboot”. Part II. Encyclopedic Dictionary of the Theory of Forensic Science. Moscow: EKOM, 2012. P. 184–185. (In Russ.).

3. International Vocabulary of Metrology. 2nd ed. Translated form English and French. Saint Petersburg: NPO “Professional”, 2010. 80 p. (In Russ.).

4. Smirnova S.A., Omel’yanyuk G.G., Bebeshko G.I. Methodological Approaches to Validation of Forensic Methods Including Measurement Methods (MI). Theory and Practice of Forensic Science. 2012. No. 1 (25). P. 50–62. (In Russ.).

5. Paneva V.I. Estimation of the Validity of the Methods of Quantitative Chemical Analysis in the Laboratory. Industrial Laboratory. Diagnostics of Materials. 2008. Vol. 74. No. 8. P. 68–72. (In Russ.).

6. Prichard E., Barwik V. Quality Assurance in Analytical Chemistry. Wiley, 2007. 316 p.

7. Cadiz R.L. Measurement Uncertainty and Chemical Analysis. Journal of Analytical Chemistry. 2008. Vol. 63. No. 1. P. 104–110. (In Russ.).

8. Bebeshko G.I., Lyubetskaya I.P., Omel’yanyuk G.G., Usov A.I. Methodological Approaches to Calculation of the Main Validation Parameters of Forensic Techniques. Industrial laboratory. Diagnostics of materials. 2020. Vol. 86. No. 4. P. 66–74. (In Russ.). https://doi.org/10.26896/1028-6861-2020-86-4-66-74

9. Smirnova S.A., Omel’yanyuk G.G., Bebeshko G.I., Yudin N.V. The Experience of Validation Measurement Method “The Determination of Benzo(a)Pyrene Concentration in the Objects of Soil and Geological Origin by Means of HPLC Fluorimetry Detecting Method” for Conducting Soil Forensic Expert Examination. Theory and Practice of Forensic Science. 2012. No. 3 (27). P. 79–91. (In Russ.).

10. Smirnova S.A., Omel’yanyuk G.G., Bebeshko G.I., Popov V.V. Modern Approaches to Metrological Evaluation of Forensic Research Methods. 6th European Academy of Forensic Science Conference EAFS-2012 (August 20– 24, 2012). The Hague. Abstractbook. Рoster presentation. P. 363.

11. Smirnova S.А., Bebeshko G.I., Omel’yanyuk G.G. Experience of Using the Uncertainty Concept in the Forensic Expertise of Substances and Materials. The 21st International Symposium on the Forensic Sciences of the Australian and New Zealand Forensic Sciences Society (September 23–27, 2012). Hobart. Conferen cebook. Management and Quality Accuracy. No. 195. P. 296–300.

12. Bebeshko G.I., Lyubetskaya I.P., Brunova L.P., Khanukaeva M.A., Omel’yanyuk G.G. Measuring Dyed Fiber Color with MSFU-K Microscope Spectrophotometer: Methodology Validation. Theory and Practice of Forensic Science. 2018. Vol. 13. No. 2. P. 71–80. (In Russ.). https://doi.org/10.30764/1819-2785-2018-13-2-71-80

13. Bebeshko G.I., Omel’yanyuk G.G., Nikulina M.V., Valitova A.R. Experimental Validation of a Methodology for Determining Soil pH and Specific Electrical Conductance in Samples of Geological and Soil Evidence in Forensic Environmental Investigations in the Absence of Standard Samples. Theory and Practice of Forensic Science. 2017. Vol. 12. No. 2. P. 66–74. (In Russ.). https://doi.org/10.30764/1819-2785-2017-12-2-66-74

14. Mendmo A., Patrtarca M., Magnusson B. Understanding the Meaning of Accuracy, Trueness and Precision. Accred. Qual. Assur. 2006. Vol. 12. P. 45–47.

15. Рulido A., Ruisánchez I., Boquе́́ R., Rius F.X. Uncertainty of Results in Routine Qualitative Analysis. Trends Anal. Chem. 2003. Vol. 22. No. 10. P. 647–654. https://doi.org/10.1016/S0165-9936(03)01104-X

16. Pulido A., Ruisánchez I., Boqué R., Rius F.X. Estimating the Uncertainty of Binary Test Results to Assess Their Compliance with Regulatory Limits. Analytica Chimica Acta. 2002. Vol. 455. P. 267–275.

17. Ellison S.L.R., Fearn T. Characterizing the Performance of Qualitative Analytical Methods: Statistics and Terminology. Trends Anal. Chem. 2005. Vol. 19. No. 6. P. 468–476. https://doi.org/10.1016/j.trac.2005.03.007

18. Trullols E., Ruisánhez I., Rius F.X., Huguet J. Validation of Qualitative Methods of Analysis That Use Control Samples. Trends Anal. Chem. 2005. Vol. 24. No. 6. P. 516–524. https://doi.org/10.1016/j.trac.2005.04.001

19. Panteleimonov A.V., Nikitina N.A., Reshetnyak E.A., Loginova L.P., Bugaevskii A.A., Kholin Yu.V. Binary Response Procedures of Qualitative Analysis: Methodological Characteristics and Calculation Aspects. Methods and Objects of Chemical Analysis. 2008. Vol. 3. No. 2. P. 128–146. (In Russ.).

20. Mil’man B.L. Introduction to Forensic Identification. St. Petersburg: VVM, 2008. 179 p. (In Russ.).

21. Mil’man B.L. Identification of Chemical Compounds. Trends Anal. Chem. 2005. Vol. 24. No. 6. P. 493–508. https://doi.org/10.1016/j.trac.2005.03.013

22. Smith A.M., Tess N.M.S. The Distinction between Discriminability and Reliability in Forensic Science. Science and Justice. 2021. Vol. 61. P. 319–331. https://doi.org/10.1016/j.scijus.2021.04.002

23. Smirnova S.A., Afanasyev I.B., Bebeshko G.I., Omel’yanyuk G.G. Validation of Expert Method “Detection of Solid Heavy Elements Based Gunshot Residue Particles on the Surface of Various Objects by Mean of Scanning Electron Microscopy and X-Ray Microanalysis”. Industrial Laboratory. Diagnostics of Materials. 2021. Vol. 87. No. 12. P. 63–72. (In Russ.). https://doi.org/10.26896/1028-6861-2021-87-12-63-72

24. Omel’yanyuk G.G., Bebeshko G.I., Korol S.G. Methodological Approaches for Assessing the Competence of Forensic Laboratories by Interlaboratory Professional Testing. Theory and Practice of Forensic Science. 2011. No. 4 (24). P. 52–62. (In Russ.).

25. Boldyrev I.V. ISO/IEC 17025:2017. Practical Recommendations for Use. St. Petersburg: PSC “Profession”, 2018. 128 p. (In Russ.).

26. Smirnova S.A., Gradusova O.B., Nesterina E.M., Bebeshko G.I., Omel’yanyuk G.G., Lyubetskaya I.P. A Technique for Diagnostics of Micro-Inclusions in the Objects of Soil and Geological Origin: Validation and Practical Application. Industrial Laboratory. Diagnostics of Materials. 2021. Vol. 87. No. 6. P. 70–77. (In Russ.). https://doi.org/10.26896/1028-6861-2021-87-6-70-77

27. Gradusova O.B., Peleneva M.V., Nesterina E.M. Atlas of Micro-Inclusions in Soils. Moscow: RFCFS, 2014. 92 p. (In Russ.).

28. Smirnova S.A., Bebeshko G.I., Lyubetskaya I.P., Omel’yanyuk G.G., Usov A.I. Probability-Based Validation of the Forensic Method “Microscopic Analysis of Textile Fibers”. Theory and Practice of Forensic Science. 2019. Vol. 14. No. 2. P. 92–99. (In Russ.). https://doi.org/10.30764/1819-2785-2019-14-2-92-99

29. Smirnova S.A., Bebeshko G.I., Omel’yanyuk G.G., Usov A.I., Khaziev Sh.N. Evidentiary Basis Development with Support of Forensic Research Assessment. Industrial laboratory. Diagnostics of materials. 2020. Vol. 86. No. 10. P. 66–76. (In Russ.) (In Russ.). https://doi.org/10.26896/1028-6861-2020-86-10-66-76