Outlier Analysis, Charu C. Aggarwal, 2017 (Springer)DOI: 10.1007/978-3-319-47578-3 - This book provides an extensive overview of outlier detection techniques, covering various methods and applications.
Isolation forest, Fei Tony Liu, Kai Ming Ting, Zhi-Hua Zhou, 20082008 Eighth IEEE International Conference on Data Mining (IEEE)DOI: 10.1109/ICDM.2008.17 - The original research paper introducing the Isolation Forest algorithm, a machine learning method for anomaly detection.
Estimating the support of a high-dimensional distribution, Bernhard Schölkopf, John C. Platt, John Shawe-Taylor, Alex J. Smola, Robert C. Williamson, 2001Neural computation, Vol. 13 (MIT Press)DOI: 10.1162/089976601750264965 - A foundational paper that describes the One-Class Support Vector Machine, a machine learning technique applicable to anomaly detection.
Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, Aurélien Géron, 2019 (O'Reilly Media) - A practical guide that includes implementation details for various machine learning algorithms, including anomaly detection methods, using popular Python libraries.