Coffee is one of the most popular drinks in the world. It contains antioxidants and health-promoting nutrients that can boost one’s energy and focus. However, defective beans mixed in with raw beans can easily affect the flavor and even be harmful to human health. The traditional human visual inspection of defective beans is extremely laborious and time-consuming and may result in low-quality coffee due to worker stress and fatigue. We propose a lightweight and explainable intelligent coffee bean quality inspection system that uses deep learning (DL) and computer vision (CV) technologies to assist operators in detecting defects, including mold, fermentation, insect bites, and crushed beans. We use knowledge distillation (KD) to achieve model compression. The basic explainable convolutional neural network (CNN) model is established using the explainable AI (XAI) method. The implemented system has a high identification rate, low complexity, and low power consumption, and can explain the judgment criteria of the complex classification model.