Skip to main content
Artificial Intelligence
Computer Vision Devices and Systems with Artificial Intelligence
Identity Verification
AI Golf Coach
AI Football Coach
Classification with CNN
Vertebra X-Ray Retrieval

Summary:             Facial Motion Video, Promo Video
Many functions in our daily life require identity verification. Traditional identity verification methods, including passwords, ID cards, or other physical devices, all have their weaknesses or inconvenience. Biometric-based identity verification techniques are popular and more secure than traditional methods but cannot prevent passive verification or provide liveness assurance. We developed a new identity verification technique to address these challenges. This unique approach, called Concurrent Two-Factor IDentification (C2F-ID), analyzes spatial facial features and facial actions concurrently to verify identity. It requires only a short video of the frontal face expressing a unique user-produced facial action. In our study, a metric learning model using recurrent neural network (RNN) was selected as the backbone architecture to demonstrate its feasibility. The experiment result of our proof-of-concept study using this network showed a very encouraging 85% accuracy. Development of more sophisticated networks specifically for this unique application is underway. More video samples are being collected. The improvement in the performance of this unique identity verification approach is quite promising.
1.     Z. Sun, D. J. Lee, D. Zhang, W.C. Chen, and X. Li, “Concurrent Two-Factor Identity Verification Using Active Facial Features”, IEEE International Conference on Image Processing (submitted on 01/24/2021).
2.     Z. Sun, D. J. Lee, D. Zhang, and X. Li, “Concurrent Two-factor Identity Verification Using Facial Identity and Facial Actions,”, SPIE Electronic Imaging, Intelligent Robotics and Industrial Applications using Computer Vision, Virtual, January 11, 2021.

Summary:             Video
A golf swing requires full-body coordination and much practice to perform the complex motion precisely and consistently. The force from the golfer’s full-body movement on the club and the trajectory of the swing are the main determinants of swing quality. In this research, we introduce a unique motion analysis method to evaluate the quality of a golf swing. The primary goal is to evaluate how close the user’s swing is to a reference swing. We use 17 skeleton points to evaluate the resemblance and report a notation that incorrectly indicates body movement. This evaluation result can be used as real-time feedback to improve player performance. Using this real-time feedback system repeatedly, the player will be able to train their muscle memory to improve their swing consistency. We created our data-set from a professional golf instructor including good and bad swings. Our result demonstrates that such a machine learning-based approach is feasible and has great potential to be adopted as a low-cost but efficient tool to improve swing quality and consistency.
1.     J.J. Liu, J.D. Newman, and D.J. Lee, “Using Artificial Intelligence to Provide Visual Feedback for Golf Swing Training”, SPIE Electronic Imaging, Intelligent Robotics and Industrial Applications using Computer Vision, Virtual, January 11, 2021.
2.     J.J. Liu, J.D. Newman, and D.J. Lee, “Body Motion Analysis for Golf Swing Evaluation”, International Symposium on Visual Computing (ISVC), LNCS 12509, pp. 566–577, Virtual, October 5-7, 2020.

Annotation and analysis of sports videos is a challenging task that, once accomplished, could provide various benefits to coaches, players, and spectators. In particular, American Football could benefit from such a system to provide assistance in statistics and game strategy analysis. Manual analysis of recorded American football game videos is a tedious and inefficient process. As a first step of our research for this unique application, we focus on locating and labeling individual football players from a single overhead image of a football play immediately before the play begins. A pre-trained deep learning network is used to detect and locate the players in the image. A ResNet is used to label the individual players based on their corresponding player position or formation. Our player detection and labeling algorithms obtain greater than 90% accuracy, especially for those skill positions on offense (Quarterback, Running Back, and Wide Receiver) and defense (Cornerback and Safety). Results from our preliminary studies on player detection, localization, and labeling prove the feasibility of building a complete American football strategy analysis system using artificial intelligence.
1.     J.D. Newman, J.W. Lin, D. J. Lee, and J.J. Liu, “Automatic Annotation of American Football Video Footages for Game Strategy Analysis”, SPIE Electronic Imaging, Intelligent Robotics and Industrial Applications using Computer Vision, Virtual, January 11, 2021.

The field of computer vision has come a long way in solving the problem of image classification. Not too long ago, handcrafted convolutional kernels were a staple of all computer vision algorithms. With the advent of Convolutional Neural Networks (CNNs), however, handcrafted features have become the exception rather than the rule, and for good reason. CNNs have taken the field of computer vision to new heights by solving problems that used to be unapproachable or unthinkable. Robotic Vision Lab focuses on using CNNs to solve real-world problems beyond object recognition and image classification. Another research focus is the optimization and hardware implementation of CNNs for real-time embedded applications.
Journal Publications:
6.     W.C. Chen, D. Zhang, M. Li, and D.J. Lee, "STCAM: Spatial-Temporal and Channel Attention Module for Dynamic Facial Expression Recognition", IEEE Transactions on Affective Computing, (Online on 09/29/20)
5.     Y.Y. Li,  D. Zhang, and D. J. Lee , "Automatic Fabric Defect Detection with a Wide-And-Compact Network", Journal of Neurocomputing, vol. 329, pp. 329-338, February 2019.
4.     Y.Y. Li, D. Zhang, and D. J. Lee, "IIRNet: A Lightweight Deep Neural Network Using Intensely Inverted Residuals for Image Recognition", Image and Vision Computing, vol. 92, Article# 103819, December 2019.
3.     T.S. Simons and D.J. Lee, "A Review of Binarized Neural Networks", Electronics, vol. 8(6), pp. 661-25 pages, June 2019.
2.     T.S. Simons and D.J. Lee, "Jet Features: Hardware-Friendly, Learned Convolutional Kernels for High-Speed Image Classification", Electronics, vol. 8(5), pp. 588-20 pages, May 2019.
1.     J.N. Teng, D. Zhang, D.J. Lee, "Recognition of Chinese Food Using Convolutional Neural Network", Multimedia Tools and Applications, vol. 78(9), pp. 11155-11172, May 2019.

Object recognition is a well studied but extremely challenging field. We developed a novel approach to feature construction for object detection called Evolution-COnstructed Features (ECO features) in 2010. ECO features are automatically constructed by uniquely employing a standard genetic algorithm to discover multiple series of transforms that are highly discriminative. We have successfully applied this algorithm to many visual inspection applications including automated apple stem and calyx detection, shrimp shape quality grading, fish species recognition, invasive carp removal, fruit and food quality grading, and road condition and pavement quality evaluation. Demo Video.
Journal Publications:
9.     Z.H. Guo, M. Zhang, D.J. Lee, and T.S. Simons"Smart Camera for Quality Inspection and Grading of Food Products", Electronics, vol. 9(3), pp. 505-18pages, March 2020. 
8.     Z.H. GuoM. Zhang, and D.J. Lee, "Efficient Evolutionary Learning Algorithm for Real-Time Embedded Vision Applications", Electronics, vol. 8(11), pp.1367-18 pages, November 2019. 
7.     M. Zhang, D.J. Lee, K.D. Lillywhite, and B.J. Tippetts, "Automatic Quality and Moisture Evaluations Using Evolution Constructed Features,” Computers and Electronics in Agriculture, vol. 135, p. 321-327, April 2017.
6.     D. Zhang, D.J. Lee, M. Zhang. B.J. Tippetts, and K.D. Lillywhite, "Object Recognition Algorithm for the Automatic Identification and Removal of Invasive Fish,” Biosystems Engineering, vol. 145, p. 65-75, May 2016.
5.     D. Zhang, K.D. Lillywhite, D.J. Lee, and B.J. Tippetts"Automated Fish Taxonomy using Evolution-COnstructed Features for Invasive Species Removal”, Pattern Analysis and Applications, vol. 18/2. p. 451-459, May 2015. 
4.     D. Zhang, K.D. Lillywhite, D.J. Lee, and B.J. Tippetts"Automatic Shrimp Shape Grading Using Evolution Constructed Features”, Computers and Electronics in Agriculture, vol. 100, p. 116-122, January 2014.
3.     K.D. Lillywhite, D.J. Lee, B.J. Tippetts, and J.K Archibald, "A Feature Construction Method for General Object Recognition”, Pattern Recognition, vol. 46/12, p. 3300-3314, December 2013.
2.     D. Zhang, K.D. Lillywhite, D.J. Lee, and B.J. Tippetts"Automated Apple Stem End and Calyx Detection using Evolution-COnstructed Features,” Journal of Food Engineering, vol. 119/3, p. 411-418, December 2013. 
1.     K.D. LillywhiteB.J. Tippetts, and D.J. Lee, "Self-Tuned Evolution-COnstructed Features for General Object Recognition”, Pattern Recognition, vol. 45/1, p. 241-251, January 2012.

Vertebra shape can effectively describe various pathologies found in spine x-ray images. Some critical regions on the shape contour help determine whether the shape is pathologic or normal. We developed a technique to automatically select nine points from the boundary contour, represent vertebra shape using multiple open triangles, and relevance feedback to retrieve vertebra X-ray images.
Journal Publications:
4.     D.J. Lee, S.K. Antani, Y.C. Chang, K. Gledhill, L.R. Long, and P. Christensen"CBIR of Spine X-ray Images on Inter-vertebral Disc Space and Shape Profiles”, special issue on "Knowledge Discovery in Medicine", Data & Knowledge Engineering Journal,  vol. 68/12, p. 1359-1369, December 2009.
3.      X.Q. Xu , D.J. Lee, S.K. Antani, L.R. Long, and J. K Archibald, " Using Relevance Feedback with Short-term Memory for Content-based Spine X-ray Image Retrieval”,  Journal of Neurocomputing , vol. 72/10-12, p. 2259-2269,  June 2009. 
2.      X.Q. Xu , D.J. Lee, S.K. Antani, and L.R. Long, "A Spine X-ray Image Retrieval System Using Partial Shape Matching”, IEEE Transactions on Information Technology in Biomedicine, vol. 12/1, p. 100-108, January 2008.
1.      S.K. Antani, D.J. Lee, L.R. Long, and G.R. Thoma, "Evaluation of Shape Similarity Measurement Methods for Spine X-Ray Images”, Special issue on  "Multimedia Database Management Systems" of the Journal of Visual Communication and Image Representation, vol. 15/3, p. 285-302, September 2004.