FALL 2024

Substance Abuse Treatment Engagement

Naina Panjwani * , Jay Campanell * , Andrew Choi , Joanne Lee , Jimena Arce Cantu , Daniel Yang
* Project Lead   

Existing approaches to predicting treatment success in individuals with substance use disorders predominantly utilize model-specific explainable AI (XAI) techniques, which are constrained by their reliance on specific model architectures and thus limit generalizability. This project advances prior research by investigating the application of model-agnostic and post-hoc XAI methods, which provide model-independent explanations and are applied after model training, respectively. Through the integration of random forest models with these sophisticated XAI methodologies, the study aims to identify key factors that influence treatment success or failure. The findings are intended to optimize resource allocation, enable timely interventions, and enhance treatment outcomes for individuals with substance use disorders.

Digital Well-being for College Students

Darius Mahjoob * , Rachita Jain * , Spencer Tran , Kailin Xia , Catherine He , Yixue Zhao * ,  Wei Xuan * 
* Project Lead    * Project Advisor

Understanding college student mental health often relies on self-reported surveys, which lack depth and real-time insights. The College Experience Dataset, a five-year mobile sensing study, combines demographics, sensor data, and ecological momentary assessments (EMA) to explore mental health and resilience. Analyzing patterns like the pandemic's impact, gender differences, and anomalies in behaviors such as sleep or phone usage remains complex. This project aims to use machine learning to uncover meaningful insights, enhancing understanding and guiding interventions.

Nanopore for Radiation Damage Detection

Vidur Mushran * , Mo Jiang , Pratyush Jaishanker , Ryan Nene , Anisha Chitta , Vayun Mathur , Aram Modrek * ,  Khoi Huynh * 
* Project Lead    * Project Advisor

Assessing radiation exposure and DNA damage in human cells currently relies on methods that lack the ability to provide real-time, quantitative insights into molecular changes. Nanopore sequencing offers a promising approach by enabling high-resolution analysis of DNA and RNA structures through electrical signal detection. However, interpreting nanopore signals for radiation-induced DNA damage remains a complex and underexplored challenge. This project focuses on developing a machine learning-based model to predict radiation exposure levels and detect specific biomarkers of DNA damage, facilitating applications in precision radiotherapy dosing, exposure monitoring in space and nuclear environments, ecological assessments after nuclear incidents, and early cancer prevention through timely intervention.

Recovering Text from Scans of Preserved Scrolls

Leslie Moreno * , Jaiv Doshi * , Aditya Kumar , Aryan Gulati
* Project Lead   

Efforts to decode the ancient Herculaneum scrolls, preserved in the aftermath of Mount Vesuvius, have made significant strides through machine learning and computer vision, with the 2023 Vesuvius Challenge awarding over $1,000,000 for breakthroughs such as recovering Epicurean philosophy from Scroll 1. However, current ink detection methods, reliant on crackle patterns, fail to generalize to other scrolls. This project seeks to scale these techniques by leveraging unlabeled segmented data through self-supervised denoising and fine-tuning on labeled fragments, with the virtual unwrapping process—transforming 3D X-ray volumes into 2D surfaces—serving as the foundation for training models capable of generalizing across various scrolls, ultimately unlocking the hidden historical knowledge within.

Multimodale Hate and Misinformation Detection Toolkit

Jonathan Aydin * , Siddarth Rudraraju * , Youqi Huang , Arjun Bedi , Malina Freeman , Jonathan Gomez * 
* Project Lead    * Project Advisor

Current tools for detecting hate speech and misinformation on social media often lack transparency and adaptability, making it challenging for researchers and policymakers to address these issues effectively. This project aims to develop an explainable, open-source toolkit for analyzing multimodal social media video content, leveraging a dataset of over 32TB of videos, 200 million posts, and 1 million images from the Parler platform. While existing models focus primarily on single-modal data or closed systems, this toolkit will integrate advanced machine learning techniques to analyze text, audio, and visual components simultaneously. By prioritizing explainability, the project seeks to uncover patterns in hate speech and misinformation, fostering a better understanding of their spread and enabling more effective intervention strategies.

Post-generation ASR Hypothesis Reranking Utilizing Visual Contexts (2.0)

Marcus Au * , Tommy Shu * , Yirui Song , Zitong Huang , Catherine Lu
* Project Lead   

The Automatic Speech Recognition (ASR) pipeline proposed in "Multimodal Speech Recognition for Language-Guided Embodied Agents" (Chang et al.) processes both unimodal (audio-only) and multimodal (audiovisual) data to generate multiple ranked hypotheses based on a given ground truth statement. However, the model often fails to rank the hypothesis with the lowest Word Error Rate (WER) as the top choice. To address this issue, we propose a multimodal reranking pipeline that leverages the same visual cues used in the ASR process.

Computer Vision and Machine Learning on Optical Coherence Tomography for Middle Ear Pathology Detection (3.0)

Claude Yoo * , Lucia Zhang * , Irika Katiyar , Will Dolan , Matthew Rodriguez , Lauren Sun , Sana Jayaswal , Brian Applegate * 
* Project Lead    * Project Advisor

Current diagnostic methods for middle ear diseases in otology are primarily qualitative and limited to examining only the surface of the tympanic membrane (TM). Optical Coherence Tomography (OCT) offers a non-invasive, quantitative imaging technique that enables three-dimensional reconstruction of the TM and middle ear, providing more detailed information than traditional methods. However, manually interpreting OCT scans can be time-consuming and challenging, and while OCT-based disease detection models are well-established in retinal imaging and ophthalmology, their application in otology remains relatively unexplored. This project focuses on creating a multi-classification machine learning model capable of identifying conditions such as retraction pockets, perforations, and cholesteatomas, and distinguishing them from healthy ear scans.

Next