Publications

2024

1. In Press

   Language model scaling laws and zero-sum learning

   Andrei Mircea, Ekaterina Lobacheva, Supriyo Chakraborty, and 2 more authors

   In NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning, Oct 2024

   Abs

   This work aims to understand how, in terms of training dynamics, scaling up language model size yields predictable loss improvements. We find that these improvements can be tied back to loss deceleration, an abrupt transition in the rate of loss improvement, characterized by piece-wise linear behavior in log-log space. Notably, improvements from increased model size appear to be a result of (1) improving the loss at which this transition occurs; and (2) improving the rate of loss improvement after this transition. As an explanation for the mechanism underlying this transition (and the effect of model size on loss it mediates), we propose the zero-sum learning (ZSL) hypothesis. In ZSL, per-token gradients become systematically opposed, leading to degenerate training dynamics where the model can’t improve loss on one token without harming it on another; bottlenecking the overall rate at which loss can improve. We find compelling evidence of ZSL, as well as unexpected results which shed light on other factors contributing to ZSL.

2. Gradient Dissent in Language Model Training and Saturation

   Andrei Mircea, Ekaterina Lobacheva, and Irina Rish

   In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, Jun 2024

   Abs PDF Poster

   We seek to shed light on language model (LM) saturation from the perspective of learning dynamics. To this end, we define a decomposition of the cross-entropy gradient, which forms a shared low-dimensional basis for analyzing the training dynamics of models across scales. Intuitively, this decomposition consists of attractive and repulsive components that increase the logit of the correct class and decrease the logits of incorrect classes, respectively. Our analysis in this subspace reveals a phenomenon we term }textit{gradient dissent}, characterized by gradient components becoming systematically opposed such that loss cannot be improved along one component without being degraded along the other. Notably, we find that complete opposition, which we term }textit{total dissent}, reliably occurs in tandem with the saturation of smaller LMs. Based on these results, we hypothesize that gradient dissent can provide a useful foundation for better understanding and mitigating saturation.

3. Human-artificial intelligence teaming for scientific information extraction from data-driven additive manufacturing research using large language models

   Mutahar Safdar^*, Jiarui Xie^*, Andrei Mircea^*, and 1 more author

   Apr 2024

   Abs arXiv PDF

   Data-driven research in Additive Manufacturing (AM) has gained significant success in recent years. This has led to a plethora of scientific literature to emerge. The knowledge in these works consists of AM and Artificial Intelligence (AI) contexts that have not been mined and formalized in an integrated way. It requires substantial effort and time to extract scientific information from these works. AM domain experts have contributed over two dozen review papers to summarize these works. However, information specific to AM and AI contexts still requires manual effort to extract. The recent success of foundation models such as BERT (Bidirectional Encoder Representations for Transformers) or GPT (Generative Pre-trained Transformers) on textual data has opened the possibility of expediting scientific information extraction. We propose a framework that enables collaboration between AM and AI experts to continuously extract scientific information from data-driven AM literature. A demonstration tool is implemented based on the proposed framework and a case study is conducted to extract information relevant to the datasets, modeling, sensing, and AM system categories. We show the ability of LLMs (Large Language Models) to expedite the extraction of relevant information from data-driven AM literature. In the future, the framework can be used to extract information from the broader design and manufacturing literature in the engineering discipline.

4. Towards reproducible machine learning-based process monitoring and quality prediction research for additive manufacturing

   Yan Lu, Zhuo Yang, Jiarui Xie, and 4 more authors

   In International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, Mar 2024

   Awarded Abs arXiv PDF

   AMS Best paper award

   Machine learning (ML)-based cyber-physical systems (CPSs) have been extensively developed to improve the print quality of additive manufacturing (AM). However, the reproducibility of these systems, as presented in published research, has not been thoroughly investigated due to a lack of formal evaluation methods. Reproducibility, a critical component of trustworthy artificial intelligence, is achieved when an independent team can replicate the findings or artifacts of a study using a different experimental setup and achieve comparable performance. In many publications, critical information necessary for reproduction is often missing, resulting in systems that fail to replicate the reported performance. This paper proposes a reproducibility investigation pipeline and a reproducibility checklist for ML-based process monitoring and quality prediction systems for AM. The pipeline guides researchers through the key steps required to reproduce a study, while the checklist systematically extracts reproducibility-relevant information from the publication. We validated the proposed approach through two case studies: reproducing a fused filament fabrication warping detection system and a laser powder bed fusion melt pool area prediction model. Both case studies confirmed that the pipeline and checklist successfully identified missing information, improved reproducibility, and enhanced the performance of reproduced systems. Based on the proposed checklist, a reproducibility survey was conducted to assess the current reproducibility status within this research domain. By addressing this research gap, the proposed methods aim to enhance trustworthiness and rigor in ML-based AM research, with potential applicability to other ML-based CPSs.

2023

1. Balaur: Language Model Pretraining with Lexical Semantic Relations

   Andrei Mircea, and Jackie Cheung

   In Findings of the Association for Computational Linguistics: EMNLP 2023, Dec 2023

   Abs PDF

   Lexical semantic relations (LSRs) characterize meaning relationships between words and play an important role in systematic generalization on lexical inference tasks. Notably, several tasks that require knowledge of hypernymy still pose a challenge for pretrained language models (LMs) such as BERT, underscoring the need to better align their linguistic behavior with our knowledge of LSRs. In this paper, we propose Balaur, a model that addresses this challenge by modeling LSRs directly in the LM’s hidden states throughout pretraining. Motivating our approach is the hypothesis that the internal representations of LMs can provide an interface to their observable linguistic behavior, and that by controlling one we can influence the other. We validate our hypothesis and demonstrate that Balaur generally improves the performance of large transformer-based LMs on a comprehensive set of hypernymy-informed tasks, as well as on the original LM objective. Code and data are made available at https://github.com/mirandrom/balaur

2021

1. Bridging the gap between supervised classification and unsupervised topic modelling for social-media assisted crisis management

   Mikael Brunila^*, Rosie Zhao^*, Andrei Mircea^*, and 2 more authors

   In Proceedings of the Second Workshop on Domain Adaptation for NLP, Apr 2021

   Abs PDF Code Poster

   Social media such as Twitter provide valuable information to crisis managers and affected people during natural disasters. Machine learning can help structure and extract information from the large volume of messages shared during a crisis; however, the constantly evolving nature of crises makes effective domain adaptation essential. Supervised classification is limited by unchangeable class labels that may not be relevant to new events, and unsupervised topic modelling by insufficient prior knowledge. In this paper, we bridge the gap between the two and show that BERT embeddings finetuned on crisis-related tweet classification can effectively be used to adapt to a new crisis, discovering novel topics while preserving relevant classes from supervised training, and leveraging bidirectional self-attention to extract topic keywords. We create a dataset of tweets from a snowstorm to evaluate our method’s transferability to new crises, and find that it outperforms traditional topic models in both automatic, and human evaluations grounded in the needs of crisis managers. More broadly, our method can be used for textual domain adaptation where the latent classes are unknown but overlap with known classes from other domains.

2. Discourse-Aware Unsupervised Summarization for Long Scientific Documents

   Yue Dong^*, Andrei Mircea^*, and Jackie Chi Kit Cheung

   In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Apr 2021

   Abs PDF Code Poster

   We propose an unsupervised graph-based ranking model for extractive summarization of long scientific documents. Our method assumes a two-level hierarchical graph representation of the source document, and exploits asymmetrical positional cues to determine sentence importance. Results on the PubMed and arXiv datasets show that our approach outperforms strong unsupervised baselines by wide margins in automatic metrics and human evaluation. In addition, it achieves performance comparable to many state-of-the-art supervised approaches which are trained on hundreds of thousands of examples. These results suggest that patterns in the discourse structure are a strong signal for determining importance in scientific articles.

2020

1. Real-time Classification, Geolocation and Interactive Visualization of COVID-19 Information Shared on Social Media to Better Understand Global Developments

   Andrei Mircea

   In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Dec 2020

   Abs PDF Poster

   As people communicate on social media during COVID-19, it can be an invaluable source of useful and up-to-date information. However, the large volume and noise-to-signal ratio of social media can make this impractical. We present a prototype dashboard for the real-time classification, geolocation and interactive visualization of COVID-19 tweets that addresses these issues. We also describe a novel L2 classification layer that outperforms linear layers on a dataset of respiratory virus tweets.

2. Using deep learning and social network analysis to understand and manage extreme flooding

   Andrei Romascanu^*, Hannah Ker^*, Renee Sieber, and 6 more authors

   Journal of Contingencies and Crisis Management, Dec 2020

   Abs PDF

   Combining machine learning with social network analysis (SNA) can leverage vast amounts of social media data to better respond to crises. We present a case study using Twitter data from the March 2019 Nebraska floods in the United States, which caused over $1 billion in damage in the state and widespread evacuations of residents. We use a subset of machine learning, deep learning (DL), to classify text content of 11,982 tweets, and we integrate that with SNA to understand the structure of tweet interactions. Our DL approach pre-trains our model with a DL language technique, BERT, and then trains the model using the standard training dataset to sort a dataset of tweets into classes tailored to crisis events. Several performance measures demonstrate that our two-tiered trained model improves domain adaptation and generalization across different extreme weather event types. This approach identifies the role of Twitter during the damage containment stage of the flood. Our SNA identifies accounts that function as primary sources of information on Twitter. Together, these two approaches help crisis managers filter large volumes of data and overcome challenges faced by simple statistical models and other computational techniques to provide useful information during crises like flooding.