Andrew Silva

I am a Machine Learning Researcher in Apple’s MLR group, working on improving interactions and LLM personalization.

Before Apple, I was a Research Scientist at Toyota Research Institute, working on adapting LLMs to facilitate driver-vehicle interactions. I received my PhD from Georgia Tech, working in the CORE Robotics lab at Georgia Tech under professor Matthew Gombolay, and before that I was a Research Scientist in the same lab and in the RAIL lab with Dr. Sonia Chernova.

I’ve worked as an intern at Apple and Google Brain Robotics on private federated learning, agile robot adaptation, and personalization for large-language models.

I am also a recipient of the 2021 Apple Scholars in AI/ML PhD fellowship and was named a DAAD AInet Fellow!

See below for examples and links to my recent work!

Research

Dream2Assist

In Dream2Assist we present a framework that combines a rich world model with an assistive agent, resulting in a robot partner that can infer human intentions and then take actions to help satisfy human objectives.

Published Link — Paper Site — Preprint Link

Paper embeddings moving as the recommender learns preferences.

Personalized arXiv Feed

A quick write-up on a personalized paper recommender that I wrote to help researchers keep up with the deluge of arXiv papers everyday!

Link to write-up

Model diagram from FedPC: Federated Learning for Language Generation with Personal and Context Preference Embeddings.

FedPerC

In FedPC, we present a new approach to personalization for federated deployment of large language models, using personal and context embeddings for each user.

Published Link

Explainability conditions compared in the user study.

Evaluating Explainability

Explainability is increasingly popular and important for machine learning research and deployment, but little work is evaluated with real humans. In this user study, we compare seven explainability conditions with real human users.

Published Link — Preprint Link

Figure of clustered data from Cross-Loss Influence Functions to Explain Deep Network Representations

CLIF

Influence functions can help to show why a model make certain decisions, but previously have only been proven for matched train/test objectives. We show that influence functions can work with unsupervised or self-supervised learning.

Published Link

FedEmbed

We present a new approach to private, personalized federated learning, leveraging personal embeddings and clustering of users with similar preferences.

arXiv Link

LanCon-Learn

We present an approach to language-conditioned multi-task learning, using language-based command embeddings rather than conventional one-hot goal specifications.

Published Link — Preprint Link

Multimodal Punctuation Prediction

Speech-to-text systems often transcribe raw audio into text, but do not always consider the structure of text and how that might affect the meaning. In this work, we explore multi-modality and we introduce context-dropout to improve punctuation prediction from raw audio and text.

Published Link — arXiv Link

ProLoNets

Reinforcement learning agents waste hundreds of hours simply learning the rules of the world. With ProLoNets, we can hard-code heuristics directly into RL-agent’s neural network weights before training even begins, enabling faster learning in challenging domains.

Published Link — arXiv Link — GitHub Link

Examples of discrete decision trees from Optimization Methods for Interpretable Differentiable Decision Trees Applied to Reinforcement Learning

Interpretable RL

After learning the weights of a differentiable decision tree for an RL task, it’s possible to convert the network into a discrete, ordinary decision tree while preserving performance. This offers an interpretable, small tree that can be used by the agent to improve human trust and efficiency.

Published Link — Blog Post

Recent writing

All posts →