Workflow diagram showing 166 experiment configurations submitted to Slurm, dispatched as parallel training jobs to NVIDIA A100 GPU nodes, and connected to shared storage for checkpoints, logs, and metrics

Researcher Story

Teaching Vision–Language AI to See with Fewer Tokens

A compact, self-supervised token summarizer could help vision–language AI systems process images with substantially less computational overhead.

Modern artificial intelligence systems that connect images and language power applications such as image search, automatic captioning, and visual question answering. These systems typically represent each image as hundreds of visual tokens—small units of image information that can be processed by a language model.

Processing so many tokens makes inference computationally expensive and increases the amount of information that the language model must handle.

This research develops a compact, trainable token summarizer that is inserted into a pretrained vision model. The summarizer compresses hundreds of visual tokens into a much smaller collection of informative summary tokens while preserving the information needed by the larger model.

The summarizer is trained using a self-supervised approach. It learns by predicting and matching the pretrained model’s internal representations, so it does not require human-labeled training data or retraining of the large pretrained model.

How RCD Supports the Research

This project is empirical and computationally intensive. Progress depends on running many controlled training experiments on large models and comparing the results head to head. All of that computation runs on Tennessee Tech’s Research Computing & Data cluster.

Each experiment is submitted as a GPU training job through Slurm and scheduled on the cluster’s NVIDIA A100 GPU nodes. A typical run uses one A100 GPU, approximately 14 CPU cores, and 96 GiB of memory for several hours. The largest experiments may run considerably longer.

Regular checkpoints and automatic requeue allow an interrupted job to resume from its most recent saved state. This improves the resilience of long-running experiments while helping keep the shared GPU resources productively and fairly allocated.

Dependency-aware batch submission allows a single command to launch and correctly order dozens of related experiments. Shared cluster storage collects the checkpoints, logs, and performance metrics generated by every run for later analysis.

RCD Resources at a Glance

  • NVIDIA A100 GPU nodes scheduled through Slurm
  • Approximately 14 CPU cores and 96 GiB of memory per job
  • Multi-hour and long-running GPU training jobs
  • Checkpointing and automatic requeue for interrupted jobs
  • Dependency-aware submission of ordered experiment batches
  • Shared storage for checkpoints, logs, and metrics
  • 166 experiment configurations across nine sweep generations and three model families

This infrastructure makes it possible to conduct the study at the scale it requires. To date, the work spans nine generations of experimental sweeps and 166 distinct configurations across three model families—a volume of careful, one-factor-at-a-time experimentation that would be impractical without RCD’s scheduled GPU hardware, resilient job workflows, and shared storage.

Research Team

Mahmoud Attallah
Graduate Research Assistant
Department of Computer Science


Dr. Amr Hilal
Faculty Advisor
Department of Computer Science

Bar chart summarizing 166 experiment configurations across nine sweep generations and three model families. The largest generation contains 57 LLaVA experiments, followed by 29 multi-host experiments, while the remaining generations contain between 7 and 17 experiments each.

Nine generations of experimental sweeps have produced 166 configurations across three vision–language model families and approximately 190 completed runs. Each bar represents experiments performed, not model performance.