The proposal
The July 28 preprint MODUS, coauthored by Apple Machine Learning Research and academic partners, handles text, RGB, depth, surface normals, edges, segmentation, grounding, and DINO features in one decoder-only model. A common token sequence replaces modality-specific heads, losses, and task pipelines.
New compositions
Beyond direct mappings such as RGB to depth or text to image, the model performs chained generation through an intermediate modality, such as RGB to edge to surface normal. It can also predict grounding or VQA outputs from generated images and use that second modality to rank cross-modal consistency.
Reported evaluation
The paper reports MMMU 51.1, GenEval 0.81, and RefCOCO validation 54.5 from one model. Selecting among four text-to-image candidates using its own VQA and grounding scores raises GenEval from 0.81 to 0.84. Some baseline results were reproduced by the authors, and the paper discloses that several authors also developed the compared 4M model.
Limits and significance
This is not peer reviewed and does not beat every specialist. Its reported MMMU result, for example, is below the GPT-4o number in the paper's table. The contribution is composability and verification across representations, not one universal top score. Code and materials are open, but licensing, compute, bias, and failure cases need deployment-specific review.