Today we share more on PaLM-E! (http://palm-e.github.io) Thread with blog post link at the end. PaLM-E can do a lot of things across robotics, vision, and language… but let’s look at a few capabilities in detail, step by step

For one, “Let’s think step by step” comes to multimodal models! Zero-shot chain-of-thought has been one of these emergent behaviors that has caught considerable interest in researching LLM capabilities… With PaLM-E-562B, zero-shot visual chain-of-thought comes “included”.

Multimodal chain-of-thought can be very helpful to get a sense of what the model is picking up on. While the question here is only a 1-bit (yes/no) answer, the chain-of-thought provides much more than 1 bit of information on what the model sees.

Here’s a many-step zero-shot CoT example (prompt by @Ayzaan Wahid!). Note large VQA training datasets (VQAv2, OKVQA, etc.) typically only have 1-, 2-, 3-word answers, so these many-step answers are considerably out-of-distribution.

Here’s another multimodal reasoning question addressed with chain-of-thought, this time doing visual math questions, no OCR required despite needing spatial-textual context, just does everything all in one model. This prompt by @Fei Xia!

Moving on from chain-of-thought, another capability of PaLM-E that “just comes included” is the ability to do multi-image reasoning… despite only ever being trained on single-image examples.

For this multi-image reasoning, since PaLM-E flexibly supports multimodal sentences, it can answer questions about specific relationships between images. While the previous example was a “what matches?” question, this one is a “what’s different?” question.

Extending multi-image further, we can do more than just 2 images... For this, let’s look at a capability we showed last year with Socratic Models (https://socraticmodels.github.io/…, led by @Andy Zeng), where we could do long-form egocentric video understanding, some examples here:

In Socratic Models, this worked by writing out a language-based world-state history – a timestamped log of textually-represented events:

With PaLM-E, we can do this end-to-end, all in one model, with no explicit textual intermediate stage. A wide set of temporal/visual reasoning capabilities are in scope. Lots of potential AR & Robotics applications here.

Quantitatively, PaLM-E-562B sets a new state-of-the-art, 66.1, on OK-VQA dataset. This number is also achieved with a *generalist* (one model), also trained on diverse robotics and vision data, and without a final task-specific finetuning stage on OK-VQA.

In a recent-ish podcast (recorded in October, released in January), I had a few comments on where large-scale multimodal models are headed and “one big model” approach... (see around 42 minutes here) https://twitter.com/gradientpub/status/1611408050987876355?s=20…

Interesting to look back at that interview now – finishing out the results of PaLM-E has definitely shifted my perspective! (btw, thanks @The Gradient (sigmoid.social/@thegradient) + @sigmoid.social/@andrey for having me on!)

Another capability of PaLM-E-562B is that it's, quantitatively, an excellent language model. Roughly as good as PaLM-540B. Notable that scaling the model significantly reduces catastrophic language forgetting https://twitter.com/DannyDriess/status/1632904698108821505?s=20…

But of course, avoiding forgetting is a low bar :) I’ve been glad to see that folks are picking up on the **transfer** story of PaLM-E – for example see r/MachineLearning: https://www.reddit.com/r/MachineLearning/comments/11krgp4/r_palme_an_embodied_multimodal_language_model/…

For robotics, PaLM-E is a rapid learner of new planning tasks, requiring only a handful of samples to start generalizing well in a given domain. Here we plot PaLM-E sample complexity relative to baseline – the difference is solely transfer learning. (Subset of Table 2)

PaLM-E can do few-shot and zero-shot generalization – it never had training data for “push the red blocks to the coffee cup”, and only had ever seen this coffee cup in 3 images. See website for the never-before-seen “turtle” object too.

Towards wrapping up here, in addition to all our co-authors, I want to especially give a shout-out and thanks to all the Google teams who helped make the effort possible! Especially the folks behind training PaLM and the large ViTs from which PaLM-E is built.

Here is the link for the blog post: https://twitter.com/GoogleAI/status/1634252301303947272?s=20…

And I want to close with a Haiku. Prompt in gray by @Brian Ichter, and the completion written by PaLM-E-562B: