Google’s New Auto-Caption Algorithm Could Forever Change Image SEO

By Harris Roberts

Google may soon be changing their image ranking algorithm. This week they revealed a new technology that can auto-caption complex images.

It has always been relatively easy to rank images in Google’s image search, but this new technology may allow Google to caption and rank photos without the alt tag.  We predict this will change soon. More importantly, if it does, websites misusing alt tags may be negatively penalized.

Related: Check out our 7 favorite productivity tools to improve efficiency at your marketing agency.

Until now, Google has had no way to understand the content on an image. This week Google research released a new report detailing the capabilities of this technology.

Automatically captioned: “Two pizzas sitting on top of a stove top oven.”

The algorithm recognizes individual objects and then creates a caption. In an example, their algorithms would caption this picture as “Two pizzas sitting on top of a stove top oven”. Google could use this technology to automatically caption photos and use that data to create image search results.


Today an alt attribute is used by Google to understand what the content of an image is. It is also helpful for users with visual impairments who may be using a screen reader. Google then uses this description to rank images in a search.

It is very easy to rank an image for a set longtail keyword. Google recommends using less than 10-15 words to describe the image. It is also thought that the keywords in your alt tags help your page’s overall SEO.


What this new technology allows Google to do is scan images and create incredibly accurate captions. These captions could someday make the alt tag obsolete in image search.

If this is true, then Google can compare your existing alt tag and their version (hopefully, you’ve been using a genuine marketing strategy!). Theoretically, they could even penalize websites and images that have high “mismatch scores”.

Historically, Google has penalized websites that have over optimized content. This could have an impact on marketers or marketing agencies who are using deceptive alt tags to game the Google search algorithm.

For example, the image at the top of this post could have the alt tag “Best Margarita Pizza In Boston Massachusetts”. It’s not blatantly off, but obviously targeted towards a keyword search and not the image content.


We always recommend using white hat techniques and great inbound content to improve your search engine rankings. Nonetheless, that’s not to say we haven’t thrown a few relevant keywords into our image alt tags.

Until now, there has been no real way for Google to check the accuracy of your alt tags! With this new image captioning technology though, Google could (and most likely will) check the accuracy of your alt tags.

If you are using alt tags to game Google’s image rankings at your marketing agency, we recommend revising your alt tags. We don’t predict that the technology will roll out this year, but, as the accuracy increases, it seems obvious for Google to add to their search algorithm.

Google’s goal is to provide the best search results to its users. Using this new technology, they will be able to objectively filter images without human interpretation.

If you disagree or have thoughts, please add them in the comments below or tweet me at @harrisroberts!