Enable <picture> element support

Current in progress PR: #73

Related trac ticket: https://core.trac.wordpress.org/ticket/42920

See this comment for a detailed description: #21 (comment)

Original description:

Use picture element to enable modern formats like AVIF with fallback to more widely supported format like WebP or JPEG, including responsive srcset markup like current images. Issue - adamsilverstein/modern-images-wp#1 . references: The anatomy of responsive images - JakeArchibald.com & markup: I just want an image on my page

Support markup with one or more image fallbacks
Enable serving WebP/AVIF with JPEG fallback, for more complete browser support.

A good initial test case for this feature would be generating WebP images and JPEG images on upload, then using the picture element to specify the WebP with JPEG as the fallback. Although this scenario isn't likely to be that useful, it will set the path for the same approach with other formats.

I've been digging into how images are stored and generated a bit further and will post an update here soon with a proposed path forward.

I started working on a POC in a branch, PR incoming.

Some good reading:

Responsive images - https://web.dev/serve-responsive-images/
Thepicture element: https://web.dev/learn/design/picture-element/

Some additional details about my findings and the approach:

Use cases for implementing the `<picture>` element

The picture element makes sense when the image has more than one mime type, or when the upload type does not match the original type and the user wants to provide a fallback. Picture element could also be used for art directed images.

Use Case 1: uploaded type converted for sub sizes (one mime type, different than original)

Description: User uploads JPEG, sub sized images are WebP. For maximum compatibility, user wants to display WebP in a picture element, falling back to the uploaded JPEG for browsers that don’t support WebP.

Expected output:

<picture>
	<source
		srcset="{WebP srcset}"
		sizes=""
	>
	<img
		alt=""
		src="{original image}"
	>
</picture>

Use Case 2: two mime types

Description: At some point in the future when a user uploads JPEG images, WordPress will be capable of converting these to AVIF images (AVIF support was added in PHP 8.1). To get the best performance and widest compatibility, the user wants to display a picture element showing first the AVIF format to supporting browsers, then falling back to JPEG images for other browsers.

Expected output:

<picture>
	<source
		type="image/avif"
		srcset="{AVIF srcset}"
		sizes=""
	>
	<img
		alt=""
		srcset="{JPEG srcset}"
		sizes=""
		src="{original image}"
	>
</picture>

Use Case 3: multiple mime types

Description: In this case the user wants to display a picture element with src sets for AVIF, falling back to WEBP when AVIF isn’t supported, and finally a fallback to JPEG when WEBP isn’t supported.

Expected output:

<picture>
	<source
		srcset="{AVIF srcset}"
		type="image/avif"
		sizes=""
	>
	<source
		srcset="{WebP srcset}"
		type="image/webp"
		sizes=""
	>
	<img
		alt=""
		srcset="{JPEG srcset}"
		sizes=""
		src="{original image}"
	>
</picture>

Sub sized image generation

When users upload an image to WordPress, the system generates sub sized images for front end display (a sub sized image is created for each available size smaller than the uploaded image, in the same mime type format as the upload). Data about the original image as well as each sub-sized image created is stored in post meta in an array with the key _wp_attachment_metadata. This meta data includes an array keyed on sizes that contains the sub-sized image data, including the file, width, height and mime type. In the future the sub sized images might not be in the same mime format as the original (as is the case when a JPEG is uploaded and WebP is output as the default), or may contain multiple mime type sub sized images, for example AVIF and JPEG images.

The default data for a typical image would look like this after uploading(image_meta truncated):

array (
  'width' => 1500,
  'height' => 1000,
  'file' => '2021/11/road-1.jpg',
  'sizes' =>
  array (
    'medium' =>
    array (
      'file' => 'road-1-300x200.jpg',
      'width' => 300,
      'height' => 200,
      'mime-type' => 'image/jpeg',
    ),
    'large' =>
    array (
      'file' => 'road-1-1024x683.jpg',
      'width' => 1024,
      'height' => 683,
      'mime-type' => 'image/jpeg',
    ),
    'thumbnail' =>
    array (
      'file' => 'road-1-150x150.jpg',
      'width' => 150,
      'height' => 150,
      'mime-type' => 'image/jpeg',
    ),
    'medium_large' =>
    array (
      'file' => 'road-1-768x512.jpg',
      'width' => 768,
      'height' => 512,
      'mime-type' => 'image/jpeg',
    ),
    'post-thumbnail' =>
    array (
      'file' => 'road-1-825x510.jpg',
      'width' => 825,
      'height' => 510,
      'mime-type' => 'image/jpeg',
    ),
  ),
  'image_meta' =>
 	array (  ),
  'original_image' => 'road-1.jpg',
)

Front end image display

WordPress automatically iterates through the post content and adds srcset attributes for images placed from the media library (images placed by URL don't work currently). It uses a regex to find media-library images, then calls wp_get_attachment_image to get each image source. wp_get_attachment_image in turn calls wp_get_attachment_image_srcset to generate the srcset.

The image source code for a typical image looks like this (line breaks for clarity):

<img
	loading="lazy"
	width="1024"
	height="683"
	src="https://wpdev.localhost/wp-content/uploads/2021/12/road-1-1024x683.jpg"
	alt=""
	class="wp-image-2617"
	srcset=
		"https://wpdev.localhost/wp-content/uploads/2021/12/road-1-1024x683.jpg 1024w,
		https://wpdev.localhost/wp-content/uploads/2021/12/road-1-300x200.jpg 300w,
		https://wpdev.localhost/wp-content/uploads/2021/12/road-1-768x512.jpg 768w,
		https://wpdev.localhost/wp-content/uploads/2021/12/road-1.jpg 1500w"
	sizes="(max-width: 1024px) 100vw, 1024px"
>

Detour: image edits and sub sizes

The wp_calculate_image_srcset function, includes code to handle images changed by edits in the media library where each edited file name gets an edit hash appended (eg. “-e1640909072390”) to the filename. Sub sized images of the edited image get the same appended hash so they can be matched to the edited image.

For example if you rotate an image using the media library tools, the edit action creates a series of new files with hash extensions, and the post meta ‘sizes’ array is updated to reference the new files. The edited file urls will then be used for srcset generation. Note that the original_image meta data does not change, and the original image is never edited or removed, WordPress never changes your uploaded image!

Edited image data:

array (
  'width' => 2560,
  'height' => 1920,
  'file' => '2021/12/IMG_1893-1-1-scaled-e1640909072390.jpg',
  'sizes' =>
  array (
    'medium' =>
    array (
      'file' => 'IMG_1893-1-1-scaled-e1640909072390-300x225.jpg',
      'width' => 300,
      'height' => 225,
      'mime-type' => 'image/jpeg',
    ),
    'large' =>
    array (
      'file' => 'IMG_1893-1-1-scaled-e1640909072390-1024x768.jpg',
      'width' => 1024,
      'height' => 768,
      'mime-type' => 'image/jpeg',
    ),
    'thumbnail' =>
    array (
      'file' => 'IMG_1893-1-1-scaled-e1640909072390-150x150.jpg',
      'width' => 150,
      'height' => 150,
      'mime-type' => 'image/jpeg',
    ),
    'medium_large' =>
    array (
      'file' => 'IMG_1893-1-1-scaled-e1640909072390-768x576.jpg',
      'width' => 768,
      'height' => 576,
      'mime-type' => 'image/jpeg',
    ),
    '1536x1536' =>
    array (
      'file' => 'IMG_1893-1-1-scaled-e1640909072390-1536x1152.jpg',
      'width' => 1536,
      'height' => 1152,
      'mime-type' => 'image/jpeg',
    ),
    '2048x2048' =>
    array (
      'file' => 'IMG_1893-1-1-scaled-e1640909072390-2048x1536.jpg',
      'width' => 2048,
      'height' => 1536,
      'mime-type' => 'image/jpeg',
    ),
  ),
  'image_meta' =>
  array (  ),
  'original_image' => 'IMG_1893-1-1.jpg',
)

Note that cropping an image directly in the block editor works differently - a new image is created in the media library with the name “-cropped” and cropped dimensions, all sub-sizes are re-generated for that image.

Enabling additional mime types, a cookbook

How can we enable additional mime types in WordPress to support uses cases 2 & 3?

For example, in use case 3, WordPress would create sub sized images in several formats when users upload images (in any supported format). Sub sized images would be created in AVIF, WEBP and JPEG formats.

Proposed approach

Internally, the sizes array keys can be any string, so when creating additional images beyond the first mime type - for each mime type we can prepend the mime type to the key names, so a WebP medium image would have the key images/webp-medium.
wp_calculate_image_srcset would be updated to accept an optional mime type.
- When a mime type is not included, the function would work as is, except it would filter out any additional mime type image sizes from the meta data.
The function wp_get_attachment_image would be updated to automatically output a picture element when alternate mime type sub sized images are found in the images post meta.
- Mime type order and availability for picture element support would be filterable on a per-image basis.
- When the original image doesn’t match the sub sized images, it could be used as a fallback
- When multiple mime types are provided, they can be used as source elements for the picture element with the final type used for the fallback image srcset.
Code must also consider image edits
- As described above we need to ensure edited images are handled properly.
Filters should be enabled so developers can adjust each part of the output, especially the source type, srsset and sizes elements. Existing filters should work to enable developer fine tuning, we should validate that and ensure all context is passed (eg mime type).
Consider whether to require theme support of html5 eg with current_theme_supports( 'html5' )

For the purposes of the plugin we can test approaches by filter the post content.

New DOM element node <picture> may break theme or custom CSS selectors. How are you going to fix it?
Does <img> element CSS class/style applies only to <img> or to other <source> tags or to <picture> tag?

@ddur

Hey @ddur, thanks for the questions:

New DOM element node may break theme or custom CSS selectors. How are you going to fix it?

Good point, we probably want to let themes opt into the behavior so they can adjust CSS accordingly. We could make opting in as simple as checking current_theme_supports( 'picture' ) (as suggested on the trac ticket). Maybe with core image blocks we can improve that by adjusting CSS?

Does element CSS class/style applies only to or to other tags or to tag?

Good question that needs more research; I'm guessing you would need to apply to the picture or both the img and picture elements, the element displayed may depend on browser support for the image format or for the picture element itself.

Reading up on the various possible uses for the picture element here: https://dev.opera.com/articles/responsive-images/ I created a demo pages with picture elements and linked elements for testing: preview & source

A few notes:

Chrome only seems to use responsive images correctly when sizes are provided as part of the media query (first example), the second/third examples use srcset with sizes, specifying the source with the type attribute and work in firefox and safari, in chrome dev the responsive image fail to work with this approach, maybe I have something incorrect in my markup? Appreciate any help here!
I created jpeg, webp and avif versions of images at various sizes.
I added a "WebP" tag to the WebP images so they are easy to identify. I don't have an editor that works with AVIF to add the tags there.

@adamsilverstein

@adamsilverstein is this something you are continuing to work on once back from the break? If so, could we assign the ticket to you please?

How about 'img' element and loading=lazy attribute? Does it apply to picture-source elements?

@ddur

@ddur great questions!

New DOM element node may break theme or custom CSS selectors. How are you going to fix it?

Good point, this has been raised before. One approach would be an opt-in approach for the progressive enhancement, themes would declare support for 'picture-element' to get the auto-feature for multiple mime types. Alternately, any theme or plugin could call wp_get_attachment_image with a parameter indicating they want a picture element.

Our main initial use case for this will be images with multiple mime types (for example AVIF with jpeg fallback), however we need to ensure other use cases (for example varying density or art direction) are also covered by the approach we choose.

Does element CSS class/style applies only to or to other tags or to tag?

I'm not sure, this needs more investigation and likely a developer note to show how to properly apply css when using the picture element.

How about 'img' element and loading=lazy attribute? Does it apply to picture-source elements?

Yes it does, according to this it is sufficient to set the attribute on the contained img tag.

@adamsilverstein

@adamsilverstein is this something you are continuing to work on once back from the break? If so, could we assign the ticket to you please?

I have self assigned, however this is a large task and could use several developers working on it, I will think about how we can break out smaller tasks.

We've been using <picture> tags for many years now on dozens of sites and it has been great. For legacy browsers we used to include a common set of JS to boot up support for HTML5 features in general, including the <picture> tag, but we haven't included that in several years.

I've never been a fan of the srcset attribute on <img> for responsive, so we just stuck to media queries if needed, and it has worked without any issues as far as I remember, Worst case, someone gets the <img> tag.

For the most part we use this as an upgrade to WebP for users, but we occasionally use it for art direction and then almost always to crop hero images.

I think the hardest leap for people from a styling perspective is that they need to consider the <picture> tag as basically a wrapper <div> around an image. So switching from just an <img> to one wrapped in a <picture> tag needs to be treated the same. Sometimes it isn't a problem, but if you are in a grid or flex context, the <img> might not behave how you initially expected, because the container might be getting partially styled instead.

Cross-posting with #996 (comment):

I just checked and Gutenberg is explicitly selecting the img tag in the Image block:
.wp-block-image {
	img {
		height: auto;
		max-width: 100%;
		vertical-align: bottom;
		box-sizing: border-box;
	}
This is done commonly in Gutenberg:
https://github.com/search?q=repo%3AWordPress%2Fgutenberg+%2Fimg+%7B%2F+language%3ASCSS&type=code
https://github.com/search?q=repo%3AWordPress%2Fgutenberg+%2Fimg%2C%2F+language%3ASCSS&type=code

This is also super common in themes generally: https://wpdirectory.net/search/01HX83JEWYSX880A2CN2W8G5GA

This was a big headache for the AMP plugin when we converted img to amp-img, as we had to rewrite all of the selectors mentioning img to use amp-img. It is possible, but only if you are processing all the CSS.

Update: It turns out that descendant selectors targeting img do work with the picture element, but it doesn't work with child selectors. See #996 (comment).

Navigation Menu

Repository navigation

Enable <picture> element support #21

Use cases for implementing the `<picture>` element

Use Case 1: uploaded type converted for sub sizes (one mime type, different than original)

Expected output:

Use Case 2: two mime types

Expected output:

Use Case 3: multiple mime types

Expected output:

Sub sized image generation

Front end image display

Detour: image edits and sub sizes

Enabling additional mime types, a cookbook

Proposed approach

17 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Enable <picture> element support #21

Description

Activity

adamsilverstein commented on Dec 30, 2021

adamsilverstein commented on Dec 31, 2021

Use cases for implementing the <picture> element

Use Case 1: uploaded type converted for sub sizes (one mime type, different than original)

Expected output:

Use Case 2: two mime types

Expected output:

Use Case 3: multiple mime types

Expected output:

Sub sized image generation

Front end image display

Detour: image edits and sub sizes

Enabling additional mime types, a cookbook

Proposed approach

ddur commented on Jan 2, 2022

adamsilverstein commented on Jan 3, 2022

adamsilverstein commented on Jan 3, 2022

eclarke1 commented on Jan 11, 2022

ddur commented on Jan 13, 2022

adamsilverstein commented on Jan 21, 2022

adamsilverstein commented on Jan 21, 2022

17 remaining items

cjhaas commented on Apr 5, 2022

westonruter commented on May 6, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Use cases for implementing the `<picture>` element