getUserMedia started life as a nice little API that allowed the developer to grab video and/ or audio from a device.
navigator.getUserMedia({audio: true, video: true}, success, error);
It’s shipping for camera access in Opera 12 desktop and Opera Mobile 12 and implemented in Chrome nightlies.
The spec has ballooned since I last looked at it in detail. There is a new “constraints API” which adds significant complexity, while the language of the spec isn’t rigorous (yet). This is likely to delay standardisation yet the simplest use-cases that it originally met are the core features.
Originally, it had a method for authors to hint to the user agent which camera to use – you could suggest “user” for the camera that faces you, or “environment” for the camera that faces outwards. This has disappeared from the spec, to be dealt with later, once the complex capabilities & constraints API is fleshed out.
Should the author be able to specify which camera should be used? It seems to me that this would be a good idea. For example, my mobile phone camera is usually set to use the rear-facing “environment” camera, but if I were to start up a video-conferencing app (or any of the fun “Photo booth” apps on Shiny Demos) I’d like the app to switch to using the forward-facing “user” camera without my having to manually switch camera. (Some devices don’t have hardware switches, so would require me to go to a system preference to change which camera to use, which would be a major pain in the arse.)
I strongly feel that the Working Group should allow authors to choose which camera to use before fleshing out advanced capabilities & constraints APIs which are beyond the use-cases of many authors.
However, although most phones will continue to have two cameras, some devices like the Mito 611 have more than two. How do we write a specification for hinting which camera to use with multiple cameras?
All the cool kids are doing responsive design, it seems. This means fluid designs, having some hot media query action to reformat your layout depending on screen size, and ensuring your images are flexible so they don’t break out of container, generally by setting them to {max-width:100%;}.
Having images scaling down presents a problem though, if you’re a performance-minded sort of chap(ette). Why send a 300K 400 x 800 image that would look lovely on tablet device attached to wifi, but which takes ages to download on a 3G phone, and which gets resized by the browser to fit a 320px wide screen? Not only are you wasting your user’s bandwidth and time, but not every browser is created equal and not every resize algorithm makes pleasing end results. The CSS 4(!) image-rendering property can help with this but it only hints to the browser.
Sending the right-sized image to devices without wasting bandwidth is one of the knottiest problems in cross-device and responsive design at the moment. In the 24ways advent calendar series of articles, the subject has been discussed twice in eight articles (by Matt Wilcox and Jake Archibald). There are numerous other techniques, as well, such as tinySrc and the Filament Group’s Responsive Images.
All these are very clever, different solutions to solve the same problem, and they all rely on scripts, or cookies, or server-side cleverness or (in the case of Jake’s ideas) dirty hacks and spacer GIFs. I’m reminded of Image Replacement techniques from more than 6 years ago, which were over-engineered solutions to the problem better solved by CSS web fonts.
Let’s recap. We have several images, of different sizes, and want to send only the most appropriately-sized image to the browser. The circumstances differ. The canonical use case is to send smaller, lower-resolution images to smaller screen-sizes on the assumption that connection speed is slow and they have low-resolution displays. This isn’t the case, though. Some people are using retina displays on fast wifi networks. SO, while currently CSS Media Queries allow us to detect screen width and pixel density, we need new media features such as network speed/ bandwidth.
The DAP is working on a Network Information API, there’s a Phonegap version for native apps, and a Modernizr detect, but using script for this seems much harder than being able to access it via Media Queries, and if you just want to rearrange CSS layout, CSS is the place to detect it. (Purists may argue that network connection isn’t a characteristic of device, but in your face, purists!)
Once you have a media query, you can swap images in and out using the CSS content property:
<img id=thingy src=picture.png alt="a mankini">
…
@media all and (max-width:600px) {
#thingy {content: url(medium-res.png);}
}
@media all and (max-width:320px) {
#thingy {content: url(low-res.png);}
}
@media all and (network-speed:3g) {
#thingy {content: attr(alt);}
}
A browser that doesn’t support Media Queries, or doesn’t report “true” for any of the Media Queries shows the picture.png, which is the src of the img. A browser that is less than 600px replaces picture.png with medium-res.png. A 320px or narrower browser replaces picture.png with low-res.png. A browser that is only connected via 3g replaces the image with its alt text.
I first researched this technique in 2008 as a way of doing image replacement without extra markup (ironically enough). The first two media queries only works in Opera and Chrome at the moment, as they’re the only browsers that fully support content without :before or :after. (The network-speed media query works nowhere as I just made it up).
Recently, Nicolas Gallagher experimented with generated content for responsive images, and unfortunately discovered that there are no advantages to the technique because browsers always download picture.png, even if they never show it because they immediately replace it. Perhaps this could be optimised away by browsers, but there would still be the double-download problem with current browsers.
My mind turned to HTML5 video. Here we have an element that has the ability to specify multiple sources (because of different codecs) and can also switch sources according to media characteristics. It borrows syntax from Media Queries but puts them in the HTML:
<video>
<source src=high-res.webm media="min-width:800px">
<source src=low-res.webm>
<!-- fallback content -->
</video>
I firmly believe that if something as new-fangled as video can be responsive in a declarative manner (without needing script, APIs, cookies or server magic) then responsive images should be similarly simple.
Previously, on the mailing list, a new <picture> element was proposed. I mashed up that idea with the already-proven video code above and came up with a strawman:
<picture alt="angry pirate">
<source src=hires.png media="min-width:800px">
<source src=midres.png media="network-speed:3g">
<source src=lores.png>
<!-- fallback for browsers without support -->
<img src=midres.png alt="angry pirate">
</picture>
This would show a different source image for different situations. Old (=current) browsers get the fallback img element. As I said, this is a strawman; it ain’t pretty. But it would work. (It’s discussed in my recent Smashing Magazine article HTML5 Semantics.)
I prefer the media queries to be in the HTML for two reasons: you don’t need to have ids or complex selectors to target a particular image, and (more importantly) many content authors use a CMS and have no ability to edit the CSS. (Although the nasty <style scoped> could solve this.)
On the other hand, I might be over-engineering the whole problem. I chatted informally with my colleague Anne van Kesteren, glamorous Dutch WHATWG inner circle member. There’s a school of thought that says everything will be 300ppi and networks will be fast enough, so this is really an intermediate problem until everyone starts using highres graphics and all displays go from 150 to 300. Standards are long term, and by the time we have a standardised version, the problem might have gone away.
What do you think?
(Matt Machell pithily pointed out “if only browsers hadn’t forgotten the old school lowsrc attribute for images“.)
Looks like our chums at WHATWG are discussing this too.