10799 2010-09-29 03:57:12 +0000 drawImage/pattern filters underspecified 2010-12-08 00:19:11 +0000 1 1 1 Unclassified HTML WG pre-LC1 HTML Canvas 2D Context (editor: Ian Hickson) unspecified All All RESOLVED FIXED P2 normal --- 1 vladimir ian excors ian mike public-html-admin public-html-wg-issue-tracking public-html-bugzilla oldest_to_newest 39857 0 915 vladimir 2010-09-29 03:57:12 +0000 Created attachment 915 testcase This came from mozilla bugzilla bug https://bugzilla.mozilla.org/show_bug.cgi?id=600390. The canvas spec doesn't really say anything about filters, and especially how they apply to drawImage/patterns. The attached testcase demonstrates the problem -- it doesn't render identically on any of the canvas-implementing browsers (firefox, opera, chrome, ie9 -- don't have safari handy to test). The two sets in the testcase are using a canvas as a source and using an image, both with the same contents; usually there's no difference between the two. The large squares are, from left to right: 1) straight drawImage 0,0,60,60 -> 0,0,300,300 2) stairstepped drawImage, doing 2 draws per horizontal band 3) stairstepped fill() using a no-repeat pattern source 4) 300x300 rectangle fill, but with source offset by 150,150 5) same as #4, but rotated as well to examine antialiasing effects I -think- Firefox renders it the most "correctly", even though it's almost never what you want; but mathematically it's correct, and it causes the least change when under transforms -- with no repeat, the area outside of the source's defined bounds is transparent black; so at edge pixels when scaling up, when you sample acros that boundary, you get half of the transition between transparent black and the edge color. IE9 renders "what I want", but it's also the most difficult to specify precisely. Opera also gets close, except all their paths are offset by 5 pixels for some reason. Chrome is quite bad here, because I think it's trying to be smart about using the source rect for drawImage as a standalone image in its own right -- this causes problems if you, for example, try to draw a scaled-up image piecemeal, as you'll get seams. It also uses pad/clamp behaviour for a pattern that's specified as no-repeat -- however, the spec doesn't actually define no-repeat, so their interpretation is as good as any other. 40103 1 ian 2010-09-30 08:53:28 +0000 Philip`: If you have any advice here, before I take a look properly, experience suggests it would be very helpful. 40325 2 excors 2010-09-30 22:32:57 +0000 Hmm, I'm assuming this is something that can resolved without having to change the spec's basic assumption that it's drawing to an infinitely precise buffer (which is why it doesn't talk about antialiasing or rounding etc). Even with infinitely precise output, it matters how you compute each point from the low-resolution input image, so it seems a useful thing to specify. It sounds like perhaps the desired behaviour is for the computation of the sampled colour at (possibly non-integer) position (x, y) on a w*h bitmap image to be something like (in random pseudocode since it helps me think about this): filtered(x, y, clamped) = if not (0.0 <= x <= w and 0.0 <= y <= h) return transparent black else compute filtered value at (x, y) (using standard bilinear interpolation or nearest neighbour or whatever), such that whenever the filtering algorithm chooses to sample the image at pixel index (ix, iy): if clamped: clamp ix to (0 .. w-1), iy to (0 .. h-1); else: wrap ix to (0 .. w-1), iy to (0 .. h-1); then return the pixel at the new (ix, iy). drawimage(x, y) = if not (sx <= x <= sx+sw and sy <= y <= sy+sh) return transparent black else filtered(x, y, true) pattern_norepeat(x, y) = filtered(x, y, false) pattern_repeatx(x, y) = filtered(x % w, y, false) pattern_repeaty(x, y) = filtered(x, y % h, false) pattern_repeat(x, y) = filtered(x % w, y % h, false) The relevant points are: * drawImage clamps to the edge pixels when doing bilinear interpolation, instead of wrapping or padding with transparency outside the image. * All patterns wrap instead of clamping (even no-repeat). * drawImage clamps to the edge pixels of the whole image, not to the edges of the source rectangle segment of the image. * Areas of patterns outside the image region are transparent black. It looks like this is what IE9 does. Opera mostly does this, except patterns get clamped in their non-repeating directions (e.g. repeat-x wraps on left/right edges and clamps on top/bottom; no-repeat clamps on both) (and there's the weird offset thing). Firefox interpolates with transparency around the edges (instead of wrapping/clamping), except in repeating patterns where it wraps. Chrome is nothing like this and all its scaling looks terrible if you go above 16x. 40577 3 ian 2010-10-04 23:00:23 +0000 Could I convince one of you to write this up formally? I'd be happy to massage such text into RFC2119ese, but I'm concerned that I don't really understand this issue well enough to write text that makes sense. (In particular, I can't see any difference between what any of the four browsers on my platform are doing with the attached test case — they all look identical to me.) 40588 4 vladimir 2010-10-04 23:37:33 +0000 Hm, I'm not sure what Philip wrote up is correct/possible to implement efficiently.. but I'd have to study it in more detail. 40590 5 vladimir 2010-10-04 23:42:42 +0000 I'm surprised you don't see a difference.. for reference, here's Firefox and Chrome: http://i.imgur.com/QLrAo.png And Opera and IE9: http://i.imgur.com/ugLYQ.png Some differences: - fuzzy edge vs sharp edge in the fourth/fifth large squares - note fuzzy inner edges vs sharp inner edges in the first three squares, esp on chrome where bits of the second square are incorrectly sharp (due to the stairstep rendering) - blue fill on fourth and fifth in chrome instead of white (transparent) fill - no linear filtering in third, fourth and fifth in chrome, compared to first and most of the second 40598 6 excors 2010-10-05 00:03:28 +0000 My intention was that this was basically EXTEND_REPEAT for patterns and EXTEND_PAD for images (given that you set the extend mode for a gfxPattern representing the whole image, not just the source rectangle), so I think I'm missing where the problem would be (unless it's that those EXTEND modes can't be implemented efficiently). Hmm, I just realised this EXTEND_PAD thing interact with the change hidden in http://html5.org/tools/web-apps-tracker?from=5372&to=5373 ("Pixels of the source rectangle that are not entirely within the source image must be treated as transparent black") - not sure if that's going to be an issue. 40601 7 vladimir 2010-10-05 00:31:19 +0000 Nope, PAD and REPEAT are totally fine hw-wise, but I'm not sure how that works with the "no-repeat" canvas pattern mode.. you still have the question of what to do at the edges, no? According to what you put above, for no-repeat it would just use x y without clamping, thus giving transparent black.. which would give you the fuzzy edges on scaled-up drawImage. (This is essentially what firefox does currently.) 40780 8 excors 2010-10-07 15:14:41 +0000 What I (I think) put above for no-repeat patterns was that it wouldn't clamp, but would instead wrap. It's equivalent to repeat patterns for the purposes of filtering, it's just cut off sharply to transparent outside the area of the first repetition. (This is what IE9 seemed to do.) If no-repeat were to have some other filtering behaviour, then repeat-x and repeat-y ought to have that behaviour on their non-repeating edges (for consistency). They should still have the wrap behaviour on their repeating edges, which means different behaviour in different directions on one pattern. Opera does that but it looks like the Cairo API assumes a single mode for both directions - would that be a problem? 41276 9 ian 2010-10-14 01:00:06 +0000 Would it be possible to have the proposal written in terms that the spec uses? Pseudo-code isn't really workable since we'd have to define the syntax of the pseudo-code to make it formal, which is why I don't really think we can use comment 2 as written (even with RFC2119 massaging). 42013 10 ian 2010-11-03 08:26:13 +0000 Philip, I'm reassigning this to you on the basis that I have no idea what I'm doing here and you do. Please feel free to reassign it to me if you can't do anything here. I likely would just close it NEEDSINFO though. If you are able to do something, all I need is text that says the technical bits, so that I can then massage it into spec prose. 43148 11 excors 2010-12-07 17:25:18 +0000 (Sorry, been a bit distracted and didn't get around to this earlier.) I think this should do what I suggested (though if this text is adopted I make no guarantees that I will not file further bugs saying it's all wrong): For drawImage: Remove the text "Pixels of the source rectangle that are not entirely within the source image must be treated as transparent black." Somewhere in this section, add the text: """ If the original image data is a bitmap image, the value painted at a point in the destination rectangle is computed by filtering the original image data. The user agent may use any filtering algorithm (for example bilinear interpolation or nearest-neighbor). When the filtering algorithm requires a pixel value from outside the original image data, it must instead use the value from the nearest edge pixel. (That is, the filter uses 'clamp-to-edge' behavior.) """ For createPattern, add the text: """ If the original image data is a bitmap image, the value painted at a point in the area of the repetitions is computed by filtering the original image data. The user agent may use any filtering algorithm (for example bilinear interpolation or nearest-neighbor). When the filtering algorithm requires a pixel value from outside the original image data, it must instead use the value from wrapping the pixel's coordinates to the original image's dimensions. (That is, the filter uses 'repeat' behavior, regardless of the value of <var>repetition</var>.) """ 43150 12 ian 2010-12-07 20:56:56 +0000 EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document: http://dev.w3.org/html5/decision-policy/decision-policy.html Status: Accepted Change Description: see diff given below Rationale: Seems reasonable. 43151 13 contributor 2010-12-07 20:57:05 +0000 Checked in as WHATWG revision r5708. Check-in comment: <canvas> drawImage() and pattern filtering issue http://html5.org/tools/web-apps-tracker?from=5707&to=5708 43156 14 vladimir 2010-12-07 21:16:59 +0000 (In reply to comment #11) > For createPattern, add the text: > > """ > If the original image data is a bitmap image, the value painted at a point in > the area of the repetitions is computed by filtering the original image data. > The user agent may use any filtering algorithm (for example bilinear > interpolation or nearest-neighbor). When the filtering algorithm requires a > pixel value from outside the original image data, it must instead use the value > from wrapping the pixel's coordinates to the original image's dimensions. (That > is, the filter uses 'repeat' behavior, regardless of the value of > <var>repetition</var>.) Hm, isn't this inconsistent? For example, if the pattern uses no-repeat, and is used to fill a rectangle larger than the pattern image size, this seems to say that the no-repeat setting is ignored. That seems odd and unexpected. The clamp-to-edge behavior is fine for drawImage, but for patterns, I think the only thing that mathematically makes sense is for the area outside of the source pixels to be treated as transparent black for no-repeat. 43160 15 excors 2010-12-07 22:16:11 +0000 By "in the area of the repetitions", I meant the area where the previous paragraph says the pattern will be painted (anchored in a certain point and repeated in certain directions). Outside that area, the pattern won't be painted at all - it's only inside the area that the wrapping filter is used. So no-repeat patterns and drawImage will both draw a single copy of the image with sharp edges when scaled up, but will filter differently when computing the pixels just inside the edges. An alternative way to see it is that the pattern is always based on an infinitely-repeating-in-both-directions image, with filtering applied to that, and then the repeat-x/y/no-repeat setting is cropping the resulting image in one or both directions. This is similar to how drawImage takes the entire input image, filters that, and then crops to the specified source rectangle, rather than cropping before filtering. So I think it's logically justifiable in that way. 43162 16 vladimir 2010-12-07 22:24:07 +0000 Maybe, though I don't think the language makes that clear. (If I was implementing canvas reading just the spec, I'd be scratching my head at that language.) However, you're basically describing an algorithm that's more than just sampling from a pattern with repeat; I'm not sure there's any need to do that, given that canvas already provides the primitives needed to get the effect you describe (draw with a repeating pattern, clip to a rectangle). 43168 17 excors 2010-12-08 00:19:11 +0000 The language isn't ideal, but I'm not sure how better to describe it in prose - if you can suggest changes that would make it clearer, that would be great. Canvas provides the primitives to emulate any filtering effect (take the source image and surround it with transparency or repeats or clamped edges, then clip to a rectangle) so I'm not sure what you mean with that. The advantage I see in the repeat-in-both-directions pattern filtering is just that it seems the easiest way for browsers to converge (without being a clearly crazy behaviour to converge on). IE appears to implement it already; Opera is not far off. Firefox doesn't support repeat-x or repeat-y yet so it will have to change in any case, and per comment 8 it looks like the Cairo API doesn't like different extend modes in different directions, so it will be simpler to use consistent modes. Is this incorrect? 915 2010-09-29 03:57:12 +0000 2010-09-29 03:57:12 +0000 testcase t2.html text/html 2967 vladimir PCFET0NUWVBFIEhUTUw+Cgo8aHRtbD4KPGhlYWQ+CjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+CmNh bnZhcywgaW1nIHsKICAgIGJvcmRlcjogMXB4IHNvbGlkIGJsYWNrOwogICAgbWFyZ2luOiA1cHg7 Cn0KPC9zdHlsZT4KPC9oZWFkPgo8Ym9keT4KPHA+Q2FudmFzPC9wPgo8Y2FudmFzIGlkPSJjMSIg d2lkdGg9IjYwIiBoZWlnaHQ9IjYwIj48L2NhbnZhcz48YnI+CjxjYW52YXMgaWQ9ImMyIiB3aWR0 aD0iMzAwIiBoZWlnaHQ9IjMwMCI+PC9jYW52YXM+CjxjYW52YXMgaWQ9ImMzIiB3aWR0aD0iMzAw IiBoZWlnaHQ9IjMwMCI+PC9jYW52YXM+CjxjYW52YXMgaWQ9ImM0IiB3aWR0aD0iMzAwIiBoZWln aHQ9IjMwMCI+PC9jYW52YXM+CjxjYW52YXMgaWQ9ImM1IiB3aWR0aD0iMzAwIiBoZWlnaHQ9IjMw MCI+PC9jYW52YXM+CjxjYW52YXMgaWQ9ImM2IiB3aWR0aD0iMzAwIiBoZWlnaHQ9IjMwMCI+PC9j YW52YXM+CjxwPkltYWdlPC9wPgo8aW1nIGlkPSJpbWcxIiBzcmM9ImRhdGE6aW1hZ2UvcG5nO2Jh c2U2NCxpVkJPUncwS0dnb0FBQUFOU1VoRVVnQUFBRHdBQUFBOENBWUFBQUE2L05seUFBQUFiVWxF UVZSb2dlM1B3UWtBSUF3RXdmVGZ0TFlnS09iUUlkd3pzRk5WWS95MTlnQmdZT0FyNE5WTCtnVUdC dTZQQmdZR0JnWUdEb29HQmdZR0JnWU9pZ1lHQmdZR0JnNktCZ1lHQmdZR0Rvb0dCZ1lHQmdZT2ln WUdCZ1lHQmc2S0JnWUdQZ04rWnUwQndNREFHNXZZZ3dXRnEwdUErUUFBQUFCSlJVNUVya0pnZ2c9 PSI+PGJyPgo8Y2FudmFzIGlkPSJjNyIgd2lkdGg9IjMwMCIgaGVpZ2h0PSIzMDAiPjwvY2FudmFz Pgo8Y2FudmFzIGlkPSJjOCIgd2lkdGg9IjMwMCIgaGVpZ2h0PSIzMDAiPjwvY2FudmFzPgo8Y2Fu dmFzIGlkPSJjOSIgd2lkdGg9IjMwMCIgaGVpZ2h0PSIzMDAiPjwvY2FudmFzPgo8Y2FudmFzIGlk PSJjMTAiIHdpZHRoPSIzMDAiIGhlaWdodD0iMzAwIj48L2NhbnZhcz4KPGNhbnZhcyBpZD0iYzEx IiB3aWR0aD0iMzAwIiBoZWlnaHQ9IjMwMCI+PC9jYW52YXM+CgoKPHNjcmlwdCB0eXBlPSJ0ZXh0 L2phdmFzY3JpcHQiPgoKdmFyIGMgPSBbXSwgY3ggPSBbXTsKZm9yICh2YXIgaSA9IDE7IGkgPD0g MTE7ICsraSkgewogICAgY1tpXSA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKCJjIiArIGkpOwog ICAgY3hbaV0gPSBjW2ldLmdldENvbnRleHQoIjJkIik7Cn0KCnZhciBpbWcxID0gZG9jdW1lbnQu Z2V0RWxlbWVudEJ5SWQoImltZzEiKTsKCmN4WzFdLmZpbGxTdHlsZSA9ICIjMDAwMGZmIjsKY3hb MV0uZmlsbFJlY3QoMCwgMCwgNjAsIDYwKTsKY3hbMV0uZmlsbFN0eWxlID0gIiMwMGZmMDAiOwpj eFsxXS5maWxsUmVjdCgxMCwgNSwgMTAsIDUwKTsKY3hbMV0uZmlsbFJlY3QoMjUsIDUsIDEwLCA1 MCk7CmN4WzFdLmZpbGxSZWN0KDQwLCA1LCAxMCwgNTApOwoKZnVuY3Rpb24gc3RhaXJTdGVwKHNy YywgZHN0Y3gsIGRzdGN4MikgewogICAgZHN0Y3guY2xlYXJSZWN0KDAsIDAsIDMwMCwgMzAwKTsK CiAgICBkcCA9IGRzdGN4Mi5jcmVhdGVQYXR0ZXJuKHNyYywgIm5vLXJlcGVhdCIpOwoKICAgIGZv ciAodmFyIGkgPSAwOyBpIDwgNjA7ICsraSkgewoJZHN0Y3guZHJhd0ltYWdlKHNyYywKCQkJaSwg aSwgNjAtaSwgMSwKCQkJaSo1LCBpKjUsIDMwMC1pKjUsIDUpOwoKCWlmIChpID4gMCkgewoJICAg IGRzdGN4LmRyYXdJbWFnZShzcmMsCgkJCSAgICAwLCBpLCBpLCAxLAoJCQkgICAgMCwgaSo1LCBp KjUsIDUpOwoJfQoKCWRzdGN4Mi5zYXZlKCk7Cglkc3RjeDIuYmVnaW5QYXRoKCk7Cglkc3RjeDIu cmVjdChpKjUsIGkqNSwgMzAwLWkqNSwgNSk7Cglkc3RjeDIuc2NhbGUoNSwgNSk7Cglkc3RjeDIu ZmlsbFN0eWxlID0gZHA7Cglkc3RjeDIuZmlsbCgpOwoJZHN0Y3gyLnJlc3RvcmUoKTsKCglkc3Rj eDIuc2F2ZSgpOwoJZHN0Y3gyLmJlZ2luUGF0aCgpOwoJZHN0Y3gyLnJlY3QoMCwgaSo1LCBpKjUs IDUpOwoJZHN0Y3gyLnNjYWxlKDUsIDUpOwoJZHN0Y3gyLmZpbGxTdHlsZSA9IGRwOwoJZHN0Y3gy LmZpbGwoKTsKCWRzdGN4Mi5yZXN0b3JlKCk7CiAgICB9Cn0KCmZ1bmN0aW9uIGRyYXdPZmZzZXQo c3JjLCBkc3RjeCwgZHN0Y3gyKSB7CiAgICB2YXIgZHAgPSBkc3RjeC5jcmVhdGVQYXR0ZXJuKHNy YywgIm5vLXJlcGVhdCIpOwogICAgZHN0Y3guYmVnaW5QYXRoKCk7CiAgICBkc3RjeC5yZWN0KDAs IDAsIDMwMCwgMzAwKTsKICAgIGRzdGN4LnRyYW5zbGF0ZSgxNTAsIDE1MCk7CiAgICBkc3RjeC5z Y2FsZSg1LCA1KTsKICAgIGRzdGN4LmZpbGxTdHlsZSA9IGRwOwogICAgZHN0Y3guZmlsbCgpOwoK ICAgIHZhciBkcDIgPSBkc3RjeDIuY3JlYXRlUGF0dGVybihzcmMsICJuby1yZXBlYXQiKTsKICAg IGRzdGN4Mi5iZWdpblBhdGgoKTsKICAgIGRzdGN4Mi5yZWN0KDAsIDAsIDMwMCwgMzAwKTsKICAg IGRzdGN4Mi50cmFuc2xhdGUoMTUwLCAxNTApOwogICAgZHN0Y3gyLnJvdGF0ZShNYXRoLlBJLzQp OwogICAgZHN0Y3gyLnNjYWxlKDUsIDUpOwogICAgZHN0Y3gyLmZpbGxTdHlsZSA9IGRwMjsKICAg IGRzdGN4Mi5maWxsKCk7Cn0KCmN4WzJdLmRyYXdJbWFnZShjWzFdLCAwLCAwLCA2MCwgNjAsIDAs IDAsIDMwMCwgMzAwKTsKc3RhaXJTdGVwKGNbMV0sIGN4WzNdLCBjeFs0XSk7CmRyYXdPZmZzZXQo Y1sxXSwgY3hbNV0sIGN4WzZdKTsKCmZ1bmN0aW9uIGRvSW1hZ2UoKSB7CiAgICB2YXIgaW1nRmly c3QgPSA3OwogICAgY3hbaW1nRmlyc3RdLmRyYXdJbWFnZShpbWcxLCAwLCAwLCA2MCwgNjAsIDAs IDAsIDMwMCwgMzAwKTsKICAgIHN0YWlyU3RlcChpbWcxLCBjeFtpbWdGaXJzdCsxXSwgY3hbaW1n Rmlyc3QrMl0pOwogICAgZHJhd09mZnNldChpbWcxLCBjeFtpbWdGaXJzdCszXSwgY3hbaW1nRmly c3QrNF0pOwp9CiAgICAKaWYgKCFpbWcxLmNvbXBsZXRlKSB7CiAgICBpbWcxLm9ubG9hZCA9IGRv SW1hZ2U7Cn0gZWxzZSB7CiAgICBkb0ltYWdlKCk7Cn0KCjwvc2NyaXB0Pgo8L2JvZHk+CjwvaHRt bD4K