Philip`: If you have any advice here, before I take a look properly, experience suggests it would be very helpful.

Hmm, I'm assuming this is something that can resolved without having to change the spec's basic assumption that it's drawing to an infinitely precise buffer (which is why it doesn't talk about antialiasing or rounding etc). Even with infinitely precise output, it matters how you compute each point from the low-resolution input image, so it seems a useful thing to specify.

It sounds like perhaps the desired behaviour is for the computation of the sampled colour at (possibly non-integer) position (x, y) on a w*h bitmap image to be something like (in random pseudocode since it helps me think about this):

  filtered(x, y, clamped) =
    if not (0.0 <= x <= w and 0.0 <= y <= h)
      return transparent black
    else
      compute filtered value at (x, y) (using standard bilinear interpolation or nearest neighbour or whatever), such that whenever the filtering algorithm chooses to sample the image at pixel index (ix, iy):

        if clamped: clamp ix to (0 .. w-1), iy to (0 .. h-1);
	else: wrap ix to (0 .. w-1), iy to (0 .. h-1);

        then return the pixel at the new (ix, iy).

  drawimage(x, y) =
    if not (sx <= x <= sx+sw and sy <= y <= sy+sh)
      return transparent black
    else
      filtered(x, y, true)

  pattern_norepeat(x, y) = filtered(x, y, false)

  pattern_repeatx(x, y) = filtered(x % w, y, false)

  pattern_repeaty(x, y) = filtered(x, y % h, false)

  pattern_repeat(x, y) = filtered(x % w, y % h, false)

The relevant points are:

* drawImage clamps to the edge pixels when doing bilinear interpolation, instead of wrapping or padding with transparency outside the image.

* All patterns wrap instead of clamping (even no-repeat).

* drawImage clamps to the edge pixels of the whole image, not to the edges of the source rectangle segment of the image.

* Areas of patterns outside the image region are transparent black.

It looks like this is what IE9 does. Opera mostly does this, except patterns get clamped in their non-repeating directions (e.g. repeat-x wraps on left/right edges and clamps on top/bottom; no-repeat clamps on both) (and there's the weird offset thing). Firefox interpolates with transparency around the edges (instead of wrapping/clamping), except in repeating patterns where it wraps. Chrome is nothing like this and all its scaling looks terrible if you go above 16x.

Could I convince one of you to write this up formally? I'd be happy to massage such text into RFC2119ese, but I'm concerned that I don't really understand this issue well enough to write text that makes sense. (In particular, I can't see any difference between what any of the four browsers on my platform are doing with the attached test case  they all look identical to me.)

Hm, I'm not sure what Philip wrote up is correct/possible to implement efficiently.. but I'd have to study it in more detail.

I'm surprised you don't see a difference.. for reference, here's Firefox and Chrome:

  http://i.imgur.com/QLrAo.png

And Opera and IE9:

  http://i.imgur.com/ugLYQ.png

Some differences:
- fuzzy edge vs sharp edge in the fourth/fifth large squares
- note fuzzy inner edges vs sharp inner edges in the first three squares, esp on chrome where bits of the second square are incorrectly sharp (due to the stairstep rendering)
- blue fill on fourth and fifth in chrome instead of white (transparent) fill
- no linear filtering in third, fourth and fifth in chrome, compared to first and most of the second

My intention was that this was basically EXTEND_REPEAT for patterns and EXTEND_PAD for images (given that you set the extend mode for a gfxPattern representing the whole image, not just the source rectangle), so I think I'm missing where the problem would be (unless it's that those EXTEND modes can't be implemented efficiently).

Hmm, I just realised this EXTEND_PAD thing interact with the change hidden in http://html5.org/tools/web-apps-tracker?from=5372&to=5373 ("Pixels of the source rectangle that are not entirely within the source image must be treated as transparent black") - not sure if that's going to be an issue.

Nope, PAD and REPEAT are totally fine hw-wise, but I'm not sure how that works with the "no-repeat" canvas pattern mode.. you still have the question of what to do at the edges, no?  According to what you put above, for no-repeat it would just use x y without clamping, thus giving transparent black.. which would give you the fuzzy edges on scaled-up drawImage.  (This is essentially what firefox does currently.)

What I (I think) put above for no-repeat patterns was that it wouldn't clamp, but would instead wrap. It's equivalent to repeat patterns for the purposes of filtering, it's just cut off sharply to transparent outside the area of the first repetition. (This is what IE9 seemed to do.)

If no-repeat were to have some other filtering behaviour, then repeat-x and repeat-y ought to have that behaviour on their non-repeating edges (for consistency). They should still have the wrap behaviour on their repeating edges, which means different behaviour in different directions on one pattern. Opera does that but it looks like the Cairo API assumes a single mode for both directions - would that be a problem?

Would it be possible to have the proposal written in terms that the spec uses? Pseudo-code isn't really workable since we'd have to define the syntax of the pseudo-code to make it formal, which is why I don't really think we can use comment 2 as written (even with RFC2119 massaging).

Philip, I'm reassigning this to you on the basis that I have no idea what I'm doing here and you do. Please feel free to reassign it to me if you can't do anything here. I likely would just close it NEEDSINFO though.

If you are able to do something, all I need is text that says the technical bits, so that I can then massage it into spec prose.

(Sorry, been a bit distracted and didn't get around to this earlier.)

I think this should do what I suggested (though if this text is adopted I make no guarantees that I will not file further bugs saying it's all wrong):

For drawImage:

Remove the text "Pixels of the source rectangle that are not entirely within the source image must be treated as transparent black."

Somewhere in this section, add the text:

"""
If the original image data is a bitmap image, the value painted at a point in the destination rectangle is computed by filtering the original image data. The user agent may use any filtering algorithm (for example bilinear interpolation or nearest-neighbor). When the filtering algorithm requires a pixel value from outside the original image data, it must instead use the value from the nearest edge pixel. (That is, the filter uses 'clamp-to-edge' behavior.)
"""

For createPattern, add the text:

"""
If the original image data is a bitmap image, the value painted at a point in the area of the repetitions is computed by filtering the original image data. The user agent may use any filtering algorithm (for example bilinear interpolation or nearest-neighbor). When the filtering algorithm requires a pixel value from outside the original image data, it must instead use the value from wrapping the pixel's coordinates to the original image's dimensions. (That is, the filter uses 'repeat' behavior, regardless of the value of <var>repetition</var>.)
"""

EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: Accepted
Change Description: see diff given below
Rationale: Seems reasonable.

Checked in as WHATWG revision r5708.
Check-in comment: <canvas> drawImage() and pattern filtering issue
http://html5.org/tools/web-apps-tracker?from=5707&to=5708

(In reply to comment #11)
> For createPattern, add the text:
> 
> """
> If the original image data is a bitmap image, the value painted at a point in
> the area of the repetitions is computed by filtering the original image data.
> The user agent may use any filtering algorithm (for example bilinear
> interpolation or nearest-neighbor). When the filtering algorithm requires a
> pixel value from outside the original image data, it must instead use the value
> from wrapping the pixel's coordinates to the original image's dimensions. (That
> is, the filter uses 'repeat' behavior, regardless of the value of
> <var>repetition</var>.)

Hm, isn't this inconsistent?  For example, if the pattern uses no-repeat, and is used to fill a rectangle larger than the pattern image size, this seems to say that the no-repeat setting is ignored.  That seems odd and unexpected.

The clamp-to-edge behavior is fine for drawImage, but for patterns, I think the only thing that mathematically makes sense is for the area outside of the source pixels to be treated as transparent black for no-repeat.

By "in the area of the repetitions", I meant the area where the previous paragraph says the pattern will be painted (anchored in a certain point and repeated in certain directions). Outside that area, the pattern won't be painted at all - it's only inside the area that the wrapping filter is used. So no-repeat patterns and drawImage will both draw a single copy of the image with sharp edges when scaled up, but will filter differently when computing the pixels just inside the edges.

An alternative way to see it is that the pattern is always based on an infinitely-repeating-in-both-directions image, with filtering applied to that, and then the repeat-x/y/no-repeat setting is cropping the resulting image in one or both directions. This is similar to how drawImage takes the entire input image, filters that, and then crops to the specified source rectangle, rather than cropping before filtering. So I think it's logically justifiable in that way.

Maybe, though I don't think the language makes that clear.  (If I was implementing canvas reading just the spec, I'd be scratching my head at that language.)

However, you're basically describing an algorithm that's more than just sampling from a pattern with repeat; I'm not sure there's any need to do that, given that canvas already provides the primitives needed to get the effect you describe (draw with a repeating pattern, clip to a rectangle).

The language isn't ideal, but I'm not sure how better to describe it in prose - if you can suggest changes that would make it clearer, that would be great.

Canvas provides the primitives to emulate any filtering effect (take the source image and surround it with transparency or repeats or clamped edges, then clip to a rectangle) so I'm not sure what you mean with that.

The advantage I see in the repeat-in-both-directions pattern filtering is just that it seems the easiest way for browsers to converge (without being a clearly crazy behaviour to converge on). IE appears to implement it already; Opera is not far off. Firefox doesn't support repeat-x or repeat-y yet so it will have to change in any case, and per comment 8 it looks like the Cairo API doesn't like different extend modes in different directions, so it will be simpler to use consistent modes. Is this incorrect?