Minutes 2006-08-04: Friday morning Present: Norm (chair) Murray (host) Jeni (scribe) Henry Mohamed Alex Richard Norm: What do we want to say about pipelines, pipeline libraries, recursive pipelines etc. First: is it reasonable to have a pipeline inside another pipeline? Henry: I would like to, for modularity. It's a choice to package up steps into a named pipeline. Richard: You should be able to do that, but in other programming languages you have multiple functions, but usually do not put functions inside other functions. If you do have functions inside functions, it's usually to give the inner function access to information in the outer function. There's hiding the function from the outside environment... Henry: Hiding isn't a big deal. Richard: If we don't want to access names in the outer pipeline, then you don't need this. Henry: I don't want the inner pipelines to access information from the outside one. In Java, if I'm a novice, I write classes inside other classes. Jeni: But in Java, you have to create another file. In our language, you don't, so why would you want to embed it? Murray: What's the difference whether it's embedded or not? Henry: When I pass the file to another user, it's no longer obvious which pipeline should be run. Murray: Not yet, but we could provide a mechanism to say which one is going to be run. But I was really asking Richard why he cared... Richard: Because in other languages, there are other semantics associated with nesting a function inside another one, to do with accessing information in the outer function. Henry: That's why it doesn't work for me. Suppose my common code used a pipeline parameter. Murray: Why isn't it all in scope when the functions are at the same level? It doesn't matter if our pipeline language works differently from programming languages. Norm (and others): Yes it does. Richard: We also might, in the future, want to provide some semantics to nested pipelines. Here's a Java example: class foo { int a; class bar { int b, c; ... a = b + c; } } 'a' is available in the inner class 'bar', but not from outside. Murray: In the case the function is outside, you have to pass the arguments. If the function is inside, you don't have to pass the argument into the function: it's just there. Henry: I want it to be a software engineering choice. If I have: S1 S2 S3 ... If I want to package S2 and S3 into a named pipeline. I'm happy to have named inputs and outputs, and to have it encapsulated, but I want to have parameters passed in automatically. I want parameters to be lexically scoped, but not ports. Richard: A choose can use ports from outside: normally ports are lexically scoped as well. Alex: What would be the problem of saying that you have to declare parameters: if you want to pass the parameter, then you should declare it. Henry: I could live with that. But then it doesn't matter whether it's inside or outside. I have my mind on the simple user with a single pipeline element. Richard: Nesting should correspond to scoping. Alex: Nesting with encapsulation makes sense to me: the pipelines are only accessible in the parent pipeline. Norm: It seems odd that it's only one level deep. Henry: I agree with Alex. Alex: But because it's a black box, this doesn't solve the pipeline library problem. Henry: We have that as well. Jeni: We need pipeline libraries, and they do what we need to do, so why make the language more complex by adding this ability? Henry: You can't use that argument, because removing constructs from the language doesn't make it simpler to use the language. Murray: Using nested pipelines makes absolute sense to me. Example: ... Can you call 'a' from 'b'? Henry: No. Murray: Then I don't understand. Norm: If 'b' can't call 'a', then my user who wants to modularise something that's common from 'a' and 'b' to 'd', and pulls it out, but can't call it, is completely baffled. Murray: The step asks me to run 'b'. Surely I should be aware of 'a'. That's what makes sense to the naive user. Henry: So named pipelines always get put into the pipeline library. You can run one of those by name. Norm: So now 'a', 'b' and 'c' are all peers and all callable from each other. Alex: The library changes as you go in: you add things to it. When you go inside 'b', 'c' is added to the pipeline scope. Murray: Naive user. That 'c' is inside of 'b', and the only way I can run 'c' is by invoking 'b'. So I can run a pipeline that's inside of me, or outside of me, but no one else can run pipelines that are inside me. Richard: I agree that we can do this, but if we do, we will have to decide a lot of things that are quite complicated, and we should leave it 'til version 2.0. Murray: That's a good reason for not doing this. Alex: Personally, pipeline libraries are useful, but let's leave *them* to version 2.0. Because import is complicated. Henry: I assumed that you'd just specify all your pipeline libraries on the command line. Norm: I think we need to have pipeline libraries. I think Richard's right that pipeline libraries with pipelines all at the same level is sufficient. We might later think it's too much work for naive users. We can always do that later. Richard: I think we should do it later. We shouldn't pre-empt the semantics of nested pipelines, which we might add later. Murray: We have to do the libraries, with some include mechanism. I like the nesting, but I understand Richard's argument that this is too much for us to take it on right now. I don't think we should make the decision now: I think we should include it in the document, say we're uncertain, and then later pull it, unless users come back saying that they really need it. Alex: Don't we have group? Can't we use that? Henry, Norm: It's not the same thing. Murray: Can we call this procedure rather than pipeline? Norm: Let's talk about that later. Murray: How is this nesting thing not like groups? Richard, Norm: Groups get executed when you come across them: they just provide some scope: you can't call them again. Murray: Can we conflate them? Norm: I don't like the idea of asking the public whether we should do something. All we'll ever get from the public is "yes, we should do it". We should give them the minimum, and get them to ask for more. Alex: So can we talk about pipeline libraries? Henry: contains zero or more elements. We're done. Jeni: We need defaulting. Henry: contains zero or more elements, and a default-pipeline attribute that points to one of them. Norm: Let's get agreement on pipeline libraries. Alex: We shouldn't have default-pipeline. We just supply the QName when we call the pipeline. Norm: If you have to point to the library, then it's no cost to provide the name as well. To review: A pipeline library contains zero or more pipelines, all of which have names. (Zero-or-more or one-or-more...) I don't feel strongly about defaulting. Richard: I want to just refer directly to the library, just like in C, you have a 'main'. A library can have a default pipeline in it, that gets executed if you get given the library. Norm: Java has this functionality. It seems no effort, and has some use. Mohamed: What about including other libraries? Richard: We should use import rather than include. Include implies textual inclusion. With import, the pipeline library might be already compiled, and the only things that are available are some packaged information. Alex: Can you import inside the pipeline library? Richard: Yes, you have to import from the pipeline library. Norm: I suggest we leave off default-pipeline attribute for now. The has a source attribute that points to the imported library. It can go in and in Murray: I think pipeline libraries should have a name for debugging purposes, so if I loaded it, debugging information would be raised. Richard: I think it should have a name as well. Norm: OK, optionally have a name. Jeni: We shouldn't allow within Richard: You might have a single element in a file; you should be able to import pipelines into it. Jeni: No: if you need to reuse pipelines, you have to ramp up to having a pipeline library. Mohamed: You should import pipeline by QName rather than URI. Alex: I would be happy with a that excluded the URI, and tell the implementation you need pipelines by name. Richard: So do you expect a catalog mechanism so that I can get libraries by URI when I'm not connected to the 'net? Is this our problem? Norm: This isn't our problem, just as it isn't XSLT's or Schema's problem: it's implementation-defined how the documents are retrieved given a URI. Henry: If I have a pipeline and Richard says he has a library. I thought that I had to say on the command line where the library is, but everyone said that was crazy. So I need an import library statement that I can put in my pipeline. Jeni: You add around it and add ...much discussion about the requirement for naive users to add in their standalone skipped... Norm: I'm looking for a compromise. Suppose we go back to the GCC model: you supply the pipeline libraries at the command line. Richard: I think pipelines are going to be little things that they want to run. They don't want to have to do this at the pipeline. I think we should allow within when is a document element. But in a pipeline library, you have to put it at the top leve. Alex: So if I rip out a pipeline from the pipeline library and try to run it, then it would be invalid. Plus if I put a pipeline into a library, I need to move the into the top level of the pipeline library. Murray: What was the logic behind not having the wrapper with a standalone pipeline and putting the inside that wrapper? Norm: Most users are going to have simple pipelines, and they're not going to want to write the wrapper. Alex: If can have inside it, then it should be able to do that within a pipeline library. Jeni: Are the imported pipelines visible within the itself or in the entire library? Alex: Only in the that contains the . Norm: Recap: We will have a element that can contain pipelines. It has an optional name. You can import pipelines from another pipeline library. A pipeline can also stand by itself, which can import other pipeline libraries. You can import a standalone pipeline. Jeni: Circularity? Norm: If you import a library that you've already imported, you don't worry: all the pipelines you import are available. Murray: I should be able to have an import in a pipeline in a pipeline library, so I can cut and paste. ... Norm: What about saying that a standalone pipeline can't be imported. We have a syntactic warp in allowing import within a pipeline in one place and not another; this is a way of getting around it. Richard: To go back: if A imports B and C, then C shouldn't be able to access pipelines in B. Alex: In XSLT, you can. Richard: In C you can't. Norm: In XSLT you can. Richard: It means that there are libraries that will work in some contexts but not another. Norm: We can say that if any pipeline library contains a step that references a pipeline that isn't imported then it's an error. Richard: So names are globally scoped. A / \ B C | D A can see things in B and C and D. B can only see things in B. C can see things in C and D. D can only see things in D. Richard: So everything in the libraries that you import gets automatically exported. What about circularity. A <-+ / \ | B C | | | D -+ Henry: Where you start is the top (A). You stop at D. (agreement) Norm: The name for the import statement is with an attribute called 'source' (this is consistent with what we do with ). Alex: In pipeline libraries, we also have to deal with declaring components. Norm: Yes, we need to deal with extension components. Alex: We should put it in the pipeline libraries. DECISION: We have pipeline libraries with document elements, with an optional name attribute and containing multiple pipelines. We have standalone pipelines with document element. Both can have * as children of the document element. This points to either a pipeline library or a standalone pipeline. As well as the built-in components and implementation-defined components, a pipeline library or a standalone pipeline has in scope all the pipelines of all the pipeline libraries or standalone pipelines that it imports, recursively. No consensus on a default pipeline to run within a pipeline library. BREAK Inputs and outputs. Henry: This isn't a proposal for naming, it's an analysis that may help. A component is a named box with named things that data comes into and named things that data comes out of. We have the ability to replicate them, and use things to connect these boxes together. I propose declaring components with: and parameters go in here as well, but this discussion doesn't incorporate parameters. We have something new now, which covers four language constructs: group, for-each/viewport, choose and when. These are all containers for steps, with their own paired in/out at the top and out/in at the bottom. Choose actually looks almost like this, but the things inside are containers as well. This is similar to what we've talked about before, except that source->href and ref->source. So how to do we do the in/out and the out/in for the containers. We have a combination of and : Richard: What about pipelines? Henry: Pipelines are like components, in that they have some named ports at the top and the bottom. But we can't call them inputs and outputs. The value of the source attribute must always be the name of a Component ! the name of a port on a component or the name of a port on oface. Richard: I think pipelines have all of these things. You need to say what inputs they have, just like for component definitions. And you need to define the inputs for within the pipeline, and you need to bind an input for use within the pipeline. General agreement. Richard: An input for a pipeline doesn't have a source. Henry: It *could*. Richard: But it doesn't *need* it. got its source from . Jeni makes the point that the out-facing ports may have different names from the in-facing ports within the container. Henry combines them by making them siblings and writes up: Henry: I'd like to digest this for a while before we discuss names. Richard objects to the naming of one thing and another thing since an input is a port. We decide to think on the naming for a while. --- Core components --------------- Norm: We've talked about various components like XInclude, validate, XSLT. What are the others? List: XInclude XSLT[1|2] validate* xquery load save identity httprequest aggregate disaggregate subsequence escape (string to XML for RSS) unescape (XML to string for RSS) XPath[1|2]filter wrap wrap-sequence insert (attributes|elements|change values) ns-rename delete (subtrees|attributes) rename (attributes|elements) strip whitespace absolutize (absolutize selected URIs) prettyprint exec os-access (get directory/environment variable etc) sort (sorts elements) regex (destructures a string) bitbucket/sink doc-replace (replaces an input with another one) diff c14n encrypt decrypt sign verify label (adds IDs to all elements) line number push-tag (wrap selected elements with a wrapper) soap-exchange SPARQL manifest/packaging render XSL-FO/SVG/MathML tagsoup wikify sgml-in schema-check apply (pipeline) grddl (returns RDF from XML document) STX (streaming transformation) NVDL (namespace validation) uptranslate downtranslate forward-chain-RDF replicate load-escaping-entity-references save-disable-output-escaping (During the course of generating the list) Henry: I need two versions of load/save/identity, for different arities. Richard: We've agreed that a sequence of one document is acceptable to a port with an arity of 1. Henry: I think we should either declare arities and enforce them statically, but if we're not doing that, then we don't need two versions of load/save/identity. Jeni: We need, for example, load as well as a href/source attribute, to allow the URI to be, for example, passed in as a parameter. ... We have agreement that xml:base processing happens automatically, but we have to talk about what happens in terms of the base URI of outputs. We also need to talk about security at some point. ... Alex: We should have modules of components that vendors may implement. (general agreement) ... Murray: What about entities? Henry explains a case where the entities were escaped on load and unescaped on save. We need to talk about character encodings in the pipeline: we need to provide a way of preserving a character encoding through the components. Murray: I use entities for reuse: I don't want them expanded. Norm: You have to use XInclude or other mechanism. Richard: Nothing else in the XML stack does this. Henry: I want a load-while-escaping-entities step. Richard: We could have a component that turns the DTD into an XML document that can be passed through to a later component, that can then reconstruct the DTD for the entities. We want to come back to preserving entities. ... Henry: I'd like to talk about built-in parameters which have information from the XML declaration. Richard: Encoding and version are in the Infoset already. ... Alex: I want to have some general declarations on serialization parameters. Henry: We should put those on the output port declaration, to give hints to the implementation. ... What about core components? If no one objects, they're included... XInclude XSLT 1.0 validate identity aggregate Alex objects to load because he wants httprequest. Murray objects to all the rest. We decide to take a different tack. BREAK FOR LUNCH