CSS/SVG FXTF Proposal: Custom Filter Effects Modules

December 10, 2012; revision 2

Filter Effects 1.0 enables an exciting new custom filter capability for page authors to load programs to alter the rendering of element geometry and shading. This proposal details a design for the flexible use of this capability that integrates custom filter modules into CSS and takes advantage of the design of HTML, XML, HTTP, and URI.

Introduction

Custom filter effects will soon provide web developers with easy-to-use, cross-platform, declarative element rendering capabilities unmatched in any other environment. To describe the graphical effects to be applied, developers will need to indicate what functions to map over element meshes and element pixels as well as parameters to those functions. There exist many possible representations for this configuration information but the thesis of this proposal is that, at minimum, this functionality should be exposed in CSS and define generic resource semantics in alignment with the rest of the Web.

The shaders that interpret the operational model described in the Filter Effects 1.0 Working Draft may be expressed in a number of different languages and dialects in the future. It is a primary goal of this proposal to easily accomodate future shading languages and effect package formats while providing an easy and efficient interface to this sophisticated functionality. By leveraging the existing design of CSS, HTML, XML, and HTTP, Custom Filter Effects Modules achieves a wide variety of flexible features at little specification cost and minimal implementation cost.

It is important to distinguish the 4 different abstractions we are concerned with in this work:

FX Languages
FX Operational Models
FX Function Resources
FX Module Parameterizations

FX Languages are specifications of the algebraic expressions that may be used in FX Function Resources. Many different languages may be used with any individual FX Operational Model and thus with a given FX Module Parameterization. FX Languages are represented by Internet Standard Media Types as used in HTTP Accept and Content-Type headers.

FX Operational Models are specifications of the data types that may be used in FX Module Parameterizations. FX Operational Models loosely correspond to CSS 3 Font formats but it is important to note that, by their nature, operational models may vary significantly in behavior. Filter Effects 1.0 defines a single FX Operational Model detailing discretization of element surfaces into meshes for deformation and color blending operations on element texels as well as parameters to those transformations.

FX Function Resources are the actual expressions of mesh and pixel transforms in a particular FX Language. At the present time, most FX Function Resources are expressed in OpenGL ES 2.0 GLSL 1.0 (ESSL) as this FX Language is also in use by Khronos' WebGL.

FX Module Parameterizations are the primary focus of this document as concrete expressions of the specific combination of FX Function Resources with parameters for an individual use scenario.

Parameterization as CSS Expression

A means of defining and naming specific parameterizations of shader function resources is required for abstraction of filter effects. In CSS Level 3 modules, this kind of abstraction is accomplished with @-rules that group resource links with their associated parameters. To satisfy the use cases of the Filter Effects 1.0 Working Draft, specifically functional maps over element mesh vertices and element pixel values, any abstraction will require expressions of or links to expressions of these maps. To support simple creation, incorporation, and customization of custom filter effects in CSS authors' workflows, it makes sense to expose the operational model of Filter Effects 1.0 Working Draft to CSS using CSS syntax.

Below is an example of an FX Module Parameterization in a named @filter rule with two different FX Operational Models, fx1 and msfx1. As these rules are given the same name, 'curl', they should be equivalent effects chosen between by the user agent according to its support for the fx1 and msfx1 operational models.

@filter curl {
  type: "fx1";
  meshfn: url("/fx/curl.fx#mesh") url("/fx/curl.fx#mesh-fallback");
  pxfn: url("/fx/curl.fx#px");
  mesh: 4 4;
  mix: source-in multiply;
  parameters: identityMatrix mat3(1,0,0,0,1,0,0,0,1);
}

@filter curl {
  type: "msfx1";
  ms-src: url("/fx/curl.fx#ms-fx-bundle");
}

filter: custom(curl);

Among the benefits of this approach are ease of modification/customization and ability for effects to be in-lined in independent or embedded stylesheets. It is anticipated that libraries of custom filter effects modules will be constructed for easy use of advanced rendering capabilities. These libraries may be effectively abstracted through the use of the CSS @import rule. If a significant number of operational models come into use, conditional library loading may be achieved with @import media queries. Finally, these filter effects libraries, being described in CSS, may bundle related stylesheet declarations.

Function Resource Request HTTP Behavior

In every high-performance web application, both network throughput and network latency must be carefully managed. To reduce required throughput, a web application should strive to eliminate the transmission of unnecessary or unused data. To reduce latency, a web application should keep round-trip dependency depth low and use fewer requests when possible.

In pursuit of these goals, Custom Filter Effects Modules uses media types and fragment identifiers to retrieve only the function resources and packages that the client can understand in as few requests as possible. Specifically, every client will have a set of FX Languages and container formats which it can interpret in each of its FX Operational Models. A client should issue a single HTTP request for each distinct (excluding fragment identifier) FX Function Resource URI with an Accept header containing the union of the media types for every different FX Operational Model in which that URI is referenced. In this way, the client informs the server what types of languages and packages it can interpret so the server will not transmit uninterpretable data.

In the previous example, the user agent may issue a request for the resolution of "/fx/curl.fx" against the ambient base URI with an Accept header specifying "text/html, application/xml, application/essl, application/othersl, application/vnd.msfx+xml" as it understands both the "fx1" model and uses ESSL or OtherSL as FX Language in that model with HTML or generic XML packaging as well as the "msfx1" model which uses a proprietary effect packaging format and language. The server may respond to this request with a message indicating Content-Type of "application/xml" encapsulating "application/essl" source which the client will then interpret as detailed below.

HTML and XML Fragment Integration with Function Resources

As demonstrated in the previous section, the use of fragment identifiers can greatly reduce the number of required HTTP requests for function resources through request collation. This generic and fundamental feature of the Web platform requires user agent knowledge of specific compound media types. We address two major compound media types, HTML and XML, that immediately deliver enormous value to the fledgling custom filter effects ecosystem.

The HTML 'id' attribute identifies page elements that correspond to secondary resources and may be referred to with fragment identifiers. FX Function Resources may be enclosed in HTML script tags and identified by their unique ids. This functionality is free as user agents already understand the compound nature of HTML resources. By requiring only that referenced FX Function Resources ultimately resolve to an expression in an interpretable FX Language, Custom Filter Effects Modules enable single-request (of HTML) applications and easy dynamic effect loading. As HTML can also be used as an effect transport format, it can serve as immediately renderable documentation of a given effect.

The standardization of the xml:id generic attribute provides every generic XML interpreter with a simple way to resolve secondary resources. By requesting "application/xml" as one of the types for an FX Function Resource, a user agent indicates that it can understand generic XML resources and shading language secondary resources through the use of xml:id and fragment identifiers. This facility allows web developers to use any XML vocabulary they like to transport their filter effects.

Conclusion

By specifying an FX Module Parameterization data type for each FX Operational Model using existing CSS syntax, Custom Filter Effects Modules eases the understanding, debugging, and customization of effects by CSS authors. In doing this, all semantics for filter effect rendering are contained in either CSS or a referenced FX Function Resource. FX Function Resources themselves become first-class web resources easily used in other contexts like WebGL.

Another benefit to this approach is the minimal semantic implementation and heavy reliance on well-established web standards like URI, HTTP, HTML, and XML. A hallmark of these standards is their forward-looking extensibility features. Custom Filter Effects Modules leverages these extensibility features to immediately enable a host of powerful techniques for sophisticated page rendering.

Identification, interaction, and representation are orthogonal concepts, meaning that technologies used for identification, interaction, and representation may evolve independently.

Thanks

Dirk Schulze and his team at Adobe provided many of the basic ideas as well as the motivation for this proposal. The CSS/SVG Filter Effects Task Force has been crucial to bringing advanced graphical capabilities to the Web and without their devotion to technical excellence none of this would be possible. Thank you!