Filtering and Transformation for High-Volume  XML Message Brokering

Overview

People

Technology

Research Papers

Related Work

Software

Overview

In distributed environments, including Web Services, application integration, and personalized content delivery, XML will be used as the wire format in which data to be exchanged is encoded. In this emerging infrastructure, XML message brokers will serve as central exchange points for messages sent between applications and/or users. Three of the main functions of such brokers are: filtering, transformation, and routing. 

• Filtering matches messages to predicates representing interest specifications. 
• Transformation restructures matched messages according to recipient-specific requirements. 
• Routing involves the transmission of the customized data to the recipient.

YFilter aims to provide fast, on-the-fly matching of XML encoded data to a large number of interest specifications, and transformation of the matching XML data based on recipient-specific requirements. Over the past two years, it has been developed into an efficient query processor on streaming data that high-capacity XML message brokering systems can be built on. 

Applications

We present three example applications where XML message brokering systems using YFilter can be deployed .

Personalized Content Delivery
Application Integration
Network Monitoring

Query Language

In YFilter, interest specifications are written in a subset of XQuery, an emerging XML query language. When these interest specifications enter an XML message broker, they become "standing queries". That is, they will be continuously applied to all incoming messages. 

Filtering-based queries can be written using path expressions. Path expressions are primarily used to address parts of an XML message. A path expression consists of a sequence of location steps. Each location step consists of an axis, a node test and zero or more predicates. An axis specifies the hierarchical relationship between the nodes, e.g. parent-child or ancestor-descendent relationship. A node test can be an element name or a wildcard operator '*'. Predicates can be applied to the text data, the attributes, or the positions of the addressed elements. They may also contain other path expressions (i.e. nested path expressions) to impose additional structural constraints.

Those queries that specify both filtering and transformation requirements can be written using a certain subset of FLWOR (for-let-where-order by-return) expressions provided by XQuery. The for clause and let clause together produce variable bindings, that are further filtered using the where clause, and finally cause results to be generated using order by and return clauses. All of these clauses use path expressions to address elements in an XML message.

XML Message Broker Architecture

We propose an architecture of XML message brokers built using YFilter, which is shown in the following figure. The primary inputs to a message broker are the queries that represent interest specifications and the XML messages. The output is a stream of XML messages to be delivered to users or applications whose interests are satisfied. These XML messages can be the original incoming messages or the customized messages transformed from the incoming ones based on the individual query specifications.

XML Message Broker Architecture

The major components of this architecture are:

• Query listener: It receives queries.
• XQuery parser: It parses queries and sends the parsing results to other components.
• Message listener: It receives XML messages.
• XML parser: Using an event-based XML parsing tool, it incrementally generates an internal representation of the XML message, and passes the parsing events to drive the execution of YFilter.
• YFilter: The compiler of YFilter constructs execution plans for all the queries. When processing a message, the runtime system of YFilter matches all queries with the message, extracts message components for the matched queries, and organizes the extracted components in an intermediate format for efficient translation into customized output messages. Note that for filtering-based queries, YFilter only performs the matching function and then returns a boolean result, True or False. Internally, YFilter consists of a path matching engine at the bottom that provides matches of path expressions, and a post-processing module on top that post-processes the path matches for the remaining portions of the queries (see our research papers for details).
• Message factory: For each matched query, it generates an output message. If the result returned from YFilter for this query is a boolean value True, the output is simply the incoming message. If the returned result contains extracted message components, the message factory combines these components with constant tags in queries, and writes them into result messages.
• Message sender: It sends customized XML messages to users or applications.