WebApp Search Changes

November 30, 2012 by Monomon   Comments (0)

, , , , ,

In recent iterations of the WebApp search, we decided to make it more developer-friendly.
Constructing a search restriction was complicated by the fact that there were multiple ways of referring to MAPI properties.

How it works
Zarafa search conforms to MAPI; search terms are defined in the MAPI search restriction format (SRestriction).
This is a tree structure which contains operations, such as logical operators (AND, OR), and operands, which could be subrestrictions, property value tests, etc. (see diagram below). The type and number of operands depend on the preceding operators.
Search restriction structure

When a WebApp user initiates a search, the client builds up this restriction using the Zarafa.core.data.RestrictionFactory and passes it on to the server. The server then sets the restriction on a given folder or folders, and returns the table rows that match. Depending on the store, references to the matching rows may be placed in a 'Search folder'.

Most elements of a search restriction are constants - operators such as AND are mapped to integers before getting passed to the MAPI module.
Restriction constants are defined in mapi/mapidefs.php on the server and in Zarafa.core.mapi.Restrictions on the client.

MAPI properties
Properties correspond to columns in a MAPI table.
MAPI properties are identified by the so-called proptags. Proptags are integers containing a unique property identifier in the high-order 16 bits and a property type in the low-order 16 bit (see figure).

Props are normally written as hexadecimal integers.
Let us take a message's subject as example. Its proptag is 0x0037001E. The first half is the property id, the second - its type. Type constants are defined in mapi/mapidefs.php.

Each proptag is also assigned to a constant name that is easier to remember. These are defined in mapi/mapitags.php.
Continuing the above example, PR_SUBJECT is defined as mapi_prop_tag(PT_TSTRING, 0x0037). The result of this function is 0x0037001E.

Named properties
MAPI's tagged properties only go up to 0x7FFF. Properties in the range 0x8000-0xFFFE are called 'named properties' in MAPI.

Named properties allow service providers to extend messages with additional information.
They are written as strings: "PT_SYSTIME:PSETID_Appointment:0x820e". The format is <datatype>:<dataset>:<id>. The dataset part is a way to group logically related properties. It is possible to refer to a normal property from a named one.

Named properties are not persistent across sessions, so the server has to assign and retrieve the unique identifiers for given properties each session.
A mapping is obtained using the function getPropIdsFromStrings, located in mapi/mapi.util.php (in MAPI this is GetIDsFromNames).
Named properties present an additional way to refer to properties.

The problem
When transfering a restriction from the client to the server, it is passed as JSON. This means that all operators, properties, etc. will be sent as strings. Therefore, we need to know how to parse these elements when received on the server.
Operator mapping has been cloned to the client side, so now an AND operation would be sent as '0' in the restriction. All we need to do on the server is parse operators as integers.

Properties were trickier. Their sheer number already presented an issue if we were to copy the mapping to the client. In addition, there are different ways of referring to them.

The change
The WebApp server already contained a mapping from string names to proptags in class.properties.php. These are defined per class, e.g. for recipients, appointments, e-mail, etc.
For example, let us assume we are in class.appointmentlistmodule.php. $this->properties['subject'] would equal PR_SUBJECT, and thus 0x0037001E.
We decided to use this mapping to translate properties in restrictions sent from the client. This would make it easy for WebApp developers to list properties to search on, and create their own restrictions, by using names that are easy to use and memorize. It didn't require parsing the names on the server, but simply acquiring them from the mapping. This mapping is also initialized for each session, so it could be used for looking up named properties as well.

Now a client restriction can use logical names for properties, which get translated to proptags on the server, the way 'body' is used here:
        [ RELOP, RELOP_LE ],
        [ ULPROPTAG, 'body' ],
        [ VALUE, { 'body' : 'search term' }]

Note the format used - the structure is not associative, but rather the first array item is the operator, followed by an array of operands. This is what the restriction looks like in code.

It is still possible to fall back to either constant names or hexidecimal integers when forming these restrictions on the client - you could use either 'display_name', 'PR_DISPLAY_NAME', or '0x3001001E'.
Why would you need the latter? For instance, there might be properties not in the mapping, or the server mapping may not be available for a given module. Named properties cannot be used directly, however.

Default contexts have a SearchFields.js file, specifying the fields to be used in the search box. If you have a look at client/zarafa/mail/data/SearchFields.js, you will see the fields that mail can be searched on, grouped into objects. The listed property names are directly inserted into the restriction and translated on the server.

It has just become a lot easier to create custom search modules for WebApp.