Searching XPages REST service and returning the real count in the results.

In this article I will show you how I managed to get the search results count into a REST service and why I wanted to do that.

Introduction

Back in August 2012 I posted this on Stackoverflow. I will answer the question there but I don’t feel like I have “answered” it, just worked around it.

How do I make the XPages REST control return the overall number of filtered matches?

http://stackoverflow.com/questions/11786930/how-do-i-make-the-xpages-rest-control-return-the-overall-number-of-filtered-matc

Problem

When you create a viewItemFileService REST service using the the ExtensionLibrary control you get a response which looks like this:

{
    "@timestamp":"2013-03-12T23:25:57Z",
    "@toplevelentries":1299,
    "items":
    [
      {
          "@entryid":"100-8DE5EBF5E35C17F285257B1B00254A8F",
          "@unid":"8DE5EBF5E35C17F285257B1B00254A8F",
          "firstname":"Benjamin",
          "lastname":"Imran",
          "address":"2112 Hall Street",
          "city":"Las Vegas",
          "state":"NV",
          "zip":89121,
          "country":"US",
          "email":"Benjamin.A.Wayman@mailinator.com"
      },

which is fine…and from that you can ascertain that there are 1299 documents in the view (@toplevelentries)

The problem is when you execute a search against this REST service (for example ‘[address] contains street’) you still get @toplevelentries: 1299

This causes me great angst when it comes to the EXTJS grids because I want to use the Infinite Scrolling capability. What that does is calculates based on the size of the view (1299) how large a scroll bar to create and then display 25 entries at a time and load them as necessary. This is really smart except after you have searched the REST service. The grid still thinks there are 1299 documents even when there is only a subset matching the search results. This causes a lot of blank scrolling 😦

In my example the search ‘[address] contains street’ returns 338 matches. But this causes the following to happen with the infinite grid

rest1

And that sucks. Also note 52 pages in the pager!!!!

The EXTJS code to get the number of documents in the service looks like this:

 var store = Ext.create('Ext.data.Store', {
        // allow the grid to interact with the paging scroller by buffering
        buffered: true,
        pageSize: 25,
        leadingBufferZone: 25,
        autoLoad: true,
        autoDestroy: true,
        autoSync: true,
        model: 'Person',
        proxy: {
            type: 'rest',
            url: 'xRestService3.xsp/marky?count=25',
            reader: {
                type: 'json',
                root: 'items',
                totalProperty : '@toplevelentries' // This is the determining value
            },
            writer: {
                type: 'json'
            }
        }
    });

And as you can see form the code @toplevelentries is always 1299

So how do we get the 338 number to the grid without causing:

  • Some major hacktastic, back end to front end, extra ajax calls, and  redrawing the grid on the fly shenanigans
  • Manually re-writting the REST service from scratch (which I could do but would cause much pain and suffering)
  • Performance issues with XPiNC connecting to the server
  • My eyes to bleed…..

This has baffled me for months and here’s how I got around it.

Solution

In the REST service you can create a custom column. In this case we are just returning the requestScope value for markyCount in the column “theRealCount”:

<xe:this.columns>
	<xe:restViewColumn name="theRealCount">
		<xe:this.value>
		<![CDATA[#{javascript: return requestScope.get("markyCount")}]]>
		</xe:this.value>
	</xe:restViewColumn>
</xe:this.columns>

And markyCount is going to be the 338 value from the search. So how do we get that?

The REST service has an id value which can be computed – and I really don’t care what the id is because I am not binding it to a control on the form. So I used the opportunity for some server side computation of the id to create a little FTSearch.

In the example below I am simply getting the view used by the REST service (byFirstName) doing an FTsearch on it and then putting the count in the markyCount requestScope.

<xe:this.id>
	<![CDATA[${javascript:
		var theView:NotesView = database.getView("byFirstName")
		var count = theView.FTSearch("[address] contains street");
		requestScope.put("markyCount", count)
		return "searchMe" //irrelevant value
		}
	]]>
</xe:this.id>

So what this does is create a REST service output which looks like this:

{
    "@timestamp":"2013-03-12T23:25:57Z",
    "@toplevelentries":1299,
    "items":
    [
      {
          "@entryid":"100-8DE5EBF5E35C17F285257B1B00254A8F",
          "@unid":"8DE5EBF5E35C17F285257B1B00254A8F",
          "firstname":"Benjamin",
          "lastname":"Imran",
          "address":"2112 Hall Street",
          "city":"Las Vegas",
          "state":"NV",
          "zip":89121,
          "country":"US",
          "email":"Benjamin.A.Wayman@mailinator.com",
          "theRealCount": 338  //this is the new column
      },

Note theRealCount column with 338 in it. This is then added to every column output by the REST service. So yes the downside is that we are adding computation and output data size to the feed but it appears to make an insignificant amount of difference to the output speed of the service.

We then change the EXTJS code to look at the theRealCount value in the first Item and not @toplevelentries

 var store = Ext.create('Ext.data.Store', {
        // allow the grid to interact with the paging scroller by buffering
        buffered: true,
        pageSize: 25,
        leadingBufferZone: 25,
        autoLoad: true,
        autoDestroy: true,
        autoSync: true,
        model: 'Person',
        proxy: {
            type: 'rest',
            url: 'xRestService3.xsp/marky?count=25',
            reader: {
                type: 'json',
                root: 'items',
                totalProperty : 'items[0].theRealCount' // This is the determining value
            },
            writer: {
                type: 'json'
            }
        }
    });

And we get this – no more blank lines and 14 pages in the pager 🙂

rest2

14 thoughts on “Searching XPages REST service and returning the real count in the results.

  1. Is my mind playing tricks or where you’re doing this:

    items[0].theRealCount

    couldn’t you just get the number of array entries instead? Something like ‘items.length’ instead of all the messing about with columns and extra FTSearch?

      • Vitor,

        If I understand you correctly – If I was loading everything at one time then yes it would.

        But in my case this would not help me with paging or infinite scrolling because I am not loading the whole data set in one go (only 25 at a time).

      • You’re absolutely right, I missed the “load them as necessary” bit (need more coffee).

        Does the FTSearch give you a big overhead? Could a separate REST service be used to get the total, would it be any better?

      • It appears not to give any noticable overhead – i am sure there is on but the over all response is within 50 milliseconds so I am ok. I also have a relatively small dataset – no idea what the impact would be if you were scaling to 100,000 docs.

        That said if you were viewing 100,000 docs then using a grid is probably not the most user friendly way of doing that 🙂

        A separate service would not work because the way the grid looks for the “total” appears to be based on the dataset being used by the grid.

  2. Thank you very much.
    Do you have a tips to display the icons?
    Especially display icons that depend on the value of one or more fields of the document.

    thanks again

  3. Just a thought… you don’t have to quite re-write the rest service from scratch if you go the java route – although it is a bit more work that this….

    You need to create a custom database servlet by extending the base servlet class in the extension library. The documentation here is pretty good and walks you through it… http://goo.gl/FysDN .

    To get your count correct rather than implementing RestViewJsonService as it does in the example, simple create a class that extends RestViewJsonService .. lets call it MarkysJsonService, and override the getCount() method.

    If you look at the source for ResetViewJsonService here http://goo.gl/Bb1FI you will see there are a bunch of simple functions you can override (sort order, search, view name etc), just override each of them as needed to parse the URL extjs passes in and you are good to go.

    * Disclaimer: I haven’t tried this and can almost guarantee it will not be as easy as it seems 🙂

  4. This blog has become for me a point of reference for those who develop in domino + extjs.
    There will be some example for creating or editing a document?
    It would be very interesting.
    Thank you very much.

    • Stephano – yes there will be in a couple of weeks probably. I also have a piece in the works for editing the grid inline and using a CRUD update to update the document directly from the grid 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s