A couple of Tips on how to optimize your XPages REST Service

In this article will demonstrate how simple planning and insight can make your XPages EXTLib REST service more efficient.

Problem

Whatever you are using your REST service for, wither it is to feed charts, grids, other application or whatever it is, they take up data bandwidth and processing time.

Consider the following notes view

rest1

If I display that in a simple REST service one of the entries looks like this:


  {
      "@entryid":"1-45995DDEB92AA70C882579F0008280CB",
      "@unid":"45995DDEB92AA70C882579F0008280CB",
      "@noteid":"1A12",
      "@position":"1",
      "@siblings":200,
      "@form":"Contact",
      "$13":"Adriana Kerr",
      "FirstName":"Adriana",
      "LastName":"Kerr",
      "EMail":"adriana_kerr@renovations.com",
      "City":"Norwalk",
      "State":"CA",
      "Created":"2012-04-30T23:45:26Z",
      "$10":"Adriana Kerr"
  },

This batch of data takes up 462 bytes. The first 6 lines the attributes @entryid to @form you have no control over. But you do have controll over the others.

The field name displayed comes from the computed column value in the view. If you have a column which is just a single field the programmatic name will reflect the field Name.

rest2

But if you have a column value which not just a fieldname then you will have a programmatic id assigned

rest3

1) Taking control of the column names

The point is that these values can be changed

rest4

And in doing this your rest service changes:


  {
      "@entryid":"1-45995DDEB92AA70C882579F0008280CB",
      "@unid":"45995DDEB92AA70C882579F0008280CB",
      "@noteid":"1A12",
      "@position":"1",
      "@siblings":200,
      "@form":"Contact",
      "1":"Adriana Kerr",
      "2":"Adriana",
      "3":"Kerr",
      "4":"adriana_kerr@renovations.com",
      "5":"Norwalk",
      "6":"CA",
      "7":"2012-04-30T23:45:26Z",
      "8":"Adriana Kerr"
  },

And this batch of data takes up 426 bytes (as opposed to 462 before). This right there represents a saving of 8% which might not seem like much but this is only a small view with only a few columns with only short field name.

If you scale this up to a large view with 25 columns and large field names that can make 2000 bytes per entry – I just performed a quick test on one of my larger REST services and saved over 25% of the data (which is what inspired me to write the article).

“Soooo what” I hear you say – well consider 500 data entries in my world. That ramps up very quickly to a 1M data transfer – saving 25% on that is a significant bandwidth saving – not to mention the reduction in load on your browser having to parse that amount of data.

XPiNC memory 

This is especially pertinent in the XPiNC world which I have the unfortunate pleasure of playing with – it is Firefox 3.5 and has memory issues. I have nothing quantifiable to back this up but there is no doubt about it in my mind – a large data volume causes issues in the browser memory management. This is highlighted by a slowdown of the interface – you know the things we USED to have a problem with but don’t any more with Chrome and Firefox browsers.

2) Better control and only send what you need….

This should be common sense but if you are upgrading an existing database and want to take advantage of existing views, might not always be apparent.

By default the view send all the columns when you select  defaultColumns=”true”

<xe:restService id="restService3" pathInfo="contactjson">
	<xe:this.service>
		<xe:viewJsonService viewName="AllContacts"
				var="entry" contentType="text/plain"
				count="10" defaultColumns="true">
		</xe:viewJsonService>
	</xe:this.service>
</xe:restService>

By going into the all properties and selecting the columns you want you have more control and save bandwidth/memory management

In this case I put the column names back and programmatically added the columns to the REST service

rest5

And then I added the id and firstName column programmatically to the REST service – but as you can see I forced the name of the field in this case to be 1 and 2

<xe:restService id="restService3" pathInfo="contactjson">
	<xe:this.service>
		<xe:viewJsonService viewName="AllContacts"
				var="entry" contentType="text/plain"
				count="10" defaultColumns="false">
			<xe:this.columns>
				<xe:restViewColumn name="1" columnName="id"></xe:restViewColumn>
				<xe:restViewColumn name="2" columnName="firstName"></xe:restViewColumn>
			</xe:this.columns>
		</xe:viewJsonService>
	</xe:this.service>
</xe:restService>

This creates the optimized JSON feed with the smaller field name, without changing it at the column.

  {
      "@entryid":"1-45995DDEB92AA70C882579F0008280CB",
      "@unid":"45995DDEB92AA70C882579F0008280CB",
      "@noteid":"1A12",
      "@position":"1",
      "@siblings":200,
      "@form":"Contact",
      "1":"Adriana Kerr",
      "2":"Adriana",
  },

This code is (for example) all that is needed from this view and at 263 Bytes it is 39% smaller than the original
Caveats

  • If the bulk of your data is in the field value and not in the field names you will not see such a large improvement.
  • Changing the column names in a view needs to be done carefully as those names could be referenced elsewhere.
  • When you have a $10 programmatic column name and you change the formula in the column the name is recomputed – be aware that this can change

16 thoughts on “A couple of Tips on how to optimize your XPages REST Service

  1. Is Adriana Kerr the secret clone daughter of Adriana Lima and Miranda Kerr? Because if so, can you please display a phone number column as well? I mean, seriously; playas gotta play.

  2. Interesting but I would say this is only useful if there’s only 1 developer and you weren’t intending to leverage the API somewhere else – say passing to another Web Developer.

    It would drive me potty to try and work with that JSON data, imagine using the dot notation syntax to get to the first name.

    BTW I think It is possible to remove the programmatic entries but you have to write your own custom Java beans / OSGI plugin which extends the inbuilt REST services.

    I did some tests with GZiped JSON Data against AMF, which is a binary format, and it worked out pretty close for reasonably large sets of data so it might be worth checking your Domino server has GZip turned on.

  3. We create the REST APIs for our applications manually by selecting/filtering view data (e.g. all entries, subcategory navigator or ft search result collection), traversing view entries (leveraging the ViewNavigator caching if possible) and writing the JSON data to the servlet output writer with an improved version of the org.json.JSONObject classes or Jackson JSON processor.
    I just like to know what’s going on in the code and reinventing the wheel is my hobby ;-).

    I’m wondering how they retrieve “@form” from the view. The last time I checked the Extlib code, it was accessing the document to read this field, which reduces performance a lot. I think there was a comment in the code that this should be removed or fixed asap.

  4. Interesting indeed as Mark Barton already noted but I really would hate to see a rest service which returns data in fields with names like 1 , 2 or 3. Names of variables should always, in any case, tell the developer what he can expect from it. When using numbers as names developers have no clue what the data is used for and therefore can build unwanted results.

    So that tip is actualy not a tip. I would even call it a little bug you are introducing. It would be best to see if you can somehow compact the data another way such as generating only the columns you need ( as you already pointed out).

  5. I am sure there is a balance of usability and data overload. If this is a public REST service I agree field names might be best – but for an internal application which is suitably documented in the appropriate place maybe LN would work for LastName.

    In my case I had fields as long as txtLastKnownAddress and what’s even worse when the field is blank in the service

    I believe LKN would be sufficient in this case and be 80% smaller

    Maybe my example is an extreme case but highlights to possibility and considerations

  6. Found it in the Extlib code (version 853.20121217-1354) in class com.ibm.domino.services.rest.das.view.RestViewNavigatorFactory:

    // Added by Tim Tripcony
    // Hoping to add support for Form Formulas later on
    @Override
    public String getForm() throws ServiceException {
    String result = “”;
    if (isDocument()) {
    try {
    Document document = entry.getDocument();
    if (document != null) {
    result = document.getItemValueString(ITEM_FORM);
    }
    } catch (NotesException ex) {
    throw new ServiceException(ex,””); // $NON-NLS-1$
    }
    }
    return result;
    }

    They fetch the entry document, but there is no recycle call afterwards. Have you tried calling your REST services with many documents, e.g. several thousands? Might lead to a crash, because Notes is running out of handles.

  7. You can also specify which system columns (the ones with @ as prefix) to return by using the systemColumns property of the REST Service.
    Noticed a slightly reduced loading time.

    I also find using viewItemFileService faster than using viewJsonService.

  8. I know I am late to this party but there’s another setting people might find useful. If you set systemColumns = 0, all those @Siblings, @Read, @Position … values will be gone. You can save a nice chunk of bandwidth right there.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s