[Exist-open] pagination

[Eduard Drenth]

Dear all,

Does exist-db implement or promote pagination techniques or best practices?

Eduard Drenth, Software Architekt

***@fryske-akademy.nl

Doelestrjitte 8
8911 DX Ljouwert
[phone]

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

[Joe Wicentowski]

Hi Eduard,

> Does exist-db implement or promote pagination techniques or best practices?

Sure, many eXist apps implement pagination of query results. Could
you be more specific about what you have in mind?

Joe

[Eduard Drenth]

Hi,

Suppose I have a query yielding lots of results and I do this:

mylib:somequery($query)[$start to $end]

I want to prevent that each time first the query will be executed, then the result set will be built and after this it will be filtered.

Is there a way to achieve this, or is the 'engine' smart enough to optimize?

A seperate subject is that in the response I would like to see a total (which may imply executing an extra count query).

In SQL you have LIMIT and OFFSET. Totals in SQL is often achieved by executing a count() query.

Regards,

Eduard Drenth, Software Architekt

***@fryske-akademy.nl

Doelestrjitte 8
8911 DX Ljouwert
[phone]

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

________________________________________
From: Joe Wicentowski <***@gmail.com>
Sent: Tuesday, June 6, 2017 3:59 PM
To: Eduard Drenth
Cc: exist-***@lists.sourceforge.net
Subject: Re: [Exist-open] pagination

Hi Eduard,

> Does exist-db implement or promote pagination techniques or best practices?

Sure, many eXist apps implement pagination of query results. Could
you be more specific about what you have in mind?

Joe

[Joe Wicentowski]

Hi Eduard,

eXide stores the results of the most recently executed query in a
request attribute. For the source code showing this, see
https://github.com/wolfgangmm/eXide/blob/develop/controller.xql#L164-L230
and https://github.com/wolfgangmm/eXide/blob/develop/modules/session.xql#L38-L102.
The relevant functions are session:set-attribute() and
session:get-attribute(); see
http://exist-db.org/exist/apps/fundocs/view.html?uri=http://exist-db.org/xquery/session#get-attribute.1
and http://exist-db.org/exist/apps/fundocs/view.html?uri=http://exist-db.org/xquery/session#set-attribute.2.
eXide uses URL rewriting and MVC pipeline architecture to facilitate
storage/retrieval of these with the set-attribute action; see
http://exist-db.org/exist/apps/doc/urlrewrite.xml#D1.2.8.

Also you might find the cache module useful. It's not enabled by
default, but you can uncomment this line in your conf.xml file to
enable it: https://github.com/eXist-db/exist/blob/develop/conf.xml.tmpl#L915.

Joe

On Tue, Jun 6, 2017 at 11:00 AM, Eduard Drenth
<***@fryske-akademy.nl> wrote:
> Hi,
>
> Suppose I have a query yielding lots of results and I do this:
>
> mylib:somequery($query)[$start to $end]
>
> I want to prevent that each time first the query will be executed, then the result set will be built and after this it will be filtered.
>
> Is there a way to achieve this, or is the 'engine' smart enough to optimize?
>
> A seperate subject is that in the response I would like to see a total (which may imply executing an extra count query).
>
> In SQL you have LIMIT and OFFSET. Totals in SQL is often achieved by executing a count() query.
>
> Regards,
>
> Eduard Drenth, Software Architekt
>
> ***@fryske-akademy.nl
>
> Doelestrjitte 8
> 8911 DX Ljouwert
> [phone]
>
> gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43
>
> ________________________________________
> From: Joe Wicentowski <***@gmail.com>
> Sent: Tuesday, June 6, 2017 3:59 PM
> To: Eduard Drenth
> Cc: exist-***@lists.sourceforge.net
> Subject: Re: [Exist-open] pagination
>
> Hi Eduard,
>
>> Does exist-db implement or promote pagination techniques or best practices?
>
> Sure, many eXist apps implement pagination of query results. Could
> you be more specific about what you have in mind?
>
> Joe

[Dannes Wessels]

Context please?

Regards

Dannes

--
www.exist-db.org

> On 6 Jun 2017, at 10:17, Eduard Drenth <***@fryske-akademy.nl> wrote:
>
> Does exist-db implement or promote pagination techniques or best practices?

[Eduard Drenth]

Hi,

It was more a general question, with growing size of datasets and increasing number of queries I would like to know if pagination is available to keep things up and running.

- large collections
- many xquery queries with lots of results
- pagination for performance and stability

input: limit and offset
output: limit, offset, total

I hope this is context enough? Bye,

Eduard Drenth, Software Architekt

***@fryske-akademy.nl

Doelestrjitte 8
8911 DX Ljouwert
[phone]

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

________________________________________
From: Dannes Wessels <***@exist-db.org> on behalf of Dannes Wessels <***@exist-db.org>
Sent: Tuesday, June 6, 2017 12:12 PM
To: Eduard Drenth
Cc: exist-***@lists.sourceforge.net
Subject: Re: [Exist-open] pagination

Context please?

Regards

Dannes

--
www.exist-db.org

> On 6 Jun 2017, at 10:17, Eduard Drenth <***@fryske-akademy.nl> wrote:
>
> Does exist-db implement or promote pagination techniques or best practices?

[Loren Cahlander]

You can search the XQuery WikiBook for examples on pagination:

https://en.wikibooks.org/wiki/XQuery/Limiting_Result_Sets <https://en.wikibooks.org/wiki/XQuery/Limiting_Result_Sets>
https://en.wikibooks.org/wiki/XQuery/Keyword_Search#Paginate_and_Summarize_Results <https://en.wikibooks.org/wiki/XQuery/Keyword_Search#Paginate_and_Summarize_Results>


> On Jun 6, 2017, at 6:23 AM, Eduard Drenth <***@fryske-akademy.nl> wrote:
>
> Hi,
>
> It was more a general question, with growing size of datasets and increasing number of queries I would like to know if pagination is available to keep things up and running.
>
> - large collections
> - many xquery queries with lots of results
> - pagination for performance and stability
>
> input: limit and offset
> output: limit, offset, total
>
> I hope this is context enough? Bye,
>
> Eduard Drenth, Software Architekt
>
> ***@fryske-akademy.nl
>
> Doelestrjitte 8
> 8911 DX Ljouwert
> [phone]
>
> gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43
>
> ________________________________________
> From: Dannes Wessels <***@exist-db.org> on behalf of Dannes Wessels <***@exist-db.org>
> Sent: Tuesday, June 6, 2017 12:12 PM
> To: Eduard Drenth
> Cc: exist-***@lists.sourceforge.net
> Subject: Re: [Exist-open] pagination
>
> Context please?
>
> Regards
>
> Dannes
>
> --
> www.exist-db.org
>
>> On 6 Jun 2017, at 10:17, Eduard Drenth <***@fryske-akademy.nl> wrote:
>>
>> Does exist-db implement or promote pagination techniques or best practices?
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Exist-open mailing list
> Exist-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/exist-open

[Wolfgang Meier]

> It was more a general question, with growing size of datasets and increasing number of queries I would like to know if pagination is available to keep things up and running.

As a rule of thumb, paginate immediately after executing a query to build the result sequence. Do not apply any post-processing, but extract the items to be shown on the current page, using e.g. fn:subsequence($result, $offset, $max), store the rest into the HTTP session with session:set-attribute and call post-processing to format the output only on the subsequence. Background: within eXist, references to nodes stored inside the database do not consume a lot of memory as they are just pointers. However, if you post-process and format those results, you create in-memory XML fragments which do consume memory.

Cheers,

Wolfgang

[Eduard Drenth]

Thanks, good to know about the pointers also. Keeping the results in session means that the first call may take a while, subsequent calls directly address the results in the session. Am I right?

Eduard Drenth, Software Architekt

***@fryske-akademy.nl

Doelestrjitte 8
8911 DX Ljouwert
[phone]

gpg: https://sks-keyservers.net/pks/lookup?op=get&search=0x065EF82A1E02CC43

________________________________________
From: Wolfgang Meier <***@exist-db.org>
Sent: Wednesday, June 7, 2017 3:33 PM
To: Eduard Drenth
Cc: Dannes Wessels; exist-***@lists.sourceforge.net
Subject: Re: [Exist-open] pagination

> It was more a general question, with growing size of datasets and increasing number of queries I would like to know if pagination is available to keep things up and running.

As a rule of thumb, paginate immediately after executing a query to build the result sequence. Do not apply any post-processing, but extract the items to be shown on the current page, using e.g. fn:subsequence($result, $offset, $max), store the rest into the HTTP session with session:set-attribute and call post-processing to format the output only on the subsequence. Background: within eXist, references to nodes stored inside the database do not consume a lot of memory as they are just pointers. However, if you post-process and format those results, you create in-memory XML fragments which do consume memory.

Cheers,

Wolfgang

[Wolfgang Meier]

> Thanks, good to know about the pointers also. Keeping the results in session means that the first call may take a while, subsequent calls directly address the results in the session. Am I right?

Yes, that’s right.

Wolfgang