Table Paging and Sorting with filtered client-side data

Registered by Randy Bertram on 2014-11-17

Summary
=======
An improved filtering UI will enable tables to have standard behavior (sorting, paging, row count) by maintaining the appropriate client-side data. The Eucalyptus “magic search” widget will combine client-side filtering and API filtering.

Motivation
========
Current table UX is prohibitive for >20 items. This BP:
- Adds "Previous" button, paging backward through client-side data.
- Sorts all items (not just current page) for any column, using client-side data.
- Shows current position in entire dataset by getting total count from API (when available).
- Filters multiple columns via table search on client-side data.
- Filters individual columns on server-side data (via APIs) in the same search widget.
- Because of the above, the table needs fewer rows, so you can see header & footer on screen at once.

This design relies only on existing APIs. Elastic Search will solve the same problems with a better UX, but when it is not available it would gracefully degrade to this design.

Description
=========
As background, an AngularJS smart-table can show multiple pages of client-side data. The smart-table can page down and up through the client-side data, and also can sort it by column.

This BP focuses on the interplay between the complete dataset on the server-side and the partial dataset on the client-side. This involves two enhancements to the smart-table:
1. A table filter that filters the client-side data, but also drives API filters as needed.
2. An indicator of size/position in the dataset, and whether the client-side data is complete.

Refer to http://invis.io/U21P372EA for 6 screenshots that walk through a scenario. At the end, a flowchart shows the actual design. The screenshots do not show latest table design or Eucalyptus Magic Search, but refer here for the latter: http://coderslike.us/2015/01/07/magic-search-facets-and-text-in-a-single-widget-for-efficient-search-ux/

The max number of rows to be loaded client-side could be set in the Horizon Settings screen. The default will be determined by performance experimentation, but guesses range from 100 - 1000.

This BP assumes that API filtering can easily get the dataset down to 100-500 rows, which would then be paged/sorted/searched on the client-side. If API filtering can easily get the client-side data down to a single page, then paging/sorting/searching may not be important. The assumption is that it often takes more effort to API-filter from (say) 100 rows down to 20 rows than it would take to page and sort and search.

The design of this BP focuses on the Instances table. However, the initial implementation (e.g. Kilo) would be the Users table, which is being converted to AngularJS first.

UX
===
See http://invis.io/U21P372EA

Testing
======
Test all paths in the flowchart http://invis.io/U21P372EA for each table.
For each column, test with i18n strings:
- filtering by keystroke on the client-side data
- filtering by facet (e.g. API filtering)
Test multiple facets at once.
Test removing facets.

Outside Dependencies
==================
https://blueprints.launchpad.net/nova/+spec/server-count-api is required to get the total number of rows for the Instances table, when it is too many to load client-side. For tables without this API, we will gracefully degrade to something like: “500+”.

Requirements Update Required
========================
Eucalyptus Magic Search must be available.
Angular Smart-Table and other AngularJS libs will already be available.

Doc Impact
=========
Document the new Settings field for max number of rows to load client-side.
Possibly document the behavior of the new filters--but hopefully it will be self-explanatory.

Blueprint information

Status:
Complete
Approver:
David Lyle
Priority:
High
Drafter:
Randy Bertram
Direction:
Approved
Assignee:
Randy Bertram
Definition:
Approved
Series goal:
Accepted for kilo
Implementation:
Implemented
Milestone target:
milestone icon 2015.1.0
Started by
David Lyle on 2015-02-20
Completed by
David Lyle on 2015-03-20

Related branches

Sprints

Whiteboard

asahlin: Implementation of this blue print would provide a huge improvement to table usability.

Nov 27, 2014 tsufiev: This proposal would greatly simplify pagination, especially when showing/going 'Prev' is involved, so I'm almost convinced! The thing that bothers me a bit is whether it will be possible, once the proposal is implemented, to pass filtering parameters with GET-request (see https://review.openstack.org/#/c/128253/)? Also, I guess, that the separate filter in your concept corresponds to the current 'server'-type filter while the 'search included items' corresponds to the current 'query'-type filter, am I right?

Dec 1, 2014 rbertram: tsufiev, I reviewed your filter/GET patch & like it. I don't know why it wouldn't be possible to use GET with this BP. A first patch would possibly use the existing filter design as-is for the top filter, since (as you pointed out) it is just the existing server-type filter that currently sits in the table. Later, the whole window might become Angular, and we could still use GET over AJAX and also keep the URL up-to-date for bookmarking and such with HTML5 pushstate.

Dec 11, 2014 rbertram: A new approach is proposed in Invision (http://invis.io/U21P372EA), which addresses some UX and other problems. It meets the same goals, but combines the existing API filtering with client-side filtering in the same widget. I will update the BP when it stabilizes (late Dec or early Jan).

New direction:
- Combine all filtering into one "multifilter".
- Design for Instances. But implement first in Users.
- Use Angular for client-side table & cache, leveraging work by Thai and others.
- Still get total count from API, and show if overflowed.

Gerrit topic: https://review.openstack.org/#q,topic:bp/filtered-client-side-table,n,z

Addressed by: https://review.openstack.org/151386
    WIP Magic Search in Angular User

Addressed by: https://review.openstack.org/156810 -- merged
    WIP REST API for Users filter

Addressed by: https://review.openstack.org/157481
    Magic Search Enablement

Gerrit topic: https://review.openstack.org/#q,topic:demo-magic-search,n,z

Gerrit topic: https://review.openstack.org/#q,topic:enable-magic-search,n,z

Addressed by: https://review.openstack.org/158027
    WIP Magic Instances

Addressed by: https://review.openstack.org/166080
    WIP Smart-Table Client Side Magic Search

Addressed by: https://review.openstack.org/166374
    Update Magic Search

Addressed by: https://review.openstack.org/133767
    WIP angularize identity users.

Addressed by: https://review.openstack.org/166394
    Modified hzSelectAll to watch changes in row length

Gerrit topic: https://review.openstack.org/#q,topic:bp/angularize-identity-tables,n,z

Addressed by: https://review.openstack.org/200261
    Add keystone version to REST API

Addressed by: https://review.openstack.org/202265
    WIP Magic Search in Angular Users table

Addressed by: https://review.openstack.org/221391
    Magic Search in Angular Users table

Addressed by: https://review.openstack.org/245288
    Adding Magic Search codebase to Horizon

Addressed by: https://review.openstack.org/233755
    Add Magic Search Filtering to NG Flavors

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.