Query for Post List

Discussion for adding a way to query a list of posts from the Micropub endpoint (https://indieweb.org/Micropub-extensions#Query_for_Post_List)

Below are some thoughts from the brainstorming page.

Use Cases

[[headless_CMS|headless cms]] / Separation of data storage and data display
Private note taking / journaling
Find and update existing posts eg. change a "want to read" to "read" after complete
Full content management UIs
Find and edit draft / hidden posts
Synchronizing various documents eg. bookmarks
Discover if a user has liked / replied to a specific page

Query Parameters

In order to achieve the above use cases there would need to be a robust query system in place. It would need to be able to query for:

url (this filters and returns the single post with that url. This is how q=source currently functions)
posts based on post type (similar to Query for supported vocabulary #1)
pages of posts with defined limits
posts with specific properties
posts with combinations of properties
posts without certain properties
posts with properties that match a certain value (eg, all posts that have in-reply-to == 'example.com')

Other Considerations

Any returned item in a list would need to include a url property so that micropub clients could use the url for update and delete requests etc.
Would require one or more scopes

Here are the notes about IndieWeb implementation from the brainstorming page:

j4y_funabashi implemented an extension to the q=source query on my micropub endpoint, if no url is provided it returns a list of posts. Not yet paginated but that is the plan.
eddiehinkle implemented the q=source query extension.
- When no url is provided, it returns a list of posts.
- It doesn't support pagination yet, but currently returns all posts within the last month
- It does support a post-type filter that will filter out what posts are returned based on the post type discovery, similar to Query for supported vocabulary #1
grantcodes implemented an experimental extension that uses the mongodb query syntax since his posts are stored in MongoDB

@cleverdevil

@cleverdevil Post List Query discussion has moved here, for when you want to work on integrating into Known

It has been discussing that using paging should probably be similar to Microsub (https://indieweb.org/Microsub-spec#Example_Paging_Workflow) where the server provides both after and before strings for pagination. The server will then return results as appropriate. When you reach the point that there are no more results available, the results don't contain an after string.

The pagination tokens are opaque to the client, so they can be anything the server wants: page numbers, date string, post ids, etc.

It probably makes sense to include the limit parameter from Microsub as well. So that would put us at:

https://example.tld/micropub?q=source

&post-type=type - filters the returned posts to only posts that match the included post type. This would be a post type from the Post Type Discovery
&after=pagination token - returns older posts
&before=pagination token - returns newer posts
&limit=number - limits the returned results by the number provided

As already included in the original spec:

&url=POST URL - returns a single post that matches this url

I think the specific property queries might be more difficult to map into the url parameters.

They could potentially follow a similar format to the micropub update spec

In particular like the delete parameter which can either specify just the property name to delete or the value to delete.

In my mind there needs to be 2 types of queries on properties at least:

A specific property exists
A specific property contains a given value - eg. post-status = draft
This could get more complex, if you wanted to match an entire array (eg. category = ['foo', 'bar']) or just contains the given parameter

There are also a number of others that may be worth having, but I am not so sure are as vital:

A property does not exists / is empty
A property value does not equal a given value
A property value is less than, greater than etc. compared to a given value

Another one worth considering would be a sort or order parameter. I can imagine clients wanting to either retrieve posts in chronological or reverse chronological order at a minimum.

And potentially even ordering by other properties, eg. Alphabetical by post title. Although I can't think of a solid use case for that right now.

@EdwardHinkle

I've updated my implementation so I support a bunch of new features expanding on what @EdwardHinkle already mentioned

When using q=source there is now a lot of other parameters available:

post-type=type - Filters the returned posts to only posts that match the included post type. This would be a post type from the Post Type Discovery
after=pagination token - Returns older posts (I just use page numbers for now)
before=pagination token - Returns newer posts (I just use page numbers for now)
limit=number - Limits the returned results by the number provided
order=reverse - Returns posts in a reverse order
exists[]=property name - Returns posts that include the given properties
not-exists[]=property name - Returns posts that do not include the given properties
property-property-name=value - Returns posts that contain the given property value e.g. property-post-status=draft

@grantcodes

FYI, I have a local branch of Known that supports:

q=source
&url=
&limit=
&offset=

I'm closely watching this thread, and if we can come to a consensus, I should be able to have a fully compliant implementation for Known in short order.

Things that need to be decided, from what I can tell:

What's the official approach to pagination?
Of @grantcodes' and @EdwardHinkle's additional query parameters, which should be adopted?

I will say that the &post-type parameter would be a bit difficult to implement in Known at this stage, because Known doesn't currently have any such notion. That said, a mechanism could be created that would map Known "Entity" types to post-types from the Post Type Discovery spec. I'd likely need direction from @benwerd and @mapkyca on how to do it.

Can someone write down here a valid use case for post type queries? I'm sure there were some but I can't remember why they'd be needed if property queries are supported

Here is a link to the previous reasons cited: https://chat.indieweb.org/dev/2018-08-09#t1533837539815600

To summarize, there are instances where just a single property existing does not define some more complicated post types. So it means you either have to define complicated requirements in URL parameters (including that certain properties DON’T exist) or you use the PTD by using a post-type parameter.

Some examples:

both Podcasts, Articles and Issues have a Title. So how do you get all articles but not podcast episodes or issues when they all have titles?
Determining the difference between Video and Photo posts because Videos often have a photo property.
Using PTD protects against new experimental post types getting returned erroneously because it happens to have the same property as another post type.

I like the post-type filter, and I guess in combination with the post-type discovery, we can either expose them or not in the micropub clients. In the case of the Drupal module: I map every micropub post to a different content type in Drupal. When a q=source would come in with a post-type filter, it would be very easy to filter out, so I'm +1 on that filter :)

I'm going to experiment with this the next week to add this to the drupal module and into indigenous for android. Since I've added basic update post support, having a list to quickly edit (instead of having to copy over urls and such), a very simple list would make my life extremely easy!

So the post list and post-type filter is in the latest release of the indigenous android client and drupal module. The post list listens to the post-type discovery as well, so if none is found, filtering is not possible at the moment. I'll gradually add more filter options (like limit) and paging in the next week.

So now that this is becoming more widely spread, I am starting to have issues with implied post types and micropub queries. Mostly around whether something should be a specific post type or if it should be any post type with a specific property.

My 2 main use cases, (which I think are totally valid, and either in wide spread use or wanted):

Drafts - Obviously there is post-status = draft But drafts are not defined by post type discovery at the moment. So at the moment to get a list of drafts there would either need to be a post-type=draft query or the client and server would both need to support property-post-status=draft.
Private journal entries - Again there is no post type defined for journals, and this time no property that is specific to journals. Personally I just make a note with post-status=private and category=journal. If I wanted to expose these to a micropub client I would likely need to make my own custom post type.

Having written these out I think my main issue is around the creation of custom post types and whether they should be actual post types, or just properties and how that works with micropub queries

@grantcodes

@grantcodes it looks like half your post is missing because you only mention 1 use case

@EdwardHinkle

@EdwardHinkle Yeah, published early by accident, updated now.

Answer to my question: Indigenous for Android expects that this list is the full post's content, as it renders some of the information (as seen in https://media.jvt.me/5a9abbb797.png) and I found would break if any nulls were found.

(Originally published at: https://www.jvt.me/mf2/2020/07/sm4ec/)

Do we have a sense of what properties we should consider “standardized” from this list? I’ve reworked my implementation (it includes support for custom pages and some markup for tags so I can add more information to their standalone pages). Nothing really changes with regards to that but it does means that I’m looking for the field type now as well to determine h-entry, h-x-page and h-x-category (the latter being custom for reasons).

(Originally published at: https://v2.jacky.wtf/post/ef485fa8-33af-4f02-8844-572373a4fd74)

For whatever it's worth, here's an example of the "standard" properties that Micro.blog always returns. At a minimum, I'd think most servers should return content, published, and url. The URL will be needed for editing.

{
  "items": [
    {
      "type": "h-entry",
      "properties": {
        "uid": [ 12345 ],
        "name":[ "Title here" ],
        "content": [ "Hello world." ],    
        "published": [ "2020-07-14T18:54:18+00:00" ],
        "post-status":[ "published" ],
        "url": [ "https://www.manton.org/2020/..." ]
      }
    }
  ]
}

I'd prefer to keep it as a fairly minimal list, maybe even just the type, uid and url, and then require a follow-up call to q=source&url=... if a given post is required. This (possibly) reduces work on the server to render all the information but should make it simpler for folks using static sites as they then don't need to render everything for q=source compared to calls for q=source&url=...

Servers should at least have the option to return name, content, etc. It doesn't have to be required, but it is really important for any app that is showing a list of posts. Doing an extra query if you're showing a list of 100+ posts would be really wasteful.

That's fair enough - if we can clarify that they're not required properties, then that'd allow clients to be built more resiliently 👍

Throwing a wrench into the mix? How do people see a query for all posts that have a property matching a particular value? Like if the audience is set to a particular URL?

Is anyone implementing queries like exists[] and not-exists[] in their systems? I don't see an independent ticket for it on here but I'm curious to see implementation approaches.

I have a tracking ticket for Koype for implementing exists[] and not-exists[]. I plan to use this in Lwa to highlight if a sort of reaction was made to an entry before (like, in a channel, a post has been liked, replied to or shared already) to prevent duplication of content.

There's quite a few things here in this issue. Would it make sense to split it up maybe into some sort of list of items? I don't know if @EdwardHinkle would be willing to add them to the bottom of the top-entry so it'd be easier to link to (with headings) on the Wiki.

(Originally published at: https://jacky.wtf/2022/7/OWkL)

@jalcine

Good idea @jalcine. I feel like some of the older suggestions in this issue (like basic support for ?q=source to get the post list) are widely supported and stable. We should mark those as "done" and create new issues for any experimental extensions where there's no consensus yet.

@jalcine

@jalcine @manton Eddie has been busy with life...I can take a run at it...I split off some things in the past

Issue closed as remaining sub-topics now split into separate issues.

@dshanske

Thanks @dshanske!

Navigation Menu

Repository navigation

Query for Post List #4

Use Cases

Query Parameters

Other Considerations

21 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Query for Post List #4

Description

Use Cases

Query Parameters

Other Considerations

Activity

EdwardHinkle commented on Aug 9, 2018

EdwardHinkle commented on Aug 9, 2018

EdwardHinkle commented on Aug 9, 2018

EdwardHinkle commented on Aug 9, 2018

grantcodes commented on Aug 10, 2018

grantcodes commented on Aug 10, 2018

grantcodes commented on Aug 12, 2018

cleverdevil commented on Sep 6, 2018

grantcodes commented on Sep 9, 2018

EdwardHinkle commented on Sep 9, 2018

swentel commented on Sep 16, 2018

swentel commented on Sep 21, 2018

grantcodes commented on Oct 15, 2018

EdwardHinkle commented on Oct 15, 2018

grantcodes commented on Oct 15, 2018

21 remaining items

jamietanna commented on Jul 11, 2020

jalcine commented on Jul 18, 2020

manton commented on Jul 25, 2020

jamietanna commented on Jul 25, 2020

manton commented on Jul 25, 2020

jamietanna commented on Jul 25, 2020

jalcine commented on Jun 24, 2021

jalcine commented on Feb 6, 2022

jalcine commented on Mar 14, 2022

jalcine commented on Jul 3, 2022

manton commented on Jul 4, 2022

dshanske commented on Jul 4, 2022

dshanske commented on Jul 5, 2022

manton commented on Jul 5, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions