Skip to content

[BUG] Slow query execution (> 5 seconds) leads to Fider being unavailable (HTTP 502) #1343

@niels-hb

Description

@niels-hb

Fider Cloud or Self Hosted
Self Hosted Kubernetes: getfider/fider:v0.28.0

Describe the bug
Starting today (presumably after we reached a certain threshold of data in our Fider installation), we noticed that the website is extremely unresponsive or completely unavailable. All requests are timing out.

After some debugging, we noticed that one specific query started by Fider is taking up an unreasonable amount of total query time on our PostgreSQL database.

To Reproduce

  1. Install Fider
  2. Generate lots of user data
  3. Connect to the database
  4. Execute query (see below) once with and once without the last ORDER BY statement
  5. Without the ORDER BY statement, the query execution time is significantly improved.

Expected behavior
The queries used by Fider should be optimized for minimum runtime duration.

Additional context
Problematic ORDER BY:

ORDER BY 
  (
    (
      COALESCE(recent_votes_count, 0)* 5 + COALESCE(recent_comments_count, 0) * 3
    )-1
  ) / pow(
    (
      EXTRACT(
        EPOCH 
        FROM 
          current_timestamp - created_at
      )/ 3600
    ) + 2, 
    1.4
  ) DESC

Full query:

SELECT 
  * 
FROM 
  (
    WITH agg_tags AS (
      SELECT 
        post_id, 
        ARRAY_REMOVE(
          ARRAY_AGG(tags.slug), 
          NULL
        ) as tags 
      FROM 
        post_tags 
        INNER JOIN tags ON tags.ID = post_tags.TAG_ID 
        AND tags.tenant_id = post_tags.tenant_id 
      WHERE 
        post_tags.tenant_id = $1 
        AND tags.is_public = true 
      GROUP BY 
        post_id
    ), 
    agg_comments AS (
      SELECT 
        post_id, 
        COUNT(
          CASE WHEN comments.created_at > CURRENT_DATE - INTERVAL '30 days' THEN 1 END
        ) as recent, 
        COUNT(*) as all 
      FROM 
        comments 
        INNER JOIN posts ON posts.id = comments.post_id 
        AND posts.tenant_id = comments.tenant_id 
      WHERE 
        posts.tenant_id = $1 
        AND comments.deleted_at IS NULL 
      GROUP BY 
        post_id
    ), 
    agg_votes AS (
      SELECT 
        post_id, 
        COUNT(
          CASE WHEN post_votes.created_at > CURRENT_DATE - INTERVAL '30 days' THEN 1 END
        ) as recent, 
        COUNT(*) as all 
      FROM 
        post_votes 
        INNER JOIN posts ON posts.id = post_votes.post_id 
        AND posts.tenant_id = post_votes.tenant_id 
      WHERE 
        posts.tenant_id = $1 
      GROUP BY 
        post_id
    ) 
    SELECT 
      p.id, 
      p.number, 
      p.title, 
      p.slug, 
      p.description, 
      p.created_at, 
      COALESCE(agg_s.all, 0) as votes_count, 
      COALESCE(agg_c.all, 0) as comments_count, 
      COALESCE(agg_s.recent, 0) AS recent_votes_count, 
      COALESCE(agg_c.recent, 0) AS recent_comments_count, 
      p.status, 
      u.id AS user_id, 
      u.name AS user_name, 
      u.email AS user_email, 
      u.role AS user_role, 
      u.status AS user_status, 
      u.avatar_type AS user_avatar_type, 
      u.avatar_bkey AS user_avatar_bkey, 
      p.response, 
      p.response_date, 
      r.id AS response_user_id, 
      r.name AS response_user_name, 
      r.email AS response_user_email, 
      r.role AS response_user_role, 
      r.status AS response_user_status, 
      r.avatar_type AS response_user_avatar_type, 
      r.avatar_bkey AS response_user_avatar_bkey, 
      d.number AS original_number, 
      d.title AS original_title, 
      d.slug AS original_slug, 
      d.status AS original_status, 
      COALESCE(agg_t.tags, ARRAY[] :: text[]) AS tags, 
      COALESCE(null, false) AS has_voted 
    FROM 
      posts p 
      INNER JOIN users u ON u.id = p.user_id 
      AND u.tenant_id = $1 
      LEFT JOIN users r ON r.id = p.response_user_id 
      AND r.tenant_id = $1 
      LEFT JOIN posts d ON d.id = p.original_id 
      AND d.tenant_id = $1 
      LEFT JOIN agg_comments agg_c ON agg_c.post_id = p.id 
      LEFT JOIN agg_votes agg_s ON agg_s.post_id = p.id 
      LEFT JOIN agg_tags agg_t ON agg_t.post_id = p.id 
    WHERE 
      p.status != 6 
      AND p.tenant_id = $1 
      AND p.status = ANY($2)
  ) AS q 
WHERE 
  1 = 1 
ORDER BY 
  (
    (
      COALESCE(recent_votes_count, 0)* 5 + COALESCE(recent_comments_count, 0) * 3
    )-1
  ) / pow(
    (
      EXTRACT(
        EPOCH 
        FROM 
          current_timestamp - created_at
      )/ 3600
    ) + 2, 
    1.4
  ) DESC 
LIMIT 
  30;

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugsomething is broken, we need to fix it

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions