Adding Thumbs Up/Down Feedback Buttons to Slack Bot Responses

Intro

After setting up analytics, I could see how many questions users were asking and track response times. But I had no idea whether the answers were actually helpful.

Response time tells you the bot is fast. It doesn't tell you the bot is good.

I wanted a simple way for users to rate responses - thumbs up or thumbs down - without interrupting their flow. The feedback would flow into the same BigQuery pipeline I already had, letting me track satisfaction alongside other metrics.

The Implementation

1. Creating Response Blocks with Buttons

Slack's Block Kit lets you add interactive elements like buttons to messages. I created a helper function to wrap every response with feedback buttons:

1# app/handlers/slack_events.py
2
3def _create_response_blocks(response_text: str) -> list:
4    return [
5        {
6            "type": "section",
7            "text": {
8                "type": "mrkdwn",
9                "text": response_text,
10            },
11        },
12        {
13            "type": "actions",
14            "elements": [
15                {
16                    "type": "button",
17                    "text": {
18                        "type": "plain_text",
19                        "text": "👍",
20                        "emoji": True,
21                    },
22                    "action_id": "feedback_thumbs_up",
23                },
24                {
25                    "type": "button",
26                    "text": {
27                        "type": "plain_text",
28                        "text": "👎",
29                        "emoji": True,
30                    },
31                    "action_id": "feedback_thumbs_down",
32                },
33            ],
34        },
35    ]

The action_id is what Slack uses to route button clicks to the right handler.

2. Using Blocks in Message Handlers

I updated both the mention and DM handlers to use blocks instead of plain text:

1@app.event("app_mention")
2def handle_mention(event, say, logger):
3    # ... existing code ...
4
5    try:
6        with track_response("mention_response", user, channel, clean_text) as tracker:
7            gemini = get_gemini_service()
8            response = gemini.generate_response_sync(user, clean_text)
9            tracker.set_response(response)
10            result = say(
11                text=response,
12                blocks=_create_response_blocks(response),
13                thread_ts=thread_ts,
14            )
15            tracker.set_message_ts(result["ts"])
16    except Exception as e:
17        logger.error(f"Error with Gemini: {e}")
18        say(text=ERROR_MESSAGE, thread_ts=thread_ts)

A few important details:

text is still required - It's the fallback for notifications and accessibility
result["ts"] - The say() function returns the sent message's timestamp, which I save for linking feedback to responses later
set_message_ts() - A new method I added to the tracker (more on this below)

3. Adding Feedback Logging

I extended the analytics module with a new log_feedback function:

1# app/utils/analytics.py
2
3def log_feedback(
4    user_id: str,
5    channel: str,
6    message_ts: str,
7    feedback: str,
8) -> None:
9    _log_json(
10        {
11            "event_type": "feedback",
12            "user_id": user_id,
13            "channel": channel,
14            "message_ts": message_ts,
15            "feedback": feedback,
16            "feedback_value": 1 if feedback == "thumbs_up" else -1,
17        }
18    )

The message_ts field is the key - it links feedback to the original response in BigQuery.

I also added message_ts to the response tracker so both logs share the same identifier:

1class ResponseTracker:
2    def __init__(self, ...):
3        # ... existing fields ...
4        self.message_ts: Optional[str] = None
5
6    def set_message_ts(self, message_ts: str) -> None:
7        self.message_ts = message_ts
8
9    def log_success(self) -> None:
10        _log_json({
11            # ... existing fields ...
12            "message_ts": self.message_ts,
13        })

4. Handling Button Clicks

Slack Bolt makes handling button clicks simple with the @app.action() decorator:

1@app.action("feedback_thumbs_up")
2def handle_thumbs_up(ack, body, client, logger):
3    ack()
4    user = body["user"]["id"]
5    channel = body["channel"]["id"]
6    message_ts = body["message"]["ts"]
7    message_blocks = body["message"].get("blocks", [])
8
9    log_feedback(user, channel, message_ts, "thumbs_up")
10    logger.info(f"Thumbs up from {user} on message {message_ts}")
11
12    response_blocks = [b for b in message_blocks if b.get("type") != "actions"]
13    response_blocks.append({
14        "type": "context",
15        "elements": [{"type": "mrkdwn", "text": "👍 _フィードバックありがとうございます_"}],
16    })
17    client.chat_update(channel=channel, ts=message_ts, blocks=response_blocks)
18
19
20@app.action("feedback_thumbs_down")
21def handle_thumbs_down(ack, body, client, logger):
22    ack()
23    user = body["user"]["id"]
24    channel = body["channel"]["id"]
25    message_ts = body["message"]["ts"]
26    message_blocks = body["message"].get("blocks", [])
27
28    log_feedback(user, channel, message_ts, "thumbs_down")
29    logger.info(f"Thumbs down from {user} on message {message_ts}")
30
31    response_blocks = [b for b in message_blocks if b.get("type") != "actions"]
32    response_blocks.append({
33        "type": "context",
34        "elements": [{"type": "mrkdwn", "text": "👎 _フィードバックありがとうございます_"}],
35    })
36    client.chat_update(channel=channel, ts=message_ts, blocks=response_blocks)

Key points:

ack() - Must be called immediately to acknowledge the button click (Slack requires a response within 3 seconds)
client.chat_update() - Updates the original message to remove buttons and show confirmation
Removing action blocks - Filter out the actions block to prevent double-clicking
Adding context - Show a small "thank you" message where the buttons were

5. Enabling Interactivity in Slack

This part tripped me up initially. After deploying, clicking buttons showed an error about the app not being configured for interactive responses.

To fix this:

Go to api.slack.com/apps
Select your app
Click Interactivity & Shortcuts in the sidebar
Toggle Interactivity to On
Set the Request URL to the same endpoint you use for events: https://your-domain.com/slack/events
Click Save Changes

Slack Bolt automatically routes interactive payloads to the right @app.action() handler.

Testing

I wrote tests to verify the button creation and handler behavior:

1# tests/test_feedback.py
2
3class TestCreateResponseBlocks:
4    def test_creates_section_with_response_text(self):
5        blocks = _create_response_blocks("Hello, world!")
6
7        assert len(blocks) == 2
8        assert blocks[0]["type"] == "section"
9        assert blocks[0]["text"]["text"] == "Hello, world!"
10
11    def test_creates_actions_with_feedback_buttons(self):
12        blocks = _create_response_blocks("Test response")
13
14        actions = blocks[1]
15        assert actions["type"] == "actions"
16        assert len(actions["elements"]) == 2
17        assert actions["elements"][0]["action_id"] == "feedback_thumbs_up"
18        assert actions["elements"][1]["action_id"] == "feedback_thumbs_down"
19
20
21class TestLogFeedback:
22    def test_logs_thumbs_up_feedback(self, capsys):
23        log_feedback(
24            user_id="U123",
25            channel="C456",
26            message_ts="1234567890.123456",
27            feedback="thumbs_up",
28        )
29
30        captured = capsys.readouterr()
31        log_entry = json.loads(captured.out.strip())
32
33        assert log_entry["event_type"] == "feedback"
34        assert log_entry["feedback"] == "thumbs_up"
35        assert log_entry["feedback_value"] == 1

For handler tests, I mocked the Slack app and captured the registered handlers to test them in isolation without making real API calls.

BigQuery Integration

Since feedback logs use the same _log_json function with message="bot_analytics", they flow into BigQuery automatically through the existing log sink.

The schema now includes feedback events:

1SELECT
2  timestamp,
3  jsonPayload.event_type,
4  jsonPayload.message_ts,
5  jsonPayload.feedback,
6  jsonPayload.feedback_value
7FROM `PROJECT.DATASET.run_googleapis_com_stdout_*`
8WHERE jsonPayload.event_type = "feedback"
9ORDER BY timestamp DESC

To join feedback with the original response:

1SELECT
2  r.timestamp as response_time,
3  r.jsonPayload.question,
4  r.jsonPayload.response_time_ms,
5  f.jsonPayload.feedback
6FROM `PROJECT.DATASET.run_googleapis_com_stdout_*` r
7LEFT JOIN `PROJECT.DATASET.run_googleapis_com_stdout_*` f
8  ON r.jsonPayload.message_ts = f.jsonPayload.message_ts
9  AND f.jsonPayload.event_type = "feedback"
10WHERE r.jsonPayload.event_type IN ("dm_response", "mention_response")
11ORDER BY r.timestamp DESC

Looker Studio Update

I had to update my Looker Studio query to handle the new event type. Feedback logs don't have a success field, so I changed:

1jsonPayload.success

to:

1IFNULL(jsonPayload.success, NULL) as success

This prevents the "Field name success does not exist" error for feedback events.

The Log Flow

Here's how it all connects:

User asks a question
Bot responds with dm_response or mention_response log (includes message_ts)
User clicks thumbs up/down
Bot logs feedback event (same message_ts)
Message updates to show confirmation, buttons disappear
Both logs flow to BigQuery, linked by message_ts

Wrapping Up

Adding feedback buttons took about an hour:

Block Kit buttons - Wrap responses in blocks with action buttons
Action handlers - Log feedback and update the message
Visual feedback - Remove buttons after click to prevent duplicates
Slack config - Enable interactivity in the app settings

Now I can track not just how many questions users ask, but whether the answers actually help. The next step would be analyzing which types of questions get negative feedback and improving the knowledge base accordingly.