EDU-1502: Adds bigQuery page #2432

franrob-projects · 2025-02-18T18:09:46Z

This PR:

Adds new bigQuery page.
- mainly generalizes exciting content for Delivery Hero content.

coderabbitai · 2025-02-18T18:09:54Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

m-hulbert

I think this needs some light restructuring to make it consistent with the other integrations and flow a bit more succinctly.

Links also need updating with/docs/ amongst other things, and we can add it into the TOC and appropriate overview now too.

m-hulbert · 2025-03-13T17:25:03Z

content/bigquery.textile

@@ -0,0 +1,109 @@
+---
+title: BigQuery rule


I'm not sure what you're doing with the others, but tempted to say drop 'rule' from all of them.

Also consider including GCP or Google like we do for AWS maybe?

m-hulbert · 2025-03-13T17:25:46Z

content/bigquery.textile

+meta_description: "Stream realtime event data from Ably into Google BigQuery using the Firehose BigQuery rule. Configure, and analyze your data efficiently."
+---
+
+Stream events published to Ably directly into a table in "BigQuery":https://cloud.google.com/bigquery?utm_source=google&utm_medium=cpc&utm_campaign=emea-es-all-en-dr-bkws-all-all-trial-e-gcp-1707574&utm_content=text-ad-none-any-dev_c-cre_574561258287-adgp_Hybrid+%7C+BKWS+-+EXA+%7C+Txt+-+Data+Analytics+-+BigQuery+-+v1-kwid_43700072692462237-kwd-12297987241-userloc_1005419&utm_term=kw_big+query-net_g-plac_&&gad_source=1&gclid=Cj0KCQiAwtu9BhC8ARIsAI9JHanslQbN6f8Ho6rvEvozknlBMbqaea0s6ILK-VA9YpQhRr_IUrVz6rYaAtXeEALw_wcB&gclsrc=aw.ds&hl=en for analytical or archival purposes. General use cases include:


Watch your link here, this includes a lot of unnecessary params such as UTM

m-hulbert · 2025-03-13T17:26:10Z

content/bigquery.textile

+* Historical auditing of messages.
+
+<aside data-type='note'>
+<p>Ably's BigQuery integration rule for "Firehose":/integrations/streaming is in development status.</p>


We need to clarify this in light of the product status changes.

m-hulbert · 2025-03-13T17:26:53Z

content/bigquery.textile

+* Create a BigQuery table in that dataset:
+** Use the "JSON schema":#schema.
+** For large datasets, partition the table by ingestion time, with daily partitioning recommended for optimal performance.
+* Create a Google Cloud Platform (GCP) "service account":https://cloud.google.com/iam/docs/service-accounts-create?utm_source=chatgpt.com with the minimal required BigQuery permissions.


UTM link again.

m-hulbert · 2025-03-13T17:29:22Z

content/bigquery.textile

+<p>Ably's BigQuery integration rule for "Firehose":/integrations/streaming is in development status.</p>
+</aside>
+
+h3(#create-rule). Create a BigQuery rule


This should be a H2 and the first part isn't creating a rule; it's configuring GCP resources.

m-hulbert · 2025-03-13T17:31:16Z

content/bigquery.textile

+
+h4(#api-rule). Create a BigQuery rule using the Control API
+
+The following steps to create a BigQuery rule using the "Control API:":https://ably.com/docs/api#control-api


I'm not sure this should be steps. This could just be an API call?

m-hulbert · 2025-03-13T17:32:08Z

content/bigquery.textile

+
+h3(#schema). JSON Schema
+
+You can run queries directly against the Ably-managed BigQuery table. For example, if the message payloads are stored as raw JSON in the data column, you can parse them using the following query:


This says 'the following query' but is showing a payload.

m-hulbert · 2025-03-13T17:32:41Z

content/bigquery.textile

+
+h3(#queries). Direct queries
+
+Run queries directly against the Ably-managed table. For instance, to parse JSON payloads stored in @data@:


This needs some explanation around the data property otherwise it's a little out of context.

m-hulbert · 2025-03-13T17:33:35Z

content/bigquery.textile

+The following explains the components of the query:
+
+|_. Query Function |_. Purpose |
+| @CAST(data AS STRING)@ | Converts the data column from BYTES (if applicable) into a STRING format. |
+| @PARSE_JSON(…)@ | Parses the string into a structured JSON object for easier querying. |
+| @WHERE channel = “my-channel”@ | Filters results to retrieve messages only from a specific Ably channel. |


I'm not sure we need to explain the query like this - we could write this in a short sentence before the example.

m-hulbert · 2025-03-13T17:34:18Z

content/bigquery.textile

+<aside data-type='note'>
+<p>Parsing JSON at query time can be computationally expensive for large datasets. If your queries need frequent JSON parsing, consider pre-processing and storing structured fields in a secondary table using an ETL pipeline for better performance.</p>
+</aside>


I think you could make this part of the ETL section below since it's a sub-section rather than calling it out as a note.

m-hulbert

Couple of small comments to improve understanding here.

m-hulbert · 2025-04-04T10:20:28Z

src/data/nav/platform.ts

@@ -139,6 +139,10 @@ export default {
              name: 'Pulsar',
              link: '/docs/integrations/streaming/pulsar',
            },
+            {
+              name: 'BigQuery',


Suggested change

name: 'BigQuery',

name: 'Google BigQuery',

m-hulbert · 2025-04-04T10:21:45Z