Daniel Leightley

Using Python to query and extract data from Firestore, Storage Buckets and BigQuery

Sep 1, 2023

Google Firebase is a set of cloud-based development tools that helps mobile app developers like me, build, deploy, and scale our applications. It is easy to use, and cost effective. That is why I've built all our frontend digital infrastructure at KCMHR with Google Firebase using NodeJS and Typescript to programme a RESTful API.

Google Firebase is a great platform for rapid development using serverless technology. This means that we can focus on frontend development and not server management. A major bonus is that it is supported by Google, so you get all the top-notch security support bundled in. But one aspect that is a little difficult is getting data out of Google Firebase. It isn't that easy, or straightforward.

That is why I wanted to share three scripts I've created to gather data from Google Firebase, Google Storage Buckets and Google BigQuery - the three main areas where you'll store data on the platform.

There are multiple ways in which you can query the data, the ways below, for me, are the simplest way. Hereafter I will provide a Gist for each method, with the code commented outlining what is happening.

Each method requires Python 3.5, with the following packages installed as a minimum:

  • pandas
  • firebase_admin
  • google.cloud

You will also need to download your service account key either from Google Firebase OR Google Cloud Console.

Getting data from Google Firestore

If the Gist does not load, you can access it here.

Getting data from Storage Buckets

If the Gist does not load, you can access it here.

Getting data from BigQuery

If the Gist does not load, you can access it here.

It is important to note that when using BigQuery you need to elevate permissions on the service account to enable it to run BigQueries.

Daniel Leightley