Tutorial - Generate Logs in Google Cloud with Vertex AI

Vertex AI was announced at Google I/O 2021. More than just a rebranding of the Google AI Platform, this product starts to unify a lot of different APIs (including AutoML) under one product offering. Google states in a press release that this allows companies to start to implement MLOps easier.

In this blog, we are going to do the equivalent of "Hello World" for Data Science using the Vertex AI platform. In short, we are going to use a Vertex AI "Jupyter" Notebook to communicate with the logging service of Google Cloud. Think of a notebook as a way of running Python code in an iterative manner that allows you to capture the results along the way.

TLDR - Show me the code!

If you use a Vertex AI notebook, you can easily test out the python library for Cloud Logging within Google Cloud. The notebook that you need in order to test Cloud Logging can be downloaded here.

Prerequisites

1. You need to have a Google Cloud Account. If you don't already have an account, take a look at this video which shows how to set up an account.
2. You need to enable the Cloud Logging API. After you have set up an account, you can find details about enabling this API on the Google Cloud Platform .
3. You need to go to the notebooks section of Vertex AI. Enable the notebooks API if you see a corresponding warning.

Create a Vertex AI notebook

Within the Vertex AI console, the first step that we need to do is we need to create a notebook instance. This instance is this is going to be backed by a CPU, so the key is to run this exercise, and when you're done, be sure to delete the notebook instance so that you don't incur additional fees.

As seen above, I'm creating a notebook called test-logging in us-central1 (and you should create your instance in a location that is close to you). I create this notebook with libraries such as TensorFlow and Pandas that would typically be used in data science, and I do this by selecting the TensorFlow Enterprise 2.5 environment.

After clicking create, you will see the test-logging notebook appear in the console, and click on "OPEN JUPTYERLAB" in order to continue.

Once you are in JupyterLab, click on Python [conda env:root] in order to open up a notebook for experimentation.

Now go ahead and enter the following Python code into the notebook.

# Code that sets logging
import logging
import google.cloud.logging_v2 as logging_v2
from os import environ

client = logging_v2.client.Client()

# set the format for the log
google_log_format= logging.Formatter(
fmt='%(name)s | %(module)s | %(funcName)s | %(message)s',
datefmt='%Y-%m-$dT%H:%M:%S')


handler = client.get_default_handler()
handler.setFormatter(google_log_format)

cloud_logger = logging.getLogger("vertex-ai-notebook-logger")
cloud_logger.setLevel("INFO")
cloud_logger.addHandler(handler)

log = logging.getLogger("vertex-ai-notebook-logger")
log.info("This is a log from a Vertex AI Notebook!")

If any of the above code looks unfamiliar, or if you have not used the google-cloud-logging library before, I strongly encourage you to take a look at this video which discusses how to set up the format for logging and how to get a python logger to output information to the cloud.

Verify Results

After running this test code in the notebook, you can head over to the Logs Explorer to see your results.

Finally, remember to go back into the notebook console within Vertex AI to:

1. Select the "instance name" that you created (such as "test-logging")
2. Click the delete icon at the top of the console in order to delete the instance

When successful, you should see something on the console that states " You don't have any notebook instances in this project yet."

Conclusion

In this blog, we briefly reviewed Vertex AI notebooks, and we looked at how those notebooks can communicate with the centralized logging in Google Cloud.

It would be great to hear your thoughts about this blog, you can reach me through my company (White Owl Education) which is on Twitter at @whiteowled .