Microservices observability tools provide you with the ability to instrument your applications to collect and present telemetry data in Cloud Monitoring, Cloud Logging, and Cloud Trace from gRPC workloads deployed on Google Cloud and elsewhere. Microservices observability works with any deployment that has been granted permission to access Monitoring, Logging, and Trace by enabling the Microservices API.
In this tutorial you learn how to use Microservices observability features by building a simple gRPC application on Google Cloud using Compute Engine and instrument your application with Microservices observability, and view them come up live on Monitoring and Logging.
Objectives
At a high level, you do the following.
As a service developer, you can:
- Create a gRPC application with the language of your choice (C++, Go, or Java).
- Opt in your application and control the Microservices observability plugin.
- Deploy the application on a Compute Engine VM.
As a service operator, you consume the collected data in various ways:
- View traces on Trace.
- View metrics on the monitoring dashboard called Microservices (gRPC) Monitoring.
- View metrics on Metrics Explorer.
- Inspect log entries on Logs Explorer.
Costs
In this document, you use the following billable components of Google Cloud:
To generate a cost estimate based on your projected usage,
use the pricing calculator.
When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.
Before you begin
Console
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Roles required to select or create a project
- Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
-
Create a project: To create a project, you need the Project Creator role
(
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission. Learn how to grant roles.
-
Verify that billing is enabled for your Google Cloud project.
-
Enable the Compute Engine, and Microservices API APIs.
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (
roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enablepermission. Learn how to grant roles. -
Ensure that you have the Create Service Accounts IAM role
(
roles/iam.serviceAccountCreator) and the Project IAM Admin role (roles/resourcemanager.projectIamAdmin). Learn how to grant roles. -
In the Google Cloud console, go to the Create service account page.
Go to Create service account - Select your project.
-
In the Service account name field, enter a name. The Google Cloud console fills in the Service account ID field based on this name.
In the Service account description field, enter a description. For example,
Service account for quickstart. - Click Create and continue.
-
Grant the following roles to the service account: Logging > Logs Viewer > Logs Writer, Monitoring > Monitoring Editor > Metrics Writer, Trace > Trace Admin > Trace Agent.
To grant a role, find the Select a role list, then select the role.
To grant additional roles, click Add another role and add each additional role.
- Click Continue.
-
In the Service account users role field, enter the identifier for the principal that will attach the service account to other resources, such as Compute Engine instances.
This is typically the email address for a Google Account.
-
Click Done to finish creating the service account.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Roles required to select or create a project
- Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
-
Create a project: To create a project, you need the Project Creator role
(
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission. Learn how to grant roles.
-
Verify that billing is enabled for your Google Cloud project.
-
Enable the Compute Engine, and Microservices API APIs.
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (
roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enablepermission. Learn how to grant roles. -
Ensure that you have the Create Service Accounts IAM role
(
roles/iam.serviceAccountCreator) and the Project IAM Admin role (roles/resourcemanager.projectIamAdmin). Learn how to grant roles. -
In the Google Cloud console, go to the Create service account page.
Go to Create service account - Select your project.
-
In the Service account name field, enter a name. The Google Cloud console fills in the Service account ID field based on this name.
In the Service account description field, enter a description. For example,
Service account for quickstart. - Click Create and continue.
-
Grant the following roles to the service account: Logging > Logs Viewer > Logs Writer, Monitoring > Monitoring Editor > Metrics Writer, Trace > Trace Admin > Trace Agent.
To grant a role, find the Select a role list, then select the role.
To grant additional roles, click Add another role and add each additional role.
- Click Continue.
-
In the Service account users role field, enter the identifier for the principal that will attach the service account to other resources, such as Compute Engine instances.
This is typically the email address for a Google Account.
-
Click Done to finish creating the service account.
- Read the Microservices observability overview.
- Read about the two supported environment variables, decide which to use, and determine the values required by the environment variable.
cli
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
-
Install the Google Cloud CLI.
-
If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.
-
To initialize the gcloud CLI, run the following command:
gcloud init -
Create or select a Google Cloud project.
Roles required to select or create a project
- Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
-
Create a project: To create a project, you need the Project Creator role
(
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission. Learn how to grant roles.
-
Create a Google Cloud project:
gcloud projects create PROJECT_ID
Replace
PROJECT_IDwith a name for the Google Cloud project you are creating. -
Select the Google Cloud project that you created:
gcloud config set project PROJECT_ID
Replace
PROJECT_IDwith your Google Cloud project name.
-
Verify that billing is enabled for your Google Cloud project.
Enable the Compute Engine, and Microservices API APIs:
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (
roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enablepermission. Learn how to grant roles.gcloud services enable compute.googleapis.com
microservices.googleapis.com -
Set up authentication:
-
Ensure that you have the Create Service Accounts IAM role
(
roles/iam.serviceAccountCreator) and the Project IAM Admin role (roles/resourcemanager.projectIamAdmin). Learn how to grant roles. -
Create the service account:
gcloud iam service-accounts create SERVICE_ACCOUNT_NAME
Replace
SERVICE_ACCOUNT_NAMEwith a name for the service account. -
Grant roles to the service account. Run the following command once for each of the following IAM roles:
roles/logging.logWriter, roles/monitoring.metricWriter, roles/cloudtrace.agent:gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com" --role=ROLE
Replace the following:
SERVICE_ACCOUNT_NAME: the name of the service accountPROJECT_ID: the project ID where you created the service accountROLE: the role to grant
-
Grant the required role to the principal that will attach the service account to other resources.
gcloud iam service-accounts add-iam-policy-binding SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com --member="user:USER_EMAIL" --role=roles/iam.serviceAccountUser
Replace the following:
SERVICE_ACCOUNT_NAME: the name of the service accountPROJECT_ID: the project ID where you created the service accountUSER_EMAIL: the email address for a Google Account
-
Ensure that you have the Create Service Accounts IAM role
(
-
Install the Google Cloud CLI.
-
If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.
-
To initialize the gcloud CLI, run the following command:
gcloud init -
Create or select a Google Cloud project.
Roles required to select or create a project
- Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
-
Create a project: To create a project, you need the Project Creator role
(
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission. Learn how to grant roles.
-
Create a Google Cloud project:
gcloud projects create PROJECT_ID
Replace
PROJECT_IDwith a name for the Google Cloud project you are creating. -
Select the Google Cloud project that you created:
gcloud config set project PROJECT_ID
Replace
PROJECT_IDwith your Google Cloud project name.
-
Verify that billing is enabled for your Google Cloud project.
Enable the Compute Engine, and Microservices API APIs:
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (
roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enablepermission. Learn how to grant roles.gcloud services enable compute.googleapis.com
microservices.googleapis.com -
Set up authentication:
-
Ensure that you have the Create Service Accounts IAM role
(
roles/iam.serviceAccountCreator) and the Project IAM Admin role (roles/resourcemanager.projectIamAdmin). Learn how to grant roles. -
Create the service account:
gcloud iam service-accounts create SERVICE_ACCOUNT_NAME
Replace
SERVICE_ACCOUNT_NAMEwith a name for the service account. -
Grant roles to the service account. Run the following command once for each of the following IAM roles:
roles/logging.logWriter, roles/monitoring.metricWriter, roles/cloudtrace.agent:gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com" --role=ROLE
Replace the following:
SERVICE_ACCOUNT_NAME: the name of the service accountPROJECT_ID: the project ID where you created the service accountROLE: the role to grant
-
Grant the required role to the principal that will attach the service account to other resources.
gcloud iam service-accounts add-iam-policy-binding SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com --member="user:USER_EMAIL" --role=roles/iam.serviceAccountUser
Replace the following:
SERVICE_ACCOUNT_NAME: the name of the service accountPROJECT_ID: the project ID where you created the service accountUSER_EMAIL: the email address for a Google Account
-
Ensure that you have the Create Service Accounts IAM role
(
- Read the Microservices observability overview.
- Read about the two supported environment variables, decide which to use, and determine the values required by the environment variable.
Create and connect to a Compute Engine VM
Use these instructions to create and connect to a Compute Engine VM instance. On the VM you deploy your application and then instrument the application with Microservices observability.
Create a VM instance:
gcloud compute instances create grpc-observability-vm \ --image-family=debian-11 \ --image-project=debian-cloud \ --service-account=SERVICE_ACCOUNT_NAME@$PROJECT_ID.iam.gserviceaccount.com
Connect to the VM instance:
gcloud compute ssh --project=$PROJECT_ID grpc-observability-vm
Deploy your application to the Compute Engine VM
You can either deploy an application of your choice to the Compute Engine VM that you created in the previous step, then skip this step, or you can use an example to continue with instructions in your preferred language.
C++
After you connect to the VM instance, run the following command.
sudo apt-get update -y sudo apt-get install -y git build-essential clang git clone -b v1.54.0 https://github.com/grpc/grpc.git --depth=1
Go
Make sure that you have Go installed.
sudo apt-get install -y git sudo apt install wget wget https://go.dev/dl/go1.20.2.linux-amd64.tar.gz sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf \ go1.20.2.linux-amd64.tar.gz export PATH=$PATH:/usr/local/go/bin
Clone the
gRPC-Goexamples.git clone https://github.com/grpc/grpc-go.git cd grpc-go/ git checkout -b run-observability-example 875c97a94dca8093bf01ff2fef490fbdd576373d
Java
After you connect to the VM instance, make sure that you have Java 8 or later installed.
sudo apt update sudo apt upgrade sudo apt install git sudo apt-get install -y openjdk-11-jdk-headless
Clone the
grpc-javarepository.export EXAMPLES_VERSION=v1.54.1 git clone -b $EXAMPLES_VERSION --single-branch --depth=1 \ https://github.com/grpc/grpc-java.git
Create gRPC Google Cloud Observability config file
You need separate gRPC Google Cloud Observability config file to enable
Microservices observability for both server and client. The location of
this file is exported as GRPC_GCP_OBSERVABILITY_CONFIG_FILE in the future
steps. Use the following instruction on how to set up the different parameters
in the config file.
Example GRPC_GCP_OBSERVABILITY_CONFIG_FILE
{
"project_id": "your-project-here",
"cloud_logging": {
"client_rpc_events": [
{
"methods": ["google.pubsub.v1.Subscriber/Acknowledge", "google.pubsub.v1.Publisher/CreateTopic"],
"exclude": true,
},
{
"methods": ["google.pubsub.v1.Subscriber/*", "google.pubsub.v1.Publisher/*"],
"max_metadata_bytes": 4096,
"max_message_bytes": 4096,
}],
"server_rpc_events": [{
"methods": ["*"],
"max_metadata_bytes": 4096,
"max_message_bytes": 4096
}],
},
"cloud_monitoring": {},
"cloud_trace": {
"sampling_rate": 0.5,
}
"labels": {
"SOURCE_VERSION": "J2e1Cf",
"SERVICE_NAME": "payment-service-1Cf",
"DATA_CENTER": "us-west1-a"
}
}
The following sections contain instructions for enabling data collection in your
configuration for the individual components. If you used the gRPC example in
this tutorial, you can use this config as is (after updating
your-project-here) or use this as a template for your application.and an
example showing the configuration information in an environment variable.
Enable metrics
To enable metrics, add the cloud_monitoring object to the configuration and
set its value to {}.
For more information about metrics, see Metrics definitions.
Enable tracing
To enable tracing, do the following:
- Add the
cloud_traceobject to the configuration. - Set the
cloud_trace.sampling_rateto0.5which to randomly traces 50% of RPCs.
If you plan to enable tracing across services, ensure that the services support the propagation of trace context received from upstream (or started by itself) to downstream.
For more information about tracing, see Trace definitions.
Enable logging
To enable logging, do the following:
- Add the
cloud_loggingobject to the configuration. - Add a pattern to either or both of
client_rpc_eventsandserver_rpc_eventsspecifying the set of services or methods for which you want to generate transport-level event logging and the number of bytes to log for headers and messages.
For more information about logging, see Log record definitions.
Instrument your applications for the observability plugin
To instrument your applications so that they can use the Microservices observability plugin, use the following instructions for your preferred language.
C++
You can use C++ with Microservices observability as of gRPC C++ v1.54. The example repository is in GitHub.
Observability support is only available through the Bazel build system. Add the target
grpcpp_gcp_observabilityas a dependency.Opting in Microservices observability requires an additional dependency (an observability modul) and the following code changes to existing gRPC clients, servers, or both:
#include <grpcpp/ext/gcp_observability.h> int main(int argc, char** argv) { auto observability = grpc::GcpObservability::Init(); assert(observability.ok()); … // Observability data flushed when object goes out of scope }Before any gRPC operations, including creating a channel, server, or credentials, invoke the following:
grpc::GcpObservability::Init();This returns
absl::StatusOr<GcpObservability>which should be saved. The status helps determine whether observability was successfully initialized. The accompanyingGcpObservabilityobject controls the lifetime of observability, and automatically closes and flushes observability data when it goes out of scope.
Go
- Microservices observability plugins are supported for gRPC Go versions
v1.54.0and later. The example repository is in GitHub.
With the Go module, opting in Microservices observability requires an observability module and the following code:
import "google.golang.org/grpc/gcp/observability"
func main() {
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
if err := observability.Start(ctx); err != nil {
log.Warning("Unable to start gRPC observability:", err)
}
defer observability.End()
…
}
The observability.Start call parses the configuration from environment
variables, creates exporters accordingly, and injects collection logic to client
connections and servers created after the call. The deferredobservability.End
call cleans up resources and ensures that buffered data is flushed before the
application closes.
After the application code is updated, run the following command to update the
go.mod file.
go mod tidy
Java
To use Microservices observability with Java applications, modify your build to
include the grpc-gcp-observability artifact. Use gRPC version 1.54.1 or
later.
In the build snippets in the Gradle and Maven build tool sections,
grpcVersion is set to the value 1.54.1.
The example repository is in GitHub.
- To successfully instrument your Java applications for Microservices
observability, add the following code to
main().
...
import io.grpc.gcp.observability.GcpObservability;
...
// Main application class
...
public static void main(String[] args) {
...
// call GcpObservability.grpcInit() to initialize & get observability
GcpObservability observability = GcpObservability.grpcInit();
...
// call close() on the observability instance to shutdown observability
observability.close();
...
}
Note that you must call GcpObservability.grpcInit() before any gRPC channels
or servers are created. The GcpObservability.grpcInit() function reads the
Microservices observability configuration and uses that to set up the global
interceptors and tracers that are required for the logging, metrics, and trace
features in each channel and server created. GcpObservability.grpcInit()
is thread safe and must be called exactly once. It returns an instance of
GcpObservability that you must save in order to call close() later.
GcpObservability.close() de-allocates resources. Any channel or servers created
afterwards don't perform any logging.
GcpObservability implements java.lang.AutoCloseable, which is closed
automatically if you use try-with-resources as follows:
...
import io.grpc.gcp.observability.GcpObservability;
...
// Main application class
...
public static void main(String[] args) {
...
// call GcpObservability.grpcInit() to initialize & get observability
try (GcpObservability observability = GcpObservability.grpcInit()) {
...
} // observability.close() called implicitly
...
}
Use the Gradle build tool
If you are using the Gradle build tool, include the following:
def grpcVersion = '1.54.1'
...
dependencies {
...
implementation "io.grpc:grpc-gcp-observability:${grpcVersion}"
...
}
Use the Maven build tool (pom.xml)
If you are using the Maven build tool, include the following:
<properties>
...
<grpc.version>1.54.1</grpc.version>
...
</properties>
...
<dependencies>
...
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-gcp-observability</artifactId>
<version>${grpc.version}</version>
</dependency>
...
</dependencies>
Run your application
Follow the instructions in this section only if you used the gRPC example for
the tutorial. You can modify the run command to target your application
binary.
Run Server
C++
- Create a SSH session into the VM.
Export environmental variables. Use steps defined above to create
server_config.json.export GOOGLE_CLOUD_PROJECT=$PROJECT_ID export GRPC_GCP_OBSERVABILITY_CONFIG_FILE="$(pwd)/examples/cpp/gcp_observability/helloworld/server_config.json"Run the server application
shell cd grpc tools/bazel run examples/cpp/gcp_observability/helloworld:greeter_server
Go
- Create a SSH session into the VM.
Export environmental variables. Use steps defined above to create
server_config.json.export GRPC_GCP_OBSERVABILITY_CONFIG_FILE=./server/serverConfig.jsonRun the server application
shell go run ./server/main.go
Java
- In the examples directory, open the README file and follow the instructions in the file.
- When the instructions tell you to open another terminal window, issue
this command:
shell gcloud compute ssh --project=$PROJECT_ID grpc-observability-vm
Run Client
C++
- Create another SSH session into the VM.
Export environmental variables. Use steps defined above to create
client_config.jsonfile.export GOOGLE_CLOUD_PROJECT=$PROJECT_ID export GRPC_GCP_OBSERVABILITY_CONFIG_FILE="$(pwd)/examples/cpp/gcp_observability/helloworld/client_config.json"Run the client application
cd grpc tools/bazel run examples/cpp/gcp_observability/helloworld:greeter_client
Go
- Create another SSH session into the VM.
- Export environmental variables. Use steps defined
above to create
client_config.jsonfile.shell export GRPC_GCP_OBSERVABILITY_CONFIG_FILE=./client/clientConfig.json Run the client application
cd grpc-go/examples/features/observability go run ./client/main.go
Java
- In the examples directory, open the README file and follow the instructions in the file.
- When the instructions tell you to open another terminal window, issue this
command:
shell gcloud compute ssh --project=$PROJECT_ID grpc-observability-vm
Clean up
To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.
Delete the project
Delete a Google Cloud project:
gcloud projects delete PROJECT_ID
Delete individual resources
-
Delete the instance:
gcloud compute instances delete INSTANCE_NAME
What's next
- For detailed information about configuration data, trace definitions, metrics definitions, and log definitions see Microservices observability reference.
- For information about viewing logs, traces, and metrics, see View traces, metrics, and log entries