If you are running an application on a GCP Compute Engine VM instance and want its logs to be collected and streamed into Cloud Logging, then you will have to use an agent that GCP provides called Ops Agent. It’s a single agent that uses Fluent Bit for logging and the OpenTelemetry Collector for metrics collection.
Let’s go through the different steps to achieve this.
Step 1: Save Your Logs
The first step is to make changes in your application to store your logs in a file or chose another way from where they can be made available (Step 3 below) to the Ops Agent. Let’s assume we are saving our logs in a file and proceed with the next steps.
Step 2: Install Ops Agent
You can automate installing the agent across a fleet of VMs but in this article, we will install it on a single machine. Just SSH into your VM instance (
gcloud compute ssh instance-name) and then run the following commands:
$ curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh $ sudo bash add-google-cloud-ops-agent-repo.sh --also-install
These commands will install and start running the agent as a systemd service (
systemctl status google-cloud-ops-agent). There are also ways to install the agent from the Cloud Console UI if you prefer that over the command line.
Step 3: Configure the Ops Agent
We must configure the agent to pipe logs from the file (or other sources) where our application stores them. The agent’s config file is available at
Here is a minimal set of configurations to get started:
logging: receivers: # Custom name of the receiver specified app_logs: type: files include_paths: - /home/user/path/to/logs.txt service: pipelines: # Custom pipeline name specified by us default_pipeline: # Receiver names from the receivers section above receivers: [app_logs]
The configuration is a fairly straightforward YAML file where we describe one or more receivers with their types and source paths. Receivers are where the agent will collect the logs from.
In the end, we define a pipeline name (
default_pipeline) which is supposed to create a data flow to Cloud Logging. We can have one or more pipelines as the config gets more complex.
We can specify another section called
processors (that we haven’t) to define the processing rules like filtering out (excluding) logs that match a particular pattern.
Step 4: Restart the Agent
With the configuration in place, restart the agent for it to take effect:
$ sudo service google-cloud-ops-agent restart # or systemctl restart google-cloud-ops-agent
Step 5: View the Logs
Finally, view the logs by following these steps:
- Head over to Logs Explorer in your Google Cloud Console.
- Click on the
Log Namedropdown filter and search for the receiver name you specified in your Ops Agent configuration. In our case it was
app_logs(above). Once you find it, select and the logs will start showing below.
Hope that helps!