If you are running an application on a GCP Compute Engine VM instance and want its logs to be collected and streamed into Cloud Logging, then you will have to use an agent that GCP provides called Ops Agent. It’s a single agent that uses Fluent Bit for logging and the OpenTelemetry Collector for metrics collection.

Let’s go through the different steps to achieve this.

Step 1: Save Your Logs

The first step is to make changes in your application to store your logs in a file or chose another way from where they can be made available (Step 3 below) to the Ops Agent. Let’s assume we are saving our logs in a file and proceed with the next steps.

Step 2: Install Ops Agent

You can automate installing the agent across a fleet of VMs but in this article, we will install it on a single machine. Just SSH into your VM instance (gcloud compute ssh instance-name) and then run the following commands:

$ curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh
$ sudo bash add-google-cloud-ops-agent-repo.sh --also-install

These commands will install and start running the agent as a systemd service (systemctl status google-cloud-ops-agent). There are also ways to install the agent from the Cloud Console UI if you prefer that over the command line.

Step 3: Configure the Ops Agent

We must configure the agent to pipe logs from the file (or other sources) where our application stores them. The agent’s config file is available at /etc/google-cloud-ops-agent/config.yaml.

Here is a minimal set of configurations to get started:

logging:
  receivers:
    # Custom name of the receiver specified
    app_logs:
      type: files
      include_paths:
      - /home/user/path/to/logs.txt
  service:
    pipelines:
      # Custom pipeline name specified by us
      default_pipeline:
        # Receiver names from the receivers section above
        receivers: [app_logs]

The configuration is a fairly straightforward YAML file where we describe one or more receivers with their types and source paths. Receivers are where the agent will collect the logs from.

In the end, we define a pipeline name (default_pipeline) which is supposed to create a data flow to Cloud Logging. We can have one or more pipelines as the config gets more complex.

We can specify another section called processors (that we haven’t) to define the processing rules like filtering out (excluding) logs that match a particular pattern.

The documentation on Ops Agent configuration is really detailed and you should check it out to learn more about the different configuration options across receivers, processors and pipelines.

Step 4: Restart the Agent

With the configuration in place, restart the agent for it to take effect:

$ sudo service google-cloud-ops-agent restart
# or systemctl restart google-cloud-ops-agent

Step 5: View the Logs

Finally, view the logs by following these steps:

1. Head over to Logs Explorer in your Google Cloud Console.
2. Click on the Log Name dropdown filter and search for the receiver name you specified in your Ops Agent configuration. In our case it was app_logs (above). Once you find it, select and the logs will start showing below.

Hope that helps!