Logging

Terraform Cloud Agents log helpful messages that tell operators about agent behavior, including communication with Terraform Cloud APIs, specific commands run, actions taken, and runtime management.

Log output

Terraform Cloud Agents write log messages directly to stdout/stderr. This lets the operator capture the logs in a variety of different logging systems, gives CLI users a way to see a log of the agent's behavior directly in their terminal. By default, the agent does not automatically persist the log output in any way. The user must write the logs to a file or collect them with a process or container supervisor if persistence is required.

Log verbosity and levels

The volume of logs and the level of detail they contain are controlled by log levels. There are 5 levels supported by the agent:

• error contains only critical error messages. The agent process is otherwise silent.
• warn contains all error-level messages, as well as informational messages such as system messages directly from the Terraform Cloud platform.
• info contains all warn-level messages and high-level information about the agent and the workflows it is executing. In normal circumstances, this is the safest and most helpful log level for day to day operation.
• debug contains all info-level messages, plus additional informational messages which provide further context about behavior, data, and events.
• trace contains all debug-level messages, plus verbose process logs such as the line-by-line output of the terraform command.

Data format

By default, Terraform Cloud Agents emit log lines in a human-friendly text format. This is convenient for running the TFC Agent locally and streaming the logs directly to a terminal, or for use in log systems where raw logs are consumed directly by operators. The default text format looks something like the following:

2023-04-14T17:12:43.002Z [INFO]  core: Job received: job_type=plan job_id=run-xxx

It is also possible to configure the Terraform Cloud Agent to produce JSON- formatted logs. This format will cause each log line to be serialized as an individual JSON object, and is more ideal for logging systems which are capable of parsing and performing post-processing on log data for each line. JSON logging mode is enabled by passing the -log-json CLI flag, or setting the environment variable TFC_AGENT_LOG_JSON=1. The JSON format contains additional verbose information in each log message, and looks something like this:

{"@level":"info","@message":"Job received","@module":"core","@timestamp":"2023-04-13T23:35:28.653853Z","agent_id":"agent-xxx","agent_name":"name","agent_pool_id":"apool-xxx","agent_version":"1.7.1","job_id":"run-xxx","job_type":"plan"}

Log data sensitivity

Log levels have a progressive level of data sensitivy. The info, warn, and error levels are considered generally safe for log collection and don't include sensitive information. The debug log level may include internal system details, such as specific commands and arguments including paths to user data on the local filesystem. The trace log level is the most sensitive and may include personally identifiable information, secrets, pre-authorized internal URLs, and other sensitive material.

Flash Messages

Flash Messages are a type of log that HashiCorp may send to agents from time to time. These messages may be used to communicate important or breaking changes to the agent. Flash Messages will be emitted when HashiCorp adds a new one, or when starting up an agent for the first time. Their output looks something like:

2021-09-22T15:20:59.269Z [WARN]  notice: A breaking change is incoming.

Flash Messages are version specific, and may only apply to the specific version of the agent you are running.

Adding monitoring and alerting for these notice messages may help you operate Terraform Cloud Agents more easily.