Skip to content

Deploying agent engine with Secret environment variables from the Secret Manager #5647

Closed
Closed
Deploying agent engine with Secret environment variables from the Secret Manager#5647
Assignees
changsongd
Labels
api: vertex-aiIssues related to the googleapis/python-aiplatform API.

Description

@lposilovic
lposilovic
opened on Jul 25, 2025

I'm trying to deploy the agent using Vertex AI Agent Engine using secrets as env_vars through python. The agent is deployed succesfully if I'm not using the secrets in the environment variables, but fails when using secrets. Deployment actually passes, but the agent won't start.

The agent is being deployed using github actions and a IAM service account. Service account used has Secret Manager Secret Accessor permission. I also set up the same permission in the secret configuration in the secret manager for this service account. The secrets and the agent are being deployed in the same gcp project. Error and logs are not very much useful. I guess there is some kind of access needed that I didn't yet set up?

Environment details

  • OS type and version: Ubuntu 24.04.02 (github action runner)
  • Python version: 3.12.3
  • google-cloud-aiplatform version: 1.104.0

Steps to reproduce

  1. Set up secrets in the secrets manager, give the service account permission (Secret Manager Secret Accessor) for this secret
  2. Deploy the agent with code (as per documentation):
    remote_app = agent_engines.create(
        app,
        requirements=[agent_whl_file],
        extra_packages=[agent_whl_file],
        env_vars={
            "AWS_REGION": aws_region,
            "AWS_ACCESS_KEY_ID": {"secret": "AWS_ACCESS_KEY_ID", "version": "latest"},
            "AWS_SECRET_ACCESS_KEY": {
                "secret": "AWS_SECRET_ACCESS_KEY",
                "version": "latest",
            },
        },
    )

Stack trace

INFO:__main__:Initializing Vertex AI with the following configurations:
INFO:__main__:Project: ***
INFO:__main__:Location: ***
INFO:__main__:Staging Bucket: ***
ERROR:__main__:Agent 2544419440255041536 not found
INFO:__main__:Deploying app...
DEBUG:__main__:Deploying agent to agent engine:
INFO:vertexai.agent_engines:Identified the following requirements: ***'cloudpickle': '3.1.1', 'google-cloud-aiplatform': '1.104.0', 'pydantic': '2.11.7'***
Identified the following requirements: ***'cloudpickle': '3.1.1', 'google-cloud-aiplatform': '1.104.0', 'pydantic': '2.11.7'***
WARNING:vertexai.agent_engines:The following requirements are missing: ***'cloudpickle', 'google-cloud-aiplatform', 'pydantic'***
INFO:vertexai.agent_engines:The following requirements are appended: ***'cloudpickle==3.1.1', 'pydantic==2.11.7'***
INFO:vertexai.agent_engines:The final list of requirements: ['kitro_cs-0.1.0-py3-none-any.whl', 'cloudpickle==3.1.1', 'pydantic==2.11.7']
The following requirements are missing: ***'cloudpickle', 'google-cloud-aiplatform', 'pydantic'***
The following requirements are appended: ***'cloudpickle==3.1.1', 'pydantic==2.11.7'***
The final list of requirements: ['kitro_cs-0.1.0-py3-none-any.whl', 'cloudpickle==3.1.1', 'pydantic==2.11.7']
DEBUG:urllib3.util.retry:Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:urllib3.connectionpool:[https://oauth2.googleapis.com:443](https://oauth2.googleapis.com/) "POST /token HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): storage.googleapis.com:443
DEBUG:urllib3.connectionpool:[https://storage.googleapis.com:443](https://storage.googleapis.com/) "GET /storage/v1/b/***?projection=noAcl&prettyPrint=false HTTP/1.1" 200 757
INFO:vertexai.agent_engines:Using bucket ***
Using bucket ***
DEBUG:urllib3.connectionpool:[https://storage.googleapis.com:443](https://storage.googleapis.com/) "POST /upload/storage/v1/b/***/o?uploadType=resumable HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:[https://storage.googleapis.com:443](https://storage.googleapis.com/) "PUT /upload/storage/v1/b/***/o?uploadType=resumable&upload_id=ABgVH8_mG8tK5rSL9iQbaTBxxbtXznU8eQPV2Mgu2APU5lnTvWECe2RVzx58N7XSTFnKr98jChcJ0oHZ3MCILs5sIBRU25bw2_2AeF4zhgdv2yc HTTP/1.1" 200 869
DEBUG:urllib3.connectionpool:[https://storage.googleapis.com:443](https://storage.googleapis.com/) "GET /download/storage/v1/b/***/o/agent_engine%2Fagent_engine.pkl?alt=media HTTP/1.1" 206 13037
INFO:vertexai.agent_engines:Wrote to ***/agent_engine/agent_engine.pkl
Wrote to ***/agent_engine/agent_engine.pkl
DEBUG:urllib3.connectionpool:[https://storage.googleapis.com:443](https://storage.googleapis.com/) "POST /upload/storage/v1/b/***/o?uploadType=multipart HTTP/1.1" 200 852
Writing to ***/agent_engine/requirements.txt
Creating in-memory tarfile of extra_packages
INFO:vertexai.agent_engines:Writing to ***/agent_engine/requirements.txt
INFO:vertexai.agent_engines:Creating in-memory tarfile of extra_packages
DEBUG:urllib3.connectionpool:[https://storage.googleapis.com:443](https://storage.googleapis.com/) "POST /upload/storage/v1/b/***/o?uploadType=multipart HTTP/1.1" 200 867
Writing to ***/agent_engine/dependencies.tar.gz
INFO:vertexai.agent_engines:Writing to ***/agent_engine/dependencies.tar.gz
INFO:vertexai.agent_engines:Creating AgentEngine
Creating AgentEngine
INFO:vertexai.agent_engines:Create AgentEngine backing LRO: projects/***/locations/***/reasoningEngines/388532224883949568/operations/8115121181423566848
INFO:vertexai.agent_engines:View progress and logs at https://console.cloud.google.com/logs/query?project=***
Create AgentEngine backing LRO: projects/***/locations/***/reasoningEngines/388532224883949568/operations/8115121181423566848
View progress and logs at https://console.cloud.google.com/logs/query?project=***
Traceback (most recent call last):
  File "/home/runner/work/***/***/agents/***/deployment/deploy.py", line 127, in <module>
    resource_name = deploy_agent(
                    ^^^^^^^^^^^^^
  File "/home/runner/work/***/***/agents/***/deployment/deploy.py", line 77, in deploy_agent
    remote_app = agent_engines.create(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/***/***/agents/***/.venv/lib/python3.12/site-packages/vertexai/agent_engines/__init__.py", line 161, in create
    return AgentEngine.create(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/***/***/agents/***/.venv/lib/python3.12/site-packages/vertexai/agent_engines/_agent_engines.py", line 523, in create
    created_resource = operation_future.result()
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/***/***/agents/***/.venv/lib/python3.12/site-packages/google/api_core/future/polling.py", line 261, in result
    raise self._exception
google.api_core.exceptions.InvalidArgument: 400 Reasoning Engine resource [projects/***/locations/***/reasoningEngines/388532224883949568] failed to start and cannot serve traffic. Please refer to our documentation (https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/troubleshooting/deploy) for checking logs and other troubleshooting tips. 3: Reasoning Engine resource [projects/***/locations/***/reasoningEngines/388532224883949568] failed to start and cannot serve traffic. Please refer to our documentation (https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/troubleshooting/deploy) for checking logs and other troubleshooting tips.

In the Logs Explorer though no error is reported, this is the last log:

INFO 2025-07-25T08:18:00.840894547Z DONE
NOTICE 2025-07-25T08:18:33.616264Z [protoPayload.serviceName: aiplatform.googleapis.com] [protoPayload.methodName: google.cloud.aiplatform.v1.ReasoningEngineService.CreateReasoningEngine] [protoPayload.resourceName: projects/***/locations/us-central1] [protoPayload.authenticationInfo.principalEmail: ***@***.iam.gserviceaccount.com] audit_log, method: "google.cloud.aiplatform.v1.ReasoningEngineService.CreateReasoningEngine", principal_email: "***@***.iam.gserviceaccount.com"
  {
    "protoPayload": {
      "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
      "status": {},
      "authenticationInfo": {
        "principalEmail": "***@***.iam.gserviceaccount.com",
        "principalSubject": "serviceAccount:***@***.iam.gserviceaccount.com"
      },
      "requestMetadata": {
        "callerIp": "64.236.177.0",
        "callerSuppliedUserAgent": "grpc-python/1.73.1 grpc-c/48.0.0 (linux; chttp2),gzip(gfe)",
        "requestAttributes": {
          "time": "2025-07-25T08:14:52.497012701Z",
          "auth": {}
        },
        "destinationAttributes": {}
      },
      "serviceName": "aiplatform.googleapis.com",
      "methodName": "google.cloud.aiplatform.v1.ReasoningEngineService.CreateReasoningEngine",
      "authorizationInfo": [
        {
          "resource": "projects/***/locations/***",
          "permission": "aiplatform.reasoningEngines.create",
          "granted": true,
          "resourceAttributes": {},
          "permissionType": "ADMIN_WRITE"
        }
      ],
      "resourceName": "projects/***/locations/***"
    },
    "insertId": "pv6lnld16me",
    "resource": {
      "type": "audited_resource",
      "labels": {
        "project_id": "***",
        "method": "google.cloud.aiplatform.v1.ReasoningEngineService.CreateReasoningEngine",
        "service": "aiplatform.googleapis.com"
      }
    },
    "timestamp": "2025-07-25T08:18:33.616264Z",
    "severity": "NOTICE",
    "logName": "projects/***/logs/cloudaudit.googleapis.com%2Factivity",
    "operation": {
      "id": "projects/***/locations/***/reasoningEngines/388532224883949568/operations/8115121181423566848",
      "producer": "aiplatform.googleapis.com",
      "last": true
    },
    "receiveTimestamp": "2025-07-25T08:18:33.767350476Z"
  }

Thank you for your assistance!

Activity

  1. product-auto-label
    added
    api: vertex-aiIssues related to the googleapis/python-aiplatform API.
    on Jul 25, 2025
  2. yeesian
    assigned
    yeesian
    and
    changsongd
    and unassigned
    yeesian
    on Jul 29, 2025

  3. abetekgeo commented on Sep 3, 2025

    @abetekgeo
    abetekgeo
    on Sep 3, 2025

    I am having a similar issue. Are there any updates on this?

  4. changsongd commented on Oct 6, 2025

    @changsongd
    changsongd
    on Oct 6, 2025

    The agent is being deployed using github actions and a IAM service account. Service account used has Secret Manager Secret Accessor permission.

    The deployed agent uses Vertex AI Reasoning Engine Service Agent as the identity by default. Can you confirm if the permission is granted to the Reasoning Engine SA? Follow the instructions in Set up the identity and permissions for your agent to grant your agent the Secret Manager Secret Accessor (roles/secretmanager.secretAccessor) permission.

    You can also create your custom service account and attach it to the agent.

    More details about using secret can be found here: https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/deploy#environment-variables.

  5. lposilovic commented on Oct 7, 2025

    @lposilovic
    lposilovic
    on Oct 7, 2025
    Author

    Hi @changsongd! Thank you for the response!

    I think the documentation is wrong though, since the agent_engine.create doesn't have a config parameter (or I'm looking at something wrong in the code).

    Current documentation:
    Image

    Current code on the main:
    Image

    The solution was actually implemented the day after my issue in this commit. We should deploy the agent with a custom service account this way (and I checked this works now):

        remote_app = agent_engines.create(
        app,
        requirements=[agent_whl_file],
        extra_packages=[agent_whl_file],
        service_account="your_custom_service_account",
        env_vars={
            "AWS_REGION": aws_region,
            "AWS_ACCESS_KEY_ID": {"secret": "AWS_ACCESS_KEY_ID", "version": "latest"},
            "AWS_SECRET_ACCESS_KEY": {
                "secret": "AWS_SECRET_ACCESS_KEY",
                "version": "latest",
            },
        },
    )

    Thank you for pointing me to the service account! I'll close the issue since this solved my problem 😁

  6. lposilovic
    closed this as completedon Oct 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: vertex-aiIssues related to the googleapis/python-aiplatform API.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions