Repository navigation
[Storage] Easily generate signed URLs when using workload identity #13307
Description
Hi, I am open to writing a PR to make code changes for this feature but wanted to start with an issue to get feedback.
When developing locally I can generate signed URL simply with:
require "google/cloud/storage"
storage = Google::Cloud::Storage.new(
project_id: "<project_id>",
credentials: ENV["GCP_CREDENTIALS_PATH"]
)
bucket = storage.bucket "<bucket_name>"
remote_file = bucket.file "foo/bar", skip_lookup: true
url = remote_file.signed_url method: "PUT", content_type: "application/json"
However, when running my code from within an eks pod with access grant via workload identitiy (https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity)
I get the error:
/usr/local/bundle/gems/google-cloud-storage-1.31.1/lib/google/cloud/storage/file/signer_v2.rb:88:in `determine_issuer': Service account credentials 'issuer (client_email)' is missing. To generate service account credentials see https://cloud.google.com/iam/docs/service-accounts (Google::Cloud::Storage::SignedUrlUnavailable)
Based on https://github.com/salrashid123/gcpsamples/blob/master/gcs_keyless_signedurl/main.py I was able to write some equivalent ruby code that works for generating a working signed URL (not the most well-factored code, just a proof-of-concept)
require "google/cloud/storage"
storage = Google::Cloud::Storage.new
bucket = storage.bucket "<bucket_name>"
remote_file = bucket.file "foo/bar", skip_lookup: true
path = "/#{bucket.name}/#{remote_file.name}"
url_manually = "https://storage.googleapis.com#{path}"
access_token = JSON.parse(`curl -s -H 'Metadata-Flavor: Google' http://metadata/computeMetadata/v1/instance/service-accounts/default/token`)["access_token"]
connection = Faraday.new(url: "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/#{service_account_email}:signBlob") do |conn|
conn.request :json
end
expiration = 1.day.from_now.to_i
verb = "PUT"
content_md5 = ""
content_type = "application/json"
signature_string = "#{verb}\n#{content_md5}\n#{content_type}\n#{expiration}\n#{path}"
response = connection.post do |req|
req.headers["Authorization"] = "Bearer #{access_token}"
req.body = {delegates: [], payload: Base64.encode64(signature_string).gsub("\n","")}
end
signed_blob = JSON.parse(response.body)["signedBlob"]
signed_url = url_manually + "?" + URI.encode_www_form("GoogleAccessId": service_account_email, "Expires": expiration, "Signature": signed_blob)
So... I was thinking perhaps we could/should just change the implementation of signed_url to detect the conditions under which key-based signing will fail and automatically attempt this meta data service token + signBlob service method.
Or perhaps it's already exposed somewhere else within the depths of this ruby gem? (in which case in the very least the error message you see could be updated to point that out)
Thanks,
Jacob
Update based on https://googleapis.dev/ruby/google-cloud-storage/latest/Google/Cloud/Storage/File.html#signed_url-instance_method I have a better version
require "google/cloud/storage"
storage = Google::Cloud::Storage.new
bucket = storage.bucket "<bucket_name>"
remote_file = bucket.file "foo/bar", skip_lookup: true
issuer = "<service account email>"
signer = lambda do |string_to_sign|
IAMCredentials = Google::Apis::IamcredentialsV1
iam_client = IAMCredentials::IAMCredentialsService.new
scopes = ["https://www.googleapis.com/auth/iam"]
iam_client.authorization = Google::Auth.get_application_default scopes
request = Google::Apis::IamcredentialsV1::SignBlobRequest.new(
payload: string_to_sign
)
resource = "projects/-/serviceAccounts/#{issuer}"
response = iam_client.sign_service_account_blob resource, request
response.signed_blob
end
signed_url = remote_file.signed_url method: "PUT", content_type: "application/json", issuer: issuer, signer: signer
And the only things that are "custom" to my use case in this example end up being arguments into the signed_url method. so perhaps a simplification this gem could make would be to auto-generate the signer proc
Finally, I need a way to tell which version of the signed_url method args to use.
When running locally I have a credentials JSON file. so I need only call remote_file.signed_url method: "PUT", content_type: "application/json"
Whereas from my pod I have to call remote_file.signed_url method: "PUT", content_type: "application/json", issuer: issuer, signer: signer
If I do it backwards (specify an issue and signer when not on a pod) I get
/Users/jacob/.gem/ruby/3.0.1/gems/googleauth-0.16.2/lib/googleauth/application_default.rb:76:in `get_application_default': Could not load the default credentials. Browse to (RuntimeError)
https://developers.google.com/accounts/docs/application-default-credentials
for more information
Is there some simple call I could make to this gem to determine what context I'm in?
Activity
jacobo commentedon Aug 6, 2021
looks like there is a related open feature request at googleapis/google-auth-library-ruby#270
quartzmo commentedon Oct 6, 2021
@jacobo Have you tried using ruby-cloud-env? (It's already a dependency of
google-cloud-storageviagoogle-cloud-core.)For example:
quartzmo commentedon Oct 11, 2021
Background: The
signerparameter and examples showingsign_service_account_blobusage were added in PR #7091.frankyn commentedon Jan 16, 2024
Current state of FR(1/16/23): In #7091, a new signer parameter was introduced to allow passing in a method to create signature from a payload to support signed URLs. However, it still requires boiler plate code to check the environment a client is in (GCE or local dev env) to determine how to call
#signed_urlwhen generating a Signed URL.To address this issue:
Google::Cloud.env#lookup_metadataIAMCredentials#sign_service_account_blobwhen in GCE environmentGoogle::Cloud.env#compute_engine?These steps should reduce overhead of boiler plate code using signer parameter in
signed_urland at the same existing signer / issuer support provides an escape hatch for more complex situations that can't be auto-detected.Workaround for now:
References: