Repository navigation
feat(genai): Add new samples for the Interactions API - #14640
Open
holtskinner wants to merge 4 commits into
Open
holtskinner wants to merge 4 commits into
holtskinner wants to merge 4 commits into
Conversation
product-auto-label
Bot
added
the
samples
|
Here is the summary of changes. You are about to add 49 region tags.
This comment is generated by snippet-bot.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request updates the google-genai dependency to version 2.29.0 and introduces a comprehensive suite of new example scripts and tests demonstrating the interactions API across multiple capabilities, including bounding boxes, code execution, controlled generation, image generation, text generation, thinking, and tools. Feedback on the new code highlights a potential hang in headless environments due to the use of im.show() in the bounding box example, as well as several potential FileNotFoundError crashes across multiple image generation examples where output files are saved to a directory (output_folder) without first ensuring its creation.
| # Position the text at the top-left corner of the box | ||
| draw.text((abs_x_min + 8, abs_y_min + 6), bbox.label, fill=color) | ||
|
|
||
| im.show() |
Contributor
There was a problem hiding this comment.
Using im.show() can cause issues or hang in headless environments (such as CI/CD pipelines) where no display server or default image viewer is available. To make this sample more robust and consistent with other image-processing samples in this repository, consider saving the output image to a file instead of calling im.show().
Suggested change
| im.show() | |
| import os | |
| output_dir = "output_folder" | |
| os.makedirs(output_dir, exist_ok=True) | |
| im.save(os.path.join(output_dir, "socks-bounding-box.png")) |
Comment on lines
+42
to
+44
| if interaction.output_image: | ||
| image = Image.open(BytesIO(base64.b64decode(interaction.output_image.data))) | ||
| image.save("output_folder/bw-example-image.png") |
Contributor
There was a problem hiding this comment.
Saving the image directly to "output_folder/bw-example-image.png" without ensuring that output_folder exists will raise a FileNotFoundError if the directory does not already exist. Ensure the directory is created before saving the image.
Suggested change
| if interaction.output_image: | |
| image = Image.open(BytesIO(base64.b64decode(interaction.output_image.data))) | |
| image.save("output_folder/bw-example-image.png") | |
| if interaction.output_image: | |
| import os | |
| image = Image.open(BytesIO(base64.b64decode(interaction.output_image.data))) | |
| output_dir = "output_folder" | |
| os.makedirs(output_dir, exist_ok=True) | |
| image.save(os.path.join(output_dir, "bw-example-image.png")) |
Comment on lines
+44
to
+46
| if interaction.output_image: | ||
| image = Image.open(BytesIO(base64.b64decode(interaction.output_image.data))) | ||
| image.save("output_folder/video-image.png") |
Contributor
There was a problem hiding this comment.
Saving the image directly to "output_folder/video-image.png" without ensuring that output_folder exists will raise a FileNotFoundError if the directory does not already exist. Ensure the directory is created before saving the image.
Suggested change
| if interaction.output_image: | |
| image = Image.open(BytesIO(base64.b64decode(interaction.output_image.data))) | |
| image.save("output_folder/video-image.png") | |
| if interaction.output_image: | |
| import os | |
| image = Image.open(BytesIO(base64.b64decode(interaction.output_image.data))) | |
| output_dir = "output_folder" | |
| os.makedirs(output_dir, exist_ok=True) | |
| image.save(os.path.join(output_dir, "video-image.png")) |
Comment on lines
+35
to
+37
| if interaction.output_image: | ||
| image = Image.open(BytesIO(base64.b64decode(interaction.output_image.data))) | ||
| image.save("output_folder/example-breakfast-meal.png") |
Contributor
There was a problem hiding this comment.
Saving the image directly to "output_folder/example-breakfast-meal.png" without ensuring that output_folder exists will raise a FileNotFoundError if the directory does not already exist. Ensure the directory is created before saving the image.
Suggested change
| if interaction.output_image: | |
| image = Image.open(BytesIO(base64.b64decode(interaction.output_image.data))) | |
| image.save("output_folder/example-breakfast-meal.png") | |
| if interaction.output_image: | |
| import os | |
| image = Image.open(BytesIO(base64.b64decode(interaction.output_image.data))) | |
| output_dir = "output_folder" | |
| os.makedirs(output_dir, exist_ok=True) | |
| image.save(os.path.join(output_dir, "example-breakfast-meal.png")) |
Comment on lines
+35
to
+55
| saved_files = [] | ||
| image_counter = 1 | ||
|
|
||
| for step in interaction.steps: | ||
| if step.type == "model_output": | ||
| for content in step.content: | ||
| if content.type == "text": | ||
| print(content.text) | ||
| elif content.type == "image": | ||
| image = Image.open(BytesIO(base64.b64decode(content.data))) | ||
| filename = f"output_folder/example-cats-0{image_counter}.png" | ||
| image.save(filename) | ||
| saved_files.append(filename) | ||
| image_counter += 1 | ||
|
|
||
| # Fallback to output_image if no step-level images collected | ||
| if not saved_files and interaction.output_image: | ||
| image = Image.open(BytesIO(base64.b64decode(interaction.output_image.data))) | ||
| filename = "output_folder/example-cats-01.png" | ||
| image.save(filename) | ||
| saved_files.append(filename) |
Contributor
There was a problem hiding this comment.
Saving images directly to "output_folder/" without ensuring that the directory exists will raise a FileNotFoundError if the directory does not already exist. Ensure the directory is created before saving the images.
import os
output_dir = "output_folder"
os.makedirs(output_dir, exist_ok=True)
saved_files = []
image_counter = 1
for step in interaction.steps:
if step.type == "model_output":
for content in step.content:
if content.type == "text":
print(content.text)
elif content.type == "image":
image = Image.open(BytesIO(base64.b64decode(content.data)))
filename = os.path.join(output_dir, f"example-cats-0{image_counter}.png")
image.save(filename)
saved_files.append(filename)
image_counter += 1
# Fallback to output_image if no step-level images collected
if not saved_files and interaction.output_image:
image = Image.open(BytesIO(base64.b64decode(interaction.output_image.data)))
filename = os.path.join(output_dir, "example-cats-01.png")
image.save(filename)
saved_files.append(filename)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #
Checklist
Testing
Compliance & Style
Post-Approval Actions