Skip to content

Update description note #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 15, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
Generative AI Vision with MQTT
==============================

.. contents::
:local:
:depth: 2

Materials
---------

- `AMB82-mini <https://www.amebaiot.com/en/where-to-buy-link/#buy_amb82_mini>`_ x 1

Example
-------

In this example, we will be connecting AMB82-mini to a MQTT-Broker that could display images, i.e. broker.mqttgo.io.

Open Generative AI Vision with MQTT example in “File” -> “Examples” -> “AmebaMQTTClient” -> “MQTT_GenAIVision”.

|image01|

Please modify "ssid" and "pass" according to your WiFi network SSID and password.

|image02|

Create your API key and paste it in the following section. Please refer to `GenAIVision Example <https://ameba-arduino-doc.readthedocs.io/en/latest/amebapro2/Example_Guides/Neural%20Network/Generative%20AI%20Vision.html>`_ for more details on creating API.

|image03|

Uncomment the vision prompt function according to your model.

|image04|

Compile and upload the firmware to AMB82-mini.

Go to `MQTTGO.io <https://broker.mqttgo.io/>`_, and connect to server.

Subscribe to the topics to receive the **message, image and response** published by AMB82-mini. Publish your prompt to a topic where AMB82-mini has subscribed to.

.. note :: Only message that starts with ``p/`` will be recognized as a prompt message, please insert "p/" before your prompt to trigger image capture and sending to Gen AI server.

You may refer to the snapshot below for prompt sending and response receiving over MQTT broker. **Kindly define your own topics to publish and subscribe.**

|image05|

Resources
---------

| MQTT GO Broker
| https://broker.mqttgo.io/

| openAI platform - openAI vision
| https://platform.openai.com/docs/guides/vision

| Google AI Studio - Gemini vision
| https://ai.google.dev/gemini-api/docs/vision

| GroqCloud - Llama vision
| https://console.groq.com/docs/overview

.. |image01| image:: ../../../_static/amebapro2/Example_Guides/MQTT/MQTT_GenAIVision/image01.png
:width: 658 px
:height: 681 px

.. |image02| image:: ../../../_static/amebapro2/Example_Guides/MQTT/MQTT_GenAIVision/image02.png
:width: 794 px
:height: 517 px

.. |image03| image:: ../../../_static/amebapro2/Example_Guides/MQTT/MQTT_GenAIVision/image03.png
:width: 911 px
:height: 508 px

.. |image04| image:: ../../../_static/amebapro2/Example_Guides/MQTT/MQTT_GenAIVision/image04.png
:width: 1246 px
:height: 782 px
:scale: 80%

.. |image05| image:: ../../../_static/amebapro2/Example_Guides/MQTT/MQTT_GenAIVision/image05.png
:width: 985 px
:height: 886 px
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ In the operation of MQTT, there are several roles:
topic of a message can be “PM2.5” or “Temperature”. Subscribers can
choose messages of which topics they want to receive.

In this page, there are 3 examples that connect Ameba to MQTT-Broker.
In this page, there are 4 examples that connect Ameba to MQTT-Broker.
Then send messages as publisher and receive messages from MQTT-Broker as
subscriber.

Expand All @@ -43,6 +43,8 @@ subscriber.

3. MQTT_Publish_In_Callback

4. MQTT_Image

**MQTT_Basic example**

| Open the MQTT example “File” -> “Examples” -> “AmebaMQTTClient” -> “MQTT_Basic”
Expand Down Expand Up @@ -164,6 +166,44 @@ boards connect to MQTT broker before the MQTT Explorer.

|image13|

**MQTT_Image example**

Open the MQTT example “File” -> “Examples” -> “AmebaMQTTClient” ->
“MQTT_Image”

Please modify WiFi SSID and password according to your network.

- “ssid” is the network SSID for internet access.

- “pass” is the network password for internet access.

- “mqttServer” refers to the MQTT-Broker, we will be using "broker.mqttgo.io" for this example.

- “clientId” is an identifier for MQTT-Broker to identify the connected
device.

- “publishTopic” is the topic of the published message in the example
it is “outTopic/Msg”. The devices that subscribed to “outTopic” will
receive the message.

- “publishPayload” is the content to be published.

- “subscribeTopic” is to tell MQTT-broker which topic to subscribe to
by the board.

- Image taken will be publish to "outTopic/Img".

Go to `MQTTGO.io <https://broker.mqttgo.io/>`_, and connect to server.

Subscribe to "outTopic/Msg" and "outTopic/Img", while the publish topic is set to "inTopic/Msg".

|image14|

Next, compile and upload the example code to Ameba. After Ameba is connected to MQTT server,
it publishes the message “hello world” to “outTopic/Msg” and an image to "outTopic/Img".

.. note :: You may redefine the topics for your own message and image publishing

.. |image01| image:: ../../../_static/amebapro2/Example_Guides/MQTT/Set_up_Client/image01.png
:width: 940 px
:height: 617 px
Expand Down Expand Up @@ -214,4 +254,8 @@ boards connect to MQTT broker before the MQTT Explorer.

.. |image13| image:: ../../../_static/amebapro2/Example_Guides/MQTT/Set_up_Client/image13.png
:width: 548 px
:height: 151 px
:height: 151 px

.. |image14| image:: ../../../_static/amebapro2/Example_Guides/MQTT/Set_up_Client/image14.png
:width: 997 px
:height: 849 px
1 change: 1 addition & 0 deletions source/amebapro2/Example_Guides/MQTT/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@ MQTT

Set up MQTT Client over TLS
Set up MQTT Client to Communicate with Broker
Generative AI Vision with MQTT
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Generative AI Vision
:depth: 2

.. note ::
|image_3rd_party| "Generative AI Vision" is a 3rd-party example source provided and maintained by `ChungYi Fu (Kaohsiung, Taiwan) <https://github.com/fustyles>`_
|image_3rd_party| "Generative AI Vision" is jointly developed by RTKSG SD3 and `ChungYi Fu (Kaohsiung, Taiwan) <https://github.com/fustyles>`_

|image_ameba_iot| Special thanks and credits to the efforts and contributions for all developers.

Expand All @@ -28,7 +28,7 @@ In the highlighted code snippet, fill in the "wifi_ssid" with your WiFi network

|image02|

In this documentation, we will be using Gemini API for demeonstration.
In this documentation, we will be using Gemini API for demonstration.

Create your own Gemini API key in `Google AI Studio <https://aistudio.google.com/apikey>`_.

Expand Down
Loading