Module 5 - Lambda Function Permissions

[cjbrew]

Hello!

I've been working on the Data Logging module for a while now, and I've realized that while my code is successfully sending data to the API gateway, the lambda function cannot recognize pymongo. I've been working to solve this issue using Copilot, and it looks like the issue may be that the function does not have the correct permissions to access MongoDB. The instructions say we need to do this, but do not provide any details as to what permissions are necessary. Can you please clarify?

Thanks,
Conor

[cjbrew]

Bumping this! I'd appreciate any help you can provide.

[sgbaird]

Sorry for the delay @cjbrew. @SissiFeng can you help with this?

[SissiFeng]

I am sorry for the problem you are experiencing. Let me explain the current situation and solution in detail:

1. Architecture Change Description

We've recently made an important update to our job architecture: instead of using AWS API Gateway + Lambda, we've switched to using PyMongo to connect directly to MongoDB. this change has the following benefits:

1. simplified architecture: direct connection to the database, no need for a middle tier

2. Reduced complexity: No need to configure AWS Lambda permissions.

3. Improved performance: fewer network hops

4. better development experience: easier to debug and test

5. Specific changes

2.1. Connection string fetching
You need to get the connection string from MongoDB Atlas:

1. Login to MongoDB Atlas
2. Click “Connect” 3.
3. Select “Connect your application”.
4. Copy the connection string, similarly:

mongodb+srv://<username>:<password>@cluster0.xxxxx.mongodb.net/?retryWrites=true&w=majority

2. Environment variable settings
   There are two key environment variables that need to be set:

• MONGODB_URI: MongoDB connection string
• COURSE_ID: your course ID

These variables need to be set in:

1. set in GitHub Secrets (for automated grading)

2. local development environment (for local testing)

3. Code changes
   Your insert.py should now use PyMongo to connect directly:
   ``python
   from pymongo import MongoClient
   from datetime import datetime
   import os
   from my_secrets import MONGODB_URI

Connect to MongoDB

client = MongoClient(MONGODB_URI)

Verify connection

try.
client.admin.command('ping')
print(“Successfully connected to MongoDB!”)
except Exception as e: print(f “Connection failed:”)
print(f “Connection failed: {e}”)
exit(1)

Other code...

If you are still experiencing problems:

1. Check the error messages
2. Verify environment variable settings
3. Verify network connectivity
4. View the MongoDB Atlas logs

Please be sure to complete the suggestions I've given above and perform a full debug. If you still encounter problems, please describe in detail the chokepoints you encountered (preferably with screenshots), so that I can debugging for you.

[cjbrew]

Sissi,

Thank you! I've started trying to implement your suggestions. I've run into some issues running insert.py on the microcontroller. To use pymongo in Thonny, I need to use Local Python 3 - Thonny's Python. However, when I enable this option instead of Micropython, I run into errors because the script uses modules particular to Micropython such as usocket. Do you have suggestions regarding how to proceed?

[SissiFeng]

Since everyone's code environment is set up very differently, it's normal to have parts that we need to fine-tune. It is a part of our learning path.

This is because the microcontroller environment is very different from the normal Python environment:

1. Environmental differences:

   • MicroPython is a Python implementation optimized for microcontrollers.
   • Standard Python libraries (such as pymongo) are not available in MicroPython.
   • Microcontrollers use special networking libraries (e.g., urequests instead of requests).

2. Solution:
   Try using the following insert.py to make it:

• Uses MicroPython-compatible libraries (urequests, ujson)
• maintains the same data structure and experimentation logic

Suggested scenarios:

1. In Thonny:

   • Select “MicroPython (Raspberry Pi Pico)” as the interpreter.
   • Do not use “Local Python 3”.

2. Code modification:

   • Replace pymongo with urequests.
   • Use MongoDB Data API for data insertion
   • Keep WiFi connection management

3. Test flow:

   • First ensure WiFi connection is successful
   • Verify data insertion for each experiment
   • Checking data in MongoDB

Problems you encountered:

1. ImportError: no module named 'pymongo': This is normal because MicroPython does not support `pymongo'.

   • This is normal, because MicroPython does not support pymongo.
   • This is normal because MicroPython does not support pymongo. Use the updated code, which uses urequests instead.

2. Problems with WiFi connectivity

   • Ensure WiFi credentials in my_secrets.py are correct
   • Check network connection status

3. Data Insertion Failure** Verify MongoDB URI and credentials.

   • Verify MongoDB URI and credentials
   • Verify MongoDB URI and credentials
   • Ensure that the network is stable.

Hope above solutions helpful to you.

[sgbaird]

Hi @cjbrew, I've been following this thread. @SissiFeng thanks for giving it a go - I'll handle it from here.

@cjbrew I'll try to get back to you by tomorrow.

[sgbaird]

You need to run pip install requests in the terminal, rather than adding to the python file.

[uznab16]

Thank you. That was my issue.

I've worked on my insert.py and find.py codes, and I want to test them with pytest. The see that the top of insert.py states that I need to run it in my microcontroller. However, when I try to save insert.py onto my microcontroller, I get this backend error:

**Could not write next block after having written 0 bytes to /insert.py
PROBLEM IN THONNY'S BACK-END: Exception while handling 'write_file' (OSError: [WinError Traceback (most recent call last):

File "", line 1, in

File "", line 4, in __W

OSError: 28

] __W(b'# This file needs to be run on your microcontroller\r\nfrom netman import connectWiFi\r\n\r\nfrom my_secrets import (\r\n SSID,\r\n PASSWORD,\r\n DATABASE_NAME,\r\n COLLECTION_NAME,\r\n COURSE_ID,\r\n LAMBDA_FUNCTION_URL,\r\n)\r\n\r\n# TODO: other imports here\r\nfrom pymongo.mongo_client import MongoClient\r\nimport requests\r\nimport json\r\n\r\n# Connect to WiFi\r\nconnectWiFi(SSID, PASSWORD, country="US")\r\n\r\n\r\n# Dummy function for running a color experiment\r\ndef run_color_experiment(R, G, B):\r\n """Simulate color sensor readings"""\r\n wavelengths = [410, 440, 470, 510, 550, 583, 620, 670]\r\n rw = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.9, 1.0]\r\n gw = [0.2, 0.4, 0.6, 0.8, 1.0, 0.8, 0.4, 0.2]\r\n bw = [0.9, 1.0, 0.8, 0.6, 0.4, 0.2, 0.1, 0.0]\r\n sensor_data = {\r\n f"ch{wavelength}": rw[i] * R + gw[i] * G + bw[i] * B\r\n for i, wavelength in enumerate(wavelengths)\r\n }\r\n return sensor_data\r\n\r\n\r\n# NOTE: Even though experiment_id is hard-coded here, pretend that it was\r\n# auto-generated by the "orchestrator"'): '').
See Thonny's backend.log for more info.
You may need to press "Stop/Restart" or hard-reset your MicroPython device and try again.

Process ended with exit code 1.**

Restarting does not solve the problem. Could you shed some light on this. Do I need to delete and reinstall Thonny?

[sgbaird]

Please open a new discussion - I'll respond there

[uznab16]

I'm sorry to keep adding to this thread. I'm running into issues with my lambda function. When I test in AWS, I get the follow message:

Status: Succeeded
Test Event Name: test

Response:
{
"statusCode": 500,
"body": "{"error": "Received error: 'body'. Check the AWS Lambda logs (On the function page, navigate to 'test' then 'CloudWatch Logs Live Tail', select the log group, click start, and then run the request again to monitor the full error)"}"
}

Function Logs:
START RequestId: cf9539e4-2fc8-4b6c-9244-dbfc06287716 Version: $LATEST
{'key1': 'value1', 'key2': 'value2', 'key3': 'value3'}
'body'
File "/var/task/lambda_function.py", line 30, in lambda_handler
payload = json.loads(event["body"])

END RequestId: cf9539e4-2fc8-4b6c-9244-dbfc06287716
REPORT RequestId: cf9539e4-2fc8-4b6c-9244-dbfc06287716 Duration: 3000.00 ms Billed Duration: 3000 ms Memory Size: 128 MB Max Memory Used: 60 MB

When I use mongodb_credentials_test.py in my codespaces, I am able to connect to MongDB fine, but I get the same error for the lambda function that I cannot figure out my way around:

Pinged your deployment. You successfully connected to MongoDB!
Inserted a test document into the horn collection.
Found the test document in the horn collection.
Deleted the test document from the horn collection.
Traceback (most recent call last):
File "/workspaces/5-data-logging-uznab16-1/mongodb_credentials_test.py", line 90, in
test_lambda_function_url()
File "/workspaces/5-data-logging-uznab16-1/mongodb_credentials_test.py", line 83, in test_lambda_function_url
response.status_code == 200
AssertionError: Received status code 502 and message Internal Server Error. Failed to add {'course_id': 'animated-giraffe'} to giraffe:horn via https://ygndu3usyaca365el7iarwchl40bxpjr.lambda-url.us-east-2.on.aws/.

If it helps, I noticed in the thread above that user cjbrew had slightly different error messages for both of these:
(1) they did not have
"File "/var/task/lambda_function.py", line 30, in lambda_handler
payload = json.loads(event["body"])" AND
(2) in their mongodb_credentials_test.py error message, they have something like "UoT_SDL_DB_Main:Tutorial Data" instead of the Function URL like I do

[sgbaird]

Hi @uznab16 , sorry to hear it's still giving you some trouble. Do you mind opening up a separate discussion for this one, too? If you could include the direct link to your assignment repo, too