Replies: 1 comment
-
So finally figured out that the JOB_NAME needs to reference an existing glue job. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I am using the code fragment below and the job.init call is failing with an error while getting security configuration. (see log below). I'm not suprised by the error since I can't find documentation as to the args that need to be supplied. I did find https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html but the essential parameters are missing: (IAM_ROLE, connections, etc.) . When I look at the AWS Glue Studio job editor there is a lot of Job context established in the "Job Details" tab. I expect that I need to pass similar information to job.init.
I'm trying to do local debugging of glue jobs as suggested by https://aws.amazon.com/blogs/big-data/developing-aws-glue-etl-jobs-locally-using-a-container/
Any help would be appreciated.
Code:
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
LOG:
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Traceback (most recent call last):
File "/home/glue_user/spark/python/pyspark/sql/utils.py", line 111, in deco
return f(*a, **kw)
File "/home/glue_user/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:com.amazonaws.services.glue.util.Job.init.
: java.lang.RuntimeException: error while getting security configuration for None
at com.amazonaws.services.glue.util.SecurityConfig$.get(SecurityConfig.scala:29)
at com.amazonaws.services.glue.util.Job$.init(Job.scala:67)
at com.amazonaws.services.glue.util.Job.init(Job.scala)
Beta Was this translation helpful? Give feedback.
All reactions