chris.mclennon

1

votes
1

answer
728

views

How do I select an ambiguous column reference? [duplicate]

This question already has an answer here: Enable case sensitivity for spark.sql globally 1 answer Here's some sample code illustrating what I'm trying to do. There is a dataframe with columns companyid and companyId. I want to select companyId, but the reference is ambiguous. How do I unambiguously...
chris.mclennon
1

votes
0

answer
251

views

How to import referenced files in ETL scripts?

I have a script which I'd like to pass a configuration file into. On the Glue jobs page, I see that there is a 'Referenced files path' which points to my configuration file. How do I then use that file within my ETL script? I've tried from configuration import *, where the referenced file name is co...
chris.mclennon
1

votes
2

answer
46

views

How can I run a function over a rolling window on each collective key in Spark?

I am handling some event data, and I'd like to run a function that loops down an ordered list of values for a given key and return a value. I'm not sure how to do this, or if it's possible using Spark, and am hoping for a tip in the right direction. Specifically, I have some event data, and I'd like...
chris.mclennon