I'm trying to write a google cloud function that automatically downloads data from a certain kaggle data set and upload it to a google cloud bucket. No matter how I try to include my kaggle api key in my google cloud function, the kaggle api can't find it!

I uploaded my kaggle api key to the secret manager and I wrote some code to write the key value into a file stored in /tmp, and set KAGGLE_CONFIG_DIR to /tmp. I granted the cloud function secret manager admin permissions and created added a runtime service account with permission to access the kaggle key as well just to make sure

I reference the secret via a mount path that I set here

First I tried just setting KAGGlE_CONFIG_DIR to the mount path, but that didn't work:

os.environ["KAGGLE_CONFIG_DIR"] = "/.kaggle"

Then I tried to write the key to /tmp, and set KAGGlE_CONFIG_DIR to /tmp and that didn't work either.

def get_kaggle_api_key():
    kaggle_key_path = "/.kaggle/kaggle.json"  

    #read the kaggle key
    with open(kaggle_key_path, 'r') as f:
        kaggle_api_key = f.read().strip()

    #save the key to /tmp/
    kaggle_config_path = "/tmp/kaggle.json" 
    with open(kaggle_config_path, 'w') as f:
        json.dump(json.loads(kaggle_api_key), f)

    os.environ["KAGGLE_CONFIG_DIR"] = "/tmp"

I always get this error:

OSError: Could not find kaggle.json. Make sure it's located in /www-data-home/.config/kaggle. Or use the environment method. See setup instructions at /

The setup instructions don't help, they just say to set KAGGLE_CONFIG_DIR to where the key is stored, which I did.

This is my first time trying to write a cloud function to access an api, and I'm not really sure what I'm doing. Please help!

I'm trying to write a google cloud function that automatically downloads data from a certain kaggle data set and upload it to a google cloud bucket. No matter how I try to include my kaggle api key in my google cloud function, the kaggle api can't find it!

I uploaded my kaggle api key to the secret manager and I wrote some code to write the key value into a file stored in /tmp, and set KAGGLE_CONFIG_DIR to /tmp. I granted the cloud function secret manager admin permissions and created added a runtime service account with permission to access the kaggle key as well just to make sure

I reference the secret via a mount path that I set here

First I tried just setting KAGGlE_CONFIG_DIR to the mount path, but that didn't work:

os.environ["KAGGLE_CONFIG_DIR"] = "/.kaggle"

Then I tried to write the key to /tmp, and set KAGGlE_CONFIG_DIR to /tmp and that didn't work either.

def get_kaggle_api_key():
    kaggle_key_path = "/.kaggle/kaggle.json"  

    #read the kaggle key
    with open(kaggle_key_path, 'r') as f:
        kaggle_api_key = f.read().strip()

    #save the key to /tmp/
    kaggle_config_path = "/tmp/kaggle.json" 
    with open(kaggle_config_path, 'w') as f:
        json.dump(json.loads(kaggle_api_key), f)

    os.environ["KAGGLE_CONFIG_DIR"] = "/tmp"

I always get this error:

OSError: Could not find kaggle.json. Make sure it's located in /www-data-home/.config/kaggle. Or use the environment method. See setup instructions at https://github/Kaggle/kaggle-api/

The setup instructions don't help, they just say to set KAGGLE_CONFIG_DIR to where the key is stored, which I did.

This is my first time trying to write a cloud function to access an api, and I'm not really sure what I'm doing. Please help!

• python
• google-cloud-platform
• google-cloud-functions
• kaggle
• google-secret-manager

Share Improve this question Follow edited Nov 15, 2024 at 22:04 Doug Stevenson 319k3636 gold badges456456 silver badges473473 bronze badges asked Nov 15, 2024 at 21:58 Sean MacInnisSean MacInnis 111 silver badge

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

I'm not a Kaggle user but hopefully this example will help.

There are 3 (possibly 2) steps but importantly you must create a volume and then create a volume mount:

1. Create an environment variable for KAGGLE_CONFIG_DIR
2. Create a Volume (e.g. KAGGLE-CONFIG-DIR) using the Secret
3. Create a Volume Mount (e.g. /.kaggle)

Set an environment variable (KAGGLE_CONFIG_DIR) to /.kaggle:

Create a Volume:

Create a Container Volume Mount:

And some Python:

@functions_framework.http
def root(request):
    # Get the value from the environment
    kaggle_config_dir = os.getenv("KAGGLE_CONFIG_DIR")

    # Open the file
    with open(f"{kaggle_config_dir}/kaggle.json", 'r') as file:
        content = file.read()
        # Return the content
        return f"{kaggle_config_dir}/kaggle.json: {content}:

Yields:

.kaggle/kaggle.json: {"your":"JSON"}