Generate a csv file and dump it into a sftp folder directory

[aynkodexa]

Hi all, just wanted to get an idea of how to implement this

[wesleyscholl]

@aynkodexa

I created a basic functional sftp-csv-upload example WorkflowTemplate. The workflow generates a .csv artifact and uploads it to the SFTP server. Keep in mind that a Secret is required with the SFTP connection host, username and password.

Example WorkflowTemplate:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: sftp-csv-upload
  namespace: argo
spec:
  templates:
    - name: generate-and-upload
      steps:
        - - name: generate-csv
            template: generate-csv
        - - name: upload-csv
            template: sftp-upload
            arguments:
              parameters:
                - name: uploadDir # Upload directory on the SFTP server
                  value: /upload
                - name: outputFile # Output file name for the uploaded CSV
                  value: outputFile.csv
              artifacts:
                - name: csv-file
                  from: "{{steps.generate-csv.outputs.artifacts.csv-file}}" # Pass artifact from previous step
    - name: generate-csv
      outputs:
        artifacts:
          - name: csv-file
            path: /tmp/outputFile.csv # Path to save the generated CSV
      container:
        name: ""
        image: python:3.9
        command:
          - python
        args:
          - "-c"
          - |
            import csv 
            # Data to be written to the CSV file
            data = [
              ["Name", "Age", "Location"],
              ["John Doe", 30, "New York"],
              ["Jane Smith", 25, "Los Angeles"]
            ]
            # Write the data to the CSV file
            with open("/tmp/outputFile.csv", mode='w', newline='') as file:
                writer = csv.writer(file)
                writer.writerows(data)
        volumeMounts:
          - name: temp-volume # Mount volume to persist data across steps
            mountPath: /tmp # Mount to '/tmp' directory inside the container
    - name: sftp-upload
      inputs:
        parameters:
          - name: uploadDir # Directory where the file will be uploaded on the SFTP server
          - name: outputFile # The name of the output file
          - name: secretName # Secret name containing the SFTP credentials (default: 'sftp-conn')
            default: sftp-conn
        artifacts:
          - name: csv-file # Reference to the CSV file generated by the 'generate-csv' step
            path: /tmp/outputFile.csv # Path to the CSV file
      container:
        name: ""
        image: alpine:3.12
        command:
          - /bin/sh
          - "-c"
        args:
          - >
            # Install required packages (lftp and openssh) to enable SFTP
            apk add --no-cache lftp openssh && \
            # Using lftp to upload the file to the SFTP server, auto-confirm to accept the host key
            lftp -d -u "$SFTP_USER,$SFTP_PASS" -e 'set net:max-retries 3; set net:timeout 60; set sftp:auto-confirm yes; put /tmp/outputFile.csv
            -o $UPLOAD_DIR/$OUTPUT_FILE; chmod 644 $UPLOAD_DIR/$OUTPUT_FILE; bye' "sftp://$SFTP_HOST"
        env:
          - name: SFTP_USER # SFTP username
            valueFrom:
              secretKeyRef:
                name: "{{inputs.parameters.secretName}}" # Secret name to retrieve credentials
                key: username # Secret key for the username
          - name: SFTP_PASS # SFTP password
            valueFrom:
              secretKeyRef:
                name: "{{inputs.parameters.secretName}}"
                key: password # Secret key for the password
          - name: SFTP_HOST # SFTP host
            valueFrom:
              secretKeyRef:
                name: "{{inputs.parameters.secretName}}"
                key: host # Secret key for the host
        volumeMounts:
          - name: temp-volume # Mount volume to persist the CSV file across steps
            mountPath: /tmp
  entrypoint: generate-and-upload
  volumes:
    - name: temp-volume # Volume name
      emptyDir: {} # Use emptyDir for temporary storage

Example Secret:

kubectl apply -f - <<EOF           
apiVersion: v1
kind: Secret
metadata:
  name: sftp-conn # Secret name
  namespace: argo
type: Opaque 
data: 
  username: $(echo -n "<SFTP_USER>" | base64) # Base64 encoded SFTP username
  password: $(echo -n "<SFTP_PASS>" | base64) # Base64 encoded SFTP password
  host: $(echo -n "<SFTP_HOST>" | base64) # Base64 encoded SFTP host
EOF