Uploading

Uploading your data files to the Pulsate Data Ingestion Service through SFTP is a straightforward process that can be automated using various programming languages. This section provides code examples in C#, Node.js, and Ruby to illustrate how you can upload a file to an SFTP server.

Recommendations on Upload Frequency

Before diving into the code examples, it is essential to address the best practices regarding the frequency of uploads:

• Batch Processing: Complete any batch processing of your data on your side before initiating the upload. This ensures that the data is current and consolidated.
• Daily Uploads: We recommend uploading your data files once per day. This frequency strikes a balance between up-to-date data and minimizing server load, which can be critical for larger datasets.

Examples

The following examples are a guide on how to implement SFTP upload functionality. You should always consider your own needs when implementing any functionality, as the below samples may use libraries that are not available to you.

C#

using Renci.SshNet;

public void UploadFileToSftp(string host, string username, string password, string localFilePath, string remoteFilePath)
{
    using (var sftp = new SftpClient(host, username, password))
    {
        sftp.Connect();
        using (var fileStream = File.OpenRead(localFilePath))
        {
            sftp.UploadFile(fileStream, remoteFilePath);
        }
        sftp.Disconnect();
    }
}

// Example usage:
UploadFileToSftp("your_sftp_server.com", "username", "password", @"C:\path\to\your\file.csv", "/path/on/sftp/server/file.csv");

NodeJS

const SftpClient = require('ssh2-sftp-client');
const fs = require('fs');

let sftp = new SftpClient();

const uploadFile = async (host, username, password, localFilePath, remoteFilePath) => {
  try {
    await sftp.connect({
      host: host,
      username: username,
      password: password
    });
    let readStream = fs.createReadStream(localFilePath);
    await sftp.put(readStream, remoteFilePath);
    console.log('File uploaded successfully');
  } catch (err) {
    console.error('File upload failed:', err.message);
  } finally {
    sftp.end();
  }
};

// Example usage:
uploadFile('your_sftp_server.com', 'username', 'password', './path/to/your/file.csv', '/path/on/sftp/server/file.csv');

Ruby

require 'net/sftp'

hostname = 'your_sftp_server.com'
username = 'username'
password = 'password'
local_file_path = '/path/to/your/file.csv'
remote_file_path = '/path/on/sftp/server/file.csv'

Net::SFTP.start(hostname, username, password: password) do |sftp|
  sftp.upload!(local_file_path, remote_file_path)
  puts "File uploaded successfully"
rescue StandardError => e
  puts "File upload failed: #{e.message}"
end

Notes:

• In each script, replace your_sftp_server.com, username, password, /path/to/your/file.csv, and /path/on/sftp/server/file.csv with your actual server details and file paths.

• Ensure you have the required libraries or gems installed (SSH.NET for C#, ssh2-sftp-client for Node.js, and net-sftp for Ruby).

• For production code, handle your credentials securely (e.g., environment variables, secret management services) and implement error handling and logging.

• Check for dependencies and library updates regularly to maintain security compliance.

By following these examples and guidelines, you can automate the process of uploading your CSV files to the Pulsate Data Ingestion Service securely and efficiently. If you encounter any issues during file upload, do not hesitate to contact our support team for assistance.