Can find assets that are deleted but still there with versioning

[mzaragoza]

I have an application and just saw that this bucket is over 4TB.
I wrote a script to delete all the assets that are older that 1 year.

namespace :aws do
  desc 'Query S3 bucket and delete objects older than one year'
  task delete_old_objects: :environment do
    s3_client = Aws::S3::Client.new(region: ENV['AWS_REGION'])

    bucket_name = ENV[BUCKET_NAME]

    continuation_token = nil

    loop do
      resp = s3_client.list_objects_v2(bucket: bucket_name, continuation_token: continuation_token)
      resp.contents.each do |object|
        creation_date = object.last_modified
        if (Time.now - creation_date) > (365 * 24 * 60 * 60) # Check if older than one year (365 days)
          s3_client.delete_object(bucket: bucket_name, key: object.key)
        end
      end
      break unless resp.is_truncated
      continuation_token = resp.next_continuation_token
    end
  end
end

The problem that I am having is that I cant seem to find those assets that should had been deleted, but still have a ghost or are still saved in a version

[mullermp]

That code seems like it should work. What do you mean by "have a ghost" or "still saved in a version"?

Perhaps try s3.list_object_versions and delete that way. You can modify this code to add the filtering:

  loop do
    object_versions = s3.list_object_versions(bucket: bucket)
    objects = object_versions.data.versions.map do |v|
      { key: v.key, version_id: v.version_id }
    end
    break if objects.empty?
    s3.delete_objects(bucket: bucket, delete: { objects: objects })
  end

Just some improvements you can make to your script. AWS_REGION is read automatically when you create a client, so there's no need to pass in a region parameter. Also it looks like you're handling pagination on your own - You can use resp.next_page and resp.next_page? to do pagination.