Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete S3 media individually instead of in batches #69

Merged
merged 5 commits into from
Mar 8, 2024
Merged

Delete S3 media individually instead of in batches #69

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3c53b30
update Sentry DSN
toufali Feb 22, 2024
1457200
merge main, and make sentry_dsn check more readable
toufali Feb 23, 2024
35630ba
Merge pull request #65 from MozillaSocial/sentry-update-dsn
toufali Feb 26, 2024
65d83a8
Delete S3 media individually instead of in batches
aaga Mar 7, 2024
319fef1
Merge branch 'upstream-v4.2.8' into fix-media-deletion
aaga Mar 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 5 additions & 7 deletions app/lib/attachment_batch.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,13 +95,11 @@ def remove_files
# objects can be processed at once, so we have to potentially
# separate them into multiple calls.

keys.each_slice(LIMIT) do |keys_slice|
logger.debug { "Deleting #{keys_slice.size} objects" }

bucket.delete_objects(delete: {
objects: keys_slice.map { |key| { key: key } },
quiet: true,
})
# Mozilla Social - delete individually instead of in batches,
# since GCP XML API doesn't support batch delete
logger.debug { "Deleting #{keys.size} objects" }
keys.each do |key|
bucket.object(key).delete
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting that your non-batch version is less lines and less complex than the original batch version that needs to be sliced anyway! 🚀

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

haha! the batching makes a difference on the S3 side - putting it into slices of 1000 means all those 1000 media get deleted simultaneously, rather than O(n) with the non-batch way.

but it all gets deleted eventually so it shouldn't be an issue!

end
end

Expand Down
Loading