1. Re-encoding the image is a good idea to make it harder to distribute exploits. For example imaging the recent WebP vulnerability. A malicious user could upload a compromised image as their profile picture and pwn anyone who saw that image in the app. There is a chance that the image survives the re-encoding but it is much less likely and at the very least makes your app not the easiest channel to distribute it.

2. It gives a good place to strip metadata. For example you should almost certianlly be stripping geo location. But in general I would recommend stripping everything non-essential.

3. Generating different sizes as you mentioned can be useful.

4. Allows accepting a variety of formats without requiring consumers to support them all. As you just transcode in one place.

I don't know much about the cost on the AWS side, but it seems like you are always at some sort of risk given that if the user knows the bucket name they can create infinite billable requests. Can you create a size limit on the pre-signed URL? That would be a basic line of defence. But you probably also want to validate once the URL expires the data uploaded and decide if it conforms to your expectations (and delete it if you aren't interested in preserving the original data).

Including animation in most cases. Otherwise somebody can use a single frame with very long duration that will be reviewed by moderators, then follow that frame with a different frame containing objectionable material which will eventually be shown to people.

For anyone wondering:

https://blog.cloudflare.com/uncovering-the-hidden-webp-vulne...

> The vulnerability allows an attacker to create a malformed WebP image file that makes libwebp write data beyond the buffer memory allocated to the image decoder. By writing past the legal bounds of the buffer, it is possible to modify sensitive data in memory, eventually leading to execution of the attacker's code.

Here is a canonical old article on one type of buffer overflow:

http://www.phrack.org/issues/49/14.html#article

(The webp one, iirc, is not a stack buffer overflow, but rather a heap, but that'll give you a general sense of what's going on, even if that article is a bit ... dated)

And now your server which is doing re-encoding is pwned. Gotta segment the server doing that somehow, or use pledge, capsicum, or seccomp I guess.

Not that OpenBSD is actually unhackable or anything, but I doubt many attackers would guess you're running imagemagick on OpenBSD in your image pipeline.

I rather like it for such use cases; it has the added benefit that it never, ever seems to die. I found a 6.0 machine I setup doing some kind of risky Kafka processing that had an uptime of 6 years the other week (since migrated).

It is running imagemagic to optimize images, create different resolutions and reencode them. It's only open for us to upload manually from our customers though, they can't upload themselves. Input anything, output jpg, very easy to use.

In my mind the chain of events looks like: - Alice uploads an image file to Bob's file server - Charlie's image converter server invokes a local browser with the address of the file on Bob's server, resulting in a .bmp (or .png? a little out of my element here) saved to Charlie's server. - Doris can now pull images from Charlie's server knowing they've been vetted by a major browser.

Does that make sense?

False security, if anything.

A 1080p image with 8 bits per channel would be 6 MB. Real mobile-friendly...

Someone might still be able to get an RCE on them and burn a bit of money, but they certainly wouldn't be able to move laterally.

For years now I use nginx's image filter [1] which handles file resizing quite nicely. Resized images are cached by nginx. For some usecases it works very vell and I no longer need to specify sizes beforehand, you just ask for the size by crafting your url properly.

[1] https://nginx.org/en/docs/http/ngx_http_image_filter_module....

In fact re-encoding probably would have solved this as the server could enforce the expected dimensions and rescale or reject the image.

Best to sandbox/jail/etc as tightly as possible, and limit the codecs to only what you need. You can configure the ffmpeg builds pretty granularly... default will include too much.

So not encoding is probably the safer way to go for the business.

What if the other user getting attacked is you or another admin on your team?

Now the attacker has admin access and can compromise your servers and “leak everyone’s data” just fine.

I don’t think you’ve thought this through.

If websites get blamed when users reuse passwords, they are definitely getting blamed if you distribute malicious files.

Famously, the Dangerous Kitten hacking toolset was distributed with the classic zip-in-a-jpeg technique, because imageboards used to not re-encode images.

https://web.archive.org/web/20120322025258/http://partyvan.i...

Yes, pre-signed URL can have the `Content-Length` set and amazon S3 checks it. However, note that this is true for Amazon S3 but not for others like BackBlaze or R2. Last time I tried, BackBlaze didn't support it.

You can hang an event off S3 and have a lambda that does the work / warns you of a bad upload.

Videos will add even more to your AWS bill if you're not careful. Re-encode that 4K cat video as soon as it comes in, or wire up a CDN to do it for you.

Yes, if you use the POST method you can set the content-length-range property in your presigned URL form inputs to limit min and max bytes. https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-authen...

Which immediately makes you vulnerable to zip-bomb attacks.

If you want to be more safe, you first have to do all kinds of checks of the image headers.

I vaguely remember flattening images in some manner so imagemagick wouldn't choke. Something about setting key tones? It's been a while.

One of the major lessons I pulled out of that time is that most library authors will happily fix bugs or add features if you give them money. You can also just ask them to sell you a different license if FOSS ruins your plan.

Users can't upload to your S3 storage because they lack credentials. (It would be dangerous to make it public.) But you can give them access with a specially-generated URL (generated for each time they want to upload). So your server makes a special URL, "signed" with its own authorization. That lets them upload one file, to that specific URL.

(I dunno about anybody else, but I find working with AWS always involves cramming in a lot of security-related concepts that I had never had to think about before. It puts a lot of overhead on simple operations, but presumably is mandatory if you want an application accessible to the world.)

It's nost likely more efficient, faster, and cheaper (no need to handle the traffic and hardware to proxy).

but for certain applications, s3 is simpler to use, especially when you need to scale

I am curious, so I took the opportunity to look up the strange sounding term "pre signed url". But that's just being a dilettante.

I’m honestly surprise this isn’t a value-added capability offered by AWS S3 because it’s such a common need and undifferentiated work.

I've used ClamAV before in web apps, and it was fairly easy to get working - but it has a few big flaws: it's very slow, it uses a huge amount of memory, and detection rates are... honestly, pretty terrible.

I've used Verisys Antivirus API for 2 recently projects, and it works great, a huge improvement over self-hosted ClamAV. There are a couple of other virus scan APIs available, including Scanii[1] which I think has been mentioned on HN before, and I Trend Micro has something too - but their pricing is secret, so I have to assume it's expensive.

[0] https://www.ionxsolutions.com/products/verisys-virus-api [1] https://scanii.com

ClamAV needs lots of memory to work - we needed at least 8GB in one project I worked on (where we also used commercial signatures), 4GB was just about enough for another. The more signatures you have, the more memory it uses.

ClamAV chews through a lot of CPU, and it's so very slow. For example, scanning a 25MB MSI file from my test corpus takes 65 seconds - and that's on a fairly beefy machine!

I've used both of the services linked above, and both seem excellent - detection rates and performance are far superior to ClamAV, and they Just Work(TM).

https://learn.microsoft.com/en-us/azure/defender-for-cloud/d...

Mind you, last time I designed something using this we just used ClamAV - which is pretty easy to develop against even if it is a slight pain to manage on an ongoing basis.

> a ton of additional complexity with a proven track record of CVEs

become:

> adding it in couldn’t hurt

?

Do you also handle large files e.g. videos with lambda? Don't you struggle with Lambda limits then?

We ultimately scrapped it, if anyone else has any better experience I'd love to hear it.

https://www.cisa.gov/resources-tools/services/malware-next-g...

The raw RGBA output is then received and converted back into PNG or similar. It was a bit tricky to get everything working without additional allocation and using syscalls triggered by glibc somewhere, but works pretty well now and is fast enough for my use case (around 20ms/item).

I’m a huge fan of building minimal self-contained tools, so all of the C programs statically link in the required parser libs (libavcodec/wuffs/freetype) so the resulting binaries don’t require additional dependencies on the target machine. The python wrapping code is rather straightforward as well and is only like 300 lines of code.

- serve on a different top level domain, ideally with random subdomains per uploaded file or user who provides the content. This is really most important for serving document types and probably not for images though SVG I think is the exception as it can have scripting and styling within when loaded outside of an IMG tag

- set "content-security-policy: sandbox" (don't allow scripts and definitely don't allow same origin)

- set "X-Content-Type-Options: no sniff" - disabling sniffing makes it a lot harder to submit an image that's actually interpreted as html or js later.

Transforming the uploaded file also would defeat most exploit paths that depend on sniffing the content type.

From basically the day it existed it's been considered a best practise to send that header with every request, because there might be a vulnerability introduced otherwise . I've never heard of anyone relying on it. Why can't modern browsers flip the default on this internet Explorer era magic footgun?

It isn't the default in modern browsers, or at least not in the sense you mean. The header still controls some things, but even without it, browsers will not sniff something served as content type image/png as html or anything like that. How modern browsers interpret that is different than IE.

Both have worked well for us so far but I don't know about your scale and impact on pricing (we're small scale so far).

Cloudflare Images lets you auto resize images to generate thumbnails, etc. Same with video, where they will auto-encode the video based on who's watching it where. So for us it's just a matter of uploading it, getting an identifier, and storing that identifier.

You can do the math on the ingress to your service (let's say it's EC2), and then the upload from EC2 to S3.

It appears that AWS doesn't charge for ingress [0]. "There is no charge for inbound data transfer across all services in all Regions".

S3 is half a cent per 1000 PUT requests [1], but you were already going to pay that anyway. Storage costs would also be paid anyway with a presigned URL.

You'll have more overhead on your server, but how often do people upload files? It'll depend on your application. Generally, I'd lean towards sending it to your backend until it becomes an issue, but I doubt it ever will. Having the ability to run all the post processing you want and validate it is a big win. It's also just so much easier to test when everything is right there in your application.

[0] https://aws.amazon.com/blogs/architecture/overview-of-data-t...

[1] https://aws.amazon.com/s3/pricing/

Do not serve content from S3 directly.

ISPs often deprioritize traffic from S3, so downloading assets can be very slow. I've seen kbytes/s on a connection that Speedtest.net says has a download speed of 850 mbit. Putting Cloudfront in front of S3 solves that.

I wonder why

From there I could browse through all of their customer's home directories; eventually I found the SQL database admin password, which turned out to be the same as their administrator password for the Windows server it was running on: "internet".

This was a big lesson for me in upload sanitizing.

Since this isn't enforced by the presigned PUT URL, you can't trust that the limits have been respected without inspecting whatever was uploaded to S3. You can get a lot more flexibility in what is allowed if you use an S3 presigned POST, which lets you set minimum and maximum allowed content lengths.

[0]: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPO...

I have seen a team struggle for over a month to eliminate NSFW content - avatars - uploaded by a user that lead to their site being demonetised.

*: act as if successful, including processing time, but throw away the upload

Images are easy to display but for other media files you will probably need some streaming solution, a player, etc.

We're in the audio hosting and processing space. We still don't have an api though.

For video maybe look into Wistia, Cloudflare, BunnyCdn, etc.

- upload bucket

- processed bucket

upload bucket has an event triggered on new file upload which triggers a lambda, the lambda will re-encode and do wtv you deem fit and upload to new bucket

your app will use the processed bucket

Use Cloudflare R2 for storage, public bucket for delivery, and a $10/month Hetzner server to run conversions. You don't have to take your users' money just to burn it on AWS...

If you are using AWS and are worried about cost, the first step is to not use AWS.

Once the upload completes we kick off a workflow to virus scan, unzip, decrypt, and process the file depending on what it is. We do some preliminary checks in the service looking at the file name, extension, magic bytes, that sort of stuff and reject anything that is obviously wrong.

For virus scanning, we started with ClamAV[1], but eventually bought a Trend Micro product[2] for reasons that may not apply to you. It is serverless based on SQS, Lambda, and SNS. Works fine.

Once scanned, we do a number of things. For images that you are going to serve back out, you for sure want to re-encode those and strip metadata. I haven't worked directly on that part in years, but my prototype used ImageMagick[3] to do this. I remember being annoyed with a Java binding for it.

[0] https://tus.io/ [1] https://www.clamav.net/ [2] https://cloudone.trendmicro.com/ [3] https://imagemagick.org/index.php

One of the most important, yet oft ignored on HN, principles of architecting solutions is having an unbiased view of products and then choosing what will fit within an existing organisation’s architecture.

Given the specifications shared, utilising S3 further is the correct advice.

A different suggestion would be to build a threat model.

Who are your uploaders? Who are your viewers? What are the threats?

Only when you've figured out what the threats are, can the solutions make sense.

Internal site? Internal users? Publicly available upload link? What media types? ....

Threat modelling is a thing that adds lots of value (imho) when thinking about adding new capabilities to a system.

The intention is to develop an API-driven approach that can be easily integrated into your own products as part of your file upload mechanism. We're really early stage, so we're looking for businesses and individuals to help us define what the product should look like. If you'd be up for sharing your thoughts, you can email us at info@bucketscan.com and/or complete this short product survey:

https://forms.gle/rywgnQ7zqsPuLdMd6

Also I don't think the Content-Type is actually verified by S3 so technically users can still upload malicious files such as an executable with a png extension.

On the bright side, S3 supports requesting a range of bytes. You can use that to perform validation server side afterwards to enforce it's really a png, jpg or whatever format you want. Here's examples in Python and Ruby to verify common image types by reading bytes: https://nickjanetakis.com/blog/validate-file-types-by-readin...

I don't do videos yet, but I'm kinda terrified of the idea of putting user-uploaded files through ffmpeg if/when I'll support them.

[1] https://github.com/grishka/Smithereen

[2] https://github.com/imgproxy/imgproxy

Honestly though if this is an authenticated function and you have a small user base… who cares? Is there a reasonable chance at this disrupting any end user services? Maybe it’s not the best way to spend hundreds of hours and thousands of dollars.

Granted you’re an SRE so it’s your job to ideas this. I’d just push back on defaulting to dropping serious resources on a process that might be entirely superfluous for your use case.

I ended up piping it through my server. I can limit file size, authenticate, insert it into my DB and do what not this way.

[0] https://transloadit.com/

Like you stated, an async process using a function would suffice. Previously used ClamAV for this in a private cloud solution, I’ve also used the built in anti-virus support on Azure Blob Storage if you don’t mind multi-cloud, plus an Azure Function has the ability to support blob triggers, which is a nice feature.

The file types scan is relatively simple. You just need a list of known “magic string” header values to do a comparison, and for that you only need a max of 40 bytes of the beginning of the file to do the check (from memory). Depending on your stack, there are usually some libraries already available to perform the matching.

And it goes without saying, but never trust the client, and always generate your own filenames.

https://en.m.wikipedia.org/wiki/List_of_file_signatures

What you instead want is a whitelist: Only allow properly formatted images and videos and ruthlessly reject anything else. I wrote about how I implemented this for my service in another response.

Some problems you can run in to: Exploiting image or video processing tools with crafted inputs, leading to server compromise when encoding/resizing. Having you host illegal or objectionable material. Causing extreme resource consumption, especially around video processing and storage. Having you host material that in some way compromises your clients (exploits bugs in jpeg libraries, cross site scripting, etc.)

I can't really talk about what is done at the FAANG that I worked at on this stuff, but if you are a large enough target, this is a huge threat vector.

If you can spare the CPU cycles (depending on how you optimize, they can actually be expensive) and if your images will be downloaded frequently, your users will thank you.

if you want to get into the nitty grit of filetype abuse, to learn how to possibly detect that good. ange albertini's work on polymorphic filetypes and the ezine poc||gtfo as well as lots of articles by AV code devs are available. its really hard problem and also depends a lot on what program is interpreting the files submitted. if its some custom tool.there might even be unique vectors to take into account. fuzzing and penteting the upload form and any tools handing these files can shed light on those issues potentially.

(edit: fat fingers)

The function ensures images aren’t bigger than 10mb, the image is compressed to our sizing, and put into s3.

In my case we allow up to 10mb file upload. For example just testing something right now from my iPhone I selected a 3.7mb image which was uploaded and resized to the "frame" it will go into in the UI and its now 288kb in S3. We have a few "frame" sizes which are consistent throughout our application. Now, my cloudfront is serving 288kb file instead of 3.7mb, which is good for me because I want to avoid the bandwidth costs and users honestly can't tell the difference, and then also we aren't a photo gallery app.

So in my case I need the lambda to resize the image to our "frame" sizes. I wonder if I didn't explain right. I'm ESL so just want to make sure since I'm confused about the your last sentence.

But if you do resizing as well, then ofc you need the Lambda (or some other compute engine).

On the other hand, not processing/scanning your uploads is probably more risky for your users/the rest of the internet.

It’s not particularly cheap. But it is fast and flexible and safe.

I created a program for profile pictures. It uses face recognition technology as to not deform faces when resizing photos. This may be useful to you.

https://github.com/jftuga/photo_id_resizer

- some file type and size checks in web app

- pre-signed URL

- upload bucket

- lambdas for processing and sanity checks

- processed bucket

I'd suggest existing tooling like `exiftool` to do mimetype detection and metadata stripping.

What is the business case for making the necessary changes?

Good luck.

You will usually want the profile image of user A to be shown to user B. Same for videos and other files - users likely upload them so that others can download them / consume them in some way, right?

I think where you need to start is doing some threat analysis, and proceed from there. Hosting user content can be built out from "very small" to "very big", depending on the particular threat scenario/use cases/your particular userbase. With the description that you are giving, I would say you are more at risk of building an overcomplicated ("oversecured") solution which might compromise UX for the sake of some protection that is not necessarily needed.

If you are a small team, likely you could use an image resizing / video thumbnailing proxy server such as https://www.imgix.com/ https://imgproxy.net/ etc. You generate a signed URL to it and then the service picks up the file from S3 and does $thing to it. https://www.thumbor.org/ is another such tool. There are quite a few.

Re. uploads and downloads - you have quite some options with S3. You can generate the presigned upload URL on the server (in fact: you should do just that), make it time limited and add the Content-Length of the upload to the signed headers - this way the server may restrict the size of the upload. Similarly, access to display the images can be done via a CDN or using low-TTL signed URLs... plenty of things to do.

Yeah definitely. Even optimizing the vids. I just spend time writing scripts to convert, in parallel, a massive amount of JPG, PNG, PDFs, mp4 videos and even some HEIC files customers sent of their ID (identity card or passport, basically). I did shrink them all to reasonable size.

The issue is: if you let user do anything, you'll have that one user, once in a while, that shall send a 30 MB JPG of his ID. Recto. Than Verso.

Then the signed contracts: imagine a user printing a 15 pages contract, signing/paraphing every single page, then not scanning it but taking a 30 MB picture, with his phone, in diagonal, in perspective. And sending all the files individually.

After a decade, this represented a massive amount of data.

It was beautiful to crush that data to anywhere from 1/4th to 1/10th of its size and see all the cores working at full speed, compressing everything to reasonable sizes.

Many sites and 3rd party identity verification services (whatever these are called) do put limit on the allowed size per document, which already helps.

In my case I simply used ImageMagick (mogrify), ffmpeg (to convert to x265) and... GhostScript (good old gs command). PDFs didn't have to be searchable for text so there's that too (and often already weren't at least not easily, due to users taking pictures then creating a PDF out of the picture).

This was not in Amazon S3 but basically all in Google Workspace: it was for a SME to make everything leaner, snapper, quicker, smaller. Cheaper too (no need to buy additional storage).

Backups of all the originals, full size, files were of course made too but these shall probably never be needed.

In my case I downloaded everything. Both to create backups (offsite, offline) and to crush everything locally (simply on an AMD 7700X: powerful enough as long as you don't have months of videos to encode).

> Anybody have experience with any of this? I imagine having a big team and dedicated services for media processing could work, but what about small teams?

I did it as a one-person job. Putting limits in place or automatically resizing, right after upload, a 30 MB JPG file which you know if of an ID card to a 3 MB JPG file doesn't require a team.

Same for invoking the following to downsize vids:

    ffmpeg -i input.mp4 -vcodec libx265 -crf 28 output.mp4    (I think that's what I'm using)

I didn't bother verifying that the files visually matched (once again: all the originals are available on offline backups in case something went south and some file is really badly needed) but I could have done that too. There was a blog post posted here a few years ago where a book author would visually compare thumbnails of different revisions of his book, to make sure that nothing changed too much between two minor revisions. I considered doing something similar but didn't bother.

Needless to say my client is very happy with the results and the savings.

YMMV but worked for me and worked for my client.

I'm not diminishing asking the question in principle, I'm questioning the role and forum that the question is being asked on.