MESH API Compression Chunks

API Platform
1

Hi,
I’m currently on the P2L (INT) environment building some proof of concept snippets using the MESH API, and I’ve had good success logging in, and sending/receiving standard (small) messages of flat text and attachments between mailboxes.

But, compression and chunking which is the next step is confusing me.

Let’s say I have a bitmap file (useful for testing at least), it is 15,183,054 bytes in size. (15Mb near enough)

I can chunk it into 3 chunks, of size 5,242,880, 5,242,880 and 4,697,294, the API accepts each chunk, and I can rebuild the output file on the other side by writing the three retrieved chunks to an output file in order. Fundamentally, this works great, but I wish to employ compression as per best practice.

Both in terms of cloud storage or local storage, it would not be a good approach to attempt to open a large (10Gb theoretical) file into memory, compress in memory and understand if the resultant file would be large enough for chunking, and that’s despite the fact each chunk should be compressed individually.

That said, following the guidance, if I read a 5Mb chunk of the file and compress in memory like so:
len(chunk)
5242880

len(compressed_data)
87822

When I attempt to fire 87822 over the API and I get the response of 417
b’{“message_id”:“20241018152833889568_005F1E”,“internal_id”:“20241018152833870_48bb3e98”,“detail”:[{“event”:“SEND”,“code”:“22”,“msg”:“Only the last chunk can be less than 5MiB, decompressed”}]}’

My questions are therefore

1: The error message suggests it is the decompressed chunk size which can’t be less than 5Mb - which it isn’t.
2: If the error description is itself wrong and it’s actually the compressed chunk which can’t be less than 5Mb, then I am at a loss as to how I can guarantee individually compressed chunks of a large file will be at least 5Mb. Certain parts of files will compress much better than other parts, depending on the contents. For example, a file with 5242880 zeroes at the start, and 5242880 random characters following, the first chunk will result in next to nothing compressed, whereas the latter half will be barely affected.

I am sending Mex-Content-Compressed = ‘Y’ in the request.

Any advice you can offer would be appreciated.

Regards
Dan

2

It actually does relate to uncompressed size - that’s a relief and makes sense. And the issue I was having was down to not declaring gzip in Content-Encoding.

If this question can be closed off I’d appreciate it.

3

Hi Daniel,

Thank you for posting the solution to your own query. You are right, the issue was due to the uncompressed size. The solution you provided will be useful to others who may face similar issues.

The key is to ensure that:

• Each decompressed chunk (except the last one) is greater than 5 MiB.
• Compressed chunks are not too small due to high compression ratios.
• All required headers are correctly set and accurately reflect the data sent.

By making these adjustments, you should be able to successfully send compressed, chunked messages using the MESH API.

Thanks again for continuing to be an active member of this community.

Thanks,
NHS England API Platform team

Please note: The API Platform team can only address queries relevant to the NHS England API platform, including security, rate limiting, logging, monitoring and alerting. For any API specific queries, please reach out the relevant API teams.