Feature request - Example with gzip `Content-Encoding`

[o-l-a-v]

In my testing, setting Content-Encoding: "gzip" with body being plaintext JSON, DCR diagnostic log category DcrLogErrors ( https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-logs/microsoft-insights-datacollectionrules-logs ) returns error Compressed data in invalid..

Would be nice with an example where you compress the content to GZIP so it works with Content-Encoding: "gzip".

One can likely save a lot of data on logs files in transit, which are both text and has a lot of repetative content. Example / playground:

• https://www.txtwizard.net/compression

The Log Ingestion API limitations is 1MB, compressed or not.

• https://learn.microsoft.com/en-us/azure/azure-monitor/service-limits#logs-ingestion-api

[o-l-a-v]

Managed to do it by:

• Convert Body from [string] with JSON content, to Gzipped byte array:

  $ByteArray = [byte[]]($Body.ToCharArray().ForEach{$_ -as [byte]})
  $MemoryStream = [IO.MemoryStream]::new()
  $CompressionStream = [System.IO.Compression.GZipStream]::new($MemoryStream, [Io.Compression.CompressionMode]'Compress')
  $CompressionStream.Write($ByteArray, 0, $ByteArray.'Length')
  $CompressionStream.Close()

• Ingest with header Content-Encoding: gzip and -Body ($MemoryStream.ToArray() -as [byte[]])

  # Create headers
  $Headers = [ordered]@{
      'Authorization' = [string] 'Bearer {0}' -f (
          ConvertFrom-SecureString -AsPlainText -SecureString (
              (Get-AzAccessToken -AsSecureString -ResourceUrl 'https://monitor.azure.com' -WarningAction 'Ignore').'Token'
          )
      )
      'Content-Type'  = [string] 'application/json'
  }
  
  # Ingest without compression
  $UploadResponse = Invoke-RestMethod -Uri $Uri -Method 'Post' -Body $Body -Headers $Headers -StatusCodeVariable 'StatusCode' -ResponseHeadersVariable 'ResponseHeaders'
  
  # Ingest with gzip compression (add Content-Encoding to headers)
  $UploadResponse = Invoke-RestMethod -Uri $Uri -Method 'Post' -Body (
      $MemoryStream.ToArray() -as [byte[]]
  ) -Headers (
      $Headers + @{'Content-Encoding' = [string] 'gzip'}
  ) -StatusCodeVariable 'StatusCode' -ResponseHeadersVariable 'ResponseHeaders'
  
  # Check output
  @{
      'StatusCode'      = [uint16] $StatusCode
      'ResponseHeaders' = $ResponseHeaders
  } | ConvertTo-Json

[KnudsenMorten]

Fantastic Olav. I will try in incorporate this into the product as soon as my time allow 😊 Med venlig hilsen | Best Regards Morten Knudsen [MVP] Microsoft MVP (Azure & Security) ​Microsoft Certified Trainer +45 40 178 179 ***@***.*** www.2linkit.net Blogmortenknudsen.net TeamViewer QuickSupport 2LINKIT Strandvejen 21A 6000Kolding Denmark From: Olav Rønnestad Birkeland ***@***.***> Sent: 18. december 2024 11:10 To: KnudsenMorten/AzLogDcrIngestPS ***@***.***> Cc: Subscribed ***@***.***> Subject: [EXT] Re: [KnudsenMorten/AzLogDcrIngestPS] Feature request - Example with gzip `Content-Encoding` (Issue #5) You don't often get email from ***@***.******@***.***>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Managed to do it by: * Convert Body from string to Gzipped byte array: · $ByteArray = [byte[]]($Body.ToCharArray().ForEach{$_ -as [byte]}) · $MemoryStream = [IO.MemoryStream]::new() · $CompressionStream = [System.IO.Compression.GZipStream]::new($MemoryStream, [Io.Compression.CompressionMode]'Compress') · $CompressionStream.Write($ByteArray, 0, $ByteArray.'Length') $CompressionStream.Close() * Ingest with header Content-Encoding: gzip and -Body ($MemoryStream.ToArray() -as [byte[]]) · # Create headers · $Headers = [ordered]@{ · 'Authorization' = [string] 'Bearer {0}' -f ( · ConvertFrom-SecureString -AsPlainText -SecureString ( · (Get-AzAccessToken -AsSecureString -ResourceUrl 'https://monitor.azure.com' -WarningAction 'Ignore').'Token' · ) · ) · 'Content-Type' = [string] 'application/json' · } · · # Ingest without compression · $UploadResponse = Invoke-RestMethod -Uri $Uri -Method 'Post' -Body $Body -Headers $Headers -StatusCodeVariable 'StatusCode' -ResponseHeadersVariable 'ResponseHeaders' · · # Ingest with gzip compression (add Content-Encoding to headers) · $UploadResponse = Invoke-RestMethod -Uri $Uri -Method 'Post' -Body ( · $MemoryStream.ToArray() -as [byte[]] · ) -Headers ( · $Headers + @{'Content-Encoding' = [string] 'gzip'} · ) -StatusCodeVariable 'StatusCode' -ResponseHeadersVariable 'ResponseHeaders' · · # Check output · @{ · 'StatusCode' = [uint16] $StatusCode · 'ResponseHeaders' = $ResponseHeaders } | ConvertTo-Json — Reply to this email directly, view it on GitHub<#5 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A45OWTQ7MXAQZZ2OXMU5L5L2GFCXTAVCNFSM6AAAAABTYMZ7IWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJQHEYTEMBXGY>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.******@***.***>>