Maybe it just happens to me; maybe you’ve seen this as well. You’re using PowerShell or another tool to write logging to a file, and everything looks all very cool. A few days later, you grab a coffee and open the file, only to see something like this.
Sometimes it looks like this; sometimes it looks like someone dumped a huge amount of Chinese characters into the file.
In any case, you can’t translate it. Believe me, I’ve tried a lot of translation engines!
The solution
I found a way to fix this using PowerShell.
First, make sure all commands that write to a file use UTF8 encoding. This will prevent your machine from switching to another encoding without telling you. This doesn’t fix the files that are messed up, but, hurrah! There’s some code to fix that too.
To recover the data, you can use this.
param(
[Parameter(Mandatory=$true)]
[string]$Path
)
function Convert-Utf16Block {
param([byte[]]$bytes)
# Decode the UTF-16LE block properly
return [System.Text.Encoding]::Unicode.GetString($bytes)
}
function Repair-File {
param([string]$File)
Write-Host "Processing $File..."
# Read raw bytes
$bytes = [System.IO.File]::ReadAllBytes($File)
$output = New-Object System.Collections.Generic.List[string]
$i = 0
while ($i -lt $bytes.Length) {
# Detect UTF-16LE pattern: ASCII chars stored as XX 00 XX 00 XX 00
if ($i -lt $bytes.Length - 4 -and
$bytes[$i+1] -eq 0 -and
$bytes[$i+3] -eq 0) {
# Start of a UTF-16LE block
$start = $i
$block = New-Object System.Collections.Generic.List[byte]
while ($i -lt $bytes.Length - 1 -and $bytes[$i+1] -eq 0) {
$block.Add($bytes[$i])
$block.Add($bytes[$i+1])
$i += 2
}
# Convert the UTF-16LE block
$decoded = Convert-Utf16Block $block.ToArray()
$output.Add($decoded)
}
else {
# Normal UTF-8 byte — decode as single char
$output.Add([System.Text.Encoding]::UTF8.GetString(@($bytes[$i])))
$i++
}
}
$fixedText = ($output -join "")
$newFile = "$File.fixed"
[System.IO.File]::WriteAllText($newFile, $fixedText, [System.Text.Encoding]::UTF8)
Write-Host " Repaired -> $newFile"
}
# Process a single file or all files in a folder
if (Test-Path $Path -PathType Leaf) {
Repair-File -File $Path
}
elseif (Test-Path $Path -PathType Container) {
Get-ChildItem -Path $Path -File | ForEach-Object {
Repair-File -File $_.FullName
}
}
else {
Write-Host "Invalid path: $Path"
}
This code reads the UTF-16 data and writes it out as UTF8. That’s about it. And yes, I had a little help from the AIs.
Hopefully this short blog helps you to get unstuck!
