Ewon Flexy IoT Router. A Deep dive

Jun 18, 2019

This blogpost is a direct copy of previous research i have done at PenTestPartners. The original post resides here: https://www.pentestpartners.com/security-blog/ewon-flexy-iot-router-a-deep-dive/

This blog post is purely for archive purposes only


First off I would like to thank the techs at PTP for their insights  and help during this process. I know what I know, and I don’t know what I  don’t know, so I asked for help sometimes.

I’ve learned a lot from this project e.g. how XOR works, and how to  use IDA to analyse ARM binaries better, so I thought I’d share. I’ve  listed my fails and wins along the way too, in case you think it was too  easy ?

There’s also a shorter TL;DR version available here.

Disclaimer

This was not a paid test or assessment of this device. Responsible disclosure was done at multiple points during this project.

The research started when an actual client assessment came in with  one of these devices attached to the product. I got one of those moments  that set my Spidey senses tingling, but the device itself was not the  scope of the testing.

So, I asked Pen Test Partners to buy me one. They did and I was told to “go and play”!

This whole “Deep dive” is from a black box point of view, assuming nothing about the device or any authentication given.

There’s a table of contents more for my own structuring  than  anything else, and you get the added bonus of being able to skip the  bits you’re not interested in.

So, sit back, grab a coffee (or beer) and I hope you enjoy this read.

Ewon IoT 200 Flexy Router

The reach of these devices is incredible, these are the stats that Ewon give:

Impressive.

Here’s what Shodan has to say on the matter:

Ah not bad. Remember, Shodan has plenty of reach, its not 100% perfect, but it will do.

Additionally, Ewon devices looks like they might be manufactured by HMS Industrial Networks.

HMS Networks AB is an international company in the field of  industrial communications with the motto “Connecting Devices”. HMS is  headquartered in Halmstad, Sweden and is listed on the Nasdaq Nordic  stock exchange, employing over 500 people in ten countries and with  reported sales of 101 million Euro in 2016.

Source: https://www.hms-networks.com/products-solutions

Right Let’s get into it!

Web Interface

The device turned up and I got mega excited and just straight plugged  it in. That’s as far as I got before I had to break out the instruction manual. Also, I couldn’t be bothered to hunt down the IP address it was set to.

By default, the device is set to 10.0.0.53 with a default account of:

Username: adm
Password: adm

…I’ll come back to this later.

Note: I updated to the latest firmware at the time  of writing this post. I didn’t want Ewon coming back and saying “but  that’s not latest version.. blah blah blah”

WEB – Basic Authentication

I really don’t need to go into this, but yeah, the Ewon web application is using Basic Authentication for its users:

And yes, its communicating over HTTP, not HTTPS. Again, not going  into that either as its well known, but this will be helpful later on  for us ?

“BASIC AUTHENTICATION IS NOT SECURE AND SHOULD NOT BE USED IN APPLICATIONS”

WEB – XSS

Yes, you already know what’s coming, super elite hax and pure  unadulterated pwnage because, remember kids, they are ISO 27K1  compliant….

So, we can steal some creds because the base64 cookie of the Basic authentication is so poor!

But wait, Ewon knew about these already (2015), in fact they knew  about a lot more as well. Some issues look like they have been  fixed(ish) in updates since this disclosure was released:

Ewon sa Industrial router – Multiple Vulnerabilities : Karn Ganeshen (@juushya)

Date: Thu, 24 Dec 2015 – https://seclists.org/fulldisclosure/2015/Dec/118

Karn published a lot more for the web interface, but I’ve gone in a different direction.

Obtaining the Master Decryption Key

As it turns out, once you get access to the device; either by  phishing creds via XSS or if they are left as default (adm:adm) you can  pretty much do anything. One thing I didn’t go over was separation of  authorisation between the admin account and lower level user accounts.

When viewing the web application or gaining access to the files that  store the sensitive data such as VPN Private Certificates and keys or  even the user’s passwords, they seemed to be encrypted in a custom way I  had not seen before. This intrigued me, so I dug a lot deeper.

The picture below is what an authenticated user would see after the values have been entered and stored in the configs:

Or you can just do a simple post request and grab the Encrypted VPN Private Cert (remember you can get creds ?)

Regardless of this, lets break the stored string down and get into the issue.

We will work with the password string as it’s a lot smaller for now, but the same applies to the bigger VPN certs and keys.

This is the output of the username and stored password for the admin user (my user):

[“ptp”,”#_1_EHyXHCXlKSnkcW2f7kthnIg=”]

Now the eagle eyed of you will instantly see that there is a prefix and a base64 contained value

“#_1_” = Prefix

“EHyXHCXlKSnkcW2f7kthnIg=” Base64 Encoded data.

Unfortunately, the decoded data isn’t of use:

This isn’t great, I want to know what the actual password is.

Yes, I DID already know the password from the XSS/Insecure HTTP  comms, but if I can decrypt the data, can I decrypt the VPN Private Key  as well?

I started by adding another user to the Ewon Router and setting a password to the minimum length that was accepted “aaa”.

The test user password gets encrypted to “#_1_BXWfyGY=”. We then  remove the prefix “BXWfyGY=” and decode the base64 data. It gives us  some random looking data “.u.Èf”.

There seems to be some additional characters added at the end of the  decoded string, as remember, we only put in 3x“a” as the password.  Strange behaviour!

Without fully understanding the encryption method here I decided to  modify the password to “aba” to see how it changed. “#_1_BXafyJY=” was  the outcome.

There’s clearly some similarity:

“BXWfyGY=” = aaa
“BXafyJY=” = aba

Notice how the first two characters are the same? Does that mean that its encrypted per character?

Let’s change the password again, to “aab”:

“BXWfyGY=”-> “.u.Èf” = “aaa”
“BXafyJY=” -> “.v.È.” = “aba”
“BXWciGc=” -> “.u..g” = “aab”

My encryption game is weak so I went to my colleagues. We all agreed  that it was doing the encryption per character and then adding something  at the end. One method that was mentioned was XOR.

Let’s fire up CyberChef and get it working for us!

We knew the length of the password “3” and we know what the plaintext variant was “aaa” so we let CyberChef do its thing:

BINGO! Although we still have the two ending chars at the  end, we can for now forget those. Now let’s see if we can get it do the  full 22 length password.

And yes, we can, this was very labour intensive on CyberChef  (Calculating 9.578097130411805e+52 values) so we wrote a python script  to do this and left it running:

…so does this key work on the original password for the PTP user too?

Yes, it does. It also does the Private Certificate data too:

We also tried this on numerous Ewon Flexy devices (that we owned), and it worked across the board.

What does this mean? Well, lets recap a little from this point before moving forward?

We can steal credentials from a valid user via XSS, or if we  are on the same LAN (for whatever reason), we can also grab encrypted  sensitive data and decrypt it (other user’s passwords, VPN keys and  configs) etc… We can use the Private Key and password that was stored in  the configs to join an existing VPN connection (yes, we configured it  and tried, it works)!

We disclosed this to Ewon and got a response much like Karn did.  Apparently we needed Authentication for this to be viable! (I would  agree with this if there were not multiple XSS and it used HTTPS). Also,  none of these devices should be connected to the internet.

Well guess what, A LOT ARE.

WEB – Default Creds

Let’s now go back to the default creds thing. These are well known  within the Ewon world and are well publicised. Fine, for initial setup  this is ideal BUT, when you enter the wizards, the username and password  are pre-populated for you, and guess what they are… “adm:adm”!

Ewon should put some checks here to stop people accepting the defaults and clicking “Next>”

Now, if you look at the reach that Ewon say they have with the  products they sell, and the Shodan.io Search from earlier, you make up  your own mind on how many have changed from default creds!

I will have a guess at probably not many!

Additionally, to all this, these creds are also used for the FTP service running on the same host ?

As default the Web Portal and ftp is set on Port 80 and 21  respectively, also, there is a secondary port set for the web portal on  81. More on this later.

Hardware

Let’s now jump into a hardware review of the device.

As it’s an IoT device with a lot of usable functionality I was expecting it to be a lot worse that it initially was.

Stripping the Ewon Flexy down enough to get to the board was easy  enough. It looks like a single controller for the peripherals/daughter  boards and a OS board that is slotted in on the back:

There are slots available for a total of four daughter boards to be attached, and an SD card slot and 4 ETH ports:

On the back there is a slot for the OS board and a number of test pads available:

Things of interest are the NAND Flash on the OS board, the Test pads  on the OS board (Top left 2×5 rows of round pads) and the three test  pads on the main board on the back next to the cartridge slot (near  TP12)

Hardware – JTAG

This is one of my fails.

When seeing a double row of five round test pads I instantly think  “JTAG”, but lets be honest, it could be anything It does kind of make  sense for an initial test and program of the board. You can see at the  top left corner there is a 2-20 slot connector that I suspect can be  used to quickly attach to a JTAG programmer and flash the device.

I soldered a wire to every one of the test pads and fired up a JTAGulator.

Please excuse the different coloured wires, it’s all I had to hand at the time

Turns out this is what JTAGulator said the pins were

Getting the ID for the device gave this output:

JTAG > d TDI not needed to retrieve Device ID. Enter TDO pin [1]: Enter TCK pin [4]: Enter TMS pin [2]: All other channels set to output HIGH. Device ID #1: 0000 0001001010000001 00000100001 1 (0x01281043) -> Manufacturer ID: 0x021 -> Part Number: 0x1281 -> Version: 0x0

After some Googling it looks like a Lattice Semiconductor Device, not a ATMEL.
There’s more info here: http://bsdl.info/details.htm?sid=89730f33d939207188ca59b002289d77

I’m not sure what’s going on there really, but it did halt the rest of the JTAG enumeration and connection:

> halt halt Halt timed out, wake up GDB. timed out while waiting for target halted

Hardware – Serial

Next I moved on to another set of test pads that looked interesting. A  row of three can either mean Serial or SWD to me, so first thing was to  hook up the Saleae Logic Analyser to the three pins and see what was  going on.

The three pins were accessible via the bottom of the device and even had a cut out on the plastic housing, this was ideal.

I soldered some wires to the pads and connected them to the Saleae,  to see if I could get a read of something that might help me out:

Here’s a zoom-in for clarity:

BOOTLOADER!

I tried not to get too excited at this point, but I was actually doing this!

Now I needed to focus on getting it setup to be interactive. I broke  out the USB >serial dongle, threw the wires on and used a generic  terminal to connect to the device:

Linux-3.2.11; Hardly up to date. Also looks like we can stop the boot process, hopefully to a shell:

MMC: mci: 0 In: serial Out: serial Err: serial Net: macb0 Hit any key to stop autoboot: 0 Net: enable eth switch U-Boot> help U-boot console is locked U-Boot> ? U-boot console is locked U-Boot>

Now what? Let’s mess around and see if I get lucky:

U-Boot> unlock pw Var pw not def U-boot console is locked

Oh Hello! This looks a bit more promising.

Unfortunately that was far as I got for a while, we will come back to this part later.

Hardware – NAND Flash Dump

After my intial frustrations I de-soldered the NAND flash and threw  it in the DATAMAN reader we have at the office. A few minutes later I  had a successful Flash dump. For the sake of clarity I did this twice,  in case of any errors with the first, but both were identical.

Now that we have the NAND-dump we can start the dive into getting the  firmware extracted and set about analysing the file structure  (hopefully).

First, I downloaded this, its awesome and a go-to tool for any NAND dumps.

https://bitbucket.org/jmichel/tools/src/default/

I did some Googling of the NAND FLASH chipset MT29F1G08ABADA. Even  though the chip is supposed to handle 1G of storage, this one didn’t and  the NAND dump from the reader came out at 138.4 MB.

I then looked up the chipset idcode.

https://www.usbdev.ru/databases/flashlist/flcbm93e98s98p98e/. This took quite a bit of searching to be honest and Dave (@tautology0) at PTP pointed me in the right direction for this!

Now that we’ve got that done, let’s see what Binwalk has to say:

There are a lot of interesting parts in that file. Lets first look at  the UBI part. We will use DD to extract that part from the dump.bin  file:

Nice! Let see if it’s really what it says it is:

Winning! Let’s keep going:

Excitement mode rising!

Now that we have the firmware successfully extracted it looks like we  have both the root filesystem and the Ewon Application file system.

I’m going to go over the Root file system first as I feel that will  give us a better understanding of how the Ewon system is built.

In all honesty, there was one file in “init.d” that was of interest  and nothing else. All this file did was start the Ewon service. No  additional users, no private keys for ssh, no password hashes. Boring.

Let’s get into the Ewon SquashFS.

The SquashFS file structure looks like its specifically for the Ewon environment and gets mounted at /opt/ewon/.

Most of the folders I extracted were empty but the bin/ and patch/ folder had valuable info. The patch/ folder:

These files in the patch/ folder were just for updates from the last  firmware update that was run, a few newer binaries, a few newer libs.

Again, most of that folder was quite boring, but it does show that  they regularly add linux-arm binaries and libraries in the updates for  the device.

After a tip from my colleague Andrew (@cybergibbons) I now always scan folder structures for any executable scripts (bash scripts):

These two files, although a basic bash script, only gave build information really, nothing that helps me right now.

Moving on to the bin/ folder:

A-ha! Now it’s getting interesting. We have a binary, some config  scripts and some bash scripts, oh and what looks to be a Java binary and  jar files as well. Let’s start with the config files first I think,  then get to the Ewon binary last.

The config files are quite simplistic, and as you can see, obviously for a set of different environments. At91sam9g (https://www.microchip.com/design-centers/32-bit-mpus/microprocessors/sam9), QEMU (probably for debug purposes) and RaspberryPi (again, debug and build purposes).

The configs are very similar with most of the values exactly the same, only the name and os dependant stuff has changed.

Here is a sample of the all_config.conf file:

Hmm, what does it say on the second line? “Common part to all  product” and lists the Flexy and Cosy Ewon products. Does that mean they  are the same product?

There are a lot of similarities, and some massive differences.

“designed to offer easy remote access, across the internet”

Also, Port 443 (HTTPS). How come the Cosy gets https but the Flexy doesn’t?!

I tried to run the Java binary and it errored out, I kind of expected this but it did give me info about a few more files:

Using JD-Gui (http://java-decompiler.github.io/) I decompiled the JAR file it was trying to reach:

I will probably dive into that some more later, but for now lets get stuck into the Ewon binary.

Reverse Engineering the Ewon Binary

As you can see from earlier screenshots there really wasn’t much that  we could see in the NAND Flash dump. I was expecting the web interface  to be setup using pre-built tools such as apache2, lighthttpd or nginx,  and for the FTP using vsftpd or similar. Fortunately for us (hackers)  when developers decide to roll their own variant of servers, they miss a  lot of security features or protections that other servers have battled  against.

When reverse engineering binaries it makes sense to  go straight for the strings.

I did a few basic grep’s of the strings in the binary, “password,  pass and private”. We will go through interesting ones later as I don’t  want to spoil the rest of the blog.

Throwing the binary into IDA Pro gave fantastic disassembly straight away. It wasn’t perfect, but it was more than usable:

One of the first things I wanted to investigate was the encryption on  the stored values (VPN keys, passwords) in the config files (that we  broke earlier) ?

After a while searching I came across this function cfgcrypt_DecryptCfgStr.

Here is an IDA layout graph of that function:

Those functions in a little more detail:

With the help from Dave the walking disassembler we were able to work  out that the header “#_1_” is dictating what encryption method was  used.

Also, in the actual Ewon comcfg.txt file there is a setting “CryptMode:1”. Makes sense to me.

At no point did we see any other variant of the header during this  time, however we believe that this is the reason for multiple  occurrences of the same key and IV hardcoded in the Ewon binary.

If the CryptMode is set to > 2, then the function will error out and set the value to 0.

I believe if its set to “#_0_” it will use “blowfish” and we have the  Key and IV that are used, but in this case, Its set to “#_1_”.

So what does “#_1_” do?

There is a function called “cfgcrypt_CheckCryptmode” that checks the  header of the encrypted string and returns a value depending on the  header value set, remember, it’s always been seen as “#_1_”.

However, this function is never called in the app. Strange. But there  is a similar code base in the “cfgcrypt_DecryptCfgStr” function.

This looks like a fair bit of code reuse across the board.

Essentially all the encrypt and decrypt functions come down to the “ctr_encrypt” function.

And the awesome, “ctr_decrypt” function:

I’m not going to show the specifics, but there is an XOR function contained in the “ctr_encrypt” code.

Firmware Reverse Engineering

One of the things that I was immediately stuck on from the very  beginning was the firmware. I could easily download the firmware from  the Ewon site:

But then looking at it in some detail it became apparent there was encryption in play:

It’s obvious the file has a header of sorts, due to the version  number, release date etc being in clear text, then it moves on to  garbage!

So back to the Ewon binary I went, looking for anything that referenced the firmware file name.

The first set of results came up with references to the SDcard:

So, it turns out that if the Firmware is present on the SD card it  will try and update. That’s pretty standard to be honest, but also, from  the user documentation, you can just upload the “ewonfwr.edf” file to  the FTP (if you have authentication) .The Ewon will then process the  firmware as necessary.

I hunted around for ages for something that would satisfy my interest  in the firmware file, until I stumbled upon a function called  “loem_UncryptFile” that looked like it was opening a file and calling  decrypt functionality!

Essentially this is what is happening; the Ewon binary detects if a  firmware file has been uploaded to the ftp server or put on the SD card,  loads the manifest (checks firmware version, date etc.) then decrypts  the firmware file and data.

The firmware encryption is based on Blowfish (this is referenced a  lot in the binary!) and the binary has the Key hardcoded in ?!

AWESOME!!!

So, back to the firmware file.

After looking at the data, I decided that the encrypted part started  at 0x0140h and carried on to the end of the file. I cut the file at  these points and tried to decrypt with the given key in the binary:

I was able to get a decent decrypt of the data, but it seemed like it  was only decrypting half of it. Really strange, but I was on the right  path.

Going back to the binary it seemed like the provided IV was for the  binary to create a backup of the firmware before upgrade, so where the  hell was this IV that we needed?

I reached out again to the magic channel and explained where I was  at. Dave, came back and said, if its not in the binary, it’s probably in  the firmware. This apparently is quite common.

Let’s go back to the firmware and look at the header. We are looking  for an 8 bit data stream that could be used. After removing the  encrypted junk, this is what I was left with, and the only obvious 8bit  data stream was the one highlighted at the bottom, followed by some null  bytes then the firmware encrypted data:

This was then fed back into the python script to see what happens:

We now have full decryption!

Let’s have a look at the file system and especially that “update.sh” file:

Plenty of files for our Spidey senses! But to be honest, it’s just  the Ewon /opt folder as it was from the NAND Flash Dump. So, nothing new  there!

The update.sh on the other hand! (again, a lot has been redacted):

NICE! Let’s see what this is doing in the binary, if anything:

I see potential command execution!

I changed the update.sh, remade the firmware, and reuploaded with a higher version number:

Annoyingly this didn’t work either. Looks like there is a CRC on the  file as we get a “riftp-Invalid checksum” error in the Ewon logs:

Let’s go back to the binary:

From what I can tell, there is a CRC16x2 check. Now I’m not 100% sure  if it’s on the encrypted blob in the firmware, OR, if it would be the  squashfs blob pre encryption.

Here is a comparison between two firmware versions. Obviously I made  the bits that are common across each file the same to reduce the amount  of differences:

The parts in red are the differences in the headers.

Looking a bit further into this I can see some similarities in the red bits between the files:

Let’s run some CRC checksums across the encrypted blog of the firmware file, see if we can see any relevant info in the header!

Ver 13_2s1 = 8.433 mb (Latest)  CRC16 (encrypted blob) = AF3C  CRC32 (encrypted blob) = F388096F  Ver 13_0s0 = 12,589 mb (Not Latest)  CRC16 (encrypted blob) = A65D  CRC32 (encrypted blob) = 259BC946

Wait a moment!

That is the CRC32 value of the encrypted blob! In the header file! At 0x00B0! Lets try changing that then:

Hmm NOPE. Something’s still not right.

So, it’s back to the Ewon binary. When searching through the binary I  came across this function that outlined some of the header parts:

“riftp_EDFHeaderPtr” is the start of the firmware file so let’s say “0”.

On line 144, it compares the (rfiftp_EDFHeaderPtr + 168) if its not equal too v10 (a CRC16 x 2 calculation).

At 0xA8 in the firmware, there is the CRC value. Nice! I also mapped  out a few other bits that I could from other bits of info in the binary:

One thing that confused me (and others) is the CRC calculation  function. We know (or at least think) that it is calculating the CRC16x2  on the encrypted blob from the binary code, but this is how the  calculation is done:

Dave and I decided to rewrite the CRC16x2 function to see where we could get to:

Although we are getting different results we were confident that we were very close at this point.

With this we also ran a brute force against the Firmware file, to see if any part matched the value stored in the header:

Unfortunately, this is where this ends.

We tried a ton of different CRC16 Checks, different poly’s, xor bits  etc, but nothing came close. We just could not match the value at 0xA8.

A great resource for this was pycrc (https://pycrc.org/models.html) it helped me so much.

Disclosure timeline

2019-01-29 – First contact with Ewon to report vulnerabilities

2019-02-04 – Confirmation of vulernabilities and estimated timeline provided by Ewon

2019-02-11 – Response from PTP that the timeline was too long, Ewon response as well

2019-05-23 – Final chase before disclosure

2019-05-28 – Response from Ewon, test firmware provided

2019-06-04 – Disclosure made by Ewon

2019-06-18 – PTP disclosed, updated FW 13.3s0 now available https://websupport.ewon.biz/support/product/manual-firmware-update/manual-firmware-download

We reported this to HMS Networks, the manufacturer of the Ewon Flexy on 29th January  2019. The disclosure process wasn’t completely pain free, but the end  result was that the vulnerability was acknowledged and fixed.

Initial responses to us from HMS were speedy, so a big ‘tick’ for responsiveness there.

We had some positive, constructive interactions with their security manager.

However, it took ~120 days for new firmware to be published to fix the flaw. Not so good.

Then things got a little weird when we stated that we were going to  publish this blog. They really didn’t like that – a veiled threat was  made by a senior person at HMS.

We pointed out that threats usually result in drawing more attention to vulnerabilities. The ‘Streisand Effect’?

Then things returned to normal and we had a further, really constructive discussion with their product marketing people.

They regained their ‘cool’ in our view when they published this, including credit for the finding:

https://websupport.ewon.biz/support/news/support/ewon-security-vulnerability

Mitigations and Recommendations

The really obvious stuff:

Don’t put your Ewon Flexy (or any ICS device) on the public internet. That’s asking for trouble!

Change the default credentials.

Segment your IT and OT networks with decent access controls, obvs.

Use the free of charge Talk2M security connectivity cloud with your Ewon devices.

From the vendor:

HMS have a secure remote access solution called Talk2M which should mitigate the issue also: https://www.ewon.biz/cloud-services/talk2m

We haven’t tested Talk2M, so can’t comment on its security, but any  additional layers of defence are usually a good thing. We’ll be taking a  look at the new firmware when I get some time.

New firmware is available which should fix the bug. It should be available here: https://websupport.ewon.biz/support/product/download-firmware/ewon-firmware-0.

From us:

Update your Ewon Flexy firmware and keep it up to date.

Do that with all of your industrial control systems. If you can’t  update an ICS (e.g. it’s too old / out of support / too critical to  touch) then consider building additional security controls around it.

Conclusion

I hate to leave the blog here and end it like this, but there WILL be a part two, I promise.

This has been one hell of a journey and I have personally learnt so  much, my brain actually hurts! I now have a love for IDA, but more and  more I’m moving to Ghidra (as its Free and just that little bit better  for some stuff)

The Ewon Flexy is a great device and this is not a blog to slam them  (too much). It was to see how far I could take my skills and where I  could grow and have a focus on security related stuff. This post doesn’t  have mega WTFBBQHAX but its my output, regardless of being great or  not.

If by any chance I do finish creating my own working firmware, I’m sure there is a ton more to publish another time.

A lot of this has now been disclosed to Ewon.

Thanks for reading and good luck.