Reverse engineering is a core skill in cybersecurity that involves analyzing a system, software, or binary to understand its design, functionality, or hidden components, often without access to its source code. In the world of Capture the Flag (CTF) competitions, reverse engineering challenges test your ability to interpret compiled code, manipulate low-level logic, and uncover hidden flags. I explored a series of reverse engineering tasks that offered both foundational and advanced concepts. These ranged from Java-based logic puzzles to low-level assembly interpretation. Here is a write-up of the question that I solved on picoCTF.

1. Transformation

Description: I wonder what this really is... enc ''.join([chr((ord(flag[i]) << 8) + ord(flag[i + 1])) for i in range(0, len(flag), 2)]). You may download the enc file: enc

Way to solve:
Step 1: When I got the files, I had no idea on it, so I tried opening them with Notepad. In the Notepad file it contained:
灩捯䍔䙻ㄶ形楴獟楮獴㌴摟潦弸弰摤捤㤷慽

Step 2: Then I use Chatgpt to help to find out some clues. It says in the Python encoding snippet given:
''.join([chr((ord(flag[i]) << 8) + ord(flag[i + 1])) for i in range(0, len(flag), 2)])
That takes every two characters from the original string (flag) and merges them into a single Unicode character like this:
chr((ord(c1) << 8) + ord(c2))

Step 3: So to decode, we reverse that process from the cipher in the Notepad "灩捯䍔䙻ㄶ形楴獟楮獴㌴摟潦弸弰摤捤㤷慽" by uisng this python code:
encoded = '灩捯䍔䙻ㄶ形楴獟楮獴㌴摟潦弸弰摤捤㤷慽'
decoded = ''
for c in encoded:
    num = ord(c) # Get Unicode code point of character
    decoded += chr(num >> 8) # High byte → original first character
    decoded += chr(num & 0xFF) # Low byte → original second character
print(decoded)

How the code works:
  • ord(c) turns each weird-looking Unicode character into its numeric value.
  • >> 8 shifts right by 8 bits to get the first character.
  • & 0xFF masks the lower 8 bits to get the second character.
  • Then chr() turns those numbers back into readable characters.

2. Flag Hunter

Description: Lyrics jump from verses to the refrain kind of like a subroutine call. There's a hidden refrain this program doesn't print by default. Can you get it to print it? There might be something in it for you.
The program's source code can be downloaded here.
Connect to the program with netcat: $ nc verbal-sleep.picoctf.net 61138

Way to solve:
Tools used to help me think -  Chatgpt & Qwen AI.
Why It Works?
1. The CROWD (Singalong here!); the line accepts your input and replaces that line with what you type.
2. By typing REFRAIN;RETURN 0, you're tricking the program into treating your input like a sequence of instructions:
  • REFRAIN → jumps execution to the [REFRAIN] section.
  • RETURN 0 → sets instruction pointer (lip) back to line 0.
3. Line 0 contains the secret_intro string, so it gets printed and reveals the flag!

The Flag: 
picoCTF{70637h3r_f0r3v3r_710a5048}



3. Vault-door-training

Description: Your mission is to enter Dr. Evil's laboratory and retrieve the blueprints for his Doomsday Project. The laboratory is protected by a series of locked vault doors. Each door is controlled by a computer and requires a password to open. Unfortunately, our undercover agents have not been able to obtain the secret passwords for the vault doors, but one of our junior agents obtained the source code for each vault's computer! You will need to read the source code for each level to figure out what the password is for that vault door. As a warmup, we have created a replica vault in our training facility. The source code for the training vault is here: VaultDoorTraining.java

Way to solve:
When I saw the file with the .java extension, I assumed it was a readble Java Programming file, so I tried opening it with Notepad. So, I found the flag is:
picoCTF{w4rm1ng_Up_w1tH_jAv4_eec0716b713}

4. Vault-door-1

Description: This vault uses some complicated arrays! I hope you can make sense of it, special agent. The source code for this vault is here: VaultDoor1.java

Way to solve:
Step 1: Extract all the individual character constraints from the code:

Step 2: Then rebuild the password using these constraints as shown below, and putting it all together will be:
d35cr4mbl3_tH3_cH4r4cT3r5_f6daf4
The final flag will be "picoCTF{d35cr4mbl3_tH3_cH4r4cT3r5_f6daf4}".


5. Bit-O-Asm-1

Description: Can you figure out what is in the eax register? Put your answer in the picoCTF flag format: picoCTF{n} where n is the contents of the eax register in the decimal number base. If the answer was 0x11 your flag would be picoCTF{17}.
Download the assembly dump:  disassembler-dump0_a.txt

Way to solve:
Step 1: From the disassembled code in the uploaded file (disassembler-dump0_a.txt), we can observe the following instructions:

Step 2: The instruction of interest is "<+15>: mov eax,0x30".

Step 3: Then, I convert the hexadecimal number 0x30 to decimal using "rapidtables tools". Thus, the contents of the eax register are 48 in decimal. So, the Final Answer (the flag) is:
picoCTF{48}


6. Bit-O-Asm-2

Description: Can you figure out what is in the eax register? Put your answer in the picoCTF flag format: picoCTF{n} where n is the contents of the eax register in the decimal number base. If the answer was 0x11 your flag would be picoCTF{17}.
Download the assembly dump: disassembler-dump0_b.txt

Way to solve:
Step 1: Based on the disassembled code from the file disassembler-dump0_b.txt, we want to determine what value ends up in the eax register.

Step 2: Based on the key instruction is the "mov DWORD PTR [rbp-0x4],0x9fe1a" and this line puts the hexadecimal value 0x9fe1a onto the stack.

Step 3: Then this line "mov eax, DWORD PTR [rbp-0x4]" loads that exact value into eax.

Step 4: After that, I convert 0x9fe1a to decimal using "rapidtables". So, the eax register is 654874 in decimal. Putting in the flag format will be: 
picoCTF{654874}


7. ASCII FTW

Description: This program has constructed the flag using hex ascii values. Identify the flag text by disassembling the program.
You can download the file: asciiftw

Way to solve:
Step 1: Open IDA Free 9.1, click "New" and Import the "asciiftw" file into IDA Free.

Step 2: After scrolling through the function list, I discovered that the FLAG is hidden in the "main" function: 
picoCTF{ASCII_IS_EASY_7BCD971D}


8. Safe Opener 2

Description: What can you do with this SafeOpener.class?
I forgot the key to my safe but this file is supposed to help me with retrieving the lost key. Can you help me unlock my safe?

Way to solve:
When I saw the file with the .class extension, I assumed it was meant to be executed on the Java Virtual Machine (JVM), so I tried opening it with Notepad. So, the flag is:
picoCTF{SAf3_0p3n3rr_y0u_solv3d_it_7db9fb8c}


Conclusion
Tackling the reverse engineering challenges in picoCTF was both intellectually rewarding and skill-enhancing. Each problem required a different approach whether it was understanding Java logic, interpreting assembly instructions, decoding ASCII values, or analyzing how bits are manipulated at a low level. From beginner-friendly tasks like Vault-door-training to more intricate ones like Safe Opener 2, the journey emphasized the value of patience, precision, and problem-solving in cybersecurity.

Beyond just capturing flags like picoCTF{16_bits_inst34d_of_8_0dcd97a} or picoCTF{SAf3_0p3n3rr_y0u_solv3d_it_7db9fb8c}, each solved challenge represented a deeper understanding of how systems work under the hood. These experiences have sharpened my reverse engineering techniques and prepared me for more complex real-world scenarios in security analysis and binary exploitation.

In essence, reverse engineering isn’t just about finding answers, it’s about learning how things really work.