The Distributional Hypothesis: semantic models in theory and practice

Posted on by
Reply

This was the final project for the Data Semantics course at university – A report on distributional semantics and Latent Semantic Analysis.

Here is the nicely-formatted pdf version (with references).

What is the Distributional Hypothesis

When it comes to Distributional Semantics and the Distributional Hypothesis, the slogan is often “You shall know a word by the company it keeps” (J.R. Firth).

The idea of the Distributional Hypothesis is that the distribution of words in a text holds a relationship with their corresponding meanings. More specifically, the more semantically similar two words are, the more they will tend to show up in similar contexts and with similar distributions. Stating the idea the other way round may be helpful: given two morphemes with different semantical meaning, their distribution is likely to be different.

For example, fire and dog are two words unrelated in their meaning, and in fact they are not often used in the same sentence. On the other hand, the words dog and cat are sometimes seen together, so they may share some aspect of meaning.

Mimicking the way children learn, Distributional Semantics relies on huge text corpora, the parsing of which would allow to gather enough information about words distribution to make some inference. These corpora are treated with statistical analysis techniques and linear algebra methods to extract information. This is similar to the way humans learn to use words: by seeing how they are used (i.e. coming across several examples in which a specific word is used).

The fundamental difference between human learning and the learning a distributional semantic algorithm could achieve is mostly related to the fact that humans have a concrete, practical experience to rely on. This allows them not only to learn the usage of a word, but to eventually understand its meaning. However, the way word meaning is inferred is still an open research problem in psychology and cognitive science.

Continue reading →

Brute force a crackme file password with Python

Posted on by
Reply

I was to reverse a file for a challenge, MD5 hash 85c9feed0cb0f240a62b1e50d1ab0419.

The challenge was called mio cuggino, purposefully misspelled with two g letters. It asks for three numbers. The challenge led me to a brute force of the password with a Python script, learning how to interact with a subprocess stdin and stdout (SKIP to next section if you don’t care about context but only want the code).

Looking at the assembly with Radare, the first thing it does is to check that the numbers are non-negative and in increasing order. In details, it checks that:

  1. exactly three inputs have been provided;
  2. the first two are non-negative;
  3. the third is bigger than the second;
  4. the second is bigger than the first;
  5. the third is non-negative.

Very good, so the input pattern is three non-negative integers in increasing order. Fine. No clue about what those numbers should be though, yet.

Scroll the assembly just enough to unravel the magic.

A (pointer to) string is loaded into ebx, which contains the following Italian sentence:

Mi ha detto mio cuggino che una volta e’ stato co’ una che poi gli ha scritto sullo specchio benvenuto nell’AIDS, mio cuggino mio cuggino

The assembly basically takes the characters in the string that correspond to the first and second input (for ex, 0 as first input would map to the first char, M) and checks whether they are equal. If this is not satisfied, a Nope message is shown and the binary returns.

If this is satisfied, the same check is repeated with the third input (with the first one, although this doesn’t matter). If this is satisfied as well, a tricky sub.puts_640 function is called (with 5 inputs), and a Uhm message is shown.

Going to looking into that routine is absolutely useless as it’s completely unreadable, and even makes a bunch of additional calls that are further jumbled.

Continue reading →

The one time pad and the many time pad vulnerability

Posted on by
Reply

The scope of this article is to present the one time pad cipher method and its biggest vulnerability: that of the many time pad.

The one time pad: what it is and how it works

The one time pad is the archetype of the idea of stream cipher. It’s very simple: if you want to make a message unintelligible to an eavesdropper, just change each character of the original message in a way that you can revert, but that looks random to another person.

The way the one time pad works is the following. Suppose \mathcal{M} is the clear-text message you would like to send securely, of length |\mathcal{M}| = s. First, you need to generate a string \mathcal{K} of equal length |\mathcal{K}| = s. Then, you can obtain a cipher-text version of your message by computing the bitwise XOR of the two strings:

    \[\mathcal{M} \oplus \mathcal{K}\]

The best thing is that decoding is just the same as encoding, as the XOR operator has the property that \mathcal{X} \oplus \mathcal{X} = 0 \ \forall X (and that \mathcal{X} \oplus 0 = \mathcal{X} \ \forall \mathcal{X}). The only difference is that the cipher-text is involved in the XOR, rather than the clear-text:

    \[\mathcal{C} \oplus \mathcal{K} = \mathcal{M} \oplus \mathcal{K} \oplus \mathcal{K} = \mathcal{M} \oplus 0 = \mathcal{M}\]

Below is an example of the one time pad encoding achieved with Python, with a made-up pad string.

In the first section, result holds the XOR result. In the second part, the result and one_time_pad variables are XORed together to obtain the original plain-text message again.

It is not difficult to realize that the whole strength of the algorithm lies in the \mathcal{K} pad. Of course, as an attacker, if you can obtain \mathcal{K} in some way, then it is not difficult to get the clear-text message from the ciphered one as well.

Continue reading →

Getting started with Binary reverse engineering: an example

Posted on by
Reply

For a challenge in a university security class, I was given this file to crack: reverse1. I started with reverse0, which was considerably easier than the second one. In this post I will briefly explain how I tackled reverse1. I provided the files so you can you try on your own and then came back for hints if you are stuck! If you are new to this business, as I relatively am, I advise you to start from reverse0 and crack that first.

Hashes of reverse1 file: 
MD5 – c22c985acb7ca0f373b7279138213158
SHA256 – cd56541a75657630a2a0c23724e55f70e7f4f77300faf18e8228cd2cffe8248e

Disassembling and hoping for the best

The first thing I did was to disassemble the file with Radare to have a look at the code.

The assembly is quite jumbled up, and difficult to analyse all together. A quick look tells us that trying to crack the file just by reversing the assembly is no easy task, and actually a silly idea to begin with. There’s a cycle after the password is read from standard input, then some other instructions, then another cycle… it’s difficult to get what is going on…

Instead, let’s seek the Bad password print section, and see what should happen for the code to jump there. If we are lucky enough, we may find a bunch of final checks that will send over to the Bad password section. If we can find those, we may then look at those bits of assembly to understand how to avoid going there.

Scroll down enough, and down at the bottom I can see the Bad password part, starting at 0x080484f0.

Radare helps in showing two different arrows going into this address. The related comparisons are the following:

Continue reading →

Base conversion in Ubuntu (decimal to binary)

Posted on by
Reply

Need to convert a base 10 integer in a base 2 one? Or, at any rate, convert a number from one numeration system to another? In Ubuntu, the bc utility already integrates these features. It is usually already installed, so you don’t have to anything special.

Simply run bc, and enter the following commands:

Then, all subsequent number inputs will be simply converted to their base-2 representation.

If you want to get a conversion straight ahead, without going through the opening of bc, just enter the following from a terminal:

which will convert the number 123 from base 10 to base 2.

Of course, 2 and 10 can be replaced with any other possible base!

Does C++ delete operator really free memory?

Posted on by
Reply

Well, I have been wondering about this for quite a while now, and I have tried to run some tests to better understand what’s going on under the hood. The standard answer is that after you call delete you should not expect anything good from accessing that memory spot. However, this did not seem enough to me. What is it really happening when calling delete(ptr)? Even though there no standard behavior, what could happen, anyway? Here’s what I’ve found. I’m using g++ on Ubuntu 16.04, so this may play a role in the results.

What I first expected when using the delete operator was that the freed memory would be handed back to the system for usage in other processes. Let me say this does not happen under any of the circumstances I have tried.

Memory released with delete still seem to be allocated to the program it first allocated it with new. I have tried, and there is no memory usage decrease after calling delete. I had a software which allocated around 30MB of lists through new calls, and then released them with subsequent delete calls. What happened is that, looking at the System monitor while the program was running, even a long sleep after the delete calls, memory consumption my the program was the same. No decrease! This means that delete does not release memory to the system.

In fact, it looks like memory allocated by a program is his forever! However, the point is that, if deallocated, memory can be used again by the same program without having to allocate any more. I tried to allocate 15MB, freeing them, and then allocating another 15MB of data after, and the program never used 30MB. System monitor always showed it around 15MB. What I did, in respect to the previous test, was just to change the order in which things happened: half allocation, half deallocation, other half of allocation.

So, apparently memory used by a program can increase, but never shrink. Continue reading →

Looking for help with Post Pay Counter development

Posted on by
Reply

Post Pay Counter provides an easy way to manage authors payments on a WordPress website. Three years ago was completely redesigned to meet the demands of its growth, and shortly after Post Pay Counter 2.0 was released, a PRO version followed as well. Besides that, there are now 15 addons that further extend Post Pay Counter features.

The project has always been 100% managed and developed by me, but at this point I would gladly use some help. The project started for a niche of websites, but now more and more websites that need this kind of service are emerging, and Post Pay Counter has basically no competitors. The free version has an average of 4.7/5 stars on WordPress.org. It is nearing 50 thousands downloads, and there are several satisfied users who turn PRO, and could never go back.

I have had stuff on my to-do list for years, and more is getting on it. There 14 issues on GitHub now, but the list will just grow longer as I move all my private notes over there into public issues. Several addons would need new features, new addons need to be developed and the whole ecosystem would gladly welcome a breath of fresh air coming from people other than myself. There are several things that need to be done on the website as well, and I would gladly get my hand a bit more free to work on those.

So, how does this relate to you?

Whether you are a hard core, experienced developer, or just starting out with WordPress development, I’d love to hear from you! I’d like this project to involve more people than just me and, at this point, it looks like there is something worth getting involved into. Do get in touch for anything, even if I have not explicitly mentioned it, if you feel it could be useful/interesting for the project!

Getting involved in an open source project already having a considerable user base is a very good way to learn your way around WordPress coding, make some experience or just contribute to a free project and get credit for your work. There’s really a lot to do: from tutorial writing to marketing and promoting; from bug fixing to feature implementing.

I’m not yet in the condition claim I will pay anybody for their work on the project – however, if you want to jump into addon creation, or if you want to contribute to the project with any product you could make, then brilliant, let me know! Paid addons sold on the PPC website will give you 75% of the revenue they generate. Of course you are also free not to market them in the official site. Several of the current GitHub issues are actually just ideas for future addons, so you have a place to start from!

Tips and advice on being a freelance in Information Technology

Posted on by
Reply

What follows is Javier Silva’s interview to me. The interview is mostly focused on how what it is like to be a freelance in the IT field and how to start as a programmer (and how that may evolve into a business). It was first published on his blog in Spanish. He also did a small review of my Post Pay Counter plugin.

Please, Introduce yourself!

I’m Stefano from Italy. I study mathematics, but there are very few things I am not interested into. I am a web developer, a walker, a reader, and an amateur photographer. Those are just the things that take most of my time, but don’t believe I don’t do anything else!

You work as web developer… where did you study it? or how did you learn it?

I’ve never taken any classes on web developing or on any IT subject. I have just always been into computers and technology, and by reading/replying on forums, experimenting and lots of tutorials I have learnt all that I know. When I was 12, at school we were covering divisors, prime numbers, factorization, and the like, and… you know, homework was boring as hell! I was just tired of having to figure out whether a number was prime, or what its divisors were, so I wrote a little script that did it for me. That evolved in writing more complex software and slowly learning to write decent-quality code.

Homework was boring as hell! I was just tired of having to figure out whether a number was prime, or what its divisors were, so I wrote a little script that did it for me.

There are a lot of resources out there for people willing to learn. I believe the key is to play around and experiment. And, as always, a lot of practice is important.

What is your point of view about the “programming career”? Is it a competitive profession?

I believe it definitely is. Unless you aim at working within your own city, meeting customers face to face, you really face a lot of competition. If I am hiring someone to develop something for me, and I don’t require they live in my same city, then I can pick anyone from all over the world. And good luck to convince me that you are the best developer, and that I really want you.

Continue reading →

WordPress function url_to_postid not working with Anspress

Posted on by
2

With the Anspress theme/plugin, using the WordPress native function url_to_postid() on a question permalink returns the page ID of the base page. For example,

would return the page ID of the /questions page.

To get the WP question ID of this-is-the-question, use the following:

(using get_post_types() instead of ‘questions’ allows the call to work even with permalinks of non-question contents.)

bbPress – Anonymous Subscriptions

Posted on by
4

This add-on plugin for bbPress will allow anonymous users to subscribe to topics and get email notifications when a new reply is posted. The notification email includes an unsubscribe link.

bbPress notifications will keep to go out to registered users, this plugin only extends the thing to anonymous posters as well!

Download (it’s free!)

bbPress - Anonymous Subscriptions

A case example with >100% subscription rate

This is vital for support forums, for example. On Post Pay Counter support forums, I did not want customers to sign-up: I wanted them to be able to request support in a matter of minutes, without any hassle. I liked the idea of “enter your name and email and you’re done!” But I also felt like they needed to be notified when someone replied to help. It was not compulsory, of course, but I would have wanted it as a customer.

Continue reading →