Just a simple spam overview…

Here is an overview of my recent spambox:

More spam

And yeah, it’s time to complain about all my spam again. And what you’re seeing is what I see in my spambox. About 35 different messages received within less than 12 hours. Fortunately, they’re this many because they have been sent to multiple email addresses. Those addresses are all aliases for my mailbox, though.

The interesting one is the one about eFax. I did use eFax once, many years ago when I was working on software for PBX systems. (Has something to do with phones.) So those messages could be true if I would receive them on the proper alias. I did not, so they’re fake. Anything sent to the wrong alias is fake, unless proven otherwise. Also, I am unfamiliar with the phone number in the header and it refers to the British version of eFax, while I happened to use the Dutch version. That’s enough to tell me that these are really, really fake. It’s even funnier when you check out the link, which goes to eliteom.com which happens to be a gun sales website. So, their website has been hacked.

Still, some further investigations direct me to this IP address: By using RobTex I end up at a login site for some shared hosting website running on ZPanel. Still doesn’t tell me much. It would seem the spammer has set up his own host somewhere but the link I found goes directly to a specific page, without a domain name. So, someone is using ZPanel and had their system hacked too. RobTex tells me the ZPanel host is registered by someone in Australia and hosted on servers in the USA. I might be wrong, though, but it seems that there are many layers to peel here.

Moving on, I see spam for fake medicines, a warning about a dangerous parasite that’s probably fake too, a strange invoice that’s clearly fake, some shaving solution, a few naughty messages that just contain links and are hoping I’m curious enough to click and a few more weird messages.

One type of spam is for Ruby Palace, a casino website that seems to hop around on the Internet. According to internet rumours, the registrar for Ruby Palace is located in India where they have no anti-spam laws so they can keep supporting this spammer. Again, RobTex is quite helpful here, telling me that the registrar operates in several countries but not India. So that rumour might not be true. It seems to be Australian, though. One thing to remember, though. Casino spam is offering you great profits, but they make even bigger profits from you spending your money there.

One strange email I received is from a former colleague which was sent to my LinkedIn address. That is, my new LinkedIn address because LinkedIn had already leaked my old one. A direct message to that account is very suspicious in my opinion so I’ve marked it as spam. I’ve anonymized the header to protect my and her privacy a bit. I wonder if Liz really sent this to me, although it does make some sense considering her current employer.

The message itself seems to want to exchange business referrals between members. This is done through a website called referralkey.com which seems a bit spamlike to me. Their unsubscribe page includes ads and they don’t appear to be very reliable. Still, I will just unsubscribe my LinkedIn address and if I continue to receive more spam om my LinkedIn account then I will know that LinkedIn has been hacked again

A few more spam messages, trying to sell me a funeral insurance or give me some interesting dating options. Interestingly enough, I get a lot of spam on an account I used for instantcheckmate.com and that shows you how risky it can be to just subscribe for any website. The use of aliases when subscribing is definitely good advice! Register your own domain, get a Google Apps account for one user and let Google manage your mailbox, including the many aliases you like to create. (Or pick another solution to manage lots of aliases.)

Funny… While writing this post I received two more spam messages…

The Celebrity Hacks…

Marianne SelfieFor those who are still hiding in some cave, there’s something going on called “The Fappening“. It’s a celebrity scandal that involve ‘selfies‘ taken by some famous people, most of them female and in various stages between clothed and fully nude. People claim that these celebrities should not have taken nude selfies to begin with but I strongly disagree with that opinion. People should just have respect for the privacy of others and this includes the privacy of celebrities.

Unfortunately, we live in a society where the price of a used tampon can be hundreds of dollars worth, if used by someone very famous, like Miley Cyrus or Jennifer Anniston. Preferably with a certificate explaining how it was retrieved and when it was used. Just no respect for their privacy, since people can earn lots of money with it. And that’s also true with these celebrity hacks.

To make matters worse, there are plenty of people out there who will make fake pictures of those celebrities. Some are very obviously fake. Others use a look-alike model to make the photo more real. But in this case, the photo’s seem to be mostly real pictures of those celebrities with maybe a few fake ones to make it appear an even bigger hack.

Now, telling celebrities to stop taking nudies (nude selfies) is like telling people to not use their right of free speech. It would violate their own freedom of expression. Why would the girl next-door be allowed to take nudies while Victoria Justice should not do so? Well, the girl next-door is not as interesting as a target than someone famous. Besides, thousands of girls (and boys) have ended up as victims of the same crime because they shared the nudie with their lover, and that lover would then publish the nudie once the relation ends. (Something called revenge porn.) But those pictures will maybe draw attention from 10 to 20 other viewers while a nudie of Ariana Grande would draw the attention of thousands, maybe even millions, of people.

Basically, exposing nudies of other people without their permission should be considered a criminal offense, almost as criminal as rape. (And as a copyright violation, but that’s generally a misdemeanor and often something for the Civil Court, not the Criminal Court.) So if your ex-lover uploads your nudie to a revenge porn site, he (or she) should be arrested and punished for it. And sites that allow this kind of revenge porn should be considered to be criminal organisations and anyone visiting them or uploading pictures to those sites should face criminal charges too. Harsh? Yes, but our modern society seems to require such hard actions against these offenders. Besides, there are plenty of legal ways to publish nudes. You just need the consent of the model, and plenty of models are willing to pose for such images.

Problem with the Fappening is that no one seems to know how the hacker(s) gained access to these selfies, although it is assumed that iCloud from Apple isn’t secure enough. Most celebrities seem to favour Apple products over Android products and all investigations seem to focus on the iCloud. Since the iPhone camera can synchronize any picture it takes with the cloud, it also explains how those photo’s ended up on the Internet in the first place. Thus, if an iCloud account is hacked, those photo’s can start roaming all over the Internet.

One cause of this leak is the insecurity of the iCloud. It seems as if the photos are stored without any form of encryption on the iCloud servers. I’m not 100% sure about that, but Apple has a good reason to not use encryption: decryption takes time and thus slows down the system. But I don’t know why the phone itself cannot do the encryption or decryption of those pictures.

Basically, the iCloud account would contain a private key and every device that is used to connect to this account will receive a private key, after the user requests for it. When this happens, an email should also be sent to the user account to warn her (or him) that a new public key has been generated. Thus, if a hacker gains access to the account, he will need the public key, which will warn the user.

This public key would then be used when the user is uploading or downloading photos. Thus, the encryption happens in the phone, not in the cloud. So if anyone has access to the cloud data, they still won’t be able to see the pictures. This will generate much more privacy for the user. Besides, the encryption could also happen within iTunes so the user can synchronize with her computer. And all data the user has should be encrypted before the iCloud receives it.

That would be an important security upgrade by Apple, but users should also take some steps to secure themselves. Perez Hilton has an expert naming a few options but I don’t fully agree with those. To start with, he advises that every celebrity start using a new email address. That’s just wrong, because they can’t throw away the old one.

A better option for celebrities is to register their own domain name. Most of them already have one anyways. Use this domain to generate a bunch of email aliases and use each alias for any specific contact or account that you have. For example, your address to register your iPhone would be apple.phone@example.com while your registration for Amazon would be amazon.com@example.com. Or, if you want to keep up some bookkeeping, just use random codes for every contact. For example, bb001@example.com for Apple and bb002@example.com for Amazon. (In which case you could apply filters that will label incoming emails based on those aliases.)

Maintaining your email aliases this way is quite easy with e.g. the services of Google. There are plenty of alternatives too, but I just like Google so I use them as example. All you need to do is to create one user account and connect it to your domain. If you want to give your lover, your mom or your children with their own accounts then just add a few more user accounts. And if you have some expert handling your ICT requirements, then let the expert have his own user account too. Your own account should be set up as a catch-all address and you could set it up to filter all incoming emails based on the alias that receives it. You will have to add each address in the settings of your email application too, so you can respond with that alias too, instead of your regular mail address.

Another advice is to strengthen your security questions. Please don’t do that. Security questions are just a crappy way to make people think they’re secure while it just opens an extra attack vector to your account. It’s easier to just answer these questions with about 20 random letters and quickly forget about them. Just make sure you don’t forget your password.

The third advise is similar to my advice of using a whole domain. The difference is that my option is still a single account and all your mail will be received by that single account. Thus, you can create a lot of aliases and still support them with ease. Creating multiple email accounts will become troublesome once you have over 20 of those accounts. Basically, it means that you have to check 20 accounts every day instead of just one…

And the last advice sucks if you’re a celebrity, but I have to agree with it. Still, if you want to be famous, you need people to talk about you so some private information needs to leak out. Or you should get some reputation as bad Diva or whatever, walking on stage in a dress made of meat, having your nipple pierced and filmed to show on YouTube or start dating a homeless person with a criminal record just to draw attention. (Sitting naked on a wrecking ball seems to help too…)

But other information, like the name of your dog, the names of your family members and even where you’re taking your date out for dinner are things that are better kept private. You can still have a big influence on the Internet without exposing much of your private information. Your fans will continue to follow you no matter how crazy you act online.

Do keep in mind that you need to uphold a large fan base if you want to continue to profit from your fame. Having these nudies exposed to the public is horrific, but it is also an opportunity to get more fame. For example, Paris Hilton made a sex video of herself and her lover that got exposed to the Internet. Before that happened, barely anyone knew her. But the attention of this scandal did increase her popularity and provided her a lot of new opportunities. Some of the actresses who have become victims are already trying to spin the event into new opportunities for the future. They are trying to still get something positive about all this negative attention.

Besides, beneath our clothes, we’re all naked. We all are sexual beings who often do silly things that are better left to our own private information. You, the victimized celebrities have done nothing wrong. The ones who took those pictures from your private accounts are the real criminals. They are the ones to blame, they are the ones who need to be punished for the whole thing…

You should buy stock in ‘Inspiration Mining Corp’. (NOT!)

Well, it’s about time that I start to nag about spam again. This time someone really would like people to go trade on the financial market. Preferably in mining company called “Inspiration Mining Corp” or simply IRMGF. And yes, this is very important since the spammer wants to make a huge profit from selling his shares to you so the price needs to go up fast.

IRMGF is a so-called penny stock. This means the price of it is so low, it only costs a penny to own a piece of the company. Basically, it’s almost worthless but for some it’s still interesting to trade in. Why? Because if the price goes up just a single penny, those investors will have doubled their investment! So if you buy 100,000 in stock for a penny each and manage to increase the price by just 2 cents, then your $10,000 investment will now be worth $30,000. Which is not bad for a reasonable small investment.

Problem is, with stock you never know if the price will go up or down. So it is interesting to try to manipulate the value of penny stock in all kinds of ways. The simplest way is by making people believe how ‘cheap’ the stock actually is, hoping people will start buying. And sure, some of those buyers will pay about the same as the spammers do, about a penny per stock. But this will also start to increase the value of the same stock, since people want to buy it.

But when the stock price has doubled or tripled, the spammer will immediately sell his stock to those who still continue to buy it. The spammer will earn a nice profit and has almost no risk of getting caught. (Unless it can be proven that he was responsible for the spam.) Since lots of people will buy and sell penny stock it’s just not easy to find the person who has spammed among all those suspects.

So, lets take a quick look at the IRMGF:US stock here at Bloomberg. The price has moved between 4 cents and 16 cents during the whole year. If you bought stock in December 2013 and sold it again in May/June 2014 then you would probably quadruple your investment. Not bad for just a few months waiting. But now the price has dropped to below 7 cents per stock so it is interesting to start buying again, hoping the price will go up again.

Then again, this is how the stock market works. You buy stock as an investment to keep your money safe. If things go well, you should make a small profit on your investment. If not, you should sell before the stock becomes worthless. Most people with money don’t really buy stock to make profits but to make sure their money is reasonable safe. But they will have to check the market continuously to make sure their stocks are stable enough. This is a bit time-consuming and many investors will use computers to watch the stock market for them. And probably hire a financial advisor who does nothing else but trade in stock to keep the invested value stable.

Penny stock is reasonable unreliable because the low price suggests that the company behind the stock isn’t doing so well. If they have to file for a chapter 11 because the company is dead broke, your stock will become worthless. You’d rather invest in something more stable and reliable and start selling it when you expect its value to drop.

Now, why do I start about this spam? Well, simple. For the last 5 days I’ve received hundreds of spam messages on various of my email aliases. This is practical because it tells me which companies have shared my mail address with those spammers. Adobe and LinkedIn are, of course, the usual suspects because their databases have been hacked. As a result, I still receive lots of spam on those aliases. Another company that apparently got hacked is SmithMicro where I purchased my Poser software for the CGI models.

I also noticed strange addresses like waterside__9.jpga@example.com and tayen-usenet-a@example.com which I never even created. I don’t know why those spammers are using those aliases but maybe the person who owned the specific domain before I did used those accounts.

What do the messages look like? Well, like this:

Spam 01

Spam 02

And there have been more variations of this spam.

A few things are easily noticeable. First of all the spelling in both messages is just plain bad. They included other characters in the stock name, spaces are missing in some places, “mining” is spelled wrong and a few more things. This is done on purpose to get around spam filters, although it just doesn’t seem to work with the Google spam filters.

The sender happens to be fake, though. All spammers will use fake email accounts, often collected from their own spam lists to make it seem legitimate. So responding to the sender or anyone else in the email is useless. You’d just be harassing some other innocent person. Yet many people do think it helps so they respond to complain about the spam. Or report the account to their ISP, accusing them of spam. Most providers are smart enough to recognise this, though. They won’t take actions against the fake sender because they know he’s just a victim too.

The email also has several links to make it look more legitimate. But in this case, even those links are fake. They are a combination of the email address (the part before the @ sign) and some gibberish with .com or .org after it to generate a domain. It also includes a path on the fake domain that looks legit but since the domain is fake, the whole link is fake. This spammer just doesn’t want anything that would link back to him.

So, would the IP address in the email header be any helpful? Unfortunately, not much. The computer behind that IP address is most likely part of a bigger botnet. A machine infected by malware that the spammer can use to send his spam. You could report the IP address to the related provider and hope the provider will take the specific user off the Internet until he has cleaned his computer but in general, that’s not going to happen.

Thus, these spam messages are hard to stop. The spammer is difficult to trace since a lot of people will be trading in this penny stock. Some investors might even consider investing in it since they expect the price to go up even further because of this spam. As I said, the price has been over 16 cents at one point and now the price is 11 cents. If it continues to go up, they could still make profit from it.

Nothing in the email will lead back to the original spammer, although it will expose the computers that are part of the botnet. Those computers should be taken offline but doing so is not that easy. To make it more complex, those IP numbers could just be connected to a router and a lot of computers might be behind this router. There could even be an open WiFi connection in the router that happens to be misused by someone else in the area. (Who could be innocent too, but his computer could be infected.)

Penny Stock Spam is a very difficult one to fight against because the spammer can hide himself very well. He doesn’t have to add a link to his webshop or to some infected website that could be closed within a day after it has been reported. There’s almost no trace to the spammer either. The only thing that helps against this kind of spam is to not buy the stock, not even if you’re an experienced investor and still expect some profit. You will most likely lose money on those transactions because you’re just paying the spammer himself.

But if you’re lucky, a bigger moron will still buy the stock and give you some profit. And that’s the worst part of this spam. It’s not just the spammer who will profit but some investors might also have a smaller profit from it. As I said, if it goes up just a cent, they would have made a huge profit already.

An example of bad development…

I recently received an email from a company that’s doing questionnaires. And well, I subscribed to this and did some of their questionnaires before, so I wanted to do this new one too. Unfortunately, the page loaded quite slow, only to return a very nasty error message. A message that told me that this organisation is using amateurs for developers and administrators.

Let me be clear about one thing: errors will happen. Every developer should expect weird things to happen, but this case is not an error but evidence of amateurs. So, let’s start with analyzing the message…

Server Error in ‘/’ Application.

Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.

So, what’s wrong with this? Users should never see these messages! When you develop in ASP.NET you can just tell the system to just keep these error messages only when the user is connected locally. A remote user should see a much simpler message.

This is something the administrator of the website should have known, and checked. He did not. By failing at this simple configuration setting the organisation is leaking some sensitive information about their website. Information that’s enough for me to convince they’re amateurs.

This error is also a quite common error message. Basically, it’s telling me that the system is having too many database connection open. One common cause for this error is when the code fails to close a connection after opening them. Keep that in mind, because I will show that this is what caused the error…

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

This is a standard follow-up message. The fact that users of the site would see this stack trace too is just bad.

Exception Details: System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.

A timeout error. A reference to the connection pool and the max pool size. This already indicates that there are more connections are opened than closed and the system can’t handle that correctly. There are frameworks for .NET that are better suited for this to prevent these kinds of errors. That’s because these errors happened to be very common with ASP.NET applications. And with generic database applications written in .NET.

Basically, the top of the error message is just repeating itself. Blame Microsoft for that since this is a generic message from ASP.NET itself. Developers can change the way it looks but that’s not very common. Actually, developers should prevent users from seeing these kinds of messages to begin with. Preferably, the error should be caught by an exception handler which would write it to a log file or database and send an alert out to the administrator.

Considering that I received this error on a Friday afternoon, I bet the developer and administrators are already back home, watching television like I do now. Law & Order is just on…

Source Error:

Line 1578:
Line 1579:        cmSQL = New SqlCommand(strSQL, cnSQLconfig)
Line 1580:        cnSQLconfig.Open()
Line 1581:
Line 1582:        Try

This is interesting… The use of SqlCommand is a bit old-fashioned. Modern developers would have switched to e.g. the Entity Framework or other, more modern solution for database access. But the developers of this site are just connecting to the database in code, probably to execute a query and collect the data and then should close the connection again. The developers are clearly using ADO.NET for this site. And I can’t help but wonder why. They could have used more modern techniques instead. But probably they just need to keep up an existing site and aren’t they allowed to use more modern solutions.

But it seems to me that closing the database is not going to happen here. There are too many connections already open thus this red line of code fails. The code has an existing connection called cnSQLConfig which is already open. It then tries to open and execute an SQL command that fails. Unfortunately, opening that command happens outside a try-except block and if this fails, it is very likely that the connection won’t be closed either.

If this happens once or twice, then it still would not be a big problem. The connection pool is big enough. But here it just happened too often.

Another problem is that the ADO.NET technique used here is also vulnerable for SQL Injection. This would also be a good reason to use a different framework for database access. It could still be that they’re using secure code to protect against this but what I see here doesn’t give me much confidence.

Source File: E:\wwwroot\beta.example.com\index.aspx.vb    Line: 1580

A few interesting, other facts. First of all, the code was written in Visual Basic. That was already clear from the code but this just confirms it. Personally, I prefer C# over Visual Basic, even though I’ve developed in both myself. And in a few other languages. Language should not matter much, especially with .NET, but C# is often considered more professional than BASIC. (Because the ‘B’ in BASIC stands for ‘Beginners’.)

Second of all, this piece of code has over 1580 lines of code. I don’t know what the rest of the code is doing but it’s probably a lot of code. Again, this is an old-fashioned way of software development. Nowadays, you see more usage of frameworks that allow developers to write a lot less code. This makes code more readable. Even in a main index of a web site, the amount of code should be reasonable low. You can use views to display the pages, models to handle the data and controllers to connect both.

Yes, that’s Model-View-Controller, or MVC. A technique that’s practical in reducing the amount of code, if used well enough.

And one more thing is strange. While I replaced the name of the site with ‘example.com’, I kept the word ‘beta’ in front of it. I, a user, am using a beta-version of their website! That’s bad. Users should not be used as testers because it will scare them off when things go wrong. Like in this case, where the error might even last the whole weekend because developers and administrators are at home, enjoying their weekend.

Never let users use your beta versions! That’s what testers are for. You can ask users to become testers, but then users know they can expect errors like these.

Stack Trace:

[InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.]
   System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection) +4863482
   System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory) +117
   System.Data.SqlClient.SqlConnection.Open() +122
   _Default.XmlLangCountry(String FileName) in E:\wwwroot\beta.example.com\index.aspx.vb:1580
   _Default.selectCountry() in E:\wwwroot\beta.example.com\index.aspx.vb:1706
   _Default.Page_Load(Object sender, EventArgs e) in E:\wwwroot\beta.example.com\index.aspx.vb:251
   System.Web.UI.Control.OnLoad(EventArgs e) +99
   System.Web.UI.Control.LoadRecursive() +50
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +627

And that’s the stack trace. We see the site loading its controls and resources and the ‘Page_Load’ method is called at line 251. At line 1706 the system is apparently loading country-information which would be needed to set the proper language. Then it returns to line 1580 where it probably opens some table based on information from the language file.

Again, this is a lot of code for basically loading the main page. I even wonder why it needs to load data from the database based on the country information. Then again, I was about to fill in a questionnaire so it probably wanted to load the questionnaire in the proper language. If the questionnaire is multi-lingual then that would make sense.

Version Information: Microsoft .NET Framework Version:2.0.50727.3655; ASP.NET Version:2.0.50727.3658

And here’s one more bad thing. This site still uses .NET version 2.0 while the modern version is 4.5 and we’re close to version 5.0… It would not surprise me if these developers still use Visual Studio 2005 or 2008 for this all. If that’s the case then their budget for development is probably quite low. I wonder if the developers who are maintaining this site are even experts at software development. It’s not a lot of information that I can base this upon but in short:

  • The administrator did not prevent error messages to show up for users.
  • The use of ADO.NET adds vulnerabilities related to the connection pool and SQL injection.
  • The use of VB.NET is generally associated to less experienced developers.
  • The amount of code is quite long but common for sites that are developed years ago.
  • Not using a more modern framework makes the site more vulnerable.
  • Country information seems to be stored in XML while the questionnaire is most likely stored inside the database.
  • The .NET version has been out-of-date for a few years now.

My advice would be to just rewrite the whole site from scratch. Use the Entity Framework for the database and MVC 4 for the site itself. Rewrite it in C# and hire more professional developers to do the work.

TipsViaMail keeps spamming me…

On October 3, 2013 Adobe’s Database was hacked and about 150,000,000 Adobe users have their data exposed to a bunch of hackers. Anyone who even registered a single Adobe product , like I did,now has to deal with some extra spam in their mailbox. Unfortunately for the spammers, I used an alias that was used for just Adobe and after the hack, I provided them a new alias. As a result, any email on this old alias is now considered spam.

The hacked database was published and several companies have been datamining it to find their own users inside the database and to warn those users. In my case, only Adobe gave me a warning since only Adobe knew my alias. However, some companies are misusing the same database to pretend people have subscribed to their services and are sending spam to those people. And one of those companies calls itself TipsViaEmail.

First of all, if I did subscribe to their services, I would have used a different alias for them. Instead, they’re spamming me on my Adobe alias. Why? Not really sure but I guess they’re trying to make some profit this way.

Today I received a spam message from TipsViaEmail about some new way to chat with random people through Whatsapp. A bit like “Chat roulette” but on top of the Whatsapp engine. I’m not going to post the URL to this app because my virus scanner warned me about possible malware on their site. It seems extremely unreliable to me and is likely part of a trick to collect phone numbers, email addresses and perhaps even to infect mobile phones with malware. Don’t even try their stuff!

I think TipsViaEmail makes profit because they’re paid by these malicious companies to spam a lot of people. TipsViaEmail has a source of legitimate email accounts and claims these people subscribed to their service. So, people have to prove they never subscribed, which is difficult to do. How do you prove it? Well, I can because I have a habit of assigning aliases to every company I contact. And I can show how they got my address since they used my Adobe alias that was stolen by hackers.

They keep sending me emails once in a while but in low quantities so they won’t get a bad reputation with their providers. They send these spam messages through vistomail.com, which happens to offer ways to send email anonymously. Thus Vistomail is enabling spammers to send spam.

TipsViaEmail also allows a way to unsubscribe from their services by sending an email to an address at simpel-nieuwsbrief.com or by following a link at simpeltracking2.nl. In both cases, doing so would confirm your email address to TipsForEmail, making it profitable to sell to other spammers. They might stop spamming you, but those other spammers will start spamming you afterwards. At WhatCounts they calculated how much they could make by selling an email address and they earned about $17.34 per address! So we’re not talking about pennies when we’re talking about the value of email addresses.

And TipsViaEmail got their list for free because those hackers, who published the whole database!

So first, if you ever subscribed to an Adobe product then change your password immediately! Not just the password for your Adobe account but for all other accounts you have that used the same password! The passwords in the Adobe database were encrypted, but this encryption is being broken now so they will soon be exposed.

Next, find a way to use your own aliases with your mail provider. I did this by just getting my domain name, which costs me EUR 9.95 per year. I also use Google Apps so Google handles my email, even though it’s on my domain. And no, I don’t fear the NSA spying in my mailboxes. I just won’t send top-secret stuff by email anyways. It costs me another EUR 40,00 per year. But Microsoft Outlook and Yahoo Mail also offer similar services to connect your own domain to their email services. I just prefer Google since I think they have the best spam filter.

Finally, if you notice spam arriving at any alias, contact the company responsible for leaking your alias. (Adobe in my case.) They might not know their system has been hacked. And feel free to report the email to the proper channels. SpamCop is a good option internationally. (Do be aware that their URL ends at .net, since there are many copycats misusing their name!) For Dutch people you can report them too at SpamKlacht and people in Belgium can report spam to E-Cops.

(And don’t get fooled by spammers claiming you subscribed and who offer you an unsubscribe option. Unsubscribing will confirm your address, making it more valuable!)

Tricky spammer!

As usual, spammers trying to fool me and many others, and the best way to protect you against them is by sharing how they operate. (And by using a proper spam filter, which is part of Google mail. And today some message was in my spam folder which seemed to be legitimate. Well, okay… There was another hint telling me something wasn’t right. Multiple hints even.

Received: by with SMTP id o8csp50152igy;
        Thu, 5 Jun 2014 10:35:17 -0700 (PDT)
X-Received: by with SMTP id m18mr17979380wiw.49.1401989716698;
        Thu, 05 Jun 2014 10:35:16 -0700 (PDT)
Received: from sm1.white-lines.net (sm1.white-lines.net. [])
        by mx.google.com with ESMTP id cn1si16467631wib.60.2014.
        for <vip@watb.nl>;
        Thu, 05 Jun 2014 10:35:16 -0700 (PDT)
Received-SPF: pass (google.com: domain of  designates as permitted sender) client-ip=;
Received: by sm1.white-lines.net id hi2736000dsi for ; Thu, 5 Jun 2014 17:35:15 +0200 (envelope-from )
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
From: Security Team <security@security-fix-required.com>
Return-Path: bounce-
Subject: Your website has a security leak!
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101
Date: Thu, 05 Jun 2014 17:35:15 +0200


during a routine check, we discovered that the server hosting your domain h=
as a security leak and is currently vulnerable. Your website is at risk of =
being hacked! It's also running an outdated PHP version.

For further security details and secure managed server offers, please visit=
 our website:


Thank you,

Security Division
Managed Root Server

So, what did they do to make it seem legitimate? Well, it was a simple plain-text email with just a small amount of text. Apparently someone discovered a security leak in my website and is warning me about it. Since there are always white-hat hackers on the Internet who search for such things to warn the site owners, it could be legitimate. It even seems an automated message from an automated vulnerability scanner. So, it will probably fool a few people into clicking on the link in the email.

And that was the first thing that set me off. The domain name is a bit long and the URL ends with what seems to be a GUID or other identifier. If I would click on it, the site would confirm my address as legitimate and perhaps it would redirect me to some online advertisement or even a malware site. So, first lesson: If a URL has a weird number in it, it should be automatically suspicious!

Of course, the message doesn’t give me any information, just a warning. If they had detected something, they could have included a few more details. At least, they could have named the domain that they’ve checked. I have multiple domain names so this warning tells me nothing about the site.

They also mentioned a leak in an older version of PHP in my website, but my website doesn’t use PHP. I know this blog does, but this blog is hosted. It’s not on my server. And the host is making sure it stays safe with the latest updates. (At least, I hope they do but fortunately they have many other customers too.) If they had left away the remark about PHP, it might have looked more legitimate.

The fact that they don’t leave a name is reasonable, since hackers prefer to be anonymous. But hackers would use an alias instead, not some name of some server.

Of course, it also helped that this email ended up in my spam folder. Reporting spam thus helps protect others.If it had not been in my spam folder I would have reported it as spam myself, so Google would recognise it as spam in the future.

Some further analysis by using RobTex tells me the domain is very new. It was registered today, so probably not blacklisted yet. A Google search for the domain name is also interesting. These two should offer plenty of warnings about the site.

Of course, this wasn’t the only spam message, but it was the most tempting. Another message I received tried to sell me a specific kind of blue pills. A third one tempted me with some video but not only did Google detect it as spam, My virus scanner detected the URL inside the spam as potentially malicious. And Ruby Palace wants me as visitor, even though online gambling sites are illegal in the Netherlands if they target Dutch consumers. Since the email was in Dutch, one extra law was broken.

Again, the best weapon against spam is educating people about all the tricks spammers use and to make sure spam gets reported as such. If you use Yahoo mail, Windows Live email or Google mail, reporting spam as such should be a simple option.

Why social media aren’t happy with topless women in pictures…

People generally wonder why Facebook and Twitter seem to ban all forms of nudity, including the display of bare breasts. (Well, female breasts anyways.) Other sites have less troubles with displaying a bit of nudity. And people will always wonder why e.g. Facebook is that prudish. They even have troubles with pictures of women who are breastfeeding. Raevin WhiteBut on other sites they tend to have less troubles with the same type of content.

For example Tumblr has almost no restrictions to the material posted there, as long as it is legal to publish. On Twitter you’re allowed a bit more, like posting nipples in tweets. They won’t allow pornography, though. Many sites won’t, anyways. Still, there’s a good reason for this. The people who will join a specific site do so because of the generic content of the site.

Many social sites are aiming at teens and young adults and this means that the content needs to qualify to specific rules, especially if the site operates in the USA. For example, most people won’t be happy when their teens are visiting sites that has the occasional nude image. (Like this blog, for example.) They would block those sites, thus the site can’t target those teens with advertisements.

For Facebook, this would be a problem. Facebook has plenty of advertisements but also plenty of games that attract teens and young adults. They use Facebook to meet with friends, play games and whatever more. Thus, Facebook depends on this group of people and thus has to respond when people report “inappropriate material”. And because they have plenty of teens, they are extremely strict at that. Tumblr has less troubles with this. They make money from the bloggers themselves by offering premium services and premium themes. They also provide advertisements, although those are barely noticeable.

Tumblr doesn’t really target teens so the content can include nudity and even pornography. Because of that, it’s no surprise that you can find plenty of those on Tumblr.

And WordPress? Well, WordPress is available in several versions. You can host it on your own server, you can have it hosted by a service provider or you do as I do and let it be hosted by WordPress themselves. The hosted versions might be a bit more strict because the hosting provider has a reputation to keep up. Worse, since the blogger is paying the provider, the provider might prefer to have less visitors instead of many, to save bandwidth. Nude pictures are often large amounts of data and with many visitors the provider loses bandwidth.

Self-hosted WordPress sites have no restrictions, though. The worst thing that could happen is that police will confiscate your hardware and arrest you if you happened to host some illegal content.

So, one main reason to block nudity is because people don’t want their teen children being exposed to it. (While plenty of teens might actually be specifically looking for this material and might even exchange nude selfies with friends.) Social sites will have to know the type of visitors they generally have and adjust their content to those visitors.

At SecondLife, for example, the rules for content within the game were mostly quite relaxed. People were allowed a lot in their own lands, as long as it was marked as mature or adult. But SecondLife got into troubles after it was discovered that many underage teens would play the game too. And those teens were suddenly exposed to nudity, sex and a lot of other things. So they decided to create a separate version just for teens and kicked every teen from the adult world to the “nicer” teen world. And if new teens are discovered in the adult world, they too are kicked to the “kindergarten”.

And they banned most of the adult stuff from most areas except for the adult areas. Since you have to pay a lot to have an adult area, this meant that many people just left the game. SecondLife now has some competition because some developers started to create the OpenSimulator where people could just host their own second world on their own system.

This became even more complex after some groups started to combine forces and started hosting virtual world similar to SecondLife, but for much less money or even free. Because of this and the ban on adult material, SecondLife has lost a lot of people.

There are, of course, more reasons. Sites that want to have viewers in e.g. China need to be aware of the restrictions the Chinese government puts on content. No pornography and preferably no bashing of the government itself. Sites focussing on the USA might also block pornography because there are a lot of people in the USA whose religious views are against such images.

In the UK they’re even demanding that providers just block all pornography and adult sites, which led to plenty of protests because too much was blocked. So, sites who want to target citizens in the UK better clean themselves up so they will get past those (faulty) porn filters.

Again, Facebook belongs to those, thus they definitely want to stay clean. Basically, social sites have to choose between those who claim there’s too much nudity versus those who want to have more nudity. Some want more, others want less. And social sites just tend to listen to those who have the most power. Not the majority but those who have the biggest influence. And those would be the lawmakers.

For example, mentioning the Tiananmen Square protests of 1989 will likely get you banned in China. Not practical if you want to trade with people in China. Facebook has similar problems but all over the world. In too many countries the law puts some very strict restrictions on nudity. The USA and UK aren’t even the worst of them.

Facebook is also popular among Arab people, India and plenty of other cultures that frown upon female nipples. They want advertisers everywhere to pay them so they make a lot of profit and thus they have to give in to the demands of those lawmakers. Fortunately they also want to be in Europe so they can’t be too strict on their content, but still…

Nipples are banned because it might offend advertisers in certain areas. That would even apply to pictures of women breastfeeding their child. Male nipples are generally less offensive, though. So yes, there’s discrimination in the Facebook policies. But giving in to the demand to allow more nudity would cost them some of their advertisers, thus some of their revenue. It would only be worth their trouble if people would ban Facebook because of this strict policy.

Unfortunately, no advertiser is blocking Facebook because they don’t show enough nipples. And that’s why social media block nipples…

MtGox is close to bankrupt.

TodaY I received a PDF file called “Announcement of Commencement of Bankruptcy Proceedings_212014” And basically, it tells me that MtGox, a bitcoin market, is definitely going bankrupt. But that was to be expected. I have less than a single euro in bitcoins at MtGox I have no regrets for trying out their service. But plenty of other people have made big investments in bitcoins and stored them at MtGox. Chances are that they will have lost it all, since MtGox has plenty of bills it needs to pay first.

To make it more complex, its unclear if bitcoins can be considered equal to money or not. They’re just a collection of bytes in a specific order and format and they’re worth exactly what people are willing to pay for them. It will be interesting to see what the Japanese court system will think of the value of bitcoins. People might still get their bitcoins if the Liquidator thinks they’re worthless. But if the system in Japan is similar to the Netherlands, that Liquidator could just auction off all bitcoins that MtGox still have to pay off the debts. The remaining cash would then be compensation for anyone who had their bitcoins stored there.

Of course, plenty of other countries (the USA and UK) are probably willing to dig into the action and try to get some financial compensation too. Plenty of American people have lost a lot of money because of this. But the Japanese government goes first and all others have to pick the remaining bones. And I don’t think there will be a lot of meat left on those bones…

The lesson learned from this is, of course, that bitcoins aren’t that safe. Especially if you have them stored at some bitcoin site as MtGox. You are losing control over your money and considering how much bitcoins have been worth in the past, being careless with them can cause a big financial blow. Then again, people can also lose bitcoins if they store them on their own systems. Bitcoins on your phone can get lost if your phone is stolen or damaged. Bitcoins on your computer are always at risk of getting wiped away. I’ve heard of one guy who threw away his old laptop and later learned that he had a few thousands of bitcoins on it, each worth over $1,000 in cash! A very expensive mistake, although he had mined them himself so he did not really lose money. He just made no profits from the mining.

So, please consider what you’re doing when you will use some crypto-money like bitcoins. Make sure you’re well-informed and don’t buy them in large quantities if you just want to save your money somehow. It’s better to just start mining them yourself so your losses can be under control.

And yes, banks can go bankrupt too, but crypto-currency is a bit more riskier since there’s no proof to tell that you really owned them. Once they’re gone, you won’t get them back. This is still something that you should leave to true pioneers who are willing to take risks.

The email itself:



なお、このメールアドレス(mtgox_trustee@noandt.com )は破産管財人からの送信専用であり、貴殿が本メールアドレス宛の返信等をされても内容確認及び回答などの対応はできません。
破産手続の進行等については、ウェブサイト( http://www.mtgox.com/ )で情報提供をする予定ですので、当該ウェブサイトをご確認ください。

破産者株式会社MTGOX  破産管財人弁護士小林信明

To whom it may concern,

At 5:00 p.m. on April 24, 2014, the Tokyo District Court granted the order for the commencement of the bankruptcy proceedings vis-à-vis MtGox Co., Ltd. (“MtGox”), and based upon such order, I was appointed as the bankruptcy trustee (Tokyo District Court 2014 (fu) no. 3830).
The bankruptcy trustee will implement the bankruptcy proceedings, including the administration and realization of the assets and investigation of the claims.
For the purpose of providing information to the related parties, we hereby inform you of the basic matters regarding the bankruptcy proceedings as attached.

This email address(mtgox_trustee@noandt.com) is used only for the purpose of sending messages, and we are unable to check and respond to any replies to this email address.
Since we plan to provide the information regarding the bankruptcy proceedings by posting it on the website hosted by the bankruptcy trustee ( http://www.mtgox.com/ ), please check this website.

Bankrupt MtGox Co., Ltd. Bankruptcy trustee Attorney-at-law Nobuaki Kobayashi

Betaalverzoek inzake CJIB

Once more some stupid spammer trying to get people to pay them lots of money. It was sent to my sister who could not understand how she had to pay so she asked me how. I quickly discovered that this is a big scam and told her so. And I’m posting it here to warn other people about this scam too and how scammers try new tricks every time hoping for the suckers who are scared enough to pay.

Since this scam was written in Dutch, I will continue in the Dutch language.


Mijn zus ontving vandaag deze email van het “CJIB” betreffende een verkeersboete van 155 euro. Het dreigt ermee dat haar bankrekening wordt geblokkeerd met ingang van 13 mei, wat dus al gebeurd zou zijn. Ze moet voor 19 mei betalen, dus op de dag dat ze de email ontving. En ja, dat is de manier waarop spammers proberen om hun slachtoffers mee onder druk te zetten zodat ze betalen zonder na te denken.

Wat belangrijk is, is hoe de spammers aanwijzingen geven om een prepaid credit card aan te schaffen om zo de boete mee te betalen. Vervolgens moet je naar een site toe, waar geeneens een domeinnaam aan hangt. Het is een URL met IP adres en daarbinnen een folder. Daar zie je vervolgend een vrij kaal scherm met een betaalknop.

Clip_2Clip_3Clip_5Klik je vervolgens verder dan krijg ik met Google Chrome al een waarschuwing dat de site is geblokkeerd wegens phishing. Ik neem even het risico en kom bij het volgende plaatje. Daar moet de 3B pincode worden ingevuld, waarna de oplichter de gehele creditcard kan leeghalen. Wie uiteindelijk een 19-cijferig nummer invoert krijgt vervolgens een pagina te zien die aangeeft dat de betaling succesvol was (terwijl ik een willekeurig nummer gebruikte) en ik zal binnen drie tot 5 dagen bericht krijgen van de belastingdienst.


Het bedrag van 155 euro komt mooi overeen met de hoogste waarde van de betreffende maatschappij. Gelukkig hebben ze al door dat er dergelijke nepmails over het Internet gaan zodat iedereen op Beltegoed Opwaarderen daar nog eens de waarschuwing over deze oplichterij te zien krijgt.


Jammer dat de waarschuwing onder de betaalknoppen staat en niet erboven, waar ze nog beter opvallen. Maar iedereen zou dit toch als een waarschuwing moeten zien. Hopelijk is het duidelijk genoeg maar er zullen altijd mensen zijn die in dit soort oplichterij trappen.

Hoe komt het dat er zoveel mensen in trappen? Dat is heel simpel. Dergelijke berichten worden vaak naar grote aantallen adressen verstuurd. Als 1% van de bevolking er in trapt en ze versturen het naar 100.000 adressen dan zijn dat toch al weer 1.000 slachtoffers. En dat maal 150 euro maakt het een winstgevende actie, maar wel illegaal. Gelukkig is het percentage slachtoffers nog veel lager dan 1% maar al zijn er 10 slachtoffers in die grote groep, het geld komt dan wel binnen met relatief weinig moeite.

Hoe kun je je wapenen tegen deze oplichters? Eigenlijk moet je daarvoor gewoon goed opletten en goed weten hoe bepaalde bedrijven en organisaties werken. Het CJIB zal echt niet via prepaid creditcards betaald willen worden. Het CJIB zal sowieso nooit via het Internet boetes proberen te innen.

Dergelijke constructies zijn vooral bedoeld om geld weg te sluizen zodat het slachtoffer er niet meer bij komt. Je bent het geld gewoon kwijt zodra je op deze manier hebt betaald. Ook de creditcard maatschappij kan het niet terugkrijgen omdat ze het beltegoed erop gebruiken om bijvoorbeeld een duur 06-nummer mee te bellen. Dan is de creditcard leeg en ligt het geld bij een telefoon maatschappij die het weer moet doorbetalen aan een bel-bedrijf. En van daar gaat het geld weer verder weg van het slachtoffer.

Wat ook van belang is, is dat de site nergens om mijn persoonlijke gegevens vraagt. Deze staan zelfs niet in de email. Het is gericht aan de bestuurder, zonder zelfs een nummer van een kentekenplaat te vermelden. Dat kunnen de oplichters ook niet want ze hebben deze gegevens niet. Als iemand een rekening per email verstuurt dan zou je toch meer gegevens in de email verwachten. Het gebrek aan deze persoonlijke gegevens is ook een waarschuwing.

Wie technisch iets handiger is kan ook nog eens naar de ‘headers’ van de email kijken om te bepalen waar de email vandaan komt. En dan blijkt dat de email afkomstig is van hetzelfde IP adres als de site zelf. Een adres dat ergens in Japan te vinden is. Mogelijk een Japanse computer die onderdeel is geworden van een botnet en dus misbruikt wordt zonder dat de eigenaar dit beseft. Om de oplichter te vinden is dit dus geen behulpzame manier. Daarvoor zul je het geld moeten volgen…

Maar sowieso moet je altijd oppassen met verzoeken tot betalen per email. Eigenlijk zou je dat standaard moeten weigeren, tenzij je zeker bent dat het iets betreft dat je nog moet betalen.

Nu nog even de volledige email zoals deze is ontvangen via de hotmail account van mijn zuster:

Authentication-Results: hotmail.com; spf=none (sender IP is smtp.mailfrom=cjibnoreply@cjib.nl; dkim=none header.d=cjib.nl; x-hmca=none header.id=cjibnoreply@cjib.nl
X-SID-PRA: cjibnoreply@cjib.nl
X-SID-Result: NONE
X-Message-Status: s1:n
X-Message-Delivery: Vj0xLjE7dXM9MDtsPTA7YT0wO0Q9MjtHRD0yO1NDTD02
X-Message-Info: OR3oMfwJnYHF1wanhF69C9Yey20TK9h7x9GWXuv5yaEGAfYu81s5sUj6V3GqMLsbaFOGIxV4jNuK1YTPnnwB8khYxF5czLKOeqtp5CEeiwA6KP8+eQfiSR4aZ+C9AR+10UtHFivL+rY5J1BgXCW7aHs
Received: from vps1.cpanel.net ([]) by BAY0-MC6-F21.Bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4900);
Fri, 16 May 2014 18:16:02 -0700
Received: from [] (port=27929 helo=newran)
by vps1.cpanel.net with esmtpa (Exim 4.82)
(envelope-from <cjibnoreply@cjib.nl>)
id 1WlTE6-0002gc-Bo; Sat, 17 May 2014 10:15:51 +0900
Reply-To: <noreply@cjib.nl>
From: “Centraal Justitieel Incassobureau”<cjibnoreply@cjib.nl>
Subject: Betaalverzoek inzake CJIB
Date: Sat, 17 May 2014 03:15:51 +0200
MIME-Version: 1.0
Content-Type: multipart/related;
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname – vps1.cpanel.net
X-AntiAbuse: Original Domain – hotmail.com
X-AntiAbuse: Originator/Caller UID/GID – [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain – cjib.nl
X-Get-Message-Sender-Via: vps1.cpanel.net: authenticated_id: newran/only user confirmed/virtual account not confirmed
Return-Path: cjibnoreply@cjib.nl
Message-ID: <BAY0-MC6-F21LjANJQ000b8ac21@BAY0-MC6-F21.Bay0.hotmail.com>
X-OriginalArrivalTime: 17 May 2014 01:16:02.0669 (UTC) FILETIME=[91B0C9D0:01CF716D]

This is a multi-part message in MIME format.

Content-Type: text/html;
Content-Transfer-Encoding: 7bit

<BODY bgcolor=#FFFFFF leftmargin=5 topmargin=5 rightmargin=5 bottommargin=5>
<FONT size=2 color=#000000 face=”Arial”>
<IMG align=middle border=0 width=400 height=69 src=”cid:00E9BAC800C5$03195E81$0100007f@uhxyhwczmgwjdgc”></DIV>
<DIV align=center>
<DIV align=center>
Geachte bestuurder,</DIV>
<DIV align=center>
U hebt een beschikking en vervolgens twee aanmaningen ontvangen voor het overtreden van een verkeersvoorschrift.</DIV>
Het openstaande bedrag is niet volledig op de rekening van het Centraal Justitieel Incassobureau (CJIB) bijgeschreven.</DIV>
Daarom zullen wij de bank opdracht gegeven uw rekening te blokkeren per dinsdag 13 mei 2014.</DIV>
Alleen persoonlijk bij het BKR zelf kunt u inzage krijgen in de informatie die het BKR over u ontvangt.</DIV>
Het blokkeren van rekening betekent dat de toegang tot uw rekening geblokkkeerd is met ingang 13-05-2014 voor een periode van vier werken.</DIV>
Met de 3v online krediet kunt u online op onze website de betaling voldoen. U dient hieronder te klikken op<B><I> </B></I><I>3v credit kopen</I> .</DIV>
<B> </B></DIV>
<A href=”http://beltegoedopwaarderen.nl/3v”><FONT color=#0000FF><B><U>3v</B></U></FONT></A><A href=”http://beltegoedopwaarderen.nl/3v”><FONT color=#0000FF><B><U> credit
<B> </B></DIV>
Let op: nadat uw de 3v (prepaid credit) heeft gekocht dient u de 19 cijferige nummercode hieronder te activeren om de betaling te voldoen.</DIV>
Klik hieronder op <I>aanmaning betalen</I><B><I>.</B></I></DIV>
<A href=””><FONT color=#0000FF><B><U>Aanmaning betalen</B></U></FONT></A></DIV>
Het volledige bedrag van Eur 155,00 (inclusief kosten) moet uiterlijk 19-05-2013 worden betaald. Doet u dit niet, dan wordt u per 19-05-2014 geregisteerd bij BKR.</DIV>
Voorkom blokkade van uw rekening.</DIV>
<B> </B></DIV>
<B> </B></DIV>
<IMG align=middle border=0 width=120 height=60 src=”cid:00C18EFDDDDC$00C87F7D$0100007f@uhxyhwczmgwjdgc”></DIV>
Centraal Justitieel Incassobureau.</DIV>
<DIV align=center>
<DIV align=center>
<DIV align=center>

Content-Type: image/jpeg;
Content-Transfer-Encoding: base64
Content-ID: <00C18EFDDDDC$00C87F7D$0100007f@uhxyhwczmgwjdgc>

[SNIP – Some UUEncoded data]

Content-Type: image/jpeg;
Content-Transfer-Encoding: base64
Content-ID: <00E9BAC800C5$03195E81$0100007f@uhxyhwczmgwjdgc>

[SNIP – Some UUEncoded data]



Multithreading, multi-troubling.

Recently, I worked on a small project that needed to make a catalog of image files and folders on my hard disk and save this catalog in a database. Since my CGI and my photography hobby generated a lot of images, it would be practical to have something easy to support it all. Plenty of software that already does something like this, but none that I liked. Especially since I want to connect images to derived images, group them, tag them, share them, assign licenses to them and publish them. And I want to keep track of where I’ve shared them already. Are they on Flickr? CafePress? DeviantArt? Plus, I wanted to know if they should be rated as adult. Some of my CGI artwork is naughty by nature (because nude models are easier to work with) and thus unsuitable for a broad audience.

But for this simple catalog I just wanted to store the image folder, the image filename, an image name that would be the filename without extension and without diacritics, plus the width and height of the image so I could calculate the image ratio. To make it slightly more complex, the folder name would be a relative folder name based on a root folder that’s set in the configuration. This would allow me to move the images to a different folder or use the same database on a different machine without the need to adjust all records.

So, the database structure is simple. One table that has the folders, one table containing image ratios and one for the image names and sizes. The ratio table will help me to group images based on the ratio between width and height. The folder table would do the same for grouping by folder. The Entity Framework would help to connect to this database and take away a lot of my troubles. All I have to do now is write a simple library that would fill and keep up this catalog plus a console application to call those methods. Sounds simple enough.

Within 30 minutes, the first version was ready. I would first enumerate all folders below the source folder, then for each folder in that list I would collect all image files of type PNG, JPG and BMP. The folder would be written to the folder table and the file would be put in the Image table. Just one minor challenge, though…

I want to add the width and height of the image to the image table too, and based on the ratio between width and height, I would have to either add a new ratio record, or change an existing one. And this meant that I had to read every file into memory to find its size and then look if there’s already a ratio record related to it. If not, I would need to add the new ratio record and make sure the next request for ratio records would now include the new ratio record. Plus, I needed to check if the image and folder records also exist in the database, because this tool needs to update only for new images.

The performance was horrible, as could easily be predicted. Especially since I make images and photo’s at high resolutions, so reading those files does take dozens of milliseconds. No matter that my six cores at 3.5 GHz and 32 GB of RAM turns my system in a Speed Demon, these read actions are just slow. And I did it inefficiently since I have six cores but my code is just single-threaded. So, redo from start and this time do it multithreaded.

But multithreading and the Entity Framework don’t go well together. The database connection isn’t threadsafe and thus you cannot access the database methods from multiple threads. Besides, the ratio table could generate collisions when two images with the same, new ratio are processed. Both threads would notice the ratio doesn’t exist thus both would add it. But one of those would then fail because the other would have added it first. So I needed to change my approach.

So I Used ‘Parallel.ForEach’ to walk through the folder list and then again for all files within the folder. I would collect the data in internal lists and when the file loop was done, I would loop through all images and add those that didn’t exist. And yes, that improved performance a lot and kept the conflicts with the ratio table away. Too bad I was still reading all images but that was not a big issue.Performance went up from hours to slightly over one hour. Still slow.

So one more addition. I would first read all existing folders and images from the database and if a file existed in this list, I would not read it’s size anymore since it wasn’t needed. I could skip the image. As a result, it still took an hour the first time I imported all images, but the second run would finish within a minute, since there wasn’t anything left to read or add. The speed was limited to just reading the files and folders from the database and from the disk.

When you’re operating these kinds of projects in an Agile team and you’re scrumming around, things will slow down considerably if you haven’t thought about these challenges before you started the sprint to create the code. Since the first version looks quite simple, you might have planned it as a very short task and thus end up with extremely slow code. In the next sprint you would have to consider options to speed things up and thus you will realize that making it multithreaded is a bigger task. And while you are working on the multithreaded version, you might discover the conflicts with the Entity Framework plus the possible collisions within the tables. So the second sprint might end with a buggy but faster solution with lots of exception handling to catch all possible problems. The third sprint would then fix these, if you manage to find a better solution. Else, this problem might haunt you to the deadline of the project…

And this is where teams have to be real careful. The task sounds very simple, but it’s not. These things are easily underestimated by a team and should be well-planned before you start writing code. Experienced developers will detect these problems before they start, thus knowing that they should take their time and plan carefully without writing code immediately. (I only did it so I could write this post.) The task seems extremely simple and I managed to describe it in the second paragraph of this post with just three lines. But the solution with a high performance will require me to think before I start writing code.

My last approach is the most promising, though. And it can be done by using multithreading but it’s far more complex than you’d assume at first. And it will be memory-hungry because you need to create several lists in memory.

You would have to start with two threads. One thread will read the database and generate lists of files, folders and ratios. These lists must be completely in-memory because if you keep them as queryable lists, the system would try to continuously read them. Besides, once you’re done generating these lists you will want to close the database connection. This all tells you what you already have. The second thread will read all folders and by using parallel threads it would have to read all image files within those folders. But you would not read the image sizes yet, nor calculate all ratios.

When you’re done collecting the data, you will have to compare it all. You would start by comparing the lists of folders. Folders that exist in both lists can be ignored (but not their files.) Folders that exist in the database list but not the disk list should be deleted, including all files within those folders! Folders that are on disk but not in the database need to be added. Thus you can now start two threads, each with their own database connection. One will delete all folders plus their related images from the database that have been deleted while the other adds all new folders that are found on the disk. And by using two database connections, you can speed things up. You will have to wait for both threads to finish, though. But it shouldn’t be slow.

The next step would be the comparison of images. Here you do something similar as with folders. You split the lists in three different lists. One with all images that are unchanged. One with all images that need to be deleted. And one with all images that need to be added. And you would create a separate thread with its own database connection to delete the images so your main process can start working on the ratios table.

Because we now know which images need to be added, we can go through those files using parallel processing, read the image width and height and add this information to the image file records. When we have enriched this list with these sizes, we can use a LINQ query to generate a list of all ratios of those images and removing all duplicate ratios in this list. This generates the list of ratios that we would need to check.

Before we add the new images, we will have to check the ratios table. As with the folders table, we check for all differences. However, we cannot delete ratios that we haven’t found among the images, because we skipped the images that already exist. We will do this later, though. We will first start adding the new ratios to the database. This too can be done in a separate thread but it’s pretty fast anyways so why bother? A performance gain of two seconds isn’t worth the extra effort if a process takes minutes to finish. So add the new ratios.

Once all ratios are added, we can add all images. We could do this using parallel threads, with each thread creating a new database connection and processing all images from one specific folder or with one specific ratio. But if you want to add them multi-threaded I would just recommend to divide the images in groups of similar sizes. Keep the amount of groups relative to the number of processes (e.g. 24 for my six cores) and let the system do its work. By evenly dividing the images over multiple threads, they should all take about the same amount of time.

When adding the new images, you will have to find the related folder and ratio in the database again. This makes adding images slower than adding folders or ratios because you need the extra lookup. This performance would increase if we had kept the Folders and Ratio lists as queryable lists but then we could not open and close the connections, not could we use multiple connections to add those images. And we want multiple connections to speed things up. So we accept a slightly worse performance at this point, although we could probably speed it up a bit by using a stored procedure to add the images. The stored procedure would have parameters for the image name, the image filename, the width and height, the folder name and the ratio width and height. I’m not too fond of procedures with many parameters and I haven’t tested if this would increase the performance, but in theory it should be faster, especially if the database is on a different machine than the application.

And thus a simple task of adding images to a database turns out to be complex, simply because we need better performance. It would still take hours if it has a lot of new images to add but once you have it mostly filled, it will do quite well.

But you will have to ask yourself and your team if you are capable to detect these problems before you start a new sprint. Designs are simple, because designers don’t always keep the performance in mind. These things are easily asked for because they appear very simple, but have a lot of consequences. Similar problems might arise when you work with projects that need to be secure. The design might ask for a login screen with username and password, and optionally a few OpenID providers as alternative logins, but the amount of code to manage all this data and keep it secure is quite complex. These are real moments when you need to design some technical documentation first, which is something people often forget when working on an Agile project.

Still, you cannot blame the developer if the designer just writes a few lines and the developer chooses the first, slow solution. The result would be the requested task. It is the designer who needs to be aware of these possible performance pitfalls. And with Agile, you have a team. All team members should be able to point out that this simple description would have these pitfalls, thus making it a long and complex task. They should all realise that they will have to discuss possible solutions for this and preferably they do so as a team with just one computer. (The computer would be used to find information, not to write code!) Only when they agree on the proper solution then one or two of them could start writing code. And they would know how long this task will take. Thus, the task would finish within two sprints. In the first sprint, all team members would have a small task to meet and discuss the options. In the second sprint, one or more members would have a big task of implementing the code.

Or, to keep it simple: think before you start writing code!